Dear authors, 

Please find attached my review of your manuscript "Towards robust fracture mapping: benchmarking automatic fracture mapping in 2D outcrop imagery". The study utilises a training dataset comprising of three published datasets to assess a range of traditional and deep-learning models to achieve fracture network interpretations. This is an area of growth in the community, and a systematic comparison of methods would be of use to those using, or assessing output from, these models. Whilst i found the results were well handled and provided a robust statistical approach to comparing datasets, the manuscript was difficult to follow and the research cap and implications poorly laid out. In particular i found the level of references to be very limited, with a wealth of literature in this, and the wider fracture network characterisation fields not called upon. This made it difficult to see where the work fit in, and how the results apply to fracture analysis more generally. As such, at present, i have to suggest that the manuscript is rejected due to the number of changes required to make it acceptable for publication. Although i am sure this is disappointing, I hope my review is helpful and i look forward to seeing a revised version of this work in the future. 

Please see below the attached annotated manuscript with my specific comments. 

Kind regards, 

Dr. Billy Andrews

The article "Towards robust fracture mapping: benchmarking automatic fracture mapping in 2D outcrop imagery" addresses a key element in collating quantitative structural geology : fracture trace extraction.

In this contribution, the authors compiled a benchmark of three published empirical datasets named FraXet which combine colour RGB orthoimages, corresponding digital elevation models (DEM) and human-interpreted fracture traces. FraXet was processed in a coherent fashion with traditional image edge detection filters on the one hand, and with two deep learning approaches on the other hand in order to reproduce the manual fracture traces. 

To maximize transposabiity of their findings, effort was made to render image contents comparable by standardizing image digital counts to zero mean and unit variance over patches of 256x256 pixels. Issues like departure between annotated fracture trace serving as ground truth and predicted fracture location is effectively taken into account in the procedure. I found this particlarly relevant as reference traces are never easy to draw and are subject of debate. A  reference fracture data sets is always faulty in some way given locally ill-defined surface geometry or weak radiometric contrast. In accounting for this effect, the authors are not trying to hide this difficulty and provide a clever means of mitigating the issue.

The principle and merits of each processing are well presented. The methodology is clear and well explained even for those unfamiliar with deep learning methods nitty gritties.  I enjoyed reading this work and feel more informed now than before.

Coming to manuscript improvements, I found that an object definition was lacking. Talking about "fractures" and their occurrence in rock faces may seem obvious and common-sense. Yet, a clear definition of them, particularly in term of their appearance in images and DEM would improve the purpose of the work and imply a clearer fitness-for-purpose evaluation. A descriptive paragraph explaining which raster properties fractures do display in images and DEM would be useful. Underlying my remark is the need to recall that a fracture trace is the piece-wise linear pattern (sometimes very shortly linear) drawn by the intersection of a possibly, but not always, planar rock surface discontinuity with the topography. Both surfaces needn't be absolutely planar and the projection of their intersection on an arbitrary plane (that of the orthomosaic) will affect their final appearance geometry.

Not only is the shape of a fracture traces not strictly linear, but they also have different appearances dependant on scale. Fracture lengths (and widths) cover a very broad set of sizes. Currently the text addresses this size definition question by talking about raster patches of 256 x 256 pixels. An actual metric dimension is lacking to relate image space to object space. Table 3 is not quite sufficient to grasp the descriptive scale of each data set.

Further, the appearance of fractures in an image is a function of the fracture set orientation with respect to the outcrop surface and with respect to the orthorectification plane. I perhaps missed this information but representing the domain of relative orientation might be a good way to make your paper last through time. Learning has occurred on three ortho mosaics cross-cutting fracture sets at such range of angle only. One guesses that the fracture sets are at steep angle with the projection plane so that traces are mostly narrow, but specifying parameter range explictly may help further down the line when the benchmark data will be used for other learning experiments.

To summarize, this paper is a very valuable contribution to 2D image-based structural geology analysis. 

Implementation for the broader public through a QGIS plugin would certainly expand the user base and maximize this contribution. In there, offering a function for extracting segment lengths and intersection types for apparent fracture persistence and hierarchical connection would be excellent.

Specific comments

line 135 about wide fracture expansion. The maximum filtering is clever but :

    * what neighbourhood did it consider for the kernel? 

    * what threshold qualifies as "dark pixels" ?

    * what justifies filtering only the green image band ? Wouldn't one of the principal component analysis components fit the purpose of capturing the most contrasted image better? Is it implictly evoking the Bayer matrix of digital cameras where there are twice more green-sensitive pixels than reds and blues? It wouldn't hurt to make an explanation sentence. But my processing preference would go for the richest PCA band.

line 138+ about the multi-scale label smoothing. Again, I really liked this clever approach, but:

    * what kernel size did you use for buffering the reference fracture trace? 

    * how does this kernel size relate to the digitization scale (or actual metric distance)?

    * does a single kernel size apply to all data sets irrespective of the operator and the metric size of the images?

A single kernel size seems inappropriate given the different scale of images and objects.

line 145+ data augmentation

This is a classic approach in deep learning to augment the domain of variation of the source images. 

How do you justify the gaussian blur dimension implemented? This is a simulation of out-of-focus photographs. How realistic/unrealistic is it regarding your practice of outcrop photography? Is the smoothing kernel size in range with encoutered out of focus image portions? Wouldn't it be wise to also implement the second source of image blur : motion blur? The augmentation set fails to capture motion blur, arising from slow shutter speed. This image defect will probably enter in resonnance with fracture orientation. Fracture-normal motion blur will broaden the fracture trace and reduce local contrast. Fracture-colinear motion blur will likely not affect the appearance of the fracture. 

In turn, motion blur will affect both fracture trace location and sharpness (or probability, since you introduced this notion as metric accounting for prediction precision). A comment on the matter will suffice, no need to re-run the learning process.

How do you justify the over/underexposure values applied? Are they arbitrary? Do they correspond to one f-stop of under/over-exposure relating to possible camera's on-the-fly mis-calculation? Could you say why and when this situation should occur in field photographs?

line 158 you suggest that only one channel was considered for traditional image processing filters. Which PCA channel? The text is not explicit. Is it always the first component that contains the best fracture signal?