Hyperspectral data capturing the three drill holes (KSL133, KSL131, and KSL136) were acquired using a SPECIM SiSuRock drill core scanner, which is equipped with three hyperspectral cameras. These sensors capture a broad range of hyperspectral data across different spectral regions: with the AisaFENIX sensor capturing 450 bands in the visible and near-infrared to short-wave infrared (VNIR–SWIR; 380–2500 nm) with an average spectral sampling resolution of 3.5 nm for the VNIR and 5.5 nm for the SWIR, the SPECIM FX50 sensor capturing 308 bands in the mid-wave infrared (MWIR; 2700–5300 nm) with an average sampling resolution of 8.4 nm, and the AisaOWL sensor capturing 103 bands in the long-wave infrared (LWIR; 7700–12300 nm) with an average sampling resolution of 45 nm. The extensive spectral coverage captures spectral features that are diagnostic of most common minerals, including those expected at Spremberg (quartz, anhydrite, carbonates, clays, and feldspars) . An in-depth review of the data acquisition process, as well as the details of the Kupferschiefer hyperspectral dataset, can be found in .

Sonic logger data were acquired using the USBA-21 acoustic logger that measures the travel time of acoustic waves between two magnetostrictive transducers (sources) and a piezoelectric receiver. Two sources are used to differentiate between formation and mud effects (since the travel time through the mud stays constant, ). This travel time, recorded in microseconds per metre (a measure of slowness), is given by 1Δt=tfar-tnear, where tfar is the travel time from the farther transducer, and tnear is the travel time from the nearer transducer.

For density logging, a Century-9036 density logging tool was used. The radioactive source, typically a directional Cs137 emitter, emits medium-energy gamma rays into the formation. These gamma rays undergo Compton scattering upon interacting with electrons in the formation, and the scattered gamma rays reaching the detector provide a measure related to the formation's electron density. This electron density (ρe) is then associated with the formation's bulk density (ρb) using the following relationship: 2ρe=2ρbZA, where Z is the atomic number, and A is the molecular weight of the compound. The electron density (ρe) in gcm-3 determines the response of the density tool .

Gamma-ray logging quantifies gamma radiation levels in the formation, which is influenced by uranium (U), thorium (Th), and potassium (K) concentrations. The gamma-ray intensity in American Petroleum Institute (API) units is derived from these concentrations as follows: GR(API)=8×U(ppm)+4×Th(ppm)3+16×K(wt%).

This reading allows for comprehensive characterisation of radioactive content in the formation, providing insights into its mineralogical composition .

Given the varying resolutions and sensor positions inherent in the different geophysical logging tools used to measure slowness, density, and gamma-ray values, it was necessary to resample them to derive measurements over comparable depth ranges and ensure comparability. Given that the lowest-resolution data were from the slowness log, with a sampling of 10 cm, we employed scipy's RBFInterpolator to downsample the logs. Specifically, a thin-plate spline kernel and a different number of nearest neighbours for the slowness, density (with a sampling resolution of 2 cm), and gamma-ray (with a sampling resolution of 5 cm) logs – 10, 50, and 20 nearest neighbours respectively (to make sure that the truncation distance for the interpolator was the same for each property) – were used, to downsample the logs and match the resolutions while minimising potential interpolation bias and ensuring consistent alignment across each sensor.

Finally, a rolling standard deviation of the measured petrophysical properties was computed using a 1 km window (Fig. ). Due to factors like core loss, co-registration errors are expected between the petrophysical logs and the drill core boxes. To ensure our training dataset does not contain spectra paired with incorrect petrophysical properties, we use the rolling standard deviation to eliminate points from regions of high property variance, as these will be highly sensitive to co-registration uncertainties. Hence the underlying assumption within our preprocessing steps is that by picking points only from petrophysically homogeneous regions of the drill cores, we can partially mitigate challenges caused by co-registration errors. The hyperspectral data also needed to be downsampled by 2 orders of magnitude to match the spatial resolution of the petrophysical measurements, given their ≈1 mm² spatial resolution (2 orders of magnitude higher than the downhole geophysical logs). We computed the median spectra within a defined window of 10cm×5cm (equivalent to 100pixel×50pixel) centred along the core. The results represent the (linearly mixed) spectral response of the rock volume sampled by the petrophysical loggers, with some assumptions (e.g. on the representativity of the scanned core surface).

In addition to the standard white-reference normalisation, lens correction and sensor co-registration described by , the hyperspectral data were further preprocessed to (1) remove noisy bands, (2) reduce illumination effects that result from the 3D geometry of the scanned drill cores (e.g. shadowing), and (3) enhance mineralogically significant absorption features (Fig. ).

Specifically, the VNIR–SWIR data were subset to the range between 500 and 2500 nm, while the first and last 10 bands of the MWIR and LWIR data were also removed. A hull correction was then applied to the spectra from each sensor separately, using hylite , which simultaneously amplifies spectral absorption features and suppresses confounding variability associated with drill core shape and associated inhomogeneous illumination.

The petrophysical and associated hyperspectral data form distinct clusters, as clearly shown when they are plotted in the petrophysical property space (Fig. a). After using the rolling standard deviation to remove high-variance points and associated mixed measurements (Fig. b), these clusters are even more distinct. Importantly, the cluster sizes are unbalanced due to different thicknesses of the lithologies in the boreholes, leading to an over-representation of abundant units in the training dataset.

The Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) algorithm was used to identify these clusters and mitigate their translation (as representation bias) into the deep learning models. HDBSCAN is well adapted for its ability to identify clusters of varying densities . We used HDBSCAN to automatically identify clusters in a dimensionality-reduced feature space containing the depths of each measurement (to include a spatial component) and the 30 normalised principal components of the median spectra for each data point. This clustering resulted in a total of 24 distinct groups, with one additional noisy cluster excluded from the dataset (Fig. c) to leave 2946 remaining spectrum–property pairs. Hyperparameters of HDBSCAN were selected manually and iteratively assessed based on the number and separation between the clusters shown in Fig. . We aimed to over-segment the dataset, as the clustering was solely used to help in removing the inherent bias from a larger number of points belonging to one lithology. Any cluster distribution with over-segmented clusters (i.e. the large clusters separated from the rest) would provide similar results during training.

A stratified sampling strategy using the labels from the HDBSCAN clusters was then employed to select well distributed training and validation subsets from drill cores KSL131 and KSL133 (KSL136 was kept separate for independent testing), using a sixfold-stratified train–validation split implemented by the StratifiedShuffleSplit utility in scikit-learn . This stratification ensures the data diversity is captured in both the train and validation datasets, while a weighted loss function was developed (see Sect. 3.6) to ensure the differing cluster sizes do not bias our model.

Our model was developed using PyTorch, an open-source machine learning framework known for its flexibility and strong support for deep learning applications . We implemented a multi-headed variant of a 1D convolutional neural network (CNN) architecture . The 1D CNN enhances feature extraction by incorporating convolutional layers within a multi-layer perceptron (MLP) framework, using kernels with learnable weights that are convolved with the input data .

Each spectral range – VNIR–SWIR, MWIR, and LWIR – was assigned a separate head within the CNN (Fig. ) to allow for sensor-specific feature extraction. This configuration was chosen to accommodate the different spectral sampling resolutions provided by each sensor and the distinct spectral characteristics associated with each of the respective spectral ranges. By tailoring the kernel sizes to each spectral range, the model was better adapted to identify and capture relevant features across the diverse spectral data. The extracted features were then concatenated and fed through additional fully connected layers before outputting the petrophysical predictions. To ensure balanced contributions from each of the three outputs during loss computation, the slowness, density, and gamma-ray counts were normalised by factors of 10-3, 10-1, and 10-3 respectively, to ensure that their values fall between 0 and 1.

A mean squared error (MSE) loss function was used for training, with the following key adjustments. First, to address the imbalance in the number of data points per cluster, we multiplied each output loss by a factor inversely proportional to the number of points in its assigned cluster (see Sect. 3.5). This modification ensured that the model treated each cluster equally, avoiding the disproportionate influence of more populous clusters. Each data point in the kth cluster was associated with a weight wk given by 4wk=Nk-1∑j=1NcNj-1, where Nk is the number of points in the kth cluster in the training set, and Nc is the total number of clusters.

Secondly, we introduced a penalty term into the loss function that discouraged negative outputs, as none of the predicted properties can be negative. This ensures that the model predictions stay within valid (positive) bounds by increasing the loss in proportion to the square of the sum of all negative outputs. A penalty term was used instead of, for example, a ReLU activation function to avoid issues with “dying neurons” in the final layer . The final loss function utilised for training was thus Loss=∑i=1n[1nwim(xi)-yi5+λmax-m(xi),0], where m is the model, xi is the input spectrum, yi denotes the corresponding petrophysical properties, λ is the penalty factor for negative outputs, n is the batch size, and wi is the cluster-defined weight (Eq. ) associated with the ith datum.

The training dataset was shuffled and split into six parts, with the model trained iteratively on each split. Every 100 epochs, a new split was introduced for training and validation, effectively reshuffling the data to help prevent overfitting. Over 600 total epochs, the model iteration with the lowest validation loss was chosen as the best-fitting model. The use of StratifiedShuffleSplit preserved data distribution across splits, reducing overfitting without needing dropout layers or batch normalisation, common in CNNs but sometimes performance-limiting. As mentioned previously, a spatially separated borehole (KSL136) was then used to test the model and evaluate its accuracy after the training phase was completed.

Finally, to help understand the spectral properties that influenced the model predictions, we performed a Shapley value analysis. Shapley values are derived from cooperative game theory and have been widely employed in the machine learning context to quantify each input feature's contribution to model predictions . We apply Shapley values using the Python package shap (SHapley Additive exPlanations) , specifically leveraging the DeepExplainer utility optimised for deep learning models. This allowed us to efficiently compute feature impacts, enhancing interpretability and transparency in our neural network.

To reduce computational complexity, we subsampled the data before computing Shapley values, selecting 10 random samples from each of the HDBSCAN clusters. This ensures that the samples extracted for Shapley analysis are representative of the entire dataset. We then visualised the results in the form of a raster image to help qualitatively assess the spectral regions used by the model for each of its predictions. Lastly, to better understand the impact of certain bands on the outputs, we computed the mean Shapley values for the top 10 % of each predicted output, to explore which factors resulted in a sample being predicted to be fast (i.e. bottom 10 % of the slowness), be dense, or exhibit high gamma-ray counts.

The model predictions for the training set showed scores of 0.990, 0.996, and 0.964 for the slowness, density, and gamma-ray values respectively (Fig. ), suggesting the model explains more than 96 % of the petrophysical variance in the training set. The corresponding root-mean-squared error (RMSE) for these properties was 2.544 µsm-1, 0.023 gcm-3, and 4.993 API respectively.

To better assess the model's accuracy for upscaling applications, we applied it to every 1 mm2 pixel in the hyperspectral dataset (including the test drill core), to create a high-resolution (upscaled) map of the petrophysical properties. The results identify fine-scale and cross-core variations that could not be captured by traditional running log measurements (Fig. ).

Our workflow for hyperspectral downsampling (see Sect. 3.3) was then run on these petrophysical property prediction maps to derive an estimate of the petrophysical logging data (noting that the predicted properties slowness, density, and gamma-ray readings are all additive, so they can be arithmetically averaged).

The resulting predicted logging data showed a close match to the measurements (Fig. ), both for the two training boreholes and for the independent test hole (KSL136). R2 scores for this test hole were 0.86 and 0.9 for the density and gamma-ray logs respectively, indicating very reasonable accuracy on unseen data, which even included a basement lithology that was not sampled by the other training drill cores (highlighted by the grey box labelled Tonschiefer i.e. argillaceous basement). The slowness prediction in KSL136 showed a relatively lower R2 score of 0.7, with most of the erroneous predictions lying within the unseen lithology. The measured sonic log here shows significant fluctuations, whereas our model prediction remains steady (suggesting the lithology is spectrally quite uniform).

The calculated Shapley values (Fig. ) represent the quantitative importance of every input feature in the form of a push away from the mean prediction. Since the theoretical foundation of Shapley analysis looks at every band as a separate feature and not necessarily the overall shape of the absorption feature itself, the overall contribution from a particular spectral range cannot be ascertained from looking at the raw Shapley values. However, a single band in the LWIR, on average, contributes more towards the output than a single band in the VNIR–SWIR–MWIR ranges.

To look at the total contribution each spectral range makes towards the output, we use the additive nature of Shapley values to create a stack plot (Fig. ). The plot shows that the contributions from the VNIR–SWIR range are minimal in all predictions. The net contribution from the MWIR bands is the largest, followed by the LWIR bands for slowness predictions, except for the fastest samples, where the LWIR contribution exceeds that of the MWIR bands. For the density predictions, the MWIR dominates the predictions across all the samples, closely followed by LWIR. For the gamma-ray values, the SWIR bands contribute more than in the other two properties, but the LWIR appears to be the dominant steerer of the predicted value.

The median Shapley values of the samples in the bottom 10 % of the slowness predictions (i.e. fastest predicted P wave velocities), the top 10 % of density predictions, and the top 10 % of gamma-ray predictions were calculated as well (Fig. ). The hull-corrected spectra for these samples (Fig. ) were then inspected to identify possible spectral features associated with fast, dense, or high gamma-ray predictions. From Fig. 10, it is clear that the model is focusing on known mineral absorption features (and their surrounding bands) in each of the spectral ranges. These include, but are not limited to, the carbonate absorption feature at 11 200 nm, as well as the fundamental overtones of the silicon oxide (νSiO) absorption feature at 8575 and 9200 nm. The 1400 and 1900 nm water and 2200 AlOH SWIR absorption features are also clearly being used, as are the bands around 2300 nm (which are likely influenced by the relatively broad carbonate absorption at 2345 nm).

The results highlight some spectral variability within these top 10 % groups, as would be expected, but also some commonalities. For instance, in the slowness plots (Fig. a), the samples show a significant push below the mean due to the bands around 11 200 and 4000 nm, corresponding to primary and secondary overtones of carbonate absorption features. This observation aligns with our geological understanding, as the fastest samples are from the Stassfurt dolomite formation (see legend, Fig. ). The spectra in Fig. a further support this inference, as they exhibit a clear carbonate absorption feature.

Conversely, the densest samples in the region are associated with the Werra Anhydrite formation (see legend, Fig. ). They do not exhibit the carbonate absorption feature as seen in the spectra (Fig. b), and the low Shapley values for those bands (Fig. b) confirm that the model is no longer looking at any of the carbonate absorption features.

The samples with the top 10 % of gamma-ray readings come from the Aller salt clay formation, the Stassfurt salt formation, and the Rotliegend sandstone formation. The model considers the bands corresponding to the water absorption feature, as well as the clay feature in the SWIR favourably (Fig. c). Furthermore, there is a considerable peak in the 9200–9600 nm range in the LWIR, which might correspond to the tecto- and sheet-silicate absorption features containing potassium salts from the Aller salt clay and Stassfurt salt formations, or the K-feldspar absorption feature from the Rotliegend sandstones, both of which would give rise to high gamma-ray values.