Clinical Evaluation of a Quantitative Imaging Biomarker Supporting Radiological Assessment of Hippocampal Sclerosis

Hippocampal sclerosis (HS) is the most frequently observed pathology in treatment-refractory mesial temporal lobe epilepsy [1]. For detecting such structural epileptogenic lesions, magnetic resonance imaging (MRI) with appropriate protocols is pivotal [2‐5]. Radiological signs of HS include volume loss and T2/FLAIR signal alterations [6], whereas a loss of the hippocampal internal architecture is difficult to demarcate on 3T imaging [7]. The radiological interpretation from MRI is predominantly a visual task by qualitatively assessing the images, although quantitative volumetry has been generally recommended [8]. Ultra-high-field (UHF) MR imaging at 7 Tesla has shown promising results in epilepsy [9‐11], e.g., better visualization of the hippocampal internal architecture [12]. Since 2017, 7T MRI has been cleared for clinical applications and implemented beyond research setting in neuro – and musculoskeletal imaging¹. The available evidence and current consensus recommendations support the use of 7T MRI in patients with epilepsy with specific clinical questions [13, 14]. However, only a minority of patients will have access to 7T imaging due to the limited availability of the devices and high costs.

The literature contains a plethora of suggestions for automated methods to analyze MRI in epilepsy, increasingly by employing artificial intelligence [15]. Such techniques are often evaluated by assessing the accuracy against a ground truth [16‐19], or contrasting accuracies with radiological reading alone [18, 20]. Given that medical decision support tools are usually intended to be used as supportive information for radiologists and not as a replacement, evaluation should be performed accordingly. However, even among commercial products with CE/FDA label, only a minority have tested and demonstrated clinical efficacy [21‐23] as this is currently not a requirement for CE/FDA clearance.

The quantitative neuroradiology initiative (QNI) [24] has proposed a framework comprising six steps for the technical and clinical validation necessary to embed automated image quantification software (QReports) into the clinical neuroradiology workflow. Likewise, [21] described six levels of efficacy to assess the contribution of automated tools in a diagnostic process, ranging from technical efficacy in level I over diagnostic thinking (level III) and therapeutic (IV) efficacy to societal effect in level VI.

An imaging biomarker for HS using cross-sectional area and T2 relaxometry profiles along the hippocampal posterior-anterior axis was described by [25]. In a subsequent evaluation study, [26] reported increased accuracies with a very large effect size (\(d=1.23\) for neuroradiologists) when using the additional QReports.

We have previously proposed an imaging biomarker to support the radiological assessment of HS [19]. In brief, it is based on accurate segmentation from T1-weighted MRI using deep-learning (DL) [27, 28] from which the surface-to-volume ratio of the hippocampi is calculated. The metrics are presented in a graphical report allowing direct interpretation by radiologists. A standalone evaluation against a ground truth revealed higher robustness and accuracies compared to non-DL based methods. In the present clinical validation study, we extend the evaluation by quantifying the impact when used to complement and inform expert assessment. MRI from 40 patients with epilepsy were assessed by six raters with and without additional QReports for the presence of HS and hippocampal volume abnormalities. Owing to the increased attention of amygdala enlargement in TLE [29, 30], the raters were asked to estimate amygdala volume abnormalities as well.

Including the warm-up phases, a total of 600 examinations with \(\sim 1800\) MRIs were assessed, and 3600 data points were recorded by the six raters altogether (6 raters, 50 cases with 2 readings, at least 3 images per examination, 6 answers in the reporting form). The mean accuracy across all six raters regarding presence of HS was 77.5% in the first round with the availability of MRI only (Table 1). Five out of six raters reached a higher accuracy in the second round with the additional QReport available, resulting in a mean accuracy of 86.3% (Fig. 2). The accuracies improved by a very large effect size of \(d=1.43\). All raters perceived a higher confidence of their rating in the second round. In contrast, if one would use the QReports alone with three standard deviations (SD) as the decision boundary, an accuracy of 87.5% would result (Fig. S3). The presence of HS was overestimated by the raters in the first round (in total 36 false positive [FP] ratings and 16 false negative [FN] ratings pooled across all raters), with a more balanced ratio in the second round (13 FP, 17 FN), as depicted in the pooled confusion matrices (Fig. S11).

Independent of the ground truth, inter-rater agreement improved from \(k=0.56\) to \(k=0.72\) (Fig. 3). We have observed a tendency of the neurologists (R1–R3) to change their assessment more frequently than the neuroradiologists (R4–R6). A qualitative example of the case P05 is shown in Fig. 4, where two raters changed their decision in the second round. Additional cases are discussed in the Appendix: Case P13, where two raters erroneously changed their rating (Fig. S4), possibly because the QReport was ambiguous with measures lying between two and three SD, P38 without HS that was erroneously classified as HS right by three raters in the first round (Fig. S5), and P28 as an example where additional clinical context is crucial for establishing a diagnosis (Fig. S6).

Inter-rater agreement for hippocampal volume abnormalities increased from \(k=0.50\) in the first round to \(k=0.63\) in the second round (Figs. S7–8). Also for the amygdala, where inter-rater agreements for volume were generally much lower, they increased from \(k=0.16\) to \(k=0.28\) (Figs. S9–S10). On average, the raters reported an abnormality of the amygdala volumes in 30.4% of the cases in the first round, whereas this fraction reduced to 16.7% in the second round using the QReports.

In agreement with the findings by [26] who performed a comparable study with a different imaging biomarker [25], we have observed a similar trend-level improvement of the diagnostic accuracies with a very large effect size. This suggests that these results are likely robust, and statistical significance is mainly a matter of the small number of raters in both studies.

Overall, we have observed a lower accuracy (77.5% in the first round) compared to the study of [26] (87.5%). This difference might be explained by the high complexity of our cases. The 7T MRI examinations of all patients were clinically indicated during detailed phase I or phase II evaluation [31] because the previous 3T MRI yielded equivocal results that did not answer all clinical questions.

Neurologists changed their decision more frequently than neuroradiologists during the second rating round when the QReports were available. This might indicate that radiologists with more experience in visual MRI interpretation are more confident in their assessment. No apparent dependence of the results on the seniority of the raters was noticed for the primary diagnosis. For the hippocampal volume asymmetries, more experienced raters tended to change their rating less frequently (cf. Fig. S7). Although of no statistical significance, it is noteworthy that a neurologist (with several years of experience in epileptology) reached the highest accuracy overall in the second round with the QReports.

We intentionally included not only experienced neuroradiologists in this study. Tools available to support MRI interpretation with quantitative evaluation are designed to be also used in primary and secondary care centers where images might be interpreted by radiologists and neurologists with regular training. On the other hand, clinical epileptologists in tertiary centers might use such tools as assistance in image interpretation.

Estimating amygdala volume abnormalities is a difficult task, as indicated by the low inter-rater agreements. The high fraction of 30.4% reported abnormalities in the first round might indicate an overestimation by the raters owing to the study setting. As we explicitly requested for an estimation, raters might have reported minor perceived asymmetries that would otherwise be described less frequently in a routine medical report. Nevertheless, by using the QReports the inter-rater agreement increased and the number of abnormalities decreased (16.4%). These findings suggest that results based on pure visual assessment of amygdala enlargement [29] should ideally be complemented by quantitative methods.

The investigated imaging biomarker is based on a previously proposed metric along with the QReport for communicating the results [19]. Artificial intelligence (AI) is used for high-quality anatomy segmentation, but not to predict a diagnosis directly. Instead, the surface-to-volume ratio is derived from the segmentation and depicted in a quantitative report. These results are interpretable, making it particularly suited as complementary information for expert reading and mitigating the black-box challenge of AI-based systems [40].

An abundance of learning-based methods for classification of hippocampal sclerosis from MRI [17, 18, 41‐46] stands in stark contrast to scarce clinical evaluations [21]. While the importance of scrutinizing the standalone performance of such imaging biomarkers is undisputed, estimating the potential for a future translation into clinical applications requires an assessment of the diagnostic efficacy in a setup mimicking clinical routine. Because the implementation of a decision support system is most likely alongside a radiologist (i.e., “human vs. human + machine” and not “human vs. machine”), we have designed this evaluation study as a level 3 [21] assessment accordingly.

Additional quantitative reports supporting the radiological assessment of hippocampal sclerosis decreased the inter-rater variability of raters compared to visual interpretation of 3T MR images alone. With the QReports, an increased mean accuracy by a very large effect size was observed when comparing the diagnosis to a consensus derived from 7T imaging.