Evaluation of the quality and the productivity of neuroradiological reading of multiple sclerosis follow-up MRI scans using an intelligent automation software

Multiple sclerosis (MS) is an autoimmune, potentially disabling demyelinating disease of the central nervous system (CNS), with clinical onset typically in the young adult [1]. Traditionally, MS progression have been categorized in three main type, primary-progressive MS (PPMS) affecting between 10 and 15% of MS patients, relapsing–remitting MS (RRMS) and secondary-progressive MS (SPMS) [1], although it has been recently suggested that the clinical course of MS is better described as a continuum, with contributions from concurrent pathophysiological processes that vary across individuals and over time [2]. Prior to the availability of the disease-modifying therapies, roughly 50% of patients diagnosed with RRMS would transition to SPMS within 10 years, and 90% would transition within 25 years, with a median time of about 19 years [3]. The diagnosis of SPMS is most often established retrospectively [4], on average 3 years after the actual progression started [5]. This is caused by the difficulties by the clinicians and the patients to interpret the initial symptoms indicating early progression, which they can be subtle and fluctuating [4], and by the difficulties to detect disease progression on imaging. The importance of detecting remaining inflammatory disease earlier and more systematically has further increased with the demonstration that early intervention with high-efficacy disease modifying therapies can delay the onset of SPMS [6], and with the recent availability of approved new therapies for progressive MS, such as the anti-CD20 monoclonal antibody Ocrelizumab [7] or the selective sphingosine-1-phosphate receptor modulator Siponimod [8].

MS disease monitoring with imaging is particularly challenging. It involves at least yearly follow-up magnetic resonance imaging (MRI) scans of their central nervous system and consists of the tedious, time-consuming, and error-prone manual comparison and counting of the multiple demyelinating lesions. Given time constraints and the fact lesion burden can reach hundreds in severe cases, neuroradiologists can be forced to grossly compare the lesion load, and relevant disease evolution might remain unnoticed for some time. Longitudinal subtraction maps have been used to increase sensitivity for detecting new or enlarged lesions multiple sclerosis lesions [9‐12] and reduce reading time [13], but small lesions might be difficult to detect using this technique [14].

Fully automated neuroradiological reporting of MS lesion load evolution would be of great interest, but current methods, including deep learning methods, have currently limited practicality given they are far from error free. In the last decade, deep learning methods, in particular two-dimensional and three-dimensional convolutional neural networks, have been applied to develop automatic lesion detection and segmentation [15]. Reported lesion-wise true positive rate ranges between 0.15 and 0.83, and the lesion-wise false positive rate ranges between 0.08 and 0.68 [16‐27]. Those numbers are relatively low, given that every single new or evolving lesion on a follow-up control can be considered a marker of disease progression requiring a re-evaluation of the patient’s current therapy, and therefore, the expert radiologist assessment is still required for every patient [28].

In this context, a tool based on intelligent automation, which could simplify the work of the radiologist by automating many repetitive tasks, would be of interest. Intelligent automation aims to streamline processes to improve efficiency and reduce errors [29]. It applies the concept of breaking larger complex tasks into several simpler, very-well-understood low-level steps, and automatizes the low-level steps with intelligent software, using artificial intelligence and advanced software engineering. By freeing the radiologist to concentrate on the lesion assessment, it could reduce the number of missed lesions. Intelligent automation promises to improve productivity, reduce costs, as well as more precise processes and a better user experience. Jazz [30] is an intelligent automation software dedicated for neuroradiology, that automates multiple low-level tasks such as contrast recognition, previous exams organization, and images coregistration. It includes a semi-automatic lesion annotation tool, permitting the user to easily save all relevant information in a lesion list, and has a graphical user interface intended to maximize productivity and to minimize operator fatigue and discomfort (Fig. 1).

In this retrospective study, we hypothesize that the quality and the productivity of neuroradiological reading of MS follow-up MRI scans can be improved using an intelligent automation software such as Jazz. We evaluated Jazz for the assessment of new, expanding, and contrast-enhancing MS lesions, blindly in three centers, and compared the reported lesions with the lesions described in the standard clinical report.

The following analysis was performed in three separate centers in a fully blinded and independent manner.

This study shows that a significantly larger number of new and SELs are reported when using the intelligent automation software Jazz compared to the standard clinical reporting method in a time efficient manner, by allowing the neuroradiologist to concentrate on the detection and evaluation of lesions. This is to our knowledge, the first study involving an intelligent automation software in the clinics. By permitting to toggle between coregistered current and previous exams, it uses a particularly potent neurophysiological mechanism to attract the attention of the reader to modifications in the images, in other words in the case of MS, to new and evolving lesions. Indeed, attention is a selective process and is physiologically necessary, because there are severe limits on our capacity to process visual information [31]. A theory suggests that attention is imposed by the fixed amount of overall energy available to the brain, and that because of the high-energy cost of neuronal activity, only a small fraction of the machinery can be engaged concurrently [31]. As a consequence, stimuli have to compete for limited resources [32], a notion that is supported by electrophysiological, neuroimaging and behavioral studies [33, 34]. Of the several visual stimuli that are known to capture attention, the sudden appearance of a new object and the sudden changes of an object are known to be particularly potent and to influence priority in visual search [35‐37]. This mechanism is exploited in the dynamic switching tool of Jazz, compared to a standard side-by-side comparison method.

The identification of progression in MS is typically retrospective and based on clinical symptoms [38]. Most clinical trials rely on an increase in Expanded Disability Status Scale score confirmed over 3 or 6 months to define disease progression, and might overestimate the accumulation of permanent disability by up to 30% [39]. Given the profound burden of progressive MS and the recent development of effective treatments for these patients, there is a need to establish reliable and reproducible methods for identifying progression early in the disease course. Neuroradiological MRI reports of MS are often sparse and not standardized, despite the advantages of standardization of clinical data for MS monitoring being well recognized [40]. Structured reports of MRI in patients with MS have been shown to provide more adequate information for clinical decision making than nonstructured reports [41]. Improvement in the digitization of real-world patient data could foster new insights into the epidemiology and pathophysiology of the disease, and are ultimately necessary to achieve truthful personalized patient care [42].

This study also shows that at least one SEL was described in more than half of all patients, while none were reported in the standard clinical reports. The inter-reader agreement was low though, showing that this assessment remains particularly difficult and prone to subjectivity. The use of objective criteria defining SELs might help to improve this aspect. SELs can be easily overlooked, and are particularly difficult to detect in a standard DICOM viewer. There is substantial interest in SELs as a potential marker of chronic but active MS lesions [43‐45] which may have diagnostic, prognostic and treatment implications. SELs were found to be more prevalent in patients with PPMS compared with patients with RRMS [44]. The proportion of SELs and their microstructural tissue abnormalities were associated with a higher risk of MS progression and SPMS conversion [46]. In SPMS, SELs were found to represent almost one-third of T2 lesions, associated with neurodegenerative MRI markers and related to clinical worsening [47]. Therefore, the number and volume of SELs are a promising biomarker to predict a more active, progressive disease course and could become a new target for therapeutic intervention. This is particularly interesting, because recent advances in our understanding of the mechanisms that drive SELs have fueled optimism for improved treatment of this condition [48]. A robust tool to gather SELs on magnetic resonance images, such as the one described in this work, might help to detect patients with progressive disease earlier and more systematically.

This study suffers from several limitations. Only three centers were included, and our results may not generalize to other centers. The reading using the Jazz software was not under identical conditions to the reading employed to generate the standard report, which was done during the standard clinical routine and might have included more distracting events. The reading of the standard clinical reports included at least a first reading and a second reading through a supervisor, and discussion with, and comments from the referring clinicians might have occurred. The reading with the Jazz software only included a single reading from a single reader. The ground truth was not based on a consensus reading performed by the most experienced reader in each center, which might have biased the agreement. Current and previous exams could have been acquired on different scanners, which might be a cofounder in the SEL evaluation. Further, a previous definition of SELs was based on the inclusion of a least three time points over more than 2 years [44], with local expansion assessed by the Jacobian determinant of the deformation between reference and follow-up scans, and a heuristic scored was used to favor individual SEL candidates undergoing concentric and constant change [44], while here, a more pragmatic approach was considered, using the subjective expansion assessment based on only two MR time points. Finally, no standardized method was used to define the SELs, as the evaluation was left to the subjective appraisal of each reader.

In conclusion, this study shows that significantly more new lesions and SELs can be described in a time-efficient manner using the intelligent automation software Jazz compared to the standard method.