Item 33 — Testing on external data

Describe the external data used to evaluate the completed algorithm. If no external testing is performed, note and justify this limitation. If there are differences in structure of annotations or data between the training set and the external testing set, explain the differences, and describe the approach taken to accommodate the differences.

Explanation and Elaboration: Machine learning models typically demonstrate poorer performance on external data–data drawn from different sources than their training data–due to limited ability to generalize. Generalizability is crucial in medical imaging AI, as the data seen in clinical deployments is rarely drawn from exactly the same distribution as the training dataset. Absence of external testing is a severe limitation; in most cases, work is not publishable without external testing. External datasets are ideally as different as is reasonably practical from the training data; the sources and demographics of the external data should be described, so readers can assess the extent of the difference. Ideally the structure and definitions of annotations on the external data are concordant with those on the internal data, but that may not be the case when a previously created dataset is used for external testing. When such unavoidable differences exist, the manuscript should detail the approach employed to mitigate their impact.

Examples
Next, 35 controls (mean age, 61.51 years ± 7.38) from the Frontotemporal Lobar Degeneration Neuroimaging Initiative (https://4rtni-ftldni.ini.usc.edu/) and 35 (mean age, 65.37 years ± 4.53) from the Alzheimer’s Disease Neuroimaging Initiative (https://adni.loni.usc.edu/) were used as external test sets. Finally, MR images in 71 patients with PSP–Richardson syndrome, (58 [mean age, 70.57 years ± 7.16] from the 4-Repeat Tauopathy Neuroimaging Initiative [http://4rtni-ftldni.ini.usc.edu] and 13 [mean age, 70.46 years ± 3.07] from the Center for Neurodegenerative Diseases and the Aging Brain of the University of Bari at Pia Fondazione Cardinale G. Panico) and 129 age- and sex-matched patients with PD (98 [mean age, 68.45 years ± 5.51 years] from the Parkinson’s Progression Markers Initiative dataset and 31 [mean age, 69.88 years ± 5.34 years] from the Center for Neurodegenerative Diseases and the Aging Brain) were used as clinical test dataset. T1-weighted images were acquired with 1.5-T and 3-T MRI scanners from different vendors and with a wide variety of high-resolution sequences (Table S1). All individuals provided informed consent, and the protocol was approved by the institutional review board at all sites. [Materials and methods (36), CC BY 4.0.]
Two datasets were used for external testing (Table 2): the CQ500 dataset … (436 examinations; 212 ICH; mean age, 48 years ± 29 [SD]; 158 of 436 [36.2%] women) and the CT-ICH dataset … (75 examinations; 36 ICH; mean age, 28 years ± 20; 33 of 75 [44.0%] women). The CQ500 dataset comprises scans from clinical centers in New Delhi, India, annotated with the types of hemorrhage present by three expert radiologists. For a subset of 196 scans, images were enhanced via the BHX dataset … with 6282 manual segmentations of bleeds performed by three other expert radiologists. The CT-ICH dataset was collected from Al Hilla Teaching Hospital, Iraq, from patients with traumatic brain injury with manual segmentations of bleeds performed by two expert radiologists. All images were windowed with the standard brain setting (window width = 80; window level = 40) and minimum and maximum normalized. [Materials and methods (71), CC BY 4.0.]