Item 32 — Evaluation on internal data

Document and describe evaluation performed on internal data. If there are systematic differences in the structure of annotations or data between the training set and the internal test set, explain the differences, and describe the approach taken to accommodate the differences. Document whether there is consistency in performance on the training and internal test sets.

Explanation and Elaboration: “Internal data” refers to data from the same dataset or source as the data used for training. Internal test sets are typically constructed by extracting a patient-level random sample of the overall dataset. This internal test set is often annotated more rigorously than the training data, as machine learning methods are relatively robust to errors in annotation but performance metrics are not. These differences in methods may be appropriate, but can introduce unintended systematic differences between the training and internal test sets. For instance, multiple annotation of the internal test set in the RSNA Pneumonia Challenge using overlap of bounding boxes to resolve discrepancies led to systematically smaller bounding boxes in the test set relative to the singly-annotated training set (69). Investigators should try to identify any such systematic differences they may have introduced, and where they are present, describe the impact and strategy to mitigate this source of error. Performance on the training set is not meaningful in isolation, but in combination with metrics for performance on an internal test set, it helps to identify overfitting. A small decrease in performance on the internal test set is expected; a large decrease suggests overfitting.

Examples
The internal dataset included 993 HCM patients from a multi-center HCM study … and was used for model development (i.e., training and validation) and held-out internal testing. The dataset was collected from 2003 to 2018 and included scans from 7 imaging centers: Tufts medical center (Boston, Massachusetts, USA), Minneapolis Heart Institute (Minneapolis, Minnesota, USA), Toronto General Hospital (Toronto, Canada), and four medical centers in Florence, Rome, Genova, and Bologna (Italy). Two readers reviewed the LGE images of the internal dataset, and we excluded cases where there was no agreement on presence of scar, e.g., due to low image quality or artifacts. The resulting dataset with consensus readings included 759 patients from the 7 centers (Tufts, n = 321; Minneapolis, n = 125; Toronto, n = 141; Italy, n = 172) and was split (patient-wise) into development (n = 600) and held-out testing (n = 159) cohorts. Five-fold cross-validations were employed to split the development dataset into training (80%) and validation (20%) subsets for determining the optimal model parameters. Stratified patient-wise random sampling was used to maintain same number of patients with and without scar in the testing subset. The testing subset is referred to as internal testing set because it is sampled from the same cohort used for model development. [Methods (70), CC BY 4.0]
In total, 103 MRI datasets of 67 LION study participants with available ground truth labels from both baseline and follow-ups were split into 83 datasets for training and 20 datasets for testing. The 83 datasets used for training (among which 49 were female and 34 were male datasets) were split with a ratio of 4:1 for training and validation. When partitioning into training and test datasets, one participant’s images from the different time points were strictly put into either training or test groups. Training was performed either with a two-channel input using water- and fat-separated images or with a three-channel input adding background-removed T2* maps as a third channel. T2* maps were added because some anatomic structures, such as the vena cava, looked similar to the liver on the water-fat separated images but showed different contrast on the T2* maps. The network results were then applied to the unsegmented data to evaluate the cohort characteristics. [Materials and methods (40), CC BY 4.0]