Item 19 — How data were assigned to partitions

Specify how data were partitioned for training, model optimization (often termed “tuning” or “validation”), and testing. Indicate the proportion of data in each partition (e.g., 80/10/10) and justify that selection. Indicate if there are any systematic differences between the data in each partition, and if so, why and how potential class imbalance was addressed. If using openly available data, use established splits to improve comparison to the literature. If freely sharing data, provide data splits so that others can perform model training and testing comparably.

Explanation and Elaboration: Tuning and testing partitions must be large enough to calculate performance metrics with appropriately narrow confidence intervals – see Item 21. When there are class imbalances, stratified sampling may be necessary to ensure adequate representation in all partitions. Test partitions are often annotated differently than training partitions, such as using multiple annotators for test and single annotators for training. Any such differences should be justified and potential biases introduces should be discussed.

Examples
We randomly divided eligible patients from the primary cohort into a training set (n = 507) and a validation set (n = 218) at a 7:3 ratio. [Materials and methods (44), CC-BY 4.0]
The data is split into training, validation, and testing sets with ratios of 50 %, 20 % and 30 % respectively. We train our model on the training set and use the validation set for evaluation. The best hyperparameters are then tested on unseen testing data. For Casc-UNetS, tuning is done separately for each stage. For UNetS1, hyperparameters are tuned with and without CLAHE. For UNetS2 and 3, only CLAHE MRI are used due to improved validation losses. [Methods (45), CC-BY 4.0]
The cohort of 9919 women (13 306 DBT examinations) was split into a training set of 3948 women (804 cancers [20.3%]), a validation set of 1661 women (182 cancers [11.0%]), and a test set of 4310 women (453 cancers [10.5%]) (main test). Examinations in the same woman appear within only one of these data sets. The training data set included examinations from 18 imaging facilities. The validation set was used for model selection and calibration of the AI model (Appendix E1 [online]). A subset of the main test (Fig 2C, Table E1 [online]), which we named sites test (2375 women), included examinations from four sites and was used to test the generalizability of AI to unseen sites. [Methods (46), CC-BY 4.0]