Item 26 — Method of selecting the final model

Describe the method and metrics used to select the best-performing model among all the models trained for evaluation against the held-out test set. If more than one model was selected, justify why this was appropriate.

Explanation and Elaboration: Typically, many variations on a model will be trained as part of model and hyperparameter optimization, but in general, only one final model should be evaluated on the held-out test set. Criteria for determining which of these models is selected as the “best” final model should ideally be determined in advance and must be clearly described. Absence of clear methodology for model selection raises concern that multiple models may have been evaluated against the held-out test set and the best performing model selected. Since there is some random variability in the performance of any model based on the items included in the test set, this multiple testing biases the reported metrics to be higher than the actual expected performance of the model. In some cases there may be multiple related objectives for model development and it may be appropriate to select a small number of final models that are “best” for different criteria (e.g. favoring sensitivity or favoring specificity, or for ensembling). In such cases, the decision to select more than one model requires justification and each set of selection criteria should be detailed.

Examples
We independently trained 10 deep learning models to predict malignancy from a single image. … Multiple models were trained over a few variations of data sources, image types, malignancy time windows, and binary prediction tasks to produce the model bank outlined in Table S2. Such a diversity of complementary models helped the high-level vision system achieve higher performance by providing a rich image descriptor. … Metamodels were trained using a random hyperparameter search for 50 iterations and five-fold cross-validation with four repeats. During 50 iterations, we selected the four sets of hyperparameters that yielded the best performance in terms of area under the receiver operating characteristic curve (AUC), mean rule-out-rate (ie, mean specificity in a sensitivity range of 95%–100%), rule-out rate (ie, specificity at 99% sensitivity), and ROC equality point (ie, the point at which sensitivity equals specificity). The final metamodel was an averaged ensemble of these four metamodels with uniform weights. … After the model training phase, the model operating point was calculated based on the validation dataset composed of only cancer-positive samples from the US Dataset 1 and UK Dataset 3 validation data at two target sensitivity levels: 0.99 and 0.97, resulting in thresholds T = 0.02094 and T = 0.03584. A bootstrapping method was utilized as a form of interpolation. The cancer positive validation dataset, which comprised 646 mammograms (106 from US Dataset 1, 540 from UK Dataset 3), was sampled with replacement resulting in 2000 bootstrapped datasets, each composed of 646 mammograms (not all input mammograms are present in a given bootstrap sample and some are repeated due to sampling with replacement). For each bootstrapped dataset, the operating point was chosen by selecting the mammogram from the set such that less than a fraction S of the examinations have a smaller model-predicted probability of cancer (this is equivalent to finding the S-percentile of model outputs). The final operating point was then calculated as the mean operating point chosen over the 2000 bootstrap samples. The evaluation study was performed at two sensitivity levels: S=0.99 and S=0.97. [Appendix S2. Model Development (23), CC-BY 4.0]
Because T2-weighted and synthetic MCE MRI images are anatomically coregistered, kidney masks created on T2-weighted images provide ground truth for training segmentation models on synthetic images. Thus, four Mask R-CNN models (Smodels 1, 2, 3, and 4) were trained using the synthetic images of each of the MCE MRI phases and the T2-weighted masks as input. … To train Mask R-CNN models, we used the gray-scale images as input and the binary kidney masks as ground truth (512 × 512). We set a batch size of four, a number of epochs of 100, an initial learning rate of 0.008 with a momentum optimizer of 0.9, and a gradient clipping by a norm of 10. The optimal model was selected based on the minimum of the validation loss. With the pretrained weights, the Mask R-CNN models were trained using synthetic datasets for each MCE MRI phase. [Materials and methods (57), CC-BY 4.0]