Item 28 — Metrics of model performance

Describe the metrics used to assess the model’s performance and indicate how they address the performance characteristics most important to the clinical or scientific problem…. Compare the presented model to previously published models.

Explanation and Elaboration: There are a wide variety of performance metrics available for evaluating models, some common in data science and others more common in diagnostic medicine. Select metrics that focus on measuring the aspects of performance that are most relevant to the specific problem you are addressing. Typically, these are metrics commonly found in non-AI medical diagnostics manuscripts, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC-ROC). Metrics that are dependent on prevalence, including positive and negative predictive values and overall accuracy, should be avoided unless prevalence in the test set is similar to the population to which the model will be applied. It may be appropriate to secondarily report non-medical metrics commonly used in AI to facilitate comparison with other models.

Examples
The ability of the system to detect tumors was assessed by calculating the area under the receiver operating characteristic curve (ROC AUC) and generating a confusion matrix. Automatically generated binary masks were resampled to the original image resolution using cv2.INTER_BITS interpolation before comparing with manual segmentations. The contouring performance of the proposed pipeline, as well as the doctor’s variability, were assessed by using the volumetric Dice similarity coefficient (DSC), Jaccard index (Ji), and 95th percentile Hausdorff distance (H95th). Additionally, we have evaluated quantitative contouring performance using Surface DSC and Added Path Length (APL). [Methods (61), CC-BY 4.0]
To compare pixel-level performance between the baseline and semi-supervised models, we collected 1000 sets of 529 images by conducting random sampling with replacement on the images comprising the pixel-labeled CQ500 test set. We obtained the estimated differences in the Dice similarity coefficient (ΔDSC) between the baseline and semi-supervised models on all 1000 bootstrapped sets of 529 images and confirmed that the ΔDSC are normally distributed at a confidence level of P less than .05 using a standard Kolmogorov-Smirnov test for normality. We then calculated test statistic (Z = ΔDSC/σ, where σ is the empirical SD of the DSC differences on the 1000 bootstrapped sets of 529 images) and calculated a two-sided P value as twice the probability that a standard normal random variable is less than |Z|. [Materials and methods (62), CC-BY 4.0]
In addition, to compare the diagnostic efficacy of the AI system with that of the junior radiologists as well as that of the radiologists under the assistance of AI, we calculated the sensitivity, specificity, and accuracy together with their 95% confidence intervals as well as the AUC. In addition, the Z-value test was used to compute P-values for statistical comparisons between different groups for evaluation metrics other than the AUC values, whose comparison was done instead using the DeLong test…. [Methods (63), CC-BY 4.0]