Item 38 — Estimates of diagnostic performance and their precision

For classification tasks, include estimates of diagnostic accuracy and their measures of statistical uncertainty, such as 95% confidence intervals. Apply appropriate methodology such as receiver operating characteristic analysis and/or calibration curves. When the direct calculation of confidence intervals is not possible, report nonparametric estimates from bootstrap samples. State which variables were shown to be predictive of the response variable. Identify the subpopulation(s) for which the prediction model worked most and least effectively. If applicable, recognize the presence of class imbalance (uneven distribution across data classes within or between datasets) and provide appropriate metrics to reflect algorithm performance…

Explanation and Elaboration: Quantification of uncertainty facilitates the reader’s understanding of whether or not apparent differences in performance are significant. Raw confidence values associated with classifications produced by machine learning models are generally poor estimates of the probability that the classification is correct. Calibration maps model confidence values to calibrated probabilities so they better reflect the true likelihood that a prediction is correct (75)(76). The predictive variables driving the model’s performance should be explicitly stated, enabling readers to assess their clinical relevance. Subpopulation analyses are critical to identifying groups where the model performs well or poorly, ensuring transparency in its generalizability and fairness. Addressing class imbalance is particularly important in medical datasets, where uneven distributions can skew model performance (77).

Examples
The sensitivity per patient was estimated at 64.8% (3732 of 5760 readings) without AI aid and 75.2% (4331 of 5760 readings) with AI aid, with an AI effect estimated at +10.4% (95% CI: 6.9, 13.9; one-sided P < .001 for superiority). The specificity per patient was estimated at 90.6% (5217 of 5760 readings) without AI aid and 95.6% (5504 of 5760 readings) with AI aid, with an AI effect estimated at +5.0% (95% CI: +2.0, +8.0; one-sided P = .001 for noninferiority and one-sided P = .001 for superiority). Therefore, the primary analysis was successful, and the superiority of specificity per patient with AI aid was demonstrated (hierarchical test). Other metrics for all readers are compared in Table 2 by using two-sided tests. Overall, the reading time was 6.3 seconds shorter (95% CI: –12.5, –0.1; P = .046) with AI aid than without… Planned sensitivity analyses did not change the point estimate and tended to reduce the width of the CI of the sensitivity per patient gain, up to 26% for the linear mixed-effects model compared with the primary analysis, meaning that the primary analysis is more conservative (Table E3 [online]). A post hoc sensitivity analysis removing the outlier (a rheumatologist with extremely low diagnostic performances) did not change the conclusions: There was a 9.5% (95% CI: +6.4, +12.5; one-sided P < .001 for superiority) gain of sensitivity per patient and 4.0% (95% CI: +1.8, +6.2; one-sided P < .001 for superiority) gain of specificity per patient. [Results (78), CC BY 4.0.]
The overall accuracy in generating differential diagnoses improved with increased temperature settings in both GPT-4V (T1, 49% [93 of 190 cases]; T0.5, 45% [86 of 190 cases]; T0, 41% [78 of 190 cases]) and Gemini Pro Vision (T1, 39% [74 of 190 cases]; T0.5, 36% [69 of 190 cases]; T0, 29% [55 of 190 cases]), although there was no evidence of a statistically significant difference (T1 vs T0: 49% vs 41% [P = .12] for GPT-4V and 39% vs 29% [P = .04] for Gemini Pro Vision). The overall accuracy of radiologists was 61% (115 of 190 cases), which was higher than that of Gemini Pro Vision at T1 (39% [74 of 190 cases], P < .001). There was no evidence of a statistically significant difference in overall accuracy between radiologists and GPT-4V at T1 (49% [93 of 190 cases]) after Bonferroni adjustment (P = .02). No evidence of a statistically significant difference in accuracy was observed at T1 between GPT-4V (49% [93 of 190 cases]) and Gemini Pro Vision (39% [74 of 190 cases], P = .05). [Results (79), © RSNA 2024]