Item 37 — Performance metrics and measures of statistical uncertainty

Report the final model’s performance. Benchmark the performance of the AI model against the reference standard, such as histopathologic identification of disease or a panel of medical experts with an explicit method to resolve disagreements. State the performance metrics on all data partitions and datasets, including any demographic subgroups.

Explanation and Elaboration: Evaluation of performance metrics stratified across demographic subgroups is critical to identify potential biases and ensure equitable model performance. Uncertainty measures are particularly important for these stratified analyses, as some large apparent differences in performance may not be significant due to small size of subgroups.

Examples
The average DSC between the two expert readers was high. The DSC between the algorithm and Reader 1 was 0.68 (0.07 SD) and Reader 2 was 0.70 (0.07 SD), with no significant difference detected between the two distributions (p = 0.31, t-test). [Results (73), CC-BY 4.0]
The AUC of nomogram for predicting 6-month PFS in training set, validation set and testing set are 0.847 (0.737–0.957), 0.786 (0.566–1.000) and 0.864 (0.735–0.994), respectively. … The AUC of nomogram for predicting 1-year OS in training set, validation set and testing set are 0.770 (0.635–0.906), 0.743 (0.479–1.000) and 0.818 (0.630–1.000), respectively. [Results (74), CC-BY 4.0]