Item 29 — Statistical measures of significance and uncertainty
Indicate the uncertainty of the performance metrics’ values, such as with standard deviation and/or confidence intervals. Compute appropriate tests of statistical significance to compare metrics. Specify the statistical software, including version.
Explanation and Elaboration: Uncertainty is inherent in any measurement of model performance, since metrics are calculated based on a sample (the test set). Metric values are essentially uninterpretable without quantification of this uncertainty. Wherever the manuscript makes direct comparison between values (e.g. comparing different models), an appropriate statistical test should be employed to estimate the probability that the observed difference may be due to chance (the p-value). Working with a statistician helps to ensure the analysis is accurate, detailed, and methodologically sound. Statisticians are cautioned that commonly used tests that are well understood by scientists who are not statistics experts should be preferred to novel tests; novel statistical methods should be reserved for circumstances that cannot be addressed by simpler techniques. Statistical software version should be reported to facilitate enhance reproducibility and facilitate detection of errors if the software is later determined to have bugs.
| Examples |
| Standard descriptive statistics were used on a per-study level to compare iPE detection rate and characteristics. The chi-square test was used for comparing categorical variables. The exact Clopper-Pearson interval was used for calculating CIs for proportions. Categorical data are presented as percentages, and continuous data as the mean ±1 SD. P < .05 was considered statistically significant. SPSS Statistics for Windows, version 27 (IBM Corp.), was used for all analyses. [Materials and method (64), CC BY 4.0] |
To determine whether the features generated by a model were biased, we used two-sample Kolmogorov–Smirnov tests to determine P values for the null hypothesis that the marginal distributions for a given pair of subgroups are identical in each of the first four modes of PCA. These statistical tests were performed for all relevant pairwise comparisons regarding the presence of disease, biologic sex, and race. The P values were adjusted for multiple testing using the Benjamini–Yekutieli procedure, and significance was determined at a 95% confidence level (P < .05). To evaluate and compare the disease detection performance of different models, we computed the AUC, true-positive rate (TPR), and false-positive rate (FPR). TPR and FPR in subgroups were determined at a fixed decision threshold, which was optimized for each model to yield an FPR of 0.20 on the whole patient sample. The fixed target FPR allows for immediate identification of performance deviations across subgroups. To provide a single measure of classification performance, we report the Youden J statistic (measured at the target FPR), which is defined as J = TPR − FPR. We used bootstrapping with 2000 samples to calculate 95% CIs. [Materials and method (65), CC BY 4.0] |