Item 21 — Intended sample size

Describe the size of the testing set and how it was determined. Use traditional power calculation methods, if applicable, to estimate the required sample size. For classification problems, in cases where there is no algorithm-specific sample size estimation method available, sample size can be estimated for a given area under the curve and confidence interval width…

Explanation and Elaboration: The size of the test set or partition should be chosen based on the acceptable level of uncertainty in performance metrics, not simply as an arbitrarily selected percentage of the number of items in the full dataset. Power calculations should be performed based on the specific performance metrics intended to be used as the primary measures or outcomes. Targeted uncertainty (e.g. confidence interval width) should be based on clinical judgment and justified.

Examples
To determine the sample size required for the comparison between the network and radiologists, we assumed a mean Dice score of 0.75 as the lower bound for the radiologist and of 0.80 as the lower bound for the network. Power analysis for a paired t test was performed with logit-transformed Dice scores to approximate normality …. Use of an estimated standard deviation of 1.5 across examinations (based on the network performance in the validation set) resulted in Cohen d of 0.1918. With this effect size, a power of 85% at a significance of 5% required 246 scans. We selected a sample size of 250 scans for the test set. [Materials and methods (48), CC BY 4.0]
Sample size was calculated using a previously developed equation…. To estimate sensitivity and specificity for task 4 of 0.7 … with a 95% CI, a precision of 10%, and an expected prevalence of causal findings of 20%, the number of reports should be at least 400. With approximately 20% … of emergency MRI examinations performed for headache in our institution, at least 2000 reports were screened. [Materials and methods (24), CC BY 4.0]
G*Power version 3.1 (The G*Power Team) was used to determine the sample size …. A recent study reported a median difference of 1.9 months ± 0.4 (SD) between the estimated age and the corrected chronologic age …. Therefore, a sample size of 833 was sufficient to reach a minimum 80% power and 5% error rate according to a two-sided t test. [Materials and methods (49), © RSNA 2023]