Item 16 — Source of reference standard annotations

Specify the source of reference standard annotations, citing relevant literature if annotations from existing data resources are used. Specify the number of human annotators and their qualifications (eg, level of expertise, subspecialty training). Describe the instructions and/or training given to annotators; include training materials as a supplement.

Explanation and Elaboration: Reference standards are often based on human annotations; the quality of such a reference standard is heavily dependent on the quality of the annotation process and the skill of the annotators. Authors should specify whether annotations were generated prospectively or included retrospectively, who performed them, how many annotators were involved, and their qualifications (e.g., board-certified neuroradiologist with 10 years of experience). The annotation procedure should describe whether labels were assigned independently, in consensus, or with arbitration, and whether annotators had access to auxiliary data (e.g., clinical history, prior reports). Standardized instructions or training protocols should be reported to ensure reproducibility and minimize variability.

Exampless
Manual segmentation was performed independently by two board-certified radiologists initially (T.A.D. and H.C.B., with 12 and 7 years of experience, respectively). If an existing model for a given structure was publicly available (Appendix S3), that model was used to create the first segmentation, which was then reviewed and refined manually …. [Material and methods (37), © RSNA 2025]
The included referrals (n = 1953) were classified into suitable imaging protocols and use or no use of a gadolinium-based contrast agent by two fellowship-trained neuroradiologists (M.N., 16 years of experience, and J.H., 12 years of experience) in consensus, regarded as better reference standard than original instructions from radiologists of varying experience. [Methods (38), CC BY 4.0.]