Item 14 — Definition of method(s) used to obtain reference standard
Include a clear, detailed description of methods used to obtain the reference standard; readers should be able to replicate the reference standard based on this description. Include specific, standard guidelines provided to all annotators. Avoid vague descriptions, such as “white matter lesion burden,” and use precise definitions, such as “lesion location (periventricular, juxtacortical, infratentorial), size measured in three dimensions, and number of lesions as measured on T2/FLAIR MR brain images.” Provide an atlas of examples to annotators to illustrate subjective grading schemes (eg, mild, moderate, severe) and make that information available for review.
Explanation and Elaboration: A reference standard serves as the benchmark against which models are trained and experimental results are compared, making its clarity fundamental to the integrity of the study. Without a precise explanation of how this standard was established, other researchers cannot reliably replicate the methodology, validate the findings or understand the output the model is trained to produce. Therefore, authors should provide a detailed description of their methodology, including the measurement techniques used to generate the reference standard. This should include defining annotation protocols and outlining expert consensus processes when applicable.
| Examples |
| T2-weighted MRI scans with varying lesion etiologies (traumatic, ischemic, hemorrhagic), injury chronicity (subacute, intermediate, and chronic), orientations (sagittal and axial), and voxel sizes were used for this study (Fig 1, Table 1). Lesions appearing as T2-weighted signal abnormalities (hyperintense or hypointense voxels corresponding to primary contusions, secondary cytotoxic edema, or hemorrhage) were manually annotated as a single object by two raters from site 1 (D.P. and L.F.), one rater from site 2 (A.C.S.), and by two raters (E.N.K. and J.V.) for site 3 using JIM (version 7.0; Xinapse Systems) and FSLeyes (version 1.9.0; University of Oxford) image viewers. As obtaining the reference standard spinal cord segmentation masks using a fully manual approach is time-consuming, sct_deepseg_sc … was used to initially segment the spinal cord of patients from all three sites, followed by manual corrections wherever necessary. This semiautomatic approach was also used in previous studies …. [Materials and methods (11), CC BY 4.0] |
| Follow-up information was acquired by matching with two validated national breast cancer and screening databases … accessed through the Danish Clinical Quality Program–National Clinical Registries (ie, RKKP). Positive and negative cancer outcomes were determined through histopathology-verified breast cancer or cancer-free follow-up, respectively, up until the next consecutive screening or within 24 months. [Materials and methods (22), CC BY 4.0] |