Item 15 — Rationale for choosing the reference standard
Describe the rationale for choice of the reference standard versus any alternatives. Include information on potential errors, biases, and limitations of that reference standard.
Explanation and Elaboration: A good reference standard should minimize error and ambiguity, and align with the study’s objectives. For example, in segmentation tasks, if the task is straightforward then the reference standard is often manual segmentation by an expert. If the segmentation task is challenging, or the decisions which it informs require high precision, then multiple human experts may be warranted. Alternatively, in cases where both the imaging system and segmentation contribute to error, or the contribution of each is unknown, a more rigorous evaluation may be necessary. This may be the case when an imaging modality is being used to measure a new tissue of interest it is not validated for. In this case, the reference standard may require physical measurement methods, such as dissection and direct tissue volume measurement, to establish a more objective reference. This principle applies broadly across AI research: when the chosen reference standard may obscure the contributions of different error sources or lacks precision, alternative or supplementary methods should be employed.
| Examples |
| With no society recommendations on how to annotate and track the MV points, the Yale and Lund datasets were annotated according to two different principles. A single observer at Yale University (Yale observer) placed the MV points in the intersection between the MV and the LV myocardium, whereas the 9 observers at Lund University (Lund observers) placed the MV points at the most basal part of the compact LV myocardium. Both ways yielded similar MV motions but with a different reference point. However, due to the inherent observer bias and the high number of Lund observers, these annotations presented more variability. Such differences in manual annotation are very common in deep learning applications. [Methods, manual annotation section (35), CC BY 4.0] |
| Currently, manual segmentation is considered the reference standard for planimetric and volumetric measurements of brain structures at MRI. However, this approach is time- and resource-demanding and prone to inter- and intrarater variability, especially for small brain regions, thus limiting the usability of the above-mentioned neuroimaging biomarkers in routine clinical practice …. To overcome these issues, several studies have developed semi- or fully automated segmentation methods, adopting threshold- and landmark-based approaches …, as well as atlas-based segmentation strategies …. This decade, deep learning (DL) techniques have also been employed for the rapid and accurate segmentation of brain regions …. DL models learn to capture relevant features from raw images through a sequence of contracting and expanding paths of convolutions (encoder and decoder components) and generate as final output the desired segmentation masks …. [Introduction (36), CC BY 4.0] |