Item 39 — Failure analysis of incorrect results

Provide information to help understand incorrect results. If the task entails classification into two or more categories, provide a confusion matrix that shows tallies for predicted versus actual categories. Consider presenting examples of incorrectly classified cases to help readers better understand the strengths and limitations of the algorithm. Provide sufficient detail to frame incorrect results in the appropriate medical context.

Explanation and Elaboration: The goal of the failure analysis discussion is to provide the reader with some understanding of what types of errors the model is likely to make and, to the extent possible, why it may make those errors. This promotes understanding of the risks of the present model and helps to identify areas for improvement for future models. Expected outcomes for different types of errors (e.g false positive vs false negative) are often different. For each type of error, the clinical implications of that error occurring in the deployed model should be discussed.

Examples

Figure 2: Accuracy of natural language processing score extraction algorithm, as depicted by the confusion matrix for number of clinically significant findings in a radiology report. Evaluated on the manually labeled development dataset. PI-RADS = Prostate Imaging and Reporting Data System. [Figure 2 and its caption (80), CC BY 4.0]

The few failed detections on long-axis test images were due to incorrect imaging planning, unusual LV shapes, or poor image quality. Examples and a discussion of failed detections on long-axis images can be found in Figure E2 (supplement) Figure E2: Examples of failed detection for LAX views. (a) This CH2 cine image contains unusual anatomy, due to congenital heart abnormality of this patient. Model missed two landmarks on this image. (b) This LGE image had very low signal-noise-ratio. The model correctly detected apical point but missed other landmarks. (c) An LGE image had severe aliasing artifacts, causing models to miss all three landmarks. (d) The acquisition plane of this CH4 LGE image was imperfectly placed, causing the model to miss landmarks. Red: manual landmarks; Yellow: model landmarks. [Results, Figure E2 and its caption (81), CC BY 4.0]