The impact of label uncertainty on artificial intelligence development on the example of late gadolinium enhancement artefact classifiers

Background: Artificial intelligence (AI) increasingly automates medical imaging tasks such as segmentation and classification (1,2). However, defining ground truth labels can be difficult and may be influenced by inter-reader variability. These inter-reader differences and label uncertainties may be integrated into datasets such that they bias AI tools or manifest unclear decision criteria within the dataset and in consequence the AI tools (3–5).



The AIM of this work is to quantify the impact of different labels and inter-reader agreement on AI development and testing results in CMR.

Methods: 1411 late gadolinium enhancement MagIR short axis images from 161 cases were labeled independently by two readers (reader A, reader B) as ‘no artifact’, ‘infolding’, and ‘motion’. Only artifacts overlapping the left myocardium were labeled (Fig. 1). The label ‘artifact’ is the union of the labels ‘infolding’ and ‘motion’. Reader AB is the intersection of the labels where A and B agree. Label distribution by reader is presented in Fig. 1. Accuracy was calculated to assess inter-reader agreement.

The cases were randomly separated into training, validation and test datasets (112/24/25 cases respectively). Convolutional neural networks (CNN) with equivalent parameters were trained for three classification scenarios: i) classify an image as ‘no artifact’ or ‘artifact’, ii) classify an image as ‘no artifact’ or ‘infolding’ and iii) classify an image as ‘no artifact’ or ‘infolding’ or ‘motion’, with ‘no artifact’ always being exclusive. For each scenario labels of readers A, B and AB were used to train CNN^A, CNN^B and CNN^AB respectively leading to nine CNNs.

Recall (correctly classified images per labeled images per class) was calculated to evaluate the readers and the CNNs on the test dataset for labels A, B and AB respectively.

Results: Inter-reader accuracy between reader A and B was 55.6% (785/1411 images). Qualitative investigation revealed that disagreement largely stemmed from deciding whether the artifact affected the myocardium (no artifact ⟷ artifact) and only minorly from differentiating between artifact types (see Fig. 1).

Averaged class-wise recall of the CNNs ranged between 55.4% (CNN^A on B for no artifact – infolding – motion) and 92.2% (CNN^B on AB for no artifact – infolding) depending on the test data (Fig. 3). Recall analysis shows that infoldings were easier to detect than motion for human and AI readers alike (Fig. 3). In all scenarios testing the AI only on images with agreement of labels increased the mean recall. Indicating that classifying images with lack of inter-reader agreement might also be more challenging for AIs.

Conclusion: This experiment highlights the impact of label uncertainties on developing AIs as the mean recalls differ significantly between different labels. Considering only images with inter-reader agreement in training can increase AI reliability. Identifying inconsistent or misleading labels by inter-reader reproducibility assessments can support stronger AI tools.

Figure 1: Selected late gadolinium enhancement sax example images of the dataset. Images a, b, c were labeled as ‘no artifact’ by reader A and B consistently. Images d, e, f were consistently labeled as ‘motion’ by reader A and B, the yellow arrows point out the motion impact on the myocardium. Images g, h, i were consistently labeled as ‘infolding’ by reader A and reader B with red arrows pointing out the infolding on the myocardium. Image j was labeled as ‘no artifact’ by reader A and ‘motion’ by reader B (yellow-white striped arrow), displaying the disagreement on the severity of the motion artifact required to affect the postprocessing. Image k was once labeled as ‘no artifact’ and once as ‘infolding’, as there was a disagreement if the infolding in the image (red-white striped arrow) affects the myocardium. Image l was consistently labeled as ‘artifact’, but one reader labeled the source of the artifact as ‘infolding’ and one as ‘motion’ (yellow-red striped arrow).
Figure 2: Label distribution by number of images per class and reader.
Figure 3: Resulting mean recall in percent for the nine CNNs. Scenario i) shows the binary classifier for ‘no artifact’ vs ‘artifact’, scenario ii) shows the classifier for differentiating ‘no artifact’ from ‘infolding’ and scenario iii) the multi-label approach for ‘no artifact’, ‘infolding’ and ‘motion’. The letter at the CNN indicates on which labels the CNN was trained and validated, the different columns provide which labels were used for testing the CNN. The two lower rows in the three subfigures show how reader A performed compared to ground truth labels B and vice versa for all scenarios. Note that the inter-reader mean recall of reader A and reader B for scenarios i) and iii) is similar to the performance of CNNA and CNNB respectively — that is, CNNA is compared to reader A, and CNNB to reader B. However, in scenario ii), CNNA outperforms reader A, and CNNB outperforms reader B when classifying infoldings versus artefact-free images, even though both CNNs were trained on labels from the respective human reader.