Impact of Human-AI Interaction on Diagnostic Accuracy, Confidence and Decision-Making in Cardiac Amyloidosis Artificial Intelligence Cardiac Magnetic Resonance Clinical Decision Support

Presenting Author(s)

• 

  David Chen, PhD

  Director of Artificial Intelligence
  Cleveland Clinic
  Cleveland, Ohio, United States

Primary Author(s)

• MN

  Makiya Nakashima, MSc

  Research Data Scientist II
  Cleveland Clinic
  Cleveland, Ohio, United States

Co-Author(s)

• RK

  Rishabh Kharana, MD

  Staff Physician
  Cleveland Clinic, United States
• NG

  Neha Gupta

  Physician
  Cleveland Clinic, United States
• 

  Shaoxiong Zhang, MD, PhD

  Staff Radiologist
  Cleveland Clinic
  Cleveland, Ohio, United States
• MB

  Michael A. Bolen, MD

  Staff Radiologist, Co-Section chief of CVI
  Cleveland Clinic
  Pepper Pike, Ohio, United States
• BA

  Ben Alencherry, MD

  Fellow
  Cleveland Clinic, Ohio, United States
• PC

  Po-Hao Chen, MD, MBA

  Staff
  Cleveland Clinic, United States
• RG

  Richard Grimm

  Staff
  Cleveland Clinic, United States
• TK

  Tore A. Klock

  Senior Software Developer
  Cleveland Clinic
  Cleveland, Ohio, United States
• 

  Christopher Nguyen, PhD, FSCMR

  Director, Cardiovascular Innovation Research Center
  Cleveland Clinic
  Cleveland, Ohio, United States
• MH

  Mazen Hanna, MD

  Clinician
  Cleveland Clinic
  Cleveland, Ohio, United States
• 

  Deborah Kwon, MD, FSCMR

  Director of Cardiac MRI
  Cleveland Clinic
  Cleveland, Ohio, United States

Background: Differentiating cardiac amyloidosis (CA) from hypertrophic cardiomyopathy (HCM) and other causes of left ventricular hypertrophy on cardiovascular magnetic resonance (CMR) can be challenging[1]. While artificial intelligence (AI)-based clinical decision support (CDS) tools have been shown to achieve near-human or above-human discriminative ability for identifying CA[2-4], little work has been done to examine the human-AI interaction and how exposure to such AI-CDS tools impacts human CA diagnostic accuracy and confidence. Therefore, we evaluate the impact of a CA detection AI-CDS tool with an AUC of 0.954 on diagnostic accuracy, reader confidence, and inter-reader agreement.

Methods: MRI acquisition parameters and clinical indications for all cases were made available to readers. One hundred CMR studies (50 CA, 30 HCM, 20 other cardiomyopathies) were independently interpreted in three sequential stages by four CMR physicians: (1) Cine and late gadolinium enhancement (LGE) images only (no clinical history/EKG), (2) reviewers were then shown the AI model’s probabilistic predictions for CA, HCM, and other cardiomyopathies, and (3) T1 mapping and measurements were provided. For each stage, readers provided their diagnosis and graded their level of confidence. We assessed diagnostic accuracy, AI impact (remained correct, remained wrong, correct-to-wrong, and wrong-to-correct), confidence distribution, physician perception of AI feasibility, and inter-rater agreement. The AI-CDS tool employed a space-time vision transformer model trained on 561 CMR studies (155 CA, 205 HCM, and 201 other cardiomyopathies) with cine short-axis images with temporal resolution and LGE short-axis stacks with spatial depth, concatenated into video sequences.

Results: The initial mean reader accuracy was 74%. Following AI assistance, accuracy rose to 77% reflecting diagnostic revisions in 6.8 % of cases and 5.0 % improvement in correct classifications. Incorporation of T1 mapping further elevated overall accuracy to 84.0%. The AI model achieved 84.0% accuracy independent of reader, and among the remaining misclassified cases, it correctly identified 54.8%. Inter-observer agreement improved from κ = 0.72 with Cine + LGE alone, to κ = 0.74 after AI guidance, and reached κ = 0.84 when T1 mapping was added. Readers initially reported moderate confidence in 43.5 % of cases and high confidence in 25.5 %. AI-CDS tools most improved confidence in difficult cases, raising confidence in 83.3% of low-confidence and 64.5% of low-moderate confidence diagnoses. For high-confidence cases, the improvement was 2.9%.

Conclusion: Integration of AI decision support with human interpretation slightly improves human diagnostic accuracy, although not to the level of including T1 mapping. AI-CDS improved reader confidence and agreement, suggesting these tools could enhance decision-making, facilitate standardization, and support less experienced readers in complex CMR interpretation.

Overview of study design and evaluation process. Data included 50 cases of cardiac amyloidosis (CA), 30 cases of hypertrophic cardiomyopathy (HCM), and 20 cases of other cardiomyopathies, reviewed by four human evaluators. The workflow comprised three steps: (1) Cine and LGE image review (2) AI supported diagnosis and (3) T1 mapping and measurement.
Performance across diagnostic stages. a) Progression of diagnostic accuracy from initial diagnosis to AI-assisted review and T1 mapping, compared with AI model performance alone. b) Impact of AI support on diagnostic outcomes. c) AI accuracy in remained wrong cases.
Inter-rater agreement and confidence metrics. a) Average inter-rater agreement at each diagnostic stage (b) Distribution of initial diagnostic confidence levels. c) Impact of AI support on confidence.