Evaluation of Three Artificial Intelligence Solutions for Automated Short-Axis Cine Segmentation in Cardiovascular Magnetic Resonance Imaging

Background: Automated segmentation support tools have entered cardiovascular magnetic resonance (CMR) workflows in clinical routine (1). Reliable clinical parameter quantification requires exact cardiac structure delineation (2). Evidence on the performance of industry-provided artificial intelligence (AI) tools is limited – especially for basal and apical slices, where inaccuracies concentrate. (3). This study compares three industry-provided AIs (two commercial, one research) for short-axis (SAX) cine segmentation across a diverse population.

Methods: The dataset comprised 346 CMR SAX stacks: 120 healthy volunteers and 226 patients with cardiac diseases. Datasets were segmented in end-systole and -diastole (ES, ED) by an expert and the AIs. Clinical parameter differences were assessed for left and right ventricular (LV, RV) volumes, function (SV, EF), and mass (LVM). Differences were presented as bias and boxplots. Segmentation behavior was analyzed by cardiac structure (LV Endo, LV Myo, RV Endo) and region (basal, midventricular, apical). Segmentation decision differences were assessed as false positive and negative rates (FPR, FNR), segmentation differences with Dice and area differences.

Results: Clinical parameter differences are shown in Fig. 1. For all AIs, absolute (abs.) mean differences (Δ) were low for volumes (< 10ml) and LVM (< 10g); function parameters revealed abs. ΔLVSV< 3ml, abs. ΔRVSV< 7ml, and abs. ΔLV/RVEF< 3%. Nonetheless, AIs showed individual biases: AI1 and AI2 overestimated LVM (AI1: Δ+9.4g, AI2: Δ+6g) and RV volumes (AI1 ΔEDV: +6.8ml, AI2 ΔESV: +3.9ml), while AI3 underestimated LV volumes (ΔESV: -6.4ml, ΔEDV: -7.8ml) and ΔRVEDV by -6.6ml. Segmentation decision analysis (Table 1) revealed that AI1 and AI2 contoured too many basal slices (e.g. AI1 LV Myo FPR: 17%, AI2 LV Endo FPR: 12.5%) while AI3 overlooked basal slices (e.g. AI3 LV Endo FNR: 40.5%). Basal slice differences had the largest impacts (abs. Δarea 1.7cm² – 3.9cm²). Midventricular slices had negligible decision differences (FPR< 1%, FNR=0), high Dice averages (Dice >80%), and small abs. Δarea (< 1,7cm², except AI2 RV Endo: 2.5cm²). Apical slices were less reproducible: all AIs overlooked apical slices (FNR >15%) and showed low Dice values (< 65%), however with lower impact (abs. Δarea < 1cm²). Example segmentations are provided in Fig. 2.

Conclusion: Across a diverse cohort, industry-provided AI post-processing solutions demonstrated good segmentation performance. However, relevant differences between AI solutions led to different biases for clinical parameters. Differences are caused by segmentation strategies, and inclusion choices for basal and apical slices, consistent with the surrounding literature (4). All AIs segmented midventricular slices accurately and may be integrated into clinical routine as support tools to amplify efficiency.

Table 1: Segmentation Decision and Quality Metrics

		LV Endo			LV Myo			RV
Position	Metric	AI 1	AI 2	AI 3	AI 1	AI 2	AI 3	AI 1	AI 2	AI 3
Basal	FPR [%]	10.9	12.5	2.4	17.3	23.1	3.1	24.1	13.9	5.4
	FNR [%]	3.5	4.9	40.5	2.9	3.5	39.0	5.3	18.8	31.8
	Dice [%]	76.7	82.9	78.7	64.2	68.6	65.8	69.9	66.4	78.1
	Abs. Area [cm²]	1.7	1.9	2.2	1.5	2.0	1.8	2.6	3.9	1.6
Midv.	FPR [%]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	FNR [%]	0.2	0.2	0.8	0.3	0.4	1.0	0.2	0.4	0.8
	Dice [%]	94.3	93.8	93.2	85.4	83.4	83.3	90.7	87.2	90.9
	Abs. Area [cm²]	0.9	0.9	1.0	1.6	1.5	1.5	1.7	2.5	1.6
Apical	FPR [%]	2.3	2.9	7.8	3.6	2.8	1.9	4.9	11.8	12.3
	FNR [%]	26.9	31.4	22.4	28.3	35.8	37.9	21.8	15.8	17.3
	Dice [%]	57.7	53.6	62.0	52.2	39.5	41.5	56.8	57.5	64.0
	Abs. Area [cm²]	0.9	0.9	0.8	1.3	1.6	1.5	1.1	1.3	1.0

Figure 1: Clinical Parameter Differences: Expert – AI Each subplot shows one clinical parameter for the three AI-expert comparisons, illustrated side by side by a boxplot for quantiles and a stripplot for distribution and outlier visualization. Outliers (abs. difference >50ml) are shown as an * below the AI’s boxplot. Legend: LV: left ventricle, RV: right ventricle, ESV: end-systolic volume, EDV: end-diastolic volume, SV: stroke volume, EF: ejection fraction, LVM: LV myocardial mass
Figure 2: Segmentation Examples Caption: Each three-column subsection shows the base image (first column), the expert and the AI segmentations (columns 2 & 3). Example images show excellent segmentations in the first two rows. But they also include difficult cases, such as artefacts (section 1, row 3), a large apical thrombus (section 2, row 5) and a hypertrophic heart in end-systole (section 3, row 3).