Generalist deep learning for cross-modality landmark annotation in cardiovascular magnetic resonance

Background: Cardiovascular magnetic resonance (CMR) enables detailed myocardial tissue characterisation through complementary modalities: quantitative T1 mapping for tissue properties, late gadolinium enhancement (LGE) for scar detection, and virtual native enhancement (VNE) as a contrast-free alternative [1]. Standardised analysis by the AHA 16-segment model requires precise annotation of landmarks such as the anterior right ventricular insertion and left ventricular centre, to ensure consistent segmental quantification. Manual annotation is time-consuming and subject to variability. Although recent deep learning approaches have advanced landmark detection, most have been developed for individual modalities and have not been extended to cover ShMOLLI T1 maps or VNE. This restricts their generalisability in clinical practice, where multiple modalities are routinely integrated. This study proposes a generalist deep learning framework that works robustly across CMR modalities, overcoming these limitations.

Methods: This retrospective study included 16,160 short-axis CMR images from 1,299 subjects across 20+ centres, spanning five modalities: native T1 (rest and stress), post-contrast T1, LGE (magnitude and PSIR), and VNE. A dual-stage ResNet-50v2 architecture [2] was developed: Stage 1 generated coarse predictions from preprocessed 384×384 images; Stage 2 refined predictions on spatially normalized 192×192 crops. The model was trained using data augmentation and evaluated on a held-out internal test set of 3,568 images from 257 subjects. Euclidean distance to manual annotations by four trained readers served as the performance metric. Model performance was compared with five specialist models (trained per modality) and one generalist model trained on the merged training data from all modalities. Inter-observer variability was assessed using an external test set of 42 native T1 images annotated by 11 independent observers.

Results: In the cross-modality testing (Figure 1A), the generalist model achieved an error of 2.3 mm (IQR: 1.3–4.5), outperforming all specialist models with errors smaller by a factor of 2.3 (2.0–3.7) overall. For in-distribution data, performance improved by a factor of 1.1 (1.0–1.1); for out-of-distribution data, generalist training improved results by 3.7 (2.4–4.9). In the external test set, inter-observer error was 1.3 mm (0.6–3.2), and the generalist model achieved 1.8 mm (1.1–3.0) on the same images. Accurate annotations (Figure 1B) were generated in 225 ms per sample in batch processing.

Conclusion: A generalist, dual-stage deep learning model can accurately annotate cardiac landmarks across multiple CMR modalities. It outperforms specialist models, especially in cross-modality settings, and approaches expert-level consistency. Its exposure to broader, more varied training data likely enhances generalisation and robustness, offering a scalable solution for standardised myocardial segmentation. This approach may extend to unseen or rare modalities with minimal tuning.

Figure 1. Generalist model performance and representative cross-modality predictions. (A) Euclidean error (mm, median [IQR]) and (B) examples showing predicted (red) vs reference (green) landmarks, in test set across five cardiovascular magnetic resonance modalities.