CMR-LLAMA: a pre-trained language model for identifying cardiovascular conditions and associated attributes in cardiac MRI text reports

Background: Cardiac magnetic resonance imaging (CMR) reports are an abundant source of information on a patient’s cardiovascular status. If information can be automatically extracted from these free-text reports, this could help with the development of automated clinical decision support tools, enable quality initiatives, and expedite information extraction for research. There have been few studies examining the automatic extraction of CMR data from free-text radiology reports. The studies that do lack extractions including nuances in natural language, such as uncertainty, severity, and anatomical locations of the condition. Furthermore, none of these studies have been able to show external validation of their model onto another hospital system. The goal of this study was to build a large language model to automatically extract cardiovascular conditions and their associated attributes from CMR reports, as well as evaluate model performance on an external validation set.

Methods: We built CMR-LLaMA, a large language model that uses Meta’s pre-trained LLaMA 3.3 with a custom adapter. This model reads in CMR free text and outputs the mentions of 31 cardiovascular conditions and their associated attributes, including certainty, severity, location, and pattern. This model was trained on 1,778 MRI reports and tested on 397 additional reports. We also performed external validation on 428 reports from another hospital campus with independent radiology practice and scanners.

Results: Our model shows robust performance in predicting the mention of the 31 cardiovascular conditions (average F1 = 0.85). It also showed strong performance predicting attributes including certainty (average F1 = 0.97) and severity (average F1 = 0.97). Model performance on the external validation set was generally slightly lower than the internal validation set but performance was still strong (average F1 = 0.78 for mention, 0.97 for certainty, and 0.96 for severity).

Conclusion: CMR-LLaMA has strong performance identifying a variety of condition mentions and a selection of other associated attributes. Furthermore, this model can be applied to another hospital system and still have robust results. Large language models can be used to automate data extraction from CMR reports to potentially assist with clinical and research workflow.

Summary of study methods. Cardiac MRI reports were extracted from patients’ electronic health records from the Cleveland Clinic main campus (N = 2,174). The impression section of these reports were manually tagged by 5 independent annotators for the mentions of 31 cardiovascular conditions and their associated features (certainty, severity, location, and subtype). This data was used to train CMR-LLaMA, a LLaMA-based pretrained language model developed to automate the tagging process of cardiac MRI reports. Model performance was evaluated using F1 score. The model was further evaluated on an external validation set from the Cleveland Clinic Florida hospital system (N = 428). There were no mentions of aortic atherosclerosis and aortic arteritis in our external validation set, so the F1 scores for these two conditions have been set to 0.