Performance of an Open-Source Large Language ...
Document type :
Article dans une revue scientifique: Article original
DOI :
PMID :
Permalink :
Title :
Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports
Author(s) :
Le Guellec, Bastien [Auteur]
Departement de Neuroradiologie [Lille]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Lefevre, Alexandre [Auteur]
Departement de Neuroradiologie [Lille]
Geay, Charlotte [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Shorten, Lucas [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Bruge, Cyril [Auteur]
Departement de Neuroradiologie [Lille]
Hacein-Bey, Lotfi [Auteur]
University of California [Davis] [UC Davis]
Amouyel, Philippe [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Facteurs de Risque et Déterminants Moléculaires des Maladies liées au Vieillissement - U 1167 [RID-AGE]
Pruvo, Jean-Pierre [Auteur]
Departement de Neuroradiologie [Lille]
Plateformes Lilloises en Biologie et Santé - UAR 2014 - US 41 [PLBS]
Centre de Recherche Jean-Pierre AUBERT Neurosciences et Cancer - U837 [JPArc]
Kuchcinski, Gregory [Auteur]
Lille Neurosciences & Cognition - U 1172 [LilNCog]
Departement de Neuroradiologie [Lille]
Plateformes Lilloises en Biologie et Santé - UAR 2014 - US 41 [PLBS]
Hamroun, Aghiles [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Facteurs de Risque et Déterminants Moléculaires des Maladies liées au Vieillissement - U 1167 [RID-AGE]
Departement de Neuroradiologie [Lille]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Lefevre, Alexandre [Auteur]
Departement de Neuroradiologie [Lille]
Geay, Charlotte [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Shorten, Lucas [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Bruge, Cyril [Auteur]
Departement de Neuroradiologie [Lille]
Hacein-Bey, Lotfi [Auteur]
University of California [Davis] [UC Davis]
Amouyel, Philippe [Auteur]

Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Facteurs de Risque et Déterminants Moléculaires des Maladies liées au Vieillissement - U 1167 [RID-AGE]
Pruvo, Jean-Pierre [Auteur]

Departement de Neuroradiologie [Lille]
Plateformes Lilloises en Biologie et Santé - UAR 2014 - US 41 [PLBS]
Centre de Recherche Jean-Pierre AUBERT Neurosciences et Cancer - U837 [JPArc]
Kuchcinski, Gregory [Auteur]

Lille Neurosciences & Cognition - U 1172 [LilNCog]
Departement de Neuroradiologie [Lille]
Plateformes Lilloises en Biologie et Santé - UAR 2014 - US 41 [PLBS]
Hamroun, Aghiles [Auteur]
Centre Hospitalier Régional Universitaire [CHU Lille] [CHRU Lille]
Facteurs de Risque et Déterminants Moléculaires des Maladies liées au Vieillissement - U 1167 [RID-AGE]
Journal title :
Radiology: Artificial Intelligence
Pages :
e230364
Publisher :
RSNA
Publication date :
2024-05-08
English keyword(s) :
Large Language Model (LLM)
Generative Pretrained Transformers (GPT)
Open Source
Information Extraction
Report
Brain
MRI
Generative Pretrained Transformers (GPT)
Open Source
Information Extraction
Report
Brain
MRI
HAL domain(s) :
Sciences du Vivant [q-bio]
English abstract : [en]
PurposeTo assess the performance of a local open-source large language model (LLM) in various information extraction tasks from real-life emergency brain MRI reports.Materials and MethodsAll consecutive emergency brain MRI ...
Show more >PurposeTo assess the performance of a local open-source large language model (LLM) in various information extraction tasks from real-life emergency brain MRI reports.Materials and MethodsAll consecutive emergency brain MRI reports written in 2022 from a French quaternary center were retrospectively reviewed. Two radiologists identified MRI scans that were performed in the emergency department for headaches. Four radiologists scored the reports’ conclusions as either normal or abnormal. Abnormalities were labeled as either headache-causing or incidental. Vicuna (LMSYS Org), an open-source LLM, performed the same tasks. Vicuna’s performance metrics were evaluated using the radiologists’ consensus as the reference standard.ResultsAmong the 2398 reports during the study period, radiologists identified 595 that included headaches in the indication (median age of patients, 35 years [IQR, 26–51 years]; 68% [403 of 595] women). A positive finding was reported in 227 of 595 (38%) cases, 136 of which could explain the headache. The LLM had a sensitivity of 98.0% (95% CI: 96.5, 99.0) and specificity of 99.3% (95% CI: 98.8, 99.7) for detecting the presence of headache in the clinical context, a sensitivity of 99.4% (95% CI: 98.3, 99.9) and specificity of 98.6% (95% CI: 92.2, 100.0) for the use of contrast medium injection, a sensitivity of 96.0% (95% CI: 92.5, 98.2) and specificity of 98.9% (95% CI: 97.2, 99.7) for study categorization as either normal or abnormal, and a sensitivity of 88.2% (95% CI: 81.6, 93.1) and specificity of 73% (95% CI: 62, 81) for causal inference between MRI findings and headache.ConclusionAn open-source LLM was able to extract information from free-text radiology reports with excellent accuracy without requiring further training.Show less >
Show more >PurposeTo assess the performance of a local open-source large language model (LLM) in various information extraction tasks from real-life emergency brain MRI reports.Materials and MethodsAll consecutive emergency brain MRI reports written in 2022 from a French quaternary center were retrospectively reviewed. Two radiologists identified MRI scans that were performed in the emergency department for headaches. Four radiologists scored the reports’ conclusions as either normal or abnormal. Abnormalities were labeled as either headache-causing or incidental. Vicuna (LMSYS Org), an open-source LLM, performed the same tasks. Vicuna’s performance metrics were evaluated using the radiologists’ consensus as the reference standard.ResultsAmong the 2398 reports during the study period, radiologists identified 595 that included headaches in the indication (median age of patients, 35 years [IQR, 26–51 years]; 68% [403 of 595] women). A positive finding was reported in 227 of 595 (38%) cases, 136 of which could explain the headache. The LLM had a sensitivity of 98.0% (95% CI: 96.5, 99.0) and specificity of 99.3% (95% CI: 98.8, 99.7) for detecting the presence of headache in the clinical context, a sensitivity of 99.4% (95% CI: 98.3, 99.9) and specificity of 98.6% (95% CI: 92.2, 100.0) for the use of contrast medium injection, a sensitivity of 96.0% (95% CI: 92.5, 98.2) and specificity of 98.9% (95% CI: 97.2, 99.7) for study categorization as either normal or abnormal, and a sensitivity of 88.2% (95% CI: 81.6, 93.1) and specificity of 73% (95% CI: 62, 81) for causal inference between MRI findings and headache.ConclusionAn open-source LLM was able to extract information from free-text radiology reports with excellent accuracy without requiring further training.Show less >
Language :
Anglais
Peer reviewed article :
Oui
Audience :
Internationale
Popular science :
Non
Collections :
Source :
Submission date :
2025-01-24T08:25:54Z
Files
- document
- Open access
- Access the document
- le-guellec-et-al-2024-performance-of-an-open-source-large-language-model-in-extracting-information-from-free-text.pdf
- Open access
- Access the document