EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Enhancing Precision in Detecting Severe Immune-Related Adverse Events: Comparative Analysis of Large Language Models and International Classification of Disease Codes in Patient Records

Verbesserung der Präzision beim Erkennen schwerer immunvermittelter unerwünschter Ereignisse: Vergleich von Large Language Models und ICD-Codes in Patient:innenakten

Virginia H Sun iD, Julius C Heemelaar iD, Ibrahim Hadzic iD, Vineet K Raghu iD, Chia-Yun Wu, Leyre Zubiri iD, Azin Ghamari, Nicole R LeBoeuf iD, Osama Abu-Shawer iD, Kenneth L Kehl iD, Shilpa Grover iD, Prabhsimranjot Singh und 18 weitere

Journal of Clinical Oncology · 10.12.2024 · 42 (35) · 4134-4144

DOI: 10.1200/jco.24.00326 ↗

Auf einen Blick

In 7.555 Aufnahmen von Patient:innen unter ICI-Therapie erkannte ein LLM irAEs sensitiver als ICD-Codes bei ähnlicher Spezifität. Die Validierung zeigte ebenfalls hohe Sensitivität und Spezifität.

Fragestellung

Wie gut erkennt ein LLM schwere irAEs bei hospitalisierten Patient:innen unter ICI-Therapie im Vergleich zu ICD-Codes und manueller Adjudikation?

Methode

Einzentrische Auswertung von Krankenhausaufnahmen mit manueller Adjudikation als Goldstandard; Vergleich von ICD-Codes und einem LLM mit retrieval-augmented generation. Externe Validierung an zweitem Zentrum.

Ergebnisse

Initialkohorte: 7.555 Aufnahmen; 2,0 % Kolitis, 1,1 % Hepatitis, 0,7 % Pneumonitis, 0,8 % Myokarditis. LLM: Sensitivität 94,7 % vs. 68,7 % bei ICD-Codes; Spezifität 93,7 % vs. 92,4 %. Validierung (N = 1.270): Sensitivität 98,1 %, Spezifität 95,7 %.

Schlussfolgerung der Autor:innen

LLMs sind ein nützliches Instrument zur Erkennung von irAEs und waren ICD-Codes in der Sensitivität sowie der Effizienz der Adjudikation überlegen.

Kernaussagen

  • LLM-Sensitivität höher als bei ICD-Codes: 94,7 % vs. 68,7 %
  • Spezifität ähnlich: 93,7 % vs. 92,4 %
  • Validierung: Sensitivität 98,1 %, Spezifität 95,7 %

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Clinical Oncology, erschienen am 10.12.2024. Verfasst von Virginia H Sun und 29 weiteren Autor:innen. Beteiligte Einrichtung: Harvard Medical School, Boston, MA. Hauptthemen laut MEDLINE: International Classification of Diseases, Immune Checkpoint Inhibitors. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIDDK NIH HHS, NHLBI NIH HHS.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

PURPOSE Current approaches to accurately identify immune-related adverse events (irAEs) in large retrospective studies are limited. Large language models (LLMs) offer a potential solution to this challenge, given their high performance in natural language comprehension tasks. Therefore, we investigated the use of an LLM to identify irAEs among hospitalized patients, comparing its performance with manual adjudication and International Classification of Disease (ICD) codes. METHODS Hospital admissions of patients receiving immune checkpoint inhibitor (ICI) therapy at a single institution from February 5, 2011, to September 5, 2023, were individually reviewed and adjudicated for the presence of irAEs. ICD codes and an LLM with retrieval-augmented generation were applied to detect frequent irAEs (ICI-induced colitis, hepatitis, and pneumonitis) and the most fatal irAE (ICI-myocarditis) from electronic health records. The performance between ICD codes and LLM was compared via sensitivity and specificity with an α = .05, relative to the gold standard of manual adjudication. External validation was performed using a data set of hospital admissions from June 1, 2018, to May 31, 2019, from a second institution. RESULTS Of the 7,555 admissions for patients on ICI therapy in the initial cohort, 2.0% were adjudicated to be due to ICI-colitis, 1.1% ICI-hepatitis, 0.7% ICI-pneumonitis, and 0.8% ICI-myocarditis. The LLM demonstrated higher sensitivity than ICD codes (94.7% v 68.7%), achieving significance for ICI-hepatitis ( P < .001), myocarditis ( P < .001), and pneumonitis ( P = .003) while yielding similar specificities (93.7% v 92.4%). The LLM spent an average of 9.53 seconds/chart in comparison with an estimated 15 minutes for adjudication. In the validation cohort (N = 1,270), the mean LLM sensitivity and specificity were 98.1% and 95.7%, respectively. CONCLUSION LLMs are a useful tool for the detection of irAEs, outperforming ICD codes in sensitivity and adjudication in efficiency.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

International Classification of DiseasesImmune Checkpoint InhibitorsHumansNatural Language ProcessingRetrospective StudiesElectronic Health RecordsFemaleMaleMiddle AgedAgedNeoplasms

Substanzen

Immune Checkpoint Inhibitors

Autor:innen und Institutionen

  • Harvard Medical School, Boston, MA.
  • Cardiovascular Imaging Research Center, Massachusetts General Hospital, Boston, MA.
  • Leiden University Medical Center, Leiden, the Netherlands.
  • Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Boston, MA.
  • Brigham and Women's Hospital, Boston, MA.
  • Maastricht University, Maastricht, the Netherlands.

ORCID-Profile: Virginia H Sun, Julius C Heemelaar, Ibrahim Hadzic, Vineet K Raghu, Leyre Zubiri, Nicole R LeBoeuf, Osama Abu-Shawer, Kenneth L Kehl, Shilpa Grover, Giselle A Suero-Abreu, Jessica Wu, Ayo S Falade, Nora Hathaway, Benjamin D Medoff, Alexandra-Chloe Villani, Meghan J Mooradian, Meghan E Sise, Daniel A Zlotoff, Steven M Blum, Michael Dougan, Tomas G Neilan, Kerry L Reynolds

Förderung

  • NIDDK NIH HHS
  • NHLBI NIH HHS

Zitieren

APA

Sun, V. H., Heemelaar, J. C., Hadzic, I., Raghu, V. K., Wu, C. Y., Zubiri, L., Ghamari, A., LeBoeuf, N. R., Abu-Shawer, O., Kehl, K. L., Grover, S., Singh, P., Suero-Abreu, G. A., Wu, J., Falade, A. S., Grealish, K., Thomas, M. F., Hathaway, N., Medoff, B. D., …, & Reynolds, K. L. (2024). Enhancing Precision in Detecting Severe Immune-Related Adverse Events: Comparative Analysis of Large Language Models and International Classification of Disease Codes in Patient Records. Journal of Clinical Oncology, 42(35), 4134-4144. https://doi.org/10.1200/jco.24.00326

Vancouver

Sun VH, Heemelaar JC, Hadzic I, Raghu VK, Wu CY, Zubiri L, et al. Enhancing Precision in Detecting Severe Immune-Related Adverse Events: Comparative Analysis of Large Language Models and International Classification of Disease Codes in Patient Records. Journal of Clinical Oncology. 2024;42(35):4134-4144. doi:10.1200/jco.24.00326

BibTeX
@article{Sun2024,
  author = {Sun, Virginia H and Heemelaar, Julius C and Hadzic, Ibrahim and Raghu, Vineet K and Wu, Chia-Yun and Zubiri, Leyre and Ghamari, Azin and LeBoeuf, Nicole R and Abu-Shawer, Osama and Kehl, Kenneth L and Grover, Shilpa and Singh, Prabhsimranjot and Suero-Abreu, Giselle A and Wu, Jessica and Falade, Ayo S and Grealish, Kelley and Thomas, Molly F and Hathaway, Nora and Medoff, Benjamin D and Gilman, Hannah K and Villani, Alexandra-Chloe and Ho, Jor Sam and Mooradian, Meghan J and Sise, Meghan E and Zlotoff, Daniel A and Blum, Steven M and Dougan, Michael and Sullivan, Ryan J and Neilan, Tomas G and Reynolds, Kerry L},
  title = {Enhancing Precision in Detecting Severe Immune-Related Adverse Events: Comparative Analysis of Large Language Models and International Classification of Disease Codes in Patient Records},
  journal = {Journal of Clinical Oncology},
  year = {2024},
  volume = {42},
  number = {35},
  pages = {4134--4144},
  doi = {10.1200/jco.24.00326}
}
Kostenlos starten →