EUVIMEDEUVIMED entdecken
Open Access

A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study

Gerald Gui Ren Sng iD, Yi Xiang iD, Daniel Yan Zheng Lim iD, Joshua Yi Min Tung iD, Jen Hong Tan iD, Chiaw Ling Chng iD

JMIR Formative Research · 2025

DOI: 10.2196/70863 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Gerald Gui Ren Sng und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Endocrinology, Singapore General Hospital, 20 College Road, Academia Level 3, Singapore, 169856, Singapore, 65 63214377. Hauptthemen laut MEDLINE: Thyroid Nodule, Artificial Intelligence, Thyroid Neoplasms. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Thyroid nodules are common, with ultrasound imaging as the primary modality for their assessment. Risk stratification systems like the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) have been developed but suffer from interobserver variability and low specificity. Artificial intelligence, particularly large language models (LLMs) with multimodal capabilities, presents opportunities for efficient end-to-end diagnostic processes. However, their clinical utility remains uncertain.

Objective

This study evaluates the accuracy and consistency of multimodal LLMs for thyroid nodule risk stratification using the ACR TI-RADS system, examining the effects of model fine-tuning, image annotation, prompt engineering, and comparing open-source versus commercial models.

Methods

In total, 3 multimodal vision-language models were evaluated: Microsoft’s open-source Large Language and Visual Assistant (LLaVA) model, its medically fine-tuned variant (Large Language and Vision Assistant for bioMedicine [LLaVA-Med]), and OpenAI’s commercial o3 model. A total of 192 thyroid nodules from publicly available ultrasound image datasets were assessed. Each model was evaluated using 2 prompts (basic and modified) and 2 image scenarios (unlabeled vs radiologist-annotated), yielding 6912 responses. Model outputs were compared with expert ratings for accuracy and consistency. Statistical comparisons included Chi-square tests, Mann-Whitney U tests, and Fleiss’ kappa for interrater reliability.

Results

Overall, 88.4% (6110/6912) of responses were valid, with the o3 model producing the highest validity rate (2273/2304, 98.6%), followed by LLaVA (2108/2304, 91.5%) and LLaVA-Med (1729/2304, 75%; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Thyroid NoduleArtificial IntelligenceThyroid NeoplasmsHumansUltrasonographyRisk AssessmentFemaleReproducibility of ResultsLarge Language Models

Stichwörter der Autor:innen

artificial intelligencelarge language modelsmultimodalrisk stratification systemsthyroid nodules

Autor:innen und Institutionen

  • Department of Endocrinology, Singapore General Hospital, 20 College Road, Academia Level 3, Singapore, 169856, Singapore, 65 63214377.
  • Data Science and Artificial Intelligence Laboratory, Singapore General Hospital, Singapore, Singapore.
  • Office of Insights and Analytics, SingHealth, Singapore, Singapore.
  • Department of Gastroenterology, Singapore General Hospital, Singapore, Singapore.
  • Department of Urology, Singapore General Hospital, Singapore, Singapore.

ORCID-Profile: Gerald Gui Ren Sng, Yi Xiang, Daniel Yan Zheng Lim, Joshua Yi Min Tung, Jen Hong Tan, Chiaw Ling Chng

Zitieren

APA

Sng, G. G. R., Xiang, Y., Lim, D. Y. Z., Tung, J. Y. M., Tan, J. H., & Chng, C. L. (2025). A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study. JMIR Formative Research. https://doi.org/10.2196/70863

Vancouver

Sng GGR, Xiang Y, Lim DYZ, Tung JYM, Tan JH, Chng CL. A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study. JMIR Formative Research. 2025. doi:10.2196/70863

BibTeX
@article{Sng2025,
  author = {Sng, Gerald Gui Ren and Xiang, Yi and Lim, Daniel Yan Zheng and Tung, Joshua Yi Min and Tan, Jen Hong and Chng, Chiaw Ling},
  title = {A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/70863}
}
Kostenlos starten →