EUVIMEDOpdag EUVIMED
Open access

A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study

Gerald Gui Ren Sng iD, Yi Xiang iD, Daniel Yan Zheng Lim iD, Joshua Yi Min Tung iD, Jen Hong Tan iD, Chiaw Ling Chng iD

JMIR Formative Research · 2025

DOI: 10.2196/70863 ↗

At a glance

Journal article in JMIR Formative Research, 2025. Written by Gerald Gui Ren Sng and 5 co-authors. Institution: Department of Endocrinology, Singapore General Hospital, 20 College Road, Academia Level 3, Singapore, 169856, Singapore, 65 63214377. Major MEDLINE topics: Thyroid Nodule, Artificial Intelligence, Thyroid Neoplasms. Cited 6 times so far (Crossref); 37 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Thyroid nodules are common, with ultrasound imaging as the primary modality for their assessment. Risk stratification systems like the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) have been developed but suffer from interobserver variability and low specificity. Artificial intelligence, particularly large language models (LLMs) with multimodal capabilities, presents opportunities for efficient end-to-end diagnostic processes. However, their clinical utility remains uncertain.

Objective

This study evaluates the accuracy and consistency of multimodal LLMs for thyroid nodule risk stratification using the ACR TI-RADS system, examining the effects of model fine-tuning, image annotation, prompt engineering, and comparing open-source versus commercial models.

Methods

In total, 3 multimodal vision-language models were evaluated: Microsoft’s open-source Large Language and Visual Assistant (LLaVA) model, its medically fine-tuned variant (Large Language and Vision Assistant for bioMedicine [LLaVA-Med]), and OpenAI’s commercial o3 model. A total of 192 thyroid nodules from publicly available ultrasound image datasets were assessed. Each model was evaluated using 2 prompts (basic and modified) and 2 image scenarios (unlabeled vs radiologist-annotated), yielding 6912 responses. Model outputs were compared with expert ratings for accuracy and consistency. Statistical comparisons included Chi-square tests, Mann-Whitney U tests, and Fleiss’ kappa for interrater reliability.

Results

Overall, 88.4% (6110/6912) of responses were valid, with the o3 model producing the highest validity rate (2273/2304, 98.6%), followed by LLaVA (2108/2304, 91.5%) and LLaVA-Med (1729/2304, 75%; P

Subjects

MeSH terms (MEDLINE)

Thyroid NoduleArtificial IntelligenceThyroid NeoplasmsHumansUltrasonographyRisk AssessmentFemaleReproducibility of ResultsLarge Language Models

Author keywords

artificial intelligencelarge language modelsmultimodalrisk stratification systemsthyroid nodules

Authors and institutions

  • Department of Endocrinology, Singapore General Hospital, 20 College Road, Academia Level 3, Singapore, 169856, Singapore, 65 63214377.
  • Data Science and Artificial Intelligence Laboratory, Singapore General Hospital, Singapore, Singapore.
  • Office of Insights and Analytics, SingHealth, Singapore, Singapore.
  • Department of Gastroenterology, Singapore General Hospital, Singapore, Singapore.
  • Department of Urology, Singapore General Hospital, Singapore, Singapore.

ORCID profiles: Gerald Gui Ren Sng, Yi Xiang, Daniel Yan Zheng Lim, Joshua Yi Min Tung, Jen Hong Tan, Chiaw Ling Chng

Cite

APA

Sng, G. G. R., Xiang, Y., Lim, D. Y. Z., Tung, J. Y. M., Tan, J. H., & Chng, C. L. (2025). A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study. JMIR Formative Research. https://doi.org/10.2196/70863

Vancouver

Sng GGR, Xiang Y, Lim DYZ, Tung JYM, Tan JH, Chng CL. A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study. JMIR Formative Research. 2025. doi:10.2196/70863

BibTeX
@article{Sng2025,
  author = {Sng, Gerald Gui Ren and Xiang, Yi and Lim, Daniel Yan Zheng and Tung, Joshua Yi Min and Tan, Jen Hong and Chng, Chiaw Ling},
  title = {A Multimodal Large Language Model as an End-to-End Classifier of Thyroid Nodule Malignancy Risk: Usability Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/70863}
}
Start for free →