How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis
Natural Language Processing Journal · 2026-12 · 17 · 100237
DOI: 10.1016/j.nlp.2026.100237 ↗Kurz gefasst
Fachartikel in der Zeitschrift Natural Language Processing Journal, 2026-12. Verfasst von Li Zeng und 2 weiteren Autor:innen. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Sciences and Engineering Research Council of Canada.
Abstract (Originalsprache)
To systematically assess the accuracy, completeness, and safety of information provided by three mainstream LLMs (ChatGPT-5, DeepSeek-V3.2, Gemini-2.5 Flash) regarding Hashimoto’s Thyroiditis (HT), a set of standardized clinical questions was designed and posed to the LLMs (Foundational Knowledge, Clinical Presentation and Diagnosis, Treatment and Management). Two independent endocrinologists blinded to the model sources evaluated each response. Using authoritative references from the National Institutes of Health (NIH) and the American Thyroid Association (ATA) as the gold standard, they scored each answer on factual accuracy, completeness, medical safety, and source support. The weighted total scores and rankings of the three LLMs across all metrics were calculated: Gemini-2.5 Flash (93.5) > DeepSeek-V3.2 (88) > ChatGPT-5 (85). Comparison of normalized item scores showed that Gemini-2.5 Flash scored significantly higher than both other models, whereas DeepSeek-V3.2 and ChatGPT-5 did not differ significantly. Qualitative analysis identified common strengths (e.g., chain-of-thought process, proactive engagement, generally accurate and comprehensive responses) and weaknesses (e.g., outdated or non-existent information, omission of key points, absolute claims lacking sufficient evidence). Specific LLMs demonstrate potential in providing information on HT; however, the presence of inconsistencies and risks necessitates their cautious use.
Förderung
- Natural Sciences and Engineering Research Council of Canada
Zitieren
APA
Zeng, L., Huang, E., & Huang, Y. (2026). How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis. Natural Language Processing Journal, 17, 100237. https://doi.org/10.1016/j.nlp.2026.100237
Vancouver
Zeng L, Huang E, Huang Y. How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis. Natural Language Processing Journal. 2026;17:100237. doi:10.1016/j.nlp.2026.100237
BibTeX
@article{Zeng2026,
author = {Zeng, Li and Huang, Ellen and Huang, Yizheng},
title = {How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis},
journal = {Natural Language Processing Journal},
year = {2026},
volume = {17},
pages = {100237},
doi = {10.1016/j.nlp.2026.100237}
}
EUVIMED