EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis

Li Zeng, Ellen Huang, Yizheng Huang

Natural Language Processing Journal · 2026-12 · 17 · 100237

DOI: 10.1016/j.nlp.2026.100237 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Natural Language Processing Journal, 2026-12. Verfasst von Li Zeng und 2 weiteren Autor:innen. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Sciences and Engineering Research Council of Canada.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To systematically assess the accuracy, completeness, and safety of information provided by three mainstream LLMs (ChatGPT-5, DeepSeek-V3.2, Gemini-2.5 Flash) regarding Hashimoto’s Thyroiditis (HT), a set of standardized clinical questions was designed and posed to the LLMs (Foundational Knowledge, Clinical Presentation and Diagnosis, Treatment and Management). Two independent endocrinologists blinded to the model sources evaluated each response. Using authoritative references from the National Institutes of Health (NIH) and the American Thyroid Association (ATA) as the gold standard, they scored each answer on factual accuracy, completeness, medical safety, and source support. The weighted total scores and rankings of the three LLMs across all metrics were calculated: Gemini-2.5 Flash (93.5) > DeepSeek-V3.2 (88) > ChatGPT-5 (85). Comparison of normalized item scores showed that Gemini-2.5 Flash scored significantly higher than both other models, whereas DeepSeek-V3.2 and ChatGPT-5 did not differ significantly. Qualitative analysis identified common strengths (e.g., chain-of-thought process, proactive engagement, generally accurate and comprehensive responses) and weaknesses (e.g., outdated or non-existent information, omission of key points, absolute claims lacking sufficient evidence). Specific LLMs demonstrate potential in providing information on HT; however, the presence of inconsistencies and risks necessitates their cautious use.

Förderung

  • Natural Sciences and Engineering Research Council of Canada

Zitieren

APA

Zeng, L., Huang, E., & Huang, Y. (2026). How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis. Natural Language Processing Journal, 17, 100237. https://doi.org/10.1016/j.nlp.2026.100237

Vancouver

Zeng L, Huang E, Huang Y. How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis. Natural Language Processing Journal. 2026;17:100237. doi:10.1016/j.nlp.2026.100237

BibTeX
@article{Zeng2026,
  author = {Zeng, Li and Huang, Ellen and Huang, Yizheng},
  title = {How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis},
  journal = {Natural Language Processing Journal},
  year = {2026},
  volume = {17},
  pages = {100237},
  doi = {10.1016/j.nlp.2026.100237}
}
Kostenlos starten →