EUVIMEDEUVIMED entdecken
Vergleichsstudie

Putting AI Chatbots to the Test: A Comparative Analysis of Large Language Models' Performance in the Context of Basal Cell Carcinoma

Vinzent K Ortner iD, Heather C Lent, Katrine E Karmisholt, Ulrikke Lei, Mette Gyldenløve, Silje H Omland, Diljit Kaur-Knudsen, Kati H Kainu, Katrine Togsverd-Bo, Stine R Wiegell, Johannes Bjerva, Merete Hædersdal

Experimental Dermatology · 2025

DOI: 10.1111/exd.70175 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Experimental Dermatology, 2025. Verfasst von Vinzent K Ortner und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Dermatology, Copenhagen University Hospital-Bispebjerg and Frederiksberg, Copenhagen, Denmark. Hauptthemen laut MEDLINE: Basal Cell Carcinoma, Skin Neoplasms, Language, Artificial Intelligence. Die Arbeit nennt 22 Literaturangaben. Gefördert von: Carlsbergfondet, Danish Research Center For Skin Cance, Region Hovedstadens Forskningsfond.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have been explored in various dermato‐oncological conditions. In this study, we aimed to compare different LLMs' potential to guide clinicians on the treatment of basal cell carcinoma (BCC). Four authors formulated 24 questions on the topic of clinical management of BCC. The blinded responses of three LLMs (Gemini, Copilot and ChatGPT 4.0) were presented to a panel of nine dermato‐oncologists for assessment of (i) factual accuracy, (ii) concision, (iii) comprehensiveness and (iv) overall preference. In addition, the responses were then quantitatively compared based on lexical (i.e., vocabulary) and semantic (i.e., meaning) similarity to three additional LLMs (ChatGPT 3.5, ChatGPT 4o and Claude). ChatGPT 4.0 had the highest accuracy rate (87.5%, i.e., 21/24 responses), followed by Gemini (50%) and Copilot (25%). All models scored lower for concision and comprehensiveness, with ChatGPT 4.0 in the lead (62.5% comprehensive; 54.2% concise), followed by Gemini (33.3%; 12.5%) and Copilot (16.7%; 8.3%). The panel achieved consensus on model preference in 16 questions (ChatGPT 4.0: 54.2%; Gemini: 8.3%; Copilot: 4.2%; no consensus: 33.3%). While the lexical similarity was found to be low (x̄ ~0.07–0.10 across models), the semantic similarity between the LLM responses was moderate (x̄ ~0.60–0.70 across models). LLMs may assist clinicians in settings where expert dermato‐oncological guidance is not readily available, with ChatGPT 4.0 currently outperforming both Gemini and Copilot. Since quantitative methods are unable to detect clinically relevant differences between LLMs, surveying dermatologists is necessary to identify useful models in this rapidly developing field.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Basal Cell CarcinomaSkin NeoplasmsLanguageArtificial IntelligenceHumansSemanticsLarge Language ModelsGenerative Artificial Intelligence

Autor:innen und Institutionen

  • Department of Dermatology, Copenhagen University Hospital-Bispebjerg and Frederiksberg, Copenhagen, Denmark.
  • Department of Dermatology and Allergy, Copenhagen University Hospital-Herlev and Gentofte, Hellerup, Denmark.
  • Department of Computer Science, Aalborg University, Copenhagen, Denmark.
  • Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark.

ORCID-Profile: Vinzent K Ortner

Förderung

  • Carlsbergfondet
  • Danish Research Center For Skin Cance
  • Region Hovedstadens Forskningsfond

Zitieren

APA

Ortner, V. K., Lent, H. C., Karmisholt, K. E., Lei, U., Gyldenløve, M., Omland, S. H., Kaur-Knudsen, D., Kainu, K. H., Togsverd-Bo, K., Wiegell, S. R., Bjerva, J., & Hædersdal, M. (2025). Putting AI Chatbots to the Test: A Comparative Analysis of Large Language Models' Performance in the Context of Basal Cell Carcinoma. Experimental Dermatology. https://doi.org/10.1111/exd.70175

Vancouver

Ortner VK, Lent HC, Karmisholt KE, Lei U, Gyldenløve M, Omland SH, et al. Putting AI Chatbots to the Test: A Comparative Analysis of Large Language Models' Performance in the Context of Basal Cell Carcinoma. Experimental Dermatology. 2025. doi:10.1111/exd.70175

BibTeX
@article{Ortner2025,
  author = {Ortner, Vinzent K and Lent, Heather C and Karmisholt, Katrine E and Lei, Ulrikke and Gyldenløve, Mette and Omland, Silje H and Kaur-Knudsen, Diljit and Kainu, Kati H and Togsverd-Bo, Katrine and Wiegell, Stine R and Bjerva, Johannes and Hædersdal, Merete},
  title = {Putting AI Chatbots to the Test: A Comparative Analysis of Large Language Models' Performance in the Context of Basal Cell Carcinoma},
  journal = {Experimental Dermatology},
  year = {2025},
  doi = {10.1111/exd.70175}
}
Kostenlos starten →