EUVIMEDEUVIMED entdecken
Mixed-Methods-StudieOpen Access

Performance Evaluation of Large Language Models in Multilingual Medical Multiple-Choice Questions: Mixed Methods Study

Livia Maria Strasser iD, Wilma Anschuetz iD, Fabio Dennstädt iD, Janna Hastings iD

JMIR Medical Education · 05.03.2026 · 12 · e81399-e81399

DOI: 10.2196/81399 ↗

Kurz gefasst

Mixed-Methods-Studie in der Zeitschrift JMIR Medical Education, online erschienen am 05.03.2026. Verfasst von Livia Maria Strasser und 3 weiteren Autor:innen. Beteiligte Einrichtung: Medical Knowledge and Decision Support, School of Medicine, University of St.Gallen, St.Jakobstrasse 21, St.Gallen, 9000, Switzerland, +41 71 224 32 …. Hauptthemen laut MEDLINE: Large Language Models, Multilingualism, Educational Measurement. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence continues to transform health care, offering promising applications in clinical practice and medical education. While large language models (LLMs), as a form of generative artificial intelligence, have shown potential to match or surpass medical students in licensing examinations, their performance varies across languages. Recent studies highlight the complex influence and interdependency of factors such as language and model type on LLMs’ accuracy; yet, cross-language comparisons remain underexplored.

Objective

This study evaluates the performance of LLMs in answering medical multiple-choice questions quantitatively and qualitatively across 3 languages (German, French, and Italian), aiming to uncover model capabilities in a multilingual medical education context.

Methods

For this mixed methods study, 114 publicly accessible multiple-choice questions in German, French, and Italian from an online self-assessment tool were analyzed. A quantitative performance analysis of several LLMs developed by OpenAI, Meta AI, Anthropic, and DeepSeek was conducted to evaluate their performance on answering the questions in text-only format. For the comparative analysis, a variation of input question language (German, French, and Italian) and prompt language (English vs language-matched) was used. The 2 best-performing LLMs were then prompted to provide answer explanations for incorrectly answered questions. A subsequent qualitative analysis was conducted on these explanations to identify the reasons leading to the incorrect answers.

Results

The performance of LLMs in answering medical multiple-choice questions varied by model and language, showing substantial differences in accuracy (between 64% and 87%). The effect of input question language was significant ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsMultilingualismEducational MeasurementHumansItalyGenerative Artificial Intelligence

Stichwörter der Autor:innen

LLMLLM evaluationeducationlarge language modelmedical question answeringmultiple-choice questionsnatural language processing

Autor:innen und Institutionen

  • Medical Knowledge and Decision Support, School of Medicine, University of St.Gallen, St.Jakobstrasse 21, St.Gallen, 9000, Switzerland, +41 71 224 32 00.
  • Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland.
  • Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland.
  • Institute for Implementation Science in Health Care, University of Zurich, Zurich, Switzerland.
  • Idiap Research Institute, Martigny, Switzerland.

ORCID-Profile: Livia Maria Strasser, Wilma Anschuetz, Fabio Dennstädt, Janna Hastings

Zitieren

APA

Strasser, L. M., Anschuetz, W., Dennstädt, F., & Hastings, J. (2026). Performance Evaluation of Large Language Models in Multilingual Medical Multiple-Choice Questions: Mixed Methods Study. JMIR Medical Education, 12, e81399-e81399. https://doi.org/10.2196/81399

Vancouver

Strasser LM, Anschuetz W, Dennstädt F, Hastings J. Performance Evaluation of Large Language Models in Multilingual Medical Multiple-Choice Questions: Mixed Methods Study. JMIR Medical Education. 2026;12:e81399-e81399. doi:10.2196/81399

BibTeX
@article{Strasser2026,
  author = {Strasser, Livia Maria and Anschuetz, Wilma and Dennstädt, Fabio and Hastings, Janna},
  title = {Performance Evaluation of Large Language Models in Multilingual Medical Multiple-Choice Questions: Mixed Methods Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e81399--e81399},
  doi = {10.2196/81399}
}
Kostenlos starten →