EUVIMEDEUVIMED entdecken
Qualitative StudieOpen Access

Performance of Large Language Models in the Non-English Context: Qualitative Study of Models Trained on Different Languages in Chinese Medical Examinations

Zhong Yao iD, Liantan Duan iD, Shuo Xu iD, Lingyi Chi iD, Dongfang Sheng iD

JMIR Medical Informatics · 27.06.2025 · 13 · e69485-e69485

DOI: 10.2196/69485 ↗

Kurz gefasst

Qualitative Studie in der Zeitschrift JMIR Medical Informatics, online erschienen am 27.06.2025. Verfasst von Zhong Yao und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurosurgery, Qilu Hospital of Shandong University, Jinan, China. Hauptthemen laut MEDLINE: Language, Educational Measurement, Licensure, Medical. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Research on large language models (LLMs) in the medical field has predominantly focused on models trained with English-language corpora, evaluating their performance within English-speaking contexts. The performances of models trained with non–English language corpora and their performance in non-English contexts remain underexplored.

Objective

This study aimed to evaluate the performances of LLMs trained on different languages corpora by using the Chinese National Medical Licensing Examination (CNMLE) as a benchmark and constructed analogous questions.

Methods

Under different prompt settings, we sequentially posed questions to 7 LLMs: 2 primarily trained on English-language corpora and 5 primarily on Chinese-language corpora. The models’ responses were compared against standard answers to calculate the accuracy rate of each model. Further subgroup analyses were conducted by categorizing the questions based on various criteria. We also collected error sets to explore patterns of mistakes across different models.

Results

Under the zero-shot setting, 6 out of 7 models exceeded the passing level, with the highest accuracy rate achieved by the Chinese LLM Baichuan (86.67%), followed by ChatGPT (83.83%). In the constructed questions, all 7 models exceeded the passing threshold, with Baichuan maintaining the highest accuracy rate (87.00%). In few-shot learning, all models exceeded the passing threshold. Baichuan, ChatGLM, and ChatGPT retained the highest accuracy. While Llama showed marked improvement over previous tests, the relative performance rankings of other models stayed similar to previous results. In subgroup analyses, English models demonstrated comparable or superior performance to Chinese models on questions related to ethics and policy. All models except Llama generally had higher accuracy rates for simple questions than for complex ones. The error set of ChatGPT was similar to those of other Chinese models. Multimodel cross-verification outperformed single model, particularly improving accuracy rate on simple questions. The implementation of dual-model and tri-model verification achieved accuracy rates of 94.17% and 96.33% respectively.

Conclusions

At the current level, LLMs trained primarily on English corpora and those trained mainly on Chinese corpora perform similarly well in CNMLE, with Chinese models still outperforming. The performance difference between ChatGPT and other Chinese LLMs are not solely due to communication barriers but are more likely influenced by disparities in the training data. By using a method of cross-verification with multiple LLMs, excellent performance can be achieved in medical examinations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageEducational MeasurementLicensure, MedicalHumansChinaQualitative ResearchLarge Language Models

Stichwörter der Autor:innen

ChatGPTlanguage corporalarge language modelmedical examinationnon-English

Autor:innen und Institutionen

  • Department of Neurosurgery, Qilu Hospital of Shandong University, Jinan, China.
  • Cheeloo College of Medicine, Shandong University, Jinan, China.
  • Brain and Brain-Inspired Science, Shandong University, Jinan, China.
  • School of Management, Shandong University, No. 27 Shanda South Road,250100, Jinan, China, 86 15216408181.

ORCID-Profile: Zhong Yao, Liantan Duan, Shuo Xu, Lingyi Chi, Dongfang Sheng

Zitieren

APA

Yao, Z., Duan, L., Xu, S., Chi, L., & Sheng, D. (2025). Performance of Large Language Models in the Non-English Context: Qualitative Study of Models Trained on Different Languages in Chinese Medical Examinations. JMIR Medical Informatics, 13, e69485-e69485. https://doi.org/10.2196/69485

Vancouver

Yao Z, Duan L, Xu S, Chi L, Sheng D. Performance of Large Language Models in the Non-English Context: Qualitative Study of Models Trained on Different Languages in Chinese Medical Examinations. JMIR Medical Informatics. 2025;13:e69485-e69485. doi:10.2196/69485

BibTeX
@article{Yao2025,
  author = {Yao, Zhong and Duan, Liantan and Xu, Shuo and Chi, Lingyi and Sheng, Dongfang},
  title = {Performance of Large Language Models in the Non-English Context: Qualitative Study of Models Trained on Different Languages in Chinese Medical Examinations},
  journal = {JMIR Medical Informatics},
  year = {2025},
  volume = {13},
  pages = {e69485--e69485},
  doi = {10.2196/69485}
}
Kostenlos starten →