EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study

Ahmed Bashah, Quan Huixin, Hamza Zaawari, Ali Al-Waqeerah, Eslam Ghaleb, Abdulkhaleq Salem, Natheer Wahan, Gang Chen

JMIR Medical Informatics · 27.10.2026 · 14 · e96731-e96731

DOI: 10.2196/96731 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Medical Informatics, online erschienen am 27.10.2026. Verfasst von Ahmed Bashah und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Stomatology, The First Affiliated Hospital of Dalian Medical University, No.222, Zhongshan Road, Dalian, Liaoning, 116011, China, +86 1…. Hauptthemen laut MEDLINE: Internet, Artificial Intelligence. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Publicly accessible, internet-based AI chatbots have emerged as unregulated digital health resources that clinicians and patients consult in real time, including during acute medical emergencies. In dental practice—where life-threatening events such as anaphylaxis, cardiac arrest, and severe hypoglycemia can unfold within seconds—the unsupervised use of these tools as point-of-care decision aids poses uncharacterized patient safety risks. Despite widespread deployment, no study has systematically evaluated the accuracy, temporal consistency, or multilingual reliability of contemporary large language model (LLM)–based chatbots specifically for dental medical emergency management.

Objective

This study aimed to evaluate the accuracy, temporal stability, and multilingual performance of five contemporary LLM-based chatbots when queried on the management of medical emergencies in dental practice.

Methods

A comparative study was conducted to assess five AI chatbots: ChatGPT-5.2 (OpenAI), Gemini 3.1 (Google), DeepSeek-R1 (DeepSeek AI), Grok 4.20 (xAI), and Kimi K2.5 (Moonshot AI). A total of 130 questions covering dental medical emergency management were developed and categorized into four formats: multiple-choice questions (MCQs), yes-or-no questions, open-ended WH questions, and clinical scenario questions. Each question was presented using two prompting strategies (basic and advanced) across three languages (English, Arabic, and Chinese). Data collection was performed over seven consecutive days, generating 27,300 chatbot responses. Responses were evaluated by language-specific expert reviewers and classified as correct or incorrect based on established clinical references. Statistical analysis was performed using chi-square tests to assess associations between accuracy and study variables. Pairwise post hoc comparisons of column proportions were conducted using z-tests with Bonferroni correction.

Results

A total of 27,300 responses were analyzed; 25,377 (92.96%) were correct. Question type significantly influenced chatbot performance ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

InternetArtificial IntelligenceHumansLarge Language ModelsDigital HealthReproducibility of Results

Stichwörter der Autor:innen

artificial intelligencechatbotsdental medical emergencieslarge language modelsmultilingual evaluationprompt engineering

Autor:innen und Institutionen

  • Department of Stomatology, The First Affiliated Hospital of Dalian Medical University, No.222, Zhongshan Road, Dalian, Liaoning, 116011, China, +86 18098873280.
  • College of Dentistry, Hodeidah University, Al Hudaydah, Yemen.
  • Department of Respiratory, The First Affiliated Hospital of Dalian Medical University, Dalian, Liaoning, China.
  • Department of Biochemistry and Molecular Biology, Dalian Medical University, Dalian, Liaoning, China.
  • Department of Pharmacology, Dalian Medical University, Dalian, Liaoning, China.

Zitieren

APA

Bashah, A., Huixin, Q., Zaawari, H., Al-Waqeerah, A., Ghaleb, E., Salem, A., Wahan, N., & Chen, G. (2026). Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study. JMIR Medical Informatics, 14, e96731-e96731. https://doi.org/10.2196/96731

Vancouver

Bashah A, Huixin Q, Zaawari H, Al-Waqeerah A, Ghaleb E, Salem A, et al. Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study. JMIR Medical Informatics. 2026;14:e96731-e96731. doi:10.2196/96731

BibTeX
@article{Bashah2026,
  author = {Bashah, Ahmed and Huixin, Quan and Zaawari, Hamza and Al-Waqeerah, Ali and Ghaleb, Eslam and Salem, Abdulkhaleq and Wahan, Natheer and Chen, Gang},
  title = {Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  volume = {14},
  pages = {e96731--e96731},
  doi = {10.2196/96731}
}
Kostenlos starten →