EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Arkangel AI, OpenEvidence, ChatGPT, Medisearch: Are They Objectively up to Medical Standards? A Real-Life Assessment of LLM Chatbots in Health Care

Natalia Castaño-Villegas, María Camila Villa, Katherine Monsalve Barrientos, Isabella Llano, Laura Velásquez, José Zea

Mayo Clinic Proceedings: Digital Health · 2026-06 · 4 (2) · 100356

DOI: 10.1016/j.mcpdig.2026.100356 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mayo Clinic Proceedings: Digital Health, 2026-06. Verfasst von Natalia Castaño-Villegas und 5 weiteren Autor:innen. Beteiligte Einrichtung: Arkangel AI, Bogotá, Colombia. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To compare the performance of 4 large language model chatbots in response time and quality of clinical answers, evaluated by specialists using predefined validity criteria. Participants and Methods Between June 1 and September 20, 2025, four clinical vignettes (orthopedics, pediatrics, gynecology, and psychiatry) were developed by independent experts and answered by 4 conversational agents: Arkangel AI, OpenEvidence, ChatGPT, and Medisearch. Each vignette included 4 questions (diagnosis, clinical management, research, and general knowledge). Responses were independently evaluated by external clinicians using an 8-criterion Likert scale assessing correctness, consensus agreement, absence of bias, adherence to standards of care, timeliness, patient safety, authenticity of cited references, and contextual appropriateness. Response times were summarized using medians and interquartile ranges.

Results

A total of 128 question-answer pairs (1024 evaluations) were analyzed. Overall satisfaction ranged from 71.1% (727) to 93% (952) across agents, with statistically significant differences (Kruskal-Wallis P

Autor:innen und Institutionen

  • Arkangel AI, Bogotá, Colombia.

Zitieren

APA

Castaño-Villegas, N., Villa, M. C., Barrientos, K. M., Llano, I., Velásquez, L., & Zea, J. (2026). Arkangel AI, OpenEvidence, ChatGPT, Medisearch: Are They Objectively up to Medical Standards? A Real-Life Assessment of LLM Chatbots in Health Care. Mayo Clinic Proceedings: Digital Health, 4(2), 100356. https://doi.org/10.1016/j.mcpdig.2026.100356

Vancouver

Castaño-Villegas N, Villa MC, Barrientos KM, Llano I, Velásquez L, Zea J. Arkangel AI, OpenEvidence, ChatGPT, Medisearch: Are They Objectively up to Medical Standards? A Real-Life Assessment of LLM Chatbots in Health Care. Mayo Clinic Proceedings: Digital Health. 2026;4(2):100356. doi:10.1016/j.mcpdig.2026.100356

BibTeX
@article{CastaoVillegas2026,
  author = {Castaño-Villegas, Natalia and Villa, María Camila and Barrientos, Katherine Monsalve and Llano, Isabella and Velásquez, Laura and Zea, José},
  title = {Arkangel AI, OpenEvidence, ChatGPT, Medisearch: Are They Objectively up to Medical Standards? A Real-Life Assessment of LLM Chatbots in Health Care},
  journal = {Mayo Clinic Proceedings: Digital Health},
  year = {2026},
  volume = {4},
  number = {2},
  pages = {100356},
  doi = {10.1016/j.mcpdig.2026.100356}
}
Kostenlos starten →