EUVIMEDEUVIMED entdecken
Open Access

Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study

Mitigierung kognitiver Verzerrungen in klinischen Entscheidungsprozessen durch Multi-Agenten-Gespräche mit Large Language Models: Simulationsstudie

Yuhe Ke iD, Rui Yang iD, Sui An Lie iD, Taylor Xin Yi Lim iD, Yilin Ning iD, Irene Li iD, Hairil Rizal Abdullah iD, Daniel Shu Wei Ting iD, Nan Liu iD

Journal of Medical Internet Research · 19.11.2024 · 26 · e59439

DOI: 10.2196/59439 ↗

Auf einen Blick

Eine Simulation mit 16 Fallberichten prüfte, ob ein Multi-Agenten-Framework mit GPT-4 die diagnostische Genauigkeit verbessert. Die beste Konfiguration erreichte 76 % korrekte Top-2-Differenzialdiagnosen.

Fragestellung

Kann ein Multi-Agenten-Framework mit Large Language Models kognitive Verzerrungen in der klinischen Entscheidungsfindung verringern und die Diagnosegenauigkeit verbessern?

Methode

Simulation mit 16 Fallberichten; GPT-4 in einem Multi-Agenten-Framework mit 5 Rollen; 3 Frameworks, jede Szene 5-mal wiederholt; Vergleich mit menschlichen Antworten mittels Fisher-Exact-Test.

Ergebnisse

240 Antworten wurden ausgewertet. Die Ausgangsdiagnose lag bei 0 % (0/80). Nach den Multi-Agenten-Diskussionen erreichte das beste Framework 76 % (61/80) für die Top-2-Differenzialdiagnosen; OR 3,49; P=.002.

Schlussfolgerung der Autor:innen

Das Multi-Agenten-Framework zeigte die Fähigkeit, Fehlannahmen zu re-evaluieren und zu korrigieren. Das LLM-gestützte Gesprächs-Framework erscheint vielversprechend zur Verbesserung der Diagnosegenauigkeit in diagnostisch schwierigen Situationen.

Kernaussagen

  • Ausgangsdiagnose: 0 % (0/80)
  • Bestes Framework: 76 % (61/80) Top-2-Differenzialdiagnosen
  • Im Vergleich zu Menschen signifikant höhere Genauigkeit (OR 3,49; P=.002)

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 19.11.2024. Verfasst von Yuhe Ke und 8 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore. Hauptthemen laut MEDLINE: Clinical Decision-Making. Die Arbeit wurde bisher 72-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Cognitive biases in clinical decision-making significantly contribute to errors in diagnosis and suboptimal patient outcomes. Addressing these biases presents a formidable challenge in the medical field.

Objective

This study aimed to explore the role of large language models (LLMs) in mitigating these biases through the use of the multi-agent framework. We simulate the clinical decision-making processes through multi-agent conversation and evaluate its efficacy in improving diagnostic accuracy compared with humans.

Methods

A total of 16 published and unpublished case reports where cognitive biases have resulted in misdiagnoses were identified from the literature. In the multi-agent framework, we leveraged GPT-4 (OpenAI) to facilitate interactions among different simulated agents to replicate clinical team dynamics. Each agent was assigned a distinct role: (1) making the final diagnosis after considering the discussions, (2) acting as a devil’s advocate to correct confirmation and anchoring biases, (3) serving as a field expert in the required medical subspecialty, (4) facilitating discussions to mitigate premature closure bias, and (5) recording and summarizing findings. We tested varying combinations of these agents within the framework to determine which configuration yielded the highest rate of correct final diagnoses. Each scenario was repeated 5 times for consistency. The accuracy of the initial diagnoses and the final differential diagnoses were evaluated, and comparisons with human-generated answers were made using the Fisher exact test.

Results

A total of 240 responses were evaluated (3 different multi-agent frameworks). The initial diagnosis had an accuracy of 0% (0/80). However, following multi-agent discussions, the accuracy for the top 2 differential diagnoses increased to 76% (61/80) for the best-performing multi-agent framework (Framework 4-C). This was significantly higher compared with the accuracy achieved by human evaluators (odds ratio 3.49; P=.002).

Conclusions

The multi-agent framework demonstrated an ability to re-evaluate and correct misconceptions, even in scenarios with misleading initial investigations. In addition, the LLM-driven, multi-agent conversation framework shows promise in enhancing diagnostic accuracy in diagnostically challenging medical scenarios.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Clinical Decision-MakingHumansCognitionBiasLanguageComputer SimulationDiagnostic ErrorsCommunication

Stichwörter der Autor:innen

clinical decision-makingcognitive biasgenerative artificial intelligencelarge language modelmulti-agent

Autor:innen und Institutionen

  • Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.
  • Department of Anesthesiology, Singapore General Hospital, Singapore, Singapore.
  • Information Technology Center, University of Tokyo, Tokyo, Japan.
  • Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore.
  • Institute of Data Science, National University of Singapore, Singapore, Singapore.

ORCID-Profile: Yuhe Ke, Rui Yang, Sui An Lie, Taylor Xin Yi Lim, Yilin Ning, Irene Li, Hairil Rizal Abdullah, Daniel Shu Wei Ting, Nan Liu

Zitieren

APA

Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024). Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. Journal of Medical Internet Research, 26, e59439. https://doi.org/10.2196/59439

Vancouver

Ke Y, Yang R, Lie SA, Lim TXY, Ning Y, Li I, et al. Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. Journal of Medical Internet Research. 2024;26:e59439. doi:10.2196/59439

BibTeX
@article{Ke2024,
  author = {Ke, Yuhe and Yang, Rui and Lie, Sui An and Lim, Taylor Xin Yi and Ning, Yilin and Li, Irene and Abdullah, Hairil Rizal and Ting, Daniel Shu Wei and Liu, Nan},
  title = {Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e59439},
  doi = {10.2196/59439}
}
Kostenlos starten →