Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study
Mitigierung kognitiver Verzerrungen in klinischen Entscheidungsprozessen durch Multi-Agenten-Gespräche mit Large Language Models: Simulationsstudie
Journal of Medical Internet Research · 19.11.2024 · 26 · e59439
DOI: 10.2196/59439 ↗Auf einen Blick
Eine Simulation mit 16 Fallberichten prüfte, ob ein Multi-Agenten-Framework mit GPT-4 die diagnostische Genauigkeit verbessert. Die beste Konfiguration erreichte 76 % korrekte Top-2-Differenzialdiagnosen.
Fragestellung
Kann ein Multi-Agenten-Framework mit Large Language Models kognitive Verzerrungen in der klinischen Entscheidungsfindung verringern und die Diagnosegenauigkeit verbessern?
Methode
Simulation mit 16 Fallberichten; GPT-4 in einem Multi-Agenten-Framework mit 5 Rollen; 3 Frameworks, jede Szene 5-mal wiederholt; Vergleich mit menschlichen Antworten mittels Fisher-Exact-Test.
Ergebnisse
240 Antworten wurden ausgewertet. Die Ausgangsdiagnose lag bei 0 % (0/80). Nach den Multi-Agenten-Diskussionen erreichte das beste Framework 76 % (61/80) für die Top-2-Differenzialdiagnosen; OR 3,49; P=.002.
Schlussfolgerung der Autor:innen
Das Multi-Agenten-Framework zeigte die Fähigkeit, Fehlannahmen zu re-evaluieren und zu korrigieren. Das LLM-gestützte Gesprächs-Framework erscheint vielversprechend zur Verbesserung der Diagnosegenauigkeit in diagnostisch schwierigen Situationen.
Kernaussagen
- Ausgangsdiagnose: 0 % (0/80)
- Bestes Framework: 76 % (61/80) Top-2-Differenzialdiagnosen
- Im Vergleich zu Menschen signifikant höhere Genauigkeit (OR 3,49; P=.002)
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 19.11.2024. Verfasst von Yuhe Ke und 8 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore. Hauptthemen laut MEDLINE: Clinical Decision-Making. Die Arbeit wurde bisher 72-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Cognitive biases in clinical decision-making significantly contribute to errors in diagnosis and suboptimal patient outcomes. Addressing these biases presents a formidable challenge in the medical field.
Objective
This study aimed to explore the role of large language models (LLMs) in mitigating these biases through the use of the multi-agent framework. We simulate the clinical decision-making processes through multi-agent conversation and evaluate its efficacy in improving diagnostic accuracy compared with humans.
Methods
A total of 16 published and unpublished case reports where cognitive biases have resulted in misdiagnoses were identified from the literature. In the multi-agent framework, we leveraged GPT-4 (OpenAI) to facilitate interactions among different simulated agents to replicate clinical team dynamics. Each agent was assigned a distinct role: (1) making the final diagnosis after considering the discussions, (2) acting as a devil’s advocate to correct confirmation and anchoring biases, (3) serving as a field expert in the required medical subspecialty, (4) facilitating discussions to mitigate premature closure bias, and (5) recording and summarizing findings. We tested varying combinations of these agents within the framework to determine which configuration yielded the highest rate of correct final diagnoses. Each scenario was repeated 5 times for consistency. The accuracy of the initial diagnoses and the final differential diagnoses were evaluated, and comparisons with human-generated answers were made using the Fisher exact test.
Results
A total of 240 responses were evaluated (3 different multi-agent frameworks). The initial diagnosis had an accuracy of 0% (0/80). However, following multi-agent discussions, the accuracy for the top 2 differential diagnoses increased to 76% (61/80) for the best-performing multi-agent framework (Framework 4-C). This was significantly higher compared with the accuracy achieved by human evaluators (odds ratio 3.49; P=.002).
Conclusions
The multi-agent framework demonstrated an ability to re-evaluate and correct misconceptions, even in scenarios with misleading initial investigations. In addition, the LLM-driven, multi-agent conversation framework shows promise in enhancing diagnostic accuracy in diagnostically challenging medical scenarios.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.
- Department of Anesthesiology, Singapore General Hospital, Singapore, Singapore.
- Information Technology Center, University of Tokyo, Tokyo, Japan.
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore.
- Institute of Data Science, National University of Singapore, Singapore, Singapore.
ORCID-Profile: Yuhe Ke, Rui Yang, Sui An Lie, Taylor Xin Yi Lim, Yilin Ning, Irene Li, Hairil Rizal Abdullah, Daniel Shu Wei Ting, Nan Liu
Zitieren
APA
Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024). Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. Journal of Medical Internet Research, 26, e59439. https://doi.org/10.2196/59439
Vancouver
Ke Y, Yang R, Lie SA, Lim TXY, Ning Y, Li I, et al. Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study. Journal of Medical Internet Research. 2024;26:e59439. doi:10.2196/59439
BibTeX
@article{Ke2024,
author = {Ke, Yuhe and Yang, Rui and Lie, Sui An and Lim, Taylor Xin Yi and Ning, Yilin and Li, Irene and Abdullah, Hairil Rizal and Ting, Daniel Shu Wei and Liu, Nan},
title = {Mitigating Cognitive Biases in Clinical Decision-Making Through Multi-Agent Conversations Using Large Language Models: Simulation Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e59439},
doi = {10.2196/59439}
}
EUVIMED