EUVIMEDEUVIMED entdecken
Open Access

Facilitating Trust Calibration in Artificial Intelligence–Driven Diagnostic Decision Support Systems for Determining Physicians’ Diagnostic Accuracy: Quasi-Experimental Study

Tetsu Sakamoto iD, Yukinori Harada iD, Taro Shimizu iD

JMIR Formative Research · 27.11.2024 · 8 · e58666-e58666

DOI: 10.2196/58666 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 27.11.2024. Verfasst von Tetsu Sakamoto und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, 880 Kitakobayashi, Mibu-cho, Shimotsuga-gun, Tochigi, 321-0293, Japan, 8…. Hauptthemen laut MEDLINE: Artificial Intelligence, Trust, Decision Support Systems, Clinical, Physicians. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Diagnostic errors are significant problems in medical care. Despite the usefulness of artificial intelligence (AI)–based diagnostic decision support systems, the overreliance of physicians on AI-generated diagnoses may lead to diagnostic errors.

Objective

We investigated the safe use of AI-based diagnostic decision support systems with trust calibration by adjusting trust levels to match the actual reliability of AI.

Methods

A quasi-experimental study was conducted at Dokkyo Medical University, Japan, with physicians allocated (1:1) to the intervention and control groups. A total of 20 clinical cases were created based on the medical histories recorded by an AI-driven automated medical history–taking system from actual patients who visited a community-based hospital in Japan. The participants reviewed the medical histories of 20 clinical cases generated by an AI-driven automated medical history–taking system with an AI-generated list of 10 differential diagnoses and provided 1 to 3 possible diagnoses. Physicians were asked whether the final diagnosis was in the AI-generated list of 10 differential diagnoses in the intervention group, which served as the trust calibration. We analyzed the diagnostic accuracy of physicians and the correctness of the trust calibration in the intervention group. We also investigated the relationship between the accuracy of the trust calibration and the diagnostic accuracy of physicians, and the physicians’ confidence level regarding the use of AI.

Results

Among the 20 physicians assigned to the intervention (n=10) and control (n=10) groups, the mean age was 30.9 (SD 3.9) years and 31.7 (SD 4.2) years, the proportion of men was 80% and 60%, and the mean postgraduate year was 5.8 (SD 2.9) and 7.2 (SD 4.6), respectively, with no significant differences. The physicians’ diagnostic accuracy was 41.5% in the intervention group and 46% in the control group, with no significant difference (95% CI −0.75 to 2.55; P =.27). The overall accuracy of the trust calibration was only 61.5%, and despite correct calibration, the diagnostic accuracy was 54.5%. In the multivariate logistic regression model, the accuracy of the trust calibration was a significant contributor to the diagnostic accuracy of physicians (adjusted odds ratio 5.90, 95% CI 2.93‐12.46; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceTrustDecision Support Systems, ClinicalPhysiciansHumansMaleFemaleJapanAdultCalibrationDiagnostic ErrorsReproducibility of ResultsMiddle Aged

Stichwörter der Autor:innen

AIaccuracyaccurateartificial intelligencechartdecision supportdiagnosisdiagnosticdiagnostic accuracydiagnostic decision supporthistoryreliabilityreliabletrust calibration

Autor:innen und Institutionen

  • Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, 880 Kitakobayashi, Mibu-cho, Shimotsuga-gun, Tochigi, 321-0293, Japan, 81 282-86-1111, 81 282-86-4775.

ORCID-Profile: Tetsu Sakamoto, Yukinori Harada, Taro Shimizu

Zitieren

APA

Sakamoto, T., Harada, Y., & Shimizu, T. (2024). Facilitating Trust Calibration in Artificial Intelligence–Driven Diagnostic Decision Support Systems for Determining Physicians’ Diagnostic Accuracy: Quasi-Experimental Study. JMIR Formative Research, 8, e58666-e58666. https://doi.org/10.2196/58666

Vancouver

Sakamoto T, Harada Y, Shimizu T. Facilitating Trust Calibration in Artificial Intelligence–Driven Diagnostic Decision Support Systems for Determining Physicians’ Diagnostic Accuracy: Quasi-Experimental Study. JMIR Formative Research. 2024;8:e58666-e58666. doi:10.2196/58666

BibTeX
@article{Sakamoto2024,
  author = {Sakamoto, Tetsu and Harada, Yukinori and Shimizu, Taro},
  title = {Facilitating Trust Calibration in Artificial Intelligence–Driven Diagnostic Decision Support Systems for Determining Physicians’ Diagnostic Accuracy: Quasi-Experimental Study},
  journal = {JMIR Formative Research},
  year = {2024},
  volume = {8},
  pages = {e58666--e58666},
  doi = {10.2196/58666}
}
Kostenlos starten →