EUVIMEDEUVIMED entdecken
Open Access

Measuring Expert Inter-Rater Agreement with a Semi-Automated Clinical Decision Support System: Who Agrees with What?

Yasmin Benhalim iD, Matthew W Hodgman, John R Charpie, Gabe E Owens, Ranjit Aiyagari, Daniel E Ehrmann iD

Applied Clinical Informatics · 01.08.2026 · 17 (04) · 817-824

DOI: 10.1055/a-2956-4559 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.08.2026. Verfasst von Yasmin Benhalim und 5 weiteren Autor:innen. Beteiligte Einrichtung: University of Michigan Medical School, Michigan, United States, Ann Arbor. Hauptthemen laut MEDLINE: Decision Support Systems, Clinical, Diuretics. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Assess inter-rater agreement on clinician- and clinical decision support (CDS)-driven diuretic titrations and risk classifications to benchmark a semi-autonomous CDS (OTTO-FM).

Methods

Secondary analysis of prospectively collected porcine data modeling postoperative fluid overload. Three pediatric cardiac intensivists rated items (reasonable/unsure/unreasonable) for clinician-driven (human phase; 29 items) and CDS-driven titrations with risk classification (CDS phase; 44 items). Inter-rater agreement was estimated with ordinal-weighted Gwet’s AC2.

Results

Dosing inter-rater agreement was similar between phases (human AC2: 0.79 [0.61–0.92]; CDS: 0.83 [0.68–0.92]), though unanimity was reached on only 19 of 29 (65.5%) human-phase and 31 of 44 (70.5%) CDS-phase items. Risk-label inter-rater agreement stratified sharply by label: high-risk, 0.92 (0.74–0.95) versus low-risk, 0.54 (0.28–0.75). Disagreement also differed in structure between tasks: 19 of 20 nonunanimous risk items were minority dissent (one rater against a two-rater majority), including 16 of 17 among low-risk items, whereas 7 of 13 nonunanimous CDS dosing items were three-way splits.

Conclusion

Inter-rater agreement on dosing was substantial and similar across phases, but not unanimous, suggesting that perfect reasonableness consensus with CDS may be unrealistic. Agreement on risk classification was lower, and the structure of disagreement differed between CDS dose and risk. Empirical inter-rater agreement baselines may contextualize CDS evaluation, and these differences may have implications for the design of semi-autonomous CDS if confirmed in larger studies.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Decision Support Systems, ClinicalDiureticsAnimalsObserver VariationHumansSwineAutomation

Substanzen

Diuretics

Autor:innen und Institutionen

  • University of Michigan Medical School, Michigan, United States, Ann Arbor.
  • University of Michigan Medical School, Department of Computational Medicine and Bioinformatics, Michigan, United States, Ann Arbor.
  • University of Michigan Medical School, Department of Pediatrics, Michigan, United States, Ann Arbor.
  • Congenital Heart Center at C.S. Mott Children's Hospital, Michigan, United States, Ann Arbor.

ORCID-Profile: Yasmin Benhalim, Daniel E Ehrmann

Zitieren

APA

Benhalim, Y., Hodgman, M. W., Charpie, J. R., Owens, G. E., Aiyagari, R., & Ehrmann, D. E. (2026). Measuring Expert Inter-Rater Agreement with a Semi-Automated Clinical Decision Support System: Who Agrees with What?. Applied Clinical Informatics, 17(04), 817-824. https://doi.org/10.1055/a-2956-4559

Vancouver

Benhalim Y, Hodgman MW, Charpie JR, Owens GE, Aiyagari R, Ehrmann DE. Measuring Expert Inter-Rater Agreement with a Semi-Automated Clinical Decision Support System: Who Agrees with What?. Applied Clinical Informatics. 2026;17(04):817-824. doi:10.1055/a-2956-4559

BibTeX
@article{Benhalim2026,
  author = {Benhalim, Yasmin and Hodgman, Matthew W and Charpie, John R and Owens, Gabe E and Aiyagari, Ranjit and Ehrmann, Daniel E},
  title = {Measuring Expert Inter-Rater Agreement with a Semi-Automated Clinical Decision Support System: Who Agrees with What?},
  journal = {Applied Clinical Informatics},
  year = {2026},
  volume = {17},
  number = {04},
  pages = {817--824},
  doi = {10.1055/a-2956-4559}
}
Kostenlos starten →