Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti251532 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Dorian Zwanzig und 3 weiteren Autor:innen. Beteiligte Einrichtung: HTW Berlin, Faculty 4 (Computing, Communication and Business). Hauptthemen laut MEDLINE: Artificial Intelligence, Physical Therapists. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
This paper introduces a simple approach for assessing whether laypeople or AI-based automations can adequately substitute for expert ratings in the evaluation of AI-powered Q&A systems It employs weighted Cohen’s Kappa to assess inter-rater reliability, establishing an expert agreement benchmark and comparing this to individual alternative rater-expert agreements. By visualizing these results in an inter-rater reliability matrix, it is a transparent and structured way to determine the adequacy of non-expert raters. Our findings, based on a real project, suggest that laypeople or AI, in some cases, can match or exceed expert agreement, particularly when risk aversion is a factor. The approach can be adapted to different contexts and rating attributes.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- HTW Berlin, Faculty 4 (Computing, Communication and Business).
Zitieren
APA
Zwanzig, D., Kreibich, L., Binder, U., & Dietrich, U. (2025). Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251532
Vancouver
Zwanzig D, Kreibich L, Binder U, Dietrich U. Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251532
BibTeX
@article{Zwanzig2025,
author = {Zwanzig, Dorian and Kreibich, Luca and Binder, Uta and Dietrich, Ute},
title = {Evaluating AI-Powered Q&A Systems: A Simple Approach to Determining the Need for Expert Ratings},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti251532}
}
EUVIMED