Human-in-the-Loop AI Reviewing: Feasibility, Opportunities, and Risks
Journal of the Association for Information Systems · 2024 · 25 (1) · 98-109
DOI: 10.17705/1jais.00867 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the Association for Information Systems, 2024. Verfasst von Iddo Drori und 1 weiteren Autor:innen. Die Arbeit wurde bisher 43-mal zitiert (Crossref).
Abstract (Originalsprache)
The promise of AI for academic work is bewitching and easy to envisage, but the risks involved are often hard to detect and usually not readily exposed. In this opinion piece, we explore the feasibility, opportunities, and risks of using large language models (LLMs) for reviewing academic submissions, while keeping the human in the loop. We experiment with GPT-4 in the role of a reviewer to demonstrate the opportunities and the risks we experience and ways to mitigate them. The reviews are structured according to a conference review form with the dual purpose of evaluating submissions for editorial decisions and providing authors with constructive feedback according to predefined criteria, which include contribution, soundness, and presentation. We demonstrate feasibility by evaluating and comparing LLM reviews with human reviews, concluding that current AI-augmented reviewing is sufficiently accurate to alleviate the burden of reviewing but not completely and not for all cases. We then enumerate the opportunities of AI-augmented reviewing and present open questions. Next, we identify the risks of AI-augmented reviewing, highlighting bias, value misalignment, and misuse. We conclude with recommendations for managing these risks.
Zitieren
APA
Drori, I., & Te'eni, D. (2024). Human-in-the-Loop AI Reviewing: Feasibility, Opportunities, and Risks. Journal of the Association for Information Systems, 25(1), 98-109. https://doi.org/10.17705/1jais.00867
Vancouver
Drori I, Te'eni D. Human-in-the-Loop AI Reviewing: Feasibility, Opportunities, and Risks. Journal of the Association for Information Systems. 2024;25(1):98-109. doi:10.17705/1jais.00867
BibTeX
@article{Drori2024,
author = {Drori, Iddo and Te'eni, Dov},
title = {Human-in-the-Loop AI Reviewing: Feasibility, Opportunities, and Risks},
journal = {Journal of the Association for Information Systems},
year = {2024},
volume = {25},
number = {1},
pages = {98--109},
doi = {10.17705/1jais.00867}
}
EUVIMED