AI Reviewer’s Confidence
Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing" · 31.12.2025 · 145
DOI: 10.47743/consilr2025.11 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing", online erschienen am 31.12.2025. Verfasst von Daniela Gîfu und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Due to the overwhelming number of review requests received by top-tier scientific conferences, many deserving papers do not receive the rigorous evaluation they require. Given the increasingly competent responses provided by artificial intelligence (AI), human reviewers may partially or fully delegate the initial reviewing task to such systems. This practice raises a critical question: how much confidence should we place in an AI-generated review? We observed significant divergence among responses produced by different chatbots when evaluating the same paper. To investigate this reliability issue, we constructed theHLLM-Dist corpus, which includes 200 reviews written by an expert human reviewer for top-tier international conferences, all collected prior to the launch of ChatGPT in November 2022. For the same set of papers, we tasked three chatbotsMicrosoft Copilot, Google Gemini, and DeepSeek-Chat-to generate reviews in English. The core contribution of our work is the proposal and pilot validation of a novel metric, CAIR (Confidence of AI Reviewer), designed to quantify the alignment between an AI model’s review and human-generated evaluations. As an initial step, we fine-tuned a DistilBERT model for calibration, with plans to test and normalize results using ChatGPT in subsequent phases. The primary goal of this study is to compute and compare confidence scores for the three AI systems, establishing a foundational framework that can eventually predict AI reviewer confidence without requiring extensive human evaluation for each new model. Our results demonstrate the feasibility of this approach, offering a meaningful step toward calibrating trust in AI-assisted peer review.
Zitieren
APA
Gîfu, D., Bolea, S. C., Crucianu, P. A., & Bejinariu, S. (2025). AI Reviewer’s Confidence. Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing", 145. https://doi.org/10.47743/consilr2025.11
Vancouver
Gîfu D, Bolea SC, Crucianu PA, Bejinariu S. AI Reviewer’s Confidence. Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing". 2025:145. doi:10.47743/consilr2025.11
BibTeX
@article{Gfu2025,
author = {Gîfu, Daniela and Bolea, Speranța Cecilia and Crucianu, Paula Alexandra and Bejinariu, Silviu-Ioan},
title = {AI Reviewer’s Confidence},
journal = {Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing"},
year = {2025},
pages = {145},
doi = {10.47743/consilr2025.11}
}
EUVIMED