Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports
Applied Sciences · 06.03.2026 · 16 (5) · 2546
DOI: 10.3390/app16052546 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 06.03.2026. Verfasst von Yeerpan Tuohetiyaer und 4 weiteren Autor:innen. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Jiangsu Province Major Scientific Project.
Abstract (Originalsprache)
Deep Research Agents (DRAs) generate detailed literature surveys but often suffer from hallucinations and inconsistent structures. Existing evaluation methods face significant limitations. Human evaluation is time-consuming and requires domain expertise. Meanwhile, current LLM judges struggle with long reports due to context limits and the inability to verify source reliability. To address this, we propose Deep-Research Eval. This framework standardizes the page as the basic unit for evaluation. It features an adaptive scoring system that assesses the logical quality of each page. Furthermore, it employs Paged-RAG with a constructible reference database to verify facts against specific evidence. Experiments on five agents show that our method effectively identifies errors. It achieves a strong correlation with human judgment, reaching a Composite Consistency Index (CCI) of 0.7585, an absolute increase of 0.4588 over baselines. Additionally, the Paged-RAG module improves factual verification accuracy, increasing the QA-F1 score by up to 6.9 times compared to standard retrieval methods. This work offers a scalable and practical approach for assessing AI-generated academic content.
Förderung
- Jiangsu Province Major Scientific Project
Zitieren
APA
Tuohetiyaer, Y., Zhu, Y., Hu, Y., Lu, S., & Wang, Z. (2026). Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports. Applied Sciences, 16(5), 2546. https://doi.org/10.3390/app16052546
Vancouver
Tuohetiyaer Y, Zhu Y, Hu Y, Lu S, Wang Z. Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports. Applied Sciences. 2026;16(5):2546. doi:10.3390/app16052546
BibTeX
@article{Tuohetiyaer2026,
author = {Tuohetiyaer, Yeerpan and Zhu, Yuye and Hu, Yan and Lu, Siyuan and Wang, Zhongfeng},
title = {Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports},
journal = {Applied Sciences},
year = {2026},
volume = {16},
number = {5},
pages = {2546},
doi = {10.3390/app16052546}
}
EUVIMED