EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports

Yeerpan Tuohetiyaer, Yuye Zhu, Yan Hu, Siyuan Lu, Zhongfeng Wang

Applied Sciences · 06.03.2026 · 16 (5) · 2546

DOI: 10.3390/app16052546 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 06.03.2026. Verfasst von Yeerpan Tuohetiyaer und 4 weiteren Autor:innen. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Jiangsu Province Major Scientific Project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Deep Research Agents (DRAs) generate detailed literature surveys but often suffer from hallucinations and inconsistent structures. Existing evaluation methods face significant limitations. Human evaluation is time-consuming and requires domain expertise. Meanwhile, current LLM judges struggle with long reports due to context limits and the inability to verify source reliability. To address this, we propose Deep-Research Eval. This framework standardizes the page as the basic unit for evaluation. It features an adaptive scoring system that assesses the logical quality of each page. Furthermore, it employs Paged-RAG with a constructible reference database to verify facts against specific evidence. Experiments on five agents show that our method effectively identifies errors. It achieves a strong correlation with human judgment, reaching a Composite Consistency Index (CCI) of 0.7585, an absolute increase of 0.4588 over baselines. Additionally, the Paged-RAG module improves factual verification accuracy, increasing the QA-F1 score by up to 6.9 times compared to standard retrieval methods. This work offers a scalable and practical approach for assessing AI-generated academic content.

Förderung

  • Jiangsu Province Major Scientific Project

Zitieren

APA

Tuohetiyaer, Y., Zhu, Y., Hu, Y., Lu, S., & Wang, Z. (2026). Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports. Applied Sciences, 16(5), 2546. https://doi.org/10.3390/app16052546

Vancouver

Tuohetiyaer Y, Zhu Y, Hu Y, Lu S, Wang Z. Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports. Applied Sciences. 2026;16(5):2546. doi:10.3390/app16052546

BibTeX
@article{Tuohetiyaer2026,
  author = {Tuohetiyaer, Yeerpan and Zhu, Yuye and Hu, Yan and Lu, Siyuan and Wang, Zhongfeng},
  title = {Deep-Research Eval: An Automated Framework for Assessing Quality and Reliability in Long-Form Reports},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {5},
  pages = {2546},
  doi = {10.3390/app16052546}
}
Kostenlos starten →