EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Diagnosing the limits of large language models for structured scientific evidence extraction

Zhiyin Tan, Jennifer D’Souza, Sören Auer

International Journal on Digital Libraries · 2026-12 · 27 (4)

DOI: 10.1007/s00799-026-00449-9 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal on Digital Libraries, online erschienen am 28.10.2026. Verfasst von Zhiyin Tan und 2 weiteren Autor:innen. Die Arbeit nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Niedersächsisches Ministerium für Wissenschaft und Kultur.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Scientific evidence is predominantly communicated through unstructured articles, while many downstream workflows require structured representations of study populations, variables, methods, and quantitative findings. Large language models (LLMs) offer a promising route toward automating this transformation, yet their reliability in constructing schema-constrained evidence records from parsed scientific full texts remains insufficiently understood. Evidence synthesis is a representative use case in which systematic reviews and meta-analyses require accurate, structured study information before quantitative integration can begin. We present a schema-grounded, tiered evaluation framework for structured scientific evidence extraction that increases task difficulty from single-entity identification to relational tuples and corpus-level statistical tasks. We evaluate frontier commercial and open-weight models across five scientific domains under a stricter multi-level assessment protocol, complemented by human validation of model-based judging and process-level inspection of intermediate reasoning for statistical tasks. Results reveal a clear pattern: models perform substantially better on low-complexity extraction tasks than on tasks requiring correct bindings among entities, numerical values, and methodological attributes. Intermediate-step analysis further indicates error amplification under exact downstream aggregation, where small upstream errors often prevent correct final outputs. Scaling model size yields uneven gains and does not remove these weaknesses. These findings show that recent advances have not resolved the fundamental challenges of structured scientific evidence extraction. We provide a reusable query suite and diagnostic evaluation framework to support future fine-grained benchmarks and more rigorous assessment of models for evidence-intensive scientific workflows.

Förderung

  • Niedersächsisches Ministerium für Wissenschaft und Kultur

Zitieren

APA

Tan, Z., D’Souza, J., & Auer, S. (2026). Diagnosing the limits of large language models for structured scientific evidence extraction. International Journal on Digital Libraries, 27(4). https://doi.org/10.1007/s00799-026-00449-9

Vancouver

Tan Z, D’Souza J, Auer S. Diagnosing the limits of large language models for structured scientific evidence extraction. International Journal on Digital Libraries. 2026;27(4). doi:10.1007/s00799-026-00449-9

BibTeX
@article{Tan2026,
  author = {Tan, Zhiyin and D’Souza, Jennifer and Auer, Sören},
  title = {Diagnosing the limits of large language models for structured scientific evidence extraction},
  journal = {International Journal on Digital Libraries},
  year = {2026},
  volume = {27},
  number = {4},
  doi = {10.1007/s00799-026-00449-9}
}
Kostenlos starten →