Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers
International Journal of Qualitative Methods · 19.01.2026 · 25
DOI: 10.1177/16094069261504392 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Qualitative Methods, online erschienen am 17.11.2026. Verfasst von Milad Mehdizadkhani und 3 weiteren Autor:innen. Die Arbeit nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Special Project of National Social Science Fund of China “Researching the Diplomatic Discourse System of Marxism Adapted to the Chinese Context and the Needs o…, Major Project of National Social Science Fund of china.
Abstract (Originalsprache)
As large language models (LLMs) are increasingly adopted for automated research synthesis, their capacity to accurately extract nuanced data from language-centric disciplines remains critically underexplored. Therefore, the primary aim of this study is to systematically evaluate the data extraction accuracy and specific error patterns of four leading LLMs (ChatGPT, Gemini, DeepSeek, and Grok) when processing complex academic literature. First, we constructed a verified corpus of 20 research articles, comprising 10 quantitative and 10 qualitative papers, drawn from Applied Linguistics, Discourse Analysis, and Translation Studies. Then we manually created consensus-based golden templates for all 20 papers. Using zero-shot prompts, we extracted predefined data categories and evaluated the models’ outputs through quantitative F1 score analysis and a systematic qualitative error taxonomy. Results indicate high extraction accuracy across all models. DeepSeek and ChatGPT demonstrated superior consistency in structured tasks. A distinct accuracy paradox emerged in qualitative extraction where models successfully identified broad theoretical frameworks yet frequently omitted critical narrative details and individual participant trajectories. Error analysis revealed that the category Omission was the primary failure mode with zero instances of hallucination. Significantly, inaccuracy often showed as the mitigation of critical academic discourse, particularly in ChatGPT, which tended to neutralize authorial tone. These findings suggest that while LLMs can streamline data extraction for research, language-related scholars should focus on detecting omissions and tonal shifts rather than fabrication.
Förderung
- Special Project of National Social Science Fund of China “Researching the Diplomatic Discourse System of Marxism Adapted to the Chinese Context and the Needs o…
- Major Project of National Social Science Fund of china
Zitieren
APA
Mehdizadkhani, M., Liu, L., Zheng, R., & Yang, M. (2026). Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers. International Journal of Qualitative Methods, 25. https://doi.org/10.1177/16094069261504392
Vancouver
Mehdizadkhani M, Liu L, Zheng R, Yang M. Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers. International Journal of Qualitative Methods. 2026;25. doi:10.1177/16094069261504392
BibTeX
@article{Mehdizadkhani2026,
author = {Mehdizadkhani, Milad and Liu, Linjun and Zheng, Runyu and Yang, Mingxing},
title = {Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers},
journal = {International Journal of Qualitative Methods},
year = {2026},
volume = {25},
doi = {10.1177/16094069261504392}
}
EUVIMED