EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers

Milad Mehdizadkhani, Linjun Liu, Runyu Zheng, Mingxing Yang

International Journal of Qualitative Methods · 19.01.2026 · 25

DOI: 10.1177/16094069261504392 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Qualitative Methods, online erschienen am 17.11.2026. Verfasst von Milad Mehdizadkhani und 3 weiteren Autor:innen. Die Arbeit nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Special Project of National Social Science Fund of China “Researching the Diplomatic Discourse System of Marxism Adapted to the Chinese Context and the Needs o…, Major Project of National Social Science Fund of china.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As large language models (LLMs) are increasingly adopted for automated research synthesis, their capacity to accurately extract nuanced data from language-centric disciplines remains critically underexplored. Therefore, the primary aim of this study is to systematically evaluate the data extraction accuracy and specific error patterns of four leading LLMs (ChatGPT, Gemini, DeepSeek, and Grok) when processing complex academic literature. First, we constructed a verified corpus of 20 research articles, comprising 10 quantitative and 10 qualitative papers, drawn from Applied Linguistics, Discourse Analysis, and Translation Studies. Then we manually created consensus-based golden templates for all 20 papers. Using zero-shot prompts, we extracted predefined data categories and evaluated the models’ outputs through quantitative F1 score analysis and a systematic qualitative error taxonomy. Results indicate high extraction accuracy across all models. DeepSeek and ChatGPT demonstrated superior consistency in structured tasks. A distinct accuracy paradox emerged in qualitative extraction where models successfully identified broad theoretical frameworks yet frequently omitted critical narrative details and individual participant trajectories. Error analysis revealed that the category Omission was the primary failure mode with zero instances of hallucination. Significantly, inaccuracy often showed as the mitigation of critical academic discourse, particularly in ChatGPT, which tended to neutralize authorial tone. These findings suggest that while LLMs can streamline data extraction for research, language-related scholars should focus on detecting omissions and tonal shifts rather than fabrication.

Förderung

  • Special Project of National Social Science Fund of China “Researching the Diplomatic Discourse System of Marxism Adapted to the Chinese Context and the Needs o…
  • Major Project of National Social Science Fund of china

Zitieren

APA

Mehdizadkhani, M., Liu, L., Zheng, R., & Yang, M. (2026). Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers. International Journal of Qualitative Methods, 25. https://doi.org/10.1177/16094069261504392

Vancouver

Mehdizadkhani M, Liu L, Zheng R, Yang M. Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers. International Journal of Qualitative Methods. 2026;25. doi:10.1177/16094069261504392

BibTeX
@article{Mehdizadkhani2026,
  author = {Mehdizadkhani, Milad and Liu, Linjun and Zheng, Runyu and Yang, Mingxing},
  title = {Evaluating LLM Data Extraction Accuracy and Error Patterns in Language-Related Research Across Quantitative and Qualitative Papers},
  journal = {International Journal of Qualitative Methods},
  year = {2026},
  volume = {25},
  doi = {10.1177/16094069261504392}
}
Kostenlos starten →