EUVIMEDEUVIMED entdecken
Crossref

Evaluating Data Suitability for RAG Systems in Manufacturing: A Comparison Between Human and LLM Judgments

Tom Keller, Carsten Wohlgemuth, Eike Permin

Journal of Machine Engineering · 22.06.2026 · 26 (2) · 163-174

DOI: 10.36897/jme/218709 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Machine Engineering, online erschienen am 16.04.2026. Verfasst von Tom Keller und 2 weiteren Autor:innen. Die Arbeit nennt 18 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-Augmented Generation (RAG) is widely used for manufacturing assistance, but its effectiveness depends on selecting retrievable text units. We test whether humans or Large Language Models (LLMs) can judge which case descriptions are better suited as RAG inputs. We constructed 100 synthetic manufacturing service cases, each paired with a realistic query and two comparable problem–solution variants differing in contextual completeness, granularity, and quality. Five engineers and five LLMs chose the variant expected to be more retrievable and useful. As a reference, both variants were indexed in a minimal retrieval setup with one chunk per case and evaluated with MRR@3, treating the case-matching chunk as the only relevant item among distractors. LLMs showed much higher within-group agreement than humans, yet neither cohort consistently matched retrieval-derived winners. Ties were frequent; on non-tied cases, majority decisions fell below chance and were significantly worse than random guessing in one embedding setting, while no individual rater achieved above-chance performance. Overall, the findings indicate that perceived RAG-fitness is not a reliable proxy for retrieval performance and should be grounded in retrieval-based evaluation under the target deployment setup.

Zitieren

APA

Keller, T., Wohlgemuth, C., & Permin, E. (2026). Evaluating Data Suitability for RAG Systems in Manufacturing: A Comparison Between Human and LLM Judgments. Journal of Machine Engineering, 26(2), 163-174. https://doi.org/10.36897/jme/218709

Vancouver

Keller T, Wohlgemuth C, Permin E. Evaluating Data Suitability for RAG Systems in Manufacturing: A Comparison Between Human and LLM Judgments. Journal of Machine Engineering. 2026;26(2):163-174. doi:10.36897/jme/218709

BibTeX
@article{Keller2026,
  author = {Keller, Tom and Wohlgemuth, Carsten and Permin, Eike},
  title = {Evaluating Data Suitability for RAG Systems in Manufacturing: A Comparison Between Human and LLM Judgments},
  journal = {Journal of Machine Engineering},
  year = {2026},
  volume = {26},
  number = {2},
  pages = {163--174},
  doi = {10.36897/jme/218709}
}
Kostenlos starten →