EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Context, User, and Truth: Evaluating LLM Response Quality as a Human–Model System Property

Mariusz Kania, Ryszard Klempous, Artur Bąk, Zenon Chaczko, Jakub Segen, Michał Kulbacki, Marek Kulbacki

International Journal of Electronics and Telecommunications · 29.10.2026 · 72 (4) · 1-8

DOI: 10.24425/ijet.2026.6313 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Electronics and Telecommunications, online erschienen am 29.10.2026. Verfasst von Mariusz Kania und 6 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are evaluated as if quality were a property of the model. We show that it is a property of the human–model system. In a benchmark of ten LLMs across 12 tasks with controlled prompt variation and a user study (n=80), we find that query context and user expertise jointly modulate response quality. Specifically, context enrichment improves BERTScore F1 by 0.07 for novice users but only 0.02 for experts, and model rankings shift by up to five positions across user-group conditions. More critically, we identify a fluency–accuracy dissociation: 62.5% of users prefer a factually incorrect answer when it is more fluent, and this preference is strongest among novice users who also provide the least context. Because RLHF-style alignment trains on these preferences, the human–model loop can converge on a stable equilibrium in which users are satisfied but answers are wrong. We formalise this as ΔQ ∝ ΔH(C) · σ(U), where σ(U) is the user’s prompt under specification, and verify that the model correctly predicts the monotonic ordering of context effects across user groups (3.5× ratio between novice and expert gains). We further show that the same context–prompt–receiver dynamics arise in live musical performance, where conductor gestures, orchestral generation, and heterogeneous audience reception exhibit structurally isomorphic behaviour. These results imply that single-score leaderboards are misleading, that current alignment methods may systematically optimise for perceived quality rather than factual correctness, and that evaluation frameworks should be explicitly user-parameterised.

Zitieren

APA

Kania, M., Klempous, R., Bąk, A., Chaczko, Z., Segen, J., Kulbacki, M., & Kulbacki, M. (2026). Context, User, and Truth: Evaluating LLM Response Quality as a Human–Model System Property. International Journal of Electronics and Telecommunications, 72(4), 1-8. https://doi.org/10.24425/ijet.2026.6313

Vancouver

Kania M, Klempous R, Bąk A, Chaczko Z, Segen J, Kulbacki M, et al. Context, User, and Truth: Evaluating LLM Response Quality as a Human–Model System Property. International Journal of Electronics and Telecommunications. 2026;72(4):1-8. doi:10.24425/ijet.2026.6313

BibTeX
@article{Kania2026,
  author = {Kania, Mariusz and Klempous, Ryszard and Bąk, Artur and Chaczko, Zenon and Segen, Jakub and Kulbacki, Michał and Kulbacki, Marek},
  title = {Context, User, and Truth: Evaluating LLM Response Quality as a Human–Model System Property},
  journal = {International Journal of Electronics and Telecommunications},
  year = {2026},
  volume = {72},
  number = {4},
  pages = {1--8},
  doi = {10.24425/ijet.2026.6313}
}
Kostenlos starten →