Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study
Theory and Practice in Language Studies · 03.07.2026 · 16 (7) · 2544-2552
DOI: 10.17507/tpls.1607.36 ↗Kurz gefasst
Fachartikel in der Zeitschrift Theory and Practice in Language Studies, online erschienen am 03.07.2026. Verfasst von Imed Sdiri und 1 weiteren Autor:innen.
Abstract (Originalsprache)
Reliability is a central concern in research on textbook evaluation, particularly when judgments depend on the subjective interpretation of pedagogical features. In the context of English as a Foreign Language (EFL) education, the Common European Framework of Reference for Languages (CEFR) has become the dominant benchmark for assessing curricular alignment. Yet, while numerous studies claim that textbooks are evaluated against CEFR guidelines, few report systematically on the reliability of such evaluations. This paper addresses this gap by examining inter-rater reliability in the evaluation of an EFL textbook marketed by a renowned international publisher as aligned with the CEFR A2 level. Three expert raters independently applied a validated 22-item evaluation instrument to 20 reading comprehension lessons. Agreement was measured using Fleiss’ Kappa ( ), a statistic specifically designed for categorical data involving multiple raters. The results revealed a consistently high level of inter-rater reliability, with a pooled Fleiss’ Kappa of 0.89. These findings confirm the robustness of the evaluation instrument and demonstrate that with rigorous rater training and independent coding, subjective bias in EFL textbook evaluation can be effectively minimized. The study contributes a methodological model that enhances the rigor of EFL textbook evaluation studies by advocating for evidence-based validation procedures.
Zitieren
APA
Sdiri, I., & Singh, M. K. M. (2026). Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study. Theory and Practice in Language Studies, 16(7), 2544-2552. https://doi.org/10.17507/tpls.1607.36
Vancouver
Sdiri I, Singh MKM. Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study. Theory and Practice in Language Studies. 2026;16(7):2544-2552. doi:10.17507/tpls.1607.36
BibTeX
@article{Sdiri2026,
author = {Sdiri, Imed and Singh, Manjet Kaur Mehar},
title = {Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study},
journal = {Theory and Practice in Language Studies},
year = {2026},
volume = {16},
number = {7},
pages = {2544--2552},
doi = {10.17507/tpls.1607.36}
}
EUVIMED