EUVIMEDEUVIMED entdecken
Crossref

Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study

Imed Sdiri, Manjet Kaur Mehar Singh

Theory and Practice in Language Studies · 03.07.2026 · 16 (7) · 2544-2552

DOI: 10.17507/tpls.1607.36 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Theory and Practice in Language Studies, online erschienen am 03.07.2026. Verfasst von Imed Sdiri und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reliability is a central concern in research on textbook evaluation, particularly when judgments depend on the subjective interpretation of pedagogical features. In the context of English as a Foreign Language (EFL) education, the Common European Framework of Reference for Languages (CEFR) has become the dominant benchmark for assessing curricular alignment. Yet, while numerous studies claim that textbooks are evaluated against CEFR guidelines, few report systematically on the reliability of such evaluations. This paper addresses this gap by examining inter-rater reliability in the evaluation of an EFL textbook marketed by a ‎renowned international publisher as aligned with the CEFR A2 ‎level. Three expert raters independently applied a validated 22-item evaluation instrument to 20 reading comprehension lessons. Agreement was measured using Fleiss’ Kappa ( ), a statistic specifically designed for categorical data involving multiple raters. The results revealed a consistently high level of inter-rater reliability, with a pooled Fleiss’ Kappa of 0.89. These findings confirm the robustness of the evaluation instrument and demonstrate that with rigorous rater training and independent coding, subjective bias in EFL textbook evaluation can be effectively minimized. The study contributes a methodological model that enhances the rigor of EFL textbook evaluation studies by advocating for evidence-based validation procedures.

Zitieren

APA

Sdiri, I., & Singh, M. K. M. (2026). Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study. Theory and Practice in Language Studies, 16(7), 2544-2552. https://doi.org/10.17507/tpls.1607.36

Vancouver

Sdiri I, Singh MKM. Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study. Theory and Practice in Language Studies. 2026;16(7):2544-2552. doi:10.17507/tpls.1607.36

BibTeX
@article{Sdiri2026,
  author = {Sdiri, Imed and Singh, Manjet Kaur Mehar},
  title = {Establishing Inter-Rater Reliability in CEFR-Based Textbook Evaluation: Evidence From a Fleiss’ Kappa Study},
  journal = {Theory and Practice in Language Studies},
  year = {2026},
  volume = {16},
  number = {7},
  pages = {2544--2552},
  doi = {10.17507/tpls.1607.36}
}
Kostenlos starten →