EUVIMEDEUVIMED entdecken
ÜbersichtsarbeitOpen Access · CC BY 4.0

From tetrachoric to kappa: How to assess reliability on binary scales

Sophie Vanbelle iD

British Journal of Mathematical and Statistical Psychology · 2026

DOI: 10.1111/bmsp.70021 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift British Journal of Mathematical and Statistical Psychology, 2026. Verfasst von Sophie Vanbelle. Beteiligte Einrichtung: Department of Methodology and Statistics, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands. Hauptthemen laut MEDLINE: Psychometrics. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reliability is crucial in psychometrics, reflecting the extent to which a measurement instrument can discriminate between individuals or items. While classical test theory and intraclass correlation coefficients are well‐established for quantitative scales, estimating reliability for binary outcomes presents unique challenges due to their discrete nature. This paper reviews and links three major approaches to estimate reliability for single ratings on binary scales: the normal approximation approach, kappa coefficients, and the latent variable approach, which enables estimation at both latent and manifest scale levels. We clarify their conceptual relationships, show conditions for asymptotical equivalence, and evaluate their performance across two common study designs, repeatability and reproducibility studies. Then, we extend the Bayesian Dirichlet‐multinomial method for estimating kappa coefficients to settings with more than two replicates, without requiring Bayesian software. Additionally, we introduce a Bayesian method to estimate manifest scale reliability from latent scale reliability that can be implemented in standard Bayesian software. A simulation study compares the statistical properties of the three major approaches across Bayesian and frequentist frameworks. Overall, the normal approximation approach performed poorly, and the frequentist approach was unreliable due to singularity issues. The findings offer further refined practical recommendations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PsychometricsReproducibility of ResultsBayes TheoremHumansModels, StatisticalComputer SimulationData Interpretation, StatisticalSoftware

Stichwörter der Autor:innen

agreementinterrater reliabilityintraclass correlation coefficientintrarater reliabilityobserver variationreproducibility

Autor:innen und Institutionen

  • Department of Methodology and Statistics, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands.

ORCID-Profile: Sophie Vanbelle

Zitieren

APA

Vanbelle, S. (2026). From tetrachoric to kappa: How to assess reliability on binary scales. British Journal of Mathematical and Statistical Psychology. https://doi.org/10.1111/bmsp.70021

Vancouver

Vanbelle S. From tetrachoric to kappa: How to assess reliability on binary scales. British Journal of Mathematical and Statistical Psychology. 2026. doi:10.1111/bmsp.70021

BibTeX
@article{Vanbelle2026,
  author = {Vanbelle, Sophie},
  title = {From tetrachoric to kappa: How to assess reliability on binary scales},
  journal = {British Journal of Mathematical and Statistical Psychology},
  year = {2026},
  doi = {10.1111/bmsp.70021}
}
Kostenlos starten →