EUVIMEDOpdag EUVIMED
Crossref

Accounting for Patient Characteristics in a Model‐Based Kappa of Agreement Between Two Experts' Ordinal Ratings

Kerrie P Nelson iD, Thomas J Zhou

Statistics in Medicine · 2025

DOI: 10.1002/sim.70141 ↗

At a glance

Journal article in Statistics in Medicine, 2025. Written by Kerrie P Nelson and 1 co-authors. Institution: Department of Biostatistics, Boston University, Boston, Massachusetts, USA. Major MEDLINE topics: Models, Statistical. 36 references. Funded by: NIH HHS, NCI NIH HHS, National Institutes of Health.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Cohen's kappa and other summary measures are often used in clinical studies to describe agreement and association between two experts' ordered categorical ratings. However, a key limitation of Cohen's kappa and similar measures is their inability to evaluate the impact of patient‐related factors such as family history and age on the agreement and association between experts. Strong agreement between experts is an essential component of effective clinical procedures where subjective interpretation of patients' images or test results by an expert is required, for example, in the visual assessment of breast density from a mammogram. Not accounting for important patient‐related factors can lead to inflated and biased assessments of agreement and association. In this article, our objective is to propose novel model‐based measures that appropriately account for the impact of patient‐related covariates on chance‐corrected agreement and association between two experts' ordinal ratings that overcome limitations of existing measures. Our population‐based approach is based on an ordinal generalized linear mixed model (GLMM). Rigorous simulation studies evaluating performance of the new model‐based measures in a broad range of settings are reported. Existing and new measures are compared in two clinical applications assessing breast density and multiple sclerosis. Key advantages of the new kappa measures over existing measures such as Cohen's kappa include incorporating patient‐related factors, robustness to underlying disease prevalence and marginal distributions of experts' ratings, and appropriately correcting for chance agreement. Sample R code is provided by the authors for application of proposed measures in other studies.

Subjects

MeSH terms (MEDLINE)

Models, StatisticalHumansFemaleComputer SimulationMammographyLinear ModelsBreast DensityObserver VariationMultiple SclerosisBreast Neoplasms

Author keywords

agreementassociationgeneralized linear mixed modelordinal ratingspatient factorsstratified kappa

Authors and institutions

  • Department of Biostatistics, Boston University, Boston, Massachusetts, USA.
  • Center for Biologics Evaluation and Research, Food and Drug Administration, Silver Springs, Maryland, USA.

ORCID profiles: Kerrie P Nelson

Funding

  • NIH HHS
  • NCI NIH HHS
  • National Institutes of Health
  • National Cancer Institute

Cite

APA

Nelson, K. P., & Zhou, T. J. (2025). Accounting for Patient Characteristics in a Model‐Based Kappa of Agreement Between Two Experts' Ordinal Ratings. Statistics in Medicine. https://doi.org/10.1002/sim.70141

Vancouver

Nelson KP, Zhou TJ. Accounting for Patient Characteristics in a Model‐Based Kappa of Agreement Between Two Experts' Ordinal Ratings. Statistics in Medicine. 2025. doi:10.1002/sim.70141

BibTeX
@article{Nelson2025,
  author = {Nelson, Kerrie P and Zhou, Thomas J},
  title = {Accounting for Patient Characteristics in a Model‐Based Kappa of Agreement Between Two Experts' Ordinal Ratings},
  journal = {Statistics in Medicine},
  year = {2025},
  doi = {10.1002/sim.70141}
}
Start for free →