EUVIMEDEUVIMED entdecken
Open Access

Large Language Model Risk of Bias Assessment for Studies in Environmental Health: Proof-of-Concept Study

Gaelen P Adam, Eitan Agai, Riaz Qureshi, Naeem Meman, Stephanie Qureshi, Winifred W Yu, Juleen Lam, Karen A Robinson

JMIR Bioinformatics and Biotechnology · 09.11.2026 · 7 · e99716-e99716

DOI: 10.2196/99716 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Bioinformatics and Biotechnology, online erschienen am 09.11.2026. Verfasst von Gaelen P Adam und 7 weiteren Autor:innen. Beteiligte Einrichtung: Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Mai…. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIEHS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The assessment of the risk of bias (RoB) of each study is a necessary but labor-intensive step in conducting a systematic review (SR). There is a potential for AI to reduce the time and effort needed for RoB assessment and improve consistency between reviewers, but little is known about the accuracy of AI-based RoB assessments.

Objective

In this study, we aim to evaluate the performance of a large language model (LLM) when given the questions from the RoB tool and the relevant passages from the full-text papers in assessing RoB in observational studies of environmental exposures and health outcomes.

Methods

We evaluated the performance of GPT-5-mini on 128 observational studies from a SR of per- and polyfluoroalkyl substances and health outcomes conducted by the National Academies of Sciences. The navigation guide (NavGuide), a domain-based RoB tool designed for environmental studies, was used. For each paper-domain pair, the LLM was provided with the questions and guidance from the NavGuide and the human-identified text passages addressing that domain. The LLM returned a structured RoB rating, which was compared to the human-adjudicated ratings to quantify agreement and identify patterns of discrepancy.

Results

GPT-5-mini RoB assessments for exact matches (64.5%; κ=0.199, Gwet AC2=0.882, F 1 -score=0.311) remained lower than the concordance between humans and the reference standard (87.5%-91.4%). Performance improved substantially for partial matches, reflecting agreement in direction but not necessarily in magnitude (eg, “low” and “probably low”), with percent concordance of 91.9% (κ=0.241, Gwet AC2=0.910, F 1 -score=0.614; for humans, 97.9%-98.6% concordance). Performance varied across the NavGuide domains, with the worst performance in domain 1 (selection bias) and the best in domain 4 (outcome assessment). The models tended to be more stringent than humans, often assigning higher RoB ratings (eg, “probably low” when humans assigned “low”).

Conclusions

Our results suggest that although concordance was moderate and inconsistent across domains, LLMs may be sufficiently accurate for use with human oversight. LLMs can flag cases for human adjudication or consistency checking but are not ready for use as an independent second reviewer without human oversight.

Schlagwörter

Stichwörter der Autor:innen

AIenvironmental healthlarge language modelsrisk of bias assessmentsystematic review methods

Autor:innen und Institutionen

  • Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Main St, Providence, RI, 02903, United States, 1 617-470-5685.
  • PICO Portal, St. Petersburg, FL, United States.
  • Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States.
  • Department of Epidemiology, School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO, United States.
  • Department of Public Health, California State University, East Bay, Hayward, CA, United States.
  • Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, MD, United States.

Förderung

  • NIEHS NIH HHS

Zitieren

APA

Adam, G. P., Agai, E., Qureshi, R., Meman, N., Qureshi, S., Yu, W. W., Lam, J., & Robinson, K. A. (2026). Large Language Model Risk of Bias Assessment for Studies in Environmental Health: Proof-of-Concept Study. JMIR Bioinformatics and Biotechnology, 7, e99716-e99716. https://doi.org/10.2196/99716

Vancouver

Adam GP, Agai E, Qureshi R, Meman N, Qureshi S, Yu WW, et al. Large Language Model Risk of Bias Assessment for Studies in Environmental Health: Proof-of-Concept Study. JMIR Bioinformatics and Biotechnology. 2026;7:e99716-e99716. doi:10.2196/99716

BibTeX
@article{Adam2026,
  author = {Adam, Gaelen P and Agai, Eitan and Qureshi, Riaz and Meman, Naeem and Qureshi, Stephanie and Yu, Winifred W and Lam, Juleen and Robinson, Karen A},
  title = {Large Language Model Risk of Bias Assessment for Studies in Environmental Health: Proof-of-Concept Study},
  journal = {JMIR Bioinformatics and Biotechnology},
  year = {2026},
  volume = {7},
  pages = {e99716--e99716},
  doi = {10.2196/99716}
}
Kostenlos starten →