EUVIMEDDescubrir EUVIMED
VergleichsstudieOpen access

Using Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and Human-Annotated Bibliographies

Michelle Colder Carras iD, Riaz Qureshi iD, Kevin Naaman iD, Faisal Aldayel, Mayank Date iD, Dahlia AlJuboori iD, Johannes Thrul iD

JMIR Formative Research · 2026-02-12 · 10 · e69707-e69707

DOI: 10.2196/69707 ↗

At a glance

Vergleichsstudie in JMIR Formative Research, published online on 2026-02-12. Written by Michelle Colder Carras and 6 co-authors. Institution: Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe St, Baltimore, MD, 21205, United States, 1 410-955-3…. Major MEDLINE topics: Large Language Models, Artificial Intelligence. Cited 2 times so far (Crossref); 52 references. The full text is freely available in PubMed Central.

ResumenSubjectsDatos bibliográficosCiteRelated studies

Abstract (original language)

Background

Annotated bibliographies summarize literature, but training, experience, and time are needed to create concise yet accurate annotations. Summaries generated by artificial intelligence (AI) can save human resources, but AI-generated content can also contain serious errors.

Objective

To determine the feasibility of using AI as an alternative to human annotators, we explored whether ChatGPT can generate annotations with characteristics that are comparable to those written by humans.

Methods

We had 2 humans and 3 versions of ChatGPT (3.5, 4, and 5) independently write annotations on the same set of 15 publications. We collected data on word count and Flesch Reading Ease (FRE). In this study, 2 assessors who were masked to the source of the annotations independently evaluated (1) capture of main points, (2) presence of errors, and (3) whether the annotation included a discussion of both the quality and context of the article within the broader literature. We evaluated agreement and disagreement between the assessors and used descriptive statistics and assessor-stratified binary and cumulative mixed-effects logit models to compare annotations written by ChatGPT and humans.

Results

On average, humans wrote shorter annotations (mean 90.20, SD 36.8 words) than ChatGPT (mean 113, SD 16 words) which were easier to interpret (human FRE score, mean 15.3, SD 12.4; ChatGPT FRE score, mean 5.76, SD 7.32). Our assessments of agreement and disagreement revealed that one assessor was consistently stricter than the other. However, assessor-stratified models of main points, errors, and quality/context showed similar qualitative conclusions. There was no statistically significant difference in the odds of presenting a better summary of main points between ChatGPT- and human-generated annotations for either assessor (Assessor 1: OR 0.96, 95% CI 0.12‐7.71; Assessor 2: OR 1.64, 95% CI 0.67‐4.06). However, both assessors observed that human annotations had lower odds of having one or more types of errors compared to ChatGPT (Assessor 1: OR 0.31, 95% CI 0.09‐1.02; Assessor 2: OR 0.10, 95% CI 0.03‐0.33). On the other hand, human annotations also had lower odds of summarizing the paper’s quality and context when compared to ChatGPT (Assessor 1: OR 0.11, 95% CI 0.03‐0.33; Assessor 2: OR 0.03, 95% CI 0.01‐0.10). That said, ChatGPT’s summaries of quality and context were sometimes inaccurate.

Conclusions

Rapidly learning a body of scientific literature is a vital yet daunting task that may be made more efficient by AI tools. In our study, ChatGPT quickly generated concise summaries of academic literature and also provided quality and context more consistently than humans. However, ChatGPT’s discussion of the quality and context was not always accurate, and ChatGPT annotations included more errors. Annotated bibliographies that are AI-generated and carefully verified by humans may thus be an efficient way to provide a rapid overview of literature. More research is needed to determine the extent that prompt engineering can reduce errors and improve chatbot performance.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsArtificial IntelligenceHumansGenerative Artificial Intelligence

Author keywords

ChatGPTannotated bibliographyartificial intelligenceevidence synthesisinformation managementlarge language model

Authors and institutions

  • Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe St, Baltimore, MD, 21205, United States, 1 410-955-3934.
  • Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States.
  • Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, United States.

ORCID profiles: Michelle Colder Carras, Riaz Qureshi, Kevin Naaman, Mayank Date, Dahlia AlJuboori, Johannes Thrul

Cite

APA

Colder Carras, M., Qureshi, R., Naaman, K., Aldayel, F., Date, M., AlJuboori, D., & Thrul, J. (2026). Using Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and Human-Annotated Bibliographies. JMIR Formative Research, 10, e69707-e69707. https://doi.org/10.2196/69707

Vancouver

Colder Carras M, Qureshi R, Naaman K, Aldayel F, Date M, AlJuboori D, et al. Using Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and Human-Annotated Bibliographies. JMIR Formative Research. 2026;10:e69707-e69707. doi:10.2196/69707

BibTeX
@article{ColderCarras2026,
  author = {Colder Carras, Michelle and Qureshi, Riaz and Naaman, Kevin and Aldayel, Faisal and Date, Mayank and AlJuboori, Dahlia and Thrul, Johannes},
  title = {Using Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and Human-Annotated Bibliographies},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e69707--e69707},
  doi = {10.2196/69707}
}
Start for free →