RAPID: Reliable and efficient Automatic generation of submission rePortIng checklists with large language moDels
Journal of the American Medical Informatics Association · 01.08.2025 · 32 (8) · 1340-1349
DOI: 10.1093/jamia/ocaf093 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 17.06.2025. Verfasst von Zeming Li und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR 999077, China. Hauptthemen laut MEDLINE: Checklist, Artificial Intelligence, Research Report, Information Storage and Retrieval. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Young Collaborative Research, Health and Medical Research Fund, Guangdong-Hong Kong Technology.
Abstract (Originalsprache)
Objective
To evaluate an automated reporting checklist generation tool using large language models and retrieval augmentation generation technology, called RAPID. Materials and Methods This study utilized large language models to develop a retrieval augmentation generation architecture. To assess its performance, a total of 91 published journal articles were collected and manually annotated in accordance with the CONSORT and CONSORT-AI medical reporting guidelines. These articles comprised 50 randomized controlled trials conducted without AI intervention and 41 randomized controlled trials that incorporated AI tools.
Results
Fifty RCT articles without the intervention of AI tools and 41 RCT articles with the intervention of AI tools were collected as CONSORT and CONSORT-AI datasets. All of the CONSORT reporting items (37) were included in the tool. RAPID achieved a high average accuracy rate of 92.11% and a content consistency score of 81.14% on the CONSORT dataset. Of the CONSORT-AI reporting items, 11 items related to the intervention of AI tools were included in the tool. RAPID achieved an average accuracy of 83.81% with a content consistency score of 72.51% on the CONSORT-AI dataset.
Discussion
RAPID may effectively save time and improve working efficiency for different user groups such as medical authors, researchers, editors, and reviewers.
Conclusion
RAPID has strong scalability, which can be easily adapted to different medical reporting guidelines without transfer learning on a large dataset. RAPID got state-of-the-art performance on 2 datasets for 2 different checklists compared to other methods.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR 999077, China.
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China.
- Vincent V.C. Woo Chinese Medicine Clinical Research Institute, School of Chinese Medicine, Hong Kong Baptist University, Hong Kong SAR 999077, China.
- School of Public Health, Lanzhou University, Lanzhou 730000, China.
- Chinese EQUATOR Centre, Hong Kong Baptist University, Hong Kong SAR 999077, China.
- Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, United States.
ORCID-Profile: Long Ge, Yaolong Chen, Lu Zhang
Förderung
- Young Collaborative Research
- Health and Medical Research Fund
- Guangdong-Hong Kong Technology
- Research Unit of Evidence-Based Evaluation and Guidelines
- Chinese Academy of Medical Sciences
- Vincent and Lily Woo Foundation
- Guangdong-Hong Kong Technology Cooperation Funding Scheme
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences
Zitieren
APA
Li, Z., Luo, X., Yang, Z., Zhang, H., Wang, B., Ge, L., Bian, Z., Zou, J., Chen, Y., Zhang, L., & ADVANCED Working Group (2025). RAPID: Reliable and efficient Automatic generation of submission rePortIng checklists with large language moDels. Journal of the American Medical Informatics Association, 32(8), 1340-1349. https://doi.org/10.1093/jamia/ocaf093
Vancouver
Li Z, Luo X, Yang Z, Zhang H, Wang B, Ge L, et al. RAPID: Reliable and efficient Automatic generation of submission rePortIng checklists with large language moDels. Journal of the American Medical Informatics Association. 2025;32(8):1340-1349. doi:10.1093/jamia/ocaf093
BibTeX
@article{Li2025,
author = {Li, Zeming and Luo, Xufei and Yang, Zhenhua and Zhang, Huayu and Wang, Bingyi and Ge, Long and Bian, Zhaoxiang and Zou, James and Chen, Yaolong and Zhang, Lu and ADVANCED Working Group, ADVANCED Working Group},
title = {RAPID: Reliable and efficient Automatic generation of submission rePortIng checklists with large language moDels},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {8},
pages = {1340--1349},
doi = {10.1093/jamia/ocaf093}
}
EUVIMED