Comparative evaluation of a language model and human specialists in the application of European guidelines for the management of inflammatory bowel diseases and malignancies
Endoscopy · 01.09.2024 · 56 (09) · 706-709
DOI: 10.1055/a-2289-5732 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Endoscopy, erschienen am 01.09.2024. Verfasst von Itai Ghersin und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel. Hauptthemen laut MEDLINE: Inflammatory Bowel Diseases, Practice Guidelines as Topic, Guideline Adherence, Colorectal Neoplasms. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 12 Literaturangaben.
Abstract (Originalsprache)
Background
Society guidelines on colorectal dysplasia screening, surveillance, and endoscopic management in inflammatory bowel disease (IBD) are complex, and physician adherence to them is suboptimal. We aimed to evaluate the use of ChatGPT, a large language model, in generating accurate guideline-based recommendations for colorectal dysplasia screening, surveillance, and endoscopic management in IBD in line with European Crohn’s and Colitis Organization (ECCO) guidelines.
Methods
30 clinical scenarios in the form of free text were prepared and presented to three separate sessions of ChatGPT and to eight gastroenterologists (four IBD specialists and four non-IBD gastroenterologists). Two additional IBD specialists subsequently assessed all responses provided by ChatGPT and the eight gastroenterologists, judging their accuracy according to ECCO guidelines.
Results
ChatGPT had a mean correct response rate of 87.8%. Among the eight gastroenterologists, the mean correct response rates were 85.8% for IBD experts and 89.2% for non-IBD experts. No statistically significant differences in accuracy were observed between ChatGPT and all gastroenterologists (P=0.95), or between ChatGPT and the IBD experts and non-IBD expert gastroenterologists, respectively (P=0.82).
Conclusions
This study highlights the potential of language models in enhancing guideline adherence regarding colorectal dysplasia in IBD. Further investigation of additional resources and prospective evaluation in real-world settings are warranted.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Gastroenterology, Rambam Health Care Campus, Haifa, Israel.
- Rappaport Faculty of Medicine, Technion, Israel Institute of Technology, Haifa, Israel.
- Department of Gastroenterology and Hepatology, Hillel Yaffe Medical Center, Hadera, Israel.
- Department of Gastroenterology, Galilee Medical Center, Nahariya, Israel.
ORCID-Profile: Yuri Gorelik
Zitieren
APA
Ghersin, I., Weisshof, R., Koifman, E., Bar-Yoseph, H., Ben Hur, D., Maza, I., Hasnis, E., Nasser, R., Ovadia, B., Dror Zur, D., Waterman, M., & Gorelik, Y. (2024). Comparative evaluation of a language model and human specialists in the application of European guidelines for the management of inflammatory bowel diseases and malignancies. Endoscopy, 56(09), 706-709. https://doi.org/10.1055/a-2289-5732
Vancouver
Ghersin I, Weisshof R, Koifman E, Bar-Yoseph H, Ben Hur D, Maza I, et al. Comparative evaluation of a language model and human specialists in the application of European guidelines for the management of inflammatory bowel diseases and malignancies. Endoscopy. 2024;56(09):706-709. doi:10.1055/a-2289-5732
BibTeX
@article{Ghersin2024,
author = {Ghersin, Itai and Weisshof, Roni and Koifman, Eduard and Bar-Yoseph, Haggai and Ben Hur, Dana and Maza, Itay and Hasnis, Erez and Nasser, Roni and Ovadia, Baruch and Dror Zur, Dikla and Waterman, Matti and Gorelik, Yuri},
title = {Comparative evaluation of a language model and human specialists in the application of European guidelines for the management of inflammatory bowel diseases and malignancies},
journal = {Endoscopy},
year = {2024},
volume = {56},
number = {09},
pages = {706--709},
doi = {10.1055/a-2289-5732}
}
EUVIMED