Characterizing the Increase in Artificial Intelligence Content Detection in Oncology Scientific Abstracts From 2021 to 2023
JCO Clinical Cancer Informatics · 2024
DOI: 10.1200/cci.24.00077 ↗Kurz gefasst
Fachartikel in der Zeitschrift JCO Clinical Cancer Informatics, 2024. Verfasst von Frederick M Howard und 4 weiteren Autor:innen. Beteiligte Einrichtung: Section of Hematology/Oncology, Department of Medicine, The University of Chicago, Chicago, IL. Hauptthemen laut MEDLINE: Artificial Intelligence, Abstracting and Indexing, Medical Oncology. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS, NIDCR NIH HHS.
Abstract (Originalsprache)
PURPOSE Artificial intelligence (AI) models can generate scientific abstracts that are difficult to distinguish from the work of human authors. The use of AI in scientific writing and performance of AI detection tools are poorly characterized. METHODS We extracted text from published scientific abstracts from the ASCO 2021-2023 Annual Meetings. Likelihood of AI content was evaluated by three detectors: GPTZero, Originality.ai, and Sapling. Optimal thresholds for AI content detection were selected using 100 abstracts from before 2020 as negative controls, and 100 produced by OpenAI's GPT-3 and GPT-4 models as positive controls. Logistic regression was used to evaluate the association of predicted AI content with submission year and abstract characteristics, and adjusted odds ratios (aORs) were computed. RESULTS Fifteen thousand five hundred and fifty-three abstracts met inclusion criteria. Across detectors, abstracts submitted in 2023 were significantly more likely to contain AI content than those in 2021 (aOR range from 1.79 with Originality to 2.37 with Sapling). Online-only publication and lack of clinical trial number were consistently associated with AI content. With optimal thresholds, 99.5%, 96%, and 97% of GPT-3/4–generated abstracts were identified by GPTZero, Originality, and Sapling respectively, and no sampled abstracts from before 2020 were classified as AI generated by the GPTZero and Originality detectors. Correlation between detectors was low to moderate, with Spearman correlation coefficient ranging from 0.14 for Originality and Sapling to 0.47 for Sapling and GPTZero. CONCLUSION There is an increasing signal of AI content in ASCO abstracts, coinciding with the growing popularity of generative AI models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Section of Hematology/Oncology, Department of Medicine, The University of Chicago, Chicago, IL.
- Center for Research and Analytics, American Society of Clinical Oncology, Alexandria, VA.
ORCID-Profile: Frederick M Howard, Elizabeth Garrett-Mayer, Alexander T Pearson
Förderung
- NCI NIH HHS
- NIDCR NIH HHS
Zitieren
APA
Howard, F. M., Li, A., Riffon, M. F., Garrett-Mayer, E., & Pearson, A. T. (2024). Characterizing the Increase in Artificial Intelligence Content Detection in Oncology Scientific Abstracts From 2021 to 2023. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci.24.00077
Vancouver
Howard FM, Li A, Riffon MF, Garrett-Mayer E, Pearson AT. Characterizing the Increase in Artificial Intelligence Content Detection in Oncology Scientific Abstracts From 2021 to 2023. JCO Clinical Cancer Informatics. 2024. doi:10.1200/cci.24.00077
BibTeX
@article{Howard2024,
author = {Howard, Frederick M and Li, Anran and Riffon, Mark F and Garrett-Mayer, Elizabeth and Pearson, Alexander T},
title = {Characterizing the Increase in Artificial Intelligence Content Detection in Oncology Scientific Abstracts From 2021 to 2023},
journal = {JCO Clinical Cancer Informatics},
year = {2024},
doi = {10.1200/cci.24.00077}
}
EUVIMED