EUVIMEDEUVIMED entdecken
Open Access

Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study

Joseph Mugaanyi iD, Liuying Cai iD, Sumei Cheng iD, Caide Lu iD, Jing Huang iD

Journal of Medical Internet Research · 05.04.2024 · 26 · e52935

DOI: 10.2196/52935 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 05.04.2024. Verfasst von Joseph Mugaanyi und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Hepato-Pancreato-Biliary Surgery, Ningbo Medical Center Lihuili Hospital, Health Science Center, Ningbo University, Ningbo, China. Hauptthemen laut MEDLINE: Artificial Intelligence, Language. Die Arbeit wurde bisher 82-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022.

Objective

The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities.

Methods

Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines.

Results

Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy.

Conclusions

ChatGPT’s performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceLanguageHumansReproducibility of ResultsResearch PersonnelWriting

Stichwörter der Autor:innen

AIChatGPTGPT-3.5LLMsNLPacademic discourseacademic writingaccuracyartificial intelligencechatbotcitationscross-disciplinary evaluationhumanitieslarge language modelsmachine learning algorithms

Autor:innen und Institutionen

  • Department of Hepato-Pancreato-Biliary Surgery, Ningbo Medical Center Lihuili Hospital, Health Science Center, Ningbo University, Ningbo, China.
  • Institute of Philosophy, Shanghai Academy of Social Sciences, Shanghai, China.

ORCID-Profile: Joseph Mugaanyi, Liuying Cai, Sumei Cheng, Caide Lu, Jing Huang

Zitieren

APA

Mugaanyi, J., Cai, L., Cheng, S., Lu, C., & Huang, J. (2024). Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study. Journal of Medical Internet Research, 26, e52935. https://doi.org/10.2196/52935

Vancouver

Mugaanyi J, Cai L, Cheng S, Lu C, Huang J. Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study. Journal of Medical Internet Research. 2024;26:e52935. doi:10.2196/52935

BibTeX
@article{Mugaanyi2024,
  author = {Mugaanyi, Joseph and Cai, Liuying and Cheng, Sumei and Lu, Caide and Huang, Jing},
  title = {Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e52935},
  doi = {10.2196/52935}
}
Kostenlos starten →