EUVIMEDEUVIMED entdecken
Open Access

The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries

Michelle Cung, Branden Sosa iD, He S Yang, Michelle M McDonald iD, Brya G Matthews, Annegreet G Vlug, Erik A Imel iD, Marc N Wein iD, Emily Margaret Stein, Matthew B Greenblatt iD

Journal of Bone and Mineral Research · 22.03.2024 · 39 (2) · 106-115

DOI: 10.1093/jbmr/zjad007 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Bone and Mineral Research, online erschienen am 04.01.2024. Verfasst von Michelle Cung und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pathology and Laboratory Medicine, Weill Cornell Medical College, New York, NY 10065, United States. Hauptthemen laut MEDLINE: Artificial Intelligence, Bone and Bones. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIAMS NIH HHS, NIH HHS, NICHD NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Artificial intelligence (AI) chatbots utilizing large language models (LLMs) have recently garnered significant interest due to their ability to generate humanlike responses to user inquiries in an interactive dialog format. While these models are being increasingly utilized to obtain medical information by patients, scientific and medical providers, and trainees to address biomedical questions, their performance may vary from field to field. The opportunities and risks these chatbots pose to the widespread understanding of skeletal health and science are unknown. Here we assess the performance of 3 high-profile LLM chatbots, Chat Generative Pre-Trained Transformer (ChatGPT) 4.0, BingAI, and Bard, to address 30 questions in 3 categories: basic and translational skeletal biology, clinical practitioner management of skeletal disorders, and patient queries to assess the accuracy and quality of the responses. Thirty questions in each of these categories were posed, and responses were independently graded for their degree of accuracy by four reviewers. While each of the chatbots was often able to provide relevant information about skeletal disorders, the quality and relevance of these responses varied widely, and ChatGPT 4.0 had the highest overall median score in each of the categories. Each of these chatbots displayed distinct limitations that included inconsistent, incomplete, or irrelevant responses, inappropriate utilization of lay sources in a professional context, a failure to take patient demographics or clinical context into account when providing recommendations, and an inability to consistently identify areas of uncertainty in the relevant literature. Careful consideration of both the opportunities and risks of current AI chatbots is needed to formulate guidelines for best practices for their use as source of information about skeletal health and biology.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceBone and BonesHumansBone Diseases

Stichwörter der Autor:innen

BardBingAIChatGPTartificial intelligencelarge language modelsskeletal biology

Autor:innen und Institutionen

  • Department of Pathology and Laboratory Medicine, Weill Cornell Medical College, New York, NY 10065, United States.
  • Skeletal Diseases Program, The Garvan Institute of Medical Research, Darlinghurst, 2010, Australia.
  • St Vincent's Clinical Campus School of Clinical Medicine, University of New South Wales, Kensington, 2052, Australia.
  • School of Medicine Science, Faculty of Medicine and Health, The University of Sydney, Sydney, 2006, Australia.
  • Department of Molecular Medicine and Pathology, University of Auckland, Auckland, 1142, New Zealand.
  • Center for Regenerative Medicine and Skeletal Development, School of Dental Medicine, UConn Health, Farmington, CT 06030, United States.

ORCID-Profile: Branden Sosa, Michelle M McDonald, Erik A Imel, Marc N Wein, Matthew B Greenblatt

Förderung

  • NIAMS NIH HHS
  • NIH HHS
  • NICHD NIH HHS
  • NCI NIH HHS
  • Pershing Square Sohn Cancer Research Alliance
  • Pershing Square MIND Prize Awards
  • Irma T. Hirschl Career Scientist Award
  • NIH

Zitieren

APA

Cung, M., Sosa, B., Yang, H. S., McDonald, M. M., Matthews, B. G., Vlug, A. G., Imel, E. A., Wein, M. N., Stein, E. M., & Greenblatt, M. B. (2024). The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries. Journal of Bone and Mineral Research, 39(2), 106-115. https://doi.org/10.1093/jbmr/zjad007

Vancouver

Cung M, Sosa B, Yang HS, McDonald MM, Matthews BG, Vlug AG, et al. The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries. Journal of Bone and Mineral Research. 2024;39(2):106-115. doi:10.1093/jbmr/zjad007

BibTeX
@article{Cung2024,
  author = {Cung, Michelle and Sosa, Branden and Yang, He S and McDonald, Michelle M and Matthews, Brya G and Vlug, Annegreet G and Imel, Erik A and Wein, Marc N and Stein, Emily Margaret and Greenblatt, Matthew B},
  title = {The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries},
  journal = {Journal of Bone and Mineral Research},
  year = {2024},
  volume = {39},
  number = {2},
  pages = {106--115},
  doi = {10.1093/jbmr/zjad007}
}
Kostenlos starten →