EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-SA 4.0

Popular GenAI Chatbots Vary in Capabilities to Answer Academic Reference Questions

Lisa Shen

Evidence Based Library and Information Practice · 16.03.2026 · 21 (1) · 206-208

DOI: 10.18438/eblip30949 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Evidence Based Library and Information Practice, online erschienen am 16.03.2026. Verfasst von Lisa Shen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

A Review of: Whitfield, S., & Yang, S. Q. (2025). Evaluating AI language models for reference services: A comparative study of ChatGPT, Gemini, and Copilot. Internet Reference Services Quarterly, 29(2), 153-167. https://doi.org/10.1080/10875301.2025.2478861 Objective – To assess and compare the quality of responses to reference questions by popular generative artificial intelligent (GenAI) chatbots. Design – Content analysis. Setting – Web browser platforms of four GenAI chatbots. Subjects – Responses from ChatGPT 3.5, Chat GPT 4.0, Gemini, and Copilot to a set of 28 chat reference questions submitted by Rider University Library patrons between July 1, 2023, and May 14, 2024. Methods – Transcripts of 112 responses from the four chatbots were evaluated using a content analysis scheme adapted from a previous ChatGPT study (Yang & Mason, 2024). Both researchers independently rated every response using a 10-point scale on four categories, accuracy, relevance, friendliness, and instructiveness, and analyzed the results using inferential statistics. Main Results – Responses from Gemini received the highest total score (592 out of a possible 1,120 points), followed by ChatGPT 4.0 (542), ChatGPT 3.5 (502), and Copilot (433). However, every chatbot fluctuated greatly in their performance in accuracy and relevance. A single-factor ANOVA test also found statistically significant differences between the quality of the GenAI chatbots’ responses in relevance and friendliness, with Gemini and Copilot performing the best in these categories respectively. There were no statistically significant differences between the chatbots’ performances in accuracy or instructiveness, although Gemini held the highest mean score for instructiveness. Conclusion – The researchers concluded that popular GenAI chatbots should be used to supplement, not replace, the work of reference and instruction librarians, and noted the potentials for training GenAI to address basic or local, library-specific FAQs for after-hour reference support. The authors also advised librarians to stay current with the rapid development of chatbots and other GenAI tools and to continue differentiating librarianship competencies from services offered by these programs.

Zitieren

APA

Shen, L. (2026). Popular GenAI Chatbots Vary in Capabilities to Answer Academic Reference Questions. Evidence Based Library and Information Practice, 21(1), 206-208. https://doi.org/10.18438/eblip30949

Vancouver

Shen L. Popular GenAI Chatbots Vary in Capabilities to Answer Academic Reference Questions. Evidence Based Library and Information Practice. 2026;21(1):206-208. doi:10.18438/eblip30949

BibTeX
@article{Shen2026,
  author = {Shen, Lisa},
  title = {Popular GenAI Chatbots Vary in Capabilities to Answer Academic Reference Questions},
  journal = {Evidence Based Library and Information Practice},
  year = {2026},
  volume = {21},
  number = {1},
  pages = {206--208},
  doi = {10.18438/eblip30949}
}
Kostenlos starten →