Benchmarking of Large Language Models for the Dental Admission Test
Health Data Science · 2025-01 · 5
DOI: 10.34133/hds.0250 ↗Kurz gefasst
Fachartikel in der Zeitschrift Health Data Science, 2025-04. Verfasst von Yu Hou und 7 weiteren Autor:innen. Beteiligte Einrichtung: Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, USA. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: National Institute on Aging, National cancer institute, National Institutes of Health’s National Center for Complementary and Integrative Health.
Abstract (Originalsprache)
Background
Large language models (LLMs) have shown promise in educational applications, but their performance on high-stakes admissions tests, such as the Dental Admission Test (DAT), remains unclear. Understanding the capabilities and limitations of these models is critical for determining their suitability in test preparation.
Methods
This study evaluated the ability of 16 LLMs, including general-purpose models (e.g., GPT-3.5, GPT-4, GPT-4o, GPT-o1, Google’s Bard, mistral-large, and Claude), domain-specific fine-tuned models (e.g., DentalGPT, MedGPT, and BioGPT), and open-source models (e.g., Llama2-7B, Llama2-13B, Llama2-70B, Llama3-8B, and Llama3-70B), to answer questions from a sample DAT. Quantitative analysis was performed to assess model accuracy in different sections, and qualitative thematic analysis by subject matter experts examined specific challenges encountered by the models.
Results
GPT-4o and GPT-o1 outperformed others in text-based questions assessing knowledge and comprehension, with GPT-o1 achieving perfect scores in the natural sciences (NS) and reading comprehension (RC) sections. Open-source models such as Llama3-70B also performed competitively in RC tasks. However, all models, including GPT-4o, struggled substantially with perceptual ability (PA) items, highlighting a persistent limitation in handling image-based tasks requiring visual-spatial reasoning. Fine-tuned medical models (e.g., DentalGPT, MedGPT, and BioGPT) demonstrated moderate success in text-based tasks but underperformed in areas requiring critical thinking and reasoning. Thematic analysis identified key challenges, including difficulties with stepwise problem-solving, transferring knowledge, comprehending intricate questions, and hallucinations, particularly on advanced items.
Conclusions
While LLMs show potential for reinforcing factual knowledge and supporting learners, their limitations in handling higher-order cognitive tasks and image-based reasoning underscore the need for judicious integration with instructor-led guidance and targeted practice. This study provides valuable insights into the capabilities and limitations of current LLMs in preparing prospective dental students and highlights pathways for future innovations to improve performance across all cognitive skills assessed by the DAT.
Autor:innen und Institutionen
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, USA.
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN 55455, USA.
- Kornberg School of Dentistry, Oral Health Sciences Department, Temple University, Philadelphia, PA 19140, USA.
- Wayzata High School, Plymouth, MN 55446, USA.
- School of Education, Syracuse University, Syracuse, NY 13244, USA.
- Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN, USA.
ORCID-Profile: Rui Zhang
Förderung
- National Institute on Aging
- National cancer institute
- National Institutes of Health’s National Center for Complementary and Integrative Health
Zitieren
APA
Hou, Y., Patel, J., Dai, L., Zhang, E., Liu, Y., Zhan, Z., Gangwani, P., & Zhang, R. (2025). Benchmarking of Large Language Models for the Dental Admission Test. Health Data Science, 5. https://doi.org/10.34133/hds.0250
Vancouver
Hou Y, Patel J, Dai L, Zhang E, Liu Y, Zhan Z, et al. Benchmarking of Large Language Models for the Dental Admission Test. Health Data Science. 2025;5. doi:10.34133/hds.0250
BibTeX
@article{Hou2025,
author = {Hou, Yu and Patel, Jay and Dai, Liya and Zhang, Emily and Liu, Yang and Zhan, Zaifu and Gangwani, Pooja and Zhang, Rui},
title = {Benchmarking of Large Language Models for the Dental Admission Test},
journal = {Health Data Science},
year = {2025},
volume = {5},
doi = {10.34133/hds.0250}
}
EUVIMED