EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Evaluating Source-Based Large Language Models for Preclinical Dermatology Education: Comparative Study

Frank Je-Min Lin iD, Sunghun Cho iD

JMIR Formative Research · 25.06.2026 · 10 · e88008-e88008

DOI: 10.2196/88008 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Formative Research, online erschienen am 25.06.2026. Verfasst von Frank Je-Min Lin und 1 weiteren Autor:innen. Beteiligte Einrichtung: F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD, 20814, United States…. Hauptthemen laut MEDLINE: Large Language Models, Dermatology. Die Arbeit nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have gained increasing popularity in medical education, with evidence supporting their educational value when framed through the lens of cognitive load theory. Source-based LLMs, which explicitly ground responses in user-uploaded material via retrieval-augmented generation algorithms, may offer additional educational value by using student-developed materials to conceptualize new areas of learning within a familiar framework. This has applications for areas like medical education in dermatology, which could benefit from inclusive sources and enhanced education to alleviate health care gaps. However, no prior studies have examined whether the inclusion of student-authored notes alters the response characteristics of a source-based LLM when responding to medical questions.

Objective

This study aims to conduct an observational, comparative performance evaluation study assessing the accuracy, response reproducibility, and intermodel response similarity of freely available LLMs on text-only step 1 dermatology questions, and to explore whether providing extensive student-generated notes to a source-based LLM alters these performance characteristics.

Methods

In December 2024, 4 LLMs were evaluated: NotebookLM (NLM) with uploaded preclerkship study guides (NLM w/ Notes), NLM with an uploaded blank sheet of paper (NLM w/o Notes), ChatGPT-4o Mini, and Google Gemini 1.5 Flash. Each model completed 3 trials of 121 text-based United States Medical Licensing Examination (USMLE) step 1 dermatology questions from the AMBOSS question bank. They were evaluated for overall majority-consensus accuracy, accuracy by question difficulty, intertrial reproducibility, and agreement in answer choice selection between models. Differences were analyzed through a Cochran Q omnibus test and subsequent pairwise McNemar tests with Benjamini-Hochberg correction. Response reproducibility and intermodel agreement were analyzed through Fleiss κ statistics with 95% CI.

Results

ChatGPT-4o Mini achieved the highest overall majority-consensus accuracy (102/121, 84.3%). NLM w/ Notes demonstrated the highest intertrial reproducibility (Fleiss κ=0.927, 95% CI 0.875‐0.978) and strong performance on lower-difficulty questions but comparatively reduced accuracy on higher-difficulty items. NLM w/o Notes exhibited significantly higher omission rates (38/363, 10.5% vs ≤7/363, 1.92% for other models) than other tested LLMs. Sensitivity analysis excluding omissions increased NLM w/o Notes’ accuracy from 66.9% (81/121) to 77.8% (77/99), matching NLM w/ Notes’ accuracy of 74.4% (90/121). Intermodel agreement was significantly higher between NLM w/ Notes and ChatGPT-4o Mini compared to NLM w/o Notes and Gemini 1.5 Flash.

Conclusions

Provision of student-generated notes substantially increased response reproducibility in a source-based LLM, likely reflecting consistent retrieval of similar source excerpts across trials. However, note-grounding appeared to constrain performance on higher-difficulty questions, suggesting a retrieval-augmented generation algorithm retrieval error when question stems excluded characteristic “keywords” present in lower-difficulty items. The results highlight potential challenges of a student-level, cognitive load theory–grounded educational LLM that must deal with notes not curated by experts, balance source use and internal reasoning, and meaningfully appraise uploaded sources to assess a student’s individual learning gaps.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsDermatologyHumansReproducibility of ResultsEducational Measurement

Stichwörter der Autor:innen

AIAI in the classroomLLMNotebookLMartificial intelligencecognitive load theorylarge language modelretrieval-augmented generationsource-based LLM

Autor:innen und Institutionen

  • F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD, 20814, United States, 1 2532733100.
  • Department of Dermatology, Uniformed Services University of the Health Sciences, Bethesda, MD, United States.

ORCID-Profile: Frank Je-Min Lin, Sunghun Cho

Zitieren

APA

Lin, F. J., & Cho, S. (2026). Evaluating Source-Based Large Language Models for Preclinical Dermatology Education: Comparative Study. JMIR Formative Research, 10, e88008-e88008. https://doi.org/10.2196/88008

Vancouver

Lin FJ, Cho S. Evaluating Source-Based Large Language Models for Preclinical Dermatology Education: Comparative Study. JMIR Formative Research. 2026;10:e88008-e88008. doi:10.2196/88008

BibTeX
@article{Lin2026,
  author = {Lin, Frank Je-Min and Cho, Sunghun},
  title = {Evaluating Source-Based Large Language Models for Preclinical Dermatology Education: Comparative Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e88008--e88008},
  doi = {10.2196/88008}
}
Kostenlos starten →