EUVIMEDEUVIMED entdecken
Mixed-Methods-StudieOpen Access

The Feasibility of Large Language Models in Verbal Comprehension Assessment: Mixed Methods Feasibility Study

Dorit Hadar-Shoval iD, Maya Lvovsky iD, Kfir Asraf iD, Yoav Shimoni iD, Zohar Elyoseph iD

JMIR Formative Research · 24.02.2025 · 9 · e68347

DOI: 10.2196/68347 ↗

Kurz gefasst

Mixed-Methods-Studie in der Zeitschrift JMIR Formative Research, online erschienen am 24.02.2025. Verfasst von Dorit Hadar-Shoval und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology, Max Stern Academic College of Emek Yezreel, Afula, Israel. Hauptthemen laut MEDLINE: Comprehension, Artificial Intelligence. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Cognitive assessment is an important component of applied psychology, but limited access and high costs make these evaluations challenging.

Objective

This study aimed to examine the feasibility of using large language models (LLMs) to create personalized artificial intelligence–based verbal comprehension tests (AI-BVCTs) for assessing verbal intelligence, in contrast with traditional assessment methods based on standardized norms.

Methods

We used a within-participants design, comparing scores obtained from AI-BVCTs with those from the Wechsler Adult Intelligence Scale (WAIS-III) verbal comprehension index (VCI). In total, 8 Hebrew-speaking participants completed both the VCI and AI-BVCT, the latter being generated using the LLM Claude.

Results

The concordance correlation coefficient (CCC) demonstrated strong agreement between AI-BVCT and VCI scores (Claude: CCC=.75, 90% CI 0.266-0.933; GPT-4: CCC=.73, 90% CI 0.170-0.935). Pearson correlations further supported these findings, showing strong associations between VCI and AI-BVCT scores (Claude: r=.84, P.05).

Conclusions

These findings support the potential of LLMs to assess verbal intelligence. The study attests to the promise of AI-based cognitive tests in increasing the accessibility and affordability of assessment processes, enabling personalized testing. The research also raises ethical concerns regarding privacy and overreliance on AI in clinical work. Further research with larger and more diverse samples is needed to establish the validity and reliability of this approach and develop more accurate scoring procedures.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

ComprehensionArtificial IntelligenceHumansFeasibility StudiesMaleFemaleAdultMiddle AgedWechsler ScalesYoung AdultLarge Language Models

Stichwörter der Autor:innen

AI in psychodiagnosticsWAIS-IIIWechsler Adult Intelligence Scaleartificial intelligenceethics in computerized cognitive assessmentlarge language modelspersonalized intelligence testspsychological test validityverbal comprehension assessmentverbal comprehension index

Autor:innen und Institutionen

  • Department of Psychology, Max Stern Academic College of Emek Yezreel, Afula, Israel.
  • School of Psychology, University of Haifa, Haifa, Israel.
  • Department of Education and Psychology, Open University of Israel, Raanana, Israel.
  • Department of Brain Sciences, Faculty of Medicine, Imperial College London, London, United Kingdom.
  • School of Counseling and Human Development, Faculty of Education, University of Haifa, Haifa, Israel.

ORCID-Profile: Dorit Hadar-Shoval, Maya Lvovsky, Kfir Asraf, Yoav Shimoni, Zohar Elyoseph

Zitieren

APA

Hadar-Shoval, D., Lvovsky, M., Asraf, K., Shimoni, Y., & Elyoseph, Z. (2025). The Feasibility of Large Language Models in Verbal Comprehension Assessment: Mixed Methods Feasibility Study. JMIR Formative Research, 9, e68347. https://doi.org/10.2196/68347

Vancouver

Hadar-Shoval D, Lvovsky M, Asraf K, Shimoni Y, Elyoseph Z. The Feasibility of Large Language Models in Verbal Comprehension Assessment: Mixed Methods Feasibility Study. JMIR Formative Research. 2025;9:e68347. doi:10.2196/68347

BibTeX
@article{HadarShoval2025,
  author = {Hadar-Shoval, Dorit and Lvovsky, Maya and Asraf, Kfir and Shimoni, Yoav and Elyoseph, Zohar},
  title = {The Feasibility of Large Language Models in Verbal Comprehension Assessment: Mixed Methods Feasibility Study},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e68347},
  doi = {10.2196/68347}
}
Kostenlos starten →