Evaluating Embeddings from Pre-Trained Language Models and Knowledge Graphs for Educational Content Recommendation
Future Internet · 29.12.2023 · 16 (1) · 12
DOI: 10.3390/fi16010012 ↗Kurz gefasst
Fachartikel in der Zeitschrift Future Internet, online erschienen am 29.12.2023. Verfasst von Xiu Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Educational content recommendation is a cornerstone of AI-enhanced learning. In particular, to facilitate navigating the diverse learning resources available on learning platforms, methods are needed for automatically linking learning materials, e.g., in order to recommend textbook content based on exercises. Such methods are typically based on semantic textual similarity (STS) and the use of embeddings for text representation. However, it remains unclear what types of embeddings should be used for this task. In this study, we carry out an extensive empirical evaluation of embeddings derived from three different types of models: (i) static embeddings trained using a concept-based knowledge graph, (ii) contextual embeddings from a pre-trained language model, and (iii) contextual embeddings from a large language model (LLM). In addition to evaluating the models individually, various ensembles are explored based on different strategies for combining two models in an early vs. late fusion fashion. The evaluation is carried out using digital textbooks in Swedish for three different subjects and two types of exercises. The results show that using contextual embeddings from an LLM leads to superior performance compared to the other models, and that there is no significant improvement when combining these with static embeddings trained using a knowledge graph. When using embeddings derived from a smaller language model, however, it helps to combine them with knowledge graph embeddings. The performance of the best-performing model is high for both types of exercises, resulting in a mean Recall@3 of 0.96 and 0.95 and a mean MRR of 0.87 and 0.86 for quizzes and study questions, respectively, demonstrating the feasibility of using STS based on text embeddings for educational content recommendation. The ability to link digital learning materials in an unsupervised manner—relying only on readily available pre-trained models—facilitates the development of AI-enhanced learning.
Zitieren
APA
Li, X., Henriksson, A., Duneld, M., Nouri, J., & Wu, Y. (2023). Evaluating Embeddings from Pre-Trained Language Models and Knowledge Graphs for Educational Content Recommendation. Future Internet, 16(1), 12. https://doi.org/10.3390/fi16010012
Vancouver
Li X, Henriksson A, Duneld M, Nouri J, Wu Y. Evaluating Embeddings from Pre-Trained Language Models and Knowledge Graphs for Educational Content Recommendation. Future Internet. 2023;16(1):12. doi:10.3390/fi16010012
BibTeX
@article{Li2023,
author = {Li, Xiu and Henriksson, Aron and Duneld, Martin and Nouri, Jalal and Wu, Yongchao},
title = {Evaluating Embeddings from Pre-Trained Language Models and Knowledge Graphs for Educational Content Recommendation},
journal = {Future Internet},
year = {2023},
volume = {16},
number = {1},
pages = {12},
doi = {10.3390/fi16010012}
}
EUVIMED