Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays
International Journal of Artificial Intelligence in Education · 2025-09 · 35 (3) · 1178-1217
DOI: 10.1007/s40593-024-00426-w ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Artificial Intelligence in Education, 2025-09. Verfasst von Julian F Lohmann und 7 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Psychology of Learning and Instruction, Kiel University, Olshausenstrasse 75, 24118 Kiel, Germany. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Deutsche Forschungsgemeinschaft, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Christian-Albrechts-Universität zu Kiel.
Abstract (Originalsprache)
Recent investigations in automated essay scoring research imply that hybrid models, which combine feature engineering and the powerful tools of deep neural networks (DNNs), reach state-of-the-art performance. However, most of these findings are from holistic scoring tasks. In the present study, we use a total of four prompts from two different corpora consisting of both L1 and L2 learner essays annotated with trait scores (e.g., content, organization, and language quality). In our main experiments, we compare three variants of trait-specific models using different inputs: (1) models based on 220 linguistic features, (2) models using essay-level contextual embeddings from the distilled version of the pre-trained transformer BERT (DistilBERT), and (3) a hybrid model using both types of features. Results imply that when trait-specific models are trained based on a single resource, the feature-based models slightly outperform the embedding-based models. These differences are most prominent for the organization traits. The hybrid models outperform the single-resource models, indicating that linguistic features and embeddings indeed capture partially different aspects relevant for the assessment of essay traits. To gain more insights into the interplay between both feature types, we run addition and ablation tests for individual feature groups. Trait-specific addition tests across prompts indicate that the embedding-based models can most consistently be enhanced in content assessment when combined with morphological complexity features. Most consistent performance gains in the organization traits are achieved when embeddings are combined with length features, and most consistent performance gains in the assessment of the language traits when combined with lexical complexity, error, and occurrence features. Cross-prompt scoring again reveals slight advantages for the feature-based models.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute for Psychology of Learning and Instruction, Kiel University, Olshausenstrasse 75, 24118 Kiel, Germany.
- University of Hildesheim, Hildesheim, Germany.
- University of Applied Sciences and Arts Northwestern Switzerland, Basel, Switzerland.
- University of Teacher Education Zürich, Zürich, Switzerland.
- Leibniz Institute for Science and Mathematics Education at Kiel University, Kiel, Germany.
- CATALPA, FernUniversität in Hagen, Hagen, Germany.
ORCID-Profile: Julian F Lohmann, Fynn Junge, Jens Möller, Johanna Fleckenstein, Ruth Trüb, Stefan Keller, Thorben Jansen, Andrea Horbach
Förderung
- Deutsche Forschungsgemeinschaft
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Christian-Albrechts-Universität zu Kiel
Zitieren
APA
Lohmann, J. F., Junge, F., Möller, J., Fleckenstein, J., Trüb, R., Keller, S., Jansen, T., & Horbach, A. (2025). Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays. International Journal of Artificial Intelligence in Education, 35(3), 1178-1217. https://doi.org/10.1007/s40593-024-00426-w
Vancouver
Lohmann JF, Junge F, Möller J, Fleckenstein J, Trüb R, Keller S, et al. Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays. International Journal of Artificial Intelligence in Education. 2025;35(3):1178-1217. doi:10.1007/s40593-024-00426-w
BibTeX
@article{Lohmann2025,
author = {Lohmann, Julian F and Junge, Fynn and Möller, Jens and Fleckenstein, Johanna and Trüb, Ruth and Keller, Stefan and Jansen, Thorben and Horbach, Andrea},
title = {Neural Networks or Linguistic Features? - Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits Among L1- and L2-Learners’ Argumentative Essays},
journal = {International Journal of Artificial Intelligence in Education},
year = {2025},
volume = {35},
number = {3},
pages = {1178--1217},
doi = {10.1007/s40593-024-00426-w}
}
EUVIMED