EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Assessing the reliability and relevance of DeepSeek in EFL writing evaluation: a generalizability theory approach

Huixin Gao, Harwati Hashim, Melor Md Yunus

Language Testing in Asia · 19.06.2025 · 15 (1)

DOI: 10.1186/s40468-025-00369-6 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Language Testing in Asia, online erschienen am 19.06.2025. Verfasst von Huixin Gao und 2 weiteren Autor:innen. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study explored the potential of DeepSeek to contribute to English writing assessment via generalizability (G-) theory and qualitative feedback analysis. Specifically, it assessed the reliability of holistic scores and qualitative feedback generated by DeepSeek-V3 and DeepSeek-R1 for essays written in English as foreign language (EFL) learners, and compared these scores with those assigned by four college English teachers. The data consisted of 92 College English Test Band 4 (CET-4) essays written by non-English majors at a university in Heilongjiang Province, China. All the essays were holistically scored by DeepSeek-V3, R1, and four college English teachers. In addition, all three groups of raters provided qualitative feedback on content, language use, organization, and coherence. G-theory analysis revealed that the scoring reliability of DeepSeek-V3 was consistently lower than that of DeepSeek-R1 and the teacher raters; however, DeepSeek-R1 demonstrated consistently higher reliability coefficients than the teachers did. The qualitative feedback analysis indicated that both DeepSeek-V3 and R1 consistently provided more relevant feedback on the EFL essays than did the teacher raters. Furthermore, DeepSeek-V3 and R1 were equally relevant across the content, language use, organization, and coherence aspects of the essays, whereas the teacher raters generally focused more on language use but provided less relevant feedback on content, organization, and coherence. Consequently, DeepSeek-V3 and R1 could be useful AI tools for enhancing EFL writing assessments. The implications of adopting DeepSeek for classroom writing assessments are discussed.

Zitieren

APA

Gao, H., Hashim, H., & Yunus, M. M. (2025). Assessing the reliability and relevance of DeepSeek in EFL writing evaluation: a generalizability theory approach. Language Testing in Asia, 15(1). https://doi.org/10.1186/s40468-025-00369-6

Vancouver

Gao H, Hashim H, Yunus MM. Assessing the reliability and relevance of DeepSeek in EFL writing evaluation: a generalizability theory approach. Language Testing in Asia. 2025;15(1). doi:10.1186/s40468-025-00369-6

BibTeX
@article{Gao2025,
  author = {Gao, Huixin and Hashim, Harwati and Yunus, Melor Md},
  title = {Assessing the reliability and relevance of DeepSeek in EFL writing evaluation: a generalizability theory approach},
  journal = {Language Testing in Asia},
  year = {2025},
  volume = {15},
  number = {1},
  doi = {10.1186/s40468-025-00369-6}
}
Kostenlos starten →