Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation
Algorithms · 05.11.2026 · 19 (11) · 967
DOI: 10.3390/a19110967 ↗Kurz gefasst
Fachartikel in der Zeitschrift Algorithms, online erschienen am 05.11.2026. Verfasst von Chi-Yi Hsieh und 5 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) generate fluent Chinese, but controlling lexical difficulty remains challenging. This study examined a Python-orchestrated external-control workflow using one GPT model, two sequential prompting conditions (no-word-list and lexicon-aware), three post-generation revision strategies (word-level substitution, sentence-level rewriting, and downstream regeneration), and four Test of Chinese as a Foreign Language (TOCFL) levels (A1–B2). Automatic indicators covered 96 initial drafts and 288 revision outputs. For 24 selected tasks, two experts directly rated 180 text instances; 12 unchanged outputs inherited initial-draft ratings, yielding 192 condition-specific expert-evaluation records. Task-matched comparisons used 20,000 TOCFL-stratified task-cluster bootstrap resamples and Holm-adjusted tests; automatic–expert associations were analyzed using Spearman correlations with task-cluster confidence intervals. Lexicon-aware prompting was associated with fewer initial out-of-level words and less revision, although the prompting batches were sequential. Word-level substitution and sentence-level rewriting generally showed greater source-relative embedding resemblance, whereas downstream regeneration received higher expert ratings for several language-quality dimensions but not consistently for level appropriateness. Residual out-of-level word density showed a weak, inconclusive pooled association with level appropriateness. Whole-text cosine similarity was negatively associated with semantic plausibility overall but positively associated within word-level substitution. Automatic indicators and source-blind expert judgments thus provide complementary, non-interchangeable evidence.
Zitieren
APA
Hsieh, C., Hsu, C. H., Hsieh, C., Huang, Y., Chen, I., & Wu, Y. (2026). Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation. Algorithms, 19(11), 967. https://doi.org/10.3390/a19110967
Vancouver
Hsieh C, Hsu CH, Hsieh C, Huang Y, Chen I, Wu Y. Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation. Algorithms. 2026;19(11):967. doi:10.3390/a19110967
BibTeX
@article{Hsieh2026,
author = {Hsieh, Chi-Yi and Hsu, Caroline Hsiu-Fang and Hsieh, Chi-Wen and Huang, Yi-Chi and Chen, I-Yu and Wu, Yu-Han},
title = {Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation},
journal = {Algorithms},
year = {2026},
volume = {19},
number = {11},
pages = {967},
doi = {10.3390/a19110967}
}
EUVIMED