EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation

Chi-Yi Hsieh, Caroline Hsiu-Fang Hsu, Chi-Wen Hsieh, Yi-Chi Huang, I-Yu Chen, Yu-Han Wu

Algorithms · 05.11.2026 · 19 (11) · 967

DOI: 10.3390/a19110967 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Algorithms, online erschienen am 05.11.2026. Verfasst von Chi-Yi Hsieh und 5 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) generate fluent Chinese, but controlling lexical difficulty remains challenging. This study examined a Python-orchestrated external-control workflow using one GPT model, two sequential prompting conditions (no-word-list and lexicon-aware), three post-generation revision strategies (word-level substitution, sentence-level rewriting, and downstream regeneration), and four Test of Chinese as a Foreign Language (TOCFL) levels (A1–B2). Automatic indicators covered 96 initial drafts and 288 revision outputs. For 24 selected tasks, two experts directly rated 180 text instances; 12 unchanged outputs inherited initial-draft ratings, yielding 192 condition-specific expert-evaluation records. Task-matched comparisons used 20,000 TOCFL-stratified task-cluster bootstrap resamples and Holm-adjusted tests; automatic–expert associations were analyzed using Spearman correlations with task-cluster confidence intervals. Lexicon-aware prompting was associated with fewer initial out-of-level words and less revision, although the prompting batches were sequential. Word-level substitution and sentence-level rewriting generally showed greater source-relative embedding resemblance, whereas downstream regeneration received higher expert ratings for several language-quality dimensions but not consistently for level appropriateness. Residual out-of-level word density showed a weak, inconclusive pooled association with level appropriateness. Whole-text cosine similarity was negatively associated with semantic plausibility overall but positively associated within word-level substitution. Automatic indicators and source-blind expert judgments thus provide complementary, non-interchangeable evidence.

Zitieren

APA

Hsieh, C., Hsu, C. H., Hsieh, C., Huang, Y., Chen, I., & Wu, Y. (2026). Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation. Algorithms, 19(11), 967. https://doi.org/10.3390/a19110967

Vancouver

Hsieh C, Hsu CH, Hsieh C, Huang Y, Chen I, Wu Y. Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation. Algorithms. 2026;19(11):967. doi:10.3390/a19110967

BibTeX
@article{Hsieh2026,
  author = {Hsieh, Chi-Yi and Hsu, Caroline Hsiu-Fang and Hsieh, Chi-Wen and Huang, Yi-Chi and Chen, I-Yu and Wu, Yu-Han},
  title = {Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation},
  journal = {Algorithms},
  year = {2026},
  volume = {19},
  number = {11},
  pages = {967},
  doi = {10.3390/a19110967}
}
Kostenlos starten →