EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Temporally Informed Distillation of Embedding Semantics: Beyond Continued Pretraining for Modeling Gender Ideology in Dated Texts

Yingqiu Ge, Jinghang Gu, Chu-Ren Huang

Data · 22.05.2026 · 11 (6) · 126

DOI: 10.3390/data11060126 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Data, online erschienen am 22.05.2026. Verfasst von Yingqiu Ge und 2 weiteren Autor:innen. Die Arbeit nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Research Grants Council of Hong Kong SAR, China, The Hong Kong Polytechnic University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Modeling historically situated gender ideology remains challenging for language models, as contemporary embeddings struggle to reflect temporally specific semantic structures beyond surface lexical patterns. Although large language models exhibit extensive general-purpose performance, their direct use with history-specific semantic analysis is limited by the distributional mismatch between contemporary training data and historical linguistic patterns. These constraints encourage the distillation of temporally based semantic knowledge into small student architectures. To solve this issue, we propose Temporally Informed Distillation of Embedding Semantics (TIDES), which integrates continued pretraining on temporally specific corpora with feature-level distillation from large embedding teachers. We evaluate TIDES across teacher architectures with distinct pretraining objectives. While continued pretraining provides lexical and syntactic adaptation, our results show that improvements in ideological modeling cannot be attributed to additional training exposure alone. Rather, teacher–student structural alignment is also critical to transfer effectiveness. Contrastive, encoder-aligned teachers yield substantially more stable preservation of fine-grained, historically situated semantic distinctions. These findings suggest that temporal ideology transfer is representation-dependent: ideological meaning can be shaped by the geometry and training objectives of embedding spaces. By introducing TIDES and providing evidence that architectural compatibility can influence ideological inheritance, this study advances a representation-centered account of modeling ideology in temporally grounded semantic research.

Förderung

  • Research Grants Council of Hong Kong SAR, China
  • The Hong Kong Polytechnic University

Zitieren

APA

Ge, Y., Gu, J., & Huang, C. (2026). Temporally Informed Distillation of Embedding Semantics: Beyond Continued Pretraining for Modeling Gender Ideology in Dated Texts. Data, 11(6), 126. https://doi.org/10.3390/data11060126

Vancouver

Ge Y, Gu J, Huang C. Temporally Informed Distillation of Embedding Semantics: Beyond Continued Pretraining for Modeling Gender Ideology in Dated Texts. Data. 2026;11(6):126. doi:10.3390/data11060126

BibTeX
@article{Ge2026,
  author = {Ge, Yingqiu and Gu, Jinghang and Huang, Chu-Ren},
  title = {Temporally Informed Distillation of Embedding Semantics: Beyond Continued Pretraining for Modeling Gender Ideology in Dated Texts},
  journal = {Data},
  year = {2026},
  volume = {11},
  number = {6},
  pages = {126},
  doi = {10.3390/data11060126}
}
Kostenlos starten →