EUVIMEDSkopri EUVIMED
Open access · CC BY 4.0

Language Modeling with Reduced Densities

Tai-Danae Bradley, Yiannis Vlassopoulos

Compositionality · 2021

DOI: 10.32408/compositionality-3-4 ↗

At a glance

Journal article in Compositionality, 2021. Written by Tai-Danae Bradley and 1 co-authors. Cited 4 times so far (Crossref); 57 references. The full text is freely available (open access, CC BY 4.0).

AstrattDettalji biblijografiċiCiteRelated studies

Abstract (original language)

This work originates from the observation that today's state-of-the-art statistical language models are impressive not only for their performance, but also---and quite crucially---because they are built entirely from correlations in unstructured text data. The latter observation prompts a fundamental question that lies at the heart of this paper: What mathematical structure exists in unstructured text data? We put forth enriched category theory as a natural answer. We show that sequences of symbols from a finite alphabet, such as those found in a corpus of text, form a category enriched over probabilities. We then address a second fundamental question: How can this information be stored and modeled in a way that preserves the categorical structure? We answer this by constructing a functor from our enriched category of text to a particular enriched category of reduced density operators. The latter leverages the Loewner order on positive semidefinite operators, which can further be interpreted as a toy example of entailment.

Cite

APA

Bradley, T., & Vlassopoulos, Y. (2021). Language Modeling with Reduced Densities. Compositionality. https://doi.org/10.32408/compositionality-3-4

Vancouver

Bradley T, Vlassopoulos Y. Language Modeling with Reduced Densities. Compositionality. 2021. doi:10.32408/compositionality-3-4

BibTeX
@article{Bradley2021,
  author = {Bradley, Tai-Danae and Vlassopoulos, Yiannis},
  title = {Language Modeling with Reduced Densities},
  journal = {Compositionality},
  year = {2021},
  doi = {10.32408/compositionality-3-4}
}
Start for free →