Quantifying the Dissimilarity of Texts
Information · 02.05.2023 · 14 (5) · 271
DOI: 10.3390/info14050271 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 02.05.2023. Verfasst von Benjamin Shade und 1 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University of Sydney.
Abstract (Originalsprache)
Quantifying the dissimilarity of two texts is an important aspect of a number of natural language processing tasks, including semantic information retrieval, topic classification, and document clustering. In this paper, we compared the properties and performance of different dissimilarity measures D using three different representations of texts—vocabularies, word frequency distributions, and vector embeddings—and three simple tasks—clustering texts by author, subject, and time period. Using the Project Gutenberg database, we found that the generalised Jensen–Shannon divergence applied to word frequencies performed strongly across all tasks, that D’s based on vector embedding representations led to stronger performance for smaller texts, and that the optimal choice of approach was ultimately task-dependent. We also investigated, both analytically and numerically, the behaviour of the different D’s when the two texts varied in length by a factor h. We demonstrated that the (natural) estimator of the Jaccard distance between vocabularies was inconsistent and computed explicitly the h-dependency of the bias of the estimator of the generalised Jensen–Shannon divergence applied to word frequencies. We also found numerically that the Jensen–Shannon divergence and embedding-based approaches were robust to changes in h, while the Jaccard distance was not.
Förderung
- University of Sydney
Zitieren
APA
Shade, B., & Altmann, E. G. (2023). Quantifying the Dissimilarity of Texts. Information, 14(5), 271. https://doi.org/10.3390/info14050271
Vancouver
Shade B, Altmann EG. Quantifying the Dissimilarity of Texts. Information. 2023;14(5):271. doi:10.3390/info14050271
BibTeX
@article{Shade2023,
author = {Shade, Benjamin and Altmann, Eduardo G.},
title = {Quantifying the Dissimilarity of Texts},
journal = {Information},
year = {2023},
volume = {14},
number = {5},
pages = {271},
doi = {10.3390/info14050271}
}
EUVIMED