Deduplicating corpora
International Journal of Corpus Linguistics · 30.07.2026 · 31 (3) · 299-332
DOI: 10.1075/ijcl.24197.bed ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Corpus Linguistics, online erschienen am 16.06.2026. Verfasst von Monika Bednarek und 1 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Australian Research Data Commons.
Abstract (Originalsprache)
Corpora containing newspaper articles are widely used in corpus-based discourse research. However, these corpora often contain duplicate texts due to issues such as syndication, the existence of agency copy, and largely similar print and online versions of the same article. While there are several tools that have automated deduplication functions, only a limited number of these tools allow corpus compilers to inspect duplicates before removal. In this paper, we will introduce the Document Similarity tool, which was developed to automatically remove identical texts and then provides near-identical candidates for users to examine side-by-side, giving corpus compilers more control of the deduplication process. We will describe several experiments using corpora that have been deduplicated using the Document Similarity tool in order to demonstrate its utility for corpus creation in corpus-based discourse analysis of newspapers and potentially other text types.
Förderung
- Australian Research Data Commons
Zitieren
APA
Bednarek, M., & Lee, K. K. H. (2026). Deduplicating corpora. International Journal of Corpus Linguistics, 31(3), 299-332. https://doi.org/10.1075/ijcl.24197.bed
Vancouver
Bednarek M, Lee KKH. Deduplicating corpora. International Journal of Corpus Linguistics. 2026;31(3):299-332. doi:10.1075/ijcl.24197.bed
BibTeX
@article{Bednarek2026,
author = {Bednarek, Monika and Lee, Kelvin K. H.},
title = {Deduplicating corpora},
journal = {International Journal of Corpus Linguistics},
year = {2026},
volume = {31},
number = {3},
pages = {299--332},
doi = {10.1075/ijcl.24197.bed}
}
EUVIMED