Smart bilingual focused crawling of parallel documents
Natural Language Processing · 2026-05 · 32 (3) · 302-321
DOI: 10.1017/nlp.2026.10021 ↗Kurz gefasst
Fachartikel in der Zeitschrift Natural Language Processing, online erschienen am 21.04.2026. Verfasst von Cristian García-Romero und 2 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).
Abstract (Originalsprache)
Crawling parallel texts—texts that are mutual translations—from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up leading to actual parallel content. In this work, we propose a smart crawling method that guides the crawl towards finding parallel content more rapidly. We follow a neural approach that consists in adapting a pre-trained multilingual language model based on the encoder of the Transformer architecture by fine-tuning it for two new tasks: inferring the language of a document from its Uniform Resource Locator (URL) and inferring whether a pair of URLs link to parallel documents. We evaluate both models in isolation and their integration into a crawling tool. The results demonstrate the individual effectiveness of both models and highlight that their combination enables us to address a practical engineering challenge: the early discovery of parallel content during web crawling in a given language pair. This leads to a reduction in the amount of downloaded documents deemed useless and yields a greater quantity of parallel documents compared to conventional crawling approaches.
Zitieren
APA
García-Romero, C., Esplà-Gomis, M., & Sánchez-Martínez, F. (2026). Smart bilingual focused crawling of parallel documents. Natural Language Processing, 32(3), 302-321. https://doi.org/10.1017/nlp.2026.10021
Vancouver
García-Romero C, Esplà-Gomis M, Sánchez-Martínez F. Smart bilingual focused crawling of parallel documents. Natural Language Processing. 2026;32(3):302-321. doi:10.1017/nlp.2026.10021
BibTeX
@article{GarcaRomero2026,
author = {García-Romero, Cristian and Esplà-Gomis, Miquel and Sánchez-Martínez, Felipe},
title = {Smart bilingual focused crawling of parallel documents},
journal = {Natural Language Processing},
year = {2026},
volume = {32},
number = {3},
pages = {302--321},
doi = {10.1017/nlp.2026.10021}
}
EUVIMED