EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-SA 4.0

Smart bilingual focused crawling of parallel documents

Cristian García-Romero, Miquel Esplà-Gomis, Felipe Sánchez-Martínez

Natural Language Processing · 2026-05 · 32 (3) · 302-321

DOI: 10.1017/nlp.2026.10021 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Natural Language Processing, online erschienen am 21.04.2026. Verfasst von Cristian García-Romero und 2 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Crawling parallel texts—texts that are mutual translations—from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up leading to actual parallel content. In this work, we propose a smart crawling method that guides the crawl towards finding parallel content more rapidly. We follow a neural approach that consists in adapting a pre-trained multilingual language model based on the encoder of the Transformer architecture by fine-tuning it for two new tasks: inferring the language of a document from its Uniform Resource Locator (URL) and inferring whether a pair of URLs link to parallel documents. We evaluate both models in isolation and their integration into a crawling tool. The results demonstrate the individual effectiveness of both models and highlight that their combination enables us to address a practical engineering challenge: the early discovery of parallel content during web crawling in a given language pair. This leads to a reduction in the amount of downloaded documents deemed useless and yields a greater quantity of parallel documents compared to conventional crawling approaches.

Zitieren

APA

García-Romero, C., Esplà-Gomis, M., & Sánchez-Martínez, F. (2026). Smart bilingual focused crawling of parallel documents. Natural Language Processing, 32(3), 302-321. https://doi.org/10.1017/nlp.2026.10021

Vancouver

García-Romero C, Esplà-Gomis M, Sánchez-Martínez F. Smart bilingual focused crawling of parallel documents. Natural Language Processing. 2026;32(3):302-321. doi:10.1017/nlp.2026.10021

BibTeX
@article{GarcaRomero2026,
  author = {García-Romero, Cristian and Esplà-Gomis, Miquel and Sánchez-Martínez, Felipe},
  title = {Smart bilingual focused crawling of parallel documents},
  journal = {Natural Language Processing},
  year = {2026},
  volume = {32},
  number = {3},
  pages = {302--321},
  doi = {10.1017/nlp.2026.10021}
}
Kostenlos starten →