EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Building a Modular Health Information Crawler: Leveraging Apache Nutch for the Tala-Med Search Engine

Nanxing Deng, Martin Boeker, Raphael Scheible

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti250423 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Nanxing Deng und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Computation, Information and Technology, Technical University of Munich, Germany. Hauptthemen laut MEDLINE: Search Engine, Software, Information Storage and Retrieval, Internet. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Developed in 2020, the tala-med search engine provides high-quality, evidence-based health information from trustworthy German websites while ensuring user privacy. However, it still leaves room for technical improvements. In the present work, we are replacing the Fess crawler with Apache Nutch to improve scalability and customization, as Nutch offers greater flexibility for large-scale web crawling and indexing. The new system uses Docker to integrate five services: PostgreSQL for configurations, Nutch for crawling and indexing, and ElasticSearch for search operations; a Manager orchestrates the process with custom configurations via extended Nutch-REST interface. The configuration options include domain-specific URL filters to select the crawled content. A test crawl demonstrated the system’s effectiveness, processing approximately 23k websites over 65 hours. Future work will focus on deploying the crawler long-term and generating a search index for further analysis. We have published our code under the MIT license at https://gitlab.com/mri-tum/aiim/search-platform/crawler.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Search EngineSoftwareInformation Storage and RetrievalInternetUser-Computer InterfaceGermany

Stichwörter der Autor:innen

Data SystemsHealth Information SystemsInformation Storage and RetrievalOnline SystemsSearch EngineSoftware DesignWeb-Crawler

Autor:innen und Institutionen

  • School of Computation, Information and Technology, Technical University of Munich, Germany.
  • Institute for AI and Informatics in Medicine (AIIM), TUM University Medical Center, Technical University of Munich, Germany.

Zitieren

APA

Deng, N., Boeker, M., & Scheible, R. (2025). Building a Modular Health Information Crawler: Leveraging Apache Nutch for the Tala-Med Search Engine. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250423

Vancouver

Deng N, Boeker M, Scheible R. Building a Modular Health Information Crawler: Leveraging Apache Nutch for the Tala-Med Search Engine. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250423

BibTeX
@article{Deng2025,
  author = {Deng, Nanxing and Boeker, Martin and Scheible, Raphael},
  title = {Building a Modular Health Information Crawler: Leveraging Apache Nutch for the Tala-Med Search Engine},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti250423}
}
Kostenlos starten →