EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A reproducible web-scraping pipeline for collecting geolocated German online job advertisements at scale

Matthias Kapa, Kerstin J Schaefer

MethodsX · 2026-12 · 17 · 104166

DOI: 10.1016/j.mex.2026.104166 ↗

Kurz gefasst

Fachartikel in der Zeitschrift MethodsX, 2026-12. Verfasst von Matthias Kapa und 1 weiteren Autor:innen. Beteiligte Einrichtung: Leibniz University Hannover, Institute of Economic and Cultural Geography, Schneiderberg 50, Hannover, 30167, Germany. Die Arbeit nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Niedersächsische Ministerium für Wissenschaft und Kultur.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Online job advertisements (OJAs) provide near-real-time data on labour market skill demand including location information but are predominantly accessed through costly commercial providers whose collection and processing methods are not publicly documented. We present a fully reproducible, open-source web-scraping pipeline that systematically collects OJAs from StepStone, one of Germany's largest job portals. The pipeline combines the Scrapy crawling framework with Playwright-based headless browser automation to handle dynamic JavaScript-rendered content, implements deduplication logic, and deploys a locally run large language model (AI4Privacy) to censor personally identifiable information in compliance with the General Data Protection Regulation. Applied from April to July 2023, the pipeline collected 342,452 unique OJAs. Validation against the commercial provider Lightcast confirms that, for single-portal scrapes, regional coverage (79% vs 80% of German municipalities) and spatial distribution are broadly comparable to a commercial provider, though volume and cross-portal reach differ. However, because of the smaller overall size of the single-portal dataset, there might be drawbacks when it comes to the analysis of smaller sectors. The anonymised dataset and cleaning scripts are released in machine-readable JSON format on Mendeley Data.•A Scrapy + Playwright pipeline reliably collects large-scale OJAs from a dynamic, JavaScript-rendered job portal.•A locally deployed LLM (AI4Privacy) censors personally identifiable data in job ad text before release.•Validation against Lightcast shows that the single-portal scrape reproduces the aggregate NUTS-2 regional structure of the German OJA landscape (Spearman ρ = 0.91), though direct record-level URL overlap is low owing to cross-source deduplication and URL instability.

Schlagwörter

Stichwörter der Autor:innen

Data collection pipelineGermanyLabour market dataOnline job advertisementsPrivacy anonymisationPythonScrapyWeb scraping

Autor:innen und Institutionen

  • Leibniz University Hannover, Institute of Economic and Cultural Geography, Schneiderberg 50, Hannover, 30167, Germany.
  • Utrecht University, Human Geography and Spatial Planning, Princetonlaan 8a, Utrecht, 3584 CB, the Netherlands.

Förderung

  • Niedersächsische Ministerium für Wissenschaft und Kultur

Zitieren

APA

Kapa, M., & Schaefer, K. J. (2026). A reproducible web-scraping pipeline for collecting geolocated German online job advertisements at scale. MethodsX, 17, 104166. https://doi.org/10.1016/j.mex.2026.104166

Vancouver

Kapa M, Schaefer KJ. A reproducible web-scraping pipeline for collecting geolocated German online job advertisements at scale. MethodsX. 2026;17:104166. doi:10.1016/j.mex.2026.104166

BibTeX
@article{Kapa2026,
  author = {Kapa, Matthias and Schaefer, Kerstin J},
  title = {A reproducible web-scraping pipeline for collecting geolocated German online job advertisements at scale},
  journal = {MethodsX},
  year = {2026},
  volume = {17},
  pages = {104166},
  doi = {10.1016/j.mex.2026.104166}
}
Kostenlos starten →