EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation

Qianheng Zhang, Song Gao, Chen Wei, Yibo Zhao, Ying Nie, Ziru Chen, Shijie Chen, Yu Su, Huan Sun

Transactions in GIS · 2025-11 · 29 (7)

DOI: 10.1111/tgis.70135 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions in GIS, online erschienen am 13.10.2025. Verfasst von Qianheng Zhang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in large language models (LLMs) have fueled growing interest in automating geospatial analysis and GIS workflows, yet their actual capabilities remain uncertain. In this work, we call for rigorous evaluation of LLMs on well‐defined geoprocessing tasks before making claims about full GIS automation. To this end, we present GeoAnalystBench, a benchmark of 50 Python‐based tasks derived from real‐world geospatial problems and validated by GIS experts. Each task is paired with a minimum deliverable product, and evaluation covers workflow validity, structural alignment, semantic similarity, and code quality (CodeBLEU). Using this benchmark, we assess both proprietary and open‐source models. Results reveal a clear gap: proprietary models such as ChatGPT‐4o‐mini achieve high validity (95%) and stronger code alignment (CodeBLEU 0.39), while smaller open‐source models like DeepSeek‐R1‐7B often generate incomplete or inconsistent workflows (48.5% validity, 0.272 CodeBLEU). Tasks requiring deeper spatial reasoning, such as spatial relationship detection or optimal site selection, remain the most challenging across all models. These findings demonstrate both the promise and limitations of current LLMs in GIS automation and provide a reproducible framework to advance GeoAI research with human‐in‐the‐loop support.

Förderung

  • National Science Foundation

Zitieren

APA

Zhang, Q., Gao, S., Wei, C., Zhao, Y., Nie, Y., Chen, Z., Chen, S., Su, Y., & Sun, H. (2025). GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation. Transactions in GIS, 29(7). https://doi.org/10.1111/tgis.70135

Vancouver

Zhang Q, Gao S, Wei C, Zhao Y, Nie Y, Chen Z, et al. GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation. Transactions in GIS. 2025;29(7). doi:10.1111/tgis.70135

BibTeX
@article{Zhang2025,
  author = {Zhang, Qianheng and Gao, Song and Wei, Chen and Zhao, Yibo and Nie, Ying and Chen, Ziru and Chen, Shijie and Su, Yu and Sun, Huan},
  title = {GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation},
  journal = {Transactions in GIS},
  year = {2025},
  volume = {29},
  number = {7},
  doi = {10.1111/tgis.70135}
}
Kostenlos starten →