GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation
Transactions in GIS · 2025-11 · 29 (7)
DOI: 10.1111/tgis.70135 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions in GIS, online erschienen am 13.10.2025. Verfasst von Qianheng Zhang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.
Abstract (Originalsprache)
Recent advances in large language models (LLMs) have fueled growing interest in automating geospatial analysis and GIS workflows, yet their actual capabilities remain uncertain. In this work, we call for rigorous evaluation of LLMs on well‐defined geoprocessing tasks before making claims about full GIS automation. To this end, we present GeoAnalystBench, a benchmark of 50 Python‐based tasks derived from real‐world geospatial problems and validated by GIS experts. Each task is paired with a minimum deliverable product, and evaluation covers workflow validity, structural alignment, semantic similarity, and code quality (CodeBLEU). Using this benchmark, we assess both proprietary and open‐source models. Results reveal a clear gap: proprietary models such as ChatGPT‐4o‐mini achieve high validity (95%) and stronger code alignment (CodeBLEU 0.39), while smaller open‐source models like DeepSeek‐R1‐7B often generate incomplete or inconsistent workflows (48.5% validity, 0.272 CodeBLEU). Tasks requiring deeper spatial reasoning, such as spatial relationship detection or optimal site selection, remain the most challenging across all models. These findings demonstrate both the promise and limitations of current LLMs in GIS automation and provide a reproducible framework to advance GeoAI research with human‐in‐the‐loop support.
Förderung
- National Science Foundation
Zitieren
APA
Zhang, Q., Gao, S., Wei, C., Zhao, Y., Nie, Y., Chen, Z., Chen, S., Su, Y., & Sun, H. (2025). GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation. Transactions in GIS, 29(7). https://doi.org/10.1111/tgis.70135
Vancouver
Zhang Q, Gao S, Wei C, Zhao Y, Nie Y, Chen Z, et al. GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation. Transactions in GIS. 2025;29(7). doi:10.1111/tgis.70135
BibTeX
@article{Zhang2025,
author = {Zhang, Qianheng and Gao, Song and Wei, Chen and Zhao, Yibo and Nie, Ying and Chen, Ziru and Chen, Shijie and Su, Yu and Sun, Huan},
title = {GeoAnalystBench : A GeoAI Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation},
journal = {Transactions in GIS},
year = {2025},
volume = {29},
number = {7},
doi = {10.1111/tgis.70135}
}
EUVIMED