MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology
Applied Sciences · 12.08.2026 · 16 (16) · 8032
DOI: 10.3390/app16168032 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 12.08.2026. Verfasst von Tingzhao Yu und 9 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science and Technology Innovation Program of Xiongan New Area, China Meteorological Administration.
Abstract (Originalsprache)
This paper proposes METMAP, a comprehensive 6D evaluation framework designed to assess large language models in meteorological applications. The framework encompasses six critical capabilities including meteorological knowledge comprehension, expert-level meteorological content summarization, multilingual translation of meteorological information, geospatial mapping context understanding, alignment with authoritative meteorological standards, and professional meteorological service communication. To operationalize this framework, the paper further introduces MetSEval-1k, a high-quality benchmark comprising 1083 expert-curated questions spanning operational and public-facing meteorological services. The benchmark integrates both objective multiple-choice items and subjective open-ended tasks to enable holistic model assessment. This paper conducts systematic evaluations of multiple state-of-the-art large language models using MetSEval-1k, revealing substantial performance disparities across the six dimensions. The results highlight the critical necessity for domain-specific adaptation and rigorous validation before deploying large language models in operational meteorological contexts. MetSEval-1k is released as a foundational benchmark to advance research and development of trustworthy, service-oriented artificial intelligence application in meteorology.
Förderung
- Science and Technology Innovation Program of Xiongan New Area
- China Meteorological Administration
Zitieren
APA
Yu, T., Wang, K., Li, Z., Wang, M., Chen, Y., Chen, H., Zhao, R., Xu, Y., Song, Y., & Xu, B. (2026). MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology. Applied Sciences, 16(16), 8032. https://doi.org/10.3390/app16168032
Vancouver
Yu T, Wang K, Li Z, Wang M, Chen Y, Chen H, et al. MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology. Applied Sciences. 2026;16(16):8032. doi:10.3390/app16168032
BibTeX
@article{Yu2026,
author = {Yu, Tingzhao and Wang, Kuoyin and Li, Zhimin and Wang, Muhua and Chen, Yu and Chen, Hui and Zhao, Rui and Xu, Yucheng and Song, Yingying and Xu, Baowen},
title = {MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology},
journal = {Applied Sciences},
year = {2026},
volume = {16},
number = {16},
pages = {8032},
doi = {10.3390/app16168032}
}
EUVIMED