EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology

Tingzhao Yu, Kuoyin Wang, Zhimin Li, Muhua Wang, Yu Chen, Hui Chen, Rui Zhao, Yucheng Xu, Yingying Song, Baowen Xu

Applied Sciences · 12.08.2026 · 16 (16) · 8032

DOI: 10.3390/app16168032 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 12.08.2026. Verfasst von Tingzhao Yu und 9 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science and Technology Innovation Program of Xiongan New Area, China Meteorological Administration.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper proposes METMAP, a comprehensive 6D evaluation framework designed to assess large language models in meteorological applications. The framework encompasses six critical capabilities including meteorological knowledge comprehension, expert-level meteorological content summarization, multilingual translation of meteorological information, geospatial mapping context understanding, alignment with authoritative meteorological standards, and professional meteorological service communication. To operationalize this framework, the paper further introduces MetSEval-1k, a high-quality benchmark comprising 1083 expert-curated questions spanning operational and public-facing meteorological services. The benchmark integrates both objective multiple-choice items and subjective open-ended tasks to enable holistic model assessment. This paper conducts systematic evaluations of multiple state-of-the-art large language models using MetSEval-1k, revealing substantial performance disparities across the six dimensions. The results highlight the critical necessity for domain-specific adaptation and rigorous validation before deploying large language models in operational meteorological contexts. MetSEval-1k is released as a foundational benchmark to advance research and development of trustworthy, service-oriented artificial intelligence application in meteorology.

Förderung

  • Science and Technology Innovation Program of Xiongan New Area
  • China Meteorological Administration

Zitieren

APA

Yu, T., Wang, K., Li, Z., Wang, M., Chen, Y., Chen, H., Zhao, R., Xu, Y., Song, Y., & Xu, B. (2026). MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology. Applied Sciences, 16(16), 8032. https://doi.org/10.3390/app16168032

Vancouver

Yu T, Wang K, Li Z, Wang M, Chen Y, Chen H, et al. MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology. Applied Sciences. 2026;16(16):8032. doi:10.3390/app16168032

BibTeX
@article{Yu2026,
  author = {Yu, Tingzhao and Wang, Kuoyin and Li, Zhimin and Wang, Muhua and Chen, Yu and Chen, Hui and Zhao, Rui and Xu, Yucheng and Song, Yingying and Xu, Baowen},
  title = {MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {16},
  pages = {8032},
  doi = {10.3390/app16168032}
}
Kostenlos starten →