EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Toward HydroLLM: a benchmark dataset for hydrology-specific knowledge assessment for large language models

Dilara Kizilkaya, Ramteja Sajja, Yusuf Sermet, Ibrahim Demir

Environmental Data Science · 2025 · 4

DOI: 10.1017/eds.2025.10006 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Environmental Data Science, online erschienen am 02.06.2025. Verfasst von Dilara Kizilkaya und 3 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation, National Oceanic and Atmospheric Administration.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid advancement of large language models (LLMs) has enabled their integration into a wide range of scientific disciplines. This article introduces a comprehensive benchmark dataset specifically designed for testing recent LLMs in the hydrology domain. Leveraging a collection of research articles and hydrology textbooks, we generated a wide array of hydrology-specific questions in various formats, including true/false, multiple-choice, open-ended, and fill-in-the-blank. These questions serve as a robust foundation for evaluating the performance of state-of-the-art LLMs, including GPT-4o-mini, Llama3:8B, and Llama3.1:70B, in addressing domain-specific queries. Our evaluation framework employs accuracy metrics for objective question types and cosine similarity measures for subjective responses, ensuring a thorough assessment of the models’ proficiency in understanding and responding to hydrological content. The results underscore both the capabilities and limitations of artificial intelligence (AI)-driven tools within this specialized field, providing valuable insights for future research and the development of educational resources. By introducing HydroLLM-Benchmark, this study contributes a vital resource to the growing body of work on domain-specific AI applications, demonstrating the potential of LLMs to support complex, field-specific tasks in hydrology.

Förderung

  • National Science Foundation
  • National Oceanic and Atmospheric Administration

Zitieren

APA

Kizilkaya, D., Sajja, R., Sermet, Y., & Demir, I. (2025). Toward HydroLLM: a benchmark dataset for hydrology-specific knowledge assessment for large language models. Environmental Data Science, 4. https://doi.org/10.1017/eds.2025.10006

Vancouver

Kizilkaya D, Sajja R, Sermet Y, Demir I. Toward HydroLLM: a benchmark dataset for hydrology-specific knowledge assessment for large language models. Environmental Data Science. 2025;4. doi:10.1017/eds.2025.10006

BibTeX
@article{Kizilkaya2025,
  author = {Kizilkaya, Dilara and Sajja, Ramteja and Sermet, Yusuf and Demir, Ibrahim},
  title = {Toward HydroLLM: a benchmark dataset for hydrology-specific knowledge assessment for large language models},
  journal = {Environmental Data Science},
  year = {2025},
  volume = {4},
  doi = {10.1017/eds.2025.10006}
}
Kostenlos starten →