Formal Specification and Testing for Reinforcement Learning
Proceedings of the ACM on Programming Languages · 30.08.2023 · 7 (ICFP) · 125-158
DOI: 10.1145/3607835 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Programming Languages, online erschienen am 31.08.2023. Verfasst von Mahsa Varshosaz und 3 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Innovation Fund Denmark.
Abstract (Originalsprache)
The development process for reinforcement learning applications is still exploratory rather than systematic. This exploratory nature reduces reuse of specifications between applications and increases the chances of introducing programming errors. This paper takes a step towards systematizing the development of reinforcement learning applications. We introduce a formal specification of reinforcement learning problems and algorithms, with a particular focus on temporal difference methods and their definitions in backup diagrams. We further develop a test harness for a large class of reinforcement learning applications based on temporal difference learning, including SARSA and Q-learning. The entire development is rooted in functional programming methods; starting with pure specifications and denotational semantics, ending with property-based testing and using compositional interpreters for a domain-specific term language as a test oracle for concrete implementations. We demonstrate the usefulness of this testing method on a number of examples, and evaluate with mutation testing. We show that our test suite is effective in killing mutants (90% mutants killed for 75% of subject agents). More importantly, almost half of all mutants are killed by generic write-once-use-everywhere tests that apply to any reinforcement learning problem modeled using our library, without any additional effort from the programmer.
Förderung
- Innovation Fund Denmark
Zitieren
APA
Varshosaz, M., Ghaffari, M., Johnsen, E. B., & Wąsowski, A. (2023). Formal Specification and Testing for Reinforcement Learning. Proceedings of the ACM on Programming Languages, 7(ICFP), 125-158. https://doi.org/10.1145/3607835
Vancouver
Varshosaz M, Ghaffari M, Johnsen EB, Wąsowski A. Formal Specification and Testing for Reinforcement Learning. Proceedings of the ACM on Programming Languages. 2023;7(ICFP):125-158. doi:10.1145/3607835
BibTeX
@article{Varshosaz2023,
author = {Varshosaz, Mahsa and Ghaffari, Mohsen and Johnsen, Einar Broch and Wąsowski, Andrzej},
title = {Formal Specification and Testing for Reinforcement Learning},
journal = {Proceedings of the ACM on Programming Languages},
year = {2023},
volume = {7},
number = {ICFP},
pages = {125--158},
doi = {10.1145/3607835}
}
EUVIMED