EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating language models for mathematics through interactions

Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka iD, Umang Bhatt, Thomas Lukasiewicz iD, Yuhuai Wu, Joshua B Tenenbaum, William Hart iD, Timothy Gowers iD, Wenda Li und 2 weitere

Proceedings of the National Academy of Sciences · 11.06.2024 · 121 (24)

DOI: 10.1073/pnas.2318124121 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 03.06.2024. Verfasst von Katherine M Collins und 13 weiteren Autor:innen. Beteiligte Einrichtung: University of Cambridge, Cambridge CB2 1TN, United Kingdom. Hauptthemen laut MEDLINE: Mathematics, Language, Problem Solving. Die Arbeit wurde bisher 58-mal zitiert (Crossref) und nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Alan Turing Institute (ATI), EU TAILOR, Leverhulme Trust.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs; this is insufficient for making an informed decision about which LLMs are best to use in an interactive setting, and how that varies by setting. Static assessment therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analyzing MathConverse, we derive a taxonomy of human query behaviors and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, among other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by experienced mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, may constitute better assistants. Humans should inspect LLM output carefully given their current shortcomings and potential for surprising fallibility.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

MathematicsLanguageProblem SolvingHumansStudents

Stichwörter der Autor:innen

AIhuman–computer interactionlanguage modelstheorem proving

Autor:innen und Institutionen

  • University of Cambridge, Cambridge CB2 1TN, United Kingdom.
  • University of Oxford, Oxford OX1 4BH, United Kingdom.
  • Massachusetts Institute of Technology, Cambridge, MA 02139.
  • The Alan Turing Institute, London NW1 2DB, United Kingdom.
  • New York University, New York, NY 10011.
  • Vienna University of Technology, Vienna 1040, Austria.

ORCID-Profile: Miri Zilka, Thomas Lukasiewicz, William Hart, Timothy Gowers, Adrian Weller, Mateja Jamnik

Förderung

  • Alan Turing Institute (ATI)
  • EU TAILOR
  • Leverhulme Trust
  • ERC Advanced Grant ALEXANDRIA
  • EPSRC

Zitieren

APA

Collins, K. M., Jiang, A. Q., Frieder, S., Wong, L., Zilka, M., Bhatt, U., Lukasiewicz, T., Wu, Y., Tenenbaum, J. B., Hart, W., Gowers, T., Li, W., Weller, A., & Jamnik, M. (2024). Evaluating language models for mathematics through interactions. Proceedings of the National Academy of Sciences, 121(24). https://doi.org/10.1073/pnas.2318124121

Vancouver

Collins KM, Jiang AQ, Frieder S, Wong L, Zilka M, Bhatt U, et al. Evaluating language models for mathematics through interactions. Proceedings of the National Academy of Sciences. 2024;121(24). doi:10.1073/pnas.2318124121

BibTeX
@article{Collins2024,
  author = {Collins, Katherine M and Jiang, Albert Q and Frieder, Simon and Wong, Lionel and Zilka, Miri and Bhatt, Umang and Lukasiewicz, Thomas and Wu, Yuhuai and Tenenbaum, Joshua B and Hart, William and Gowers, Timothy and Li, Wenda and Weller, Adrian and Jamnik, Mateja},
  title = {Evaluating language models for mathematics through interactions},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2024},
  volume = {121},
  number = {24},
  doi = {10.1073/pnas.2318124121}
}
Kostenlos starten →