EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Large language models as tax attorneys: a case study in legal capabilities emergence

John J Nay iD, David Karamardian, Sarah B Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H Choi, Jungo Kasai

Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · 15.04.2024 · 382 (2270)

DOI: 10.1098/rsta.2023.0159 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, online erschienen am 26.02.2024. Verfasst von John J Nay und 8 weiteren Autor:innen. Beteiligte Einrichtung: CodeX, Center for Legal Informatics, Stanford University, Stanford, CA, USA. Hauptthemen laut MEDLINE: Artificial Intelligence, Lawyers. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Better understanding of Large Language Models' (LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence and leveraging LLMs to identify inconsistencies in law. This paper explores LLM capabilities in applying tax law. We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and maths skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. We experiment with retrieving and using the relevant legal authority to assess the impact of providing additional legal context to LLMs. Few-shot prompting, presenting examples of question–answer pairs, is also found to significantly enhance the performance of the most advanced model, GPT-4. The findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels. As LLMs continue to advance, their ability to reason about law autonomously could have significant implications for the legal profession and AI governance. This article is part of the theme issue ‘A complexity science approach to law and governance’.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceLawyersHumansLanguage

Stichwörter der Autor:innen

artificial intelligencecomputational lawlarge language modelslaw informs codelaw-informed AImachine learning

Autor:innen und Institutionen

  • CodeX, Center for Legal Informatics, Stanford University, Stanford, CA, USA.
  • Stanford University, Stanford, CA, USA.
  • Northwestern Pritzker School of Law, Chicago, IL, USA.
  • Engineering, University of Michigan, Ann Arbor, MI, USA.
  • SimPPL, India.
  • Independent, Northern Ireland.

ORCID-Profile: John J Nay

Zitieren

APA

Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2024). Large language models as tax attorneys: a case study in legal capabilities emergence. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270). https://doi.org/10.1098/rsta.2023.0159

Vancouver

Nay JJ, Karamardian D, Lawsky SB, Tao W, Bhat M, Jain R, et al. Large language models as tax attorneys: a case study in legal capabilities emergence. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2024;382(2270). doi:10.1098/rsta.2023.0159

BibTeX
@article{Nay2024,
  author = {Nay, John J and Karamardian, David and Lawsky, Sarah B and Tao, Wenting and Bhat, Meghana and Jain, Raghav and Lee, Aaron Travis and Choi, Jonathan H and Kasai, Jungo},
  title = {Large language models as tax attorneys: a case study in legal capabilities emergence},
  journal = {Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences},
  year = {2024},
  volume = {382},
  number = {2270},
  doi = {10.1098/rsta.2023.0159}
}
Kostenlos starten →