How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator
Entropy · 19.11.2024 · 26 (11) · 997
DOI: 10.3390/e26110997 ↗Kurz gefasst
Fachartikel in der Zeitschrift Entropy, online erschienen am 19.11.2024. Verfasst von Subhash Kantamneni und 2 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Artificial Intelligence and Fundamental Interactions, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Rothberg Family Fund for Cognitive Science, NSF Graduate Research Fellowship, NSF grant.
Abstract (Originalsprache)
How do transformers model physics? Do transformers model systems with interpretable analytical solutions or do they create an “alien physics” that is difficult for humans to decipher? We have taken a step towards demystifying this larger puzzle by investigating the simple harmonic oscillator (SHO), x¨+2γx˙+ω02x=0, one of the most fundamental systems in physics. Our goal was to identify the methods transformers use to model the SHO, and to do so we hypothesized and evaluated possible methods by analyzing the encoding of these methods’ intermediates. We developed four criteria for the use of a method within the simple test bed of linear regression, where our method was y=wx and our intermediate was w: (1) Can the intermediate be predicted from hidden states? (2) Is the intermediate’s encoding quality correlated with the model performance? (3) Can the majority of variance in hidden states be explained by the intermediate? (4) Can we intervene on hidden states to produce predictable outcomes? Armed with these two correlational (1,2), weak causal (3), and strong causal (4) criteria, we determined that transformers use known numerical methods to model the trajectories of the simple harmonic oscillator, specifically, the matrix exponential method. Our analysis framework can conveniently extend to high-dimensional linear systems and nonlinear systems, which we hope will help reveal the “world model” hidden in transformers.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute for Artificial Intelligence and Fundamental Interactions, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
ORCID-Profile: Subhash Kantamneni, Max Tegmark
Förderung
- Rothberg Family Fund for Cognitive Science
- NSF Graduate Research Fellowship
- NSF grant
- Rothberg Family Fund for Cognitive Science, the NSF Graduate Research Fellowship
- IAIFI through NSF grant
Zitieren
APA
Kantamneni, S., Liu, Z., & Tegmark, M. (2024). How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator. Entropy, 26(11), 997. https://doi.org/10.3390/e26110997
Vancouver
Kantamneni S, Liu Z, Tegmark M. How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator. Entropy. 2024;26(11):997. doi:10.3390/e26110997
BibTeX
@article{Kantamneni2024,
author = {Kantamneni, Subhash and Liu, Ziming and Tegmark, Max},
title = {How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator},
journal = {Entropy},
year = {2024},
volume = {26},
number = {11},
pages = {997},
doi = {10.3390/e26110997}
}
EUVIMED