Physics of generative AI’s atom: Repetition, bias, and beyond
Source abstract
Abstract
We derive a first-principles physics theory of the individual “atom” at the heart of generative AI such as ChatGPT: the basic attention head. The theory shows how, when, and why its output can become repetitive or suddenly switch to potentially harmful content. In situations where a small subset of attention heads dominates generative AI’s output or multi-layer effects average out, such undesirable microscale behavior will emerge at the macroscale to threaten generative AI’s safety in medical, legal, and business settings. The theory also quantifies the impact of bias from training and fine-tuning. The theory’s 2-spin form suggests why generative AI such as ChatGPT can work so well but hints that a generalized 3-spin attention might be even better. The theory’s similarity to spin bath physics means existing physics expertise could be harnessed to help generative AI become more trustworthy and resilient to manipulation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Artificial Intelligence in Healthcare and Education · Ethics and Social Impacts of AI · Explainable Artificial Intelligence (XAI)
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Physics Department, George Washington University , Washington, District of Columbia 20052,
References (36)
Show source-provided references
1. Attention is all you need
2. 2025
3. 2025
4. The curious case of neural text degeneration
5. Training large language models on narrow tasks can lead to broad misalignment10.1038/s41586-025-09937-5 ↗
6. 2025
7. R. Chen , A.Arditi, H.Sleight, O.Evans, and J.Lindsey, “Persona vectors: Monitoring and controlling character traits in language models,” arXiv:2507.21509 [cs.CL] (2025).
8. Progress measures for grokking via mechanistic interpretability
9. N. Nanda and T.Lieberum, A mechanistic interpretability analysis of grokking, see https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking (accessed 7 May 2024).
10. N. Nanda , Paper replication walkthrough: Reverse-engineering modular addition, see https://www.neelnanda.io/mechanistic-interpretability/modular-addition-walkthrough (accessed 7 May 2024).
11. Anthropic , Tracing the thoughts of a Large Language Model, 2025, see https://www.anthropic.com/research/tracing-thoughts-language-model (accessed 28 March 2025).
12. 2025
13. E. Ameisen , J.Lindsey, A.Pearce, W.Gurnee, N. L.Turner, B.Chen, C.Citro, D.Abrahams, S.Carter, B.Hosmer, J.Marcus, M.Sklar, A.Templeton, T.Bricken, C.McDougall, H.Cunningham, T.Henighan, A.Jermyn, A.Jones, A.Persic, Z.Qi, T. B.Thompson, S.Zimmerman, K.Rivoire, T.Conerly, C.Olah, and J.Batson, Circuit tracing: Revealing computational graphs in language models, 2025, see https://transformer-circuits.pub/2025/attribution-graphs/methods.html (accessed 28 March 2025).
14. J. Lindsey , W.Gurnee, E.Ameisen, B.Chen, A.Pearce, N. L.Turner, C.Citro, D.Abrahams, S.Carter, B.Hosmer, J.Marcus, M.Sklar, A.Templeton, T.Bricken, C.McDougall, H.Cunningham, T.Henighan, A.Jermyn, A.Jones, A.Persic, Z.Qi, T. B.Thompson, S.Zimmerman, K.Rivoire, T.Conerly, C.Olah, and J.Batson, On the biology of a Large Language Model, 2025, see https://transformer-circuits.pub/2025/attribution-graphs/biology.html (accessed 28 March 2025).
15. A. Templeton , T.Bricken, C.McDougall, H.Cunningham, T.Henighan, A.Jermyn, A.Jones, A.Persic, Z.Qi, T. B.Thompson, S.Zimmerman, K.Rivoire, T.Conerly, C.Olah, and J.Batson, Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet, 2024, see https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html (accessed 28 March 2025).
16. T. Bricken , A.Templeton, J.Batson, B.Chen, A.Jermyn, T.Conerly, N. L.Turner, C.Anil, C.Denison, A.Askell, R.Lasenby, Y.Wu, S.Kravec, N.Schiefer, T.Maxwell, N.Joseph, A.Tamkin, K.Nguyen, B.McLean, J. E.Burke, T.Hume, S.Carter, T.Henighan, and C.Olah, Towards monosemanticity: Decomposing Language Models with dictionary learning, 2023, see https://transformer-circuits.pub/2023/monosemantic-features/index.html (accessed 28 March 2025).
17. Talking heads: Understanding inter-layer communication in transformer language models
18. Emergent abilities of large language models
19. Are sixteen heads really better than one?
20. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned
21. DistilBERT: A distilled version of BERT
22. BERT loses patience: Fast and robust inference with early exit
23. DeeBERT: Dynamic early exiting for accelerating BERT inference
24. Reducing transformer depth on demand with structured dropout
25. Transformer variational wave functions for frustrated quantum spin systems10.1103/physrevlett.130.236401 ↗
26. Mapping of attention mechanisms to a generalized Potts model10.1103/physrevresearch.6.023057 ↗
27. H. Ramsauer , B.Schäfl, J.Lehner, P.Seidl, M.Widrich, T.Adler, L.Gruber, M.Holzleitner, M.Pavlović, G. K.Sandve, V.Greiff, D.Kreil, M.Kopp, G.Klambauer, J.Brandstetter, and S.Hochreiter, “Hopfield networks is all you need,” arXiv:2008.02217 [cs.NE] (2021).
28. Energy transformer
29. J. Arnold , F.Holtorf, F.Schäfer, and N.Lörch, “Phase transitions in the output distribution of large language models,” arXiv:2405.17088 [cs.LG] (2024).
30. Phase transitions in large language models and the O(N) model
31. K. Nakaishi , Y.Nishikawa, and K.Hukushima, “Critical phase transition in large language models,” arXiv:2406.05335 [cond-mat.dis-nn] (2024).
32. Attention in natural language processing10.1109/tnnls.2020.3019893 ↗
33. Neural machine translation by jointly learning to align and translate
34. A theoretical analysis of the repetition problem in text generation
35. Multispecies cohesion: Humans, machinery, AI, and beyond10.1103/physrevlett.133.247401 ↗
36. Basic attention head as a building block toward understanding transformer-based generative AI10.54364/AAIML.2025.54251 ↗