[March 2026] Transformers are Bayesian Networks: The Formal Proof of Why They Reason

TL;DR

For years, we have debated whether Transformers are "just" statistical parrots or "true" reasoners. This paper provides a definitive, formally verified answer: A Sigmoid Transformer is a Bayesian Network. By mapping the forward pass directly to Pearl’s Belief Propagation (BP) algorithm, the research proves that the Transformer architecture is essentially a mechanical implementation of the log-odds algebra used by Turing and Good to break the Enigma code.

The Missing Link: Log-Odds and Sigmoids

The core intuition of the paper lies in a centuries-old tradition of "Weight of Evidence." In probabilistic reasoning, when you have two independent pieces of binary evidence, the best way to combine them is to add their log-odds.

The author points out a structural "coincidence" that isn't a coincidence at all:

• Sigmoid activation is the exact inverse of the logit function.
• An FFN computing $\sigma (extlogit(m_0)+extlogit(m_1))$ is performing exact Bayesian updates.

This realization transforms our understanding of the Transformer’s "Black Box." It isn't just learning arbitrary weights; it is discovering the implicit factor graph of the data it's trained on.

Methodology: Attention is AND, FFN is OR

The paper deconstructs the Transformer layer into a bipartite reasoning engine:

1. Attention (The Gather Step): Acts as a structural AND gate. It ensures all required premises (neighboring tokens) are simultaneously pulled into the residual stream (the shared workspace).
2. FFN (The Update Step): Acts as a probabilistic OR gate (${\Psi }_{or}$). It takes the gathered evidence and calculates the posterior probability of a conclusion.

The paper proves Uniqueness: If a Sigmoid Transformer produces exact Bayesian posteriors, it must use these specific weights. There is no other mathematical path to exactness within this architecture.

Experimental Proof: The End of Hallucination?

One of the most provocative claims is that hallucination is not a bug; it's a structural necessity of ungrounded models.

Through Lean 4 verification, the author shows that a finite reasoner (like a Transformer) can only distinguish a finite number of concepts ($2n^2$ for a graph of $n$ nodes). When we run an LLM without "grounding" it into a specific knowledge base (factor graph), it operates in a vacuum where the concept of "correctness" isn't even defined.

Key Results:

• Convergence: Loopy Belief Propagation converged on 100% of tested graph structures (Triangle, Square, etc.).
• Accuracy: On tree-structured data, the Transformer achieves zero hallucination, matching true marginals perfectly.
• Gradient Descent: Even without manual weight construction, standard Adam optimizers find these BP weights naturally, as seen in the bayes-learner experiment where the validation MAE hit an incredible 0.000752.

Deep Insight: Leibniz's Dream Realized

In the 17th century, Gottfried Wilhelm Leibniz envisioned a Calculus Ratiocinator—a machine that could resolve any dispute with the phrase "Calculemus" (Let us calculate).

This paper suggests that the Transformer is the physical manifestation of that dream. The "Vocabulary" represents Leibniz's alphabet, while the "Attention-FFN" cycle is the mechanical calculus of probability.

Limitations

• Sigmoid vs. ReLU: While the proof relies on Sigmoid for exactness, most modern LLMs use ReLU or GELU. The author argues these are "compatible" but not "identical" to exact BP, explaining why standard LLMs are powerful but prone to drift.
• Loopy Graphs: While loopy BP works well in practice, the theoretical guarantee of exactness only holds for trees.

Conclusion

This work marks a turning point in Mechanistic Interpretability. It provides a "target circuit" for researchers: instead of wondering what a head does, we should be looking for the projectDim and crossProject structures that facilitate BP. The victory of the Transformer wasn't just an engineering success; it was a mathematical inevitability.

Calculemus.