FPRM: Breaking the Depth Barrier in Looped Transformers with Fixed-Point Reasoning

Executive Summary

TL;DR: FPRM (Fixed-Point Reasoning Model) is a new class of Looped Transformer that solves the instability of deep recurrent computation. By combining pre-norm layers with learned residual scaling, it maintains stable signal propagation across thousands of iterations. It replaces complex halting modules with a simple, elegant rule: stop when the latent state stops changing (fixed-point convergence).

Background: This work sits at the intersection of Looped Transformers and Deep Equilibrium Models (DEQ). While recent models like TRM and HRM relied on "hierarchical" loops to manage depth, FPRM proves that a simpler, non-hierarchical model can outperform them if the underlying signal propagation is fixed.

The Core Conflict: Stability vs. Trainability

In a Looped Transformer, the same layer is applied repeatedly ($z_{i+1}=f(z_i;x)$). This creates an "effective depth" that can reach hundreds of layers.

• Post-norm (used by prior SOTA): Keeps activations from blowing up but "stifles" the gradient, making it hard for the model to learn complex, multi-step algorithms.
• Pre-norm: Allows gradients to flow freely but causes activations to grow exponentially, eventually crashing the training.

FPRM introduces Residual Scaling to have the best of both worlds. By scaling the residual stream $(\alpha )$ and the sub-layer output $(\beta )$, the authors mathematically prove that activations remain bounded even as depth goes to infinity.

Figure 1: FPRM scales compute more effectively than traditional Transformers as task difficulty increases.

Methodology: The Fixed-Point "Halting" Insight

The most "academic" contribution here is how the model decides when to stop. Instead of training a separate "halting head" (which is notoriously hard to optimize due to discrete decisions), FPRM essentially asks: Has the reasoning converged?

1. Iterative Step: $z_{i+1}=\alpha z_i+\beta f(Norm(z_i))$.
2. Damping: To prevent the model from "spiraling" or oscillating around a solution, they use a damped update: $z_{new}=\eta z_{next}+(1-\eta )z_{old}$.
3. Halting: Once $\Vert z_{i+1}-z_i\Vert <au$, the "thought" is complete.

This makes the "Test-Time Compute" truly adaptive. A simple Sudoku puzzle might take 20 iterations, while an "Extreme" one might take 500.

Figure 2: The architecture of FPRM, highlighting the pre-norm and scaling components.

Experimental Triumphs

FPRM was tested on the most grueling symbolic reasoning tasks:

• Sudoku-Extreme: Achieved 94.2% accuracy, crushing previous 7M and even 27M parameter models.
• ARC-AGI: On the ARC-1 benchmark, it reached 47.5%, showing that compact, recurrent models can sometimes rival massive LLMs (like DeepSeek-R1 or Claude) on pure logic puzzles.
• State Tracking: Unlike TRM, FPRM showed near-perfect length generalization by scaling its "pondering" time linearly with the input sequence length.

Table 1: FPRM outperforms hierarchical models across multiple benchmarks.

Critical Analysis & Conclusion

The Takeaway: FPRM suggests that the "Hierarchy" in previous reasoning models might have just been a band-aid for poor signal propagation. By fixing the math behind the residual stream, a "flat" looped model becomes significantly more powerful.

Limitations: While FPRM is a master of symbolic logic, it hasn't been tested on natural language. Converting these "fixed-point thoughts" into fluent text remains an open challenge.

Future Outlook: We are moving toward a world where model size matters less than "Test-Time Compute". FPRM provides a blueprint for how small, efficient models can "think" longer to solve problems that take a static-depth model by surprise.