ZAYA1-8B: Solving Olympiad Math with 0.7B Active Parameters

TL;DR

ZAYA1-8B is a technical breakthrough from Zyphra that proves reasoning density matters more than raw parameter count. By combining a 700M active parameter MoE architecture with a novel test-time compute strategy called Markovian RSA, this model outperforms giants like DeepSeek-R1-0528 and Gemini-2.5 Pro on elite math benchmarks (AIME, HMMT). It represents a shift toward "test-time scaling" where inference-time compute is used to bridge the gap between small and large models.

The Problem: The "Reasoning Tax" on Small Models

High-level reasoning (math, coding, logical puzzles) has traditionally been the domain of trillion-parameter models. Small models usually fail here for two reasons:

1. Architecture Bottlenecks: Standard linear routers in Mixture-of-Experts (MoE) models aren't expressive enough to achieve true expert specialization at small scales.
2. Inference Inefficiency: Standard Chain-of-Thought (CoT) reasoning is computationally expensive. As the model "thinks" longer, the KV-cache grows, making it slower and more memory-intensive to generate the next token.

Methodology: Rethinking the Thinking Process

1. Architectural Innovation: MoE++

ZAYA1-8B introduces three key changes to the standard Transformer:

• The ZAYA1 Router: Replaces the simple linear router with a multi-layer MLP and Exponential Depth Averaging (EDA). This forces experts to specialize more effectively, achieving better load balancing (as seen in Figure 4).
• CCA (Compressed Convolutional Attention): Performs attention in a compressed latent space. This reduces the KV-cache footprint and prefill FLOPs by 8x relative to standard attention—critical for the long paths taken in reasoning.
• Residual Scaling: Learned gating for residual streams to control gradient growth through its 40-layer depth.

Fig. 3: The ZAYA1 architecture featuring the MLP-based router and CCA block.

2. Markovian RSA: Bounded-State Thinking

The star of the show is Markovian RSA. Usually, if a model thinks for 40,000 tokens, it must attend to all 40,000 in its context window. Markovian RSA breaks this. It generates many parallel "candidates," but between rounds of reasoning, it only carries forward a tail (e.g., the last 4,000 tokens).

This treats reasoning as a Markov process: the model only needs its most recent thoughts to decide what to think next. This allows it to generate up to 740,000 tokens per problem across parallel workers while keeping the context window (and memory cost) predictably small.

Experiments: Punching Above Its Weight

The results are staggering. In math competitions like AIME and HMMT, ZAYA1-8B (0.7B active params) not only beats its weight class but competes with the world's most powerful frontier models.

Fig. 2: Scaling of performance vs. Active Parameter count. ZAYA1-8B sits at the far left but reaches the top of the chart.

Key Results Summary:

• AIME '25: 91.9% (with Markovian RSA).
• HMMT '25: 89.6%.
• LiveCodeBench: 64.8% (outperforming DeepSeek-R1-0528).

The "Reasoning Warmup" and "RLVE-Gym" RL stages were vital. The paper notes that RL was incredibly sample-efficient, moving the model's performance by +20-30 points on AIME with fewer than 1,500 update steps.

Critical Analysis: The Future of Dense Reasoning

The core takeaway from ZAYA1-8B is that Reasoning Density—the ability to utilize every active parameter for logic rather than just memorization—is the next frontier.

Limitations:

• Knowledge vs. Reasoning: While the model is a math wizard, it doesn't match 100B+ models on broad factual knowledge (MMLU-Pro). It is a "reasoning core," not a "encyclopedic core."
• Agentic Tasks: The model currently lags on multi-turn tool-use (agency), as the training focused primarily on verifiable reasoning tokens.

Conclusion

ZAYA1-8B proves that the compute-optimal path to AGI-level reasoning might not be "more parameters," but "wider search." By training a model to specifically handle parallel candidate aggregation (TTC), Zyphra has narrowed the gap between 1B and 100B parameter models, fundamentally changing how we value Small Language Models.