A closed loop that writes, trains, and reads its own results
The question behind the paper is whether an AI system can do the long, expensive part of architecture research itself, rather than pick from a menu a human wrote. Earlier neural architecture search worked inside a fixed, human-defined search space, so the system acted as a selection algorithm. ASI-Arch removes that space: it freely hypothesizes new architectures and then has to implement and test them.
The loop has four modules. A Researcher proposes an architecture and writes its code in a single pass, so the implementation cannot drift away from the stated motivation. An Engineer trains the candidate in a real code environment and computes a fitness score. An Analyzer turns each finished run into a structured write-up. An Experience Repository stores everything and feeds context back into the next proposal.
The paper is explicit about why this is hard: language models have thin built-in knowledge of architecture design, a model layer lives inside a large codebase where one constraint violation silently corrupts training, and quality can only be measured by pre-training from scratch. Each module exists to absorb one of those difficulties rather than to add capability for its own sake.
Cheap filters protect the expensive training budget
Training is the costly step, so the system tries to reject bad candidates before they reach it. The Researcher runs a novelty check that embeds a proposal's motivation, matches it against the five most similar historical motivations, and asks a specialized model to judge whether the design is genuinely new or a superficial variant. A sanity check then combines automated code analysis with model review of code quality, complexity, and causal masking.
Failures are not simply discarded. Rejected proposals come back with specific feedback for revision. During training, an implementation error sends the full error log to a debugging agent that revises the code and resubmits the run. A parallel monitor kills runs showing pathological signals: abnormally low loss, excessive compilation time, or training more than twice as slow as the baseline.
These are engineering choices, and the paper frames them as such: the stated aim is to maximize the share of GPU hours spent on genuinely viable candidates. The evidence offered is the design rationale plus the reported success of the overall run, not a controlled ablation isolating each filter's contribution.
Fitness blends measured gains with a model's judgment
Candidates are ranked by a composite fitness that averages three terms: a sigmoid-transformed improvement in loss, a sigmoid-transformed improvement in benchmark score, and a model-as-judge score between 0 and 1. The sigmoids focus on differences within 10% of baseline, which caps extreme values so no single metric dominates while still amplifying small but meaningful gains.
The judge term is there to capture what benchmark numbers miss: architectural complexity, computational efficiency, implementation elegance, and novelty relative to the baseline. This matters for reading the headline result, because a design can score well partly on judged qualities rather than purely on measured loss or benchmark improvement. The paper does not report how much of the final ranking each term drove.
Screening at 20M parameters, confirming at 1.3B
The search runs in two stages. In exploration, candidates are trained at 20M parameters on 1B tokens and evaluated on 500 samples per benchmark — fast enough to run thousands of experiments. Only architectures that beat DeltaNet on both quantitative and qualitative metrics advance.
In verification, survivors scale to 340M parameters on 1B tokens under a strict causality check confirming that perturbing later tokens does not change earlier-token hidden states. Top candidates then scale to 1.3B parameters on 15B tokens and are tested on six additional out-of-distribution benchmarks.
This staging is the paper's answer to an obvious worry: gains found under a tight compute budget could be artifacts of that budget. The multi-scale verification is designed to show the improvements persist as models grow. It is still a check at three specific sizes, not a guarantee at every scale.
105 reported state-of-the-art designs, and no saturation
Applied to linear attention, the system ran 1,773 iterative experiments and discovered 105 architectures the authors describe as state-of-the-art. The headline comparison is against DeltaNet: the best architecture improves over it by nearly three times the gain that Mamba2 achieves. That is a relative comparison of improvement margins over a shared baseline, not a claim that the new design is three times better than Mamba2 in absolute terms.
The paper also reports that discovery continued throughout the run with no saturation in sight, and that distilled design rules transferred across families — from DeltaNet to Mamba2, GLA, and HGRN2. Transfer of design rules is a different claim from transfer of a single architecture's weights or performance, and the paper presents it as evidence that the system learned something generalizable about design rather than one lucky configuration.
Shared memory turns single runs into cumulative insight
The Experience Repository keeps a dynamically updated pool of the top 50 scoring architectures. At each evolution step the Researcher samples one parent from the top 10 to modify directly and four references from positions 11 through 50 for diversity. Pool updates are batched so parallel agents work from a stable reference set, and a cold-start period freezes the pool for the first 200 explorations so convergence pressure does not set in too early.
Alongside the pool sits a cognition base: roughly 100 seminal linear-attention papers distilled into about 150 entries, each recording an applicable scenario, a core algorithm, and historical context. After each experiment, the Analyzer's summary of observed shortcomings is used as a retrieval query against the scenario field, and the top three matching entries are injected into the next proposal cycle.
The Analyzer also does something the paper treats as a distinct capability: by comparing structurally similar architectures that differ in only a few components, it infers the isolated contribution of individual design choices, approximating an ablation study without extra controlled runs. That is the mechanism by which the system can improve by understanding why designs succeed, not only by recombining what already scored well.
What the linear-attention result does and does not cover
The demonstrated scope is linear attention, a family of sub-quadratic sequence models that includes linear attention proper, state-space models such as Mamba, and linear RNNs such as RWKV and HGRN2. The 1,773 experiments, the 105 reported state-of-the-art designs, and the DeltaNet-versus-Mamba2 comparison all sit inside that setting. The paper's own framing is that the architectural landscape has become a vast combinatorial space where optimal designs depend heavily on task and constraints, which is why it targets one region of that space rather than all of it.
The authors also note that future work will explore larger architectural hypothesis spaces, keep iterating on the strong designs found here, and pursue concrete algorithmic efficiency optimizations to better understand practical efficiency. That last point is a real boundary: the fitness function rewards judged computational efficiency, but the paper does not present deployed-efficiency measurements, so the reported gains should be read as pre-training and benchmark results at the tested scales.
The most useful way to read the result is as a comparison of improvement margins under a fixed baseline. If you care whether automated search can outpace a strong human-designed linear-attention baseline, the relevant number is the roughly three-to-one margin over Mamba2's gain on DeltaNet — measured at 20M, 340M, and 1.3B parameters in this pipeline. If you care whether the discovered designs are faster in production, or whether the same loop works outside linear attention, this paper does not answer that.
Sources used in this answer
AlphaGo Moment for Model Architecture Discovery
ASI-Arch ran 1,773 closed-loop experiments on linear attention and discovered 105 state-of-the-art architectures; its best improves over DeltaNet by nearly three times the gain achieved by Mamba2.