The mismatch that self-conditioning corrects
A one-hot flow represents each token as a one-hot vector and trains a denoiser to predict the clean endpoint from a corrupted point. Near the end of the corruption schedule, training inputs sit in a very small radius around the final endpoint, so the model mostly learns to track a nearly clean target. At inference, however, the model follows its own denoiser predictions, which retain non-zero entropy and therefore never point exactly at a clean vertex of the simplex. The paper calls this gap trajectory ambiguity: the states the model actually visits when sampling are rare during training.
Self-conditioning addresses this by feeding the model information about its own previous prediction, which exposes it to inference-like states during training. The authors frame the gain not as a generic capacity boost but as a correction of that train/inference mismatch. This framing matters because it predicts where the benefit should appear and suggests that the same effect might be obtained more cheaply than by adding an extra forward pass.
DoubleCat recovers the gain without extra passes
DoubleCat implements this idea as a two-level interpolant. Instead of corrupting only toward the clean one-hot target, it adds a second corruption process on the denoiser's output space, the simplex, so the model is trained on states that resemble its own intermediate predictions. The paper reports that this improves TinyGSM accuracy more than threefold relative to the comparison it makes, and that the gains of self-conditioning can be recovered without an extra forward pass.
The accuracy numbers should be read with their setting in mind. TinyGSM is a small-scale math word-problem benchmark, and the more-than-threefold improvement is a relative statement about that benchmark rather than an absolute accuracy level. The paper's own table on self-conditioning effects also shows that switching self-conditioning on raises scores for both one-hot flows and DoubleCat, which supports the interpretation that the mechanism, not merely extra computation, drives the improvement.
Token identifiability and discrete reveals
Self-conditioning helps with a second problem the paper calls token identifiability: recognising tokens whose representation is dominated by large-norm Gaussians, which makes the expected endpoint harder to identify from the current state. Rather than relying only on self-conditioning to mitigate this, the authors address it directly by adding discrete reveals to DoubleCat, producing SemiCat. In this hybrid design, each position is revealed as a discrete token at a sampled time, after which the interpolant points at the clean token instead of a continuous mixture.
SemiCat reaches 49.94% on TinyGSM with standard training hyperparameters and a tuned sampler, which the paper describes as discrete-diffusion-level performance. The qualifier matters: the result depends on both the training configuration and a tuned sampler, so it is not a claim that the architecture alone closes the gap under any sampling budget.
Where SemiCat holds at 1.7B scale
At the 1.7B scale, SemiCat outperforms discrete diffusion on four of six likelihood-based evaluations and surpasses one-hot flows on unconditional generation. These are two different comparisons on two different kinds of measurement: likelihood-based evaluations and unconditional generation quality are not interchangeable, and the paper reports them separately rather than collapsing them into a single ranking.
The split result is the useful part. SemiCat does not win everywhere against discrete diffusion; it wins on four of six likelihood evaluations. Readers comparing methods should therefore ask which evaluation type matches their use case, because the paper's evidence supports a mixed picture rather than uniform superiority.
Embedding geometry follows the loss
The second lever is embedding geometry. Instead of one-hot vectors, embedded methods map tokens into a lower-dimensional space through a trainable codebook and run the flow there. The paper finds that cross-entropy training drives these embeddings toward a geometry whose decoding error rate closely matches the one-hot one. In other words, the embedded model does not automatically escape the behaviour of one-hot flows; the loss pushes it toward similar decoding properties.
The paper also notes that one-hot methods in fact evolve over an embedded space with anisotropic noise. This reframes the one-hot versus embedded distinction: the relevant difference is not simply whether an embedding exists, but how the training objective and noise structure shape the geometry that decoding operates on.
Hybrid corruption and noise schedules
Beyond the two main levers, the paper briefly explores further design options, including hybrid corruption processes and noise schedules, among them asynchronous schedules that assign different times to different token positions. It introduces an α-Cat generalization of hybrid corruption, where each position flows toward either a random vertex with probability α or the true token otherwise until its reveal time.
The α-Cat results on TinyGSM show accuracy improving as α rises to 0.75, recovering the effect of SemiCat, while α = 1 produces a leave-one-out denoiser that ignores the position being predicted. The authors note that variants with α > 0 partially defeat the purpose of hybrid methods because they rely on discretely revealed positions. This is a design trade-off rather than a free improvement, and it is reported at the same small benchmark scale as the main TinyGSM comparisons.
What to take from the design space
The paper's most decision-relevant claim is a priority ordering among cheap fixes. If the goal is to recover self-conditioning's benefit without an extra forward pass, DoubleCat is the lower-cost option the authors demonstrate. If the goal is to close the gap to discrete diffusion on reasoning-style benchmarks, SemiCat's discrete reveals are the mechanism the paper credits, reaching 49.94% on TinyGSM and beating discrete diffusion on four of six likelihood evaluations at 1.7B scale.
That ordering comes with boundaries the paper itself sets. The headline TinyGSM numbers come from a small benchmark and, for SemiCat, from standard training hyperparameters plus a tuned sampler. The 1.7B comparison is split across evaluation types rather than a clean sweep. And the embedding finding is a statement about where cross-entropy drives the geometry, not a guarantee that embedded flows behave differently from one-hot flows by construction. The practical reading is to choose the intervention by which failure mode matters: trajectory ambiguity points to DoubleCat, token identifiability points to SemiCat's reveals, and neither claim should be extended to settings the paper did not test.
Sources used in this answer
On the Design Space of Continuous Diffusion Language Models
Self-conditioning corrects a train/inference mismatch in continuous diffusion language models; DoubleCat recovers its gains without an extra forward pass and more than triples TinyGSM accuracy, while SemiCat adds discrete reveals and reaches 49.94% on TinyGSM.