Perplexity bounds are not comparable across model families
For autoregressive models, the standard way to measure how well a generated distribution matches the data distribution is the perplexity of held-out data, computed from the exact likelihood. Diffusion and flow-based language models cannot supply that exact likelihood. They report an upper bound of perplexity based on an evidence lower bound, or ELBO.
The paper identifies two problems with this substitute. First, an ELBO is often not available: for continuous models the bound holds only in expectation over the noise, requires the change-of-variables formula along the flow, and has been derived only for selected models. Most recent models also use self-conditioning, where the network receives its own previous prediction as input; that input arises along the sampling trajectory and cannot be recovered for a given data sequence. Second, even when an ELBO exists, it is not comparable across model families. The gap between ELBO and likelihood is a property of the noise process and its parameterization, not of the model alone. For masked diffusion the bound is tight, whereas for uniform diffusion it is looser and not even optimized by the training objective. A better perplexity may therefore reflect a tighter bound rather than a better model.
Generative perplexity measures imitation of a reference model
Because every model can be sampled, generative perplexity together with token entropy has become the common substitute. Generative perplexity is the perplexity of generated texts under a fixed reference model such as GPT-2 Large, and entropy is computed from token frequencies within sequences.
The paper argues that neither quantity measures the distance between the generated distribution and the data distribution. Both are averaged over sequences, so they do not reward variety in generated outputs; the paper points to mode collapse as an example. Generative perplexity also measures agreement with the reference model rather than with the data, so it rewards imitation of the reference model instead of actual data.
The paper's stated goal is therefore an analogous score for text distributions that requires only samples from both distributions, is a distance that is zero if and only if the two distributions agree, and reflects both quality and variety.
SOL detects distributional failures and recovers model trends
The paper reports that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. As a first step, the authors use SOL to re-evaluate a variety of models trained on OpenWebText.
Stability is a central claim. The paper reports that SOL converges as the number of samples increases, as the number of slices increases, and as the number of integration points increases, for both GPT-2 Large and Dream encoders. In particular, inexpensive SOL estimates based on only a few samples and slices already provide relatively reliable approximations to the full evaluation, which the authors present as suitable for monitoring during training.
The paper contrasts this with MAUVE, which it says depends heavily on the number of clusters chosen. For two disjoint LM1B corpora, the reported MAUVE score depends on the number of evaluation samples and clusters; keeping a fixed ratio of clusters to samples decreases the score as sample size increases, while keeping the number of clusters fixed and increasing samples causes the score to converge to one. The paper says this raises the question of which score to report and makes MAUVE scores hard to compare.
MAUVE is more sensitive to the final-token distribution
The paper reports a substantial disagreement between MAUVE and SOL for DUO's greedy-tail sampler on OpenWebText. To investigate, the authors measure token entropy across documents at a selected position. Greedy DUO's positional entropy falls from 6.32 nats at the middle position to 5.01 at the endpoint, whereas real OpenWebText remains near 6.58 at both positions.
The authors then reweight generated documents by their last token so that weighted final-token frequencies match real OpenWebText without changing the texts. After this correction, MAUVE rises from 0.492 to 0.591 for greedy DUO, but only from 0.836 to 0.843 for ancestral DUO. SOL changes by only 0.29% and 0.07%, respectively. The paper concludes that MAUVE is much more sensitive to the final-token distribution than SOL, because SOL uses hidden states from all token positions.
The paper also reports that MAUVE with the standard last-token hidden representation has a strong bias toward the end of a sequence: replacing the beginning with random tokens has little effect on the score. With a mean-pooled representation, MAUVE saturates quickly under corruption, dropping from 0.97 to 0.47 at 5% pollution and to 0.06 at 10%, after which it stays near zero. The paper says the score has used up its range by 10%, so it cannot distinguish 10% from 100% corrupted text. SOL, by contrast, is reported to increase monotonically over the full range of corruption rates.
Favorable perplexity and entropy do not guarantee a better match
The paper argues that generative perplexity and entropy should be interpreted jointly: generative perplexity should be small, while entropy should remain close to that of the data. Even together, however, the paper says they do not determine distributional quality.
For distilled DUO, ancestral and greedy-tail sampling have almost identical entropy deviations from the data, 0.069 and 0.071. Greedy-tail sampling nevertheless lowers generative perplexity substantially, from 62.3 to 46.5, while worsening SOL on all three encoders, including Dream from 30.595 to 31.802. More strongly, exact MDLM has both lower generative perplexity than VP with ODE sampling, 59.4 versus 66.1, and entropy closer to the data, 5.445 versus 5.435, yet has worse SOL on all three encoders: 2.185, 14.119, and 36.394 versus 1.775, 11.997, and 33.275. The paper concludes that even favorable generative perplexity and entropy do not guarantee a better match to the text distribution.
Sampling strategy ordering reverses between conditional and unconditional generation
The paper investigates how sampling strategies affect GPT-2-XL's agreement with the WebText distribution, comparing ancestral sampling, nucleus sampling with p equal to 0.95, and greedy decoding.
In conditional generation, the authors use 5,000 WebText documents, provide the first 35 GPT-2 tokens of each document as context, and stop at the first end-of-text token or when the total sequence length including the prefix reaches 256 tokens. Both metrics favor nucleus sampling: SOL decreases from 0.5083 to 0.2712, while MAUVE increases from 0.9407 to 0.9607. Greedy decoding performs substantially worse than both stochastic samplers, with MAUVE of 0.0697 and SOL of 3.7755.
In unconditional generation, GPT-2-XL is initialized with the end-of-text token and generates until the first subsequently generated end-of-text token or a maximum of 256 generated tokens. The ordering reverses: ancestral sampling achieves lower SOL, 0.8358 versus 0.9072, and higher MAUVE, 0.8832 versus 0.8740, than nucleus sampling. The paper notes that SOL and MAUVE agree on the sampling-strategy ordering in both settings, but the better strategy depends on whether generation is conditional or unconditional.
SOL is a sample-based distance, not a universal quality score
SOL's guarantee as a metric depends on the transformer being injective. The paper proves the zero-if-and-only-if property under that assumption for empirical distributions with uniform weights, so the metric interpretation is tied to the encoder used.
The OpenWebText comparisons are made within the same reference family and encoder, and the paper states that each model comparison uses 5,000 generated sequences and 5,000 rows selected from a 10,000-row reference pool, with real-real controls using disjoint subsets. The paper also notes that LDDM uses the same OpenWebText source and split with different packing-worker boundaries, and that LangFlow's historical training split is not independently verified. These conditions bound how far the reported rankings can be transferred across encoders, reference families, or preprocessing pipelines.
The paper's own framing is that SOL fills a gap in the evaluation protocol used for non-autoregressive models, and that re-evaluating models trained on OpenWebText is a first step. The practical priority the paper establishes is not a single leaderboard number but the choice of what to compare: SOL values are meaningful within a fixed reference family and encoder, and the paper's evidence shows that perplexity, entropy, and distributional fit can point in different directions. Readers weighing non-autoregressive text models should therefore treat a favorable generative perplexity or entropy as insufficient, and check whether the distributional distance was measured against the data distribution rather than against a reference model.
Sources used in this answer
SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics
SOL represents each sequence by the empirical measure of its hidden states under a fixed transformer and compares the distributions of these measures with a double sliced Wasserstein distance; it is proven to be a metric when the transformer is injective, and experiments show it detects distributional failures, recovers expected model trends, and provides stable sample-based estimates.