Experts that output a constant value
The model studied here is a regression predictor built from a segmentation of the input space. Each region of that segmentation is assigned one expert, and each expert is a single learnable constant: for any input falling in its region, it returns that constant. Routing an input to its region is the only computation at inference, which is why the paper calls the design Zero-Compute 1-Sparse MoE (ZC-1SMoE).
This is a deliberately stripped-down MoE. Only one expert answers each input, and no expert performs a computation beyond holding a value. The paper's argument is that this simplicity is what makes the model analytically tractable, while the use of many experts keeps the overall predictor from being too crude. The authors also note that zero-compute experts were previously found practically useful in neural-network MoEs, which motivates studying the idealized version.
Many small regions become a continuous density
The central assumption comes from classical high-rate quantization theory: there are so many experts that each input-space region is very small. Under that assumption, the discrete collection of regions can be approximated analytically as a continuous density of segments over the input domain, rather than handled region by region.
This move is what allows the paper to write down approximation error in closed form. The analysis has to account for two things at once: the probability distribution of the input, and the input-to-output function being predicted. The paper positions this joint handling as its distinctive use of quantization ideas, which were previously applied to piecewise-constant approximation of functions and to functional quantization where the computing function is known. Here the function is unknown and must be learned, which is what introduces a separate estimation-error term.
In one dimension, the optimal segmentation is explicit
For one-dimensional inputs, the paper formulates the test error of the ZC-1SMoE model class together with the segmentation and expert values that minimize it. Because regions in one dimension are intervals rather than shaped cells, the problem reduces to choosing interval boundaries and the constant inside each interval.
The paper's experiments use concrete settings to check the formulas: a cosine β function and a truncated Gaussian input density, with uniform segmentation points placed on the input axis. In that setting, the empirical approximation error is compared against the theoretical formula as the number of experts grows from small values up to about 120. The comparison is between measured approximation error and the analytic prediction, not between the MoE and an alternative model class.
Multiple dimensions require an upper bound
With multidimensional inputs, regions have shapes as well as sizes, so the paper does not claim an exact closed-form test error. Instead it formulates an upper bound on the test error and studies how to minimize that bound. The bound relies on an assumption from high-rate vector quantization theory: that the normalized moment of inertia of the regions can be approximated by a smooth function over the input space.
The bound yields a segment density that the paper uses to guide a recursive rectangular segmentation method for two-dimensional inputs. The experiments test this on a separable two-dimensional cosine and a nonseparable two-dimensional cosine, both with a truncated isotropic Gaussian input density, and compare empirical approximation error against both the theoretical formula and the upper bound as the number of experts rises to roughly 4000. The paper explicitly leaves optimizing region shapes as an open challenge, so the rectangular construction is a practical device rather than a proven optimum.
Each expert learns only from its own samples
Given a fixed segmentation, learning the expert constants is least squares, and the solution is simply the average of the training outputs routed to each region. The paper notes that this corresponds to histogram regression. The number of training examples routed to expert i is a random quantity, and the accuracy of that expert's learned constant depends on it.
The statistical analysis makes this dependence explicit. The deviation between the learned constant and the optimal constant for a region is bounded in terms of the number of samples routed there, with a confidence statement that degrades as the number of experts grows because the bound must hold simultaneously for all of them. The resulting estimation-error bound scales with the number of experts divided by the training-set size, and it requires the training set to be large enough relative to the smallest region probability. Regions that receive no training examples need a fallback, such as the average of all training outputs, which the paper uses in its empirical analysis.
More experts help approximation and hurt estimation
Combining the two analyses produces the paper's central claim: approximation error falls as the number of experts increases, because the input space is covered by smaller regions, while estimation error rises, because each constant is fitted from fewer samples. The test error therefore reflects a balance between the two, and the paper shows this tradeoff both theoretically and empirically as a function of the number of experts.
The comparison that matters is not simply more experts versus fewer. It is the number of experts relative to the amount of training data. The estimation-error bound depends on the training-set size and on how many experts must be estimated at once, so the same expert count can be reasonable for one dataset size and excessive for another. This is the paper's own framing of why the tradeoff depends on the number of experts rather than being resolved by a single recommended value.
What the analysis does not yet cover
The results apply to regression with a single-parameter constant expert per region, one active expert per input, and a fixed segmentation when expert parameters are learned. The paper states that learning the input-space segmentation itself is left for future work, so the estimation-error treatment covers expert constants but not the routing boundaries. Extending the analysis to other sparsity levels and to non-constant experts is also named as future work.
The multidimensional result is an upper bound rather than an exact error expression, and the two-dimensional experiments use a density-guided recursive rectangular segmentation whose region shapes are not optimized. Readers should therefore treat the multidimensional numbers as evidence that the bound and the guided segmentation behave as predicted, not as a demonstration that this segmentation is optimal among all possible ones.
The expert count is a data-dependent choice
The paper's most useful message is a reframing of what the number of experts controls. In this model class, adding experts buys finer input-space resolution at the price of noisier per-expert estimates, and the paper's formulas tie both sides of that exchange to measurable quantities: the input density, the gradient of the underlying function, and the number of training samples routed to each region.
That makes the expert count a data-dependent decision rather than a free capacity knob. The paper's specific contribution is to show, for zero-compute experts in regression, exactly where the approximation gain and the estimation penalty come from, and to demonstrate that the predicted balance appears in experiments across one- and two-dimensional settings. The open question it leaves is the one its own framing makes central: how the segmentation itself should be learned, not just how many regions to use.
Sources used in this answer
Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
Using high-rate quantization theory, the paper shows that in a mixture of zero-compute constant experts for regression, increasing the number of experts lowers approximation error but raises estimation error, so the test-error tradeoff depends on the number of experts relative to the training data.