How can particle search and MH refinement align audio-visual generation under a fixed budget?

Two alignment tools fail for opposite reasons

Reward-guided inference-time alignment steers a generator toward higher-reward outputs without changing model weights. For joint audio-visual generation, that means scoring candidate videos and audio against alignment objectives and pushing generation toward better-scoring samples. The paper frames the obstacle as cost: optimizing several alignment objectives requires many expensive model evaluations, so existing methods become costly when compute is limited.

The two building blocks have complementary weaknesses. Particle-based search explores globally and can run in parallel without reward gradients, but a finite particle population produces estimation error and can degrade sample quality. Metropolis-Hastings (MH) refinement improves a candidate locally, but the paper notes it needs substantial compute for global exploration because of slow mixing, and it benefits from a strong starting point. SWIM is designed around that division: search first to find a promising candidate, then refine it locally rather than paying for a larger particle population or a long global MH run.

Particles move to checkpoints with high weight dispersion

The first stage treats particle allocation as a variance-minimization problem under a fixed particle budget. Denoising is divided into checkpoints, and at each checkpoint arriving particles are scored and given selection weights. The paper measures how concentrated those weights are using the effective sample size, then converts it into an empirical squared coefficient of variation. When weights concentrate on few particles, dispersion is high; when all particles weigh equally, the estimate is zero.

The allocation theorem says the optimal population at a stage scales with the anticipated dispersion and inversely with the square root of that segment's length. In plain terms, stages expected to be more informative get more particles, while longer, more expensive segments get fewer. Because future dispersion is unknown at inference time, the online algorithm forecasts the next checkpoint's dispersion from the current one and adjusts a compute multiplier based on how much budget has already been spent. A pacing floor keeps a minimum population for later stages, and an affordability cap reserves compute for the remaining segments. The paper is explicit that the summed dispersion term is a tractable variance surrogate, not the exact variance of the final output.

MH refines a completed output instead of exploring globally

The particle funnel returns the terminal particle with the highest particle-stage reward, and that output initializes a short MH chain. Each MH step re-noises the output latent to a chosen level and re-denoises both modalities along a fresh stochastic path to form a proposal. The proposal is scored with a fixed weighted sum of alignment and quality objectives and accepted with a probability that depends on the reward difference.

The acceptance rule is worth reading carefully. Under an exact reverse model, the reward-only rule is the MH acceptance ratio for the reward-tilted target distribution; with a learned reverse model, the paper calls it approximate. A finite MH run also depends on its initialization and proposal scale, which is why the authors use particle search to supply the starting point. The particle stage itself uses a checkpoint-dependent reward: one term is applied at every checkpoint, and a second is added only after a mixing threshold, when its intermediate estimate is described as more reliable.

Matched-compute gains favor the combined pipeline

The headline comparison is at closely matched compute on JavisBench-Mini. SWIM reports mean relative gains across seven metrics of 30.7% over best-of-N and 14.1% over EvoSearch when paired with JavisDiT++, and 9.2% over best-of-N and 17.0% over EvoSearch when paired with LTX-2. These are relative gains averaged over seven metrics, not a single absolute score, so they should not be read as a uniform improvement on every individual metric.

The paper also reports that, at matched compute, combining particle search with MH refinement achieves higher rewards than either approach alone. That claim is about the combination, and it is the paper's central design argument: MH recovers additional reward from a strong candidate without requiring a larger particle population or costly global exploration.

Adaptive allocation helps most when particles are scarce

The appendix compares SWIM against a uniform-K baseline that spreads particles evenly across checkpoints. On JavisDiT++ with TV-IB, the largest score improvement is 2.5% at 400 neural function evaluations (NFE), narrowing to 1.3% at NFE=500. On SD1.5 with AestheticScore, the largest improvement is 2.3% at NFE=400, narrowing to 0.9% at NFE=1000. The paper's explanation is that larger particle populations let even uniform allocation explore the reward landscape more effectively, reducing the benefit of adaptation.

Estimator variance shows the same pattern but with a notable exception. SWIM reduces total estimator variance by 36.1% and 27.2% at NFE=200 and NFE=400 for TV-IB, and by 14.9% and 10.6% for AestheticScore. For TV-IB the gap remains 28.7% at NFE=500, while for AestheticScore it narrows to 0.8% at NFE=1000. The paper attributes the persistent audio-visual gap to the much larger search space for audio-visual generation compared with images. These variance numbers are surrogate estimator variances, not the same quantity as the reward scores.

Moderate adaptation beats both extremes

The schedule has parameters that change how aggressively compute is moved. The pacing floor gamma reserves a minimum population for future stages. With small gamma the schedule is highly adaptive and places many particles at the informative middle checkpoint but can prune too aggressively afterward; with large gamma it approaches uniform allocation and loses some adaptation benefit. The best reward in the reported sweep occurs at gamma=0.6.

The dual step size controls how fast the compute multiplier responds to budget over- or under-spending. Schedules are identical through the first two checkpoints and diverge later. Larger values prune more aggressively, and final performance is non-monotonic, with the highest reward at a step size of 1e-3 and the lowest at 1e-1. The paper's reading is that moderate adaptation preserves particles until later denoising stages, where reward estimates are more informative. When the population grows, the growth rule also matters: a rescue strategy that balances reward-guided selection with lineage preservation gives the largest improvement in VR-TA (0.099), ahead of weighted (0.089) and best (0.086), while uniform performs worst (0.068).

The advantage narrows as budgets grow

The paper's own discussion sets the boundary. At larger budgets, the smaller gains from adaptive allocation may not justify its extra schedule parameters. The persistence forecast, which assumes the next checkpoint's dispersion resembles the current one, may be suboptimal when variance changes rapidly between adjacent checkpoints. The MH acceptance rule is approximate with a learned reverse model, and a short chain's outcome depends on initialization and proposal scale.

The practical priority that follows is a budget threshold rather than a universal recommendation. The largest reported gains from adaptive allocation appear at low NFE budgets, and the paper notes that these low-budget gains are particularly relevant for expensive audio-visual models where the number of model evaluations is a major constraint. As compute rises, uniform allocation closes much of the score gap, especially in image generation, while the audio-visual variance advantage persists longer. Readers deciding whether to adopt this schedule should therefore look first at their evaluation budget and at whether their bottleneck is score or estimator variance.

Sources used in this answer