[CVPR 2024] DPCache: Rethinking Denoising as Global Path Planning for 5x Diffusion Acceleration

Executive Summary

TL;DR: DPCache is a novel, training-free framework that accelerates diffusion models (DiT, FLUX, HunyuanVideo) by treating the denoising process as a global path planning problem. By identifying the mathematically optimal "key timesteps" via dynamic programming, it achieves significant speedups (up to 4.87×) without the quality degradation typical of local caching methods.

Background Positioning: While existing methods focus on what to cache (features, tokens) or how to predict the next step (Taylor series), DPCache addresses the when. It moves the field from "greedy" local heuristics to "global" trajectory optimization, representing a significant shift in how we handle inference efficiency in the Transformer era.

Problem & Motivation

The central bottleneck in Diffusion Transformers (DiTs) is the costly iterative sampling process. Caching-based methods (like DeepCache or TeaCache) attempt to reuse features from previous timesteps to skip computation. However, they possess a "blind spot":

1. Fixed Schedules: They apply a uniform skip-rate (e.g., skip every 2 steps), ignoring that some timesteps are more critical for structural formation than others.
2. Short-sightedness: Locally adaptive methods make greedy decisions. If a jump looks "okay" now, they take it, often leading to a "drift" that becomes unrecoverable later in the trajectory.

The authors' key insight is that the denoising trajectory is content-agnostic. The "shape" of how a model generates an image is determined more by the model itself than the specific prompt. This allows for a one-time calibration to find a globally optimal path.

Methodology: The Path-Aware Cost Tensor (PACT)

DPCache formulates acceleration as finding a sequence of timesteps $\mathcal{T}$ that minimizes total deviation.

1. 3D Path-Aware Cost Tensor (PACT)

Unlike a simple 2D matrix (cost of jumping from $j$ to $k$), DPCache uses a 3D tensor $C[i,j,k]$. This accounts for the path-dependency: the error of skipping from $j$ to $k$ depends on the previous key step $i$, because the feature predictor uses the difference between $i$ and $j$ to estimate $k$.

2. Global Optimization via Dynamic Programming

With the cost tensor defined, the problem becomes: $\arg {\min }_{\mathcal{T}}\sum \mathcal{C}[t_{m-1},t_m,t_{m+1}]$ This is solved exactly using Dynamic Programming in $O(K{T}^2)$ time. This optimization happens once during a "calibration phase" (using ~10 samples) and the resulting schedule is used for all future inferences.

Experiments & Results

DPCache was tested on flagship models including FLUX.1-dev and HunyuanVideo.

• Higher Fidelity: On FLUX, DPCache (3.54× speedup) actually scored higher in ImageReward (1.007) than the 50-step baseline (0.979). This suggests the optimized path might act as a regularizer, potentially filtering out some noise-induced artifacts.
• Video Mastery: In text-to-video tasks, where temporal consistency is paramount, DPCache maintained sharp edges and smooth motion while competitors like SpeCa and TaylorSeer introduced warping or "hand-to-lifebuoy" hallucinations.
• Efficiency: Unlike TaylorSeer, which caches every layer (massive VRAM hit), DPCache only caches the final layer, keeping the memory overhead negligible (+0.36 GB for large video models).

Critical Analysis & Conclusion

Takeaway

DPCache proves that global scheduling is superior to local adaptation. By moving the intelligence from the "per-step decision" to a "one-time global plan," we can achieve massive speedups without the catastrophic drift that plagues current SOTA.

Limitations & Future Work

• Inherited Errors: Since DPCache optimizes for fidelity to the baseline, it also inherits the baseline's mistakes (e.g., misspellings in text rendering).
• Static Nature: While the trajectory is mostly content-agnostic, very complex prompts might still benefit from some level of input-adaptive correction.

The next frontier for DPCache will likely involve combining its global path planning with learnable predictors, potentially creating an inference engine that is not only faster than the baseline but also more "intelligent" in its reconstructions.