[CVPR 2025] Quant Experts: Adapting PTQ to the Heterogeneity of Vision-Language Tokens
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
The paper introduces Quant Experts (QE), a novel Post-Training Quantization (PTQ) framework for Large Vision-Language Models (VLMs). It utilizes a Mixture-of-Experts (MoE) approach—combining a shared global expert with multiple routed local experts—to achieve state-of-the-art accuracy in low-bit settings (e.g., W4A6) across models ranging from 2B to 72B parameters.
TL;DR
Quant Experts (QE) is a breakthrough in Post-Training Quantization (PTQ) for Vision-Language Models (VLMs). By recognizing that "important" channels shift dynamically across different tokens and modalities, the authors replace static error compensation with a Mixture-of-Experts (MoE) system. This allows the model to adaptively "patch" quantization errors at the token level, achieving near-lossless 4-bit performance on models as large as 70B.
Background: The Moving Target of Signal Importance
In the world of model compression, we’ve long known that a few "outlier" channels carry most of the representational weight. Standard PTQ methods like SmoothQuant find these outliers using a calibration set and apply a fixed scaling factor to protect them.
However, this paper reveals a critical flaw for multi-modal models: Importance is a moving target. A channel that is vital for processing a high-resolution image might be irrelevant when the model is generating a text-based reasoning chain. Even within a single image, "important" channels for a token representing a "cat" differ from those representing "background grass." Static methods ignore this intra-modality and inter-modality dynamics.
Problem & Motivation: The Case for Token-Awareness
The authors present two key observations:
- Dynamic Migration: The positions of high-magnitude activation channels vary significantly across tokens (see Figure 3 in the paper).
- Frequency Imbalance: Only a small fraction of important channels are "always-on" (token-independent); the rest are "sometimes-on" (token-dependent).
Existing methods like LQER or MBQ treat all high-impact channels as a single global set, leading to "over-smoothing" or insufficient protection for the rare but critical token-specific features.
Methodology: Shared and Routed Experts
To solve this, Quant Experts (QE) introduces a two-tier compensation architecture:
1. The Shared Expert (Global)
This expert targets Token-Independent channels. It uses a standard low-rank adapter () to reconstruct errors that occur consistently across all inputs. Think of this as the "baseline" correction for the model's fundamental biases.
2. The Routed Experts (Local & Adaptive)
This is where the magic happens. The authors identify Token-Dependent channels and cluster them using Spectral Clustering based on their co-occurrence patterns (using Normalized Pointwise Mutual Information).
- Each cluster represents a specific "type" of error pattern.
- A set of Routed Experts (multiple low-rank adapters) is built, each specialized for one of these clusters.
- A Lightweight Router looks at the input token and picks the expert most likely to minimize the local error.
Figure 1: The QE framework. Token-independent channels are handled by the Shared Expert, while token-dependent channels are handled by Routed Experts.
Experiments: Accuracy Recovery at Scale
The authors tested QE on the Qwen2VL and InternVL2 series across a massive battery of benchmarks (MMMU, OCRBench, TextVQA).
Key Performance Highlights:
- Near-Lossless 4-bit: On the massive Qwen2VL-72B, QE narrowed the gap to full precision (FP16) to within negligible margins, outperforming the next best method (MBQ) by over 5% in the harsh W4A6 setting.
- Generalization: Even on language-only tasks (MMLU), the token-adaptive nature of QE helped maintain accuracy, proving it's not just a "vision trick."
Figure 2: Performance comparison on Qwen2VL-2B. QE (bottom row) consistently leads across almost every benchmark.
Critical Analysis & Insight
The real beauty of QE lies in its Hardware-Algorithm Co-design. While adding MoE usually increases inference latency, the authors used low-rank adapters. Because these adapters (ranks as low as 32 or 64) are mathematically small compared to the main weight matrices, the computational overhead is a mere 2-3%.
Limitation: The routing logic currently relies on a relatively simple absolute mean error prediction. There is potential for even more sophisticated routers (perhaps using small neural networks) to further improve the selection of local experts.
Conclusion
Quant Experts shifts the paradigm of VLM quantization from "Global Smoothing" to "Local Adaptation." By acknowledging that tokens are not created equal, it allows 70B+ models to run efficiently on commodity hardware without sacrificing the "intelligence" that makes them useful in the first place.
Takeaway for Practitioners:
If you are deploying VLMs in production, static PTQ is likely leaving performance on the table. Look towards adaptive reconstruction—the future of efficient multi-modal AI.