[CVPR 2026] DHP: Breaking the Static Barrier in MLLM Training with Dynamic Hybrid Parallelism

TL;DR

Training Multimodal Large Language Models (MLLMs) on diverse video-language datasets is notoriously inefficient due to extreme data heterogeneity. Dynamic Hybrid Parallelism (DHP) solves this by replacing static parallelism grids with an adaptive framework that reconfigures communication groups on the fly. It supports non-power-of-two parallelism degrees and employs a 2D-Dynamic Programming solver to minimize makespan, achieving up to 1.36× throughput speedup over industry standards like Megatron-LM.

Problem & Motivation: The Heterogeneity Debt

Modern MLLMs are scaling to handle high-resolution images and long-form video. However, real-world data distributions are "long-tailed": a single training batch might contain a 2-second clip and a 64-second video simultaneously.

Existing frameworks (Megatron-LM, DeepSpeed) use Static Parallelism. If you configure your grid to handle the longest sequence, memory is wasted on short sequences; if you configure for the average, long sequences trigger Out-of-Memory (OOM) errors. This mismatch results in:

1. Workload Imbalance: Devices processing short sequences sit idle waiting for those handling long ones.
2. Redundant Communication: Static groups often force sequences into parallelism degrees that are too high, increasing communication overhead unnecessarily.
3. Rigidity: Most systems require parallelism degrees to be powers of two (2, 4, 8...), which is too coarse for optimal packing.

Methodology: Adaptive Orchestration

The core innovation of DHP lies in its ability to treat parallelism degrees as dynamic variables rather than static constants.

1. The Two-Stage Solving Strategy

Solving for the optimal assignment of sequences to NPU ranks is NP-hard. DHP tackles this via a two-stage approximation:

• Stage 1: Memory-aware Sequence Packing: Uses the Best-Fit Decreasing (BFD) algorithm to pack shorter sequences into the memory "headroom" of larger ones. This simplifies the optimization space by creating "Atomic Groups."
• Stage 2: 2D-Dynamic Programming: Determines the optimal parallel degree for each group to minimize the "makespan" (the time until the slowest device finishes).

2. Flexible Ring-style Context Parallelism

Unlike Ulysses-style sequence parallelism, which is tied to the number of attention heads (often requiring power-of-two degrees), DHP leverages Ring-style CP. This allows for arbitrary integer degrees (e.g., 3, 5, 6), enabling much finer control over resource allocation.

3. Asynchronous Execution

To ensure the "brain" (the CPU-based scheduler) doesn't slow down the "brawn" (the NPUs), DHP uses a producer-consumer pattern. While the NPUs calculate the current batch, the CPU is already solving the optimization problem for the next batch.

Experiments & Results

DHP was evaluated using InternVL3 and Qwen3-VL models on Ascend 910B NPU clusters (up to 64 nodes).

Performance Gains

On the OpenVid dataset—which features the most diverse sequence lengths—DHP achieved a 1.36× speedup for 8B models. Even on more uniform datasets like MSRVTT, DHP consistently provided 1.14× to 1.25× gains.

Scaling Efficiency

As the number of NPUs increases, communication overhead usually cannibalizes throughput. DHP mitigates this better than baselines. When scaling from 8 to 64 NPUs, DHP’s relative advantage over DeepSpeed actually increased from 1.02x to 1.16x, highlighting its superior scalability.

Precision and Overhead

The DHP Cost Estimator predicts execution time with under 8% error, and the Solver Time is negligible (consistently < 100ms), proving it can be fully overlapped with the multi-second computation time of LLM batches.

Critical Analysis & Conclusion

Takeaway

DHP shifts the paradigm from "fitting data into a fixed grid" to "shaping the grid around the data." By enabling non-power-of-two parallelism and near-optimal sequence packing, it effectively eliminates the "heterogeneity tax" inherent in multimodal training.

Limitations & Future Work

• Model Parallelism Rigidity: Currently, TP (Tensor Parallelism) and PP (Pipeline Parallelism) remain static because reshuffling weights in real-time is too costly. Future research might explore ways to make these dimensions elastic without incurring massive overhead.
• Cross-Node Latency: While DHP handles ring communication well, the impact of non-power-of-two groups on network topology (e.g., non-aligned InfiniBand paths) deserves further investigation.

DHP is a significant step toward "native" multimodal training systems that are as dynamic as the data they ingest.