LIVEditor: Breaking the ICL Bottleneck in Video Editing with In-Context Sparse Attention

TL;DR

Video editing is shifting towards In-Context Learning (ICL), but the quadratic cost of attention is a massive roadblock. This paper introduces LIVEditor and its core engine, In-Context Sparse Attention (ISA). By intelligently pruning "low-saliency" context tokens and using a novel 0-th order Taylor approximation for "flat" queries, ISA slashes attention latency by 60% while actually improving visual quality and instruction following over standard full-attention baselines.

Problem & Motivation: The ICL Tax

In-Context Learning has become a favorite for video editing because it allows models to simply "look" at a reference (context) and apply changes to the target (source). However, concatenating these frames doubles the sequence length, which quadruples the computational cost due to the quadratic nature of the attention mechanism.

The authors observed that existing sparse attention techniques (like Radial or Sliding Window) don't respect the fact that in an ICL setup, not all tokens are created equal. Context tokens (the reference) often contribute far less to the final output than the source tokens (the video being edited), yet standard models waste equal FLOPs on both.

Methodology: The "ISA" Secret Sauce

The authors propose In-Context Sparse Attention (ISA), which operates on three key technical insights:

1. Pre-Selection: Slicing the Redundancy

By analyzing attention heatmaps, the authors found that many context blocks have negligible impact. ISA uses a lightweight Pooling Attention mechanism to identify the most "salient" context blocks and prunes the rest early, reducing complexity from $O(N^2)$ towards a more manageable linear-ish overhead for the context portion.

2. Query Sharpness & Taylor Approximation

This is the theoretical heart of the paper. The authors prove that the error of a 0-th order Taylor expansion (approximating block interactions using mean values) is bounded by Query Sharpness.

• Sharp Queries: Highly concentrated attention, prone to high approximation error $\to$ Route to Full Attention.
• Flat Queries: Distributed attention, low approximation error $\to$ Route to 0-order Taylor Sparse Attention.

Figure 1: The ISA Workflow showing context selection and sharpness-aware query splitting.

Experiments: Faster and Better?

Usually, "sparse" means "faster but worse." LIVEditor breaks this trend. Across benchmarks like EditVerseBench, LIVEditor (ISA) outperformed its own full-attention version.

• Latency: Achieves ~60% reduction in attention module time.
• PickScore: Surpasses SOTA models like Lucy Edit and InsV2V in both frame-level and video-level aesthetics.
• Efficiency: The speedup becomes even more dramatic as sequence length increases, making it ideal for high-resolution or long-duration editing.

Figure 2: Scaling characteristics of ISA showing significant speedups over FlashAttention-2 (FA2) and SDPA.

Two-Stage Training & The 1.7M Dataset

To make LIVEditor robust, the team built a massive data pipeline, curating 1.7 million video editing pairs. They used a two-stage approach:

1. Stage I: Pre-training on large-scale mixed-quality data to learn general semantics (Object swap, style transfer).
2. Stage II: Fine-tuning on 89K high-quality "gold" samples to polish visual fidelity.

Critical Insight & Conclusion

Why does ISA work so well? The authors suggest that by pruning the context tokens, they are effectively filtering noise. In ICL, the context is a conditional prior; by removing irrelevant tokens, the model focuses on the most semantic-rich parts of the reference, leading to fewer artifacts and better alignment.

LIVEditor proves that we don't need to choose between speed and quality. By understanding the mathematical sharpness of our queries, we can route computation to where it’s needed most.

Limitations

While ISA is training-free compatible, it reaches its full potential through fine-tuning. Additionally, the "Flat Ratio" hyperparameter is sensitive—set it too low, and the approximation error can finally start to degrade fine details.