MOSS: Mastering Video Understanding via High-Order Self-Similarity
Exploring High-Order Self-Similarity for Video Understanding
This paper introduces the Multi-Order Self-Similarity (MOSS) module, which leverages higher-order Space-Time Self-Similarity (STSS) to capture complex temporal dynamics in videos. By integrating several orders of STSS, MOSS achieves SOTA performance in action recognition, video MLLMs, and robotic tasks with minimal computational overhead.
TL;DR
Researchers have long struggled to move video AI beyond "static frame recognition" towards true "motion understanding." MOSS (Multi-Order Self-Similarity) is a breakthrough neural module that doesn't just look at how pixels move (1st-order), but how motion patterns themselves relate to one another (2nd-order and beyond). This hierarchical approach allows models to segment moving objects from background clutter and distinguish complex interactions, achieving SOTA results on benchmarks like Something-Something and real-world robotics with negligible compute costs.
The Problem: Why "Motion" is Harder Than "Appearance"
Most current video models (including heavyweights like Video Transformers) are surprisingly biased toward static backgrounds. If a model sees a basketball court, it guesses "basketball," even if no one is moving.
The classical solution was Space-Time Self-Similarity (STSS)—comparing local patches across frames to find correspondences. However, 1st-order STSS has a major weakness: it is easily confused by visually similar objects. If two identical pens move in opposite directions, a 1st-order model sees "pen texture" and fails to isolate the individual trajectories.
Methodology: The Power of Hierarchy
The core insight of this paper is that Temporal Dynamics have a hierarchy. The authors generalize STSS into a recursive process:
- 1st-Order (Appearances): "This patch looks like that patch in the next frame." → Captures Motion Flow.
- 2nd-Order (Similarities of 1st-order): "These different patches are all moving in the same way." → Captures Motion Segments/Objects.
- 3rd-Order (Similarities of 2nd-order): "These motion segments are interacting in this specific layout." → Captures Complex Group Dynamics.
(a) STSS maps across different orders, showing the progression from raw motion to object-level segmentation.
The MOSS Module
To make this practical, the authors designed the MOSS Module. It takes a feature map, computes the multi-order STSS, and encodes it using a lightweight "late-fusion" encoder (spatial first, then temporal).
(Note: Based on Fig 2 in paper) The recursive transformation keeps feature dimensions consistent while drilling deeper into relational structures.
Experiments & Results: Crushing the Baselines
MOSS was tested across three massive domains:
1. Action Recognition
On Something-Something V2, a dataset where you must understand motion to succeed (e.g., "Pushing something from left to right"), MOSS-L achieved 74.4% Top-1 accuracy. It beat the baseline Side4Video while using significantly fewer FLOPs.
2. Video Large Language Models (MLLMs)
By plugging MOSS into VideoLLaMA3, the model's ability to count repetitions or describe fine-grained motion improved substantially (+2.8% on MotionBench) with only 0.6% more parameters.
3. Robotics (VLA)
This is perhaps the most impressive result. In a task called PongPredict (predicting where a virtual ball will hit a wall), standard single-frame and multi-frame models failed. MOSS-equipped robots, however, perceived the trajectory accurately, boosting success from 51.9% to 81.5%.
Performance comparison on Something-Something V1 and V2 benchmarks.
Depth Insight: Why Higher-Order Works
Look at the visualization below. In the 1st-order map, the background and query are noisy. By the 2nd-order, the model has "segmented" the hand and the object because they share a coherent motion pattern, effectively "muting" the static background noise.
Higher-order STSS effectively suppresses static regions while highlighting moving objects.
Critical Analysis & Conclusion
Takeaway: Explicitly modeling the "logic of motion" (self-similarity) is more parameter-efficient than hoping a giant Transformer learns it implicitly.
Limitations: The authors noted a "point of diminishing returns." While 1st and 2nd orders provide massive gains, 4th-order and beyond offer marginal improvements for current datasets. Furthermore, under extreme motion blur, the 1st-order similarity becomes unreliable, causing a "cascading failure" through higher orders.
Future Outlook: MOSS proves that lightweight, modular temporal "experts" can be grafted onto frozen foundation models to give them "dynamic eyes." This is a major win for edge deployment and robotics.