How does broadening temporal views improve self-supervised video representation learning?

The asymmetry that defines BraVe

Most self-supervised video methods borrow an image recipe: take two views of the same clip, augment each the same way, and train a shared backbone to align them. Because both views cover the same temporal span, the model never has to learn from information at different time scales. BraVe changes one thing: the two views are no longer symmetric in time. One view is a narrow clip of a few seconds; the other is a broad view covering the longer temporal context of the full video clip. The training objective is to predict the broad view from the narrow one.

That prediction task is what makes the representation useful, according to the paper's framing. To predict what surrounds a short event, the model must extrapolate what happened before, during and after the narrow clip, which the authors argue requires understanding the structure of events. The paper also notes that related local-to-global proxy tasks exist in the image domain through architecture design or multi-size cropping, but applying them to video is not the same as exploiting time.

Separate backbones let other modalities supervise

BraVe processes the two views with different backbones, so the broad view does not have to be another RGB crop. The paper lists optical flow, randomly convolved RGB frames, audio, or combinations of these as possible broad views. Audio is encoded as a log-mel spectrogram, and the temporal window for the audio view is enlarged relative to the narrow video view to increase the context carried by the supervisory signal. The visual narrow view and the audio broad view are kept in sync at their starting point.

This design choice matters for interpreting the results: the broad view is a supervisory target, not a second input that must share the narrow view's modality or augmentation pipeline. The paper's appendix compares broad-view combinations and reports that using multiple visual modalities in the broad view improves HMDB51 and UCF101 but decreases Kinetics-600 performance compared with using two RGB views. The authors attribute the HMDB51 and UCF101 gains to complementary information from RGB and flow plus the larger capacity of two broad backbones, and they acknowledge that this version is not directly comparable to a single broad backbone.

How broad the view is changes the result

The paper's ablations treat the temporal extent of the broad view as a variable rather than a fixed recipe. In the broad-view combination study, the narrow extent is 1.3 seconds and the broad extent is 10 seconds, and results are reported for HMDB51, UCF101 and Kinetics-600. The appendix also reports a 10-second narrow view trained with 64 frames at 6.25 FPS and 112×112 crops, evaluated with clips of the same frame rate and size, which shows that the framework can be run with a much longer narrow window than the few-second default.

These numbers should be read as within-paper comparisons under the stated training and evaluation settings, not as a general law about optimal clip length. The relevant reader takeaway is that BraVe's benefit is tied to the gap between the narrow and broad temporal windows: the broad view must actually carry more context than the narrow one for the prediction task to require extrapolation.

Syncing audio and video helps most at the start

When the broad view is audio, the temporal relationship between the two views is a design decision. The appendix compares three options: the narrow and broad views start at the same instant, the narrow view is centered on the broad view, or the two are independently randomly sampled in time. On AudioSet-trained models, starting the views together gives 68.2 on HMDB51, 92.7 on UCF101 and 69.7 on Kinetics-600; centering gives 67.3, 92.2 and 70.6; asynchronous sampling gives 67.4, 92.3 and 69.1.

The paper reads this as support for already established evidence that syncing audio and video benefits self-supervised learning, and concludes that syncing at the beginning is overall the best option. The Kinetics-600 column is the exception, where centering scores highest, so the choice is not uniformly settled across benchmarks. Readers comparing audio-based BraVe variants should therefore check which sync strategy and which benchmark a reported number comes from.

Where the transfer results come from

The headline claim is state-of-the-art self-supervised representation learning on standard video and audio classification benchmarks: UCF101, HMDB51, Kinetics, ESC-50 and AudioSet. The evaluation protocols differ by benchmark, and the paper is explicit about them. HMDB51 and UCF101 are evaluated by fine-tuning with the LARS optimizer, a batch size of 256 except for TSM-50x2 and R(2+1)D-50 at 128, 50 warm-up epochs followed by cosine decay over 550 epochs, and test-time averaging over 30 clips of 224×224. Kinetics-600 is evaluated with a linear layer trained for 100 epochs at batch size 512, reporting top-1 accuracy on the validation set. AudioSet uses a shallow one-hidden-layer MLP with 512 units trained for 20 epochs on frozen audio features.

Because these are different protocols, a number from one benchmark cannot be substituted for another. The paper's own discussion adds a scope condition: when trained with a larger backbone, BraVe is described as competitive with supervised pre-training techniques, which is a different claim from outperforming other self-supervised methods.

What the broad backbone actually learns

A natural question is whether the broad backbone is simply a better encoder. The appendix tests this by evaluating the model trained with an RGB broad view, without random convolutions, on the narrow backbone's terms. The broad network reaches 63.4 on HMDB51 and 90.3 on UCF101, while the narrow backbone reaches 63.2 and 90.5 respectively. So on RGB the broad network performs on par with the narrow one.

The paper states that its best results come when the broad view uses a different modality, such as flow or audio, or an altered modality such as random convolutions, and that in those cases the comparison is not possible. This is the paper's own interpretation of why the asymmetry helps: the value is not a bigger encoder but a supervisory signal that carries information the narrow view does not contain.

The priority is the view gap, not the backbone

Across the paper's comparisons, the variable that most consistently organizes the results is the relationship between the two views: how much longer the broad view is, whether it is synced with the narrow view, and whether it comes from a different or altered modality. The broad-backbone check shows that a same-modality broad view lands at roughly the narrow backbone's level, while flow, audio and random-convolution broad views are where the reported gains concentrate. The sync study adds that the alignment choice can move individual benchmark scores in different directions.

For a reader deciding whether BraVe's approach transfers to a new setting, the paper's evidence points to a specific priority: design the broad view so it genuinely adds temporal or modal context the narrow view lacks, and report which benchmark and evaluation protocol each gain comes from. The paper's own scope is self-supervised pre-training on Kinetics or AudioSet with the listed backbones and the stated fine-tuning, linear and shallow-classifier protocols; it does not establish that the same view gap helps at every scale or on every downstream task.

Sources used in this answer