How can motion estimation enable efficient Bayer-domain computer vision?

Two bottlenecks sit at opposite ends of the pipeline

The paper frames the efficiency problem as two separate costs. At the front end, an image signal processor (ISP) converts raw Bayer sensor data into RGB pixel by pixel before any vision model sees it; the authors describe this preprocessing as computationally intensive and non-differentiable. At the back end, conventional video frameworks process each high-resolution frame independently, so they never exploit the fact that adjacent frames largely repeat each other.

The proposed framework attacks both at once. It feeds Bayer data straight into the network and trains all models in the Bayer domain, removing the ISP from the path. Inside the network, a module called Motion Estimation-based Video Convolution (MEVC) replaces standard convolutions so that later frames reuse computation from earlier ones. The authors position this against two prior acceleration families: key-frame/non-key-frame model pairs, which still leave redundant calculations, and block-copy schemes such as BlockCopy, which decide whether to copy a block or recompute it but ignore differences between current and reference blocks, so accuracy falls quickly as the acceleration ratio rises.

Motion vectors plus sparse residuals replace full convolution

The mechanism borrows the logic of video coding. For each block in a current frame, motion estimation searches a reference frame for the most similar block and records two things: a motion vector, which is the position difference, and a residual map, which is the block difference. The current frame can then be reconstructed by predicting from the reference frame along the motion vector and adding the residual.

The paper's key step is applying this to convolution. Because convolution is a linear operator, the convolution of the current frame equals the convolution of the motion-predicted reference plus the convolution of the residual. The first term is already known from processing the reference frame, and the residual map is usually sparse, so the correction step is cheap. In practice, each video sequence is split into groups of pictures; the first frame of each group is a key frame processed with standard convolutions, and subsequent frames use MEVC modules. MEVC only swaps out convolution layers, leaving batch normalization and activation layers unchanged.

Sliding windows align block matching with convolution geometry

A structural mismatch has to be solved first. Traditional motion estimation uses large, non-overlapping blocks, whereas convolutional layers operate on overlapping patches defined by stride, with small, often odd-sized kernels such as 3×3. The paper's sliding-window motion estimation matches the motion-estimation block size to the convolution kernel size and searches in a sliding-window fashion across the input frame, folding the convolution stride into the search so each searched block corresponds to a unique output activation.

Two further details control cost and sparsity. Matching uses the sum of absolute differences criterion, and any residual value below a threshold is suppressed to zero. An early-stopping rule ends the search when residual sparsity of a candidate match exceeds a predefined threshold, given as 0.3 in the example. During prediction, motion vectors estimated from input tensors are rescaled by the layer stride so warped outputs stay aligned with convolutional activations.

The search range is the main user-controlled cost knob

The paper gives an explicit acceleration ratio built from four quantities: the matching success ratio α, the residual map sparsity β, the search range R, and layer-fixed parameters including stride s and the number of output filters. The term (2R+1)²/(s²C_out) captures motion-estimation overhead, so a larger R covers larger motion but increases search cost quadratically. α and β are measured from motion estimation, while s and C_out are fixed by the convolution layer.

This makes the search range the practical decision variable rather than a free setting. The reported experiments use a search range of 1, a threshold of 0.01 and a group-of-pictures size of 12, so the measured savings correspond to that configuration and to motions the small window can capture.

Segmentation keeps accuracy while GFLOPs fall about 70%

Semantic segmentation is evaluated on CamVid and Cityscapes with PSPNet18 and HRNet-W48 backbones, reporting mean Intersection over Union (mIoU) for accuracy and GFLOPs for computational cost. On Cityscapes with HRNet-W48, the method reaches 78.98% mIoU against 79.06% for the RGB counterpart, while GFLOPs drop from 748.68 to 228.91. On CamVid with PSPNet18, mIoU is 69.28% versus 69.43% for RGB, with GFLOPs falling from 309.02 to 90.85.

The pattern holds across the four dataset-backbone combinations: accuracy stays close to the RGB baseline, and GFLOPs fall by roughly 70% for both PSPNet18 and HRNet-W48. The authors also note that these savings exclude ISP processing, which all RGB-domain methods require but this pipeline eliminates, so they describe the reported efficiency gains as conservative.

Removing the ISP also shortens measured front-end latency

Beyond FLOPs, the paper measures system-level front-end latency on a HikRobot prototype. The conventional RGB pipeline incurs 11.08 ms of ISP latency and 12.58 ms of transmission latency, whereas the ISP-free Bayer pipeline removes ISP processing entirely and reduces transmission latency to 5.41 ms, which the authors attribute to raw Bayer data being more compact than RGB data.

The paper also summarizes published ISP power and latency statistics to argue that ISP processing carries non-negligible front-end cost, listing devices with power figures ranging from tens to hundreds of milliwatts and, for one Qualcomm ISP, latencies from 14–16 ms at 1080p→720p up to 40–42 ms at 4K→4K. These are cited statistics rather than measurements from this system, and they describe the cost being avoided rather than the accuracy of the proposed models.

Gains depend on motion that the search window can match

The efficiency claim rests on temporal redundancy being both present and findable. The acceleration ratio depends on the matching success ratio α and residual sparsity β, both measured from motion estimation, so scenes with large or poorly aligned motion relative to the search range of 1 will match less often and leave more residual to compute. The authors make this contrast explicit: prior reuse methods that omit residual compensation are sensitive to alignment errors and degrade noticeably under large or misaligned motion, and their own design adds explicit residual correction inside each convolution layer to counter that.

The evaluation covers video semantic segmentation, depth estimation and object detection with convolutional backbones, on synthetic Bayer-converted data and real Bayer video datasets. The paper presents the framework as complementary to raw-domain representation work, which studies raw-input representation and adaptation rather than convolutional computation reuse. The practical reading is therefore conditional: the over-70% FLOPs reduction with minimal accuracy loss is the result to weigh when motion is small enough for the configured search window and when the front-end ISP cost is part of what you are trying to remove.

Sources used in this answer