ORCA: Mastering the "When" and "How" of Foundation Model Errors
Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models
ORCA (Online Residual Contextual Adaptation) is a plug-and-play black-box adaptation framework designed to refine Time Series Foundation Models (TSFMs) during streaming inference. It achieves an average MSE reduction of 6.3% across 5 SOTA models (e.g., Chronos-2, Moirai-2.0, TimesFM-2.5) by learning to predict residuals based on the "context of errors."
TL;DR
As Time Series Foundation Models (TSFMs) move toward closed-source API delivery (similar to GPT-4), "White-Box" fine-tuning becomes impossible. ORCA (Online Residual Contextual Adaptation) solves this by treating the TSFM as a complete black box. By learning the context of errors—specifically conditioning on both the input features and the model's own predictions—ORCA acts as a lightweight client-side refiner that consistently slashes MSE across the latest SOTA models like Chronos-2 and Moirai-2.0.
The Shift to Black-Box Adaptation
The industry is reaching a consensus: Foundation models are the future of time series forecasting. However, a major hurdle remains—Concept Drift. Real-world data is non-stationary; a model pre-trained on 2023 data may stumble on 2026 market shifts.
Traditional adaptation requires backpropagation through the model. But if you are using a commercial API, you don't have the weights. Existing black-box attempts usually try to forecast residuals based only on past residuals. ORCA's core insight is that this is insufficient. To fix an error, you must understand the Context: What was the input that led to this mistake, and what did the model eventually predict?
Methodology: Mapping the Context of Errors
ORCA doesn't just look at the error signal; it looks at the relationship .
1. The Linear Adapter
Instead of a complex neural net that might overfit noisy residuals, ORCA uses a Linear Adapter with strong structural bias. It decomposes the context into Trend and Seasonal components before applying channel mixing.
2. The Boltzmann Router: Preventing "Negative Optimization"
One fear in online learning is that the adapter might actually make the prediction worse. ORCA introduces a Boltzmann Router inspired by statistical mechanics. It treats smoothed predictive errors as "energy states." If the adapter's energy (error) is high, the router dynamically shifts confidence back to the base foundation model, ensuring the system is "safe by design."
3. Predictive-Space Bayesian Update
To solve the Plasticity-Stability Dilemma (learning new patterns without forgetting old ones), ORCA performs updates in the "Predictive Space." It uses a Bayesian-inspired loss function that anchors the current adapter to a "prior" (a delayed copy of itself), preventing catastrophic forgetting during high-speed streaming updates.
Experimental Battleground: SOTA vs. ORCA
The authors tested ORCA against 5 of the most recent TSFMs (released 2025-2026) across 8 benchmarks.
Key Result: 90% Success Rate
Across 120 different testing scenarios, ORCA successfully refined the base model 90% of the time. In the Few cases where it couldn't improve the result, the Boltzmann Router successfully "muted" the adapter, limiting potential degradation to a negligible ~3%.
As shown in the heatmap above, the "Green" zones dominate, representing significant MSE drops. Models like Moirai-2 and TiRex saw average improvements of over 8% when paired with ORCA.
Deep Insight: Why "Context" Matters
A critical takeaway from the ablation studies (Figure 4 in the paper) is the quantification of "What to learn."
- Learning only from Past Errors: -0.7% (Actually harms performance).
- Learning only from Input: +4.5%.
- Learning from Input + Base Prediction (Context): +6.5%.
This proves that error surfaces in foundation models are highly dependent on the current scenario () and the model's intended trajectory ().
Conclusion & Future Impact
ORCA proves that you don't need access to "the brain" (weights) to fix the "output" (predictions). By focusing on the Context of Errors, ORCA provides a high-efficiency ( overhead) solution for companies using TSFM APIs. It effectively bridges the gap between universal pre-training and specialized, non-stationary real-world applications.
Future Outlook: As more TSFMs adopt MoE (Mixture of Experts) or SSM (State Space Model) architectures, client-side adapters like ORCA will be essential tools for maintaining "local" accuracy on top of "global" intelligence.