[UCLA] SUPERNOVA: Transforming Natural Instructions into General Reasoning Superpowers

TL;DR

The "Reasoning Revolution" sparked by DeepSeek-R1 has largely been confined to the "Verifiable" silos of Math and Code. SUPERNOVA breaks this wall. By repurposing the massive, human-annotated goldmine of instruction-tuning datasets (like SuperNI) for Reinforcement Learning (RL), researchers at UCLA have created a 4B model that out-reasons 8B giants, proving that how we curate reasoning data is more important than simply scaling it.

Background: The "Math-only" Trap

Current SOTA reasoning models (OpenThinker, OpenReasoner) are masters of calculus but often fail at simple logic puzzles. In fact, training specifically on math data can reduce a model's performance on general reasoning benchmarks like BBEH by 8%. This is because the "Inductive Bias" for solving a symbolic equation is fundamentally different from understanding causal relationships or temporal sequences.

The barrier to fixing this has always been Verifiable Rewards. In math, $x=5$ is easy to check. In general reasoning (e.g., "Why is the street wet?"), answers are often subjective and hard for an automated RL system to reward.

Methodology: The SUPERNOVA Pipeline

SUPERNOVA introduces a principled way to turn "squishy" instructions into "hard" RL targets.

1. From Instructions to Verifiable Questions

The authors took 83 diverse tasks from SuperNI and reformatted them. If a task was open-ended, they used GPT-family models to convert it into a Multiple-Choice Question (MCQ). This ensures that the RL algorithm (GRPO) can use a simple exact-match reward function.

2. Micro Mixing vs. Macro Mixing

This is the paper's most critical insight.

• Macro Mixing: Selecting the best tasks based on their average performance across all benchmarks.
• Micro Mixing: Selecting the best tasks specifically for each sub-skill (e.g., Task A for Causal, Task B for Temporal).

The experiments found that Micro Mixing consistently wins. Reasoning is not a monolithic skill; it’s a mosaic. A model benefits more from a diverse "diet" of tasks tailored to specific cognitive hurdles.

The "Failure" of Synthetic Interventions

In a surprising twist, the authors tried to "improve" the data by synthetically adding complexity—like making questions "long-context" or "going against priors." None of these interventions worked.

This suggests that human-annotated reasoning patterns are extremely fragile; adding synthetic noise to "toughen up" the model actually degrades the signal-to-noise ratio in RL training.

Performance: Small Models, Big Brains

The results on BBEH and Zebralogic are striking. SUPERNOVA-4B doesn't just beat its base version; it exceeds the performance of much larger models that were trained on massive, less-curated datasets.

• Zero-shot Scaling: Performance keeps improving as you increase test-time compute (Pass@1 to Pass@128), showing that the model has truly learned an "exploration" strategy rather than just memorizing answers.
• Cross-Model Transfer: The curation insights gained on Qwen models transferred perfectly to Llama-3.2-3B, proving that SUPERNOVA is a data recipe, not just a model-specific hack.

Critical Analysis & Conclusion

SUPERNOVA proves that we don't need to crawl the entire web to find reasoning data; we already have it in our old instruction-tuning sets. However, the study has limitations: the 25K sample size is small compared to industrial scales, and it remains to be seen if "Micro Mixing" scales linearly with 100B+ parameter models.

The Takeaway: Stop focusing on "more data." Focus on "Task Utility." By ranking tasks based on their actual RL performance (even if it costs more compute initially), we can train leaner, smarter, and more general reasoning agents.