[ICLR 2025] TRouter: Solving the Cold-Start Problem in LLM Routing via Synthetic Task Profiles

TL;DR

LLM routing—the art of picking the right model for a specific query to balance cost and quality—usually requires a lot of "in-domain" training data. But what if you have zero data for a new application? This paper introduces a Multi-Level Task-Profile-Guided Data Synthesis framework and TRouter, a router that uses latent task variables to achieve SOTA performance in cold-start scenarios, proving that we can "synthesize" our way out of the data scarcity trap.

Problem & Motivation: The Cold-Start Trap

Deploying LLMs involves a brutal trade-off: use a "God-mode" model like GPT-4o and go bankrupt, or use a tiny 7B model and fail the task. Routers are supposed to solve this, but they have a "Chicken and Egg" problem. To train a router, you need thousands of labeled queries from your specific domain. If you’re a startup launching a new product today, you have zero queries.

Traditional routers also suffer from Domain Shift. A router trained on math problems might be completely lost when asked to summarize legal documents. The authors observed that existing SOTA routers (like RouterDC or MetricRouter) often perform worse than simple rule-based heuristics when moved to a new domain.

Methodology: Synthesizing Domain Expertise

The authors break the problem into two parts: generating the data you don't have, and building a router that understands the "type" of the task.

1. Task-Profile-Guided Data Synthesis

Instead of just asking an LLM to "generate some questions," the framework follows a rigorous hierarchical process:

• Task Type Generator: Starts with broad domains (e.g., Programming) and recursively expands into subcategories (e.g., Debugging) and 5 levels of difficulty.
• Quality Evaluator: A "self-critique" loop that shuffles and refines these categories to ensure the taxonomy is distinct and non-overlapping.
• QA Pair Generator: Produces 40 diverse pairs per "profile" (a leaf node in the taxonomy tree), using semantic similarity filters to prevent redundancy.

2. TRouter: Task-Type-Aware Routing

Standard routers try to map a raw query directly to a performance score ($p(h|q,m)$). TRouter introduces a latent variable $t$ representing the task type. The logic is: Query → Predicted Task Type → Predicted Performance/Cost. By using the synthesized taxonomy as a "prior," TRouter regularizes the learning process, making it much more robust to noise and domain shifts.

Experiments: Performance Beyond the Baseline

The authors tested TRouter across diverse pools (Qwen3, Gemini, Doubao) and evaluation metrics (F1, Accuracy, and LLM-as-a-judge).

Key Results:

• Cold-Start Superiority: TRouter trained only on synthetic data beat all other cold-start baselines.
• Efficiency: As shown in the ablation studies, TRouter needs as few as 5-10 shots per task type to reach near-optimal performance, making it incredibly cheap to train.
• Interpretability: Unlike black-box routers, TRouter can tell you why it picked a model (e.g., "I categorized this as 'Hard Calculus', so I'm routing to the 235B model").

Critical Insight: Difficulty is the Key

The ablation study reveals a fascinating nuance: synthesizing data based on difficulty levels is far more effective than just subcategories. Difficulty-level conditioning provides a nearly linear increase in validated diverse samples, whereas domain-only conditioning quickly leads to repetitive, low-value data.

Conclusion & Future Outlook

TRouter proves that the hierarchical structure of human knowledge (Domain → Subtask → Difficulty) is a powerful inductive bias for LLM orchestration. While the method still requires a "seed" domain description, it effectively removes the primary hurdle to smart routing: the need for massive, manually-labeled datasets.

Future Work: The authors suggest moving toward "Node Offloading," where users can manually prune or update parts of the taxonomy tree as their application evolves, blending human expertise with synthetic data.