How can token-level LLM routing be served efficiently at scale?

Token-level routing breaks single-model serving assumptions

Token-level routing lets the model that generates each token change from step to step. That is a finer granularity than query-level routing, where a request is sent to one model and that model produces the whole response. The paper cites R2R as evidence of the efficiency upside: it can match a 32B model's quality while routing only 5% of tokens to the 32B model and decoding the rest with a 1.5B model, compared with routing around 40% of queries to the 32B model.

Existing serving systems such as SGLang and vLLM assume a single LLM, with all active requests synchronized at each decoding step. Token-level routing violates that assumption in three ways. Step desynchronization arises because models differ widely in per-step latency, so a synchronized batch waits for the slowest model while faster models sit idle. Batch admission delay arises because frequent model switches mean a routed request often arrives while its target model is still busy, so it waits for the next batch and leaves bubbles. Implementation complexity arises because these systems expose no interface for per-step routing decisions, forcing extensive code changes that must also be coordinated with continuous batching and prefix caching.

Request-centric programming, model-centric execution

TokenRouter's design principle splits the problem in two. Developers describe a routing algorithm by tracing how a single request flows across cooperating LLMs, using a route-send-receive interface. The runtime, meanwhile, launches a subserver for each LLM and dispatches requests asynchronously, so the system complexity of routing stays hidden from the developer.

This split matters because it decouples the routing algorithm from serving-system optimization. In the implementation, TokenRouter is built on SGLang 0.5.1, the server interface is adapted from SGLang's FastAPI front end to accept multi-model port arguments, and inter-subserver messages travel over local ZeroMQ IPC. Each subserver runs its three-stage iteration as a single event loop in a dedicated subprocess, and the only inter-subserver coordination is the peer-message channel. All evaluated routing algorithms share this runtime, each implemented as a scheduler subclass.

Delayed batching trades admission delay for fuller batches

Each subserver uses a delayed-batching scheduler: a new batch launches only when at least a threshold number of requests are waiting in that model's queue. The threshold is the key tuning knob. Set it too low and batches stay small and fragmented; set it too high and requests wait unnecessarily, and the system can even deadlock if no model accumulates enough queued requests to launch.

The paper derives the optimal thresholds rather than tuning them by hand. It models the system as a discrete-time Markov chain observed at batch-completion events, with state recording each model's queue length, running batch size, and residual execution time. Routing probabilities are assumed stationary across decoding steps, and routing decisions are assumed independent across requests. From the stationary distribution, the model computes steady-state committed-token throughput as expected committed tokens per state transition divided by expected transition time, then searches threshold vectors that maximize it subject to a feasibility condition that prevents deadlock.

Predicted thresholds match measured optima

The throughput model is validated on R2R with two models, where model 0 is the small model and model 1 the large one. With routing probability p = 0.35, measured decoding-step latencies of 6.0 ms for the small model and 27.9 ms for the large model, and a time quantum of 0.1 ms, the model predicts throughput across concurrency levels and threshold vectors.

Fixing the small-model threshold at 1 and varying the large-model threshold, predicted throughput tracks measured throughput across concurrency levels N = 2, 3, and 4. More importantly, at every evaluated concurrency level the threshold predicted to maximize throughput coincides with the empirically optimal threshold. That agreement is the paper's evidence that the model captures the effect of delayed batching well enough to guide threshold selection, not just to describe trends after the fact.

Throughput gains hold across algorithms and model pairs

Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01–64.15x higher decoding throughput than existing systems. The lower end of that range is itself informative: on some small-model/large-model pairs the gain is roughly twofold, not orders of magnitude. Table 3 reports R2R versus TokenRouter throughput in tokens per second for three pairs: 77.08 versus 197.40 (2.56x) for 0.6B/32B, 103.37 versus 285.41 (2.76x) for 1.7B/8B, and 106.67 versus 211.81 (1.99x) for 4B/8B.

The paper also reports a parallel-strategy finding that shapes these numbers: increasing the large model's tensor parallelism size is vital for throughput, while the small model's tensor parallelism size has little effect. Unless otherwise specified, the experiments therefore set large-model tensor parallelism to 2 and small-model tensor parallelism to 1. Readers comparing throughput figures should note that these are decoding throughput in tokens per second under that configuration, not end-to-end latency.

Fine-grained routing moves the Pareto frontier

Throughput alone does not settle whether token-level routing is worth serving. The paper therefore evaluates the throughput–accuracy trade-off on AMC23 using the Qwen3-0.6B and Qwen3-32B pair with greedy decoding and a maximum output length of 8,192 tokens. All requests are sent concurrently, throughput is computed per request as output tokens divided by elapsed time, and small-model/large-model memory fractions are held equal across systems for a fair comparison.

Under those conditions, TokenRouter shifts token-level routing to a new Pareto frontier, making fine-grained routing more competitive than coarse-grained routing on a state-of-the-art framework such as SGLang. The comparison includes the official R2R implementation and query-level routing baselines from RouteLLM. This is the result that changes how a reader should weigh routing granularity: the question is not only whether token-level routing saves compute, but whether the serving system can convert those savings into throughput without giving back accuracy.

Where the throughput model stops applying

The optimization rests on an assumption: the number of tokens between two consecutive sends follows a geometric distribution. The paper states this holds for most token-level routing algorithms and leaves corner cases that violate it to future work. The model also assumes stationary routing probabilities and independence across requests; the paper notes that if routing probabilities vary over time, the same analysis applies over a short interval where they can be approximated as constant.

These are conditions on the threshold optimization, not on the asynchronous subserver architecture. A routing algorithm with non-geometric send spacing may still run on TokenRouter, but its delayed-batching thresholds would not be covered by the derived optimum. The practical priority follows from that split: the architectural gain from asynchronous model-centric execution is the paper's broad claim, while the specific threshold values are validated on R2R with two models and should be re-derived for other routing patterns.

Sources used in this answer