Can a harness-engine protocol improve agentic LLM serving efficiency?

Two layers hold complementary information

Agentic serving spans two layers that see different things. The agent harness knows request dependencies, context lifecycles, and application objectives; the inference engine observes request queues, KV-cache residency, resource pressure, and supported execution modes. The paper's motivating example is concrete: an engine unaware of future context use may evict KV state immediately before it is needed, forcing costly re-prefilling, while a harness unaware of cache residency may schedule a cold request before an equivalent cache-resident one even when both are ready and can be legally reordered.

The paper argues this is not just a missing optimization but a missing abstraction. Prior work shows the value of workflow information and runtime feedback, yet each effort defines its own cross-layer signals, control semantics, and operation lifecycles, tightly coupling policies to particular harnesses and engines. HEAR is proposed as the shared control-plane abstraction that connects workflow semantics with runtime execution state.

HEAR standardizes messages, not policies

HEAR organizes communication into four semantic categories. The harness provides Execution Description and Intent plus Execution Requirements and Control; the engine returns State and Capabilities plus Execution Outcomes. Each exchange is associated with the relevant request, context version, or serving instance. The protocol further distinguishes description from instruction, preference from requirement, observation from guarantee, and acceptance from completion.

The key design choice is separating protocol semantics from optimization policies. HEAR defines what exchanges mean, while coordination policies decide what information to exchange and what actions to take. That separation is what allows diverse coordination strategies without changing workflow or model semantics, and it lets existing policies such as PBKV and Pythia keep their predictors and optimization rules while adapters expose the required workflow descriptions, engine observations, and controls through HEAR.

Cache-aware scheduling trades reuse against tail latency

The first instantiation is online cache-aware runtime coordination. At each decision point the harness describes ready requests, their contexts, and applicable execution requirements including waiting protection; the engine returns the latest relevant cache availability and queue state. The scheduling policy then assigns priorities to ready requests. Three execution-order policies are compared: FCFS uses the default engine scheduler, Cache-Aware converts engine-reported KV residency into harness-side scheduling preferences, and Cache-Aware+Guard-W additionally prevents bypass or preemption after a request waits W seconds.

On SCBench, FCFS achieves only 19.2% KV-cache reuse for follow-up prompt tokens because of prefix recomputation. Cache-Aware raises reuse to 87.6%, cuts median TTFT from 63.1 to 4.6 seconds, and shortens batch completion from 481 to 224 seconds. But repeatedly deferring cold requests increases maximum TTFT. Waiting protection bounds that behavior: Cache-Aware+Guard-40 improves P50, P95, and maximum TTFT over FCFS while retaining a 1.61x batch speedup. Raising the threshold to 60 seconds favors reuse and batch completion but weakens tail-latency protection. The two directions of HEAR are both exercised here: engine feedback identifies reuse opportunities, while harness requirements constrain how aggressively the scheduler exploits them.

No single policy wins across load levels

Mooncake Bench shows the preferred policy changing with load while the HEAR interface stays unchanged. At 50% load, limited queuing makes idle-period eviction the primary source of reuse loss, so Session-Aware outperforms execution-order control and raises reuse from 19.5% to 31.9%. At 75%, eviction and execution-order control become complementary: combining Session-Aware with Cache-Aware+Guard yields the shortest session time, highest throughput, and highest reuse. At 100%, Cache-Aware achieves better P50, session latency, and throughput, whereas Session-Aware improves tail latency and reuse.

This is the paper's clearest evidence that workflow intent and live engine state provide complementary benefits across load regimes rather than one dominating. Session-Aware itself is a KV-retention policy: after each request the harness estimates session continuation from depth and prior inter-turn intervals, explicitly marking the estimate as descriptive intent rather than a retention requirement, and the engine combines it with access recency. The estimate is updated only from completed conversations and does not use future information from the current replay.

Role-specific execution modes beat one fixed mapping

The second instantiation is workload-aware execution-mode selection for agent roles. Here the harness communicates workload descriptions and objectives while the engine supplies stable capability profiles, and the policy selects inference modes per role. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield 1.23x and 2.45x end-to-end speedups respectively, without observed task-quality degradation.

Role-level attribution shows where the gain comes from. Reader/sub-agent execution provides the largest consistent gain: selecting H2O reduces mean latency by 5.60x on BrowseComp-Plus and 7.22x on DeepResearchBench. On BrowseComp-Plus this accompanies a 76.2% reduction in prefilled tokens and an increase in cache reuse from 6.2% to 80.3%; on DeepResearchBench it accompanies 49.8% chain reuse. The preferred Main-side mode is workload-dependent: BrowseComp-Plus Main requests generate roughly 0.4K output tokens and vLLM is 1.14x faster than OmniKV, while DeepResearchBench Main requests generate roughly 4.4K tokens in a round-synchronized workflow and OmniKV is 1.08x faster than vLLM, because the longer decoding workload better amortizes mode-specific overhead.

A universal mapping costs 7.8% to 14.0%

Neither selected mapping is optimal for both workloads, and the paper quantifies the penalty for ignoring that. Applying OmniKV-H2O universally would increase BrowseComp-Plus wall time by 7.8%, whereas applying vLLM-H2O universally would increase DeepResearchBench makespan by 14.0%. The selection therefore has to account for the execution objective communicated in the harness description, not just aggregate efficiency.

The paper also reports that aggregate efficiency and P95 latency occasionally favor different mappings, which reinforces the same point: the right configuration depends on which objective the workflow declares. On DeepResearchBench, chain reuse of 49.8% reduces makespan from 5.93 to 2.42 hours, a 2.45x speedup, while preserving RACE at 41.1% versus 40.6%.

HEAR adds control-plane work, not model computation

HEAR adds control-plane metadata processing rather than model-side computation. It introduces no additional model layers or per-token GPU operators. Its online work consists of state updates, object lookups, policy-input construction, control handling, and execution-outcome reporting. The computation introduced by OmniKV, H2O, or SnapKV belongs to the corresponding execution mode rather than to the protocol itself.

The cost accounting is explicit. For cache-aware coordination, engine KV events update a harness-maintained residency directory; at each scheduling point the harness queries reusable prefix blocks, constructs scheduling preferences, and attaches the applicable waiting requirement; the engine applies these controls and reports realized execution order, cache reuse, retries, and errors. For capability-aware configuration, candidate profiling, filtering, ranking, and verification occur offline during development, while the online path loads the frozen role-to-mode mapping once at workflow initialization and performs a constant-size lookup per model request. All formal HEAR-enabled runs include the corresponding online control-plane path, so the reported end-to-end results reflect net performance after these operations.

Where the evidence stops

The results come from four conversational and research-agent benchmarks under memory-constrained, concurrent serving, with specific hardware and models: SCBench uses Qwen3-8B on one RTX PRO 6000 with a 96-GiB L2 KV cache, Mooncake Bench uses Qwen3-8B on one H100 with a 96-GiB L2 KV cache, and the role-specific experiments use GLM-4.7-Flash with Main-side and Reader/sub-agent inference on separate H100 GPUs. SCBench lacks real request timestamps, so the authors construct a closed-loop multi-turn workload with Poisson session arrivals and sampled inter-turn think times; Mooncake replays 50%, 75%, and 100% of sessions from a production trace.

Two boundaries matter for interpretation. First, the residency view the harness maintains reflects the cache state currently observed and does not constitute a residency guarantee; when a request actually executes, the engine rechecks reusable content against the current cache state. Second, the paper states that its method improves inference efficiency but does not address the safety or reliability of the underlying model outputs, and that practical deployments exchanging execution and resource metadata should enforce access control, tenant isolation, and bounded waiting. The reported speedups are measured on these benchmarks and configurations, not extrapolated to other serving stacks.

Match the policy to the load and the role

The paper's most decision-relevant message is that the coordination policy, not the protocol, is what must change with conditions. On SCBench, waiting protection is what converts cache-aware scheduling from a tail-latency problem into a 1.61x batch speedup with better P50, P95, and maximum TTFT; the 40-second threshold protects tails while the 60-second threshold favors reuse and batch completion. On Mooncake, the winning policy shifts by load: Session-Aware at 50%, the combination at 75%, and Cache-Aware for P50, session latency, and throughput at 100%, with Session-Aware still better for tail latency and reuse.

For role configuration, the paper's own comparison sets the priority order. Reader/sub-agent mode selection delivers the largest consistent gain, 5.60x on BrowseComp-Plus and 7.22x on DeepResearchBench, while Main-side choice depends on output length: vLLM for the roughly 0.4K-token BrowseComp-Plus Main requests and OmniKV for the roughly 4.4K-token DeepResearchBench Main requests. The concrete threshold to carry away is the cost of ignoring this: a universal mapping adds 7.8% to BrowseComp-Plus wall time or 14.0% to DeepResearchBench makespan. The reusable substrate is HEAR's unchanged message categories and interaction semantics; the workload-specific policy is where the gains are actually won.

Sources used in this answer