LLM-Emu: High-Fidelity LLM Serving Emulation Without the GPU Price Tag
LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
LLM-Emu is a serving-native emulation framework for vLLM that enables realistic, GPU-free evaluation of LLM serving systems. By replacing only the GPU forward path with a profile-driven latency oracle and synthetic tokens, it maintains the production HTTP stack and scheduler, achieving SOTA accuracy with less than 5.3% error in end-to-end latency.
Executive Summary
TL;DR: LLM-Emu is a runtime plugin for vLLM that allows researchers to simulate production-grade LLM inference workloads on standard CPUs. Unlike previous simulators that "mimic" scheduling, LLM-Emu is the native vLLM engine, replacing only the GPU execution with a highly accurate, profile-driven latency oracle.
Background: Within the LLM ecosystem, this work represents a shift from "offline simulation" to "native emulation." It bridges the gap between fast but inaccurate analytical models and expensive, hardware-locked physical testing.
The "Implementation Gap" in Current Simulators
Evaluating a new scheduling algorithm or admission policy usually requires a cluster of A100/H100 GPUs. Existing tools like Vidur or LLMServingSim try to solve this by simulating the engine logic. However, they face two critical issues:
- Code Drift: vLLM evolves weekly. Simulators that re-implement its logic quickly become obsolete.
- Missing Dynamics: Offline simulators ignore the "wall-clock" realities—HTTP request processing, Python GIL contention, and the actual asynchronous overlap between the CPU scheduler and GPU workers.
Methodology: The "Native Plugin" Insight
The core philosophy of LLM-Emu is: Don't simulate the engine; just fake the GPU.
By hooking into the executor boundary of vLLM, LLM-Emu allows the entire production stack (HTTP server, PagedAttention management, batching logic) to run as-is.
1. The Density-Aware Latency Oracle
Instead of complex math, LLM-Emu uses a 2D profile grid based on:
- Total Tokens: The aggregate workload of the batch.
- Concurrency: The number of active requests.
When a batch shape is encountered that hasn't been profiled, it uses a Shepard-weighted sampling algorithm to interpolate latency based on the nearest neighbors in the profile space.
Figure 1: LLM-Emu sits at the executor boundary, ensuring the rest of the stack remains untouched.
2. Timer-Resolved Futures
To maintain the "wall-clock" accuracy, LLM-Emu doesn't just return a number. It schedules a Future object that resolves only after the predicted time has passed. This forces the vLLM scheduler to wait just as it would for a real CUDA kernel, capturing genuine queuing and scheduling overheads.
Figure 2: Preserving the asynchronous overlap between the scheduler and worker.
Experiments: Crossing the "Sim-to-Real" Gap
The authors tested the system across multiple axes: hardware (RTX 8000 vs. A40), model scales (4B to 14B), and arrival patterns (Poisson vs. Bursty).
| Metric | Average Error | Peak Error |
|---|---|---|
| TPOT (Time Per Output Token) | < 1.0% | 4.75% |
| ITL (Iteration Latency) | < 1.2% | 4.69% |
| TPS (Throughput) | < 1.0% | 1.88% |
| TTFT (Time to First Token) | ~7.0% | 10.41% |
The higher error in TTFT (Time to First Token) is a known challenge—it is highly sensitive to the initial "warm-up" of CUDA graphs and the exact millisecond a request is admitted to the queue.
Critical Analysis & Future Outlook
Takeaway: LLM-Emu proves that for system-level research (like testing new load balancers or cache policies), we don't need to model every TFLOPS of a GPU. We only need to accurately model the time the GPU takes to respond.
Limitations:
- Cold-Start Profiling: You still need the target GPU for a few hours to "profile" it before you can emulate it.
- Single Node: The current implementation focuses on a single instance.
Future Work: The next frontier is cluster-level emulation. Imagine simulating a cluster of 1,000 GPUs on a single high-core-count CPU server to test global load-balancing strategies for the next generation of LLM clouds.
LLM-Emu is open-sourced, providing a much-needed tool for researchers who want to innovate on serving infrastructure without a massive hardware budget.