How does Odyssey benchmark long-horizon closed-loop driving with explicit routes?

Short rollouts cannot reveal later consequences

Existing closed-loop benchmarks typically evaluate only short segments, around 20 seconds, so they cannot show whether an early decision enables or blocks a maneuver that happens later. The paper's motivating example is a right turn that requires an earlier lane change: if the rollout ends before the turn, the planner's ability to complete the maneuver sequence is never tested.

Odyssey addresses this by building 100 scenarios, each reconstructed from a 100-second nuPlan driving log, so that navigation maneuvers and traffic interactions keep their surrounding context. Rollouts are bounded by three termination conditions: reaching the end of the designated SD route, a 200-second time limit equal to twice the recorded ground-truth sequence, or deviating more than 30 m from the designated SD route.

Explicit SD routes replace ambiguous commands

Many end-to-end benchmarks supply high-level meta-action commands such as "turn right." The paper argues these are ambiguous: if two right exits occur in succession, the command alone does not identify which one is intended, and inconsistent command flips obscure navigation intent along a continuous trajectory.

Odyssey instead provides explicit standard-definition (SD) map routes that specify which roads to follow, while sensor-based planning still determines local driving actions. Each route is constructed by matching the full logged ego trajectory to OpenStreetMap with HMM-based map matching, producing a single polyline from scenario start to destination; the matched roads and route continuity are manually checked against the logged trajectory and SD map. During simulation, the route segment given to the planner is updated according to the ego vehicle's current position.

Three metrics separate route adherence from lane preparation

Odyssey introduces SD Route Compliance (SDC) and Pre-Lane Change Score (PLCS) to assess how effectively planners follow routes and prepare for upcoming maneuvers. These are complemented by RouteDS, which extends the Driving Score with penalties for SD-route deviations and failed lane preparation. The three measures answer different questions: SDC concerns road-level adherence, PLCS concerns preparation before a maneuver, and RouteDS aggregates route progress and driving infractions.

Because these metrics have different definitions and denominators, they should not be substituted for one another. The paper reports cases where they diverge: DiffusionDrive achieves higher SDC than SafeDrive, but SafeDrive obtains higher RouteDS, which additionally accounts for route progress and driving infractions.

Rendering refinement changes both perception and driving

Odyssey reconstructs each scene with 3D Gaussian Splatting using OmniRe, with rigid transformations for vehicles, SMPL-based deformation for pedestrians, a learned deformation field for other actors, and time-dependent spherical-harmonic coefficients and opacity for traffic lights. Ground modeling inspired by HUGSIM represents the road surface separately using thin Gaussian disks initialized from LiDAR.

A diffusion-based refinement step, called Fixer, is fine-tuned on Odyssey using rendering–real image pairs built from all 100 scenes with three degradation strategies: under-trained reconstruction (30%), cycle reconstruction (40%), and cross-reference reconstruction (30%). The paper reports that pretrained Fixer improves mIoU but slightly decreases RouteDS, while fine-tuning on Odyssey further improves mIoU in both open-loop and closed-loop settings and raises RouteDS above the GS-only baseline. Inference optimizations removing inactive cross-attention layers and applying TensorRT yield a 3.7× speedup with negligible changes in restoration quality.

SD-route guidance improves road-level adherence

Across all evaluated planners and in both non-reactive and reactive traffic settings, providing SD routes improves both SDC and RouteDS. The paper illustrates the mechanism with SafeDrive, which deviates from the intended route when the driving command flips but follows it when given SD-route guidance.

The gains are not uniform across metrics. The paper states that explicit route guidance improves road-level adherence, but these gains do not consistently extend to lane preparation or collision avoidance. This is the boundary readers should keep in view: better route following is not the same as better lane selection or safer interaction with traffic.

Open-loop gains do not transfer to closed-loop

The paper tests whether trajectory scoring and reinforcement learning, both of which evaluate individual trajectories in an open-loop setting, improve closed-loop driving. For SafeDrive, IL scoring outperforms safety scoring despite similar PDMS values, and safety filtering further improves performance by selecting the highest IL-scored trajectory among candidates that pass safety checks. For ReCogDrive, GRPO improves PDMS but lowers closed-loop performance relative to IL-only training.

The authors interpret this as evidence that optimizing individual trajectories does not ensure better continuous driving, because neither objective directly optimizes how the planner responds to new observations after execution. Odyssey is designed to close this gap by evaluating planners on the consequences of their own decisions rather than on isolated trajectories.

Planner rankings reverse as the horizon grows

RouteDS decreases as the evaluation horizon grows, which the paper notes is partly expected because infraction penalties accumulate. More importantly, the relative performance of planners also changes: DrivoR achieves higher RouteDS than SafeDrive at short horizons but falls below it as driving continues. This rank reversal means conclusions drawn from short rollouts may not hold over longer ones.

The paper also reports that NA VSIM PDMS does not consistently reflect closed-loop differences. DrivoR achieves higher PDMS than SafeDrive but lower RouteDS in both traffic settings, so a higher open-loop score does not imply better closed-loop route following.

What Odyssey leaves open

Odyssey's contribution is a setting rather than a solved problem. The paper states that it highlights open questions in route representation and integration for end-to-end driving, and that explicit route guidance improves road-level adherence without consistently extending to lane preparation or collision avoidance.

For readers comparing planners, the practical priority follows from the paper's own comparisons: judge a planner by RouteDS and its components over long rollouts, not by SDC alone or by open-loop PDMS. The rank reversal between DrivoR and SafeDrive, and the divergence between DiffusionDrive's SDC and SafeDrive's RouteDS, are the specific cases that make horizon length and metric choice decisive. Benchmark code and adapted baselines are stated to be released publicly.

Sources used in this answer