Mind the Privileged-to-Camera Gap: Actor-Centric Sidecar Supervision for Camera-First Open-Loop Waypoint Prediction

2026-06-18

AI Summary

This work addresses the representational gap between perception and behavior prediction in purely vision-based autonomous driving models, which arises from the absence of explicit role-level supervision for surrounding road users when trained solely with waypoint supervision. To bridge this gap, the authors propose a role-centric “sidecar” supervision mechanism that leverages simulator-generated labels—including road user positions, post-hoc relevance annotations, and short-term motion cues—to enhance the waypoint prediction capability of a model fusing multi-view RGB images, ego-vehicle states, and route commands during training. Notably, this approach requires no additional inputs at inference time and substantially narrows the performance gap between vision-only and privileged-information methods. Evaluated in open-loop trajectory prediction, the method achieves a final displacement error (FDE) of 1.223 ± 0.01 meters, a 32.6% improvement over the baseline, with particularly pronounced gains in multi-agent interactions and scenarios involving vulnerable road users.

Abstract

Camera-first autonomous-driving models predict future ego waypoints from images, ego-state features, and route commands, but waypoint supervision alone does not explicitly supervise actor-level representations of nearby road users. We study this as supervised representation learning for open-loop waypoint prediction. The deployable model uses multi-view RGB, ego state, and route command at inference. During training, simulator-derived sidecar labels supervise actor grounding, privileged hindsight actor relevance relative to the logged ego trajectory, and selected-actor short-horizon motion; these labels are never inference inputs. We evaluate route-disjoint splits with matched architecture, optimizer, validation criterion, checkpoint selection, and three seeds. A plain waypoint-only RGB baseline obtains 1.815$\pm$0.02 m final displacement error (FDE), and the matched no-teacher non-sidecar RGB control obtains 1.716$\pm$0.02 m. Road-user sidecar supervision (RU-sidecar) reduces FDE to 1.223$\pm$0.01 m, a 32.6% reduction over the plain baseline and 28.7% over the matched no-teacher non-sidecar RGB control. It improves over the plain baseline on 1445/1494 routes and over the matched no-teacher non-sidecar RGB control on 1417/1494 routes. Actor-conditioned slices show gains in all nonempty subsets, including 29.1% reduction for samples with at least four valid sidecar actors and 30.0% when a vulnerable road user is present. Optional simulator-state teacher alignment reaches 1.186$\pm$0.15 m FDE, but higher seed variability makes it secondary. Non-deployable simulator-state diagnostics remain stronger, indicating a privileged-to-camera gap. The evidence is limited to open-loop simulation diagnostics.

Problem

• waypoint prediction
• actor representation
• camera-first autonomy
• privileged-to-camera gap
• supervised representation learning

Innovation

• sidecar supervision
• camera-first autonomous driving
• actor-centric representation
• open-loop waypoint prediction
• privileged-to-camera gap