BeyondSight: Object Permanence for End-to-End Autonomous Driving

2026-07-10

AI Summary

This work addresses the challenge that end-to-end autonomous driving systems struggle to perceive occluded traffic participants during prolonged visibility loss, leading to planning failures. To overcome this limitation, the study introduces object permanence modeling into end-to-end driving frameworks for the first time, decoupling an agent’s existence from its observability. By leveraging a temporally propagated agent query mechanism and an observation-conditioned evidence updating strategy, the proposed approach enables joint perception, prediction, and planning for unseen road users. The authors construct the nuScenes-Permanence dataset and establish a corresponding supervision and evaluation protocol. Experiments demonstrate that the method improves the mean average precision (mAP) for detecting invisible agents from 0 to 0.249 and reduces planning L2 error from 0.61 to 0.54, significantly enhancing reasoning capability in occluded scenarios.

Abstract

Autonomous driving operates in partially observable environments where actors may become fully occluded by other vehicles or infrastructure. Most end-to-end driving systems implicitly couple actor existence to instantaneous observations, causing actor hypotheses to degrade or disappear during prolonged occlusion and removing potentially critical agents from downstream prediction and planning. We introduce BeyondSight, a permanence-aware end-to-end driving framework that decouples actor existence from observability by maintaining persistent actor hypotheses over time. BeyondSight propagates actor queries temporally and updates them with observation-conditioned evidence, enabling joint perception, prediction, and planning to reason about actors even when they are temporarily unobservable. To enable principled training and evaluation of persistence-aware models, we further introduce nuScenes-Permanence, an extension of nuScenes that provides supervision and observability-conditioned evaluation for unobservable actors. Experiments show that BeyondSight substantially improves reasoning under occlusion, increasing detection performance for unobservable actors from 0 to 0.249 mAP while reducing planning error from 0.61 to 0.54 L2avg. These results highlight object permanence as an important modeling principle for robust end-to-end autonomous driving.

Problem

• object permanence
• autonomous driving
• occlusion
• end-to-end learning
• partially observable environments

Innovation

• object permanence
• end-to-end autonomous driving
• occlusion reasoning
• persistent actor hypotheses
• nuScenes-Permanence