Monocular 3D Occupancy Perception for Robots on Sidewalks via Hybrid 2D-3D Learning

2026-06-17

AI Summary

This work addresses the challenge of monocular 3D occupancy perception in sidewalk environments, where cluttered scenes and lack of structural regularity hinder existing approaches that rely on costly multi-view LiDAR-RGB paired data and dense 3D annotations. To overcome this limitation, we propose WalkOCC, a novel hybrid 2D–3D self-supervised learning framework tailored for sidewalk robots. WalkOCC leverages a small amount of LiDAR-RGB paired data to provide geometric priors while exploiting large-scale unpaired monocular images to generate pseudo 3D occupancy supervision via ray marching, jointly optimizing 2D feature representations and 3D occupancy predictions. Our method substantially reduces dependence on real 3D annotations and outperforms current self-supervised approaches in fine-grained structural understanding—such as curbs and gutters—prediction accuracy, and cross-environment generalization. We also introduce Sidewalk3D, a large-scale sidewalk dataset to support future research.

Abstract

Sidewalks in the real world are crowded, cluttered, and less structured than roads, making 3D occupancy prediction a key ingredient for the safe navigation of mobile robots such as delivery bots and electric wheelchairs. Existing occupancy learning pipelines are largely designed for on-road autonomous driving and often train on large-scale paired LiDAR-RGB datasets with dense 3D supervision and multiple camera inputs, which are costly to collect and do not adequately capture sidewalk-specific characteristics. We propose WalkOCC, a hybrid Ray-marching monocular 3D occupancy perception framework for robots operating on sidewalks. WalkOCC explicitly couples geometric grounding from LiDAR-RGB paired data with scalable learning from large-scale unpaired monocular images. It bootstraps pseudo occupancy supervision from paired sequences and jointly learns image-level representations on additional 2D-only data. It yields stable optimization and improved generalization without requiring costly 3D occupancy annotations. Extensive experiments demonstrate consistent gains in prediction accuracy, fine-grained segmentation of subtle urban structures such as curbs and gutters, and robustness to environmental and cross-embodiment shifts compared with self-supervised image-based baselines. To facilitate evaluation and benchmarking, we also introduce Sidewalk3D, a large-scale sidewalk perception dataset with LiDAR-camera paired sequences collected across multiple locations and time periods, along with 3D semantic occupancy annotations for evaluation. Code and data will be made available.

Problem

• monocular 3D occupancy
• sidewalk navigation
• mobile robots
• 3D perception
• urban scene understanding

Innovation

• monocular 3D occupancy
• hybrid 2D-3D learning
• pseudo supervision
• sidewalk perception
• cross-embodiment generalization