Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

2026-06-23

AI Summary

Existing BEV perception methods are constrained by closed-set categories and suffer from 3D geometric inconsistency when lifting 2D vision-language model semantics into the BEV space. This work proposes OVBEVSeg, the first framework for open-vocabulary BEV segmentation, which introduces a three-stage geometrically constrained mechanism: pseudo-label generation via reliable 3D projection, joint 2D-BEV optimization under BEV structural constraints, and online-efficient 3D geometric distillation. Requiring no annotations for novel categories, OVBEVSeg achieves a 15.3% higher mIoU on unseen classes in nuScenes compared to closed-set baselines, matching the performance of semi-supervised methods that use 40% labeled data, while offering 2.5× faster inference and reducing memory consumption to only 22% of conventional projection-based approaches.

Abstract

Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D-BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5x faster inference with only 0.22x the memory consumption of projection-based methods. Project page: https://hchoi256.github.io/projects/ovbevseg/.

Problem

• open-vocabulary
• BEV segmentation
• 3D geometric consistency
• autonomous driving
• vision-language models

Innovation

• open-vocabulary segmentation
• bird's-eye view (BEV)
• 3D geometric consistency
• vision-language models
• Gaussian splatting