PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

2026-07-07

AI Summary

Existing 3D dense captioning methods are limited by insufficient spatial layout diversity and constrained semantic expressiveness. This work proposes PVCap, a novel framework comprising PseudoCap and VoxelCapNet. PseudoCap generates pseudo-frames with diverse spatial configurations through instance-level stochastic mixing and leverages a teacher–student mechanism to produce high-quality pseudo-labels. VoxelCapNet introduces an end-to-end voxel-based feature network coupled with a tailored caption head to enhance semantic modeling. Notably, PVCap is the first to incorporate instance-level spatial augmentation, substantially improving the description of spatial relationships in 3D scenes. On the ScanRefer and Nr3D benchmarks, the method achieves CIDEr@0.5IoU scores that surpass the current state of the art by 11.41% and 13.99%, respectively.

Abstract

3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene. Despite the impressive results achieved by previous methods, they suffer from two limitations. First, current research often employs global rigid transformations, such as rotation, to augment scenes without changing their spatial layouts. However, diverse spatial layouts are crucial for training a 3D dense captioning model to describe spatial relations between objects. Second, previous works mainly focus on the design of the caption generation pipeline while utilizing a simple network architecture for other components, i.e., backbone and detection head, which is crucial for extracting rich semantic information for captioning. In this paper, we propose PVCap to alleviate the aforementioned problems. Our PVCap consists of PseudoCap and VoxelCapNet. Specifically, PseudoCap employs a random mixing technique on instances within the dataset, generating numerous pseudo frames with diverse spatial layouts at the instance level. By utilizing a teacher-student framework, PseudoCap obtains pseudo caption labels for these pseudo frames. This data augmentation approach significantly increases the number of training samples and enhances the model's ability to describe the environment effectively. Regarding VoxelCapNet, we introduce a robust caption network that utilizes voxel features and adapts the caption head to the voxel-based network architecture. Our VoxelCapNet can serve as a competitive baseline for future research on 3D dense captioning. Extensive experiments are conducted on two prevalent benchmarks, i.e., ScanRefer and Nr3D. Notably, our method surpasses current state-of-the-art by 11.41% and 13.99% in CIDEr@0.5IoU, respectively. Codes will be made publicly available.

Problem

• 3D dense captioning
• spatial layout
• data augmentation
• semantic information
• voxel-based architecture

Innovation

• PseudoCap
• VoxelCapNet
• 3D dense captioning
• spatial layout augmentation
• voxel-based architecture