State Space Model Meets Transformer: A New Paradigm for 3D Object Detection

2025-03-18

AI Summary

Existing DETR-based 3D indoor detection methods suffer from static scene point features in the Transformer decoder, limiting the contribution of deeper decoder layers. To address this, we propose DEST—a novel paradigm that models detection queries as dynamic states in a state-space model (SSM), with point cloud features serving as system inputs, enabling joint, linear-complexity iterative updates of queries and scene features. Key innovations include state-dependent SSM parameterization, bidirectional sequence scanning, cross-state attention, and a gated feed-forward network. On ScanNet V2 and SUN RGB-D, DEST achieves +5.3 and +3.2 AP50 gains over the GroupFree baseline, respectively, and establishes new state-of-the-art performance on both benchmarks.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

DETR-based methods, which use multi-layer transformer decoders to refine object queries iteratively, have shown promising performance in 3D indoor object detection. However, the scene point features in the transformer decoder remain fixed, leading to minimal contributions from later decoder layers, thereby limiting performance improvement. Recently, State Space Models (SSM) have shown efficient context modeling ability with linear complexity through iterative interactions between system states and inputs. Inspired by SSMs, we propose a new 3D object DEtection paradigm with an interactive STate space model (DEST). In the interactive SSM, we design a novel state-dependent SSM parameterization method that enables system states to effectively serve as queries in 3D indoor detection tasks. In addition, we introduce four key designs tailored to the characteristics of point cloud and SSM: The serialization and bidirectional scanning strategies enable bidirectional feature interaction among scene points within the SSM. The inter-state attention mechanism models the relationships between state points, while the gated feed-forward network enhances inter-channel correlations. To the best of our knowledge, this is the first method to model queries as system states and scene points as system inputs, which can simultaneously update scene point features and query features with linear complexity. Extensive experiments on two challenging datasets demonstrate the effectiveness of our DEST-based method. Our method improves the GroupFree baseline in terms of AP50 on ScanNet V2 (+5.3) and SUN RGB-D (+3.2) datasets. Based on the VDETR baseline, Our method sets a new SOTA on the ScanNetV2 and SUN RGB-D datasets.

Problem

• Improves 3D object detection using interactive State Space Models.
• Addresses fixed scene point features in transformer decoders.
• Enhances feature interaction with linear complexity in SSM.

Innovation

• Interactive State Space Model for 3D detection
• State-dependent SSM parameterization for queries
• Bidirectional feature interaction in point clouds