VMGNet: A Low Computational Complexity Robotic Grasping Network Based on VMamba with Multi-Scale Feature Fusion

2024-11-19

arXiv.org

AI Summary

To address the high computational complexity and poor real-time performance of existing deep learning–based grasping methods, this paper proposes a low-overhead, high-accuracy end-to-end grasping detection network. Our approach is the first to introduce the Vision Mamba (VMamba) architecture into robotic grasping, achieving O(N) linear computational complexity. We design a lightweight Fusion Bridge module for multi-scale feature fusion, significantly improving robustness in grasp localization and pose estimation. Additionally, we propose a subtask-adaptive weighted loss function to enhance training convergence and generalization. Experiments demonstrate that our method requires only 8.7 GFLOPs and achieves an inference latency of 8.1 ms, while attaining state-of-the-art performance on both Cornell and Jacquard datasets. In real-world multi-object scenes, it achieves a grasping success rate of 94.4%.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Machine Learning: Learning with Manifolds

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While deep learning-based robotic grasping technology has demonstrated strong adaptability, its computational complexity has also significantly increased, making it unsuitable for scenarios with high real-time requirements. Therefore, we propose a low computational complexity and high accuracy model named VMGNet for robotic grasping. For the first time, we introduce the Visual State Space into the robotic grasping field to achieve linear computational complexity, thereby greatly reducing the model's computational cost. Meanwhile, to improve the accuracy of the model, we propose an efficient and lightweight multi-scale feature fusion module, named Fusion Bridge Module, to extract and fuse information at different scales. We also present a new loss function calculation method to enhance the importance differences between subtasks, improving the model's fitting ability. Experiments show that VMGNet has only 8.7G Floating Point Operations and an inference time of 8.1 ms on our devices. VMGNet also achieved state-of-the-art performance on the Cornell and Jacquard public datasets. To validate VMGNet's effectiveness in practical applications, we conducted real grasping experiments in multi-object scenarios, and VMGNet achieved an excellent performance with a 94.4% success rate in real-world grasping tasks. The video for the real-world robotic grasping experiments is available at https://youtu.be/S-QHBtbmLc4.

Problem

• Reducing high computational complexity in robotic grasping networks
• Achieving linear computational complexity with Visual State Space
• Improving grasping accuracy through multi-scale feature fusion

Innovation

• VMamba-based network for linear computational complexity
• Multi-scale feature fusion module for accuracy
• New loss function for subtask importance differentiation