ReFu: Recursive Fusion for Exemplar-Free 3D Class-Incremental Learning

2024-09-18

arXiv.org

AI Summary

This work addresses the challenging problem of exemplar-free continual learning for 3D category incremental learning. We propose the first dual-modality continual learning framework integrating point clouds and meshes. To mitigate catastrophic forgetting, we design a recursive self-correlation matrix regularization that enforces knowledge accumulation and parameter-space constraints. Additionally, we introduce a point-cloud-guided mesh attention fusion layer to model dynamic cross-modal correlations. Critically, our method requires no storage of historical exemplars and achieves multi-stage category-incremental learning via end-to-end training. Extensive experiments on mainstream 3D benchmarks—including ModelNet and ScanObjectNN—demonstrate substantial improvements over existing exemplar-free approaches: average class-incremental accuracy increases by 5.2%, with enhanced robustness and stability across incremental stages.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce a novel Recursive Fusion model, dubbed ReFu, designed to integrate point clouds and meshes for exemplar-free 3D Class-Incremental Learning, where the model learns new 3D classes while retaining knowledge of previously learned ones. Unlike existing methods that either rely on storing historical data to mitigate forgetting or focus on single data modalities, ReFu eliminates the need for exemplar storage while utilizing the complementary strengths of both point clouds and meshes. To achieve this, we introduce a recursive method which continuously accumulates knowledge by updating the regularized auto-correlation matrix. Furthermore, we propose a fusion module, featuring a Pointcloud-guided Mesh Attention Layer that learns correlations between the two modalities. This mechanism effectively integrates point cloud and mesh features, leading to more robust and stable continual learning. Experiments across various datasets demonstrate that our proposed framework outperforms existing methods in 3D class-incremental learning.

Problem

• Integrates point clouds and meshes for 3D class-incremental learning.
• Eliminates need for exemplar storage in continual learning.
• Proposes recursive fusion to enhance knowledge retention and integration.

Innovation

• Recursive Fusion integrates point clouds and meshes.
• Eliminates exemplar storage using recursive knowledge accumulation.
• Pointcloud-guided Mesh Attention Layer enhances feature integration.