ReConText3D: Replay-based Continual Text-to-3D Generation

2026-04-15

AI Summary

This work addresses the challenge of catastrophic forgetting in text-to-3D generation models under continual learning scenarios and proposes the first continual learning framework tailored for this task. Without modifying the underlying model architecture, the method employs a semantic-aware memory replay strategy—specifically, a text-embedding-based k-Center selection algorithm—to efficiently preserve knowledge of previously learned categories. The contributions include establishing a continual learning paradigm for text-to-3D generation, introducing a compact yet diverse memory replay mechanism, and releasing Toys4K-CL, the first benchmark dataset for this setting. Experimental results demonstrate that the proposed framework achieves high-quality 3D generation on both old and new categories, significantly outperforming existing baseline approaches.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: Generation
• Computer Vision: 3D Computer Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Continual learning enables models to acquire new knowledge over time while retaining previously learned capabilities. However, its application to text-to-3D generation remains unexplored. We present ReConText3D, the first framework for continual text-to-3D generation. We first demonstrate that existing text-to-3D models suffer from catastrophic forgetting under incremental training. ReConText3D enables generative models to incrementally learn new 3D categories from textual descriptions while preserving the ability to synthesize previously seen assets. Our method constructs a compact and diverse replay memory through text-embedding k-Center selection, allowing representative rehearsal of prior knowledge without modifying the underlying architecture. To systematically evaluate continual text-to-3D learning, we introduce Toys4K-CL, a benchmark derived from the Toys4K dataset that provides balanced and semantically diverse class-incremental splits. Extensive experiments on the Toys4K-CL benchmark show that ReConText3D consistently outperforms all baselines across different generative backbones, maintaining high-quality generation for both old and new classes. To the best of our knowledge, this work establishes the first continual learning framework and benchmark for text-to-3D generation, opening a new direction for incremental 3D generative modeling. Project page is available at: https://mauk95.github.io/ReConText3D/.

Problem

• continual learning
• text-to-3D generation
• catastrophic forgetting
• 3D generative modeling
• class-incremental learning

Innovation

• continual learning
• text-to-3D generation
• replay memory
• catastrophic forgetting
• class-incremental learning