Cube: A Roblox View of 3D Intelligence

2025-03-19

AI Summary

This work addresses the lack of foundational models for 3D intelligence in the Roblox ecosystem. To tackle the challenge of modeling complex 3D geometric structures, we propose the first geometry-aware 3D shape tokenizer, enabling cross-modal semantic alignment among text, shape, and scene representations. We establish three design principles for 3D foundation models and introduce an LLM-3D collaborative reasoning framework alongside a unified text-to-shape/scene joint generation architecture. Our approach unifies support for 3D object/scene generation, character rigging, and behavioral script synthesis. Experiments demonstrate significant improvements in fidelity across text-to-shape, shape-to-text, and text-to-scene generation tasks. Moreover, the model supports cross-modal understanding and logical reasoning over 3D content. By providing a scalable, general-purpose intelligent foundation, this work advances programmable 3D content creation within Roblox and beyond.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Foundation models trained on vast amounts of data have demonstrated remarkable reasoning and generation capabilities in the domains of text, images, audio and video. Our goal at Roblox is to build such a foundation model for 3D intelligence, a model that can support developers in producing all aspects of a Roblox experience, from generating 3D objects and scenes to rigging characters for animation to producing programmatic scripts describing object behaviors. We discuss three key design requirements for such a 3D foundation model and then present our first step towards building such a model. We expect that 3D geometric shapes will be a core data type and describe our solution for 3D shape tokenizer. We show how our tokenization scheme can be used in applications for text-to-shape generation, shape-to-text generation and text-to-scene generation. We demonstrate how these applications can collaborate with existing large language models (LLMs) to perform scene analysis and reasoning. We conclude with a discussion outlining our path to building a fully unified foundation model for 3D intelligence.

Problem

• Develop a foundation model for 3D intelligence in Roblox.
• Support developers in generating 3D objects, scenes, and animations.
• Integrate 3D shape tokenization with text-to-shape and scene generation.

Innovation

• Develops 3D foundation model for Roblox
• Introduces 3D shape tokenizer solution
• Integrates with LLMs for scene analysis