FUSE: A Flow-based Mapping Between Shapes

2025-11-17

AI Summary

This work addresses the lack of reversible, modality-agnostic mappings among heterogeneous 3D shape representations—namely point clouds, meshes, signed distance functions (SDFs), and voxels. We propose a flow matching–based neural representation framework that learns continuous, invertible flows to map arbitrary input shapes to a unified anchor distribution, enabling bidirectional cross-representation alignment. To our knowledge, this is the first application of flow matching to 3D shape mapping, supporting zero-shot modality translation without requiring large-scale paired data. Our method integrates point-wise adaptive embeddings with composite forward–inverse flow modeling, substantially improving both matching accuracy and coverage across modalities. Extensive experiments on ShapeNet and downstream tasks—including UV parameterization and human scan registration—demonstrate state-of-the-art performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

We introduce a novel neural representation for maps between 3D shapes based on flow-matching models, which is computationally efficient and supports cross-representation shape matching without large-scale training or data-driven procedures. 3D shapes are represented as the probability distribution induced by a continuous and invertible flow mapping from a fixed anchor distribution. Given a source and a target shape, the composition of the inverse flow (source to anchor) with the forward flow (anchor to target), we continuously map points between the two surfaces. By encoding the shapes with a pointwise task-tailored embedding, this construction provides an invertible and modality-agnostic representation of maps between shapes across point clouds, meshes, signed distance fields (SDFs), and volumetric data. The resulting representation consistently achieves high coverage and accuracy across diverse benchmarks and challenging settings in shape matching. Beyond shape matching, our framework shows promising results in other tasks, including UV mapping and registration of raw point cloud scans of human bodies.

Problem

• Developing neural flow-based mapping for 3D shape correspondence
• Enabling cross-representation matching without extensive training
• Creating modality-agnostic maps between various 3D data formats

Innovation

• Flow-based neural mapping between 3D shapes
• Invertible cross-representation shape matching
• Modality-agnostic mapping across diverse formats