Product of Experts for Visual Generation

2025-06-10

AI Summary

This work addresses the challenge of knowledge fusion across heterogeneous visual models—including generative models, multimodal large language models, graphics engines, and physics simulators. We propose a training-free, inference-time expert ensemble framework. Our core innovation is the first training-free knowledge composition paradigm based on product distributions, enabling dynamic, cross-modal and cross-architectural expert collaboration. Coupled with annealed importance sampling (AIS), our method efficiently samples from the joint distribution of multiple experts while remaining compatible with arbitrary pre-trained visual and physical priors. The approach significantly improves generation controllability and target alignment, enabling fine-grained semantic control and interactive editing interfaces for image and video synthesis. Without any fine-tuning, it produces high-fidelity, editable outputs—overcoming fundamental flexibility and precision limitations inherent in monolithic models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Modern neural models capture rich priors and have complementary knowledge over shared data domains, e.g., images and videos. Integrating diverse knowledge from multiple sources -- including visual generative models, visual language models, and sources with human-crafted knowledge such as graphics engines and physics simulators -- remains under-explored. We propose a Product of Experts (PoE) framework that performs inference-time knowledge composition from heterogeneous models. This training-free approach samples from the product distribution across experts via Annealed Importance Sampling (AIS). Our framework shows practical benefits in image and video synthesis tasks, yielding better controllability than monolithic methods and additionally providing flexible user interfaces for specifying visual generation goals.

Problem

• Integrating diverse knowledge from multiple visual models
• Performing inference-time knowledge composition heterogeneously
• Improving controllability in image and video synthesis

Innovation

• Product of Experts for knowledge composition
• Annealed Importance Sampling for inference
• Training-free heterogeneous model integration