Omnimodal Dataset Distillation via High-order Proxy Alignment

2026-04-12

AI Summary

Existing dataset distillation methods struggle to handle heterogeneous multimodal scenarios involving three or more modalities, primarily due to the high heterogeneity and complex interactions among modalities. This work proposes HoPA, the first approach to extend dataset distillation to an omnimodal setting. HoPA introduces compact high-order proxy variables to model cross-modal alignment relationships and integrates trajectory matching with spectral analysis to enable scalable joint distillation. By leveraging a shared similarity structure, the method circumvents the combinatorial explosion inherent in pairwise modeling. Extensive experiments demonstrate that HoPA significantly outperforms existing methods across multiple benchmarks, maintaining strong model performance even under high compression ratios. Both theoretical analysis and empirical results confirm its superiority over bimodal distillation frameworks.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we identify the key determinant that bounds the endpoint discrepancy in the omnimodal setting, which is exacerbated with an increasing number of modalities. To this end, we propose HoPA, a unified method that captures high-order cross-modal alignments via a compact proxy, which is compatible with trajectory matching as well. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method against bimodal dataset distillation techniques. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression-performance trade-offs compared to existing competitors. The source code will be publicly released.

Problem

• dataset distillation
• omnimodal
• cross-modal interaction
• heterogeneous modalities
• data compression

Innovation

• Omnimodal Dataset Distillation
• High-order Cross-modal Alignment
• Proxy-based Representation
• Trajectory Matching
• Spectral Analysis