multimodal learning

Existing research lacks a systematic analysis of modality–language backbone integration mechanisms in multimodal large language models (MLLMs). Method: We systematically survey 125 MLLMs published between 2021 and 2025, proposing the first large-language-model-centric three-dimensional taxonomy—architectural integration, representation learning, and training paradigms—to unify cross-modal alignment pathways. Leveraging bibliometric analysis, architectural decoupling, and fine-grained modeling of embeddings and loss functions, we characterize evolutionary patterns in fusion granularity, joint representation design, and objective function development. Contribution/Results: This work fills a critical theoretical gap in structured analysis of MLLM fusion mechanisms, identifies shared bottlenecks—including misaligned modality-specific feature hierarchies and suboptimal cross-modal optimization objectives—and delivers a reusable theoretical framework and practical guidelines for developing robust, scalable multimodal foundation models.

Multimodal learning faces critical challenges including difficulty in cross-source information fusion, poor robustness to modality missing, and vulnerability to adversarial attacks. To address these, we propose a robust multimodal representation learning framework. Methodologically, we design a contrastive learning–based cross-modal alignment mechanism with cross-attention, enabling unsupervised and self-supervised fusion; integrate AutoML-driven dynamic architecture search to enhance adaptability to incomplete inputs and adversarial perturbations; and establish a unified benchmarking framework for comprehensive evaluation. Our approach achieves significant performance gains on vision-language understanding and speech-text joint modeling tasks. Moreover, it introduces a reproducible, extensible evaluation standard system, advancing general-purpose multimodal representation paradigms. The framework demonstrates superior robustness under modality dropout and adversarial conditions while maintaining high accuracy across diverse multimodal benchmarks.

Existing two-stage cross-modal alignment methods suffer from suboptimal semantic alignment due to distributional mismatches across modalities. To address this, we propose the first single-stage, trimodal joint contrastive learning framework for end-to-end semantic alignment of audio, visual, and textual modalities. Our approach abandons the sequential alignment paradigm, instead constructing a unified representation space via cross-modal attention and multimodal embedding. A novel triplet loss is introduced to enhance contrastive learning across all three modalities simultaneously. Evaluated on the AVCaps dataset, our method achieves the first empirical validation of single-stage alignment superiority: audio-driven visual retrieval improves by 2× over two-stage baselines, and cross-modal retrieval consistently outperforms state-of-the-art two-stage models across all modalities. These results demonstrate both the effectiveness and scalability of unified multimodal representation learning.

Traditional multimodal approaches rely on modality-specific encoders and late fusion, limiting scalability and cross-modal generalization. This work proposes a unified paradigm that reformulates diverse multimodal tasks—including text, image, audio, and video processing—as “next-frame prediction,” with all inputs and outputs represented as serialized video frames, enabling end-to-end, single-model inference. It pioneers the task-redefinition strategy—previously confined to NLP—within multimodal learning, eliminating modality-specific design in favor of a fully modality-agnostic architecture. The framework integrates cross-modal tokenization, sequential frame representation, autoregressive modeling, and a shared Transformer decoder, supporting joint multi-task pretraining. Empirical evaluation demonstrates strong zero-shot and few-shot generalization across text-to-text, image-to-text, video-to-video, video-to-text, and audio-to-text tasks. Crucially, adaptation requires only lightweight task-specific heads, underscoring its flexibility and efficiency.

Multimodal contrastive learning often captures only redundant, shared information across modalities, failing to model modality-unique and synergistic interactions. To address this, we propose CoMM, a framework that abandons explicit cross-modal feature alignment and instead maximizes mutual information among augmented representations within a unified multimodal embedding space—enabling end-to-end co-modeling. For the first time, we rigorously disentangle multimodal information into redundant, unique, and synergistic components from an information-theoretic perspective, and theoretically prove that mutual information maximization inherently balances these three components. The method is fully differentiable and requires no paired multimodal supervision. Controlled ablation studies validate the accuracy of our information disentanglement, and CoMM achieves state-of-the-art performance on seven real-world multimodal benchmarks.

Existing surveys predominantly examine isolated components of multimodal pipelines and lack empirically grounded, pedagogically oriented integration frameworks for teaching and learning contexts. Method: This study introduces the first taxonomy and analytical framework covering five core modalities—natural language, video, sensor data, human-centered signals, and environmental logs—and proposes a novel “mid-fusion” paradigm for multimodal data integration. It further innovates by applying citation graph pruning to achieve structured, high-precision literature synthesis. Contribution/Results: Through systematic review, taxonomic modeling, and multimodal fusion design, we demonstrate that multimodal synergy enables detection of fine-grained learning behaviors imperceptible to unimodal analysis. While prediction accuracy remains largely unchanged, interpretability improves significantly, yielding deeper insights into learners’ cognitive-affective states and training outcomes.

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

This paper addresses the cross-modal semantic gap in multimodal understanding through a unified alignment–translation–fusion–transfer framework. Methodologically: (1) a spatial reasoning BERT is introduced to map spatial language to 2D layouts; (2) a medical term spatial co-occurrence loss is designed to ground textual descriptions in 3D anatomical locations; (3) a structured text-to-knowledge graph fact linking benchmark with interpretability is established; and (4) a multi-stream feature fusion mechanism coupled with cross-modal knowledge distillation enables lightweight RGB-based action recognition. Key contributions include: the first spatial semantic alignment model, joint anatomical-spatial representation learning, a standardized, interpretable knowledge graph linking benchmark, and a novel unimodal distillation paradigm that achieves near-fused performance without multimodal inputs. Experiments demonstrate significant improvements across all tasks: the RGB-only model attains accuracy comparable to multimodal baselines while reducing computational overhead by over 60%.

This paper addresses the challenge of jointly modeling strong pairwise alignment and higher-order (e.g., XOR-type) inter-modal dependencies in multimodal joint representation learning. To this end, we propose ConFu, a contrastive fusion framework that jointly optimizes unimodal and fused multimodal representations within a unified embedding space. ConFu introduces, for the first time, a fused-modal contrastive loss that explicitly captures higher-order interactions and enables both one-to-one bidirectional and two-to-one cross-modal retrieval. By extending the contrastive learning objective and co-optimizing multimodal fusion encoders with the joint embedding space, ConFu achieves significant improvements over state-of-the-art methods on synthetic and real-world benchmarks—including MM-IMDB and Clotho—across cross-modal retrieval and classification tasks. Moreover, the framework exhibits strong computational scalability.

This work addresses the performance degradation in multimodal classification caused by modality imbalance by proposing deep ensembling as an alternative to explicit modality fusion, achieving effective multimodal classification through the combination of unimodal networks. The key contributions include the first demonstration that superior performance can be attained without explicit fusion, a heuristic strategy for allocating the number of ensemble models based on each modality’s predictive capability, and the construction of a controllable synthetic multimodal data framework with fitted scaling laws. Experiments show that, under identical parameter budgets, the proposed method significantly outperforms state-of-the-art late-fusion and intermediate-fusion approaches on both real-world and synthetic datasets, while the derived scaling laws reveal an asymptotic upper bound on ensemble performance.

This work addresses the suboptimal performance in multimodal representation alignment caused by modality gaps and data scarcity. To this end, the authors propose a disentangled representation learning framework based on shared and modality-specific codebooks. Leveraging a compositional vector quantization mechanism, the method decomposes multimodal features into shared semantic components and modality-unique components, and employs a progressive alignment strategy to optimize the alignment space without requiring fully paired data. The unified shared codebook effectively bridges the modality gap, while the modality-specific codebooks mitigate dominant-modality bias, enabling more balanced multimodal fusion. The approach achieves state-of-the-art performance across classification and retrieval tasks spanning nine modalities, including text, images, video, and audio.