Distilling Implicit Multimodal Knowledge into LLMs for Zero-Resource Dialogue Generation

2024-05-16

arXiv.org

AI Summary

To address the scarcity of high-quality multimodal annotations in zero-resource dialogue generation, this paper proposes an implicit multimodal knowledge distillation paradigm that transfers knowledge from modalities such as images and audio to large language models (LLMs) without requiring target-modality annotations. Methodologically, we design an implicit distillation framework grounded in contrastive learning and gradient masking, incorporate multimodal prompts to bridge cross-modal semantic gaps, and employ lightweight adapters with frozen LLM parameters for efficient fine-tuning. Our approach is the first to eliminate explicit modality alignment and annotation dependence. Experiments on image–text and speech–text dialogue tasks demonstrate substantial improvements over strong baselines (+12.7 BLEU, +9.3 METEOR), achieving near fully supervised performance at only one-fifth the training cost. This work establishes a novel paradigm for zero-resource cross-modal dialogue generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Problem

• Enhancing LLMs for zero-resource dialogue generation
• Integrating implicit multimodal knowledge into LLMs
• Overcoming scarcity of diverse dialogue datasets

Innovation

• Visual Implicit Knowledge Distillation Framework
• Implicit Query Transformer for knowledge extraction
• Bidirectional Variational Information Fusion technique