Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck

2026-05-08

AI Summary

Multimodal agents often exhibit inconsistent behaviors due to trajectory noise and cross-modal redundancy, hindering the acquisition of reliable and reusable skills. This work proposes the Conditional Multimodal Information Bottleneck (CMIB), which for the first time decomposes the information bottleneck principle into a two-stage sequential process: first extracting interpretable textual skill cards, then conditionally preserving perceptual residuals that go beyond textual descriptions. The framework explicitly models, via variational inference, the conditional dependence of multimodal compression on textual skills, thereby enabling disentangled control over interpretability and perceptual fidelity. Experimental results demonstrate that the resulting structured multimodal skills substantially enhance execution stability in agents while circumventing the computationally expensive multi-sample inference required by conventional self-consistency approaches.

Abstract

While LLM-based agents excel at planning and executing long action sequences, their execution often remains inconsistent across trials, limiting reliability. Consolidating agent consistency requires distilling trial-error trajectories into reusable skills that preserve task-relevant invariants while discarding trajectory-specific noise. However, in multimodal settings, the key challenge is not only that useful invariants are distributed across vision and language information, but that different modalities support different kinds of reusable skill content: while some skills are verbalizable and interpretable, others reside in perceptual evidence beyond text. Text-only skills may lose perceptual cues, whereas storing text and perception naively introduces redundancy and noise. Existing inference-time methods, such as self-consistency, improve reliability through costly multi-sample decoding, while internalization strategies lack a way to separate verbalizable skill content from residual perceptual information. To address this, we introduce Conditional Multimodal Information Bottleneck (CMIB), a method for multimodal skill construction. CMIB begins with a joint bottleneck over multimodal skills and derives an exact sequential decomposition: (1) a text-stage bottleneck distilling interpretable skill cards, and (2) a conditional multimodal bottleneck compressing only residual information in perception that remains predictive beyond text. Unlike naive two-stream formulations, CMIB explicitly conditions the multimodal latent on the text skill, thus structurally reducing cross-modal redundancy and enabling independent control over textual and perceptual compression. We instantiate CMIB with a variational objective that makes its conditional decomposition tractable to optimize, yielding reusable multimodal skills that improve execution stability without incurring multi-sample inference overhead.

Problem

• multimodal agent
• skill consistency
• information bottleneck
• perceptual redundancy
• verbalizable skills

Innovation

• Conditional Multimodal Information Bottleneck
• multimodal skill learning
• information bottleneck
• perceptual-textual decomposition
• agent consistency