MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents

2026-06-30

AI Summary

This work addresses key limitations in existing vision–language–action (VLA) models, particularly in temporal modeling, instruction–action alignment, and inference efficiency. The authors propose a unified framework featuring a dual-scale temporal memory hub to compress historical context, a semantic planning space constructed via mutual information–optimized implicit reasoning tokens, and a vector-level parallel action decoding mechanism. This approach substantially enhances semantic understanding and control capabilities in complex tasks, achieving state-of-the-art performance on both the LIBERO simulation benchmark and the real-world LeRobot platform. Moreover, the model demonstrates robustness through its ability to recover from unexpected execution errors.

Abstract

VLA models have emerged as a powerful paradigm for transferring semantic knowledge from web-scale data to physical robotic control. However, current single-frame architectures suffer from intrinsic limitations: temporal myopia that discards historical dynamics, reasoning gaps between high-level instructions and low-level motor commands, and inference inefficiency due to autoregressive scalar decoding. In this work, we propose MIRTH, a unified framework designed to address these challenges. MIRTH augments a pretrained VLA backbone with three key innovations: (1) dual-scale temporal memory hubs that compress long-term scene evolution and short-term motion trends into compact embeddings; (2) latent reasoning tokens optimized via a mutual-information objective carving out a semantic plan space to align multimodal context with action trajectories; and (3) a parallel action decoding scheme that replaces autoregressive generation with vector-wise prediction to maximize control throughput. Extensive evaluations on the LIBERO simulation benchmark and a real-world LeRobot platform demonstrate that MIRTH achieves state-of-the-art performance and exhibiting emergent error recovery capabilities. The codes and collected datasets are released at http://github.com/kiva12138/mirth.

Problem

• temporal myopia
• reasoning gaps
• inference inefficiency
• vision-language-action agents
• autoregressive decoding

Innovation

• temporal memory hubs
• mutual-information reasoning
• parallel action decoding
• vision-language-action agents
• semantic planning