Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments

2026-05-20

AI Summary

This work addresses the limited semantic understanding and environmental interaction capabilities of agents in vision-language tasks by proposing a unified multimodal intelligence framework. The framework introduces three key innovations: a DETR-based mechanism for fusing grid and region visual features, a lightweight multi-input Transformer attention module (LTMI) enabling efficient visual dialogue with less than one-tenth the parameters of comparable models while maintaining performance, and a two-stage language-vision fusion decoding strategy to support embodied instruction execution. The resulting GRIT model achieves a favorable balance between accuracy and speed on image captioning and attains a state-of-the-art 8.37% success rate on unseen scenarios in the ALFRED dataset.

Abstract

Advancements at the intersection of computer vision and natural language processing are crucial for applications like assistive tech, multimedia querying, and robotics. This dissertation proposes novel architectures to improve intelligent agents across three key vision-language tasks: image captioning, visual dialog, and interactive instruction following. First, we address limitations in visual representation for image captioning. Traditional models rely on region-based features from CNN detectors, which lack global context and suffer from high computational overhead. We propose GRIT (Grid and Region-based Image captioning Transformer), a transformer-only architecture. By integrating grid and region features using a DETR-based detector, GRIT enables end-to-end training and out-performs prior methods in both inference accuracy and speed. Second, we tackle visual dialog, which requires multi-turn conversation about an image. The challenge lies in efficiently modeling interactions between multiple inputs (image, question, history). We introduce LTMI (Light-weight Transformer for Many Inputs). Utilizing a specialized attention block, an LTMI layer matches the representational power of a standard Transformer extension while utilizing less than one-tenth of its parameters, as validated on the VisDial dataset. Finally, we study interactive instruction-following for embodied AI using the ALFRED dataset. We propose a framework featuring a two-stage instruction interpretation: it first decodes language directives independently of visual context to predict a tentative action-object sequence, which is then fused with visual features for final execution. Using multiple egocentric views and hierarchical attention, our method accurately localizes objects and achieves a state-of-the-art unseen success rate of 8.37%.

Problem

• vision-language understanding
• image captioning
• visual dialog
• instruction following
• embodied AI

Innovation

• vision-language modeling
• transformer architecture
• lightweight neural networks
• embodied AI
• instruction following