Integrating Vision Foundation Models with Reinforcement Learning for Enhanced Object Interaction

2025-08-07

AI Summary

This paper addresses the limited object interaction capability of embodied agents in simulated environments. We propose a synergistic framework integrating vision foundation models with reinforcement learning (RL). Methodologically, we first fuse Segment Anything Model (SAM) with YOLOv5 to construct a multi-granularity perception front-end, enabling precise extraction of object masks and bounding boxes; these visual features are then fed into a Proximal Policy Optimization (PPO) policy network for joint optimization of navigation and manipulation decisions in AI2-THOR kitchen scenes. Our key contribution is the end-to-end co-training of open-vocabulary visual understanding and embodied action policies. Experiments across four complex kitchen scenarios demonstrate significant improvements: +68% average cumulative reward, 52.5% object interaction success rate, and +33% navigation efficiency—substantially outperforming both pure RL and conventional detection-driven baselines.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper presents a novel approach that integrates vision foundation models with reinforcement learning to enhance object interaction capabilities in simulated environments. By combining the Segment Anything Model (SAM) and YOLOv5 with a Proximal Policy Optimization (PPO) agent operating in the AI2-THOR simulation environment, we enable the agent to perceive and interact with objects more effectively. Our comprehensive experiments, conducted across four diverse indoor kitchen settings, demonstrate significant improvements in object interaction success rates and navigation efficiency compared to a baseline agent without advanced perception. The results show a 68% increase in average cumulative reward, a 52.5% improvement in object interaction success rate, and a 33% increase in navigation efficiency. These findings highlight the potential of integrating foundation models with reinforcement learning for complex robotic tasks, paving the way for more sophisticated and capable autonomous agents.

Problem

• Enhance object interaction in simulated environments
• Integrate vision models with reinforcement learning
• Improve robotic task performance using SAM and YOLOv5

Innovation

• Integrates SAM and YOLOv5 for object perception
• Uses PPO agent in AI2-THOR simulation
• Enhances interaction and navigation efficiency