V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

2026-08-07

AI Summary

This work addresses the challenges of low sample efficiency in visual reinforcement learning under high-dimensional inputs and the high cost of real-world robotic data collection. The authors propose a novel visual RL architecture that, for the first time, successfully adapts efficient design principles from state-space methods to visual continuous control tasks. Built upon Soft Actor-Critic, the approach integrates a SimBA-inspired network structure, normalization layers to stabilize training, and pointwise convolutions to reduce computational overhead. Remarkably, through architectural enhancements alone—without introducing additional algorithmic innovations—the method achieves state-of-the-art or comparable performance on the DeepMind Control Suite, Adroit, and Meta-World benchmarks, while demonstrating superior computational efficiency compared to DrQ-v2.

Abstract

Improving sample efficiency remains a core challenge in reinforcement learning (RL), especially in real-world settings like robotics, where data collection is costly. This challenge is pronounced in visual RL, where high-dimensional inputs often obscure learning signals. While prior work in visual RL has focused on algorithmic solutions, such as better dynamics models or exploration strategies, recent advances in state-based RL show that architectural design alone can lead to significant gains in sample efficiency. This raises an important question: Can these architectural principles transfer to visual RL? In response, we introduce V-Simba, a simple yet effective visual RL architecture inspired by the Simba architecture from state-based RL. Built on top of Soft Actor-Critic (SAC) with data augmentation, V-Simba modifies the architecture by adding normalization layers to stabilize training and using pointwise convolutions to reduce computation. Despite its simplicity, V-Simba matches or outperforms the state-of-the-art methods across the DMC, Adroit, and Meta-World benchmarks, while being more computationally efficient than DrQ-v2. We make our code publicly available at https://github.com/DAVIAN-Robotics/V-Simba.

Problem

• sample efficiency
• visual reinforcement learning
• high-dimensional inputs
• real-world robotics
• continuous control

Innovation

• visual reinforcement learning
• architectural design
• sample efficiency
• pointwise convolutions
• normalization layers