HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning

2025-10-26

AI Summary

Existing Hierarchical Reasoning Models (HRMs) are confined to static, fully observable supervised learning tasks and cannot handle dynamic, uncertain, or partially observable environments; moreover, they lack mechanisms for reusing historical computational states. Method: We propose HRM-Agent, the first framework integrating a lightweight hierarchical reasoning model (HRM) into a reinforcement learning architecture, leveraging a recurrent neural network (RNN)-based cyclic reasoning mechanism to enable cross-temporal computation reuse and continual inference. This design supports autonomous decision-making and goal-directed navigation even under action-label scarcity. Contribution/Results: Evaluated in complex maze environments, HRM-Agent successfully accomplishes long-horizon navigation tasks while significantly improving inference efficiency and environmental adaptability. Our results empirically validate the feasibility of jointly modeling hierarchical reasoning and sequential decision-making, establishing a foundation for scalable, adaptive agents in partially observable, non-stationary settings.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Multiagent Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.

Problem

• Extending HRM to dynamic uncertain environments
• Training HRM using reinforcement learning methods
• Enabling recurrent computation reuse across time-steps

Innovation

• HRM trained with reinforcement learning only
• Learns goal navigation in dynamic uncertain mazes
• Recurrent inference reuses computation across time-steps