Model-Free RL Agents Demonstrate System 1-Like Intentionality

2025-01-30

AI Summary

This paper investigates how model-free reinforcement learning (RL) agents—lacking explicit planning mechanisms—can spontaneously exhibit human-like “System 1” decision-making: fast, intuitive, and goal-directed. Method: We establish the first systematic mapping framework linking RL paradigms with dual-process cognitive theory, empirically analyzing behavioral patterns across canonical algorithms—including Q-learning and Actor-Critic—under controlled task environments. Contribution/Results: We demonstrate that structured reactive policies alone suffice to generate purposeful behavior, without requiring internal world models or symbolic reasoning. This challenges the long-standing assumption that intentionality necessitates explicit modeling and planning. The findings provide a novel theoretical foundation for AI accountability, safety evaluation, and intent interpretability, and advance interdisciplinary AI governance research integrating cognitive psychology, jurisprudence, and experimental methodology.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper argues that model-free reinforcement learning (RL) agents, while lacking explicit planning mechanisms, exhibit behaviours that can be analogised to System 1 ("thinking fast") processes in human cognition. Unlike model-based RL agents, which operate akin to System 2 ("thinking slow") reasoning by leveraging internal representations for planning, model-free agents react to environmental stimuli without anticipatory modelling. We propose a novel framework linking the dichotomy of System 1 and System 2 to the distinction between model-free and model-based RL. This framing challenges the prevailing assumption that intentionality and purposeful behaviour require planning, suggesting instead that intentionality can manifest in the structured, reactive behaviours of model-free agents. By drawing on interdisciplinary insights from cognitive psychology, legal theory, and experimental jurisprudence, we explore the implications of this perspective for attributing responsibility and ensuring AI safety. These insights advocate for a broader, contextually informed interpretation of intentionality in RL systems, with implications for their ethical deployment and regulation.

Problem

• Reinforcement Learning
• System 1 Thinking
• AI Safety

Innovation

• Intentionality
• Reinforcement Learning
• Model-free vs. Model-based