A deep reinforcement learning agent trained for interval timing exhibits similarities to biological systems

2025-08-06

AI Summary

Understanding how artificial agents acquire biologically plausible, endogenous timing mechanisms remains an open challenge in computational neuroscience and AI. Method: We trained recurrent neural network–based deep reinforcement learning (DRL) agents on video-based interval timing tasks requiring sustained temporal marking and generation, without external timing cues. Contribution/Results: We discovered that the agents’ hidden states spontaneously develop high-amplitude, periodic neural oscillations that directly govern behavioral decisions—mirroring striatal rhythmic dynamics observed in biological timing systems. Critically, these oscillations support robust, cue-invariant timekeeping: agents maintain accurate timing performance under blank or visually perturbed inputs, confirming acquisition of a generalizable, internally generated timing mechanism. This work provides the first systematic evidence of biologically inspired, oscillatory temporal coding in DRL agents, offering an interpretable computational model that bridges artificial and neurobiological timing research.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Drawing parallels between Deep Artificial Neural Networks (DNNs) and biological systems can aid in understanding complex biological mechanisms that are difficult to disentangle. Temporal processing, an extensively researched topic, is one such example that lacks a coherent understanding of its underlying mechanisms. In this study, we investigate temporal processing in a Deep Reinforcement Learning (DRL) agent performing an interval timing task and explore potential biological counterparts to its emergent behavior. The agent was successfully trained to perform a duration production task, which involved marking successive occurrences of a target interval while viewing a video sequence. Analysis of the agent's internal states revealed oscillatory neural activations, a ubiquitous pattern in biological systems. Interestingly, the agent's actions were predominantly influenced by neurons exhibiting these oscillations with high amplitudes. Parallels are drawn between the agent's time-keeping strategy and the Striatal Beat Frequency (SBF) model, a biologically plausible model of interval timing. Furthermore, the agent maintained its oscillatory representations and task performance when tested on different video sequences (including a blank video). Thus, once learned, the agent internalized its time-keeping mechanism and showed minimal reliance on its environment to perform the timing task. A hypothesis about the resemblance between this emergent behavior and certain aspects of the evolution of biological processes like circadian rhythms, has been discussed. This study aims to contribute to recent research efforts of utilizing DNNs to understand biological systems, with a particular emphasis on temporal processing.

Problem

• Investigating temporal processing mechanisms in deep reinforcement learning agents
• Exploring parallels between artificial neural networks and biological time-keeping systems
• Understanding emergent oscillatory patterns for interval timing tasks in AI agents

Innovation

• Deep reinforcement learning agent for interval timing
• Oscillatory neural activations resembling biological systems
• Internalized time-keeping mechanism independent of environment