Stackelberg Coupling of Online Representation Learning and Reinforcement Learning

2025-08-10

AI Summary

To address the inefficiency arising from tight coupling between representation learning and control policy optimization under sparse rewards, this paper proposes a Stackelberg-gaming-based co-optimization framework: the perception network acts as the leader and the control network as the follower, with equilibrium approximated via a two-timescale algorithm within a DQN architecture to enable end-to-end joint training. This work is the first to introduce Stackelberg game theory into the representation–reinforcement learning coupling paradigm—requiring neither auxiliary tasks nor explicit decoupling constraints—thereby enabling perception features to actively adapt to control objectives. Empirical evaluation across multiple sparse-reward benchmark tasks demonstrates substantial improvements in sample efficiency (+32% on average) and final performance (+18% on average), validating both the effectiveness and generalizability of structured perceptual–control dynamic modeling.

Technology Category

• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Adversarial Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Integrated, end-to-end learning of representations and policies remains a cornerstone of deep reinforcement learning (RL). However, to address the challenge of learning effective features from a sparse reward signal, recent trends have shifted towards adding complex auxiliary objectives or fully decoupling the two processes, often at the cost of increased design complexity. This work proposes an alternative to both decoupling and naive end-to-end learning, arguing that performance can be significantly improved by structuring the interaction between distinct perception and control networks with a principled, game-theoretic dynamic. We formalize this dynamic by introducing the Stackelberg Coupled Representation and Reinforcement Learning (SCORER) framework, which models the interaction between perception and control as a Stackelberg game. The perception network (leader) strategically learns features to benefit the control network (follower), whose own objective is to minimize its Bellman error. We approximate the game's equilibrium with a practical two-timescale algorithm. Applied to standard DQN variants on benchmark tasks, SCORER improves sample efficiency and final performance. Our results show that performance gains can be achieved through principled algorithmic design of the perception-control dynamic, without requiring complex auxiliary objectives or architectures.

Problem

• Integrates representation and reinforcement learning via game theory
• Improves feature learning from sparse rewards without decoupling
• Enhances sample efficiency and performance in DQN variants

Innovation

• Stackelberg game structures perception-control interaction
• Two-timescale algorithm approximates game equilibrium
• SCORER framework improves sample efficiency and performance