reinforcement learning

This work addresses complex sequential decision-making for embodied agents (robots and virtual characters). It proposes a depth-first, self-contained learning framework that eliminates reliance on hand-engineered controllers. The method systematically examines core algorithms in deep reinforcement learning (DRL) and deep imitation learning (DIL), including Markov decision processes, policy gradient methods (REINFORCE), proximal policy optimization (PPO), behavioral cloning, DAgger, and generative adversarial imitation learning (GAIL), integrating essential mathematical and machine learning foundations as needed to ensure conceptual rigor over superficial surveying. The primary contribution is a logically coherent, dependency-free learning pathway tailored for beginners—designed to foster deep conceptual understanding and practical implementation proficiency in DRL/DIL. Learners acquire both theoretical insight and hands-on capability to independently conduct research and develop real-world applications.

This work addresses policy learning from reward-free offline trajectory data, aiming to achieve generalization and zero-shot transfer across environments and tasks. We propose a Joint Embedding Predictive Architecture (JEPA)-driven latent dynamics modeling framework integrated with model predictive control (MPC), contrasting model-free offline reinforcement learning. For the first time, we systematically demonstrate that this planning-centric paradigm significantly outperforms state-of-the-art model-free RL in three key dimensions: zero-shot transfer, trajectory stitching, and data efficiency. Our method achieves strong zero-shot generalization across diverse environmental layouts, enables cross-task planning with minimal data, and exhibits robustness to suboptimal trajectories. The core innovation lies in the tight integration of JEPA, latent-space dynamics modeling, and goal-conditioned MPC—enabling efficient, robust, and transferable policy learning without reward supervision.

Conventional adversarial inverse reinforcement learning (IRL) suffers from high computational cost, training instability, and reliance on expert action labels. Method: We propose a non-adversarial IRL framework that bypasses explicit reward function learning; instead, it directly optimizes the policy to match the successor features of expert trajectories. Crucially, it requires only a single unlabeled state-only demonstration trajectory—eliminating dependence on action supervision and thus overcoming a key limitation of behavior cloning (BC). Technically, we formulate a differentiable policy gradient objective based on the linear decomposition of successor features, fully compatible with standard actor-critic architectures. Results: Our method achieves significant performance gains over state-of-the-art IRL and BC baselines across diverse control tasks, marking the first IRL approach that is reward-free, action-label-free, and driven by a single state-only demonstration—while ensuring training stability and sample efficiency.

Addressing real-time, partially observable multi-robot decision-making under dynamic environments. Method: We propose a hybrid hierarchical architecture that tightly integrates model-free reinforcement learning (PPO/SAC) into the classical robotics stack via sub-behavior decomposition and heuristic scheduling, enabling end-to-end decision-making; augmented by multi-fidelity sim2real transfer (Gazebo → physical platform) and co-optimization of motion planning and state estimation modules. Contribution/Results: Our work introduces the first tightly coupled integration mechanism between RL modules and conventional robot software architectures, alongside a lightweight generalization strategy—“sub-behavior learning + heuristic selection”—that ensures millisecond-level latency while significantly improving robustness and environmental adaptability. The system secured first place in the Shield Challenge of the RoboCup Standard Platform League 2024. Real-robot evaluations demonstrate high task success rates, low end-to-end latency, and deployment stability.

To address the dual bottlenecks of low sample efficiency in reinforcement learning (RL) under sparse rewards and imitation learning’s (IL) reliance on high-quality expert policies, this paper proposes RPI—a dynamic fusion framework. Methodologically, RPI introduces (1) an online performance estimation–driven alternating IL/RL mechanism; (2) state-level adaptive modules—RAPS (Robust Active Policy Selection) and RPG (Value-Guided Policy Gradient)—enabling fine-grained, context-aware decisions on *when to imitate* versus *when to explore*; and (3) a black-box expert interface supporting arbitrary expert quality, seamlessly integrating behavior cloning, policy gradient updates, value-guided exploration, and robust policy selection. Evaluated across diverse sparse-reward benchmark tasks, RPI achieves significant improvements over state-of-the-art methods, demonstrating both superior sample efficiency and strong robustness to expert imperfection.

Current reinforcement learning (RL) post-training of large language models (LLMs) is overly focused on policy gradient methods such as PPO and GRPO, largely neglecting the broader RL algorithmic landscape. This work proposes a modular analytical framework centered on three core dimensions—MDP formulation, exploration strategies, and learning mechanisms—and systematically maps classical RL techniques—including value functions, off-policy learning, bootstrapped credit assignment, intrinsic motivation, tree search, and curriculum learning—onto the LLM training context for the first time. The study reveals a predominant reliance in existing approaches on actor-only, Monte Carlo–style policy optimization and explicitly identifies underexplored yet promising directions, thereby offering a clear roadmap for future algorithmic innovation in LLM alignment and training.

Reinforcement learning faces two key challenges in real-world applications—such as robotics, industrial automation, and healthcare—namely, the difficulty of designing reward functions and unsafe exploration. To address these, this paper proposes BRIDGE: a two-stage algorithm that first learns an initial policy offline from reward-free expert demonstrations via behavior cloning, then refines it online using human preference feedback. This is the first work to provide rigorous theoretical analysis for offline-to-online preference-based RL. Its core innovation is an uncertainty-weighted fusion mechanism that jointly leverages behavioral cloning and preference signals, enabling provably convergent regret bounds that improve with increasing offline data size. Empirical evaluation on MuJoCo discrete- and continuous-control benchmarks shows that BRIDGE significantly outperforms pure behavior cloning and standalone online preference RL, achieving superior sample efficiency while ensuring safe policy improvement.

This work addresses the challenges of offline reinforcement learning in large or continuous action spaces and the lack of theoretical guarantees for explicitly parameterized policies. By extending mirror descent to parameterized policies and establishing a connection with natural policy gradient, the proposed approach resolves contextual coupling across states and incorporates a pessimism principle to ensure stability. The study provides the first theoretical guarantees for general parameterized policies in offline RL, yielding a computationally tractable optimization framework that naturally supports continuous action spaces. Furthermore, it reveals an intrinsic unification between offline reinforcement learning and imitation learning, offering new insights into their algorithmic and theoretical connections.

This work identifies the core mechanism underlying performance degradation of synthetic data in Model-Based Policy Optimization (MBPO): in high-dimensional continuous control environments such as the DeepMind Control Suite (DMC), mismatched scaling between dynamics and reward models—coupled with suboptimal target state representation—amplifies error propagation and destabilizes model variance during policy optimization. To address this, we propose a scale-adaptive counterfactual action modeling framework with calibrated target state representation, enabling robust synthetic data utilization within MBPO. Empirical evaluation demonstrates that the improved method outperforms Soft Actor-Critic on 5 out of 7 DMC tasks, while retaining high sample efficiency on OpenAI Gym benchmarks. Moreover, it significantly enhances cross-environment generalization and model robustness, validating the efficacy of scale-aware modeling in model-based reinforcement learning.

This paper addresses the weak theoretical foundations of multi-agent imitation learning (MAIL). First, it establishes the first statistical lower bound for non-interactive MAIL, identifying the “full-policy deviation concentration coefficient” as the fundamental complexity measure. Second, it proposes MAIL-WARM, an interactive framework that integrates reward-free reinforcement learning with interactive imitation learning, using behavior cloning as a baseline to achieve near-optimal sample efficiency. Third, it improves the optimal sample complexity from 𝒪(ε⁻⁸) to 𝒪(ε⁻²), matching the derived lower bound and achieving minimax-optimal convergence. Empirical validation on benchmark environments—including grid-world domains—confirms the method’s efficacy and scalability.