SuS: Strategy-aware Surprise for Intrinsic Exploration

2026-01-15

AI Summary

This work addresses the limitation of traditional curiosity-driven exploration, which relies solely on state prediction errors and overlooks policy consistency and policy-relevant novelty. The authors propose the Strategy-aware Surprise (SuS) framework, which constructs an intrinsic reward from two complementary signals—Policy Stability (SS) and Strategy-aware Surprise (SuS)—and dynamically fuses them via learnable weights to enable policy-aware exploration control. Notably, SuS introduces predictive mismatch as a novel policy-relevant novelty signal for the first time, integrating policy representation learning with intrinsic motivation mechanisms in the context of large language models for mathematical reasoning. Experimental results demonstrate that SuS improves performance by 17.4% on Pass@1 and 26.4% on Pass@5, significantly enhancing the diversity of problem-solving strategies, while ablation studies confirm the essential contribution of each component.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We propose Strategy-aware Surprise (SuS), a novel intrinsic motivation framework that uses pre-post prediction mismatch as a novelty signal for exploration in reinforcement learning. Unlike traditional curiosity-driven methods that rely solely on state prediction error, SuS introduces two complementary components: Strategy Stability (SS) and Strategy Surprise (SuS). SS measures consistency in behavioral strategy across temporal steps, while SuS captures unexpected outcomes relative to the agent's current strategy representation. Our combined reward formulation leverages both signals through learned weighting coefficients. We evaluate SuS on mathematical reasoning tasks using large language models, demonstrating significant improvements in both accuracy and solution diversity. Ablation studies confirm that removing either component results in at least 10% performance degradation, validating the synergistic nature of our approach. SuS achieves 17.4% improvement in Pass@1 and 26.4% improvement in Pass@5 compared to baseline methods, while maintaining higher strategy diversity throughout training.

Problem

• intrinsic exploration
• reinforcement learning
• curiosity-driven methods
• mathematical reasoning
• strategy diversity

Innovation

• Strategy-aware Surprise
• intrinsic motivation
• reinforcement learning
• strategy stability
• exploration