LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning

2026-04-16

AI Summary

This work addresses the limited reasoning capability of large language models in long-context reinforcement learning by proposing a saliency-guided sparse weight update strategy. The method innovatively incorporates the importance of high-magnitude activations—commonly used in model quantization—into reinforcement learning, identifying salient activation patterns in query and key vectors within the attention mechanism to replace conventional uniform parameter updates with focused optimization of critical weights. This approach seamlessly integrates into mainstream algorithms such as GRPO and DAPO, achieving approximately an 8% performance gain on LongBench v2 and significantly enhancing generalization on the RULER benchmark, thereby demonstrating consistent effectiveness across diverse reinforcement learning settings.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reinforcement Learning (RL) has emerged as a critical driver for enhancing the reasoning capabilities of Large Language Models (LLMs). While recent advancements have focused on reward engineering or data synthesis, few studies exploit the model's intrinsic representation characteristics to guide the training process. In this paper, we first observe the presence of high-magnitude activations within the query and key vectors when processing long contexts. Drawing inspiration from model quantization -- which establishes the criticality of such high-magnitude activations -- and the insight that long-context reasoning inherently exhibits a sparse structure, we hypothesize that these weights serve as the pivotal drivers for effective model optimization. Based on this insight, we propose LongAct, a strategy that shifts from uniform to saliency-guided sparse updates. By selectively updating only the weights associated with these significant activations, LongAct achieves an approximate 8% improvement on LongBench v2 and enhances generalization on the RULER benchmark. Furthermore, our method exhibits remarkable universality, consistently boosting performance across diverse RL algorithms such as GRPO and DAPO. Extensive ablation studies suggest that focusing on these salient features is key to unlocking long-context potential.

Problem

• Long-context Reinforcement Learning
• Intrinsic Activation Patterns
• Large Language Models
• Saliency-guided Updates
• Sparse Optimization

Innovation

• saliency-guided sparse update
• intrinsic activation patterns
• long-context reinforcement learning
• high-magnitude activations
• weight sparsification