GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization

2026-05-27

AI Summary

Existing reinforcement learning approaches for general reasoning tasks suffer from limited generalization due to reliance on domain-specific verifiers, sparse rewards, and coarse-grained credit assignment. This work proposes an answer-conditioned policy gradient framework that enables fine-grained, token-level credit assignment without requiring domain-specific verifiers. By evaluating reasoning trajectories through the likelihood of generating the ground-truth answer, the method produces token-level credit signals conditioned on the correct answer. It integrates likelihood-guided optimization, token-level compatibility signals, and a direction-preserving gradient modulation mechanism to ensure training stability. Evaluated across eleven benchmarks spanning mathematical, STEM, and general reasoning domains, the approach achieves state-of-the-art average performance, demonstrating its effectiveness and robustness.

Abstract

Reinforcement learning with verifiable rewards improves language model reasoning, but its reliance on domain-specific verifiers, sparse outcome rewards, and coarse-grained credit assignment limits its applicability. We introduce GeneralThinker, an on-policy framework that reformulates reasoning supervision as dense answer-conditioned optimization, enabling response-level evaluation and token-level credit assignment without domain-specific verifiers. GeneralThinker evaluates generated reasoning trajectories using the likelihood of the ground-truth answer and derives token-wise compatibility signals for fine-grained credit assignment. To stabilize optimization, it constrains token-level updates through clipping and direction-preserving modulation. Across 11 benchmarks spanning mathematics, STEM, and general reasoning, GeneralThinker achieves the best average performance. Further analyses show that uncontrolled token-level modulation can destabilize training, whereas controlled modulation makes fine-grained credit assignment consistently effective.

Problem

• reinforcement learning
• domain-specific verifiers
• sparse rewards
• credit assignment
• reasoning

Innovation

• answer-conditioned optimization
• token-level credit assignment
• likelihood-guided reasoning
• domain-general reinforcement learning
• direction-preserving modulation