To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation

2026-06-28

AI Summary

This work addresses the tendency of large language models to rely on memorized answers during reinforcement learning (RL) training when RL data overlaps with pretraining or supervised fine-tuning corpora, often generating post-hoc fabricated reasoning paths. To mitigate this, the authors propose HIPPO, a novel framework that injects prompts to deliberately elicit overlapping behaviors, thereby producing contrastive pairs of authentic and spurious reasoning trajectories. Leveraging these pairs, HIPPO introduces a prompt-anchored pairwise aggregation mechanism that uses the overlapping data itself as a supervisory signal to train a lightweight discriminator. By integrating pairwise preference learning with RL optimization, HIPPO effectively identifies and suppresses reasoning shortcuts, significantly outperforming baseline methods across multiple tasks and demonstrating strong out-of-distribution generalization, which validates the authenticity and transferability of the learned reasoning processes.

Abstract

While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.

Problem

• reasoning
• fabrication
• data overlap
• shortcuts
• reinforcement learning

Innovation

• hint-injected aggregation
• pairwise reward model
• reasoning without shortcuts
• reinforcement learning for LLMs
• out-of-distribution generalization