RAG over Thinking Traces Can Improve Reasoning Tasks

2026-05-05

AI Summary

Traditional retrieval-augmented generation (RAG) systems exhibit limited performance on demanding reasoning tasks such as mathematical problem solving and code generation, primarily due to their reliance on unstructured corpora. This work addresses this limitation by introducing, for the first time, the intermediate reasoning trajectories—i.e., step-by-step thought processes—from prior problem-solving instances as a novel retrieval source. The authors propose T3, an offline structuring method that transforms these trajectories into efficient, retrievable representations, thereby establishing a new RAG framework tailored for reasoning-intensive tasks. Notably, the approach incurs no additional inference overhead and even reduces computational cost by up to 15%. Evaluated on benchmarks including AIME 2025–2026, LiveCodeBench, and GPQA-Diamond, the method significantly outperforms both existing RAG and non-RAG baselines, achieving a maximum relative performance gain of 56.3%.

Abstract

Retrieval-augmented generation (RAG) has proven effective for knowledge-intensive tasks, but is widely believed to offer limited benefit for reasoning-intensive problems such as math and code generation. We challenge this assumption by showing that the limitation lies not in RAG itself, but in the choice of corpus. Instead of retrieving documents, we propose retrieving thinking traces, i.e., intermediate thinking trajectories generated during problem solving attempts. We show that thinking traces are already a strong retrieval source, and further introduce T3, an offline method that transforms them into structured, retrieval-friendly representations, to improve usability. Using these traces as a corpus, a simple retrieve-then-generate pipeline consistently improves reasoning performance across strong models and benchmarks such as AIME 2025--2026, LiveCodeBench, and GPQA-Diamond, outperforming both non-RAG baselines and retrieval over standard web corpora. For instance, on AIME, RAG with traces generated by Gemini-2-thinking achieves relative gains of +56.3%, +8.6%, and +7.6% for Gemini-2.5-Flash, GPT-OSS-120B, and GPT-5, respectively, even though these are more recent models. Interestingly, RAG on T3 also incurs little or no extra inference cost, and can even reduce inference cost by up to $15%$. Overall, our results suggest that thinking traces are an effective retrieval corpus for reasoning tasks, and transforming them into structured, compact, or diagnostic representations unlocks even stronger gains. Code available at https://github.com/Narabzad/t3.

Problem

• retrieval-augmented generation
• reasoning tasks
• thinking traces
• math reasoning
• code generation

Innovation

• thinking traces
• retrieval-augmented generation
• reasoning tasks
• structured representation
• T3