Self-Guided Test-Time Training for Long-Context LLMs

2026-07-10

AI Summary

This work addresses the challenge that long-context large language models struggle to focus on critical evidence when processing extended inputs, leading to degraded reasoning performance as context length increases. To mitigate this, the authors propose Self-Guided TTT, a novel test-time training approach that incorporates a self-guidance mechanism prior to parameter adaptation. This mechanism enables the model to autonomously identify high-quality evidence segments most relevant to the given query and selectively fine-tune its parameters only on these segments, thereby avoiding interference from irrelevant or noisy context. By integrating self-guided evidence selection with language modeling objectives and context-aware parameter adaptation, the method achieves up to a 15% relative accuracy improvement over baseline models Qwen3-4B and Llama-3.1-8B on the LongBench-v2 and LongBench-Pro benchmarks.

Abstract

Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.

Problem

• long-context processing
• test-time training
• evidence relevance
• model adaptation
• context utilization

Innovation

• Test-Time Training
• Long-Context Reasoning
• Evidence Selection
• Self-Guided Adaptation
• Large Language Models