EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

2026-05-14

AI Summary

This work addresses the high computational and memory costs of extending context windows in large language models, which typically require full training on long sequences. The authors propose a terminal anchoring mechanism that trains exclusively on short sequences by appending a terminal prompt with position indices near the target context length, forming a two-segment input. This design preserves semantic continuity while introducing both local and long-range relative positional relationships. Combined with RoPE interpolation, shared Transformer parameters, and theoretical analysis grounded in Bernstein’s inequality, the method effectively mitigates extrapolation instability. Evaluated on LLaMA-family models, it extends the context window from 8K to 64K, achieving a RULER score of 76.03 and outperforming LCEG, LongLoRA, and full-length fine-tuning on LongBench, all while substantially reducing training costs.

Abstract

Extending the context window of large language models typically requires training on sequences at the target length, incurring quadratic memory and computational costs that make long-context adaptation expensive and difficult to reproduce. We propose EndPrompt, a method that achieves effective context extension using only short training sequences. The core insight is that exposing a model to long-range relative positional distances does not require constructing full-length inputs: we preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length. This two-segment construction introduces both local and long-range relative distances within a short physical sequence while maintaining the semantic continuity of the training text--a property absent in chunk-based simulation approaches that split contiguous context. We provide a theoretical analysis grounded in Rotary Position Embedding and the Bernstein inequality, showing that position interpolation induces a rigorous smoothness constraint over the attention function, with shared Transformer parameters further suppressing unstable extrapolation to unobserved intermediate distances. Applied to LLaMA-family models extending the context window from 8K to 64K, EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23) while requiring substantially less computation. These results demonstrate that long-context generalization can be induced from sparse positional supervision, challenging the prevailing assumption that dense long-sequence training is necessary for reliable context-window extension. The code is available at https://github.com/clx1415926/EndPrompt.

Problem

• long-context extension
• context window
• large language models
• positional embedding
• efficient training

Innovation

• EndPrompt
• long-context extension
• positional interpolation
• Rotary Position Embedding
• efficient training