Problem
Research questions and friction points this paper is trying to address.
Cross-model latent guidance
Long-form generation
Static guidance
MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation
🤖 AI Summary
为解决长文本生成中固定指导信号失效问题，提出MentorPulse方法，通过动态更新导师模型状态来保持学生模型生成质量。
📝 Abstract
Cross-model latent guidance lets a frozen large mentor encode an input once and a frozen small student generate from the resulting signal. Existing methods keep this signal fixed, assuming it stays useful as the output grows; we show this fails in long-form generation. On multi-turn instruction following, static guidance pushes a 4B student's constraint satisfaction 2.5 points below its no-guidance baseline; a training-free refresh every 16 tokens changes only the memory content and restores a 2.0-point gain over that baseline. We propose MentorPulse to keep guidance fresh at practical cost: it compresses mentor states into a capped slot memory, incrementally processes newly generated tokens, and updates the memory that the student reads through gated cross-attention without resetting the student's KV cache. Windowed Refresh Training exposes the bridge to prefix-conditioned memory. Across thirteen datasets, MentorPulse closes 52.2% of the mentor-student gap on macro average, outperforming C2C, T2T, and equal-budget LoRA, with the largest gains on long outputs. It performs best on all eleven mentor-student pairs from three model families, with margins that narrow as the capability gap grows, and a lightweight read-pattern check predicts the gain before deployment. Measured costs identify refresh intervals that dominate text guidance on long outputs.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Cross-Model Latent Guidance
MentorPulse
Windowed Refresh Training
Gated Cross-Attention
🔎 Similar Papers
No similar papers found.
