LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

2026-07-05

AI Summary

This work addresses the challenge in reinforcement learning for instruction following where static prompts mismatch the policy’s evolving capabilities, resulting in poorly discriminative reward signals. To resolve this, the authors propose the LLM-as-a-Tutor framework, which treats prompt adaptation as a policy-aware process. Leveraging a large language model as both examiner and tutor, the method identifies non-challenging prompts through pairwise comparisons and monotonically increases task difficulty by appending atomic constraints—ensuring training signals remain calibrated to the policy’s current proficiency. Notably, this approach requires no external scheduling mechanism and significantly outperforms policy-agnostic baselines and existing adaptive methods across three complex instruction-following benchmarks.

Abstract

Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.

Problem

• non-verifiable RL
• prompt difficulty
• policy capability
• reward signal
• instruction following

Innovation

• LLM-as-a-Tutor
• prompt adaptation
• policy-aware RL
• non-verifiable reinforcement learning
• difficulty calibration