Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs

2026-09-03

AI Summary

本文通过提出一个两阶段强化学习框架TCS来自动生成有效测试用例，以解决代码生成中高质量测试用例稀缺的问题。

Abstract

Reinforcement learning (RL) has substantially advanced code generation with large language models (LLMs) through executable feedback. The feedback for coding problems mainly comes from specific test cases, where high-quality test cases are often scarce since they should be both sound and discriminative. We thus turn to study the auto-generation of test cases using the learned model. We find this is naturally an adversarial RL problem: the model is expected to generate effective test cases as counterexamples, depending on the solver's current failure modes. We propose Test Cases Scaling (TCS), a two-stage RL framework for effective test generation. Both stages train a test generator from a rolling policy-aligned buffer: Stage 1 generates tests consistent with the reference solution, and Stage 2 restricts the buffer to current failure modes and learns counterexample tests. Across TACO and LiveCodeBench, TCS improves both pass@1 and inference-time answer selection according to generated tests. We find the learned test generator also enables effective selection among other LLM outputs.

Problem

• reinforcement learning
• code generation
• test cases
• adversarial

Innovation

• Two-Stage Reinforcement Learning
• Test Cases Scaling (TCS)
• Adversarial Test Generation
• Code LLMs