DecompRL: Solving Harder Problems by Learning Modular Code Generation

2026-07-02

AI Summary

This work addresses the challenge that large language models struggle to solve extremely difficult programming problems due to the near-zero probability of sampling correct solutions through conventional methods or reinforcement learning (RL). To overcome this, we propose DecompRL, the first RL approach tailored for modular code generation. DecompRL decomposes complex problems into independently solvable subfunctions, learns to generate composable hierarchical code structures, and leverages verifiable reward signals for training. This framework enables exponential combinatorial exploration of candidate solutions while allowing efficient CPU-based evaluation of composed programs, drastically reducing GPU inference costs. Experiments on LiveCodeBench and CodeContests demonstrate that DecompRL substantially outperforms both standard and diversity-enhanced RL baselines, successfully solving problems intractable to prior methods while reducing GPU token consumption by approximately 50×.

Abstract

How can Large Language Models (LLMs) solve problems they currently cannot? Repeated sampling scales test-time compute but GPU cost grows linearly with attempts, while reinforcement learning (RL) with verifiable rewards improves single-attempt accuracy at the expense of sample diversity. Both strategies ultimately fail when the base policy has near-zero probability of producing a correct solution: no amount of sampling or gradient signal can overcome a search space that is simply too large. We take a different approach: rather than sampling harder, we make the task easier by decomposing problems into smaller, independently solvable sub-functions whose implementations can be recombined. Since off-the-shelf models are not trained for this modular generation, we introduce DecompRL, an RL algorithm that explicitly learns to decompose and implement hierarchical code structures. Recombining $k$ implementations of $n$ modules yields up to $k^{n}$ candidate solutions, shifting the bottleneck from GPU inference to cheap CPU evaluation and cutting GPU token cost by $\sim$50$\times$. On LiveCodeBench and CodeContests (Qwen~2.5~7B, Code World Model~32B), DecompRL outperforms standard and diversity-optimized RL baselines beyond $10^5$ tokens per problem, solving problems that standard generation cannot reach.

Problem

• Large Language Models
• Code Generation
• Problem Decomposition
• Search Space
• Reinforcement Learning

Innovation

• modular code generation
• problem decomposition
• reinforcement learning
• hierarchical code structures
• token-efficient inference