Learning Task Decomposition to Assist Humans in Competitive Programming

2024-06-07

Annual Meeting of the Association for Computational Linguistics

AI Summary

Language models (LMs) often generate complex, hard-to-understand and hard-to-repair solutions, hindering effective human–AI collaboration—especially in programming competition settings. Method: This paper proposes a solution decomposition framework designed explicitly for human-in-the-loop repair. It introduces “AssistV”, the first metric explicitly optimizing for human repair feasibility and efficiency; constructs the first large-scale dataset annotated with real human repair behaviors; and employs a three-stage paradigm—contextual prompting, critique-refinement-ranking—supervised by human expertise via fine-tuning. Results: In 177 hours of user studies, non-expert participants solved 33.3% more problems and achieved 3.3× faster solution times, matching the performance of unassisted experts. The framework significantly enhances human–AI collaborative efficacy in competitive programming.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

When using language models (LMs) to solve complex problems, humans might struggle to understand the LM-generated solutions and repair the flawed ones. To assist humans in repairing them, we propose to automatically decompose complex solutions into multiple simpler pieces that correspond to specific subtasks. We introduce a novel objective for learning task decomposition, termed assistive value (AssistV), which measures the feasibility and speed for humans to repair the decomposed solution. We collect a dataset of human repair experiences on different decomposed solutions. Utilizing the collected data as in-context examples, we then learn to critique, refine, and rank decomposed solutions to improve AssistV. We validate our method under competitive programming problems: under 177 hours of human study, our method enables non-experts to solve 33.3% more problems, speeds them up by 3.3x, and empowers them to match unassisted experts.

Problem

• Assist humans in understanding and repairing LM-generated solutions
• Automatically decompose complex solutions into simpler subtasks
• Improve human repair efficiency using assistive value (AssistV)

Innovation

• Automated decomposition of complex solutions into subtasks
• AssistV measures human repair feasibility and speed
• In-context learning to critique and refine decomposed solutions