Structured Program Synthesis using LLMs: Results and Insights from the IPARC Challenge

2025-06-15

AI Summary

The IPARC Challenge comprises 600 anti-automated-solver synthetic image-programming synthesis tasks, designed to evaluate models’ structured programming proficiency across sequence, selection, and iteration control structures—yet no prior automated method has systematically solved all task categories. This paper proposes an LLM-driven structured inductive programming framework featuring five core mechanisms: prior structural guidance, human-in-the-loop refinement, correct-code freezing, code reuse for efficiency, and LLM-mediated human creativity elicitation. It achieves the first complete coverage of all IPARC task classes. Experiments demonstrate substantial improvements in synthesis reliability, interpretability, and reproducibility. Furthermore, the work distills a transferable human-AI collaborative programming paradigm, providing systematic empirical evidence on both the capability boundaries of LLMs in complex code generation and viable pathways for their augmentation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The IPARC Challenge, inspired by ARC, provides controlled program synthesis tasks over synthetic images to evaluate automatic program construction, focusing on sequence, selection, and iteration. This set of 600 tasks has resisted automated solutions. This paper presents a structured inductive programming approach with LLMs that successfully solves tasks across all IPARC categories. The controlled nature of IPARC reveals insights into LLM-based code generation, including the importance of prior structuring, LLMs' ability to aid structuring (requiring human refinement), the need to freeze correct code, the efficiency of code reuse, and how LLM-generated code can spark human creativity. These findings suggest valuable mechanisms for human-LLM collaboration in tackling complex program synthesis.

Problem

• Evaluating automated program synthesis on image-based tasks
• Solving 600 challenging IPARC tasks with LLMs
• Exploring human-LLM collaboration for structured code generation

Innovation

• Structured inductive programming with LLMs
• Human refinement for LLM-generated structuring
• Freezing correct code and reusing efficiently