ARC-TGI: Human-Validated Task Generators with Reasoning Chain Templates for ARC-AGI
🤖 AI Summary
Existing ARC-AGI static handcrafted puzzles are prone to overfitting, data leakage, and memorization, limiting their effectiveness in evaluating a model’s capacity for abstraction and rule induction. To address this, this work proposes an open-source task-family generator framework that produces diverse yet structurally coherent ARC tasks through compact Python programs encoding shared latent rules. The framework integrates natural language reasoning chains with partially evaluated code to ensure training samples jointly reveal rule variations while remaining human-solvable. It innovatively introduces a generation mechanism supporting task-level constraints, complemented by manual validation and local consistency checks to preserve natural coherence across task variants and their reasoning trajectories. The release includes 461 human-verified generators, covering 180 ARC-Mini, 215 ARC-AGI-1, and 66 ARC-AGI-2 tasks, enabling scalable sampling and controllable benchmarking.
Application Category
📝 Abstract
The Abstraction and Reasoning Corpus (ARC-AGI) probes few-shot abstraction and rule induction on small visual grids, but progress is difficult to measure on static collections of hand-authored puzzles due to overfitting, dataset leakage, and memorisation. We introduce ARC-TGI (ARC Task Generators Inventory), an open-source framework for task-family generators: compact Python programs that sample diverse ARC-AGI tasks while preserving a latent rule. ARC-TGI is built around a solver-facing representation: each generated task is paired with natural-language input and transformation reasoning chains and partially evaluated Python code implementing sampling, transformation, and episode construction. Crucially, ARC-TGI supports task-level constraints so that training examples collectively expose the variations needed to infer the underlying rule, a requirement for human-solvable ARC tasks that independent per-example sampling often fails to guarantee. All generators undergo human refinement and local verification to keep both grids and reasoning traces natural and consistent under variation. We release 461 generators covering 180 ARC-Mini tasks, 215 ARC-AGI-1 tasks (200 train, 15 test), and 66 ARC-AGI-2 tasks (55 train, 11 test), enabling scalable dataset sampling and controlled benchmarking.
Problem
Research questions and friction points this paper is trying to address.
ARC-AGI
overfitting
dataset leakage
rule induction
few-shot abstraction
Innovation
Methods, ideas, or system contributions that make the work stand out.
task generation
reasoning chains
rule induction
human-validated
few-shot abstraction
🔎 Similar Papers
2024-10-10Citations: 0
