CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents

2026-06-22

AI Summary

High-quality training data for terminal-based intelligent agents remains scarce, as existing synthetic methods often yield ambiguous instructions, shallow execution paths, and fragile tests that fail to provide effective learning signals. To address this, this work proposes a structured, high-fidelity task synthesis paradigm: it samples task candidates based on a multidimensional capability taxonomy, constructs them in depth through evidence-guided grounding in real technical documentation, and ensures executability and challenge via Dockerized environment instantiation, scoring-gated test generation, and strict Fail-to-Pass validation. The resulting CLI-Universe-6K dataset comprises 6,000 verifiable agent trajectories. Fine-tuning Qwen3-32B on this dataset achieves a 33.4% success rate on Terminal-Bench 2.0, establishing a new state-of-the-art among open-source models of 32B parameters or fewer and surpassing several larger models.

Abstract

While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.

Problem

• training data scarcity
• task synthesis
• terminal agents
• executable verification
• learning signals

Innovation

• task synthesis
• terminal agents
• verifiable execution
• evidence-guided grounding
• Dockerized verification