Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent

2026-06-29

AI Summary

This work addresses how to achieve long-horizon agent performance comparable to trillion-parameter models without increasing model parameter count. The authors propose a novel paradigm—“extending agent horizons”—and develop a 35B sparse mixture-of-experts model that effectively integrates knowledge from six heterogeneous domains through long-horizon decision trajectory generation, a multi-teacher domain routing strategy, and a salient token alignment mechanism. The approach employs a three-stage training pipeline comprising full-domain supervised fine-tuning, domain-specific teacher training, and multi-teacher online distillation, embedded within a closed-loop system of knowledge, action, observation, and verification. Evaluated on benchmarks including SEAL-0, IFBench, and HiPhO, the method matches or surpasses the performance of trillion-parameter models such as Kimi-K2.6 and DeepSeek-V4-pro.

Abstract

We introduce Agents-A1, a 35B Mixture-of-Experts Agentic Model that reaches trillion-parameter-level performance by scaling the agent horizon. We investigate agent-horizon scaling from two perspectives: scaling long-horizon trajectories and scaling heterogeneous agent abilities. To support this goal, we build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes, producing agentic trajectories with an average length of 45K tokens. Based on this, we train Agents-A1 with a three-stage recipe. First, we perform full-domain supervised fine-tuning to align the base model with broad agentic behaviors. Second, we train domain-level teacher models to capture specialized expertise in each domain. Third, we propose a multi-teacher domain-routed on-policy distillation with salient vocabulary alignment to improve knowledge transfer efficiency across different domains, unifying six heterogeneous domains into one deployable student model. Agents-A1 achieves strong and broad performance for long-horizon agent benchmarks. Compared with 1T-parameter model such as Kimi-K2.6 and DeepSeek-V4-pro, Agents-A1 achieves leading results on SEAL-0 (56.4), IFBench (80.6), HiPhO (46.4), FrontierScience-Olympiad (79.0), and MolBench-Bind (56.8), and remains highly competitive on SciCode (44.3), HLE (47.6) and BrowseComp (75.5). We hope this work provides the community with a practical path for scaling the horizon using a 35B agent that can reach or match the performance of 1T models on long-horizon tasks.

Problem

• agent horizon
• long-horizon tasks
• parameter efficiency
• agentic model
• trillion-parameter performance

Innovation

• agent horizon scaling
• Mixture-of-Experts
• long-horizon trajectories
• multi-teacher distillation
• heterogeneous domain unification