🤖 AI Summary
This work addresses the engineering bottlenecks that hinder efficient task adaptation of small language models in production, particularly in data curation, error diagnosis, and iterative control. The authors propose the first fully automated closed-loop adaptation system for small language models: during cold-start, it requires only a natural language task description to automatically collect data, construct evaluation sets, and jointly optimize data selection, hyperparameters, and learning strategies; in production, it diagnoses failure patterns from erroneous samples, synthesizes targeted training data, and re-trains the model under explicit regression constraints. This system achieves end-to-end automation from cold-start to continuous deployment without human intervention, autonomously discovering effective strategies such as chain-of-thought supervision and quality-aware data filtering. It yields performance gains of 1.6–83.8 points across eight cold-start tasks and consistently maintains or improves performance across all seven scenarios in the new AdaptFT-Bench benchmark—where baseline methods degrade by up to 43 points—achieving 99.3% accuracy on intent classification and 0.810 F1 on entity recognition.
📝 Abstract
Small language models are attractive for production deployment due to their low cost, fast inference, and ease of specialization. However, adapting them to a specific task remains a challenging engineering loop, driven not by training itself but by surrounding decisions: data curation, failure diagnosis, regression avoidance, and iteration control. We present Pioneer Agent, a closed-loop system that automates this lifecycle. In cold-start mode, given only a natural-language task description, the agent acquires data, constructs evaluation sets, and iteratively trains models by jointly optimizing data, hyperparameters, and learning strategy. In production mode, given a deployed model with labeled failures, it diagnoses error patterns, constructs targeted training data, and retrains under explicit regression constraints. To evaluate this setting, we introduce AdaptFT-Bench, a benchmark of synthetic inference logs with progressively increasing noise, designed to test the full adaptation loop: diagnosis, curriculum synthesis, retraining, and verification. Across eight cold-start benchmarks spanning reasoning, math, code generation, summarization, and classification, Pioneer Agent improves over base models by 1.6-83.8 points. On AdaptFT-Bench, it improves or preserves performance in all seven scenarios, while naive retraining degrades by up to 43 points. On two production-style deployments built from public benchmark tasks, it raises intent classification from 84.9% to 99.3% and Entity F1 from 0.345 to 0.810. Beyond performance gains, the agent often discovers effective training strategies, including chain-of-thought supervision, task-specific optimization, and quality-focused data curation, purely from downstream feedback.