EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

2026-07-06

AI Summary

This study addresses the lack of systematic understanding of continual learning mechanisms for real-world deployed agents by introducing EdgeBench, a benchmark comprising 134 long-horizon tasks spanning domains such as scientific discovery and software engineering. Based on approximately 38,000 hours of agent–environment interaction data, the work presents an empirical analysis that reveals, for the first time, a high-precision logarithmic S-shaped scaling law governing overall agent performance in real-world settings (R² = 0.998) and identifies a generational evolution pattern wherein agent learning capacity doubles every three months. The project further proposes a multi-level feedback mechanism and a cross-domain task construction methodology, releasing 51 public tasks alongside a comprehensive evaluation framework, thereby establishing the first high-fidelity scaling law model tailored to real-world continual learning.

Abstract

Pretraining scaling laws reveal that model capability improves predictably with data and compute. But learning from real world environments after deployment remains far less understood. Analyzing roughly 38,000 hours of agent interaction with the environment across 134 real world tasks, we find, to the best of our knowledge, the first evidence that overall performance during environment learning follows a log-sigmoid scaling law with remarkably high precision, reaching R^2 = 0.998. Across model generations, we also find that agent learning speed roughly doubles every three months. This discovery stems from EdgeBench, a suite of 134 real world tasks with ultra-long horizons, spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games. Each task sustains at least 12 hours of continuous agent operation under rich, multilevel feedback, and is built through substantial expert effort. We publicly release 51 tasks and our full evaluation framework to accelerate the study of how agents learn from real world experience.

Problem

• scaling laws
• real-world environments
• agent learning
• environment interaction
• performance improvement

Innovation

• scaling laws
• real-world learning
• EdgeBench
• agent learning
• log-sigmoid