Continual Harness: Online Adaptation for Self-Improving Foundation Agents

2026-05-11

AI Summary

Existing approaches lack an automated framework enabling embodied agents to continuously self-improve in partially observable, long-horizon tasks. This work proposes the first online self-improvement system that operates without human intervention or environment resets, achieving co-evolution of prompts, skills, and memory through trajectory replay–driven prompt optimization, dynamic sub-agent restructuring, long-context memory, and teacher-model-guided reinforcement learning. Evaluated on Pokémon Red and Emerald, the framework substantially reduces operational overhead and consistently achieves key gameplay milestones under no-reset conditions, attaining performance comparable to handcrafted expert systems.

Abstract

Coding harnesses such as Claude Code and OpenHands wrap foundation models with tools, memory, and planning, but no equivalent exists for embodied agents' long-horizon partial-observability decision-making. We first report our Gemini Plays Pokemon (GPP) experiments. With iterative human-in-the-loop harness refinement, GPP became the first AI system to complete Pokemon Blue, Yellow Legacy on hard mode, and Crystal without a lost battle. In the hardest stages, the agent itself began iterating on its strategy through long-context memory, surfacing emergent self-improvement signals alongside human-in-the-loop refinement. Continual Harness removes the human fully from this loop: a reset-free self-improving harness for embodied agents that formalizes and automates what we observed. Starting from only a minimal environment interface, the agent alternates between acting and refining its own prompt, sub-agents, skills, and memory, drawing on any past trajectory data. Prompt-optimization methods require episode resets; Continual Harness adapts online within a single run. On Pokemon Red and Emerald across frontier models, Continual Harness starting from scratch substantially reduces button-press cost relative to the minimalist baseline and recovers a majority of the gap to a hand-engineered expert harness, with capability-dependent gains, despite starting from the same raw interface with no curated knowledge, no hand-crafted tools, and no domain scaffolding. We then close the loop with the model itself: an online process-reward co-learning loop, in which an open-source agent's rollouts through the refining harness are relabeled by a frontier teacher and used to update the model, drives sustained in-game milestone progress on Pokemon Red without resetting the environment between training iterations.

Problem

• embodied agents
• continual adaptation
• self-improvement
• partial observability
• online learning

Innovation

• Continual Harness
• online adaptation
• self-improving agents
• embodied AI
• process-reward co-learning