Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesis

2026-08-11

AI Summary

This study addresses the challenge of evaluating agent continual learning in unlabeled scenarios by proposing a scaling hypothesis-based evaluation framework. The method employs a strong teacher model to sparsely correct student outputs, utilizing the degree of student-to-teacher convergence as a proxy metric for learning efficacy, thereby establishing an end-to-end evaluation paradigm that eliminates the need for annotated benchmarks. Experimental results demonstrate that the relative improvement measured by the teacher is highly correlated with gold-standard performance gains. Consequently, this work validates the effectiveness of unlabeled evaluation and provides a reliable performance measurement solution for agent continual learning when labeled data is unavailable.

Abstract

Agentic "Continual Learning Harnesses", systems that pair an LLM with retrieval or memory to improve from feedback without retraining, have shown growing value in cybersecurity. But their value is conventionally measured by gains against labeled benchmarks, an approach that often fails in operational security settings. Benchmark labels are scarce, stale, and unrepresentative, so a practitioner often cannot tell whether a given harness helps at all or which of two is better for their task. Traditional LLM-as-a-judge offers little signal because it is no stronger than the agent it evaluates, and distillation is unreliable on scarce, sporadic, and biased labels. We propose a framework for evaluating learning harnesses end-to-end without a labeled benchmark, grounded in the scaling hypothesis. A stronger teacher model provides sparsely sampled corrections to a smaller student with a continual learning harness. We score a harness by how much its student converges toward the teacher over time. Across security tasks, model families, and harness designs, we show that improvement relative to the teacher correlates with improvement relative to a held-out gold standard, validating teacher-relative lift as a proxy for true harness uplift when labels are absent. We further show that LLM-as-a-judge between similarly powered models yields no usable signal. These results suggest that a teacher-sized model can be improved through the same harness when humans provide the same kind of sparse, high-precision corrections.

Problem

• Agentic Continual Learning
• Label-free Evaluation
• Scaling Hypothesis
• Cybersecurity
• LLM-as-a-judge

Innovation

• Label-free Evaluation
• Scaling Hypothesis
• Continual Learning Harness
• Teacher-Student Convergence
• Agentic Learning