Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training

2026-06-30

AI Summary

Current test-time training (TTT) approaches for large language models predominantly rely on proxy metrics such as perplexity, which inadequately capture the models’ true capabilities in deployment scenarios—particularly regarding memory retention, personalization, or sparse learning. This work proposes the first calibration-based behavioral evaluation framework explicitly designed to validate deployment-oriented memory claims. By integrating an evidence hierarchy and standardized protocols, the framework aligns model memory assertions with observable behaviors and clearly distinguishes between streaming/domain adaptation, bridging internalization, and deployment behavior learning. Employing explicit memory baselines, failure taxonomies, single-step LoRA updates, controlled nonce facts, multi-scale Qwen3 models, and free-recall generation experiments under sparse-fact settings, the study reveals a critical disconnect: despite improvements in support and answer loss, free-recall accuracy remains at zero, empirically exposing a significant gap between proxy metric gains and genuine behavioral competence.

Abstract

Large language model test-time training (TTT) is often evaluated through local proxy metrics: models are updated on recent tokens, retrieved context, target-domain data, or verifiable task attempts, and then judged by perplexity, future-token loss, long-context performance, or reward. These metrics are well matched to claims about stream adaptation, domain adaptation, context compression, and reward-backed test-time improvement. They are weaker evidence, however, for a capability that TTT results are increasingly used to motivate: deployed assistant memory, personalization, or sparse post-deployment learning, which instead requires behavioral evidence such as later recall, paraphrase robustness, retention, locality, conflict handling, and use in downstream actions after the original support context is removed. We introduce a behavioral evaluation framework that calibrates TTT memory claims to the evidence that supports them. It has two components: a claim-calibrated evidence ladder that separates stream/domain adaptation, bridge internalization, and deployment-time behavioral learning; and an evaluation protocol with matched explicit-memory baselines and mutually exclusive failure categories. We validate the framework by auditing recent TTT and memory-adjacent work and by instantiating it as a controlled diagnostic in which, in a sparse nonce-fact setting, one-step LoRA updates lower support and answer loss across three Qwen3 model scales while generated free-form recall stays at zero, exposing a measurable gap between proxy improvement and deployment behavior. The framework gives authors and evaluators a concrete standard for aligning TTT memory claims with the evidence actually reported.

Problem

• test-time training
• deployment memory
• behavioral evaluation
• memory claims
• large language models

Innovation

• test-time training
• behavioral evaluation
• deployment memory
• evidence ladder
• LoRA updates