Towards Effective Theory of LLMs: A Representation Learning Approach

2026-05-09

AI Summary

This work addresses the challenge of distilling interpretable, predictive, and intervenable high-level descriptions from the complex microscopic computations of large language models (LLMs). It introduces Representation Effective Theory (RET), which adapts the concept of effective theories from physics to LLM analysis for the first time. By leveraging BYOL/JEPA-style self-supervised learning, RET extracts temporally consistent macroscopic variables from hidden-state trajectories. These variables effectively capture high-level semantics, reveal underlying reasoning processes, and enable early prediction of behavioral tendencies—such as sycophancy—while supporting controllable causal interventions on generation dynamics. This approach facilitates coarse-grained modeling and interpretable analysis of LLM “mental trajectories,” offering a principled framework for understanding and steering model behavior.

Abstract

We propose Representational Effective Theory (RET), a framework for describing large language model computation in terms of learned macrostates rather than microscopic details. RET learns these macrostates from hidden-state trajectories using a BYOL/JEPA-style self-supervised objective, coarse-graining activations into macrovariables that preserve higher-level structure relevant for prediction and interpretation. We evaluate whether these macrovariables are practically relevant for interpretability: RET yields temporally consistent states that reveal "mental-state" trajectories of reasoning, capture high-level semantic structure, support early prediction of behavioral outcomes such as sycophancy, and provide causal handles for steering generations toward interpretable computational phases. Together, these results suggest that LLM computation admits useful effective descriptions via RET: high-level, dynamically meaningful variables that support interpretation, prediction, and intervention.

Problem

• large language models
• representation learning
• interpretability
• effective theory
• macrostates

Innovation

• Representational Effective Theory
• macrostates
• self-supervised learning
• interpretability
• causal intervention