From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models

2026-06-18

AI Summary

How large language models (LLMs) internally represent essay quality remains unclear. This study systematically analyzes the hidden representations of multiple LLMs across diverse essay datasets and reveals, for the first time, that information about essay quality is encoded in a linearly decodable manner within these models. The authors identify “essay-scoring neurons” whose activations strongly correlate with human ratings and whose distribution dynamically shifts with essay length. Through linear probing, cross-prompt generalization, dimensionality reduction, and targeted neuron interventions—alongside comparisons with nonlinear probes—they demonstrate that quality representations gradually emerge in deeper layers, exhibit robustness to prompt variations, and partially transfer across different scoring rubrics. The marginal gains from nonlinear methods further corroborate the predominantly linear nature of this encoding.

Abstract

Recent advances in Large Language Models (LLMs) have substantially transformed Automated Essay Scoring (AES), yet the internal mechanisms underlying LLM-based scoring remain poorly understood. In this work, we systematically analyze the hidden representations of eight LLMs across two English essay datasets (ASAP++, CSEE) and one Portuguese dataset (ENEM). Using linear probing, cross-prompt generalization, dimensionality reduction, and neuron-level analyses, we find consistent evidence that essay quality information is encoded in a linearly accessible form within LLM representations. These representations emerge progressively across layers, remain robust across prompting strategies, and partially transfer across essay prompts despite differences in scoring rubrics. In addition, nonlinear probes provide only marginal and inconsistent improvements over linear probes, suggesting that most essay quality information is already linearly decodable. We further identify individual ``essay scoring neurons'' whose activations strongly correlate with essay scores and whose behavior is sensitive to targeted intervention. Moreover, the layer-wise distribution of these neurons systematically shifts with essay length, with longer essays relying more heavily on deeper layers. Overall, our findings provide evidence that LLMs encode structured representations related to essay quality and offer new insights into the interpretability of LLM-based AES systems.

Problem

• Automated Essay Scoring
• Large Language Models
• Essay Quality
• Model Interpretability
• Hidden Representations

Innovation

• linear probing
• essay scoring neurons
• cross-prompt generalization
• representation interpretability
• automated essay scoring