Emergent Semantic Role Understanding in Language Models

2026-05-09

AI Summary

This study investigates whether semantic role understanding—capturing “who did what to whom”—spontaneously emerges during language model pretraining. By freezing the parameters of decoder-only Transformers and training linear probes on their internal representations, we assess the extractability of semantic role information across varying model scales. Our findings indicate that pretrained representations already encode substantial semantic role structure, with this capability strengthening and becoming more distributed as model size increases. Nevertheless, probe performance remains substantially below that of models fine-tuned on the target task, suggesting that while pretraining partially supports semantic role understanding, it is insufficient to fully replace task-specific optimization.

Abstract

Understanding how linguistic structure emerges in language models is central to interpreting what these systems learn from data and how much supervision they truly require. In particular, semantic role understanding ("who did what to whom") is a core component of meaning representation, yet it remains unclear whether it arises from pre-training alone or depends on task-specific fine-tuning. We study whether semantic role understanding emerges during language model pre-training or requires task-specific fine-tuning. We freeze decoder-only transformers and train linear probes to extract semantic roles, using performance to infer whether role information is already encoded in pre-training or learned during adaptation. Across model scales, we find that frozen representations contain substantial semantic role information, with performance improving but not fully matching fine-tuned models. This indicates partial but incomplete emergence from pre-training alone. We show that semantic role structure emerges from language modeling objectives, but its internal implementation shifts toward more distributed representations as model scale increases.

Problem

• semantic role labeling
• language models
• emergent abilities
• pre-training
• probing

Innovation

• semantic role labeling
• emergent linguistic structure
• linear probing
• language model pre-training
• distributed representations