How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism

2026-04-07

AI Summary

This study investigates whether instruction following in large language models relies on a universal mechanism or emerges from the dynamic coordination of modular linguistic skills. Through systematic diagnostic experiments on three instruction-tuned models across nine task categories—integrating cross-task probing, inter-task transfer tests, causal ablation, and temporal modeling of the generation process—the work demonstrates for the first time that instruction following is not governed by a single general-purpose mechanism but results from the adaptive composition of multiple specialized skills. Key findings include the superior performance of task-specific probes over generic ones, transfer efficacy constrained by underlying skill similarity, and the incremental realization of constraint satisfaction throughout the generation sequence. These results support a novel “skill composition with dynamic monitoring” framework, offering crucial empirical evidence for understanding the mechanistic basis of instruction following.

Abstract

Instruction tuning is commonly assumed to endow language models with a domain-general ability to follow instructions, yet the underlying mechanism remains poorly understood. Does instruction-following rely on a universal mechanism or compositional skill deployment? We investigate this through diagnostic probing across nine diverse tasks in three instruction-tuned models. Our analysis provides converging evidence against a universal mechanism. First, general probes trained across all tasks consistently underperform task-specific specialists, indicating limited representational sharing. Second, cross-task transfer is weak and clustered by skill similarity. Third, causal ablation reveals sparse asymmetric dependencies rather than shared representations. Tasks also stratify by complexity across layers, with structural constraints emerging early and semantic tasks emerging late. Finally, temporal analysis shows constraint satisfaction operates as dynamic monitoring during generation rather than pre-generation planning. These findings indicate that instruction-following is better characterized as skillful coordination of diverse linguistic capabilities rather than deployment of a single abstract constraint-checking process.

Problem

• instruction-following
• universal mechanism
• compositional skills
• language models
• diagnostic probing

Innovation

• instruction-following
• diagnostic probing
• compositional skills
• causal ablation
• dynamic monitoring