How Instruction-Tuning Imparts Length Control: A Cross-Lingual Mechanistic Analysis

2025-09-02

AI Summary

This study investigates how instruction tuning enhances large language models’ precise control over generated text length in English and Italian. Using a cumulative weighted attribution method based on direct logit attribution, the authors quantify the contributions of individual attention heads and MLP modules across layers to length control. Results reveal that instruction tuning strengthens length regulation through deep-layer component specialization, with pronounced cross-lingual divergence: later-layer attention heads dominate in English, whereas final-layer MLPs exhibit compensatory enhancement in Italian. This work provides the first component-level mechanistic account of cross-lingual differences in length control, uncovering distinct neural pathways underlying language-specific controllability. The findings offer interpretable, neuro-symbolic insights for developing language-adaptive controllable text generation systems.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Adhering to explicit length constraints, such as generating text with a precise word count, remains a significant challenge for Large Language Models (LLMs). This study aims at investigating the differences between foundation models and their instruction-tuned counterparts, on length-controlled text generation in English and Italian. We analyze both performance and internal component contributions using Cumulative Weighted Attribution, a metric derived from Direct Logit Attribution. Our findings reveal that instruction-tuning substantially improves length control, primarily by specializing components in deeper model layers. Specifically, attention heads in later layers of IT models show increasingly positive contributions, particularly in English. In Italian, while attention contributions are more attenuated, final-layer MLPs exhibit a stronger positive role, suggesting a compensatory mechanism. These results indicate that instruction-tuning reconfigures later layers for task adherence, with component-level strategies potentially adapting to linguistic context.

Problem

• Investigating instruction-tuning's effect on length-controlled text generation
• Analyzing performance differences between foundation and instruction-tuned models
• Examining cross-lingual mechanistic differences in English and Italian models

Innovation

• Instruction-tuning specializes deeper model layers
• Attention heads in later layers improve control
• Final-layer MLPs compensate in cross-lingual contexts