A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models

2025-05-12

AI Summary

Existing LLM instruction-following evaluations rely heavily on templated, constraint-based prompts, lacking realism and fine-grained characterization of semantic constraints. Method: We propose the first three-dimensional evaluation framework—spanning *generation modes*, *constraint categories*, and *difficulty levels*—incorporating three generation modes, four semantic constraint types, and four difficulty tiers. We design an automated instruction synthesis pipeline featuring constraint expansion, conflict detection, and rewriting, yielding 1,200 code-verifiable test instances. Contribution/Results: Comprehensive evaluation of 19 state-of-the-art models reveals a sharp performance drop (77.67% → 32.96%) as constraint difficulty increases. We identify attention module parameter tuning as a critical optimization pathway. Reinforcement learning fine-tuning guided by our framework significantly improves instruction adherence without compromising general capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Instruction following evaluates large language models (LLMs) on their ability to generate outputs that adhere to user-defined constraints. However, existing benchmarks often rely on templated constraint prompts, which lack the diversity of real-world usage and limit fine-grained performance assessment. To fill this gap, we propose a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Building on this framework, we develop an automated instruction generation pipeline that performs constraint expansion, conflict detection, and instruction rewriting, yielding 1,200 code-verifiable instruction-following test samples. We evaluate 19 LLMs across seven model families and uncover substantial variation in performance across constraint forms. For instance, average performance drops from 77.67% at Level I to 32.96% at Level IV. Furthermore, we demonstrate the utility of our approach by using it to generate data for reinforcement learning, achieving substantial gains in instruction following without degrading general performance. In-depth analysis indicates that these gains stem primarily from modifications in the model's attention modules parameters, which enhance constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF.

Problem

• Evaluating LLMs' ability to follow diverse real-world constraints
• Addressing limitations of templated benchmarks in performance assessment
• Improving instruction following via multi-dimensional constraint framework

Innovation

• Multi-dimensional constraint framework for LLMs
• Automated instruction generation pipeline
• Reinforcement learning data for performance gains