EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models

2025-06-10

AI Summary

Existing LLM evaluation benchmarks primarily target single-task, low-constraint scenarios, failing to reflect instruction-following capabilities in realistic, complex workflows. Method: We propose EIFBENCH, the first benchmark systematically modeling multi-task coordination and intertwined constraints characteristic of real-world workflows. It introduces multi-task compositional modeling, explicit constraint injection mechanisms, and Segmented Policy Optimization (SegPO)—a segment-wise reinforcement learning algorithm that improves task decomposition and constraint adherence. Contribution/Results: Experiments show mainstream LLMs achieve <42% average accuracy on EIFBENCH—substantially lower than their performance on conventional benchmarks—demonstrating EIFBENCH’s strong discriminative power and challenge. This establishes a new standard for evaluating complex instruction understanding and execution capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

With the development and widespread application of large language models (LLMs), the new paradigm of"Model as Product"is rapidly evolving, and demands higher capabilities to address complex user needs, often requiring precise workflow execution which involves the accurate understanding of multiple tasks. However, existing benchmarks focusing on single-task environments with limited constraints lack the complexity required to fully reflect real-world scenarios. To bridge this gap, we present the Extremely Complex Instruction Following Benchmark (EIFBENCH), meticulously crafted to facilitate a more realistic and robust evaluation of LLMs. EIFBENCH not only includes multi-task scenarios that enable comprehensive assessment across diverse task types concurrently, but also integrates a variety of constraints, replicating complex operational environments. Furthermore, we propose the Segment Policy Optimization (SegPO) algorithm to enhance the LLM's ability to accurately fulfill multi-task workflow. Evaluations on EIFBENCH have unveiled considerable performance discrepancies in existing LLMs when challenged with these extremely complex instructions. This finding underscores the necessity for ongoing optimization to navigate the intricate challenges posed by LLM applications.

Problem

• Evaluating LLMs' ability to handle extremely complex multi-task workflows
• Addressing lack of realistic benchmarks for complex instruction following
• Improving LLM performance in constrained multi-scenario operational environments

Innovation

• Multi-task scenarios for comprehensive LLM assessment
• Segment Policy Optimization for workflow accuracy
• Diverse constraints to replicate complex environments