Instruction-Following Evaluation in Function Calling for Large Language Models

2025-09-22

AI Summary

Existing function-calling evaluation benchmarks (e.g., BFCL, tau²-Bench) assess only parameter semantic correctness, neglecting models’ adherence to syntactic formatting instructions—such as quoted strings or ISO date formats. Method: We introduce the first benchmark explicitly targeting *format instruction adherence*, comprising 750 test cases with explicit, diverse formatting constraints. Leveraging embedded JSON Schema validation rules, our benchmark enables fully automated, algorithmic evaluation. Contribution/Results: It is the first to systematically quantify large language models’ precision in executing embedded formatting directives within structured outputs. Experiments reveal alarmingly high error rates—30%–60%—even for state-of-the-art models (e.g., GPT-5, Claude 4.1 Opus) on basic formatting requirements, exposing a critical reliability gap for AI agents in real-world deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Function calling is a core capability of large language models, essential for AI agents. Existing benchmarks such as the Berkeley Function Calling Leaderboard (BFCL), tau^2-Bench (arXiv:2506.07982), and ACEBench (arXiv:2501.12851) evaluate argument correctness but do not test adherence to format instructions embedded in parameter descriptions, such as enclosing values in double quotes or using ISO date formats. We introduce IFEval-FC, a benchmark inspired by IFEval (arXiv:2311.07911) that assesses precise instruction following in function calling. IFEval-FC encodes verifiable formats directly within JSON schema descriptions, for example specifying that a value must not contain punctuation. It includes 750 test cases, each consisting of a function with an embedded format for one of its input parameters and a corresponding user query. Evaluation is fully algorithmic, ensuring objectivity, reproducibility, and scalability. Our results show that even state-of-the-art proprietary models, including GPT-5 and Claude 4.1 Opus, frequently fail to follow basic formatting rules, highlighting a practical limitation for real-world agent systems. The complete codebase and data are publicly available at https://github.com/Skripkon/IFEval-FC.

Problem

• Evaluating precise instruction following in function calling for LLMs
• Assessing adherence to format rules embedded in parameter descriptions
• Testing compliance with formatting requirements like quotes and date formats

Innovation

• Benchmark tests format instruction adherence
• Encodes verifiable formats in JSON schema
• Algorithmic evaluation ensures objectivity and scalability