MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding

2026-09-13

AI Summary

为解决多轮对话中代码代理遵循指令的问题，提出MTAC-IFBench基准，通过多样化约束和自动验证方法评估现有模型的不足。

Abstract

Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools iteratively to tackle complex tasks. Beyond achieving functional correctness, these agents must faithfully follow process instructions and constraints throughout the development lifecycle. However, existing benchmarks typically focus on final functional correctness or confine instruction-following evaluation to single-turn, general chat or simple code generation scenarios, leaving instruction-following in multi-turn agentic coding underexplored. To bridge this gap, we propose MTAC-IFBench, a comprehensive benchmark for this critical capability. It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories. With an average of 7.04 turns and 91.33 constraints per instance, it poses a rigorous challenge to current LLMs. To make the evaluation reliable, we construct a checklist for each constraint and functional requirement, and integrate verification scripts and judge agents to verify each checklist item. MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.

Problem

• multi-turn agentic coding
• instruction-following
• large language models
• software engineering

Innovation

• multi-turn agentic coding
• instruction-following
• benchmark
• constraint verification
• LLMs