PerfGuard: A Performance-Aware Agent for Visual Content Generation

2026-01-30

AI Summary

This work addresses the unreliability of current large language model (LLM) agents in visual content generation, which often stems from their disregard for the practical performance boundaries of available tools. To overcome this limitation, we propose PerfGuard, a framework that models the multidimensional performance limits of tools and integrates this knowledge into the task planning process to enable dynamic and precise tool scheduling. PerfGuard introduces a performance-aware tool selection mechanism, an adaptive preference updating strategy, and a capability-aligned planning optimization method, supported by both theoretical analysis and empirical evaluation. Experimental results demonstrate that PerfGuard significantly outperforms existing approaches in tool selection accuracy, execution reliability, and alignment with user intent, thereby enhancing the practicality and robustness of complex AI-generated content (AIGC) tasks.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The advancement of Large Language Model (LLM)-powered agents has enabled automated task processing through reasoning and tool invocation capabilities. However, existing frameworks often operate under the idealized assumption that tool executions are invariably successful, relying solely on textual descriptions that fail to distinguish precise performance boundaries and cannot adapt to iterative tool updates. This gap introduces uncertainty in planning and execution, particularly in domains like visual content generation (AIGC), where nuanced tool performance significantly impacts outcomes. To address this, we propose PerfGuard, a performance-aware agent framework for visual content generation that systematically models tool performance boundaries and integrates them into task planning and scheduling. Our framework introduces three core mechanisms: (1) Performance-Aware Selection Modeling (PASM), which replaces generic tool descriptions with a multi-dimensional scoring system based on fine-grained performance evaluations; (2) Adaptive Preference Update (APU), which dynamically optimizes tool selection by comparing theoretical rankings with actual execution rankings; and (3) Capability-Aligned Planning Optimization (CAPO), which guides the planner to generate subtasks aligned with performance-aware strategies. Experimental comparisons against state-of-the-art methods demonstrate PerfGuard's advantages in tool selection accuracy, execution reliability, and alignment with user intent, validating its robustness and practical utility for complex AIGC tasks. The project code is available at https://github.com/FelixChan9527/PerfGuard.

Problem

• tool performance
• visual content generation
• LLM-powered agents
• performance boundaries
• AIGC

Innovation

• Performance-Aware Agent
• Tool Performance Modeling
• Adaptive Preference Update
• Visual Content Generation
• LLM-Powered Planning