COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows

2026-07-02

AI Summary

Existing workflow-based image generation agents struggle to reuse past experiences and user preferences, resulting in low efficiency and poor reliability for repetitive tasks. This work proposes a self-evolving skill mechanism that formulates workflow construction as a typed graph editing task. By integrating staged tool invocation, an automatic rollback mechanism, and a region-level vision-language model (VLM) verifier, the approach translates visual failures into actionable repair suggestions. Structured, reusable skills are continuously refined through joint distillation of execution trajectories, error logs, and VLM feedback. Implemented within the ComfyUI platform, the method achieves state-of-the-art generation scores across four benchmarks, three agent variants, and two image backbones, with human evaluators showing a significant preference for the skill-evolving version.

Abstract

Agents are increasingly used to construct workflows and assist humans in completing recurring tasks more efficiently. As these workflows become repeated and domain-specific, agent memory and reusable skills become increasingly important: agents should be able to recall workflow patterns, execution constraints, and user preferences from previous runs. We study this problem in workflow-based image generation and introduce COMFYCLAW, an agentic skill evolution harness for controlling ComfyUI workflows. COMFYCLAW formulates workflow construction as typed graph editing, exposes tools organized by construction stage, automatically reverts invalid edits, and uses a region-level vision-language model (VLM) verifier to translate visual failures into actionable repair suggestions. The framework further evolves a progressively disclosed skill library, where trajectories, execution errors, and verifier feedback from previous runs are distilled into reusable Agent Skills. Across four benchmark splits, three agent models, and two image backbones, COMFYCLAW achieves the best average image-generation evaluation score across all six agent configurations, outperforming a verifier-only baseline without skill evolution. Human annotations further show that annotators prefer COMFYCLAW over variants without skill evolution. Our results suggest that skill evolution is an effective mechanism for improving agent reliability and performance in recurring visual workflow construction.

Problem

• workflow-based image generation
• agent memory
• reusable skills
• skill evolution
• visual workflow construction

Innovation

• skill evolution
• workflow construction
• vision-language model
• graph editing
• agent memory