ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents

2026-03-20

AI Summary

This work addresses the fragility and verifiability challenges of automatically generated skills in multimodal web agents, which often stem from implicit representations of action semantics, state assumptions, and success criteria. To overcome this, the paper introduces a contract-based skill representation that transforms draft skills into executable contracts comprising explicit preconditions, step-wise specifications, postconditions, recovery rules, and termination checks—enabling, for the first time, verifiable, repairable, and cross-model-transferable skills. Built upon GLM-4.6V and Qwen3.5-Plus, the proposed framework supports precondition/postcondition reasoning, step-level error localization, and localized patching without requiring full regeneration. Evaluated on VisualWebArena and MiniWoB, the approach achieves up to 28.1% and 81.0% absolute gains in skill success rates, respectively, and yields up to a 47.8-point performance improvement when repaired skills are transferred across models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Computer Vision: Multi-modal Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite rapid progress in multimodal GUI agents, reusable skill acquisition remains difficult because on-demand generated skills often leave action semantics, state assumptions, and success criteria implicit. This makes them brittle to execution errors, hard to verify, and difficult to repair. We present ContractSkill, a framework that converts a draft skill into a contracted executable artifact with explicit preconditions, step specifications, postconditions, recovery rules, and termination checks. This representation enables deterministic verification, step-level fault localization, and minimal patch-based repair, turning skill refinement into localized editing rather than full regeneration. Experiments on VisualWebArena and MiniWoB with GLM-4.6V and Qwen3.5-Plus show that ContractSkill improves self-generated skills from 9.4% and 10.9% to 28.1% and 37.5% on VisualWebArena, and from 66.5% and 60.5% to 77.5% and 81.0% on MiniWoB. Repaired artifacts also transfer across models, improving the target model's self-generated-skill baseline by up to 47.8 points and 12.8 points on the two benchmarks, respectively. These results suggest that agent skills are better treated as explicit procedural artifacts that can be verified, repaired, and shared across models.

Problem

• reusable skill acquisition
• multimodal GUI agents
• execution errors
• skill repair
• implicit action semantics

Innovation

• ContractSkill
• repairable skills
• multimodal web agents
• executable contracts
• skill verification