SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills

2026-07-09

AI Summary

This work addresses the critical security risks—such as privacy leaks and misconfigurations—that arise when large language model (LLM) agents invoke skills without respecting their internal logical constraints. To tackle this issue, we propose SkillLogic, a framework that formally defines eight categories of logical relationships inherent in skills. Leveraging static analysis of over 5,000 publicly available skills combined with manual verification, we construct SLBench, the first executable benchmark for evaluating logical compliance in skill invocation. Our experiments reveal that state-of-the-art LLMs exhibit logical violations in up to 70% of cases on SLBench. To mitigate this, we introduce SLGuard, a lightweight runtime defense mechanism that reduces violation rates by 63% on targeted cases, substantially enhancing the reliability and safety of skill-based agent interactions.

Abstract

Agent skills extend LLM agents with reusable procedures, tools, and domain-specific workflows, but their safety depends on resolving dependencies among interacting instructions. We introduce SkillLogic, a framework for analyzing logical relations in skill files and constructing executable tests from them. Our taxonomy covers eight relation types, including preconditions that gate valid actions, constraints that limit how allowed actions may be performed, and fallbacks that specify recovery behavior after failure. Using SkillLogic, we scan over 5000 public skills and find that 70% contain at least one logical relation. We then construct SLBench, an 86-case executable benchmark from high-confidence, high-impact, and locally testable relations. Evaluating Codex and Claude Code across six LLM backbones shows unsafe rates up to 70%, with violations leading to privacy leaks, unsafe configuration changes, and incomplete cleanup. The human audit attributes failures to both agent capability gaps and low-salience skill text. We further show that SLGuard, a lightweight inference-time scaffold, reduces violations by 63% on targeted cases. Our results establish logical-relation following as a distinct reliability challenge for skill-guided agents.

Problem

• logical relations
• LLM agents
• skill safety
• dependency resolution
• reliability challenge

Innovation

• SkillLogic
• logical relations
• SLBench
• LLM agents
• SLGuard