environment scripting

Existing large language model agents lack systematic evaluation of procedural memory transfer across tasks, roles, and models. This work proposes the AFTER benchmark, comprising 382 real-world enterprise tasks, to establish the first procedural memory evaluation framework tailored for enterprise-level workflows. The study reveals a fundamental trade-off between general-purpose and specialized skills and introduces a multi-model execution trajectory fusion strategy. Experimental results demonstrate that single-round skill optimization improves performance by 3.7–6.7 points, while the fused multi-model approach achieves 73.1% accuracy in cross-model evaluations, significantly outperforming individual models.

It remains unclear whether large language model agents can distill reusable procedural skills from task experience. This work proposes SkillEvolBench, a benchmark comprising 180 role-conditioned tasks that systematically disentangles procedural abstraction from foundational capabilities, prior knowledge, and experience reuse. The framework evaluates skill evolution through compact trajectory compression, verifier feedback, and multidimensional deployment tests—including contextual transfer, adversarial shortcuts, and compositional generalization. Experiments reveal that current agents predominantly exhibit local adaptation, with distilled skills often underperforming direct reuse of original trajectories. Moreover, merely increasing the number of skills or computational resources fails to yield consistent performance gains, highlighting fundamental challenges in the formation of procedural knowledge.

Large language model (LLM) agents often struggle to reuse past experiences in recurring scenarios, leading to computational redundancy and behavioral instability. To address this, this work proposes the Skill-MDP framework, which converts interaction histories into executable skills and introduces a non-parametric variant of Proximal Policy Optimization (PPO). By integrating semantic gradient-guided skill generation with a PPO-based gating mechanism, the approach enables procedural memory learning and validation without requiring parameter updates. Coupled with a score-driven memory maintenance strategy, the method significantly enhances skill reuse and task performance across in-domain, cross-task, and cross-agent settings, while achieving extremely high compression ratios in storing procedural memories.

This work addresses the scalability bottleneck in AI tutoring systems caused by the labor-intensive, manual construction of structured procedural skill models. To overcome this limitation, the authors propose a human-in-the-loop text-to-model generation approach that leverages large language models to automatically transform instructional texts into procedural skill models conforming to the Task-Method-Knowledge (TMK) ontology. The method integrates ontology-constrained prompting with template-driven generation and incorporates expert validation of causal logic and failure conditions. This framework preserves model structural integrity and semantic alignment while substantially reducing expert modeling effort. Evaluated in a graduate-level AI course, the approach produced 23 skill models with 50–70% less expert time investment, and the generated models demonstrated high reproducibility under fixed inputs.

Current evaluation methods struggle to finely assess how large language model agents actually utilize reusable skills and why they fail. This work proposes "skill coverage" as a test adequacy metric tailored to agent skills, translating natural language skill instructions into semi-structured behavioral constraints and evaluating whether these constraints are covered and successfully executed based on execution traces. This approach decouples skill usage from task outcomes, enabling actionable failure attribution. Experiments on SkillsBench reveal that existing agents cover only 38.66%–45.51% of skill constraints; further, reinforcing skills based on failed constraints yields an average task recovery rate of 16.0% across previously failed tasks.

Existing benchmarks struggle to disentangle the influence of an agent’s skill content from its organizational structure on runtime behavior. This work proposes the Progressive Disclosure paradigm for skill organization, which integrates semantically controlled skill variants with multi-round matching evaluation. For the first time, it systematically demonstrates that—holding task knowledge constant—the structure of skills alone significantly affects large language model agents’ procedural knowledge retrieval and reasoning trajectories, with effects modulated by task characteristics. Evaluated via the SkillJuror framework across 82 tasks, this approach increases the average number of skill invocations per trajectory from 1.18 to 3.85 and effective adoptions from 1.33 to 3.92, while yielding 17 additional verified successes (+4.1%) in 410 matching trials.

This work addresses the challenge in scientific computing where models struggle to transfer knowledge from individual tasks to broader, reusable capabilities. The authors propose SciConsolidate, a novel framework that, for the first time, extracts procedural knowledge from runtime success and failure trajectories and employs a development-validation gating mechanism to filter effective knowledge. To bridge the gap between abstract knowledge and executable code, the approach integrates failure-driven synthesis of unanswerable queries with strong-model-guided concretization supervision. Additionally, a matched teacher-branch architecture is introduced to significantly enhance the performance of smaller models. Experimental results demonstrate that Qwen3.5-9B achieves gains of 6.25 and 3.89 points over program-free supervised fine-tuning—and improvements of 11.25 and 5.62 points over the base model—on main tasks and sub-steps, respectively, validating the efficacy of the proposed methodology.

Current language model agents struggle to effectively recognize, invoke, and coordinate multiple skills to accomplish complex tasks. This work proposes the first scalable, verification-based synthetic data generation framework that automatically constructs 4,000 skill-oriented tasks and 27,164 high-quality executable trajectories from 2,000 publicly available skills. The framework integrates rule-based validation, agent self-feedback repair, supervised fine-tuning, and cross-agent interface evaluation to ensure data fidelity and task complexity. The resulting SkillEval benchmark supports both single-skill and multi-skill compositional tasks, significantly enhancing diverse models’ skill invocation capabilities across multiple scenarios. Moreover, performance consistently improves as the breadth of covered skills expands, demonstrating the framework’s effectiveness in fostering scalable, skill-aware agent behavior.

This work addresses the semantic misalignment between retrieved general-purpose skills and the current task, environment, or other skills during execution—where skills are semantically relevant but suffer from execution-level mismatches. To resolve this, the paper proposes SkillAligner, a framework that treats retrieved skills as tunable drafts at inference time without requiring additional training. SkillAligner performs a one-shot joint adaptation to simultaneously customize skills for the target task, align their interfaces, and coordinate multi-skill interactions by resolving dependencies, eliminating conflicts, and removing redundancies, thereby producing a compact and unified execution plan. Experiments demonstrate that SkillAligner significantly improves task success rates across diverse agent benchmarks and model backbones, effectively mitigates performance degradation caused by skill integration, and reduces inference overhead.

This work addresses the high barrier to entry in traditional game modification, which typically requires programming expertise and lacks mechanisms for real-time, in-game adjustments via natural language. The paper presents the first voice-driven system that integrates a large language model (LLM) into a live game-editing pipeline, enabling players to dynamically modify over one hundred configuration parameters in a retro-style Space Invaders–like game using spoken natural language commands. The system supports immediate alterations to gameplay mechanics, visuals, interactions, and audio without exposing underlying code, establishing a seamless “play–edit–feedback” loop. User studies indicate that the editing experience is positively received with moderate cognitive load, effectiveness independent of programming background, enhanced usability through instant feedback, increased enjoyment from core mechanic modifications, and the emergence of diverse editing strategies—including exploratory, goal-directed, and iterative parameter tuning.