Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows

2026-07-01

AI Summary

This work addresses the silent failures—such as missing fields, hallucinated API calls, or premature termination—that commonly arise when large language models are deployed in enterprise SaaS workflows due to misalignment between pretraining objectives and real-world tasks. To tackle this, the authors propose Reinforcement Learning with Verifiable Rewards (RLVR), a method that directly optimizes agent behavior for Atlassian API interactions without requiring human annotations, online APIs, or learned reward models. Leveraging high-fidelity simulators of Jira and Confluence and employing the GRPO algorithm with Qwen3-series models, RLVR significantly improves average task rewards from 0.35–0.92 to 0.95–1.00 across four non-degenerate scenarios. Notably, it achieves a perfect reward of 1.00 on Confluence page creation, demonstrating, for the first time, effective alignment and reliable execution of smaller models in enterprise API settings.

Abstract

Large language models are trained to predict the next token, not to act inside a specific API. In niche enterprise SaaS workflows -- where success means hitting the right endpoint with the right nested arguments in the right order -- this objective mismatch shows up as silent failures: dropped required fields, hallucinated tools, or early stops after a single read. We ask whether Reinforcement Learning with Verifiable Rewards (RLVR), applied directly in the target environment, closes the gap. As a proof of concept we build a suite of five synthetic environments emulating the Jira REST v3 and Confluence v2 APIs at schema fidelity; rewards are computed entirely from the tool-call trace, with no live API, no learned judge, and no human label in the loop. Scoring prompted Qwen3-1.7B and Qwen3.5-4B on the same checkers that drive GRPO training, we find that on the four scenarios whose rewards are non-degenerate the RL-trained policy lifts average reward from a 4B-baseline range of 0.35--0.92 to 0.95--1.00, with the largest single gain on Confluence page creation ($0.35 \rightarrow 1.00$). We position this as a preliminary step toward outcome-optimised small models for niche enterprise APIs, and foreground two limitations a workshop reader should weigh: hand-crafting verifiable rewards does not scale beyond the handful of endpoints reported here, and one of our five scenarios (ticket-transition) has a saturating reward shape that the prompted 4B already maxes out.

Problem

• next-token prediction
• enterprise SaaS workflows
• tool-use agents
• silent failures
• API interaction

Innovation

• Reinforcement Learning with Verifiable Rewards
• tool-use agents
• enterprise API automation
• schema-faithful simulation
• outcome-optimised language models