Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents

2026-06-25

AI Summary

This work addresses the reproducibility and auditability challenges in large-scale LLM-based coding agents, which are often compromised by instruction drift. The authors propose a verifier-driven research paradigm wherein human-defined hypotheses are tested by multiple LLM agents executing structured tasks in parallel, with research integrity enforced through deterministic Python verifiers. These verifiers encode domain-specific rules as executable scripts—rather than relying on natural language understanding—to guarantee task isolation, result immutability, and full traceability throughout the workflow. This approach substantially enhances reproducibility and scalability. Evaluated on the BEA 2026 lexical complexity task, the system achieved first place in the closed track and second in the open track, reducing RMSE by 29.9% and 35.9% over baselines, successfully eliminating four data-leakage feature sets at a total cost of approximately \$498.

Abstract

LLM coding agents make it tempting to automate empirical research by delegating experiments to them directly, but naive delegation does not scale to large projects: low-rate instruction lapses compound into broken, irreproducible artefacts. To address this problem, we present Glite ARF, an open-source Python framework for running many LLM coding agents in parallel on a research repository without sacrificing reproducibility or auditability. The framework defines a three-role stack: a human researcher chooses which hypotheses to test, coding agents (Claude Code, Codex CLI) implement individual tasks under a fixed structure, and deterministic Python verifier scripts enforce task isolation, immutability of completed work, a corrections overlay, and a materialised project overview. We call this verifier-driven research: the rules of the research process live in code that fails loudly when violated, not in prose that agents are merely asked to follow. Using Glite ARF, we developed our submission to the BEA 2026 vocabulary-difficulty shared task, placing first in the closed track and second in the open track on all three target languages (Spanish, German, Mandarin) and reducing the official baseline RMSE by 29.9% (closed) and 35.9% (open). The campaign comprised 273 tracked tasks (146 experiment runs) across 129 feature sets, run by up to twelve parallel agents orchestrated from a single laptop - with some model training on rented A100s - at approximately \$450 in LLM API spend (\$498 total third-party cost), and structured per-fold provenance let us catch and strip four target-leaking feature sets, correcting an implausible 0.609 RMSE to 0.802. Across three campaigns in three domains, the framework's structural machinery adds only about 1% of wall-clock time. Framework and a public demo project accompany this paper.

Problem

• LLM coding agents
• reproducibility
• empirical research automation
• instruction lapses
• irreproducible artefacts

Innovation

• verifier-driven research
• parallel LLM coding agents
• reproducibility
• task isolation
• deterministic verification