Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization

2026-05-10

AI Summary

This study addresses the challenges of COBOL system modernization—namely, the scarcity of domain experts, the scale of legacy codebases, and stringent correctness requirements—and investigates the efficacy of large language model (LLM)-based orchestration strategies in automated COBOL-to-Python translation. For the first time, orchestration strategy is isolated as the sole variable within a unified experimental framework, enabling a direct comparison between deterministic and LLM-driven approaches. The findings reveal that deterministic orchestration achieves functional correctness on par with LLM-based control while significantly enhancing robustness, reducing inter-run performance variability, and cutting token consumption by up to 3.5×. These results demonstrate that deterministic orchestration offers superior stability and cost efficiency without compromising translation accuracy.

Abstract

Modernizing legacy COBOL systems remains difficult due to scarce expertise, large and long-lived codebases, and strict correctness requirements. Recent large language model (LLM)-based modernization systems increasingly rely on agentic workflows in which the model controls multi-step tool execution. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows. We present a controlled empirical study of deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using a unified experimental framework, we hold the language models, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. We evaluate both approaches using functional correctness, robustness across repeated stochastic runs, and computational efficiency. Across multiple models, deterministic orchestration achieves comparable computational accuracy to LLM-controlled orchestration while improving worst-case robustness and reducing performance variability across runs. Deterministic execution also reduces token consumption by up to 3.5x, leading to substantially lower operational cost. These results suggest that, in structured modernization workflows with explicit validation stages, fixed execution policies provide more stable and cost-efficient behavior than fully agentic orchestration without reducing translation quality.

Problem

• COBOL-to-Python modernization
• LLM-controlled orchestration
• deterministic orchestration
• software modernization
• execution control

Innovation

• deterministic orchestration
• LLM-controlled orchestration
• COBOL-to-Python modernization
• empirical study
• token efficiency