Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

2026-08-11

AI Summary

This study addresses a critical gap in the evaluation of multilingual agents, which traditionally focuses solely on final answers while neglecting auditable action trajectories that impact cost and latency. Introducing action-policy consistency as a core metric, the work systematically analyzes 2.38 million reasoning trajectories from eight models across 41 languages and six benchmarks. Through large-scale trajectory collection, similarity normalization, and preregistered causal validation, the authors identify and control for five confounding factors. Their findings reveal that non-English tasks commonly route through English intermediaries—a pathway shown to be causally necessary. Under greedy decoding, state-of-the-art models retain cross-lingual policy consistency at rates of 71–73%, with model identity accounting for only 5.7% of variance. Policy collapse occurs below 10B parameters, and a single regular expression error can inflate performance misestimation by up to 26-fold.

Abstract

When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.

Problem

• cross-lingual policy retention
• tool-using agents
• action consistency
• multilingual evaluation
• behavioral reproducibility

Innovation

• cross-lingual policy retention
• tool-using agents
• action trace evaluation
• multilingual reasoning
• policy consistency