Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents

2026-05-18

AI Summary

This study addresses the limited generalizability of behavior rules for software engineering agents derived from single-framework studies. Through a large-scale experimental ecosystem encompassing 126 agent configurations, 43 frameworks, and 64,380 SWE-bench runs, the authors systematically investigate the relationship between behavioral signals and problem-solving performance by controlling either the large language model (LLM) or the framework layer. Employing variance decomposition, behavioral feature statistics, and directional consistency analysis, they find that framework-level factors explain behavioral differences more significantly than LLM choice. Notably, in nearly half of the configurations, key behavioral signals—such as error rates—exhibit opposing effects across frameworks, revealing for the first time that identical behaviors can carry divergent or even contradictory semantic interpretations depending on the framework. These findings challenge the assumption of cross-framework universality in single-framework-derived rules and underscore the necessity of multi-framework validation.

Abstract

Behavioral studies of LLM-based software engineering agents extract operational rules about which trajectory shapes correlate with higher resolution rates: that a test step follows a code modification, that error cascades are short, or that trajectories are compact. Each rule is typically derived from a single framework, and whether it transfers, in sign as well as magnitude, to structurally different agent designs has not been directly tested. We address this at ecosystem scale: 64,380 SWE-bench runs from 126 agent configurations spanning 43 frameworks, where each configuration pairs an LLM with a framework (e.g., SWE-Agent, OpenHands) that supplies its tools and workflow. We separate framework effects from LLM effects by holding each layer fixed in turn, then measure one behavior-outcome effect per configuration and examine how those effects agree or disagree. Swapping the framework while the LLM is held fixed produces large behavioral differences in every action feature. On most signals, configurations disagree not merely in magnitude but in direction. Error rate is the cleanest case: 47 configurations resolve more issues when their error rate is lower, while 48 resolve more when it is higher. Five other continuous features and three of seven binary patterns from prior SE literature show similar directional disagreement. Framework identity accounts for more of this variation than LLM family: for mean turns, framework explains 64% of the between-configuration variance against the LLM's 10%. The implication is that the same observable behavioral signal can carry opposite meaning for different agent configurations. Behavioral findings from any single framework therefore warrant cross-configuration validation before being claimed as general.

Problem

• software engineering agents
• behavioral analysis
• framework generalization
• LLM-based agents
• cross-framework validation

Innovation

• cross-framework analysis
• behavioral semantics
• software engineering agents
• LLM-agent interaction
• ecosystem-scale evaluation