A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code

2026-07-02

AI Summary

This study systematically evaluates whether code generated by large language models (LLMs) satisfies the fault independence assumption underlying N-version programming. To this end, we introduce the first comprehensive assessment framework that integrates structural diversity analysis, behavioral failure pattern comparison under identical test cases, reliability modeling via majority voting, and manual root-cause inspection. Our experiments span 224 programming problems and reveal that even heterogeneous LLM ensembles achieve only 0.43–0.44 of the theoretical reliability gain, while homogeneous ensembles fall below 0.3. Moreover, the majority of failures share common root causes. These findings demonstrate that LLM-generated code exhibits pervasive non-independent faults, thereby challenging a core premise of N-version programming.

Abstract

N-Version Programming (NVP) improves software reliability by executing multiple independent implementations and combining outputs, but its adoption is limited by high cost and the assumption of failure independence, which empirical studies have challenged. Recent advances in Large Language Models (LLMs) reduce the cost of generating multiple implementations, shifting focus to whether their failures are independent. We propose the first systematic methodology to assess failure independence in LLM-generated code and apply it to 224 problems across twelve models, five languages, and three prompting strategies. We analyze both structural and behavioral diversity (i.e., whether implementations fail on the same test cases), complemented by N-version reliability analysis under majority voting and manual inspection of the generated code. Structural diversity analysis shows that implementations from the same model are highly similar, while different models produce more distinct solutions. The same trend appears in behavioral diversity, with implementations from different models showing higher diversity yet still failing on the same tests far more often than expected under independence. N-version reliability analysis reinforces this: three- and five-version ensembles realize only 0.43 and 0.44 of the reliability gain achievable under independence, dropping below 0.3 when ensembles are built from the same model. Manual fault analysis shows that even different failure patterns often share root causes. Overall, these results suggest LLM-generated solutions do not satisfy NVP's failure independence assumption, though heterogeneous models help partially. They also validate our methodology as a tool for systematically evaluating failure independence as models evolve.

Problem

• failure independence
• N-Version Programming
• LLM-generated code
• software reliability
• behavioral diversity

Innovation

• failure independence
• N-version programming
• LLM-generated code
• behavioral diversity
• reliability analysis