Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study

2026-07-03

AI Summary

This study addresses the longstanding challenge of scaling software diversity due to high costs by systematically exploring large language models (LLMs) as a low-cost, effective source of diversity. For the first time, the classical diversity research paradigm is extended to LLMs, evaluating their feasibility through multidimensional heterogeneous ensembles—spanning different models, decoding temperatures, and programming languages. Candidate programs generated by these ensembles undergo unified compilation, sandboxed execution, and exhaustive testing to empirically analyze failure behavior overlap between LLM-generated and human-written programs. Results demonstrate that heterogeneous LLM combinations—particularly those involving cross-language generation or pairing with human-authored code—significantly reduce common-cause failures and substantially enhance system reliability, thereby validating the practical utility of LLM-driven software diversity.

Abstract

Software diversity has been extensively studied as a means of reducing the risk of common-mode failures. Classic work showed that the central issue is whether failures of diversely redundant components overlap in ways that limit the reliability gains. Traditional software diversity is costly to obtain, since it requires multiple implementations as well as the corresponding validation, maintenance, and deployment effort. Recent advances in Large Language Models (LLMs) may change this. LLMs enable inexpensive code generation: they produce many candidate implementations of the same specification quickly, across different models, decoding settings, and programming languages. This raises a natural question: can LLMs serve as practical generators of software diversity, and how much reliability improvement can that diversity actually provide? In this paper, we extend classical empirical studies of software diversity in human-written programs to LLM-generated code. We study three specifications using both historical human-written programs and large pools of LLM-generated ones evaluated under a common compilation, sandboxing, and exhaustive test suite. We explore LLM diversity along multiple axes, including model family, generation temperature, and programming language. Reliability improvement is evaluated in a 1-out-of-2 configuration across both homogeneous and heterogeneous program populations, including within-LLM pairings and pairings across programming languages and across LLM-generated and human-written programs. The results show that combining LLM-generated programs, especially in heterogeneous settings, can yield reliability gains, although this is partly conditioned by the programming language and generation setting. Taken together, these findings suggest that LLMs provide a scalable source of comparatively low-cost programs whose diversity can be leveraged for reliability improvement.

Problem

• software diversity
• reliability improvement
• large language models
• common-mode failures
• empirical study

Innovation

• LLM-generated code
• software diversity
• reliability improvement
• empirical study
• heterogeneous redundancy