Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey

2026-05-12

AI Summary

This work addresses the exacerbation of the oracle problem in software testing caused by the generative and open-ended nature of large language models (LLMs), which traditional methods struggle to handle. Through a systematic review of 93 studies, it proposes and constructs a novel bidirectional synergy framework between metamorphic testing (MT) and LLMs. On one hand, MT is leveraged to evaluate LLM behaviors concerning hallucination, fairness, and robustness; on the other, LLMs’ semantic understanding and code generation capabilities are harnessed to automate key MT tasks—namely, metamorphic relation discovery, input transformation, and test execution. The study establishes a unified taxonomy encompassing both “MT for LLMs” and “LLMs for MT,” thereby providing a structured foundation and a co-evolutionary pathway for AI quality assurance.

Abstract

Large language models (LLMs) have introduced substantial challenges to software quality assurance due to their generative, probabilistic, and open-ended nature, which intensifies the oracle problem and limits the applicability of traditional testing methods. Metamorphic testing (MT), which checks necessary relations among multiple related executions rather than relying on exact expected outputs, has emerged as a promising approach for testing LLMs and other oracle-deficient systems. At the same time, the strong semantic understanding, reasoning, and code generation capabilities of LLMs create new opportunities to automate the traditionally labor-intensive phases of MT. This survey systematically reviews 93 primary studies and characterizes this reciprocal relationship as the bidirectional empowerment of MT and LLMs. We propose a taxonomy spanning two complementary directions: MT for LLMs, which uses MT to verify, validate, assess, and understand LLMs and LLM-based systems across issues such as hallucination, fairness, robustness, code reliability, retrieval-augmented generation, dialogue, and autonomous agents; and LLMs for MT, which leverages LLMs to support metamorphic relation discovery, input transformation and synthesis, executable test implementation, and agentic closed-loop testing. By synthesizing these developments, this survey provides a structured foundation for understanding the evolving synergy between MT and LLMs and highlights future directions for building more rigorous, scalable, and trustworthy AI quality assurance methodologies.

Problem

• large language models
• metamorphic testing
• oracle problem
• software quality assurance
• AI testing

Innovation

• Metamorphic Testing
• Large Language Models
• Bidirectional Empowerment
• Oracle Problem
• AI Quality Assurance