Beyond Surface Similarity: Evaluating LLM-Based Test Refactorings with Structural and Semantic Awareness

2025-06-07

AI Summary

Existing automated unit test refactoring by LLMs lacks reliable, multidimensional evaluation metrics. Method: This paper proposes CTSES, the first composite evaluation framework that jointly models behavioral preservation, semantic consistency, readability, and structural validity. CTSES integrates CodeBLEU, METEOR, and ROUGE-L to enable joint quantification of renaming, structural reorganization, and semantic equivalence. We conduct large-scale experiments on the Defects4J and SF110 Java benchmarks using GPT-4o and Mistral-Large-2407, enhanced with Chain-of-Thought prompting. Results: Evaluated across 5,000+ test suites, CTSES significantly outperforms prior metrics (p < 0.01) and achieves high agreement with developer judgments and human evaluations (Spearman’s ρ = 0.89). The framework substantially enhances the trustworthiness and practical utility of LLM-driven test refactoring.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Evaluation and Analysis
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large Language Models (LLMs) are increasingly employed to automatically refactor unit tests, aiming to enhance readability, naming, and structural clarity while preserving functional behavior. However, evaluating such refactorings remains challenging: traditional metrics like CodeBLEU are overly sensitive to renaming and structural edits, whereas embedding-based similarities capture semantics but ignore readability and modularity. We introduce CTSES, a composite metric that integrates CodeBLEU, METEOR, and ROUGE-L to balance behavior preservation, lexical quality, and structural alignment. CTSES is evaluated on over 5,000 test suites automatically refactored by GPT-4o and Mistral-Large-2407, using Chain-of-Thought prompting, across two established Java benchmarks: Defects4J and SF110. Our results show that CTSES yields more faithful and interpretable assessments, better aligned with developer expectations and human intuition than existing metrics.

Problem

• Evaluating LLM-based test refactorings with structural and semantic awareness
• Balancing behavior preservation, lexical quality, and structural alignment
• Improving assessment fidelity and interpretability for refactored unit tests

Innovation

• Composite metric CTSES balances behavior and structure
• Integrates CodeBLEU, METEOR, and ROUGE-L metrics
• Evaluated on GPT-4o and Mistral-Large-2407 refactorings