ASSURE: Metamorphic Testing for AI-powered Browser Extensions

2025-07-07

AI Summary

To address the challenges of non-determinism, context sensitivity, and tight environment coupling in testing LLM-driven browser extensions, this paper introduces the first modular, automated testing framework specifically designed for AI-powered browser extensions. The framework integrates mutation testing—novelly adapted to this domain—metamorphic testing, and security invariant verification to enable diverse test case generation, interactive automated execution, and behavioral consistency checking. By decoupling LLM behavior from the browser execution environment, it implements a pluggable, configurable validation pipeline. Empirical evaluation across six widely used AI extensions uncovered 531 distinct defects—including security vulnerabilities and content inconsistencies—achieved a 6.4× improvement in test throughput, and localized critical security issues within an average of 12.4 minutes, thereby significantly enhancing feasibility for integration into agile development workflows.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The integration of Large Language Models (LLMs) into browser extensions has revolutionized web browsing, enabling sophisticated functionalities like content summarization, intelligent translation, and context-aware writing assistance. However, these AI-powered extensions introduce unprecedented challenges in testing and reliability assurance. Traditional browser extension testing approaches fail to address the non-deterministic behavior, context-sensitivity, and complex web environment integration inherent to LLM-powered extensions. Similarly, existing LLM testing methodologies operate in isolation from browser-specific contexts, creating a critical gap in effective evaluation frameworks. To bridge this gap, we present ASSURE, a modular automated testing framework specifically designed for AI-powered browser extensions. ASSURE comprises three principal components: (1) a modular test case generation engine that supports plugin-based extension of testing scenarios, (2) an automated execution framework that orchestrates the complex interactions between web content, extension processing, and AI model behavior, and (3) a configurable validation pipeline that systematically evaluates behavioral consistency and security invariants rather than relying on exact output matching. Our evaluation across six widely-used AI browser extensions demonstrates ASSURE's effectiveness, identifying 531 distinct issues spanning security vulnerabilities, metamorphic relation violations, and content alignment problems. ASSURE achieves 6.4x improved testing throughput compared to manual approaches, detecting critical security vulnerabilities within 12.4 minutes on average. This efficiency makes ASSURE practical for integration into development pipelines, offering a comprehensive solution to the unique challenges of testing AI-powered browser extensions.

Problem

• Testing non-deterministic behavior of AI browser extensions
• Addressing context-sensitivity in LLM-powered extension testing
• Bridging the gap between web environments and LLM testing

Innovation

• Modular test case generation engine for extensions
• Automated execution framework for web-AI interactions
• Configurable validation pipeline for behavioral consistency