A Model-based Testing Technique for Amazon Lex Task-based Chatbots

2026-07-01

AI Summary

Existing testing approaches for chatbots often generate overly simplistic scenarios and lack effective test oracles, making it difficult to ensure the quality of task-oriented Amazon Lex bots. This work proposes LexTester—the first systematic, model-driven testing framework tailored for Amazon Lex—which constructs dialogue graphs to capture all possible interactions and leverages multiple coverage criteria to automatically generate high-coverage, executable test suites. Empirical evaluation on five real-world Lex bots demonstrates that, compared to Botium, LexTester nearly doubles test complexity, achieves dialogue element coverage rates of 83–95%, and improves defect detection efficiency by up to fourfold, all while incurring comparable time overhead.

Abstract

Task-based chatbots are nowadays widely adopted software systems, usually integrated into real-world applications and communication channels, designed to assist users in completing tasks through conversational interfaces. Like any other software, even chatbots are prone to bugs. Despite their increasing pervasiveness in everyday activities, existing techniques for assessing their quality still exhibit several limitations, such as the simplicity of generated test scenarios and oracle weaknesses. In this paper, we present LexTester, an automated model-based testing technique for Amazon Lex chatbots. The technique explores the conversational space of the chatbot under test to generate a Dialog Graph of all possible interactions, from which an executable test suite is generated according to different coverage strategies. LexTester was evaluated against the state-of-the-practice chatbot testing tool Botium on five Amazon Lex chatbots, consistently outperforming it in all subjects, generating more tests with nearly double complexity, achieving overall 83-95% coverage of conversational elements, and improving fault detection effectiveness by up to four times at comparable time costs.

Problem

• chatbot testing
• model-based testing
• test oracle
• conversational systems
• software quality

Innovation

• model-based testing
• chatbot testing
• dialog graph
• coverage strategy
• Amazon Lex