AS400-DET: Detection using Deep Learning Model for IBM i (AS/400)

2025-06-16

AI Summary

To address the challenge of automating GUI component testing for legacy IBM i (AS/400) systems, this work introduces the first domain-specific annotated dataset for this platform—comprising 1,050 screen images, including 381 Japanese-language interfaces—and covers seven critical interactive UI elements (e.g., text labels, input fields, tables). We propose an end-to-end detection framework that integrates multi-scale feature extraction with interface-aware geometric and semantic priors, combining an enhanced YOLOv8 backbone with a modified Mask R-CNN head. Evaluated on our benchmark, the method achieves 82.3% mAP—substantially outperforming off-the-shelf general-purpose detectors—while supporting real-time inference and robust identification of core controls such as command-line prompts and function keys. This work bridges critical gaps in data and methodology for intelligent mainframe GUI maintenance, establishing a reusable technical foundation for modernizing test automation of legacy enterprise systems.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Computer Vision: Object Detection & Categorization
• Application Domains: Internet of Things, Sensor Networks & Smart Cities

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This paper proposes a method for automatic GUI component detection for the IBM i system (formerly and still more commonly known as AS/400). We introduce a human-annotated dataset consisting of 1,050 system screen images, in which 381 images are screenshots of IBM i system screens in Japanese. Each image contains multiple components, including text labels, text boxes, options, tables, instructions, keyboards, and command lines. We then develop a detection system based on state-of-the-art deep learning models and evaluate different approaches using our dataset. The experimental results demonstrate the effectiveness of our dataset in constructing a system for component detection from GUI screens. By automatically detecting GUI components from the screen, AS400-DET has the potential to perform automated testing on systems that operate via GUI screens.

Problem

• Automatic GUI component detection for IBM i system
• Human-annotated dataset with 1,050 IBM i screen images
• Deep learning-based system for automated GUI testing

Innovation

• Deep learning model for GUI detection
• Human-annotated dataset with 1,050 images
• Automated testing via component detection