From Technical Excellence to Practical Adoption: Lessons Learned Building an ML-Enhanced Trace Analysis Tool

2025-08-02

AI Summary

Industrial adoption of machine learning–enhanced system trace analysis tools remains limited due to the “excellence paradox”: advanced technical capabilities often compromise usability, transparency, and user trust. Method: We propose an adoption-oriented design paradigm grounded in three principles—cognitive compatibility, expert knowledge embedding, and transparency-driven trust building—and implement TMLL, a tool library integrating system tracing, interpretable ML, and human-in-the-loop mechanisms to support semi-automated analysis, incremental workflow integration, and verifiable outputs. Contribution/Results: Validated by Ericsson experts and integrated into the Eclipse Foundation ecosystem, TMLL was evaluated via a survey of 40 industry and academic experts. Results show 77.5% prioritize result credibility and 67.5% prefer controllable semi-automation. This work establishes both a theoretical framework and practical pathway for deploying trustworthy AI tools in industrial settings.

Technology Category

• Machine Learning: Evaluation and Analysis
• Humans and AI: Human-in-the-loop Machine Learning
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

System tracing has become essential for understanding complex software behavior in modern systems, yet sophisticated trace analysis tools face significant adoption gaps in industrial settings. Through a year-long collaboration with Ericsson Montréal, developing TMLL (Trace-Server Machine Learning Library, now in the Eclipse Foundation), we investigated barriers to trace analysis adoption. Contrary to assumptions about complexity or automation needs, practitioners struggled with translating expert knowledge into actionable insights, integrating analysis into their workflows, and trusting automated results they could not validate. We identified what we called the Excellence Paradox: technical excellence can actively impede adoption when conflicting with usability, transparency, and practitioner trust. TMLL addresses this through adoption-focused design that embeds expert knowledge in interfaces, provides transparent explanations, and enables incremental adoption. Validation through Ericsson's experts' feedback, Eclipse Foundation's integration, and a survey of 40 industry and academic professionals revealed consistent patterns: survey results showed that 77.5% prioritize quality and trust in results over technical sophistication, while 67.5% prefer semi-automated analysis with user control, findings supported by qualitative feedback from industrial collaboration and external peer review. Results validate three core principles: cognitive compatibility, embedded expertise, and transparency-based trust. This challenges conventional capability-focused tool development, demonstrating that sustainable adoption requires reorientation toward adoption-focused design with actionable implications for automated software engineering tools.

Problem

• Barriers in translating expert knowledge into actionable insights
• Challenges integrating trace analysis into existing workflows
• Lack of trust in automated results without validation

Innovation

• Embeds expert knowledge in interfaces
• Provides transparent explanations for results
• Enables incremental adoption by users