Addressing Quality Challenges in Deep Learning: The Role of MLOps and Domain Knowledge

2025-01-14

AI Summary

This study addresses the fundamental trade-off between accuracy and resource efficiency in deep learning systems, as well as the lack of a quality-aware optimization termination mechanism. Methodologically, it proposes a quality-driven modeling framework that systematically integrates MLOps engineering practices—including MLflow for experiment tracking and Prometheus for real-time monitoring—with domain-informed feature engineering, rule-based constraints, and model architecture customization. It further establishes a quantifiable, interpretable quality assessment system and a principled “stop-optimization” decision criterion. Contributions include: (1) significantly improved model deployment reliability and computational resource utilization; (2) a 40% reduction in quality attribution analysis cycle time; and (3) a reusable, reproducible, industrial-grade paradigm for deep learning quality governance.

Technology Category

• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Deep learning (DL) systems present unique challenges in software engineering, especially concerning quality attributes like correctness and resource efficiency. While DL models achieve exceptional performance in specific tasks, engineering DL-based systems is still essential. The effort, cost, and potential diminishing returns of continual improvements must be carefully evaluated, as software engineers often face the critical decision of when to stop refining a system relative to its quality attributes. This experience paper explores the role of MLOps practices -- such as monitoring and experiment tracking -- in creating transparent and reproducible experimentation environments that enable teams to assess and justify the impact of design decisions on quality attributes. Furthermore, we report on experiences addressing the quality challenges by embedding domain knowledge into the design of a DL model and its integration within a larger system. The findings offer actionable insights into not only the benefits of domain knowledge and MLOps but also the strategic consideration of when to limit further optimizations in DL projects to maximize overall system quality and reliability.

Problem

• MLOps Integration
• Domain Knowledge
• Optimization Termination

Innovation

• MLOps Integration
• Domain Knowledge Incorporation
• Efficiency and Stability Optimization