Trustworthy Scheduling for Big Data Applications

2026-01-26

AI Summary

This work addresses the lack of transparency in existing schedulers, which often fail to provide actionable guidance for meeting service-level objectives (SLOs) while optimizing task execution time and resource utilization. To bridge this gap, the authors propose X-Sched, a middleware that integrates counterfactual explanation techniques with scheduling systems for the first time. Leveraging machine learning models such as random forests, X-Sched generates interpretable and actionable resource allocation recommendations in containerized environments. Empirical evaluation using real-world data demonstrates that X-Sched not only maintains performance targets but also significantly enhances the trustworthiness, transparency, and practical utility of scheduling decisions.

Technology Category

• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Algorithmic accountability and transparency on the web
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Recent advances in modern containerized execution environments have resulted in substantial benefits in terms of elasticity and more efficient utilization of computing resources. Although existing schedulers strive to optimize performance metrics like task execution times and resource utilization, they provide limited transparency into their decision-making processes or the specific actions developers must take to meet Service Level Objectives (SLOs). In this work, we propose X-Sched, a middleware that uses explainability techniques to generate actionable guidance on resource configurations that makes task execution in containerized environments feasible, under resource and time constraints. X-Sched addresses this gap by integrating counterfactual explanations with advanced machine learning models, such as Random Forests, to efficiently identify optimal configurations. This approach not only ensures that tasks are executed in line with performance goals but also gives users clear, actionable insights into the rationale behind scheduling decisions. Our experimental results validated with data from real-world execution environments, illustrate the efficiency, benefits and practicality of our approach.

Problem

• trustworthy scheduling
• explainability
• Service Level Objectives
• containerized environments
• resource configuration

Innovation

• Explainable Scheduling
• Counterfactual Explanations
• Containerized Environments
• Service Level Objectives
• Machine Learning for Resource Management