Performance Prediction for Large Systems via Text-to-Text Regression

2025-06-26

AI Summary

To address the challenge of feature engineering for raw textual data (e.g., configurations, logs) in performance metric prediction for large-scale systems like Google Borg, this paper proposes the first end-to-end text-to-text regression paradigm tailored for system performance forecasting. The method employs a randomly initialized 60M-parameter encoder-decoder architecture that directly consumes long raw text sequences and outputs numerical predictions—natively supporting uncertainty quantification and complex distribution modeling. Key contributions include: (1) eliminating conventional tabular preprocessing pipelines, enabling zero-shot initialization and rapid task adaptation with only 500 labeled samples; and (2) achieving an average rank correlation coefficient of 0.99 across the entire Borg cluster—substantially outperforming baselines—with mean squared error reduced by two orders of magnitude.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In many industries, predicting metric outcomes of large systems is a fundamental problem, driven largely by traditional tabular regression. However, such methods struggle on complex systems data in the wild such as configuration files or system logs, where feature engineering is often infeasible. We propose text-to-text regression as a general, scalable alternative. For predicting resource efficiency on Borg, Google's massive compute cluster scheduling system, a 60M parameter encoder-decoder, trained from random initialization, achieves up to a near perfect 0.99 (0.9 average) rank correlation across the entire fleet, and 100x lower MSE than tabular approaches. The model also easily adapts to new tasks in only 500 few-shot examples and captures the densities of complex outcome distributions. Ablation studies highlight the importance of using encoders, increasing sequence length, and the model's inherent uncertainty quantification. These findings pave the way for universal simulators of real-world outcomes.

Problem

• Predicting metric outcomes for large complex systems
• Overcoming limitations of traditional tabular regression methods
• Adapting to new tasks with minimal few-shot examples

Innovation

• Text-to-text regression for system prediction
• 60M parameter encoder-decoder model
• Few-shot adaptation with 500 examples