Multitask Learning with Learned Task Relationships

2025-10-12

AI Summary

In federated/decentralized learning over heterogeneous data, conventional consensus strategies suffer from statistical suboptimality, while existing multitask or personalized approaches either over-rely on prior task relationship assumptions or lack structured modeling. Method: We propose a multitask personalized framework that jointly learns task relationships and local models. Innovatively, we employ a Gaussian Markov Random Field (GMRF) to model implicit task dependencies, simultaneously estimating the unknown precision matrix and local model parameters—enabling adaptive collaboration without requiring prior knowledge of task structure. Contribution/Results: Theoretical analysis characterizes the statistical quality of the learned task relationships. Experiments demonstrate that our method significantly outperforms mainstream baselines across diverse heterogeneous settings, achieving both structural interpretability and optimization efficacy.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Classical consensus-based strategies for federated and decentralized learning are statistically suboptimal in the presence of heterogeneous local data or task distributions. As a result, in recent years, there has been growing interest in multitask or personalized strategies, which allow individual agents to benefit from one another in pursuing locally optimal models without enforcing consensus. Existing strategies require either precise prior knowledge of the underlying task relationships or are fully non-parametric and instead rely on meta-learning or proximal constructions. In this work, we introduce an algorithmic framework that strikes a balance between these extremes. By modeling task relationships through a Gaussian Markov Random Field with an unknown precision matrix, we develop a strategy that jointly learns both the task relationships and the local models, allowing agents to self-organize in a way consistent with their individual data distributions. Our theoretical analysis quantifies the quality of the learned relationship, and our numerical experiments demonstrate its practical effectiveness.

Problem

• Learning task relationships in federated multitask learning
• Overcoming statistical suboptimality in heterogeneous data settings
• Jointly learning task relationships and local models

Innovation

• Learns task relationships via Gaussian Markov Random Field
• Jointly optimizes local models and task dependencies
• Enables self-organization without prior relationship knowledge