Semantic-based Distributed Learning for Diverse and Discriminative Representations

2026-04-20

AI Summary

This work addresses the limitation of conventional distributed learning methods in classification tasks, which often yield unstructured embeddings that lack both diversity and discriminability among samples of the same class. To overcome this, the authors propose a semantic-driven distributed learning framework that enforces representation variance constraints and incorporates a semantic sharing mechanism. Under i.i.d. and non-i.i.d. settings, the framework employs primal-dual optimization and a clustered virtual node strategy, respectively, to achieve structured representation learning. Notably, the approach ensures representational diversity and discriminability without requiring uniform network architectures and provides theoretical convergence guarantees. Experimental results on MNIST, CIFAR-10, and CIFAR-100 demonstrate significant improvements in representation quality and effective capture of global semantic structures.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In large-scale distributed scenarios, increasingly complex tasks demand more intelligent collaboration across networks, requiring the joint extraction of structural representations from data samples. However, conventional task-specific approaches often result in nonstructural embeddings, leading to collapsed variability among data samples within the same class, particularly in classification tasks. To address this issue and fully leverage the intrinsic structure of data for downstream applications, we propose a novel distributed learning framework that ensures both diverse and discriminative representations. For independent and identically distributed (i.i.d.) data, we reformulate and decouple the global optimization function by introducing constraints on representation variance. The update rules are then derived and simplified using a primal-dual approach. For non-i.i.d. data distributions, we tackle the problem by clustering and virtually replicating nodes, allowing model updates within each cluster using block coordinate descent. In both cases, the resulting optimal solutions are theoretically proven to maintain discriminative and diverse properties, with a guaranteed convergence for i.i.d. conditions. Additionally, semantic information from representations is shared among nodes, reducing the need for common neural network architectures. Finally, extensive simulations on MNIST, CIFAR-10 and CIFAR-100 confirm the effectiveness of the proposed algorithms in capturing global structural representations.

Problem

• distributed learning
• representation collapse
• structural representation
• non-i.i.d. data
• embedding diversity

Innovation

• distributed learning
• diverse representations
• discriminative representations
• semantic sharing
• non-i.i.d. data