Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation

2024-12-11

arXiv.org

AI Summary

Existing federated learning (FL) benchmarks are confined to single-label image classification, failing to capture complex semantic heterogeneity across clients—such as discrepancies in the joint distributions of objects, relations, and attributes. This work introduces FL-PSG, the first FL benchmark tailored for complex semantic tasks, specifically targeting federated panoramic scene graph generation (PSG). We propose a semantic-aware data clustering strategy coupled with a controllable heterogeneous data allocation mechanism, enabling, for the first time, coordinated heterogeneity modeling across multiple semantic elements (objects, relations, attributes). Furthermore, we design an end-to-end federated PSG framework and a robust optimization algorithm. Extensive experiments reveal systematic performance degradation of state-of-the-art PSG models under federated settings, and demonstrate that our approach significantly improves convergence speed and generalization capability.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning (FL) has recently garnered attention as a data-decentralized training framework that enables the learning of deep models from locally distributed samples while keeping data privacy. Built upon the framework, immense efforts have been made to establish FL benchmarks, which provide rigorous evaluation settings that control data heterogeneity across clients. Prior efforts have mainly focused on handling relatively simple classification tasks, where each sample is annotated with a one-hot label, such as MNIST, CIFAR, LEAF benchmark, etc. However, little attention has been paid to demonstrating an FL benchmark that handles complicated semantics, where each sample encompasses diverse semantic information from multiple labels, such as Panoptic Scene Graph Generation (PSG) with objects, subjects, and relations between them. Because the existing benchmark is designed to distribute data in a narrow view of a single semantic, e.g., a one-hot label, managing the complicated semantic heterogeneity across clients when formalizing FL benchmarks is non-trivial. In this paper, we propose a benchmark process to establish an FL benchmark with controllable semantic heterogeneity across clients: two key steps are i) data clustering with semantics and ii) data distributing via controllable semantic heterogeneity across clients. As a proof of concept, we first construct a federated PSG benchmark, demonstrating the efficacy of the existing PSG methods in an FL setting with controllable semantic heterogeneity of scene graphs. We also present the effectiveness of our benchmark by applying robust federated learning algorithms to data heterogeneity to show increased performance. Our code is available at https://github.com/Seung-B/FL-PSG.

Problem

• Establishing FL benchmark for complex semantic datasets
• Managing semantic heterogeneity across clients in FL
• Enabling FL evaluation for multi-semantic vision tasks

Innovation

• Data clustering with semantics for heterogeneity
• Data distributing via controllable semantic heterogeneity
• First FL benchmark for multi-semantic vision tasks