InLINE: Inner-Layer Information Exchange for Multi-task Learning on Heterogeneous Graphs

2024-10-29

arXiv.org

AI Summary

To address the challenges of extremely sparse positive labels in logistics customer expansion on heterogeneous graphs and negative transfer in multi-task learning (MTL) caused by conflating task-shared and task-specific structural patterns, this paper proposes an intra-layer fine-grained information exchange mechanism. Unlike conventional coarse-grained cross-task interaction relying on output embeddings, our method introduces a novel structural disentanglement expert network coupled with a disentanglement gating mechanism, enabling task-directed allocation of relation-level features. Integrated within a heterogeneous graph neural network and a joint MTL optimization framework, the approach achieves a 6.3% improvement in Macro F1 on the DBLP dataset and a 3.6% gain in AUC on an industrial-scale logistics dataset. It effectively mitigates task performance collapse, offering a new paradigm for heterogeneous graph MTL that is both interpretable and high-performing.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Knowledge Representation and Reasoning: Description Logics
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Bridging structured and unstructured data

Abstract

Heterogeneous graph is an important structure for modeling complex relational data in real-world scenarios and usually involves various node prediction tasks within a single graph. Training these tasks separately may neglect beneficial information sharing, hence a preferred way is to learn several tasks in a same model by Multi-Task Learning (MTL). However, MTL introduces the issue of negative transfer, where the training of different tasks interferes with each other as they may focus on different information from the data, resulting in suboptimal performance. To solve the issue, existing MTL methods use separate backbones for each task, then selectively exchange beneficial features through interactions among the output embeddings from each layer of different backbones, which we refer to as outer-layer exchange. However, the negative transfer in heterogeneous graphs arises not simply from the varying importance of an individual node feature across tasks, but also from the varying importance of inter-relation between two nodes across tasks. These inter-relations are entangled in the output embedding, making it difficult for existing methods to discriminate beneficial information from the embedding. To address this challenge, we propose the Inner-Layer Information Exchange (InLINE) model that facilitate fine-grained information exchanges within each graph layer rather than through output embeddings. Specifically, InLINE consists of (1) Structure Disentangled Experts for layer-wise structure disentanglement, (2) Structure Disentangled Gates for assigning disentangled information to different tasks. Evaluations on two public datasets and a large industry dataset show that our model effectively alleviates the significant performance drop on specific tasks caused by negative transfer, improving Macro F1 by 6.3% on DBLP dataset and AUC by 3.6% on the industry dataset compared to SoA methods.

Problem

• Addresses extreme positive label sparsity in customer expansion
• Enhances multi-task learning by sharing structural patterns
• Improves heterogeneous graph neural networks for logistics

Innovation

• Hierarchical structure sharing for multi-task GNNs
• Stage-wise sharing mechanisms for heterogeneous graphs
• Structure-aware framework improves customer expansion accuracy