GZSL-MoE: Apprentissage G{é}n{é}ralis{é} Z{é}ro-Shot bas{é} sur le M{é}lange d'Experts pour la Segmentation S{é}mantique de Nuages de Points 3DAppliqu{é} {à} un Jeu de Donn{é}es d'Environnement de Collaboration Humain-Robot

2025-09-23

AI Summary

To address the generalized zero-shot learning (GZSL) challenge in 3D point cloud semantic segmentation—where training data lacks unseen classes and adaptation to dynamic human-robot collaboration scenarios is difficult—this paper proposes a novel method integrating generative modeling with a Mixture of Experts (MoE) mechanism. For the first time, MoE architectures are embedded into both the generator and discriminator of Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) to jointly model feature distributions of seen and unseen classes. Additionally, pre-trained KPConv features are leveraged to enhance point cloud representation robustness. Evaluated on the COVERED dataset, the method achieves significant improvements in both seen and unseen class segmentation performance, outperforming state-of-the-art approaches in H-score and harmonic mean metrics. These results validate its effectiveness and generalization capability for recognizing previously unknown objects in real-world collaborative settings.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generative Zero-Shot Learning approach (GZSL) has demonstrated significant potential in 3D point cloud semantic segmentation tasks. GZSL leverages generative models like GANs or VAEs to synthesize realistic features (real features) of unseen classes. This allows the model to label unseen classes during testing, despite being trained only on seen classes. In this context, we introduce the Generalized Zero-Shot Learning based-upon Mixture-of-Experts (GZSL-MoE) model. This model incorporates Mixture-of-Experts layers (MoE) to generate fake features that closely resemble real features extracted using a pre-trained KPConv (Kernel Point Convolution) model on seen classes. The main contribution of this paper is the integration of Mixture-of-Experts into the Generator and Discriminator components of the Generative Zero-Shot Learning model for 3D point cloud semantic segmentation, applied to the COVERED dataset (CollabOratiVE Robot Environment Dataset) for Human-Robot Collaboration (HRC) environments. By combining the Generative Zero-Shot Learning model with Mixture-of- Experts, GZSL-MoE for 3D point cloud semantic segmentation provides a promising solution for understanding complex 3D environments, especially when comprehensive training data for all object classes is unavailable. The performance evaluation of the GZSL-MoE model highlights its ability to enhance performance on both seen and unseen classes. Keywords Generalized Zero-Shot Learning (GZSL), 3D Point Cloud, 3D Semantic Segmentation, Human-Robot Collaboration, COVERED (CollabOratiVE Robot Environment Dataset), KPConv, Mixture-of Experts

Problem

• Addresses 3D point cloud semantic segmentation with unseen classes
• Integrates Mixture-of-Experts into generative zero-shot learning model
• Enhances semantic segmentation for human-robot collaboration environments

Innovation

• Uses Mixture-of-Experts layers for feature generation
• Combines GZSL with MoE for 3D segmentation
• Applies KPConv for real feature extraction