Sparse Hybrid Linear-Morphological Networks

2025-04-12

AI Summary

Training hybrid linear-morphological networks remains challenging due to poor trainability and limited pruning potential. To address this, we propose an embedded linear-morphological architecture: morphological layers—including maxout pooling and fully connected morphological layers—are inserted between linear layers of a CNN backbone, coupled with a sparsity-aware initialization strategy that accelerates convergence and enhances L1 sparsity. This design jointly leverages morphological modeling capabilities and inherent weight sparsity. On the MTAT music tagging task, our method converges faster than ReLU, maxout, and dense max-plus baselines while achieving marginally higher accuracy. On CIFAR-10, it demonstrates robustness and generalizability when replacing standard classification heads. Our core contributions are twofold: (i) the first systematic investigation of morphological layer placement on pruning efficacy, and (ii) a morphology-specific sparse initialization mechanism that significantly improves both model compressibility and training efficiency.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Lexical Semantics and Morphology

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We investigate hybrid linear-morphological networks. Recent studies highlight the inherent affinity of morphological layers to pruning, but also their difficulty in training. We propose a hybrid network structure, wherein morphological layers are inserted between the linear layers of the network, in place of activation functions. We experiment with the following morphological layers: 1) maxout pooling layers (as a special case of a morphological layer), 2) fully connected dense morphological layers, and 3) a novel, sparsely initialized variant of (2). We conduct experiments on the Magna-Tag-A-Tune (music auto-tagging) and CIFAR-10 (image classification) datasets, replacing the linear classification heads of state-of-the-art convolutional network architectures with our proposed network structure for the various morphological layers. We demonstrate that these networks induce sparsity to their linear layers, making them more prunable under L1 unstructured pruning. We also show that on MTAT our proposed sparsely initialized layer achieves slightly better performance than ReLU, maxout, and densely initialized max-plus layers, and exhibits faster initial convergence.

Problem

• Hybrid linear-morphological networks improve training and pruning efficiency
• Morphological layers replace activation functions to enhance sparsity
• Sparse initialization boosts performance and convergence in hybrid networks

Innovation

• Hybrid linear-morphological network structure
• Sparsely initialized morphological layers
• Enhanced pruning via induced sparsity