Topology-Aware Activation Functions in Neural Networks

2025-07-17

AI Summary

Traditional activation functions (e.g., ReLU) struggle to capture the topological structure of low-dimensional data manifolds, limiting neural networks’ representational capacity on geometrically complex data. To address this, we propose two topology-aware activation functions: SmoothSplit—a differentiable activation with explicit topological “cutting” capability—and ParametricSplit—a learnable variant that adaptively modulates splitting strength. These are the first activations to embed explicit topological operations directly into the nonlinearity. Built upon the ReLU framework, both require no architectural modifications and support end-to-end training. Experiments on synthetic manifolds and real-world low-dimensional datasets demonstrate that ParametricSplit significantly improves classification and reconstruction performance—outperforming ReLU, Swish, and other baselines in low-dimensional settings and remaining competitive in high-dimensional tasks. The implementation is publicly available.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This study explores novel activation functions that enhance the ability of neural networks to manipulate data topology during training. Building on the limitations of traditional activation functions like $mathrm{ReLU}$, we propose $mathrm{SmoothSplit}$ and $mathrm{ParametricSplit}$, which introduce topology "cutting" capabilities. These functions enable networks to transform complex data manifolds effectively, improving performance in scenarios with low-dimensional layers. Through experiments on synthetic and real-world datasets, we demonstrate that $mathrm{ParametricSplit}$ outperforms traditional activations in low-dimensional settings while maintaining competitive performance in higher-dimensional ones. Our findings highlight the potential of topology-aware activation functions in advancing neural network architectures. The code is available via https://github.com/Snopoff/Topology-Aware-Activations.

Problem

• Enhancing neural networks' ability to manipulate data topology
• Overcoming limitations of traditional activation functions like ReLU
• Improving performance in low-dimensional data scenarios

Innovation

• Introducing topology-aware activation functions
• Proposing SmoothSplit and ParametricSplit functions
• Enhancing neural networks' data topology manipulation