A Composite Activation Function for Learning Stable Binary Representations

2026-05-12

AI Summary

This work addresses the challenge of training binary neural networks with the non-differentiable Heaviside activation function, which impedes stable optimization via gradient-based methods. The authors propose a novel smooth approximation, termed HTAF, constructed through a composite of sigmoid and hyperbolic tangent functions. HTAF exhibits both high gradient magnitude near zero and slow tail decay, enabling effective optimization with standard gradient descent. Notably, HTAF is employed for the first time to build an interpretable Implicit Concept Bottleneck Model (ICBM), facilitating stable discretization in spiking neural networks, binary networks, and deep Heaviside networks. Experiments demonstrate that the proposed approach achieves competitive or superior predictive performance across multiple architectures and image datasets while significantly enhancing the stability of discrete feature learning compared to existing standard models.

Abstract

Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.

Problem

• binary representations
• Heaviside activation
• non-differentiability
• gradient-based optimization
• stable training

Innovation

• composite activation function
• binary representation
• gradient stability
• Heaviside approximation
• interpretable models