BOLD: Boolean Logic Deep Learning

2024-05-25

Neural Information Processing Systems

AI Summary

Deep learning training incurs high energy consumption, primarily due to floating-point arithmetic and frequent data movement. Method: This work introduces the first end-to-end training paradigm operating entirely in the Boolean domain—eliminating gradient descent and floating-point computation—and establishes a trainable deep network grounded in Boolean variational principles. It integrates Boolean logic optimization, discrete-domain training algorithms, and hardware-software co-design, underpinned by a unified energy-efficiency evaluation framework jointly considering architecture, memory hierarchy, and dataflow. Contribution/Results: The approach achieves full-precision baseline accuracy on ImageNet; surpasses state-of-the-art methods in semantic segmentation; delivers competitive performance in image super-resolution and Transformer-based language understanding; and significantly reduces both training and inference energy consumption. This work provides the first systematic, Boolean-native solution for energy-efficient AI training.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Deep learning is computationally intensive, with significant efforts focused on reducing arithmetic complexity, particularly regarding energy consumption dominated by data movement. While existing literature emphasizes inference, training is considerably more resource-intensive. This paper proposes a novel mathematical principle by introducing the notion of Boolean variation such that neurons made of Boolean weights and inputs can be trained -- for the first time -- efficiently in Boolean domain using Boolean logic instead of gradient descent and real arithmetic. We explore its convergence, conduct extensively experimental benchmarking, and provide consistent complexity evaluation by considering chip architecture, memory hierarchy, dataflow, and arithmetic precision. Our approach achieves baseline full-precision accuracy in ImageNet classification and surpasses state-of-the-art results in semantic segmentation, with notable performance in image super-resolution, and natural language understanding with transformer-based models. Moreover, it significantly reduces energy consumption during both training and inference.

Problem

• Reducing energy consumption in deep learning via Boolean logic
• Training Boolean-based neurons efficiently without gradient descent
• Achieving high accuracy while minimizing computational resource use

Innovation

• Boolean weights and inputs for neurons
• Training in Boolean domain using Boolean logic
• Reduces energy consumption in training and inference