Boosted Trees on a Diet: Compact Models for Resource-Constrained Devices

2025-10-30

AI Summary

To address the deployment challenges of gradient-boosted decision trees (GBDTs) on resource-constrained IoT devices—characterized by limited computational capacity, low power budgets, and minimal memory—the paper proposes a training-time lightweight compression method. Specifically, it introduces an explicit reward mechanism for feature and threshold reuse within frameworks such as LightGBM, jointly optimizing tree structure and memory-efficient parameter layout. This approach explicitly constrains parameter redundancy during training, substantially reducing storage overhead. Experiments across multiple benchmark datasets demonstrate 4×–16× model compression ratios, with negligible accuracy degradation (<1%), drastic reductions in memory footprint, and significant improvements in inference latency and energy efficiency. The method thus enables autonomous, efficient, and sustainable machine learning inference at the edge.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Deploying machine learning models on compute-constrained devices has become a key building block of modern IoT applications. In this work, we present a compression scheme for boosted decision trees, addressing the growing need for lightweight machine learning models. Specifically, we provide techniques for training compact boosted decision tree ensembles that exhibit a reduced memory footprint by rewarding, among other things, the reuse of features and thresholds during training. Our experimental evaluation shows that models achieved the same performance with a compression ratio of 4-16x compared to LightGBM models using an adapted training process and an alternative memory layout. Once deployed, the corresponding IoT devices can operate independently of constant communication or external energy supply, and, thus, autonomously, requiring only minimal computing power and energy. This capability opens the door to a wide range of IoT applications, including remote monitoring, edge analytics, and real-time decision making in isolated or power-limited environments.

Problem

• Compressing boosted trees for resource-constrained IoT devices
• Training compact ensembles with reduced memory footprint
• Enabling autonomous operation with minimal computing power

Innovation

• Compressed boosted decision trees for lightweight models
• Feature and threshold reuse reduces memory footprint
• Compact ensembles enable autonomous IoT device operation