Efficient Decision Trees for Tensor Regressions

2024-08-04

arXiv.org

AI Summary

This paper addresses scalar regression and tensor-to-tensor regression tasks with tensor-valued inputs. We propose the Tensor-Input Tree (TT) model—a novel decision tree architecture natively supporting tensor inputs. TT is the first additive tree ensemble framework extended to tensor–tensor regression and introduces efficient randomized and deterministic splitting algorithms that jointly ensure theoretical interpretability and computational efficiency. A rigorous theoretical analysis establishes a generalization error bound for TT. Empirical evaluation on both synthetic and real-world datasets demonstrates that TT achieves 2–4 orders of magnitude faster training than tensor-input Gaussian processes and other baselines, while maintaining comparable or superior prediction accuracy. The core contributions are: (i) a tensor-adapted decision tree structure; (ii) an additive ensemble framework for tensor regression; and (iii) scalable, efficient learning algorithms with provable guarantees.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We proposed the tensor-input tree (TT) method for scalar-on-tensor and tensor-on-tensor regression problems. We first address scalar-on-tensor problem by proposing scalar-output regression tree models whose input variable are tensors (i.e., multi-way arrays). We devised and implemented fast randomized and deterministic algorithms for efficient fitting of scalar-on-tensor trees, making TT competitive against tensor-input GP models. Based on scalar-on-tensor tree models, we extend our method to tensor-on-tensor problems using additive tree ensemble approaches. Theoretical justification and extensive experiments on real and synthetic datasets are provided to illustrate the performance of TT.

Problem

• Develops tensor-input trees for scalar-on-tensor regression
• Extends method to tensor-on-tensor regression using ensembles
• Provides efficient algorithms and theoretical performance analysis

Innovation

• Tensor-input tree method for regression
• Fast randomized deterministic algorithms
• Additive tree ensemble approaches