Neural Network Conversion of Machine Learning Pipelines

2026-03-26

AI Summary

This work proposes a systematic method to convert non-neural machine learning pipelines—such as those based on random forests—into neural networks, enabling unified inference and joint optimization. Leveraging knowledge distillation, the approach treats the traditional model as a “teacher” that guides the training of a neural “student” network. The framework further integrates neural architecture search with a random forest–inspired hyperparameter selection strategy to optimize the student model. Notably, this is the first effort to employ an entire non-neural machine learning pipeline as the teacher in knowledge distillation, thereby extending the scope of this technique. Experimental evaluation across 100 OpenML tasks demonstrates that the student networks consistently replicate the performance of their teacher models, confirming the feasibility and effectiveness of the proposed conversion framework.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Transfer learning and knowledge distillation has recently gained a lot of attention in the deep learning community. One transfer approach, the student-teacher learning, has been shown to successfully create ``small'' student neural networks that mimic the performance of a much bigger and more complex ``teacher'' networks. In this paper, we investigate an extension to this approach and transfer from a non-neural-based machine learning pipeline as teacher to a neural network (NN) student, which would allow for joint optimization of the various pipeline components and a single unified inference engine for multiple ML tasks. In particular, we explore replacing the random forest classifier by transfer learning to a student NN. We experimented with various NN topologies on 100 OpenML tasks in which random forest has been one of the best solutions. Our results show that for the majority of the tasks, the student NN can indeed mimic the teacher if one can select the right NN hyper-parameters. We also investigated the use of random forest for selecting the right NN hyper-parameters.

Problem

• knowledge distillation
• transfer learning
• neural network
• random forest
• machine learning pipeline

Innovation

• knowledge distillation
• transfer learning
• neural network conversion
• random forest
• machine learning pipeline