A Kernel Fisher Discriminant Analysis-Based Tree Ensemble Classifier: KFDA Forest

2026-06-27

AI Summary

This work addresses the challenge of simultaneously achieving high classification accuracy and model diversity in nonlinear data by proposing a novel ensemble method that integrates kernel Fisher discriminant analysis (KFDA) with decision trees. The approach applies KFDA on bootstrap samples combined with random feature grouping, leveraging the kernel trick to enhance between-class separability while reducing within-class variance. The resulting discriminant projection directions are explicitly aligned with the splitting axes of decision trees, thereby improving both individual classifier performance and ensemble diversity. Experimental results demonstrate that the proposed method significantly outperforms state-of-the-art ensemble classifiers across multiple real-world datasets from the UCI and KEEL repositories.

Abstract

In general, an ensemble classifier is more accurate than a single classifier. In this study, we propose an ensemble classifier called the kernel Fisher discriminant analysis forest (KFDA Forest), which is a tree-based ensemble method that applies KFDA. To promote diversity, bootstrap is used, and variable sets are randomly divided into K subsets. KFDA is performed on each subset to increase classification accuracy. KFDA maximizes the distance between classes while minimizing the distance within classes. KFDA can also be applied to classification problems in a nonlinear data structure using the kernel trick because it can transform the input space into a kernel feature space, commonly named a rotation, rather than performing a dimensionality reduction. Because new feature axes and KFDA projections are parallel, decision trees are used as a base classifier. To compare the proposed method with existing ensemble methods, we apply these to real datasets from the UCI and KEEL repositories.

Problem

• ensemble classifier
• nonlinear classification
• tree-based method
• class discrimination
• kernel method

Innovation

• Kernel Fisher Discriminant Analysis
• Ensemble Classifier
• Tree-based Method
• Kernel Trick
• Feature Space Transformation