Supervised Quadratic Feature Analysis: An Information Geometry Approach to Dimensionality Reduction

2025-01-31

AI Summary

Existing linear supervised dimensionality reduction methods struggle to balance discriminative power and computational efficiency: Linear Discriminant Analysis (LDA) models only class means and is constrained by the linear separability assumption, whereas nonlinear methods supporting quadratic discrimination incur high computational cost and lack interpretability. This paper proposes Symmetric Positive-Definite Quadratic Feature Analysis (SQFA), the first linear dimensionality reduction framework that jointly optimizes class-conditional first- and second-order statistics within an information-geometric setting on the SPD manifold—explicitly encoding both intra-class and inter-class covariance structures. Theoretically, SQFA is proven equivalent to Quadratic Discriminant Analysis (QDA), thereby bridging the expressiveness–efficiency gap between LDA and metric learning. Experiments demonstrate that SQFA achieves quadratic classification performance significantly surpassing LDA—approaching kernel-based methods—while retaining LDA-level computational complexity, thus offering both interpretability and practicality.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Natural Language Processing: Question Answering
• Computer Vision: Representation Learning for Vision

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Supervised dimensionality reduction aims to map labeled data to a low-dimensional feature space while maximizing class discriminability. Despite the availability of methods for learning complex non-linear features (e.g. Deep Learning), there is an enduring demand for dimensionality reduction methods that learn linear features due to their interpretability, low computational cost, and broad applicability. However, there is a gap between methods that optimize linear separability (e.g. LDA), and more flexible but computationally expensive methods that optimize over arbitrary class boundaries (e.g. metric-learning methods). Here, we present Supervised Quadratic Feature Analysis (SQFA), a dimensionality reduction method for learning linear features that maximize the differences between class-conditional first- and second-order statistics, which allow for quadratic discrimination. SQFA exploits the information geometry of second-order statistics in the symmetric positive definite manifold. We show that SQFA features support quadratic discriminability in real-world problems. We also provide a theoretical link, based on information geometry, between SQFA and the Quadratic Discriminant Analysis (QDA) classifier.

Problem

• Linear Dimensionality Reduction
• Complex Boundary Handling
• Computational Efficiency

Innovation

• SQFA
• Information Geometry
• Quadratic Discriminant Analysis