Function Basis Encoding of Numerical Features in Factorization Machines

2023-05-23

AI Summary

Factorization Machines (FMs) struggle to model numerical features efficiently and expressively. Method: This paper proposes a theory-driven, piecewise function approximation-based encoding method: numerical features are mapped to differentiable vectors of B-spline basis function evaluations and integrated into the FM architecture, enabling flexible inter-segment coefficients, strong representational capacity, and low computational overhead. Contribution/Results: It is the first work to reformulate FM’s numerical feature processing from a functional approximation perspective, supporting end-to-end differentiable training. Extensive experiments on multiple public and synthetic datasets demonstrate significant improvements in prediction accuracy. Online A/B testing in a production advertising system shows substantial gains in CTR and conversion rate, while training and inference latency remain virtually unchanged.

Abstract

Factorization machine (FM) variants are widely used for large scale real-time content recommendation systems, since they offer an excellent balance between model accuracy and low computational costs for training and inference. These systems are trained on tabular data with both numerical and categorical columns. Incorporating numerical columns poses a challenge, and they are typically incorporated using a scalar transformation or binning, which can be either learned or chosen a-priori. In this work, we provide a systematic and theoretically-justified way to incorporate numerical features into FM variants by encoding them into a vector of function values for a set of functions of one's choice. We view factorization machines as approximators of segmentized functions, namely, functions from a field's value to the real numbers, assuming the remaining fields are assigned some given constants, which we refer to as the segment. From this perspective, we show that our technique yields a model that learns segmentized functions of the numerical feature spanned by the set of functions of one's choice, namely, the spanning coefficients vary between segments. Hence, to improve model accuracy we advocate the use of functions known to have strong approximation power, and offer the B-Spline basis due to its well-known approximation power, availability in software libraries, and efficiency. Our technique preserves fast training and inference, and requires only a small modification of the computational graph of an FM model. Therefore, it is easy to incorporate into an existing system to improve its performance. Finally, we back our claims with a set of experiments, including synthetic, performance evaluation on several data-sets, and an A/B test on a real online advertising system which shows improved performance.

Problem

• Factorization Machines
• Numerical Features
• Real-time Recommendation Systems

Innovation

• B-spline basis functions
• Factorization Machines
• Digital feature transformation