Robust Regression with Students T: The Role of Degrees of Freedom

2026-02-27

AI Summary

This study addresses the sensitivity of linear regression to outliers by modeling the error terms with a Student’s t-distribution, thereby balancing robustness and estimation efficiency. It systematically compares frequentist (adjusted profile likelihood) and Bayesian approaches for estimating the degrees-of-freedom parameter, and benchmarks them against Huber loss and fixed degrees-of-freedom strategies. The findings reveal that precise calibration of the degrees-of-freedom parameter is as critical to the performance of t-distribution-based robust regression as the choice of distribution itself. Experiments on both synthetic and real-world datasets demonstrate that the adjusted profile likelihood approach yields highly accurate regression coefficients, achieving performance nearly equivalent to maximum likelihood estimation under known true degrees of freedom, thus highlighting its advantage in simultaneously ensuring robustness and statistical efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: State Estimation

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Linear regression estimators are known to be sensitive to outliers, and one alternative to obtain a robust and efficient estimator of the regression parameter is to model the error with Student's $t$ distribution. In this article, we compare estimators of the degrees of freedom parameter in the $t$ distribution using frequentist and Bayesian methods, and then study properties of the corresponding estimated regression coefficient. We also include the comparison with some recommended approaches in the literature, including fixing the degrees of freedom and robust regression using the Huber loss. Our extensive simulations on both synthetic and real data demonstrate that estimating the degrees of freedom via the adjusted profile log-likelihood approach yields regression coefficient estimators with high accuracy, performing comparably to the maximum likelihood estimator where the degrees of freedom are fixed at their true values. These findings provide a detailed synthesis of $t$-based robust regression and underscore a key insight: the proper calibration of the degree of freedom is as crucial as the choice of the robust distribution itself for achieving optimal performance.

Problem

• robust regression
• Student's t distribution
• degrees of freedom
• outliers
• regression estimation

Innovation

• Student's t distribution
• degrees of freedom estimation
• robust regression
• adjusted profile likelihood
• outlier resistance