The Post Double LASSO for Efficiency Analysis

2025-05-20

AI Summary

To address the challenge of jointly estimating frontier functions and inefficiency terms in high-dimensional “wide data” settings—where conventional stochastic frontier analysis (SFA) methods suffer from model misspecification bias—we propose the first integration of post-double LASSO into SFA. Our approach constructs Neyman-orthogonal moment conditions, enabling doubly robust estimation of both the production frontier and the inefficiency term. This framework mitigates endogeneity and selection bias under high-dimensional sparse designs, while ensuring both model selection consistency and asymptotic normality of parameter estimates. Simulation studies and empirical applications demonstrate that our method significantly improves the accuracy, robustness, and statistical reliability of efficiency measurement compared to standard SFA and single-stage LASSO-based approaches. The proposed estimator thus provides a generalizable, high-dimensional econometric solution for efficiency evaluation in big-data contexts.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Big data and machine learning methods have become commonplace across economic milieus. One area that has not seen as much attention to these important topics yet is efficiency analysis. We show how the availability of big (wide) data can actually make detection of inefficiency more challenging. We then show how machine learning methods can be leveraged to adequately estimate the primitives of the frontier itself as well as inefficiency using the `post double LASSO' by deriving Neyman orthogonal moment conditions for this problem. Finally, an application is presented to illustrate key differences of the post-double LASSO compared to other approaches.

Problem

• Addresses inefficiency detection challenges in big data
• Uses machine learning to estimate frontier primitives
• Introduces post-double LASSO for efficiency analysis

Innovation

• Uses big data for efficiency analysis challenges
• Applies machine learning to estimate frontier primitives
• Introduces post double LASSO with orthogonal conditions