Trading off performance and human oversight in algorithmic policy: evidence from Danish college admissions

2024-11-22

arXiv.org

AI Summary

This study addresses the trade-off between algorithmic decision-making and human oversight in Danish university admissions, aiming to enhance both admission efficiency and educational equity. Method: We propose a deep sequential modeling framework (LSTM/Transformer) that predicts students’ degree completion probability using pre-enrollment multi-source data, integrated with fairness-constrained learning and counterfactual sensitivity analysis. Contribution/Results: To our knowledge, this is the first systematic quantification—within a real-world higher education policy context—of the triadic trade-off among predictive performance, group fairness, and interpretability across AI models, human judgment, and conventional metrics (e.g., GPA). We uncover student–program matching patterns to inform welfare-oriented public policy design. Empirical results demonstrate significant improvements in prediction accuracy and group-level fairness; policy simulations indicate substantial economic gains under a 10% enrollment reduction when using AI-assisted decisions—yet highlight risks of reduced transparency and diminished human oversight.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Student dropout is a significant concern for educational institutions due to its social and economic impact, driving the need for risk prediction systems to identify at-risk students before enrollment. We explore the accuracy of such systems in the context of higher education by predicting degree completion before admission, with potential applications for prioritizing admissions decisions. Using a large-scale dataset from Danish higher education admissions, we demonstrate that advanced sequential AI models offer more precise and fair predictions compared to current practices that rely on either high school grade point averages or human judgment. These models not only improve accuracy but also outperform simpler models, even when the simpler models use protected sociodemographic attributes. Importantly, our predictions reveal how certain student profiles are better matched with specific programs and fields, suggesting potential efficiency and welfare gains in public policy. We estimate that even the use of simple AI models to guide admissions decisions, particularly in response to a newly implemented nationwide policy reducing admissions by 10 percent, could yield significant economic benefits. However, this improvement would come at the cost of reduced human oversight and lower transparency. Our findings underscore both the potential and challenges of incorporating advanced AI into educational policymaking.

Problem

• Predicting student dropout risk before enrollment using AI models
• Improving accuracy and fairness in college admissions decisions
• Balancing AI benefits with reduced human oversight transparency

Innovation

• Advanced sequential AI models for precise predictions
• Better student-program matching using AI
• Economic benefits from simple AI models