Identifying and Mitigating Gender Cues in Academic Recommendation Letters: An Interpretability Case Study

2026-04-14

AI Summary

This study addresses the persistence of implicit gendered language in anonymized letters of recommendation, which can inadvertently reveal an applicant’s gender and compromise admissions fairness. Leveraging DistilBERT, RoBERTa, and Llama 2, the authors classify gender in residency recommendation letters and employ TF-IDF combined with SHAP to identify key implicit gender cues. Building on these insights, they propose a fairness framework integrating textual auditing and model-based intervention to generate more gender-neutral revisions by removing such cues. Experimental results show that even in anonymized letters, models achieve 68% gender classification accuracy; after cue removal, accuracy drops by 5.5% and macro F1 by 2.7%, yet performance remains above random chance, underscoring the tenacity of implicit bias and its potential influence on high-stakes decision-making.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Letters of recommendation (LoRs) can carry patterns of implicitly gendered language that can inadvertently influence downstream decisions, e.g. in hiring and admissions. In this work, we investigate the extent to which Transformer-based encoder models as well as Large Language Models (LLMs) can infer the gender of applicants in academic LoRs submitted to an U.S. medical-residency program after explicit identifiers like names and pronouns are de-gendered. While using three models (DistilBERT, RoBERTa, and Llama 2) to classify the gender of anonymized and de-gendered LoRs, significant gender leakage was observed as evident from up to 68% classification accuracy. Text interpretation methods, like TF-IDF and SHAP, demonstrate that certain linguistic patterns are strong proxies for gender, e.g. "emotional'' and "humanitarian'' are commonly associated with LoRs from female applicants. As an experiment in creating truly gender-neutral LoRs, these implicit gender cues were remove resulting in a drop of up to 5.5% accuracy and 2.7% macro $F_1$ score on re-training the classifiers. However, applicant gender prediction still remains better than chance. In this case study, our findings highlight that 1) LoRs contain gender-identifying cues that are hard to remove and may activate bias in decision-making and 2) while our technical framework may be a concrete step toward fairer academic and professional evaluations, future work is needed to interrogate the role that gender plays in LoR review. Taken together, our findings motivate upstream auditing of evaluative text in real-world academic letters of recommendation as a necessary complement to model-level fairness interventions.

Problem

• gender bias
• letters of recommendation
• gender leakage
• implicit gender cues
• fairness in evaluation

Innovation

• gender bias
• recommendation letters
• interpretability
• large language models
• fairness auditing