Explained, yet misunderstood: How AI Literacy shapes HR Managers' interpretation of User Interfaces in Recruiting Recommender Systems

2025-09-08

AI Summary

This study investigates how HR managers’ AI literacy influences their perception and comprehension of eXplainable AI (XAI) in recruitment recommendation systems, revealing literacy-driven cognitive biases and an “explanation effectiveness gap.” Method: An online experiment compared a baseline dashboard against three XAI visualization modalities—feature importance overlays, counterfactual explanations, and model-standard benchmarks—to assess subjective trust and objective understanding. Results: While XAI significantly increased subjective trust, it failed to improve objective comprehension; complex explanations even reduced decision accuracy. Only feature importance overlays enhanced understanding—but exclusively among high-literacy users. Contribution: This work provides the first empirical evidence that AI literacy critically moderates XAI utility. It introduces the “literacy-aligned explanation” framework, advocating for explanation granularity and modality to be calibrated to users’ cognitive capabilities—thereby informing human-centered XAI design and targeted AI literacy development in HR contexts.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

AI-based recommender systems increasingly influence recruitment decisions. Thus, transparency and responsible adoption in Human Resource Management (HRM) are critical. This study examines how HR managers' AI literacy influences their subjective perception and objective understanding of explainable AI (XAI) elements in recruiting recommender dashboards. In an online experiment, 410 German-based HR managers compared baseline dashboards to versions enriched with three XAI styles: important features, counterfactuals, and model criteria. Our results show that the dashboards used in practice do not explain AI results and even keep AI elements opaque. However, while adding XAI features improves subjective perceptions of helpfulness and trust among users with moderate or high AI literacy, it does not increase their objective understanding. It may even reduce accurate understanding, especially with complex explanations. Only overlays of important features significantly aided the interpretations of high-literacy users. Our findings highlight that the benefits of XAI in recruitment depend on users' AI literacy, emphasizing the need for tailored explanation strategies and targeted literacy training in HRM to ensure fair, transparent, and effective adoption of AI.

Problem

• AI literacy's impact on HR managers' UI interpretation
• XAI effectiveness in recruiting recommender systems
• Tailored explanation strategies for HRM adoption

Innovation

• Used three XAI styles in dashboards
• Tested AI literacy impact on perception
• Tailored explanation strategies for HRM