Trustworthy AI Software Engineers

2026-02-06

AI Summary

This study addresses the challenge of trustworthiness in AI software engineers operating within human-AI collaborative software engineering teams, where current understandings of trustworthiness remain ambiguous and suffer from a measurement gap in evaluation. To bridge this gap, the work proposes a multidimensional trustworthiness framework centered on four pillars: technical quality, transparency and accountability, cognitive humility, and alignment with social ethics. Integrating insights from software engineering theory, AI agent research, and ethical design methodologies, the study develops a conceptual model and evaluative perspective for trustworthy AI software engineers. This contribution provides a theoretical foundation for the design, assessment, and governance of AI software engineers, advocating for an ethics-by-design paradigm that fosters well-calibrated trust in human-AI collaboration.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Responsible Web: Fairness, accountability, transparency and ethics of web technologies
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

With the rapid rise of AI coding agents, the fundamental premise of what it means to be a software engineer is in question. In this vision paper, we re-examine what it means for an AI agent to be considered a software engineer and then critically think about what makes such an agent trustworthy. \textit{Grounded} in established definitions of software engineering (SE) and informed by recent research on agentic AI systems, we conceptualise AI software engineers as participants in human-AI SE teams composed of human software engineers and AI models and tools, and we distinguish trustworthiness as a key property of these systems and actors rather than a subjective human attitude. Based on historical perspectives and emerging visions, we identify key dimensions that contribute to the trustworthiness of AI software engineers, spanning technical quality, transparency and accountability, epistemic humility, and societal and ethical alignment. We further discuss how trustworthiness can be evaluated and demonstrated, highlighting a fundamental trust measurement gap: not everything that matters for trust can be easily measured. Finally, we outline implications for the design, evaluation, and governance of AI SE systems, advocating for an ethics-by-design approach to enable appropriate trust in future human-AI SE teams.

Problem

• Trustworthy AI
• AI Software Engineer
• Human-AI Collaboration
• Software Engineering
• Trustworthiness

Innovation

• Trustworthy AI
• AI Software Engineer
• Human-AI Collaboration
• Ethics-by-Design
• Epistemic Humility