GitHub is uniquely positioned to lead the industry through this transition. We have direct feedback and deep insight into real production workflows from millions of developers, and the scale to build evaluation systems that truly reflect developer success. We’re looking for a Principal Technical Program Manager to help us build the future of AI evaluation. The Applied Science team for GitHub Copilot sits at the intersection of frontier AI research and the world's largest developer platform. We ship AI-powered experiences (ex: code completion, code review, coding agents) used by millions of professional developers every day. As a member of the team, you will help lead GitHub Copilot's AI evaluation strategy end-to-end — from benchmark design and lifecycle governance, through evaluation infrastructure and internal adoption, to community engagement and public transparency. You are the person who ensures that every model swap, product harness, and feature launch is measured against what actually matters to developers — and that the world can see the results.

Partner with Applied Science researchers to translate cutting-edge evaluation research into production systems: adaptive testing (IRT), agent-centric co-evolution, adversarial benchmarking, and telemetry-driven benchmark generation.

Lead the deprecation of saturated benchmarks and design their next-generation replacements — including procedurally-generated code evaluations that can't be memorized and adaptive testing systems that skip trivial questions for frontier models.

Build GitHub's community benchmark submission program — enabling external researchers, enterprises, and open-source developers to contribute domain-specific evaluations — and publish GitHub's first external benchmark transparency reports showing how models perform on real developer workflows.

Design and operationalize multi-tier evaluation frameworks — from fast automated regression suites and LLM-as-judge systems, through expert human evaluation, to production A/B testing — so teams can iterate in hours, not weeks.

Design feedback-to-benchmark pipelines that convert thumbs-down signals, user frustrations, and support tickets into candidate regression tests — systematizing informal practices into scalable, automated systems.

Establish evaluation as a first-class discipline across GitHub Copilot — creating the rituals, dashboards, and communication cadences that make evaluation results accessible and actionable for every team.