About the job
We are training some of the largest models in the world on the latest hardware across multiple environments. To do this reliably at xAI’s pace, we need engineers who have battle-tested experience keeping massive distributed infrastructure up and running 24/7, including on-prem and cloud-based infrastructure. This is a joint xAI/X role: you will own 24×7 reliability for the world’s largest GPU training superclusters and one of the highest-QPS production systems on the planet (X). You will own the availability, performance, and evolution of xAI’s core compute, storage, and networking infrastructure. This is not an ops-only role — strong coding is a hard requirement. You will design, implement, and ship systems software, automation, and tooling in Python and/or Rust that directly impact training throughput and cluster utilization. You will be expected to participate in a team on-call rotation and to contribute to ushering xAI into the next generation of infrastructure management across multiple data centers and cloud environments.
Responsibilities
Define and execute the technical strategy for infrastructure reliability and scalability
Build and maintain the automation, observability, and control planes that keep multi-datacenter, hybrid cloud/on-prem environments healthy
Lead incident response, deep-dive root cause analysis, and post-mortems that drive real fixes
Identify, instrument, and eliminate systemic failure patterns (capacity, network, hardware, storage, software)
Design and implement high-leverage systems software (daemons, controllers, schedulers, etc.) in Python and Rust.
Push the state of the art in large-scale GPU cluster operations and AI workload reliability
Qualifications
Minimum
5+ years shipping production software and/or operating distributed infrastructure at scale
Expert-level knowledge of Linux systems, TCP/IP networking, and systems programming
Strong coding skills with proven production experience in Rust (strongly preferred) and at least one of Python, Go, or C++.
Deep experience with large-scale distributed systems in on-prem and cloud environments (GCP experience a plus)
Hands-on expertise with container orchestration (Kubernetes, Borg-class systems, or custom schedulers), container runtimes, and infrastructure-as-code (Puppet/Chef/Ansible/Terraform)
Intimate understanding of common failure modes in distributed systems and how to mitigate them (blast radius control, failure domains, canaries, chaos engineering, etc.)
Track record of participating in (or building) effective on-call rotations in high-stakes environments
Bachelor’s degree in Computer Science, Electrical Engineering, or equivalent real-world experience
Preferred
Significant contributions to large-scale GPU clusters or AI/ML infrastructure
Experience in on-call rotations and incident response in high-stakes environments.
Strong problem-solving skills and ability to thrive in a fast-paced, ambiguous setting.
Experience with high-performance networking (RDMA, RoCE, Infiniband) and low level configuration (eBPG, xdp, io_uring)
Comfortable with deployment, support, monitoring, administration, and troubleshooting across on-prem, cloud and hybrid infrastructures.