As a Principal Machine Learning Engineer focused on distributed vLLM infrastructure in the llm-d project, you will collaborate with our team to tackle the most pressing challenges in scalable inference systems and Kubernetes-native deployments. Your work with distributed systems and cloud infrastructure will directly impact enterprise AI deployments. You would be joining the core team behind 2025's most popular open source project on GitHub. If you want to solve challenging technical problems in distributed systems and cloud-native infrastructure the open-source way, this is the role for you.

Develop and maintain distributed inference infrastructure leveraging Kubernetes APIs, operators, and the Gateway Inference Extension API for scalable LLM deployments.

Create system components in Go and/or Rust to integrate with the vLLM project and manage distributed inference workloads.

Design and implement KV cache-aware routing and scoring algorithms to optimize memory utilization and request distribution in large-scale inference deployments.

Enhance the resource utilization, fault tolerance, and stability of the inference stack.

Contribute to the design, development, and testing of various inference optimization algorithms.

Actively participate in technical design discussions and propose innovative solutions to complex challenges.

Provide timely and constructive code reviews.

Mentor and guide fellow engineers, fostering a culture of continuous learning and innovation.