About the job
Join Apple Services Engineering to build the next generation of AI evaluation systems. We are seeking a staff machine learning platform engineer to lead the architectural design and development of the high availability services and internal tools powering self-service evaluation at scale. You will partner with researchers to operationalize their innovations, transforming complex workflows into intuitive, developer-first platforms. We are looking for builders who thrive in the ambiguity of new initiatives and are passionate about creating scalable infrastructure.
Responsibilities
System Design & Implementation: Lead the architecture for the core evaluation engine and distributed services, designing, coding, and shipping high-quality Python services. Own the technical direction for platform abstractions that will scale across Apple Services Engineering.
Technical Leadership & Collaboration: Mentor engineers across the team, conduct code reviews, and drive technical decision-making. Be the person others look to when facing the hardest engineering problems, and foster a culture of technical excellence and rapid delivery through example and collaboration.
Operationalizing Science: Lead the technical partnership with scientists to translate novel metrics, judge prompts, and scoring algorithms into scalable, production-grade services. Define the frameworks and architectural patterns to evaluate not just simple responses, but also multi-turn agent trajectories and tool usage.
System Integration: Serve as the primary technical bridge between the research organization and the broader engineering ecosystem, driving how our tools integrate seamlessly with existing ML infrastructure and developer workflows.
Engineering Rigor: Champion the software development lifecycle (SDLC) for the team, defining the standards for comprehensive automated testing (CI/CD), and instrumenting monitoring to ensure high availability and reliability.
Qualifications
Minimum
8+ years of hands-on software engineering experience, with a track record of owning the technical direction of a platform or infrastructure domain.
Strong proficiency in the Python ecosystem (e.g., FastAPI, Pydantic, Pandas). You write production-grade code and lead architectural discussions on day one.
Customer Obsession & Product Thinking: You have owned the technical roadmap for an internal platform, presented it to senior stakeholders, and shipped against it. You independently translate vague requirements from other teams into concrete engineering specifications and platform roadmaps.
Demonstrated experience leading technical partnerships with Data Scientists or Researchers: You have taken research code and shipped it as a production service and built the abstractions, testing frameworks, and deployment pipelines that made the next handoff faster than the last..
Strong expertise in API Design & Platform Infrastructure: You have designed and owned APIs and SDKs that other developers rely on, with a focus on versioning, backward compatibility, and developer experience at scale.
Operational excellence background: You have architected and owned CI/CD pipelines, containerization (Docker/Kubernetes), and monitoring (Datadog/Prometheus) for production services, and have been accountable for their reliability.
Bachelors in Computer Science or related field, Masters preferred.
Preferred
Deep familiarity with AI Evaluation Frameworks: You have built, extended, or contributed to modern evaluation tools like DeepEval, Ragas, TruLens, or LangSmith. You understand how to implement and scale model-based evaluation workflows across a large organization.
Evaluation Service Deployment: Own the deployment, scaling, and operational health of evaluation services in production - including high-throughput evaluation job orchestration (queueing, prioritization, concurrency, auto-scaling), and defining SLAs for evaluation pipeline latency and availability.
Observability & Reliability: Experience instrumenting production ML evaluation pipelines including tracking evaluation job throughput, queue depth, judge model latency SLAs, scoring drift over time, and failure modes specific to non-deterministic LLM-based evaluation workflows.
Deep understanding of Generative AI & Agents: You understand the engineering challenges of relying on LLMs and Agents as software components—specifically managing token economics, handling rate limits, and evaluating non-deterministic, multi-step reasoning capabilities. You have built production systems that depend on these components and have solved these problems at scale.
Builder Experience: You have thrived in startup-like environments, navigating high ambiguity to deliver complex technical roadmaps from scratch.