About the job
As a Principal Engineer on the Walmart Advertising technology team, you will define and lead the test engineering strategy for AI-powered advertising systems — spanning ML model validation, intelligent audience pipelines, real-time bidding infrastructure, and programmatic data platforms. You'll architect test frameworks, embed quality into AI/ML development lifecycles, and operate as a senior technical leader across engineering, data science, and product.
Responsibilities
Architect and own enterprise-scale test automation frameworks for APIs, data pipelines, UI, and services layers — built for maintainability, scale, and CI/CD integration
Drive shift-left quality — embedding automated quality checks at every stage of the development lifecycle, from schema design through production monitoring
Define and enforce quality gates in CI/CD pipelines at an organizational level; own release criteria and go/no-go decisions for major platform changes
Lead data pipeline testing at scale — ETL/ELT validation, schema governance, event-level tracking accuracy, audience segment integrity, and aggregation correctness (BigQuery, Spark, Kafka)
Own test strategy for AI/ML systems — including model validation, data drift detection, feature pipeline integrity, inference correctness, and algorithmic fairness testing for advertising models (lookalike, propensity, audience prediction)
Define model quality gates in ML training and deployment pipelines — validating model outputs against golden datasets, statistical thresholds, and business KPIs before promotion to production
Build automated evaluation frameworks for GenAI and LLM-integrated components — including prompt regression testing, output consistency checks, and hallucination detection pipelines
Partner with data scientists to implement shadow testing, A/B experiment integrity validation, and statistical significance guardrails for algorithmic experiments
Implement performance and chaos engineering for latency-sensitive systems — bidding engines, ad decisioning, and audience lookup at millions of QPS
Represent quality in architecture and design reviews — surfacing testability gaps, observability blind spots, and data integrity risks before they are built in
Author quality frameworks, testing playbooks, and SDET standards adopted across WMX and Growth Tech
Define SLOs and production quality posture — integrating quality signals into observability dashboards and driving proactive incident prevention
Mentor and grow Staff and Senior SDETs; raise the engineering bar through design reviews, code reviews, and hiring
Qualifications
Minimum
8+ years in quality/test engineering; 3+ years at Staff or Principal SDET level in distributed, large-scale systems
Proven expertise testing AI/ML systems — model validation, feature store integrity, inference pipeline testing, or data quality for ML training datasets
Expert-level experience designing test automation frameworks for APIs, data, and services layers (Pytest, Selenium, Appium, WebDriver, or equivalent)
Deep expertise in data pipeline quality — ETL/ELT validation, schema governance, data completeness, transformation correctness at scale (BigQuery, Hive, Spark)
Strong experience testing event-driven and async systems — Kafka pipelines, REST APIs, webhook integrations
SQL as a first-class testing tool — writing complex queries against large analytical datasets to validate data correctness
Experience enforcing quality gates in CI/CD at organizational scale
Preferred
Experience with LLM/GenAI quality — prompt regression testing, output evaluation frameworks, hallucination detection, or RAG pipeline validation
Expertise in programmatic advertising quality — RTB auction integrity, DSP/SSP integrations, ad serving correctness, measurement/attribution validation
Hands-on with GCP data platform: BigQuery, Dataflow, Pub/Sub, Vertex AI
Experience in privacy and compliance testing — GDPR, CCPA, consent signal propagation, data deletion verification, CMP validation
Proficiency with performance and load testing for high-throughput, low-latency systems
Familiarity with MLOps tooling — experiment tracking (MLflow, Vertex AI), model registries, feature stores, and deployment pipelines
Experience with data clean room and privacy-preserving measurement validation
Background in building and growing SDET teams — hiring, mentoring, technical interview design