About the job
The Opportunity
Adobe Express Data Platform is the intelligence backbone for millions of creators- a billion-event-per-day system spanning streaming, feature serving, agent data APIs, and a lakehouse that powers every personalization decision, experiment, and AI workflow. We are evolving it into a streaming-first, self-healing, agent-ready Lakehouse and we need engineers who challenge the status quo, move fast, and default to an agentic-first approach for every problem they encounter.
This is a systems-first engineering role. You won’t build ML models, you’ll build the foundational infrastructure that makes AI, analytics, and autonomous agents possible at scale. You’ll bring the conviction that any manual, repetitive, or slow platform workflow is a candidate for agentic automation and the engineering skill to make that real.
Responsibilities
Design and build streaming-first data pipelines that collapse end-to-end latency from hours to minutes, through event-driven architectures.
Own and extend the ML Attribute Store — building low-latency online serving capabilities alongside batch feature computation with unified batch/streaming aggregation to prevent training-serving skew.
Build MCP-compatible Agent Data APIs and tool servers that make the lakehouse discoverable and queryable by autonomous AI agents through standardized protocols, semantic layers, and catalog-driven data discovery.
Develop agentic framework — automated anomaly detection, duplicate event cleanup, transient event lifecycle management with audit trails, pipeline self-healing, and root cause analysis automation.
Drive operational excellence: observability, incident detection and response automation, performance tuning, cost optimization, and on-call ownership for mission-critical platform services.
Collaborate across Data Science, Personalization, Engineering Operations, Product, and Experimentation teams to translate platform capabilities into self-serve infrastructure that reduces engineering toil for non-platform teams.
Use and champion AI-powered developer tools (Claude Code, Cursor, GitHub Copilot, or similar) to accelerate personal and team engineering velocity.
Qualifications
Minimum
6+ years of experience in data platform engineering, distributed systems, or backend infrastructure at scale.
Deep hands-on experience with Apache Spark, Databricks, Delta Lake, or equivalent lakehouse technologies (Iceberg, Hudi).
Proven track record building and operating large-scale pipelines processing billions of events daily with sub-hour latency SLAs.
Strong experience with streaming systems: Kafka, Kinesis, Flink, Spark Structured Streaming, or Delta Live Tables.
Proficiency in Python and/or Scala; SQL fluency required. Java or Go is a plus.
Experience with cloud platforms (AWS or Azure), containerization (Docker, Kubernetes), and CI/CD for data pipelines.
Preferred
Experience building AI-powered developer tools, self-serve data platforms, or code generation agents that reduce engineering toil.
Experience migrating batch-first data architectures to streaming-first without disrupting downstream consumers — including dual-write patterns, shadow pipelines, and incremental cutover strategies
Experience building autonomous monitoring systems that detect, diagnose, and remediate pipeline failures without human intervention — circuit breakers, auto-rollback, and intelligent retry logic
Familiarity with Adobe-native data and analytics solutions (CJA, AEP, Adobe Analytics) and data governance automation including FinOps practices, cost attribution, and compliance frameworks.
Contributions to open-source data or AI infrastructure projects, published engineering blog posts, or conference talks.
BS/MS in Computer Science, Engineering, or equivalent practical experience.