About the job
YOUR CHARTER
- Data at Scale: Own the pipelines and storage systems that feed petabyte-scale multimodal datasets into model training.
- Sustainable Platforms: Build tooling and systems that are automated and efficient, enabling processing at scale and handling many small heterogeneous datasets.
Responsibilities
Design, automate, maintain, and optimize Python ETL pipelines (Spark/Ray) for large-scale multimodal data.
Build and maintain data cataloging, lineage, quality tooling, integrity verification, access controls, and lifecycle management systems.
Provide guidance, internal tools, and documentation to colleagues on data best practices.
Serve as a custodian of the company’s datasets, ensuring overall data health, quality, and discoverability.
Qualifications
Minimum
Data Engineering: Knowledge of Python ETL pipelines and supporting infrastructure, data formats, and storage systems at scale.
ML Data Ops: Experience managing datasets, annotations, and data versioning for model training.
Basic ML Knowledge: Solid grasp of ML fundamentals is essential to collaborate effectively with researchers and make sound data platform decisions.
Agentic Engineering: Skilled at writing high-quality specifications for AI agents, while maintaining effective human review of AI-generated work.
Preferred
No preferred qualifications listed.