Member of Technical Staff - Distributed Training Engineer

Liquid AI
San Francisco / Boston2025-07-29Hybrid

About the job

Our Training Infrastructure team is building the distributed systems that power our next-generation Liquid Foundation Models. As we scale, we need to design, implement, and optimize the infrastructure that enables large-scale training. This is a high-ownership training systems role focused on runtime/performance/reliability (not a general platform/SRE role). You’ll work on a small team with fast feedback loops, building critical systems from the ground up rather than inheriting mature infrastructure.

Responsibilities

Design and build core systems that make large training runs fast and reliable

Build scalable distributed training infrastructure for GPU clusters

Implement and tune parallelism/sharding strategies for evolving architectures

Optimize distributed efficiency (topology-aware collectives, comm/compute overlap, straggler mitigation)

Build data loading systems that eliminate I/O bottlenecks for multimodal datasets

Develop checkpointing mechanisms balancing memory constraints with recovery needs

Create monitoring, profiling, and debugging tools for training stability and performance

Qualifications

Minimum

Hands-on experience building distributed training infrastructure (PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, Megatron-LM TP/PP)

Experience diagnosing performance bottlenecks and failure modes (profiling, NCCL/collectives issues, hangs, OOMs, stragglers)

Understanding of hardware accelerators and networking topologies

Experience optimizing data pipelines for ML workloads

Preferred

MoE (Mixture of Experts) training experience

Large-scale distributed training (100+ GPUs)

Open-source contributions to training infrastructure projects