Site Reliability Engineer, Post Training

Highlights

• Own reliability of large-scale post-training and RL training jobs end-to-end.
• Embed with researchers to debug live runs and speed up iteration.
• Build self-healing monitoring and tooling for model training clusters.

What they offer

• Health, dental, and vision benefits
• Unlimited PTO
• Paid parental leave
• Relocation support as needed

Ideal candidate

A production-minded engineer who thrives on-call and loves debugging complex distributed failures side-by-side with researchers to keep frontier training runs moving fast.

Requirements

• 4+ years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems in production
• Track record debugging complex failures across distributed systems — networking, hardware, kernel, or scheduler issues
• Strong software engineering skills in Python and/or Go/C++, with the judgment to know when to script a fix versus build a system
• Solid grounding in Linux systems internals and networking fundamentals
• Comfortable owning production systems, including participating in on-call rotations
• Experience operating GPU or TPU training clusters at scale
• Familiarity with post-training and RL techniques (e.g., RLHF, PPO, DPO) and infrastructure challenges specific to them
• Experience with distributed training frameworks (e.g., PyTorch, Ray) and job schedulers (e.g., Slurm, Kubernetes)
• Experience with high-performance networking (e.g., InfiniBand, RDMA, NCCL) and its role in distributed training performance
• Experience building observability tooling purpose-built for ML training
• Track record of thriving in fast-changing, research-driven environments