Site Reliability Engineer, Production
Thinking Machines Lab · San Francisco · New York
CHECKED 12H AGO
Frontier labonsite$300k–$350k
At a glance
Highlights
- Drive end-to-end reliability for Tinker fine-tuning API platform
- Own monitoring and incident response across full training path
- Harden multi-tenant isolation to maximize utilization safely
What they offer
- Generous health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support as needed
Ideal candidate
A reliability-driven engineer who loves owning production end-to-end and partnering with researchers to make frontier training robust, resilient, and fast to recover.
Requirements
- Bachelor's degree or equivalent experience in computer science, engineering, or similar
- Experience in distributed systems, cloud infrastructure, or site reliability engineering
- Proficiency writing software to solve reliability problems, including building tooling and automation
- Experience with production incident response, postmortems, and systematic reliability improvement
- Strong communication skills and track record of coordination across engineering and research teams
- Deep experience operating production cloud services at scale
- Background in distributed training frameworks and how infrastructure failures surface in training behavior
- Track record building checkpoint and recovery systems for long-running distributed jobs
- Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads
- visa sponsorship
- We sponsor visas
Lab profile
About Thinking Machines Lab
Builds frontier AI that extends human will and judgment through customizable models and interfaces.
What they build
- Inkling frontier models
- Tinker model customization
- Human-AI communication interfaces
Known for
- Frontier model training
- Post-training and reinforcement learning
- Evaluation infrastructure
- GPU supercomputing and reliability