Staff Software Engineer, Inference

Highlights

• Lead architecture for low-latency, high-throughput AI inference at massive scale.
• Optimize scheduling, batching and memory for real-time AI workloads.
• Power AI breakthroughs for leading labs, startups and global enterprises.

What they offer

• Tuition Reimbursement
• Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-forming support through Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with generous employer match
• Flexible PTO
• Catered lunch each day in office and data center locations
• Casual work environment

Ideal candidate

A performance-obsessed distributed systems leader who loves optimizing low-latency infrastructure at scale and driving cross-team technical direction in fast-moving, high-growth environments.

Requirements

• 8-12+ years of experience building and operating large-scale distributed systems or cloud platforms
• Proven experience leading cross-team technical initiatives impacting multiple services or organizations
• Strong programming skills in Go, Python, or C++
• Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design
• Strong understanding of distributed systems, networking, and performance optimization
• Experience designing and operating low-latency, high-throughput systems with strict P95/P99 latency requirements
• Hands-on experience with inference systems, including batching or micro-batching strategies, caching, and memory optimization
• Experience improving system performance using metrics-driven approaches
• Familiarity with mixed precision (BF16, FP8) and streaming inference workloads