Senior Software Engineer, Inference

Highlights

• Lead designs to improve latency, throughput and reliability across services.
• Evolve large-scale AI inference platform to meet strict P99 SLAs at scale.
• Own areas like request routing, adaptive scheduling and cost-per-token analytics.

What they offer

• Tuition Reimbursement
• Ability to Participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations

Ideal candidate

Distributed systems builder with 5-8 years experience who obsesses over tail latency and reliability. You mentor others, raise engineering standards, and thrive owning complex, multi-service areas in fast-moving environments.

Requirements

• 5-8 years industry experience building distributed systems or cloud services
• Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems and performance
• Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry)
• Practical knowledge of inference internals: batching, caching, mixed precision (BF16/FP8), streaming token delivery
• Proven track record improving tail latency (P95/P99) and service reliability through metrics-driven work
• Contributions to inference frameworks (vLLM, Triton, TensorRT-LLM, Ray Serve, TorchServe)
• Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies