Distributed LLM Inference Engineer

Highlights

• Push boundaries of performance for large-scale Batch and Online inference.
• Ship end-to-end inference solutions used by open-source users and customers.
• Help achieve market-leading position for AI infrastructure at scale.

What they offer

• Stock Options
• Healthcare plans, with premiums covered at 99% for employees and dependents
• 401k Retirement Plan
• Education & Wellbeing Stipend
• Paid Parental Leave
• Fertility Benefits
• Paid Time Off
• Commute reimbursement
• 100% of in-office meals covered

Ideal candidate

A performance-obsessed distributed systems engineer who loves fast product iteration and large-scale ML inference, excited to push efficiency and contribute to open source.

Requirements

• Familiarity with running ML inference at large scale with high throughput and low latency
• Familiarity with deep learning and deep learning frameworks (e.g. PyTorch)
• Solid understanding of distributed systems and ML inference challenges
• ML systems knowledge
• Experience using Ray
• Experience with LLM engines like vLLM, TensorRT-LLM and community collaboration
• Contributions to deep learning frameworks such as PyTorch, TensorFlow
• Contributions to deep learning compilers such as Triton, TVM, MLIR
• Prior experience working on GPUs / CUDA