Member of Technical Staff - Evaluations

Highlights

• Own internal eval suite end-to-end that drives post-training decisions.
• Design held-out evals for agentic workflows, computer use, cybersecurity.
• Build evals for small edge models and release open-source evals with papers.

What they offer

• Competitive base salary plus equity
• Unlimited PTO plus company-wide Refill Days
• Comprehensive health coverage
• Paid leave for pregnancy, parenting, medical and family care

Ideal candidate

Proactive, white-box evaluator who owns benchmarks end-to-end, digs into outputs before trusting scores, and cares about rigorous, reproducible evals for small edge models that ship within weeks.

Requirements

• Strong eval design: turn capability descriptions into accurate, generalizable, hard-to-game evals
• Prior evals built from scratch such as a benchmark, eval paper, or contributions to lm-eval-harness, lighteval, OLMES, HELM, Inspect
• Fluent in eval fundamentals: contamination, saturation, judge bias, and variance
• Hands-on experience running and managing evals at scale in a team setting
• Debugs unexpected scores by reading individual model outputs before trusting numbers
• Strong software engineering to own eval infrastructure and debug inference and infra failures