Member of Technical Staff, Evals

Highlights

• Design and validate new evaluations and benchmarks across training.
• Audit public benchmarks and fix subtle bugs that mislead conclusions.
• Build tools that make evaluation results accessible across Magic.

What they offer

• Equity is a significant part of total compensation
• 401(k) plan with 6% salary matching
• Generous health, dental, and vision insurance for you and dependents
• Unlimited paid time off
• Visa sponsorship and relocation support to San Francisco

Ideal candidate

A rigorous, detail-obsessed ML experimenter who distrusts unreproducible results and cares deeply about measurement quality and correctness.

Requirements

• Experience developing model evaluations, benchmarks, or ML experiments
• Strong programming skills and experience with machine learning systems
• Track record of owning technical projects end-to-end
• Skepticism toward results that cannot be reproduced, validated, or explained
• Experience analyzing model performance and interpreting experimental results
• Comfortable navigating ambiguity and validating measurements
• Commitment to raising rigor in AI evaluations