Senior Product Operations Manager, Evaluation Quality

Highlights

• Own the quality bar for human evaluations guiding product and AI decisions.
• Build calibration and QA systems for global attorney eval pipeline scaling 10x.
• Turn raw eval results into decision-grade signal for Product and Research.

What they offer

• Equity
• Bonus

Ideal candidate

Senior operator who loves setting rigorous quality bars for expert human work and turning messy judgment into trusted, reproducible signal. High-ownership, documentation-driven, and energized by partnering with legal experts to ship AI.

Requirements

• 6+ years in product operations, research operations, evaluation/QA operations, or quality program management
• Track record owning quality inputs like guidelines, instructions, benchmarks, and QA procedures for expert-driven work
• Experience onboarding, training, and calibrating distributed expert raters and running improvement feedback loops
• Experience running quantitative and qualitative analyses and statistical tests on evaluation data
• Record scaling evaluation quality processes under shipping pressure with focus on documentation and reproducibility
• Ability to translate nuanced domain-expert judgment into repeatable, documented standards
• Strong cross-functional coordination across Product, Engineering, Research, and vendors
• Clear communicator who builds stakeholder credibility and trust with high ownership and bias to action