Member of Technical Staff, Evals and Post-Training
Profound · New York, New York · San Francisco, California
Applied AIonsite$230k–$325k
At a glance
Highlights
- Build eval systems that measure how well AI performs marketing tasks.
- Turn eval results and expert feedback into training signal for models.
- Wire evals into CI, launch reviews, and model selection decisions.
What they offer
- equity
Ideal candidate
A builder with 2+ years in measurement and evals who loves separating signal from noise and has strong opinions on agent behavior. You thrive in fast startups and care about defining quality for AI marketers.
Requirements
- 2+ years building evaluation or measurement systems, including AI evals, experimentation, ranking, or search quality
- Strong Python and data skills with track record shipping production systems
- Deep familiarity with LLM evaluation including LLM-as-judge, benchmark design, agent harnesses, and calibration
- Sound experimental design with instinct to separate signal from noise in non-deterministic systems
- Familiarity with modern post-training methods including SFT, preference optimization, and RL
- Strong opinions about how models and agents should behave and keeps up with current research
- Comfortable operating in fast-paced startup environment with rapid iteration cycles
Lab profile
About Profound
AI marketing platform for research, content, and campaigns with visibility into AI answers.
What they build
- AI marketing platform
- AI Marketer agent
- AI visibility analytics and reporting
Known for
- Used by 15% of the Fortune 500
- Customers include Comcast, Estee Lauder and Walmart
- Built on 2B+ real user prompts
Posted · Last seen