Research Scientist, Open Benchmarks Red Team
Snorkel AI · New York City, NY (Hybrid) · San Francisco, CA (Hybrid) · United States (Remote) · Remote
Infrastructurehybrid$200k–$375k
At a glance
Highlights
- Red-team frontier AI benchmarks to expose grading failures and shortcuts.
- Develop and validate fixes for benchmark issues with original authors.
- Create reusable methods to detect recurring weaknesses across datasets.
Ideal candidate
A curious researcher-engineer who loves breaking AI evaluations to make them trustworthy and partnering with authors to turn adversarial findings into lasting improvements.
Requirements
- Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation.
- Strong Python programming skills and experience building, running, and debugging AI agents or evaluation systems.
- Ability to identify weaknesses in existing evaluation methods, develop new approaches, and rigorously test whether they work.
- Experience designing rigorous experiments, analyzing model behavior, and investigating unexpected failures.
- Familiarity with LLM/agent benchmarks, evaluation frameworks, or automated grading systems.
- Experience independently driving research and engineering projects from problem identification through implementation and validation.
- Strong communication and collaboration skills.
Lab profile
About Snorkel AI
Frontier AI data lab building datasets, benchmarks, and evals for frontier and agentic AI.
What they build
- Datasets for frontier AI
- Benchmarks and evals
- Custom AI data solutions
Known for
- Research-driven AI data development
- Combines technology with expert data development
- Works with leading AI labs and enterprises
Key facts
- Founded in 2019
- Spun out of Stanford AI Lab
Posted · Last seen