Skip to content
← Snorkel AI

Research Scientist, Open Benchmarks Red Team

Snorkel AI · New York City, NY (Hybrid) · San Francisco, CA (Hybrid) · United States (Remote) · Remote

Infrastructurehybrid$200k–$375k
Apply on the lab’s site ↗

At a glance

Highlights

  • Red-team frontier AI benchmarks to expose grading failures and shortcuts.
  • Develop and validate fixes for benchmark issues with original authors.
  • Create reusable methods to detect recurring weaknesses across datasets.

Ideal candidate

A curious researcher-engineer who loves breaking AI evaluations to make them trustworthy and partnering with authors to turn adversarial findings into lasting improvements.

Requirements

  • Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation.
  • Strong Python programming skills and experience building, running, and debugging AI agents or evaluation systems.
  • Ability to identify weaknesses in existing evaluation methods, develop new approaches, and rigorously test whether they work.
  • Experience designing rigorous experiments, analyzing model behavior, and investigating unexpected failures.
  • Familiarity with LLM/agent benchmarks, evaluation frameworks, or automated grading systems.
  • Experience independently driving research and engineering projects from problem identification through implementation and validation.
  • Strong communication and collaboration skills.

Lab profile

About Snorkel AI

Frontier AI data lab building datasets, benchmarks, and evals for frontier and agentic AI.

What they build

  • Datasets for frontier AI
  • Benchmarks and evals
  • Custom AI data solutions

Known for

  • Research-driven AI data development
  • Combines technology with expert data development
  • Works with leading AI labs and enterprises

Key facts

  • Founded in 2019
  • Spun out of Stanford AI Lab
More about Snorkel AI →
Apply on the lab’s site ↗

Posted · Last seen