Skip to content
← Hark

Member of Technical Staff, Pretraining Evaluations

Hark · San Jose · HQ

Frontier labonsite$180k–$450k
Apply on the lab’s site ↗

At a glance

Highlights

  • Build evaluation harness tracking held-out loss across pretraining runs.
  • Design new benchmarks for capabilities existing evaluations miss.
  • Extend evaluation tools to image, video, speech and audio tasks.

Ideal candidate

A rigorous experimentalist fascinated by model capabilities and measurement. You care about reproducibility, statistical rigor, and turning evaluation insights into better training decisions.

Requirements

  • Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
  • Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
  • Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
  • Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.
  • Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.
  • A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.

Lab profile

About Hark

Building proactive, personalized multimodal AI with next-generation hardware for human-machine interaction.

Known for

  • Proactive personalized intelligence
  • Multimodal speech, text and vision
  • Persistent memory
  • Next-generation AI hardware
More about Hark →
Apply on the lab’s site ↗

Posted · Last seen