Member of Technical Staff, Pretraining Evaluations
Hark · San Jose · HQ
Frontier labonsite$180k–$450k
At a glance
Highlights
- Build evaluation harness tracking held-out loss across pretraining runs.
- Design new benchmarks for capabilities existing evaluations miss.
- Extend evaluation tools to image, video, speech and audio tasks.
Ideal candidate
A rigorous experimentalist fascinated by model capabilities and measurement. You care about reproducibility, statistical rigor, and turning evaluation insights into better training decisions.
Requirements
- Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
- Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
- Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
- Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.
- Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.
- A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.
Lab profile
About Hark
Building proactive, personalized multimodal AI with next-generation hardware for human-machine interaction.
Known for
- Proactive personalized intelligence
- Multimodal speech, text and vision
- Persistent memory
- Next-generation AI hardware
Posted · Last seen