Research Engineer, Web Crawling
Thinking Machines Lab · San Francisco
CHECKED 13H AGO
Frontier labonsitestaff$350k–$475k
At a glance
Highlights
- Own distributed web crawlers sourcing Inkling pretraining data at internet scale.
- Build petabyte-scale pipelines turning raw crawls into usable pretraining data.
- Build specialized crawlers for high-value and hard-to-reach data sources.
What they offer
- generous health, dental, and vision benefits
- unlimited PTO
- paid parental leave
- relocation support as needed
Ideal candidate
A seasoned builder who has owned internet-scale crawling systems and loves turning messy web data into high-quality pretraining fuel. You care about reliability at petabyte scale and partnering with model teams to move quality.
Requirements
- 8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems
- Track record of owning crawler or data-acquisition infrastructure at internet scale
- Strong software engineering skills in Python, Go, or Rust with distributed systems experience
- Working knowledge of practical and legal considerations for large-scale web collection (robots.txt, rate limiting, licensing)
- visa sponsorship
- We sponsor visas and will work through the visa process together
Lab profile
About Thinking Machines Lab
Builds frontier AI that extends human will and judgment through customizable models and interfaces.
What they build
- Inkling frontier models
- Tinker model customization
- Human-AI communication interfaces
Known for
- Frontier model training
- Post-training and reinforcement learning
- Evaluation infrastructure
- GPU supercomputing and reliability