Member of Technical Staff - Web Crawl Engineer

Highlights

• Build web-scale crawling infrastructure across billions of URLs.
• Own URL discovery, scheduling, and crawl orchestration systems.
• Power frontier AI with fresh, diverse web data at scale.

What they offer

• Top-tier compensation with salary and equity
• Stock options for all contributors
• Comprehensive medical, dental, vision and life insurance plus wellness allowance
• Lunch and dinner provided in the office daily
• 22 weeks paid parental leave for all new parents
• Unlimited paid time off in the U.S. and 30 days in the U.K.
• Visa sponsorship and long-term immigration support
• Regular off-sites, happy hours and team celebrations

Ideal candidate

Distributed-systems engineer passionate about web-scale data collection and how web data shapes frontier models. Collaborates closely with researchers and thrives balancing coverage, freshness, quality, and efficiency.

Requirements

• Experience building large-scale web crawling, search indexing, content acquisition, or internet-scale data collection systems
• Strong understanding of crawling architectures, URL frontier management, scheduling, and distributed crawl coordination
• Experience with large-scale distributed systems using technologies such as Ray, Spark, Beam, Flink, or similar frameworks
• Familiarity with content extraction, HTML parsing, browser automation, rendering systems, and modern web technologies
• Experience operating systems that process petabyte-scale datasets
• Strong systems engineering skills, including reliability, observability, performance optimization, and debugging
• Experience designing experiments and using data to improve crawl quality, coverage, and efficiency
• Excellent communication skills and ability to reason about system tradeoffs and operational constraints