Software Engineer, Web Crawling

About the role

Exa is an applied AI lab building a search engine that crawls the entire web, trains embedding models on it and uses vector databases to retrieve results. It powers search for Cursor, Cognition and HubSpot and has raised $350m. As a Web Crawler engineer you would build a crawler that handles Google-scale crawling, working in person in Zurich.

Key responsibilities

• Build a distributed crawler able to handle 100M+ pages per day
• Optimize crawl politeness and rate limiting across thousands of domains
• Design systems to detect and handle dynamic content, JavaScript rendering and anti-bot measures
• Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

Required profile

• Extensive experience building and scaling web crawlers, or willingness to ramp up very quickly
• Experience with a high-performance language such as C++ or Rust
• Familiarity with TypeScript, Playwright, modern web design and Chrome DevTools Protocol (CDP)
• Comfortable optimizing a system to an exceptional degree
• Interest in finding high-quality knowledge

What we offer

• Visa sponsorship for international candidates (STEM OPT, OPT, H1B, O1, E3); the company works hard to get you a visa once hired
• Premium healthcare benefits (medical, dental, vision) and fertility benefits
• 16 weeks of fully paid parental leave for all new parents
• A monthly wellness stipend for all employees