> Markdown version of [/jobs/ext/2716150-web-crawler-engineer](https://www.wearedevelopers.com/jobs/ext/2716150-web-crawler-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Web Crawler engineer

- **Company:** ONE STOP COLLECTIBLE CORP
- **Location:** San Francisco, CA, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** JavaScript (Programming Language), Dynamic Content
- **Published:** September 4, 2026
- **Apply:** https://startup.jobs/software-engineer-web-crawling-exa-7810050

## About the Role

* You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
 * You have experience with some high performance language (C++, Rust, etc.)
 * You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
 * You're comfortable optimizing a system to an exceptional degree
 * You care about the problem of finding high quality knowledge and recognize how important this is for the world

## Description

* Build a distributed crawler that can handle 100M+ pages per day
 * Optimize crawl politeness and rate limiting across thousands of domains
 * Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
 * Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency


## Related Videos

- [From clicks to cribs - How to find your dream home with web scraping](https://www.wearedevelopers.com/videos/767-from-clicks-to-cribs-how-to-find-your-dream-home-with-web-scraping)
- [How to Stop Choosing JavaScript Frameworks and Start Living](https://www.wearedevelopers.com/videos/118-how-to-stop-choosing-javascript-frameworks-and-start-living)
- [Let's get visual - Visual testing in your project](https://www.wearedevelopers.com/videos/540-let-s-get-visual-visual-testing-in-your-project)
- [How to scrape modern websites to feed AI agents](https://www.wearedevelopers.com/videos/1446-how-to-scrape-modern-websites-to-feed-ai-agents)
- [Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations](https://www.wearedevelopers.com/videos/2046-modern-web-data-extraction-techniques-tools-and-legal-ethical-considerations)
- [Extreme Rules Lawyering with the ECMAScript Specifications](https://www.wearedevelopers.com/videos/100116-extreme-rules-lawyering-with-the-ecmascript-specifications)

## Related Articles

- [Dev Digest 113 - Debugging above the cloud](https://www.wearedevelopers.com/magazine/422-dev-digest-113-debugging-above-the-cloud)
- [Dev Digest 106 - DHH in HD](https://www.wearedevelopers.com/magazine/394-dev-digest-106-dhh-in-hd)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)
- [Dev Digest 133 - Back to Front](https://www.wearedevelopers.com/magazine/474-dev-digest-133-back-to-front)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Dev Digest 136 - No JS(on) of mine](https://www.wearedevelopers.com/magazine/482-dev-digest-136-no-js-on-of-mine)