Python Web Scraping Engineer

My client is hiring a Senior Python Scraping Engineer to design, build, and operate high-volume, highly resilient web scraping systems, with a specific and sustained focus on scraping Google at scale.

This is a specialist role. It is not suited to generalist data engineers who have only worked on light or opportunistic scraping. You will be working in hostile, rapidly changing environments where naïve techniques fail and deep expertise in bot detection and evasion is essential.

The role sits at the intersection of data engineering, reverse engineering, and large-scale systems reliability, and plays a critical role in delivering accurate, timely, and trusted data.

My client is a leader in adopting AI-assisted and agentic coding practices, and this role is ideal for engineers who actively use AI tools to improve productivity, reasoning, and system design., * Design and operate large-scale scraping systems handling 10+ million requests per day, primarily targeting Google and Google-like platforms

• Build robust scrapers for dynamic, JavaScript-heavy environments using browser automation and hybrid approaches
• Continuously adapt to changes in markup, request flows, ranking logic, and anti-automation mechanisms
• Engineer extraction pipelines with a strong emphasis on correctness, consistency, and observability
• Implement and maintain advancedanti-bot evasion strategies, including:

• Proxy and request routing strategies
• Browser and headless fingerprinting
• CAPTCHA handling and mitigation

• Monitor system health, detect anomalies early, and debug complex production issues
• Optimise performance, cost, and latency across large-scale scraping infrastructure
• Collaborate closely with data engineers, data scientists, and product teams to ensure scraped data is reliable and usable
• Produce clear documentation and operational runbooks to support long-term maintainability

• Expert-level web scraping skills using Python
• Direct, hands-on experience scraping Google at scale - this is essential
• Deep understanding of:

• Anti-bot and bot-detection systems
• Browser and network fingerprinting
• CAPTCHA systems and mitigation techniques
• Scaling scraping infrastructure reliably

• Strong knowledge of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour
• Experience with tools such as Playwright, Selenium, Puppeteer, or equivalent frameworks
• Comfortable designing asynchronous and concurrent scraping architectures
• Proven experience running scraping systems at scale in cloud environments
• Excellent debugging skills and the ability to reason about complex failure modes
• Strong communication skills, with the ability to clearly explain complex technical behaviour

Tenth Revolution Group are the go-to recruiter for Data & AI roles in the UK offering more opportunities across the country than any other recruitment agency. We're the proud sponsor and supporter of SQLBits, Power Platform World Tour, and the London Fabric User Group. We are the global leaders in Data & AI recruitment.