> Markdown version of [/jobs/ext/1918571-senior-back-end-engineer-website-scanning](https://www.wearedevelopers.com/jobs/ext/1918571-senior-back-end-engineer-website-scanning). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Back-End Engineer (Website Scanning) - **Company:** ASGN Incorporated - **Location:** Redwood City, CA, United States (Remote available) - **Experience:** Expert - **Salary:** $160,000.0 - $200,000.0 - **Contract:** Permanent contract - **Skills:** Query Performance, HTML, Java (Programming Language), Test Suite, Application Programming Interfaces (APIs), Airflow, Amazon Web Services, Systems Engineering, Automation of Tests, Unit Testing, Microsoft Azure, Cascading Style Sheets (CSS), Profiling, Code Review, Databases, Continuous Integration, Cron, Extract Transform Load (ETL), Relational Databases, Software Debugging, Distributed Systems, Elasticsearch, Github, Monitoring of Systems, Web Browsers, Python (Programming Language), PostgreSQL, Message Broker, Microsoft Message Queuing, MongoDB, MySQL, Nginx, Node.Js, NoSQL, OAuth, Parsing, Performance Tuning, RabbitMQ, Redis, Prometheus, Reverse Proxy, Secure Coding, Selenium, Web Crawlers, XPath, Google Tag Manager, Datadog, Data Logging, Multithreading, Google Cloud, Test-Driven Development (TDD), Data Ingestion, Captcha, Grafana, Concurrency, Caching, Technical Debt, AWS Lambda, Indexer, Puppeteer (Software), Backend, Rate Limiting, Git, Containerization, Gitlab-ci, Integration Tests, Kubernetes, Information Technology, Playwright, Google Cloud Functions, Apache Kafka, Graphql, Celery, Data Pipelines, Serverless Computing, Docker, Jenkins, Golang, Microservices - **Published:** August 4, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=72c7497e60ca6585 ## About the Role * Bachelors or Masters degree in Computer Science, Engineering, or equivalent practical experience. * 5+ years building back-end systems with production experience in Python, Go, Java, Node.js, or Rust; demonstrable projects in at least two of these languages. * Strong experience with distributed systems, microservices architecture, concurrency (asyncio, goroutines, multithreading), and message brokers (Kafka, RabbitMQ, AWS SQS). * Proven experience designing and operating web crawling and scraping systems, including headless browsers (Puppeteer, Playwright, Selenium), HTTP clients, HTML parsing (BeautifulSoup, lxml, Cheerio), and selector strategies (CSS/XPath). * Familiarity with proxy management, IP rotation, CAPTCHA mitigation strategies, fingerprinting and anti-bot evasion, robots.txt and sitemap parsing, and rate limiting/throttling implementation. * Practical knowledge of databases (PostgreSQL, MySQL, MongoDB, Redis), search/indexing (Elasticsearch), and data pipeline/ETL patterns. * Experience with containerization (Docker), orchestration (Kubernetes), and cloud platforms (AWS, GCP, Azure); experience with serverless platforms is a plus. * Experience with CI/CD tools (Git, Jenkins, GitLab CI, GitHub Actions), logging and observability stacks (ELK, Prometheus, Grafana, Datadog), and performance tuning at scale. * Strong understanding of security best practices (OAuth, JWT, input sanitization), API rate compliance, error handling and retry/backoff patterns, and caching strategies. * Comfortable with test-driven development, unit and integration testing, and automation of test suites and deployments. * Excellent debugging, profiling, and performance optimization skills; ability to triage incidents and implement long-term fixes. * Strong communication skills, ability to work cross-functionally, mentor teammates, and drive projects from concept to production. ## Description Our product tackles important scanning + remediation challenges in the privacy + consent space (think of cookies + consent management, GDPR / CCPA, google tag manager, data privacy, etc.) and our product is used by stakeholders seeking to ensure compliance w/ standard data privacy laws as well as law firms who need to capture, assess and triage issues related to privacy + security., We are seeking a senior Back-End Software Engineer focused on building and scaling high-performance web scanning and data ingestion systems. You will design, implement, and operate robust distributed services that crawl, scrape, process, and store large volumes of web data while maintaining reliability, compliance, and efficiency at scale. The role combines deep systems engineering, web automation, anti-bot evasion strategies, and platform-level scalability work across cloud and containerized environments., * Design, implement, and maintain scalable microservices and distributed systems for web crawling, scraping, and data ingestion using Go or Rust or Python * Build and operate headless browser automation and scraping pipelines using Puppeteer, Playwright, Selenium, or lightweight HTTP clients (Requests, AIOHTTP, OkHttp, Axios) with robust HTML parsing (BeautifulSoup, lxml, Cheerio) and CSS/XPath selector strategies. * Develop and optimize REST and GraphQL APIs to serve downstream consumers * Implement concurrency and parallelism patterns (asyncio, goroutines, multithreading, multiprocessing) and task queues (Celery, RQ) to maximize throughput while controlling resource usage and backpressure. * Architect and tune message-driven pipelines with brokers like Kafka, RabbitMQ, AWS SQS, and task scheduling with Airflow or Cron for ETL workflows. * Design storage and indexing solutions using relational databases (PostgreSQL, MySQL), NoSQL stores (MongoDB, Redis), and search systems (Elasticsearch) with attention to schema design, normalization, and query performance. * Build robust proxy management, IP rotation strategies, robots.txt and sitemap parsing, rate limiting/throttling, retry/backoff patterns, and CAPTCHA handling and solving strategies as needed. * Implement fingerprinting and anti-bot evasion techniques, caching strategies, reverse proxy configurations (Nginx), and secure handling of credentials, tokens, and sensitive data (OAuth, JWT). * Containerize services with Docker and deploy and manage them on Kubernetes or serverless platforms (AWS Lambda, Google Cloud Functions), leveraging cloud platforms (AWS, GCP, Azure) for scalability and resilience. * Instrument systems for observability and reliability using logging and monitoring tools (ELK, Prometheus, Grafana, Datadog), set up alerts, and conduct post-incident analysis and performance tuning. * Collaborate with product, data, and security teams to define SLAs, compliance boundaries, and automated tests; own CI/CD pipelines using Git, GitHub Actions, Jenkins, or GitLab CI for repeatable deployments. * Lead efforts to harden systems with input sanitization, secure coding practices, and automated unit/integration testing and test automation to ensure correctness and reduce technical debt. * Mentor engineers, perform code reviews, and contribute to architectural decisions that improve scalability, maintainability, and developer productivity. ## Related Videos - [From clicks to cribs - How to find your dream home with web scraping](https://www.wearedevelopers.com/videos/767-from-clicks-to-cribs-how-to-find-your-dream-home-with-web-scraping) - [The Resilience of the World Wide Web](https://www.wearedevelopers.com/videos/1281-the-resilience-of-the-world-wide-web) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [WeAreDevelopers LIVE – AI, Freelancing, Keeping Up with Tech and More](https://www.wearedevelopers.com/videos/1786-wearedevelopers-live-ai-freelancing-keeping-up-with-tech-and-more) - [Is your backend a hodgepodge of queues, event stores and cron jobs? Durable Execution to the Rescue.](https://www.wearedevelopers.com/videos/744-is-your-backend-a-hodgepodge-of-queues-event-stores-and-cron-jobs-durable-execution-to-the-rescue) - [NoLoJS - Avoiding JavaScript Cruft with HTML and CSS - Aaron T. Grogg](https://www.wearedevelopers.com/videos/1806-nolojs-avoiding-javascript-cruft-with-html-and-css-aaron-t-grogg) ## Related Articles - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [Is Software Engineering Over-Saturated?](https://www.wearedevelopers.com/magazine/418-is-software-engineering-over-saturated) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [What’s the Difference Between Frontend and Backend Development?](https://www.wearedevelopers.com/magazine/240-what-s-the-difference-between-frontend-and-backend-development) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)