WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 β€’ Oct 6, 2023

From clicks to cribs - How to find your dream home with web scraping

Alexander Lichter

Automate your competitive apartment search with a custom web scraper. Learn to bypass CAPTCHAs with Puppeteer and send instant property alerts straight to a Telegram bot.

Pause
Mute Enter Fullscreen
#1 about 3 min

Challenges of finding dream homes in major cities

How competitive real estate markets create stress and demand constant monitoring.

#2 about 1 min

Automating routine website checks to save time

Calculating the time saved justifies creating custom scripts for repetitive tasks.

#3 about 1 min

Extracting website data using web scraping techniques

Bypassing manual entry limitations by extracting bulk information across web platforms.

#4 about 3 min

Scraping data payloads from single page applications

Inspect network requests to uncover and replicate underlying JSON API calls.

#5 about 7 min

Extracting data from static HTML using parsing libraries

Querying DOM elements with tools like Cheerio to isolate structured attributes.

#6 about 3 min

Writing unit tests and parsers for HTML extraction

Using testing frameworks to validate accurate extraction of complex real estate data.

#7 about 2 min

Automating the fetch and parse workflow for multiple sites

Combining fetching utilities with parsing logic to create consistent object arrays.

#8 about 4 min

Diffing JSON data and scheduling mobile push notifications

Storing parsed entries in a flat file database and triggering Telegram alerts.

#9 about 4 min

Bypassing authentication and captchas using headless browser automation

Handling anti-bot protections through simulated sessions and manual intervention fallbacks.

#10 about 2 min

Concluding thoughts on scraper tools and audience questions

Final insights on data extraction choices and third-party solving services.

Matching moments

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann Chris Heilmann +4 Β· LIVE

2:54 min

Technical workflow for generating self-healing scrapers

Ariel Shulman Ariel Shulman +1 Β· World Congress 2026 Europe

3:09 min

Adapting scraping tools using headless browser setups

Fabien Vauchelles Β· LIVE

3:43 min

Scaling web scraping infrastructure to bypass strict security restrictions

Tim Ruscica Β· Coffee With Developers

3:40 min

Overcoming modern anti-bot mechanisms and network access restrictions

Vidas Bacevičius Vidas Bacevičius · World Congress 2025

2:20 min

Handling http network errors and anti-scraping challenges

Domagoj Marić Domagoj Marić · Europe 2026 Virtual

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
almost 3 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens HΓΌbner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Upcoming sessions on this topic

Open session

Build a Zero-Cost AI Agent in Your Browser

  • Aishwarya Mathuria

    Adobe

    Computer Scientist

  • Nishant Thakur

    Adobe

    Senior Computer Scientist

Open session

Teaching AI Coding Agents to Build It Right the First Time

  • Sajeetharan Sinnathurai

    Microsoft

    Principal Product Manager

Open session

React Under Fire: Rendering Performance for Streaming Agent UIs

  • Abhirupa Mitra

    Slack

    Senior Member of Technical Staff

Open session

Your New Teammates: Architecting a Multi-Agent Workforce

  • Henk Boelman

    GitHub

    Developer Advocate

Open session

Speed or Security? Why Not Both for Your Coding Agents

  • Ajeet Raina

    Docker

    Developer Advocate

Open session

ULTRON: Your Agent Has No Hands - Real-Time Voice, MCP, and Ten Phones on Stage

  • Sagar Tamang

    Two Spoon Technologies

    AI Engineer

Related videos

Cracking the Code: Decoding Anti-Bot Systems! thumbnail
57:47
LIVE β€’ over 2 years ago

Cracking the Code: Decoding Anti-Bot Systems!

Fabien Vauchelles

How to scrape modern websites to feed AI agents thumbnail
19:20
World Congress 2025 β€’ about 1 year ago

How to scrape modern websites to feed AI agents

Jan Curn

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15
World Congress 2025 β€’ about 1 year ago

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations thumbnail
36:42
Europe 2026 Virtual β€’ 4 months ago

Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations

Domagoj Marić

Automate everything via NodeJS and Puppeteer thumbnail
43:11
JS Congress β€’ almost 5 years ago

Automate everything via NodeJS and Puppeteer

Miki Lombardi

Data is Key: Scraping Metadata from Websites thumbnail
21:17
World Congress 2023 β€’ about 3 years ago

Data is Key: Scraping Metadata from Websites

Lars KΓΆlker

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.