WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Oct 6, 2023

From clicks to cribs - How to find your dream home with web scraping

Alexander Lichter

Automate your competitive apartment search with a custom web scraper. Learn to bypass CAPTCHAs with Puppeteer and send instant property alerts straight to a Telegram bot.

Pause
Mute Enter Fullscreen
#1 about 3 min

Challenges of finding dream homes in major cities

How competitive real estate markets create stress and demand constant monitoring.

#2 about 1 min

Automating routine website checks to save time

Calculating the time saved justifies creating custom scripts for repetitive tasks.

#3 about 1 min

Extracting website data using web scraping techniques

Bypassing manual entry limitations by extracting bulk information across web platforms.

#4 about 3 min

Scraping data payloads from single page applications

Inspect network requests to uncover and replicate underlying JSON API calls.

#5 about 7 min

Extracting data from static HTML using parsing libraries

Querying DOM elements with tools like Cheerio to isolate structured attributes.

#6 about 3 min

Writing unit tests and parsers for HTML extraction

Using testing frameworks to validate accurate extraction of complex real estate data.

#7 about 2 min

Automating the fetch and parse workflow for multiple sites

Combining fetching utilities with parsing logic to create consistent object arrays.

#8 about 4 min

Diffing JSON data and scheduling mobile push notifications

Storing parsed entries in a flat file database and triggering Telegram alerts.

#9 about 4 min

Bypassing authentication and captchas using headless browser automation

Handling anti-bot protections through simulated sessions and manual intervention fallbacks.

#10 about 2 min

Concluding thoughts on scraper tools and audience questions

Final insights on data extraction choices and third-party solving services.

Matching moments

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann +4 · LIVE

2:54 min

Technical workflow for generating self-healing scrapers

Ariel Shulman Ariel Shulman +1 · WWC Europe 2026

3:43 min

Scaling web scraping infrastructure to bypass strict security restrictions

Tim Ruscica · Coffee With Developers

3:40 min

Overcoming modern anti-bot mechanisms and network access restrictions

Vidas Bacevičius Vidas Bacevičius · WWC 2025

6:28 min

Scraping property data from multiple real estate platforms

Alp Uguray · JS Congress

2:14 min

Understanding the basic mechanics of automated web scraping

Vidas Bacevičius Vidas Bacevičius · WWC 2025

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
over 2 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
over 2 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
over 2 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
over 2 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
over 2 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Small LLM in your Browser: Huge Opportunities for Web Applications

Daniel Ostrovsky

UI/UX Architect at Payoneer | AI Architect | Full Cycle Development Expert | Public Speaker | Open Source Contributor |

Daniel Ostrovsky
Open session

World Congress 2026 North America

Agentic commerce's dirty little secret, and what to do about it.

Joe Monastiero

Founder & CEO at visualAI

Joe Monastiero
Open session

World Congress 2026 North America

rm -rf: Horror Stories From Unsandboxed AI Agents (and How Docker Fixes This)

Rishab Kumar

Staff Developer Evangelist @ Twilio

Rishab Kumar
Open session

World Congress 2026 North America

RoboCoders: Judgment Day: AI-Assisted Engineering Applied - The Battle of Agents

Baruch Sadogursky, Viktor Gamov

Baruch Sadogursky
Viktor Gamov
Open session

World Congress 2026 North America

Designing APIs That Survive AI Agents at Scale

Phani Pendurthi

Mastercard, Principal Software Engineer

Phani Pendurthi
Open session

World Congress 2026 North America

DeepAgents: Build Multi-Agent AI Systems That Actually Work

Anagha Rumade, Anjana Umapathy, Apoorva Jaiswal

Anagha Rumade
Anjana Umapathy
Apoorva Jaiswal

Related videos

How to scrape modern websites to feed AI agents thumbnail
19:20
WWC 2025 • 12 months ago

How to scrape modern websites to feed AI agents

Jan Curn

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15
WWC 2025 • 12 months ago

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

Automate everything via NodeJS and Puppeteer thumbnail
43:11
JS Congress • over 4 years ago

Automate everything via NodeJS and Puppeteer

Miki Lombardi

Data is Key: Scraping Metadata from Websites thumbnail
21:17
WWC 2023 • almost 3 years ago

Data is Key: Scraping Metadata from Websites

Lars Kölker

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more thumbnail
1:06:14
LIVE • 8 months ago

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more

Chris Heilmann, Daniel Cranney, Ondra Urban, Coo, Gtm At Apify

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more thumbnail
1:01:25
LIVE • over 1 year ago

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more

Chris Heilmann, Daniel Cranney

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.