WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Oct 6, 2023

From clicks to cribs - How to find your dream home with web scraping

Alexander Lichter

Automate your competitive apartment search with a custom web scraper. Learn to bypass CAPTCHAs with Puppeteer and send instant property alerts straight to a Telegram bot.

Pause
Mute Enter Fullscreen
#1 about 3 min

Challenges of finding dream homes in major cities

How competitive real estate markets create stress and demand constant monitoring.

#2 about 1 min

Automating routine website checks to save time

Calculating the time saved justifies creating custom scripts for repetitive tasks.

#3 about 1 min

Extracting website data using web scraping techniques

Bypassing manual entry limitations by extracting bulk information across web platforms.

#4 about 3 min

Scraping data payloads from single page applications

Inspect network requests to uncover and replicate underlying JSON API calls.

#5 about 7 min

Extracting data from static HTML using parsing libraries

Querying DOM elements with tools like Cheerio to isolate structured attributes.

#6 about 3 min

Writing unit tests and parsers for HTML extraction

Using testing frameworks to validate accurate extraction of complex real estate data.

#7 about 2 min

Automating the fetch and parse workflow for multiple sites

Combining fetching utilities with parsing logic to create consistent object arrays.

#8 about 4 min

Diffing JSON data and scheduling mobile push notifications

Storing parsed entries in a flat file database and triggering Telegram alerts.

#9 about 4 min

Bypassing authentication and captchas using headless browser automation

Handling anti-bot protections through simulated sessions and manual intervention fallbacks.

#10 about 2 min

Concluding thoughts on scraper tools and audience questions

Final insights on data extraction choices and third-party solving services.

Matching moments

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann Chris Heilmann +4 · LIVE

2:54 min

Technical workflow for generating self-healing scrapers

Ariel Shulman Ariel Shulman +1 · World Congress 2026 Europe

3:43 min

Scaling web scraping infrastructure to bypass strict security restrictions

Tim Ruscica · Coffee With Developers

3:40 min

Overcoming modern anti-bot mechanisms and network access restrictions

Vidas Bacevičius Vidas Bacevičius · World Congress 2025

2:20 min

Handling http network errors and anti-scraping challenges

Domagoj Marić Domagoj Marić · Europe 2026 Virtual

6:28 min

Scraping property data from multiple real estate platforms

Alp Uguray · JS Congress

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
almost 3 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Related videos

How to scrape modern websites to feed AI agents thumbnail
19:20
World Congress 2025 • about 1 year ago

How to scrape modern websites to feed AI agents

Jan Curn

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15
World Congress 2025 • about 1 year ago

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations thumbnail
36:42
Europe 2026 Virtual • 3 months ago

Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations

Domagoj Marić

Automate everything via NodeJS and Puppeteer thumbnail
43:11
JS Congress • almost 5 years ago

Automate everything via NodeJS and Puppeteer

Miki Lombardi

Data is Key: Scraping Metadata from Websites thumbnail
21:17
World Congress 2023 • about 3 years ago

Data is Key: Scraping Metadata from Websites

Lars Kölker

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more thumbnail
1:06:14
LIVE • 10 months ago

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more

Chris Heilmann, Daniel Cranney, Ondra Urban, Coo, Gtm At Apify

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.