WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

How to scrape modern websites to feed AI agents

Jan Curn

Feeding raw HTML to your LLM guarantees garbage outputs. Learn to bypass modern bot blockers to extract clean data. Transform your AI into a web-integrated operator.

Pause
Mute Enter Fullscreen
#1 about 1 min

Role of web scraping in training language models

Extensive public internet crawls remain the foundational mechanism enabling the vast reasoning capabilities of modern language models.

#2 about 4 min

Overcoming language model limitations with context engineering

Retrieval-augmented generation pipelines directly append external facts into prompt streams to prevent outdated intelligence without triggering costly parameter updates.

#3 about 2 min

Extracting dynamic web content using headless browsers

Rendering dynamic client-side applications necessitates migrating from naive HTTP requests to deploying scripted headless browsers.

#4 about 3 min

Bypassing anti-bot protections with proxies and emulation

Scraping systems rely on varied residential proxy rotations and hardware parameter emulation to remain undetected by aggressive firewall endpoints.

#5 about 2 min

Cleaning extraction payloads and scaling crawler infrastructure

Sanitizing retrieved documents with aggressive tag filtering heavily curtails token expenditures during the text embedding phase.

#6 about 2 min

Automating web extraction with cloud-based components

Pre-configured environment modules allow rapid configuration of deep site crawling routines that export directly into compatible markdown formats.

#7 about 4 min

Connecting crawler payloads directly to vector databases

Piping markdown outputs automatically into embedded index caches facilitates immediate deployment of custom generative bots.

#8 about 1 min

Integrating data extraction directly with orchestration frameworks

Packaging extraction routines into standardized containers unlocks immediate interoperability across popular orchestration libraries like Langchain.

#9 about 2 min

Building resilient agent architectures via model context protocols

Adopting dynamic connection protocols empowers autonomous systems to seamlessly adjust to changing endpoint schemas without static API maintenance.

#10 about 3 min

Implementing dynamic tool discovery within agent workflows

Equipping models with protocol-driven discovery endpoints allows them to intuitively locate and deploy specialized automation tools strictly as demanded.

#11 about 1 min

Monetizing community-built extraction scripts via an ecosystem

Exposing modular data scripts to broad developer ecosystems allows engineers to directly capture revenue by servicing niche aggregation needs.

Matching moments

4:20 min

Automating tasks securely through agentic browser protocols

Christian Liebel Christian Liebel · WWC Europe 2026

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann +4 · LIVE

5:54 min

Standardizing agent interactions with the Web MCP proposal

Chris Heilmann +2 · LIVE

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · WWC 2023

3:46 min

Integrating AI into web performance engineering workflows

Perf + AI

1:52 min

Adapting modern web architecture for artificial intelligence crawlers

Eric Enge · Coffee With Developers

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
10 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
12 months ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
12 months ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
12 months ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
12 months ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
12 months ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

MicroAgents: The Microservices of the Agentic Era

Ashish Shubham

Runs the show bussiness at ThoughtSpot

Ashish Shubham
Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

AI-Powered Incident Triage: How We Built GenAI Agents with MCPs to Automate On-Call Workflows

Prakshal Doshi

Site Reliability Engineer

Prakshal Doshi
Open session

World Congress 2026 North America

DeepAgents: Build Multi-Agent AI Systems That Actually Work

Anjana Umapathy, Anagha Rumade, Apoorva Jaiswal

Anjana Umapathy
Anagha Rumade
Apoorva Jaiswal
Open session

World Congress 2026 North America

Agentic commerce's dirty little secret, and what to do about it.

Joe Monastiero

Founder & CEO at visualAI

Joe Monastiero
Open session

World Congress 2026 North America

Securing AI Agent Infrastructure: Identity, Attestation, and Trust at Scale

Abdel Fane

Founder of OpenA2A

Abdel Fane

Related videos

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15
WWC 2025 • 12 months ago

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify) thumbnail
40:29
Coffee With Developers • about 2 months ago

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

MCP Mashups: How AI Agents are Reviving the Programmable Web thumbnail
23:17
WWC 2025 • 12 months ago

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP thumbnail
25:32
WWC 2025 • 12 months ago

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP

Viktoria Semaan

API, MCP or MCP App? Choosing the right surface for AI agents thumbnail
29:59
WWC Europe 2026 • about 1 month ago

API, MCP or MCP App? Choosing the right surface for AI agents

Rishabh Budhiraja, Rishi Deorukhkar

On a Secret Mission: Developing AI Agents thumbnail
29:03
WWC 2025 • 12 months ago

On a Secret Mission: Developing AI Agents

Jörg Neumann

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.