WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

How to scrape modern websites to feed AI agents

Jan Curn

Feeding raw HTML to your LLM guarantees garbage outputs. Learn to bypass modern bot blockers to extract clean data. Transform your AI into a web-integrated operator.

Pause
Mute Enter Fullscreen
#1 about 1 min

Role of web scraping in training language models

Extensive public internet crawls remain the foundational mechanism enabling the vast reasoning capabilities of modern language models.

#2 about 4 min

Overcoming language model limitations with context engineering

Retrieval-augmented generation pipelines directly append external facts into prompt streams to prevent outdated intelligence without triggering costly parameter updates.

#3 about 2 min

Extracting dynamic web content using headless browsers

Rendering dynamic client-side applications necessitates migrating from naive HTTP requests to deploying scripted headless browsers.

#4 about 3 min

Bypassing anti-bot protections with proxies and emulation

Scraping systems rely on varied residential proxy rotations and hardware parameter emulation to remain undetected by aggressive firewall endpoints.

#5 about 2 min

Cleaning extraction payloads and scaling crawler infrastructure

Sanitizing retrieved documents with aggressive tag filtering heavily curtails token expenditures during the text embedding phase.

#6 about 2 min

Automating web extraction with cloud-based components

Pre-configured environment modules allow rapid configuration of deep site crawling routines that export directly into compatible markdown formats.

#7 about 4 min

Connecting crawler payloads directly to vector databases

Piping markdown outputs automatically into embedded index caches facilitates immediate deployment of custom generative bots.

#8 about 1 min

Integrating data extraction directly with orchestration frameworks

Packaging extraction routines into standardized containers unlocks immediate interoperability across popular orchestration libraries like Langchain.

#9 about 2 min

Building resilient agent architectures via model context protocols

Adopting dynamic connection protocols empowers autonomous systems to seamlessly adjust to changing endpoint schemas without static API maintenance.

#10 about 3 min

Implementing dynamic tool discovery within agent workflows

Equipping models with protocol-driven discovery endpoints allows them to intuitively locate and deploy specialized automation tools strictly as demanded.

#11 about 1 min

Monetizing community-built extraction scripts via an ecosystem

Exposing modular data scripts to broad developer ecosystems allows engineers to directly capture revenue by servicing niche aggregation needs.

Matching moments

2:21 min

Core concepts and mechanics behind Web MCP

Alex Nahas · Coffee With Developers

4:20 min

Automating tasks securely through agentic browser protocols

Christian Liebel Christian Liebel · World Congress 2026 Europe

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann Chris Heilmann +4 · LIVE

3:35 min

Impact of AI data scraping on open web content

Léonie Watson · A11y + AI

5:54 min

Standardizing agent interactions with the Web MCP proposal

Chris Heilmann Chris Heilmann +2 · LIVE

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · World Congress 2023

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
11 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
about 1 year ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
about 1 year ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
about 1 year ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
about 1 year ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
about 1 year ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Related videos

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15
World Congress 2025 • about 1 year ago

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify) thumbnail
40:29
Coffee With Developers • 3 months ago

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

MCP Mashups: How AI Agents are Reviving the Programmable Web thumbnail
23:17
World Congress 2025 • about 1 year ago

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP thumbnail
25:32
World Congress 2025 • about 1 year ago

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP

Viktoria Semaan

API, MCP or MCP App? Choosing the right surface for AI agents thumbnail
29:59
World Congress 2026 Europe • 3 months ago

API, MCP or MCP App? Choosing the right surface for AI agents

Rishabh Budhiraja, Rishi Deorukhkar

On a Secret Mission: Developing AI Agents thumbnail
29:03
World Congress 2025 • about 1 year ago

On a Secret Mission: Developing AI Agents

Jörg Neumann

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.