WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

How to scrape modern websites to feed AI agents

Jan Curn

Feeding raw HTML to your LLM guarantees garbage outputs. Learn to bypass modern bot blockers to extract clean data. Transform your AI into a web-integrated operator.

How to scrape modern websites to feed AI agents thumbnail

Checking access…

Playback and chapters load privately for Free videos.

Matching moments

2:21 min

Core concepts and mechanics behind Web MCP

Alex Nahas · Coffee With Developers

4:20 min

Automating tasks securely through agentic browser protocols

Christian Liebel Christian Liebel · World Congress 2026 Europe

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann Chris Heilmann +4 · LIVE

3:35 min

Impact of AI data scraping on open web content

Léonie Watson · A11y + AI

5:54 min

Standardizing agent interactions with the Web MCP proposal

Chris Heilmann Chris Heilmann +2 · LIVE

1:48 min

Exploring AI integrations in modern agile development workflows

Markus Walker Markus Walker · World Congress 2023

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing · AWS

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel +2

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe +3

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler +3

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher +3

AI Sovereignty: What Does It Take? thumbnail
29:13

AI Sovereignty: What Does It Take?

Boris Hecker +3

Upcoming sessions on this topic

Open session

Build a Zero-Cost AI Agent in Your Browser

  • Aishwarya Mathuria

    Adobe

    Computer Scientist

  • Nishant Thakur

    Adobe

    Senior Computer Scientist

Open session

Beyond File Dumps: Context Engineering for Coding Agents

  • Animesh Dutta

    Arm

    Senior Software Engineer

Open session

Built-In AI in the Browser: Practical Capabilities for Modern Web Apps

  • Piyush Gupta

    Expedia

    Senior Engineering Manager

Open session

The Developer’s Guide to the Agentic Git Forge: Agents That Remember

  • Haimantika Mitra

    Entire

    Senior Developer Advocate

Open session

Teaching AI Coding Agents to Build It Right the First Time

  • Sajeetharan Sinnathurai

    Microsoft

    Principal Product Manager

Open session

Your New Teammates: Architecting a Multi-Agent Workforce

  • Henk Boelman

    GitHub

    Developer Advocate

Related videos

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius · Oxylabs

World Congress · 2025

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify) thumbnail
40:29

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

Coffee With Developers · Jun 2026

MCP Mashups: How AI Agents are Reviving the Programmable Web thumbnail
23:17

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones · Block

World Congress · 2025

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP thumbnail
25:32

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP

Viktoria Semaan · Databricks

World Congress · 2025

API, MCP or MCP App? Choosing the right surface for AI agents thumbnail
29:59

API, MCP or MCP App? Choosing the right surface for AI agents

Rishabh Budhiraja +1

World Congress Europe · 2026

On a Secret Mission: Developing AI Agents thumbnail
29:03

On a Secret Mission: Developing AI Agents

Jörg Neumann · NeoGeeks

World Congress · 2025

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.