> Markdown version of [/events/world-congress-2025/sessions/590-how-to-scrape-modern](https://www.wearedevelopers.com/events/world-congress-2025/sessions/590-how-to-scrape-modern). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# How to scrape modern websites to feed AI agents


- **Date:** Thursday, Jul 10, 2025
- **Time:** 13:30–14:00 (30 min)
- **Room:** Stage 10
- **Event:** World Congress 2025

## Recording

[Watch recording](https://www.wearedevelopers.com/videos/1446-how-to-scrape-modern-websites-to-feed-ai-agents)

## Description

It was web scraping that quietly enabled the AI revolution.

All major generative AI models and large language models (LLMs) have been trained on massive datasets extracted from the web. But the story doesn’t end there. Today’s AI agents still rely on up-to-date information from online sources to stay relevant, reduce hallucinations, and go beyond their training data. Once again, the web becomes the richest source of context.

Yet scraping the modern web is far from trivial. From anti-bot protections to dynamic, client-side rendering and long-term maintenance issues, building scalable data pipelines has become increasingly complex.

In this talk, we’ll demonstrate how to:

- Set up resilient, maintainable data extraction workflows using cloud-based web scraping Actors,
- Clean messy HTML and structure it into meaningful content,
- Feed that data into a vector database to power a Retrieval Augmented Generation (RAG) pipeline,
- And integrate scraping tools directly into AI agents using the Model Context Protocol (MCP).

Whether you're building copilots, smart assistants, or knowledge-augmented LLM apps, this talk will equip you with the techniques and tools to make the web your AI's best friend.

## Speaker

### [Jan Curn](https://www.wearedevelopers.com/@jan-curn)

Founder and CEO of Apify 


## Related talks at this congress

- [Scrape, Train, Predict: The Lifecycle of Data for AI Applications](https://www.wearedevelopers.com/events/world-congress-2025/sessions/889-scrape-train-predict) — Vidas Bacevičius
- [MCP Mashups: How AI Agents are Reviving the Programmable Web](https://www.wearedevelopers.com/events/world-congress-2025/sessions/559-mcp-mashups-how-ai) — Angie Jones
- [From A2A to MCP: How AI’s “Brains” are Connecting to “Arms and Legs”](https://www.wearedevelopers.com/events/world-congress-2025/sessions/800-from-a2a-to-mcp-how) — Brad Axen
- [Prompt API & WebNN: The AI Revolution Right in Your Browser](https://www.wearedevelopers.com/events/world-congress-2025/sessions/888-prompt-api-webnn-the) — Christian Liebel