Jan Curn
How to scrape modern websites to feed AI agents
#1about 1 minute
Why web data is essential for training large language models
LLMs are trained on massive web datasets like Common Crawl, but this leads to knowledge cutoffs and hallucinations.
#2about 2 minutes
How RAG provides LLMs with up-to-date context
Retrieval-Augmented Generation (RAG), or context engineering, feeds external, live data to LLMs to produce more accurate and timely answers.
#3about 3 minutes
Navigating the complexities of modern web scraping
Modern websites use dynamic JavaScript rendering and anti-bot measures, requiring headless browsers, proxies, and CAPTCHA solvers to access data.
#4about 2 minutes
Cleaning messy HTML and scaling data extraction
To avoid the 'garbage in, garbage out' problem, you must clean HTML by removing cookie banners and ads, and manage complexities like sitemaps and robots.txt.
#5about 3 minutes
Demo of scraping a website with Apify Actors
A demonstration shows how to use the Apify Website Content Crawler to perform a deep crawl of a website and extract its content into markdown.
#6about 2 minutes
Building a RAG chatbot with scraped data and Pinecone
The scraped website data is uploaded to a Pinecone vector database, enabling a chatbot to answer questions using the site's specific content.
#7about 1 minute
Using the Model Context Protocol for AI agent integration
The Model Context Protocol (MCP) provides a fluid, dynamic interface for AI agents to communicate with and discover tools, unlike static traditional APIs.
#8about 3 minutes
Demo of dynamic tool discovery using MCP
An AI agent uses MCP to dynamically search the Apify store for a Twitter scraper, add it to its context, and then use it to fetch live data.
Related jobs
Jobs that call for the skills explored in this talk.
Wilken GmbH
Ulm, Germany
Senior
Kubernetes
AI Frameworks
+3
Picnic Technologies B.V.
Amsterdam, Netherlands
Intermediate
Senior
Python
Structured Query Language (SQL)
+1
Matching moments
04:57 MIN
Increasing the value of talk recordings post-event
Cat Herding with Lions and Tigers - Christian Heilmann
02:54 MIN
Automating video post-production with local scripts
Cat Herding with Lions and Tigers - Christian Heilmann
03:15 MIN
The future of recruiting beyond talent acquisition
What 2025 Taught Us: A Year-End Special with Hung Lee
03:28 MIN
Why corporate AI adoption lags behind the hype
What 2025 Taught Us: A Year-End Special with Hung Lee
03:48 MIN
Automating formal processes risks losing informal human value
What 2025 Taught Us: A Year-End Special with Hung Lee
05:18 MIN
Incentivizing automation with a 'keep what you kill' policy
What 2025 Taught Us: A Year-End Special with Hung Lee
04:27 MIN
Moving beyond headcount to solve business problems
What 2025 Taught Us: A Year-End Special with Hung Lee
03:13 MIN
How AI can create more human moments in HR
The Future of HR Lies in AND – Not in OR
Featured Partners
Related Videos
26:15Scrape, Train, Predict: The Lifecycle of Data for AI Applications
Vidas Bacevičius
1:06:14WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more
Chris Heilmann, Daniel Cranney, Ondra Urban & COO & GTM at Apify
23:17MCP Mashups: How AI Agents are Reviving the Programmable Web
Angie Jones
25:32Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP
Viktoria Semaan
28:04Build RAG from Scratch
Phil Nash
24:12From A2A to MCP: How AI’s “Brains” are Connecting to “Arms and Legs”
Brad Axen
35:16How AI Models Get Smarter
Ankit Patel
21:17Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation
Carl Lapierre
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Forschungszentrum Jülich GmbH
Jülich, Germany
Intermediate
Senior
Linux
Docker
AI Frameworks
Machine Learning
FRG Technology Consulting
Intermediate
Azure
Python
Machine Learning
autonomous-teaming
München, Germany
Remote
API
React
Python
TypeScript
Accenture
Charing Cross, United Kingdom
REST
React
GraphQL
React Native
Continuous Integration
UL Solutions
Barcelona, Spain
Python
Machine Learning