LLMs are trained on massive web datasets like Common Crawl, but this leads to knowledge cutoffs and hallucinations.
Retrieval-Augmented Generation (RAG), or context engineering, feeds external, live data to LLMs to produce more accurate and timely answers.
Modern websites use dynamic JavaScript rendering and anti-bot measures, requiring headless browsers, proxies, and CAPTCHA solvers to access data.
To avoid the 'garbage in, garbage out' problem, you must clean HTML by removing cookie banners and ads, and manage complexities like sitemaps and robots.txt.
A demonstration shows how to use the Apify Website Content Crawler to perform a deep crawl of a website and extract its content into markdown.
The scraped website data is uploaded to a Pinecone vector database, enabling a chatbot to answer questions using the site's specific content.
The Model Context Protocol (MCP) provides a fluid, dynamic interface for AI agents to communicate with and discover tools, unlike static traditional APIs.
An AI agent uses MCP to dynamically search the Apify store for a Twitter scraper, add it to its context, and then use it to fetch live data.
msg
Ismaning, Germany
26:15Vidas Bacevičius
1:06:14Chris Heilmann, Daniel Cranney, Ondra Urban & COO & GTM at Apify
23:17Angie Jones
25:32Viktoria Semaan
21:17Carl Lapierre
28:04Phil Nash
24:12Brad Axen
30:42Andre Cipriani Bandarra & François Beaufort
Jobs that call for the skills explored in this talk.
The Rolewe
Charing Cross, United Kingdom
Agile Talent
Collaboration Betters The World GmbH
autonomous-teaming
München, Germany
Capgemini
Westlake, United States of America
Capgemini
Westlake, United States of America
