Vidas Bacevičius
Scrape, Train, Predict: The Lifecycle of Data for AI Applications
#1about 2 minutes
Understanding the fundamentals of web scraping
Web scraping is the automated collection of data from websites using a scraper program and proxy servers to handle the request-response cycle.
#2about 2 minutes
Exploring business use cases for scraped data
Scraped data can be used to analyze past trends like SEO rankings and competitor pricing or to predict future trends like market demand.
#3about 4 minutes
Training AI models with custom scraped data
Public datasets like Common Crawl have limitations, so custom web scraping provides fresher, more relevant, and multimodal data for training superior AI models.
#4about 3 minutes
Powering real-time AI with retrieval augmented generation
Retrieval augmented generation (RAG) uses live web scraping to integrate the most current external knowledge directly into an LLM's response generation process.
#5about 7 minutes
Overcoming blocking techniques and messy HTML
Web scrapers face major challenges from anti-bot measures like fingerprinting and CAPTCHAs, as well as from inconsistent and messy HTML structures.
#6about 5 minutes
Using AI classification models to improve scraping
AI classification models trained on labeled HTML data can automatically validate responses to detect blocks and adaptively parse messy content without hardcoded selectors.
#7about 3 minutes
Demonstration of an AI copilot for automated scraping
An AI-powered tool can take a natural language prompt and a list of URLs to automatically generate parsing instructions and extract structured data.
#8about 1 minute
The symbiotic relationship between AI and web scraping
Web scraping provides the fresh, high-quality data that AI models need to function, while AI makes the scraping process itself smarter and more resilient.
Related jobs
Jobs that call for the skills explored in this talk.
Featured Partners
Related Videos
19:20How to scrape modern websites to feed AI agents
Jan Curn
21:17Data is Key: Scraping Metadata from Websites
Lars Kölker
35:16How AI Models Get Smarter
Ankit Patel
28:29Unlocking Value from Data: The Key to Smarter Business Decisions-
Taqi Jaffri, Kapil Gupta, Farooq Sheikh and Tomislav Tipurić
28:09The AI-Ready Stack: Rethinking the Engineering Org of the Future
Jan Oberhauser, Mirko Novakovic, Alex Laubscher, Keno Dreßel
29:00AI in Action: Real Use Cases with Real Impact - Hanna Hennig, Michael Ameling, Tobias Regenfuss
Hanna Hennig, Michael Ameling, Tobias Regenfuss and Mike Butcher
26:08From clicks to cribs - How to find your dream home with web scraping
Alexander Lichter
21:17Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation
Carl Lapierre
From learning to earning
Jobs that call for the skills explored in this talk.
ML & Data Engineer - GenAI & Cloud Infra - Pur Player Data IA ML
WAKE IT UP
Paris, France
Python
Ansible
Terraform
Continuous Integration
Senior - AI Engineer
Predictable Machines
Municipality of San Sebastian, Spain
API
GIT
Kotlin
Docker
TypeScript
Senior - AI Engineer
Predictable Machines
Municipality of Santiago de Compostela, Spain
API
GIT
Kotlin
Docker
TypeScript