Vidas Bacevičius
Scrape, Train, Predict: The Lifecycle of Data for AI Applications
#1about 2 minutes
Understanding the fundamentals of web scraping
Web scraping is the automated collection of data from websites using a scraper program and proxy servers to handle the request-response cycle.
#2about 2 minutes
Exploring business use cases for scraped data
Scraped data can be used to analyze past trends like SEO rankings and competitor pricing or to predict future trends like market demand.
#3about 4 minutes
Training AI models with custom scraped data
Public datasets like Common Crawl have limitations, so custom web scraping provides fresher, more relevant, and multimodal data for training superior AI models.
#4about 3 minutes
Powering real-time AI with retrieval augmented generation
Retrieval augmented generation (RAG) uses live web scraping to integrate the most current external knowledge directly into an LLM's response generation process.
#5about 7 minutes
Overcoming blocking techniques and messy HTML
Web scrapers face major challenges from anti-bot measures like fingerprinting and CAPTCHAs, as well as from inconsistent and messy HTML structures.
#6about 5 minutes
Using AI classification models to improve scraping
AI classification models trained on labeled HTML data can automatically validate responses to detect blocks and adaptively parse messy content without hardcoded selectors.
#7about 3 minutes
Demonstration of an AI copilot for automated scraping
An AI-powered tool can take a natural language prompt and a list of URLs to automatically generate parsing instructions and extract structured data.
#8about 1 minute
The symbiotic relationship between AI and web scraping
Web scraping provides the fresh, high-quality data that AI models need to function, while AI makes the scraping process itself smarter and more resilient.
Related jobs
Jobs that call for the skills explored in this talk.
Picnic Technologies B.V.
Amsterdam, Netherlands
Intermediate
Senior
Python
Structured Query Language (SQL)
+1
Wilken GmbH
Ulm, Germany
Senior
Kubernetes
AI Frameworks
+3
Matching moments
04:57 MIN
Increasing the value of talk recordings post-event
Cat Herding with Lions and Tigers - Christian Heilmann
02:39 MIN
Establishing a single source of truth for all data
Cat Herding with Lions and Tigers - Christian Heilmann
03:28 MIN
Why corporate AI adoption lags behind the hype
What 2025 Taught Us: A Year-End Special with Hung Lee
03:15 MIN
The future of recruiting beyond talent acquisition
What 2025 Taught Us: A Year-End Special with Hung Lee
03:48 MIN
Automating formal processes risks losing informal human value
What 2025 Taught Us: A Year-End Special with Hung Lee
05:18 MIN
Incentivizing automation with a 'keep what you kill' policy
What 2025 Taught Us: A Year-End Special with Hung Lee
03:13 MIN
How AI can create more human moments in HR
The Future of HR Lies in AND – Not in OR
04:27 MIN
Moving beyond headcount to solve business problems
What 2025 Taught Us: A Year-End Special with Hung Lee
Featured Partners
Related Videos
19:20How to scrape modern websites to feed AI agents
Jan Curn
21:17Data is Key: Scraping Metadata from Websites
Lars Kölker
35:16How AI Models Get Smarter
Ankit Patel
28:29Unlocking Value from Data: The Key to Smarter Business Decisions-
Taqi Jaffri, Kapil Gupta & Farooq Sheikh and Tomislav Tipurić
29:00AI in Action: Real Use Cases with Real Impact - Hanna Hennig, Michael Ameling, Tobias Regenfuss
Hanna Hennig, Michael Ameling & Tobias Regenfuss and Mike Butcher
28:09The AI-Ready Stack: Rethinking the Engineering Org of the Future
Jan Oberhauser, Mirko Novakovic, Alex Laubscher & Keno Dreßel
26:08From clicks to cribs - How to find your dream home with web scraping
Alexander Lichter
17:01Five things in tech that matter and we have to make work
Christian Heilmann
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Forschungszentrum Jülich GmbH
Jülich, Germany
Intermediate
Senior
Linux
Docker
AI Frameworks
Machine Learning
Eurecat
Barcelona, Spain
C++
Python
PyTorch
Computer Vision
UL Solutions
Barcelona, Spain
Python
Machine Learning
fortice
Remote
£156K
Python
Data analysis
Amazon Web Services (AWS)