Lars Kölker
Data is Key: Scraping Metadata from Websites
#1about 2 minutes
How social media sites generate link previews
Social media platforms scrape hidden metadata like titles and descriptions from URLs to transform a simple link into a rich preview.
#2about 1 minute
Defining web scraping and its primary use cases
Web scraping is the practice of gathering data from websites without an API, often used when APIs are missing, rate-limited, or too expensive.
#3about 2 minutes
Why CSS selector-based scraping is brittle
Relying on specific CSS selectors for scraping creates a fragile solution that is tied to a single site and breaks whenever the source code changes.
#4about 4 minutes
Generic scraping with schema.org and JSON-LD
Schema.org provides a standardized vocabulary for structured data, enabling the creation of generic scrapers using formats like JSON-LD.
#5about 5 minutes
Using meta tags for structured data extraction
Protocols like Open Graph (OGP) and Twitter Cards extend standard HTML meta tags to provide rich, structured metadata for social sharing and scraping.
#6about 4 minutes
The oEmbed protocol for embedded content
The oEmbed protocol offers a standardized endpoint for retrieving embeddable representations of a URL, which is essential for sites like Instagram.
#7about 1 minute
Showcasing a powerful multi-protocol scraper
A demonstration shows how combining different scraping techniques can extract rich information, including product prices and author images, from various websites.
#8about 3 minutes
Q&A on legality, rate limits, and frameworks
The speaker addresses audience questions regarding the legality of scraping, managing rate limits, and recommended frameworks like Beautiful Soup.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
01:57 MIN
Presenting live web scraping demos at a developer conference
Tech with Tim at WeAreDevelopers World Congress 2024
05:33 MIN
Navigating a fragmenting web and AI content scraping
Fireside Chat: Can Regulation Improve Accessibility? - Léonie Watson
03:54 MIN
Designing a scalable architecture for data collection
Cracking the Code: Decoding Anti-Bot Systems!
09:56 MIN
The evolution of web scraping for modern applications
WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more
03:28 MIN
Navigating the complexities of modern web scraping
How to scrape modern websites to feed AI agents
00:41 MIN
The symbiotic relationship between AI and web scraping
Scrape, Train, Predict: The Lifecycle of Data for AI Applications
02:42 MIN
Demonstration of an AI copilot for automated scraping
Scrape, Train, Predict: The Lifecycle of Data for AI Applications
04:04 MIN
Training AI models with custom scraped data
Scrape, Train, Predict: The Lifecycle of Data for AI Applications
Featured Partners
Related Videos
43:52Web-based Information Visualization
Johanna Schmidt
50:38Python Data Visualization @ Deepnote (w/ PyViz overview)
Radovan Kavický
26:15Scrape, Train, Predict: The Lifecycle of Data for AI Applications
Vidas Bacevičius
26:08From clicks to cribs - How to find your dream home with web scraping
Alexander Lichter
19:20How to scrape modern websites to feed AI agents
Jan Curn
48:10Data Science on Software Data
Markus Harrer
1:06:14WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more
Chris Heilmann, Daniel Cranney, Ondra Urban & COO & GTM at Apify
32:12Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata
Clemens Vasters
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
&why GmbH
Berlin, Germany
€50-70K
Junior
Intermediate
Senior
React
Next.js
TypeScript
SMG Swiss Marketplace Group
Sachseln, Switzerland
Senior
OCLC
Leiden, Netherlands
Remote
Intermediate
Python
METRO Markets GmbH
Düsseldorf, Germany
GIT
Python
PyTorch
XGBoost
TensorFlow
+1
White Light Digital Marketing
Remote
£30-36K
Junior
API
Python
Data analysis
+2