> Markdown version of [/videos/708-data-is-key-scraping-metadata-from-websites](https://www.wearedevelopers.com/videos/708-data-is-key-scraping-metadata-from-websites). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data is Key: Scraping Metadata from Websites

Tired of scrapers breaking after minor UI updates? Stop relying on brittle CSS selectors. Build resilient metadata extractors using JSON-LD and Open Graph protocols.

- **Speakers:** Lars Kölker
- **Event:** World Congress 2023
- **Published:** September 27, 2023
- **Duration:** 21:17
- **URL:** https://www.wearedevelopers.com/videos/708-data-is-key-scraping-metadata-from-websites

## Summary

Generating rich link previews requires robust metadata extraction, but traditional parsing using complex CSS selectors is highly brittle and tied rigidly to specific site structures. When official APIs are unavailable, prohibitively expensive, or heavily rate-limited, developers need a generalized, resilient approach to scrape web data without breaking upon minor UI updates. Relying on raw HTML structure fundamentally limits the scalability of scrapers across diverse domains.

Modern metadata scraping overcomes these limitations by targeting structured contextual data rather than raw DOM elements. The most robust method utilizes the schema.org specification mapped via json-ld syntax, which serves as the standard recommended by Google for search engine indexing. When json-ld data is absent, developers can securely fall back on the open graph protocol and embedded twitter cards. These fallback formats inject rich object properties directly into standard html meta tags, universally exposing actionable insights like product pricing, stock availability, and article authorship.

For heavily structured platforms like Instagram or Facebook, oembed provides a standardized format for retrieving embedded representation data via targeted json or xml endpoints. Locating oembed data typically relies on sniffing alternate link header tags, which allows publishers to effectively decouple link previews from database-driven html file modifications. To build platform-agnostic extractors, developers must aggregate these diverse markup layers. While tools like beautiful soup streamline this logic in Python, JavaScript ecosystems often necessitate custom abstraction. Regardless of the underlying technology, operators must maintain legal compliance by respecting robots.txt directives and managing rate limits responsibly to avoid overwhelming host architectures.

**Keywords:** metadata extraction, link preview generation, structured data markup, schema.org specification, json-ld syntax, open graph protocol, twitter cards fallback, oembed endpoints, dom querying limitations, css selector fragility, api monetization constraints, html meta tags, robots.txt compliance, rate limit management, beautiful soup parsing

## Chapters

1. **How social media platforms generate link previews** (00:05) — Social platforms execute transformations that read hidden markup to display titles, descriptions, and dynamic content on links.
1. **Defining web scraping and recognizing proper use cases** (01:25) — Scraping becomes essential when target systems lack transparent APIs or when existing endpoints become prohibitively expensive.
1. **Limitations of specific CSS selectors for web scraping** (02:45) — Hardcoding CSS selectors restricts scraper versatility and creates breaking points whenever an underlying website updates its structure.
1. **Using schema.org for generic structured data extraction** (05:04) — Schema.org libraries define standard rules for machine-readable types and generalized properties across various content models.
1. **Extracting schema attributes mapped to JSON-LD structures** (06:59) — Adopting Google's JSON-LD format reveals precise relational metadata stored inside predefined script elements.
1. **Extracting basic key-value data with common meta elements** (08:29) — Standard HTML attributes like application name and description offer a foundational layer for key-value extraction routines.
1. **Using Open Graph Protocol to construct rich objects** (09:36) — Evaluating Open Graph specification tags accesses dedicated attribute values originally standardized by Facebook for media sharing.
1. **Extracting Twitter Cards markup as valuable scraping fallbacks** (11:48) — Falling back on Twitter-specific metadata summary tags preserves robust link scraping capabilities whenever Open Graph formats miss details.
1. **Utilizing the oEmbed format to query rich endpoints** (13:37) — Fetching an endpoint defined via oEmbed's alternate link attributes returns complex API representations of deep resources.
1. **Demonstrating successful metadata scraping techniques for generic previews** (17:10) — Chaining standard schema fallbacks populates functional link previews containing critical information like live pricing and stock changes.
1. **Navigating legality boundaries and request rate limits safely** (18:12) — Reviewing platform terms of service and restricting endpoint concurrency significantly reduces the compliance risk involved in external scraping protocols.
1. **Selecting suitable extraction frameworks for structured schema parsers** (20:08) — Dedicated Python libraries streamline generalized document parsing, though JavaScript environments often require custom logic to evaluate JSON-LD schemas effectively.

## Related Moments

- [The technical evolution of modern web scraping infrastructure](https://www.wearedevelopers.com/videos/1764-wearedevelopers-live-web-scraping-agents-actors-and-more) (from "WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more")
- [Exploring data collection challenges from user-generated platforms](https://www.wearedevelopers.com/videos/78-leverage-cloud-computing-benefits-with-serverless-multi-cloud-ml) (from "Leverage Cloud Computing Benefits with Serverless Multi-Cloud ML ")
- [Technical workflow for generating self-healing scrapers](https://www.wearedevelopers.com/videos/100244-marketing-x-product-how-we-stopped-gaslighting-each-other-and-built-ai-products-that-actually-work) (from "Marketing x Product: How We Stopped Gaslighting Each Other and Built AI Products That Actually Work")
- [Concluding thoughts on scraper tools and audience questions](https://www.wearedevelopers.com/videos/767-from-clicks-to-cribs-how-to-find-your-dream-home-with-web-scraping) (from "From clicks to cribs - How to find your dream home with web scraping")
- [Optimizing social media link previews with header attributes](https://www.wearedevelopers.com/videos/1340-wearedevelopers-live-performance-and-ai-social-media-decline-developer-events-and-more) (from "WeAreDevelopers LIVE - Performance and AI?, Social Media decline, Developer Events and more")
- [Scraping web data by intercepting JSON HTTP responses](https://www.wearedevelopers.com/videos/322-automate-everything-via-nodejs-and-puppeteer) (from "Automate everything via NodeJS and Puppeteer")

## Related Articles

- [The Web We Broke (And Why AI Agents Are Paying the Price) - AgentCon Berlin](https://www.wearedevelopers.com/magazine/735-the-web-we-broke-and-why-ai-agents-are-paying-the-price-agentcon-berlin)
- [Dev Digest 133 - Back to Front](https://www.wearedevelopers.com/magazine/474-dev-digest-133-back-to-front)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)
- [WebMCP: Empowering Agents as First-Class Citizens of the Web](https://www.wearedevelopers.com/magazine/696-webmcp-empowering-agents-as-first-class-citizens-of-the-web)

## Related Jobs

- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/645320-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/644637-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/646086-remote-senior-full-stack-engineer) at **Edge Impulse**