WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
Aug 22, 2024

Command the Bots: Mastering robots.txt for Generative AI and Search Marketing

Fili Wiese

Did you know a 500-level error on your robots.txt can deindex your entire site? Discover the layered server directives developers need to safely block LLM training bots.

Command the Bots: Mastering robots.txt for Generative AI and Search Marketing thumbnail

Checking access…

Playback and chapters load privately for Free videos.

Matching moments

53 sec

Identifying technical hurdles with aggressive AI web crawlers

Klaus-M. Schremser Klaus-M. Schremser · World Congress 2026 Europe

1:31 min

Blocking AI training crawlers by default on networks

Stephanie Cohen Stephanie Cohen +1 · World Congress 2025

2:13 min

Defending domains against aggressive indexing crawlers

Chris Heilmann Chris Heilmann +2 · LIVE

2:27 min

Combating automated artificial intelligence content with brand authority

Eric Enge · Coffee With Developers

1:51 min

Managing automated agents and maintaining secure application access

Ramona Schwering Ramona Schwering · Coffee With Developers

1:52 min

Adapting modern web architecture for artificial intelligence crawlers

Eric Enge · Coffee With Developers

Upcoming sessions on this topic

Open session

Build a Zero-Cost AI Agent in Your Browser

  • Aishwarya Mathuria

    Adobe

    Computer Scientist

  • Nishant Thakur

    Adobe

    Senior Computer Scientist

Open session

Engineering AI Strategy: Beyond the Hype

  • Omkar Srivastava

    ADSK

    Engineering Leader

Open session

Beware of Strangers Bearing Code: Open Source Trust in the Agent Era

  • Vikram Vaswani

    Consultant

Open session

Built-In AI in the Browser: Practical Capabilities for Modern Web Apps

  • Piyush Gupta

    Expedia

    Senior Engineering Manager

Open session

Teaching AI Coding Agents to Build It Right the First Time

  • Sajeetharan Sinnathurai

    Microsoft

    Principal Product Manager

Open session

The Illusion of Green: Why AI-Generated Browser Automation Tests Pass Locally but Fail in CI

  • Puja Jagani

    BrowserStack

    Lead of Open Source and Developer Advocacy

Related videos

ChatGPT vs Google: SEO in the Age of AI Search - Eric Enge thumbnail
36:44

ChatGPT vs Google: SEO in the Age of AI Search

Eric Enge

Coffee With Developers · Jan 2025

AI Search Insights from OtterlyAI: What We Tested, What Failed, and What Actually Works thumbnail
29:54

AI Search Insights from OtterlyAI: What We Tested, What Failed, and What Actually Works

Klaus-M. Schremser · OtterlyAI

World Congress Europe · 2026

WeAreDevelopers LIVE – SEO, GEO, AI Slop & More thumbnail
1:02:24

SEO, GEO, AI Slop & More

Chris Heilmann +2

LIVE · Nov 2025

Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations thumbnail
36:42

Modern Web Data Extraction: Techniques, Tools, and Legal & Ethical Considerations

Domagoj Marić · Pontis Technology

Europe Virtual · 2026

Cracking the Code: Decoding Anti-Bot Systems! thumbnail
57:47

Cracking the Code: Decoding Anti-Bot Systems!

Fabien Vauchelles

LIVE · May 2024

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more thumbnail
1:01:25

Browser Extensions, Honey Scam, Jailbreaking LLMs and more

Chris Heilmann +1

LIVE · Jan 2025

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.