January 8, 2021
min read

The funniest robots.txt files only developers will understand

Thomas Limbüchler

The robots.txt file is based on the Robots exclusion standard. The file tells Google's web crawler and other bots crawling the web which content to capture. The robots.txt file is thus usually generated by programs and is thus not intended for human eyes. Still, some developers have fun "decorating" their robots.txt files with funny texts or ASCII images.

The robots.txt – a playground for grown-ups

Many of these hidden messages remain true to the filename as they refer to well-known robots. The robot Bender from the popular cartoon series Futurama is part of a whole series of robots.txt files. Or are in a nod to Isaac Asimov's laws of robotics.

Others take up general internet phenomena such as keyboard cat. While companies such as Nike or Starbucks hide their company logo. References to job vacancies are also popular as it shows an unusual, high level of interest in a company.

Bender is a popular motif when creating robots.txt (robotstxt.org.ru):

Back to the Future (arenaflowers.com):

The Google founders want to protect themselves from the terminator (google.de):

The originator of this file should be clear (nike.com):

The robots.txt file of YouTube comes from the future (youtube.com):

There are also Asimov fans at Yelp (yelp.com):

Tripadvisor uses robots.txt to recruit new employees (tripadvisor.com):

Again Bender (tindeck.com):

Keyboard cat in the robots.txt file (sharkseo.com):

And again Bender (reddit.com):

The funniest robots.txt files only developers will understand

January 8, 2021
min read

Subscribe to DevDigest

Get a weekly, curated and easy to digest email with everything that matters in the developer world.

Learn more

From developers. For developers.