The robots.txt file is based on the Robots exclusion standard. The file tells Google's web crawler and other bots crawling the web which content to capture. The robots.txt file is thus usually generated by programs and is thus not intended for human eyes. Still, some developers have fun "decorating" their robots.txt files with funny texts or ASCII images.
The robots.txt – a playground for grown-ups
Many of these hidden messages remain true to the filename as they refer to well-known robots. The robot Bender from the popular cartoon series Futurama is part of a whole series of robots.txt files. Or are in a nod to Isaac Asimov's laws of robotics.
Others take up general internet phenomena such as keyboard cat. While companies such as Nike or Starbucks hide their company logo. References to job vacancies are also popular as it shows an unusual, high level of interest in a company.
Bender is a popular motif when creating robots.txt (robotstxt.org.ru):
Back to the Future (arenaflowers.com):
The Google founders want to protect themselves from the terminator (google.de):
The originator of this file should be clear (nike.com):
The robots.txt file of YouTube comes from the future (youtube.com):
There are also Asimov fans at Yelp (yelp.com):
Tripadvisor uses robots.txt to recruit new employees (tripadvisor.com):
Again Bender (tindeck.com):
Keyboard cat in the robots.txt file (sharkseo.com):
And again Bender (reddit.com):
