Death to Robots.txt

Robots.txt was born on July 3rd 1994 into a life of devoted service, during a time of enormous social change on the Internet.

Robots.txts’ father, Martijn Koster, recognised in 1994 that robots aren’t good or bad - they were simply not going away. The Internet needed a way to manage them to avoid operational problems and grief for webmasters as the bots flooded requests to the web-servers and caused them to crash.

Back in 1994 Robots.txt was a simple yet powerful way of managing crawlers, yet hackers have long exploited the data in robots.txt itself to find vulnerabilities and bypass security. Hackers use robots.txt like a handy map of your website to discover the key paths you actively want to protect, giving them the data to impersonate legitimate bots that you do allow, and then systematically crawl your entire website.

In 2023 however there is a better way.

VerifiedVisitors allows webmasters to manage all bot traffic automatically with our Machine Learning based service. We read and actually enforce the rules you have made in robots.txt that the potentially harmful bots ignore. It’s all automated using our Virtual CISO scanning.

Discover more with our free trial here.