less than 1 minute read

If you think they might be hiding something from you, it’s entirely possible that they are.

For those unfamiliar with the robots.txt function, this file keeps all web-robots, such as those employed by search engines, from crawling through the listed directories. The interesting question, for me, is whether I’m more paranoid about what they don’t want me to know, or they’re more paranoid about making sure I don’t know it.

[Thanks, Marcel.]

Leave a comment

Discuss on Mastodon