nitter/public/robots.txt
minus b6d1947747 Block search engines via robots.txt (#631)
Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
2022-09-14 09:08:40 +02:00

2 lines
26 B
Text