NoLog.cz/nitter
6
0
Fork 1
Code Issues Pull requests 1 Releases Packages 1 Wiki Activity
nitter/public/robots.txt

5 lines
59 B
Text
Raw Normal View History

Block search engines via robots.txt (#631) Prevents instances from being rate limited due to being senselessly crawled by search engines. Since there is no reason to index Nitter instances, simply block all robots. Notably, this does *not* affect link previews (e.g. in various chat software).
2022-06-04 17:48:25 +02:00
User-agent: *
Disallow: /
Fix Twitterbot rule in robots.txt
2022-06-04 17:55:35 +02:00
User-agent: Twitterbot
Disallow:
Reference in a new issue Copy permalink