NoLog.cz/nitter-nolog
Archived
6
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
This repository has been archived on 2023-09-07. You can view files and clone it, but cannot push or open issues or pull requests.
nitter-nolog/public/robots.txt

5 lines
58 B
Text
Raw Normal View History

Explicitly allow Twitterbot to generate previews
2022-06-04 17:52:47 +02:00
User-agent: Twitterbot
Allow: /
Block search engines via robots.txt (#631) Prevents instances from being rate limited due to being senselessly crawled by search engines. Since there is no reason to index Nitter instances, simply block all robots. Notably, this does *not* affect link previews (e.g. in various chat software).
2022-06-04 17:48:25 +02:00
User-agent: *
Disallow: /
Reference in a new issue Copy permalink