nocebo/diaspora
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update robots.txt comments

Browse source 
clarify comments and update permanently moved URL
This commit is contained in:
rivendale 2013-06-14 12:58:02 -03:00
parent c6d2c714e9
commit 9aaf7ed422
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
public/robots.txt
View file

@ -1,6 +1,10 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # See http://www.robotstxt.org/orig.html for documentation on how to use the robots.txt file
# #
# To ban all spiders from the entire site uncomment the next two lines: # To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /
# To ban all spiders from only specific directories such as /people /u or /tag etc.
User-Agent: * User-Agent: *
Disallow: /people/ Disallow: /people/
Disallow: /u/ Disallow: /u/