nocebo/diaspora
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #4222 from rivendale2010/patch-1

Browse source 
Update robots.txt comments
This commit is contained in:
Jonne Haß 2013-06-14 09:21:08 -07:00
commit d475a7d935
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
public/robots.txt
View file

@ -1,6 +1,10 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # See http://www.robotstxt.org/orig.html for documentation on how to use the robots.txt file
# #
# To ban all spiders from the entire site uncomment the next two lines: # To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /
# To ban all spiders from only specific directories such as /people /u or /tag etc.
User-Agent: * User-Agent: *
Disallow: /people/ Disallow: /people/
Disallow: /u/ Disallow: /u/