question about robots.txt

[Ajay]

would the following block all bots from your site ?

User-agent: *
Disallow:

or do you need the / after Disallow:, like so:

User-agent: *
Disallow: /

[lyzyrdgyzyrd]

I don't know about the first one but the second one works

[Ajay]

EDIT: nevermind i found the answer...

[lyzyrdgyzyrd]

BTW this does not "block" bots, it just tells them "NO".

Some don't listen ;)

[Incognito]

Do we not need the bots to index our sites ?

[Ajay]

yeah, but I like to block them (well, the ones that can be blocked ;) ) while i'm copying scripts over and working on stuff.


Do we not need the bots to index our sites ?

[Incognito]

Ok just cheking LOL

[mcfox]

EDIT: nevermind i found the answer...
Do share. It's kind of the point of a forum. :winkiss:

[lyzyrdgyzyrd]

Do we not need the bots to index our sites ?

There are some rogue bots that may as well be kept at bay if they can be. And maybe not so much so in the "myspace resource" niche, but many webmasters prefer to block imagebots.

[Incognito]

There are some rogue bots that may as well be kept at bay if they can be. And maybe not so much so in the "myspace resource" niche, but many webmasters prefer to block imagebots.


Yep I block the image ones from my pixel membership site.
Don't want all my hand drawn stuff plastered everywhere LOL

[Ajay]

my bad.

disallow: - just means that the bot in question is allowed to access all pages.
and disallow: / - blocks everything on your site from certain bots.

found the info at http://www.robotstxt.org/wc/faq.html

Do share. It's kind of the point of a forum. :winkiss: