Ask The Experts About Robots.txt

Robert asks…

Is it possible to view an archive of a site that has robots.txt?

I am looking for an archive of a post that someone had made on a site, but they have since left the site, and thus their post is gone as well. The site does not allow robots, so google and the likes do not have cached pages for it.

Any help?

admin answers:

If you know the exact post url, you could see it as long as it wasn’t deleted, or set to private.

Powered by Yahoo! Answers

Incoming search terms:

Ask The Experts About Robots.txt Disallow

Betty asks…

Can you “allow” a search engine in a robots.txt rather than a disallow?

I want to block ALL search engines from one of the sub directory folders on my website. I want to allow one search engine though so that my staff can search that section easily using a site search engine.

There are so many search engines out there, I wouldn’t be able to name them all in my long list of disallows so I am wondering is there is any line of code to include in a robots that would allow me to disallow all search engines except for the one???

Please advise. Thanks!

admin answers:

Yes, you can. I’m not sure of the code due to I have no need for it. But this is a little tool that you can use to allow and disallow bots.

Powered by Yahoo! Answers

Page 1 of 41234
line
footer
Powered by Wordpress | Designed by Elegant Themes