Easy Robots.txt Builder
Posted by Michael D Jensen on April 25th, 2007
Clickability has created a really nice Robots.txt Builder that helps you to configure your Robots.txt file. You can easily build a Robots.txt file to disallow robots into your file structure. There are options for easily adding web search robots, image search, contextual ads, web archivers, and even “bad robots”. The bad robots puts in a default list of a ton of robots that you can keep out.
With the announcement of the new sitemap autodiscovery code for Robots.txt, I hope they add something for this, even though its pretty easy to implement yourself.
Visited 3391 times April 25th, 2007 Michael D Jensen
Related Posts:
Subscribe
3 Comments Add your own
1. The Dino | May 25th, 2007 at 6:01 am
Nice tool. Thanks for the tip.
2. Noah | January 8th, 2008 at 1:28 pm
Appreciate the tool. I’ve installed and we’ll see if there are any changes.
3. simmons | October 16th, 2008 at 6:22 pm
Hi everyone…I am John from Mexico…I have seen your forum…Nice information provided here which is very useful to everyone…The bad robots puts in a default list of a ton of robots that you can keep out…thanks for posting…Let me know more about this one…..
==================================================================
simmons
Link Building
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed