At Search Engine Strategies New York it was announced that you can now have your sitemap automatically discovered by configuring it in your Robots.txt file. It is simple and easy to do, you’ll just need to know the URL or web address of your sitemap.

First, open your Robots.txt file on your server for editing. Then you will need to add the following line to the end of the file (it can be anywhere, but the end is probably a good place).

Sitemap: http://www.mydomain.com/sitemap.xml

Save the Robots.txt file with the new line for the sitemap URL. There you go! Your whole file may look something like this:

User-agent: *
Disallow: /somefolder/
Disallow: /somethingelse/
Sitemap: http://www.soloseo.com/sitemap.php

Search engines already come to your Robots.txt file when they visit your domain, so on their next crawl they will automatically find your sitemap file.

If you have a new site/domain you will probably still want to submit the sitemap URL to the search engines. To submit you can either submit the URL through their interfaces or use a ping.

Submit Sitemap to Google or Ping Google with your Sitemap
Submit Sitemap to Yahoo or Ping Yahoo with your Sitemap
Submit Sitemap to MSN Live.com
Info for Ask.com Sitemaps or Ping Ask.com by hitting this address: http://submissions.ask.com/ping?sitemap=http://www.yourdomain.com/sitemap.xml

 

10 Responses to How to Configure Sitemap Autodiscovery in Robots.txt

  1. [...] Sitemap autodiscovery [Solo SEO] [...]

  2. [...] With the announcement of the new sitemap autodiscovery code for Robots.txt, I hope they add something for this, even though its pretty easy to implement yourself. Digg | StumbleUpon | Del.icio.us | Newsvine | Furl | Reddit | Yahoo! MyWeb [...]

  3. [...] With the announcement of the new sitemap autodiscovery code for Robots.txt, I hope they add something for this, even though its pretty easy to implement yourself. [...]

  4. 1. In your sitemap, you have specified static html pages. What if the majority of your site consists of html pages brought in by Ajax? http://www.soloseo.com/sitemap.php

    2. You mentioned that your site was indexed in one day? How were the keywords selected that related to your test site and were these the ones used in the test? (assume there was no ppc.)

    Thanks.

  5. [...] Create and submit sitemap to Google Webmaster Central. Ping Google, and put the sitemap URL in “robots.txt”. [...]

  6. iCalvyn says:

    i did not know the yahoo sitemap and msn sitemap, only submit to google before.

    Let me submit mine now, thank dude :)

  7. [...] Create and submit sitemap to Google Webmaster Central. Ping Google, and put the sitemap URL in “robots.txt”. [...]

  8. [...] 6) Created a sitemap, pinged Google, and put the sitemap in my Robots.txt. Logged into Google Webmaster Central and submitted my sitemap there. [...]

  9. [...] Create and submit sitemap to Google Webmaster Central. Ping Google, and put the sitemap URL in “robots.txt”. [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

*

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>