SoloSEO

How to Configure Sitemap Autodiscovery in Robots.txt

Posted by Michael D Jensen on April 12th, 2007

At Search Engine Strategies New York it was announced that you can now have your sitemap automatically discovered by configuring it in your Robots.txt file. It is simple and easy to do, you’ll just need to know the URL or web address of your sitemap.

First, open your Robots.txt file on your server for editing. Then you will need to add the following line to the end of the file (it can be anywhere, but the end is probably a good place).

Sitemap: http://www.mydomain.com/sitemap.xml

Save the Robots.txt file with the new line for the sitemap URL. There you go! Your whole file may look something like this:

User-agent: *
Disallow: /somefolder/
Disallow: /somethingelse/
Sitemap: http://www.soloseo.com/sitemap.php

Search engines already come to your Robots.txt file when they visit your domain, so on their next crawl they will automatically find your sitemap file.

If you have a new site/domain you will probably still want to submit the sitemap URL to the search engines. To submit you can either submit the URL through their interfaces or use a ping.

Submit Sitemap to Google or Ping Google with your Sitemap
Submit Sitemap to Yahoo or Ping Yahoo with your Sitemap
Submit Sitemap to MSN Live.com
Info for Ask.com Sitemaps or Ping Ask.com by hitting this address: http://submissions.ask.com/ping?sitemap=http://www.yourdomain.com/sitemap.xml

10 comments Visited 18544 times April 12th, 2007 Michael D Jensen

Related Posts:

  • Easy Robots.txt Builder
  • 24 Hour Site Indexing Works Again
  • Yes, you SHOULD Submit a Sitemap to Google
  • New SEO Tool – XML Sitemap Generator
  • Free SEO Tool: XML Sitemap Ping Tool
  • Entry Filed under: How To,Search Engines,SEO,Sitemaps,Tools

    10 Comments Add your own

    • 1. Around the Net 4/16/2007 &hellip  |  April 16th, 2007 at 11:29 pm

      [...] Sitemap autodiscovery [Solo SEO] [...]

    • 2. Easy Robots.txt Builder |&hellip  |  April 25th, 2007 at 8:24 am

      [...] With the announcement of the new sitemap autodiscovery code for Robots.txt, I hope they add something for this, even though its pretty easy to implement yourself. Digg | StumbleUpon | Del.icio.us | Newsvine | Furl | Reddit | Yahoo! MyWeb [...]

    • 3. How to Add Sitemap to Rob&hellip  |  April 25th, 2007 at 12:30 pm

      [...] You can find more information in the excellent Solo SEO Blog [...]

    • 4. Making Money Online &raqu&hellip  |  May 16th, 2007 at 1:57 am

      [...] With the announcement of the new sitemap autodiscovery code for Robots.txt, I hope they add something for this, even though its pretty easy to implement yourself. [...]

    • 5. Sheryl Hamlin  |  January 8th, 2008 at 12:00 pm

      1. In your sitemap, you have specified static html pages. What if the majority of your site consists of html pages brought in by Ajax? http://www.soloseo.com/sitemap.php

      2. You mentioned that your site was indexed in one day? How were the keywords selected that related to your test site and were these the ones used in the test? (assume there was no ppc.)

      Thanks.

    • 6. How To Get Indexed by Goo&hellip  |  March 13th, 2008 at 5:33 pm

      [...] Create and submit sitemap to Google Webmaster Central. Ping Google, and put the sitemap URL in “robots.txt”. [...]

    • 7. iCalvyn  |  March 15th, 2008 at 12:01 am

      i did not know the yahoo sitemap and msn sitemap, only submit to google before.

      Let me submit mine now, thank dude :)

    • 8. Web Design & SEO &raq&hellip  |  March 23rd, 2008 at 12:09 pm

      [...] Create and submit sitemap to Google Webmaster Central. Ping Google, and put the sitemap URL in “robots.txt”. [...]

    • 9. Official Blog - Nairahost&hellip  |  March 25th, 2008 at 8:48 pm

      [...] 6) Created a sitemap, pinged Google, and put the sitemap in my Robots.txt. Logged into Google Webmaster Central and submitted my sitemap there. [...]

    • 10. . . . ::: HeroTurko.Net T&hellip  |  April 4th, 2008 at 3:52 pm

      [...] Create and submit sitemap to Google Webmaster Central. Ping Google, and put the sitemap URL in “robots.txt”. [...]

    Leave a Comment

    Required

    Required, hidden

    Some HTML allowed:
    <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

    Trackback this post  |  Subscribe to the comments via RSS Feed

    Solo SEO Blog


    Subscribe


    Add to Google
    Subscribe in Bloglines

    Calendar

    April 2007
    S M T W T F S
    « Mar   May »
    1234567
    891011121314
    15161718192021
    22232425262728
    2930