Once we have created the all important sitemap.xml file, we must inform the search engine crawlers of its location and there are several methods of doing his:
- send an HTTP request
- upload the sitemap.xml file directly to the search engines
- add the location of the sitemap in your robots.txt file
In this blog entry I will only address # 3. While I do upload my sitemap.xml files to the major search engines, I also add the location of the sitemap file to the robots.txt file because all crawlers that look at the robots.txt file wil be directed to my sitemaps and not just the search engines where I manually uploaded the site map. The following is the actual robots.txt file from http://webdesign-ri.com
User-agent: * Sitemap: http://webdesign-ri.com/sitemap.xml Sitemap: http://webdesign-ri.com/sitemap.htm Disallow: /cgi-bin/ Disallow: /case_studies/
In the second line I have provided the path for the sitemap.xml file and in the 3rd line I have added a path to the sitemap.htm file.
