//Using Robots.txt to inform crawlers of a sitemap

Using Robots.txt to inform crawlers of a sitemap

Once we have created the all important sitemap.xml file,  we must inform the search engine crawlers of its location and there are several methods of doing his:

  1. send an HTTP request
  2. upload the sitemap.xml file directly to the search engines
  3. add the location of the sitemap in your robots.txt file

In this blog entry I will only address # 3. While I do upload my sitemap.xml files to the major search engines, I also add the location of the sitemap file to the robots.txt file because all crawlers that look at the  robots.txt file wil be directed to my sitemaps and not just the search engines where I manually uploaded the site map. The following is the actual robots.txt file from http://webdesign-ri.com

User-agent: *
Sitemap: http://webdesign-ri.com/sitemap.xml
Sitemap: http://webdesign-ri.com/sitemap.htm
Disallow: /cgi-bin/
Disallow: /case_studies/

In the second line I have provided the path for the sitemap.xml file and in the 3rd line I have added a path to the sitemap.htm file.

The following two tabs change content below.
After a long career in sales, marketing and engineering design, within the manufacturing sector, George launched Web Strategic Marketing in 2009 to enable businesses to be able to effectively compete in the new world of Internet and Social Media Marketing. With the ability to develop marketing plans and marketing strategies, web site development (HTML and WordPress), search engine optimization, social media marketing, write copy and shoot product/commercial photography Web Strategic Marketing is your one-stop-shop for your Internet marketing needs.

Latest posts by George Ross (see all)

By |2018-07-31T05:21:04-05:00September 12th, 2009|Social Media|0 Comments

Leave A Comment

Humans please complete * Time limit is exhausted. Please reload CAPTCHA.