Google Sitemap Standard Adopted by Three of the Leading International Search Engines

Each of the three major search engines (Google, Yahoo! and Microsoft’s Live Search) have announced joint support of Google’s sitemaps protocol.

2007-04-11: Ask announces support of the sitemap standard. It is not yet clear if they actually use sitemaps. While Google and Yahoo do process sitemaps, Microsoft does not yet use them.

A new site, www.sitemaps.org, has been created to support the sitemaps protocol. While the Yahoo blog indicates Yahoo is apparently already accepting submissions, there is not mention of this on their Site Explorer submission form. Microsoft is committed to supporting sitemaps after finishing internal testing which is currently underway.

We heartily commend each of the search engines for this constructive collaboration. Large companies often resist industry collaboration - the “it wasn’t invented here” syndrome. Collaboration on sitemaps is the second joint effort this year - each of the major engines now supports the noodp meta tag introduced by Microsoft’s Live Search to exclude open directory / dmoz titles from search results. Google signed-up in July, Yahoo! committed in October. Unfortunately, Ask hasn’t yet pronounced on either initiative.

Keep in mind that while a sitemap can help search engines find otherwise hidden content, they shouldn’t really be used for this purpose – the underlying site architecture issues should be addressed directly.

The true value of a sitemap is to pro-actively notify search engines of new and updated content. This is a win-win for both parties - search engines can better focus their crawling resources and site owners will (hopefully) see their content updated in the engines sooner. Google’s original sitemap format has many advantages over other attempts, such as Yahoo!’s, which we discussed in our article Yahoo! Sitemap Feed Submission… worth the Effort?.

2007-04-11: The sitemap protocol has been extended to support automatic detection by search engines via a robots.txt entry:

Sitemap: <sitemap_location>

We suggest that you avoid using this approach as it facilitates content theft. You should notify each search engine by using a URL encoded ping syntax:

• Ping Google: http://www.google.com/webmasters/sitemaps/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz
• Ping Yahoo: http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz
• Ping Bing: http://www.bing.com/webmaster/ping.aspx?siteMap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz
• Ping Ask.com: http://submissions.ask.com/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz

Similar Posts:

• The Google Webmaster Dashboard, a.k.a. Google Sitemaps
• Now there are 6 ways to keep website content out of search engines
• Search Engine Crawlers: Who’s visiting my site and why?
• 6 methods to control what and how your content appears in search engines
• Search engine detection in Microsoft adCenter Analytics

If you're new here, please subscribe to my RSS feed or subscribe to me on Twitter, which is updated on a more frequent – and more meaningless – basis in English and Italian. Finally, if you're a Sphinn user, Sphinn love is welcome :-). Thanks for visiting!