Rare is the web professional who doesn’t know that building a great website isn’t usually enough to guarantee its success. Sites have to be visible in search engines for the keywords and phrases web navigators are most likely to associate with the site’s content. An entire industry has grown up around SEO, search engine optimization. Yawn, you say.
What about the reverse side of the coin, keeping content out of search engines? Should be easy, no? Maybe not. In February, we looked at 5 ways to stop Google and the other search engines from downloading and indexing a website’s pages.
Unfortunately, it appears that the folks behind the personal lubricant astroglide didn’t understand the implications of leaving sensitive customer data on a public web server. They, and their customers, found out how search engines can be all too effective in finding content - as long as it is in a public area and there is a public link to it! Too bad astroglide blamed Google rather than admitting the error of their ways.
The astroglide case isn’t an isolated incident. A consortium of French and German Press in Belgium, Copiepresse, has been battling the search engines to keep Belgium news out of search results – all while blissfully ignoring the circa 1996 robots.txt protocol. It does appear that Copiepresse is finally making progress in learning how to manage what content appears in search engines using existing web conventions.
As of last week, there is an additional tool available to manage content in Yahoo!. By adding a class=”robots-nocontent” attribute to an html tag, a webmaster can specify that content within the html tag shouldn’t appear in Yahoo! As we note in our related class=”robots-nocontent” article, we welcome the additional granularity that this option offers to specify how a search engine indexes a page. Unfortunately its value will be limited until all the major search engines adopt it or a similar syntax.
So have you protected your sensitive content from search engine crawlers?
(No Ratings Yet)
Loading ...
0 responses so far ↓
There are no comments yet...Kick things off by filling out the form below.
Leave a Comment