Several months ago a client inspired me to write a comprehensive guide to keeping website content out of search engines. Usually website owners are focused on the opposite side of search engine optimization, insuring web content is well indexed. Yet, as many can attest, search engines can be all too efficient at finding documents they shouldn’t. Thus, the need to understand what options exist, how they work and which search engines support them.

One problem with the techniques available up until now is that options for digital media have been limited. The official way to keep video, audio and pdf files out of search engines was through the robots.txt protocol, not a very efficient tool when setting indexing options on a file level.

[Read more →]

Rare is the web professional who doesn’t know that building a great website isn’t usually enough to guarantee its success. Sites have to be visible in search engines for the keywords and phrases web navigators are most likely to associate with the site’s content. An entire industry has grown up around SEO, search engine optimization. Yawn, you say.

What about the reverse side of the coin, keeping content out of search engines? Should be easy, no? Maybe not. In February, we looked at 5 ways to stop Google and the other search engines from downloading and indexing a website’s pages.

[Read more →]