Google Crawling and Execution of JavaScript: where are we at today?

Share

For a long time, Google’s advice to website developers was to keep things simple to ensure search engine spiders could successfully crawl and process website content:

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.¹

In reality, Google often found links in Flash objects, significantly improving this ability as announced last June (creating much confusion by misrepresenting this as a new feature rather than an improvement). And despite the hoopla, there are still many good reasons to avoid Flash.

JavaScript has been another can of worms. Early search engine crawlers and indexers were not sophisticated enough to process JavaScript. Yet astute observers will note that Google has downloaded external JavaScript files linked in html <script> tags for awhile now (and css for that matter); others noted Google has extracted and followed easily identifiable links in JavaScript code for several years now. Clearly Google hasn’t been completely JavaScript blind.

Yet actual interpretation of JavaScript code presents many problems, as evidenced by the difficult deployment several years ago of the much-maligned Google Web Accelerator.

Yesterday I had two people contact me about Google crawling and actually executing JavaScript (rather than just extracting obvious links). Due to the previous experience with the Google Web Accelerator, I admit I was a bit skeptical. I hadn’t seen an announcement by Google or Google representatives. Yet today, while investigating this hypothesis of Google spidering and following JavaScript, I saw this post on Search Engine Land about Google successfully interpreting a JavaScript based menu system. This particular case may be no more than what Google was already doing 5 years ago.

To what extent does Google use JavaScript today, and what are the implications – both pro and con?

I don’t yet have an up-to-date answer. But while working to figure this out, I’d love to know your experience.

Similar Posts:

• How do search engines, such as Google, handle JavaScript and CSS?
• Flash is still a problem for SEO (and the web) despite Google announcement
• Creating Search Engine Friendly Drop-down menus using CSS
• Web Analytics Embedded JavaScript Page Tracking Code: Place at the top or bottom of the page?
• Web Analytics: Embedded JavaScript Page Tracking Code vs. Web Server Log Files

If you haven't already, you might subscribe to my feed by Email, RSS feed and/or follow me on Twitter, which is updated on a more frequent – and more meaningless – basis. Finally, if you're a Sphinn user, Sphinn love is welcome :-). Thanks for visiting!

Share