Gbrowser Redux
An interesting post on an allegedly new Googlebot. I’ve got no clue about the truth or accuracy of it, but the article thinks Googlebot is no longer a lynx like browser, but based on Mozilla. It would make sense, so Google can take better advantage of things like CSS, JavaScript. Perhaps it’s using <canvas/> to create screenshots for thumbnails?
Again, no clue regarding accuracy, but it’s an interesting read.
Tags: css, gbrowser, google-browser, javascript
February 23rd, 2006 at 7:43 pm
The user agent string is nothing new. I blogged about it back in 2004.
I can confirm that Googlebot is now downloading CSS and JS files. It seems to do so very sparingly (on my site, only once in the last few weeks). Alternate stylesheets are being downloaded as well.
February 23rd, 2006 at 7:48 pm
Correction: there seems to be a new user agent that is slightly different:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Googlebot 2.1
This is the one that is downloading the CSS and JS files, and it is also downloading some of the images in the stylesheets (so far, it has downloaded from my site one list-style-image from the default stylesheet and one background-image from the default stylesheet).
February 23rd, 2006 at 8:03 pm
Okay, here’s yet another correction (sorry). There was only one instance of the above user agent string ever hitting my site, and it very much appears like an actual person, not a bot. The first page hit has a Google search results page as the referrer, regular stuff was requested as Firefox or Mozilla would do, and two images from the default stylesheet were requested, and then nothing. Seems like someone who just clicked a Google result and quickly hit the back button. Perhaps he/she is using a modified user agent string in hopes of getting around some types of website filters (referrer filters for images, possibly some subscription services, etc.)
My logs don’t show any requests for CSS/JS files from previously known Googlebot user agents.
February 24th, 2006 at 2:38 am
It seems not likely that a “new” Googlebot would be version 2.0, as Google’s crawler version has been 2.1 for months (I started logging UAs in february 2005, and then Googlebot version was already 2.1).