My observations/conclusions/insights (if you can call it that) can be found after the raw data.

Backend

Frontend

Frontend (cont)

Observations

I retooled this for 2012 based on how web development and the internet has changed as well as the data that’s available. The most noteworthy thing is that EVERY campaign uses open source. Perhaps it’s saving money in this economy? Windows Server isn’t free after all. Most use it extensively. Regardless who wins, that candidate would be very hypocritical to support the (unlikely regardless) “open source is communism” mantra. If this isn’t proof that open source is as mainstream as ever in America, I don’t know what is. Apache is a huge winner. So is jQuery, WordPress, Drupal, even Nginx and Varnish made a showing (they weren’t even on the radar in 2008).

Lots of websites are using the HTML5 doctype now. That doesn’t mean they are using HTML5, but many are moving in that direction. Web Fonts were spotted. Tables were very rare.

Shockingly, CDN usage and HTTP compression were pretty rare. Given Google will host popular javascript frameworks (jQuery for example), if you can’t afford the CPU to gzip data you could let Google host it for free. Lots of cloud hosting though.

Between popular CMS’s, and popular plugins/modules for those CMS’s, there’s little diversity in these sites this time around. It was obvious last time, it’s much more obvious this time. Mono-culture has set in regarding the technicals of these sites.

One thing that really stood out is the amount of analytics on each site. This election is really a data war. Knowing as much as possible about voters and the candidates base. Clearly this is an escalation from 2008. There is also a proliferation of real-time analytics usage this time around. Data is everything in the business world, it’s becoming that way in campaign websites as well.

Misc Notes

Data collection method: The data for this blog post was mostly done on the evening of August 30, 2011 and August 31, 2011 by myself. The character set was however Firefox 6.0 interprets the page. HTML validation was checked by submitting to the W3C validator. All other analysis was done by eye and using tools like cURL. Some things were a little bit of a judgment call, such as CSS layout. I didn’t generally penalize if a table was used, it depends how it was used, and the context. I viewed source on all of them, and spent some time looking around while collecting data. I didn’t view every page on every site, since that would drive me insane. The data is based on the homepage of the site however I did make a brief attempt to hunt for feeds since some only include a link on a “news” page. If I couldn’t find it quick enough, it doesn’t exist.

Secrets In Websites III?: Yes there will likely be a third installment. I don’t know when, I don’t know what will be included. I do have some ideas and notes. It takes time to put these together, and I’m not exactly drowning in free time these days.

* Roy Moore’s website is Flash in an iframe. For purposes of this analysis I’m using the page containing the flash object.

Lets take a look:

66.xxx.xxx.xxx - - [06/Dec/2009:20:17:43 +0000] "GET /test HTTP/1.1" 301 20 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I guess Google has a deal with Twitter. Googlebot indexed just a few seconds after it was sent. As far as I know nothing is actually announced. This is the first evidence I know of a potential deal of some sort. I’d be shocked if Google is scraping the site this quickly.

Edit: Stephen Duncan pointed out in the comments that this was announced in October. Totally forgot about that.

208.xxx.xxx.xxx - - [06/Dec/2009:20:17:47 +0000] "GET /test HTTP/1.0" 301 - "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8"

This is Topsy, a twitter search engine. Never saw this site before. Few tests and I actually kind of like the output.

89.xxx.xxx.xxx - - [06/Dec/2009:20:17:58 +0000] "GET /test HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot"

Tweetmeme mines Twitter links and attempts to build a Digg-like index based on retweets rather than Diggs.

75.xxx.xxx.xxx - - [06/Dec/2009:20:18:05 +0000] "GET /test HTTP/1.1" 301 - "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
72.xxx.xxx.xxx - - [06/Dec/2009:20:20:25 +0000] "GET /test HTTP/1.1" 301 - "-" "Python-urllib/2.5"

Can’t identify these AWS hosted services.

70.xxx.xxx.xxx - - [06/Dec/2009:20:20:53 +0000] "GET /test HTTP/1.1" 301 20 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
70.xxx.xxx.xxx - - [06/Dec/2009:20:24:23 +0000] "GET /test HTTP/1.1" 301 20 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

This is actually Microsoft. Microsoft’s Bing search engine indexes Twitter. I’m not sure why they indexed twice in such close intervals that seems odd for this day and age.

Mining logs a little deeper it looks like when tweets meet certain criteria (such as retweeted) there are other bots that spider them. It also looks like other search engines may be indexing at a slower rate (Baidu for example).

There are several others from AWS and a few other dedicated providers. These servers are obviously trying to keep a low profile, they don’t even have reverse DNS.

So there you go. Just a matter of seconds after a link hits Twitter this all happens.

Here’s a few more from another Tweet that weren’t in the first set:

Edit: More!:

75.xxx.xxx.xxx - - [06/Dec/2009:20:49:42 +0000] "GET /test HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)"

Feedtrace is some sort of twitter mining service currently in beta.

67.xxx.xxx.xxx - - [06/Dec/2009:20:49:45 +0000] "GET /test HTTP/1.0" 301 - "-" "Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)"

Chainn is a social data mining service with a few apps that make use of the data it collects.

Get detailed reporting within minutes after an action occurs on your website. Quickly identify dips in key site metrics or monitor the performance of new content. Seeing the impact of website and marketing changes immediately makes it much easier to optimize them. Yahoo! Web Analytics also maintains historical data so you can go back at any time to review old data for new insight, or compare the present to the past without any changes to your page tags.

Interesting. I wonder if this will light a fire under Google’s butt to deliver real-time analytics as well. Urchin wasn’t really designed for real-time data. Google’s obviously done a lot of work with it to build Google Analytics. I wonder if that’s the next step for them.

The Code

WordPress.com

You’ll notice a comment sure to make any web developer laugh on WordPress.com’s login page

	<link rel="stylesheet" href="http://wordpress.com/wp-admin/wp-admin.css?version=MU" type="text/css" />
	<!--[if IE]>
		<style type="text/css">#login h1 a { margin-top: 35px; } #login #login_error { margin-bottom: 10px; }</style>< ![endif]-->
	<!-- Curse you, IE! -->

The guys behind WordPress a while back took the site BrowseHappy under its wing. WordPress has always been a strong believer in web standards, so this isn’t surprising (though still amusing). Did you also know that the guys behind it (Automattic) don’t have job titles? Unless you consider “Chief BBQ Taste Tester” to be a real job title. Matt, I hope your job doesn’t kill you with a heart attack.

Facebook

The geniuses over at Facebook feel the same and put the following on the top of their IE conditionally included stylesheets:

/*  ------------------------------------------------------------------------
                    Facebook | IE/PC Hacks | getfirefox.com
    ------------------------------------------------------------------------  */

popurls

The ever so popular popurls has the following comment in the header of the page.

<!--

  __   __
 (  \,/  )
  \_ | _/  IN THE FUTURE EVERY URL WILL BE POPULAR FOR 1.5 SECONDS
  (_/ \_)                  - thomas and the wise popurls butterfly

-->

RedHat

RedHat was one of the earlier corporate sites to redo itself into a standards based design. They have great respect for those who came before them. In their master css file they have the following tribute as well as a little remark about Netscape 4.x:

/* 	redhat.com MASTER style sheet

	a tip of the red hat to Zeldman, Bowman, Meyer, Shea, Cederholm, Newhouse, Holzschlag,
	and many, many other css and web standards pioneers who have inspired us. 

	the css, layout and validation status of redhat.com is a work-in-progress. numerous
	web-building worker bees are working furiously to correct the bugs, minimize the hacks
	and validate the code. stay tuned. 

-------------------------------------------------------------------- created June 2004 */
@import url("global.css");
...
@import url("dig.css");

/* ---------------------------------------------------------------- ns4 styles - bah! */

table {
	border: 1px;
	}
...

Panic Software

Panic Software has a cool little piece of code for those who browse the product page for Coda (awesome product btw) with IE and don’t have at least version 6.0:

		<!--[if lte IE 6]><p id="iewarning"><img src="/extras/ripoff/images/ie-warning.gif" alt="IE Warning" title="We hear Firefox is nice!" /></p>< ![endif]-->

I hear it’s pretty nice too.

Panic also has a comment in the head of their homepage that reads:

<!-- This homepage design is not long for this world. Enjoy it while you can!   -->

Twitter

Twitter (who redirects to drop the ‘www’ btw) is a very popular service these days. In their html they mark which server served up the data. You’ll see it in the form:

  <!-- served to you through a copper wire by bennu.twitter.com at 24 Nov 19:08 in 11 ms (d 0 / r 8). thank you, come again. -->

Copper eh? No fiber in your data center? I won’t judge, as long as your bandwidth is plentiful.

WordPress.com

Here’s a bonus from WordPress. While many analytics programs use a 1px transparent “tracker gif” to manage statistics, WordPress did something a little different. At the very bottom on the left hand side, you can see the face of WordPress analytics in all it’s tiny glory.

Mozilla

This technically applies to more than just Firefox. You’d be surprised to see how many times kungFuDeathGrip is in the code base.

Many Sites using Google Products/Services

Many people have noticed strange Google tags on sites such as:

code
<!--googleoff: index-->
all
<!--googleon: index-->
over

This isn’t a “SEO” practice, despite some misconception on the web. This is used by the Google Search Appliance, a product made by Google which many websites use to power their own search engines to tell the engine what to read and what to ignore. It wouldn’t be practical for Google to use these “in the wild”. The reason is that spammers could effectively hide an alternate website within those comments. Google’s business is based largely on accurate search results. Spammers have already tried to abuse the css property display: none;. This would be even better. You can find code like this on Apple.com among many other sites.

Webmasters can however optimize their side for AdSense using a technique recommended by Google:

<!-- google_ad_section_start -->

<!-- google_ad_section_end -->

This tells Google to give weight to a certain part of your page when deciding what ad to display on the page. This is good for cases where you feel other material on your page is influencing the ads and resulting in off-topic ads.

Infrastructure/Platform

Microsoft

Microsoft‘s offering against Linux and Apache is IIS on Windows. Which one would expect they themselves use. What they don’t tell you is that they also have used Akamai (with over 25,000 servers), which uses Linux. They have used Akamai for many things like DNS, and caching files. Rather than “Powered By Windows Server” maybe they should append “hiding behind Linux”.

Myspace.com

Myspace.com was previously Adobe/Macromedia’s model customer because it was written in ColdFusion, and said to be the biggest ColdFusion site on the net (and one of the biggest sites on the net). Many think it still is, but it’s not. While many url’s suggest it might be because they end in .cfm it’s actually running ASP.net and has been since aprox, 2006. You can confirm this by viewing the headers on some of their pages. You’ll see:

X-AspNet-Version: 2.0.50727

MTV.com

MTV.com‘s site has search powered by a Google Search Appliance. MTV is also owned by Viacom who sued Google, the parent company of YouTube. The folks at MTV awesomely admitted the irony during relaunch on their blog.

Global Crossing

Tier 1 networking provider Global Crossing really wants you to know how fast they are. Doing a trace could turn up something like this:

  7    15 ms    13 ms    14 ms  COMCAST-IP-SERVICES-LLC.tengigabitethernet1-4.ar5.NYC1.gblx.net [64.208.222.58]
  8    14 ms    13 ms    13 ms  tengigabitethernet1-4.ar5.NYC1.gblx.net [64.208.222.57]

Yes that’s right, they use 10 GigE! Just FYI.

Goofy

Firefox 2.0

In Firefox 2.0, go to “About Firefox” (under the help menu for Windows, under the Firefox menu for Mac), and click on credits. You’ll notice Stephen Colbert. He wrote it single handedly, but added some other names because he’s a nice guy. Bonus: I’m on the list too. Above him because I’m better than him. That’s right, I said it.

Handy

Chase

Chase for some reason puts it’s login form in plain text. The submit url is https, but it doesn’t feel right. They do have a SSL enabled login page, but for some reason they hide it. Here it is for those interested:

https://chaseonline.chase.com/online/home/sso_co_home.jsp

Google

For some reason, most of Google’s services are insecure by default. By simply going to https, you can use SSL for added security.
Gmail: https://mail.google.com
Google Calendar: https://www.google.com/calendar
Google Reader: https://www.google.com/reader

On the next page is the 2008 US Presidential Candidate Campaign sites…

This of course raises an interesting question. How do tabs influence this metric? Take the following situation as an example. A user visits a home page, and opens a link in a new tab. Then finds another link and opens it in a new [background] tab. That’s 3 tabs in 1 visit (assume visit to be 30 minutes).

Before tabs, most browser sessions would look like this:

There’s now an increasing number that will look like this (gray is a tab not in view):

If we assume total time on the site is time between the first and last page, we potentially undercount the total time on sites that list information (for example Digg). The total time to make those clicks could be < 10 seconds, but the time spent reading those two page alone might be > 10 minutes. Many tab power-users from what I’ve read around the web over the years essentially use them as a way to bookmark their “to read” list (including myself). It also undercounts sites like Gmail which are ajax based (1 page) but can be used for several minutes.

If we use javascript to “ping” (call back by placing a tracker gif) the analytics service every x seconds to see if the page is still open, we potentially double count since a user can’t be in 3 tabs at once. The clock would be counting 3 seconds for every 1 second the user is actually looking at the page.

This raises the question: are sites that are heavily used by Firefox, Safari, Opera and IE7 site underestimated or overestimated because of the way users browse the site? How do you accurately tell how long a view is when a user can have multiple tabs?

Another example is someone who keeps their webmail open in a tab all afternoon for easy access. They may only check it 1x measuring no more than 1 minute in actual attention. But it’s open for 5 hrs. What is the real time on the page? You can measure my interaction (opening/closing mail). But what if I’m reading an email for an hour (it’s a really complicated one)? How does that compare to just leaving it open in the background?

This is really no different than using new windows, the difference being that most people seem to have found windows to be annoyance, while tabs are a “feature”. The increase in usage and popularity in a time where visit length matters raises an interesting question. How do you measure it?

One assumption is that it’s just a small percentage of the population, which is likely true. The problem with this assumption is that it’s one subject to change as the browserscape matures and users learn about new features. Another assumption is to just account for all time a page is open, even if it’s not visible. The downside I see here is that it’s pretty inaccurate. As a content producer I’d like to know if my content is used, or just loaded on a users computer. If I were an advertiser I’d care even more.

I’m not sure how analytics firms approach this. In a sense it’s similar to the “hotel problem“. Perhaps just something you need to decide upon and live with.