My observations/conclusions/insights (if you can call it that) can be found after the raw data.

Backend

Frontend

Frontend (cont)

Observations

I retooled this for 2012 based on how web development and the internet has changed as well as the data that’s available. The most noteworthy thing is that EVERY campaign uses open source. Perhaps it’s saving money in this economy? Windows Server isn’t free after all. Most use it extensively. Regardless who wins, that candidate would be very hypocritical to support the (unlikely regardless) “open source is communism” mantra. If this isn’t proof that open source is as mainstream as ever in America, I don’t know what is. Apache is a huge winner. So is jQuery, WordPress, Drupal, even Nginx and Varnish made a showing (they weren’t even on the radar in 2008).

Lots of websites are using the HTML5 doctype now. That doesn’t mean they are using HTML5, but many are moving in that direction. Web Fonts were spotted. Tables were very rare.

Shockingly, CDN usage and HTTP compression were pretty rare. Given Google will host popular javascript frameworks (jQuery for example), if you can’t afford the CPU to gzip data you could let Google host it for free. Lots of cloud hosting though.

Between popular CMS’s, and popular plugins/modules for those CMS’s, there’s little diversity in these sites this time around. It was obvious last time, it’s much more obvious this time. Mono-culture has set in regarding the technicals of these sites.

One thing that really stood out is the amount of analytics on each site. This election is really a data war. Knowing as much as possible about voters and the candidates base. Clearly this is an escalation from 2008. There is also a proliferation of real-time analytics usage this time around. Data is everything in the business world, it’s becoming that way in campaign websites as well.

Misc Notes

Data collection method: The data for this blog post was mostly done on the evening of August 30, 2011 and August 31, 2011 by myself. The character set was however Firefox 6.0 interprets the page. HTML validation was checked by submitting to the W3C validator. All other analysis was done by eye and using tools like cURL. Some things were a little bit of a judgment call, such as CSS layout. I didn’t generally penalize if a table was used, it depends how it was used, and the context. I viewed source on all of them, and spent some time looking around while collecting data. I didn’t view every page on every site, since that would drive me insane. The data is based on the homepage of the site however I did make a brief attempt to hunt for feeds since some only include a link on a “news” page. If I couldn’t find it quick enough, it doesn’t exist.

Secrets In Websites III?: Yes there will likely be a third installment. I don’t know when, I don’t know what will be included. I do have some ideas and notes. It takes time to put these together, and I’m not exactly drowning in free time these days.

* Roy Moore’s website is Flash in an iframe. For purposes of this analysis I’m using the page containing the flash object.

Google, Bitgravity, CDNetworks, DNS.com and Edgecast have deployed support for edns-client-subnet. The idea is pretty simple. It passes part of your IP address (only part as to keep it semi-anonymous) in the request. A server that supports this extension can use it to geotarget and find a CDN node closest to you. Previously the best that could be done was using the location of the DNS server, which in many cases could be far away.

Still missing is support from some heavyweights like Akamai, who is the largest CDN, Limelight Networks and Level3. This is a pretty solid proposal with minimal negative implications. They are only passing part of the origin IP address, so it wouldn’t be a privacy invasion. In theory any website you browse could already harvest the IP you are using, this is just making part of it accessible to a partner who is already serving data on their behalf.

I’ve worked on large websites and setup some websites from DNS to HTML. As a result I’ve got some experience in this realm.

How DNS Works

To understand why this is, you first need to know how DNS works. When you connect to any site, your computer first makes a DNS query to get an IP address for the server(s) that will give the content you requested. For example, to connect to this blog, you’re computer asks your ISP’s DNS servers for robert.accettura.com and it gets an IP back. Your ISP’s DNS either has this information cached from a previous request, or it asks the websites DNS what IP to use, then relays the information back to you.

This looks something like this schematically:

[You] --DNS query--> [ISP DNS] --DNS query--> [Website DNS] --response--> [ISP DNS] --response--> [You]

Next your computer contacts that IP and requests the web page you wanted. The server then gives your computer the requested content. That looks something like this:

[You] --http request--> [Web Server] --response--> [You]

That’s how DNS works, and how you get a basic web page.

How a CDN Works

Now when you’re website gets large enough, you may have servers in multiple data centers around the world, or contract with a service provider who has these servers for you (most contract). This is called a content distribution network (CDN). Parts of, or your entire website may be hosted with a CDN. The idea is that if you put servers close to your users, they will get content faster.

Say the user is in New York, and the server is in Los Angeles. You’re connection may look something like this:

New York : 12.565 ms  10.199 ms
San Jose: 98.288 ms  96.759 ms  90.799 ms
Los Angeles: 88.498 ms  92.070 ms  90.940 ms

Now if the user is in New York and the server is in New York:

New York: 21.094 ms  20.573 ms  19.779 ms
New York: 19.294 ms  16.810 ms  24.608 ms

In both cases I’m paraphrasing a real traceroute for simplicity. As you can see, keeping the traffic in New York vs going across the country is faster since it reduces latency. That’s just in the US. Imagine someone in Europe or Asia. The difference can be large.

The way this happens is a company using a CDN generally sets up a CNAME entry in their DNS records to point to their CDN. Think of a CNAME as an alias that points to another DNS record. For example Facebook hosts their images and other static content on static.ak.facebook.com. static.ak.facebook.com is a CNAME to static.ak.facebook.com.edgesuite.net. (the period at the end is normal). We’ll use this as an example from here on out…

This makes your computer do an extra DNS query, which ironically slows things down! However in theory we make up the time and then some as illustrated earlier by using a closer server. When your computer sees the record is a CNAME it does another query to get an IP for the CNAME’s value. The end result is something like this:

$ host static.ak.facebook.com
static.ak.facebook.com is an alias for static.ak.facebook.com.edgesuite.net.
static.ak.facebook.com.edgesuite.net is an alias for a749.g.akamai.net.
a749.g.akamai.net has address 64.208.248.243
a749.g.akamai.net has address 64.208.248.208

That last query is going to the CDN’s DNS instead of the website. The CDN gives an IP (sometimes multiple) that it feels is closest to whomever is requesting it (the DNS server). That’s the important takeaway from this crash course in DNS. The CDN only sees the DNS server of the requester, not the requester itself. It therefore gives an IP that it thinks is closest based on the DNS server making the query.

The use of a CNAME is why many large websites will 301 you to from foo.com to www.foo.com. foo.com must be an A record. To keep you behind the CDN they 301.

Now lets see it in action!

Here’s what a request from NJ for an IP for static.ak.facebook.com looks like:

$ host static.ak.facebook.com
static.ak.facebook.com is an alias for static.ak.facebook.com.edgesuite.net.
static.ak.facebook.com.edgesuite.net is an alias for a749.g.akamai.net.
a749.g.akamai.net has address 64.208.248.243
a749.g.akamai.net has address 64.208.248.208

Now lets trace the connection to one of these responses:

$ traceroute static.ak.facebook.com
traceroute: Warning: static.ak.facebook.com has multiple addresses; using 64.208.248.243
traceroute to a749.g.akamai.net (64.208.248.243), 64 hops max, 52 byte packets
 1  192.168.x.x (192.168.x.x)  1.339 ms  1.103 ms  0.975 ms
 2  c-xxx-xxx-xxx-xxx.hsd1.nj.comcast.net (xxx.xxx.xxx.xxx)  25.431 ms  19.178 ms  22.067 ms
 3  xe-2-1-0-0-sur01.ebrunswick.nj.panjde.comcast.net (68.87.214.185)  9.962 ms  8.674 ms  10.060 ms
 4  xe-3-1-2-0-ar03.plainfield.nj.panjde.comcast.net (68.85.62.49)  10.208 ms  8.809 ms  10.566 ms
 5  68.86.95.177 (68.86.95.177)  13.796 ms
    68.86.95.173 (68.86.95.173)  12.361 ms  10.774 ms
 6  tengigabitethernet1-4.ar5.nyc1.gblx.net (64.208.222.57)  18.711 ms  18.620 ms  17.337 ms
 7  64.208.248.243 (64.208.248.243)  55.652 ms  24.835 ms  17.277 ms

That’s only about 50 miles away and as low as 17ms latency. Not bad!

Now here’s the same query done from Texas:

$ host static.ak.facebook.com
static.ak.facebook.com is an alias for static.ak.facebook.com.edgesuite.net.
static.ak.facebook.com.edgesuite.net is an alias for a749.g.akamai.net.
a749.g.akamai.net has address 72.247.246.16
a749.g.akamai.net has address 72.247.246.19

Now lets trace the connection to one of these responses:

$ traceroute static.ak.facebook.com
traceroute to static.ak.facebook.com (63.97.123.59), 30 hops max, 40 byte packets
 1  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  2.737 ms  2.944 ms  3.188 ms
 2  98.129.84.172 (98.129.84.172)  0.423 ms  0.446 ms  0.489 ms
 3  98.129.84.177 (98.129.84.177)  0.429 ms  0.453 ms  0.461 ms
 4  dal-edge-16.inet.qwest.net (205.171.62.41)  1.350 ms  1.346 ms  1.378 ms
 5  * * *
 6  63.146.27.126 (63.146.27.126)  47.582 ms  47.557 ms  47.504 ms
 7  0.ae1.XL4.DFW7.ALTER.NET (152.63.96.86)  1.640 ms  1.730 ms  1.725 ms
 8  TenGigE0-5-0-0.GW4.DFW13.ALTER.NET (152.63.97.197)  2.129 ms  1.976 ms TenGigE0-5-1-0.GW4.DFW13.ALTER.NET (152.63.101.62)  1.783 ms
 9   (63.97.123.59)  1.450 ms  1.414 ms  1.615 ms

The response this time is from the same city and a mere 1.6 ms away!

For comparison www.facebook.com does not appear to be on a CDN, Facebook serves this content directly off of their servers (which are in a few data centers). From NJ the ping time averages 101.576 ms, and from Texas 47.884 ms. That’s a huge difference.

Since www.facebook.com hosts pages specifically outputted for the user, putting them through a CDN would be pointless since the CDN would have to go to Facebooks servers for every request. For things like images and stylesheets a CDN can cache them at each node.

Wrapping It Up

Now the reason why using a DNS service like Google’s DNS or OpenDNS will slow you down is that while a DNS query may be quick, you may no longer be using the closest servers a CDN can give you. You generally only make a few DNS queries per pageview, but may make a dozen or so requests for different assets that compose a page. In cases where a website is behind a CDN, I’m not sure that using even a faster DNS service will ever payoff. For smaller sites, it obviously would since this variable is removed from the equation.

There are a few proposals floating around out there to resolve this limitation in DNS, but at this point there’s nothing in place.

• Running Microsoft-IIS 6.0 and ASP.NET 2.0.50727. The Bush administration ran Apache on what I think was some sort of Unix. Data is gzip’d.
• Whitehouse.gov is using Akamai as a CDN and for DNS service.
• Using jQuery 1.2.6 (someone should let them know 1.3 is out). Also using several plugins including jQuery UI, jcarousel, Thickbox. Also using swfobject.
• Pages tentatively validate as XHTML 1.0 Transitional! I’m shocked by this. I’ve checked several pages all with the same result.
• Using WebTrends for analytics. Bush Administration also did.
• IE Conditional Stylesheets and a print stylesheet.
• RSS feeds are actually Atom feeds.
• The website is setting two cookies that I can see WT_FPC and ASP.NET_SessionId which expire at the end of the session which is not prohibited in federal government as per OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002 (using Google Cache for that link since I can’t find it anywhere else, our government should really keep those in a more permanent location).

I should note that this is quite different in architecture than the Obama campaign site which ran PWS/PHP, no notable JS library, feed, and Google Analytics.

Update [1/20/2009 @ 9:00 PM EST]:

• Jason Kottke points out that the new whitehouse.gov sports a much slimmer robots.txt file.
• Content is now under Creative Commons license. Way to go Lawrence Lessig.

The definition of broadband in the US is now being redefined as 768kbps. They will now collect upstream data, and use census-track data. This is a major win since it will more accurately show how many people really do have broadband, and more importantly how many do not.

I personally disagree on the number and think it should be at least 2Mbps, but it’s a win regardless.

The pacific rim annihilates the United States when it comes to broadband. According to Akamai’s State Of The Internet for Q1 2008 high broadband (greater than 5Mbps) is where we really start to show our deficiencies. Here’s a look at broadband which they define as simply greater than 2Mbps:

Pretty pathetic considering our last Vice President invented the Internet . We are the largest in terms of sq miles, but when you consider the US population density, the bulk of our land is very sparsely populated. 80.8% of the US population lives in an urban setting [Warning: PDF].

Japan by comparison has 66.0% of it’s population in an urban setting. Belgium has a surprising 91.5% which may account for it’s #2 position. Switzerland has 44.4% yet makes 3rd place threatening Belgium’s position.

I’m far from the first one to complain about the poor state of broadband. BusinessWeek and CNet both have relatively good discussions about the topic.

The future of media is clearly moving online as people demand to consume it on their schedule as they desire. Take a look at some of the statistics and it’s clearly a large industry. I suspect the lack of broadband infrastructure will be a real problem in the next several years as the rest of the world becomes very easy to distribute media to, and the US still faces challenges.

Solution? Highly debatable, but if so many other countries can do something about it, I suspect it’s achievable here in the US as well. I suspect that the taxes made from companies that do business on the internet from ecommerce to advertising would make this a decent investment for the US government to at least partially back. The more places companies make money, the more places the government does. That may be necessary as not all markets are profitable enough for telco’s to bother with. There have been various attempts to jumpstart this effort, but none to date have been successful.

It’s not only about just having access, it’s also the cost. As BusinessWeek points out in the article above, broadband in the US is not cheap.

Perhaps wireless will finally allow for competition and lower prices, at least that’s what everyone is hoping for. The question is if it will happen, if the technology will be there (wireless is generally high latency), and if it will be affordable for the common man.

I suspect in the next 4 years this will become and even bigger topic of discussion as some of the top ranking countries start to reach the point of saturation.

The Code

WordPress.com

You’ll notice a comment sure to make any web developer laugh on WordPress.com’s login page

	<link rel="stylesheet" href="http://wordpress.com/wp-admin/wp-admin.css?version=MU" type="text/css" />
	<!--[if IE]>
		<style type="text/css">#login h1 a { margin-top: 35px; } #login #login_error { margin-bottom: 10px; }</style>< ![endif]-->
	<!-- Curse you, IE! -->

The guys behind WordPress a while back took the site BrowseHappy under its wing. WordPress has always been a strong believer in web standards, so this isn’t surprising (though still amusing). Did you also know that the guys behind it (Automattic) don’t have job titles? Unless you consider “Chief BBQ Taste Tester” to be a real job title. Matt, I hope your job doesn’t kill you with a heart attack.

Facebook

The geniuses over at Facebook feel the same and put the following on the top of their IE conditionally included stylesheets:

/*  ------------------------------------------------------------------------
                    Facebook | IE/PC Hacks | getfirefox.com
    ------------------------------------------------------------------------  */

popurls

The ever so popular popurls has the following comment in the header of the page.

<!--

  __   __
 (  \,/  )
  \_ | _/  IN THE FUTURE EVERY URL WILL BE POPULAR FOR 1.5 SECONDS
  (_/ \_)                  - thomas and the wise popurls butterfly

-->

RedHat

RedHat was one of the earlier corporate sites to redo itself into a standards based design. They have great respect for those who came before them. In their master css file they have the following tribute as well as a little remark about Netscape 4.x:

/* 	redhat.com MASTER style sheet

	a tip of the red hat to Zeldman, Bowman, Meyer, Shea, Cederholm, Newhouse, Holzschlag,
	and many, many other css and web standards pioneers who have inspired us. 

	the css, layout and validation status of redhat.com is a work-in-progress. numerous
	web-building worker bees are working furiously to correct the bugs, minimize the hacks
	and validate the code. stay tuned. 

-------------------------------------------------------------------- created June 2004 */
@import url("global.css");
...
@import url("dig.css");

/* ---------------------------------------------------------------- ns4 styles - bah! */

table {
	border: 1px;
	}
...

Panic Software

Panic Software has a cool little piece of code for those who browse the product page for Coda (awesome product btw) with IE and don’t have at least version 6.0:

		<!--[if lte IE 6]><p id="iewarning"><img src="/extras/ripoff/images/ie-warning.gif" alt="IE Warning" title="We hear Firefox is nice!" /></p>< ![endif]-->

I hear it’s pretty nice too.

Panic also has a comment in the head of their homepage that reads:

<!-- This homepage design is not long for this world. Enjoy it while you can!   -->

Twitter

Twitter (who redirects to drop the ‘www’ btw) is a very popular service these days. In their html they mark which server served up the data. You’ll see it in the form:

  <!-- served to you through a copper wire by bennu.twitter.com at 24 Nov 19:08 in 11 ms (d 0 / r 8). thank you, come again. -->

Copper eh? No fiber in your data center? I won’t judge, as long as your bandwidth is plentiful.

WordPress.com

Here’s a bonus from WordPress. While many analytics programs use a 1px transparent “tracker gif” to manage statistics, WordPress did something a little different. At the very bottom on the left hand side, you can see the face of WordPress analytics in all it’s tiny glory.

Mozilla

This technically applies to more than just Firefox. You’d be surprised to see how many times kungFuDeathGrip is in the code base.

Many Sites using Google Products/Services

Many people have noticed strange Google tags on sites such as:

code
<!--googleoff: index-->
all
<!--googleon: index-->
over

This isn’t a “SEO” practice, despite some misconception on the web. This is used by the Google Search Appliance, a product made by Google which many websites use to power their own search engines to tell the engine what to read and what to ignore. It wouldn’t be practical for Google to use these “in the wild”. The reason is that spammers could effectively hide an alternate website within those comments. Google’s business is based largely on accurate search results. Spammers have already tried to abuse the css property display: none;. This would be even better. You can find code like this on Apple.com among many other sites.

Webmasters can however optimize their side for AdSense using a technique recommended by Google:

<!-- google_ad_section_start -->

<!-- google_ad_section_end -->

This tells Google to give weight to a certain part of your page when deciding what ad to display on the page. This is good for cases where you feel other material on your page is influencing the ads and resulting in off-topic ads.

Infrastructure/Platform

Microsoft

Microsoft‘s offering against Linux and Apache is IIS on Windows. Which one would expect they themselves use. What they don’t tell you is that they also have used Akamai (with over 25,000 servers), which uses Linux. They have used Akamai for many things like DNS, and caching files. Rather than “Powered By Windows Server” maybe they should append “hiding behind Linux”.

Myspace.com

Myspace.com was previously Adobe/Macromedia’s model customer because it was written in ColdFusion, and said to be the biggest ColdFusion site on the net (and one of the biggest sites on the net). Many think it still is, but it’s not. While many url’s suggest it might be because they end in .cfm it’s actually running ASP.net and has been since aprox, 2006. You can confirm this by viewing the headers on some of their pages. You’ll see:

X-AspNet-Version: 2.0.50727

MTV.com

MTV.com‘s site has search powered by a Google Search Appliance. MTV is also owned by Viacom who sued Google, the parent company of YouTube. The folks at MTV awesomely admitted the irony during relaunch on their blog.

Global Crossing

Tier 1 networking provider Global Crossing really wants you to know how fast they are. Doing a trace could turn up something like this:

  7    15 ms    13 ms    14 ms  COMCAST-IP-SERVICES-LLC.tengigabitethernet1-4.ar5.NYC1.gblx.net [64.208.222.58]
  8    14 ms    13 ms    13 ms  tengigabitethernet1-4.ar5.NYC1.gblx.net [64.208.222.57]

Yes that’s right, they use 10 GigE! Just FYI.

Goofy

Firefox 2.0

In Firefox 2.0, go to “About Firefox” (under the help menu for Windows, under the Firefox menu for Mac), and click on credits. You’ll notice Stephen Colbert. He wrote it single handedly, but added some other names because he’s a nice guy. Bonus: I’m on the list too. Above him because I’m better than him. That’s right, I said it.

Handy

Chase

Chase for some reason puts it’s login form in plain text. The submit url is https, but it doesn’t feel right. They do have a SSL enabled login page, but for some reason they hide it. Here it is for those interested:

https://chaseonline.chase.com/online/home/sso_co_home.jsp

Google

For some reason, most of Google’s services are insecure by default. By simply going to https, you can use SSL for added security.
Gmail: https://mail.google.com
Google Calendar: https://www.google.com/calendar
Google Reader: https://www.google.com/reader

On the next page is the 2008 US Presidential Candidate Campaign sites…

Fast forward to 2007: FoxTorrent is by RedSwoosh (now owned by Akamai).

I’d personally love to see something like this ship built in. It’s a great feature. BitTorrent is a great protocol for distributing large downloads without having to buy expensive infrastructure. Akamai’s interest is proof of that.

FoxTorrent has a blog if you want to keep an eye on it. FoxTorrent is MIT licensed as well. It seems like a very interesting product. I’ll have to dig into this and look at it a bit closer.

[Hat tip: TechCrunch]

14,000 servers in 1,100 networks in 65+ countries.

Just makes you wonder how vulnerable the internet really is. Yea, it’s a web, and not based on a central hub. But it obviously still has problems.

On another note, what a wonderful NOCC.