Data centers worldwide now consume more energy annually than Sweden. And the amount of energy required is growing, says Jonathan Koomey, a scientist at Lawrence Berkeley National Laboratory. From 2000 to 2005, the aggregate electricity use by data centers doubled. The cloud, he calculates, consumes 1 to 2 percent of the world’s electricity.

To put that in a little more perspective, the 2009 census for Sweden puts the population at 9,263,872. Sweden’s population is just slightly higher than New York City (8,274,527 in 2007) or the state of New Jersey (8,682,661 estimate in 2008). Granted Sweden’s population density is 20.6/km² compared to New York City’s 10,482/km² or New Jersey’s 438/km². Population density is important since that says a lot about energy consumption. Dense populations require less energy thanks to communal resources. I still suspect the average Swede uses less electricity than the average American anyway. All these numbers were pulled from Wikipedia.

The US Department of Energy does have data on power consumption and capacity as well as forecasts on consumption and production. The obvious downside in the data is the reliance on coal, oil and gas which have environmental impacts as well as political impacts and costs (we know about the instabilities of the oil market). This is why companies with lots of servers like Google are looking very carefully at power generation alternatives such as hydroelectric and solar.

We all benefit from data center efficiency. Lower cost computing is a big advantage to startups and encourages more innovation by removing price barriers. It’s also an advantage to the general public since the technology and tricks learned eventually trickle down to consumers. We already are seeing more efficient power supplies, some even beating the original 80 PLUS certification.

Perhaps if we started tracking “performance per watt” in addition to “watts per square foot” we’d be looking at things from a more sustainable perspective.

Data center capacity and consumption is pretty interesting when you look at all the variables involved. Growth, power costs, facility size, technology available, even foreign politics play a role in what it costs to operate.

The Classic Example: UPS

Started in 1907 and adopting the name United Parcel Service in 1919 UPS has no real “high tech” background unless you include the Ford Model T. That doesn’t mean it couldn’t become more efficient. UPS has made a science of the delivery business. For example it’s famous for it’s “no left” policy. Simply put they found that avoiding left turns means less time waiting at lights which means less fuel is wasted. The more efficient routing formerly done by humans now computerized saves them 3 million gallons of fuel in 2007 alone. Lets do the math:

Assuming they run 100% diesel at an average cost of $2.87/gallon in 2007 [doe] multiplied by 3 million that’s $8.61 million dollars by trying to avoid left turns.

Not bad for a souped up mapping application.

By having their drivers do things like turning of the ignition while unbuckling their seat belt at the same time, and scanning for the doorbell while walking towards the door (it’s easier to see from a distance than up close) they can shave time off of their routes.

Then of course there’s package tracking. While customers might like to know in what city their weight loss taps are sitting tracking systems help reduce loss and monitor package routing for optimal efficiency.

Cutting Utility Bills: Google

Being the largest search engine, a large ad network, email provider, analytics firm, mapping service, video site, and whatever else they do means Google needs a ton of servers. Cramming servers into data centers and keeping them cool to prevent hardware failures is a complicated and expensive task. Keeping the whole thing powered is also really expensive. Google has scrutinized server designs to eliminate all waste possible. This has resulted in Google having more horsepower at a lower cost than their competitors. Having more capacity at a lower cost means Google can do more at a lower cost than their competitors. I won’t discuss Google in too much detail since they did a great job themselves recently and I mentioned it the other day in another blog post: Google’s Data Center Secrets.

Shipping Efficiency: Amazon

Amazon’s long been improving efficiency by using data collection and analysis to encouraging their customers to spend more. Their algorithms to recommend related products you might be interested in is one of the best out there. Their ordering system is streamlined to prevent customers from bailing before completion. Their products are SEO’d to appear on the top of Google searches. That doesn’t mean Amazon can’t improve other parts of their business.

Amazon several months ago started a Frustration-Free Packaging program. Here’s how they describe it:

The Frustration-Free Package (on the left) is recyclable and comes without excess packaging materials such as hard plastic clamshell casings, plastic bindings, and wire ties. It’s designed to be opened without the use of a box cutter or knife and will protect your product just as well as traditional packaging (on the right). Products with Frustration-Free Packaging can frequently be shipped in their own boxes, without an additional shipping box.

The key here is “can frequently be shipped in their own boxes”. By shipping a box alone rather than packaging they can skip a step in their warehouses (and the packaging materials that go with packaging something for delivery). This also lowers the weight as those extra boxes don’t weigh 0 oz. The frustration free packaging is also the perfect shape for efficiently filling trucks and strong enough to not crush easily thus lowering returns due to damage.

Amazon now even has a feedback form [login required] for users to share what they think of their package. This has the added bonus of helping further reduce the inefficient shipping practices so common right now.

Amazon’s also done a significant amount of work on their infrastructure to make their servers scale well using tech such as EC2 and S3. By selling capacity to other companies they able to take advantage of economy of scale as well as diversify their business beyond just retail. Of course they are planning their data centers to have access to cheap power.

These aren’t haphazard attempts at increasing efficiency, these are well calculated engineered approaches to removing even the smallest inefficiencies with the knowledge of how they compound as operations scale. Aren’t they clever?

Buying JungleDisk makes sense since Rackspace wants to get into the cloud storage business. JungleDisk is one of the bigger Amazon S3 products out there. By adding Rackspace support to the product they can quickly attempt to get into that market. If they will succeed depends on their offering’s cost. Their press release suggests $0.15/GB, but that doesn’t say if they will bill based on requests and bandwidth (which is where Amazon S3 gets expensive). Also interesting is this little nugget:

Also later this year, Limelight Networks will team with Rackspace to allow developers to easily distribute content to millions of end users around the world and bring scalable content delivery and application acceleration services to the masses.

This is competing with Amazon’s attempts at starting a CDN later this year. It’s worth noting that these are both pretty primitive CDN’s since they require you to register objects before the CDN hosts them. Modern CDN’s like Limelight and Akamai allow you setup a CNAME so that their CDN essentially acts as a middle layer between your origin servers and your users. This requires no preregistering since the CDN can just check the origin for any asset requested. Caching is configured via configuration files and via standard http headers. I’m not sure how useful these CDN’s will be to most. Registering objects and uploading to another platform is a giant pain as opposed to just setting up a transparent CNAME. The difference is one requires development time, the other doesn’t.

Acquiring Slicehost makes sense since they apparently have technology that will be useful to Rackspace. They are making a bet that startups in need of hosting on virtual machines (which is much more complicated to manage than typical shared hosting) will produce a decent market in the future. With the economic downturn, at least in the short term this may not look like the most useful purchase. In the long run this may pay off handsomely. They have decent competition in that space and it’s quickly growing. Rackspace’s size may help it weather a downturn better than others though.

They closed 5.18 +0.22 (4.44%) today, despite the DOW being -514.45, so I guess I’m not alone in my assessment.

Amazon S3 is cloud based computing. Essentially when you send them a file using their REST or SOAP interface Amazon stores it on multiple nodes in their infrastructure. This provides redundancy and security (in case a data center catches fire for example). Because of this design it’s often though that cloud based computing is invincible to problems. This is hardly the fact. Just like any large system, it’s complicated and full of hazards. It takes only a small software glitch, or an unaccounted for issue to cause the entire thing to grind to a halt. More complexity = more things that can fail.

Amazon S3 is popular because it’s cheap and easy to scale. It’s pay-per-use based on bandwidth, disk storage, and requests. Because that allows for websites to grow without having to make a large infrastructure investment, it’s popular for “Web 2.0″ companies trying to keep their budgets tight. Notably sites like Twitter, WordPress.com, SmugMug and Amazon.com themselves all use Amazon S3 to host things like images.

Many sites, notably Twitter, and SmugMug didn’t have a good day today. WordPress.com and Amazon.com operated like normal. The obvious reason for this is WordPress.com and Amazon.com are much better in terms of infrastructure and design.

WordPress.com uses S3, but proxies that with Varnish. There’s a brief description here, and a more detailed breakdown here. According to Barry Abrahamson, WordPress.com does 1500 image requests per second across and 80-100 are served through S3. They have (slower) back up’s in house for when S3 is down and can failover if S3 has a problem. This means they can leverage S3 to their advantage, but aren’t down because of S3. Using Varnish allows them to keep the S3 bill down by using their own bandwidth (likely cheaper since they are a large site and can get better rates on bandwidth). This also and lets them have this have a good level of redundancy. Awesome job.

Amazon.com uses S3 themselves. If you look at images on the site, they are actually served from g-ecx.images-amazon.com. Which is actually:

g-ecx.images-amazon.com. 38     IN      CNAME   ant.mii.instacontent.net.

instacontent.net is actually part of Mirror Image, a CDN. This is essentially outsourcing what WordPress.com is doing in terms of caching. It’s similar to Akamai’s services. A CDN’s biggest advantage is lowering latency by using servers closer to the customer, which are generally going to feel faster. The other benefit is that they cache content for when the origin is having problems. Because Amazon has a layer on top of S3, they have an added level of protection and remained up and images loaded.

Twitter serves most images such as avatars right off of S3. This means when S3 went down, there were thousands of dead images on their pages. No caching, not even a CNAME in place. Image hosting is the least of their concerns. Keeping the service up and running is their #1 concern right now. The service was still usable, just ugly. Many users take advantage of third party clients anyway.

Using a CDN or having the infrastructure in house is obviously more expensive (it makes S3 more of a luxury than a cost savings measure), but it means your not depending on one third party for your uptime.

Defining uptime, security, and privacy

For the intents of the discussion at hand, “uptime” is defined as the application being accessible and functional to the user. Note putting a “fail whale”? image up so that the page loads doesn’t not qualify as functional. For all intents and purposes the service is down. One should also note that traffic goes through different routes to get to different users, hence a site can be down for one person, but up for millions of others. The vast majority (95%+) should be able to use the service for it to really be considered “up”.

“Security” defined as the assurance that privacy, data integrity, and account access are restricted in accordance with typical site functionality and users understanding. “Privacy” is defined as not allowing any unauthorized person or entity to manipulate, view, copy, handle, destroy, or know about the existence of data without explicit approval from the user.

Why applications fail

Applications can fail for many reasons, but most can be lumped into a handful of categories. At the highest level you have in-house and upstream reasons. In-house can be defined as something you can control, for example software or servers you control, while upstream is typically a vendor or partner, for example ISP, colo facility, etc, which there is less control over (other than submitting a ticket). Generally startups have more upstream services and bring more things in-house as time goes on. For example, Facebook relies on colo facilities for their servers. They now plan to build their own (more control, and hopefully will ensure lower costs as well).

On a lower level you break things down to hardware and software. Hardware failures are inevitable. Computers suck in a 24×7 environment. We deal with that since they are better than people, who still insist on sleep (lazy bastards). Hard drives fry, motherboards fail, fans die resulting in “thermal events”?. Generally it’s pretty easy to deal with this. You can use RAID so 1 hard drive isn’t critical after all, moving parts are the most prone to failure. You can also have more than one server powering a successful application. If one dies, the load goes to other boxes running on the grid. You can put them in different data centers so if there was a problem at one, your still up and running. This obviously comes at a cost. Services like Google App Engine, Amazon’s S3 and Amazon’s EC2 help lower the cost, but also result in hardware being handled by an upstream provider. Amazon and Google are very redundant, but they too can and have failed.

Software generally fails because it either wasn’t designed to scale, or it was hatefully put together to meet a deadline. Startups are infamous for this as the business guys just want things done quick and cheap and don’t care about reliability until it’s too late (they will also deny this until the end of time). All major software platforms can scale when done correctly. Many people say Perl can’t scale, but it has for a decade, look at IMDB, Amazon and Slashdot among the many. Even more claim PHP can’t scale, but Facebook and Yahoo seem to run fine. Python (YouTube), Ruby-On-Rails (YellowPages.com, Hulu, 43things) ASP.NET (MySpace and Microsoft) all seem workable in high traffic situations. It’s not what you use, but how you use it. These run on Apache, IIS, Oracle, MySQL, among others. The platform is rarely (if ever) the problem. The implementation almost always is.

There’s also the possibility that everything is fine and dandy, but somewhere along the internet from the servers to some of your users there’s a problem. ISP’s encounter tons of problems with people snagging their fiber and tearing a line, to DoS attacks and viruses reeking havoc. When this happens close to the user, no sites are accessible, when this happens further away several sites may be inaccessible or slow. Users often wrongly attribute this to a site or application being slow or down when that’s hardly the case. Using a data center with good connectivity reduces these cases. Having data centers distributed around the globe is even better, but often not economical. The best a business can do is submit a ticket and wait. If it’s frequent enough they can move somewhere else.

Why security fails

Security failure is almost too complex of a topic to discuss without holding a complete college course. The most obvious answer is that someone is cleverer than the person in charge of security, and outwitted or outsmarted them. It could be in the physical form (stealing a server or hard drive with data), or in the electronic form (Phishing, XSS, DoS). It could be a “hacker”, or it could be an application failure that results in a security glitch.

Many websites take several measures to protect your privacy. They require “strong” passwords, maybe even require you to change them. For things like banking you may have “security questions” to answer. Perhaps even a key fob to provide two factor authentication.

Most security failure can be traced to stupidity. For example using “password” for your password, or replying to an email asking for your password. A poorly configured server can also be a vulnerability. Then all you need is someone who wants to exploit that. If the data is of any value, that person exists.

When businesses fail

Hackers want your data, business want to keep it secure, but don’t want to spend too much time/effort on it since the formula is time = money. There’s really not much more to explain here.

Sometimes it’s not even the business you know your dealing with. You may be working with company X, but they may use company Y, Z, A1, A2, A3, and A4 to actually provide their services. Your data may be accessible by any or all of them.

Then there’s the possibility of a business going out of business. They may give you a chance to download your data and move it elsewhere or they may even do it for you. They may also just shut down abruptly and disappear of the face of the earth. Goodbye data.

Take control of your data

I may sound cynical for effectively saying applications fail, many people could potentially see your data, and there’s nothing you can do about it. I’m not, I am a realist, and I know what goes on behind the scenes. There actually is something you can do about it: Take control of your data. Keep control of your data.

1. Know who has your data, what they might do to it, who they might share it with, and what they will do to protect it. Companies (at least reputable ones) post privacy policies for a reason. Check them out or Google for some info on that company. The results may surprise you. For example if you delete something from Google Docs it may take 3 weeks for it to actually be deleted on their servers. This isn’t uncommon, but many people assume once you delete it, the company deletes it. That’s not the case.
2. Think about accessibility. What happens if that application has an outage? What happens if your ISP has a problem? Or your cable line got cut? Using an online office suite is a great way to keep documents accessible from home or work, but not so great when you can’t access them. Storing them on a USB drive may prove useful, at least as a backup. If you’ve got a business, this is especially true. You may also want to consider a 2nd way to get online should your ISP have problems (giving a wireless card and a laptop to certain employees may also have the perk of allowing employees to be more mobile).
3. Decide the fate of your data. I personally prefer to keep a copy of everything so if a company goes under, I still have my data. I host my own blog, and my own photos. I keep backups of all that too, in multiple locations. I know I’ll be around as long as I care about keeping that data online. I’m not going out of business. If I am, I don’t care about that data anymore . I always have my data. You should too.

Keep control of your data

Just because you’ve figured out how to protect your data, doesn’t mean you’re done. You need to reevaluate yourself every time you start using something else, or change your usage patterns. You don’t have to keep your data offline, just understand what putting it online really means. Offline backups aren’t a bad idea. Having backups on another service is also an option, but may be even more complicated.

This is somewhat more complicated in the case of things like social networks, but things like Data Portability are slowly becoming a reality.

Google in general has been pretty good with leaving the options to take your data back. Gmail lets you use IMAP to download all your mail, Google Reader lets you export an OPML feed, Google Docs lets you save all your docs to your computer. It’s important to know what the services you rely on let you do with your data. Don’t just assume you can easily get it out.

You’re responsible for your fate

It’s easy to blame Google, Microsoft, Yahoo, or Twitter for your problems, but that’s really a poor excuse. You’re responsible for the choices you make, and what you rely on. If what you’re relying on isn’t giving you what you need, you need to find something else, or reevaluate if your putting your priorities in the right place.

I now present to you…

Accettura’s Law Of Business Computing

where
people = prone to frequent failures
technology = expensive, complex, frequent failure

business computing = people + technology = complex frequent failures that are costly in nature.

You can see how this works right? Best way to avoid that cost? Make sure your technology is redundant, and your people’s interaction is controlled to prevent failure from leaking into the technology.

This should be in Wikipedia and every Business and CompSci textbook. That way everything that a student touches or thinks about in this industry is done with this in mind. Build with the knowledge in mind that the fail whale will just make you a relic before you even hit your prime.

That said, get over Twitter being down and stop complaining.

Edit: All done. Successfully moved to it’s new home.

Update: Server is snug in it’s new home. Will be moving again to a permanent new data center.

Fast forward to 2007: FoxTorrent is by RedSwoosh (now owned by Akamai).

I’d personally love to see something like this ship built in. It’s a great feature. BitTorrent is a great protocol for distributing large downloads without having to buy expensive infrastructure. Akamai’s interest is proof of that.

FoxTorrent has a blog if you want to keep an eye on it. FoxTorrent is MIT licensed as well. It seems like a very interesting product. I’ll have to dig into this and look at it a bit closer.

[Hat tip: TechCrunch]

These projects could include a wind farm in India or a small-scale run of the river hydroelectric project in Brazil. We’re also looking to invest in emerging clean technologies.

Interesting. I wonder if we will see things like carbon neutral VoIP, carbon neutral bandwidth, carbon neutral data centers / colocation / hosting?