Engineering Efficiency

Internet companies have the unique ability to scale quicker than any other industry on earth. Never before has a company been able to position itself from being nothing more than an idea to being in the living rooms of millions around the globe in a matter of hours. While this introduces seemingly unlimited opportunities to grow it also allows for exponential waste if a company isn’t careful. It’s interesting to see how they scale. Scaling businesses in many ways isn’t very different than scaling servers and software stacks.

The Classic Example: UPS

Started in 1907 and adopting the name United Parcel Service in 1919 UPS has no real “high tech” background unless you include the Ford Model T. That doesn’t mean it couldn’t become more efficient. UPS has made a science of the delivery business. For example it’s famous for it’s “no left” policy. Simply put they found that avoiding left turns means less time waiting at lights which means less fuel is wasted. The more efficient routing formerly done by humans now computerized saves them 3 million gallons of fuel in 2007 alone. Lets do the math:

Assuming they run 100% diesel at an average cost of $2.87/gallon in 2007 [doe] multiplied by 3 million that’s $8.61 million dollars by trying to avoid left turns.

Not bad for a souped up mapping application.

By having their drivers do things like turning of the ignition while unbuckling their seat belt at the same time, and scanning for the doorbell while walking towards the door (it’s easier to see from a distance than up close) they can shave time off of their routes.

Then of course there’s package tracking. While customers might like to know in what city their weight loss taps are sitting tracking systems help reduce loss and monitor package routing for optimal efficiency.

Cutting Utility Bills: Google

Being the largest search engine, a large ad network, email provider, analytics firm, mapping service, video site, and whatever else they do means Google needs a ton of servers. Cramming servers into data centers and keeping them cool to prevent hardware failures is a complicated and expensive task. Keeping the whole thing powered is also really expensive. Google has scrutinized server designs to eliminate all waste possible. This has resulted in Google having more horsepower at a lower cost than their competitors. Having more capacity at a lower cost means Google can do more at a lower cost than their competitors. I won’t discuss Google in too much detail since they did a great job themselves recently and I mentioned it the other day in another blog post: Google’s Data Center Secrets.

Shipping Efficiency: Amazon

Amazon’s long been improving efficiency by using data collection and analysis to encouraging their customers to spend more. Their algorithms to recommend related products you might be interested in is one of the best out there. Their ordering system is streamlined to prevent customers from bailing before completion. Their products are SEO’d to appear on the top of Google searches. That doesn’t mean Amazon can’t improve other parts of their business.

Amazon several months ago started a Frustration-Free Packaging program. Here’s how they describe it:

The Frustration-Free Package (on the left) is recyclable and comes without excess packaging materials such as hard plastic clamshell casings, plastic bindings, and wire ties. It’s designed to be opened without the use of a box cutter or knife and will protect your product just as well as traditional packaging (on the right). Products with Frustration-Free Packaging can frequently be shipped in their own boxes, without an additional shipping box.

The key here is “can frequently be shipped in their own boxes”. By shipping a box alone rather than packaging they can skip a step in their warehouses (and the packaging materials that go with packaging something for delivery). This also lowers the weight as those extra boxes don’t weigh 0 oz. The frustration free packaging is also the perfect shape for efficiently filling trucks and strong enough to not crush easily thus lowering returns due to damage.

Amazon now even has a feedback form [login required] for users to share what they think of their package. This has the added bonus of helping further reduce the inefficient shipping practices so common right now.

Amazon’s also done a significant amount of work on their infrastructure to make their servers scale well using tech such as EC2 and S3. By selling capacity to other companies they able to take advantage of economy of scale as well as diversify their business beyond just retail. Of course they are planning their data centers to have access to cheap power.

These aren’t haphazard attempts at increasing efficiency, these are well calculated engineered approaches to removing even the smallest inefficiencies with the knowledge of how they compound as operations scale. Aren’t they clever?

Network Perils

It’s been a week of networking pain. For the past few weeks Comcast has been using a low DHCP lease time. 30 minutes to be exact. This is typical of when they are doing network upgrades/repairs and is what a normal network administrator does. It’s similar to lowering the TTL for DNS.

Renewing the DHCP lease is normally a pretty transparent process, but this time around it’s been causing network connections to drop. While this process is relatively quick, it still resulted in a brief network outage that would kill connections. Occasionally it created a spike when things came back online, which made it feel even longer due to the resulting lag.

No configurations have changed in a long time other than a firmware upgrade a few months ago. Strange I thought. Why is it insisting on on loosing the IP and rediscovering, rather than just renewing? I let it go for a few days hoping Comcast’s low DHCP lease time would be temporary. After 2 weeks I decided to dig deeper.

After a few emails with Motorola’s tech support (Motorola bought Netopia in 2007) they came to the conclusion that the renew-lease ACK packet wasn’t reaching the router. They suspected the firewall wasn’t allowing it to pass, as a result it was dropping the IP, and requesting it again from the DHCP server. They suggested opening up UDP/67 and UDP/68 on the firewall. This seems to resolve the problem. I’m still seeing the lease drop at about 1:00 AM for the past 2 nights, but that’s really a low priority issue and may indeed be on Comcast’s end. It’s possible the router was renewing the hard way every 24-72 hours for a few months, but I know Comcast’s DHCP lease time has been lowered before and the router didn’t exhibit this behavior. Perhaps the firmware upgrade changed the firewalls behavior? I don’t recall that in the docs. Regardless, it’s fixed.

Now today, the UPS for the router, modem and file server’s battery died. Yet another pain. I was able to swap the battery with a similar model UPS from another computer for now. I can deal with that other computer later.

Now maybe I can take my networking hat off for a little while.

Tracking Packages With RSS

What will it take for USPS, FedEx, DHL, and the USPS to offer RSS feeds to track packages. It seems like such a natural idea. Yet none have implemented the feature themselves. It appears the USPS is trying to modernize their image, but still no RSS tracking of packages.

There are several third party sites out there that do this, but to me that’s just asking for trouble since:

  1. What are they doing with my info? Is this really secure to use a stranger as a middle man?
  2. Screen scraping can be unreliable and break. I like reliability.

It’s not that complicated to build, it’s essentially a different template on an existing system. I’d bet they even have it since many large retailers show tracking info within their order status pages. They just don’t make it available to consumers.

Am I really the only one who wants this?