Measurement Lab

Google today unwrapped Measurement Lab (M-Lab) which can measure connection speed, run diagnostics and check if your ISP is blocking or throttling things via it’s blog.

In general it’s a good idea, but it’s nothing new. Sites like dslreports.com and SpeedTest.net have been measuring speed and providing diagnostics for years. The BitTorrent test however isn’t replicated by many yet.

One thing that isn’t addressed is how they will detect if an ISP is adjusting their routing to handle M-Lab servers specially. What stops an ISP from not throttling from one of Google’s “36 servers in 12 locations in the U.S. and Europe” but throttling all other data? Perhaps Vint Cerf and friends have a plan in mind but it seems to me this could be a cat and mouse game.

BitTorrent For HTTP Failover

There is a proposal circulating around the web to create a X-Torrent HTTP header for the purpose of pointing to a torrent file as an alternative way to download a file from an overloaded server. I’ve been an advocate of implementing BitTorrent in browsers in particular Firefox since at least 2004 according to this blog, and like the idea in principal but don’t like the implementation proposed.

The way the proposal would work is a server would send the X-Torrent HTTP header and if the browser chose to use the BitTorrent it would do that rather than leach the servers bandwidth. This however fails if the server is already overloaded.

Unnecessary Header

This is also a little unnecessary since browsers automatically send an Accept-Encoding Requests header which could contain support for torrents removing the need for a server to send this by default. Regardless the system still fails if the server is overloaded.

Doesn’t Failover

A nicer way would be to also utilize DNS which is surprisingly good at scaling for these types of tasks. It’s already used for similar things like DNSBL and SPF.

Example

Assume my browser supports the BitTorrent protocol and I visit the following url for a download:

http://dl.robert.accettura.com/pub/myfile.tar.gz

My request would look something like this:

Get: /pub/myfile.tar.gz
Host: dl.robert.accettura.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5
Accept: */*
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate,torrent
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://robert.accettura.com/download

The servers response would look something like this:

Date: Sun, 18 Jan 2009 00:25:54 GMT
Server: Apache
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/x-bittorrent

The content would be the actual torrent. The browser would handle as appropriate by opening a helper application or handling it internally. If I didn’t have torrent in my Accept-Encoding header, I would have been served via HTTP like we are all accustomed.

Now what happens if the server is not responding? A fallback to the DNS level could be done.

First take the GET and generate a SHA1 checksum for the GET, in my example that would be:

438296e855494825557824b691a09d06a86a21f1

Now to generate a DNS Query in the format [hash]._torrent.[server]:

438296e855494825557824b691a09d06a86a21f1._torrent.dl.robert.accettura.com

The response would look something like a Base64 encoded .torrent file broken up and served as TOR or TXT records. Should the string not fit in one record (I think the limit is 512 bytes) the response could be broken up into multiple records and concatenated by the client to reassemble.

Odds of a collision with existing DNS space is limited due to the use of a SHA1 hash and the _torrent subdomain. It coexists peacefully.

Downside

The downside here is that if your server fails your DNS is going to take an extra query from any client capable of doing this. There is slight latency in this process.

Upside/Conclusion

The upside is that DNS scaling has come a long way and is rarely an issue for popular sites and web hosts. DNS can (and often is) cached by ISP’s resulting in an automatic edge CDN thanks to ISP’s. ISP’s can also mitigate traffic on their networks by caching on their side (something I also suggested in 2004).

BitTorrent may be used for illegal content, but so is HTTP. I think costs for ISP’s and websites could be significantly cut by making BitTorrent more transparent as a data transfer protocol.

April Fools 2008

As usual, my list of April Fools that I saw today:

Open Source And Recessions

There’s an interesting blog post on Open Source and recessions worth reading. Essentially the question is this: Does a recession have a negative impact on open source?

I’d say the answer is somewhat more complex than a simple yes/no. There are many different types of projects out there with entirely different circumstances. However I suspect a projects impact could be gaged on a few key aspects of it’s operation:

Purpose – The purpose of the project is likely the most critical aspect. For example, I don’t think there would be any significant impact on projects like the Linux kernel which is essential to many products out there including server infrastructure that powers much of the web and many companies computer systems. Then you have consumer products like TiVo, Google Android etc. Because it’s purpose is so broad there are enough people with a financial interest in seeing development continue. WebKit, Mozilla, Apache, are good examples of this. They have broad usage by many. Something specific to a more obscure task would have more trouble due to it’s more limited market.

Development Team – Of course for a project to succeed it needs one or more developers. During a recession one could theorize that many would be less inclined to participate. This may not necessarily so. First of all, quite a bit of open source development is loosely sponsored. Several projects have actual staff, paid employees who write open source code. For example Apple employees people to work on WebKit. Mozilla has staff working on Firefox. There are people paid to work on Linux (Red Hat, IBM, Novell, etc.) and many other open source projects. There are also companies who contribute some code that would be of strategic value to them. There’s also those who are simply willing to sponsor some work they want to see happen. All of which fund developers of larger open source projects. But would developers who aren’t sponsored or employed to code still participate? I theorize most still would as they don’t depend on it for income during good times, presumably a job during a recession wouldn’t generally prohibit participation and more than a job during years of economic growth. There’s also the impact of college students who participate partially for the educational aspect. The early 2000’s was a recession and still showed a fair amount of growth of open source. In fact many of todays stars really started to take shape during that period. For example:

Funding – Somewhat obvious: Funding is key. Who pays the developers (partially the last aspect I discussed)? Who pays for the projects needs (servers, etc.)? Many of the more popular projects (almost all of the above) have either an organization of for-profit company around built around it. That company often sponsors the needs of the project. Unless the needs of that companies product/service is no longer needed during the recession, funding likely remains. That’s partially the first aspect I discussed.

It’s my belief the larger and more popular open source projects would feel a minimal impact during a recession. I think history has shown this, and common sense agrees. They are mostly low development cost, adequately funded (often from diverse sources), stable, and have a broad team of developers. The projects that are in trouble are the ones who have very few or only 1 developer, even worse if they share the same sponsor, even worse if there is little community around the project. Most projects would generally experience a slight slowdown in development the degree would depend on the above. A few may go dormant for a period of time. Thanks to things like GPL licensing, another developer can pick up should there be a market in the open source ecosystem.

Overall I don’t think open source would be nearly as impacted as most businesses during a recession. The model is very different. Open source when successful has a community and many different sponsors. The diversity allows the project to survive even when recession causes some sponsors to need to reduce or eliminate involvement. Open Source also by definition is used to this type of environment. It’s used to developing on a budget, soliciting sponsors to help cover costs, etc.

The interesting thing about recession is that it impacts everyone, but the degree to which someone is impacted varies. For example construction and housing are generally harder hit than other industries. People tend to cut back on new home purchases before they cut back on other things. Each of those industries has computing needs, sometimes met by open source. This all feeds into the open source ecosystem.

I’d suggest that all of the projects I have mentioned here will do ok during a recession. Many with a slowdown, but all will still continue as long as they provide value. A notable situation is Mozilla’s income comes largely from Google which is based on ad revenue. During a recession and bubble bursting this would likely dramatically reduce the revenue brought in. This isn’t being ignored. As the 2006 Financial FAQ states:

First, the cash reserve is of course a form of insurance against the loss of income. We will continue to maintain enough of a reserve to allow us flexibility in making product decisions….

It seems that an open source project with a diverse stream of funding from individuals and companies of various industries, as well as developers in different situations is in the best position to survive.

It’s an interesting topic.

FoxTorrent

I said back in 2004 that Firefox needs built in support for BitTorrent. My idea was it would be integrated into the download manager so that it was “just another protocol” and would be transparent to a typical user. I still stand by that.

Fast forward to 2007: FoxTorrent is by RedSwoosh (now owned by Akamai).

I’d personally love to see something like this ship built in. It’s a great feature. BitTorrent is a great protocol for distributing large downloads without having to buy expensive infrastructure. Akamai’s interest is proof of that.

FoxTorrent has a blog if you want to keep an eye on it. FoxTorrent is MIT licensed as well. It seems like a very interesting product. I’ll have to dig into this and look at it a bit closer.

[Hat tip: TechCrunch]

BitTorrent

I said over a year ago Firefox should support BitTorrent. Opera now has support. No it wouldn’t support piracy, or any of the other silly comments people have given over the past year about why it shouldn’t be supported. Launching yet another app to download a file is silly. If Firefox dropped FTP support (because it too is just another protocol, and doesn’t relate to web browsing). There would be a boat load of complaints. It’s not a kitchen sink. It’s a usable protocol. What people do with it is up to them. There are many legitimate BitTorrent uses. Linux distro’s use it for example. So do some gaming sites (those giant demo’s). The protocol isn’t illegal by any means. Just some individuals misuse of the protocol. The web is used for illegal purposes all the time, it doesn’t mean Firefox encourages it.

I’m a fan of the protocol. The problem is that it’s still to complex for average joe. Current clients are awkward and slow (while better than previous generations). Why do I need another program to download a file? And for those currently using BitTorrent, imagine the benefits of 60 Million users with BitTorrent capable browsers who can easily participate in torrents.

I still believe it’s a good idea. Congrats to Opera for realizing the benefit of the protocol and realizing the end user can benefit from it. That’s what software is about: benefiting the end user.

More on Firefox 0.9

I’ve been on the trunk for a while. A few days ago I went back to 0.8, just to see how life has improved since.

All I can say is get downloading if you haven’t already! Yes, it’s that good of a release.

The new download is only 4.7 MB in size. Thanks to 7-Zip. That’s welcome to all who hate large downloads. Oh, and BitTorrent is also available now.

Speed, extension/theme manager, as well as a profile migration tool make it the prime time to start a ‘switch’ campaign. As I said before. Lets make a goal of getting 1 person to switch. If everyone would do it, Mozilla would double it’s marketshare.

There’s also a new theme, as mentioned earlier, and has made shockwaves around the earth several times.

So come on people, get downloading. Spread the word. Get someone to switch. If you do, be sure to make a comment here. Lets see more people discover the internet doesn’t have to suck.

ISP’s should run BitTorrent Cache’s

I’ve went on a bit about BitTorrent before. And in part is has happened (regarding Mozilla). We at least have torrents on the homepage!

Now to send a little messages to ISP’s:

BitTorrent could be an ISP’s best friend. Think networking basics for a minute: Staying within the network is faster, and more reliable. If a user subscribes to Comcast, their connection to Comcast’s network is optimal. Theoretically faster than anything else the can access. Also, Comcast doesn’t need outbound bandwidth by peering, or purchasing bandwidth when a user is using internal content (savings).

If an ISP were to embrace something like BitTorrent, it would really be an advantage to ISP’s. When something new is released, such as a Game, Linux Distro, or other large file, people go and download it all at once. To accommodate that takes some bandwidth. There’s no good reason why an ISP can’t handle the bulk of that internally, and provide faster downloads to their users (great marketing), and lower operational costs.

If an ISP were to setup perhaps a cache, simply to provide fast internal downloading through a method like BitTorrent there would be significant benefit to all parties. File hosts save bandwidth, consumers get files quicker, and ISP’s relieve uplink bandwidth, as well as get something new to market.

Even if the cache only mirrored the very popular things, perhaps took the top 10 of the past 24hrs. That would make a significant difference.

Mozilla Needs BitTorrent

Background/What is BitTorrent

BitTorrent is a protocol rapidly gaining popularity on the Internet for Peer 2 Peer (P2P) downloading large files. For information on how BitTorrent works, visit here.

Why BitTorrent?

BitTorrent feeds are used on quite a few of the big downloads that people encounter (see below). BitTorrent is also designed to prevent abusers. If your not uploading, your not downloading. So it’s less prone to leaching, which would turn a useful protocol into a waste. It’s also open source (MIT License).

How do we know it’s not a fad?

It already has a significant penetration in the large download segment. Here are a few uses of BitTorrent I ran across this week browsing the web:

Also noteworthy is Download.com alone has over 524,089 downloads to date.

Advantages for Mozilla?

There are several distinct advantages for Mozilla supporting BitTorrent “out of the box”. The first, and most prominent is a good feature to offer end users. While all Windows users will need a download to be able to use a BitTorrent, Mozilla users will simply be able to download like any other protocol. BitTorrent’s major shortfall is an ugly UI. Mozilla Firefox has a new download manager thanks to Ben Goodger’s recent overhaul. BitTorrent could use the same UI, and fit right in. Mozilla can make this protocol friendly, and attractive. Mozilla could essentially become the “required tool” to use BitTorrent recommended by websites with Torrents on them.

The second advantage is that it could be used a distribution system by Mozilla.org. During the Firefox release, servers were slow at best. We could alleviate some of the load by supporting BitTorrent. Just like you can download Mozilla with Mozilla, we can continue this by supporting it internally.

What about bloat?

Mozilla needs to put things together, more than start a new project. A good implementation would utilize the Download Manager’s UI, a clean, simple way of tracking all downloads. Perhaps just an icon to denote it’s a BitTorrent download. BitTorrent also uses HTTP.

The goal is to be slim and good, not bloated. Even the official BitTorrent client is slim. No UI options. Just where to save the file. Extreme simplicity.

Why not an extension?

Because it’s another step 😉

Users don’t want to do more than they need to. It could be included and nicely integrated so that it isn’t bloat (there’s no new UI necessary). And it could then be an effective distribution method for Mozilla updates in the future. Mozilla itself can be updated via a BitTorrent feed on the Mozilla.org website.

Conclusion

Supporting BitTorrent would prove to be an advantage to Mozilla. Not only would it be a great feature for end users, but it would prove useful in Mozilla’s attempt to provide fast download options for its end users.