On H.264 Revisited

Once again the debate over H.264 has come up in the Mozilla community. I’ve been a strong advocate of the WebM/VP8 codec given its liberal license and abilities and still am, but agree H.264 needs to be supported. It’s a requirement for mobile (B2G), and becoming necessary on the desktop.

A little over a year ago Chrome talked about dropping support for H.264. To date they have not done so, or given any indication that is even still in the plans as far as I know. In 2010 Adobe said they would be supporting WebM (link in that same blog post). They too have failed to live up to their promises. In either case I’ve found no indication on the internet they ever plan to go forward with those plans.

I suspect in Google’s case they were pressured by various providers and mobile partners who don’t want to encode or support another encoding. Google’s been trying to woo anyone/everyone for the purposes of Google TV and presumably YouTube. It’s likely just not worth it for them to push. There are various theories floating around about Adobe including a lack of clear Flash strategy in an HTML5 world. Adobe does however have a “tools” strategy. Perhaps time will tell.

Furthermore Apple and Microsoft are fundamentally opposed to WebM as they are both licensors for H.264. The odds of them supporting something that hurts their bottom line unless the rest of the web is threatening to leave them behind is nearly 0.

I question however if it should be bundled vs. using system codecs. Windows XP aside, system codecs mean that Microsoft and Apple are essentially responsible for making it work as well as the expense. Plugins could be used for OS’s that don’t ship with the appropriate codecs.

It’s time to put some effort into a JavaScript player for WebM and make that liberally licensed. Browsers still aren’t quite there, but eventually the day will come when that’s workable. The web will then gain the ability to have video play on any (modern) device. Just not natively. That is the backdoor for an open codec.

The real issue is larger than the <video/> element. It’s software patents and their ability to undermine innovation and progress. It’s important to keep this in mind. Just look at mobile. It’s completely possible that the entire mobile industry could come to a halt over patent lawsuits and fear of lawsuits. All it takes is a company willing to press the button. Google spent $12.5 billion in what is essentially the patent equivalent of nuclear proliferation. That’s how real the threat is perceived. H.264 is arguably a fart in a hurricane.

90% Of Web Traffic Is Video

There’s a lot of talk about the prediction from YouTube’s Robert Kyncl that video will be 90% of internet traffic. This sounds shocking until you realize it’s not, and doesn’t really mean much in context of the internet.

This number presumably is at least partially based on a prediction by Cisco who makes routers that carry that internet traffic. They even bought Flip (an unfortunate tale) simply to help drive video on the internet and presumably drive up the need for routers, their main business. They might have a bias.

The reality is video already is, and will continue to be the vast majority of Internet traffic. However that doesn’t mean people are spending as much time on it as the numbers will make you believe. Video per minute of online activity is just insanely high bandwidth and expensive. The percentage of bandwidth is meaningless, what matters is where people spend their time.

Video compresses very poorly in comparison to text and audio (interestingly, speech compresses fairly well these days). Video these days either streaming or progressive download is still a very bandwidth intensive activity.

This webpage is likely under 160 KB for you and loads in about 300 ms. You’ll spend statistically a few minutes reading this, perhaps slightly less if you just skim for the numbers. The takeaway is experiencing: this blog post is about 160 KB over a few minutes assuming you visit the site, and don’t use an RSS reader (go ahead RSS users, try it).

Now lets figure out what video is typically. Most web video these days is still H.264, and generally 640×480 (480p) or 1280×720 (720p). Most laptops today are pretty cheap and just can’t handle 1080p, certainly their displays aren’t good enough to show a measurable difference, not to mention the buffering time annoys people, so we’ll ignore it’s existence.

I want to make this a little realistic, so to figure out a bitrate, we’ll use the standard formula for what Adobe calls ‘ideal’ in the H.264 Primer, and what most seem to use these days:

[image width] x [image height] x [framerate] x [motion rank] x 0.07 = [desired bitrate]

For 1280×720 that means:

1280 x 720 x 24 x 2 x 0.07 = 3,096,576 bps = ~3000 kbps

And for 640×480:

640 x 480 x 24 x 2 x 0.07 = 1,032,192 bps = ~1000 kbps

This sounds about right. As of today, Wikipedia lists the bandwidth range for 720p at 2–2.9 Mbit/sec and for 480p at 0.8–1 Mbit/sec. So we know we’re on track.

That’s all the math we really need to do. Google can do the conversions for us:

For 720p:

3000 kbps = 1,318.35938 Megabytes / hour = 22 MB / min

For 480p:

1000 kbps = 439.453125 Megabytes / hour  = 7.3 MB / min

Going back to the example of this blog permalink, at 160 KB for a few minutes of reading vs. 7,475 KB for one minute of 480p video. That is why video is 90% of traffic.

Video growth is huge, but it’s not nearly as one sided as the stats will lead you to believe.

On Chrome Dropping H.264

The Chrome team announced they are dropping support for H.264.

WebM Support

WebM support will be growing quickly as Firefox 4 rolls out (Firefox upgrade adoption is legendary). Chrome commands sizable market share and is pushing the Chrome OS platform. Opera is also supporting WebM.

Apple and Microsoft could join the party and bundle WebM support along with the other codecs they support at any time, though they are licensors for H.264 and wouldn’t benefit from WebM market penetration. Microsoft’s implementation does allow for VP8 support if a codec is installed. I’m not aware of anything for Safari and am rather certain nothing can be done for the iPhone without Apple intervening.

On the hardware side AMD, ARM, Nvidia are backing WebM. Broadcom announced support, as did Qualicomm and TI. These are major vendors for mobile chips. Intel is working on stuff too.

H.264 Trouble

H.264 is problematic and bad for the web for many reasons I’ve mentioned here before as well as great posts by roc and shaver. I’ll leave it at that rather than rehash.

There was buzz a while back about H.264 being “free” (quotes intentional), but it’s not really “free” if you read the fine print. As Peter Csathy of Sorenson Media notes:

But, you say, MPEG LA recently announced that it will no longer charge royalties for the use of H.264. Yes, it’s true – MPEG LA recently bowed to mounting pressure from, and press surrounding, WebM and announced something that kind of sounds that way. But, I caution you to read the not-too-fine print. H.264 is royalty-free only in one limited case – for Internet video that is delivered free to end users. Read again: for (1) Internet delivery that is (2) delivered free to end users. In the words of MPEG LA’s own press release, “Products and services other than [those] continue to be royalty-bearing.”

That’s hardly “free”. That’s just one potential use case that’s now royalty exempt. The reason they are doing that is presumably if they can get H.264 adoption high enough, all the other cases will be paying and therefore subsidizing this one case.

WebM is licensed a little different: Patent wise, it’s irrevocably royalty free. License is about as liberal as you can get.

There’s no proprietary html, css, or images (GIF was, now it’s dead) used across the web. Why should video be any different? The key to success and growth has always been an open platform that’s low cost and encourages innovation.

Implementing Today

For anyone who suggests that this further fragments the market, that’s not really true. Adobe Flash actually creates an excellent shim to help migrate away from Flash to <video/>. Allow me to explain:

Adobe will soon be supporting WebM through Flash. Adobe already support H.264 in Flash. For legacy browsers and those who won’t support WebM, you have the option of delivering a Flash experience just like most websites do today. There are websites doing this today via Flash and H.264. For modern browsers you can just use <video/>. Once your non-WebM market share drops low enough, you can get rid of the Flash experience. Soon enough you’ll be able to push WebM to your Flash users. The benefit of switching your Flash experience to WebM as a middle step would be one encoding for both delivery mechanisms vs. using H.264 and WebM in parallel. Of course if you’re supporting mobile you likely need H.264 for a bit longer but likely use a smaller resolution and different profile for mobile consumption.

No matter what there will be two delivery mechanisms for those looking to push video using HTML5 to users today. The only thing that changes is the lean towards standardizing on the actively developed WebM codec vs. H.264.

All new technology has speed bumps, that’s the cost of being on the bleeding edge. However this is a positive turn as things are now starting to line up. The most awesome thing is that the codec, HTML5 specs, and some of the most popular browsers in the world are open and inviting feedback and contributions to improve things.

Mac Finally Gets H.264 Decoding In Flash

Adobe today pushed an update that enabled H.264 hardware decoding in Flash 10.1. It only works on certain newer Mac’s and there are an assortment of caveats in which Flash will revert to software decoding according to a Flash Engineer.

I’ve only played with it for a few minutes on my Core i7 MacBook Pro, and things seem very speedy and my CPU didn’t see much of a spike. Hopefully enough videos will take advantage of hardware decoding that this will be a nice improvement.

I still believe WebM is the better future, but H.264 hardware decoding does make Flash less painful for the moment.

WebM

In August 2009 after the On2 announcement, I suggested that Google might open source a codec in hopes of derailing OGG which it feels is inferior as well as h.264 which is patent-encumbered. Google took VP8, the successor to the popular VP7 codec and started The WebM Project. To quote the project page:

WebM is an open, royalty-free, media file format designed for the web.

WebM defines the file container structure, video and audio formats. WebM files consist of video streams compressed with the VP8 video codec and audio streams compressed with the Vorbis audio codec. The WebM file structure is based on the Matroska container.

Google describes the license as “BSD-style”. A very good move since it’s liberal enough to encourage widespread open and proprietary inclusion. GPL is to viral for some potential adopters.

Software Support

For the browser side, Chromium and Firefox Nightly builds support WebM starting today. Opera and Google Chrome to come shortly.

Google also created patches against FFmpeg for encode as well as decode and created DirectShow filters which are available for download. I suspect by way of libavcodec we’ll see support in lots of other products in the near future.

Microsoft will support VP8 in Internet Explorer 9 if you have the VP8 codec installed. Not quite “support”, but better than nothing.

Adobe is also supporting VP8 in Flash, which means content producers can eventually kill VP7 and VP6 encoding and use VP8 to reach most of their audience. This is very important as encoding videos into several formats is costly and time consuming (I know this very well).

Hardware Support

Google has already said they are working with video and silicon vendors to add VP8 hardware acceleration to their chipsets. I suspect newer phones in the near future will be supporting it. Especially if they run Android.

Content

Google is supporting WebM in the HTML5 test for YouTube which I mentioned a few months ago. I suspect we’ll see lots more support in the very near future.

Supporters

Even more telling of the potential than the above is the list of supporters which contains some big names who can put a lot of weight behind hardware/software/content support. AMD (who owns ATI), NVIDIA, Marvell (lots of mobile chipsets), Qualcomm (think mobile chipsets), TI, Broadcom, ARM on the hardware side alone is impressive. If the majority of them add hardware support to their upcoming offerings, that will be game changing. On the software side leaves 1.5 holdouts in the web video world: Apple (1) and Microsoft (0.5).

This is a game changer.

YouTube HTML5 + Firefox

Google has been a long time supporter of HTML5. They recently launched a HTML5 beta of YouTube however it will only work in Safari and Chrome. The reason for this is not due to the actual markup but the video codec chosen. YouTube is using h.264, the same codec used for YouTube HD via Flash. This works in Safari and Chrome because Safari uses QuickTime to render <video/> and Google licensed h.264 for Chrome. Firefox however doesn’t include the proprietary codec for licensing reasons. It’s not a matter of cost but principle.

IE is supported through “Chrome Frame” which is essentially the Chrome browser in IE’s chrome. Your really just browsing the YouTube site with Chrome. Google could use this as a way to get people away from Flash and IE and onto Chrome one way or another.

I discussed the h.264 debate in more depth a few months ago.

You have to wonder why we don’t want anything proprietary slipping into HTML5, or want proprietary image formats (GIF turned us off to that) but exceptions are made for video.

Edit 1/23/2010: More on the topic:

Edit 5/21/2010: Thoughts on WebM.

Google Buys On2

Google today announced they are buying On2 Technologies. This is one of their more significant purchases despite the relatively low price tag of $106.5 million since it’s video technology and Google is the largest video source on the web right now.

On2 is really an unknown to most people but their product has an amazing reach thanks to Adobe Flash. VP6 notably was included in Flash 8 and really brought about the age of Flash video (think YouTube). On2 also has VP7 which is considered a H.264 competitor. VP3 was released as open source and lives on as OGG Theora.

Of course by buying On2 Google will not need to pay any licensing for it’s VP7 technology, they can then bundle it into Chrome, Android and Google Chrome OS (finally giving Linux decent web video support). They could also open source it similar to these platforms in hopes that it will gain ubiquity.

This does however leave me wondering if this pending On2 deal had any bearing on the decision to leave HTML 5 <video/> codec ambiguous. It’s noteworthy since Google is very involved in the HTML 5 efforts. As I mentioned last month licensing is really key. If VP7 were open sourced and it’s licensing were compatible to meet Apple and Mozilla’s needs (which could lead to inclusion in Safari and Firefox respectively), OGG Theora is potentially dead overnight. Given Google’s strategy so far of making technology open source in efforts to encourage adoption, I wouldn’t rule this out, though it would likely take a while to evaluate everything and make sure they legally have that option. Timeline could also come into play here. The web isn’t necessarily going to wait for Google. These reviews can potentially take a long time. No guarantee others will incorporate it either, though it’s a pretty good deal should licensing work.

It’s also interesting that now Microsoft has Windows Media Player, Apple has QuickTime, and Google has On2′s codec bundle. It’s not exactly a “player”, but considering it’s usage it’s just as important.

It’s going to be very interesting to see how this plays out. One thing that seems relatively certain is that Google just made web video more interesting.

Debating Ogg Theora and H.264

Since the big HTML 5 news that there will be no defined codec for <audio/> or <video/> there has been a lot of discussion about the merits of such a decision, and what led to it. To quote Ian Hickson’s email:

Apple refuses to implement Ogg Theora in Quicktime by default (as used by Safari), citing lack of hardware support and an uncertain patent landscape.

Google has implemented H.264 and Ogg Theora in Chrome, but cannot provide the H.264 codec license to third-party distributors of Chromium, and have indicated a belief that Ogg Theora’s quality-per-bit is not yet suitable for the volume handled by YouTube.

Opera refuses to implement H.264, citing the obscene cost of the relevant patent licenses.

Mozilla refuses to implement H.264, as they would not be able to obtain a license that covers their downstream distributors.

Microsoft has not commented on their intent to support

I think everyone agrees this is going nowhere and isn’t likely to change in the near future. For the sake of moving HTML5 forward, this is likely the best decision.

Here’s how I interpret everyone’s position:

Apple’s Argument

One of the undeniable perks behind H.264 right now is that there is hardware decoding available and used on on certain devices. One of the most notable is the iPhone. Using hardware decoding means your not using the CPU which results in better performance, and most importantly better battery life.

Thus far there’s no hardware Theora decoder on the market (if you know of any let me know, my research says none), which I suspect is why Apple is hesitant to jump on board. Until there’s hardware that’s proven to perform well, be cost-effective in the quantities Apple needs, and not be bombarded with patent infringement claims, I suspect they’d rather settle with H.264. The patent part is critical. Apple can update software to comply with patent wars pretty quickly, as many other companies have done with software in the past. Hardware is not so easy. Last minute hardware changes are harder to deal with than software because of the many things it impacts, and the inability to update at a later date.

I’m almost positive the lack of hardware support is the exact same reason Apple has been so against Flash support. Remember the YouTube application isn’t using VP6 like regular flash, it’s using H.264 (that’s why it took so long for all of YouTube to be available on the iPhone).

If there’s enough Theora content out there, there will likely be Theora decoder hardware made to meet market demand. To get to this point will be difficult with the amount of VP6 (Flash) and H.264 content already on the web. H.264 alone has a major head start in applications. VP6 has several years of video on the web now (and I still don’t think it has a hardware decoder on the market though that might be due to licensing again).

In the long run, I think mobile technology will improve enough to make this a somewhat unnecessary constraint. Mobile CPU’s and GPU’s are just starting to get to the caliber needed for video. Performance per watt should improve. Battery technology is just starting to get pushed to the limits. This is a good thing for Theora in the long run, but the question is how long?

Until it can be played with minimal impact on battery life, I don’t think any company who has a heavy investment in mobile will want to jump on board.

Google’s Argument

Google has money and can license H.264. Shocker. Google however has trouble when it comes to Chromium. I suspect Google doesn’t care too much about which way this goes since what they support in Chrome doesn’t mandate that YouTube support it. However if the encoding quality for a given bitrate is good enough, it becomes a viable option.

Regarding the quality argument, I’ll simply point to this comparison. I the quality today is comparable already, and likely to get better as the encoders improve. I’ll leave this discussion here.

Opera’s Argument

Opera says H.264 is to expensive to license. I don’t know what the costs are, and what they would be for Opera, but I’ll take their word on it. After all, the do have a product available for free download. While commercial and closed source, they don’t have Google’s revenue stream and I respect that.

Mozilla’s Argument

Mozilla can’t license for downstream Gecko use etc. I’m sure a good part of the argument is also that requiring licensing fees to use <video/> is bad for the web and open source. I agree.

Microsoft’s Argument

No comment. Historically they implemented <marquee/> but not the <blink/>. Make of that what you will.

<video/> could be supported by plugin if needed. I recall Adobe supporting SVG by plugin a few years ago.

Where to go from here?

I think there are a few possible outcomes. As for what I think are the most likely:

  1. There’s a push for hardware decoding that makes Theora on mobile technically possible and working well. If Apple legally is satisfied and jumps on board that changes the game. As I stated earlier I think Google is mostly ambivalent since they support both right now. Opera doesn’t want H.264 anyway, so they are cool. IE 8 can likely be handled by a plugin. Apple really is the deciding factor. Theora is the future.
  2. See what the web does. I suspect at least for a long while the web will just stick with Flash since it works on almost all desktops. For mobile the iPhone and Android make up pretty much the bulk of the mobile video market and that doesn’t look like it’s changing so fast. Content providers that want mobile will encode for mobile. That means 3 target platforms, not ideal but reasonable. H.264 and whatever Adobe adopts is the future.

I know how the media is interpreting all of this. How do other developers, and open source folks see it?

Big Buck Bunny

Big Buck Bunny

Big Buck Bunny, the new open movie made using Blender is out. It’s rather good, and impressive when you realize it’s made with open source products, meaning the only barrier to making one yourself (assuming you’ve got a rendering farm, or the patience to let your workstation churn out the pixels) is your skills. You can download it from the website (h.264 available) or watch on YouTube. I’d recommend the download so you can appreciate the HD quality. Some more screenshots can be found on Wikimedia Commons.

The first open movie was Elephants Dream back in 2007. Elephants dream used proprietary audio software. As far as I can tell, Big Buck Bunny didn’t.

Between the two I think I like Elephants Dream more. It was a little darker, but struck me as a little more entertaining. That’s my personal opinion though. It will be interesting to see what the next one is.

iPhone 2nd Generation

So more is coming out about the next iPhone, which we all know is going to be 3G. Someone found evidence of it in the recent update to the iPhone SDK. The SGOLD2 chipset will be replaced with the SGOLD3, which supports 3G networks (as we all expected). Looking at the specs some interesting things come out:

  • ARM 926 CPU capable of running up to 312 MHz – This isn’t that much more than the existing iPhone which is said to be underclocked to preserve battery life. Don’t expect much change here.
  • 5 Megapixel camera support – Capable, but don’t expect to see 5MP. I suspect it will be upgraded to 3MP and no higher to conserve costs.
  • MPEG4 / H.263 hardware accelerator – Sigh, no H.264 support. That’s a bummer. Apple could still use hardware support via another chip.
  • Support for video telephony, streaming, recording and playback – I wonder if Apple plans to utilize this. Video telephony could work over 3G networks (AT&T already did it with an LG CU500v). But it would require a potentially reworking the location of the camera on the phone.
  • 3G upgradeable with WCDMA coprocessor – Very interesting since this could allow Apple to offer the iPhone on CDMA networks, though the largest (Verizon) is going to become LTE a varient of GSM. That’s the largest CDMA network in the US. Still CDMA support will be needed for Japan.

Walt Mossberg initially said this is going down in 60 days, but now he’s retracted that statement. I still think a June timeline sounds about right. WWDC 2008 is June 9-13, 2008. Sounds about right.