Protecting Photo Privacy Via Browsers

Browsers can do more to protect users from inadvertently violating their own privacy. The NY Times today had an article about a topic that has been discussed in various circles several times now. The existence of geotagging data in photos. Many cameras, in particular smart phones like the iPhone can tag photos with GPS data. This is pretty handy for various purposes including organizing photos at a later date, iPhoto for example does a pretty nice job of it. Most photo applications however don’t make this information very visible, as a result many users don’t even know it exists, others simply forget.

What the problem looks like

The data, embedded in a photo looks something like this:

GPSLatitude                    : 57.64911
GPSLongitude                   : 10.40744
GPSPosition                    : 57.64911 10.40744

Which I could map.

Proposal

I propose that browsers need to have a content policy for when users upload images that can better protect them from uploading information they may not even realize. Here’s what I’m imagining:

The first time a user attempts to upload a photo that has EXIF or XMP data containing location they are prompted if they want it stripped from the image they are uploading. The original file remains unharmed, just the uploaded version won’t have the data. They can also choose to have the browser remember their preference to prevent being prompted in the future. They can revise their choice in the preferences window later if they want. This isn’t to different from how popups are handled. I thnk that per-site policy might be too confusing and not warranted, but perhaps I’m wrong.

Warning users about hidden information they may be revealing is a worthwhile effort. It’s only a matter of time before someone uses a “contest” or some other form of social engineering to solicit pictures that may reveal location data for users. Evildoers always find creative ways to exploit people.

Caveat

There are a notable caveat to this approach. The most notable is that flash uploaders would bypass this security measure though individual uploaders could do it themselves, or Adobe could do it, but I don’t think that’s enough of a turnoff to this approach. The same caveat applied to “private browsing” in browsers.

Prior Work

As far as I know no browser actually implements a security feature like this yet. There are a few Firefox Add-ons like Exif Viewer and FxIF (both written in pure JavaScript) that look at EXIF data but nothing that intercepts uploads.

Who Can Do It First?

I’m curious who can do it first. By add-on (seems like it should be possible at least in Firefox), and dare I say include in a browser itself? If this were earlier in the year I would have added this to the Summer of Code ideas list. Instead I’m just throwing it into the wind until 2011 rolls around.

Debating Ogg Theora and H.264

Since the big HTML 5 news that there will be no defined codec for <audio/> or <video/> there has been a lot of discussion about the merits of such a decision, and what led to it. To quote Ian Hickson’s email:

Apple refuses to implement Ogg Theora in Quicktime by default (as used by Safari), citing lack of hardware support and an uncertain patent landscape.

Google has implemented H.264 and Ogg Theora in Chrome, but cannot provide the H.264 codec license to third-party distributors of Chromium, and have indicated a belief that Ogg Theora’s quality-per-bit is not yet suitable for the volume handled by YouTube.

Opera refuses to implement H.264, citing the obscene cost of the relevant patent licenses.

Mozilla refuses to implement H.264, as they would not be able to obtain a license that covers their downstream distributors.

Microsoft has not commented on their intent to support

I think everyone agrees this is going nowhere and isn’t likely to change in the near future. For the sake of moving HTML5 forward, this is likely the best decision.

Here’s how I interpret everyone’s position:

Apple’s Argument

One of the undeniable perks behind H.264 right now is that there is hardware decoding available and used on on certain devices. One of the most notable is the iPhone. Using hardware decoding means your not using the CPU which results in better performance, and most importantly better battery life.

Thus far there’s no hardware Theora decoder on the market (if you know of any let me know, my research says none), which I suspect is why Apple is hesitant to jump on board. Until there’s hardware that’s proven to perform well, be cost-effective in the quantities Apple needs, and not be bombarded with patent infringement claims, I suspect they’d rather settle with H.264. The patent part is critical. Apple can update software to comply with patent wars pretty quickly, as many other companies have done with software in the past. Hardware is not so easy. Last minute hardware changes are harder to deal with than software because of the many things it impacts, and the inability to update at a later date.

I’m almost positive the lack of hardware support is the exact same reason Apple has been so against Flash support. Remember the YouTube application isn’t using VP6 like regular flash, it’s using H.264 (that’s why it took so long for all of YouTube to be available on the iPhone).

If there’s enough Theora content out there, there will likely be Theora decoder hardware made to meet market demand. To get to this point will be difficult with the amount of VP6 (Flash) and H.264 content already on the web. H.264 alone has a major head start in applications. VP6 has several years of video on the web now (and I still don’t think it has a hardware decoder on the market though that might be due to licensing again).

In the long run, I think mobile technology will improve enough to make this a somewhat unnecessary constraint. Mobile CPU’s and GPU’s are just starting to get to the caliber needed for video. Performance per watt should improve. Battery technology is just starting to get pushed to the limits. This is a good thing for Theora in the long run, but the question is how long?

Until it can be played with minimal impact on battery life, I don’t think any company who has a heavy investment in mobile will want to jump on board.

Google’s Argument

Google has money and can license H.264. Shocker. Google however has trouble when it comes to Chromium. I suspect Google doesn’t care too much about which way this goes since what they support in Chrome doesn’t mandate that YouTube support it. However if the encoding quality for a given bitrate is good enough, it becomes a viable option.

Regarding the quality argument, I’ll simply point to this comparison. I the quality today is comparable already, and likely to get better as the encoders improve. I’ll leave this discussion here.

Opera’s Argument

Opera says H.264 is to expensive to license. I don’t know what the costs are, and what they would be for Opera, but I’ll take their word on it. After all, the do have a product available for free download. While commercial and closed source, they don’t have Google’s revenue stream and I respect that.

Mozilla’s Argument

Mozilla can’t license for downstream Gecko use etc. I’m sure a good part of the argument is also that requiring licensing fees to use <video/> is bad for the web and open source. I agree.

Microsoft’s Argument

No comment. Historically they implemented <marquee/> but not the <blink/>. Make of that what you will.

<video/> could be supported by plugin if needed. I recall Adobe supporting SVG by plugin a few years ago.

Where to go from here?

I think there are a few possible outcomes. As for what I think are the most likely:

  1. There’s a push for hardware decoding that makes Theora on mobile technically possible and working well. If Apple legally is satisfied and jumps on board that changes the game. As I stated earlier I think Google is mostly ambivalent since they support both right now. Opera doesn’t want H.264 anyway, so they are cool. IE 8 can likely be handled by a plugin. Apple really is the deciding factor. Theora is the future.
  2. See what the web does. I suspect at least for a long while the web will just stick with Flash since it works on almost all desktops. For mobile the iPhone and Android make up pretty much the bulk of the mobile video market and that doesn’t look like it’s changing so fast. Content providers that want mobile will encode for mobile. That means 3 target platforms, not ideal but reasonable. H.264 and whatever Adobe adopts is the future.

I know how the media is interpreting all of this. How do other developers, and open source folks see it?

3rd Party Web Browsers For iPhone

There’s some buzz around the web regarding 3rd party web browsers for the iPhone now appearing in the App Store. This really isn’t as good as it sounds. In fact, it’s misleading. From what I can tell they are all using WebKit (Safari) API’s. UIWebView to be specific. These are just applications that serve as an alternative UI for WebKit.

This isn’t even totally new as there are several apps that have done this in the past, the most popular being Twitterrific who ships a “mini-browser” for the purpose of viewing links in tweets without leaving the application. What’s new is that an application’s sole purpose can be a browser. Though there’s no official word on a policy that I know of.

That means no you will not see Java, you will not see Flash, you will not see Firefox. You may perhaps see some user experience improvements which of course are welcome, but not another “browser”.

Mobile Browsing UI

It’s interesting to watch mobile web browsing UI develop. This is really the first time since web browsers existed that they have received a large overhaul. Sure things like tabs are “major”, but when you really look at it, Safari, Chrome, IE, Firefox are all strikingly similar to the original Mosaic (this is 1.0 running on Windows XP):

NCSA Mosaic UI

I’m not sure who’s idea it was to put the title in the UI like that, especially in a time when displays were small. That was a gigantic waste of space. The address bar in this version is read only, you need to select open and enter your URL there. Other than that, it’s pretty much the same browser UI since 1993. That’s right, 15 years of really the same user interface. The window to the web has always looked that way. There’s now bookmarking, a fancier address bar, favicons, and a search box. Firefox goes nuts by letting users install add-ons. Overall: Not very different.

There’s a few reasons why it hasn’t changed too much. First of all, it’s a pretty good design. Minus some quirks which were worked out pretty fast, it’s effective. If it wasn’t the web wouldn’t have caught on. Secondly, people know how to use it already. Why make people re-learn?

The mobile space is different yet surprisingly the same. Like days of old there’s a need to conserve screen space. Unlike days of old there’s no reason to believe it will get bigger since small phones are always desirable. Until screens are foldable, the iPhone is about as big as you’ll see. Even when phones get thinner and lighter, the screen size won’t likely get any larger since it will be awkward to hold and put in your pocket.

With a touch screen you can only make items in the UI as small as a fingerprint. Any smaller and they are unusable to people. A stylus while clunkier and more awkward allows for a much more compact UI. This leaves very little space to get a lot accomplished. Too add to the complexity of the problem websites are designed for big displays meaning there’s a lot to cram into a small space.

Apple’s allegedly making a pretty interesting change to iPhone 2.2. Safari will break out the search box into a more desktop-like separate box. This results in a smaller address bar and the reload icon being moved inside the url bar. I think the reason for this is to better parody the desktop, and remind users they can search from the browser chrome.

iPhone 2.2 Safari With Search Box

To be perfectly honest, I’m not sure the address bar is even needed in a mobile browser on a touchscreen device. Unlike a desktop you can’t type directly into it because of the small size. Your essentially going to another UI to enter the text anyway. Why not just make it a button? You could argue you need the address bar as a way to know where you are. Of course you can likely merge it with the Title to accomplish that. All that’s needed in the main UI is the title and hostname. That can be all in the title of the window. I think I’d prefer a back button more than the address bar on a mobile device. Of course if I could tap or tilt the device to go backwards or forwards that would be cool too. One less thing for the UI.

The most similar to this is Fennec.

On a side note, thanks to Apple’s insane SDK licensing and app store policy it is unlikely to ever live on an iPhone. Maybe one day Apple will realize that just like 3rd party applications (something they were originally against), an even more open device would be even more enticing. But I digress.

Nintendo DSAnother concept I’d really love to see and experiment with is a dual screen format. Similar to that of the Nintendo DS. This would be perfect for a flip phone style smart phone. As phones can be made thinner folding them over is an option to keep the physical device small enough for portability but the display size can then be doubled. By the time the iPhone can be made half the thickness (remember the iPod G1 was much thicker than it is now) this is feasible.

There are several fun things about this design. First of all you essentially Optimus Maximus keyboard on your phone. Secondly you can now separate the content from the chrome in applications. Perfect for things like web browsers. This is also handy for watching movies as controls don’t overlay video but are still available. It also would be great for multi-tasking.

That’s where I predict things will ultimately go. We’re once again in the era of Bar form phones. Anyone remember the Nokia 1100/5110/3210/3310 fad a few years ago? Then flip phones came back in style. The flip phone style also has the advantage of protecting the internal display from scratches and involuntary button pressing.

It will be fun to see how the interface evolves. I’m relatively certain despite all the different UI prototypes surfacing right now regarding web browsers, as they mature they will adopt features from each other and become surprisingly similar to each other.

iPhone Safari image via Wired. Nintendo DS image via Wikipedia Commons]

Initial Thoughts On Google Chrome

I figured I’d blog my initial thoughts on Google chrome. Rather than a hard to read essay, I figured bullet points are easier to read/scan, so that’s what I’ll do.

Announcement

  • Another One Bites the Dust – The classic Queen song was played both before and after the presentation. I can’t help to think this was an intentional joke about a browser biting the dust.
  • From Scratch – How can a browser be both written from scratch, and based on the existing WebKit at the same time? Is this a bug? ;-)
  • Ben Goodger – Did a good job selling the UI.
  • BSD License – All of it is BSD licensed. Very interesting that it’s not pulling a Safari. Chromium Project, V8.
  • Incognito Mode – Ben/Brian discussed it without mentioning the word “Porn”, though you can somewhat tell that they were both dying to use the “P” word…. uhh for research… for a friend!
  • Static Content Performance – As shown in the demonstration: IE7: 220.64ms, Chrome: 77.28ms. Less is more.
  • Dynamic Content Performance – As shown in the demonstration: IE7: 5.8 RPH, Chrome: 569.3RPM. More is more.
  • Search Box Missing – I agree with merging the search box with the url bar. It makes sense and results in a cleaner UI.

My Testing

  • Build – This is build 0.2.149.27 (1583)
  • UseragentMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13. Noteworthy that they keep “Safari” in there. This looks like what I’d expect.
  • Bug Type – “reporter” feature has a “Bug type” called “browser crash… go boom”. I like jokes in software.
  • Tubes – The gears plugin in the chrome install has the file description “These are the Gears that power the tubes!”. Awesome Ted Stevens reference.
  • History View – I always wanted the history view in Firefox to be in a page rather than a sidebar. I understand the thought behind the implementation, but when you see how Chrome is laid out, it’s clearly superior.
  • No PPC Support – Because of how V8 works, I strongly suspect Google Chrome will never support Mac/PPC despite it still being supported by Apple until (likely) 10.6. In the press conference they said anyone can port V8 to another platform, if you have 3-4 months. Doubt they will use any other engine for that one platform. I’d say it’s toast.
  • Chrome API – The announcement mentioned plans for an extension API. Looking at the files it ships, it looks like almost all dll’s. Because of this I doubt we’ll see a method as simple as Firefox. Not sure how the interface is put together but it doesn’t look like anything as flexible as XUL, Boxely or XAML.
  • about: – So far I’ve found about:, about:memory, about:plugins.
  • Slickspeed Test – Ran a test and found Chrome did 115ms, 130ms, Safari 3.1 did: 28ms, 68ms, while Firefox 3 did 179ms, 294ms (prototype, jQuery). Interesting that Safari selectors are still faster.
  • Walt Mossberg – Everyone’s favorite tech writer Walt Mossberg wrote:

    Despite Google’s claims that Chrome is fast, it was notably slower in my tests at the common task of launching Web pages than either Firefox or Safari. However, it proved faster than the latest version of IE — also a beta version — called IE8.

  • Icon – Daniel Glazman identified the source of inspiration for the icon.
  • Phone Home – Matt Cutts has a blog post on when Chrome phones home
  • Omnibox – It’s ability to find new search engines is pretty neat… wonder how well that really works in day to day browsing though. Otherwise it’s essentially an Awesomebar to me.
  • Application Shortcut – The Prism-like functionality is just that. Essentially it just passes the url as a param into the Chrome exe via a shortcut. Looks like unless Gears is used you’ll use the Favicon which looks pretty bad.
  • Download Manager – The download manager is pretty cool, but if you download a number of files, I think this interface can get pretty cumbersome. Just like Thunderbird with a lot of attachments.
  • Font Rendering – It looks like they are using GDI Text rendering to avoid that blurry mess that Safari uses on Windows. I suspect Apple will do the same soon.

Overall it seems pretty smooth. From what I’ve seen, the process model does result in more memory in total than Firefox 3, since most tabs I open stay open for quite a while. It’s clearly still a little rough, but it’s not even out for 12hrs yet.

I await a Mac release. I just realized Pinkerton is working on Chromium as well, so I have a feeling the Mac release won’t suck but will be a real port that looks and feels like a Mac application should.

I don’t think it was mentioned in the press conference, but the Chromium blog is open.

Wired has a great article on inside the project. In addition to the names I mentioned yesterday, Bryan Ryner was also involved in at least the prototype.

I’ll make one prediction: The code most likely to find it’s way into other browsers is the GreenBorders stuff. It was originally for IE/Firefox, making it most suitable for possible adaptation to be included in other browsers. I’m not sure how much of it remains and how easy to adapt it would be though.

I’ll leave this “review” right here and unfinished since it’s still an ongoing project. Just wanted to share my initial thoughts. I’ll follow up at some point in the future when I feel it’s right to do so.

Any thoughts/additions? Feel free to leave a comment.

Edit [9/2/2008 @ 10:55 PM EST] – Added prediction about GreenBorders and link to Wired Article.

Opera’s Evangelism

Opera is said to be sending evangelism emails to websites that have compatibility problems with their browser. What’s interesting is that they are customizing the emails with actual fixes for the problems. This is pretty clever. In theory it will improve the problems regarding compatibility and make the web more standards compliant (which is where Opera excels).

One thing I do question is if webmasters will read it, at least where it matters. Most large companies have a contact form, or an email address, but it’s often forwarded to customer support, or sometimes just into a giant bin where a handful get processed. Will the information get to the people who need it? I suspect it will for small companies who read all the email they get from the web. For large companies, I doubt it, and that’s where I think it matters the most. The bigger sites that the majority of the web visit.

Regardless, it’s interesting to see, for me in particular since I wrote reporter. I suspect the best efforts are still to encourage the industry as a whole to adopt best practice. Considering the move to go mobile, and be more flexible on the front-end, using standards is just becoming more of a requirement. I think that will ultimately end up being the winning effort. It’s already winning as newer sites are generally pretty good when it comes to standards. The old ones will take time.

With Safari 3 and Opera 9.5 out, Firefox 3 taking off, IE 8 coming soon, it’s pretty obvious that standards are the future.

Safari’s New JS Interpreter: SquirrelFish

There’s an announcement on the Safari blog about SquirrelFish, their new JS interpreter. To sum it up:

SquirrelFish is a register-based, direct-threaded, high-level bytecode engine, with a sliding register window calling convention. It lazily generates bytecodes from a syntax tree, using a simple one-pass compiler with built-in copy propagation.

Some performance data can be found here, as well as here, which even tests against Tamarin (slated for inclusion in Mozilla2). I think the motive for this move might have been best summarized here:

  1. I can imagine the “performance per watt� power consumption for SquirrelFish is also much lower. Good for my iPhone’s battery life.

Especially with the iPhone going 3G next week which will consume more power, making a web browser be as efficient as possible with CPU cycles not only makes the experience better, but will save battery life. This doesn’t just impact the iPhone as Google’s Android also includes WebKit.

David Mandelin has some analysis and comparison to the Mozilla work being done on his blog.

It’s pretty interesting stuff.

Rebreaking The Web

It’s happening again. Once upon a time, browser vendors started adding their own features without consulting with each other and agreeing upon standards. What they created was a giant mess of inconsistencies across browsers and platforms that is still in effect today. Ask any web developer and they can tell you of the pains that they have suffered trying to make seemingly trivial things work everywhere consistently. It’s no easy task. Before IE 7, even an ajax required something along the lines of:

var httpRequest;
if (window.XMLHttpRequest) { // Mozilla, Safari, …
    httpRequest = new XMLHttpRequest();
} else if (window.ActiveXObject) { // IE
    httpRequest = new ActiveXObject("Microsoft.XMLHTTP");
}

That’s right, IE 6 didn’t support the native xmlHttpRequest object (more here). This is just one of many examples in JavaScript and CSS. document.all anyone?

The end result of this problem became to be known as the “Web Standards” movement. Simply put it’s an idea that code should follow a standard that results in consistent output across all browsers on various platforms. Write once, run anywhere. While it’s taken years for this to manifest, it’s slowly become a reality. Firefox, Safari, Opera have fairly consistent rendering (at least in comparison to the mess of just a few years ago on the browser scene. IE 6 was fairly poor in terms of modern web development, but IE 7 made progress, and IE 8 is Microsoft’s greatest effort to date to bring their browser up to speed.

Continue reading