Perception Of Performance

Google is pervasive about associating Chrome with being fast. It’s was their primary pitch when they first announced it. Back when Firefox went 1.0, it wasn’t so much about speed but “not sucking” as all the geeks liked to say. Given IE 6 was the competition, that was likely the best marketing on earth. Sure it was faster, but sucking fast wasn’t nearly as good as not sucking. Not sucking encompassed the missing features, broken rendering, crashing, constant parade of security problems. It summarized the product surprisingly well for not being an official slogan by any means.

Google now launched Chrome for iOS. On the desktop Chrome and Safari both use WebKit, Chrome applies it’s own touches to make things faster. Notably they have their own JS engine. Safari also has it’s own JS engine. This is the secret sauce of performance. In the iOS world however Apple being the totalitarian dictator decided that iOS will provide WebKit and JS. If your app has any web browser functionality it will utilize these API’s and not implement it’s own engine. Verbatim:

2.17 Apps that browse the web must use the iOS WebKit framework and WebKit Javascript

Google Chrome for iOS however is Google integration into a reskinned experience of Safari. It’s the same browser. Just a new UI bolted on with some Google features integrated in. It’s not a separate browser. It’s a UI.

That however doesn’t stop Google’s marketing machine (I’d argue Apple marketing’s top rival) from putting “fast” as the second word:

Browse fast with Chrome, now available on your iPhone, iPod touch and iPad. Sign in to sync your personalized Chrome experience from your computer, and bring it with you anywhere you go.

It goes on to clarify:

  • Search and navigate fast, directly from the same box. Choose from results that appear as you type.

So Google isn’t truly misleading. It’s just very strategic wording.

The truth of the matter however is that Google Chrome on iOS is substantially slower than Safari. Safari uses Nitro to accelerate JavaScript, which powers most of the complicated websites that will slow down a browser on any modern device. Apple however restricts Nitro to Safari, and doesn’t let third party apps like Google Chrome use it. This is still the case as of iOS 5, and I believe is the case in iOS 6, though I haven’t personally verified that.

How much slower is Google Chrome on iOS in comparison to Safari? Well Here’s a SunSpider test I did on my iPad 3:

Safari

============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 1817.9ms +/- 0.2%
--------------------------------------------

3d: 214.7ms +/- 1.1%
cube: 72.3ms +/- 0.7%
morph: 57.9ms +/- 0.9%
raytrace: 84.5ms +/- 2.2%

access: 224.9ms +/- 0.6%
binary-trees: 44.4ms +/- 1.7%
fannkuch: 96.2ms +/- 0.6%
nbody: 56.0ms +/- 0.0%
nsieve: 28.3ms +/- 2.7%

bitops: 141.0ms +/- 0.4%
3bit-bits-in-byte: 23.4ms +/- 1.6%
bits-in-byte: 29.5ms +/- 1.3%
bitwise-and: 37.8ms +/- 1.5%
nsieve-bits: 50.3ms +/- 0.7%

controlflow: 15.7ms +/- 2.2%
recursive: 15.7ms +/- 2.2%

crypto: 123.3ms +/- 0.6%
aes: 70.5ms +/- 0.5%
md5: 29.4ms +/- 1.3%
sha1: 23.4ms +/- 1.6%

date: 274.4ms +/- 0.7%
format-tofte: 139.8ms +/- 1.1%
format-xparb: 134.6ms +/- 0.7%

math: 175.1ms +/- 0.3%
cordic: 61.5ms +/- 0.8%
partial-sums: 74.4ms +/- 0.7%
spectral-norm: 39.2ms +/- 0.8%

regexp: 70.8ms +/- 0.6%
dna: 70.8ms +/- 0.6%

string: 578.0ms +/- 0.5%
base64: 78.3ms +/- 1.9%
fasta: 68.1ms +/- 0.9%
tagcloud: 109.5ms +/- 1.2%
unpack-code: 207.5ms +/- 1.2%
validate-input: 114.6ms +/- 0.7%

Google Chrome

============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 7221.0ms +/- 0.1%
--------------------------------------------

3d: 802.7ms +/- 0.2%
cube: 230.9ms +/- 0.6%
morph: 297.3ms +/- 0.5%
raytrace: 274.5ms +/- 0.1%

access: 1112.0ms +/- 0.2%
binary-trees: 98.4ms +/- 1.1%
fannkuch: 609.6ms +/- 0.2%
nbody: 247.9ms +/- 0.2%
nsieve: 156.1ms +/- 0.4%

bitops: 957.2ms +/- 0.2%
3bit-bits-in-byte: 210.4ms +/- 0.6%
bits-in-byte: 232.9ms +/- 0.2%
bitwise-and: 188.5ms +/- 0.4%
nsieve-bits: 325.4ms +/- 0.2%

controlflow: 129.5ms +/- 0.3%
recursive: 129.5ms +/- 0.3%

crypto: 493.3ms +/- 0.2%
aes: 214.3ms +/- 0.4%
md5: 140.2ms +/- 0.3%
sha1: 138.8ms +/- 0.5%

date: 381.1ms +/- 0.3%
format-tofte: 214.2ms +/- 0.2%
format-xparb: 166.9ms +/- 0.5%

math: 770.7ms +/- 0.2%
cordic: 316.6ms +/- 0.2%
partial-sums: 243.2ms +/- 0.3%
spectral-norm: 210.9ms +/- 0.4%

regexp: 1340.2ms +/- 0.2%
dna: 1340.2ms +/- 0.2%

string: 1234.3ms +/- 0.6%
base64: 175.7ms +/- 0.5%
fasta: 205.6ms +/- 0.2%
tagcloud: 284.0ms +/- 2.3%
unpack-code: 370.1ms +/- 0.9%
validate-input: 198.9ms +/- 0.6%

Quite a bit slower.

So really, if you’re using Chrome on iOS, it’s because you absolutely love the design and integration with Google’s services, and are willing to trade off considerable JavaScript performance for those perks.

That however doesn’t stop many people from thinking it’s fast. Just in the past few minutes I’m able to find these Tweets among the thousands streaming across the web. I won’t mention or link to them directly (you could find them however if you wanted):

“Chrome for iOS is FAST, takes the mobile browsing experience to a new level.”

“I like it! It’s fast and can sync with Chrome desktop, which I use all of the time.”

“Liking #chrome on #iOS very slick, fast and clean looking”

“using chrome on my iphone right now.. cant believe how fast it is”

“That chrome for iOS is freaking fast but so basic. No tweet button, no add-on. Man I kinda disappointed. I give ‘em 1 ‘fore the update”

“Chrome for iOS? Hell yes!! So fast! #chrome”

“Google Chrome for iOS is fast.”

“Holy hell Chrome is fast on the iPad.”

The most touted feature isn’t actually a feature. It’s technically not even there. The numbers and the technology insist that it’s not (they prove it’s actually slower). But that’s what everyone is ranting and raving about. You could argue Google’s UI is faster, but I’d be highly skeptical that Google’s found Cocoa tricks Apple engineers haven’t. Perhaps a UI transition or two makes you think it’s faster or more responsive, however even that I can’t find any evidence of.

All the hard work the Google engineers did squeezing their services into a compact simple to use UI are ignored in favor of this non-existent feature. And as a developer who can’t ignore such a thing, I will say they did a great job with their UI.

I present to you, the power of marketing!

17 thoughts on “Perception Of Performance

  1. I present to you, the power of marketing!

    Which is funny, since that’s traditionally where Apple has won out – good UI, and better marketing.

  2. Well, right and wrong.

    First, on a perceptual front, there’s no doubt that marketing does convince people of things. And there’s also no doubt that sometimes people come to bizarre conclusions on their own for inexplicable reasons. So I have no doubt that some people happy about Chrome’s performance (but also some set of people who have come to all other possible conclusions as well) are somewhere between “misled” and “smoking crack”.

    Also, you’re right that by not being able to use V8 (or even Nitro), we’re a hell of a lot slower in executing JS on iOS than mobile Safari. Sunspider is a particular craptastic benchmark to use to make any kind of a point, but the point would stand regardless.

    OTOH, as many other vendors have — rightly — argued at times when our marketing folks started emphasizing JS speed, JavaScript is not all things. In particular, Chrome for iOS does have a few tricks up its sleeve:

    (1) We use our own network stack. AFAIK (I haven’t done any work on iOS Chrome directly), it supports SPDY and the numerous other things we’ve done to try and increase network speed (many of which, such as an increased number of parallel connections and DNS prefetching, have been done by other vendors as well, including Mozilla).
    (2) We support aggressive prefetching, especially for the omnibox and top search results, which really does have truly enormous effects on actual usage that are harder to capture in a benchmark. (Omnibox prefetching on desktop, where typing is easier and faster than on a phone, already can save several seconds for a single page load.)
    (3) The omnibox itself reduces cognitive load, and thus leads to both a perception and reality of a faster and easier UI.
    (4) Inasmuch as people start doing tasks that Chrome provides good accommodation for — e.g. loading sites on their phones that they were last looking at on a synced desktop machine, or flipping over into incognito mode — having UI affordances for those tasks makes them faster and easier.

    These aren’t just marketing, and you even note that we “did a great job” with our UI. But you don’t seem willing to credit that UI, let alone the other bits we still do have control of under-the-hood, with actually making people’s browsing truly faster. Chrome is not definitively slower than mobile Safari; it’s slower at Javascript, and it happens to be faster at some other things in some circumstances. An unfortunately vague and nuanced statement, but I’m an engineer, not a marketer, and I’m not going to lie for my company’s product. Or easily let misimpressions stand uncorrected. :)

    (Postscript: I don’t think you’d find a person on the Chrome team who wouldn’t love to be able to actually ship the full-fledged, multiprocess, V8-using, way-the-heck-better-port-of-WebKit-than-the-iOS-team-uses Chrome on iOS, and make it definitively fast in the way that this release isn’t. But you won’t get me to lay odds on when Apple will ever allow such a thing.)

    • iOS Chrome may currently be faster than Safari for some things, but Apple can catch up in those areas by copying your features, whereas Chrome’s JS performance is permanently capped by the lack of a JIT so you can never catch up in that area. I don’t like your odds.

      To be honest I’m surprised you’d risk dilution of Chrome’s brand this way. There’s going to be plenty of confusion among Web developers and users about what “Chrome” means. Lots of features that are “in Chrome” won’t be in iOS Chrome. Some sites that “work in Chrome” won’t work in iOS Chrome.

      But more competition is good, and good luck to you.

      • Judging from the positive reviews, comments (pick your tech website – HN, Reddit, Engadget) and user feedback, Chrome on iOS is well on its way to becoming a real choice on the platform. Rather than fight Apple to change its policies, the chrome team have created a buttery smooth and fast UI that provides the perception of performance. Based on my usage, the responsiveness of the UI makes up for some of the underlying inefficiencies of a UIWebView.

        Perception is reality. In chrome’s case, the general perception is that it is a fast browser on the desktop. Fast is subjective – a typical iOs user cares first and foremost about interface responsiveness. Chrome is using tricks that ensure this. Therefore, the brand value is not diminished. It is stil fast! Maybe not in the true developer benchmark sense of the word. But honestly, sunspider is an unknown entity outside the developer crowd. Not many users are going to download chrome and make running these benchmarks their first order of business.

      • And if Apple indeed copies all the good parts of Chrome, then users win, and I’m happy.

        The web developer fragmentation effect has definitely been a real concern, and not everyone on the team has felt like the overall benefits of iOS Chrome outweigh the costs.

      • Manoj, slow JS means that interface responsiveness of Web content will often suffer … and aren’t modern browsers supposed to be about “content, not chrome”?

      • Robert, I understand that at a conceptual level. JavaScript performance, especially for a desktop browser is a table stake these days. The mobile use-case is slightly different. Mobile content is optimized, and not extremely JS-heavy (games are native apps, JS-heavy web apps like gmail are native apps as well). Even Facebook announced today that they are going to make a native objective-c app for iOS.

        As things stand, a speedy Chrome that loads content without discernible lag, and provides a number of cool features like desktop sync, unlimited tabs, etc. is going to garner attention and wide-spread adoption.

        If you don’t believe me, check the App Store; Chrome is the #1 free app today.

    • Thanks for the info.

      FWIW I very much want to see a real native Chrome with v8 on iOS. Just like I want to see a native Firefox using Gecko on iOS.

  3. It would be interesting to see data on network pre-fetching effects. From my qualitative observations, I’ve never seen it help much. I imagine its biggest winners are users who search in the urlbar and then stare at the results for several seconds before picking one of the top results though, which just seems like a really bad interaction (but I’d believe that it happens occasionally, especially with the Omnibar which I’ve always found confusing to read).

    It would be even more awesome if the Chrome team was actually open source though. At least there’s one (pretty awesome and super fast) open source mobile browser out there right now!

    • The “13 years saved per day” stat in yesterday’s I/O keynote referred specifically to Chrome’s omnibox prefetching. Even for expert users like me, it can save maybe a couple hundred milliseconds as I type a few more letters than I needed to (while inline autocomplete has already guessed the correct site) and then hit enter. A few hundred milliseconds on a navigation is a big win; if we got that from a WebKit rendering speedup we’d be overjoyed.

      I don’t know what the ultimate plans are for open-sourcing of the iOS Chrome UI layer. Certainly the backend is to be unforked and open-sourced. I hope the frontend will be too.

  4. And if you would do good marketing, you should also offer Firefox for iOS with webkit. Of course it’s only a new UI plus a few features like your own sync etc. But the end user does not care / does not even know what Gecko and Webkit are.

  5. I wanted to point out benchmark results at AWFY, but apparently guys have pulled the Nitro results. The reason is, Nitro is faster in most tests, mainly SunSpider ones.

  6. Measuring via tests is fine and all…for academia, but when it comes to user’s and how they actually perceive the responsiveness of the app *experience* that is what matters.

    For example, if you use Safari, it does not know my commonly used bookmarks or even URLs from my desktop experience. So guess what. I have to *manually* type it in and that is SLOW. But with Chrome for iOS, if I type “b” it immediatley pulls up bloomberg.com and along with its prefetching mechanism, the site loads quickly. This is FAST compared to Safari.

    With any good product launch, there is certainly marketing dollars behind it. But this isn’t new from Chrome — they’ve been marketing speed as the number one differentiator for its browser from day one.

    The thing that matters is how the perceived, human being experience actually is and currently, Chrome wins that battle hands down.

  7. I am curious how Microsoft lost anti-trust law suits in regards to bundling IE with Windows but Apple is A-Okay not only bundling their browser technology with iOS but also making it impossible for any alternatives… Very puzzling to me

  8. Pingback: A Smattering of Selenium #99 « Official Selenium Blog

  9. Pingback: Three cheers for Android browser competition | Android Developers

Leave a Reply

Your email address will not be published. Required fields are marked *

Connect with Facebook

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>