Full SPDY Ahead

For those not keeping score, Twitter, and Facebook have both come out publicly in favor of SPDY. Twitter is already using it in production. It sounds like Facebook will be soon. Mozilla implemented it in Firefox. Opera has SPDY. Google, the author of SPDY is using it in production.

This leaves Microsoft and Apple as the holdouts. Microsoft’s HTTP + Mobility is SPDY at it’s core. Microsoft hasn’t started supporting SPDY in any products, but it seems inevitable at some point. They are a holdout in implementation but not opposed to SPDY it seems.

Apple is the last major holdout. SPDY hasn’t been announced for iOS 6 or Mac OS X 10.8. As far as I’m aware Apple hasn’t made any statement suggesting support or opposition to SPDY. However I can’t see why they would oppose it. There’s nothing for them to disapprove of, other than it’s not using their IP. I’d be surprised if they don’t want to implement it.

However given SPDY is a rather backwards compatible thing to support, I don’t see this holding back adoption. Nginx is adding support for SPDY (thanks to WordPress creator Automattic), and Google is working on mod_spdy for Apache. That makes adoption for lots of large websites possible.

While the details of SPDY and the direction it will go are still in flux, it seems nearly certain that SPDY is the future of the web. Time to start digging into how to adopt it and ease the transition. The primary concerns I see are as follow:

  1. TLS Required – While not explicitly required, SPDY essentially builds on TLS and virtually any real world application needs it. This means purchasing SSL certificates for any website you wish to use SPDY with. Some have argued performance and scalability, but Google, Facebook and Twitter use SSL extensively on commodity hardware.
  2. IP Address – Unless you use Server Name Indication (SNI), which almost no websites do because of compatibility, you need an IP address for every hostname that you use TLS with. That means until IPv6 is widely adopted, it will be putting further strain on the remaining IPv4 pool.

Both of the above concerns increase complexity and cost of building websites at scale and for those who are on a very tight budget (the rest of us will manage). Because of this, I don’t think we’ll see a 100% SPDY or HTTP 2.0 web for quite some time. Don’t expect SPDY for shared hosting sites anytime soon.

In a world of increasing surveillance and user data being integrated into everything, the benefits of TLS will be realized. Both Facebook and Twitter acknowledge it’s importance in preventing user data from getting into the wrong hands.

I, For One, Welcome Our New SPDY overlord.

Innovator’s Patent Agreement

Twitter today announced the Innovator’s Patent Agreement, their attempt to help defuse the patent chaos currently going on in the tech world. The real meat of the post is this:

It is a commitment from Twitter to our employees that patents can only be used for defensive purposes. We will not use the patents from employees’ inventions in offensive litigation without their permission. What’s more, this control flows with the patents, so if we sold them to others, they could only use them as the inventor intended.

I do like the GPL like virility of this. The one thing that I questioned was the term “defensive purposes”. That sounded vague. The IPA itself defines this:

(a) against an Entity that has filed, maintained, threatened, or voluntarily participated in an intellectual property lawsuit against Assignee or any of Assignee’s users, affiliates, customers, suppliers, or distributors;

(b) against an Entity that has filed, maintained, or voluntarily participated in a patent infringement lawsuit against another in the past ten years, so long as the Entity has not instituted the patent infringement lawsuit defensively in response to a patent litigation threat against the Entity; or

(c) otherwise to deter a patent litigation threat against Assignee or Assignee’s users, affiliates, customers, suppliers, or distributors.

This sounds like a big improvement to me. I’d still like to see something more explicit that deters Assignees and Inventors from agreeing to claims other than defensive purposes. While it says Assignees must get written permission from Inventors “without additional consideration or threat”. I question if that’s practical given the Assignee is generally the employer and controls the Inventors paycheck, medical benefits, and possible bonus.

Still, it’s an improvement over the current state of things. Kudos to Twitter for innovating and getting some dialog going. This still doesn’t get rid the need for some patent reform.

Getting RSS Feeds For Twitter Users

Want an RSS feed for a particular Twitter user? This used to be linked off the profile page(s) but since disappeared. It’s still available if you know where it is:


http://api.twitter.com/1/statuses/user_timeline.rss?screen_name={username}

Replace {username} with a username (example).

You could also put a bunch of users into a list and query that using:


http://api.twitter.com/1/lists/statuses.atom?slug={listName}&owner_screen_name={username}&include_entities=true

Replace {username} with the list owners username and {listName} with the list name (example). Strangely that’s only available in atom format.

I still find RSS handy for those accounts I don’t want in my stream but want to keep an eye on, as well as those I want to programmatically access or manipulate.

Twitter Client Gripes

Like many in my trade, I keep a Twitter client open all day. 140 characters works very well between compile times, reloads, uploads. I still use RSS extensively, but Twitter fills the gaps nicely as my brain is always looking for information to absorb (feel free to follow if you don’t).

To this day it amazes me that I can’t find a perfect Twitter client. Tweetie back in its day was pretty damn close, but since it was bought by Twitter, it went downhill to the point of being unusable on the iPhone. Amazingly priced at “free” it’s not worth the price. These days TweetBot is as close to perfect as I can find on the iPhone and I’d recommend it to anyone who is frustrated with Twitter for iPhone.

Largely due to neglect the Mac client is still usable to me, however it’s hardly awesome. Why doesn’t “command /” reliably bring the window to focus? Why can’t I set my preferred url shortener? The developer console has lots of weird select and focus issues. I could go on…

From where I sit, these are the most annoying things Twitter still hasn’t figured out:

  • Search Blows – This one everyone always complains about. Search isn’t good, and only goes a few days back. It’s a miserable experience.
  • Amnesia – Twitter has a very limited memory. You can only search a few days back. Your timeline can only go so far back. Even DM’s can only be retrieved a mystery period back. Everything eventually disappears. I actually backup my tweets to a MySQL database so I can search anything I’ve ever tweeted. Most don’t have this luxury. Perhaps they should just partner with Google and let Google handle their archive/search problem. Let Google pay for the data, and for the right to solve this problem.
  • DM Downgraded – This one is pretty specific to the new Twitter “Let’s Fly” UI. Direct Messages, are very obscured and buried. Yea I get it, you want everything out in the open. It’s annoying however to hide useful and sometimes important UI.
  • Incomplete Clients – There’s no interface that seems to do everything. If you want to know how many RT’s or Favorites posts have, the best UI seems to be the website. If you want to use a custom URL shortener, Twitter for iPhone has you covered. Twitter for Mac has no UI to show what client a tweet was created with, mobile with its limited screen size does. It also has no way to see RT stats for a tweet. Want to be notified when you have a mention or DM? iPhone or desktop client is best (that’s not the web clients fault). Amazingly these UI’s all come from the same company. Facebook (now) does a pretty good job on feature parody across web/mobile clients.
  • What are favorites/lists – I don’t think anyone has fully figured out what these really are and how they should be used. Is there a value to maintaining a list? It seems most use favorites as bookmarks to read later, some use it for marking tweets they really like. I know I’ve done both. Facebook hasn’t figured out lists completely either, though I feel they’ve at least given them a useful purpose for power users.
  • Spam – I think if a user signs up and just @replies a link to 50 people, an algorithm should be able to detect they are a spammer and stop it.
  • Placeholder – The thing that annoys me the most is they still haven’t figured out how to leave a placeholder on your timeline. Why can’t I just pickup where I left off? I need to search for it. Facebook never solved for this problem either. Amazon’s Kindle (and apps) solved for this brilliantly. Surely Twitter could adopt an API to solve for this. As someone who restarts their browser often due to work I’m doing, this makes the web UI unusable.

So what am I ignoring in terms of annoying Twitter client things?

Use SSL By Default

Twitter is now the latest site defaulting to HTTPS. Kudos to them. I love seeing the web get more secure, even if it’s one site at a time.

If you’ve got a site where login is required, please make sure to use SSL. It’s not that costly anymore. Even this blog uses SSL where necessary.

It’s not needed for general public consumption things (like this webpage), but anywhere a session can be hijacked or confidential data could be transfered, SSL is a good idea. When not possible to default, at least make it an option (I do this for safepasswd.com).

Web App Stores Via Twittter/Facebook

It seems likely to me that Facebook and Twitter will eventually be competing with Apple in terms of App stores. Facebook sort of already is with their extensive apps platform, however that’s just competing for developer attention. Twitter doesn’t really have an equivalent today (developers mainly build clients and interact with data), but don’t underestimate their clout.

The reason I say this is that Facebook and Twitter have become identity gatekeepers on the net. Already you can login to many sites via accounts with one of the two sites. Creating the API’s to handle purchase/subscriptions and transparently handling the billing to effectively turning a HTML5 site into an “app” is the next logical step. They could undercut Apple and still walk away with a handsome profit for not doing terribly much more than leveraging their size and reach. These apps would work on any device with a web browser. Desktop or mobile.

Given both sites need to diversify revenue streams (something Google never figured out), it seems only logical to make this step. $0.99 for Angry Birds seems more than plausible.

And yes, there are offline abilities in a browser.

Who Indexes Tweets

I was curious who is indexing the links that people tweet on Twitter. It’s obvious someone does since links get ‘clicks’ almost immediately after submission. To do this presumably they are tapping into the xmpp firehose.

Lets take a look:

66.xxx.xxx.xxx - - [06/Dec/2009:20:17:43 +0000] "GET /test HTTP/1.1" 301 20 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I guess Google has a deal with Twitter. Googlebot indexed just a few seconds after it was sent. As far as I know nothing is actually announced. This is the first evidence I know of a potential deal of some sort. I’d be shocked if Google is scraping the site this quickly.

Edit: Stephen Duncan pointed out in the comments that this was announced in October. Totally forgot about that.

208.xxx.xxx.xxx - - [06/Dec/2009:20:17:47 +0000] "GET /test HTTP/1.0" 301 - "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8"

This is Topsy, a twitter search engine. Never saw this site before. Few tests and I actually kind of like the output.

89.xxx.xxx.xxx - - [06/Dec/2009:20:17:58 +0000] "GET /test HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot"

Tweetmeme mines Twitter links and attempts to build a Digg-like index based on retweets rather than Diggs.

75.xxx.xxx.xxx - - [06/Dec/2009:20:18:05 +0000] "GET /test HTTP/1.1" 301 - "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
72.xxx.xxx.xxx - - [06/Dec/2009:20:20:25 +0000] "GET /test HTTP/1.1" 301 - "-" "Python-urllib/2.5"

Can’t identify these AWS hosted services.

70.xxx.xxx.xxx - - [06/Dec/2009:20:20:53 +0000] "GET /test HTTP/1.1" 301 20 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
70.xxx.xxx.xxx - - [06/Dec/2009:20:24:23 +0000] "GET /test HTTP/1.1" 301 20 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

This is actually Microsoft. Microsoft’s Bing search engine indexes Twitter. I’m not sure why they indexed twice in such close intervals that seems odd for this day and age.

Mining logs a little deeper it looks like when tweets meet certain criteria (such as retweeted) there are other bots that spider them. It also looks like other search engines may be indexing at a slower rate (Baidu for example).

There are several others from AWS and a few other dedicated providers. These servers are obviously trying to keep a low profile, they don’t even have reverse DNS.

So there you go. Just a matter of seconds after a link hits Twitter this all happens.

Here’s a few more from another Tweet that weren’t in the first set:

Edit: More!:

75.xxx.xxx.xxx - - [06/Dec/2009:20:49:42 +0000] "GET /test HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)"

Feedtrace is some sort of twitter mining service currently in beta.

67.xxx.xxx.xxx - - [06/Dec/2009:20:49:45 +0000] "GET /test HTTP/1.0" 301 - "-" "Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)"

Chainn is a social data mining service with a few apps that make use of the data it collects.

Capturing User Innovation

Building a new product is always fun. You draft ideas, generate wireframes, mockups, prototypes, you build your app, you tweak it, you release it. In the case of software and web applications you also get to update it and make it better. If it’s hardware, you work on a 2nd revision to be sold a year later to people who didn’t adopt early (jab at early adopters).

One of the most interesting things is how users actually use the product you make, if they use it at all. Do they use it a little or a lot? Do they use it as intended? Do they find things missing? To robust for their taste? Or do they just find uses and modifications that all the engineers involved never in a million years would have contemplated?

Continue reading

Loren Brichter On Tweetie

Loren Brichter is the author of the popular Twitter application Tweetie, an iPhone only application until the Mac version was released on Monday. MacWorld has a great little interview with Loren. One thing I really admire is that Loren really understands how to build a good application. Performance, ease of use, simplicity are all taken into account. Not just features and toys.

I thought this particular nugget was the highlight though:

..AIR apps are like modern day Java applets… sure, they run on every platform. But they also suck on every platform.

I’ve yet to find an Adobe AIR application I like even though several have great ideas behind them. Even on Windows, where I presume AIR has the biggest market share they all look strange, the UI is garbage and the performance is abysmal. On the Mac it gets even worse. Creating a Mac theme won’t help as my expectations for a Mac UI are different than they are on Windows or Linux. Java apps have the same issues.

I think this is why more and more “applications” are becoming web based. If your going to feel awkward and unnatural to the user anyway, why even bother with the installation barrier? Why not just be web based so you don’t have to download and install. As awkward as they may be, those that add Adobe Flash tend to make the problem worse by adding more strange feeling UI to their application. Adobe Flash does do good video, it’s a big reason YouTube became popular, but it’s really no replacement for user interface. Hopefully in 2017 when HTML5 is wrapping up we’ll have this problem solved.

User Generated Content Ownership

Since the creation of the <form/> elements people have been wondering about the ownership and copyright of content created online. From email and message boards in Web 1.0 to blogs and Twitter in Web 2.0 the same fundamental questions remain.

Lately, Twitter has been the focus. Twitter is actually pretty clear about it’s claims to user generated content:

  1. We claim no intellectual property rights over the material you provide to the Twitter service. Your profile and materials uploaded remain yours. You can remove your profile at any time by deleting your account. This will also remove any text and images you have stored in the system.
  2. We encourage users to contribute their creations to the public domain or consider progressive licensing terms.

It’s pretty clear that Twitter is taking a hands off approach, but it doesn’t let users decide what they want. I’m personally a fan of Creative Commons so my suggestion would be to let decide in their account settings how they wish to license and choose between CC licenses. That of course makes retweeting complicated to put it nicely (it’s more like a minefield). That’s likely the reason they avoid the licensing issue. Sure you can put some sort of an icon next to the tweet to indicate the licensing, but what if someone retweets it? Or modifies it ever so slightly? Is it a new tweet? How many characters must change for it to be a new one? This is where it gets murky.

Yahoo owned Flickr choose to solve this problem by letting users choose what copyright they want to impose, and include a Creative Commons option. A very graceful solution though admittedly their situation is much simpler than Twitter’s since they don’t have to deal with complexities like retweeting which would make things very complicated.

WordPress.com isn’t as clear in regards to it’s claims (or lack of) to copyright. Though they are far from locking people in considering you can delete stuff at any time and download your entire blog and move it elsewhere. Matt‘s been pretty open about giving users choice including the ability to leave WordPress.com. There is of course room for improvement to clarify their stance on copyright ownership.

Even Google has been criticized for copyright concerns regarding services like Google Docs.

They could adopt the Richard Stallman stance to “intellectual property” (his airquotes), though that would alienate at least as many as it attracts.

While Twitter might be the hot topic today it’s hardly a problem exclusive to Twitter. It’s an issue for virtually any site out there that accepts third party content. It gets more complicated when content can be remixed and redistributed.

The reality is people should know what rights they are giving up by putting content on these or any other services, but people rarely do. Perhaps a great Creative Commons project would be to create the same simplified icon/license system but for websites that allow users to submit content. The licenses would indicate what the impacts of the Terms of Service jargon are in plain English. It’s essentially the inverse of what they do now. Label the service as well as the content.

So what’s the best solution?