Thunderbird Sync With iPhone/iPod touch

I’ve gotten quite a few emails over the past several months from people who want to know how to sync their iPhone (or iPod touch) with Thunderbird. Quite a few are disappointed to find mozPod doesn’t support these newer devices. It’s not quite my fault as I’ve yet to see any indication that it’s practical to implement.

I should note I do not own either device, but from what I’ve read, it doesn’t matter. Hopefully by 2nd or 3rd generation when I’m on the market Apple will have come to their senses.

The problem is that Apple hasn’t provided a good method for anyone to provide sync with the iPod. Even on older devices it wasn’t pretty, but it was workable. There is no real sync API outside of iSync, which is Mac only and not a public API. Instead what was done is mount the iPod file system and send it your data. While not awesome, this has been workable and provided many Thunderbird users with mobile data.

On the iPhone and iPod touch it’s not even possible to easily mount the filesystem. The best method I’ve heard of is hacking it so you can mount your phone using sshfs with something like FUSE. I know myself and some Linux and Mac users (horray for MacFUSE!) could manage, but I wouldn’t dare try to explain to someone how to do that. From what I’ve read the iPhone manages data using SQLite 3. Therefore it seems possible to use mozStorage to interface with it. Perhaps someone with more experience with mozStorage and db’s generated outside mozStorage would know better about any potential compatibility issues.

Calendar:
/var/root/Library/Calendar/Calendar.sqlitedb

Address BookL
/var/root/Library/AddressBook/AddressBook.sqlitedb
/var/root/Library/AddressBook/AddressBookImages.sqlitedb

Seems to me Apple could turn this into an API pretty easily by re-creating the old iPod style USB storage device mounting and give us access to these files.

In my mind the ideal implementation would be for Apple to just have a iTunes embed a mini web server locally accessible (obviously) and create a REST api to work with all data on the device. That would open up a whole new way of interacting with the device. Essentially you would interact with 127.0.0.1:[port#] and GET/POST your way through the interface. Perfect for Calendar, Address Book, Notes, or whatever else Apple comes up with.

To get a vCard for someone:

  GET /contact/vcard/?last_name=Jones HTTP/1.1
  Host: localhost
  User-Agent: mozPod/0.3

vcard can also be done as xml (ooh DOM parsing for iPhone!), so there’s lots of possibilities.

To add a contact:

  POST /contact/add/ HTTP/1.1
  Host: localhost
  User-Agent: mozPod/0.3
  Content-Length: 450
  Content-Type: application/x-www-form-urlencoded

  fname=John&lname=Doe&address=1%20Large%20Road...

Though it would likely make sense to go with a more vcard like naming structure.

For calendar iCalendar or vCalendar would make most sense:

 GET /calendar/ical/?start=1-1-1970&end=1-19-2038 HTTP/1.1
  Host: localhost
  User-Agent: mozPod/0.3

You get the idea. Dead simple access. You could then use something likecurl to manipulate the iPhone’s address book via your existing application. It would be only accessible locally, so it’s as secure as your system. It would actually make things more secure since it would be one less reason for tinkering on the actual device, and it would open up a whole new market of iPod/iPhone software to go with all those accessories that have helped boost Apple’s market share. SyncML is cool, but SyncML is also a beast and reminds me of SOAP, WSDL etc.

mozPod 0.2a1

mozPod 0.2a1 is available. It’s alpha because it hasn’t been as well tested as of yet. I wanted to get it out before Thunderbird 2.0 ships, and I’ve been getting a fair number of requests for it lately.

I’ve released MozPod 0.2a1 as an interim release for Thunderbird 2.0 users who want to use mozPod and see some new features. I decided to not support mozPod 0.1 on Thunderbird 2.0 to keep things easier to manage.

This is an alpha release and likely has some bugs. I wanted to get it out for those who want to start testing. This would be an ideal time as people want to move to Thunderbird 2.0.

Here’s the changes that matter:

  • Feature – Preliminary support for Lightning (if installed).
  • Enhancement – Thunderbird 2.0 support.
  • Enhancement – Some performance tweaks.
  • Fix – Sync all available AB’s.
  • Fix – Correctly handle notes that are more than one line.
  • Fix – Skip over LDAP servers in Address Book without failing.
  • Fix – Try to not hold lock on disks.

As usual, if you like it and want to encourage me to spend a little more time on it, feel free to do so. I do request some feedback. Let me know how it works for you.

I’ve got more extension goodness on the way. I’m planning to get to a real mozPod 0.2 release in the next few weeks. There may be a new extension on the way as well…

You download it from this link: mozPod 0.2

In Search Of The Missing Sync?

I had another silly thought this evening, and thought I’d just mention it for anyone interested. I’ve been reading through some bugs, and wiki documents on the state of synchronization of various devices with Mozilla based products.

It would be great to have a generic XPCOM wrapper for interacting with Bluetooth support on various platforms (yes, you read that right, and trust me it’s not that crazy). It would need to have the ability to detect what Bluetooth implementation is used by the user for Windows (Broadcom/WIDCOMM vs. Windows XP), Apple’s Bluetooth implementation and of course Linux (I believe it’s Bluez, but I’m not to familiar with Bluetooth on Linux). Note that this does not mean having Mozilla drivers for Bluetooth hardware. Just an easy way for developers to interact with Bluetooth without the concept of platform specific software.

With projects like Lightning, Thunderbird maturing, and MiniMo it’s important to be able to interact with other devices for the sake of forward progress, and this may prove to be the best solution for many cases.

There are several approaches to device sync from what I’ve seen from things that are device specific likePalmSync to the more generic iSync, but the problem that comes up is that it’s a lot of work for relatively limited devices. For example, even with iSync, as robust as it is, your still limited in devices it supports, and it’s Mac only (leaving Windows and Linux users in the cold). Others still require commercial software (Palm) to work with iSync. So for all the work, all Windows users have no benefit from iSync support. For all the effort on PalmSync for Windows done thus far (and that’s a fair amount of effort), no Mac users have been able to benefit thus far. SyncML has some potential, but the lack of support thus far still leaves many without any support. Anyone without a newer device likely won’t ever have SyncML support, so again the problem is a limited device set. And still not all newer devices support SyncML, and it could even depend on your service provider, as they sometimes disable features.

The advantage of Bluetooth is standardization. Lets face it, love it or hate it Bluetooth is becoming more and more available. More and more portable devices are shipping with it. By having support for it, Mozilla products gain compatibility with devices as they come out. It also would provide good device support over multiple platforms. For example, Thunderbird’s Address Book could have a small database of device types (phone/PDA models) and the format of an address book they support (such as vCard) or Calendar (vCalendar or iCal).

It looks like Synchronization Profile (SYNC) and File Transfer Profile (FTP) would be necessary to get most devices going (seems not all support SYNC, so supporting FTP would allow you to at least dump your Address Book or just an Address Book card from time to time onto your phone.

If included with Firefox, there would be an easy way for product developers, and extension developers to add a “send with Bluetooth” menu option. Save snippets of a web page, an hCard, the possibilities are endless.

This wouldn’t work for every device (such as the iPod thus far which is Bluetooth free, but I’ve got you covered on that ;-)), but it’s a big gain with supporting 1 protocol.

I could even see this as an interesting approach to keeping bookmarks in sync. Perhaps store your bookmark file on your cell phone… well you get the picture, mobile devices are without question the future, the question we need to ask is, how do we transport data back and forth?

It doesn’t look like something I’m personally up to implementing (I’m not very strong when it comes to this stuff), but it would be pretty cool to see someone take on. The key to this is that it needs to support SYNC and FTP, as well as be as multi-platform as possible. That’s not an easy task, but the possibilities are rather impressive.

I’m curious what others think of such an idea. I’m not saying it’s 100% workable, or warrantied for any particular use. Just some idea in my head I felt like putting into text and sending down the series of tubes we call the internet.

Lightning Strikes The iPod

I started working to implement support for Lighting (project to integrate Calendar into Thunderbird) to sync with Apple iPods via mozPod. Didn’t take to long before I had a successful sync. It’s not done yet, and likely some big evil bugs (read: including but not limited to loss of data or first born child), but it’s well on the way!

That’s right, we now have the ability to sync contacts and calendar to the iPod on Mac/Windows (Linux still on the todo list, though it’s mostly there). It will require Thunderbird 1.5 or later. No release date just yet.

How cool is that? 😀

mozPod 0.2 Status

I have slowly been working on a new mozPod release, it’s just not going very fast, as it’s still a lower priority project. So far, it seems to be pretty well accepted. Here are a few of the changes planned, or already completed:

  • Preliminary Lightning Support
  • Having an LDAP server setup in your Address Book won’t cause the sync to fail
  • Some code cleanup, optimization, and bug fixes

For the record, I will be dropping support for Thunderbird 1.0.x in mozPod 0.2. Most people seem to be upgrading, and it’s just not worth the hassle. Many (or most) seem to have had problems with MozPod and Thunderbird 1.0 anyway.

Yes, development is a little slow, but it’s free (unless you feel like saying thanks), so don’t complain ;-).

Update: It’s out.

mozPod 0.1 Released

I decided to release mozPod 0.1 as a late Christmas Gift. This is of course a 0.1 release (beta quality at best), so of course feedback is desired. I’m positive bugs remain, and I’m positive there are some larger bugs still in that code. Testing is needed to locate them. So if you download, make sure to come back here and let me know how it goes for you. I know there is no unicode support just yet, and the Thunderbird UI may get a little slugish during sync (though once sync is done it should go back to normal).

Download

MozPod

For those who don’t know, mozPod is a Mozilla Thunderbird extension to allow for easy transfer of the users address book to the popular Apple iPod for mobile use. The extension has a minimal UI and is designed to be as automatic as possible.

Thanks

Special thanks to David Bienvenu (who hopefully will get a blog sooner than later 😉 ), and Boris Zbarsky for their help, and everyone who volunteered and participated in testing.

iPod Sync Private Testing

Before I do a more general release, I want to get a few more technical people to give it a try, so I can get any big issues resolved.

Please email me (using the form) the following info if you wish to participate:

Name
Email
OS
iPod Version
Experience

You will need to be using Thunderbird 1.5. This is rather untested stuff, so I do put an emphasis on only technical users with experience testing and doing QA with Mozilla products.

If you don’t think you qualify, just wait a little longer until this round is over. I do plan on a 0.1 release by the end of the year.

Update [12-26-2005]: It’s out, and available here.

iPod Sync Unteaser

Unfortunately it’s not done yet. The big issue remaining is to ensure the Thunderbird UI doesn’t completely freeze up (which many people apparently don’t like). I just don’t feel it’s right to release like this, since it’s annoying and problematic. My first stab at it was unsuccessful, so I’m taking it on from another angle, but need some solid time to sit down and really dig into it. I’ve been hoping to also handle non-ASCII characters as well, but that might get pushed out to the next release. My plan was to have done this already, but real life interferes with my plans at times. This week is Thanksgiving (read: I’m not coding when I can be eating, as I’m sure most of you can relate with), and I’ve got finals rapidly approaching (read: less coding time). I’ve also got a few other projects that I’m balancing here. There’s lots of great announcements coming up on this blog in coming months, so keep an eye on that RSS feed.

So how much longer?
The correct answer is “when it’s done”. My true hope is by the end of the year at the latest (which is only a couple weeks). Hopefully earlier than that (I’d really like to have it by early December). It all depends how much time I get to work on it.

I guess that’s just life… everything takes a little longer than you want.

Daniel Glazman is sitting on two really sweet extensions (CaScadeS II and OpenWengo), and hacking away at them ever so diligently. So I’m not the only one making people crazy by not going fast enough (seriously, check out some of those screenshots on his blog. Really cool stuff he’s been working on).

While your waiting, check out the Mozilla Quality Blog for ways you can help ensure 1.5 totally rocks. There’s no such thing as too much testing.

Update [12-26-2005]: It’s out, and available here.

iPod Sync

Looks like things are delayed for several days. The biggest issue is not freezing the Thunderbird UI during the sync. Boris Zbarsky was kind enough to take a stab at getting threading going, but that doesn’t look like it’s going to work out. I’ve got another approach in mind, thanks to David Bienvenu, but I’ve yet to get that worked out. In addition to that, I’ve got enough going on that I don’t have the necessary time to code/debug that right away. This is still a hobby and not a job [shameless plug I’ll be looking for one this summer], so please be patient.

So earliest possible timeframe is some point next week. More realistically is 2-3 weeks, because there are several things (of various sizes) I would like to actually do before I start with some testing. Rather than a release I’ll likely take a few beta testers reading this blog (so keep reading if your interested), and expand from there based on feedback.

For those wondering, I will be supporting Lightning, though likely not until there’s more integration with Thunderbird. I’ll judge that as we go along. Lightning support is still in question. Having two instances would be rather awkward, both polling for the iPod device. Something that just shouldn’t happen. Not to mention Sunbird is still somewhat early in development. I’ll evaluate that as time goes on, based on demand, and Sunbirds development.

It’s great to see so much support for this project. It’s more than I actually expected to see.

Update [11/08/2005 11:05 AM EST]: Corrected Sunbird/Lightning confusion.

Update [12-26-2005]: It’s out, and available here.