Category: Open Source

The code is out there

Categories
Mozilla Open Source

Why Open Source Is Pretty Awesome

At some point I think it’s easy to take things for granted. Being able to alter software to meet your needs is an awesome power.

Today, a tweet rehashed an annoyance regarding a tactic on websites to alter copy/paste and put a link with tracking code in your clipboard. I could opt out, but that doesn’t fix when websites roll their own. It’s a fairly simple thing to implement. In my mind there’s little (read: no) legitimate justification for oncopy, oncut or onpaste events.

So I did an hg pull while working on some other stuff. I came back and wrote a quick patch, started compiling and went back to working on other stuff.

Then came back to a shiny new Firefox build with a shiny new preference that disabled the offending functionality. A quick test against a few websites shows it works as I intended by simply killing that event. You can’t do these things with closed source.

Of course I found the relevant bug and added a patch for anyone interested.

A 15 minute diversion and my web browsing experience got a little better. Sometimes I forget I’ve got experience on that side of the wire too 😉 .

Categories
Mozilla Open Source

On webOS Going Open Source

webOS is going open source. I’ll start by saying I’m rooting for webOS. I’m skeptical webOS will have much success given the announcement. An OS is a huge undertaking. A mobile OS is even more difficult.

Define “open source”

The press release says “underlying code of webOS available under an open source license”. Technically Apple can say the same thing with OS X and iOS*. Working on or with an OS is an investment. A very large investment. If it’s not complete or nearly complete, it’s not going to fly. Similarly unless the license is free enough, it’s not worth the investment. It sounds like it will be pretty inclusive and liberally licensed (Apache could be a good choice), but until that happens, I wouldn’t place any bets. Especially with HP’s seemingly bizarre behavior lately.

Ecosystem/Community

Building an ecosystem and community around that is going to be tough. Years ago with no competition except a stale IE. AOL gave $2M US Dollars to start the Mozilla Foundation and that had open source legs for years already under Netscape. While few people knew of “Mozilla” and even “Firefox” both in name and concept were a while away, it was a popular browser on Linux and in some more technical crowds. webOS is starting off against Google Android. Google has resources. Google isn’t Microsoft in this story. Google won’t be Microsoft.

Mozilla was also “just” a browser with much less surface area than a mobile OS. By that I mean hardware and dealing with the Linux community intricacies. Releasing the source alone won’t do it. HP reportedly had about 500 engineers working on webOS. That’s the type of effort it takes. Google puts substantial resources behind Android.

Lastly, people don’t install open source OS’s on their phones. They don’t install any OS’s on their phones except upgrades. That means hardware partners are critical for any viability. Hardware vendors already have deals and plans with Google. This is going to be tough to penetrate. Mozilla never had much luck getting desktops to ship with Firefox. The vast majority of users choose Firefox. On desktops, at least for now that is an option. On mobile hardware that’s not generally the case.

Even if someone comes up with a way to root and “upgrade” Android and/or iPhone devices to run webOS, you can be sure hardware vendors and mobile providers will be in front of Congress the next morning to outlaw the practice and stop it (or claim it’s “wiretapping”). Given the money behind App Stores and mobile payments, which is already a mess, there’s too much money there. These “rogue” devices could be banned from major networks if it got traction.

I’d love to see it survive and thrive. I’d love to see a PC like community of hardware vendors. But it’s going to be an uphill battle.

More than likely, pieces will be taken and strapped to Android as a HTML5 based Adobe Air like platform for building/deploying apps. It may also find some use in non-mobile purposes from entertainment devices to home alarms. As more devices become ARM based computers vs. microcontrollers, webOS like Android could be a way to get started building an interface. I see that as being more likely than continuing as a mobile OS.

A successful open source project takes a lot more than most give it credit for. Source alone doesn’t do it. It’s the community and ecosystem that sustains a project, not a tarball.

* I’d consider Android half open considering it does source dumps and develops largely in private.

Categories
Open Source Programming

DBSlayer + Node.js

Lately it seems the rage among developers is to take node.js and combine it with something else unusual. So here’s my contribution.

DBSlayer is a project by NYTimes a few years ago that seems to be somewhat forgotten but is pretty cool. It’s another MySQL proxy, but with a slight twist. Rather than use a binary protocol, or XML, they went with JSON. It supports things like connection pooling, round-robin distribution to slaves, automatic fallover, and it’s mutithreaded. It’s pretty fast and easy to work with. It’s almost like turning a MySQL database into a REST API. You pass a SQL query as a query argument and it gives you a JSON response.

Once you start it you can do something query using a request like:

http://localhost:9090/db?%7B%22SQL%22%3A%22SELECT%20*%20FROM%20facts%20WHERE%20id%3D1%3B%22%7D%20

That will give you a JSON object containing the result of your query.

So doing that in node.js is roughly:

var sql = ‘{"SQL":"SELECT * FROM facts WHERE id=7;"}’;
 
var http = require(‘http’);
var client = http.createClient(9090, ‘localhost’);
 
var request = client.request(‘GET’, ‘/db?’ + escape(sql), {});
request.end();
request.on(‘response’, function (response) {
  console.log(‘STATUS: ‘ + response.statusCode);
  console.log(‘HEADERS: ‘ + JSON.stringify(response.headers));
  response.setEncoding(‘utf8’);
  response.on(‘data’, function (chunk) {
    console.log(‘BODY: ‘ + chunk);
  });
});

Running that looks like this:

$ node test.js
STATUS: 200
HEADERS: {"date":"Fri, 27 May 2011 02:02:27 GMT","server":"dbslayer/beta-12","content-type":"text/plain; charset=utf-8","content-length":"290","connection":"close"}
BODY: {"RESULT" : {"HEADER" : ["id" , "fact" , "author" , "ip" , "timestamp"] , "ROWS" : [[7 , "1+1=2" , "raccettura" , "127.0.0.1" , 123456]] , "TYPES" : ["MYSQL_TYPE_LONG" , "MYSQL_TYPE_VAR_STRING" , "MYSQL_TYPE_VAR_STRING" , "MYSQL_TYPE_VAR_STRING" , "MYSQL_TYPE_LONGLONG"]} , "SERVER" : "db"}

You could obviously clean that up and create a little library to hide the HTTP parts.

It’s an interesting JS centric way to abstract your database while maintaining SQL level control. JSON is becoming the new XML.

Categories
Around The Web Internet Open Source

Developing A Thick Skin Is Bullshit

From Sarah Lacy’s TechCrunch post Conan O’Brien’s Love/Hate Relationship with the Internet is a very interesting insight from Coco:

O’Brien said the biggest thing that held him back from both writing and performing was a fear of being criticized because he’s incredibly sensitive. He punched a big hole in one of the biggest clichés in fame—that you just have to develop a thick skin. He says he’s still just as sensitive and criticism still hurts just as much. The secret is to just keep going anyway, because you will get criticized no matter how brilliant you are.

This is clearly something that’s gotten more pronounced in a Web age, but there may be a silver lining to that. In a time when every video, photo, blog post and Tweet can easily be trashed by others, people learn that criticism is inevitable early on.

The sensitivity to criticism I suspect is a motivator to O’Brien himself and I’m certain to others. It’s hard for it to not have an impact, even on the most jaded of individuals.

I think that has become true for anyone in any industry, not just night show hosts. To quote a movie title “We Live In Public” thanks largely to the Internet.

I learned long ago to take criticism and praise with a grain of salt. Neither are always genuine or accurate, especially on the Internet. Anonymity does make people more aggressive, but it also sometimes makes them more honest. Most people have a hard time giving criticism to you’re face even when asked to do so. It’s almost too easy when you can type it from a distance. Praise can have many false motivators that can often be hard to detect online.

It’s never “fun” to get a really nasty or critical blog comment, email, article, blog post written etc. about you or something you’ve done. This is especially true if you’ve dedicated a lot of time and effort. Regardless at some point you need to ask yourself: Is there any truth to this? Can I do something better? Once you’ve done that, it’s time to move on and plow forward. You won’t always be able to learn something, but that’s OK.

From my experience being a web developer, writing code that’s open source, and blogging means you’re going to get feedback, welcome or not. With certainty 100% of it will not be positive. Some will be negative, some will just be vile. Some however will be constructive. It’s to your advantage to use it.

Criticism and debate are a critical part of academia. Given programming’s still strong ties to academia, perhaps more so than many other industries may be why it seems so natural to accept for me and at least some of my peers. It’s like being graded in school. Or perhaps it’s because I started at about 14 or 15 years old as a developer. I got used to this sort of thing pretty early on during my formative teen years and it’s more natural for me. I guess I’m not really a true part of the trophy kids generation despite falling into the age group.

Categories
Open Source Web Development

Introducing “It’s All Good”

Several months ago I was looking for a good way to monitor not just my server, but the actual services on the server. Just responding to a ping doesn’t mean everything is OK. As the old saying goes “if you can’t find it, build it”. The result of this is a little project called It’s All Good.

At its core it’s a light framework for checking various aspects of a server and deciding if things are operating within defined parameters or not. So far it has “out of the box” support for:

  • CPU Load – As simple as it sounds. Check that your CPU load doesn’t exceed a threshold you define.
  • Disk Usage – Sets off an alarm when your server is running low on disk space.
  • SMTP Ping – This makes a connection to your SMTP server to check that it’s online and operational.
  • MySQL Check – Checks to see if it can make a successful connection to a MySQL server.
  • HTTP(s) Check – This can connect to a HTTP or HTTPS server and check that it connected successfully as well as check that for a condition on the page. This is handy to make sure a web app is up and running or that your SSL cert isn’t expired.

Like I said, it’s just a framework, so adding other checks are relatively easy. There’s lots more I want to include (memory, disk IO, process monitor for example). It’s designed to monitor the host, not a series of servers (though technically doable). This isn’t Nagios, it’s a way to get a quick glance at your key services on a host.

On its own it doesn’t send any notifications. It’s designed to be combined with the keyword monitoring feature of services like Pingdom, Monitis, Host-Tracker, SiteUptime, or Howsthe.com among others. This way you not only check services, but the server itself. If anything fails, you will be notified by your monitoring provider.

It’s All Good also has a UI for an admin to view which can give you the status and a basic rundown of its polling data. It’s also designed to so that it’s pretty easy to read on mobile devices like the iPhone, making it a great dashboard for on the go.

Lastly it’s designed to be pretty light and quick, so unless you are monitoring a ton of things on your server, it shouldn’t have any real overhead.

So far I’ve only implemented real support for the checks for Linux. I suspect most will work on BSD, and Darwin (though not all). Windows still needs some help. Patches are welcome. I’d also like to support things like IP whitelist/blacklists (automated via RSS fetches), and lots of modules to extend what it can keep track of.

Licensed GPL v2.

Categories
Apple Open Source Software

VirtualBox 3.2 Beta Supports Mac OS X Guests

Interestingly one of the new features in Sun Oracle’s VirtualBox 3.2 Beta is:

  • Experimental support for Mac OS X guests

I’m curious how they implemented that so that they steer clear of Apple’s legal team. I’m also curious how that runs. I may need to give that a try.

Categories
Open Source Software

Kernel Upgrade Fun

Server Load

A few days ago I did a kernel upgrade from 2.6.24 to 2.6.32.1. Surprisingly the load on the server has dropped slightly. The server is generally under minimal load, just the way I like it so a drop is particularly surprising. It was restarted just a few weeks prior, so I don’t think the restart had an impact on load. Unscientifically it appears the box is under the same level of usage as prior to the upgrade. The two spikes that delimit the restart are due to some log processing.

Server Load

Categories
Networking Open Source Web Development

Yahoo Traffic Server Open Sourced

Way back in 2002 Yahoo acquired Inktomi who was largely know for their search products. Their software powered some early search engines like HotBot in the pre-Google days. One of their lesser known products was something called Traffic Server. Even if it was lesser known it was still used by ISP’s including AOL, who in those days was big. Their business disappeared with the great bubble and they were acquired by Yahoo, who was using Traffic Server themselves ever since.

Fast forward to 2009. Yahoo is now in the process of opening up Traffic Server as an Apache project. It’s already in incubator. Yahoo says it’s capable of 30,000 requests per server. Noteworthy is that this runs on generic hardware.

These days most websites use either Squid, Nginx, Pound or Varinish on the open source side. On the proprietary side there’s Citrix NetScaler, Foundry (now Brocade) ServerIron, Zeus ZXTM or F5’s Big-IP. The proprietary side can be either expensive software running on generic hardware or an appliance (which is generally a Intel based server with a custom modified Linux install for low maintenance and top performance).

At this point it’s apparently not 64-bit and doesn’t have native IPv6 support. However it appears to be usable and likely competitive with some of the other stuff out there already. Yahoo has been using it all along, and I hear they are pretty popular (problems aside).

It should be noted that commercial CDN’s aren’t really an alternative for reverse proxy or load balancer since they still require a robust and redundant origin. If anything they will reduce your requirements, not eliminate them.

Given everyone’s interest in scaling computing quickly and cheaply this is pretty noteworthy open source event. It tends to be an afterthought but these applications can be critical. Squid handles 78% of Wikipedia’s requests. Given all their traffic, you can see how it matters.

It will be interesting to see if a community builds around Traffic Server and if it sees adoption.

Categories
Mozilla Open Source Politics

WhiteHouse.gov Goes Open Source

I noted in January that WhiteHouse.gov relaunched for the Obama administration using a closed source infrastructure (it was using ASP.NET on IIS 6.0) running a proprietary CMS.

It has now relaunched using open source Drupal. Also interesting is that it’s no longer broadcasting any headers regarding it’s server. Considering Drupal is by far better tested on a Unix OS andApache, I’m wondering if they dropped Windows Server/IIS 6.0 in favor of some sort of Linux and Apache. I can’t find any hint at what they are using.

It’s noteworthy that Drupal was already used on recovery.gov and has been used in politics by way of CivicSpace for the Dean campaign in 2004.

Via Drupal it’s still using jQuery (verison 1.2.6). It’s also now using RSS rather than ATOM for feeds, which I presume is by way of the switch to Drupal rather than an intentional effort.

Another interesting change is they tweaked the doctype from XHTML Transitional to XHTML+RDFa.

Pretty much everything else is still the same including the design. Analytics is still done using WebTrends (holdover from the Bush administration) and Akamai still sits in front of their servers.

For CSS hackers: They still choose conditional CSS for IE compatibility.

Their pages don’t fully validate anymore, though there is no terrible markup either.

Video is still done using Flash, maybe they’ll consider adopting HTML5 video. They could do so and fallback to Flash. The latest versions of Firefox, Safari, and Chrome could take advantage of it today. The rest of the browsers would get the Flash experience. That would be the next major step in opening up. Mark Pilgrim has a good primer if they need.

Edit [9/26/2009 @ 1:45 PM EST]: Tim O’Reilly confirms it is indeed running on LAMP, specifically Red Hat Linux with Apache, MySQL and obviously PHP. Apache Solr is used for search.

Categories
Apple Open Source

Apple Kills ZFS Support

I mentioned over a year ago that Apple was porting Sun’s ZFS file system to Mac OS X. While it was available as read-only on Leopard it seems to have been completely pulled from Snow Leopard. For something that was suspected to be the future of disk storage for Mac OS X, that seemed odd. Now Apple has officially discontinued the project.

I had heard about the ongoing NetApp vs. Sun patent war where NetApp feels that ZFS is too close to WAFL. It seems likely that Apple doesn’t want to get involved in that. Apple even has a fear of that potential with OGG Theora. Once the transition was made to ZFS it would be a costly and time-consuming effort to swap with something else since Mac OS has never been very file system neutral.

A new theory is that the Oracle/Sun deal leaves the company developing two filesystems: ZFS and Btfs. It sounds like Oracle’s Btfs is the more likely future. If Apple switched to ZFS they would have been left as the only platform using it. Linux can’t fully switch since the CDDL license isn’t fully compatible with GPL meaning it would need to be implemented through FUSE. Btfs is coming along for Linux.

Reading through Btfs, it seems like a lot of the big advantages of ZFS are already in Btfs though it lacks full disk encryption. It does however add online resizing. It’s also GPL and has support from RedHat, Novell, IBM and was accepted into the Linux mainline kernel as of 2.6.29rc1. That means it already has a much more robust community and seems likely to be widely accepted in UNIX land.

So will Apple switch to Btfs for Mac OS X 10.7 or 10.8? I think the two possibilities are that it will either build something in-house, or switch to Btfs. I think Btfs offers a compelling set of features and would allow Apple to brag about more compatibility with other OS’s and potentially adopt features as the file system matures at a low-cost. It’s possible we’ll hear something as soon as WWDC 2010.

Edit [10/25/2009 @ 6:30 PM EST]: Confirmed via Daring Fireball