Tag Archives: nytimes

DBSlayer + Node.js

Posted on by
Reply

Lately it seems the rage among developers is to take node.js and combine it with something else unusual. So here’s my contribution.

DBSlayer is a project by NYTimes a few years ago that seems to be somewhat forgotten but is pretty cool. It’s another MySQL proxy, but with a slight twist. Rather than use a binary protocol, or XML, they went with JSON. It supports things like connection pooling, round-robin distribution to slaves, automatic fallover, and it’s mutithreaded. It’s pretty fast and easy to work with. It’s almost like turning a MySQL database into a REST API. You pass a SQL query as a query argument and it gives you a JSON response.

Once you start it you can do something query using a request like:

http://localhost:9090/db?%7B%22SQL%22%3A%22SELECT%20*%20FROM%20facts%20WHERE%20id%3D1%3B%22%7D%20

That will give you a JSON object containing the result of your query.

So doing that in node.js is roughly:

var sql = ‘{"SQL":"SELECT * FROM facts WHERE id=7;"}’;
 
var http = require(‘http’);
var client = http.createClient(9090, ‘localhost’);
 
var request = client.request(‘GET’, ‘/db?’ + escape(sql), {});
request.end();
request.on(‘response’, function (response) {
  console.log(‘STATUS: ‘ + response.statusCode);
  console.log(‘HEADERS: ‘ + JSON.stringify(response.headers));
  response.setEncoding(‘utf8’);
  response.on(‘data’, function (chunk) {
    console.log(‘BODY: ‘ + chunk);
  });
});

Running that looks like this:

$ node test.js
STATUS: 200
HEADERS: {"date":"Fri, 27 May 2011 02:02:27 GMT","server":"dbslayer/beta-12","content-type":"text/plain; charset=utf-8","content-length":"290","connection":"close"}
BODY: {"RESULT" : {"HEADER" : ["id" , "fact" , "author" , "ip" , "timestamp"] , "ROWS" : [[7 , "1+1=2" , "raccettura" , "127.0.0.1" , 123456]] , "TYPES" : ["MYSQL_TYPE_LONG" , "MYSQL_TYPE_VAR_STRING" , "MYSQL_TYPE_VAR_STRING" , "MYSQL_TYPE_VAR_STRING" , "MYSQL_TYPE_LONGLONG"]} , "SERVER" : "db"}

You could obviously clean that up and create a little library to hide the HTTP parts.

It’s an interesting JS centric way to abstract your database while maintaining SQL level control. JSON is becoming the new XML.

Common Application Bugs

Posted on by
Reply

There’s a curious article in the NY Times about the Common Application‘s technical glitches. The Common Application is a uniform way of filling out one application to apply to many colleges, as opposed to filling out an application for each individual college.

As a web developer, this struck me as particularly odd:

As it turns out, applicants do not have, say, 150 words to discuss their most meaningful extracurricular activities; they have something closer to 1,000 characters (Max said he eventually figured this out). And because some letters may take up more space than others, one applicant’s 145-word essay may be too long, while another’s 157-word response may come up short, Mr. Killion said.

“A capital W takes up 10 times the space of a period,” he said. “If a student writes 163 characters that include lots of Ws and m’s and g’s and capital letters, their 163 characters are going to take many more inches of space than someone who uses lots of I’s and commas and periods and spaces.”

Asked why the problem had not been fixed, Mr. Killion said, “Believe me, if there’s a way to do it, we’d do it. Maybe there’s a way out there we don’t know about.”

Sounds like the folks behind the common application need to go back to middle school and learn about variable width and fixed width fonts. If they had switched to fixed width fonts in the <textarea/> and used the same number of cols and font size it should be pretty accurate. I’m guessing some designer insisted on Helvetica or whatever they are using.

That said, are they actually printing these things out? Is there no way to do this electronically in 2010?

Back in my day (2002), I was advised to go the paper route since many still felt/feared that electronic applications weren’t being fairly considered (and in some cases not processed correctly). That was also the first year public colleges could join the Common Application as I recall. I suspect I was the last class where the majority did it on paper. I guess it’s still an improvement for a technophobic educational system.

Google Releases Protocol Buffers

Posted on by
1

Google today released Protocol Buffers. Protocol Buffers is their “language-neutral, platform-neutral, extensible mechanism for serializing structured data”. In general it’s pretty interesting stuff, and looking over the docs, seems pretty well thought out.

I agree XML is bulky and wasteful for the task. There’s a reason why many web developers prefer JSON rather than actual XML when using xmlHttpRequest: XML parsing can be a real performance killer. JSON in my mind is currently the winner in this department since it’s light weight, simple, and a can be interpreted by pretty much any language on the planet (may need to install a module, gem, extension, or include a class). The downside to JSON is that it doesn’t really allow you to define structure. JSON also is still not binary format, so you have a performance penalty to parse the string. The upside is that JSON is rather easy for humans to read (great for debugging). The NY Times even made a database abstraction layer called DBSlayer that interfaces using JSON.

Serialized PHP has become somewhat popular (Yahoo Developer Network API’s support it), but it’s language specific, though interpreters that can read/write it exist for other languages including Perl, Python and Java. It’s also somewhat complicated for what it provides. At a glance it’s a string of garbage until you break it down.

It looks like Google already has support for Java, Python, and C++. It’s only a matter of time before Perl, PHP, and Ruby get support for Protocol Buffers as well.

I could see Protocol Buffers being pretty useful in combination with Memcached.

It’s great to see Google open sourcing stuff like this.