Tuesday, March 30, 2010

WTF? - SQLite files as a network messaging format

A colleague told me about an idea his team is considering that at first had me going ewww!

They are thinking of replacing the use of Google protocol buffers with a SQLite file for their network communication format.

Huh?  It didn't make sense at first.  But then I chewed on it some more.

Their protocol is primarily about sending one or more queries, and getting back potentially quite large tabular data sets as a result.  What better format than a compressed relational data format?

The SQLite database format is in the public domain.  It is self-describing.  It is compressed, saving network overhead.  The format is portable across platforms.  It is tabular in nature.  You can use the SQLite APIs to to inspect, slice and dice the results, both for processing as well as for troubleshooting/debugging.  You can also use the SQLite API (e.g. SQL INSERT) to incrementally build up a request or response before sending it over the network.

SQLite is embedded, so you don't have to deal with connecting to the server and yada yada, and it's quite cheap to create new databases (each request and response is a new SQLite database).

The more I think about it, the more it seems like a very compelling approach...  It goes against everything I've learned about network protocols, but for their use case, it actually makes a lot of sense.

1 comment:

Ashwin Jayaprakash said...

The Sqlite author Richard Hipp said in a talk recently that some customers - Adobe being the biggest use Sqlite for its platform independent file format - http://javaforu.blogspot.com/2010/01/sqlite-talk-select-from-sqliteinternals.html

So, I'm not surprised that there are others using it in more or less similar ways.