Wednesday, January 30, 2008

Kenya: memories and heartbreak

I have been reading the news about Kenya with a growing deep sadness. I spent one year there as a child with my family, on a little island called Rusinga off the shore of Lake Victoria. My Mom is a paleontologist and my Dad is a geologist, and they were doing research for Louis Leakey. Later at a town called Kaptagat while the kids went to school and my Dad worked at a mine.

View Larger Map

It was a wonderful time for me, and I continue to be filled with gratitude for the land and the people. It touched me, and all my family, in a very deep way, something that goes beyond words and description.

As I read about the places of the worst violence: Kisumu, Eldoret, my heart aches. I remember those towns. I remember the people - the fierce but loving Masai, the Luo with their incredibly brilliant smiles and warm hearts, and the Kikuyu. All with such open hearts, living in a beautiful land. Now I read how many are filled with hate, looting and killing, about how 60% of the people live in slums, and I just shake my head.

For some reason I had always thought that some day I could maybe go back, maybe with my kids when they're older, and revisit this place that was such an integral part of my childhood. Now I seriously doubt that if I went, it would be even close to the same. It really is true, you can't go back.

I do hope that the people of Kenya can pull themselves out of this madness and find a way to live and love.

As an ancient prayer from the Rig Veda says:

Let us be united;
Let us speak in harmony;
Let our minds apprehend alike.
Common be our prayer,
Common be the end of our assembly;
Common be our resolution;
Common be our deliberations.
Alike be our feelings;
Unified be our hearts;
Common be our intentions;
Perfect be our unity.

Monday, January 28, 2008

Friday, January 25, 2008

NetBeans plugin for copying text between Java and SQL/XML/HTML/...

You've been there: you want to write a SQL query, test it, and then paste it into Java. But Java requires a string that is concatenated with '+' for readability. What a pain to have to go back and forth between the query editor and the Java editor.

Wade Chandler (our hero again) has written a nice plugin that let's you convert back and forth between "Java-style" string-concatenated text and "plain style" of SQL and other plain text editors. Wade explains this more in this email.

Here's an example. I start with a nice long query in the query editor that does what I want it to do...


Then I use Wade's plugin to paste into Java:


et voila!


Thursday, January 24, 2008

Setting up an SSH tunnel to talk to a remote database

On the nbusers list, Gary Greenberg asked

I have a database (Postgresql) running on the server with the SSL access only.

I.e. to run the query against it I need to login in via PUTTY or other SSL client.

Does anyone know about SSL proxy or whatever that will allow me to create a database connection in the NB6 Services tab?

I gave a pretty lame answer, because I've actually never had to do this, but Wade Chandler had an excellent description of what to do, which I will share here

It sounds like you are saying you have to use SSH. Do you mean the only way to access the server is with SSH? If so then you can use SSH tunnels. I do this often.

You will use which ever SSH client you have. Putty in this case. You then setup a tunnel which in Putty I believe should be under SSH tunnels or tunnels in the configuration section where you setup your connections or connect. You'll have a local port you will later connect to using regular TCP/IP and the server or remote port which the end SSH connection hooks to your local port and makes a tunnel to. Be sure and save your connection once you make your changes so you don't lose them for the next run.

Once you make your SSH connection on your local machine to the SSH server then your DB client server address will actually point to and the local port number you setup in your SSH tunnel. SSH will then link your local port ${whatever_it_is} to the other port on the other end of your SSH connection. So, you actually connect with regular TCP/IP to your local machine on a port which is tunneled over SSH to a TCP/IP port on the other end of your SSH connection. This is why you connect to your local box/machine as the server with the database running on it though it is is tunneled to the real connection and machine.

I have heard about SSH tunneling in the past, and never understood clearly it worked, never having the need. But this description was so clear, I just Got It. Thanks, Wade!


Thanks to Eduardo for pointing me to this great video. Now that's the way to seal a deal!

The phonetic translation from Kaj's blog had me roaring - hell and gore!

Hell and gore
Shung Hop father Allan Allan ley.
Hell and gore
Shung Hop father Allan ley.

Oh handsome in the hell and tar
hand hell are in the half and four.
Hell and gore
Shung Hop father Allan ley

My sister married a Swede a few years ago, and the wedding was full of songs and drinks.

I saw the same thing in Norway when I used to visit the database team there in Trondheim - very polite and reserved (mostly) during the day, but go out for beer with them and the roaring laughter and singing starts. I can almost see the horned hats :)

Skål! May we both learn and grow from each other!

What to do when Google Data API requires a captcha

I posted quite a while ago about a little program I wrote to take my delicious links tagged with "blogthis" and post them to my blog.

This has been working great for over a year, but today I got the following error:$CaptchaRequiredException: Captcha required

I remember reading about this when I was researching the API, but my problem is my app is not a web app and I can't dynamically route someone to a captcha page. After much Googling about Google (hm), I discovered you can unlock your account by going to the Google Unlock Captcha page. Type in what you see, submit, and the program works again. Cool!

Arun Gupta: RESTful representation of "sakila" using GlassFish and NetBeans IDE

Of worthy note, Arun Gupta, our Web 2.0/web services evanglist, has a nice blog showing how to quickly put together a RESTful representation of MySQL's Sakila database...

Wednesday, January 23, 2008

The Moral Instinct - New York Times

A very interesting conclusion, that morality is greater than us, not because of a moral, judgemental God, but because it is a natural consequence of how the world works. I personally see "the nature of things" *as* God - God is Existence itself.

Good Math, Bad Math : Databases are hammers; MapReduce is a screwdriver.

I was a little stunned at the MapReduce criticism from Stonebraker, and I think this is a great response

Monday, January 21, 2008

The Moral Instinct - New York Times

A very interesting conclusion, that morality is greater than us, not because of a moral, judgmental God, but because it is a natural consequence of how the world works. I personally see "the nature of things" as God - God is Existence itself.

AS IS CLAUSE - Deal With It

I love these "AS IS" clauses in software license agreements. I thought I'd check out Amazon Simple DB on my day off (yes, you can tell I'm a geek when this is what I do with my day off).

The limited beta is closed, so I guess I'll have to wait. But I signed up to be notified if a slot opens up in the future, and I saw this in the Amazon Web Services license agreement. Standard boilerplate I'm sure, but it still strikes me how little software companies are willing to warrant. It shows just how unreliable software really is.
Sheesh! The only thing one can hope for is that they really don't want anything bad to happen because of the impact on the perception of their services. Although I don't know how one could claim that Amazon Web Service was not giving "quiet enjoyment." :)

Steve Jobs Macworld 2008 Keynote in 60 Seconds

Thanks to TechCrunch for this link - better and better and better.

Friday, January 18, 2008

Introducing Sun's new database: Elephine

Hey, what's the point of having three databases? Let's combine them all!

Thanks to Petr Zaharadnik by way of Zdenek Kotala for this one. Hee hee.

Cringely agrees with me

I have my theory about why the MacBook Air has no optical drive. It looks like Mr. Cringely agrees with me.
What about the Air's lack of an optical drive? It's hard to find a place for an optical drive in such a thin computer, but isn't Steve Jobs the guy who when he returned to Apple railed against notebooks without removable media, like the PowerBook 100 and 2400 and the various PowerBook Duos? Why did Steve change his mind now? Because Steve wants to replace optical drives of any sort with bits provided over the network, preferably from iTunes. That's also why we didn't see an Apple Blu-ray announcement this week and -- if Jobs has his way -- we'll never see one.

NetBeans and MySQL: What To Do?

I have been getting a number of internal and external emails and phone calls offering ideas and asking me for opinions for what to do in NetBeans now that Sun has a new Very Popular Database.

In case you don't know, I am currently the person responsible for database tooling in NetBeans. Until yesterday, this was going along fairly smoothly, but things just got a little hot around here :).

First of all, I suspect that some folks over in MySQL may have some thoughts and opinions about this :), and so far I haven't had a chance to talk to them. So that's a very important missing piece of the puzzle. I need to figure out when and how I can do that.

But meanwhile, let me share my thoughts and idea around this.

First, I do want to say that we already have pretty good integration with MySQL. The MySQL JDBC driver is included with NetBeans 6.0, and it's pretty darn easy to set up a connection and start browsing your MySQL database


NetBeans is excellent for database programming because the database tooling is integrated with a fully fledged IDE. That means you can do things like generate JPA entities from database tables, generate a desktop CRUD application, generate a RESTful Web Service from a database, and visually bind web page components to database tables.

OK, but what more can we do :).

Well, lots of things. All sorts of things. But what makes sense? That's going to take some thought.

Just as an example, NetBeans is a development environment. Most of the folks I talk to don't believe it should be a database administration tool, because administration is so specific to each database. Sure, we will own and support MySQL, but Sun also distributes and supports PostgreSQL and Java DB, and there are a few other important databases out there that many of our customers use. Do we build admin tools for all of these as well?

That said, I can imagine making it very easy to bring up the MySQL admin tool from NetBeans (and let's not forget the wildly popular open source MySQL admin tool, phpmyadmin. For that matter, we should make it easy to bring up pgAdmin for PostgreSQL. Java DB, bless it's heart, really doesn't need an admin tool, except for start and stop, and we already do that (OK, we could add support for setting properties).

Now that I think of it, probably the best thing would be to let the user register the admin tool they want to use for a given database driver/connection.

There are lots of other things we could do. What about stored procedure support, including debugging? What about integrating MySQL's data modeling functionality into NetBeans? What about making it easy to create a sample database? And so on.

I've set up a Wiki page to gather them all. There is a lot of creative energy around this. Tell us your ideas. Put them on the Wiki page, add a comment to this blog, add a feature request in Issuezilla, or send me an email (david at vancouvering dot com). Let's get this ball rolling!

Say it ain't so: Ruby on Rails not thread-safe, no prepared statements?

Maybe everyone already knows this. But I've been hearing about Ruby on Rails for over a year now, and I know people working on it, and I am reading all this stuff about putting Ruby on Rails on Glassfish and getting it into the enterprise. I did hear some mumblings about scaling issues with Ruby on Rails. But I didn't know it was as basic as Ruby on Rails not being thread-safe.

Hm, maybe this guy got it wrong. But no, here is another post. And another.

The author of the last post, Greg Luck, says something else that just blows my mind as a database programmer - Ruby on Rails doesn't support prepared statements. Really? So I checked for other sources. Yep, it's true.

Somebody in Greg's comments said something about how Greg obviously didn't know about ActiveRecord caching. Um, that's not really the point. The point is, when you do hit the database, why on earth would you force it to recompile and re-optimize the SQL each time?

I have worked with multi-threaded applications since 1988. Multi-threading is not new. Pre-compiled SQL showed its power ofter fifteen years ago, and every database vendor worth its salt supports it. How can it be possible that an entirely new framework for server-side database applications is not thread-safe and does not support prepared statements? It's one of those things that makes one's jaw drop.

Maybe I'm just completely missing something. Maybe I'm living in the stone age. Someone explain to me where I've gone wrong. I strive to be open-minded and anyone who knows me knows I regularly change my mind if I am convinced I have it wrong.

But as it stands, if someone were to suggest to me that we should use Ruby On Rails for a web-scale server-side application, I would sit him or her down for coffee (or a strong drink), explain the limitations above, and beg them to think about this very seriously, and at the minimum, please, run some tests before committing to this framework.

Optimizing Java DB in unit test environments

A number of folks use Java DB/Apache Derby for their unit testing. In this environment, your requirements are different:

  • You don't care about durability. If some data is lost, fine, run the test again
  • You do care about getting through your tests as quickly as possible
  • You want to regularly start with a fresh database, and you don't want "crud" left around at the end of the unit test. In particular, you don't want your test database left around.
Because you don't care about durability, you don't necessarily have to write to disk. You may not be aware of this, but some directories on your system may not be mounted to a physical disk, but instead are mapped to memory. In particular, /tmp is normally mapped to memory.

So if you set your database to point to /tmp, you should get some significant performance improvements, e.g. jdbc:derby:/tmp/mydb.

I'm not sure how you would do this on Windows. Here is one possible tip - note I haven't tried this myself. Just the first hit on Google.

Another thing you can do to speed things up is to configure Java DB to not bother synching to disk. You do this by setting derby.system.durability=test as either a system property when you start your VM, or in the file. Specifically, when you set this property,
the store system will not force I/O synchronization calls for:
  • the log file at each commit.
  • the log file before a data page is forced to disk.
  • page allocation when a file is grown.
  • for data writes during checkpoints.
You can see this documentation page for more information.

Now, what about removing that pesky database directory when you're done with a test? Java DB does not provide any mechanism for doing that, so you're going to have to do this yourself.

There are two approaches to doing this. First of all, you could drop all the tables after each test run and re-create them, re-using the same database each time.

Alternately, you can have a "model" database set up, and after each test run you use Java file APIs to blow away the old database directory [1] and replace it with a fresh "model".

Note: before you try to remove the database, you should probably shut it down. Do this by connecting to it with the ;shutdown=true" property added to your URL.

The trick is, your code needs to know where this database directory lives. Here are some possible approaches:
  • If you are fully qualifying paths to your databases, you know exactly where they live. But usually you wouldn't do this, because it makes your tests non-portable.
  • You can let test runners specify a property that indicates where the database directories should go. Then you prepend this to the JDBC URL, and also use it when you're blowing away the directory. For example:

public class MyTest {

private Connection conn;
private String dbdir;
private String dbname;

public void setUp() {
dbdir = getTestProperty("database.dir", "/tmp");
dbname = getTestProperty("", "mydb");
copyModel(dbdir, dbname);
String url = "jdbc:derby:" + dbdir + "/" + dbname;
conn = connectToDb(url);

public void tearDown() {
conn = connectToDb(url = ";shutdown = true");
deleteDatabase(dbdir, dbname);
  • You can use some knowledge about how Java DB determines where to place a database. It follows the basic simple rule: if derby.system.home is set, place the database directory there. If that's not set, use the Java system property user.dir. So you can follow the same logic to figure out where your database is and then blow it away (and put a new one there in its place).

[1] You can write the code yourself to blow away a directory, or use the handy FileUtils.deleteDirectory() from Apache Commons IO.

Wednesday, January 16, 2008

MySQL and Sun - my thoughts

OK, I'm a database guy, I work at Sun, I have a blog. I guess I need to say something here about the MySQL acquisition.

Well, OK - Wow. I really got caught off guard by this one. I wasn't paying attention, I guess. I didn't think MySQL was in the market, and I didn't know Sun was looking. I work a lot with the folks who must have been involved in the due diligence with this deal, but not a word. A very well kept secret.

That said, here are my thoughts.

Let The Community In

You have heard me criticize an open source product that is controlled solely by a single company. "This is my opinion and does not necessarily reflect the position of the company I work for", but I hope that, as part of Sun, that changes for MySQL.

I would like to see us allowing external contributors become committers to MySQL. I think it behooves us to make it clear that MySQL is ultimately owned by the global community, not by Sun. But that's just my manifesto, I'll leave it to the execs to make the Big Important Decisions around this.

Make It Free

I also think it's time to throw out the dual-licensing model, and, just like our other software, make it free to deploy and embed, and charge at the point of value: support and services.

What About Sun's Investment in PostgreSQL?

Well, as far as I can tell, it's pretty much "stay on course" for this. Josh Berkus still has a job. I think PostgreSQL has features and a community that is focused more on the IT data center and data warehousing environments. PostgreSQL and Sun's hardware are a very good pair.

MySQL has an affinity for the Internet, web-facing, fast-changing, fast-paced application environments.

Yes, it's a little fuzzy, and I'm sure we'll get questions about what to use when, but I personally see no reason to throw PostgreSQL out the window just because we acquired MySQL.

What About the Database Team?

Hey, why have a blog if I can't do wild speculation?

I can see Marten Mickos reporting to Rich Green, and aligning all database technology under him. I just hope he can let go of (or ignore) any MySQL biases he may have and make sure he serves all of our database technology products/projects (Java DB, PostgreSQL, MySQL) equally.

I can see some of the MySQL database tools staying under the database group (the migration and admin tools) and others migrating over to NetBeans over time (the Query Designer and MySQL Workbench). And I think you'll see all of these tools becoming free - charging for tools has not historically been Sun's strategy. But I could be wrong...

Check back with me in a year and we'll see how wildly off I was with these predictions.

How Does It Affect Me?

I'm not sure yet. I can imagine me flying to Germany to meet the MySQL tools team (stopping by in London to see my new niece) and discussing strategy and future plans. I can see the database team under NetBeans getting a bit bigger (right now, in case you didn't know this, it's just me). I can see me downloading and playing with the MySQL tools (thank goodness for VMWare Fusion, otherwise I'd need to borrow a Windows box to run MySQL Workbench).

I don't know, what do you think? What should we be doing with NetBeans and tooling in light of this acquisition? Any ideas?

One final thought: now that I know MySQL was on the market, I'm very glad Oracle didn't buy them. That would have had a sobering effect on the open source community. I guess they were busy with other deals. Thanks, BEA! :)

Tuesday, January 15, 2008

No optical drive? Gee, I guess I have to rent my movie through iTunes

Just read the details that the new MacBook Air has no optical drive. The rationale is that it makes the machine lighter and have longer battery life.

Yeah, sure.

You know what I think? If you have an optical drive, you'll buy or rent DVDs when you want to watch a movie on your laptop. If you don't have an optical drive, what do you do? You rent them through iTunes. Kaching!

This explains why the MBA is so reasonably priced. It's a loss leader to Apple's real business - entertainment media distribution. You don't have to be an MBA to figure that one out... :)

Oooh, I want it

This just in - a new Mac laptop, and I quote:
A lot of others only get 1.5 hours -- this is phenomenal battery life. 3 pounds, 13.3-inch full size display, full keyboard, multi-touch gestures, 0.16-inch thickness, iSight... 1.6GHz C2D, 2GB RAM standard, 80GB drive, 64GB SSD option, 802.11n standard Bluetooth 2.1 + EDR...

All these and more in the thinnest notebook in the world -- we are pricing this at $1799.
Did you read that about a 64GB Solid State Drive? Do you know how fast and low-power those things are?

My shiny MBP Core-Duo seems so, oh, old-fashioned now. Good job in creating tech greed. Now I just need to show my work or my wife why this is absolutely essential for me to do my job. Maybe I can pull out the eco-card with their mercury/arsenic-free screen and eco-friendly electronic parts... Yeah, that's it...

Monday, January 14, 2008

You just don't get it

There are times when you realize other people just don't see things the way you do. They just don't get it. For example...

Geek Jokes

After I showed it to her online, with a big smile on my face, my wife bought me a T-shirt for Christmas that says
There are 10 types of people in the world. Those who understand binary and those who don't.
I love this T-shirt. Either you get it, or you don't. In particular, you probably don't get why anyone would think it was funny.

Another favorite of mine is the T-Shirt that says <BODY> on the front and </BODY> on the back. Priceless. Snork snork.

Fixing My Own Computer

I recently bought a newer (larger, faster) hard drive for my MacBook Pro. iFixIt has very nice instructions on how to upgrade your hard drive here. Note all the tiny little screws. I was surprised by how many tiny little screws there were. I also had to go to Radio Shack to get a T6 Torx screwdriver. I actually got a cool little box with a screwdriver and exchangable tiny bits for electronic repair.

At dinner (my laptop in pieces, not done yet) I mentioned to my wife how surprised I was at all the little screws, how Apple had made it so complicated compared to other laptops I had worked with, and how it was taking me longer than I thought. She said "why didn't you just take it to a repair person?" I looked at her, stunned. I was hurt. How could she say such a thing? She just didn't get it.

It was as if she was struggling late at night over a pie for a pot-luck the next day, and I were to say "honey, why don't you save yourself some time and just buy a pie from the local bakery?" Sheesh...

Smacking My Son

Recently I was hanging outside our house with my 2-year-old son and our neighbors, a Mom and her daughter. At some point my son ran SLAM straight into me and we both laughed. The Mom and daughter both sucked in their breath.

I said, "you know, I don't think this is something women usually get. Boys love this sort of thing." Then I gave him a good smack on his forehead with my palm, kind of like Homer doing 'doh!' or the V-8 commercial. They both exclaimed in alarm. He roared with laughter. After some time the Mom said "no, we don't get it" and shook her head.

Ah, well...

Derby tips and tricks Wiki page

The Apache Derby Wiki has a very useful page collecting hints and tips for using Apache Derby -- worth keeping in your bookmarks if you run into issues or have questions.

The Derby Wiki has a lot of useful information, including information on upcoming releases and pages on how to contribute to Derby if you are so inclined.

How to initialize an embedded Java DB database using JPA/TopLink Essentials

The question of how to initialize an embedded Java DB database when you are using JPA/TopLink Essentials came up recently on the NetBeans users list.

I answered the question there and thought I'd answer it here as well... These steps are applicable even when you're not using TopLink, with the exception of how you configure your persistence unit to auto-create tables for you.

There are three parts to initializating an embedded database. Note that this is for a fresh install of the application. If you are upgrading an application, that is a whole other kettle of fish that I will answer another time...
  • Create the database
  • Create tables
  • Load initial data set
I'll cover each of these separately. See also the last section, where I talk about using a model database, which handles all of these steps in one fell swoop.

Create the Database

If you change your persistence.xml so that the URL includes the parameter ";create=true" then Java DB will automatically create the database for you

Create the Tables

You can have TopLink to this for you by setting a property for the
persistence unit in your persistence.xml file:

<property name="toplink.ddl-generation" value="create-tables">

See the Oracle TopLink extensions documentation for more information

Load the Initial Data Set

There are a number of ways you can do this:
  • You can write some code that creates new JPA objects, initializes them, and commits them using JPA
  • You can write some code that uses JDBC to load your data set
  • Use a SQL script and the Derby tool 'ij' to load the SQL. This requires including derbytools.jar in your application. See this blog entry for a reference to one user who was successful doing it this way.

Three in One Blow: Using a Model Database

Another way to create the database, tables and data is to use a model database. You create your database as you want it, and then include the database directory in your application jar file. Derby databases are completely portable across any platform, so you don't need to worry about having different versions for, say, different byte-ordering architectures or for 32-bit/64-bit architectures.

Then at startup if the database directory is not there, you can extract it from your application jar and you're good to go.

Note this is what NetBeans does. When you register Java DB, or if Java DB is detected when you register Glassfish or you use JDK 6, we automatically extract a zip of a sample database that is included in NetBeans and create a connection to it in the Services tab. It works quite nicely.

Tuesday, January 08, 2008

Fake Steve: Breakfast with an Apple lawyer

OMG, this is hilarious, tears in the eyes funny, and scary at the same time. He nailed it with Gandhi. Corporate America has no ethics, black heart to the core. I suspect Apple is Very Very Angry now (cue to Looney Tunes Martian).

Monday, January 07, 2008

Ruby support in NetBeans continues to gain fans

Just saw this in the Ruby mailing list for NetBeans:
I love the Ruby and Ruby on Rails support in NetBeans! I believe it's
the best Ruby IDE out there. Please keep up the good work. I've
abandoned RadRails for NetBeans.
Want to know more? See and

Give me a good hash function, and I can move mountains

Knut Anders Hatlen is one of the resident geniuses over in Java DB land. He just recently started a blog, and this entry is a great read, instructing us in performance optimization, hashing, and one of the hidden gems in NetBeans, a simple Generate Code menu item that generates a good hash code.

As I mentioned to the Java DB team on email, it's these little things that make all the difference. Sure, the various whizbang doodads are nice, but give me a good editor that improves my productivity and quality while writing code, and I'll gladly raise a pint in your name and kiss your dog in gratitude.