Dooth Booty at MySQL Conference next week

Off to the MySQL conference next week. I still have to remind myself that MySQL is now part of Sun - it's so easy to think of "them" as a separate company that we're just partnering with. It really is something that takes a while to get your head around.

I'll be doing a ton of dooth booty, um booth duty, on Tuesday and Wednesday. Come to the NetBeans booth in the morning and I'll be there, and maybe I can get you a T-Shirt. XXXL of course :)

We'll be showing off what you can do with NetBeans and MySQL, and I also would love to talk to folks about what you want for database tooling in an IDE (and what you don't want).

I'm looking forward to hearing from Mårten Mickos and Jonathan Schwårtz. It should be very interesting to see where these guys see MySQL going next. And then there is Werner Vogels, my "hero" of eventual consistency. Should be good.

I also hope to meet a number of my colleagues from MySQL as well as dear old friends from the Java DB, PostgreSQL and HADB teams (which are now part of the New Database Team).

It should be a time of good information, good friends, and good cheer.

Who is the Dick on My Site? - Dick Hardt

I am just noting one of the best talk titles I have heard in a while. This one is by Dick Hardt on web identity. I'm going just to hear this person who has the cojones and sense of humor to publish such a title for a keynote.

http://en.oreilly.com/mysql2008/public/schedule/detail/2460

Characterization Tests

I love this very realistic and somewhat funny article on how to test stuff where you have no clear idea exactly how a piece of code should behave, but regardless you do want to assure that if you make changes it keeps behaving the way it used to.

http://www.artima.com/weblogs/viewpost.jsp?thread=198296

Do Messages Want to Be Asynchronous? - Beautiful Code

A nice article which comes to the conclusion that asynchronous messaging is the logical end result of applying "Tell, Don't Ask" methodology to your systems.

http://beautifulcode.oreillynet.com/2007/10/do_messages_want_to_be_asynchr.php

Tiny useful iPhone headset feature

I got a new iPhone recently, and I had been trying to find a good headset that could double as headphones, handle switching between calls and music, had good sound, was easy to put in my pocket, etc.

I had bought one version, returned it dissatisfied, and was trying another one before I left the Apple Store. The sound was horrible, and I complained to a "Concierge". He said "well, why don't you just use the microphone on your headset that came with the phone?" Huh? What button?

Well, if you look closely at the picture, you'll see it on the right. A tiny little microphone. And if you squeeze it between your fingers, it will answer/hang up the phone or pause your music.




I just tried it out and it works perfectly. I am duly impressed.

Microsoft Damage Control

I am downloading the MS SQL Server JDBC driver for some testing I need to do, and saw this in their license agreement:

LIMITATION ON AND EXCLUSION OF REMEDIES AND DAMAGES. You can recover from Microsoft and its suppliers only direct damages up to U.S. $5.00. You cannot recover any other damages, including consequential, lost profits, special, indirect or incidental damages.

That is so generous of them - they are willing to pay your $5.00 if their software causes some sort of damage. For instance, the JDBC driver has a bug that causes data corruption in my central IT database, and I can get $5.00 from Microsoft to cover the damages.

Generous to a fault, those guys are. What will the stockholders have to say about that?

Broadway Photo - Stay Away!

I was this close to ordering our new video camera from Broadway Photo after seeing their amazing prices at half the going rate. Just before I clicked the purchase button, I thought "hm, at those prices, I bet there are no returns" and decided to research their return policy.

I ended up Googling for it, and to my horror got a list of folks saying this shop was a complete scam. I was astounded. They looked totally legit, a very nicely put together web site, but these folks are reputed for calling you to buy very expensive accessories, never shipping the produce, or shipping you used versions of it. I couldn't believe how close I was to losing a very big chunk of cash to these guys.

I'm amazed they're still in business, but I thought I'd do my part to expose them...

E Pluribus Unum

I read Barack Obama's talk yesterday, and it really struck me. I know everybody is talking about how it has to do with race, but his real theme, if you look more closely, is about forming "a more perfect union" - the vision of the Founding Fathers that this country is on a path, a path of union.

I recently read a biography of Abraham Lincoln, and he also studied very carefully the Constitution and tried to interpret the intentions of the founders of our nation. He saw that these founders recognized that the nation, the union, was not perfect, and that although they believed slavery was anathema to their vision of the country, it had to be allowed to exist until a future time.

It reminds me of the story I heard of a cathedral in Europe where they designed it to hold a massive dome on the top, when nobody knew how to create such a dome. But they knew it would take 200 years to build the cathedral, and they had this astounding faith that in that time people would figure out how to build the dome. And that is actually what happened.

I was thinking about this theme of "a more perfect union," and it struck me what the motto is on all of our money - E Pluribus Unum - Out of the Many, One. The Founding Father's vision was not just about uniting a number of separate states into one country, but it was about uniting people - different people from different countries, religions and philosophies into one people.

I think this is what is inspiring people about the vision Mr. Obama is holding for us - he continues to emphasize what is common among us all, about what is good in this country. And he says, unified, we can meet and address our problems. Divided, we fall. As Abraham Lincoln said, "A house divided cannot stand."

Unity is such a common theme in religions, and so it should be. The Indian scriptures say that the root cause of fear is the feeling of being separate, being different. It is the sense of difference, of separateness, that pits man against man, tribe against tribe, husband against wife. But in finding what is common, seeing that we are all the same, promotes love and brings strength to all.

Mr. Obama said something else that struck me: that focusing on these differences of race, of color, and so on, are distractions. And I realized that distraction is key to those who would not have us look at our real problems and solve them together. There are those who benefit from the status quo. And the best thing to do is to keep us distracted, focusing on things like gun control, death penalty, issues of race and sex, immigration, and all these things that keep us separate from and afraid of each other.

I think the Internet has its problems socially, but it has done one thing. It has enabled the individual to participate in democracy like never before. Obama's campaign is it is not financed by the traditional money interests, by lobbyists and large donors. It is financed by you and me, by one small donation after another. So Obama is not beholden to these entrenched interests. He would not have made it, with the platform he is putting forth, if he were getting his money from traditional sources.

It also strikes me that because of the Internet and the read/write web, people can watch the full video of his speech on YouTube, and are not spoon-fed it by the main media. So people can listen to it all, take it in, think, and let it affect their perception of what this nation is about.

And that gives me hope - perhaps there is a way we can bring our democracy - of the people, by the people, and for the people - back to the people, to a place where our dialog is about unity and solving our problems together, not tribal bickering. Audacious hope, yes, but there it is :)

I offered this prayer from the ancient Rg Veda for Kenya and now I offer it for our United States, and for the world:

Let us be united;
Let us speak in harmony;
Let our minds apprehend alike.
Common be our prayer,
Common be the end of our assembly;
Common be our resolution;
Common be our deliberations.
Alike be our feelings;
Unified be our hearts;
Common be our intentions;
Perfect be our unity.

Jeremiah Wright's Speech "Audacity to Hope"

Another perspective of the polarizing figure of Reverend Wright...

http://andrewsullivan.theatlantic.com/the_daily_dish/2008/03/for-the-record.html

MySQL support in NetBeans 6.1

I was going to blog about this, but James beat me to it.

Thanks, James! I think he covers it really well.

I just wanted to comment on why we did this. It's nothing that revolutionary, but if you have MySQL installed on your machine, then these features can help you to get started very quickly. In particular, you can get a connection to MySQL very quickly and start exploring, and you can easily create the sample databases used by tutorials and blogs throughout NetBeans.

Other things of note that didn't make it into the beta but will be there by the time NetBeans 6.1 releases:

• I fixed a bug so that you can again create stored procedures and triggers in the SQL editor. This broke a when we started executing each statement separately. No, if you say 'delimiter X' where X is any string besides ';', then the semi-colons inside the procedure/trigger definition won't confuse our parser. This is the same approach used by the mysql command-line tool

• I'll be putting up a module in NetBeans update center that will make it very easy to create the Sakila sample database in NetBeans

What about the next release of NetBeans? Well, that depends on a lot of things, including what results from my discussions with the MySQL team, what I hear from you, and how many people are available to contribute. Two things I'm thinking about are improving our SQL editing support (including some support embedded within the PHP and Java code editors) and, thinking big here, some form of database schema refactoring.

Addiction and Wall Street

As I read about how the money-and-power addicts of Wall Street (fueled of course by investor's fear and greed), and read about how the government, and thus our tax money, has to come in and save the day, particular for a slimeball company like Bear Stearns, it is very frustrating. And it reminds me of what I know about addiction.

The pattern of addiction is complex, but one aspect of it goes like this. The addict splurges, indulging in whatever it is he or she indulges in. It feels great. It feels like nothing can go wrong. And then at some point the bottom falls out. The addict gets caught in a scandal, or loses his job, and/or the spouse or partner says they've had enough and threaten to leave.

The addict begs, pleads, says it will never happen again. They promise to reform themselves. And the enabling partner decides to give it another shot, and stands firm with the addict.

One would hope that the inflictors of this disaster upon our economy pay the price, or at least don't walk away with bills in suitcases, and that financial regulation is put in place to prevent this sort of thing in the future.

And we'll hope the addict has reformed themselves and won't do it again. I mean they promised, right?

Say Doh! and get a Pez

IMG_0024.jpg, originally uploaded by David Mezzo Van Couvering.

Central Perk, the coffee shop I sometimes frequent for wireless work, has a vast sea of collectible toys, including this shelf of Simpson's Pez Dispensers...

Of note: Jumpchart -- HTML design using Wiki syntax

I was pointed to a very interesting site called Jumpchart, that lets you design web pages using Wiki-like syntax, and then export the result as a set of HTML pages and CSS. A very nice way to quickly pull together a wireframe of your app and then use that to get started. Take a look...

TechCrunch: Yahoo Embraces Hadoop

This is very encouraging, given how important search is and what a proprietary stranglehold Google has on the underlying technology.

http://www.techcrunch.com/2008/02/20/yahoo-search-wants-to-be-more-like-google-embraces-hadoop/

Sittin’ on the CouchDB « ZefHemel.com

A very nice blog describing one person's experience of working with CouchDB. I like it because he compares this with using relational databases and helps a RDB user "migrate" his concepts to CouchDB

http://www.zefhemel.com/archives/2008/03/11/sittin-on-the-couchdb

Threads in a process and servers in a cluster: same issues, different context

I was just re-reading my last two posts, and it triggered an epiphany: language architectures like Scala and Erlang and server architectures like the H-Store, Gigaspaces, memcached (and others) share a very common model: each unit runs independently and collaboration between units is accomplished through asynchronous messaging. This messaging can be thought of as sharing data (e.g. asynchronous replication), or as sending messages that contain data, but at a certain level the result is the same.

Developers struggling with threading, consistency and locking issues in a single process are actually dealing with the same issues that system architects are struggling with when trying to figure out how to scale out a system where data needs to be shared across processing units. Concurrency, consistency and performance - same issues, different context.

Just thought I'd share that little lightbulb moment.

Stonebraker's H-Store: There's something happenin' here

It was fascinating for me to read this overview by Curt Monash of Michael Stonebraker's new database architecture for the web called the H-Store. It is a complete overturning of the traditional database architecture. And I quote:

On the system side, some of their most radical suggestions include:

• No disks or other persistent storage at all.
• No multi-threading.
• No locks
• No redo logs (and perhaps not a lot of undo logs either).

Wow. And here's something else: the engine has a single API: execute transaction(parameters). In other words, all it does is stored procedures. And the language they are proposing for these stored procedures? Ruby.

We're not talking small changes here. What is motivating these kinds of changes? The web. In particular, traditional databases and database APIs like JDBC and ODBC are tuned for a very old style of application, where the user is working over a set of data on a single connection which is holding some level of lock on the database. In Sybase we used to call this "browse mode," and then this got standardized in SQL with the concept of cursors. I remember we had to do a lot of work in the engine and the client API to support browse mode/cursors, because our customers were demanding it.

But now in many cases, this style of interaction with a database is completely irrelevant. Cursors just do not make sense for a web-based application, where the user comes and goes and the last thing you want to do is hold on to a connection across requests. Instead, you have to use loose transactional models like optimistic concurrency to handle conflicts. And when you let go of locks and long-running transactions, you can start letting go of a lot of other aspects of the traditional database.

And then there is web scale. Standard client/server apps needed to scale, but not at the level of consumer-based web applications. A traditional database's transactional, single machine, disk-based model just doesn't cut it. I was at a PHP conference last year, and was amazed to see that the "standard" way of scaling a web application that talks to the database is to do your own partitioning and run multiple instances of a database. When an application has to handle all of the details of managing a self-partitioned database, we know things have gone astray.

The list above is a great summary, but I recommend you spend the time to read the 10-page paper (PDF). It is quite an interesting read. It's not every day you read quotes like There is no redo log, no concurrency control, and no distributed commit processing and H-Store ran 70,416 TPC-C transactions per second. In contrast, we could only coax 850 transactions per
second from the commercial system, in spite of several days of tuning by a professional DBA.

All of this leads me to some deeper (or more strategic) thoughts... The relational database is not going anywhere. But if those of us in the database space are not careful, we could very well be caught off guard by a new wave of database technologies that do the job for a lot of people but have nothing to do with traditional relational databases.

How does that Buffalo Springfield song go - stop, hey, what's that sound, everybody look what's goin' down...

On threading

I've been quiet for a while because I've been heads down working on some new features for NetBeans 6.1 (and now, debugging those features).

One class of bugs has been knocking me down a number of times: there is a chunk of code that is not thread-safe in a very "interesting" way. I am not historically a GUI programmer, and I have been exposed to the amazing intricacies of threading in a GUI application like NetBeans.

I won't go into details (AWT thread cough cough), but suffice it to say that I have had to put on my multi-threaded hat in deeper ways than I have had to for a while.

In the background I've also been reading about Scala, Erlang, threading and scalability. I will ask my readers to please help me, because I can't find the original references, but some comments I have been percolating in the back of my brain as I wade through my bugfixes.

First of all, someone posted that any language that required users to manage their own concurrency was a broken language. That got me thinking - hey, it is pretty crazy that programmers have to deal with this. With Java we got rid of most memory management issues (not all, think weak references), and we have a wonderful exception model, and very clear typing semantics (a reference can't suddenly become an int, for example, like it can in C). All in all it made a safer sandbox for programmers. But then here is this threading and locking complexity that has engendered books upon books, incredibly popular JavaOne talks and so on.

Then I keep hearing about Erlang and Scala, and in particular how they don't do threading, they do message passing, and there is absolutely no shared data between process units. Ah, a light came on in my head. If you follow this model, you avoid ending up in the ditch (or stuck in gridlock).

I remember working on an incredible product at Sybase called Open Server, ahead of its time in many ways, the invention of Tom Haggin, number three employee at Sybase, and someone I was honored to work with.

Open Server was a server that understood the Sybase client/server protocol called TDS, with an open API that let you plug in event handlers to process client requests and send results.

Since it was a server, it had to be multi-threaded. Its recommended mechanism for inter-thread communication was to avoid shared data and use in-memory message queues. You want to talk to another thread, you post a message on its queue. It was so simple and so effective, and so different from how things are done in most desktop applications like NetBeans. At the time I didn't think much about it, I just thought it was an interesting invention.

But my tribulations with Java and threading, my memories of the elegance of the message-passing model, and the ongoing challenge of scaling in multi-core systems, have reminded me of the power of this model, and increased my interest in languages like Scala and Erlang. Now if I can just pull myself out from under these bugs long enough to take a look at them...

Praise be, and deep thanks to Koffi Annan

I just read this opinion piece by Roger Cohen in the New York Times about what Secretary General Koffi Annan was able to accomplish in Kenya. All I can feel is great gratitude, and hope that it will stick.

Here's to the all too rare success of diplomatic intervention. The prayers of many of us appear to have been answered.

Good signs from Monty

Monty just posted a blog about the status of Maria and his proposal to have an architectural meeting at the MySQL Conference. Then he said something at the end that I thought was pretty promising. I hope he can make this fly!

I hope that within Sun we will get resources to change our current polices, priorities and in some ways the whole engineering organization to make the development model much friendlier to outside participants. It should be as easy for an outsider to get a patch into the MySQL server as someone working for MySQL. This is one of the things I would like to spend my time on inside of Sun!

You go, Monty!

Amazon, Hadoop and the New York Times

I just read this interesting entry on the Amazon Web Services blog talking about the combination of Amazon EC2 and Hadoop. I was mildly interested but then this quote caught my eye.

Everyday, I hear new stories about running Hadoop on EC2. For example, The New York Times used 100 Amazon EC2 instances and a Hadoop application to process 4TB of raw image TIFF data (stored in S3) into 1.1 million finished PDFs in the space of 24 hours at a computation cost of just $240.

Pretty breathtaking...

Short-sightedness and the iPhone

My wife and I have this deal that when I get a bonus (thanks, Sun!), we split half of it between us and the other half goes into the Money Pit.

So I had this $$ burning a hole in my pocket, and I have just finished a Big Push to get some MySQL tooling into NetBeans (a topic for another blog). I deserved a treat. I also felt my patriotic duty to inject Consumer Confidence and Cash into the economy.

I've been wanting to switch wireless services, I really find that I want to use the Internet, and Verizon charges an arm and a leg for its data service. AT&T is downright reasonable. But of course you can't take your phone with you (that's just horrific, IMHO), and so my Treo is tethered to Verizon. I thought I had bought that Treo, but I realize now I was only renting it, it actually belongs to Verizon.

Meanwhile, iPhones are popping up all around me, and I interviewed many users, a number of them quite picky and techie, like me. And they all love them. Then I hesitated because it didn't have GPS. Well, that's been solved, sort of, with its cell tower triangulation.

A friend convinced me to take a look at the Blackberry. It has all the bells and whistles, and even has video recording and real GPS. So I went into the AT&T store and started looking at it. And this was where things started to break down. I just couldn't figure it out, and the print was so tiny. I'm getting older: gray hair, bad back and bad eyes. I could not for the life of me even read what the applications were; I had to wear my reading glasses just to use the phone.

Then I tried the browser and it was O So Sad. All the images crammed into a tiny screen. It was like living on the 7 and 1/2th floor.

Then I ambled over to the sleek iPhone, like a rabbit into a fox's den. Come Closer it cooed. Feel my sleek lines. Press my buttons. Try the Two-Fingered Zoom Gesture. Look at my Browser. Come to me...

Well, when I could see real web pages, and zoom in so I could actually read them, and the numbers and letters are actually readable for me without putting on my glasses, I was finally sold. A phone for Old People.

I feel a little dirty, like I have just made a Faustian deal with Mr. Jobs, but I do have to admit I love this new iPhone.

Jon Udell makes a good case for a simple db-to-web tool

Jon Udell makes a very good case for providing functionality that makes a database very easily web-accessible. The NetBeans DB-to-REST tool is a good example of this, and is a very good start.

http://blog.jonudell.net/2008/02/20/overcoming-data-friction/

A one week hellish trip to nowhere

A family acquaintance, whose name I won't mention for reasons that will be obvious, sent me this missive.

Ye gods, what a path through Purgatory... It sounds like one of my nightmares where I get lost on trains and planes and can't even remember where I was going in the first place (metaphor for life? nah...)

All this time he's been sending me email, and I'm trying to figure out how he's up so late at night in India. Now I realize he was doing his email from hotels, airport benches and who knows where else. Sigh...

Another family acquaintance responded with his normal humorous form "and I was just fighting freeway traffic!"

It's snowing in New York. How do I know? Because I'm in New York, instead of New Delhi, India. I wish it were a long story, but it was only a long plane ride.

After leaving London on a 7:30 am flight after not sleeping the night before (the taxi came at 4:30 am), I arrived in Helsinki at noon, then hung around the tidy airport for an hour until called to queue for my flight to New Delhi. I handed over my boarding card and passport, and watched with half a mind (my mode of concentration up to this
point) until the Finn Air employee told me that I didn't seem to have a visa. "Oh, I don't need a visa for India," I said.

Some clickety-clacks on the computer and a phone call later, I was told, with ruthless Scandinavian efficiency, that indeed I did need a visa, and while the plane was going to New Delhi, I was not. Americans, it seems, are not welcome without reservation the world over. Americans who can't be bothered to check visa requirements doubly so...

It was Saturday, all offices closed until Monday, usual 10-day wait for visas. For a microsecond I toyed for with the idea of a wintertime vacation in Helsinki, but then looking out the window at the 2;30pm sunset, I decided to go home.

That's when I began to get very acquainted with the high cost of bad planning. From 2:30 pm Helsinki time on Sunday, this was my path.

1. Exit (forlornly) the Helsinki departure lounge.

2. Haggle with Finn Air to get a ticket back to London. They eventually gave me my return leg from Helsinki without charge.

3. Back through security

4. 6 pm flight to London, arrive 8-ish. Wait an hour for my bag, which with ruthless Scandinavian efficiency Finn Air told me I had to check (1 kg. overweight)

5. Hie myself to American Airlines, get there just before the ticket desk closed. Of course I had just missed the last flight, but could take another flight the next day ($200 change fee).

6. A night of purgatory at the Comfort Inn Heathrow, only 1/2 hour away and "cheap" at 75 pounds ($150). Hot water worked fine, but no cold water. No washcloth either, but I managed with the corner of a towel and a decent time to let the water cool. I could have gone into London, but that's a long trip and friend Peter was away at a
conference and I'd thrown his keys through the mail slot. And I was ashamed.

7. Back to Heathrow and "has your luggage been with you at all times?" (Yes, for the last week.)

8. Arrive New York Sunday night.

During this time I told just one person what had happened, the guy I was supposed to meet the next morning in New Delhi.

No harm done; the schmoozing that I was to do in India has been re- scheduled back here in the States, but I feel like a fool. Since then, I've told a few other people, citing "family reasons."

Hotel Facebook: you just can't leave

This New York Times article about how impossible it is to remove your data from Facebook underscores the need for data property rights.

McCain: Bomb, bomb, bomb Iran

Ouch. I'm sure some of it's taken out of context, but my goodness. If you haven't seen the original video "Yes We Can," you might want to see it first...

New York Times - Berkeley At its Best (Worst?)

Sigh... Good ol' Berkeley. My home town and alma mater. Sniff. By the way, I like the way patchouli smells, and I like being hip. :) And anyway, we have the best coffee, bar none. So there.

http://www.nytimes.com/2008/02/12/us/12berkeley.html

Rich Green: A vision for virtual machines, development and cloud computing

I think Rich nails the long-term vision, how you can have development "stacks" ready to install on your dev machine, and deploy "stacks" ready to push to the hosted environment. Note Amazon EC2 already does image-based hosting... a brave new world...

http://blogs.sun.com/richgreen/entry/on_snowglobes_xvm_and_freedom

Sun now has its own desktop VM product - VirtualBox

I just read the news that Sun is buying innotek, maker of the open source desktop virtual machine product, VirtualBox. I used VirtualBox before and it was nice, but since it seemed like a small concern and it was not clear what it's long-term prospects would be, I moved over to VMWare.

But now it's powered by Sun, as a fan of open source over proprietary source, as a Solaris user (and as a Sun employee :)), it's time to take another look at VirtualBox.

Fake Steve on Google, "the cloud" and Microsoft: funny yet serious

Fake Steve has a great conversation with Squirrel Boy, and as much as I laughed, you know, I think he's really nailed it. Although of course Sun will always stay independent and be one of the big boys :)

http://fakesteve.blogspot.com/2008/02/cloud-computing-oy-enough-already.html

Data Property Rights, Not Portability - GigaOM

I think Nitin nails this one - there is an ongoing attitude towards personal data in the networked world which has continued to bother me, and "the right to leave" only scratches the surface of the issues.

http://gigaom.com/2008/02/06/data-property-rights-not-portability/

Thirteen unlucky steps for installing MySQL 5 on Windows Vista

I haven't used Vista, and this doesn't motivate me to do so... What a mess!

http://forums.mysql.com/read.php?11,195569,195569#msg-195569

With Amazon SimpleDB, the world's a string, and I'm feeling a little frayed

Everything in Amazon SimpleDB is stored as a string and compared as a string. The consequences of this are explored in some detail in the new article on Amazon's pages.

What's the reason for doing this? In another article the rationale is explained:

This provides application designers with the flexibility of not predefining different data types for their attributes, but rather changing them dynamically, as the application requires. A good example of why this flexibility is useful is “1984” – should it be treated as an integer, a title of a book, or a date? What happens if the application decides to store it as an integer, only to later realize that it was meant to be the title of a book and should be treated as a string? Amazon SimpleDB provides the flexibility of storing all data in one format, allowing developers to make data type decisions in the application layer without the data store enforcing constraints.

Some of the consequences are pretty stunning, especially when you're working with numbers.

The first step for representing the number ranges is to ensure that every number in the dataset is positive. This can be easily achieved by choosing an offset, such that it is larger than the module of the smallest expected negative number in your dataset. For example, if the smallest expected number in your dataset is -12,000, choosing offset = 100,000 may be safe.

Then they follow with an example:

• Original dataset: {654, -12000, 3610, 0, -23}
• Negative number offset: 100,000
• Dataset with offset applied: {100654, 88000, 103610, 100000, 99977}

Then there is zero-padding:

Once all the numbers in the dataset are positive, there is another step necessary to ensure that values are properly represented for lexicographical comparisons. For example, number 2 and 10, if converted directly into strings "2" and "10" will not compare properly, as "10" comes before "2" in lexicographical order. However, if we zero-pad number 2 to be represented as "02", the comparison will execute as expected. But what happens if we decide to add number 200 to the dataset at a later point? Zero-padding with a single "0" will not work anymore. Therefore, application designers may follow the next steps for zero-padding:

1. Determine the largest possible number in your dataset. Remember, that if you are using offsetting to take care of the negative number conversions, you have to take that into account as well by ensuring that your largest possible number is determined after you have added the offset to all the values.
2. Based on the largest possible number, determine the maximum number of digits before the decimal point that you may have in your dataset.
3. Convert all the numbers in your dataset to the proper string representation by appending as many "0" as necessary in front of each number to ensure that the total number of characters, representing the portion of the number before the decimal point, matches the maximum number of digits determined in Step 2 above.

Example:

• Original dataset: {14.58, -12536.791, 20071109, 655378.34, -23}
• Negative number offset: 100,000
• Dataset with offset applied: {100014.58, 87463.209, 20171109, 755378.34, 99977}
• Zero-padded dataset representation: {00100014.58, 00087463.209, 20171109, 00755378.34, 00099977}
• Original query: ['attribute' > '500']
• Converted query: ['attribute' > '00100500']

Simple, right?

What if you don't happen to know what your maximum or minimum value will be for the lifetime of your application? Oh well, you gave it your best shot, right? What are the consequences if you guess wrong? So some queries will be inaccurate. No biggie...

I'm not religiously attached to the relational data model and the theory behind it. I can see the value in having "loosely typed" data, which is what Simple DB is doing. The argument is this way you don't have make assumptions about your data early on.

But the problem is, you actually do have to make assumptions. You have to know (or guess) how big and how small the numbers may get.

And because you are applying padding and shifting, this means you really are make a firm decision early on that the value is an integer, and really are never planning to treat it as anything else.

So the argument of flexibility doesn't seem to be holding water for me. I start wondering if it's the constraints of their architecture, and not user flexibility, that is driving this model. It's like, SimpleDB is simple because it is a simple data model (key/values stored a strings), not because it is simple to use.

I remember at Sybase, one of its selling points was that with the relational model, and with stored procedures and triggers, you can centralize your business rules in the database. This means that regardless of what applications, old or new, happen to touch your data, they can't mess up the integrity of your data. This was, and is, very popular with larger companies trying to manage hundreds of applications touching their databases built by as many developers across the globe, experienced and otherwise.

With SimpleDB, the responsibility to maintain business rules and data integrity lies with the application tier. If you have lots of applications touching your data, it seems to me it's a recipe for confusion at the least and potential data corruption at the worst.

So, if I were building an architecture on top of Simple DB, I would seriously consider putting a very solid and unencroachable layer between my application code and SimpleDB to enforce the structure and rules I need to keep the data clean and accurate.

As I continue to blog, I am hearing and learning from readers that there are many, many perspectives and requirements, and usually the answer for any architectural choice is "it depends."

So, what do you think? How would you approach these aspects of SimpleDB? What's to like, what's not to like?

James Governor: a great analyst article about Sun

I've enjoyed James' blogs over the years, but to me this is one of the most well written, insightful, and fun blogs I've read from him. It doesn't hurt that it has good things (very good things) to say about Sun.

http://www.redmonk.com/jgovernor/2008/02/08/strong-leaders-strange-bedfellows-and-the-art-of-war-by-sun-2/

The House & other Arctic musings: -40 Fahrenheit

I really enjoy this blog from north of the Arctic circle. "no machine likes -40 and you can almost hear them scream as parts snap or shake off of their mounts." Yeesh!

http://kiggavik.typepad.com/the_house_other_arctic_mu/2008/02/-40.html

Some rave reviews for Apache Derby

Someone just pointed me to this page reviewing Derby on ohloh.net.

Some pretty positive reviews! Here are some treasure quotes. Note the range from embedded to client/server, with some very heavy data loads. The amazing thing is that these are some heavy users, and the derby-user list is very quiet, maybe 10 emails a day. It Just Works (TM).

The sign of a good embedded database system is not who’s running it, but rather who doesn’t know they’re running it! The ideal database is one that sits there in the background, working hard, and never raising a peep nor complaining. That’s Apache Derby!

Our software has been deployed by tens of thousands of organizations ranging from small schools all the way up to prestigious academic organizations and corporate banks. Although we offer the option to choose other external databases, the majority stick with Apache Derby as the embedded default. Many PaperCut NG installations handle thousands of print transactions a day and host datasets in the millions of rows. Apache Derby has proven that an embedded zero-administration database can be robust and scalable in a range of environments and operating systems.

I'm currently using derby for a distributed client architecture. I've tested with 3~5 million records that need to be keep in sync through all the clients. Derby has worked beautifully, selects, inserts all very fast. And I've even seen several searches with the same indexes run faster on Derby than Oracle.

We use derby in our search engine project which is in beta stage. Having tried alternatives (mysql , postgre , oralce ) derby provieded us unmatched performence over large datasets our current dataset is around 30GB for each server we read more than a million rows at a time and insert around 300 records per second we are confident that derby will not let us down along the way. we are planning to hit over 100GB mark for both of our servers during the next couple of months

Get paid for improving NetBeans

I finally tuned into this new NetBeans program. If you'd like to make some money and you love to code, you can participate in the NetBeans Innovators Grants. See the page for details, but here's the money quote:

The NetBeans Innovators Grant is a process to provide grants to developers or teams of developers to work on an open source project. A total of 10 large projects will be chosen and awarded a grant of US$ 11,500 dollars. Another 10 smaller projects will be chosen and awarded a grant of US$ 2,000 dollars. Awards will only be awarded upon actual project completion. Projects that excel may receive one of two possible gold awards of US$ 11,000 dollars or two possible silver awards of US$ 5,000 dollars.

Submission deadline is March 3.

This Wiki page shows some ideas for projects already listed (I put in my plug for some database features like SQL command history (small) and database schema refactoring (large)).

Derby documentation quick search in Firefox - Knut Anders Hatlen's Weblog

Knut Anders has an excellent tip on how to make it easy to search Derby documentation using Firefox. Thanks, Knut!

http://blogs.sun.com/kah/entry/derby_documentation_quick_search_in

Marion Nestle » Healthy people are too expensive for society?

I love this quote:

... obese people and smokers cost less to treat. Of course they do. They die sooner. Healthy people are expensive ... Economists have an interesting way of looking at things

http://whattoeatbook.com/2008/02/05/healthy-people-are-too-expensive-for-society/

Using the pre-installed SXDE VMWare image on MacOSX

As I mentioned in a previous post, you can download SXDE 01/08 as a VMWare image.

The download page, however, assumes you know what to do with the files (two large zip files) once you have them.

It wasn't that clear to me, although I figured it out after some searching around. I thought I'd describe them here for you.

• Unzip the two files. This creates two separate directories, with a number of files. Copy all the files into a single directory. Your directory listing should look something like this:
  

  
  SunSXDE108-s001.vmdk    SunSXDE108-s005.vmdk    SunSXDE108.vmsd
  SunSXDE108-s002.vmdk    SunSXDE108-s006.vmdk    SunSXDE108.vmx
  SunSXDE108-s003.vmdk    SunSXDE108.nvram        vmware.log
  SunSXDE108-s004.vmdk    SunSXDE108.vmdk
  

  
  
• Edit the SUNSXDE108.vmx file and add this line:
  

  
  monitor_control.disable_longmode = 1
  

  
  
• Bring up VMWare Fusion, choose "File->Open" and point it to SunSXDE108.vmx

• Log in as "root", password "sxde"

• When you log in, it warns you you are logging in as a privileged user, and suggests you not do this. I wholeheartedly agree. But log in anyway, so you can create a new user account. SXDE very graciously starts up the user admin GUI for you to do this


• Log out of root and log in as the new user


• VMWare tools don't work completely so are not installed. This requires a fix from VMWare, which I hope we'll see soon. This is a cutting-edge distribution, after all...

• Fix the screen resolution. On my 17" MacBook Pro, much trial and error led me to pick 1680x1050 - not too small, not too large, just right.

• Then you can follow the various to-dos I wrote up on how to get going with NetBeans, Ruby, MySQL, PostgreSQL, Apache, etc.

That should do it!

Sequel, Es-Qyew-Ell, Squell

Now that Sun has purchased MySQL, there have been some internal discussions about the right pronunciation. I always said "Sequel" and "MySequel." When I was at Sybase, we called it "Sequel Server" and I hear tell that's the pronunciation Oracle uses too.

But the official pronunciation of MySQL is "My-Ess-Qyew-Ell." Ah well, I have to train my tongue out of respect for our new colleagues, even though Sybase and Oracle were there first.

This all reminds me of the time when I was at Sybase (pronounced "sea bass" in France) when I was sent onsite to a big news agency on Long Island whose client application (written in Ada of all things) kept crashing. It was a nasty timing bug that was so sensitive that print statements would make it go away. This was with fully asynchronous code where you registered call backs, and asynchronous system traps (ASTs - remember?) would call the next step when the I/O completed. Talk about fun.

After three days of head banging I was sorely tempted to tell them to just put in print statements, but I knew that wouldn't fly...

I finally found it by reading, line by line, the application code until I logically determined what the problem was, and it was fixed with a few lines of code.

Anyway, my point is, the folks at this shop called SQL "Squell". Eeeww. I always had an uncomfortable feeling when I heard it pronounced that way, like they were squashing a snail.

Then of course there is Apache Derby. And Linux. Can't we all just get along?

Solaris Express Developer Edition with NetBeans 6, AMP and Ruby

Solaris Express Developer Edition (SXDE) is a quarterly distribution of the latest and greatest Solaris stuff. The releases are named after the quarter, and the release that just came out today is called SXDE 01/08. You can download it here. Note you can get the DVD image, or if you're using VMWare like me, you can just get the VMWare image -- pretty cool!

This release is of particular interest because for the first time it includes NetBeans out of the box, in this case NetBeans 6.0.

This release also includes an integrated runtime stack for Apache-MySQL-PHP and for Ruby, along with tools to make it easy to set up and manage these stacks. Getting Apache and MySQL up and running was quite a snap!



I've been playing around with this release for a while, making sure NetBeans plays nicey nicey with the runtime stacks. There are a few hiccups, but for the most part it works quite well.

I wrote up a number of how-tos on the NetBeans wiki site, like how to get started with using the Ruby runtime in NetBeans or how to start up the PostgreSQL service. If you're interested, get SXDE, install it (in VMWare if you're like me), and use these how-tos to help you get your feet wet.

One thing I thought was pretty cool was how easy it was to get going with debugging in PHP. You just have to enable debugging, set a breakpoint, and go.




I have been working with Solaris for a number of years, and these days the leaps and bounds they are making in improving the desktop experience is quite astounding. The installer is actually reasonable. I am also looking forward to the new package management system, IPS, that is coming out as part of Project Indiana.

So, try it out, or stay tuned, as you see fit, but this Solaris thing is definitely coming along. The report from Europe isn't so bad either... :)

Kenya: memories and heartbreak

I have been reading the news about Kenya with a growing deep sadness. I spent one year there as a child with my family, on a little island called Rusinga off the shore of Lake Victoria. My Mom is a paleontologist and my Dad is a geologist, and they were doing research for Louis Leakey. Later at a town called Kaptagat while the kids went to school and my Dad worked at a mine.


View Larger Map

It was a wonderful time for me, and I continue to be filled with gratitude for the land and the people. It touched me, and all my family, in a very deep way, something that goes beyond words and description.

As I read about the places of the worst violence: Kisumu, Eldoret, my heart aches. I remember those towns. I remember the people - the fierce but loving Masai, the Luo with their incredibly brilliant smiles and warm hearts, and the Kikuyu. All with such open hearts, living in a beautiful land. Now I read how many are filled with hate, looting and killing, about how 60% of the people live in slums, and I just shake my head.

For some reason I had always thought that some day I could maybe go back, maybe with my kids when they're older, and revisit this place that was such an integral part of my childhood. Now I seriously doubt that if I went, it would be even close to the same. It really is true, you can't go back.

I do hope that the people of Kenya can pull themselves out of this madness and find a way to live and love.

As an ancient prayer from the Rig Veda says:

Let us be united;
Let us speak in harmony;
Let our minds apprehend alike.
Common be our prayer,
Common be the end of our assembly;
Common be our resolution;
Common be our deliberations.
Alike be our feelings;
Unified be our hearts;
Common be our intentions;
Perfect be our unity.

Giant Crystal Cave Comes to Light

Amazing, amazing cave. Look at the sight of these. Just beautiful. The world contains wonder after wonder.

http://news.nationalgeographic.com/news/2007/04/photogalleries/giant-crystals-cave/index.html

NetBeans plugin for copying text between Java and SQL/XML/HTML/...

You've been there: you want to write a SQL query, test it, and then paste it into Java. But Java requires a string that is concatenated with '+' for readability. What a pain to have to go back and forth between the query editor and the Java editor.

Wade Chandler (our hero again) has written a nice plugin that let's you convert back and forth between "Java-style" string-concatenated text and "plain style" of SQL and other plain text editors. Wade explains this more in this email.

Here's an example. I start with a nice long query in the query editor that does what I want it to do...



Then I use Wade's plugin to paste into Java:



et voila!

Setting up an SSH tunnel to talk to a remote database

On the nbusers list, Gary Greenberg asked

I have a database (Postgresql) running on the server with the SSL access only.

I.e. to run the query against it I need to login in via PUTTY or other SSL client.

Does anyone know about SSL proxy or whatever that will allow me to create a database connection in the NB6 Services tab?

I gave a pretty lame answer, because I've actually never had to do this, but Wade Chandler had an excellent description of what to do, which I will share here

It sounds like you are saying you have to use SSH. Do you mean the only way to access the server is with SSH? If so then you can use SSH tunnels. I do this often.

You will use which ever SSH client you have. Putty in this case. You then setup a tunnel which in Putty I believe should be under SSH tunnels or tunnels in the configuration section where you setup your connections or connect. You'll have a local port you will later connect to using regular TCP/IP and the server or remote port which the end SSH connection hooks to your local port and makes a tunnel to. Be sure and save your connection once you make your changes so you don't lose them for the next run.

Once you make your SSH connection on your local machine to the SSH server then your DB client server address will actually point to 127.0.0.1 and the local port number you setup in your SSH tunnel. SSH will then link your local port ${whatever_it_is} to the other port on the other end of your SSH connection. So, you actually connect with regular TCP/IP to your local machine on a port which is tunneled over SSH to a TCP/IP port on the other end of your SSH connection. This is why you connect to your local box/machine as the server with the database running on it though it is not...it is tunneled to the real connection and machine.

I have heard about SSH tunneling in the past, and never understood clearly it worked, never having the need. But this description was so clear, I just Got It. Thanks, Wade!

Skål!

Thanks to Eduardo for pointing me to this great video. Now that's the way to seal a deal!



The phonetic translation from Kaj's blog had me roaring - hell and gore!

Hell and gore
Shung Hop father Allan Allan ley.
Hell and gore
Shung Hop father Allan ley.

Oh handsome in the hell and tar
hand hell are in the half and four.
Hell and gore
Shung Hop father Allan ley

My sister married a Swede a few years ago, and the wedding was full of songs and drinks.

I saw the same thing in Norway when I used to visit the database team there in Trondheim - very polite and reserved (mostly) during the day, but go out for beer with them and the roaring laughter and singing starts. I can almost see the horned hats :)

Skål! May we both learn and grow from each other!

What to do when Google Data API requires a captcha

I posted quite a while ago about a little program I wrote to take my delicious links tagged with "blogthis" and post them to my blog.

This has been working great for over a year, but today I got the following error:

com.google.gdata.client.GoogleService$CaptchaRequiredException: Captcha required

I remember reading about this when I was researching the API, but my problem is my app is not a web app and I can't dynamically route someone to a captcha page. After much Googling about Google (hm), I discovered you can unlock your account by going to the Google Unlock Captcha page. Type in what you see, submit, and the program works again. Cool!

Arun Gupta: RESTful representation of "sakila" using GlassFish and NetBeans IDE

Of worthy note, Arun Gupta, our Web 2.0/web services evanglist, has a nice blog showing how to quickly put together a RESTful representation of MySQL's Sakila database...

http://blogs.sun.com/arungupta/entry/restful_representation_of_sakila_using

The Moral Instinct - New York Times

A very interesting conclusion, that morality is greater than us, not because of a moral, judgemental God, but because it is a natural consequence of how the world works. I personally see "the nature of things" *as* God - God is Existence itself.

http://www.nytimes.com/2008/01/13/magazine/13Psychology-t.html?_r=1&oref=slogin

Good Math, Bad Math : Databases are hammers; MapReduce is a screwdriver.

I was a little stunned at the MapReduce criticism from Stonebraker, and I think this is a great response

http://scienceblogs.com/goodmath/2008/01/databases_are_hammers_mapreduc.php

The Moral Instinct - New York Times

A very interesting conclusion, that morality is greater than us, not because of a moral, judgmental God, but because it is a natural consequence of how the world works. I personally see "the nature of things" as God - God is Existence itself.

http://www.nytimes.com/2008/01/13/magazine/13Psychology-t.html

AS IS CLAUSE - Deal With It

I love these "AS IS" clauses in software license agreements. I thought I'd check out Amazon Simple DB on my day off (yes, you can tell I'm a geek when this is what I do with my day off).

The limited beta is closed, so I guess I'll have to wait. But I signed up to be notified if a slot opens up in the future, and I saw this in the Amazon Web Services license agreement. Standard boilerplate I'm sure, but it still strikes me how little software companies are willing to warrant. It shows just how unreliable software really is.

WE AND OUR LICENSORS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, QUIET ENJOYMENT, AND ANY WARRANTIES ARISING OUT OF ANY COURSE OF DEALING OR USAGE OF TRADE. WE AND OUR LICENSORS DO NOT WARRANT THAT THE SERVICE OFFERINGS WILL FUNCTION AS DESCRIBED, WILL BE UNINTERRUPTED OR ERROR FREE, OR FREE OF HARMFUL COMPONENTS, OR THAT THE DATA YOU STORE WITHIN THE SERVICE OFFERINGS WILL BE SECURE OR NOT OTHERWISE LOST OR
DAMAGED.

Sheesh! The only thing one can hope for is that they really don't want anything bad to happen because of the impact on the perception of their services. Although I don't know how one could claim that Amazon Web Service was not giving "quiet enjoyment." :)

Steve Jobs Macworld 2008 Keynote in 60 Seconds

Thanks to TechCrunch for this link - better and better and better.

http://www.youtube.com/watch?v=Yz1-cPx0cIk

Introducing Sun's new database: Elephine

Hey, what's the point of having three databases? Let's combine them all!



Thanks to Petr Zaharadnik by way of Zdenek Kotala for this one. Hee hee.

Cringely agrees with me

I have my theory about why the MacBook Air has no optical drive. It looks like Mr. Cringely agrees with me.

What about the Air's lack of an optical drive? It's hard to find a place for an optical drive in such a thin computer, but isn't Steve Jobs the guy who when he returned to Apple railed against notebooks without removable media, like the PowerBook 100 and 2400 and the various PowerBook Duos? Why did Steve change his mind now? Because Steve wants to replace optical drives of any sort with bits provided over the network, preferably from iTunes. That's also why we didn't see an Apple Blu-ray announcement this week and -- if Jobs has his way -- we'll never see one.

NetBeans and MySQL: What To Do?

I have been getting a number of internal and external emails and phone calls offering ideas and asking me for opinions for what to do in NetBeans now that Sun has a new Very Popular Database.

In case you don't know, I am currently the person responsible for database tooling in NetBeans. Until yesterday, this was going along fairly smoothly, but things just got a little hot around here :).

First of all, I suspect that some folks over in MySQL may have some thoughts and opinions about this :), and so far I haven't had a chance to talk to them. So that's a very important missing piece of the puzzle. I need to figure out when and how I can do that.

But meanwhile, let me share my thoughts and idea around this.

First, I do want to say that we already have pretty good integration with MySQL. The MySQL JDBC driver is included with NetBeans 6.0, and it's pretty darn easy to set up a connection and start browsing your MySQL database



NetBeans is excellent for database programming because the database tooling is integrated with a fully fledged IDE. That means you can do things like generate JPA entities from database tables, generate a desktop CRUD application, generate a RESTful Web Service from a database, and visually bind web page components to database tables.

OK, but what more can we do :).

Well, lots of things. All sorts of things. But what makes sense? That's going to take some thought.

Just as an example, NetBeans is a development environment. Most of the folks I talk to don't believe it should be a database administration tool, because administration is so specific to each database. Sure, we will own and support MySQL, but Sun also distributes and supports PostgreSQL and Java DB, and there are a few other important databases out there that many of our customers use. Do we build admin tools for all of these as well?

That said, I can imagine making it very easy to bring up the MySQL admin tool from NetBeans (and let's not forget the wildly popular open source MySQL admin tool, phpmyadmin. For that matter, we should make it easy to bring up pgAdmin for PostgreSQL. Java DB, bless it's heart, really doesn't need an admin tool, except for start and stop, and we already do that (OK, we could add support for setting properties).

Now that I think of it, probably the best thing would be to let the user register the admin tool they want to use for a given database driver/connection.

There are lots of other things we could do. What about stored procedure support, including debugging? What about integrating MySQL's data modeling functionality into NetBeans? What about making it easy to create a sample database? And so on.

I've set up a Wiki page to gather them all. There is a lot of creative energy around this. Tell us your ideas. Put them on the Wiki page, add a comment to this blog, add a feature request in Issuezilla, or send me an email (david at vancouvering dot com). Let's get this ball rolling!

Say it ain't so: Ruby on Rails not thread-safe, no prepared statements?

Maybe everyone already knows this. But I've been hearing about Ruby on Rails for over a year now, and I know people working on it, and I am reading all this stuff about putting Ruby on Rails on Glassfish and getting it into the enterprise. I did hear some mumblings about scaling issues with Ruby on Rails. But I didn't know it was as basic as Ruby on Rails not being thread-safe.

Hm, maybe this guy got it wrong. But no, here is another post. And another.

The author of the last post, Greg Luck, says something else that just blows my mind as a database programmer - Ruby on Rails doesn't support prepared statements. Really? So I checked for other sources. Yep, it's true.

Somebody in Greg's comments said something about how Greg obviously didn't know about ActiveRecord caching. Um, that's not really the point. The point is, when you do hit the database, why on earth would you force it to recompile and re-optimize the SQL each time?

I have worked with multi-threaded applications since 1988. Multi-threading is not new. Pre-compiled SQL showed its power ofter fifteen years ago, and every database vendor worth its salt supports it. How can it be possible that an entirely new framework for server-side database applications is not thread-safe and does not support prepared statements? It's one of those things that makes one's jaw drop.

Maybe I'm just completely missing something. Maybe I'm living in the stone age. Someone explain to me where I've gone wrong. I strive to be open-minded and anyone who knows me knows I regularly change my mind if I am convinced I have it wrong.

But as it stands, if someone were to suggest to me that we should use Ruby On Rails for a web-scale server-side application, I would sit him or her down for coffee (or a strong drink), explain the limitations above, and beg them to think about this very seriously, and at the minimum, please, run some tests before committing to this framework.

Optimizing Java DB in unit test environments

A number of folks use Java DB/Apache Derby for their unit testing. In this environment, your requirements are different:

• You don't care about durability. If some data is lost, fine, run the test again
• You do care about getting through your tests as quickly as possible
• You want to regularly start with a fresh database, and you don't want "crud" left around at the end of the unit test. In particular, you don't want your test database left around.

Because you don't care about durability, you don't necessarily have to write to disk. You may not be aware of this, but some directories on your system may not be mounted to a physical disk, but instead are mapped to memory. In particular, /tmp is normally mapped to memory.

So if you set your database to point to /tmp, you should get some significant performance improvements, e.g. jdbc:derby:/tmp/mydb.

I'm not sure how you would do this on Windows. Here is one possible tip - note I haven't tried this myself. Just the first hit on Google.

Another thing you can do to speed things up is to configure Java DB to not bother synching to disk. You do this by setting derby.system.durability=test as either a system property when you start your VM, or in the derby.properties file. Specifically, when you set this property,

the store system will not force I/O synchronization calls for:

• the log file at each commit.
• the log file before a data page is forced to disk.
• page allocation when a file is grown.
• for data writes during checkpoints.

You can see this documentation page for more information.

Now, what about removing that pesky database directory when you're done with a test? Java DB does not provide any mechanism for doing that, so you're going to have to do this yourself.

There are two approaches to doing this. First of all, you could drop all the tables after each test run and re-create them, re-using the same database each time.

Alternately, you can have a "model" database set up, and after each test run you use Java file APIs to blow away the old database directory [1] and replace it with a fresh "model".

Note: before you try to remove the database, you should probably shut it down. Do this by connecting to it with the ;shutdown=true" property added to your URL.

The trick is, your code needs to know where this database directory lives. Here are some possible approaches:

• If you are fully qualifying paths to your databases, you know exactly where they live. But usually you wouldn't do this, because it makes your tests non-portable.
  
• You can let test runners specify a property that indicates where the database directories should go. Then you prepend this to the JDBC URL, and also use it when you're blowing away the directory. For example:
  

public class MyTest {

  private Connection conn;
  private String dbdir;
  private String dbname;

  public void setUp() {
      dbdir = getTestProperty("database.dir", "/tmp");
      dbname = getTestProperty("database.name", "mydb");
      copyModel(dbdir, dbname);
      String url = "jdbc:derby:" + dbdir + "/" + dbname;
      conn = connectToDb(url);
  }

  public void tearDown() {
      conn = connectToDb(url = ";shutdown = true");
      deleteDatabase(dbdir, dbname);
  }

• You can use some knowledge about how Java DB determines where to place a database. It follows the basic simple rule: if derby.system.home is set, place the database directory there. If that's not set, use the Java system property user.dir. So you can follow the same logic to figure out where your database is and then blow it away (and put a new one there in its place).

---

[1] You can write the code yourself to blow away a directory, or use the handy FileUtils.deleteDirectory() from Apache Commons IO.

MySQL and Sun - my thoughts

OK, I'm a database guy, I work at Sun, I have a blog. I guess I need to say something here about the MySQL acquisition.

Well, OK - Wow. I really got caught off guard by this one. I wasn't paying attention, I guess. I didn't think MySQL was in the market, and I didn't know Sun was looking. I work a lot with the folks who must have been involved in the due diligence with this deal, but not a word. A very well kept secret.

That said, here are my thoughts.


Let The Community In

You have heard me criticize an open source product that is controlled solely by a single company. "This is my opinion and does not necessarily reflect the position of the company I work for", but I hope that, as part of Sun, that changes for MySQL.

I would like to see us allowing external contributors become committers to MySQL. I think it behooves us to make it clear that MySQL is ultimately owned by the global community, not by Sun. But that's just my manifesto, I'll leave it to the execs to make the Big Important Decisions around this.


Make It Free

I also think it's time to throw out the dual-licensing model, and, just like our other software, make it free to deploy and embed, and charge at the point of value: support and services.


What About Sun's Investment in PostgreSQL?

Well, as far as I can tell, it's pretty much "stay on course" for this. Josh Berkus still has a job. I think PostgreSQL has features and a community that is focused more on the IT data center and data warehousing environments. PostgreSQL and Sun's hardware are a very good pair.

MySQL has an affinity for the Internet, web-facing, fast-changing, fast-paced application environments.

Yes, it's a little fuzzy, and I'm sure we'll get questions about what to use when, but I personally see no reason to throw PostgreSQL out the window just because we acquired MySQL.

What About the Database Team?

Hey, why have a blog if I can't do wild speculation?

I can see Marten Mickos reporting to Rich Green, and aligning all database technology under him. I just hope he can let go of (or ignore) any MySQL biases he may have and make sure he serves all of our database technology products/projects (Java DB, PostgreSQL, MySQL) equally.

I can see some of the MySQL database tools staying under the database group (the migration and admin tools) and others migrating over to NetBeans over time (the Query Designer and MySQL Workbench). And I think you'll see all of these tools becoming free - charging for tools has not historically been Sun's strategy. But I could be wrong...

Check back with me in a year and we'll see how wildly off I was with these predictions.

How Does It Affect Me?

I'm not sure yet. I can imagine me flying to Germany to meet the MySQL tools team (stopping by in London to see my new niece) and discussing strategy and future plans. I can see the database team under NetBeans getting a bit bigger (right now, in case you didn't know this, it's just me). I can see me downloading and playing with the MySQL tools (thank goodness for VMWare Fusion, otherwise I'd need to borrow a Windows box to run MySQL Workbench).

I don't know, what do you think? What should we be doing with NetBeans and tooling in light of this acquisition? Any ideas?

One final thought: now that I know MySQL was on the market, I'm very glad Oracle didn't buy them. That would have had a sobering effect on the open source community. I guess they were busy with other deals. Thanks, BEA! :)

No optical drive? Gee, I guess I have to rent my movie through iTunes

Just read the details that the new MacBook Air has no optical drive. The rationale is that it makes the machine lighter and have longer battery life.

Yeah, sure.

You know what I think? If you have an optical drive, you'll buy or rent DVDs when you want to watch a movie on your laptop. If you don't have an optical drive, what do you do? You rent them through iTunes. Kaching!

This explains why the MBA is so reasonably priced. It's a loss leader to Apple's real business - entertainment media distribution. You don't have to be an MBA to figure that one out... :)

Oooh, I want it

This just in - a new Mac laptop, and I quote:

A lot of others only get 1.5 hours -- this is phenomenal battery life. 3 pounds, 13.3-inch full size display, full keyboard, multi-touch gestures, 0.16-inch thickness, iSight... 1.6GHz C2D, 2GB RAM standard, 80GB drive, 64GB SSD option, 802.11n standard Bluetooth 2.1 + EDR...

All these and more in the thinnest notebook in the world -- we are pricing this at $1799.

Did you read that about a 64GB Solid State Drive? Do you know how fast and low-power those things are?

My shiny MBP Core-Duo seems so, oh, old-fashioned now. Good job in creating tech greed. Now I just need to show my work or my wife why this is absolutely essential for me to do my job. Maybe I can pull out the eco-card with their mercury/arsenic-free screen and eco-friendly electronic parts... Yeah, that's it...

You just don't get it

There are times when you realize other people just don't see things the way you do. They just don't get it. For example...

Geek Jokes

After I showed it to her online, with a big smile on my face, my wife bought me a T-shirt for Christmas that says

There are 10 types of people in the world. Those who understand binary and those who don't.

I love this T-shirt. Either you get it, or you don't. In particular, you probably don't get why anyone would think it was funny.

Another favorite of mine is the T-Shirt that says <BODY> on the front and </BODY> on the back. Priceless. Snork snork.

Fixing My Own Computer

I recently bought a newer (larger, faster) hard drive for my MacBook Pro. iFixIt has very nice instructions on how to upgrade your hard drive here. Note all the tiny little screws. I was surprised by how many tiny little screws there were. I also had to go to Radio Shack to get a T6 Torx screwdriver. I actually got a cool little box with a screwdriver and exchangable tiny bits for electronic repair.

At dinner (my laptop in pieces, not done yet) I mentioned to my wife how surprised I was at all the little screws, how Apple had made it so complicated compared to other laptops I had worked with, and how it was taking me longer than I thought. She said "why didn't you just take it to a repair person?" I looked at her, stunned. I was hurt. How could she say such a thing? She just didn't get it.

It was as if she was struggling late at night over a pie for a pot-luck the next day, and I were to say "honey, why don't you save yourself some time and just buy a pie from the local bakery?" Sheesh...

Smacking My Son

Recently I was hanging outside our house with my 2-year-old son and our neighbors, a Mom and her daughter. At some point my son ran SLAM straight into me and we both laughed. The Mom and daughter both sucked in their breath.

I said, "you know, I don't think this is something women usually get. Boys love this sort of thing." Then I gave him a good smack on his forehead with my palm, kind of like Homer doing 'doh!' or the V-8 commercial. They both exclaimed in alarm. He roared with laughter. After some time the Mom said "no, we don't get it" and shook her head.

Ah, well...

Derby tips and tricks Wiki page

The Apache Derby Wiki has a very useful page collecting hints and tips for using Apache Derby -- worth keeping in your bookmarks if you run into issues or have questions.

The Derby Wiki has a lot of useful information, including information on upcoming releases and pages on how to contribute to Derby if you are so inclined.

How to initialize an embedded Java DB database using JPA/TopLink Essentials

The question of how to initialize an embedded Java DB database when you are using JPA/TopLink Essentials came up recently on the NetBeans users list.

I answered the question there and thought I'd answer it here as well... These steps are applicable even when you're not using TopLink, with the exception of how you configure your persistence unit to auto-create tables for you.

There are three parts to initializating an embedded database. Note that this is for a fresh install of the application. If you are upgrading an application, that is a whole other kettle of fish that I will answer another time...

• Create the database
• Create tables
• Load initial data set

I'll cover each of these separately. See also the last section, where I talk about using a model database, which handles all of these steps in one fell swoop.

Create the Database

If you change your persistence.xml so that the URL includes the parameter ";create=true" then Java DB will automatically create the database for you

Create the Tables

You can have TopLink to this for you by setting a property for the
persistence unit in your persistence.xml file:

<property name="toplink.ddl-generation" value="create-tables">
</property>


See the Oracle TopLink extensions documentation for more information

Load the Initial Data Set

There are a number of ways you can do this:

• You can write some code that creates new JPA objects, initializes them, and commits them using JPA
• You can write some code that uses JDBC to load your data set
• Use a SQL script and the Derby tool 'ij' to load the SQL. This requires including derbytools.jar in your application. See this blog entry for a reference to one user who was successful doing it this way.

Three in One Blow: Using a Model Database

Another way to create the database, tables and data is to use a model database. You create your database as you want it, and then include the database directory in your application jar file. Derby databases are completely portable across any platform, so you don't need to worry about having different versions for, say, different byte-ordering architectures or for 32-bit/64-bit architectures.

Then at startup if the database directory is not there, you can extract it from your application jar and you're good to go.

Note this is what NetBeans does. When you register Java DB, or if Java DB is detected when you register Glassfish or you use JDK 6, we automatically extract a zip of a sample database that is included in NetBeans and create a connection to it in the Services tab. It works quite nicely.

Fake Steve: Breakfast with an Apple lawyer

OMG, this is hilarious, tears in the eyes funny, and scary at the same time. He nailed it with Gandhi. Corporate America has no ethics, black heart to the core. I suspect Apple is Very Very Angry now (cue to Looney Tunes Martian).

http://fakesteve.blogspot.com/2007/12/breakfast-with-apple-lawyer.html

Ruby support in NetBeans continues to gain fans

Just saw this in the Ruby mailing list for NetBeans:

I love the Ruby and Ruby on Rails support in NetBeans! I believe it's
the best Ruby IDE out there. Please keep up the good work. I've
abandoned RadRails for NetBeans.

Want to know more? See http://ruby.netbeans.org and http://wiki.netbeans.org/wiki/view/Ruby

Give me a good hash function, and I can move mountains

Knut Anders Hatlen is one of the resident geniuses over in Java DB land. He just recently started a blog, and this entry is a great read, instructing us in performance optimization, hashing, and one of the hidden gems in NetBeans, a simple Generate Code menu item that generates a good hash code.

As I mentioned to the Java DB team on email, it's these little things that make all the difference. Sure, the various whizbang doodads are nice, but give me a good editor that improves my productivity and quality while writing code, and I'll gladly raise a pint in your name and kiss your dog in gratitude.

My daughter, Santa Lucia, and the SF Chronicle...

Ariel, my daughter, goes to the East Bay Waldorf School, and a few weeks ago we were asked at the last minute to create a crown for her that would hold eleven candles. So my wife Linda spend a good two hours on Thursday night cutting up coat hangers and putting something together with needle-nose pliers and a glue gun.

Ariel was going to be Santa Lucia, the saint of light in the darkness of winter. Waldorf celebrates holidays from many traditions, including those of Europe. In European villages a girl is chosen to represent Santa Lucia, and she is dressed in robes and a crown of candles.

Ariel came home saying there was a photographer there flashing picture after picture in her face, and she said she would be in the paper. Sure enough, the following Saturday, there she was. And it wasn't just a little picture below the fold, but it was a huge picture filling the whole top fold of the Home section, as well as a thumbnail on the front page. The phone was ringing off the hook that morning with friends asking us if we'd seen the paper. You can read the whole story here at SFGate.

Doesn't she look beautiful - beatific is more like it. She is an amazing little girl (well, not so little any more). And yes, that is a towel on her head. Even a goddess of light needs to protect her hair from the wax...