Tuesday, December 25, 2007

My daughter, Santa Lucia, and the SF Chronicle...

Ariel, my daughter, goes to the East Bay Waldorf School, and a few weeks ago we were asked at the last minute to create a crown for her that would hold eleven candles. So my wife Linda spend a good two hours on Thursday night cutting up coat hangers and putting something together with needle-nose pliers and a glue gun.

Ariel was going to be Santa Lucia, the saint of light in the darkness of winter. Waldorf celebrates holidays from many traditions, including those of Europe. In European villages a girl is chosen to represent Santa Lucia, and she is dressed in robes and a crown of candles.

Ariel came home saying there was a photographer there flashing picture after picture in her face, and she said she would be in the paper. Sure enough, the following Saturday, there she was. And it wasn't just a little picture below the fold, but it was a huge picture filling the whole top fold of the Home section, as well as a thumbnail on the front page. The phone was ringing off the hook that morning with friends asking us if we'd seen the paper. You can read the whole story here at SFGate.

Doesn't she look beautiful - beatific is more like it. She is an amazing little girl (well, not so little any more). And yes, that is a towel on her head. Even a goddess of light needs to protect her hair from the wax...

Friday, December 21, 2007

A RESTful version of Amazon's SimpleDB - subbu.org

Hm, someone dug in to SimpleDB's REST API and found it to be not very RESTful. Doing everything over GET? Sounds like someone didn't do their homework...


Thursday, December 20, 2007

Fear and greed: the engine of global warming

Corporations and governments seem to act with no moral compass, like refusing to do anything about greenhouse gases even as the world roasts.

Why is this?

Public companies are whores to the Wall Street pimps - if you don't come in with cash tonight, you get beat up. So corporate leaders who are afraid of losing their jobs and the bags of money that come with it will act against what might be their own moral gauge of what is right in order to bring in the top dollar to the company's stockholders.

In the same way, elected officials are highly dependent on funding to win an election, and so they also become whores to those who have money - namely large corporations. An elected official is driven by greed for power or the fear of losing their position to go against their own moral gauge of what is right.

Then there is the engine behind the pressure to act unethically: Wall Street (or your equivalent in your country). These big greedy investment institutions are leading us pell mell to destruction.

But who is Wall Street serving? Where do they get all their money?

From you and I. We want immediate results or we dump our mutual funds. We are afraid of losing money, and we want to make more money. Fear and greed.

So at all levels, it is fear and greed which is driving this, starting with you and me. And what drives fear and greed? Well, I don't know about anyone else, but for me it is tied to deep beliefs in my own smallness, in feeling limited, contracted, vulnerable.

So how do I fight global warming? I try to fight these feelings I have about myself, and remind myself of my own greatness and goodness. In this way I can start making choices that are not based on fear and greed.

An ancient scripture from India says: dhanam atma jnanam: "Knowledge of the Self is charity". As I follow this thread of corrupt and unethical behavior down to its source, I can see the deep wisdom of this statement.

Eventual consistency - following the Middle Path

One of the features of Amazon's SimpleDB is that a write to the database may not be immediately reflected to all readers, and that the user of the interface needs to be aware of this and work with it accordingly.

For those of us used to working with relational databases, which generally provide support for full read consistency, this is a somewhat revolutionary thought. But Werner Vogels, CTO of Amazon.com, explains in his blog that in scalable systems where data must be shared across machines, consistency inherently limits availability, particularly in larger systems where network partitioning is common:
Eric [Brewer] presented the CAP theorem, which states that of three properties of shared-data systems; data consistency, system availability and tolerance to network partition one can only achieve two at any given time. A more formal confirmation can be found in a paper by Gilbert and Lynch.

A system that is not tolerant to network partitions can achieve data consistency and availability, and often does so by using transaction protocols. To make this work, client and storage systems are part of the same environment and they fail as a whole under certain scenarios and as such clients cannot observe partitions. An important observation is that in larger distributed scale systems, network partitions are a given and as such consistency and availability cannot be achieved at the same time. This means that one has two choices on what to drop; relaxing consistency will allow the system to remain highly available under the partitionable conditions and prioritizing consistency means that under certain conditions the system will not be available.

Both require the client developer to be aware of what the system is offering. If the system emphasizes consistency, the developer has to deal with the fact that system may not be available to take for example a write. If this write fails because of system unavailability the developer will have to deal with what to do with the data to be written. If the system emphasizes availability, it may always accept the write but under certain conditions a read will not reflect the result of a recently completed write. The developer then has to make a decision about whether the client requires access to the absolute latest update all the time. There is a range of applications that can handle slightly stale data and they are served well under this model.
He then goes on to describe one form of consistency, called eventual consistency, where there is a time lag between an update and the ability of all clients to read that update.
Eventual consistency. The storage system guarantees that if no new updates are made to the object eventually (after the inconsistency window closes) all accesses will return the last updated value. The most popular system that implements eventual consistency is DNS, the domain name system. Updates to a name are distributed according to a configured pattern and in combination with time controlled caches, eventually of client will see the update.
I love this -- eventual consistency. I think this is the way things actually work on the large scale in the real world. And when you design a system this way, it allows you to, in a sense, breathe, and you end up with something that is much more tolerant and scalable.

It reminds me of the story of the Buddha. He was trying to find God, and he was so strict and severe in his austerities that he was practically starving. He was feeling lost, wondering why all this effort wasn't leading him to God. Then he overheard a man instructing a student how to tune a stringed instrument: "don't tune the string too loose, or it will make no sound. Don't tune it too tight, or the string will break." And Buddha saw this as a message to him to follow the Middle Path, to relax and breathe while still following a healthy discipline.

In a way, requiring strict consistency is like tuning the string too tight. The constraints are so severe that the system is brittle and easily breaks.

So, Grasshopper, when building scalable systems, may we follow the Middle Path of Eventual Consistency.

Monday, December 17, 2007

Ahhh, vacation

Amazing, I am taking three weeks of vacation. How often does this happen? So, I know you all are hanging on my every word, and will feel deep sorrow at my lack of posts, but, alas, you must carry on without me. Have a wonderful holiday and a happy New Year!

See you in 2008!

Friday, December 14, 2007

Throw out your relational db for all but the tough queries

Amazon Simple DB continues the trend that for many many use cases the relational model and SQL queries, even pure data consistency, are overkill. Don MacAskill on his SmugMug blog has this to say about Simple DB:
Like many of you, I’m sure, we’re using much of our RDBMS as a fairly simple data store and aren’t using many advanced RDBMS capabilities. All of those queries could just use SimpleDB and then we could devote our DB iron to just the rare complex queries. We’re not alone - tons of web apps are gonna love this.
I think I have to agree with him here. The relational model is powerful, but it is hard to learn, hard to do right, hard to scale, tends to sacrifice performance for consistency, and adds a lot of complexity to your application. And more and more people are discovering that in many cases, you just don't need it.

Vinod Khosla: We Can Replace Oil in 20 Years or Less

Earth2Tech blog (a great blog on green tech) gives us a video of Vinod Khosla on removing dependency on oil and coal in 20 years. "Don't listen to what most environmentalists say, because they come up with ideas that can't scale"


SmugBlog: Don MacAskill » Companies That Listen: Sun

When you're working hard and you see all the warts, sometimes it's great to read this kind of fan mail. Made my day.


Build a company with no office and no hardware - no kidding

With Amazon SimpleDB, I think it is now entirely possible to build an massively scalable web-based product without investing in a single piece of server hardware.

Think about that. The up-front costs for starting up a company have never been smaller. You don't even need an office (I'm working at Central Perk this morning).

Sometimes I wish I had a good idea, just so I could have fun building a company like this.

Thursday, December 13, 2007

Solid state disk array delivers 800 MB/s with .1 ms access time

800 mb second

I wrote a while back about a dream laptop that included a solid state drive. These things are compelling because they are damn fast, and use a lot less power than a disk drive. As a developer working with databases and NetBeans (which is very I/O intensive with all the class indexing and module loading it does), a fast disk means a fast life.

So I dream about getting a solid state drive - it's just they're so expensive - $650 for 32GB - ouch. But when the prices start going down -- I'm there!

The other area where this could kick the pants off of things is in the server room. Dominic Strippoli does the numbers on a RAID array of solid state drives. This setup ain't cheap, but with the right controller installed (the standard one just couldn't keep up), he was able to get 800 MB/second sustained read with .1ms access times. Using these drives, he got Windows Vista to boot up in five seconds!

Imagine running your database on an array of these things. Yowza! I hope Sun storage is keeping an eye on this trend...

Wednesday, December 12, 2007

Open source databases: the looming tsunami?

A couple of blogs came across my reader today. The first was by Allan Packer, starting what looks like a series of blogs where he asks the question bound to get him a lot of hits: are proprietary databases doomed?
With the combination of essential features, improved performance, robust support, and compelling price, OSDBs today bear a striking resemblence to Linux a few years ago. Many believe that the wave looming on the horizon is a tsunami.
On the same day in my Dzone database search RSS feed I encountered a link to Brian Hurt's blog, Postgres for the win! (I just noticed it's a 2006 entry, so I guess it's been re-discovered :)).

Brian is doing some serious work with PostgreSQL, and PostgreSQL is succeeding for him
At this point, the only complaint I have is that Postgres is doing so much with so little that I’ll never get cool hardware to play with. I’ll be stuck with cheap low-end boxes forever. Sigh.

The next person who claims “Postgres is slow” is going to get an earful from me. Maybe Postgres 7.0 was slow, I don’t know- but I know for damned sure that Postgres 8.1 isn’t.
If you read Brian's blog, you'll see PostgreSQL is succeeding for him not just because it is fast and can scale, but because he has access, in terms of documentation, community, and the source code itself. It's open-ness gave him the ability to figure out how to solve his problem.

What's my take on this? Well, I think it may happen sooner, or it may happen later, but at some point, the open source databases are going to be just fine for a majority of users. And if you can get enterprise level support, well, all the better!

Tuesday, December 11, 2007

Illegitimi non carborundum

I read a great editorial by Sean Olender this weekend in the Chronicle on the massive fraud taking place within financial institutions around sub-prime mortgages, and how the government is apparently working with the institutions, in private, to basically prevent the instigators of this fraud from being held accountable.

This seems to happen over and over again. Whenever the economy is doing well, we don't hear about these things. But at some point the apple cart falls over, and everyone sees the ugly crawling bugs underneath. And usually the crooks get away, as they are in cahoots with our government. I find myself surprised, yet not surprised, outraged and resigned.

I hate it that government is in the pockets of people with money. I hate it that criminals of the poor classes are punished with a vengeance, filling up our jails, while white collar criminals are regularly let go with a slap of a hand while they walk aways with millions or billions of our hard-earned money. And our government is one of the better ones in terms of corruption and criminal activity.

I feel myself powerless to do anything about all this. I ask myself: what's the point of trying to live a good life if we're constantly being screwed over by the greedy and immoral and power-hungry?

But then I recognize that all this impotent anger really isn't helping me or anyone else. Not that I won't take action where I can, but to reduce my own stress I gently remind myself to step back, and take on the vision of Shiva, the vision of eternity. I remember how ancient and eternal this universe is. I remember that Death visits us all, and does not recognize or pay homage to wealth, power, race or position. I remember that in the game of life, Nature always wins.

From this perspective, it becomes clear to me that the best I can do is live a life in harmony with the eternal throb of Nature. You can feel it in the emptiness of nature - in the desert, or on the ocean, or in the mountains. That deep throb of stillness and eternity.

From this perspective, I begin to see that all of this is happening as a great dance. It's like watching the waves on the ocean, a constant play of shadow and light. And I remind myself: illegitimi non carborundum - don't let the bastards grind you down.

Saturday, December 08, 2007

Tweaks to make Solaris Express work in VMWare

A quick blog to tell you two things you need to do to get Solaris Express to work with VMWare Fusion:

First, if you're using build 79, there is a bug where, during installation, you need to give Solaris just a little bit less than the available disk space. If you don't, the installation fails. This is logged as a P1 bug and hopefully will get fixed by the next build.

Second, after installation, VMWare will complain that you're trying to run a 32 bit operating system in a 64 bit host (or vice versa, I can't remember). To fix this, in add this line to the .vmx file in your image directory:

monitor_control.disable_longmode = 1

Thursday, December 06, 2007

JRuby performance: impressive improvements

Tim Bray pointed me to The Great Ruby Shootout, where Antonio Cangiano does an incredibly detailed analysis of the relative performance of the various popular Ruby runtimes out there.

There is a lot to look at, and I recommend you take a gander, but I wanted to highlight this paragraph:
JRuby is truly impressive, being the only implementation that’s able to pass all the tests without raising errors or timing out. Not only that, but in my previous benchmark, JRuby was several times slower than 1.8.5, whereas now it’s faster than the main Ruby interpreter. There have been huge improvements going on over the last few months. Really, just compare the times in the old table with the ones in the new table: unbelievable progress. JRuby is the only one, aside from 1.8.6, which is able to run Rails in production. I wouldn’t be surprised if JRuby turned out to be, on average, just as fast as Ruby 1.9, in a year or so.
That's some pretty impressive news. Kudos to Charlie, Thomas and the entire JRuby community!

Burningbird » Biofuel and Global Warming

Three cheers to Shelley. The more I hear, the more I don't like. Sounds like a boondoggle for get-rich-quickers. We need truly sustainable energy, not a game of whack-a-mole with our natural resources.


Wednesday, December 05, 2007

Now THAT is the way to learn math

Our daughter is going to a Waldorf School in the East Bay. We heard about Waldorf from a friend of ours who is a teacher, and the more we heard, the more we liked. Ariel is in second grade, and she and we are liking this school and the way it teaches children more and more.

It's based on a very rich philosophy called anthroposophy, which I have barely scratched the surface of understanding. An important principle is that you involve the entire being of the child when teaching them - physical, emotional, mental and spiritual. You don't just sit them at a desk and give them homework and tests. As some examples, they have a movement class every week, and their first exposure to times tables is by doing step dances in a circle, emphasizing the two-beat, the three-beat, the four-beat, etc.

The latest revelation for me was how she is learning her times tables visually, and is getting her first introduction into geometry. Take a look at the beautiful 10-pointed star below that Ariel drew

10 Pointed Star

Note how the circle has the numbers 0 to 9 around it. The way she draws the star is she places her pen at '0', and then says "seven times one is seven" and draws a line to the 7. Then she says "seven times two is fourteen" and draws a line to the 4, and so on. In this way she draws a ten-pointed star, and then she spends time coloring it in to make it beautiful.

In the last week she has done this about six times at home, and this is without having any homework assigned. She just loves to do it.

I find myself really happy for her, and a little bit jealous - I am terrible at math, in part because I struggled with it at school (in particular my times and addition tables) and now I find myself freezing up when trying to do basic math operations. If I had learned math this way, I think I would have loved it...

tecosystems » “Hybrid” Source, MySQL, and the Economics of Open Source

Stephen O'Grady does an interesting analysis on MySQL going "hybrid source" with their Workbench product. I really like the principle of "pay at the point of value." I would think in most cases that's services and support, but let's see if this works...


Tuesday, December 04, 2007

A new look

Those of you who actually go to my blog page rather than read my posts through an aggregator will notice I just did a face lift. I just thought my old layout was a bit bland and nondescript.

I'm not a UI expert and I don't have hours to spend on this, but I pulled a template called "K2" from a site called Gecko and Fly (thanks, guys!) and tweaked the CSS a bit to my liking.

At the same time I changed the name of the blog, I kind of like the new one. Makes me smirk when I read it anyway. Often when people ask me how to spell my last name, I say "it's like you were going to Vancouver." Sometimes people get it, sometimes they don't...

Monday, December 03, 2007

NetBeans 6.0 is out!

NetBeans 6.0 is now available from the NetBeans web site. Congratulations to a huge effort by a lot of people.

I've been working on 6.0 for a while now, and I would have a hard time going back. Some of my favorite features include:
  • Really fast detection of errors while I code
  • Showing all the errors in red and all changes in green on the side bar, and if I click on a red or green mark it takes me to that line.
  • Greying out of unused variables, and color-coding local variables
  • No more two-phase refactoring, unless I request it.
  • Solid Ruby support, including cool things like auto-completion of ActiveRecord methods by using migration data.
  • PostgreSQL and MySQL JDBC drivers included and registered out of the box
In the Database Explorer fixed two things that were real annoyances to me (and others):
  • No more quoting in SQL DDL! (this caused no end of confusion for users)
  • Ability to remember password for DB connections (I was so tired of having to re-type passwords
There were also a lot of things that happened under the covers which perhaps weren't so noticeable, including integration of a number of packs/plugins, including VisualWeb Pack, Enterprise Pack, Mobility. and UML. This took a lot of work, and the amazing thing was it was done in a way that maintained the over consistency and solid out-of-the-box experience that is one of NetBean's real strengths.

So, anyway, kudos to all, and now onto the next release! :)

Friday, November 30, 2007

Steve Jobs talks about the 3G IPhone Slip

Just read it, sometimes Fake Steve is so funny, and dead on in a dark sort of way.


Live Upgrade step-by-step for Solaris Express

I have heard lots of good things about Live Upgrade -- it allows you to upgrade to a new version of the OS while giving you the option to go back to your old version. Pretty cool!

Live Upgrade requires you set up a partition that's the same size as your root partition. In the past, if you didn't do that yourself, you can find out you don't have enough space to do that. I and others gave some feedback about that.

So I was very happy to notice that when I installed SXDE 09/07 it said that if I had given it more than 20GB of space (which I had), then the install would automatically set up a partition to make it possible to do a live upgrade.

Sure enough, when the install was done, there was a mount called /second_root that was exactly the same size as my / partition:

# /usr/ucb/df
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0d0s0 9294575 6742458 2459172 74% /
/dev/dsk/c0d0s4 9294575 9233 9192397 1% /second_root

So, after doing some research on the Web on how to do this, here are the steps I followed. You can also find the official instructions here.

Back Up your VMWare Image File

I didn't need to tell you to do this, did I?

Download the Image for the new OS

There are weekly builds of OpenSolaris called Nevada or Solaris Express Community Edition (SCDE) that have not gone through the same level of validation as Solaris Express Developer Edition (SXDE), which comes out about once a quarter (next one due in January).

So if you're brave and want the latest and greatest, get Nevada/SXCE. The latest one is build 78, with build 79 coming out some time next week. The nice thing is with Live Upgrade you can turn back the clock if you Otherwise, get the latest SXDE.

Mount the Image

Once you have the image, you can mount it into your current Solaris image using VMWare.
Choose Virtual Machine->CD/DVD->Choose Image... and then browse to your .iso file.

Upgrade LiveUpgrade

You heard me right. Before you can do Live Upgrade, you need to install the latest version of Live Upgrade. This seriously bit me in my last attempt about six months ago. It makes sense when you think about it, but, you know, hindsight is 20/20...

Find out where the CD is mounted to (it's usually under /cdrom), and then run something like the following:

# /cdrom/sol_11_x86/Solaris_11/Tools/Installers/liveupgrade20

and then let it do its thing.

Create a Live Upgrade Unit

is going to contain a copy of/. This is what will be upgraded, and then you will "switch" roots when you want to run in the new operating system. If you want to go back, the old one is still there, and you can "switch" back. Cool, huh?

So, first step is to make a copy of your root partition. This is done with lu_create.

First, you need to get the device name for /second_root. Do a df to find out. From above, you can see mine is c0d0s4.

Take a note of that, and then unmount /second_root and comment it out from /etc/vfstab. This is important, lu_create won't work if it thinks /second_root is in use.

Once you've done that, run the following command

# lucreate -c sxde907 -n nv78 -m /:c0d0s4:ufs

Where sxde907 is the name you want to give your current OS version, nv78 is whatever name you want to name your new OS version, and c0d0s4 is whatever the device name is for your /second_root partition.

This will run for quite some time, as it is copying your entire root partition. Time to go get a cup of coffee...

Note: keep an eye on things in the Performance Monitor. I noticed code>vmware-guestd taking up all the CPU and lucreate was asleep - if this happens, you can safely kill vmware-guestd, and then things will move along. I think this issue is fixed once you install the VMWare tools (I'm not sure though).

Run the Upgrade

This is done pretty simply. If your new image is mounted in the standard place, it will look like this:

# luupgrade -u -n nv78 -s /cdrom/sol_11_x86

Where nv78 is whatever name you gave to your new partition.

Activate the New Partition, And Go


# luactivate nv78

to activate your new boot environment. This command, when run, provides very useful information on how to get *back* to your current boot environment if for some reason reboot fails, so do take note.

Make sure you fully shutdown, versus "soft reboot" by using the shutdown command or the init command. The simplest way to do this is to do a full shutdown; if necessary (it shouldn't be) shut down the virtual machine using VMWare's code>Virtual Machine->Shut Down Guest.

When you reboot, you'll be in the new operating system version. If you ever want to go back, run luactivate again to activate the old bood environment, e.g.

# luactivate sxde907

Note that the startup menu lets you pick which boot environment you want to run in. Pretty nice!

My only complaint is that it took longer for me to run the upgrade than it did to make a clean install. But I guess that's to be expected, upgrading is a more complicated process.


I noticed that PostgreSQL did not start up in build 78.

Also I got an error saying there was a version mismatch for the 'vmhgfs' filesystem, and the HGFS module could not be started. However, I could see my shared directory no problem.

The screen resolution stuff appears to be kind of broken in build 78. This has been reported and hopefully will be fixed in subsequent releases. It's workable, but not nearly as nice as with SXDE 09/07.

Uncle, Uncle!

My little sister had a baby girl last night. I am so frustrated she is all the way in London! I feel very proud and happy. Her name is Charlotte, after my dear great aunt who recently passed away, and her middle name is Josephine, after our grandmother.

Welcome, little Charlotte Josephine Örnebring.

Thursday, November 29, 2007

VMWare Fusion 1.1 performance - give it some time

I am continuing to happily run Solaris Express Developer Edition under VMWare Fusion 1.1 on my Mac. This is now my primary development environment

I had some nasty performance issues at first. I have a 5400RPM disk drive, and when I was running Solaris, I would experience regular "lock downs" where my computer became almost completely unresponsive while the disk chewed and chewed and chewed.

I finally moved my disk image to an external USB drive, and was much much happier.

Then earlier this week, I tried running my image on my main disk again, and it actually is working great, no issues at all.

I think the issue is that VMWare allocates disk space on demand. So when I was first getting going, installing lots of stuff, pulling down source trees and so on, VMWare kept asking the Mac for more disk space for its image.

But now the disk size has stabilized, and that's not happening any more.

If I were to do it again, I think I might do the following:
  • Put the image on a separate disk initially
  • Write a script to create a big file in my image filesystem, then delete it. This forces allocation of space
  • Move the image back to my main disk.
Alternately, you could just keep the image on the external disk, but that's a bit of a pain.

I suspect if I had a 7200 RPM disk, this wouldn't be so much of an issue. I think I'll put that in as a budget request for next quarter :)

Tuesday, November 27, 2007

SQL Server Express - nothing "Express" about it!

I have been trying most of the day to get SQL Server Express installed on a lab Windows XP machine. I need this to try and reproduce a problem somebody is having with NetBeans and SQL Server.

What a friggin nightmare!

Someone prior to me had installed the beta of SQL Server Express, which now appears to have expired and won't start. So I downloaded the new version, which refused to install, saying that "Beta Components of .Net, Visual Studio, or SQL Server 2005 are causing problems. Please fix this" or something to that effect.

Um, OK... Not very helpful.

I tried uninstalling the old version. Get this - it is installed but it refuses to uninstall because it says it is missing the .Net 2.0 framework. Gads!

Ok, back to Microsoft, download and install .Net 2.0. Same problem. Reboot. Same problem. The @#%@% system tells me .Net 2.0 is installed, but it refuses to uninstall. And the new version refuses to install, same error.

What, are you doing everything you can to make me not use your software??!! Amazing...

For the sake of my customer, I'll have to try this again tomorrow. But if I were a developer, I'd be throwing this thing out the window and moving to Java DB.

Landon Fuller builds a dev preview of Java 6 on Mac OSX Leopard

Wow. Thank goodness for open source. Three cheers for Landon!


Berkeley Mayor Tom Bates on Solar Financing Plan

City financing of up-front costs of solar installation: really cool idea, and, living in Berkeley, one I hope we can take advantage of.


Nati Shalom's Blog: Architecture You Always Wondered About: Lessons Learned at Qcon

Nati Shalom, CTO and founder of GigaSpaces, has a very interesting blog describing the architectures of some of the Big Scale Web Guys out there -- what the patterns were, what lessons were learned, etc. Check it out...


Monday, November 26, 2007

Auto-creating your embedded Java DB database - some working code

A lot of folks are surprised when an embedded database doesn't work the same as a client/server database. In particular, they don't initially realize that it is the application's responsibility to create the necessary tables and load them with any initial data the first time the application is run in a new location. Many of use are used to the database just being out there on a server somewhere and aren't used to having this responsibility.

I asked for a blog or tutorial on how to do this on the Derby user mailing list, and one user, Alan Burlison, was kind enough to point me to his work. And I quote:

No tutorial, but some working code

This runs inside Tomcat and creates a new database the first time it is run, subsequent runs use the existing database. The database runs in embedded server mode, so whilst Tomcat is running you can access it externally as well.

The database is initialised via SQL script files that are contained in the application WAR file, using the ij utility (derbytools.jar). ij can be run from inside an application as well as from the command-line, and ij + SQL scripts is the easiest way I've found so far of setting up a database, because the SQL scripts can be debugged from the command-line before being embedded in the app.

More info on PostgreSQL in Solaris Express

I got a tip from one of the folks on the PostgreSQL team here at Sun. There is a lot of very useful information on Postgres 8.2, which ships with Solaris, through the man page.

So try

man postgres_82

And have a read. It tells you a lot more useful details than what I provided, and can prevent you from getting a bit lost and confused trying to set this up.

Photos from Antarctica

Penguin photo

My brother and Dad are back, and the first set of photos are in!

My brother is a wonderful photographer, and these shots attest to that.

Monday, November 19, 2007

NetBeans, Ruby on Rails, and PostgreSQL on Solaris

I needed to track down an issue with PostgreSQL and the Ruby on Rails support in NetBeans today, and in the process learned what you need to do to get PostgreSQL working on Solaris with NetBeans.

There are two approaches you can do. You can use the Cool Stack Ruby on Rails package, which is optimized for Solaris. Or you can use the JRuby/Ruby On Rails support that comes with NetBeans.

These instructions are for NetBeans 6 RC1, and assume you have installed the bundle with Ruby support or installed the plugin from the update center (see Tools->Plugins).

In either case, you need to install the Postgres driver gem for Ruby. This is easily done, just go to Tools->Ruby Gems, select the New Gems tab, search for "postgres" and pick "postgres-pr". This is the Ruby implementation. I had trouble installing the native implementation, I think I'm missing the postgres header files it needs.


If you want to use JRuby and the Rails that comes with NetBeans, you're ready to go. When you create a new Ruby on Rails project, pick "postgres" as the database.


Now, there appears to be a problem with postgres-pr and JRuby, when it tries to use the Postgres UNIX file socket to connect to PostgreSQL. So edit your database.yml file and uncomment the following lines:

host: localhost
port: 5432

With that, you should be good to go.

If you want to use the Cool Stack implementation, first download and install the Ruby package. Then you need to point NetBeans to the CoolStack Rails implementation. Go to Tools->Options, and pick the tab for Ruby. Point Ruby Interpreter to the location for the interpreter, which by default should be /opt/coolstack/bin/ruby.


When using Cool Stack Rails, you can use the UNIX socket, so you don't need to modify your database.yml file.

For more information on Postgres and Ruby, see the Ruby Wiki page.

Starting PostgreSQL as a service in Solaris Express

I just needed to do this, so I thought I'd document the fairly steps to set up PostgreSQL as a service in Solaris Express Developer Edition version 9/07

PostgreSQL is already installed in Solaris Express, and it's set up as a service, but it is disabled by default. Once you have this set up, then PostgreSQL will always be up and running, even after reboots.
  1. Log in as the postgres user, which is pre-created as part of the install of SXDE. I didn't know what the password was, so I had to log in as root first, then change the password using passwd postgres.
  2. Fix the PATH environment variable so that /usr/postgres/8.2/bin is at the front. The default Postgres commands in /usr/bin are for version 8.1 of PostgreSQL
  3. Make sure the directory /var/postgres/8.2/data exists; if not, create it
  4. Initialize your database using the command initdb /var/postgres/8.2/data
  5. Log out of the postgres user
  6. Make sure your default user has rights to manage services. Go to Administration->Users and Groups, select your user, and choose Properties, then select the User Privileges tab, and make sure either System Administrator or Service Management is enabled.
  7. Enable the service using svcadm enable postgresql:version_82
  8. Run svcs -x. If it shows postgresql, there was a problem, and it will tell you where the log file is to see what went wrong. Run svcs | grep postgres and it should show it as enabled.
  9. Create a user in Postgres that matches your current user, so you can log in to the database as that user by running createuser userid. Alternately you can run createuser postgres and log in to the database as postgres.
  10. Verify you can log in to the Postgres server by running the psql command.

Is NetBeans 6 ready to roll? You be the judge.

Survey Says!

For this release of NetBeans, we have a survey that users of NetBeans can fill out to tell us whether the release candidate for NetBeans 6, labeled RC1, is fit for production.

Check out this survey link and tell us what you think

Wednesday, November 14, 2007

Putting a face on Mr. D: NetBeans and SunStudio plugins for DTrace

DTrace has won accolades within the Solaris community as delivering something nothing short of miraculous for understanding the root cause of performance issues without becoming a bottleneck to your system itself.

But DTrace is a command-line interface, and one thing it needs is a UI.

Well, the tools teams have been working hard to make this happen. In the latest release of Solaris Express Developer Edition, Sun Studio comes with a plugin called D-Light that helps you visualize DTrace probes.

Nasser Nouri has also delivered a DTrace plugin for NetBeans which looks quite nice.

Sun also provides DTrace support in their optimized AMP stack.

So this is all goodness for those out there who want to deploy AMP solutions to a platform that gives you the tools you need deep observability into your application so you can find and fix performance issues quickly.

Tuesday, November 13, 2007

Solaris Express Dev Edition on VMWare Fusion: It's a Go!

I just finished installing Solaris Express Developer Edition 9/07 on VMWare Fusion 1.1 on my MacBook Pro.

Well, I don't know what to say. I just can't think of any complaints. The install was simple and painless, I was able to install the VMWare Fusion tools without a hitch [1], networking Just Works, Firefox Just Works, NetBeans 5.5 Just Works, including Java DB is Just There and Works. File sharing works. Navigation between Solaris and Mac applications works.

I have tried many early installs of Solaris, and I have never been in this baffling situation of having absolutely nothing to complain about (and believe me, I don't keep my mouth shut).

I can go to full screen mode, and it stays in full screen even when I suspend and resume (unlike Ubuntu under Fusion).

I'm downloading Ant and am going to give a NetBeans build a go. We'll see how it goes; I am getting myself ready to shift over to a real developer's platform, that actually has Java 6 and a working version of Open Office and top-level AMP and PostgreSQL support. I'll keep the Mac around for consumer-oriented stuff like playing Rhapsody, but I think in general it's going to start moving to the background of my day...

One other nice thing about running Solaris in Fusion on the Mac -- I can suspend my computer. If you have Solaris installed natively, no suspend. They're working on it, but meanwhile, this is a pretty good workaround.

[1] Well, just one hitch, per Felipe's blog. After installing and you reboot, VMWare thinks you are booting a 64 bit OS in a 32 bit guest. You have to do add the following to the .vmx configuration file:

monitor_control.disable_longmode = 1

After you successfully boot this way, VMWare recommends you remove this line as it can confuse some applications.

Thursday, November 08, 2007

DC PHP Conference Takeaway Thoughts

Well, I'm heading back from the PHP conference in DC. This was a simple, small, conference, but I came away from it quite impressed. The speakers were good, the organization was informal but solid, and (this is something quite unusual) the food was quite good. If you're interested, they're already planning to have the next one in the same venue (George Washington University campus of all places, but actually quite cozy) in early June of 2008.

I'm not a PHP programmer myself -- the main goal of my visit was to gain a better understanding of "who" a PHP developer is (if you can make such broad generalizations) and what their priorities are for database tooling. So I spent a lot of time at lunch and breaks chatting up my table buddies and asking them lots of questions. Sometimes I'd even pull out my survey and walk through it with them.

Many of the developers I met are the single "IT guy/gal" in their department. They had more computer skills than the rest in the team, and just pulled out the web and PHP books when they needed to build something. Over time they did this more and more and all of a sudden they were doing PHP for their day job.

In tems of tooling, a vast number of them use and love phpmyadmin (duh, say some of you, I'm sure:)), so I am definitely going to take a look at that.

In general, the primary database tool they care about is SQL query editing and execution. There are frameworks out there like PDO that hide the SQL from the developer, but this is used by a very small minority (I couldn't find anyone using PDO).

Another theme I heard was that Entity/Relationship (E/R) diagrams are useful, not as a design tool, but primarily as a communication tool. Developers need to understand existing schemas, and also communicate their schemas to the end users. There was a lot of head nodding around a feature to allow you to print an E/R diagram to PDF to make it easily distributable over the web and email.

I also particularly wanted to know how important visual query building is. This is a feature that lets you build a query by dragging tables onto a pallette, selecting/deselecting columns, adding filters, joining tables with a line, and the resulting SQL query is built for you. The strong answer I got here was "ah, that would be nice, but I don't really do that. I mostly just hack SQL on the command line."

Many folks don't use an IDE and don't know what value it could have for them. One of them even said "there was this one tool I used that color coded my code for me and when I typed a letter it would show me all methods that started with that letter, man that was so cool!" So I could see that if they get started on an IDE, they would really love it.

However, one thing I noticed is: these are keyboard guys, they want fast response time and fast round trip. They will be turned off by an IDE if it feels "heavy." Luckily NetBeans is modular, and we can provide a PHP bundle that is quite lightweight and should have a reasonable startup time.

Another interesting theme I gathered was that many PHP developers don't have an entire LAMP stack on their machine. They work in the office, connected to the network, and use remote access to their server to deploy and test, even during development.

I need to spend some time thinking about the implications of this for database tooling. In particular, if the database server is only accessible through HTTP, then all our database tooling that requires a direct database connection is not going to be of much value... I can see why phpmyadmin is so popular, because it is web based...

I could imagine building a set of REST services for database tooling like schema metadata management, query editing, and so on, and access these services through NetBeans, allowing you to manage a database over the web through NetBeans. Pretty cool, now that I think of it...

Java One 2008 : Do you have something to say?

Now is the time to submit your presentation or BOF for consideration for Java One 2008.

I submitted one for a BOF demonstrating the upcoming database tool features for NetBeans, and a technical session on trends in persistence and databases. We'll see if they get in.

What about you? Do you have anything you want to say or share with the Java community? Now's your chance.

My father and brother are off to Antarctica

Photo of Ushuaia

I'm not sure what inspired them to do this, but they are off to the Antarctic. Intrepid family. Here is a missive from my father just before they head off from the tip of Argentina.

I know I am supposed to be in Antarctica, but that's tomorrow. Today I am in the southernmost hotel in Argentina and it has free internet. So instead of catching up on my sleep as Tony is doing over there, I'll pop this note into the virtual bottle and float it out on the digital ocean for you guys to find.

I've never been in Alaska but it has to be like this, with long deep fjords winding past snow- capped mountains. Ushuaia, where we are, was probably a little fishing town 30 years ago. Now it's booming, from the tourist trade, with outfitters and souvenir shops and fancy cafes, block after block, with hotels and pensions up every side street. The main street is full of cars, though it's easily 100 miles by dirt road to Patagonia and civilization.. It looks somewhat like Aspen and somewhat like Anchorage. There are 11 flights a day from Buenos Aires, bringing kayakers and trout fishermen and hikers, plus planeloads of people coming to meet their Antarctic cruise ships. And other cruise ships, doing the round South America thing, drop anchor every day. Steak and wine are cheap - $15 for a big porterhouse - but most prices are similar to New York.

We are on the north side of the Beagle Channel, first mapped by the ship that took Charles Darwin to the Galapagos. South of us, the snowy peaks are in Chile, which wraps around the southern tip of Tierra del Fuego. We will leave by sailing east, not south, until we clear the channel.

Speaking of suddenly growing towns, there is not a stop sign, let alone a signal, anywhere in Ushuaia. As a result there are no accidents, since everyone is scared to death. This reminded Tony of the Mr Barnes who founded the amazing Barnes Museum of modern art in Philadelphia, said to be on money he got from stealing patents. "How do you think he died?" asked Tony. Bad chemicals? Outraged fellow chemist? "Nope. He ran a red light." And? "And, served him right. He never stopped for red lights in his whole life. He grew up without them and he objected when they started putting them up, so he ran them all to express his disapproval. He got away with it until he was well into his sixties."

Today, us early arrivers - 10 in all - hired a van to take us out to Harberton Ranch, the first ranch ever established in Tierra del Fuego. It is a piece of about 5,000 acres given to a missionary by the Argentine government in honor of his service with the surviving Indians (the tribe of near-naked savages who went around with snow melting on them, and from which Darwin abducted or borrowed the man they named Jeremy Buttons, who went back to England and was a sensation until he caught a cold or something and died.

[ed note: there is a great book called To the Edge of The World, a wonderful historical novel all about Darwin's visits to Tierra Del Fuego, including Jeremy Buttons and the Fuegian Indians. Highly recommended.]

It is still in the family, but the present owner, a goofy-looking but nice guy named Tom Goodall, had the good luck to marry Natalie, a girl from Ohio who came down to study Fuegian plants. Which she did, but she also liked to pick up whale bones and skulls along the beaches. One day a marine mammal guy from the Smithsonian came by, and she invited him to look at what she had. Oh, OK, I guess -- but when he saw them he nearly dropped dead. Among the 60 or 70 skulls were several that were known by only one other specimen, and there was one that was new to science. Long story short, she switched to marine mammals, started beach combing in earnest, got money from Total oil company to build a museum, and now has a collection of whale, porpoise and dolphin material -- in this little private museum sitting at the end of a dirt road 35 miles from Ushuaia -- that is considered the second most important in the world.

If you look at a world map you can see why. Tierra del Fuego sticks down into the Circumantarctic Current like a fish-hook, and everything that dies or is sick in the whole Southern Ocean washes up on its eastern shore. Tom's brother has a ranch up there, and she and her students go up and look for stranded dead whales, and whale bones. She is now stout and white-haired with bad knees and breast cancer, but we got a three hour tour thanks to Darrel, who organized our trip and who had been to Antarctica 40 times as an expedition leader (Enid and I met him on an Indian Ocean cruise, however). For instance, she has skulls of five differerent species of beaked whales, and nobody else has more than 2 or 3. It was totally amazing. With Ushuaia booming, they now get busloads of tourists every day and are building a new restaurant on the hill above the museum. They also run boats out to an island they own, where gentoo and magellanian penguins make nests in burrows among the grass.

We went out there, and walked carefully along paths marked out between the penguin nests. It was just like the Galapagos - they just looked at us, bored. Not frightened at all. But I about had a heart attack when a huge skua gull leaped into the air and hovered in front of my face with her sharp beak gaping in threat -- I had come just a little too close to her egg on the ground.

What a great start for what looks like a great trip! I will keep collecting notes, but this may be the last from us for a while.

Love to all

Wednesday, November 07, 2007

My Mac becomes a VMWare host

As promised, I'm moving away from Mac as my primary working environment. I need Java 6, it doesn't have the best integration with the AMP stack, which I need to work with right now, and so on.

But I'm not fully giving up on Mac OSX. It is now my hosting environment for VMWare Fusion. I am typing this blog in fullscreen mode in Ubuntu under VMWare. I am realizing that for many a computer could easily turn from a single computer to a hosting environment for many computers. This is especially useful for developers, because it gives you flexibility, freedom, and the ability to easily test on many platforms. A real wave of the future.

I installed Indiana, and I am working on installing Solaris Express Developer Edition. I'd like to work with Solaris - I like ZFS and I remember how blazingly fast Solaris is, particularly with disk I/O. Indiana had an incredibly nice install experience, it's actually hard to believe after previous experiences with the Solaris installation. This is just an amazing, amazing improvement and the team deserves very high accolades.

However, Indiana doesn't have things fully integrated yet (it's missing Java, PostgreSQL, MySQL, and so on). I tried using the new package management system but nothing seemed to be available. I'm going to keep working on getting this set up, but I'm not quite ready to dedicate my day-to-day working life under Indiana. Indiana looks very promising and I think ultimately that's where I'm going to end up.

Next up is Solaris Express Developer Edition. I've heard the install is also vastly improved, and it has a lot more things integrated, including NetBeans. So I'll let you know how that goes.

Oracle's big presence at PHP Conference: Can you say LOAP?

Oracle has a very strong presence at this PHP conference. They are a "Presidential" sponsor, had a keynote, and handed out a slew of CDs in the conference tote bag. As a matter of fact, their stuff was the only real thing included in the tote bag.

Let's see, what do we have here, let me open the box...
  • Four (count them ) 4 copies of Enterprise Linux 4 - Oracle's supported distribution of Enterprise Linux
  • Enterprise Linux 5 and an associated source CD
  • Oracle Database 11g for Linux
  • Database Developer CD for Linux
  • Java and PHP Developer DVD (Preview)
At the keynote, they talked about their free version of Oracle that you can embed in your application and "you don't even have to tell us" (this is included with the Database Developer CD).

To me this means that Oracle definitely notices that there are a growing number of apps built with the LAMP stack, and I suspect they are trying to make a presence here -- they want to see more LOAP (Linux-Oracle-Apache-PHP) and less LAMP.

It's funny how they are trying to associate themselves with the open source community without actually going open source. More than once I have heard Oracle reps assert that the value of open source is free, and that since they now have a free version, all things should be equal.

As I have mentioned before, I think they are missing the point. Or they get the point, but are not in a position business-wise to say publicly that they get the point. Their best and only possible response to an open source threat is to make a (slightly crippled) version of their software available for free and market it to developers.

Solaris was in the same boat with Linux, and it was suffering. But we actually took the plunge and made Solaris open source, so now it really is a level playing field.

Oracle's cash cow is still database revenue. So, as Nitin Borwankar predicts, they cannot mess with that business model, and this is limiting their options to how they react to open source databases.

Perhaps some day more of their revenue will shift to their application stack, support and services, and they could potentially sacrifice the cow and open source their database and make the fully functional version absolutely free.

They could start charging at the point of value - when the user purchases services and support for their mission-critical production application. But right now they appear to be staying with their existing business model, and are doing their best to stop or slow the open source database juggernaut. We'll just have to see how that strategy pans out...

At PHP Conference in Washington, DC

Here I am in Washington, DC at the PHP developer's conference.

It is a very small conference compared to say JavaOne and OSCON, but that makes it quite cozy and friendly. There are maybe 200 folks here. It's quite surprising, given how many PHP developers there are, and this appears to be the only conference in the USA besides the Zend conference that just happened.

I took the red-eye last night (first time on Jet Blue, very nice!), rented a car, and got immediately stuck in nasty stop-and-go traffic coming out of the Dulles airport. I had rented a GPS navigator, so I found the closest Metro station, parked, and did public transit into the city. I am not a car driver (I take BART to Menlo Park twice a week into Menlo Park and love it). I'll pick up my car this evening and take it to my hotel in Alexandria ($100 a night, thanks Priceline! Sun Travel couldn't find anything below $400 a night).

What, might you ask, is a Java programmer like me doing at a PHP conference in DC?

Well, I want to find out more about the PHP community, and start learning what we might be able to do in the database tooling area of NetBeans to make NetBeans useful/attractive to PHP developers.

It's not my area to build a good PHP editor, but one thing we could do is make it very easy to work with databases in PHP. But I need to find out what PHP developers care about around databases. In some ways I suspect it's what all developers care about: schema management, querying and data management. But there are subtleties, like how important is a visual query tool versus text-based query tools. How important are ORM APIs like PDO and frameworks? What kind of caching is done, if any, with things like memcached (good results from coming to this conference, already, I just found out about Facebook's open-source PHP caching technology called APC). Maybe it would be really cool to do in-place query editing, where you get code completion and syntax highlighting for SQL right inside the PHP editor, but would anyone care?

I did a survey around these kinds of things, but so far only 12 PHP developers have responded. So I'm also trying to recruit more PHP developers to fill in this very short survey. If you're a PHP hacker, please fill in the survey, it takes 10 minutes maximum.

In the meantime, as the conference moves along, I'll report back to you anything I find interesting.

Friday, November 02, 2007

CAFEBABE: A vote for Java 6 on the Mac

In an attempt to have our voice heard, let's spread the word. If you want Java 6 on the Mac, or at least have Apple tell us when they plan to deliver it rather than be silent, then post a blog with this magic cookie:
(13949712720901 is "CAFEBABE" in hex, the magic cookie for Java class files).

Search for this string using your favorite search engine, and you can see how many votes there are.

This apparently had great impact with Nova Spivak's effort to bring attention to the plight of the Burmese people, although I do find the Java 6 on the Mac issue a little less important than the suffering and repression of an entire country.

Thanks to Henry Story for the idea and for kicking this off.

And thanks to Fredericiana for the fun photo.

'Project Indiana' OpenSolaris Preview Debuts

A binary distribution of OpenSolaris with Linux Userland CLI support and a spanking new network-based package management system. Now *that* sounds cool!


Wednesday, October 31, 2007

Derby vs. SQLite - one user chimes in

I haven't used SQLite enough to compare it in any cogent way with Derby. But recently there was a thread on the derby-user list asking to compare Derby with SQLite. A lot of interesting answers, but there was one from Ryan Bobko who apparently has used both, and he had some points to make that I thought I'd share.

Personally, I think there's no comparison between Derby and SQLite.
Derby's an actual database with "all" the bells and whistles, but still
a really compact size. SQLite is an extremely fast database-like system
with a much smaller subset of features and SQL compliance. Plus, if
you're writing in Java, go with Derby. If you're writing in C or C++,
give SQLite a run-through.

Off the top of my head, SQLite doesn't support foreign key constraints,
or use column types (everything is a string, unless it's an int, which
is actually a string). In the quirks department, I've noticed join order
can have a dramatic effect on performance. What's really nice is that
the whole database is a single file, which makes using it as a save file
in your application really nice. Also, startup times are zero. I think
Derby takes a second or two to startup. Both systems support
transactions. Derby can be used in a multi-user mode, while SQLite is
strictly mono-user.

I know both databases claim to be zero-administration, but I'd say
SQLite more serious about it. I don't even know how to configure SQLite.
Derby certainly works great without administration, but there are a
whole lot of options you can muck with if you like.

Monday, October 29, 2007

No Java 6 on Mac Leopard - time to plan a move

I had been waiting patiently for the new version of OS X, code-named Leopard, since early this year. Why? Because I need Java 6. Now I find out that it's not there, and no work from Apple when it will be there.

So, I sigh, and consider that Apple is abandoning the Java developer base, and I am seriously considering making a move. I think I'll start with putting Solaris Express Developer Edition under VMWare Fusion, which sounds like it's just getting better and better (you can get a pre-installed image here).

If that works well, I may set up Solaris under BootCamp and start using Solaris as my primary platform. Even if suspend still doesn't work. At least under VMWare if you suspend your Mac, you suspend Solaris too...

Thursday, October 18, 2007

NetBeans Database Tooling - Show Us The Way

I just put together a very quick survey where you can provide us some guidance for database tooling features for the next release of NetBeans. The survey doesn't cover all the features (thank goodness, talk about overwhelm), but the key ones where I'm not sure whether they are important or not.

Your input here is key, very valuable. It takes about ten minutes. Click here and tell us what you want.

Monday, October 15, 2007

Quiet Time, and, how to suppress derby.log in Apache Derby

You probably won't hear much from me in the next few weeks - big deliverable. But I'll Be Back...

A quick tip from the Derby user's list, that I thought was very useful, particuarly in embedded situations where you don't want to leave detritus around when you're running Derby.

A user asks:

I'm about to distribute my java app with an embedded derby database,
but it's really not polite for apps to write unsolicited files to
disc. Anyway of suppressing the derby.log file?

Knut Anders from the Java DB team has a great suggestion:

You could take a look at the derby.stream.error.field property:


If you create a stream object which swallows whatever is passed in to
its write() method and put it in a public static field in the class
MyApp, like this

public static final OutputStream DEV_NULL = new OutputStream() {
public void write(int b) { }
then you can suppress the derby.log file by setting
   derby.stream.error.field to "MyApp.DEV_NULL".

Friday, October 12, 2007

I, Cringely . The Pulpit . Oh, and We Also Saved the World | PBS

Bob Cringely pulls together a very interesting story about Google and high-flying wind power. I don't know if I can believe his numbers, but they are very compelling: 1/2 cent per kilowatt, and only needing 3,600 sq miles of land to power the entire USA.


Thursday, October 11, 2007

A very personal view on the costs American health care

Photo of my Aunt

Recently my aunt passed away after a long bout with breast cancer, the youngest of six brothers and sisters in my Dad's family.

My cousin, my aunt's elder daughter, just sent this out to the family list. I think her message speaks for itself.

I want to say Thank You to all of you. You have been so supportive over the past 5 years and there are no words that can express those feelings that I have. This is truly a difficult time for me and my family.

My mother never had a mammogram until she was diagnosed. There were times, mostly, when she did not have medical insurance. She was afraid that if they ever found anything, she would not be able to get insurance and be treated properly. And if she had to be on county care, she wouldn't get good care.

Does this sound right to you? If you ever have an opportunity to make a difference in the policies that this country has, do it. That is the single most frustrating thing that I have dealt with through this ordeal. If she had had a mammogram at age 50, she would have caught this cancer early, and we would still have her around now. She had those tumors, TWO, for 10 years.

Thank you again for all of your kind words and support. They do not go unappreciated.

Monday, October 08, 2007

Heck with SQL: is a persistent hash all you need?

Jamie Flournoy compares ActiveRecord to VB: "The easy stuff is easy, but the hard stuff is impossible," he quote James Gosling as saying about VB.

One of the things Jamie says in this very long, well-thought-out article, by someone who obviously has experience with both Rails and databases, gave me pause.

Jamie referred to some of the leaders of the Rails community saying that SQL is a design mistake and that most web apps only need a persistent hash.

This statement follows shortly after me reading an article by Diego Parilla about the impact of ORM on the database, and finding out that Amazon uses basically a highly available, persistent hash for much of its database storage on the web tier.

Michael Stonebraker is even saying that relational databases can not meet everybody's needs any more, and talks about the need for specialized data stores for specific use cases.

I actually like a lot of this. We need to simplify, simplify, when it comes to building web applications - I've been appalled at what a huge mess it is for quite some time, and have found myself avoiding building web apps if at all possible because of this. Also, SQL ties you to a database (yeah, I know, it's a "standard"), and also ties your application to a particular incarnation of the data model in the database. So for all these reasons, eliminating the need for SQL is goodness.

My concern is that if you're not careful, you start with a hash table, but then start implementing your own database on top of it. Jamie mentions this, and I have seen this too:

Satisfying queries is the database’s job, period. It’s just hideously slow to try and do an inner join in the application across a network link to a database. If you find yourself doing this, that’s a pretty good sign that your architecture is broken.

It's easy to think that all you need is a hash table, and when you're building simple, basic web apps, that's probably all you need. Heck, that's all Amazon needs, for the most part. But there are times when you want more. You really do need to do a complex query, or a stored procedure, or text search. You want to be able to run useful reports without having to do joins and sorts within your application.

I think Rails is getting a lot of things right by hiding a lot of complexity from you (as do the Java ORM technologies).

But I agree with Jamie that for those times when you need SQL, it's nice to have the option -- at least until ORM gets so friggin' smart that writing in SQL is like writing in assembly - sure you could do it, but who wants to, and why?

Bob Cringley, the inspiration for FON

I didn't know this. Bob Cringely wrote an article on WiFi sharing that inspired the founders of FON. You never know where your next idea will come from...


Recording Artist: Don't be a ZFS Hater

Drew Thaler, an ex-MacOS filesystem engineer, has some very positive things to say about ZFS. "ZFS is a fine candidate to replace HFS+ eventually. It's not going to happen overnight, no. ... . But several years from now? Absolutely."


Friday, October 05, 2007

Out-of-the-box database notifications: JMX and Derby in Java 6

A very interesting article that shows how you can use Java DB's Java procedure mechanism, triggers, and JMX to build a very nice change-notification system to detect changes to your database.


Thursday, October 04, 2007

Where are my database tables in NetBeans?

All right, another common question, and this one (understandably) can cause a lot of frustration: where the heck are my tables?!

So I think it's time to record some common causes of this. I'm actually finding these 'tips and tricks' blogs very helpful because when the question comes up again I can just point to my blog and save my fingers some typing...

A lot of users create tables in their database, and then are flummoxed when they can't find them in NetBeans.

Here are some possible reasons why this can happen:

Bad Metadata

Older JDBC drivers, particularly Oracle drivers, are notorious for not handling metadata correctly. NetBeans reports exceptions in the log when it can't get metadata, but you may not have noticed that. The visible result is you see no tables when you open the connection, even though you know you created thoses tables. Head banging ensues.

So, make sure you upgrade to the latest JDBC drivers and register them with NetBeans, and see if that helps.

Two Different Schemas

Make sure you're in the right schema. If you create a schema and then create the tables under that schema, they may not show up by default in NetBeans. To fix this, in the connection dialog for NetBeans, choose [Advanced] and then select the correct schema.

Two Different Databases with the Same Name

This is possible particularly with Java DB. Java DB will create databases directories in the location where you start the VM which hosts the database. If you connect to Java DB using the embedded driver, that's the VM of your tool.

So if you use one tool to create your tables, and then use NetBeans to look at the tables, you may actually be looking at two different databases, and so in NetBeans the tables won't be there.

Let's follow this scenario more closely so you can see what I mean:

- You use a tool to create tables. You use a URL like "jdbc:derby:mydb;create=true". Note that if the database does not exist, it is created automatically. The database is created in the directory where your tool was started.

- Then you open up NetBeans, and you connect to your database using the same URL. But NetBeans is started in a different directory, and a different database is created

- You browse the connection in NetBeans, and your tables aren't there! Bang bang goes the head.

The best way to solve this is use an absolute path to the database when you specify the database name in the URL. So instead of "jdbc:derby:mydb" use "jdbc:derby:/the/path/to/my/db/mydb" or "jdbc:derby:C:/the/path/to/my/db" (note that you use forward slashes even for Windows)

Update: Some great comments from my Dear Readers, who are respectfully correcting my proposed solution. Take a look at them if you know what's good for you :) But to summarize:

  • an alternate to absolute path is setting derby.system.home (or, in NetBeans, setting the database directory, see my previous post on how to do this). Absolute path doesn't always work, and particularly is a problem in production deployments where you don't control your environment
  • don't use "create=true" after you've created your database, to avoid creating databases willy nilly (although this approach eliminates some usability where in embedded deployments you don't have to worry about the database not existing)
  • if you don't know where your database is, search for it
Anyway, my readers say it better than me, so read their comments.

Be productive: stop working

I've been reading a great book called The Power of Full Engagement. These guys worked with top performing athletes, analyzing what makes the best the best, and they came up with a very important conclusion: the best took breaks and re-energized themselves, even if it was a quick pause between serves in tennis. And now they are applying this to the rest of us - what makes us the best performers?

They found that there is something powerful in stopping, getting off the treadmill, taking some deep breaths, walking around, and do anything but think about work. This has an almost magical ability to untie knots in our mental processes and show us new ways to look at a problem. We all know this at some level, but this book has an impact because it's based on solid research -- you just can't argue with it.

I read this book when I was in the middle of a big push at work, and it seemed that the harder I worked, the longer it took to get anything done. I was also exhausted and grumpy, not enjoying life. We all know this mode. My old boss used to call it the "death march."

So I adopted the principles: I made myself not work insane hours, and tried to take more breaks during the day. And I became much happier and much more effective.

Recently I've also been in physical therapy for my lower back, after a very nasty episode where I was in bed and couldn't move for three days. And guess what one of their practices for me is, this time to help me keep track of my physical posture during the work day: take breaks.

Oh, and there is my failing eyesight. One of the key tips to help the eyes not get over-strained -- you guessed it: take breaks.

OK, OK, I get the point. My problem is, I forget to take breaks. I get so completely engrossed in my work, especially when I am coding on some cool feature, that I lose track of time.

I knew they must exist, and sure enough, I found a very nice tool for the Mac called RSIGuard. This tool tracks your mouse and keyboard activity, and then reminds you to take a break. During the break, it shows you little videos of stretches you can do to reinvigorate yourself and avoid physical injury like RSI. You can fine-tune how frequently and for how long you want to take breaks. They also have these nice little "microbreaks" that kind of "wake me up" for a few seconds, and I take a deep breath, adjust my posture, and move on.

So, I highly recommend it. If you are a real bulldog, you can even have it enforce your break by locking up your keyboard and screen. I don't like working that way, I just get pissed off. I like a more gentle approach, and I am a good doggie and take my breaks. And you know what, I really like it.

Off for a cuppa decaf and a walk around the cafe...

Amazon's Dynamo - Serious Non-Relational State Storage

This is getting a lot of buzz, but for us database folks, a very interesting and detailed analysis of a different way to store and query state that meets requirements not met by relational databases.


The Future of Web Startups

Paul Graham has a great essay that captures a lot of the trends we're seeing with startups and where he sees it all going. Easier, cheaper, standardized, more of them, and who needs college?


The Inevitable March of Recorded Music Towards Free: Tech Crunch

I have tried telling this to friends, and they think I'm kind of crazy. But I do believe it is inevitable. Maybe we'll also go back to the good ol' days where everyone sang each others' songs without having to pay license fees.


Wednesday, October 03, 2007

How to change the host and port for Java DB server in NetBeans

This issue has come up a number of times on the mailing list, so I thought I'd post it here for posterity.

The question is: how do you configure the Java DB Network Server, like set its host and port number, for the server that NetBeans starts and stops for you?

The sad answer is: you can't from the IDE. There is a bug logged for this.

However, you can do this by editing the derby.properties file.

First, find out where your database directory is. In NetBeans 6, go to Tools->Java DB Database->Settings and see the value for Database Location

For NetBeans 5.5, go to the Preferences/Options panel for NetBeans, choose Advanced Options, and then under IDE Configuration->Server and External Tools Settings->Java DB Database look at Database Location.

Under that directory, edit derby.properties. The file should be there; if it's not, create it.

Then add

# Change host setting so you will accept connections from any host

# Change port number if you don't want to use the default of 1527

Then for both NetBeans 5.5 and NetBeans 6, do Tools->Java DB Database->Stop Server and then Start Server.

The new properties should now be applied.

Friday, September 28, 2007

Google Search is Down

Well, some things you always take for granted. In the six-plus years I've used Google Search, it's never been down, but for the past ten minutes http://www.google.com is not responding. Thinking about how many layers of redundancy they must have, I can only imagine what nightmare of cascading failures caused this.

Update: Google Reader appears to be affected too, I get nothing but the page header...

Wednesday, September 26, 2007

Why programmers hate relational databases: Greg Jorgensen

Greg Jorgensen writes a fun blog about the common complaints programmers have against relational databases and his responses to them.

Most professional programmers know how it feels to see an amateur, unfamiliar with Knuth or any programming book containing equations, implement their own sort routine. That’s how people who understand relational theory feel when they see a badly-designed database. Relational theory and RDBMSs are old and well-established now, so it’s hard not to think a lot of programmers are willfully ignorant.

I have seen this too: "I don't need a friggin' relational db, just give me a flat file." Sometimes this works, but all too often I see the same programmers adding locking semantics, querying capabilities, indexes, and so on, and I just shake my head.

Rails and JPA (Instead of ActiveRecord) : Brian Leonard gives it a go

I have been seeing some blogs complaining about ActiveRecord and its limitations. I don't know ActiveRecord and what users need from it well enough to know if these are valid complaints, but Brian Leonard shows a possible alternative: JPA with JRuby.

Tuesday, September 25, 2007

Whoa! WiFi on the bus!

I was just sitting here working offline on my bus ride from the Union City BART to the Sun Menlo Park campus as I normally do when I noticed the guy in front of me looking at the BART schedule on his laptop.

I looked for some fancy schmancy network card in his PCMCIA slot, but nothing there. Then I thought, what the heck, and what do you know, and I looked for wifi signals. What do you know, there was an ssid called ac_transit_wifi_bus.

Whoee! It's a nice fast signal too. So here I am zooming across the Dumbarton Bridge and able to work online.

View Larger Map

Now if they could only get it on BART, I'd be a very happy camper.