Starting MySQL in NetBeans as root/administrator

As you may or may not know, NetBeans 6.1 and 6.5 provide the ability to let you start and stop your MySQL server directly from NetBeans. To be honest I wasn't sure how valuable this feature would be when I implemented it, but in my own work I've found it to be very nice. It's a small thing but it helps me maintain my "flow" when working with the database.

One challenge is that often MySQL can only be run with administrative/root privileges.

On Linux and Solaris you can use 'gksu' to start a program as root, and a dialog will appear asking you for the root password. So if you right-click on the MySQL node and choose Properties and then select the Admin tab, you can make gksu the command name and the actual command to start and stop MySQL is part of the command arguments. Here is a screen shot to give you the idea:



Now, gksu doesn't exist on Mac. The command to bring up a dialog for the administrator password is somewhat complex. So I've written a little script called "macsu" that looks like this:

#!/bin/sh/usr/bin/osascript -e "do shell script \"$@\" with administrator privileges"

Then in your Properties window you can do the following:



Et, voila, you get a prompt when you run the start command:



Note that my 'macsu' command doesn't handle "--" options well, so if you want to do "--console" you might have to wrap your mysql start command in your own script file, and then invoke that file from macsu.

To be honest I have no idea what you can do on Windows. Does anyone have any tips?

Why is the default answer always a web app?

On one of our NetBeans mailing lists, a new programmer asked a question, how should they build their application that needs to run on ten client machines talking to a MySQL database.

The quick answer was "build a web app."

I just don't understand this. The web app architecture is - c'mon, admit it - ugly for applications that need to have dynamic behavior (e.g. almost every app out there). It was never built for that. HTTP/HTML is a static, document-oriented protocol and markup. HTML came from SGML, which is all about documents and publishing, not dynamic database-driven OLTP applications.

But the browser is ubiquitous. It's the absolute best way to distribute a service out to millions of people. Its the visual channel into the Internet. So we've worked with it. Thousands of developers have grown up building apps using HTML/CSS/JavaScript, session state, multiple tiers and so on.

But it wasn't always that way. When I was fresh out of school, the browser didn't exist, and we built simple client/server applications. The client talked directly to the database using a database-oriented protocol.

There were issues, of course, the primary one being how to manage versioning and upgrades across thousands of machines, each with their own operating system and versions, some of them sporadically connected (e.g. on laptops). One of the key advantages of the web is that you install it in one place, and everybody automatically gets the upgrade. And it runs on everybody's machine. It's so cool.

But today there is Java (write once, run anywhere) and Java Web Start (auto-upgrade to all your client machines). They are mature and battle tested. If your application is only going to run on the local network, there is absolutely no need to create a multi-tiered HTML/CSS/JavaScript monster. You can use traditional client/server, with rich clients written in Java and deployed by Java Web Start talking directly to your database using JPA/Hibernate/JDBC. You can use the visual design tools in NetBeans or Eclipse to visually build your application. No hacking <table><tr><td> garbage.

Having a middle tier also allows you to disintermediate between the clients and the database, which can give you a chance to improve performance and scale through things like caching, pooling, clustering, replication and so on. But none of that is needed when you have a small number of users, so why create complexity when it's not required?

Now, if you're building an application to be accessible over the Internet, or if you need to scale, it's a different story. You need a middle tier, and one that speaks HTTP if you're going over the Internet.

But even then you can use Java and Java Web Start, and your client application can talk to the middle/web tier using a service API. There is no reason why the server should be responsible for composing the UI for your application. It can be a server, not a UIer.

If you have Java antibodies, you can use Flex or Silverlight. Or take a look at Cappucino.

There are reasons why these RIA architectures are popular - they are meant for dynamic applications and dynamic behavior, unlike HTML, which really was never intended to be anything more than a way to display static documents. With these RIA environments, finally we're getting back to building apps with tools and an underlying architecture that match the dynamic application paradigm.

Now, folks like Google want you to keep building browser-based HTML applications, using JavaScript for dynamic behavior to muck with the DOM (note their new browser with superior JavaScript performance). Why? So they can scan your text-oriented application and inject targeted ads. Not possible with RIAs where the application is compiled into byte codes.

But just because it works for Google doesn't mean it works for you or your users, or that it's the best architecture. It's just the architecture we've had to work with for the past fifteen years.

Now, there are times when a web app makes sense, absolutely. The answer, as always, is "it depends." But when someone asks you how to build a database-oriented app please don't just default to "web app". There are other, potentially superior, options out there...

Drag and drop service consumption in NetBeans PHP

Very nice screenshot showing how easy it is to generate PHP code to consume various web services, such as Amazon S3, Facebook, Delicious, Google Maps, etc. Includes a link to a tutorial on how to use this feature for Yahoo News Service.

http://blogs.sun.com/netbeansphp/entry/software_as_a_service_in

Hey, Mr. Senator, can I have some of that?

I hear tell that the gummint is handing out cash to anyone in trouble.

Well, hey, Mr. Senator. My bank won't give me no loan, and it's costin' me more and more for food and clothing. Baby needs a new pair of shoes. Can you give me some of that there 700 BILLION dollars? Where do I sign up?

Looking to talk to a few good PHP developers

Well, our team is hitting code freeze today, and so it's time to look ahead and think about database tooling for NetBeans.next.

Last year I interviewed 10 different Java developers about how they use databases, and it was very enlightening. But I didn't talk to very many PHP developers.

I'd like to change that, and YOU can help. If you develop with PHP and would like to help the NetBeans db team improve your productivity when it comes to working with databases, then let me know, and let's talk. You can reach me at david dot vancouvering at sun dot com.

Shock politics

When I read the headline today that Bush is putting heavy pressure on Congress to accept the bailout plan, where we throw at a minimum 700 billion dollars (or more) at Wall Street with no strings attached, saying it's a crisis and we must act quickly, and when there are serious concerns with it, I smelled hidden agenda.  You can just feel it in your bones.

Last week my wife told me that she had read an article where it explained that a big agenda of the current administration before it leaves is to empty out the government coffers as much as possible.  I walked away scratched my head wondering why on earth anyone would want to do that.

Then today I read this article by Naomi Klein and a light went on in my head.  If we're drained of cash, then it becomes much easier to push through "cash-saving" agendas such as privatizing social security, lower corporate taxes and cut spending on the poor.  It severely hobbles any new administration to do anything progressive.

Eight years ago I would never have believed such a cockamamy "conspiracy theory."  But these days, after watching what has happened with the Iraq war, with the EPA, and with the politicization of the Justice Department, well, I'm not so sure.  I have the ugly feeling this may actually be a strategy.

If it's true, and they succeed, heaven help us.  If such backroom strategists are really thinking this way, I don't think they really have any idea what the true consequences of such a strategy might be.  Talk about short-sighted loot-and-burn out-and-out greed.  It makes one shudder.

No, not *that* Palin

Get Fuzzy is a great comic, but I think I haven't laughed at any of them more than these two. Priceless.

Consensus does not mean we all agree

I have had a number of colleagues come to me over the years since I've worked in open source expressing frustration and almost hopelessness as they found themselves trying to obtain consensus for their open source contribution.

What I tell them is that just because the community is consensus-based does not mean everyone has to agree with you. 

Here's one definition of consensus I found on the web: General agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests.  My dictionary on my Mac makes it even simpler: general agreement.
 
The key word here is general agreement.  I also like the phrase "absence of sustained opposition in substantial issues."

Many people get frustrated working in open source because it feels like any attempt to make a substantial change gets bogged down in numbing committee-like discussions.

Now, I won't deny it, there is overhead in working with open source.  You do need to get feedback, and make a best effort to address comments or concerns.

But that doesn't mean you iterate and iterate and iterate in some Sisyphean nightmare trying to get everyone to accept your change, or that you hunker down and make a change you disagree with. 

If you're a committer, you take the feedback, incorporate what makes sense to you, and if there is absence of sustained opposition, you check it in.  If you are not a committer, then all you need to do is find one committer who supports your change and they can check it in for you.

In the Apache model, sustained opposition is in the form of a veto.  A veto is taken very seriously - if you veto, you had better have a very good reason, and you had better offer an alternative. 

In the OpenSolaris ARC model, reviewers provide comments in the form of TCAs (Technical Change Advised) and TCRs (Technical Change Required).  It's only the TCRs you have to worry about.  Your change won't be accepted withough addressing the TCRs, and generally there are few if any TCRs.

I like the TCR/TCA model because it (a) explicity identifies which issues are causing you to veto a change and (b) requires you to be very clear about what needs to change, rather than just saying "-1".

I learned the principle of "consensus does not mean we all agree" the hard way.  When I was new on the Apache Derby project (my first open source project) I spent three months trying to get agreement on a change that would allow sharing of code between the network-based JDBC driver and the embedded JDBC driver.  It was insane.  I would try to please one person or group, and then another set of people would complain, back and forth.  Finally I got a private email from one of the veterans on the project who said "you're never going to make everyone happy, David.  Nobody's threatening to veto.  Just check it in."

Oh.

So now I offer the same advice to people when they complain to me that they feel if they have to bend over backwards for person X or person Y or they say something like "OK, fine, I'll do it your way, but I don't like it."  Relax.  Consensus does not mean we all agree.  Just check it in.

Barack Obama's Economic Plan

Wow. Direct, to the point, and ( (although I'm not sure about the tax rebate) it actually makes sense. Read more here.

• Revamp tax system to work for the average American, not the rich
• Regulate Wall Street
• Energy independence in 10 years
• Crack down on lobbyists
• Responsible end to war in Iraq

It won't be easy. But we're Americans. We've met tough challenges before and we can again.

Nicholas Kristof - Need a Job? $17,000 an Hour. No Success Required.

Pointed, painful, outrageous, hilarious.

John McCain seems to think that the problem is that C.E.O.’s are greedy. Well, of course, they are. We’re all greedy. The real failure is one of corporate governance, which provides only the flimsiest oversight to curb the greed of executives like Mr. Fuld....
John Kenneth Galbraith, the great economist, once explained: “The salary of the chief executive of a large corporation is not a market award for achievement. It is frequently in the nature of a warm personal gesture by the individual to himself.”

http://www.nytimes.com/2008/09/18/opinion/18kristof.html?ref=opinion

Paul Krugman - Blizzard of Lies

It is stunning to see advertisements approved by John McCain stating flat, complete, and utter lies.

What it says, I’d argue, is that the Obama campaign is wrong to suggest that a McCain-Palin administration would just be a continuation of Bush-Cheney. If the way John McCain and Sarah Palin are campaigning is any indication, it would be much, much worse.

http://www.nytimes.com/2008/09/12/opinion/12krugman.html?ref=opinion

Ludovic Champenois's Blog: GlassFish V3 Prelude Update Tool

Ludo gives a nice demo of the Glassfish V3 update tool in action. An app server that is incrementally upgradable like Ubuntu. Pretty nice!

http://weblogs.java.net/blog/ludo/archive/2008/09/glassfish_v3_pr.html

Word wrapping with a JLabel

I am sure a lot of you already know about this, but it took me a while to find this in Google-land, so I thought I'd share it here.

If you want to create a JLabel that does word wrap dynamically, so that if you resize the window the label "does the right thing", then wrap the label text with <HTML> and </HTML>

Memcached User-Defined Functions for Java DB : Knut Anders Hatlen's Weblog

Wow, now this is cool. Some sample code that shows you how to turn Java DB into a memcached engine. Take a look at this snippet:

ij> create trigger insert_trigger after insert on my_table
referencing new as n
for each row
call memcached_set(n.id, n.x);
0 rows inserted/updated/deleted


http://blogs.sun.com/kah/entry/memcached_udf_for_java_db

Drew Westen: What Obama Needs to Do in the Final Sixty Day

Long but great essay with a lot of zing about the problems with standard Democratic strategy and some excellent suggestions (IMHO) or Mr. Obama. I don't know about you, but I am horrified with the decisions McCain is making and am impressed by the ruthlessness of the Republican political machine. I would desperately like to see Obama and his campaign take McCain/Palin by the jugular and stop namby-pambying.
This isn't just about politics. This is about pulling out all the stops to make sure we don't have four more years of horrible Republican policy - I mean, look around at the mess in this house (country)! It's deplorable, and it's been done on the backs of the middle class and poor.

http://www.huffingtonpost.com/drew-westen/what-obama-needs-to-do-in_b_125051.html

Bob Herbert - Liberals, Hold Your Heads Up

Great editorial by Bob Herbert reminding all of us the amazing accomplishments of liberals in the US. I highly respect and agree with conservative's championing of traditional values such as honesty, sacrifice, hard work and family.

But let's remember the love, hard work, sacrifice (including jail time, verbal and physical abuse, even lives lost) given by liberals fighting for the repressed and ignored - women, blacks, the elderly, gays, our health, the environment.

Without these honorable fights against a highly resistant conservative status quo, where would we (and our brothers and sisters and children and parents) be today? Hear hear, Bob.

http://www.nytimes.com/2008/09/09/opinion/09herbert.html?_r=1&em&oref=slogin

Skype ignores PayPal siphoning hijack scheme | The Register

Ouch. This is nasty. I use both Skype and Paypal, and now I wonder if I should move on to something else. I stopped using EBay years ago after a number of nasty experiences.

http://www.theregister.co.uk/2008/09/02/mysterious_skype_hijackings/

Sun and NetApp gain external storage market share | The Register

You know, we make excellent products, and it's nice to see people taking notice...

http://www.theregister.co.uk/2008/09/05/sun_and_netapp_gain_market_share_gartner_and_idc_say/

Patrick Keegan's Blog: A Personal Data Storage Application With Embedded Java DB

Patrick has done a very useful blog showing how you can use JPA with embedded Java DB. Thanks, Patrick!

http://weblogs.java.net/blog/pkeegan/archive/2008/08/a_personal_data_1.html

Economic View - Is History Siding With Obama’s Economic Plan? - NYTimes.com

This article review a book that reveals statistical facts that reflect my experience at any rate. The Republicans talk about being for the common man and woman, and perhaps many of them are, but the ones in power have shown by their actions that they are very much for the rich and powerful.

The two Great Partisan Divides combine to suggest that, if history is a guide, an Obama victory in November would lead to faster economic growth with less inequality, while a McCain victory would lead to slower economic growth with more inequality. Which part of the Obama menu don’t you like?

http://www.nytimes.com/2008/08/31/business/31view.html?em

U.S. Rescue Seen at Hand for 2 Mortgage Giants - NYTimes.com

Nice to know that wild speculation is being bailed out on the backs of taxpayers.

I know, I know, better than not doing anything, but it sure reminds me of sending drunken Uncle Bob some cash so he doesn't end up in jail.

Promises of reform to follow, I suspect, followed by a short memory, debates, and a watered down bill in Congress.

http://www.nytimes.com/2008/09/06/business/06fannie.html?hp

And here, this is what really gets my goat:

But privately, senior officials have been critical of top executives at the companies, particularly Freddie Mac. They have raised concerns about major risks to taxpayers of a bailout of companies whose executives have received huge compensation packages. Mr. Syron, for instance, collected more than $38 million in compensation since he joined the company in 2003.

Must be nice, to have the taxpayers foot $38 million in personal income. Consequences, what consequences?

What's even worse is that the reason investors didn't help out these two companies was because everyone knew that the US government would bail them out sooner or later. Fait accomplis...

Sigh...

Major Republican Hypocrisy - The Daily Show | Comedy Central

A friend showed me this video. Hilarious and also very seriously revealing. The two quotes from Karl Rove are particularly damning. Warning - a little strong language in some of the commentary.

http://www.thedailyshow.com/video/index.jhtml?videoId=184086&title=sarah-palin-gender-card

TechCrunch: Cappuccino Brings Cocoa-Like Programming To The Web

OK, this can change everything. If you can use a full-fledged programming language like Objective-C to build web apps, and you don't have to know HTML and CSS (and who would want to if you had a choice), well, now that's interesting.

Cool thing is if you learn this, then you can write stuff for Cocoa as well. As a programmer, anything that lets you do two in one blow is very attractive. I have avoided writing web apps for years because of the hideous, painfully low-level programming environment. This could switch me over.

Also, think about it - the full application is running in Javascript on the IDE. This means the server isn't busy building HTML/CSS, but it simply in the job of delivering Javascript and providing back-end services for rich web clients. Cool.

http://www.techcrunch.com/2008/09/04/cappuccino-brings-cocoa-like-programming-to-the-web/

Brian Aker - Deleting Your Work, Starting over is good for the soul...

It felt good to delete yesterday's tree and start over again. More often than not this is a best practice :)

A nice short blog about something I have done a number of times. It's good to know I'm not the only one, and yes, now that I think of it, it is a good thing.

http://krow.livejournal.com/608110.html

NetBeans 6.1 has a new patch release available

Hi, all.  I thought I'd share this message from our sustaining team:

Hi All,


I would like to inform you that a new patch for NetBeans IDE 6.1 is now available on the Update Center. The patch includes fixes in modules for BPEL, C/C++, Composite Application, Database, Editing Files, GUI Builder, GlassFish, IDE Platform, Java, Java EE, Java Persistence, NetBeans Plugin Development, Platform, RESTful Web Services, SOA, TAX Library, WSDL, Web Services, XML Productivity Tools, XML Schema Support, XML Tools Java Ext, XML and Schema and XSL Support. 

The patch is available in English, Japanese, Simplified Chinese and Brazilian Portuguese languages.


For more information, see http://wiki.netbeans.org/wiki/view/NetBeans6.1PatchesInfo.


The update is easy to install when the NetBeans IDE is running. Just click on the globe in the status bar or go to Tools->Plugins->Updates tab, then follow the installation instructions.


Regards,
-Rudolf Balada

Government arrests protesters - in USA, not China

Pretty ugly.  I just find this astounding, in the country where the principle of free speech was established.

Engineering @ Facebook : How We Scale Out

Very interesting details into how one of the biggest MySQL customers scales out, particularly how they handle consistency issues with an asynchronous replication and caching architecture.

http://www.new.facebook.com/notes.php?id=9445547199

MySQL Workbench 5.1 Alpha for Linux in Less Than 30 Days

Mike Zinner tells us that his team will be providing an early access of MySQL 5.1 for Linux this month. Those guys are on a roll...

http://dev.mysql.com/workbench/?p=138

Licking summer

Just came back from a very nice family reunion up at Grant's Pass, Oregon.

At one point my little four-year-old-first-cousin-once-removed David (he calls me "cousin me" because we both have the same name) led me up the side of Applegate River next to his house exploring, and we found ourselves picking (and eating) delicious juicy blackberries in the toasty summer sun.

It seemed so idyllic, and I said "mmmm.... this is the taste of summer..."

David was silent for a moment and then he said "we're eating summer..." I said "yeah, I guess we are!"

Then he said, "no, we're licking summer, because it keeps coming back every year. So you can't eat summer ... or spring .... or winter. I guess if you ate all the trees then you could eat fall, because all the trees would be gone and there would be no more fall."

So there we were, licking summer, and enjoying every minute of it.

Miserable bastards (at Apple)

I was listening to a fun song on iTunes, I right-clicked on it and I saw "Create Ringtone..." Cool! So I went ahead and decided to see what it would be like to create a ringtone from a song.

Miserable. Bastards.

And yes, that is the name of the song. Just coincidence. Really.

My mothers are getting married

My Mom came out over 25 years ago, and she has been with the same partner, Maryann for over 20 years.

They're coming to visit us here in California this week, and because gay marriage was just made legal here, they're finally going to marry.

My sister asks "But will it last?" :)

Congratulations, Mom (and Mom)!

NetBeans Wiki: Best Practices With JPA And BeansBinding

A great technical guide on the NetBeans Wiki which quickly and clearly goes through how to use beans binding with JPA, including the question that comes up all the time: how to use the Beans Binding support with Java DB embedded.

People get very confused because they see the tables and data in the Database Explorer but they aren't there when they run their app. And I quote: "Don't be confused if you don't see the data (or even tables) when you run your application in spite of the fact that you can see them in Services window of NetBeans. Recall that Derby is looking for the DB in the working directory and the working directory of NetBeans is different from the working directory of your project."

http://wiki.netbeans.org/BestPracticesWithJPAAndBeansBinding

Sun Rise, Sun Set - Josh Berkus leaves Sun.

Sorry to see Josh go...

http://it.toolbox.com/blogs/database-soup/sun-rise-sun-set-26078

Arnon Rotem-Gal-Oz - Why the Database as a Service is a bad idea

What Arnon says, I've been thinking for a while. But that's not stopping anyone. It just *seems* to fit so perfectly.

http://www.rgoarchitects.com/nblog/2008/08/17/WhyTheDatabaseAsAServiceIsABadIdea.aspx

So, what is it that you *do*, Dad?

I kept a promise yesterday by taking my eight-year old daughter into the office. I missed the official "Take Your Daughter To Work Day" but I really wanted to do this.

She was very excited and she told me she really loved my work and wanted to go again.

I found this mystifying, as this was our day:

• Take BART from North Berkeley to Union City (45 minutes)
• Take the Dumbarton Express bus over the Dumbarton Bridge to the Menlo Park campus
• Get some breakfast
• Meet with the *one* person in my team who (a) works in the Bay Area and (b) was in the office
  
• Color and listen to stories on her MP3 player while I worked
• Go to lunch
• Color and listen to stories while I worked
• Go to coffee
• Color and listen to stories while I worked
• Take the bus and BART back home

I tried to get her to tell me what it was she loved about going into the office, but she didn't come up with anything specific.

You know what I think. I think it's just great that she and I spent the day together. She knows it, I know it, and it had nothing to do with what we did. Just in itself, it was precious.

Something to seriously keep in mind.

Sakila Sample database plugin now available from NetBeans Daily Build

As I promised in an earlier post, I have created a plugin that makes it very easy for you to load the MySQL Sakila sample database from NetBeans.  Sakila is a serious sample database, with tons of data, and a number of views, stored procedures, triggers and functions to allow you to really explore and test all the various features of MySQL.

If you tried in the past to load Sakila from NetBeans, you were probably quickly dissatisfied.  The editor currently can't handle such a large file, and it didn't handle the delimiters correctly, and it also completely barfed when the script tried to insert a raw PNG file.

So, I fixed all those, and now you simply have to install the plugin, then choose Create Database from your MySQL node in the Database Explorer, and you're off and running.

Here are the steps you need to follow. 

First of all, you need to get a recent 6.5 development build, this does not work with NetBeans 6.5 beta.  Go to http://bits.netbeans.org/dev/nightly/latest/ and get the nightly build from there.

Then you go to Tools->Plugins->Available and choose Sakila Sample Database.  To filter the list, type in Sakila in the matchbox.  Check the install checkbox and choose Install



Now open the Services tab, expand Databases, and connect to your MySQL server through the MySQL node.  If you don't have this node, right-click on Databases and choose Register MySQL Server.

Once you are connected, right-click on the node and choose Create Database...



This brings up a little dialog with a drop-down list of sample names.  Note that sakila is on the list - that was added by the plugin. 



Choose sakila and click OK.

You are then presented with a dialog to create a new connection for the new database. 



Type in the appropriate user and password, press OK, and then press OK when asked to choose a schema (I really should fix this, you don't choose schemas in MySQL as a schema and a database are the same thing).

At this point the plugin merrily creates all the tables, views, stored procedures and triggers for Sakila and then loads up all the data.  One thing I need to fix is to add a progress bar so you know when it's done, but just keep your eye on the output window(s).  (Another thing to fix: don't have multiple output windows.  But hey, this is still in beta :)).

Now you should have all of Sakila loaded up, easy as pie.



Please let me know if you have any feedback, particularly by logging issues.

In India, the cell phone is a religious experience

This should probably go on the list of "things you don't know about me" (well, some of you do), but over the millennium I spent a year in India, living in an ashram north of Mumbai.

One thing that I came to understand during that time was how much religion is part of the warp and woof of Indian life. Temples and images are everywhere - even truck drivers and cab drivers have their favorite deities covering their dashboards and painted over the outside of their trucks.


I was having coffee with an Indian friend of mine today, and he told me that another thing that is very popular in India is cell phones, and that these two things have combined in a wonderful and odd sort of way.

He said one of the most popular applications people download onto their cell phones in India is a "daily worship" application. As I understand it, you pick your favorite deity, and each morning when you open your cell phone it pops up an image of your deity, it plays mantras for the deity, and even shows incense waving in front of it.

The thought of this made me smile and laugh. I also found myself filled with love and respect for the way in which spirituality and devotion is anchored into the very core of Indian life.

NetBeans development process rocks!

I have worked on a lot of large software projects in the 17+ years I have been in the industry, and I don't say it lightly that I think NetBeans has one of the best development processes I've ever seen.

I've been on (and seen) projects that were so overburdened with processes and documents that it was like walking through a lake of molasses.

I've also been on (and seen) projects where there was so little process that life was exciting, interesting, crazy, but basically one chaotic scramble after another.

NetBeans follow a beautiful middle path - not too much process, but not too little. I'm sure I'm missing a lot of important points and aspects that make this all fall together, but these are the ones that stand out to me:

Planning
The planning phase is fast and lightweight. Each release is given a release boss, and the release boss works to define a theme for the release. The release boss also decides on the dates for milestones and the final release. Then each team is asked to decide what features they feel best meet the theme, and for what milestones those features will be delivered.

Normally the feature list is written up quickly on a Wiki page. Each feature is given an issue number in Issuezilla and is assigned a person who owns it and a milestone date.

The release boss uses reporting features in Issuezilla to track all of the features committed for a given milestone, and you have the plan. Pretty simple. We all know what to do, and when we have to do it by, and we get to work.

Specification and Design

You are expected to specify the user experience and interface for your features. But it's in a very lightweight manner, on a Wiki page. Here is an example UI Spec that I wrote. Pretty simple, but it gets the point across.

Certain defining features get a lot of attention and are reviewed carefully by our User Experience team, but for most smaller features you just need to write up a quick spec, have it reviewed by your team, and you are done.

Milestones
I love milestones. I think every large team should have milestones. Basically these are a mini-release, where everyone has to get features in by a certain date, there is a stabilization period with extended QA and bugs are fixed, and then it gets published on the NetBeans site. Milestones normally happen about once every 4-6 weeks.

The main reason why I like these as a developer is because it keeps me focused. A six month development cycle tends to make you be lazy early on in the cycle. With milestones coming at you once a month, you get very focused, and you are working hard all the way through the release, instead of working insanely at the end. This also means that it is much more likely that we deliver on time, or if we're late, we know sooner rather than later. And what I have seen is that NetBeans has consistently delivered what we say we deliver, when we say we're going to deliver it. There is a lot behind this, but milestones are a key part of this.

Simple, Strong Quality Criteria
Every release has quality criteria. These are soft and hard goals in terms of the number of P1, P2 and P3 bugs. There is a total overall goal, and each team is also assigned specific goals. For some reason a lot of projects I have been on have not been that strict about this, and it has affected the quality of their product

Nightly Builds Available to the Community
What this means is everybody is using our stuff all the time, all the way through the release process, and we hear from them. I think this is great. More and more projects are doing this, and I really can't recommend it highly enough. And if you don't know about Hudson, you really should check it out.

The Team
This is a lot harder to reproduce, but as usual is a key part of it.

The theme running through this all of this and a thousand more details and steps, is a dedication to quality and predictability, without overburdening the teams with process and busywork. It's a difficult balance, and it's executed quite well here.

So I think that is probably the underpinning of why NetBeans development process rock: the team. From developers to management to QE to release engineering, I have never seen such a smart, dedicated, hardworking group of folks. I am really honored to work with them.

Drop Memcached, Add More MySQL Servers | High Scalability

A very interesting summary of a different way of looking at things: rather than add a new layer with memcache, consider creating more shards to the point where all your data is running in memory. "Hitting disk is the problem. Shard more and get your data in memory."

http://highscalability.com/strategy-drop-memcached-add-more-mysql-servers

Van Couvering IS a verb!

I just found out that I've been verbized.  From the NetBeans users mailing list:

I've googled, nabbled, Vancouveringed (yes, it is a verb now) and even read the manual (gasp), and I'm at a loss.  I have no confidence that data changes will still be 'there' when I copy the DB folder to another location.

Well, I give up.  I guess I will have to change my blog title now. 

Anyway I've been reminded by various English types that if "Vancouvering" were anything it would be a gerund, not a verb.  But "Vancouvering is Not a Gerund" just doesn't have the same ring to it.  And this way my blog title is slightly more confusing ("um, yeah, did I ever say it was?), and that's always good, makes you scratch your head a bit.

W00t! NetBeans 6.5 beta is a go!

Great news, we just got a "go" for NetBeans 6.5 beta, and you can get it here.

There is a lot of great stuff in this release.  Solid PHP, JavaScript and AJAX support.  Groovy/Grails support. 

The Java debugger and editor also have some nice improvements, including the very nice feature that you Eclipse users are used to, compile on save. 

And of course a bunch of cool new database features.

One thing that's not listed yet that I'm just about to put out is support for creating the Sakila sample database directly from NetBeans.  This will be available as a plugin on the update center.  I'll post more about this when it's ready.

Are you engaged or are you vegging?

I just recently got clear on a distinction between being engaged and vegging.  Both of them can happen on a computer.

When I am engaged, my mind is active, I feel productive, and I feel full of energy and creativity.

When I am vegging, it is as if my mind is shut off.  Even if my jaw is not slack, I feel slack-jawed.  I am not on the computer for any reason but to check out.

I like it when I'm engaged.  I dont' like it when I'm vegging.  It affects me not only when I do it but after, and I also find I miss out.  I am on an anti-veg campaign.  My first battle-line - no vegging at night.  If I have to work in the evenings, but when I'm done, no vegging.  No looking at endless blogs.  No link-following, no You-Tubing, no Twittering, no Facebooking.  Also, no iPhone vegging.

I do need breaks - one can't work non-stop, it's actually non-productive.  But I have so many other great things I could be doing with my break time.  Standing up and stretching, going into the kitchen to make myself a cuppa.  Breathe deep and relax.  Hang out with the kids for a few minutes.  Read the paper.

How about you?  Do you know when you're vegging and when you're engaged?

Tim Boudreau's Blog: The Capability Pattern - Future-Proof Your APIs

Tim is doing an excellent series on API design. I'm learning a lot in each read. This latest one is on how to use API/SPI and final to create APIs that retain compatibility.

http://weblogs.java.net/blog/timboudreau/archive/2008/08/the_capability.html

Creating a Custom Java Desktop Database Application - NetBeans 6.1 Tutorial

A new tutorial is out showing how to build a Java Desktop Database app. It's using some of the new MySQL support we put into NetBeans 6.1 to make it easier to work with MySQL from NetBEans.

http://www.netbeans.org/kb/61/java/gui-db-custom.html

Real-world experience with the new NetBeans PHP and JavaScript editors

One of the challenges for someone who builds products but doesn't necessarily use them is that you can get so busy with your day job that you don't know what it's like to use the product.

I do use NetBeans during my day job, writing Java code for the NetBeans product itself. So I am definitely getting my hands dirtier than when I was working on the app server or Java DB. But NetBeans is vast, and there are some areas that I just don't get enough exposure to, but which are directly relevant to my job.

In particular, a large chunk of database applications today are web applications written with PHP and JavaScript. I am quite aware of the new JavaScript and PHP editors, but had not had much experience with them.

Yesterday I spent the whole day trying to help a friend in need who had waded in too deep with a contract he has. He has no database experience, and was trying to customize a Wordpress plugin that uses PHP, JavaScript and MySQL.

So I downloaded the daily build of the PHP bundle of NetBeans for 6.5 and started working. I had done a previous PHP project using vi, and although I was fairly productive it was somewhat painful. So I was so happy to see so many of the things in the NetBeans Java editor available for JavaScript and PHP: completion, inline documentation, easy refactor/rename, goto declaration, find usages, error highlighting and so on.

I felt much more comfortable and productive in this environment. The PHP support even has FTP integration to make it easy to push to a remote site.

But I also spent a lot of time doing SQL in PHP, and had a number of ideas on how to make it easier to work with SQL in a code editor .

One thing is that for this project, as it is for many, MySQL is on a remote server on the Internet. Our entire Database Explorer is useless in such a scenario, as it requires a JDBC driver connecting to a server available on the LAN.

SSH Tunnel Support

There are HTTP JDBC proxy drivers, but what most people usually do is set up a ssh tunnel using a tool like PuTTY for Windows or OpenSSH for UNIX systems. Wade Chandler provided useful instructions for how to do this with PuTTY. Here is another good page on how to do this.

But I can imagine finding a way to make it very easy to configure a Database Explorer connection to use an ssh tunnel. This simple thing could be a big win for a lot of our users. I'm not sure what platform-specific dependencies there are and if this is possible in a Java environment, but I'm going to check it out.


Add Column Wizard

I have been thinking about how to do SQL completion in the PHP editor for a while. But during my work yesterday another itch came up that I would like to scratch. As I worked on the code, I needed to add a column. It would be great to be able to add the column right there in the PHP editor - some kind of right-click or keystroke to add the column that I have highlighted in my text editor, and a quick dialog comes up to get the details of the column.


Insert DDL Wizard

The Wordpress plugin I was working with does something I have heard a lot of people do: right there in the PHP code, detect if the tables exist and then create the tables (or alter the tables if you are doing an upgrade).

This is different from what I'm used to - I had always created the SQL scripts as separate files - but when you think about the web deployment model, it makes sense.

So, a nice feature would be to provide an Insert Code wizard to generate all the CREATE TABLE statements in your code editor. Generating the ALTER TABLE statements is trickier, but hey, let's set our goals high :)

What do you think? What else could we do to make it easier to work with databases in your language editor?

Stephen O'Grady » The Burden of Choice

A very nice blog, by one of my favorite bloggers.

most [open source] projects could use $5,000. What they could use more, however, is some advice on how to spend it.

http://redmonk.com/sogrady/2008/08/07/heres-5000-let-me-help-you-spend-it/

New plugin for PL/SQL editor in NetBeans

Alexandre Sumbatov was a participant in the NetBeans Innovator Grant project, and he just published the result of his work, a NetBeans plugin that provides the ability to edit and compile Oracle PL/SQL objects within NetBeans.  Just download this plugin, and you can browse, edit, save, and compile stored procedures, triggers, functions, package specs, package bodies, type specs and typed bodies all within NetBeans.  Take a look, try it out, and let Alexandre know what you think.  And thanks to Alexandre for his hard work on this!

Harvesting Money in a Hungry World - Op-Ed - NYTimes.com

An important editorial, well expressed, but I suspect likely to fall on deaf ears. It will likely take massive starvation and uprisings to get anyone to pay attention to this situation.

http://www.nytimes.com/2008/08/01/opinion/01hanson.html?ref=opinion

Tim O'Reilly puts out the call to join the Federation and fight the Borg

I've been thinking a lot over the past few years about data on the web, and in particular open data. How can you build a system such that your data doesn't get locked in to a particular vendor. In his recent blog, Tim O'Reilly takes it a step further, and discusses his concerns about a Web 2.0 OPEC where three or four cloud vendors lock everyone in because they become a black hole for data.

There's a lot of very interesting stuff in his article, and is worth reading twice. Here's my favorite quote, a call to action:

If you care about open source for the cloud, build on services that are designed to be federated rather than centralized. Architecture trumps licensing any time.

He even refers to Prophet, a peer-to-peer database system I didn't even know existed.

With Prophet, it’s easy to build custom database applications and synchronize your data with your friends and coworkers – all without a central server.

Prophet supports arbitrary topology synchronization and has an advanced conflict resolution database which helps make sure that the entire database ‘network’ works its way toward consistency even when disconnected users make conflicting choices about how to resolve any given conflict.

This is way cool - those of you who've been reading my blog know I was thinking about this a while ago. And one of the reasons I think this is very important (and Tim does too) is it gets rid of the need for a centralized data repository like Google or Facebook. You can collaborate and broadcast your content without having to rely on Big Brother to hold it for you. That's fine, thank you very much, I'll keep it for myself.

So I'm definitely going to take a closer look at Prophet...

And here's to Tim's call to action: architecture over licensing! Join the Federation and fight the Data Borg!

jwz - Damned Dirty Ape Mask

From Tim Bray's twitter feed. Serious LOL funny, a MUST read :)

"A chimpanzee face stared back at me from the floor, impassive, but mocking."

http://jwz.livejournal.com/914926.html

Manish Bharani: NetBeans Database Explorer gets a facelift

Manish provides a nice summary of what's new in the Database Explorer for 6.5. Manish and his team did a great job providing us with a lot of this functionality - the power of open source, even within Sun!

http://manishbharani.blogspot.com/2008/07/netbeans-database-explorer-gets.html

MySQL Workbench 5.1: Mac/Linux support, and a new Query Editor

Mike Zinner, team lead for the MySQL Workbench team, just posted some screen shots for their up-and-coming new version of MySQL Workbench for the Mac.

Note the screen shot above for the new SQL editor, which will replace the old MySQL Query Browser.

Now, some of you may notice, that looks awfully like the query editor in NetBeans.

Yes, there is some crossover, and we've been talking with each other. It would be nice if we could just wave a magic wand and all this database tooling from Sun were in a single tool. But it's not that easy, particularly if one is written in C and one is written in Java. And it's not clear that's actually what we want.

Sometimes one size doesn't fit all. We need to think carefully about two different types of users: those who want an IDE that works with multiple databases and is focused on serious application development productivity, and those who don't necessarily want an IDE or don't want NetBeans, or Java, but do want a top-notch tool for working with MySQL.

Anyway, we're still discussing, and I hope to pass on what our final decisions are once we have them. Regardless of where we end up, you as the user are going to win with both an excellent IDE and a most excellent database tool.

JavaFX Preview available today

A little bird told me the JavaFX preview is available for download today.

So I got the NetBeans 6.1 with Java FX download from their main page and gave it a spin.  It is a preview, and it still has some rough edges, but with absolutely no knowledge of Java FX I was able to work through the tutorial and get something up and running pretty quickly, all within the NetBeans environment I know so well.

If you're interested in building rich client apps and want to see what Java FX is about, I recommend trying this preview edition, and get your feedback to the team.

Piracy is alive and well

The picture above is from a shopping area in downtown Cuernavaca. There was a whole building with stall after stall covered from floor to ceiling with pirated DVDs and CDs and MP3s. It was a wild thing to walk through, with music blaring from either side.

As I walked out of the building, there was a policeman standing on the sidewalk just next door minding his own business. Obviously this was no concern of his.

I remember when DVDs came out, one of their big things was copy protection. Shyeah, that worked!

Leal’s Blog » Globo.com moves from Oracle to MySQL

"Globo.com is the biggest portal here in Latin America (newspapers, magazine, Television, and radio), and when the speaker said the line: 'We are moving from Oracle to MySQL', everyone was with the same face as you. :)"

http://www.posix.brte.com.br/blog/?p=151

NetBeans has compile on save - Roumen's Weblog

Very cool.

http://blogs.sun.com/roumen/entry/netbeans_has_compile_on_save

I got apps!

Well, I haven't splunked down the money and extra data usage fees for the iPhone 3G, but that doesn't mean I can't get my apps.

I just upgraded to the iPhone OS 2.0, and thus got the app store. An evening of browing and I got some fun stuff. I'd give you links but everything is available only through the iTunes store, so you'll have to go find it yourself. These are all free, I'm not ready to pony up for anything yet...

• Labyrinth LE - remember that old marble game where you tilt the board to move the marble along without falling into a hole. Imagine doing that with your phone, and getting multiple boards to boot. Cool!
• Phone Saber - different colors and special effects - you are an iJedi knight!
• Apple Remote - use your iPhone as a remote for iTunes. Very nice.
• Pandora and AOL radio - streaming Internet radio on my phone - say no more. Just give me Rhapsody and I'm a happy camper
  
• Location-sensitive Yelp - cool!
• midomi - Can't remember that song but it's stuck in your head? Hum it to midomi and it will identify it.
• Location-sensitive pictures - there are a whole bunch of these to choose from
  
• NearPics - show me pictures near where I am.
  
• Google search - searches both the Internet and my phone (e.g. contacts). Nice.

Well, now we're talking. It's like a whole new phone. Although, as we all know, "phone" is not necessarily the right name for this, this, ... thing...

New NetBeans Tutorial: Creating a Simple Web Application Using a MySQL Database

Nice tutorial to get you started building a database app. I love the name of the application: IFPWAFCAD or the International Former Professional Wrestlers' Association for Counesling and Development. Snork!

http://www.netbeans.org/kb/60/web/mysql-webapp.html

Beyond Server Farms In The Sky - InformationWeek

Interesting analysis of what your choices are when you want to scale. I like the phrase "could computing" :)

http://www.informationweek.com/blog/main/archives/2008/07/beyond_server_f.html

Sun may or may not be about to obliterate Oracle and Microsoft | The Register

Warning: colorful language in link. But very funny description of transactional memory (TM), to be supported on Sun's new Rock chip. I have heard about TM for a while within Sun, and yes, there is a very good fit here for databases. 'Nuff said.

http://www.theregister.co.uk/2008/07/28/sun_dziuba_tm/

Kudos for NetBeans exception reporting

I've personally been impressed by the way NetBeans handles reporting an exception for you (and I personally have had a number of bug reports come to me this way, which I don't think I would have otherwise been informed of). It really does help improve the quality of the product.

But here is one user who I think describes the quality of this feature quite well:

So I used the Mercurial support in NetBeans 6.1 (as I have been for some
time), and something went wrong. NetBeans 6.1 did not crash, but recovered and saved the java exception for me, I viewed the exception stacktrace and it was obvious it was a crash inside the NetBeans 6.1 Mercurial support code. There was an option of reporting the crash so I hit the "Report" button to report the problem, then continued on with my work.

Then a few minutes later, I noticed that I had a Firefox tab open to a netbeans site... It said:


Upload of NetBeans Error Report Complete
Thank you very much for sending error information to our server.

After cross-checking your report against others recently received,
your report has been classified as a duplicate of report #64798.
Your comment and logging information were added to this one.

If you believe this issue is severe and you are registered with
netbeans.org, you can use the above link to check all the generated
information in our web application, and you can enter this issue
directly into Issuezilla. To see all reported issues, visit the
Exceptions page.

Thanks again for your help.

The NetBeans development team!


It automatically detected a matching java exception in a previously filed bug report, detecting the duplicate bug report.

So NetBeans 6 did everything right as far as I'm concerned:

1. The tool did not crash or stop or even have to restart due to the error
2. It allowed me the option of reporting the problem
3. It made it trivial to report the problem
4. It automatically submitted the report and data seemlessly and easily
5. It detected it was a duplicate and let me know it was.

Good work guys.

Kaj Arnö - Speaking in Tongues

I was having lunch with Kaj in Prague when he was practicing his speech from his notes, and the locals really were quite impressed. I have never given a talk in a foreign country, but I'm inspired to follow Kaj's example should the opportunity arise.

http://blogs.mysql.com/kaj/2008/06/19/the-why-and-how-to-of-localising-presentations-beyond-english/

Gmail and Google Calendar making the leap offline? » VentureBeat

Looks like Google Gears and SQLite are one of the definite winners in the offline RIA technology wars. Pretty cool that Wordpress is providing offline blogging, and I look forward to offline Gmail.

http://venturebeat.com/2008/07/17/gmail-and-google-calendar-making-the-leap-offline/

Back in the saddle, ruminating on the economics of "free"

I'm back from a nice long vacation, and as usual it was a time of reflection.

I would say that the most interesting and valuable ideas I gained were from reading an excellent book I found on the bookshelf of the home we were doing an exchange at, called The Undercover Economist by Tim Harford.

Mr. Harford layed out in plain and fun English some economic principles I hadn't heard of before that I am still chewing on, including the power of the free market to reveal the truth about what people really value - they put their money where their mouth is - and thus allowing the market to respond in the most efficient way.

But I also liked his descriptions of thing that regularly get in the way of a true free market.  Corruption and influence peddling is one for sure.  Not taking into account external costs, such as the consumption of resources and the dumping of waste, is another important factor.   He argues that when you charge, or "tax", for external costs, it brings more truth into the market, and the market responds.  An example that he cites and which I saw directly was the fees charged to cars entering into the center of London, and how that impacted pollution, noise, and transit services.  It really is quite amazing.

After reading his book, I realized that in technology we often think something is or should be "free" when it's really not.  For example, email has the appearance of being "free", but it actually has a big external cost, particularly managing spam.  Companies spend millions of dollars trying to manage spam; spam management itself consumes large amounts of power and CPU.

If there were nominal charge for sending an email, say 1 cent per email, we would immediately throttle back the insane amount of spam being distributed throughout the system.

The problem is, not only is there no centralized point of control on the Internet, there is also no real identity.  Would it be possible to set up a system where you are required to provide strong identity before you can access a service like email?  You would think this would have been done already if it were possible...


I do think ultimately this problem will need to be solved, and I like the ideas of applying charges for external costs as a way to solve it.

Now this is a vacation

Well, it's wonderful here.  We are thoroughly enjoying our visit.  We are in a huge beautiful house, we are with a friend and her son as well as my wife's aunt, we are visiting places, hanging out, cooking delicious meals, going out for food, and just soaking in the sunshine and the fact that we left all of our many-layered obligations behind us for three weeks.

It was a lot of work getting our house ready, but it was worth it.

The picture above, by the way, is from Las Estacas water park.  This is a park surrounding a river that bubbles up from the ground, and then you can ride down in an inner tube or a life jacket and look up at the incredible lush greenery or look down into the crystal clear water.   Or you can swing from a rope and jump straight into the water, or wander through the lotus garden.



To add to the romance and drama, every night there is thick moist stillness and crickets, then a rumble of far off thunder, then wind and lightning and BOOM a huge thunderstorm, power sometimes goes out, rain pours down in sheets, and the next morning all is clear again.

Did I mention the mangoes, the bunches of tiny bananas, delicious Mexican food, a massive kitchen, an inner courtyard, and the fact that our friend sharing this home with us is a chef?

What can I say?  This really is the life...

Someone is having fun with me

We are getting ready for a wonderful three week vacation to Cuernavaca, Mexico.  We are doing a home exchange with a family of four who have a home there.  Their place is huge, has a swimming pool and a trampoline, and we are going to have loads of fun.

But then there is getting ready.  We really want the place to work well for our guests, and in the past things have happened that were no fun, like the sink clogging up.  And I guess Someone Out There is having a little fun, because in the weeks before leaving, we've had one thing come up after another, and for each one, I am so glad that it didn't happen to our guests:

• Our sprinkler system got cracked and required a complete overhaul (yes, I did this myself, many trips to the hardware store for pipes and valves and glue and mud and getting sprayed in the face and...) 

• Our car battery ran out of steam and we kept having to get jumped (we're doing a car exchange too), so the car is at the shop today getting its battery replaced

• The day before we were going to get its wings clipped,one of our cockatiels escaped and flew up into a neighbor's tree and wouldn't come down.  We drove out there with the bird cage and the other bird, treats and a pillowcase.  No luck, and this morning we had to try again. This time it was in another tree, and we finally coaxed it down with a millet stick (his favorite treat).  We were very lucky.  We lost our first cockatiel for good this way (these two are from an animal shelter). 

• We had mice and rats in our garage and our food was being raided.  Took two weeks to get rid of them, I'll spare you the details 

• At 4:30 am two nights ago I awoke to a roaring sound.  I stumbled downstairs to the kitchen and walked into a pool of water that spread from under the kitchen sink and all the way into the living room.  The ominous roar came from under the sink, and taking a deep breath I opened it and water was spraying everywhere.  Of course, the shutoff valve was at the back of the opening, so I dove in and spluttering and being smacked by water I got it shut off.  Nice way to start your day... Linda and I spent the rest of the morning into lunchtime cleaning up and replacing the water filter, which at that moment had decided to give up the ghost in a dramatic way.

Can you imagine any of that happening to our guests while we were away?  Or worse yet (with the water thing) happening when we had no guests?  So as crazy as it's been, I actually feel blessed that all these things happened now, instead of later.

Theo Schlossnagle - Traffic spikes through the heart of your app

Theo Schlossnagle's blog is a very interesting read indeed.

What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.

That's something to think about.

One likes to think that cloud computing is the silver bullet for scalability - use their stuff, build a system that assumes scale, and no worries, mate. But this makes it clear that there are basic operational issues that need to be kept in mind. Doing it manually may not cut it.

Amazon for example provides no tools for handling traffic increases automatically. It's your job to monitor and kick off new instances. But if this is happening in a matter of 60 seconds, you'd better be very sure you know what to do, and quickly.

Personally, I've never had to deal with this, so I have no tips for you. If I were to start up a new site, I'd go find people who do and have a long talk with them.

New DB Features coming in NetBeans 6.5

Some cool stuff coming down the pike for NetBeans 6.5 database tooling.

SQL Editor code completion - lots of you have been asking for this, and it's in, with more on the way. Note how completion brings up the columns if the SELECT statement already specifies the table name.






SQL History - You can now look at the history of all SQL you've executed, and can to text filtering and/or filter by connection URL, and then select and insert a statement.




Editable, sortable results, multiple result tabs

This isn't integrated yet, but I had to show it to you. With this you can sort the results by double-clicking on a row header (we should have done this a long time ago), modify a row, insert and delete rows, there is full pagination support, and you can optionally create a new tab for each statement that returns results, very nifty for comparing results back and forth.

This feature actually came from the SOA team under a completely different part in Sun - open source works inside as well as outside. Thanks to Ahi and Nithya and Nilesh for this!




Tired of typing in a JDBC URL?

Many of our users, and they're growing, aren't even doing Java, but when they want to connect to a database, the dialog is all JDBC-ish. We've fixed that - now you just type in the parameters and we figure out the URL for you.

It turns out this is no walk in the park, as each vendor has their own parameters and URL format. So for now it works for MySQL, PostgreSQL, Java DB, Oracle and Microsoft SQL Server. For the rest, sorry, you still need to type in the URL, until I can get to them.

BTW, if you still want to see and edit the URL, you can, and we'll update the fields automatically. Nifty.

Jerry Seinfeld on George Carlin’s Life and Comedy

I loved George Carlin as a kid. I once thought of a great joke - what is the worst ice cream flavor you ever heard of. I went to a Carlin concert, and he had the same joke, with the best answer: "bologna swirl." Carlin already did it.

http://www.nytimes.com/2008/06/24/opinion/24seinfeld.html?em&ex=1214452800&en=33d98b4f11507cbf&ei=5087%0A

Query languages for Hadoop, AND tooling for the non-relational stores

Tom White just blogged about three, count them three, different query languages being built for Hadoop.

• Pig, from Yahoo! and now incubating at Apache, has an imperative language called Pig Latin for performing operations on large data files.
  
• Jaql, from IBM and soon to be open sourced, is a declarative query language for JSON data.
  
• Hive, from Facebook and soon to become a Hadoop contrib module, is a data warehouse system with a declarative query language that is a hybrid of SQL and Hadoop streaming.

As someone working on database tooling, I have been thinking about what we at NetBeans might do to make it easier for developers to build apps against these new platforms/products like Hadoop and CouchDB and Amazon services.

These query languages are an example - I can see an interactive query editor that lets you write and test queries directly against your Hadoop or CouchDB engine, much as you can today with SQL tools.

Another interesting question is - if I write my application in terms of domain objects, what things can a tool do to make it easy to map these domain objects to an underlying non-structured store like Hadoop or CouchDB or SimpleDB.

Definitely some opportunities here to make developers' lives easier.

Kaj Arnö - Happily using NetBeans to build MySQL

After getting a demo from a NetBeans engineer on the C/C++ extensions to NetBeans, Kaj was converted - "now this was enough even for a sceptic like me to become eager."

http://blogs.mysql.com/kaj/2008/06/18/netbeans-as-ide-for-developing-mysql-itself/

Alex Bunardzic » Software Development Detox Part 5: Session

Someone who is in my boat about session. Although I still wonder if you are overburdening the client if you say "on the web each request contains all the information necessary for the server code to make a decision on what to do next."

http://jooto.com/blog/index.php/2008/06/20/software-development-detox-part-5-session/

Alex Bunardzic » World Wide Web is About Self-Serve

Interesting blog challenging us to question if there is a need for services, when really the web is basically about resources. I think the real service is in quality of service, making sure the resource is available when you need it.

http://jooto.com/blog/index.php/2008/06/19/world-wide-web-is-about-self-serve/

AOL Radio

I have been a long-time member of the "AOL sucks" club. I signed my name in blood as a member of this club when I tried for three days to get Internet access working for my laptop in someone's home that had AOL as the Internet provider. AOL has historically provided a horrific user interface which constantly thinks it knows better than you and gets in your way whenever you want to do anything different from the standard.

But I've found one piece of AOL that is just, what can I say, nice... It's AOL Radio, which I bumped into when reading a NY Times article about how AOL Radio will soon be available on the iPhone. I tried the Acoustic Rock station, and it's really quite high quality, and the user interface, is, well, simple and - gasp - easy to use!

I have tried a number of interfaces for free Internet radio, and they are either impossible to use, have a horrible selection, or are filled with ads. AOL Radio has none of these limitations. And all you need is Flash to get going. They do seem to have song-skipping disabled, but that's supposed to come back soon.

Pretty nice, and from such an unexpected source! :)

Facebook's Thrift in Apache Incubator

From the Facebook blog:

Thrift is a lightweight software framework for enabling communication between programs written in different programming languages, running on different computers, or both.

It includes support for C, Java, C++, Ruby, Erlang, Perl, Haskell and many others.

It sounds very interesting. I first heard of Thrift because it's used by a very interesting Amazon-like stack called Thrudb. What's nice is because it works with Thrift, you can use it from any language.

However, I looked at the interface for calling the Thrift APIs, and it's a bit ugly, or at least it takes some getting used to.

Anyway, something to keep an eye on, it may have legs.

Rhapsody on my Mac

I really like the Rhapsody music service. I have been trying various ways to get this to work on my Mac.

There is a plugin for the browser that lets you access Rhapsody natively on the Mac. The only problem is that it regularly crashes Firefox, and Rhapsody support clearly has fixing this as a very low priority. I tried Safari, and that was working OK, until I upgraded to OSX 10.5.3, and now Safari regularly crashes my entire machine! Sorry, no go.

So I finally gave in, and am happily running Rhapsody natively on my Win XP VMWare instance. I reduced the memory on the instance to 512MB otherwise if I have two instances of NetBeans up my system memory starts thrashing. But once I did that I am happy as a clam. Mozart is lilting to me as I code, and all is well with the world.

Open Source EC2 - the beginning of a Scale Stack

In my last post I mentioned that what I wanted to see was the industry coalesce around an open source "scale stack" [1].

We may be seeing the beginnings of this. The High Scalability site (great blog, by the way) recently posted a blog talking about a new kid in town coming out of UC Santa Barbara called Eucalyptus . They are providing an open source implementation an elastic compute infrastructure that is interface-compatible with Amazon's EC2 which you can take and deploy on your own hardware.

This is very encouraging, and I think is a smart approach. Rather than try to build some standard that is lost in committee for years, use the de-facto standard, which in this space is Amazon.

Another piece of the puzzle is Hadoop, an open source implementation of map/reduce. Hadoop also has a distributed file system - one thing that might be worth investigating is building an S3 layer on top of Hadoop's file system.

What about the queuing service? Well, one possibility is to but an SQS API on top of ActiveMQ or OpenJMS.

Throw in CouchDB, and you're starting to get a very interesting stack indeed. I'm not sure about putting a SimpleDB interface on top of this - CouchDB is pretty darn interesting in its own right, and I think the jury is still out on SimpleDB.

[1] I am not sure if he wants me to mention his name, so I won't, but I want to acknowledge that the idea for an open source stack based on Amazon's APIs is not my own, but comes from a colleague at Sun. I think it's a great idea, and may it come to fruition.

The exponential cost of contention

I enjoy Nati Shalom's blog, although it always has that taste of having the agenda of pushing Gigaspace's solutions. But putting that aside his posts are always well thought-out and well written.

I think his latest blog on the Economies of Non-Scale really drive some points home about scalability, or more to the point, the cost of non-linear scalability.

If 90% of our application is free of contention, and only 10% is spent on a shared resources, we will need to grow our compute resources by a factor of 100 to scale by a factor of 10! Another important thing to note is that 10x, in this case, is the limit of our ability to scale, even if more resources are added.

...

1. The cost of non-linearly scalable applications grows exponentially with the demand for more scale.

2. Non-linearly scalable applications have an absolute limit of scalability. According to Amdhal's Law, with 10% contention, the maximum scaling limit is 10. With 40% contention, our maximum scaling limit is 2.5 - no matter how many hardware resources we will throw at the problem

That's something to chew on. These are real costs, both to your business, to your users, and to the environment. Even if you only have a teeny 10% contention in your system, that 10% will nail you faster than you can say ACID semantics. And as has become very clear to me the final breaking point, the final point of contention in any traditional web application architecture, is the database. Get rid of that and you're home free.

How do you do that? Well, there are a lot of people trying to solve this problem with things like space-based architectures, eventual consistency, distributed map/reduce and Stonebraker's H-Store architecture. Anything to let each instance stand on its own and not have to serialize with the rest of the system at any point, in any form.

Some people argue that scalability is so hard that you shouldn't think about it until you need to. But I really believe that if you do enough to educate yourself and make some wise choices, you will be very glad you did.

What I'd like to see is the industry coalescing around some best practices, an open source "scale stack" ala LAMP, tools, community, and hosting environments like Amazon, that allow developers to easily build applications that will scale from the get go. That's where I want to see things go. That way you don't have to throw your hands up and hope for the best. Because as you can see, the costs can be deadly.

Something to think about...

Greed and vision

I was listening to NPR this morning to a fascinating analysis by someone who has talked to everyone from homeowners to lenders to Wall Street about the home loan fiasco. He said everybody up and down the chain was taking unreasonable risks, and everyone was making tons of money off of it.

Then he quoted a financial expert, I wish I could remember who, who said

It's amazing how hard it is for someone to see something when they are being paid a small fortune not to see it.

I just had to laugh, that really sums it up.

It isn't because you're dirty, it isn't because you're clean

My daughter is currently on antibiotics, and she's staying home from school because she has a sniffle.

Why is such an innocent sniffle the cause for keeping her at home, and why is she on antibiotics? Pertussis, or whooping cough.

My daughter goes to a private school, so vaccinations are optional. Many chose not to vaccinate for pertussis (this is actually one we did), and these are the consequences. The Department of Health for the county closed the school for the day on Friday, and is requiring that everyone, regardless of whether they've been vaccinated or not, either take a 5-day protocol of antibiotics or stay home for 21 days. They also have implemented a no-symptoms policy, so anybody with any symptoms, including a sniffle, has to stay home. Sorry, Ariel.

So, swirling around this is the whole debate of vaccinations. As a father, I was stunned at the number and frequency of vaccinations we are supposed to give our children, and which you are required to have before you can enter public schools. I understand, this is all in the name of public health, but as a parent you have to use your intuition a lot, and this just intuitively feels like overkill, literally. It reminds me of the latest discoveries I read about in Science News where there are bacteria in dirt that, when inhaled, create a very strong immunity. Children who are kept away from dirt and kept super-clean are more prone to disease.

I understand vaccinating against horrific and fatal diseases. But many of these vaccinations are for diseases that many of us older folks had as a child: chicken pox, measles, mumps. It was just part of growing up. And there are some real issues here. I am no doctor, but as I understand it, if you are vaccinated against chicken pox and never get it as a kid, then you have a chance of getting it as a teenager, and then it can be *really* bad, whereas if you get it as a kid, then you get a much stronger immunity and generally don't get it when you're older.

I understand then challenge of maintaining public health over trying to protect your child from an army of doctors with needles. I mean, I would be horrified if my child or someone else's child died because of my unwillingness to vaccinate. But at the same time, I wonder what is truly the right thing to do here. We are often so focused on eliminating all and any suffering from this world that we fail to see the ultimate consequences. As an example, part of the reason the plague pounced upon Europe so badly in the middle ages is because humans had been so successful at eradicating the wolf, which, you guessed it, was the primary predator for rats.

I couldn't tell you what the unexpected consequences may be for so much vaccination, but I can tell you that it feels like overkill, and I personally recommend that as a parent you do your research and try to make informed choices about which vaccinations you give your kids, rather than blindly doing none or blindly following the protocol laid out by the powers that be.

MySpace to share data

This just in:
MySpace is going to allow users to share information from their site.

This is definitely a step in the right direction, but notice that sharing is done either by copying/synching, or through their specific APIs. It doesn't fully extract the content into a separate layer with application-provider-agnostic service APIs.

One can imagine an OpenSocial API on top of a social network data service that is used by numerous providers but owned by none.

Giant Food & Biotech Corporations Make Billions in Profit from Growing Global Food Crisis

Big Food is serious big business, and thus is inherently evil (profits over social needs, always - the corporate credo).

http://www.organicconsumers.org/articles/article_12088.cfm

Internet content by reference, not by value

You may have noticed that I am very interested in how data is managed on the Internet as a platform, at a web scale. In that light, I have been having some very illuminating and interesting conversations with an old friend and colleague, Nitin Borwankar. His thoughts on data property rights and DRY data are concepts that if implemented could result in a major shift in how we manage data on the web.

Data property rights is about laying out a "bill of rights" for data that goes far beyond "the right to move". It also includes the right to access, modify, remove and own your data. So often it happens that once you upload your content to a site, you no longer have full rights to that content, as if somehow in the act of uploading it it is no longer yours. It's like living in a serfdom where you do all the work to plow, seed, tend and harvest the land, but the fruit of your labor is not yours, just because you are using the land that someone else owns.

DRY data is about following the principle of Don't Repeat Yourself for web content. Web applications need to start applying this principle, so that rather than you having to load copies of your content across multiple sites (and losing ownership of it in the process), you place it in one location (your "home" on the Web, as it were), and then you refer application providers to that one place. They can focus on providing added value (for instance, referring it to your friends, enabling collaboration, or helping you organize it or present it in useful ways) rather than on the overhead of building and deploying a scalable storage architecture.

Nitin calls this architecture YINAS (YINAS Is Not A Silo).



The value of DRY for the user is obvious - I only have to put my stuff in one place, and I get to really own my stuff, rather than the vendor owning it. DRY is also very valuable for the vendor, as they can save overhead and complexity by delegating the work of scalable storage and indexing to a "data service provider" rather than having to do it themselves. It's even good for the environment, because you need fewer disk farms sucking up power and space. I guess the only folks who would lose out are the storage and power vendors :)

It's funny, it makes so much sense, but nobody is really doing this.

I pulled Tim Bray aside at Java One to talk to him about these ideas after reading his blog about changing his address, and he suggested that concepts are good, but a simple proof of concept is better. Hm... let me think about that ... :)