Thursday, August 28, 2008

Engineering @ Facebook : How We Scale Out

Very interesting details into how one of the biggest MySQL customers scales out, particularly how they handle consistency issues with an asynchronous replication and caching architecture.

http://www.new.facebook.com/notes.php?id=9445547199

MySQL Workbench 5.1 Alpha for Linux in Less Than 30 Days

Mike Zinner tells us that his team will be providing an early access of MySQL 5.1 for Linux this month. Those guys are on a roll...

http://dev.mysql.com/workbench/?p=138

Monday, August 25, 2008

Licking summer

Just came back from a very nice family reunion up at Grant's Pass, Oregon.

At one point my little four-year-old-first-cousin-once-removed David (he calls me "cousin me" because we both have the same name) led me up the side of Applegate River next to his house exploring, and we found ourselves picking (and eating) delicious juicy blackberries in the toasty summer sun.

It seemed so idyllic, and I said "mmmm.... this is the taste of summer..."

David was silent for a moment and then he said "we're eating summer..." I said "yeah, I guess we are!"

Then he said, "no, we're licking summer, because it keeps coming back every year. So you can't eat summer ... or spring .... or winter. I guess if you ate all the trees then you could eat fall, because all the trees would be gone and there would be no more fall."

So there we were, licking summer, and enjoying every minute of it.

Wednesday, August 20, 2008

Miserable bastards (at Apple)

I was listening to a fun song on iTunes, I right-clicked on it and I saw "Create Ringtone..." Cool! So I went ahead and decided to see what it would be like to create a ringtone from a song.


Miserable. Bastards.

And yes, that is the name of the song. Just coincidence. Really.

My mothers are getting married

Charlotte with Babu and Grandma

My Mom came out over 25 years ago, and she has been with the same partner, Maryann for over 20 years.

They're coming to visit us here in California this week, and because gay marriage was just made legal here, they're finally going to marry.

My sister asks "But will it last?" :)

Congratulations, Mom (and Mom)!

Tuesday, August 19, 2008

NetBeans Wiki: Best Practices With JPA And BeansBinding

A great technical guide on the NetBeans Wiki which quickly and clearly goes through how to use beans binding with JPA, including the question that comes up all the time: how to use the Beans Binding support with Java DB embedded.

People get very confused because they see the tables and data in the Database Explorer but they aren't there when they run their app. And I quote: "Don't be confused if you don't see the data (or even tables) when you run your application in spite of the fact that you can see them in Services window of NetBeans. Recall that Derby is looking for the DB in the working directory and the working directory of NetBeans is different from the working directory of your project."

http://wiki.netbeans.org/BestPracticesWithJPAAndBeansBinding

Sun Rise, Sun Set - Josh Berkus leaves Sun.

Sorry to see Josh go...

http://it.toolbox.com/blogs/database-soup/sun-rise-sun-set-26078

Arnon Rotem-Gal-Oz - Why the Database as a Service is a bad idea

What Arnon says, I've been thinking for a while. But that's not stopping anyone. It just *seems* to fit so perfectly.

http://www.rgoarchitects.com/nblog/2008/08/17/WhyTheDatabaseAsAServiceIsABadIdea.aspx

So, what is it that you *do*, Dad?

SANY0008.JPG

I kept a promise yesterday by taking my eight-year old daughter into the office. I missed the official "Take Your Daughter To Work Day" but I really wanted to do this.

She was very excited and she told me she really loved my work and wanted to go again.

I found this mystifying, as this was our day:
  • Take BART from North Berkeley to Union City (45 minutes)
  • Take the Dumbarton Express bus over the Dumbarton Bridge to the Menlo Park campus
  • Get some breakfast
  • Meet with the *one* person in my team who (a) works in the Bay Area and (b) was in the office
  • Color and listen to stories on her MP3 player while I worked
  • Go to lunch
  • Color and listen to stories while I worked
  • Go to coffee
  • Color and listen to stories while I worked
  • Take the bus and BART back home
I tried to get her to tell me what it was she loved about going into the office, but she didn't come up with anything specific.

You know what I think. I think it's just great that she and I spent the day together. She knows it, I know it, and it had nothing to do with what we did. Just in itself, it was precious.

Something to seriously keep in mind.

Sakila Sample database plugin now available from NetBeans Daily Build

NetBeans Logo
MySQL Logo

As I promised in an earlier post, I have created a plugin that makes it very easy for you to load the MySQL Sakila sample database from NetBeans.  Sakila is a serious sample database, with tons of data, and a number of views, stored procedures, triggers and functions to allow you to really explore and test all the various features of MySQL.

If you tried in the past to load Sakila from NetBeans, you were probably quickly dissatisfied.  The editor currently can't handle such a large file, and it didn't handle the delimiters correctly, and it also completely barfed when the script tried to insert a raw PNG file.

So, I fixed all those, and now you simply have to install the plugin, then choose Create Database from your MySQL node in the Database Explorer, and you're off and running.

Here are the steps you need to follow. 

First of all, you need to get a recent 6.5 development build, this does not work with NetBeans 6.5 beta.  Go to http://bits.netbeans.org/dev/nightly/latest/ and get the nightly build from there.

Then you go to Tools->Plugins->Available and choose Sakila Sample Database.  To filter the list, type in Sakila in the matchbox.  Check the install checkbox and choose Install

sakila-plugin.jpg

Now open the Services tab, expand Databases, and connect to your MySQL server through the MySQL node.  If you don't have this node, right-click on Databases and choose Register MySQL Server.

Once you are connected, right-click on the node and choose Create Database...

sakila-create-db.jpg

This brings up a little dialog with a drop-down list of sample names.  Note that sakila is on the list - that was added by the plugin. 

sakila-choose-db.jpg

Choose sakila and click OK.

You are then presented with a dialog to create a new connection for the new database. 

sakila-create-connection.jpg

Type in the appropriate user and password, press OK, and then press OK when asked to choose a schema (I really should fix this, you don't choose schemas in MySQL as a schema and a database are the same thing).

At this point the plugin merrily creates all the tables, views, stored procedures and triggers for Sakila and then loads up all the data.  One thing I need to fix is to add a progress bar so you know when it's done, but just keep your eye on the output window(s).  (Another thing to fix: don't have multiple output windows.  But hey, this is still in beta :)).

Now you should have all of Sakila loaded up, easy as pie.

sakila-view-data.jpg

Please let me know if you have any feedback, particularly by logging issues.

Friday, August 15, 2008

In India, the cell phone is a religious experience

This should probably go on the list of "things you don't know about me" (well, some of you do), but over the millennium I spent a year in India, living in an ashram north of Mumbai.

One thing that I came to understand during that time was how much religion is part of the warp and woof of Indian life. Temples and images are everywhere - even truck drivers and cab drivers have their favorite deities covering their dashboards and painted over the outside of their trucks.

truck&drivers_04
I was having coffee with an Indian friend of mine today, and he told me that another thing that is very popular in India is cell phones, and that these two things have combined in a wonderful and odd sort of way.

He said one of the most popular applications people download onto their cell phones in India is a "daily worship" application. As I understand it, you pick your favorite deity, and each morning when you open your cell phone it pops up an image of your deity, it plays mantras for the deity, and even shows incense waving in front of it.

The thought of this made me smile and laugh. I also found myself filled with love and respect for the way in which spirituality and devotion is anchored into the very core of Indian life.

Thursday, August 14, 2008

NetBeans development process rocks!

I have worked on a lot of large software projects in the 17+ years I have been in the industry, and I don't say it lightly that I think NetBeans has one of the best development processes I've ever seen.

I've been on (and seen) projects that were so overburdened with processes and documents that it was like walking through a lake of molasses.

I've also been on (and seen) projects where there was so little process that life was exciting, interesting, crazy, but basically one chaotic scramble after another.

NetBeans follow a beautiful middle path - not too much process, but not too little. I'm sure I'm missing a lot of important points and aspects that make this all fall together, but these are the ones that stand out to me:

Planning
The planning phase is fast and lightweight. Each release is given a release boss, and the release boss works to define a theme for the release. The release boss also decides on the dates for milestones and the final release. Then each team is asked to decide what features they feel best meet the theme, and for what milestones those features will be delivered.

Normally the feature list is written up quickly on a Wiki page. Each feature is given an issue number in Issuezilla and is assigned a person who owns it and a milestone date.

The release boss uses reporting features in Issuezilla to track all of the features committed for a given milestone, and you have the plan. Pretty simple. We all know what to do, and when we have to do it by, and we get to work.

Specification and Design

You are expected to specify the user experience and interface for your features. But it's in a very lightweight manner, on a Wiki page. Here is an example UI Spec that I wrote. Pretty simple, but it gets the point across.

Certain defining features get a lot of attention and are reviewed carefully by our User Experience team, but for most smaller features you just need to write up a quick spec, have it reviewed by your team, and you are done.
A milestone on the A140 in Horsham St.
Milestones
I love milestones. I think every large team should have milestones. Basically these are a mini-release, where everyone has to get features in by a certain date, there is a stabilization period with extended QA and bugs are fixed, and then it gets published on the NetBeans site. Milestones normally happen about once every 4-6 weeks.

The main reason why I like these as a developer is because it keeps me focused. A six month development cycle tends to make you be lazy early on in the cycle. With milestones coming at you once a month, you get very focused, and you are working hard all the way through the release, instead of working insanely at the end. This also means that it is much more likely that we deliver on time, or if we're late, we know sooner rather than later. And what I have seen is that NetBeans has consistently delivered what we say we deliver, when we say we're going to deliver it. There is a lot behind this, but milestones are a key part of this.

Simple, Strong Quality Criteria
Every release has quality criteria. These are soft and hard goals in terms of the number of P1, P2 and P3 bugs. There is a total overall goal, and each team is also assigned specific goals. For some reason a lot of projects I have been on have not been that strict about this, and it has affected the quality of their product

Nightly Builds Available to the Community
What this means is everybody is using our stuff all the time, all the way through the release process, and we hear from them. I think this is great. More and more projects are doing this, and I really can't recommend it highly enough. And if you don't know about Hudson, you really should check it out.

The Team
This is a lot harder to reproduce, but as usual is a key part of it.

The theme running through this all of this and a thousand more details and steps, is a dedication to quality and predictability, without overburdening the teams with process and busywork. It's a difficult balance, and it's executed quite well here.

So I think that is probably the underpinning of why NetBeans development process rock: the team. From developers to management to QE to release engineering, I have never seen such a smart, dedicated, hardworking group of folks. I am really honored to work with them.

Wednesday, August 13, 2008

Drop Memcached, Add More MySQL Servers | High Scalability

A very interesting summary of a different way of looking at things: rather than add a new layer with memcache, consider creating more shards to the point where all your data is running in memory. "Hitting disk is the problem. Shard more and get your data in memory."

http://highscalability.com/strategy-drop-memcached-add-more-mysql-servers

Van Couvering IS a verb!

I just found out that I've been verbized.  From the NetBeans users mailing list:
I've googled, nabbled, Vancouveringed (yes, it is a verb now) and even read the manual (gasp), and I'm at a loss.  I have no confidence that data changes will still be 'there' when I copy the DB folder to another location.
Well, I give up.  I guess I will have to change my blog title now. 

Anyway I've been reminded by various English types that if "Vancouvering" were anything it would be a gerund, not a verb.  But "Vancouvering is Not a Gerund" just doesn't have the same ring to it.  And this way my blog title is slightly more confusing ("um, yeah, did I ever say it was?), and that's always good, makes you scratch your head a bit.

W00t! NetBeans 6.5 beta is a go!


Great news, we just got a "go" for NetBeans 6.5 beta, and you can get it here.

There is a lot of great stuff in this release.  Solid PHP, JavaScript and AJAX support.  Groovy/Grails support. 

The Java debugger and editor also have some nice improvements, including the very nice feature that you Eclipse users are used to, compile on save

And of course a bunch of cool new database features.

One thing that's not listed yet that I'm just about to put out is support for creating the Sakila sample database directly from NetBeans.  This will be available as a plugin on the update center.  I'll post more about this when it's ready.

Monday, August 11, 2008

Are you engaged or are you vegging?


I just recently got clear on a distinction between being engaged and vegging.  Both of them can happen on a computer.

When I am engaged, my mind is active, I feel productive, and I feel full of energy and creativity.

When I am vegging, it is as if my mind is shut off.  Even if my jaw is not slack, I feel slack-jawed.  I am not on the computer for any reason but to check out.

I like it when I'm engaged.  I dont' like it when I'm vegging.  It affects me not only when I do it but after, and I also find I miss out.  I am on an anti-veg campaign.  My first battle-line - no vegging at night.  If I have to work in the evenings, but when I'm done, no vegging.  No looking at endless blogs.  No link-following, no You-Tubing, no Twittering, no Facebooking.  Also, no iPhone vegging.

I do need breaks - one can't work non-stop, it's actually non-productive.  But I have so many other great things I could be doing with my break time.  Standing up and stretching, going into the kitchen to make myself a cuppa.  Breathe deep and relax.  Hang out with the kids for a few minutes.  Read the paper.

How about you?  Do you know when you're vegging and when you're engaged?

Tim Boudreau's Blog: The Capability Pattern - Future-Proof Your APIs

Tim is doing an excellent series on API design. I'm learning a lot in each read. This latest one is on how to use API/SPI and final to create APIs that retain compatibility.

http://weblogs.java.net/blog/timboudreau/archive/2008/08/the_capability.html

Sunday, August 10, 2008

Creating a Custom Java Desktop Database Application - NetBeans 6.1 Tutorial

A new tutorial is out showing how to build a Java Desktop Database app. It's using some of the new MySQL support we put into NetBeans 6.1 to make it easier to work with MySQL from NetBEans.

http://www.netbeans.org/kb/61/java/gui-db-custom.html

Thursday, August 07, 2008

Real-world experience with the new NetBeans PHP and JavaScript editors

One of the challenges for someone who builds products but doesn't necessarily use them is that you can get so busy with your day job that you don't know what it's like to use the product.

I do use NetBeans during my day job, writing Java code for the NetBeans product itself. So I am definitely getting my hands dirtier than when I was working on the app server or Java DB. But NetBeans is vast, and there are some areas that I just don't get enough exposure to, but which are directly relevant to my job.

In particular, a large chunk of database applications today are web applications written with PHP and JavaScript. I am quite aware of the new JavaScript and PHP editors, but had not had much experience with them.

Yesterday I spent the whole day trying to help a friend in need who had waded in too deep with a contract he has. He has no database experience, and was trying to customize a Wordpress plugin that uses PHP, JavaScript and MySQL.

So I downloaded the daily build of the PHP bundle of NetBeans for 6.5 and started working. I had done a previous PHP project using vi, and although I was fairly productive it was somewhat painful. So I was so happy to see so many of the things in the NetBeans Java editor available for JavaScript and PHP: completion, inline documentation, easy refactor/rename, goto declaration, find usages, error highlighting and so on.


I felt much more comfortable and productive in this environment. The PHP support even has FTP integration to make it easy to push to a remote site.

But I also spent a lot of time doing SQL in PHP, and had a number of ideas on how to make it easier to work with SQL in a code editor .

One thing is that for this project, as it is for many, MySQL is on a remote server on the Internet. Our entire Database Explorer is useless in such a scenario, as it requires a JDBC driver connecting to a server available on the LAN.

SSH Tunnel Support

There are HTTP JDBC proxy drivers, but what most people usually do is set up a ssh tunnel using a tool like PuTTY for Windows or OpenSSH for UNIX systems. Wade Chandler provided useful instructions for how to do this with PuTTY. Here is another good page on how to do this.

But I can imagine finding a way to make it very easy to configure a Database Explorer connection to use an ssh tunnel. This simple thing could be a big win for a lot of our users. I'm not sure what platform-specific dependencies there are and if this is possible in a Java environment, but I'm going to check it out.


Add Column Wizard

I have been thinking about how to do SQL completion in the PHP editor for a while. But during my work yesterday another itch came up that I would like to scratch. As I worked on the code, I needed to add a column. It would be great to be able to add the column right there in the PHP editor - some kind of right-click or keystroke to add the column that I have highlighted in my text editor, and a quick dialog comes up to get the details of the column.


Insert DDL Wizard

The Wordpress plugin I was working with does something I have heard a lot of people do: right there in the PHP code, detect if the tables exist and then create the tables (or alter the tables if you are doing an upgrade).

This is different from what I'm used to - I had always created the SQL scripts as separate files - but when you think about the web deployment model, it makes sense.

So, a nice feature would be to provide an Insert Code wizard to generate all the CREATE TABLE statements in your code editor. Generating the ALTER TABLE statements is trickier, but hey, let's set our goals high :)

What do you think? What else could we do to make it easier to work with databases in your language editor?

Stephen O'Grady » The Burden of Choice

A very nice blog, by one of my favorite bloggers.
most [open source] projects could use $5,000. What they could use more, however, is some advice on how to spend it.

http://redmonk.com/sogrady/2008/08/07/heres-5000-let-me-help-you-spend-it/

Monday, August 04, 2008

New plugin for PL/SQL editor in NetBeans

Alexandre Sumbatov was a participant in the NetBeans Innovator Grant project, and he just published the result of his work, a NetBeans plugin that provides the ability to edit and compile Oracle PL/SQL objects within NetBeans.  Just download this plugin, and you can browse, edit, save, and compile stored procedures, triggers, functions, package specs, package bodies, type specs and typed bodies all within NetBeans.  Take a look, try it out, and let Alexandre know what you think.  And thanks to Alexandre for his hard work on this!

Friday, August 01, 2008

Harvesting Money in a Hungry World - Op-Ed - NYTimes.com

An important editorial, well expressed, but I suspect likely to fall on deaf ears. It will likely take massive starvation and uprisings to get anyone to pay attention to this situation.

http://www.nytimes.com/2008/08/01/opinion/01hanson.html?ref=opinion

Tim O'Reilly puts out the call to join the Federation and fight the Borg



I've been thinking a lot over the past few years about data on the web, and in particular open data. How can you build a system such that your data doesn't get locked in to a particular vendor. In his recent blog, Tim O'Reilly takes it a step further, and discusses his concerns about a Web 2.0 OPEC where three or four cloud vendors lock everyone in because they become a black hole for data.

There's a lot of very interesting stuff in his article, and is worth reading twice. Here's my favorite quote, a call to action:
If you care about open source for the cloud, build on services that are designed to be federated rather than centralized. Architecture trumps licensing any time.
He even refers to Prophet, a peer-to-peer database system I didn't even know existed.
With Prophet, it’s easy to build custom database applications and synchronize your data with your friends and coworkers – all without a central server.
Prophet supports arbitrary topology synchronization and has an advanced conflict resolution database which helps make sure that the entire database ‘network’ works its way toward consistency even when disconnected users make conflicting choices about how to resolve any given conflict.
This is way cool - those of you who've been reading my blog know I was thinking about this a while ago. And one of the reasons I think this is very important (and Tim does too) is it gets rid of the need for a centralized data repository like Google or Facebook. You can collaborate and broadcast your content without having to rely on Big Brother to hold it for you. That's fine, thank you very much, I'll keep it for myself.

So I'm definitely going to take a closer look at Prophet...

And here's to Tim's call to action: architecture over licensing! Join the Federation and fight the Data Borg!

jwz - Damned Dirty Ape Mask

From Tim Bray's twitter feed. Serious LOL funny, a MUST read :)

"A chimpanzee face stared back at me from the floor, impassive, but mocking."

http://jwz.livejournal.com/914926.html