Kaj Arnö - Happily using NetBeans to build MySQL

After getting a demo from a NetBeans engineer on the C/C++ extensions to NetBeans, Kaj was converted - "now this was enough even for a sceptic like me to become eager."

http://blogs.mysql.com/kaj/2008/06/18/netbeans-as-ide-for-developing-mysql-itself/

Alex Bunardzic » Software Development Detox Part 5: Session

Someone who is in my boat about session. Although I still wonder if you are overburdening the client if you say "on the web each request contains all the information necessary for the server code to make a decision on what to do next."

http://jooto.com/blog/index.php/2008/06/20/software-development-detox-part-5-session/

Alex Bunardzic » World Wide Web is About Self-Serve

Interesting blog challenging us to question if there is a need for services, when really the web is basically about resources. I think the real service is in quality of service, making sure the resource is available when you need it.

http://jooto.com/blog/index.php/2008/06/19/world-wide-web-is-about-self-serve/

AOL Radio

I have been a long-time member of the "AOL sucks" club. I signed my name in blood as a member of this club when I tried for three days to get Internet access working for my laptop in someone's home that had AOL as the Internet provider. AOL has historically provided a horrific user interface which constantly thinks it knows better than you and gets in your way whenever you want to do anything different from the standard.

But I've found one piece of AOL that is just, what can I say, nice... It's AOL Radio, which I bumped into when reading a NY Times article about how AOL Radio will soon be available on the iPhone. I tried the Acoustic Rock station, and it's really quite high quality, and the user interface, is, well, simple and - gasp - easy to use!

I have tried a number of interfaces for free Internet radio, and they are either impossible to use, have a horrible selection, or are filled with ads. AOL Radio has none of these limitations. And all you need is Flash to get going. They do seem to have song-skipping disabled, but that's supposed to come back soon.

Pretty nice, and from such an unexpected source! :)

Facebook's Thrift in Apache Incubator

From the Facebook blog:

Thrift is a lightweight software framework for enabling communication between programs written in different programming languages, running on different computers, or both.

It includes support for C, Java, C++, Ruby, Erlang, Perl, Haskell and many others.

It sounds very interesting. I first heard of Thrift because it's used by a very interesting Amazon-like stack called Thrudb. What's nice is because it works with Thrift, you can use it from any language.

However, I looked at the interface for calling the Thrift APIs, and it's a bit ugly, or at least it takes some getting used to.

Anyway, something to keep an eye on, it may have legs.

Rhapsody on my Mac

I really like the Rhapsody music service. I have been trying various ways to get this to work on my Mac.

There is a plugin for the browser that lets you access Rhapsody natively on the Mac. The only problem is that it regularly crashes Firefox, and Rhapsody support clearly has fixing this as a very low priority. I tried Safari, and that was working OK, until I upgraded to OSX 10.5.3, and now Safari regularly crashes my entire machine! Sorry, no go.

So I finally gave in, and am happily running Rhapsody natively on my Win XP VMWare instance. I reduced the memory on the instance to 512MB otherwise if I have two instances of NetBeans up my system memory starts thrashing. But once I did that I am happy as a clam. Mozart is lilting to me as I code, and all is well with the world.

Open Source EC2 - the beginning of a Scale Stack

In my last post I mentioned that what I wanted to see was the industry coalesce around an open source "scale stack" [1].

We may be seeing the beginnings of this. The High Scalability site (great blog, by the way) recently posted a blog talking about a new kid in town coming out of UC Santa Barbara called Eucalyptus . They are providing an open source implementation an elastic compute infrastructure that is interface-compatible with Amazon's EC2 which you can take and deploy on your own hardware.

This is very encouraging, and I think is a smart approach. Rather than try to build some standard that is lost in committee for years, use the de-facto standard, which in this space is Amazon.

Another piece of the puzzle is Hadoop, an open source implementation of map/reduce. Hadoop also has a distributed file system - one thing that might be worth investigating is building an S3 layer on top of Hadoop's file system.

What about the queuing service? Well, one possibility is to but an SQS API on top of ActiveMQ or OpenJMS.

Throw in CouchDB, and you're starting to get a very interesting stack indeed. I'm not sure about putting a SimpleDB interface on top of this - CouchDB is pretty darn interesting in its own right, and I think the jury is still out on SimpleDB.

[1] I am not sure if he wants me to mention his name, so I won't, but I want to acknowledge that the idea for an open source stack based on Amazon's APIs is not my own, but comes from a colleague at Sun. I think it's a great idea, and may it come to fruition.

The exponential cost of contention

I enjoy Nati Shalom's blog, although it always has that taste of having the agenda of pushing Gigaspace's solutions. But putting that aside his posts are always well thought-out and well written.

I think his latest blog on the Economies of Non-Scale really drive some points home about scalability, or more to the point, the cost of non-linear scalability.

If 90% of our application is free of contention, and only 10% is spent on a shared resources, we will need to grow our compute resources by a factor of 100 to scale by a factor of 10! Another important thing to note is that 10x, in this case, is the limit of our ability to scale, even if more resources are added.

...

1. The cost of non-linearly scalable applications grows exponentially with the demand for more scale.

2. Non-linearly scalable applications have an absolute limit of scalability. According to Amdhal's Law, with 10% contention, the maximum scaling limit is 10. With 40% contention, our maximum scaling limit is 2.5 - no matter how many hardware resources we will throw at the problem

That's something to chew on. These are real costs, both to your business, to your users, and to the environment. Even if you only have a teeny 10% contention in your system, that 10% will nail you faster than you can say ACID semantics. And as has become very clear to me the final breaking point, the final point of contention in any traditional web application architecture, is the database. Get rid of that and you're home free.

How do you do that? Well, there are a lot of people trying to solve this problem with things like space-based architectures, eventual consistency, distributed map/reduce and Stonebraker's H-Store architecture. Anything to let each instance stand on its own and not have to serialize with the rest of the system at any point, in any form.

Some people argue that scalability is so hard that you shouldn't think about it until you need to. But I really believe that if you do enough to educate yourself and make some wise choices, you will be very glad you did.

What I'd like to see is the industry coalescing around some best practices, an open source "scale stack" ala LAMP, tools, community, and hosting environments like Amazon, that allow developers to easily build applications that will scale from the get go. That's where I want to see things go. That way you don't have to throw your hands up and hope for the best. Because as you can see, the costs can be deadly.

Something to think about...

Greed and vision

I was listening to NPR this morning to a fascinating analysis by someone who has talked to everyone from homeowners to lenders to Wall Street about the home loan fiasco. He said everybody up and down the chain was taking unreasonable risks, and everyone was making tons of money off of it.

Then he quoted a financial expert, I wish I could remember who, who said

It's amazing how hard it is for someone to see something when they are being paid a small fortune not to see it.

I just had to laugh, that really sums it up.

It isn't because you're dirty, it isn't because you're clean

My daughter is currently on antibiotics, and she's staying home from school because she has a sniffle.

Why is such an innocent sniffle the cause for keeping her at home, and why is she on antibiotics? Pertussis, or whooping cough.

My daughter goes to a private school, so vaccinations are optional. Many chose not to vaccinate for pertussis (this is actually one we did), and these are the consequences. The Department of Health for the county closed the school for the day on Friday, and is requiring that everyone, regardless of whether they've been vaccinated or not, either take a 5-day protocol of antibiotics or stay home for 21 days. They also have implemented a no-symptoms policy, so anybody with any symptoms, including a sniffle, has to stay home. Sorry, Ariel.

So, swirling around this is the whole debate of vaccinations. As a father, I was stunned at the number and frequency of vaccinations we are supposed to give our children, and which you are required to have before you can enter public schools. I understand, this is all in the name of public health, but as a parent you have to use your intuition a lot, and this just intuitively feels like overkill, literally. It reminds me of the latest discoveries I read about in Science News where there are bacteria in dirt that, when inhaled, create a very strong immunity. Children who are kept away from dirt and kept super-clean are more prone to disease.

I understand vaccinating against horrific and fatal diseases. But many of these vaccinations are for diseases that many of us older folks had as a child: chicken pox, measles, mumps. It was just part of growing up. And there are some real issues here. I am no doctor, but as I understand it, if you are vaccinated against chicken pox and never get it as a kid, then you have a chance of getting it as a teenager, and then it can be *really* bad, whereas if you get it as a kid, then you get a much stronger immunity and generally don't get it when you're older.

I understand then challenge of maintaining public health over trying to protect your child from an army of doctors with needles. I mean, I would be horrified if my child or someone else's child died because of my unwillingness to vaccinate. But at the same time, I wonder what is truly the right thing to do here. We are often so focused on eliminating all and any suffering from this world that we fail to see the ultimate consequences. As an example, part of the reason the plague pounced upon Europe so badly in the middle ages is because humans had been so successful at eradicating the wolf, which, you guessed it, was the primary predator for rats.

I couldn't tell you what the unexpected consequences may be for so much vaccination, but I can tell you that it feels like overkill, and I personally recommend that as a parent you do your research and try to make informed choices about which vaccinations you give your kids, rather than blindly doing none or blindly following the protocol laid out by the powers that be.

MySpace to share data

This just in:
MySpace is going to allow users to share information from their site.

This is definitely a step in the right direction, but notice that sharing is done either by copying/synching, or through their specific APIs. It doesn't fully extract the content into a separate layer with application-provider-agnostic service APIs.

One can imagine an OpenSocial API on top of a social network data service that is used by numerous providers but owned by none.

Giant Food & Biotech Corporations Make Billions in Profit from Growing Global Food Crisis

Big Food is serious big business, and thus is inherently evil (profits over social needs, always - the corporate credo).

http://www.organicconsumers.org/articles/article_12088.cfm

Internet content by reference, not by value

You may have noticed that I am very interested in how data is managed on the Internet as a platform, at a web scale. In that light, I have been having some very illuminating and interesting conversations with an old friend and colleague, Nitin Borwankar. His thoughts on data property rights and DRY data are concepts that if implemented could result in a major shift in how we manage data on the web.

Data property rights is about laying out a "bill of rights" for data that goes far beyond "the right to move". It also includes the right to access, modify, remove and own your data. So often it happens that once you upload your content to a site, you no longer have full rights to that content, as if somehow in the act of uploading it it is no longer yours. It's like living in a serfdom where you do all the work to plow, seed, tend and harvest the land, but the fruit of your labor is not yours, just because you are using the land that someone else owns.

DRY data is about following the principle of Don't Repeat Yourself for web content. Web applications need to start applying this principle, so that rather than you having to load copies of your content across multiple sites (and losing ownership of it in the process), you place it in one location (your "home" on the Web, as it were), and then you refer application providers to that one place. They can focus on providing added value (for instance, referring it to your friends, enabling collaboration, or helping you organize it or present it in useful ways) rather than on the overhead of building and deploying a scalable storage architecture.

Nitin calls this architecture YINAS (YINAS Is Not A Silo).



The value of DRY for the user is obvious - I only have to put my stuff in one place, and I get to really own my stuff, rather than the vendor owning it. DRY is also very valuable for the vendor, as they can save overhead and complexity by delegating the work of scalable storage and indexing to a "data service provider" rather than having to do it themselves. It's even good for the environment, because you need fewer disk farms sucking up power and space. I guess the only folks who would lose out are the storage and power vendors :)

It's funny, it makes so much sense, but nobody is really doing this.

I pulled Tim Bray aside at Java One to talk to him about these ideas after reading his blog about changing his address, and he suggested that concepts are good, but a simple proof of concept is better. Hm... let me think about that ... :)

Meeting Good Friends at CommunityOne

Lots going on these days, hard to keep up. I have been particularly busy, and that's why I've been quiet. I suspect you probably don't mind too much, dear reader, because if you're like me, you probably experience information overload on a regular basis.

I was at CommunityOne today, and, as seems to be happening these days, I spent most of my time saying Hi to old and new friends catching up and sharing our thoughts and feelings about what is going on in the industry.

I ran into Ted Leung and he shared some of the work he's doing to try and get some of the powerful OpenSolaris features like DTrace working in the various dynamic language environments.

Simon Phipps was hanging out (literally, his feet dangling from the RedMonk UnConference stage during a break) and we ruminated on the fact that Sun seems to be held to a higher standard than many companies in the industry - people see us as a Good Company and so they get very upset when it looks like we're being Corporate and Just Like Everyone Else. There is a sense of betrayal and lost hope. So in a way, it's a positive reflection on Sun - we in general have been good to the community, and on the up and up, so any indication that we're Falling to the Dark Side creates a passionate response.

I bumped into Joe Keller, who was one of my first VPs at Sun when the little company I was at was acquired by Sun. Now he's working for Kapow! (now there's a name), and it's fascinating what they do. He described it, and here's how I interpret it: they're doing data integration at Web Scale. Their customers define a master database schema, look at the web pages that they're interested in (say governmental pages for each state in the USA), and define a mapping between the HTML elements on these pages and the elements in their schema. Then the Kapow! software does its magic, scraping these screens at various semantic levels (screen, HTML, through APIs, etc.), and slurping this data into their database schema. Customers then use their data warehousing solution to analyze the resulting data. As an example, one customer of Kapow! is the company Sun hires to do background checks on potential employees. Yipes!

I saw my old boss Dan Leighton, now an independent consultant, but who was instrumental in setting up the database group and defining our database strategy, working for Rich Green, which ultimately led to our involvement in Java DB, PostgreSQL, and, of course MySQL. So he was happily surprised to hear of Sun's acquisition of MySQL.

I bumped into Rob Stephens, someone else who came from the Clustra acquisition and is now in the Solaris org. He told me I should check out the new release of VirtualBox, the open source virtualization solution whose parent company, which was recently acquird by Sun. Rob's got a very strong sysadmin background, so when he says it's good, I trust him. I had tried out VirtualBox about a year ago, and he says it's really improved. Downloaded, installed, and now pulling down the first GA release of OpenSolaris and will put it in its virtual box and put it through its paces.

I ran into Francois Orsini, who is demonstrating some very interesting stuff with Java DB and Grizzly. I ran into a Sun colleague working on a big Rails project, and we talked about web platforms (Rails vs. PHP vs. Java), scalability, and the place of the relational database in a platform that needs web scale. All very interesting stuff. We'll see how the day goes tomorrow!

OpenSolaris on Amazon EC2

Wow, great news. OpenSolaris has some very powerful technology, particularly DTrace and ZFS. Having this available as an Amazon Machine Instance (AMI) on EC2 is sweet!

http://www.sun.com/third-party/global/amazon/index.jsp

Windows Live Dev : Introducing Live Mesh

Very nice, very interesting stuff. This is in line with a lot of what I've been thinking about. Too bad your stuff has to go onto *their* servers and is tied to a very proprietary company. Imagine doing the same thing in an open source model.

http://dev.live.com/blogs/devlive/archive/2008/04/22/279.aspx

Paul Graham's Business Strategy: Be Good

"Being good is a particularly useful strategy for making decisions in complex situations because it's stateless... it's the only algorithm that works on [a large] scale." Great article.

http://www.paulgraham.com/good.html

Seeped in MySQL

This is day three of the MySQL conference for me - usually when people hit burn-out, in my experience. And I'm definitely getting there - I left the Sun party early, my tongue and ears tired of talking and listening.

I have met a lot of great folks here - I was on booth duty a lot and people just kept coming by. This includes saying hello to our new colleagues from MySQL, hello to my old buddies in the database group, and also hello to very good old friends from my Sybase days.

Just as an example, I ran into Robert Hodges, who I used to sit near in the B-Treeum (our building next to the Atrium) in Emeryville. He was the lead on the message queue product, and I led the app server team. Now he's the CTO of Continuent.

Hi, Robert!



I also ran into Wen Hu. I worked for him when he was the lead architect for a massively parallel database engine Sybase was working on with Teradata called Navigation Server. Haven't heard of it? Oh well. Then he moved on to implement the partitioning and parallel query support for the Sybase database kernel. I think MySQL is just now implementing something like that. Now Wen is the CTO for ScaleDB. Sorry, no photo of Wen.

Both of these guys are working on scalability solutions for databases. That is the one theme that repeated here over and over: scalability. All these web properties are encountering scalability requirements that actually go way beyond enterprise scale requirements. You basically need to be able to scale, well, infinitely, and linearly. It's amazing to me that this tiny database engine originally built for quick departmental web apps is now having to address requirements that go way beyond anything Oracle or its peers from the 1980s was built for.

And all the folks I talk to agree, standard transactional/relational approaches just ain't gonna cut it. Something has to give. There's even an acronym for this that I just learned at this conference: CAPS - Consistency, Availability, Performance, Scalability[1].

As Robert explained it at his talk, these are four qualities, and you can't have all of them at once. Something has to give. For a lot of folks, what they are most ready to let go of is consistency. For example, if I write something, I want to see it show up right away. But another reader may not see it for a while (we're talking on the order of seconds, maybe a minute). And that's fine. You get this if, for example, you do asynchronous replication to multiple nodes/sites, rather than trying to do consistent, synchronous replication. And as you can imagine, asynchronous replication scales much better than synchronous replication.

This is the model Amazon has taken, to great success, and I believe it's time for this to spread to the mainstream. The important thing (again, thanks to Robert for crystallizing this for me) is that the levels of consistency need to be clearly defined and guaranteed, like the transaction isolation levels.

I talked to some MySQL folks, and they seem to be looking very seriously at this issue of CAPS, and at these alternate storage solutions like Hadoop/BigTable, SimpleDB and CouchDB. So I look forward to seeing if and how the MySQL folks address this growing need.

---

[1] I have searched for a reference to CAPS on Google, and can find nothing, so it must be something new. If anyone has a link, please let me know.

Phanfare, S3 and Google Apps

I'm looking very seriously at moving to Phanfare as a replacement for Flickr and YouTube for my photos and videos. Their video quality is much better, and I can use it for video archival as well as display. My friends and family can even download the videos at DVD quality and play them offline or even burn them to DVD. Now that's cool.

The cost: $59 a month. Worth it in my book, and since I'm paying up front I'm pretty sure I don't have to worry about advertising polluting my content. Hey, waddya know, a different business model than free-with-ads, which I am so tired of.

I also like it that they're on S3.

They also have a good blog. I particularly liked their analysis of the benefits of the Amazon Web Services over Google Apps. I think they provide a great testimonial for the value of following open standards - something Sun has always believed in.

This just in: Google Docs offline

I was putzing around with a meeting agenda on Google Docs and noticed an "Offline - New!" link at the top right.

Sure enough, it prompts you to download Google Gears and you can work with your documents offline.

Killer feature. Ouch to Microsoft.