Monday, June 30, 2008

Theo Schlossnagle - Traffic spikes through the heart of your app

Theo Schlossnagle's blog is a very interesting read indeed.
What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.
That's something to think about.

One likes to think that cloud computing is the silver bullet for scalability - use their stuff, build a system that assumes scale, and no worries, mate. But this makes it clear that there are basic operational issues that need to be kept in mind. Doing it manually may not cut it.

Amazon for example provides no tools for handling traffic increases automatically. It's your job to monitor and kick off new instances. But if this is happening in a matter of 60 seconds, you'd better be very sure you know what to do, and quickly.

Personally, I've never had to deal with this, so I have no tips for you. If I were to start up a new site, I'd go find people who do and have a long talk with them.

Friday, June 27, 2008

New DB Features coming in NetBeans 6.5

Some cool stuff coming down the pike for NetBeans 6.5 database tooling.

SQL Editor code completion - lots of you have been asking for this, and it's in, with more on the way. Note how completion brings up the columns if the SELECT statement already specifies the table name.

completion-1.jpg

completion-2.jpg


SQL History
- You can now look at the history of all SQL you've executed, and can to text filtering and/or filter by connection URL, and then select and insert a statement.

history.jpg


Editable, sortable results, multiple result tabs

This isn't integrated yet, but I had to show it to you. With this you can sort the results by double-clicking on a row header (we should have done this a long time ago), modify a row, insert and delete rows, there is full pagination support, and you can optionally create a new tab for each statement that returns results, very nifty for comparing results back and forth.

This feature actually came from the SOA team under a completely different part in Sun - open source works inside as well as outside. Thanks to Ahi and Nithya and Nilesh for this!

dataview.jpg


Tired of typing in a JDBC URL?

Many of our users, and they're growing, aren't even doing Java, but when they want to connect to a database, the dialog is all JDBC-ish. We've fixed that - now you just type in the parameters and we figure out the URL for you.

It turns out this is no walk in the park, as each vendor has their own parameters and URL format. So for now it works for MySQL, PostgreSQL, Java DB, Oracle and Microsoft SQL Server. For the rest, sorry, you still need to type in the URL, until I can get to them.

BTW, if you still want to see and edit the URL, you can, and we'll update the fields automatically. Nifty.

new-connection.jpg

Tuesday, June 24, 2008

Jerry Seinfeld on George Carlin’s Life and Comedy

I loved George Carlin as a kid. I once thought of a great joke - what is the worst ice cream flavor you ever heard of. I went to a Carlin concert, and he had the same joke, with the best answer: "bologna swirl." Carlin already did it.

http://www.nytimes.com/2008/06/24/opinion/24seinfeld.html?em&ex=1214452800&en=33d98b4f11507cbf&ei=5087%0A

Friday, June 20, 2008

Query languages for Hadoop, AND tooling for the non-relational stores

Tom White just blogged about three, count them three, different query languages being built for Hadoop.
  • Pig, from Yahoo! and now incubating at Apache, has an imperative language called Pig Latin for performing operations on large data files.
  • Jaql, from IBM and soon to be open sourced, is a declarative query language for JSON data.
  • Hive, from Facebook and soon to become a Hadoop contrib module, is a data warehouse system with a declarative query language that is a hybrid of SQL and Hadoop streaming.

As someone working on database tooling, I have been thinking about what we at NetBeans might do to make it easier for developers to build apps against these new platforms/products like Hadoop and CouchDB and Amazon services.

These query languages are an example - I can see an interactive query editor that lets you write and test queries directly against your Hadoop or CouchDB engine, much as you can today with SQL tools.

Another interesting question is - if I write my application in terms of domain objects, what things can a tool do to make it easy to map these domain objects to an underlying non-structured store like Hadoop or CouchDB or SimpleDB.

Definitely some opportunities here to make developers' lives easier.

Thursday, June 19, 2008

Kaj Arnö - Happily using NetBeans to build MySQL

After getting a demo from a NetBeans engineer on the C/C++ extensions to NetBeans, Kaj was converted - "now this was enough even for a sceptic like me to become eager."

http://blogs.mysql.com/kaj/2008/06/18/netbeans-as-ide-for-developing-mysql-itself/

Alex Bunardzic » Software Development Detox Part 5: Session

Someone who is in my boat about session. Although I still wonder if you are overburdening the client if you say "on the web each request contains all the information necessary for the server code to make a decision on what to do next."

http://jooto.com/blog/index.php/2008/06/20/software-development-detox-part-5-session/

Wednesday, June 18, 2008

Alex Bunardzic » World Wide Web is About Self-Serve

Interesting blog challenging us to question if there is a need for services, when really the web is basically about resources. I think the real service is in quality of service, making sure the resource is available when you need it.

http://jooto.com/blog/index.php/2008/06/19/world-wide-web-is-about-self-serve/

Monday, June 16, 2008

AOL Radio

I have been a long-time member of the "AOL sucks" club. I signed my name in blood as a member of this club when I tried for three days to get Internet access working for my laptop in someone's home that had AOL as the Internet provider. AOL has historically provided a horrific user interface which constantly thinks it knows better than you and gets in your way whenever you want to do anything different from the standard.

But I've found one piece of AOL that is just, what can I say, nice... It's AOL Radio, which I bumped into when reading a NY Times article about how AOL Radio will soon be available on the iPhone. I tried the Acoustic Rock station, and it's really quite high quality, and the user interface, is, well, simple and - gasp - easy to use!

I have tried a number of interfaces for free Internet radio, and they are either impossible to use, have a horrible selection, or are filled with ads. AOL Radio has none of these limitations. And all you need is Flash to get going. They do seem to have song-skipping disabled, but that's supposed to come back soon.

Pretty nice, and from such an unexpected source! :)

Friday, June 13, 2008

Facebook's Thrift in Apache Incubator

From the Facebook blog:
Thrift is a lightweight software framework for enabling communication between programs written in different programming languages, running on different computers, or both.

It includes support for C, Java, C++, Ruby, Erlang, Perl, Haskell and many others.

It sounds very interesting. I first heard of Thrift because it's used by a very interesting Amazon-like stack called Thrudb. What's nice is because it works with Thrift, you can use it from any language.

However, I looked at the interface for calling the Thrift APIs, and it's a bit ugly, or at least it takes some getting used to.

Anyway, something to keep an eye on, it may have legs.

Thursday, June 12, 2008

Rhapsody on my Mac

I really like the Rhapsody music service. I have been trying various ways to get this to work on my Mac.

There is a plugin for the browser that lets you access Rhapsody natively on the Mac. The only problem is that it regularly crashes Firefox, and Rhapsody support clearly has fixing this as a very low priority. I tried Safari, and that was working OK, until I upgraded to OSX 10.5.3, and now Safari regularly crashes my entire machine! Sorry, no go.

So I finally gave in, and am happily running Rhapsody natively on my Win XP VMWare instance. I reduced the memory on the instance to 512MB otherwise if I have two instances of NetBeans up my system memory starts thrashing. But once I did that I am happy as a clam. Mozart is lilting to me as I code, and all is well with the world.

Tuesday, June 10, 2008

Open Source EC2 - the beginning of a Scale Stack

In my last post I mentioned that what I wanted to see was the industry coalesce around an open source "scale stack" [1].

We may be seeing the beginnings of this. The High Scalability site (great blog, by the way) recently posted a blog talking about a new kid in town coming out of UC Santa Barbara called Eucalyptus . They are providing an open source implementation an elastic compute infrastructure that is interface-compatible with Amazon's EC2 which you can take and deploy on your own hardware.

This is very encouraging, and I think is a smart approach. Rather than try to build some standard that is lost in committee for years, use the de-facto standard, which in this space is Amazon.

Another piece of the puzzle is Hadoop, an open source implementation of map/reduce. Hadoop also has a distributed file system - one thing that might be worth investigating is building an S3 layer on top of Hadoop's file system.

What about the queuing service? Well, one possibility is to but an SQS API on top of ActiveMQ or OpenJMS.

Throw in CouchDB, and you're starting to get a very interesting stack indeed. I'm not sure about putting a SimpleDB interface on top of this - CouchDB is pretty darn interesting in its own right, and I think the jury is still out on SimpleDB.

[1] I am not sure if he wants me to mention his name, so I won't, but I want to acknowledge that the idea for an open source stack based on Amazon's APIs is not my own, but comes from a colleague at Sun. I think it's a great idea, and may it come to fruition.

Thursday, June 05, 2008

The exponential cost of contention

I enjoy Nati Shalom's blog, although it always has that taste of having the agenda of pushing Gigaspace's solutions. But putting that aside his posts are always well thought-out and well written.

I think his latest blog on the Economies of Non-Scale really drive some points home about scalability, or more to the point, the cost of non-linear scalability.
If 90% of our application is free of contention, and only 10% is spent on a shared resources, we will need to grow our compute resources by a factor of 100 to scale by a factor of 10! Another important thing to note is that 10x, in this case, is the limit of our ability to scale, even if more resources are added.

...

1. The cost of non-linearly scalable applications grows exponentially with the demand for more scale.

2. Non-linearly scalable applications have an absolute limit of scalability. According to Amdhal's Law, with 10% contention, the maximum scaling limit is 10. With 40% contention, our maximum scaling limit is 2.5 - no matter how many hardware resources we will throw at the problem
That's something to chew on. These are real costs, both to your business, to your users, and to the environment. Even if you only have a teeny 10% contention in your system, that 10% will nail you faster than you can say ACID semantics. And as has become very clear to me the final breaking point, the final point of contention in any traditional web application architecture, is the database. Get rid of that and you're home free.

How do you do that? Well, there are a lot of people trying to solve this problem with things like space-based architectures, eventual consistency, distributed map/reduce and Stonebraker's H-Store architecture. Anything to let each instance stand on its own and not have to serialize with the rest of the system at any point, in any form.

Some people argue that scalability is so hard that you shouldn't think about it until you need to. But I really believe that if you do enough to educate yourself and make some wise choices, you will be very glad you did.

What I'd like to see is the industry coalescing around some best practices, an open source "scale stack" ala LAMP, tools, community, and hosting environments like Amazon, that allow developers to easily build applications that will scale from the get go. That's where I want to see things go. That way you don't have to throw your hands up and hope for the best. Because as you can see, the costs can be deadly.

Something to think about...

Tuesday, May 27, 2008

Greed and vision

I was listening to NPR this morning to a fascinating analysis by someone who has talked to everyone from homeowners to lenders to Wall Street about the home loan fiasco. He said everybody up and down the chain was taking unreasonable risks, and everyone was making tons of money off of it.

Then he quoted a financial expert, I wish I could remember who, who said
It's amazing how hard it is for someone to see something when they are being paid a small fortune not to see it.
I just had to laugh, that really sums it up.

Monday, May 12, 2008

It isn't because you're dirty, it isn't because you're clean

My daughter is currently on antibiotics, and she's staying home from school because she has a sniffle.

Why is such an innocent sniffle the cause for keeping her at home, and why is she on antibiotics? Pertussis, or whooping cough.

My daughter goes to a private school, so vaccinations are optional. Many chose not to vaccinate for pertussis (this is actually one we did), and these are the consequences. The Department of Health for the county closed the school for the day on Friday, and is requiring that everyone, regardless of whether they've been vaccinated or not, either take a 5-day protocol of antibiotics or stay home for 21 days. They also have implemented a no-symptoms policy, so anybody with any symptoms, including a sniffle, has to stay home. Sorry, Ariel.

So, swirling around this is the whole debate of vaccinations. As a father, I was stunned at the number and frequency of vaccinations we are supposed to give our children, and which you are required to have before you can enter public schools. I understand, this is all in the name of public health, but as a parent you have to use your intuition a lot, and this just intuitively feels like overkill, literally. It reminds me of the latest discoveries I read about in Science News where there are bacteria in dirt that, when inhaled, create a very strong immunity. Children who are kept away from dirt and kept super-clean are more prone to disease.

I understand vaccinating against horrific and fatal diseases. But many of these vaccinations are for diseases that many of us older folks had as a child: chicken pox, measles, mumps. It was just part of growing up. And there are some real issues here. I am no doctor, but as I understand it, if you are vaccinated against chicken pox and never get it as a kid, then you have a chance of getting it as a teenager, and then it can be *really* bad, whereas if you get it as a kid, then you get a much stronger immunity and generally don't get it when you're older.

I understand then challenge of maintaining public health over trying to protect your child from an army of doctors with needles. I mean, I would be horrified if my child or someone else's child died because of my unwillingness to vaccinate. But at the same time, I wonder what is truly the right thing to do here. We are often so focused on eliminating all and any suffering from this world that we fail to see the ultimate consequences. As an example, part of the reason the plague pounced upon Europe so badly in the middle ages is because humans had been so successful at eradicating the wolf, which, you guessed it, was the primary predator for rats.

I couldn't tell you what the unexpected consequences may be for so much vaccination, but I can tell you that it feels like overkill, and I personally recommend that as a parent you do your research and try to make informed choices about which vaccinations you give your kids, rather than blindly doing none or blindly following the protocol laid out by the powers that be.

Friday, May 09, 2008

MySpace to share data

This just in:
MySpace is going to allow users to share information from their site.


This is definitely a step in the right direction, but notice that sharing is done either by copying/synching, or through their specific APIs. It doesn't fully extract the content into a separate layer with application-provider-agnostic service APIs.

One can imagine an OpenSocial API on top of a social network data service that is used by numerous providers but owned by none.

Giant Food & Biotech Corporations Make Billions in Profit from Growing Global Food Crisis

Big Food is serious big business, and thus is inherently evil (profits over social needs, always - the corporate credo).

http://www.organicconsumers.org/articles/article_12088.cfm

Thursday, May 08, 2008

Internet content by reference, not by value

You may have noticed that I am very interested in how data is managed on the Internet as a platform, at a web scale. In that light, I have been having some very illuminating and interesting conversations with an old friend and colleague, Nitin Borwankar. His thoughts on data property rights and DRY data are concepts that if implemented could result in a major shift in how we manage data on the web.

Data property rights is about laying out a "bill of rights" for data that goes far beyond "the right to move". It also includes the right to access, modify, remove and own your data. So often it happens that once you upload your content to a site, you no longer have full rights to that content, as if somehow in the act of uploading it it is no longer yours. It's like living in a serfdom where you do all the work to plow, seed, tend and harvest the land, but the fruit of your labor is not yours, just because you are using the land that someone else owns.

DRY data is about following the principle of Don't Repeat Yourself for web content. Web applications need to start applying this principle, so that rather than you having to load copies of your content across multiple sites (and losing ownership of it in the process), you place it in one location (your "home" on the Web, as it were), and then you refer application providers to that one place. They can focus on providing added value (for instance, referring it to your friends, enabling collaboration, or helping you organize it or present it in useful ways) rather than on the overhead of building and deploying a scalable storage architecture.

Nitin calls this architecture YINAS (YINAS Is Not A Silo).



The value of DRY for the user is obvious - I only have to put my stuff in one place, and I get to really own my stuff, rather than the vendor owning it. DRY is also very valuable for the vendor, as they can save overhead and complexity by delegating the work of scalable storage and indexing to a "data service provider" rather than having to do it themselves. It's even good for the environment, because you need fewer disk farms sucking up power and space. I guess the only folks who would lose out are the storage and power vendors :)

It's funny, it makes so much sense, but nobody is really doing this.

I pulled Tim Bray aside at Java One to talk to him about these ideas after reading his blog about changing his address, and he suggested that concepts are good, but a simple proof of concept is better. Hm... let me think about that ... :)

Monday, May 05, 2008

Meeting Good Friends at CommunityOne

Lots going on these days, hard to keep up. I have been particularly busy, and that's why I've been quiet. I suspect you probably don't mind too much, dear reader, because if you're like me, you probably experience information overload on a regular basis.

I was at CommunityOne today, and, as seems to be happening these days, I spent most of my time saying Hi to old and new friends catching up and sharing our thoughts and feelings about what is going on in the industry.

I ran into Ted Leung and he shared some of the work he's doing to try and get some of the powerful OpenSolaris features like DTrace working in the various dynamic language environments.

Simon Phipps was hanging out (literally, his feet dangling from the RedMonk UnConference stage during a break) and we ruminated on the fact that Sun seems to be held to a higher standard than many companies in the industry - people see us as a Good Company and so they get very upset when it looks like we're being Corporate and Just Like Everyone Else. There is a sense of betrayal and lost hope. So in a way, it's a positive reflection on Sun - we in general have been good to the community, and on the up and up, so any indication that we're Falling to the Dark Side creates a passionate response.

I bumped into Joe Keller, who was one of my first VPs at Sun when the little company I was at was acquired by Sun. Now he's working for Kapow! (now there's a name), and it's fascinating what they do. He described it, and here's how I interpret it: they're doing data integration at Web Scale. Their customers define a master database schema, look at the web pages that they're interested in (say governmental pages for each state in the USA), and define a mapping between the HTML elements on these pages and the elements in their schema. Then the Kapow! software does its magic, scraping these screens at various semantic levels (screen, HTML, through APIs, etc.), and slurping this data into their database schema. Customers then use their data warehousing solution to analyze the resulting data. As an example, one customer of Kapow! is the company Sun hires to do background checks on potential employees. Yipes!

I saw my old boss Dan Leighton, now an independent consultant, but who was instrumental in setting up the database group and defining our database strategy, working for Rich Green, which ultimately led to our involvement in Java DB, PostgreSQL, and, of course MySQL. So he was happily surprised to hear of Sun's acquisition of MySQL.

I bumped into Rob Stephens, someone else who came from the Clustra acquisition and is now in the Solaris org. He told me I should check out the new release of VirtualBox, the open source virtualization solution whose parent company, which was recently acquird by Sun. Rob's got a very strong sysadmin background, so when he says it's good, I trust him. I had tried out VirtualBox about a year ago, and he says it's really improved. Downloaded, installed, and now pulling down the first GA release of OpenSolaris and will put it in its virtual box and put it through its paces.

I ran into Francois Orsini, who is demonstrating some very interesting stuff with Java DB and Grizzly. I ran into a Sun colleague working on a big Rails project, and we talked about web platforms (Rails vs. PHP vs. Java), scalability, and the place of the relational database in a platform that needs web scale. All very interesting stuff. We'll see how the day goes tomorrow!

OpenSolaris on Amazon EC2

Wow, great news. OpenSolaris has some very powerful technology, particularly DTrace and ZFS. Having this available as an Amazon Machine Instance (AMI) on EC2 is sweet!

http://www.sun.com/third-party/global/amazon/index.jsp

Thursday, April 24, 2008

Windows Live Dev : Introducing Live Mesh

Very nice, very interesting stuff. This is in line with a lot of what I've been thinking about. Too bad your stuff has to go onto *their* servers and is tied to a very proprietary company. Imagine doing the same thing in an open source model.

http://dev.live.com/blogs/devlive/archive/2008/04/22/279.aspx