Shut up and write those tests

One of the things I love about my new job is that this team is committed to the craft of software development, and is introducing me to Deep Testing.  I was pointed to this excellent book, Working Effectively with Legacy Code by Michael Feathers, and to this amazing little toolkit called Mockito.


Working Effectively with Legacy Code assumes a Test-Driven Development attitude and philosophy, and then tells you how to go into a hairball of existing code and get it to be testable.  Chapter titles include "I Can't Get This Class into a Test Harness" and "I Don't  Understand the Code Well Enough to Change It," and includes techniques for breaking that bugaboo of unit testing, hard-coded dependencies, like "Extract and Override Method Call," "Introduce Instance Delegator" and "Subclass and Override Method."  Excellent stuff.

Mockito is a mocking framework that is just insanely easy to use.  Here is a piece of code I wrote where the class to test had a large number of dependencies, and I just mocked out all but the relevant ones.  This took me about ten minutes to write, and it was an excellent way to test impact of the changes I was making:

Checkpointable mockCheckpointable = mock(Checkpointable.class);
CrawlerContent mockCrawlerContent = mock(CrawlerContent.class);
List crawlerItems = new ArrayList();
crawlerItems.add(mockCrawlerContent);

ContentStream mockContentStream = mock(ContentStream.class);

when(mockCheckpointable.getItems()).thenReturn(crawlerItems);
when(_mockCrawlerContent.getContentStream()).thenReturn(mockContentStream);

_messageManager.sendCheckpoint(mockCheckpointable);

verify(_mockContentBuffer).handleContentFromCrawler(mockContentStream, mockCrawler);
verify(_mockWalkUpdater).handleCheckpoint(mockCheckpointable);

With tools like this in hand, plus an attitude of commitment to excellence, I now firmly believe you can wrestle your code to the ground and get it into a unit test framework and start improving not only the testability and quality but overall architecture of your code.  I don't care if your code is full of massive, crufty code you can't understand, with lots of accesses to the network and the database, you can write unit tests that test modules independently. 

So shut up and write those tests.  You have no excuses.

Ayatollah you so - Daily Show

Very funny clip, and sweet too, of the Daily Show in Iran

NYT: Saudi royal family lavishes $$$ on terrorist groups

Looks like the lawyers for the families of Sept 11 got tired of trying to get the justice system to hear them, so they took it to the press. They gave the New York Times a stack of damning evidence linking the House of Saud to Al Qaeda, Taliban, and others.  From the article:

A self-described Qaeda operative in Bosnia said in an interview with lawyers in the lawsuit that another charity largely controlled by members of the royal family, the Saudi High Commission for Aid to Bosnia, provided money and supplies to the terrorist group in the 1990s and hired militant operatives like himself.

Another witness in Afghanistan said in a sworn statement that in 1998 he had witnessed an emissary for a leading Saudi prince, Turki al-Faisal, hand a check for one billion Saudi riyals (now worth about $267 million) to a top Taliban leader.

And a confidential German intelligence report gave a line-by-line description of tens of millions of dollars in bank transfers, with dates and dollar amounts, made in the early 1990s by Prince Salman bin Abdul Aziz and other members of the Saudi royal family to another charity that was suspected of financing militants’ activities in Pakistan and Bosnia.

If this is true, this is a Big Deal.  I don't fully understand why the Saudis would want to support these organizations.  Maybe somebody more versed in the ways of the Middle East can explain it to me.

Medical records = transparency = trouble for medical industry

Via @timoreilly, an article in Technology Review gives its take on why the medical industry has been so resistant to going electronic. Surprise - it's all about the money...

Dangling $19 billion in front of a $2.4 trillion industry is not nearly enough to get it to reveal the financial secrets that electronic health records are likely to uncover--and upon which its huge profits depend. In those medical records lie the ugly truth about the business of medicine: sickness is profitable. The greater the number of treatments, procedures, and hospital stays, the larger the profit. There is little incentive for doctors and hospitals to identify or reduce wasteful spending in medicine.

Health care: you get what you pay for

Great article by Atul Gawande in the New Yorker where he does some serious snooping around as to why health care costs are skyrocketing.  His conclusion: it's how we pay the doctors.

Somewhere in the United States at this moment, a patient with chest pain, or a tumor, or a cough is seeing a doctor. And the damning question we have to ask is whether the doctor is set up to meet the needs of the patient, first and foremost, or to maximize revenue.

Iran: it wasn't supposed to happen this way

A great guest editorial in the New York Times  from a student in Iran. It starts slow, but has a wonderful finish, really gives you the sense that everybody is surprised by the sweep of events, even those participating directly in them...

Everyone watched everyone else and we wondered how all of this could be happening. Who were all of these people? Where did they come from? These were the same people we pass by unknowingly every day. We saw one another, it feels, for the first time. Now in the second week, we continue to look at one another as we walk together, in marches and in silent gatherings, toward our common goal of having our vote respected.

...

It wasn’t supposed to happen this way. Until last week, Mr. Moussavi was a nondescript, if competent, politician — as one of his campaign advisers put it to me, he was meant only to be an instrument for making Iran a tiny bit better, nothing more. Iranians knew that’s what they were getting when they cast their votes for him. Now, like us, Mr. Moussavi finds himself caught up in events that were unimaginable, each day’s march and protest more unthinkable than the one that came before.

The Guardian seems to have it right

While other news media seem to be flailing in their attempts to keep up with the speed and detail of information coming from social media, the Guardian in the UK has a live page (updated every minute) that seems to get the balance right: keep an eye on the stream of information, and use journalistic discipline to avoid rumors, innuendo and falsehoods.  As an example, Twitter is posting this picture, and The Guardian says

A picture claimed to be of today's opposition rally has been posted to Twitpic. It shows thousands of people marching on a flyover labelled the "Krimkhan bridge". The date of the picture is impossible to confirm, please let us know if you have more information.

 I think this is great, as it gives me a source of information I feel I can trust that is also up-to-date and using new media sources and not just the traditional reporting resources.

New York Times on social media as the only way to get the word out

A good article that seems to capture the power of Twitter and other social media tools to break down the walls put up by repressive regimes.

I've been reading the Twitter #iranelection stream, and a lot of it is chaotic, doubtful, over-enthusiastic, or silly.  But there are also gems in there, with links to articles and lots of pictures showing the emotion of the demonstrators and the violent repression that is going on.

It definitely gives a different more real and emotionally engaging view of what is unfolding from traditional media, with an article here or there, mostly talking about what they've heard is going on through Twitter, and only showing "sanctioned" pictures of pro-government rallies.

But to me what is most important is not what I can learn about what is going on, but what Iranians can communicate with each other even as the government tries to shut down communications.  I read in a NYT article that demonstrations are being organized through Twitter.  That was great news to hear.  And I pray for them, because the risk of serious injury or death is very real, and they are still moving forward.  Incredibly bravery and determination.

Thinking about getting a gTLD - talk to Antony

Here's a nice interview with my brother, Antony Van Couvering, CEO of Minds+Machines, talking about gTLDs (generic Top-Level Domains), why you might want one, why they're useful, and giving you a good overview of what is involved.

For a brand owner, having your brand name as the destination for your customers on the web is ideal. How much money do major corporations spend on branding, and how effective is Internet visibility? To me, it’s a no-brainer to brand yourself instead of a domain extension like .COM which has nothing to do with your brand. For a city, it’s a tremendous aid to tourism and local business, with great possibilities for civic initiatives. Another key advantage, if you keep your TLD clean (in other words, take steps to discourage warehousing, non-working sites, and template sites), is that search engines will pay attention. For instance, .NYC will be about New York City in a meaningful way, and if Google wants to deliver relevant searches (which it does), then it will favor the .NYC TLD when someone types in “nyc” or “new york city” or “manhattan” or any other New York-related term.

Do we *really* want SQL in HTML5?

Jan Lehnardt's tweet pointed me to this ongoing conversation about the new SQL support in HTML5. Vladimir Vukicevic I think summarizes the issue quite well:

There's been a lot of interest around the Web Storage spec (formerly part of whatwg HTML5), which exposes a SQL database to web applications to use for data storage, both for online and offline use. It presents a simple API designed for executing SQL statements and reading result rows. But there's an interesting problem with this; unlike the rest of the HtML5, this section defines a core piece of functionality in terms of an undefined chunk referenced as "SQL".

The initial implementations of Web Storage are both based on SQLite, and expose the dialect of SQL understood by SQLite to web content. I'm actually a big fan of SQLite, and was one of the advocates for pulling it into the Gecko platform. However, SQLite implements a variant of SQL, with a number of deviations from other SQL engines, especially in terms of the types of data that can be placed in columns.

Web content that is created to use database storage with SQLite as the backing is unlikely to work with any other backend database. Similarly, if another database was chosen as a browser's backing implementation, web content that works with it is unlikely to work with anything else. This is a serious interop problem, the root of which is that there really isn't a useful core SQL standard. SQL92 is generally taken as a base, but is often extended or altered by implementations. Even beyond the parser issues (which could be resolved by defining a strict syntax to be used by Web Storage), the underlying implementation details will affect results.

So, the only option is for the Web Storage portion of the spec to state "do what SQLite does". This isn't specified in sufficient detail anywhere to be able to reimplement it from the documents, so it would be even worse — "do what this exact version of SQLite does", because there are no guarantees that SQLite won't make any incompatible changes. For example, a future SQLite 4 may introduce some changes or some new syntax which wouldn't be supported by earlier versions. Thus, it requires every single browser developer to accept SQLite as part of their
platform.

There then follows a lot of discussion about how to handle this, from creating an ORM layer to fully specifying the standard subset of SQL that will be supported in HTML5 (as if there were a valid standard). One person argues that we can simply refer to the ISO SQL standard and then describe extensions to that standard.

OK, well, I have worked with relational databases for over 20 years, and I have seen how completely stuck customers are with a given database vendor. I don't care how "standard" the standard is, it's merely an illusion. There is a reason Oracle can charge such huge sums of money and smile as customers fume. There is a reason Sybase is still in business. It is almost impossible to move a large database application from one vendor to another.

We all know how difficult it is to write code today that is compatible across all browsers. Imagine if each browser had its own SQL database that "implemented the standard." That's where we're heading, and it sure seems hellish to me. I guess if we forced every browser to use SQLite, we'd be OK, but that seems to placing an awful lot of reliance on a single technology. And as Vladimir points out, that doesn't solve issues of incompatibility across versions of SQLite.

It seems to me the requirements are to have rich storage, particularly for Javascript objects, and a way to query over this storage. I know saying is easier than doing, but shouldn't we be defining an API that defines JSON storage and a JSON query language? The Persevere folks have already defined a JSON query language, and they seem to have put a lot of thought into it. Perhaps that could be used as a start.

I can't say I have a solution, but I'm just raising a flag that creating a standard as important as HTML5 with a placeholder for "SQL" is an invitation for major incompatibilities which we'll be digging ourselves out of for years to come.

Michael sings the train blues

Michael got a little geetar for Christmas, and I was sitting here working yesterday when I stopped and listened to him singing along. I couldn't believe it - serious train blues. I got up and went into the living room and there he was sitting in the middle of his train tracks singing along, "Red Train Going Down the Tracks!".

Listen to him singing about the piston at the end. What to say, infinite cuteness.

Middle age: learning acceptance and contentment

To me a lot of what middle age is about - or perhaps life in general - is accepting your life as it is, and finding a place of contentment, an angle of repose.

We often have ideas about how our life should be going, but that's not the way it's going, and we fight and fight and fight. But at some point we are taken down, usually by some bodily failure (which is where middle age comes in) and we are forced to let go and accept. I suspect that this is the seed of wisdom.

This is also the message I see regularly in Indian scriptures. Krishna says "it is better to live your own dharma poorly than someone else's well". The great statements of Vedanta say "All this is God" and "Nothing but God exists." To me, one aspect of this teaching is that if I am seeing my life as "not good enough" or not where I want to be, then this implies I am not seeing God in my life.

Sometimes this is very hard to accept.

Over the last three weeks I went through a debilitating flu and had to go through a root canal (partially caused by grinding my teeth). I took these as signs of a life out of balance and did some self-inquiry.

First of all, I was exhausted, and secondly I was stressed. This much was obvious. Next question was why.

I have just started a new job after being laid off by Sun. I didn't really expect to make this change, and I don't think I had fully accepted the change. At Sun I was working with some serious rock stars of the industry, and doing some "cool stuff" in open source. In my new job, it's a great team and fun work, but I have been unhappy because (to me) it just wasn't that "cool."

I have also been watching all this interesting stuff going on in the web tier - distributed data stores like CouchDB, rich client development using Javascript/HTML5, Flex and JavaFX, potentials for peer-to-peer collaboration, etc. But none of this has anything to do with my day job.

So I started trying to do this stuff on the side. Keep in mind I'm a father of 2 young kids, and I also just took away two hours from my day for commute time. So I was squeeezing this into late nights and early mornings. Thus the exhaustion. And the grinding teeth: I was pushing and pushing myself, I didn't want to let it go and just do my day job - that would be admitting surrender.

But after these illnesses and further thought and I saw that (a) this wasn't a maintainable lifestyle and (b) I was being proud and egotistical. My day job was "below me" and I was meant for "greater things." Hints of Raskolnikov. I wasn't accepting my life as it is.

So with a sigh I saw it was time to let go. No more late nights trying to hack Javascript and CouchDB. Although part of me feels like I'm letting go the chance to ride this wave, I also have to admit I've been much happier and less stressed since this decision.

It's such a fascinating process. This wasn't the first time I've gone through it, and I doubt it's the last.

Great article on CSS development

I have done very very little CSS development, but I'm starting to dabble a bit.  @petr3 posted a link on Twitter to an excellent article by Jonathan Christopher that both lays out some nice principles on CSS design and provides some valuable links to earlier articles he wrote how to write better CSS and improve your CSS processes.

The principles he mentions reminds me a lot of software design in general:

• find the patterns and reuse wherever possible
• work from generic to specific
• work to keep clarity as you maintain it

Definitely worth a read...

What a concept: a steady state economy

From the first time I heard of this idea that growth was essential to prosperity, it's had me shaking my head. I first heard it in terms of an individual company: a public company is always under pressure to grow, grow, grow. You can't just have one coffee shop, you have to have two. You can't just be a single company with a single focus, you have to conglomerate. Grow grow grow! It just seemed wrong for a company to be forced to grow when it could be doing perfectly well where it is. See the story of the Southern fisherman and the Northern industrialist.

Then I heard about this concept in terms of the economy in general.  I read how a steady state was considered the same as stagnation - like some horrible third circle of hell.  But you just do the math in your head, and you wonder where all this growth is going to take you.  Sure enough, it's not sustainable, and bubbles burst.  And I just sigh when I hear all these economists (who got us into this mess) trying to get us back to a trajectory of growth.  Can't we try something else for a change?

So, I thought I'd share with you this very insightful and entertaining article by Herman Daly, posted on the Oil Drum.  Herman Daly has fun skewering the growth economists (who I'm sure are ostracizing him as we speak) and then goes on to describe a vision of a healthy steady state economy.  You know, the old general store instead of Wal-Mart.

Here's one juicy skewer quote:

Some economists in fact think of nature as the set of extractive subsectors of the economy (forests, fisheries, mines, wells, pastures, and even agriculture….). The economy, not the ecosystem or biosphere, is seen as the whole; nature is a collection of parts. If the economy is the whole then it is not a part of any larger thing or system that might restrain its expansion. If some extractive natural subsector gets scarce we will just substitute other sectors for it and growth of the whole economy will continue, not into any restraining biospheric envelope, but into sidereal space presumably full of resource-bearing asteroids and friendly highly-evolved aliens eager to teach us how to grow forever into their territory. Sources and sinks are considered infinite.

Then Mr. Daly goes on to offer 10 proposals to help reach a healthy steady state economy.  I'm not sure I grok or even agree with all of them, but it's a starting point for a healthy dialog about this.  I recommend having a read...

JSR308 (pluggable type checking) is looking promising

Watching the Twitter stream for JavaOne, I saw a number of folks say they were quite impressed with the talk called "Preventing Bugs with Pluggable Type Checking."  I checked out the link for JSR308, and this looks very promising.

Basically, you can add annotations that check for things such as NonNull or Nullable, and then incorporate a checker that supports these annotations into the compiler.

For example, you can say

import checkers.nullness.quals.*;

public class GetStarted {
    void sample() {
        @NonNull Object ref = new Object();
    }
}

and then say

javac -processor checkers.nullness.NullnessChecker GetStarted.java

and the checker will ensure that the @NonNull object indeed is not being set to null anywhere.
Take a look at the page for JSR308, this looks like a good new addition.

Learn CouchDB through online experimentation - highly recommended

Wow, this is very nice.  It's an all JavaScript emulation of CouchDB.  What makes this super-cool is that this allows you to interactively play around with both CouchDB documents and views.  This is particularly useful when you're trying to understand how views work.  Definitely recommended if you're interested in learning CouchDB.