Now *that's* a meal

Via my cuz, supposedly taken by a road crew in Arizona. As my son would say - eeewwww (grin) !

Wherefore the blog posts?

I am surprised myself to find out how less often I'm blogging now that I'm a more active user of Twitter. I mean, why should that be - two very different mediums, and two very different purposes.

I think it must be a matter of focus and energy. When I have something I want to share, I go to the medium I am most active on/familiar with. Before Twitter (and long after Twitter was around) that was my blog.

But now I find I can share a lot of my thoughts quickly on Twitter. But I also miss the longer musings that I would do on my blogs.

I keep *wanting* to post more stuff on my blog, but it keeps not happening.

It could also be that I'm much busier. Life on NetBeans was a little low key - working from home, the team in Prague, and my feature low on the priority list. Not so at my new job - I've got tight deadlines, I'm the office each day, busy busy busy. So sitting down to write a blog post is often just not an option. I can barely squeeze in a tweet.

I know some of my Dear Readers are not Tweet-aholics and maybe even are avid Twee-totalers. So I did want to share one option - you can view my Twitter stream as an RSS feed. I only post a few things a day at most, so it's not like it will be overwhelming. In particular, this is where I post many of my links of things I have found interesting/compelling on the 'net.

Here's my Twitter RSS feed: http://twitter.com/statuses/user_timeline/15160781.rss

Cheers,

David

The future of UI will be boring

I'm not officially a UI guy, but I liked this blog post:

http://www.scottberkun.com/blog/2010/the-future-of-ui-will-be-boring/

The wiring that powers your home, the plumbing that brings you water, the roads you go to and from work on, work in mostly the same way they always have. This is ok. Lack of upgrade is not a sign of failure. Your genetic code is hundreds of thousands of years old, and seems to be working quite well if you’re reading this. It might just be more important to consider how the tech in question will make your dreams, or someone else’s, come true. If you worry more about the ends, rather than the means, revolution in UI is less important than you suspect it is. I mean, I’m a writer. If a quill pen was good enough for Shakespeare, what do I really have to complain about?

Continuous Deployment - releasing two-three times a day...

Very interesting case study on applying Lean principles to deployment. Serious elimination of waste.

http://www.startuplessonslearned.com/2010/01/case-study-continuous-deployment-makes.html

Continuous Deployment is Continuous Flow applied to software. The goal of both is to eliminate waste. The biggest waste in manufacturing is created from having to transport products from one place to another. The biggest waste in software is created from waiting for software as it moves from one state to another: Waiting to code, waiting to test, waiting to deploy. Reducing or eliminating these waits leads to faster iterations which is the key to success.

Neither Enterpriser or Hacker Be

I know of what Tim speaks around large, unwieldy Enterprise-level projects.  They are money pits and resource pits.  I suspect most of the engineers working on them aren't really having a good time.

But Tim seems to glom Web developers and agile methodologies into the same boat, and I think it's important to make a distinction.  There are strong agile-style developers and there are Hackers.  Hackers do follow iterative development and are focused on user experience, but often testing and reusable design go out the window, and monolithic spaghetti garbage heaps are often the result.

When I was interviewing during the Web 1.0 boom, a common job description was something like "Emergency Senior Engineer."  I would go into a shop that had hacked their way through the first few years and now they were seriously stuck, and were looking for someone with some grey hairs to help clean things up.

It's definitely very hard to get engineers with an Enterprise mentality to adopt an agile approach.  Enterprisers are very uncomfortable when they aren't writing out all the specs and plans up front - they are wanting to cover all the bases and get signoff before they start working - as Tim said "Enterprise IT has spent decades growing a defensive culture based on the premise that you only get noticed when you screw up, so that must be avoided at all costs."

But Hackers are very uncomfortable with having to write tests first, thinking about reusable design, spend time cleaning up code they touch, and so on.  Many times I've seen a roll of the eyes when I bring up these kinds of things.  I think this is because these guys love just whacking out code, are purely focused on technology, and don't have the patience or interest to think about maintainability.  Normally a few hellish releases where you can't get anything done because all you're doing is fixing bugs straightens out this attitude.  But by then it's often too late for the project at hand, without major rewrite.

That said, if I were to be forced to choose between an Enterprise culture or a Hacker culture, I'd probably choose the latter.  Enterprisers take years (and years) to get something to market, and it's usually not what anybody actually wants.  Hackers get hideously coded stuff out quickly, get feedback, and probably have money coming in when it comes time to rewrite.

Also, let's talk about culture.  Enterprise projects have this horrible stuffy dead feeling, like you're working on the 7 and 1/2th floor.  Everything's about specs and standards and ISO and most of the time you're in meetings and doing reviews.  Hackers are often a bit wacko and over-stimulated, but at least there's energy there.

Luckily, right now I'm on a team committed to agile practices, and it's definitely a lot of fun.  Our customers are pretty happy too.  And this, in an enterprise!  Maybe there's hope...

Steampunk LCD - classy and cool

At lunch yesterday a colleague from Sun introduced me to the term "steampunk."  Steam what?  Wikipedia showed me I was way behind the times.

I was thinking back to movies and books I've read, realizing they were "steampunk" and I didn't even know it, like The Golden Compass.

This same friend today pointed me to this awesome page where someone shows you step by step how they souped up their LCD monitor to be a steampunk monitor.  My goodness, look at this thing, that is just cool and classy.  That keyboard is pretty cool too, although I'm not sure it's practical (but probably good for preventing RSI).

What wakes men up, what wakes women up

From MindLab, by way of the New York Times. Note the complete lack of "baby crying" from the men's top ten list :)

Top 10 sounds most likely to wake women:

* Baby crying
* Dripping tap
* Rowdiness outside
* Snoring
* Buzzing fly
* Drilling/workmen
* Sirens
* Car alarm
* Howling wind
* Noise from drains

Top 10 sounds most likely to wake men:

* Car alarm
* Howling wind
* Buzzing fly
* Snoring
* Noise from drains
* Crickets chirping
* Sirens
* Clock ticking
* Drilling/workmen
* Dripping tap

Some things I would like to add to my list:

* The sound of a mouse scratching in the drywall right behind my bed
* My wife throwing a pillow on my head (long story)

* Too much heat and still air (sadly, what wakes my wife up is too much cold and a draft)

* My iPhone buzzing under my pillow (well, that's on purpose, that's my alarm)

Going Postel

In our code base here, we have a utility class called Observable that makes it easy to apply the observer pattern to a class.

I was using it, and in the process (having been bitten by this in NetBeans), I wrote a unit test to see what would happen if a user of this class added Observers and then dereferenced them without removing them. I used a form of memory leak testing described here.

Sure enough, there was a memory leak, because the underlying list of observers in the Observable class was still hanging on to the observers.

So I modified the class to use WeakReference and ReferenceQueue to detect when an observer was no longer strongly referenced, and then removed "stale" references from the observer list.

When I submitted the code for review, our technical director here, who had written the original Observable code, pushed back, saying it was adding unnecessary complexity to the class.

I argued that my experience with listeners/observers was that people often did not do the right thing and forgot to remove their observers, and Evil Things ensued.

He pointed me to this wonderful, hilarious, incredibly well-written article by Joel Spolsky about (among other things) Martian headsets, the problem of standards, and the disaster caused by Jon Postel's robustness principle ("“Be conservative in what you do, be liberal in what you accept from others.”) I highly recommend you read it, but in case you don't, the point was, this principle encouraged sloppy and incorrect web pages to proliferate, which has put us into a terrible compatibility mess today.

I argued that although this made a lot of sense, we weren't building a standard or pseudo-standard with lots of implementations, and look what Java did to ease programmer's lives by introducing garbage collection to the masses.

He made an interesting point that services should probably follow the robustness principle, but lower-level building blocks, where implementation is more transparent, should expect users to use them correctly and not adapt to incorrect use.

I finally relented because (a) he's the boss, and at some point just to make progress you let the boss make the call :) and (b) I had to admit that we had never actually encountered this problem after being in the field for many years, so perhaps I was inventing a dragon where none existed.

It still makes me nervous. I saw nasty problems with zombie observers in NetBeans. But I'm willing to give it a shot. In general I am happier with code that is simpler and cleaner and only complicate it if you have to. I think we'll just have to see how it goes.

I'm curious, what do you think?

Applying OO principles to interpersonal relationships

I formally admit my geekiness by seeing the interesting similarities between good OO design and healthy interpersonal relationships.

Healthy boundaries
Objects need to collaborate and interact and depend on each other, but it's important to maintain healthy boundaries.  Other objects are not allowed to touch your private parts, and you get to say what is visible and what is not.

Multitasking

Objects must know how to handle multiple requests at once while maintaining their state.  


Grace Under Pressure
Well-designed objects need to be able to handle stress and overload.

Honoring your commitments

Objects provide public interfaces, which provide an agreement as to what they are willing to do.  You can not change your agreements without talking about it first.

Interdependence

No object is an island unto itself; all objects have dependencies on other objects.  But an object should not be overly dependent on others, as this means it becomes fragile to changes made in the other object.

However, it's important to recognize there are some OO principles that should not be applied to relationships.

Single Responsibility Principle - each object has one and only one responsibility in a system.  Shyeah, right.

Pluggability - you should be able to replace one object with another that plays the same role.  No thanks.

TDD taken to logical conclusion - write acceptance tests first

I'm reading an interesting book called Growing Object-Oriented Software Systems by Steve Freeman and Nat Price (based on @mfeathers recommendation).

For the most part the first two chapters are nothing new for me (I'm expecting that will change in subsequent chapters), but there was one thing that made me stop and step back.  To me TDD has been about unit tests, but they strongly recommend something that I had never though of before but which made a lot of sense.  Not only do you write tests first at the unit level, you also write your end-to-end acceptance test first.

This gave me pause.  I am so used to QA reactively writing acceptance tests either after or in parallel with feature development.  But the more I thought of it, the more it made sense.  It defines a contract in black-and-white, and it also helps you focus on building just as much as you need and no more.

Of course the acceptance tests will break, and then you work towards making it pass by writing unit tests that break and making those unit tests work.  This really makes sense, and I'm going to try this with my next project.  I talked with our QA engineer, and he loves the idea and wants to give it a go.

If it works, I'll talk to the rest of the team about it.  This is the kind of thing that turns standard development process on its head and has to be introduced very gently... :)

LinkedIn looking for top-notch database engineers

I recently talked with Jean-Luc Valliant, the CTO at LinkedIn, about a new project they're starting. Of course I was interested, but I'm here at Symantec now and jumping ship after just six months is just not how I do things.

But I did say I'd put the word out. They're looking for top-level database theorists - people who have a solid understanding of and have contributed to database research. They're also looking for people with real-world experience building and deploying high performance, highly scalable database solutions, dealing with things like storage, caching, sharding/clustering, messaging and so on.

If you're interested, contact Jean-Luc Valliant at LinkedIn. If you have trouble reaching him, let me know at david at vancouvering dot com and I'll connect you up.

H1N1 Un-scare

I really appreciated this newsletter from our pediatrician, a different perspective on it all, so I thought I'd pass it on...

By the way, Michael and I already got it, and it looks like Linda is getting it too.  Not fun, it hangs on for a while, but no worse than any normal flu.  It was weird having cold-like symptoms and also feeling a bit flu-ish.

---

Dear Families,

Don't panic. Breathe. Wait, don't breathe, you might catch a germ!  Stay home! Wear masks!  Wash your hands constantly!  Remove your children from all public activities!  Stay home! Build a flu bunker!

Well, you get the idea.  There has been a lot of media coverage of the flu this week.  Unfortunately, the media's job is to obtain viewers, not to give you balanced, non-sensational medical information.  We have been inundated with H1N1 flu questions this past week. Everyone seems much more nervous, due in large part to the escalating news coverage.  Everyone remembers the one exception covered on 60 minutes, but let's set some basic facts straight about this flu:

1.  It is milder than typical winter influenza.

2.  This is a very contagious strain, so it is likely that many members of our community will get it.  The incubation period is 3 days.  If you have no symptoms 5 days after exposure, you likely will not get the virus.

3. If you get the virus, 30% of you won't even get a fever and will think you have just a cold.  If you do get fever, it will typically be for 2-3 days.  A few children have had fever for as long as 5-7 days, but this is very unusual in our patients who are treating their flu with natural means.

4.  Most of the people who have gotten very ill have significant pre-existing medical conditions like chronic lung disease.  In fact, proportionally, more "healthy" people get very ill with regular winter flu than have gotten very ill with this flu.

5.  So far, our families that have had the flu have recovered quickly and easily, with little need for bloodwork, antibiotics, chest x-rays, etc.  Even children with a history of asthma or wheezing seem to be less likely to wheeze with this illness.  We are very happy overall with how our patients are weathering this storm. 

6.  If you have had a flu like illness (fever, muscle aches, headache the first day, with runny nose and cough symptoms) then you may have already had H1N1!  The CDC says unless you know for sure it was H1N1 you should still get the vaccine, but this is likely overkill and you probably don't need it.

7.  Initially we were quite concerned about the H1N1 vaccine because it was going to contain a controversial new ingredient that had not been used in regular vaccines in the U.S.  Luckily, the CDC changed their mind and opted not to use this new adjuvant, squalene.  As far as we know, the vaccine that will be supplied to us from the government will be manufactured in a manner similar to our usual preservative-free flu vaccine.  So, we feel more comfortable with the idea of our patients receiving this vaccine.

We will have a limited supply of preservative-free H1N1 vaccine.  We will prioritize vaccine for high-risk families such as those with pregnant women, or children of any age with a history of pneumonia or asthma.   Once these families are covered, we can open up our supplies to any families that desire the vaccine.  Adults should get their vaccine from their primary care provider.  Pregnant women should be sure to ask for the preservative free version as it does not contain mercury.

8.  Let us be clear, again, that we feel most people don't need this vaccine.  This is a mild flu and it may be better to get it now and have immunity for the next time it goes around, or next year's strain.

Online equation solver

I'm working with Retouched Bloom Filters (PDF), and am trying to figure out how to calculate the size of a filter given a desired false positive and false negative rate. I tried to do some of the math, but I'm just too rusty.

Then I remembered my old friend The Web, and sure enough a search for an equation solver uncovered this site. Very useful - enter in a linear equation, and the variable you want to solve for, et voila!

For example, from the paper above, the false negative rate is given by

fn = 1 - (1 - s/(p1 * m))^k

where fn is the false negative rate, s = the number of bits reset, p1 is the probability a given bit is set in the filter after all n elements have been added to it (about 1/2 for an optimized filter), m is the size of the filter, and k is the number of hash functions.

I wanted to know what s is, the number of bits to reset, if I know the other values. So I entered in the equation above to the equation solver, and like magic, I have

s = (1-(1-fn)^(1/k))*m*p1

Nice!

BTW, once I have the numbers worked out to my satisfaction, I'll post them here.

More on blatant counterfeiting in Wall Street

A well written, if expletive, article by Matt Taibbi in the Rolling Stone.

This is again about naked short selling - shorting stock that you don't actually have, and someone else then selling this same stock, thus driving the stock down.  It turns out that buying and selling stuff you don't acdtually have is also done in commodities, mortgages, and bonds.

A paper presented at the American Bankruptcy Institute earlier this year reports that up to a third of all notes for mortgage-backed securities may have been "misplaced or lost" — meaning they're backed by IOUs instead of actual mortgages.

He can't get his hands on the smoking gun, but all arrows do seem to point towards Wall Street robbing us all blind, with wolves guarding the hen house in Washington.

This week I watched a homeless man standing with handcuffs while two policemen talked to him and searched his stuff.  Meanwhile there are these companies in the center of our financial world who are basically greedy schmucks laughing while they rob us for billions and billions of dollars.  Greed gone wild.

The nation's largest financial players are able to write the rules for own their businesses and brazenly steal billions under the noses of regulators, and nothing is done about it. A thing so fundamental to civilized society as the integrity of a stock, or a mortgage note, or even a U.S. Treasury bond, can no longer be protected, not even in a crisis, and a crime as vulgar and conspicuous as counterfeiting can take place on a systematic level for years without being stopped, even after it begins to affect the modern-day equivalents of the Rockefellers and the Carnegies. What 10 years ago was a cheap stock-fraud scheme for second-rate grifters in Brooklyn has become a major profit center for Wall Street. Our burglar class now rules the national economy. And no one is trying to stop them.

If history is any lesson, this will end, and end ugly.  It's like watching a tidal wave build up behind you, seeing the water all pull away and you wonder if anyone else is noticing.

Pitch for ReviewBoard for code reviews

Back in the nineties, code reviews were painful, horrible experiences. You would be handed a thick parchment of source output and line numbers, and you had to go into a meeting for two hours to review all the code. The waterfall method at work.

Things got a little better when people would quickly send out diffs or attach them to a bug, but not by much. In the open source world, the reviews would go on for days with twenty levels of indents on long long email threads. At the end I'd scratch my head and wonder where things stood.

In my new job here at Symantec, I was introduced to ReviewBoard, an open source tool for code reviews. Wow. What a difference a tool makes. You make a change, and then use a tool to submit it for review. It gets the diff and submits it to Review Board. You then go in and specify who the reviewers are, submit, and you're done.

As a reviewer, you can look at the diffs in context with the source, and then click on any line and you can add a comment. When you're done, submit, and an email is sent out. The change owner can then go to specific comments and reply - each comment can have its own dialog which is logged in the tool. All this is done with AJAX style interaction, which is sweet.

The change owner can submit a new patch for review, and if they use the same changeset id, the tool lets you as a reviewer see the diff between the original and the updated version, so you can focus on what's changed since the last review. When you're satisfied, you click on "ship it" and you're done.

Atlassian also has a code review tool but (a) it's not free and (b) it doesn't handle iterative reviews - where you address comments by a few small changes. Their tool doesn't let you see the differences between iterations in the review, and that's pretty crucial when you have a large patch and the new revision only has a small set of updates. Also, it's difficult to submit a change before it is committed. So I think overall ReviewBoard is the better tool.

Because of this tool, we can now have a policy that every change be submitted for review before it's committed. Of course we are flexible with this, but this is the overall policy. I don't think such a policy would be bearable or enforceable without a tool like ReviewBoard. Thanks, guys!

Calling out to hungry spirits

I was listening to Door of Faith by Krishna Das, and I just had to share this beautiful poem he sang in English. It really touched my heart. To me, this is the heart of God, which we all carry within us.

Calling out to hungry hearts
Everywhere through endless time
You who wander, you who thirst
I offer you this heart of mine

Calling out to hungry spirits
Everywhere through endless time
Calling out to hungry hearts
All lost and left behind

Gather round and share this meal
Your joy and your sorrow I make it mine.

Short a stock, then bring the company down: sounds like a plan

This blog tells quite an incredible story about the CEO of Overstock.com, Patrick Byrne, bringing to light the strategy of some hedge funds to short a stock and then use various mechanisms to bring the company down, and how the media appears to be involved in a coverup.  I don't know how well sourced this material is, but it's a good read anyway, and to be honest, it doesn't surprise me at all.

A small group of powerful hedge fund managers stop at nothing to annihilate the companies they sell short. Their tactics include: blackmail, smear campaigns, espionage, fraud, harassment, extortion, bribery, rumor-mongering, sabotage, off-shore money laundering, political cronyism, frivolous lawsuits, witness tampering, biased financial research, false identities, bogus credit ratings, bribery, libelous blogs, bad science, forgery, wiretapping, counterfeiting, collusion, lying, cheating, threats and theft.

Sounds like the Wall Street I know and ... well, definitely not love.

Greed gone wild.

"When I despair, I remember that all through history the way of truth and love has always won. There have been tyrants and murderers and for a time they seem invincible, but in the end, they always fall.. think of it, always." - Mahatma Gandhi

Answers to life's persistent questions

I was recently working on some algebraic problems when I was studying a Retouched Bloom Filter (PDF). I was trying to see if I could calculate how much smaller the Bloom Filter could be depending on how many bits I reset to 0.

I found myself stumped - it has just been too long since I did this stuff in college. My numbers came out all wrong.

So I found it quite appropriate when I found these real-life answers, copied from an email sent to me.

Concurrency.next and the suckiness of explicit concurrency management

Tim Bray has a nice blog (with lots of fun comments) about multicore chips, parallelization and concurrency, and how this may bring about a new set of popular languages. Nice analysis, most of it makes sense.

But what he said last struck me the most, because it's been bothering me for a while too and I thought I was a loner.

Assumption · I’m taking the following as an axiom: Exposing real pre-emptive threading with shared mutable data structures to application programmers is wrong. Once you get past Doug Lea, Brian Goetz, and a few people who write operating systems and database kernels for a living, it gets very hard to find humans who can actually reason about threads well enough to be usefully productive.

When I give talks about this stuff, I assert that threads are a recipe for deadlocks, race conditions, horrible non-reproducible bugs that take endless pain to find, and hard-to-diagnose performance problems. Nobody ever pushes back.

When I started working NetBeans, where you the coder are responsible for concurrency, rather than the app server container, I was stunned, literally stunned, at how ugly things could get so easily. I almost immediately introduced deadlocks, data corruptions, and other icky stuff, and found myself desperately reading the bibles on concurrency and thread safety.

But I also found myself shaking my head. Why are we all doing this as if it makes sense for programmers to have to worry about this. This level of complexity and shooting-yourself-in-the-footedness indicates to me something inherently wrong with the programming model. It's like a hotel asking arriving guests to coordinate with each other to assign themselves rooms.

I had heard about the asynchronous, immutable state, message passing model of Erlang and Scala, and these seemed to me what we were looking for. You write an actor, it does what it does, and doesn't share its state with anyone except through copies. Simple, nice, elegant, parallelizable.

It's very hard to move from one language to another, so I'm with Java for now, but I'm looking forward to an opportunity to try out Scala or Erlang for Real Work. I'm sure the opportunity will come soon enough.

It's kind of nice that the new chip architectures are forcing programmers to think about parallelism, and thus good parallel concurrency models, and thus looking at the asynchronous/message-passing architectures of Scala and Erlang. Maybe we'll get out of this synchronized-volatile-deadlock-semaphore-thread-pool-latch madness that exists today in Java.

Excellent article on doing UI right

This is a great article on Software CEO about a new McAfee product (yes, I know, Symantec's sworn enemy :)), how successful it was in terms of usability, and what went behind that.

Lots of great tips about how to build your engineering process around your users, from soup to nuts.  Hiring an outside design team to design your UI - what a concept! :)

Here is just one tip to whet your appetite:

UI tip #6: Let user demand defend against code creep.
" Our team strove to understand what is necessary versus what's desirable," Ries says. "Everything you add in to the product affects documentation, support, and code.

"With every feature, you have to make a design decision. We tried to make reasonable and good assumptions about setting limits; we actively eliminated options, and then validated those options with the users.

"We ran into lots of feature that are good ideas, but they were clipped until we can get enough evidence that there's enough demand. There are always next versions that we can use to include them."