I recently talked with Jean-Luc Valliant, the CTO at LinkedIn, about a new project they're starting. Of course I was interested, but I'm here at Symantec now and jumping ship after just six months is just not how I do things.
But I did say I'd put the word out. They're looking for top-level database theorists - people who have a solid understanding of and have contributed to database research. They're also looking for people with real-world experience building and deploying high performance, highly scalable database solutions, dealing with things like storage, caching, sharding/clustering, messaging and so on.
If you're interested, contact Jean-Luc Valliant at LinkedIn. If you have trouble reaching him, let me know at david at vancouvering dot com and I'll connect you up.
Wednesday, October 28, 2009
Tuesday, October 27, 2009
H1N1 Un-scare
I really appreciated this newsletter from our pediatrician, a different perspective on it all, so I thought I'd pass it on...
By the way, Michael and I already got it, and it looks like Linda is getting it too. Not fun, it hangs on for a while, but no worse than any normal flu. It was weird having cold-like symptoms and also feeling a bit flu-ish.
Dear Families,
Don't panic. Breathe. Wait, don't breathe, you might catch a germ! Stay home! Wear masks! Wash your hands constantly! Remove your children from all public activities! Stay home! Build a flu bunker!
Well, you get the idea. There has been a lot of media coverage of the flu this week. Unfortunately, the media's job is to obtain viewers, not to give you balanced, non-sensational medical information. We have been inundated with H1N1 flu questions this past week. Everyone seems much more nervous, due in large part to the escalating news coverage. Everyone remembers the one exception covered on 60 minutes, but let's set some basic facts straight about this flu:
1. It is milder than typical winter influenza.
2. This is a very contagious strain, so it is likely that many members of our community will get it. The incubation period is 3 days. If you have no symptoms 5 days after exposure, you likely will not get the virus.
3. If you get the virus, 30% of you won't even get a fever and will think you have just a cold. If you do get fever, it will typically be for 2-3 days. A few children have had fever for as long as 5-7 days, but this is very unusual in our patients who are treating their flu with natural means.
4. Most of the people who have gotten very ill have significant pre-existing medical conditions like chronic lung disease. In fact, proportionally, more "healthy" people get very ill with regular winter flu than have gotten very ill with this flu.
5. So far, our families that have had the flu have recovered quickly and easily, with little need for bloodwork, antibiotics, chest x-rays, etc. Even children with a history of asthma or wheezing seem to be less likely to wheeze with this illness. We are very happy overall with how our patients are weathering this storm.
6. If you have had a flu like illness (fever, muscle aches, headache the first day, with runny nose and cough symptoms) then you may have already had H1N1! The CDC says unless you know for sure it was H1N1 you should still get the vaccine, but this is likely overkill and you probably don't need it.
7. Initially we were quite concerned about the H1N1 vaccine because it was going to contain a controversial new ingredient that had not been used in regular vaccines in the U.S. Luckily, the CDC changed their mind and opted not to use this new adjuvant, squalene. As far as we know, the vaccine that will be supplied to us from the government will be manufactured in a manner similar to our usual preservative-free flu vaccine. So, we feel more comfortable with the idea of our patients receiving this vaccine.
We will have a limited supply of preservative-free H1N1 vaccine. We will prioritize vaccine for high-risk families such as those with pregnant women, or children of any age with a history of pneumonia or asthma. Once these families are covered, we can open up our supplies to any families that desire the vaccine. Adults should get their vaccine from their primary care provider. Pregnant women should be sure to ask for the preservative free version as it does not contain mercury.
8. Let us be clear, again, that we feel most people don't need this vaccine. This is a mild flu and it may be better to get it now and have immunity for the next time it goes around, or next year's strain.
By the way, Michael and I already got it, and it looks like Linda is getting it too. Not fun, it hangs on for a while, but no worse than any normal flu. It was weird having cold-like symptoms and also feeling a bit flu-ish.
Dear Families,
Don't panic. Breathe. Wait, don't breathe, you might catch a germ! Stay home! Wear masks! Wash your hands constantly! Remove your children from all public activities! Stay home! Build a flu bunker!
Well, you get the idea. There has been a lot of media coverage of the flu this week. Unfortunately, the media's job is to obtain viewers, not to give you balanced, non-sensational medical information. We have been inundated with H1N1 flu questions this past week. Everyone seems much more nervous, due in large part to the escalating news coverage. Everyone remembers the one exception covered on 60 minutes, but let's set some basic facts straight about this flu:
1. It is milder than typical winter influenza.
2. This is a very contagious strain, so it is likely that many members of our community will get it. The incubation period is 3 days. If you have no symptoms 5 days after exposure, you likely will not get the virus.
3. If you get the virus, 30% of you won't even get a fever and will think you have just a cold. If you do get fever, it will typically be for 2-3 days. A few children have had fever for as long as 5-7 days, but this is very unusual in our patients who are treating their flu with natural means.
4. Most of the people who have gotten very ill have significant pre-existing medical conditions like chronic lung disease. In fact, proportionally, more "healthy" people get very ill with regular winter flu than have gotten very ill with this flu.
5. So far, our families that have had the flu have recovered quickly and easily, with little need for bloodwork, antibiotics, chest x-rays, etc. Even children with a history of asthma or wheezing seem to be less likely to wheeze with this illness. We are very happy overall with how our patients are weathering this storm.
6. If you have had a flu like illness (fever, muscle aches, headache the first day, with runny nose and cough symptoms) then you may have already had H1N1! The CDC says unless you know for sure it was H1N1 you should still get the vaccine, but this is likely overkill and you probably don't need it.
7. Initially we were quite concerned about the H1N1 vaccine because it was going to contain a controversial new ingredient that had not been used in regular vaccines in the U.S. Luckily, the CDC changed their mind and opted not to use this new adjuvant, squalene. As far as we know, the vaccine that will be supplied to us from the government will be manufactured in a manner similar to our usual preservative-free flu vaccine. So, we feel more comfortable with the idea of our patients receiving this vaccine.
We will have a limited supply of preservative-free H1N1 vaccine. We will prioritize vaccine for high-risk families such as those with pregnant women, or children of any age with a history of pneumonia or asthma. Once these families are covered, we can open up our supplies to any families that desire the vaccine. Adults should get their vaccine from their primary care provider. Pregnant women should be sure to ask for the preservative free version as it does not contain mercury.
8. Let us be clear, again, that we feel most people don't need this vaccine. This is a mild flu and it may be better to get it now and have immunity for the next time it goes around, or next year's strain.
Tuesday, October 20, 2009
Online equation solver
I'm working with Retouched Bloom Filters (PDF), and am trying to figure out how to calculate the size of a filter given a desired false positive and false negative rate. I tried to do some of the math, but I'm just too rusty.
Then I remembered my old friend The Web, and sure enough a search for an equation solver uncovered this site. Very useful - enter in a linear equation, and the variable you want to solve for, et voila!
For example, from the paper above, the false negative rate is given by
fn = 1 - (1 - s/(p1 * m))^k
where fn is the false negative rate, s = the number of bits reset, p1 is the probability a given bit is set in the filter after all n elements have been added to it (about 1/2 for an optimized filter), m is the size of the filter, and k is the number of hash functions.
Then I remembered my old friend The Web, and sure enough a search for an equation solver uncovered this site. Very useful - enter in a linear equation, and the variable you want to solve for, et voila!
For example, from the paper above, the false negative rate is given by
fn = 1 - (1 - s/(p1 * m))^k
where fn is the false negative rate, s = the number of bits reset, p1 is the probability a given bit is set in the filter after all n elements have been added to it (about 1/2 for an optimized filter), m is the size of the filter, and k is the number of hash functions.
I wanted to know what s is, the number of bits to reset, if I know the other values. So I entered in the equation above to the equation solver, and like magic, I have
s = (1-(1-fn)^(1/k))*m*p1
Nice!
s = (1-(1-fn)^(1/k))*m*p1
Nice!
BTW, once I have the numbers worked out to my satisfaction, I'll post them here.
Saturday, October 17, 2009
More on blatant counterfeiting in Wall Street
A well written, if expletive, article by Matt Taibbi in the Rolling Stone.
This is again about naked short selling - shorting stock that you don't actually have, and someone else then selling this same stock, thus driving the stock down. It turns out that buying and selling stuff you don't acdtually have is also done in commodities, mortgages, and bonds.
This week I watched a homeless man standing with handcuffs while two policemen talked to him and searched his stuff. Meanwhile there are these companies in the center of our financial world who are basically greedy schmucks laughing while they rob us for billions and billions of dollars. Greed gone wild.
This is again about naked short selling - shorting stock that you don't actually have, and someone else then selling this same stock, thus driving the stock down. It turns out that buying and selling stuff you don't acdtually have is also done in commodities, mortgages, and bonds.
A paper presented at the American Bankruptcy Institute earlier this year reports that up to a third of all notes for mortgage-backed securities may have been "misplaced or lost" — meaning they're backed by IOUs instead of actual mortgages.He can't get his hands on the smoking gun, but all arrows do seem to point towards Wall Street robbing us all blind, with wolves guarding the hen house in Washington.
This week I watched a homeless man standing with handcuffs while two policemen talked to him and searched his stuff. Meanwhile there are these companies in the center of our financial world who are basically greedy schmucks laughing while they rob us for billions and billions of dollars. Greed gone wild.
The nation's largest financial players are able to write the rules for own their businesses and brazenly steal billions under the noses of regulators, and nothing is done about it. A thing so fundamental to civilized society as the integrity of a stock, or a mortgage note, or even a U.S. Treasury bond, can no longer be protected, not even in a crisis, and a crime as vulgar and conspicuous as counterfeiting can take place on a systematic level for years without being stopped, even after it begins to affect the modern-day equivalents of the Rockefellers and the Carnegies. What 10 years ago was a cheap stock-fraud scheme for second-rate grifters in Brooklyn has become a major profit center for Wall Street. Our burglar class now rules the national economy. And no one is trying to stop them.If history is any lesson, this will end, and end ugly. It's like watching a tidal wave build up behind you, seeing the water all pull away and you wonder if anyone else is noticing.
Friday, October 16, 2009
Pitch for ReviewBoard for code reviews
Back in the nineties, code reviews were painful, horrible experiences. You would be handed a thick parchment of source output and line numbers, and you had to go into a meeting for two hours to review all the code. The waterfall method at work.
Things got a little better when people would quickly send out diffs or attach them to a bug, but not by much. In the open source world, the reviews would go on for days with twenty levels of indents on long long email threads. At the end I'd scratch my head and wonder where things stood.
In my new job here at Symantec, I was introduced to ReviewBoard, an open source tool for code reviews. Wow. What a difference a tool makes. You make a change, and then use a tool to submit it for review. It gets the diff and submits it to Review Board. You then go in and specify who the reviewers are, submit, and you're done.
As a reviewer, you can look at the diffs in context with the source, and then click on any line and you can add a comment. When you're done, submit, and an email is sent out. The change owner can then go to specific comments and reply - each comment can have its own dialog which is logged in the tool. All this is done with AJAX style interaction, which is sweet.
The change owner can submit a new patch for review, and if they use the same changeset id, the tool lets you as a reviewer see the diff between the original and the updated version, so you can focus on what's changed since the last review. When you're satisfied, you click on "ship it" and you're done.
Atlassian also has a code review tool but (a) it's not free and (b) it doesn't handle iterative reviews - where you address comments by a few small changes. Their tool doesn't let you see the differences between iterations in the review, and that's pretty crucial when you have a large patch and the new revision only has a small set of updates. Also, it's difficult to submit a change before it is committed. So I think overall ReviewBoard is the better tool.
Because of this tool, we can now have a policy that every change be submitted for review before it's committed. Of course we are flexible with this, but this is the overall policy. I don't think such a policy would be bearable or enforceable without a tool like ReviewBoard. Thanks, guys!
Things got a little better when people would quickly send out diffs or attach them to a bug, but not by much. In the open source world, the reviews would go on for days with twenty levels of indents on long long email threads. At the end I'd scratch my head and wonder where things stood.
In my new job here at Symantec, I was introduced to ReviewBoard, an open source tool for code reviews. Wow. What a difference a tool makes. You make a change, and then use a tool to submit it for review. It gets the diff and submits it to Review Board. You then go in and specify who the reviewers are, submit, and you're done.
As a reviewer, you can look at the diffs in context with the source, and then click on any line and you can add a comment. When you're done, submit, and an email is sent out. The change owner can then go to specific comments and reply - each comment can have its own dialog which is logged in the tool. All this is done with AJAX style interaction, which is sweet.
The change owner can submit a new patch for review, and if they use the same changeset id, the tool lets you as a reviewer see the diff between the original and the updated version, so you can focus on what's changed since the last review. When you're satisfied, you click on "ship it" and you're done.
Atlassian also has a code review tool but (a) it's not free and (b) it doesn't handle iterative reviews - where you address comments by a few small changes. Their tool doesn't let you see the differences between iterations in the review, and that's pretty crucial when you have a large patch and the new revision only has a small set of updates. Also, it's difficult to submit a change before it is committed. So I think overall ReviewBoard is the better tool.
Because of this tool, we can now have a policy that every change be submitted for review before it's committed. Of course we are flexible with this, but this is the overall policy. I don't think such a policy would be bearable or enforceable without a tool like ReviewBoard. Thanks, guys!
Calling out to hungry spirits
I was listening to Door of Faith by Krishna Das, and I just had to share this beautiful poem he sang in English. It really touched my heart. To me, this is the heart of God, which we all carry within us.
Calling out to hungry hearts
Everywhere through endless time
You who wander, you who thirst
I offer you this heart of mine
Calling out to hungry spirits
Everywhere through endless time
Calling out to hungry hearts
All lost and left behind
Gather round and share this meal
Your joy and your sorrow I make it mine.
Thursday, October 15, 2009
Short a stock, then bring the company down: sounds like a plan
This blog tells quite an incredible story about the CEO of Overstock.com, Patrick Byrne, bringing to light the strategy of some hedge funds to short a stock and then use various mechanisms to bring the company down, and how the media appears to be involved in a coverup. I don't know how well sourced this material is, but it's a good read anyway, and to be honest, it doesn't surprise me at all.
Greed gone wild.
"When I despair, I remember that all through history the way of truth and love has always won. There have been tyrants and murderers and for a time they seem invincible, but in the end, they always fall.. think of it, always." - Mahatma Gandhi
A small group of powerful hedge fund managers stop at nothing to annihilate the companies they sell short. Their tactics include: blackmail, smear campaigns, espionage, fraud, harassment, extortion, bribery, rumor-mongering, sabotage, off-shore money laundering, political cronyism, frivolous lawsuits, witness tampering, biased financial research, false identities, bogus credit ratings, bribery, libelous blogs, bad science, forgery, wiretapping, counterfeiting, collusion, lying, cheating, threats and theft.Sounds like the Wall Street I know and ... well, definitely not love.
Greed gone wild.
"When I despair, I remember that all through history the way of truth and love has always won. There have been tyrants and murderers and for a time they seem invincible, but in the end, they always fall.. think of it, always." - Mahatma Gandhi
Thursday, October 08, 2009
Answers to life's persistent questions
I was recently working on some algebraic problems when I was studying a Retouched Bloom Filter (PDF). I was trying to see if I could calculate how much smaller the Bloom Filter could be depending on how many bits I reset to 0.
I found myself stumped - it has just been too long since I did this stuff in college. My numbers came out all wrong.
So I found it quite appropriate when I found these real-life answers, copied from an email sent to me.
I found myself stumped - it has just been too long since I did this stuff in college. My numbers came out all wrong.
So I found it quite appropriate when I found these real-life answers, copied from an email sent to me.
Thursday, October 01, 2009
Concurrency.next and the suckiness of explicit concurrency management
Tim Bray has a nice blog (with lots of fun comments) about multicore chips, parallelization and concurrency, and how this may bring about a new set of popular languages. Nice analysis, most of it makes sense.
But what he said last struck me the most, because it's been bothering me for a while too and I thought I was a loner.
When I started working NetBeans, where you the coder are responsible for concurrency, rather than the app server container, I was stunned, literally stunned, at how ugly things could get so easily. I almost immediately introduced deadlocks, data corruptions, and other icky stuff, and found myself desperately reading the bibles on concurrency and thread safety.
But I also found myself shaking my head. Why are we all doing this as if it makes sense for programmers to have to worry about this. This level of complexity and shooting-yourself-in-the-footedness indicates to me something inherently wrong with the programming model. It's like a hotel asking arriving guests to coordinate with each other to assign themselves rooms.
I had heard about the asynchronous, immutable state, message passing model of Erlang and Scala, and these seemed to me what we were looking for. You write an actor, it does what it does, and doesn't share its state with anyone except through copies. Simple, nice, elegant, parallelizable.
It's very hard to move from one language to another, so I'm with Java for now, but I'm looking forward to an opportunity to try out Scala or Erlang for Real Work. I'm sure the opportunity will come soon enough.
It's kind of nice that the new chip architectures are forcing programmers to think about parallelism, and thus good parallel concurrency models, and thus looking at the asynchronous/message-passing architectures of Scala and Erlang. Maybe we'll get out of this synchronized-volatile-deadlock-semaphore-thread-pool-latch madness that exists today in Java.
But what he said last struck me the most, because it's been bothering me for a while too and I thought I was a loner.
Assumption · I’m taking the following as an axiom: Exposing real pre-emptive threading with shared mutable data structures to application programmers is wrong. Once you get past Doug Lea, Brian Goetz, and a few people who write operating systems and database kernels for a living, it gets very hard to find humans who can actually reason about threads well enough to be usefully productive.
When I give talks about this stuff, I assert that threads are a recipe for deadlocks, race conditions, horrible non-reproducible bugs that take endless pain to find, and hard-to-diagnose performance problems. Nobody ever pushes back.
When I started working NetBeans, where you the coder are responsible for concurrency, rather than the app server container, I was stunned, literally stunned, at how ugly things could get so easily. I almost immediately introduced deadlocks, data corruptions, and other icky stuff, and found myself desperately reading the bibles on concurrency and thread safety.
But I also found myself shaking my head. Why are we all doing this as if it makes sense for programmers to have to worry about this. This level of complexity and shooting-yourself-in-the-footedness indicates to me something inherently wrong with the programming model. It's like a hotel asking arriving guests to coordinate with each other to assign themselves rooms.
I had heard about the asynchronous, immutable state, message passing model of Erlang and Scala, and these seemed to me what we were looking for. You write an actor, it does what it does, and doesn't share its state with anyone except through copies. Simple, nice, elegant, parallelizable.
It's very hard to move from one language to another, so I'm with Java for now, but I'm looking forward to an opportunity to try out Scala or Erlang for Real Work. I'm sure the opportunity will come soon enough.
It's kind of nice that the new chip architectures are forcing programmers to think about parallelism, and thus good parallel concurrency models, and thus looking at the asynchronous/message-passing architectures of Scala and Erlang. Maybe we'll get out of this synchronized-volatile-deadlock-semaphore-thread-pool-latch madness that exists today in Java.
Excellent article on doing UI right
This is a great article on Software CEO about a new McAfee product (yes, I know, Symantec's sworn enemy :)), how successful it was in terms of usability, and what went behind that.
Lots of great tips about how to build your engineering process around your users, from soup to nuts. Hiring an outside design team to design your UI - what a concept! :)
Here is just one tip to whet your appetite:
Lots of great tips about how to build your engineering process around your users, from soup to nuts. Hiring an outside design team to design your UI - what a concept! :)
Here is just one tip to whet your appetite:
UI tip #6: Let user demand defend against code creep.
" Our team strove to understand what is necessary versus what's desirable," Ries says. "Everything you add in to the product affects documentation, support, and code.
"With every feature, you have to make a design decision. We tried to make reasonable and good assumptions about setting limits; we actively eliminated options, and then validated those options with the users.
"We ran into lots of feature that are good ideas, but they were clipped until we can get enough evidence that there's enough demand. There are always next versions that we can use to include them."
Subscribe to:
Posts (Atom)