Survivorship Bias and Product Development
A couple of months ago I came across an interesting article by the author of 'You Are Not So Smart' about a fallacy known as 'Survivorship Bias' which Wikipedia defines as:
The logical error of concentrating on the people or things that "survived" some process and inadvertently overlooking those that didn’t because of their lack of visibility.
I particularly liked the story describing how Abraham Wald helped the US military overcome an instance of this error when trying to work out where to place armour on their bomber planes:
The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered. But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all. The mistake, which Wald saw instantly, was that the holes showed where the planes were strongest. The holes showed where a bomber could be shot and still survive the flight home, Wald explained. After all, here they were, holes and all. It was the planes that weren’t there that needed extra protection, and they had needed it in places that these planes had not. The holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit.
I find the concept interesting on its own but as I’m working at a product company I was trying to understand how applicable it is for my particular context.
With respect to a product I think you’d say that non survivors are people who started using your product and then gave at some stage for whatever reason. Those people may not reach out for help so you lose the opportunity to get their feedback about the problems with your product.
At the other end of the scale you have people who are very happy with your product (the survivors) and they will often give you feedback about things that they’d like to see in the product for which they may currently have found a work around for.
What interests me at the moment is understanding whether the survivors' opinions about how to improve the product are the same as that of the non survivors - survivorship bias suggests this probably isn’t the case so we need to find a way to learn what’s hurting the non survivors.
One way to do this is to conduct user testing where we get potential users of our product to use it and then observe the pain points that they encounter. Although we don’t get to see in which areas people would be prepared to persevere to achieve their goal, it is still useful for identifying weak points.
Another thing that my colleagues and I have been doing is discussing times at which we felt like giving up while using the product but persevered and ended up being survivors.
The reason looking at this is interesting is that we tend to be affected by the affect heuristic and although we might be prepared to overlook a weakness of the product others might go elsewhere.
I’m sure there is more to this than what I’ve thought about so far so if you have any cool insights please share them in the comments.
About the author
I'm currently working on short form content at ClickHouse. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.