Survivorship Bias: Why the way we think about team culture is wrong

Let’s play a game.

You, four friends, and I will sit around a table.

On that table lies a revolver, capable of carrying six bullets.

That’s right. We’re playing Russian Roulette.

Your friend draws the fortune of being the first one to go. Good for him. In a traditional game, he has the best odds of “winning” (surviving). He spins the cylinder, puts the gun to his forehead, and pulls the trigger. Click. Empty. We all breathe a sigh of relief for him.

But there’s a small wrinkle in our game. Instead of just moving to the next notch in the cylinder, we choose to spin the cylinder for each successive person in the game. In a traditional game of Russian Roulette, the odds of you being the unlucky one increases with each person that survives. If you’re the 4th one to go and the bullet is still in play, there is a 1-in-3 chance you’re going to meet your maker. But in this version, the odds remain 1-in-6 for each person until the bullet has been fired.

Because I prefer not to write about the gory details of someone’s life coming to an end, I won’t continue to detail this game of Russian Roulette.

Instead, let’s talk about coaching, team culture, and survivorship bias.

Nobody wins in Russian Roulette

Let’s pretend that you played the alternate version of Russian Roulette and — incredibly — everyone survives. As the gun clicks one final time, you take a collective sigh of relief.

You’ve all lived to see another day. But this forces us to reflect and ask a couple of critical questions.

1. Was the fact that you survived the result of luck or skill?
2. Was it a good decision to play?

The answer to our first question should be obvious. Unless you have figured out a way to hack the revolver, the game is entirely based on luck, perhaps better-called randomness.

The second question is more difficult to answer. In our hypothetical game where everyone survives, you could make an argument that the game was a good idea. You all survived. Luck was on your side.

But that would be wrong. As Nassim Nicholas Taleb points out in Fooled by Randomness, you can’t judge the quality of a decision by the results, but rather by the costs of things that could have happened, but didn’t. Just like successfully driving home drunk isn’t a good decision regardless of the outcome, playing Russian Roulette is never a good idea.

We intuitively grasp this, but we have a harder time grasping the underlying psychological biases that cause us to overlook the impact of randomness in our lives.

Alternative histories and survivorship bias

An alternative history is anything that could have happened but didn’t. 

If Nassim Taleb is right when he says that “one cannot judge a performance in any given field by the results, but by the costs of the alternative,” then we must consider the decision to play Russian Roulette not by the result of survival, but by the alternative history of one of your comrades (or you or me) sprawled out on the floor covered in blood.

Former World Series of Poker champion Annie Duke calls this “resulting”, which she defines as “assuming that our decision-making is good or bad based on a small set of outcomes”. If you go all-in on the deal with a 2-7 off-suit, it’s a poor decision–even if you win.

When you consider alternative histories, you quickly realize that any gamble must be evaluated based on the set of all other outcomes that COULD have happened but didn’t.

This idea of alternative histories brings us to survivorship bias.

Survivorship bias is what causes us to only concentrate on people, teams, and successes that survived past a particular point. This can lead to overly-optimistic beliefs because we only analyze successes.

Failed startup founders don’t write books. So, we only hear the stories of massive success–failures are ignored.

Nearly every self-help book that has ever been written tells you to set goals. Wrapped in the story of someone “successful” (who could have been a failure in an alternative history), our brains make the seemingly logical connection: if I want to be successful, I need to set goals. 

But for it to be true that goals always lead to success, we would need to find that nobody who ever failed set goals–which isn’t true. As Scott Adams pointed out years ago, winners and losers have the same goals.

Survivorship bias also causes us to create causation where it doesn’t exist. Just like the businessman telling you to set goals because it worked for them, we often look to the winners to figure out what works and what doesn’t without accounting for the fact that the losers were often doing the same thing. There are many examples of this, but I want to discuss something particularly relevant for coaches.

If your culture’s so good, why don’t you win more games?

Cody Royle recently put out this two-part thread on culture.

The popular narrative is winning and culture are perfectly correlated; we laud any team who wins as having great culture.

That line of thinking, when reversed, suggests teams who don’t win have rotten culture (which I don’t believe to be true).

Perhaps our thinking is faulty.

— Cody Royle (@codyroyle) August 16, 2020

As Cody points out, survivorship bias causes us to assume winning = good culture. Interview any coach after a championship run and they’re likely to point to the team’s culture as the driver of that success. This further ingrains that association in our heads. Winning = good culture.

Now, please don’t hear what I’m not saying. I’m not saying that having a strong culture (however you define that) is unimportant. Having a strong culture is extremely important. But I think we massively overstate the role team culture has in winning.

Winning and losing teams have many shared objectives. One of them is to build a strong, winning culture. Nobody wants to be a part of a weak culture.

But the simple fact that winners and losers have the same goal suggest that it’s likely there are losing teams that have a strong culture. We might just not know about it yet.

I’m proposing that we need to rethink the way we talk about culture. Instead of falling for the winning = good culture bias, we need to look deeper when evaluating teams, coaches, and players.

Is a regression in a team’s performance a sign of a deteriorating culture? Or (in the case of the San Antonio Spurs), did the team’s best players retire? Is it possible for a team to have a great culture while simultaneously missing the playoffs?

Conversely, should we believe that the Houston Astros won a ring in 2017 because of their great culture? Their culture certainly played a role in their championship, but I’ll let you decide whether theirs was a culture you want to be a part of.

---

Your turn!

I want to hear from you. Do you agree? Disagree? Do you have any ideas for how we can better evaluate teams and draw out the factors that actually led to their success?

The comments are open for this one.

Let me know what you think.