Beware of Survivorship Bias in Cybersecurity

Survival Bias is a type of selection bias that occurs when a non-representative subset of data is used to draw conclusions. What does this mean?

Consider Figure 1 below, let’s say you were tasked to examine the damage of aircraft that had returned from missions and recommend the best place to add additional armor on them to raise the possibility of returning safe to base. Where would you recommend placing the extra armor? Around the wings and tail, right? It is an obvious answer because these are the spots where the planes mostly got hit.

Let us not rush to conclusions. We are trying to draw a conclusion based on aircraft that made it back safely while ignoring those that did not. During World War II, the statistician Abraham Wald pointed out that this was the damage on the planes that made it home, so he proposed that the Allies should add extra armor on areas where there are no dots at all because those are the places where the planes won’t survive when hit. By considering the aircraft that never returned, the armor was placed around the engines saving lives of countless pilots.

Wald was aware that accurate analysis and reliable conclusions require including all the data, not just the survivors.

One of the major flaws in cybersecurity is survivorship bias. As humans, we are all subject to decision-making biases but overcoming biases in cybersecurity is crucial. Survivorship bias has its influence in cybersecurity decision-making; technology investment, resource allocation, threat analysis, investigation conclusion, strategy development, etc.

A malware detected and deleted by your anti-malware solution on a workstation is an example of survivorship bias where security team might think that their anti-malware solution is effective and the targeted system is not compromised. In such case, it is important to ask ourselves how this malware got there in the first place. Which layer of defense has failed (people, technology, and process)? Was there any credential compromised? Is it part of a network-wide cyber-attack? Another example is when organizations blindly rely on off-the-shelf cybersecurity products to protect their environment while overlooking the importance of building a security aware culture and closing the skills gap of their IT and Security team.

Business leaders and security professionals should not let survivorship bias influence their decision-making. Business owners lack the education on the state of corporate cybersecurity risk while many security professionals don’t always look at cybersecurity issues from the 3 security pillars perspective (people, process and technology) and still fall victims to the marketing hype where cybersecurity vendors are hyping up their technology by impossible claims and overpromised features.

Business leaders and security professionals should put more effort to gain more insight about the entire situation and perform more analytical and logical reasoning. It is important that the IT and Security team don’t work in silos and look into problems from a different perspective not only from a technical perspective. Protecting the organization is not something the IT and security people can magically solve on their own. It requires commitments, contribution and support from across the organization.

As cybersecurity professionals, you can overcome survivorship bias by thinking out of the box, understanding the situation, performing more analytics, asking questions, looking into problems from different perspectives, and trying to avoid assumption-based decision-making.