You Can Measure Security – The Remix – Metrics Matter

Rothman posted some brief thoughts on Metricon and how we measure the wrong things (here) I have posted on the importance of measuring security and the impotence of the security industry to actually measure anything (here), which you should read because it is funny as hell if nothing else.

Usually I would defer to Mr. Security Metrics, Andrew Jaquith (here), to argue but since I have not read his book (send me a copy Andrew – I am just too damn lazy/busy to log into Amazon and order one) and because I’m sure he gave it his Yankee best in debating with the mouth from the South, I thought I would expand on my earlier thoughts.

I agree that we have been measuring the wrong things; it provides little value to measure if we were attacked yesterday or how many viruses or spam we stopped at the gateway. I do, however, believe there is tremendous value in measuring the effectiveness and efficiency of an IT security program. We all agree that information security is evolving beyond a siloed organization that drops a box in, stares at the shiny lights and waits around to get hacked (or more appropriately spends a ridiculous amount of time tuning controls and culling through log files). Many security controls are becoming or have become operationalized, additionally an increasing number of organizations of all sizes are turning to system integrators and MSSP’s to manage aspects of their day to day security operations, both situations require the ability to gain insight into the effectiveness and efficiencies of these programs, especially when accountability  against contractual agreements and SLA’s are demanded – which they should be. This evolution is happening against the backdrop of regulatory compliance pressures, hostile threat environment and more and more organizations looking to the IT department to add value to the business beyond simply keeping the lights on. If we as security professionals ever expect security to earn respect as a recognized IT discipline, viewed as important to the business as network infrastructure, data protection, and business applications than we need to start acting more like business professionals and less like firewall jockeys with Perl/Python-foo who can rattle off the latest attack techniques from Blackhat but become glossy-eyed in the face of questions around security program effectiveness. Game over, we are all going to die is the wrong answer.

So how does one measure security? What are the right metrics? How do we move beyond the silly % of spam/virus/DoS attempts per day type metrics.  The answers to these will take longer than a single posting, but I will focus in on 2 key areas: operational efficiency and security effectiveness. Let’s look at operational efficiency first.

The ability for an organization to respond quickly to the velocity of change in the threat environment requires an agility that can be measured by looking at the effectiveness of an organization to affect change within an acceptable amount of time, as may be defined in an SLA. Some metrics that might be measured, assuming a time interval of 24 hours and pervasive visibility into the assets under management (the latter is difficult for most btw – that is most organizations are blind to 15-30% of assets they have under management – but that is a story for another day):

How efficient are we in updating our environment?  That is within a 24 hour period, once a change has been authorized, how quickly can we update what percentage of our environment?

– How efficient are we in updating new dat files?

– How efficient are we in modifying configurations, such as disabling vulnerable services, changing registry settings, etc?

– How efficient are we in issuing new new personal firewall policies and verifying they have been applied? (i.e. deny inbound RPC, deny port 445, etc)

– How efficient are we in distributing critical patches?

– How efficient are we in re-configuring IPS signatures?

– How efficient are we at doing x?

This information can be expanded to include historical analysis and trending to gain visibility into the introduction of new technologies or processes, or the end of others. For example, over the last 6 months we maintain 60% update efficiency for new dat file distribution, however since we switched over to incorporate technology widget y we have experienced a 20% increase in efficiency and are now on average 80% update efficient. Why is this important to measure? Well simply put an IT organizations operational efficiency is one measure of how quickly they can adapt and respond to a changing external threat environment, additionally this is a measure that can be used to understand the effectiveness of new technologies or process.  BTW – If you are not 80-90% update efficient, you need to take some action today. There are many different aspects to operational efficiency but I wanted to touch on one that should be measured and understood by both security, IT ops, and those who manage these teams.

Let’s take a look at security effectiveness

You can argue, and effectively I might add, that the amount of threat variability is too broad to ever answer the question of how secure are we in a realistic way. Well, sort of, no amount of controls will protect me from a giant asteroid or other wrath of God type event, and if our governments go mad and nukes start flying then it is unlikely that a virtual UTM will do much of anything, unless it is made of bronze which might allow it to last for several thousand years as a remnant of our futile existence. But we cannot live our lives this way, nor can we run our security programs against that type of risk potential. There are many things we do understand and we should be able to answer the following questions that address our security effectiveness:

– How many assets do we have under management? How many should we have? What is the delta?

– Have we implemented a security configuration management program (here)? How many machines deviate from our corporate policies?

– How vulnerable are we against a database of known vulnerabilities? How do we know?

– When was the last time we performed a penetration test? What was the result? Did we improve since last time, have we attempted to address deficiencies or are we still as vulnerable as ever?

– Do we monitor user activity? Access to critical applications? Can someone generate a report to prove it?

I can go on here, but there are so many different metrics that can be used to measure a security programs effectiveness. That doesn’t actually mean you will never experience a significant attack, but at least you have some visibility into how effective the controls you implement are against the threats you are aware of. Remember there is no such thing as perfect security, there is no way to defend against all bad things. The goal of a security is to limit the probability of a successful incident and when one does occur, which it will, to limit it’s impact on the organization and to do that you need complete visibility and control with situational awareness coupled with metrics that can be used to tune controls to ensure you are not just repeating the same mistakes over and over again.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s