Security metrics, which I have posted on in the past (here), and (here), are almost as elusive as security ROI. But unlike the mystical pink unicorn that is security ROI, security metrics are real, tangible and meaningful. Why is it then that we have so much difficulty in defining metrics that are both simple in their implementation and significant in their impact on the organization? I believe much of this stems from two flaws in how most organizations approach information security.
The first problem is that, for the most part, security is a reactive, ad-hoc discipline, primarily focused on responding to an incident. This drives post-incident metrics such as how many virus outbreaks did we experience, or how many attacks did our IDS detect, or how much SPAM did our anti-spam thingie block. These might be useful in determining, well, those things above, but they are hardly telling of the effectiveness or efficiency of one’s IT security program.
The second problem is how an organization communicates between groups. Operations, audit & compliance, and security are examples of domains within an organization that use a very different language to communicate problem/resolutions.
Vulnerability assessment is a great example of the problem of cross-organizational communication. Security will look at vulnerability assessment data from the perspective of unique, distinct conditions, operations will look at the data with an eye towards what remediation must be done, and audit & compliance might be concerned with how the data is relevant to regulatory initiatives. Operationally these are all very different ways of describing environmental variables, and it is very difficult to satisfy each of these groups with a simple metric of how vulnerable are we? – to what? or How many vulnerabilities exist in our environment? – Why does it matter? Operations doesn’t care how many, unique, distinct vulnerabilities some VA scanner found – their charter is availability.
A common language that is driven by policy and used in terms of the business is critical to ensuring cross-organizational communication. Ideally we would be able to draft metrics that address effectiveness and efficiency, how effective is our IT security and operations program and how efficient are we in detecting and remediating change. Most of this would require a move towards a policy driven approach and SLA’s to monitor adherence to plan, which we will look at in a future post. I did want to take a minute and list some metrics that every organization must be able to address today, because if you cannot answer these basic questions about your environment, with any degree of accuracy, then all the metrics we will come up will fall short.
1. How many computing devices are actively connected to my network right now and how many of these do we actually own?
2. Of these how many do we actively manage (have full visibility into and command and control of)?
3. What percentage of these are compliant with basic security policies, including…?
a. Endpoint security is up to date and configured in compliance with corporate policy (Anti Virus, Anti Spyware, Personal Firewall, HIPS, Encryption, et al)
b. Systems are configured against a security baseline as defined by NIST, NSA, DISA, CIS, etc…
c. Systems are patched to corporate standards
4. How effective is our change management process? And how quickly can we affect change in the environment. For example, once a decision has been made to change some environmental variable (modify PFW settings, configuration change to the device itself, update to dat files, reconfigure HIPS/PFW settings, etc) what percentage of the environment can we verify conforms these changes within a 24 hour period?
5. What audit mechanisms are in place to detect changes to a corporate COE (common operating environment), how often do we monitor for non-compliance, what is the process for remediating non-compliant devices, and how long does it take from detection to remediation?
If your organization can repeatably and verifiably answer these 5 questions, you are well on your way to metrics nirvana.