The Two Camps of Cybersecurity Metrics
Why risk quantification and program performance aren’t the same thing
I’ve bought every book on cybersecurity metrics (there aren’t that many), and I can sort them all into just two camps: Risk Quantification and Program Performance.
Both camps add value to an infosec program, but they are not the same, nor are they replacements for one another. They are different processes, focused on measuring different things for different reasons. And in practice, they need each other.
Camp 1: Risk Quantification
Cyber Risk Quantification (CRQ), Quantitative Risk Analysis, and Risk Modeling are different names for the same idea: using probability theory to model security risks and usually arriving at a monetary distribution for losses.
This approach has grown in popularity through methodologies like the Factor Analysis of Information Risk (FAIR), tools from vendors such as SAFE and Qualys, and the broader movement to move away from qualitative heat maps for risk assessment.
Ultimately, Risk Quantification is focused on modeling what could happen, in other words, risk.
Camp 2: Program Performance
Program performance metrics (KPIs, control monitoring, and operational measures) are about tracking what has already happened and how the program is functioning today.
They are often straightforward to calculate (and sometimes straightforward to collect). Their primary purpose is to measure and communicate the performance of controls, processes, and operations.
Examples include:
Percent of privileged accounts with MFA enabled
Mean time to resolve vulnerabilities
Number of security incidents contained within SLA
Percent of endpoints enrolled in asset tracking
These are implementation, effectiveness, and efficiency metrics. They don’t put a dollar value on potential losses, but they do tell you whether the program is working as intended and where attention or resources should go.
Where CRQ is about modeling potential futures, program performance metrics are about measuring current and past performance.
Both Camps Matter
Program performance metrics measure control coverage and operational execution: things like MFA adoption, patch times, or incident response SLAs. They show whether the program is being implemented and how well it is functioning day to day.
Risk quantification, on the other hand, models what could happen. It uses probability theory and loss modeling to forecast potential futures in monetary terms.
Both perspectives are necessary. Together, they give leadership both lenses: how the program is performing today, and what the financial downside could look like tomorrow.