Introduction

Time-to-event data is one of the most common scenarios I come across in security, yet the majority of our industry is analyzing this data all wrong.

Time-to-event analysis occurs when where we are measuring and analyzing the duration until a specified event occurs. This “specified event” can vary, such as vulnerability exploitation, an EDR alert firing, or even updating a security policy. Regardless, we are measuring durations from a starting point to an event occurring.

For example, imagine we are measuring participant times in a 100 meter dash race. We may ask, “How long does it take for participants to sprint from the starting line to the finish line?”

To answer that, let’s measure the time until each participant reaches the event we are measuring: crossing the finish line.

While we have a table with the time-to-finish for each participant, we might want to aggregate the data into a centrality measure in order to make a statement about the entire population, such as the mean time-to-finish for the race participants.

On average, it took the participants ~12.4 seconds to complete the 100 meter dash. However, there is a problem. In the race analogy we wait for all participants to cross the finish line before we calculate the mean. However, in many of the time-to-event scenarios we face in security, we analyze the data prior to all observations reaching the event.

To give you an example, it’s more like we measure the time until each participant crosses the finish line, but we stop measuring at the 13 second mark. In this scenario, our table will look a little different:

When we calculate the average, what do we do? We might decide the exclude those that did not finish the race while we were measuring. Let’s calculate the average, excluding participant 3:

We calculated a mean of 11.8 seconds, 0.6 seconds faster than the previous example. We could have any number of participants who were still sprinting at the 13 second mark and our mean would continue to be 11.8 seconds.

However, if we did not stop measuring at the 13 second mark and instead observed the time-to-finish for participants 3-6, the mean would likely be significantly higher. Saying it takes participants on average 11.8 seconds to complete the 100 meter dash in this scenario is misleading.

This is the core problem with time-to-event data, especially in security. We often don’t observe the start and event time for every observation. Whenever we decide to stop measuring, there will be some vulnerabilities that have yet to be patched, some alerts that have not been triaged, etc.

Censored Data

Censored data refers to an observation where the information about the time-to-event is incomplete, usually because the event did not occur within the study’s timeframe or the subject was lost to follow-up. Our 100 meter dash participants who had not yet completed the race when we stopped measuring are an example of censored data: we know that these participants took greater than 13 seconds to complete the race (because they were still sprinting when we stopped measuring), but we do not know how long they took. They could have finished in 13.1 seconds, or 130 seconds.

In security, we often encounter censored data when measuring the time to an event. In fact, the censoring problem is the primary reason we can’t use the mean.

For example, when measuring the efficiency of a Security Operations Center (SOC) in detecting and responding to security events, security teams may measure the following time-to-event data for an incident:

1. Time-to-Detect: The duration from the first activity of the incident to an alert/incident being raised to the SOC

2. Time-to-Resolve: The duration from the alert/incident being raised and the incident resolution

Note: Your SOC may define these differently. As long as you have a start and stop time defined, it doesn’t matter. I chose these definitions to keep it simple for this demonstration.

The security team pulls incident data from the prior week and starts the calculate the Mean Time-to-Detect (MTTD) and Mean Time-to-Resolve (MTTR).

Everything looks great when calculating the MTTD: The list of incidents all have a timestamp for when the incident was created in the SIEM, so they calculate the time to detect and average it. However, when they review the closed timestamps, there is an issue. Not all incidents were resolved last week, and as such, some of the incidents do not have a closed timestamp.

The security team could calculate the MTTR for only the incidents that were closed. However, as we saw in the race analogy, that will cause the metric to be misleading.

Despite the data being censored, the security team does know some information about the time-to-resolve for incidents that have not yet been resolved. They know that they took at least as long until the end of the measurement period. That knowledge should be included in the calculation, somehow…

Survival Analysis

The good news is that time-to-event data is a well researched domain of statistics, and one that has been solved for. We just don’t commonly use these tools in security measurement (yet).

Survival analysis is the statistical modeling of how much time elapses before an event occurs. Survival analysis can be used to model the time until almost any “event” which makes it an excellent application for a variety of time-to-event scenarios in security.

Subjects and Events

In survival analysis, you will see reference to subject(s). The subject is who/what we are measuring the duration of until it experiences the event. It’s important to note that the subject is not the same as the event being analyzed, but instead the subject experiences the event.

Let’s look at some examples inside and outside of the security domain:

1. We are analyzing how long a patient survives a critical medical condition after treatment is administered.

   1. Subject: Patients

   2. Event: Death

2. We are analyzing how long new tires last before being replaced.

   1. Subject: Tires

   2. Event: Replacement

3. We are analyzing how long software vulnerabilities remain in the environment until they are remediated.

   1. Subject: Software vulnerabilities

   2. Event: Remediation

4. We are analyzing how long it takes the organization to review a new vendor request for security requirements.

   1. Subject: New vendor requests

   2. Event: Vendor security determination

5. We are analyzing how quickly security incidents are detected.

   1. Subject: Security incident

   2. Event: Detection

Survival

Survival refers to the subject not experiencing the event. The term survival is used because survival analysis was originally developed for use in modeling patient survival from time of treatment until death (morbid, I know). In this scenario, the terms are pretty unambiguous: what is the probability a patient will survive until a specific time after treatment?

The techniques used in survival analysis, however, can be applied to any time to event data. We just have to reframe how we think about it. If survival means the subject does not experience the event:

• When analyzing time-to-resolve of incidents, survival means the incident has not been resolved

• When analyzing time-to-patch for software, survival means the software has not been patched

• When analyzing the time-to-exploitation for vulnerabilities, survival means the vulnerability has not been exploited.

Depending on the event being analyzed, survival may be a good thing, or may be a bad thing (and in security it is often the latter, which can be a bit unintuitive).

Survival Time

Survival time is the duration (time) until a subject experiences the event. For example:

• When analyzing time-to-resolve of incidents, an incident’s survival time is how long the incident took to be resolved.

• When analyzing time-to-patch for software, a software’s survival time is how long it took to patch.

• When analyzing the time-to-exploitation for vulnerabilities, a vulnerability’s survival time is how long it took to be exploited.

You can also think of this as the “time-to-x”. For example, calculating the Mean Time-to-Resolve (MTTR) is equivalent to calculating the mean survival time. But remember, we often do not know the survival time for all subjects, and as such the mean is misleading (we cannot use it).

The Survival Function: S(t)

The survival function is a mathematical function that expresses the probability that a subject’s survival time (the time elapsed before the event occurs) is greater than a specified time. In statistics, you will often see the survival function notated as S(t), where t = time.

For example, if we derive the survival function for MTTR, we could compute the probability of an incident to be resolved 10 minutes, 30 minutes, 2 hours, or even one week after the incident’s first activity (or another start time).

We might calculate these values as follows:

• The survival function at time 10 minutes equals 0.7

• The survival function at time 30 minutes equals 0.5

• The survival function at time 2 hours equals 0.15

• The survival function at time one week equals 0.02

Or said another way:

• There is a 70% chance an incident is not resolved within 10 minutes.

• There is a 50% chance an incident is not resolved within 30 minutes.

• There is a 15% chance an incident is not resolved within 2 hours.

• There is a 2% chance an incident is not resolved within one week.

The Cumulative Distribution Function: F(t)

The cumulative distribution function is a mathematical function that expresses the probability that a subject’s survival time (the time elapsed before the event occurs) is less than a specified time. You will often see the cumulative distribution function notated as F(t), where t = time.

This function is rather straightforward once you see the relationship. If we can interpret the survival function as “There is a 70% chance an incident is not resolved within 10 minutes”, we can also say “There is a 30% chance an incident is resolved within 10 minutes.”.

How do we get that? We subtract 70% from 100% to get 30%. In other words, it’s the inverse. That’s exactly how we can compute the cumulative distribution function given we know the survival function.

The Kaplan-Meier Estimator

We won’t know the true survival function, meaning we don’t know the exact probability of a subject to experience the event or not. Instead, we use statistical techniques to take the data we do have and estimate the survival function.

There are multiple techniques to do this, but a popular one is the Kaplan-Meier estimator (named after Edward Kaplan and Paul Meier who originally published the method in 1958).

Note: Using a Kaplan-Meier Estimator is as simple as running KaplanMeierFitter() in a statistical library such as Python’s Lifelines. However, I will briefly explain the math so you have a basic understanding of what’s happening under the hood.

The concept Kaplan and Meier introduced is relatively simple: break up the duration into intervals, calculate the probability of surviving during each interval, and then multiply them together. We multiply them together because a subject must survive each preceding interval. For example, in order for a subject to have a chance of surviving until day 20, they must first survive from day 0-1, then 1-2, then 2-3, and so on.

Let’s build a simple table to demonstrate this:

The table has the following columns:

1. Interval → The time intervals where events occurred in the dataset

2. Number at risk → The number of subjects at risk of experiencing the event (survivors) during the time interval

3. Number of events → The number of subjects who experienced the event during the time interval

4. Number censored → The number of subjects who were censored

5. Survival rate → The calculated survival rate

6. Hazard rate → The calculated hazard rate (we will cover this later)

Looking at interval 0, we can see that all 20 subjects are at risk and no events were observed (this is usually assumed in survival analysis as time 0 is the exact time we started measuring). As such, all participants survived until interval 0 and the hazard rate is 0.

When we move down to time interval 1, we see that we have 20 subjects at risk. Of those 20 at risk, 2 of them experienced the event during this interval, while 18 of them survived the interval. Looking at time interval 2, we can see how the 18 survivors from interval 1 are at risk to experience the event during interval 2.

However, when looking at time interval 3, we get 16, not the listed 15:

In order to handle our censored data, we need to remove any censored subjects from the at risk pool:

To calculate the survival time, we take the number of at-risk subjects, subtract the number of events, and then divide it by the number of at-risk subjects. We then multiply that value by the value of the preceding time interval.

You don’t need to worry much about calculating the survival function by hand. What’s important to know is that we are calculating the probability a subject survives past the interval, accounting for censored subjects.

The Survival Curve

Looking at our previous table, you’ll notice we have a column for time and a column for the survival rate. Let’s keep just these two columns:

This table shows the survival function S(t), where t=time. For example:

• S(0) = 1

• S(1) = 0.9

• S(2) = 0.8

• S(3) = 0.64

• etc.

We can represent this graphically by plotting the interval column on the x-axis and survival rate on the y-axis:

Notice that the above survival curve is a step-function (it looks like a set of stairs). Each vertical drop in the curve is where one or more events were observed. For example, events were observed at durations 1, 2, 3, 4, etc.

Reading the Survival Curve

We can read this curve in one of two ways:

1. Starting from the x-axis: What is the survival probability at time 2?

2. Starting from the y-axis: At what time is the survival probability 0.4?

Confidence Intervals

Remember that the Kaplan-Meier Estimator is estimating the survival function. Because we are estimating and don’t know the true curve, you’ll often see the curve plotted with confidence intervals (usually as a shaded region).

Confidence intervals essentially plot the upper and lower bounds of where we think the survival curve is at a specific time. Said another way, we are 95% confident the survival time at time t is within the shaded region.

Comparing Survival Functions

When analyzing time to event we data, you’ll likely want to compare times between time periods or groups. For example, we might want to compare the functions for incident resolution today from last quarter to understand how performance has changed over time. Or, we may want to compare the functions between groups, such as incident type or severity. There are multiple ways to do this with survival analysis.

Median Survival Time

Note: To calculate the median survival time we must have had at least 50% of subjects experience the event within the measurement period.

The median survival time is a good alternative to using the mean (such as Mean-Time-to-Resolve) as it provides a centrality measure for the dataset that accounts for censored subjects.

We can calculate the median survival time by looking at the time where the survival function equals 0.5 (50%). We can interpret this as, “50% of subjects will experience the event by the median survival time”. Or, given our example below, “50% of subjects will experience the event by time interval 5.”

We can use the median survival time to compare against other groups (such as the median survival time for command and control incidents compared to malware) or compare the median survival time over time (such as the median survival time for Q1 vs Q2).

Restricted Mean Survival Time (RMST)

While we can’t use the mean due to censored data, we can use the Restricted Mean Survival Time (RMST). The RMST calculates the average survival time within a specified time period. For example, you could calculate the RMST from the baseline (time=0) up until time 10.

If we have the RMST for two curves using the same period, we can subtract them to find the area between the two curves:

The most difficult part in using the RMST is selecting an appropriate time period to analyze. While I won’t go to in-depth on choosing a time period for the RMST, one method is setting the time period to whichever curve has the smallest maximum time period (which was used in the figures above). In some scenarios, there will be a time period to analyze that is significant (related to the problem being modeled). Regardless, when comparing values between two curves, ensure they are both using the same time period.

LogRank Test

We may want to test if two survival functions are statistically different. For example, we may take a dataset and create two survival functions, one for the time-to-resolve for phishing incidents, and one for non-phishing incidents. When we plot the two curves they look like this:

At first glance it appears that phishing incidents are in fact resolved faster than non-phishing incidents. Let’s quickly compare the median survival times:

While it seems pretty clear that phishing incidents are resolved faster than non-phishing incidents, that won’t always be the case. To test if the curves are truly different (and not just caused by random chance) we use the LogRank Test.

The LogRank Test will return something called a p-value. The p-value is our indicator of statistical significance. If the p-value is less than 0.05, we can conclude that the two survival curves are in fact statistically different. However, if the p-value is equal to or greater than 0.05, we cannot conclude that the two curves are different.

The Hazard Function

The hazard function is a mathematical function that returns the instantaneous rate at which the event occurs at a specific time, given that it hasn’t happened before that time. In other words, the hazard function computes the risk of an event occurring at a specific time. You will often see the hazard function notated as h(t), where t = time.

The hazard rate is less intuitive to interpret then the survival probability. The biggest difference is that the hazard is not a probability, but a rate. For example, the speed your vehicle is traveling in miles per hour (mph) is a rate. If you are traveling at 60 mph, you would travel 60 miles within one hour assuming the rate did not change. In reality, however, your speed will change over time. You may be traveling at 40 mph in the city, then accelerate to 75 mph on the highway. Even during your travels your speed will fluctuate.

Just like mph, our hazard rate is a rate of units/time:

• Miles per hour

  • distance/time

• Hazard rate

  • events/time

We can visualize the hazard function by plotting the hazard rate on the y axis and time on the axis:

Hazard Function Shapes

How the hazard function changes over time is particularly of interest. For example, if the hazard function increases over time, that means the likelihood of the subject experiencing the event increases faster as it ages. On the other hand, if the hazard function decreases over time, that means the likelihood of the subject experiencing the event decreases as it ages.

1. Increasing Hazard Rate

An increasing hazard function means the conditional probability of the event occurring increases as time passes.

• Example: As a machine gets older, the chance of it failing in the next hour is higher than it was when the machine was new.

2. Decreasing Hazard Rate

A decreasing hazard function means the conditional probability of the event occurring decreases as time passes.

• Example: If a transplant patient survives the first 48 hours, their risk of immediate complication often drops significantly.

3. Constant Hazard Rate

A constant hazard function means the conditional probability of the event is not changed over time.

• Example: The object doesn’t “age” in terms of its risk; it is just as likely to fail today as it is a year from now.

Let’s think through the expected hazard function for different scenarios:

Scenario 1

We are analyzing the duration from when a software patch is available until the affected software is patched by the organization.

1. We may expect that the hazard increases over time in the beginning as the organization prioritizes, schedules, tests, and eventually rolls out software patches.

2. However, once the software has survived long enough without being patched, the hazard may fall significantly, as the organization has either accepted the risk and will not patch or is not aware/managing patching of the software entirely.

Scenario 2

We are analyzing the duration from employee leave until assets (such as a laptop) have been retrieved.

1. We may expect that the hazard is relatively high in the beginning as many employees are in the office on their last day to return assets.

2. Some employees are remote, and as such the hazard will continue to climb over the short term as assets are retrieved.

3. Eventually, the hazard will fall when enough time has passed without successful retrieval, causing the organization to instead implement compensating controls such as remotely wiping the devices or pursuing legal action.

Scenario 3

We are analyzing the duration from an alert being raised in the SOC until it is resolved.

1. We may expect that the hazard starts low when the alert is first triggered, and the likelihood of the alert being resolved continues to increases over time.

Cumulative Hazard Function

The cumulative hazard function calculates the total accumulated risk of experiencing the event over time. While the hazard function calculates the risk at a specific time, the cumulative hazard function calculates the total risk of the current time and the time that has already passed.

Think of it this way: The hazard function assumes that the subject has already survived until a specific point in time, so, given that fact, what does that tell us about their risk of experiencing the event now? In our example of retrieving assets after employee leave, the fact that the assets haven’t been retrieved in 90 days means they less likely to be retrieved right now than they were in the first 7 days.

The cumulative hazard function, on the other hand, takes into account that for a subject to survive until a specific time, they must also face the risk of every time point before that. In our example of retrieving assets after employee leave, while the rate of retrieval may be low at 90 days, most assets are retrieved prior to 90 days as the likelihood an asset is retrieved within 90 days is very high.

We can visualize the cumulative hazard function by plotting the cumulative hazard on the y-axis and time on the x-axis:

Because the cumulative hazard function is cumulative (integrates the hazard function), it is always positive and always increasing. The hazard function tells us the rate at which the cumulative hazard function changes (the derivative). Let’s look at the hazard function to see the relationship:

We can see that the hazard starts high and then decreases over time. We can see the same behavior in the cumulative hazard function, where the rate of change is faster in the beginning and then slows down.

Partitioned Survival Analysis

So far, we have been analyzing a single duration. However, in many security applications of survival analysis there are multiple steps or phases we are analyzing. One example of this would be incident lifecycles in a SOC. In a SOC, there are multiple discrete phases. One way to organize these phases is detection, triage, investigation, and response:

We could perform survival analysis on each stage independently by shifting the start time for each phase as the event time for the previous:

1. Detection

   1. Start = First Activity

   2. Event = Detection

2. Triage

   1. Start = Detection

   2. Event = Triaged

3. Investigation

   1. Start = Triaged

   2. Event = Determination

4. Response

   1. Start = Determination

   2. Event = Closure

In this case, we would get four distinct survival curves that models that phase’s survival time. However, we can also model the entire process by setting the start time for each phase as the time the subject becomes at risk for the first phase. This is called partitioned survival analysis. For example:

1. Detection

   1. Start = First Activity

   2. Event = Detection

2. Triage

   1. Start = First Activity

   2. Event = Triaged

3. Investigation

   1. Start = First Activity

   2. Event = Determination

4. Response

   1. Start = First Activity

   2. Event = Closure

We can visualize this partitioned model on a chart:

When analyzing portioned models, it can be useful to plot it as a stacked area chart, as the area between the curves can be thought of the subjects within those states:

When looking at a specific duration, we can see the proportion of incidents in each state:

50 minutes after the incidents’ first activity:

• 100% are detected

• 97.2% are triaged

• 70% are investigated

• 24.2% are resolved

Note that in the partitioned model we are looking at the lifecycle as a whole. Each subsequent step’s duration is dependent on the duration of the previous steps since each steps’ duration is calculated using the incident’s first activity as the start time.

This is important when comparing survival curves in a partitioned model. We might see an improvement in incidents being resolved faster from Q1 to Q2, but this improvement could be due to faster detection, triage, investigation, and/or investigation. In order to identify improvement in a specific step, we need to measure the area between the curves (the shaded areas on our stacked area chart). The area between the curve is equivalent to the individual survival curve if analyzed independently.

Conclusion

Survival analysis is one of the most transformative statistical tools the security professional can learn. Throughout security you will encounter time-to-event problems, and all to often we use the wrong statistical techniques (such as the mean) to measure them.

While survival analysis can seem complicated at first, once you have the basics down you will be much better equipped to answer key time-to-event questions, improve decision making, and translate findings into easy to understand statements.

But this is just the beginning of survival analysis. While this guide will arm you with the tools needed to answer most questions, there is a big world of advanced techniques that build off this foundation.

Earlier this week I finally released The Security Metrics Workbook into the world. Its a short workbook focused on defining security metrics (what are we measuring and why) that I had been working on for months prior. At the time of this writing I have given out 1,000+ copies for free (both physical and digital).

Truth be told, when I hit publish on the launch post on LinkedIn, I was quite nervous what the reception from the wider community would be. For a book titled “The Security Metrics Workbook” the scope of content was actually rather narrow: defining security metrics. This is but a tiny slice of the pie of the entire process:

The reason I wrote this in the first place, however, is quite ironic. From my perspective there were three general areas of security metrics: Performance Indicators, Risk Indicators, and Control Indicators.

Most of the guidance I was reading, however, took a perspective in one (or maybe two) of these areas. The reality is that a metrics program encompasses all three, and they are used for entirely different purposes and outcomes. I believe that most of the disagreement on security metrics and how they are used (including the quantitative risk management movement) are rooted in this disconnect.

So, when I finally decided the best way to distribute my thoughts on the subject was longform writing, I sat down to create this thing with a couple goals in mind:

1. It cannot be long. I want almost everyone who starts reading it to finish reading it.

2. It needs to be pragmatic and useful.

3. Readers should DO something with it, not just consume it (hence the workbook form).

I sure hope I met those three goals. You’ll have to let me know.

I’ve bought every book on cybersecurity metrics (there aren’t that many), and I can sort them all into just two camps: Risk Quantification and Program Performance.

Both camps add value to an infosec program, but they are not the same, nor are they replacements for one another. They are different processes, focused on measuring different things for different reasons. And in practice, they need each other.

Camp 1: Risk Quantification

Cyber Risk Quantification (CRQ), Quantitative Risk Analysis, and Risk Modeling are different names for the same idea: using probability theory to model security risks and usually arriving at a monetary distribution for losses.

This approach has grown in popularity through methodologies like the Factor Analysis of Information Risk (FAIR), tools from vendors such as SAFE and Qualys, and the broader movement to move away from qualitative heat maps for risk assessment.

Ultimately, Risk Quantification is focused on modeling what could happen, in other words, risk.

Camp 2: Program Performance

Program performance metrics (KPIs, control monitoring, and operational measures) are about tracking what has already happened and how the program is functioning today.

They are often straightforward to calculate (and sometimes straightforward to collect). Their primary purpose is to measure and communicate the performance of controls, processes, and operations.

Examples include:

• Percent of privileged accounts with MFA enabled

• Mean time to resolve vulnerabilities

• Number of security incidents contained within SLA

• Percent of endpoints enrolled in asset tracking

These are implementation, effectiveness, and efficiency metrics. They don’t put a dollar value on potential losses, but they do tell you whether the program is working as intended and where attention or resources should go.

Where CRQ is about modeling potential futures, program performance metrics are about measuring current and past performance.

Both Camps Matter

Program performance metrics measure control coverage and operational execution: things like MFA adoption, patch times, or incident response SLAs. They show whether the program is being implemented and how well it is functioning day to day.

Risk quantification, on the other hand, models what could happen. It uses probability theory and loss modeling to forecast potential futures in monetary terms.

Both perspectives are necessary. Together, they give leadership both lenses: how the program is performing today, and what the financial downside could look like tomorrow.

Information security teams track all sorts of metrics. The problem? No two teams track the same set. Different controls, goals, requirements, and tools means every program looks different.

Yet, across SMBs and Fortune 500s alike, one thing is consistent: everyone uses Excel. And honestly, it makes sense.

The Security Metrics Toolkit builds on that reality. It gives you a simple way to run a complete security metrics program while staying grounded in Excel. And when you’re ready to automate, you can migrate certain metrics into data pipelines and APIs without breaking the rest of your program.

In short: the toolkit handles the reporting and dashboards, so you can focus on defining what matters, how to measure it, and why it matters.

Providing Structure & Consistency

Most metrics programs are ad hoc. Lots of good intentions, but they don’t have a common structure. The Toolkit solves this.

It’s built on the methodology from NIST SP 800-55, “Measurement Guide for Information Security”, which organizes metrics into four categories:

• Implementation

• Effectiveness

• Efficiency

• Impact

It also gives you a framework for maintaining a formal metrics inventory. Each metric can be documented with:

• Unique ID

• Status (Active/Inactive)

• Dates added/updated

• Type and category

• Title and purpose

• Calculation and target

• Measurement frequency

• Owner and data sources

• Retirement dates and reasons

That structure is what turns scattered measurements into a sustainable program.

Simple Measurement Log

At the heart of the Toolkit is a unified measurement log. The schema is deliberately simple:

• Metric ID

• Measurement value

• Measurement date

• Optional notes

Manual logging may feel old-school, but it comes with a hidden benefit: accountability. The person responsible for recording the metric isn’t always the same as the person accountable for the outcome. By keeping that distinction clear, you avoid the common pitfall of owners pushing back or disputing measurements.

Of course, not everything needs to be manual. When you’re ready, the log schema supports automated data pulls from APIs, databases, or data lakes, seamlessly feeding into the reporting.

Intuitive Reporting That “Just Works”

The Toolkit’s biggest advantage is time saved on reporting. No more ad hoc Excel charts or endless PowerPoint slides. Instead, you get three ready-to-use views:

1. Metrics Matrix™

A compact, at-a-glance view of your entire program.

• Red action dots highlight metrics falling short of targets.

• Group and filter by type, category, or owner.

• Quickly spot where attention is needed.

2. Metric Cards

Individual metric dashboards with:

• IDs, names, current values, and targets

• Change since last measurement

• Clear “on/off target” indicators

• Mini trend charts showing % of target achieved over time

3. Target Deviation Cards

A sharper look at gaps.

• Show how far a metric is above or below target

• Visualize deviations without being cluttered by raw values

• Automatically color-coded by desired direction

These layouts let stakeholders get answers quickly, whether they want the big picture or a deeper dive.

Making Metrics Easy

The purpose of the Security Metrics Toolkit is simple: make running a metrics program easier.

• It doesn’t lock you into specific integrations.

• It doesn’t prescribe what to measure.

• It works with any metric, from any source.

Your job is to pick the right measures for the right reasons. The Toolkit takes care of the rest: structure, consistency, and reporting.

You can grab the Security Metrics Toolkit here, or try it for yourself here.

Model your first FAIR scenario — in Power BI, for free.

Most cybersecurity professionals know that qualitative risk assessments don’t cut it anymore.

Executives are asking for real numbers. Audit committees want to know how likely a scenario is — not whether it’s red, yellow, or green. Security teams are trying to prioritize work based on data, not gut feel.

But jumping into quantitative risk modeling feels like a heavy lift.

• FAIR sounds promising, but complex.

• Most tooling is built for big orgs or big budgets.

• Spreadsheets get messy fast.

• And even when you do model something, explaining it is half the battle.

We wanted to change that.

Meet the Securemetrics CRQ Community Edition

This free Power BI template lets you model a single FAIR scenario with clarity and structure — no code, no spreadsheets, and no giant SaaS platform required.

It’s built for:

• Individuals exploring cyber risk quantification

• Teams piloting FAIR

• Consultants who want a clean demo or teaching tool

• Anyone curious about modeling risk with numbers, not colors

What’s Inside

The Community Edition includes:

• Structured Inputs:

  • Threat Event Frequency (TEF)

  • Vulnerability

  • Primary Loss Magnitude

  • Secondary Event Frequency & Loss
    All modeled using BetaPERT distributions or constants.

• Built-in Monte Carlo simulation:
  10,000 iterations using the open-source pyfair library

• Power BI visuals for:

  • Loss distributions

  • Expected loss & percentiles

  • LEF/LM decomposition

• Clean, editable layout
  Designed to help you present the results — not just analyze them.

Setup is Simple

All you need is:

1. Power BI Desktop (free)

2. Python (3.9+)

3. The pyfair library (pip install pyfair)

4. Your scenario inputs

That’s it. Open the template, enter your parameters, and simulate.

Licensing

This version is free for personal, internal, and educational use.

If you’re doing client work or internal reporting at scale, we’ve got something better coming soon.

What’s Next: CRQ Pro

The Community Edition is just the beginning. The Pro Edition will support:

• Multi-scenario modeling

• Comparisons and trend reporting

• Export-ready visuals

• Commercial use licensing

Why We Built This

Quantitative risk modeling shouldn’t be locked behind paywalls or certification programs. It should be a skill you can learn hands-on — by running real scenarios, seeing real numbers, and experimenting in a real environment.

That’s why we made this free.

If you’ve been curious about FAIR, cyber risk quantification, or what a better risk conversation could look like — this is your starting point.

Download the Community Edition now

GRC + engineering?

Governance, risk & compliance (GRC) + engineering can seem like an oxymoron, but I believe in 5-10 years it is how many organizations will approach implementing, measuring, and managing information security programs. In fact, 8 GRC practitioners from companies such as Apple, Zoom, & Netflix have come together to create the GRC Engineering Manifesto, reminiscent of the Manifesto for Agile Software Development that changed the way we develop and deliver software.

GRC Engineering is a fundamental shift in the way GRC is done, with a focus on building GRC as a product for stakeholders leveraging open data models and automation.

What does this look like in practice?

Leveraging data for continuous and automated monitoring instead of manual, disconnected efforts.

But where does all this data come from? How is it organized? And how is it harmonized from disparate sources and workstreams into a delivery mechanism that supports stakeholder's needs?

Introducing Emergence, a security graph for GRC engineering

Harmonizing the disparate data sources and delivering it to stakeholders is a major roadblock in the adoption of GRC engineering, especially outside of technology companies.

Emergence, a security graph for GRC, aims to solve this.

Emergence is comprised of 3 components:

1. An open-source graph model built on Neo4j

2. GRC-as-code analytics & monitoring

3. Templated & self-serve reporting in BI tooling

Open-source graph model

The core of Emergence is an open-source security graph model with two functions:

1. Represent and connect GRC constructs such as, but not limited to policy statements, requirements, controls, & risks.

2. Ingest, normalize, and connect data from across the environment to inform the GRC program from tooling such as endpoint security, vulnerability scans, human resources, identity and access management, & cloud environments.

Because of the design, organizations can extend the graph model to ingest and normalize data from a variety of sources without relying on a vendor-supported integration.

GRC-as-code

Emergence provides an interface for data-source-agnostic analytics & queries to automate and embed requirements, processes, and objectives.

For example, you can now ask in a single query:

1. Which controls are not defined in any policy or procedure?

2. Does the operating system of a device impact the conformance to certain controls?

3. How many of our controls only apply to one requirement?

You can also automate common manual processes:

1. Perform risk quantification whenever a risk is added or updated

2. Assess level of effort to support new contractual requirements

3. Evidence collection & implementation assessments

Stakeholder-specific reporting

Emergence exposes the security graph to common business intelligence tools & provides out-of-the-box reporting for unique stakeholders.

GRC is comprised of a diverse group of stakeholders and activities. While there is benefit in centralizing GRC data, the opposite is true for the user experience.

Emergence supports micro, stakeholder specific user experiences.

Whether it is a software developer, network engineer, human resources manager, identity and access management administrator, operations manager - each stakeholder has a unique role to play.

With this in mind, you can create better reporting delivered to the right people to enable their workflow, or let them create their own.

Enough reading, see my progress:

Join the people flipping GRC upside-down

An open-source security graph, GRC-as-code, stakeholder-specific reporting.

These are the tools that allow an organization to fundamentally shift their GRC program to drive adoption throughout the organization and better align to business objectives.

If I peaked your interest - join the movement.

Subscribe now

The rise of BI has lead many organizations to put a claim that they embrace “data-driven decision making”. Why is it then, that when we look at our security programs, they are rarely making data-driven decisions?

Early Adopters: Security Operations

The use of security data has been best adopted within Security Operations (SecOps) due to the large amounts of log data being centralized, normalized, and analyzed for detections. Most Security Information & Event Management (SIEM) tools also provide the capabilities to build dashboards based on this data for leadership & analysts to use.

While these capabilities have spawned the use of data visualization & analytics to understand large amounts of security data, there are several challenges that lead me to believe the next evolution will not be seen in SIEM platforms for two key reasons:

1. Cost-to-value ratio

2. Platform use case

Cost-to-Value Ratio

It’s no secret that SIEM tools are expensive, and this is mainly due to their data ingestion costs. The current trend in security operations is to ingest only the data needed in an effort to optimize costs; in other words, SIEM tools have become a game of pick & choose.

Paired with the explosion of security data generated by a modern environment, the SIEM becomes a hard place to win the battle over ingesting more data. But why would we need to ingest more data?

There are still a variety of data sources and data types that are not typically ingested by SIEM tools that provide value in decision-making such as:

• Vulnerability management

• Security audit & assessment results

• Penetration testing

• Security awareness training

• Policy, procedures, & other governance data

Making the case to ingest additional data into a tool with uneconomical ingestion and storage costs for the sake of enabling more analytics use cases seems hard pressed - especially when the current conversations are “how we can ingest less?” instead of “what else can we ingest?”

In short: The cost-to-value ratio of ingesting data sources into a SIEM that do not contribute to detection is insufficient.

Platform Use Case

SIEMs were built for a purpose: the detection & investigation of security incidents. While analytics and visualization is rather streamlined within that context, its when you stray outwards to GRC, Security Testing, Security Awareness Training, and other contexts that the viability of the SIEM as the analytic tool of choice quickly fades.

Data visualization, for example, often lives within the tool for use by SOC managers and analysts. While a portion of the access constraint is alleviated with the introduction of cloud-hosted SIEMs, it’s far-fetched to expect all relevant stakeholders to access the SIEM in order to view and consume security analytics.

Because SIEMs are primarily security tools, there is also fewer members of the analytics community familiar with the platform compared to other industry-standard tools such as R, Python, Tableau, or PowerBI. In fact, many organizations may already have the talent in-house that are familiar with these tools.

The Next Wave: Unhooking Analytics from Detection

There is a phenomena in information security that we often think our problems are unique to the field, instead opting to reinvent the wheel instead of look for solutions in adjacent disciplines.

The next wave of analytics for information security programs looking to expand beyond the perimeters of security operations likely won’t be in the confines of traditional security tools — SIEM or otherwise — as they’ll quickly discover the two roadblocks for adoption:

1. Cost-to-value ratio

2. Vertical use cases

Instead, organizations will borrow from the Business Intelligence (BI) playbook and look to build data pipelines connecting security data and centralizing it in lower-cost data lakes. In fact, we are already starting to see this transition with organizations retaining data not actively used in SIEM detection but still of value in other storage methods.

Leveraging Previously Untapped Data

By lowering the cost-to-value for ingestion & unhooking analytics from detection, security teams can begin to harness new data sources that were previously neglected.

You can think of this phenomena in the context of oil extraction. In the oil industry, certain deposits were considered too difficult & costly to reasonably extract. Instead, they went unused until new technologies — such as horizontal drilling and hydraulic fracturing — shifted the cost-to-value ratio.

Just as these new technologies enabled the oil industry to leverage previously underutilized oil deposits, using different technologies for security analytics turns previously unviable data sources into positive return-on-investment (ROI).

Governance, Risk, & Compliance (GRC) data, for example, is of significant value: laying the foundation of a security program’s goals, objectives, and controls. Despite this value, many organizations leave this data untapped in their security analytics programs, instead living in spreadsheets, documents, and presentations.

To tackle this, we’ve seen an emerging “GRC Tools” market that aims to operationalize GRC activities through software - however, most fall flat in their ability to act as analytics tools across an organization’s security data estate, or even just their GRC data at that.

In addition, most security data currently being analyzed is structured in nature. However, there is a large variety of unstructured data generated, such as but not limited to security testing reports, security questionnaires, contracts, security advisories, and threat intelligence. Implementing dedicated analytic pipelines opens the door for this data to generate insights with the use of natural language models that can analyze unstructured data at scale.

Breaking Down Data Siloes

It’s not enough to just tap into new data sources - after all, many of these sources have their own siloed analytics capabilities housed in that domain’s tools, such as vulnerability management data.

We’ve often heard of ambitions to break down siloes in organizations and improving cross-department collaboration and communication. While this has been an on-going objective in organizations across the board — the concept has merit. Centralized security analytics democratizes it across the organization and unlocks novel insights from previously siloed data.

While the security operations team may be the main authors and consumers of security analytics in the old world, they are rarely the only team that analytics impact. Helpdesk, Identity & Access Management, HR, Infrastructure, and Development are only a few of the domains that not only contribute security data, but directly benefit from its analysis and insights.

Security data is often siloed: generated, analyzed, and consumed all within the confines of a discrete piece of the organization. For example, the development team may analyze the results of their application security testing, software composition analysis, and unit testing; however this data rarely makes it outside the walls of the development team - when it does, it takes the form of slide decks and other upwards reporting to management. Centralizing this data to be correlated across the entire security data estate instead unlocks new value and insights.

Enabling Distribution

Using traditional BI infrastructure also allows organizations to reduce friction in distributing analytics across the organization by leveraging the same tools stakeholders are already used to using to consume data.

With decentralized security analytics, data is analyzed and visualized within the individual security tools. For example, your phishing simulation tool may have the ability to report on the results of recent tests and generate some charts. Your vulnerability scanner is likely able to generate a report or Excel output that can be filtered. But what happens when an adjacent team needs to view these reports?

You create an inefficient web of access, where teams access security analytics in a variety of places, using a variety of tools.

In scenarios that granting direct access is not viable, such as executive reporting, we see employees cobbling together slide decks with manually input metrics and graphs.

Centralized security analytics, on the other hand, leverages the same infrastructure as the rest of the business, cleaning up the mess of information distribution.

Whether the business is using Microsoft Fabric, open source, or even homegrown analytics pipelines, security can be incorporated into the existing data culture and take advantage of any existing maturity in managing the lifecycle of BI resources.

It changes the game when security professionals take advantage of the headways already made by the BI community and contribute intelligence to the organization.

Conclusion

In conclusion, the landscape of data analytics in information security is undergoing a significant transformation. As we have explored, the limitations of traditional SIEM tools in terms of cost-to-value ratio and platform use case are prompting organizations to seek alternative solutions for their security analytics needs. The future lies in unhooking analytics from detection and embracing the principles of Business Intelligence (BI) to centralize, analyze, and distribute security data more effectively and economically.

By adopting lower-cost data lakes and leveraging technologies that can process both structured and unstructured data, security teams can tap into a wealth of previously underutilized data sources. This shift not only promises a better return on investment but also enables a more comprehensive understanding of the security landscape. Furthermore, breaking down data siloes and democratizing security analytics across the organization will foster better collaboration, enhance decision-making, and ultimately strengthen the organization's security posture.

As we move forward, it is clear that the integration of security analytics into the broader BI infrastructure will be a game-changer. It will allow security professionals to benefit from the advancements in BI, contribute valuable insights to the organization, and ensure that security data is not just collected, but harnessed to its full potential. The new world of data analytics in information security is not just about collecting more data—it's about making smarter, more strategic use of the data we have to protect our digital assets and infrastructure in an ever-evolving threat landscape.