Subject Matter Expertise in regards to risk assessments and loss models – especially in absence of large data sets – is a very acceptable method of model or tool input. However, there are prerequisites. There needs to be calibration or estimation training; usually with the goal of making estimates of which the SME feels 90% confident of the values they are providing. There needs to be reasonable logic in choosing priors (if available). Are the estimates for threat event frequency based off information your organization has, the industry your organization is in, etc. The reality is that the threat landscape changes from organization to organization.

I am a strong advocate of FAIR / FAIRLite because when used right – it usually results in better decision making, consistent use of a simple methodology, and easier conversations with management.

Finally – pardon the soap box moment; it is not directed at you – IT folks – especially information security folks need to take a step back and really understand that uncertainty and risk go hand in hand. I strongly believe that if those in the information security industry that frown upon and discourage risk quantification would learn more about the risk management profession – we could mature this discipline within our profession very quickly. This is no doubt easier said then done.

I think this might all be great, if we had some reasonable way to determine the numbers which go in.

You say “For example, if I perform 1001 simulations where a value between $1 and $10 dollars is drawn– I would add up the sum of all the simulations and divide it by 1000.”

How do you go about determining that?

But this is not always the right approach. For example, in the case of the exponential distribution, it is usually leveraged when in scenarios where you need to represent the time between events. In absence of a large amount of data – this would seem to be problematic for most information security events.

The beauty of the betaPERT is that the SME estimates can take the form of curves related to other continuous distributions (e.g. Log normal, F, maybe even exponential). There are philosophical types out there that even debate the normal distribution being past its prime. They can debate all they want. In the mean time – I believe that how RMI is leveraging the betaPERT in FAIRLite offers the most flexibility from a risk quantification perspective.

Finally, regarding the sniff test – I absolutely agree. I recently “modeled” some risk issues where the aggregate expected loss magnitude value a very large amount ~$12M. There was no way this was close to being accurate. The loss estimates that were provided were no where close to being realistic. SNIFF test WIN in this case – not the right risk amount. The ironic part was that a few model iterations later, where the aggregate expected loss magnitude was around a tenth of the original – one of the people that had provided input to the original estimates stated the more realistic model was too much. You should have seen the look I received when I showed this individual the $12M model with the original estimates.

Good discussion.

Thank you for the very helpful clarifications.

Fascinating stuff, especially trying to get your hands around the tail events in a believable and meaningful way.

My years of work in medical outcomes taught me the importance of what I would call the “sniff” test – while it’s true that our brains aren’t really wired to think correctly about probability, at the end of the day, every analysis has to pass the test of common sense.

I have been thinking for some time to try sampling an exponential or Pareto distribution instead of a normal distribution for comparison purposes, but my math skills are a bit rusty. It seems like that might be another way to get at estimates of tail events. Maybe you know how to do this?

We agree about LEF – some types of events like AV are likely to occur much more frequently than once per year.

Patrick

Minimum: 1
Most Likely: 4
Maximum: 8

Using NORMSINV with the MEAN and STDEV from all the simulation values may give you some values further in the tail; possible zeros or maybe a 9, possibly a 10. So how does this relate to betaPert – betaPert is bounded by the subject matter expertise. While that may be more then sufficient for generating a distribution that accounts for 90% certainty – leveraging the approach above for modeling purposes extends that certainty further.

Regarding your loss event comment, your comment regarding ‘common sense tells us that the first loss event…”. I do not disagree with your statement. But this may not be the case for a lot of risk scenarios – especially in the world of AV. In addition, we have to remember that we are here to give the decision maker the best information we have. We can not assume that there will be mitigation activity taken just because one loss event occurred. Regardless, in those scenarios where you think a one time, large loss magnitude event is going to result in response / mitigation activity that result in the future (after the initial loss event) LEF being near 0, I would articulate that to management. For me personally, I would not adjust the loss impact downwards until after I have had a conversation with management.

Thanks for the comment. Let me know if further clarification is needed. Also, please do share anything you are working on that you think would benefit the community at large.

My comments from this morning seem to have disappeared, so here goes again.

Median: if there are an odd number of values in the sample or population, then it’s the middle value, as you say. If the number of values in the sample is even, then some people say to take the average of the two values closest to the middle. Take your pick.

Question: What do you mean by the following language?

“a distribution that leverages the LM mean and standard deviation …”

Is this a calculation of some sort? If so, what are you doing with the mean and std deviation?

Comment: since FAIR Lite uses a betaPERT function that gives weight to the most likely estimate, why would you want to go through the exercise that you have? The sampling of the distribution is, by definition, skewed towards the most likely value.

Comment: for some types of events – a large data breach, say – even if the simulation calculates multiple loss events per year, common sense tells us that once the event happens, it is not likely to happen again, during the same year, at least. In this case, we have to set LEF = 1 and calculate LM from there.

Comment: I have found it useful to do FAIR Lite analyses in groups of three: a most optimistic, a most likely, and a least likely scenario. By presenting the results together, I can express a range of estimates that I, at least, find useful in making a decision. I have a model that does this, if you would like to see it.

Thanks for you post!

Best regards,

Patrick Florer
Dallas