Where does qualitative risk analysis belong, and where should it stop?
Session 16 · ERQI Global Virtual Risk Summit
Qualitative vs. quantitative risk analysis: the debate
Sriram Raghavan, Tony Martin-Vegue, Chris Patteson & Adam Lamantia, moderated by David White
Overview
The Summit's closing debate found more agreement than its title promised. Four practitioners agreed that the method should follow the decision, with simple scoring useful for early triage and dollars needed once controls, risk transfer or the board come into it. They split on whether qualitative ratings should ever be formalized in decisions that carry consequences.
Session 16 recording, ERQI Global Virtual Risk Summit 2026
Where qualitative earns a place
Chris Patteson opened with a defense of simple scoring. His work in quantification began after WannaCry and NotPetya, when finance told him to retire red, yellow and green charts, pointing to 400 million dollars the company had lost. He still uses one-to-five scoring early in the annual cycle at very large organizations, because it canvasses front-line staff cheaply and shows where rigorous modeling is needed. "Heat map's just a visualization," he said, and the real question is what sits on the axes. Once discussion turns to controls, risk transfer or board impact, he moves to dollars, and he sets up frameworks so successors can see that continuum.
Sriram Raghavan approached it from the consumer of risk information. Executives need abstraction to decide quickly, but a decision outlives the person who made it, so a quantified baseline gives later owners something to test and update. Adam Lamantia summarized the trade-off. Qualitative is quick and accessible to leadership. Quantitative is more defensible and reproducible when new analysts arrive, and suits decisions that cascade into others.
The case against formalizing it
Tony Martin-Vegue diverged most. He would not run a Monte Carlo simulation on a 1,000-dollar decision and uses gut checks to start an analysis. His test is whether a method is decision useful, and ordinal ratings fail because they cannot be tested, compared or added. He has asked about 60 people to rate one data breach scenario, and found roughly 30% said low, 30% high and 40% medium. "With qualitative risk, we're not actually rating the risk, we're rating the rater."
He agreed that telling people their risk matrix is useless won't win them over, but said that in his own view 30 years of use doesn't make it right, comparing it to bloodletting, an accepted medical treatment for hundreds of years that medicine has since abandoned. He also argued that red, yellow, green carries its own false precision, hiding the many small incidents and undercounting the rare large one. Raghavan noted that consumers often ask for simple colors. Patteson and Lamantia replied that a simple output can sit on a rigorous analysis, and a high, medium or low label can be traced back to a mathematical scale when someone asks.
Culture, tails and human harm
Lamantia named mismanaging the cultural shift as the biggest reason he sees quantification programs fail. Patteson advised against calling anyone's baby ugly, and elicits scenarios by asking operations staff what a line outage cost. Martin-Vegue translates security into return on investment and risk exposure.
On tails, he separated the matrix as a rudimentary model from the heat map as a visualization. Plotting a P50, P95 or mean loses the tail, so he supplements it with ranges or a loss exceedance curve. Patteson used car insurance to explain annual expected loss, and Raghavan warned that a 10 million dollar annual figure can accumulate past 100 million over 20 years. Asked whether some risks should not be quantified, Martin-Vegue said he would not run a risk quantification on human life. White suggested that organizations model injuries instead, since they usually cost more and are easier to discuss.
“With qualitative risk, we're not actually rating the risk, we're rating the rater.”
Tony Martin-Vegue, after asking about 60 people to rate one data breach scenario
Key takeaways
1
Match the method to the decision
Qualitative scoring suits early triage. Dollars belong where controls, risk transfer and board impact are discussed.
Watch at 25:132
Test the labels
Martin-Vegue's exercise showed little agreement on a single breach scenario, so ordinal ratings tend to reflect the rater.
Watch at 19:463
Show ranges and tails
A single point on a heat map hides the tail, so pair it with ranges or a loss exceedance curve.
Watch at 44:524
Speak the language of business
Return on investment and risk exposure hold executive attention better than colors, adjectives or security jargon.
Watch at 31:105
Manage the cultural shift
Build on existing practice rather than dismissing it, and keep a path from simple scoring to quantification.
Watch at 39:54
