Session 12 · ERQI Global Virtual Risk Summit
Good enough to decide on: model risk in cyber risk quantification
Laura Voicu & Graeme Keith
Overview
Laura Voicu's Enterprise Risk Quantification Institute (ERQI) paper, "Model Risk Testing and Analysis," adapts banking model-risk practice to cyber risk quantification (CRQ). Its central test is simple: a model is good enough when plausible uncertainty in it would not change the decision it informs. In a recorded conversation, Graeme Keith asks her where cyber models fail, how to test them, and what to ask a vendor.
Session 12 recording, ERQI Global Virtual Risk Summit 2026
Testing against the decision
CRQ output drives risk appetite, insurance and control investment, Voicu said, and if the model is wrong, the decision is wrong. Banking learned this after 2008, and cyber has not yet. Adoption of FAIR ran ahead of scrutiny, and a flawed model shows itself only after a loss.
Her test is to run the model under two or three credible alternatives (different severity or frequency assumptions, different expert inputs or data sources). If every version points to the same action, such as buying the $10 million limit or funding the MFA rollout, the model is robust enough for that decision. If they diverge, choosing which assumptions to believe is the decision, and the decision-maker belongs in that conversation. Keith liked the extra freedom this gives: when a model is not good enough, improve it or change the decision it supports.
The same logic explains how an imperfect model stays useful. Model-wide biases, such as under-reported calibration data, hit every option equally and cancel when ranking control investments. Voicu also scales rigor to consequence, because "complexity is a very, very bad proxy for consequence." A single spreadsheet feeding a board risk appetite number deserves more scrutiny than a sophisticated Bayesian patch-ranking engine.
Why cyber data is hard
In cyber, Voicu said, nearly everything an actuary takes for granted is missing at once. Confirmed loss events number in the hundreds to low thousands over about ten years. Quietly paid ransoms and absorbed business email compromise never reach a database, and insurance claims are truncated below the retention and censored at the policy limit. Most models also assume independence, which she called very wrong: Log4j, MOVEit and shared cloud outages cluster losses, and aggregating scenarios as independent can understate the tail by a factor of two or more. Her advice is to treat tail estimates as their own model-risk problem.
Pushback and vendors
Keith asked whether the paper would send discouraged teams back to gut feel. Voicu said it is not an argument against quantification, and an unvalidated FAIR analysis still beats heat maps and peer benchmarks. The lowest of the paper's three assurance levels needs only documented assumptions and limitations, which she said costs an afternoon. Asked whether it would harden into a compliance checklist, she said to keep asking what decision the testing changed. If nothing changed, either the models are sound or the findings are not reaching the right people.
On vendors, her target is opacity, not vendors as such. In a demo, she would move an input and ask which way the output should go before clicking. Keith argued the principles apply to risk quantification well beyond cyber, wherever there is no established modeling tradition.
“Complexity is a very, very bad proxy for consequence.”
Laura Voicu, on scaling validation effort to what rests on the model
Key takeaways
1
Test against the decision
Ask whether plausible alternative assumptions would change the action. If not, the model is good enough for that use.
Watch at 11:342
Scale rigor to consequence
Spend validation effort where a material financial commitment rests, not where the model is most complex.
Watch at 16:393
Treat the tail as its own problem
Report P95 or P99 with sensitivity bands, state how few extreme observations support them, and sanity check against known extremes such as Equifax.
Watch at 27:244
Start small
A model card with purpose, assumptions, limitations and conditions of use, a qualitative band around the headline number, and two or three recalibration triggers need no validation function.
Watch at 50:125
Ask vendors for the logical chain
Voicu's minimum is a conceptual explanation of how inputs become loss estimates, outputs that move in the expected direction, and uncertainty bounds rather than point estimates. A companion checklist has 24 questions.
Watch at 40:57
