What a Risk Committee Actually Asks Before You Go Live
Teams usually prepare for the wrong meeting. They arrive with accuracy metrics and leave with eleven questions about everything else. Here is the list, and the artefact that answers each one.
A risk committee is not evaluating your model. It is evaluating whether the organisation can defend a decision the model made, eighteen months from now, to someone who was not in the room. Once you accept that, the questions stop feeling arbitrary.
The Eleven Questions
- What decision does this make, and what happens if it is wrong? Answered by a one-page decision description, not an architecture diagram.
- Where does every input come from? Answered by data lineage — source system, owner, refresh interval, and what happens when it is stale.
- Who can override it, and is the override recorded? Answered by the override path in the product, plus a sample of the audit log.
- How do you know it is still working? Answered by the monitoring plan: which metrics, which thresholds, who is paged.
- What does it do with a customer it has never seen before? Answered by the out-of-distribution policy and the fallback decision.
- Can it discriminate? Answered by a fairness assessment across protected characteristics, with the methodology stated.
- Who approved the current version, and when? Answered by a model register entry with a version, a date and a name.
- How would we turn it off? Answered by a documented kill switch that someone has actually tested.
- What is the challenger, and how does it compare? Answered by a side-by-side over the same period.
- Can you reproduce a decision from six months ago? Answered by versioned inputs, code and model weights.
- Who owns it after go-live? Answered by a named person in the operating model — not a team, a person.
Build the Pack While You Build the System
The teams that clear committee in one sitting are not better at presenting. They produced these artefacts as a by-product of building, because each one is genuinely useful during development — lineage prevents silent breakage, monitoring catches drift, the register stops two people shipping different versions.
The teams that take three sittings write the pack afterwards, from memory, under time pressure. The gaps in the pack are the gaps in the system; the committee is simply the first party to read them out loud.
What to Bring
One binder, eleven tabs, one page each. Accuracy metrics go in an appendix. Lead with the decision, the failure mode and the person who owns it — in that order.
Shahaf Lavi
Founder, Zero Evoke