Most AI governance programs are strongest at design and weakest at measurement. Frameworks are written, rubrics published, gates established, and then the organization has remarkably little idea whether any of it is working. This is the least glamorous part of second-line defense practice and the part that separates a real control function from a well-documented intention.
The distinction at the center of this article is one that governance discussions routinely blur: the difference between measuring the model and measuring the control.
Two Different Measurement Problems
Model performance asks whether the system does its job well: accuracy, groundedness, precision and recall, latency, cost. This is largely a first-line responsibility, though the second line must be able to interrogate the methodology.
Control operating effectiveness asks whether the governance apparatus is functioning: whether the inventory is complete, whether tiers are assigned correctly, whether gates are actually applied, whether conditions are complied with, whether human checkpoints do real work, and whether escalation happens when it should. This is squarely a second-line defense function, and it is where programs are thinnest.
The distinction matters because the two can diverge in both directions. A portfolio of highly accurate models governed by controls nobody follows is a program in trouble. A well-followed control framework applied to a portfolio nobody has evaluated is equally in trouble. A serious function measures both and reports them separately.
Evaluation as a Second-Line Instrument
The traditional model validation toolkit assumed a stable specification and reproducible outputs. Generative systems require a different instrument:a structured evaluation harness that scores outputs against a representative test set on dimensions appropriate to the use.
That harness is a governance asset as much as an engineering one. It converts an unfalsifiable claim, the system works well, into an evidenced one, and it makes conditional approval enforceable, because a condition expressed as an accuracy threshold can be verified rather than merely asserted.
The dimensions worth scoring vary by system type:
Groundedness and citation accuracy for retrieval systems: is the answer supported by the retrieved source, and is the citation correct?
Factual accuracy against a gold-standard set, with particular attention to the hardest and rarest cases rather than the average.
Refusal and fallback behavior: does the system decline when evidence is insufficient, or does it produce a confident guess?
Consistency: does the same question, asked repeatedly, produce materially different answers?
Safety and policy adherence, including behavior under adversarial input and prompt injection through retrieved or user-supplied content.
Action correctness for agentic systems: not merely whether the answer was right, but whether the action taken was the right action, with the right parameters, in the right sequence.
Bias and disparate outcome testing wherever the system informs decisions affecting people.
Two methodological points determine whether the numbers mean anything:
The test set must be representative of production reality rather than of the cases the builder anticipated, which requires drawing from actual traffic, not only from designed examples.
The aggregate figure is the least informative statistic available. What matters is the shape of the error distribution, which categories fail, whether failures cluster in the complex and ambiguous cases, and whether the system signals its own uncertainty.
A system that is ninety-five percent accurate overall but concentrates its errors in the highest-value decisions is worse than the headline suggests.
Key Risk Indicators for the Portfolio
Beyond individual systems, the second line must monitor the estate. A workable indicator set covers five areas.
1. Coverage and inventory integrity
Estimated proportion of the actual AI estate that is registered: triangulated from enterprise model API consumption, procurement records, and periodic discovery, since the unregistered population cannot be counted directly.
Registrations completed at design time versus at or after release.
Inventory entries with incomplete mandatory fields, particularly tool permissions and out-of-scope uses.
Systems whose recorded model version differs from the version actually in production.
2. Tiering integrity
Distribution of the portfolio across tiers, tracked over time.
Rate at which second-line validation revises a first-line self-assessment, and the direction of revision. A persistent upward correction indicates systematic under-tiering.
Systems that have crossed a mandatory top-tier trigger since approval without being re-tiered.
3. Gate and condition compliance
Proportion of high-tier systems in production that completed the gate.
Conditions overdue, by age and by tier.
Approvals granted with an exception or waiver, and the ageing of open waivers.
Median review turnaround: an efficiency measure that is also a risk measure, because a slow gate is a gate people learn to avoid.
4. Operating behavior in production
Human override rate at checkpoints. This is the most diagnostic single indicator in the set, and it is informative at both extremes. A rate near zero suggests the checkpoint is ceremonial, reviewers approving without meaningful engagement. A very high rate suggests the system is not fit for the task. A healthy checkpoint shows a stable, non-trivial override rate and reviewers spending measurable time on the hardest items.
Drift indicators: change in output distribution, retrieval quality, or evaluation scores relative to the approved baseline.
Vendor model change events and the elapsed time before affected systems are re-evaluated.
Incident volume and severity, with root cause classified as model failure, control failure, or misuse.
Escalation volume, which is a health indicator rather than a failure indicator: a portfolio generating no escalations is more likely under-governed than perfectly designed.
5. Culture and literacy
Reach and completion of AI governance training across the population that builds with AI.
Proportion of new use cases arriving through intake versus discovered after the fact.
Time from a builder's first design conversation to first contact with the second line.
That last measure deserves attention.When builders talk to the risk function during design rather than at release, the program is working. When contact happens at release, the function is a bottleneck. When it happens after an incident, there is no function.
Designing the Dashboard
A dashboard exists to provoke decisions, not to display data. Three design rules keep it useful.
1. Every indicator needs a threshold and a defined consequence. An indicator without a trigger point is a chart. If coverage falls below a defined level, or overdue conditions exceed a defined count, something specific must happen: a report to committee, a pause on new approvals in that domain, a targeted discovery exercise.
2. Report trend, not only level. A single number is nearly meaningless without direction. Rising under-tiering corrections or lengthening condition ageing tell a story that a point-in-time figure conceals.
3. Separate the strength of the control from the size of the perimeter. This is the most common way governance reporting misleads. A program showing full compliance across a small registered population, in an organization whose telemetry implies a far larger estate, is reporting the size of its blind spot rather than the strength of its controls. Coverage should sit at the top of the dashboard, above every compliance metric, precisely so that it conditions how the rest is read.
Red Teaming and Adversarial Assurance
Passive monitoring detects problems the system reveals. Adversarial testing finds problems it conceals.For high-tier systems this should be a periodic requirement rather than an occasional exercise, and it should specifically probe prompt injection through retrieved or user-supplied content, attempts to induce action outside the intended scope, extraction of sensitive material from context or retrieval, and behavior at the boundaries of the intended population.
For agentic systems, one test matters more than the rest:attempt to make the agent do something its permissions technically allow but its purpose does not contemplate. The gap between what an agent is for and what it can reach is where the significant failures live.
Reporting Honestly
Second-line reporting to a risk committee or board has an obligation the rest of the organization does not: it must be candid about its own limitations. That means stating estimated coverage rather than implying completeness, distinguishing what has been evaluated from what has merely been registered, reporting incidents and near misses rather than only successes, and identifying where the function lacks the access, tooling, or expertise to provide the assurance it is nominally giving.
A committee that receives only good news is not being governed; it is being managed. The credibility of the function over time rests far more on the accuracy of its self-assessment than on the favorability of its numbers.
Measurement tells the second line whether its controls are working. The next article addresses a more structural question:whether the controls can be built into the systems themselves, rather than layered around them.
Explore more: Start Here · Topics · Archive · About