Across this series I have argued that data readiness, agent data domains, and lineage are what make generative and agentic AI defensible. But defensibility rests on a capability I have only gestured at until now:the ability to say, with evidence, whether the system is any good. Every five question test in this series has assumed a golden set, a ground truth, an evaluation dataset. This article is about that assumed thing, because it turns out to be one of the hardest and most neglected data assets in the entire AI estate.
The neglect is understandable. Building the model feels like the work. Measuring it feels like overhead. But in a regulated institution the relationship is inverted. The evaluation data is what lets you validate, monitor, improve, approve, and defend. A model you cannot measure is a model you cannot govern, and a model you cannot govern does not belong in production. Measurement is not the overhead around the real work. It is the real work.
Why Measuring Generative AI Is Genuinely Hard
Traditional model validation had a comfortable property:a ground truth to compare against. A credit model predicted default or no default, and reality eventually revealed which. You could compute accuracy, precision, and recall against outcomes that actually happened, and you could back test the model against years of historical data to prove it would have performed. Generative AI breaks this comfort. When a model summarizes a policy, drafts a memo, or answers a question, there is often no single correct output and no future event that settles the score. Quality becomes multi-dimensional and partly subjective: was the answer accurate, complete, grounded in the source, appropriately hedged, and free of anything it should not have said?
This is why measurement for generative AI is itself a data problem before it is a metrics problem. You cannot compute a score without first constructing the thing you score against. That construction, curating questions, defining what good looks like, and capturing it as reusable data, is the work most teams skip and later regret when validation or an examiner asks for evidence they never built.
Ground Truth Is a Curated Asset, Not a Found One
Ground truth is the set of known correct answers you measure the system against. In generative AI, ground truth rarely exists readymade. It has to be curated deliberately, by people who know the domain, and then maintained as the domain changes. That makes it one of the most valuable and most underfunded data assets an AI program owns.
What disciplined ground truth curation requires:
1. Representative coverage
Questions drawn from real usage, not invented in a conference room. The evaluation set should reflect what users actually ask, including the awkward, the ambiguous, and the adversarial, because those are where systems fail.
2. Expert defined answers
Correct answers defined by qualified people, with the reasoning captured. In finance this often means the answer plus the authoritative source it derives from, so the standard is defensible and not one reviewer's opinion.
3. Provenance on the truth itself
A record of who defined each answer, when, and against which source version. Ground truth without its own lineage cannot be trusted or defended, and it drifts out of date silently.
4. Maintenance over time
As policies, products, and regulations change, the correct answers change with them. A ground truth set that is not maintained slowly becomes a measure of how the world used to be.
Notice that ground truth is subject to every data governance discipline from Article 4. It has owners, versions, lineage, and a lifecycle. The dataset you measure against must be governed as rigorously as the system you measure, or your metrics rest on sand.
The Seduction and Danger of LLM as Judge
Because human evaluation is slow and expensive, a popular pattern has emerged:use a language model to judge the outputs of another language model. LLM as judge is genuinely useful. It scales evaluation to volumes humans cannot reach and can score dimensions like grounding and relevance reasonably well. It is also quietly dangerous if adopted without discipline, and in a regulated setting the danger is the part that matters.
The core risk is circularity.If a model judges a model, what validates the judge? An LLM judge has its own biases, blind spots, and failure modes, and it can confidently agree with a wrong answer for the same reasons the original model produced it. Treating the judge's scores as ground truth rather than as an estimate to be validated is how a program convinces itself a system is performing well while the evidence quietly rots.
The disciplines that make LLM as judge safe are, again, data disciplines:
Calibrate the judge against human ground truth
The judge must be validated against a human curated golden set before its scores are trusted, and re-validated when it or the system changes. An unvalidated judge is an opinion with a number attached.
Keep humans in the loop where it counts
Use the judge to scale evaluation, not to replace the human standard. High stakes and edge cases still warrant human review, with the judge triaging volume rather than owning the verdict.
Log the judge's own lineage
Judge outputs are evaluation data too, with the same provenance and versioning obligations. Which judge, which version, which prompt, produced which score matters when someone asks how you know the system works.
Prefer pairwise comparison over absolute scoring
Asking a judge to score an answer good or bad in isolation is noisy, because the model has no anchor for what the scale means. Asking it to choose which of two responses is better is a far more reliable signal, since relative comparison is easier and more stable than absolute judgment. Where you can, evaluate pairwise rather than with a yes or no verdict.
Randomize order to defeat position bias
LLM judges have a well-documented position bias, tending to favor whichever response they see first. Randomize the order of the two candidates across evaluations, and ideally score each pair in both orders, so the position cancels out rather than quietly skewing your results.
Use a different, stronger model as the judge
Using the same model to generate and to judge maximizes shared blind spots, since it will forgive exactly the errors it is prone to make. A different model, and ideally a more capable one than the system under test, gives a more independent and more discerning assessment. An independent judge is the whole point.
From One Time Evaluation to Continuous Monitoring
A score at launch tells you the system was acceptable once. It says nothing about today. Generative and agentic systems drift: the underlying model is updated, the corpus changes, usage patterns shift, and behavior moves with them. Measurement therefore cannot be a gate you pass once.It must be a continuous discipline, which means the evaluation data has to be production infrastructure, not a onetime project artifact.
In practice this means running the golden set as a regression suite whenever anything in the estate changes, monitoring live outputs against quality signals rather than only pre-launch samples, capturing user feedback and routing failures back to the owners who can fix them, and watching for the slow degradation that no single output reveals but a trend does. This is the same monitoring and remediation loop the rest of the series has described, now powered by evaluation data. Without a maintained golden set, continuous monitoring has nothing to measure against and quietly becomes continuous hoping.
What Regulators Expect You to Have Measured
Supervisory expectations make measurement non negotiable. SR 11-7 built model risk management on validation and ongoing monitoring, both of which are meaningless without evaluation data. The EU AI Actrequires accuracy, robustness, and performance to be tested and documented for higher risk systems. The NIST AI RMF measure function is explicitly about assessing and tracking AI performance and trustworthiness over time. Each framework assumes the institution can produce evidence that the system was tested, is monitored, and performs within stated bounds. That evidence is evaluation data. An institution without curated ground truth and a monitoring history is not in a position to satisfy any of these, however good its models happen to be.
The Test I Give Every System
Before I accept that an AI system is measured rather than merely launched, I ask five questions.
Do you have a curated golden set with expert defined answers, owned and versioned like any other governed dataset?
Does it represent what users actually ask, including the hard and adversarial cases?
If you use an LLM judge, has it been calibrated against human ground truth, and do you re-validate it on change?
Do you run evaluation continuously as a regression suite, or did you measure once at launch and stop?
And can you produce, for a regulator or your own risk function, a documented history showing the system has performed within bounds over time?
Systems that can answer these are measured, monitored, and defensible. Systems that cannot are running on the memory of a single good demo, and in this domain the gap between a demo that scored well once and a system proven to perform is exactly the gap between a pilot and production.
The next and final article in the series steps back to the operating model: who owns AI data across all of these domains, how data product thinking applies, and how to fund the unglamorous foundation work that every article in this series has ultimately been about.
Explore more: Start Here · Topics · Archive · About