The preceding articles describe what a second-line AI function should do. This last article in the series is about building it: turning principles into a scored instrument, a control standard, and a sequenced plan that produces something real within a quarter.
Two failure modes bracket this work: The framework that is comprehensive, well written, but never operated, a forty-page policy, filed, and activity without structure, where reviews happen case by case with no standard behind them and decisions cannot be reconstructed. The path between runs through making the framework concrete enough to apply and small enough to finish.
Turning Tiering Into a Scored Instrument
In an earlier article in the series, we set out the dimensions on which AI systems should be assessed, but dimensions alone leave too much to judgment.Converting them into a scored rubric with explicit anchors makes assessments consistent between reviewers, defensible when contested, and auditable afterwards.
A workable rubric scores seven dimensions on a one, three, or five scale: a coarse scale deliberately, because finer gradations invite a precision the underlying judgment cannot support.
Autonomy. One point: the system suggests and a person authors the output. Three: the system drafts and a person reviews before use. Five: the system acts or publishes without human review.
Audience. One: a single internal user. Three: an internal enterprise process or many internal users. Five: an external client, subscriber, or the public.
Consequence of error. One: convenience or productivity only. Three: operational, financial, or reputational impact contained internally. Five: market-moving, contractually relied upon, or published as fact.
Data sensitivity.One: public or non-sensitive data. Three: internal or confidential business data. Five: client-identifiable data, personal data, or material non-public information.
Reversibility.One: fully reversible before any impact. Three: correctable with effort and notification. Five: irreversible, externally distributed, or already acted upon.
Regulatory exposure. One: no specific obligation identified. Three: transparency or disclosure obligations apply. Five: falls in a regulated use category or triggers sectoral obligations.
Third-party dependency. One: fully internal model and infrastructure. Three: managed vendor with contractual visibility. Five: external model with limited visibility into changes or training data.
Scores aggregate into three tiers: Critical at twenty-six to thirty-five, Elevated at sixteen to twenty-five, and Standard at seven to fifteen.
A small number of characteristics should override the score and force the top tier outright, because they describe exposures that no favorable score elsewhere should be able to offset: the system publishes to clients or the public without human review; it takes autonomous action with an irreversible external effect; it writes to a book of record or a regulatory reporting system; it modifies its own rules or objectives at runtime; or it processes material non-public or client-identifiable data in a client-facing path.
Control Requirements by Tier
Tiering is only useful if something different actually happens at each tier. This is where most frameworks stop short, leaving proportionality as a stated intention rather than a specified standard. The remedy is a matrix that states, control by control, what is required at each level.
Standard tier
Register in the inventory, document intended use and limitations in short form, self-attest, and log. Deployment proceeds on first-line self-attestation, sampled by the second line. Annual re-review. Standard vendor process for any third party. No evaluation suite, adversarial testing, or bias assessment required.
The great majority of the estate should sit here and should never require second-line involvement on an individual basis. A framework that pulls routine internal tools into review has misallocated its scarcest resource.
Elevated tier
A full intended-use statement, reviewed. An evaluation suite with a golden dataset. Groundedness and citation-fidelity testing where retrieval is used. Targeted adversarial testing scoped to the use case. Bias assessment where the use case affects people. A named reviewer role and a defined review point. Metric monitoring with periodic review. Second-line sign-off for deployment. Re-review on material change, as defined by the second line. Enhanced AI assessment for third parties. A documented decommission plan.
Critical tier
Everything above, strengthened at each point. The intended-use statement is reviewed and challenged. The evaluation methodology receives independent second-line review. Groundedness testing carries a numeric pass threshold operating as a launch gate. Adversarial testing is mandatory pre-launch and at least annually. Bias assessment is required and documented with a remediation path. Human oversight names an accountable reviewer with a defined sampling rate and retained evidence. Monitoring is real-time with alert thresholds and an on-call owner. The risk committee approves deployment. Any model or version change triggers re-review, including vendor-initiated changes. Third-party assessment is full, with an exit plan that is documented and rehearsed. The decommission plan is likewise rehearsed rather than merely written.
Two rows in that matrix carry disproportionate weight. The re-review trigger at critical tier, any version change, including vendor-initiated, is what closes the silent-update gap that otherwise invalidates evaluation evidence without anyone noticing. And the named accountable reviewer with a defined sampling rate is what separates real human oversight from the most common governance fiction: a human-in-the-loop requirement with no human named.
Sequencing Principles
The order of construction matters more than the speed of it, and four principles determine it.
Inventory before policy. A rule cannot be written for a population that has not been counted. Policy written first will be either too strict for the estate that exists or irrelevant to it.
Tier before control. Control requirements are meaningless until there is a basis for applying them differentially. Uniform controls are the signature of a framework built without tiering.
Pilot before publication. A framework that has never been run against a live system contains defects its authors cannot see.
Attach before building. Use existing governance forums, existing procurement processes, and existing issue-management tooling wherever they exist. Every parallel structure is a permanent maintenance obligation.
The First Ninety Days
Days one to thirty, listen and locate
Meet the adjacent leaders first: engineering, responsible AI, privacy, security, procurement. Agree the seams before writing anything.
Map what already exists: Meaningful governance almost always exists in pockets. Proposing to build something that is already there is the fastest way to lose credibility in the first month.
Build a provisional inventory bottom-up: from platform telemetry, API gateway logs, cloud spend, and vendor contracts rather than from a survey. Surveys find what people remember; telemetry finds what is actually running.
Identify the top ten exposures: the client-facing, published, agentic, or sensitive-data systems. These make the business case for the function.
Understand the client-assurance pipeline: Inbound AI due-diligence questionnaires from customers are frequently the most persuasive argument for the program, because they translate governance into revenue.
Deliverable: a current-state assessment, an agreed division of labour with adjacent functions, and a provisional inventory with the top ten exposures named.
Days thirty-one to sixty, frame and tier
Publish the tiering standard, corrected by what the first thirty days revealed.
Write the policy on one page. A one-page policy with a linked standard and a linked procedure gets read. A forty-page policy gets filed.
Pilot on three live systems: one agentic, one client-facing, one internal. Run each through the full framework and record every place it broke. Then publish the breakages, which demonstrates that the second line tests itself before it tests others.
Stand up the working group: monthly, working level, with a real agenda. The committee comes later; the working group is what makes the committee useful.
Draft the intake questionnaire at ten questions maximum. Every additional question halves compliance.
Deliverable:an adopted rubric, a one-page policy, three completed pilots with documented framework defects, and a functioning working group.
Days sixty-one to ninety, prove and report
Produce the first aggregate risk report. It will be incomplete, and it should say so plainly. An honest partial picture beats a polished one resting on a fiction of full coverage.
Baseline the initial indicator set.Improvement cannot be shown without a starting line, and the starting line must be captured before the program has had time to improve it.
Put a real issue through the issue process.A process that has never carried a live issue is untested. Find one, rate it, assign it, track it.
Present the roadmap and the resourcing ask. With pilot evidence and the top-ten exposures behind it, the resourcing conversation becomes arithmetic rather than advocacy.
Deliverable: a first executive dashboard, a baselined indicator set, a live issue-management process, and an evidence-backed twelve-month roadmap.
Months Four to Twelve, Scaling
Coverage:The inventory moves from provisional to comprehensive, and shadow-AI discovery becomes a tracked metric rather than a recurring surprise.
Control standards:Published standards for each named risk domain, bias, explainability, robustness, hallucination, drift, data protection, so that first-line teams design against a known bar.
Third party:AI-specific assessment embedded inside procurement and vendor onboarding rather than bolted on afterwards, with concentration tracked.
Automation:Inventory, evidence collection, and attestation move into governance tooling and delivery pipeline hooks. The objective is that registering a system becomes easier than not registering it.
Team:Build to a small senior team across advisory, framework and policy, and reporting and operations. Advisory capacity is almost always the binding constraint.
Enablement:Role-based training for engineers, product managers, and business owners. This is the fastest lever available and the one with the widest reach per unit of effort.
Regulatory watch:A standing quarterly process converting regulatory movement into framework changes with a documented rationale.
A Note on Reporting Maturity
In the first two quarters most indicators will be partial or unavailable. Report them anyway, marked as partial, with the date by which they will be complete. A dashboard that only shows the metrics already looking healthy trains its audience to distrust it later, and forfeits the most valuable thing a new function has: a credible baseline that shows what changed.
Closing the Series
Twelve articles have covered the first line that builds and owns, the second line that sets standards and challenges and measures and designs control into the architecture, the third line that tests whether any of it is true, the third-party estate outside direct control, the mechanics of failure, the surrounding legal terrain, and now the operating model and the build itself.
The thread running through all of it is that AI does not demand an entirely new risk discipline. It demands that an old one be rebuilt honestly for an object that is probabilistic, general-purpose, frequently third-party, and increasingly capable of acting rather than advising.
Explore more: Start Here · Topics · Archive · About