As organizations accelerate the adoption of Generative and Agentic AI, traditional Model Risk Management (MRM) frameworks are being pushed beyond their original design limits. Built for deterministic models with predictable outputs, legacy MRM approaches must now evolve to address probabilistic systems, dynamic behaviors, and increasingly autonomous decision-making. This article introduces a modernized MRM framework for the AI era, one that extends established principles such as SR 11-7 while incorporating new capabilities like prompt governance, hallucination monitoring, agentic workflow oversight, and real-time risk controls across the full AI lifecycle.
1. Traditional versus Generative and Agentic AI MRM Activities
This section compares traditional MRM with Generative and Agentic AI activities.
2. AI Model Inventory Requirements
The model inventory is the foundational record of all AI systems subject to MRM governance. For Generative AI and Agentic AI, standard inventory fields must be extended to capture the additional complexity of these systems.
The following fields are recommended in addition to standard inventory data:
Model architecture type:LLM (GPT, Claude, Llama, etc.), RAG system, agentic workflow, multi-agent system, fine-tuned variant
Access modality:API-accessed (third-party), self-hosted, hybrid; includes vendor name and model version
System prompt version:Version-controlled record of active system prompt(s); change log with dates and change owners
RAG configuration:Vector store type, knowledge base sources, retrieval parameters, last update date
Tool access inventory:Complete list of tools the model/agent can invoke, with permission levels and approval status
Autonomy tier classification:Tier 1 (Assistive) through Tier 4 (Fully Autonomous) per the risk continuum framework
Human oversight protocol:Documented human review requirements, escalation thresholds, and override procedures
Hallucination tolerance threshold:Defined acceptable hallucination rate for the use case; basis for ongoing monitoring alerts
Training data lineage:Sources, vintage, PII handling, bias pre-screening status (for fine-tuned models)
Third-party risk assessment:Vendor MRM assessment status, SLA for model change notifications, exit provisions
3. Risk Tiering and Prioritization
Not every Gen AI deployment requires the same level of MRM investment. A risk-tiered approach ensures that the most consequential systems receive the most rigorous governance while lower-risk assistive tools can be deployed more efficiently.
4. The Validation Challenge for Generative and Agentic AI Applications
Model validation for traditional models relies on clear logic: compare model outputs against known outcomes, assess statistical performance metrics, and evaluate sensitivity to input assumptions. This methodology breaks down for generative AI systems because there is often no single correct output, ground truth datasets are difficult to construct at scale, and the space of possible inputs is essentially unbounded.
Validation for Gen AI must therefore be redesigned around empirical behavioral testing, adversarial probing, output consistency measurement, and use-case-specific evaluation rubrics, supplemented by traditional performance metrics where applicable.
5. Validation Methodology
Component 1: Conceptual Soundness Review
The validator must assess whether the architectural choices in the Gen AI system are appropriate for the intended use case. This includes:
Is the chosen base model appropriate for the task? (e.g., is a reasoning-optimized model used for multi-step analysis?)
Is the RAGarchitecture, if present, designed to minimize hallucination and maximize retrieval relevance?
Is the system prompt clearly and unambiguously instructing the model to behave in the intended manner?
Are temperatureand other generation parameters calibrated for the required output consistency?
Is the tool access inventory appropriate and minimal (principle of least privilege)?
Component 2: Benchmark and Performance Testing
Where quantitative evaluation is possible, validators should construct benchmark test sets with human-annotated ground truth.
Metrics should be tailored to the specific use case:
Document summarization:ROUGE-L, BERTScore, factual consistency rate, key fact retention rate
Regulatory Q&A:Answer accuracy rate, citation correctness, hallucination rate, refusal rate for out-of-scope queries
Code generation:Functional correctness, security vulnerability rate, style compliance, test pass rate
Agentic task completion:Task completion rate, error rate per step, tool invocation accuracy, hallucinated tool calls
Component 3: Red-Team and Adversarial Testing
Red-teaming is non-negotiable for any AI system deployed in a regulated environment.The red team attempts to elicit harmful, biased, incorrect, or non-compliant outputs through adversarial prompting, edge case construction, and attack simulation. Red-team testing should be conducted:
Pre-deployment:As part of the validation process, before any production release
Post-significant change:After system prompt revisions, model version updates, RAG changes, or tool additions
Periodically:At least annually for high-risk systems; semi-annually or quarterly for highest-risk Tier 3/4 deployments
By independent parties:Internal red teams should be supplemented by external adversarial testing for critical systems
Red-Team Testing Categories for Financial Services Gen AI:
Hallucination elicitation: Prompts designed to induce factual errors about regulations, products, market data, and client information
Prompt injection simulation: Embedding malicious instructions in documents, emails, and data inputs processed by the system
Jailbreak and instruction override: Attempts to bypass system prompt constraints through role-play, hypothetical framing, or adversarial phrasing
Bias and fairness probing: Systematically varying protected characteristics in prompts to test for discriminatory output patterns
Data extraction attempts: Prompts designed to elicit sensitive training data, other users' information, or confidential system information
Agentic abuse scenarios: For agentic systems, scenarios designed to cause the agent to take unauthorized actions, exceed its authority, or misuse tool access
Component 4: Bias and Fairness Evaluation
For AI systems that produce outputs used in regulated decisions, bias evaluation must be systematic, documented, and repeated when the system changes.The evaluation should test for both explicit bias (the model produces different outputs when protected characteristics are mentioned) and implicit bias (the model's outputs correlate with protected characteristics even when not explicitly referenced).
Component 5: Ongoing Monitoring Plan
Validation must produce not just an approval decision but a validated monitoring plan that specifies exactly what will be measured in production, at what frequency, with what alert thresholds, and what actions will be triggered when thresholds are breached. This monitoring plan becomes the operational guide for post-deployment risk management.
6. Governance and Oversight
6.1 Governance Architecture for Generative and Agentic AI Systems
6.2 Hub-and-Spoke AI Governance Model
For large institutions deploying Gen AI across multiple business lines, a hub-and-spoke governance structure balances the need for enterprise-wide consistency with the operational flexibility required at the business line level. The hub provides standards, frameworks, and oversight. The spokes execute within those standards and report to the hub.
6.3 Key AI Governance Policies
The following policies are essential building blocks of an enterprise Gen AI governance framework. Each should be formally documented, approved by appropriate governance bodies, and reviewed at least annually:
Gen AI Use Case Policy:Defines approved and prohibited use cases for Gen AI in the institution; specifies the risk-tiering process for new use cases; identifies use cases requiring board or regulatory approval before deployment
Prompt Change Management Policy:Establishes that system prompt changes are subject to change management controls equivalent to model changes; requires version control, documentation, validation review for material changes, and rollback capability
Third-Party AI Model Policy:Governs the use of externally developed AI models accessed via API or license; requires vendor MRM assessments, contractual provisions for change notification, and exit planning
Agentic AI Deployment Policy:Specifies the additional controls required for agentic AI deployments including, bounded autonomy requirements, tool access approval, kill switch implementation, and human oversight minimums by risk tier
AI Incident Response Policy:Defines AI-specific incident categories (hallucination causing material harm, agentic failure, prompt injection, data leakage); specifies escalation paths, containment procedures, and post-incident review requirements
AI Training Data Policy:Governs data used to train or fine-tune AI models; covers PII handling, data licensing, bias pre-screening, and documentation requirements for training data provenance
7. Bounded Autonomy and Human-in-the-Loop Design
7.1 The Bounded Autonomy Principle
Bounded autonomy is the design philosophy that agentic AI systems should operate within explicitly defined constraints on their scope, authority, tool access, and decision-making. Rather than granting agents open-ended autonomy and attempting to control risk through monitoring alone, bounded autonomy embeds risk controls directly into the agent's architecture.
The bounded autonomy principle does not require minimal AI capability; it requires precisely calibrated AI capability. An agent can be highly capable within its defined boundaries while remaining incapable of causing harm outside them. This is analogous to the principle of least privilege in cybersecurity: grant only the permissions necessary for the intended function, and no more.
7.2 Bounded Autonomy Design Principles
Minimal Footprint:Agents should be granted only the tool access, data permissions, and computational resources strictly necessary for their assigned task. Permissions should be scoped to the minimum required and revoked when no longer needed.
Explicit Action Boundaries:The agent's permitted actions must be explicitly enumerated, not implied. The agent should operate on an allow-list of permitted actions rather than a blocklist of prohibited ones.
Irreversibility Thresholds:Actions that are irreversible or difficult to reverse (financial transactions, external communications, data deletions) require elevated authorization. This can be either explicit human approval or cryptographic authorization tokens.
Temporal Constraints:Agent sessions and task executions should be time-bounded. An agent that has been running for longer than expected should be paused and reviewed rather than allowed to continue indefinitely.
Scope Containment:The agent's operational scope should be technically enforced, not just instructed. System architecture should prevent the agent from accessing data sources, tools, or systems outside its defined scope.
Transparent Reasoning:For high-stakes actions, the agent should be required to explain its reasoning before executing, enabling human reviewers to identify misalignment before consequences occur.
Graceful Degradation:When the agent encounters ambiguity, missing information, or situations outside its defined scope, it should escalate to human review rather than attempting to resolve uncertainty autonomously.
7.3 Human-in-the-Loop Design Patterns
Human-in-the-loop (HITL) controls are not a single design pattern but a spectrum of oversight mechanisms that can be calibrated to the risk level of specific actions or decisions.
The following HITL patterns, from highest to lowest oversight intensity, provide a toolkit for agentic system designers:
7.4 Kill Switches and Circuit Breakers
Every agentic AI system deployed in a financial services environment must have a technically implemented and operationally tested kill switch capability, defined as the ability to immediately halt the system's actions in response to detected risk events.
Kill switch design should address:
Halt mechanisms: At the task level (stop a specific running workflow), the agent level (suspend a specific agent), and the system level (halt all agentic activity across the enterprise)
Automated trigger conditions: Which monitoring metrics or anomaly patterns automatically trigger the kill switch without human intervention?
Human-triggered protocols: Who has authority to engage the kill switch? What is the escalation path during off-hours?
State preservation: When halted, does the system preserve sufficient state to support incident investigation and controlled restart?
Testing requirements: Kill switch functionality must be tested at least quarterly for critical agentic systems; test results must be documented
Recovery procedures: Controlled restart protocols that require explicit human authorization and verification before resuming operation
8. Conclusion
As organizations move from experimentation to enterprise-scale deployment, MRM can no longer remain a static compliance function. It must become a dynamic, architecture-driven discipline embedded directly into how AI systems are designed, deployed, and operated. The shift from deterministic models to probabilistic, autonomous systems requires governance that is continuous, adaptive, and deeply integrated with system behavior. Institutions that evolve their MRM frameworks in this way will mitigate risk while unlocking the full potential of Generative and Agentic AI with confidence, control, and credibility.
Explore more: Start Here · Topics · Archive · About