Every generative AI conversation I have with executives eventually arrives at the same place. We start by talking about models, use cases, agents, and orchestration patterns. We end by talking about data. The teams that are scaling generative and agentic AI in production are not the ones with the most sophisticated models. They are the ones that treated data as a first-class product long before the first prompt was written.
That observation is the reason for this series. Over the coming weeks I will publish a set of articles dedicated entirely to the role of data in generative and agentic AI, with a focus on what it takes to get this right inside a large financial institution. This first article frames the problem and lays out the roadmap.
Why Data Moved Back to Center Stage
For a brief period after ChatGPT arrived, it felt like data engineering had been demoted. Foundation models came pretrained on the open internet, and the promise was that you could simply prompt your way to value. That illusion did not survive contact with the enterprise.
The moment organizations tried to make these models useful for real work; three truths became unavoidable. First, the model does not know your business. It has never seen your policies, your product taxonomy, your client hierarchies, your risk limits, or your general ledger. Second, the model cannot be trusted with facts it was not grounded in. Hallucination is not a bug you patch. It is a structural property you manage with retrieval, context, and controls. Third, the quality of every output is bounded by the quality of the data you feed into the context window. A frontier model reasoning over a stale, duplicated, poorly governed document repository will confidently produce polished nonsense.
In other words, generative AI did not eliminate the data problem. It relocated it. The center of gravity shifted from training pipelines to the data that surrounds the model at inference time: the documents you retrieve, the metadata that describes them, the entitlements that gate them, and the lineage that lets you defend an answer in front of a regulator.
Generative AI Changed What Counts as Data
Traditional data management grew up around structured data. Tables, schemas, keys, and quality rules expressed as constraints. Generative AI forced a dramatic expansion of scope. The most valuable inputs to a large language model are often unstructured: policy documents, procedures, contracts, emails, call transcripts, research, and code. Most institutions have decades of this content, and almost none of it was ever managed with the discipline applied to a regulatory data warehouse.
This creates a new agenda. Chunking strategies, embedding pipelines, vector stores, document freshness, version control for knowledge, and metadata that captures ownership and sensitivity are now core data management concerns. Retrieval augmented generation made the knowledge base a production system. If a policy document is outdated, the model will faithfully retrieve the outdated version and present it with total confidence. Data quality has become answer quality.
Agentic AI Raises the Stakes Again
Generative AI reads your data. Agentic AI reads it, reasons over it, and then acts on it. That distinction changes the risk profile completely.
An agent that reconciles breaks, drafts journal entries, monitors liquidity metrics, or triages client requests is consuming data through tools, producing data through actions, and accumulating data through memory. Every tool call returns data the agent will treat as ground truth. Every action the agent takes writes data back into systems of record. Every memory the agent retains becomes context that shapes future behavior. Data is no longer just the fuel. It is the control surface.
This is where governance becomes inseparable from architecture. In my 5 Pillar Governance Model, at least three pillars are fundamentally data problems. Inventory and Classification requires knowing what data each model and agent touches. Independent Validation requires evaluation datasets, ground truth, and lineage. Use Controls and Guardrails requires entitlement aware retrieval, data loss prevention at the context window, and output checks grounded in authoritative sources. Supervisory expectations reinforce the same point. SR 11-7 always demanded rigor around data used in model development and validation, and the emerging expectations for generative and agentic systems extend that rigor to retrieval corpora, prompts, tool outputs, and agent memory. The EU AI Act and NIST AI RMF land in the same place from different directions: show me your data, its quality, its provenance, and its controls.
The Uncomfortable Truth for Financial Institutions
Here is the pattern I saw repeatedly while building and scaling an AI transformation organization inside a global bank. AI ambitions are set at the level of the firm's best data. AI outcomes are delivered at the level of the firm's average data. The gap between those two levels is where programs stall, budgets evaporate, and pilots fail to reach production.
Ruchir Gupta made this point sharply in a recent LinkedIn post on enterprise AI readiness. In his framing, real AI adoption is the product of five multiplying factors, and the one most organizations underestimate is R, enterprise readiness: data and processes coherent enough that you would trust an AI to reason on top of them. His core observation is that the semantic problem was always there. Organizations never agreed on what their own data means, and as he puts it, "The model just makes that disagreement impossible to ignore." His customer example lands especially hard in financial services: if support, billing, and marketing each hold a different status for the same account, the AI does not know which reality governs the workflow. A human reading a confusing report shrugs and asks a colleague. An AI reasoning over the same contradiction produces a confident, wrong answer and moves on. And as institutions move from AI analysis toward AI mediated execution, that same unresolved contradiction stops producing a wrong report and starts producing a wrong transaction. Gupta's prescription matches my own experience: the fix is not architecture and it is not prompting. It is ownership, with standards set centrally and stewardship at the domain level, funded as an ongoing job rather than a project that ends.
Closing that gap is not glamorous work. It is data ownership, metadata, lineage, quality rules, access management, and lifecycle discipline applied to both structured and unstructured content. But it is the highest leverage investment an institution can make in its AI program, because it compounds. Every improvement in the data foundation raises the ceiling for every current and future use case, generative or agentic, at the same time.
What This Series Will Cover
Each article in this series will take one layer of the data stack and examine it through a generative and agentic AI lens, with practical guidance for financial services. The planned roadmap:
Data readiness for RAG:how unstructured content becomes a governed, production grade knowledge asset.
The retrieval pipeline as a data pipeline:chunking, embeddings, vector stores, and the data quality dimensions that determine retrieval accuracy.
GraphRAG and structured knowledge:using knowledge graphs to give models the entity relationships that flat retrieval misses.
Data for agents:tool outputs, agent memory, and action logs as first class data domains with owners and controls.
Data governance and lineage for AI:what SR 11-7 style rigor looks like when applied to corpora, prompts, context, and evaluation datasets.
Privacy, entitlements, and security:entitlement aware retrieval, sensitive data in the context window, and preventing leakage through outputs.
Measuring quality:evaluation datasets, ground truth curation, and the data behind LLM as judge patterns.
The operating model:who owns AI data, how data product thinking applies, and how to fund the unglamorous foundation work.The Thesis of the Series
If I had to compress the entire series into one sentence, it would be this: in generative and agentic AI, your data strategy is your AI strategy, and everything else is implementation detail. Models will keep improving and commoditizing. Orchestration frameworks will come and go. The durable, defensible asset is a well governed, well understood, well controlled data foundation that any model, present or future, can safely reason over and act upon.
The next article in the series will go deep on data readiness for RAG, including the specific quality dimensions that determine whether retrieval helps your model or quietly poisons it.
If your organization is wrestling with any of these questions, I would welcome the conversation. Follow along, share your experience in the comments, and let me know which topics you want the series to prioritize.
Explore more: Start Here · Topics · Archive · About