AI Vendor Due Diligence: The Reality
Your AI vendor just renewed the contract. They sent over the usual package: a SOC 2 Type II report, an ISO 27001 certificate, a penetration testing summary dated nine months ago. Procurement checked the boxes. Legal confirmed the DPA. The deal closed.
None of those documents tells you what data trained the model you just licensed. None of them describes whether your customers’ inputs are being used to improve future versions. None of them explains what happens when the vendor swaps the underlying foundation model mid-contract.
That gap — between the assurance stack organizations know how to read and the AI-specific risks they are actually absorbing — is what this piece is about.
1. The illusion of vendor assurance
The SOC 2 was designed in a different era, for a different threat model. It tells you whether a vendor has access controls, incident response procedures, and availability commitments. Those are meaningful signals for infrastructure and SaaS procurement. For AI systems, they are necessary but nowhere near sufficient.
The reason is structural: the SOC 2 audit boundary is the vendor’s organizational perimeter. It does not extend to the third parties who built the foundation model the vendor is wrapping. It does not cover the dataset curators who assembled the pre-training corpus. It does not describe the alignment or safety fine-tuning choices made by the model developer upstream.
Most AI products today are wrappers — applications built on top of a foundation model licensed from a separate provider. When your legal team signs a DPA with the application vendor, they are securing commitments about how that vendor handles data. They are not necessarily securing any commitments about the model layer. That layer may involve a different legal entity, a different jurisdiction, and a completely different data-handling regime.
ISO 27001 has the same limitation. It tells you the vendor has an information security management system. It does not tell you what data shaped the model’s behavior, what populations were over- or under-represented in training, or whether outputs carry embedded biases that will surface at scale in your specific use case.
2. Three questions no standard questionnaire asks
The 200-question security spreadsheets that procurement teams send — and that vendors spend weeks completing — were designed for data infrastructure, not for AI systems. They ask about encryption, access logging, patch management, and business continuity. In their default form, they do not ask three questions that have become decisive for AI procurement.
Training-data provenance. Where did the pre-training data come from? Was it scraped from the public web, licensed from publishers, or sourced from specific proprietary datasets? Did any of it include personal information — names, medical records, financial data — processed without a lawful basis? Was children’s data in scope? These questions matter because model behavior is shaped by training data. A model trained on scraped web content carries the biases, factual errors, and demographic distortions of that corpus. A model trained on licensed premium content carries different characteristics. Your organization absorbs those characteristics when you deploy the product — regardless of what the SOC 2 says.
Fine-tuning and evaluation data. Was the model fine-tuned on customer data, including data from the vendor’s prior customers? What benchmarks were used to measure safety, accuracy, and harm — and were they designed for a general audience or for the specific regulatory context your organization operates in? This is where cross-contamination risk lives. If a vendor fine-tuned a shared model on aggregated customer inputs before deploying it to you, proprietary signals from one customer’s data could influence outputs for another. Most questionnaires never ask this directly, and many vendor privacy policies are worded in ways that technically permit the practice while obscuring it.
Downstream model-swap rights. Does your contract require the vendor to notify you when they change the underlying foundation model? Does it give you the right to re-evaluate before the new model goes live in your environment? Does it specify what happens to your data when a model version is deprecated and replaced? The model-swap scenario is more common than procurement teams realize. Foundation model providers update frequently, sometimes with significant behavioral changes. Application vendors absorb those updates — often automatically — and pass them through to enterprise customers. If your risk assessment was performed on Model A and Model B ships six months later with different training data and different alignment tuning, your assessment is no longer valid. Most contracts do not address this.
3. Three composite scenarios
The following are illustrative composites drawn from publicly reported incidents and common patterns in AI procurement. They do not describe any specific organization’s confidential matter.
Scenario A — The invisible subprocessor. A mid-sized financial services firm adopted an AI-powered document review tool. The vendor held SOC 2 Type II certification and signed a comprehensive DPA. What the DPA did not make explicit was that the vendor’s core inference infrastructure ran on a foundation model licensed from a third-party provider whose own data handling was governed by terms of service, not a negotiated agreement. When the firm’s DPO conducted a subprocessor audit, they discovered the foundation model provider functioned as a processor in their jurisdiction but had no contractual relationship with the firm. The vendor’s SOC 2 said nothing about the arrangement.
Get Isabela Rodrigues’s stories in your inbox
Join Medium for free to get updates from this writer.
Scenario B — The silent model update. A healthcare organization deployed an AI summarization tool for clinical documentation. The initial risk assessment was thorough: the team reviewed the model’s known training data sources, evaluated outputs for clinical accuracy, and approved deployment for a specific use case. Eight months later, the vendor silently updated the underlying model. The new version produced more fluent summaries but had different failure modes — it was more likely to omit contraindication warnings in edge cases. The organization discovered this not through a vendor notification, but through a routine clinical audit. Their contract included no model-versioning clause.
Scenario C — The fine-tuning blind spot. A legal technology vendor offered a contract analysis tool fine-tuned on aggregated client contract data — marketed as “trained on thousands of real-world agreements.” A law firm client, reviewing the vendor’s privacy policy before renewal, noticed it permitted the vendor to use “anonymized and aggregated” client data to improve its models. The firm could not determine whether its confidential client contracts had contributed to the training corpus now benefiting competitor firms on the same platform. The vendor’s SOC 2 confirmed the data was encrypted. It confirmed nothing about how it had been used.
In each case, the controller is, in the formal sense, compliant. In the operational sense, the assurance stack has stopped describing what is actually happening.
4. What real AI vendor due diligence requires
The fix is not a longer questionnaire. It is a different kind of question — and an operational shift in how procurement teams treat AI vendors compared to infrastructure vendors.
Map the model supply chain before reviewing certifications. Identify the foundation model provider, any fine-tuning or inference infrastructure, and the contractual relationships between each party. Treat the model layer as a subprocessor regardless of how the vendor classifies it in its own documentation.
Add model-specific clauses instead of relying on standard DPA templates. That means explicit written confirmation that customer data will not be used for training or fine-tuning without affirmative consent; a model-versioning clause requiring advance notice before any change to the underlying model; a re-evaluation right that lets you test the new model before it goes live in production; and a right to audit the vendor’s subprocessor list, specifically including the foundation model provider.
Request a model card alongside — not instead of — the SOC 2. A model card describes intended use cases, known limitations, evaluation results, and training-data characteristics. Not every vendor will have one. A vendor that cannot provide any documentation about the provenance and behavior of the model it is selling you is itself a risk signal.
Build a periodic review cadence into the relationship. AI vendor due diligence is not a point-in-time activity. Tie it to the vendor’s product release cycle and your own regulatory calendar. A re-review should trigger whenever the vendor announces a new model version, whenever a regulatory change affects your use case, and at minimum once a year.
5. Don’t wait for the regulator
There is a version of this conversation that turns into a regulatory demand: wait for the EU AI Act’s transparency requirements, and vendors will have to disclose training data. That version is partially right. High-risk AI systems under the EU AI Act do carry transparency obligations, and general-purpose AI model providers face their own documentation duties. But those regimes cover specific classes of system, they bind the model provider rather than the application vendor you are actually contracting with, and their practical enforcement is still maturing toward the granularity that day-to-day procurement decisions require.
Waiting for regulation to solve vendor due diligence is a risk-management strategy that depends on the regulator moving faster than the market. The current evidence suggests the opposite.
The shift available now is contractual and operational, not regulatory. It requires legal and procurement teams to treat AI vendors as a category distinct from infrastructure vendors — asking different questions at the outset, building different clauses into agreements, and maintaining a different kind of ongoing oversight.
Closing
The SOC 2 was built for the infrastructure layer. The model layer is different — and the organizations that recognize the difference earliest will be the ones that avoid the vendor-risk scenarios already accumulating in the market.
Frameworks describe what the law expects. Implementation describes what the user actually experiences. For AI vendor due diligence, the gap between those two is where the real risk lives.
Isabela works at the intersection of AI governance, data privacy, and regulatory implementation across Canada, Brazil, the USA, and the EU. Views are personal and do not represent any employer.
This is Part III of the Operational AI Governance Series. Part I examined where children’s privacy frameworks break down once AI enters the system. Part II addressed the translation problem — how the same governance framework produces incompatible obligations across jurisdictions, in the same request. Part IV will turn to DPIAs for AI systems: when they are required, what they actually need to say, and why most current templates fall short.
Suggested tags: AI Governance · Vendor Due Diligence · Data Privacy · AI Procurement · Responsible AI
