From Shadow AI to Structured Governance: A Framework for AI Management
While the entire world goes crazy on exploring how AI can help enterprise operations from a technological perspective, has security governance kept up the pace on the compliance/regulatory aspect?
For the past 2 years, I have been asked countless times: How do we manage AI?
When facing any cybersecurity management issue, my consistent approach has always been: Firstly, identify what needs to be managed before really think about how to manage it.
For the following articles, I intend to explore this topic through two pillars: AI information asset management and AI application management policies.I hope my readers can grasp the logic behind cybersecurity management while taking away a security management template for yourselves.
Know the Unknowns: AI Asset Inventory
1. AI Model
AI Model refers to the foundational machine learning or large language models deployed within the organization, including pre-trained models accessed via APIs, fine-tuned models, and self-trained models. Examples include ChatGPT Enterprise, Microsoft Copilot, Google Gemini, open-source models like LLaMA, and custom-trained models for specific business functions such as fraud detection or customer sentiment analysis.
Inventorying AI models marks the sanctioned/authorized modles of an organization. Without proper inventory, organizations cannot assess whether models align with responsible AI principles, regulatory requirements, or appropriate security controls.
2. AI Agent
AI Agents are autonomous AI systems that can make decisions and take actions based on their own outputs without human intervention at each step, often leveraging MCP (Model Context Protocol) servers to connect with external systems and tools. Examples include customer service chatbots that resolve tickets independently, IT helpdesk agents that execute troubleshooting commands, automated document processing workflows, code generation and testing agents, and RPA bots enhanced with AI capabilities.
Organizations must understand the scope of each agent’s authority in order to implement appropriate fail-safe mechanisms.
3. API Keys & Credentials
API Keys & Credentials are authentication tokens and service account credentials that grant access to AI services, cloud platforms, and enterprise systems. Examples include OpenAI API keys, Anthropic Claude API tokens, Azure OpenAI credentials, AWS Bedrock access keys, service principal accounts, and machine identities used for automated AI workflows.
Without tracking these assets, organizations face shadow AI risks where employees use personal API keys to process corporate data through unapproved channels.
4. System Prompt
System Prompts are the foundational instructions and behavioral guidelines that define how AI models interact with users and process requests. Examples include standardized customer service response templates, legal compliance instructions embedded in AI interactions, safety guardrails preventing harmful outputs, role definitions for specialized AI assistants, and prompt templates for document generation or data analysis tasks.
Organizations that fail to manage prompt assets cannot ensure consistent, secure, or Complaint AI behavior across deployments.
5. RAG (Retrieval-Augmented Generation)
RAG (Retrieval-Augmented Generation) is the technique of enhancing AI model responses by retrieving relevant information from external knowledge bases in real-time during inference. Examples include AI assistants that query internal documentation, customer support bots accessing product manuals and FAQs, code assistants referencing internal repositories, compliance bots retrieving current regulations, and Q&A systems pulling from corporate wikis.
Without tracking RAG configurations, organizations lose visibility into what internal data AI systems can access or expose.
6. Vector Database
Vector Databases store high-dimensional vector embeddings that enable semantic search and similarity matching for RAG systems and AI applications. Examples include Pinecone, Weaviate, Qdrant, Chroma, and pgvector storing encoded versions of company documents, product catalogs, customer interactions, technical documentation, and training materials.
Get Yoyo Lin’s stories in your inbox
Join Medium for free to get updates from this writer.
Vector databases actively serve production AI queries and require continuous monitoring, access controls, and data governance to ensure they don’t become vectors for data exfiltration or unauthorized information access.
A Demo
Here is an example of AI information asset inventory.
It demonstrates a scenario of a mid-sized enterprise progressively implementing AI applications in daily operations and customer service during its digital transformation journey.
For Routine Operation, the enterprise adopts Microsoft Copilot to enhance employee document processing efficiency, while the Human Resources and General Affairs (HRGA) department has built an internal document processing AI agent that leverages RAG technology to connect with internal procedure documents on SharePoint, automatically responding to employee policy inquiries.
In the Customer Service domain, the Sales & Marketing (S&M) department has introduced ChatGPT Enterprise and developed a customized customer complaint management chatbot (CMS Chatbot), utilizing MCP Server architecture to integrate customer complaint SOPs and FAQ knowledge bases, providing real-time and consistent customer service responses.
| Business Workflow | Item | Description | Owner | Type | Qty. | PII | Business Sensitive Data |
| ----------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | ----- | -------- | ---- | --- | ----------------------- |
| Routine Operation | AI Model - Microsoft Copilot | Host by MSFT | Infra | Data | 1 | N | Y |
| Routine Operation | AI agent - document processing | RAG - Internal Procedure Docs (sharepoint: Doc/AI)<br>System Prompt - Managed by HRGA | HRGA | Software | 1 | N | Y |
| Routine Operation | API Keys & Credentials - Microsoft Copilot | API Key: None<br>Account Owner: Infra Manager | Infra | Data | N/A | N | Y |
| Routine Operation | Internal Procedure Docs | sharepoint: Doc/ | HRGA | Data | 13 | N | Y |
| Routine Operation | Prompt Libraries | sharepoint: Prompt Lib/<br>Access Control: AD Auth<br>StoringPrompts for whole company | Infra | Data | N/A | N | Y |
| Customer Service | AI Model - ChatGPT Enterprise | Host by OpenAI | S&M | Data | 1 | N | Y |
| Customer Service | API Keys & Credentials - OpenAI | OpenAI API Key: AI agent - CMS Chatbot <br>Account Owner: S&M Manager | S&M | Data | N/A | N | Y |
| Customer Service | AI agent - CMS Chatbot | MCP Server - Docker image run on Web AP<br>RAG - https://example.com/CMD_agent<br>System Prompt - Managed by S&M | S&M | Service | 1 | N | N |
| Customer Service | Customer Compliant SOP and FAQ | sharepoint: S&M/ | S&M | Data | N/A | N | N |
| Customer Service | Vendor - CMS Chatbot | Vendor provides maintainance and consultancy | S&M | Service | 1 | N | Y |Handle the Knowns: AI Application Management
Now that we have established visibility over AI assets through comprehensive inventory, the next critical question emerges: how do we actually govern these assets?
The transition from “knowing what you have” to “managing what you know” demands a structured policy framework that addresses the unique risks and operational realities of AI deployments. Unlike traditional IT assets, AI systems introduce dynamic behaviors, autonomous decision-making capabilities, and complex data dependencies that require purpose-built management approaches.
There are three key management domains that I believe cover for most scenarios.
1. AI Model Management
Management establishes approval and review processes for sanctioned models;
## AI Model Management
1. All AI models deployed within the organization must receive prior approval from authorized management and undergo regular reviews. The approval process shall consider the following factors:
- Vendor reputation or reliability of open-source communities
- Alignment with Responsible AI values and principles, including fairness, transparency, accountability, and ethical considerations
- Extent of testing and validation the model has undergone
- Regulatory compliance status, including applicability under the European Union AI Act
2. Unsanctioned models are strictly prohibited from being used to deliver external client services or process sensitive internal data.2. AI Agent Management
AI Agent Management defines operational boundaries, accountability structures, and risk mitigation requirements for autonomous systems. Key areas include establishing trust boundaries, enforcing access controls and sensitive data protection, and setting limits on agent agency. It also addresses external service applications like AI crisis response and transparency, alongside strict input/output validation to mitigate risks.
## AI Agent Management
### Core Principles
1. Trust Boundaries and Privilege Separation
- Clearly define and enforce trust boundaries between entities (e.g., End User, Agent Host, LLM, MCP Server/Client, Enterprise API Backend), based on the principle of least privilege.
- Implement mandatory isolation and privilege separation at every layer.
2. Access Control
- Manage agent permissions using conditional access policies at both user and group granularity.
3. Sensitive Data Protection
- Prohibit the use or storage of sensitive data (such as PII, trade secrets) in supporting resources (e.g., system prompts, RAG systems, vector databases).
- Enforce isolation and access controls to prevent unauthorized disclosure or tampering.
4. Regular Risk Assessment
- Perform regular third-party assessments for OWASP LLM/Agentic AI Top 10 risks, including: prompt injection, insecure output handling, training data poisoning, denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and model theft.
5. Agency Limits
- AI agents must not make legally binding decisions, decisions with significant financial impact, or decisions directly affecting personnel rights and benefits.
6. Documentation and Accountability
- All agent operations and decisions must be documented and logged.
- Responsible unit must be assigned with ultimate accountability for all agent behaviors and violations.
7. Application Isolation
- Maintain strict separation between internal and public-facing applications, including independent controls, monitoring, and alerting mechanisms.
8. Approved Sources Only
- Only agents, tools, plugins, and models sourced from vetted, reputable, and approved sources shall be utilized. The integration of unverified third-party components into any workflow is strictly prohibited
### External Service Applications
1. AI Crisis Response
- Establish robust organizational mechanisms for AI crisis response, including public statements, apologies, and corrective reporting channels, to enable rapid recovery from reputation-impacting incidents.
2. Transparency and Disclosure
- Clearly disclose in product interfaces, contracts, and documentation the limitations, risks, and sources of bias inherent in AI models.
3. Decision-Making Transparency
- Provide full transparency on AI decision rationale, data sources, and the extent of AI involvement in critical decisions.
### Input/Output Validation
1. Input Validation
- Apply strict sanitization and schema validation to all incoming data, preventing prompt injection, malicious commands, and abnormal content formats.
2. Output Sanitization
- Ensure all agent-generated outputs are automatically sanitized to prevent confidential information leakage to external or low-privileged recipients via workflow.3. AI API Key Management
AI API Key Management implements credential lifecycle controls to prevent shadow AI proliferation. Some of you may have already a Key Management procedure. You may want to merge the two.
## AI API Key Management
1. API key usage shall be documented with records of user identity, project name, and data access scope. Each project or application must use independent API keys, with cross-project sharing strictly prohibited.
2. API keys for trial or testing purposes must have defined expiration periods, not exceeding 14 calendar days by default. Production environment keys must be rotated at least once every 90 days. Immediate rotation may occur upon authorized management approval following specific events such as employee departure or suspected leakage. Key rotation should incorporate automated revocation functionality where available; otherwise, periodic reviews must be conducted to immediately delete unused keys.
3. Keys must be at least 32 characters in length (64 characters recommended) and generated using secure random generators. All API communications must employ encrypted transmission protocols such as HTTPS.
4. Storing API keys in plaintext within source code, configuration files, or version control systems is strictly prohibited. API keys shall be encrypted and stored using enterprise-grade key management systems such as Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault.
5. All API keys must be configured with appropriate usage quotas, including daily/monthly call limits, cost consumption budget caps, IP address whitelist restrictions, and accessible API endpoint scope. API key usage must maintain logs including but not limited to call timestamp, source, IP address, and request content.The journey from AI chaos to AI governance begins not with policies, but with visibility. Start utilizing these template and feel free to reach out!
Beyond Interviews and Spreadsheets: Continuous AI Discovery
Manual inventory through interviews and departmental surveys is a necessary starting point, but it is not sufficient on its own. Shadow AI — employees using personal API keys, unapproved SaaS AI tools, or self-hosted open-source models — by definition escapes self-reported inventories. Organizations need passive, technical discovery mechanisms to detect what people don’t declare.
One of the most effective approaches is API traffic inspection through network-layer monitoring. By deploying inline or out-of-band packet capture on internal network segments, security teams can parse HTTP/S traffic to identify outbound calls to known AI service endpoints (e.g., api.openai.com, api.anthropic.com, generativelanguage.googleapis.com). Correlating these API calls with source IP, user identity, and payload metadata reveals undocumented AI usage patterns and credential exposure risks in real time.
Dedicated AI Security Posture Management (AI-SPM) and API security platforms extend this capability further. Tools such as CloudCoffer MatrixShield, Noname Security (now part of Akamai), and Salt Security specialize in discovering, classifying, and monitoring API traffic across the enterprise. Applied to AI contexts, these platforms can fingerprint AI service API calls, detect anomalous consumption patterns indicative of data exfiltration, enforce policy-based blocking of unsanctioned endpoints, and generate continuous inventory feeds that augment the manual asset register.
The governance posture is therefore not a one-time exercise but a continuous discovery loop: manual interviews establish the baseline, network monitoring catches the gaps, and AI-SPM tooling maintains living visibility as the AI landscape evolves. Together, these layers transform your AI asset inventory from a static spreadsheet into a dynamic, auditable control.
