<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arun Prabhakar on Medium]]></title>
        <description><![CDATA[Stories by Arun Prabhakar on Medium]]></description>
        <link>https://medium.com/@arunp14sec?source=rss-b95d52bf9dde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yFw91_8miL2Z4eIHTosCMg.jpeg</url>
            <title>Stories by Arun Prabhakar on Medium</title>
            <link>https://medium.com/@arunp14sec?source=rss-b95d52bf9dde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 14:58:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arunp14sec/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agent Identity Assurance — A Foundational Element for Trustworthy AI Adoption in Enterprise]]></title>
            <link>https://pub.towardsai.net/agent-identity-assurance-a-foundational-element-for-trustworthy-ai-adoption-in-enterprise-c42a1140e326?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/c42a1140e326</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 16:31:01 GMT</pubDate>
            <atom:updated>2026-10-14T16:31:01.853Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bcr2m6ySssYWJlo6PxLiYw.png" /><figcaption>Image generated by AI</figcaption></figure><p>As AI Agents increasingly act on behalf of users and within enterprise applications and systems, a distinct and governable identity becomes essential. But is knowing which Agent is acting enough? Can we determine who authorized it, what authority it carries, where it is running, and what it can ultimately reach? More importantly, do we have sufficient evidence to trust that identity throughout the Agent’s lifecycle?</p><p>For decades, enterprise identity programs have been built around a relatively predictable set of actors — people, applications, service accounts, and workloads. The underlying principles are familiar: identify the subject, authenticate it, authorize appropriate access, govern its credentials, and retain accountability for its actions. <strong>AI Agents do not replace these identity foundations. What changes is the operating model.</strong> An Agent can plan, invoke tools, access enterprise resources, delegate tasks to other Agents, and cross-trust boundaries with varying degrees of autonomy. This combination is turning identity from a supporting platform capability into an increasingly important security and assurance question for <a href="https://generativeai.pub/responsible-multi-agent-systems-towards-a-trustworthy-ecosystem-cb79c282bdd8">Agentic systems</a>.</p><p>This article builds on the emerging work around AI Agent identity and authorization, while focusing specifically on the assurance question — <strong>how do we determine whether an AI Agent’s identity, authority, access, and actions are sufficiently trustworthy?</strong></p><h4><strong>A quick note on perspective, research, and AI usage:</strong></h4><p>Agent Identity is still an evolving space, and I approach this discussion as a practitioner applying the lens of Security and Risk assurance. The article draws on my research, hands-on experimentation, and iterative exploration of practical enterprise scenarios and assurance workflows, sharing what I am learning rather than presenting a definitive Identity standard.</p><p>AI was leveraged to support written presentation, editorial refinement, language review, and illustrations. The technical interpretations, perspectives, and conclusions reflect my own analysis and judgment.</p><h3><strong>Agent Identity Assurance — Why It matters now?</strong></h3><p>At this point, a natural question is: if the identity community is already working on how AI Agents should be identified, authenticated, authorized, and governed, <strong>why do we need a separate discussion on Agent Identity Assurance?</strong></p><p>The industry response is already taking shape. <a href="https://www.nccoe.nist.gov/projects/software-and-ai-agent-identity-and-authorization">NIST/NCCoE Software and AI Agent Identity and Authorization</a> initiative is exploring how existing identity standards and best practices can be applied to software and AI agents, while the <a href="https://openid.net/wp-content/uploads/2025/10/Identity-Management-for-Agentic-AI.pdf">OpenID Foundation’s work on Identity Management for Agentic AI</a> examines agent identity across authentication, authorization, workload identity, delegated authority, and auditability, while the <a href="https://labs.cloudsecurityalliance.org/agentic/agentic-identity-governance-framework-v1/">CSA Agent Identity Governance Framework</a> brings additional attention to Agent-specific concerns such as delegated authority, ephemeral identities, privilege scoping, monitoring, and decommissioning. <strong>These efforts help establish and govern Agent Identity,</strong> <strong>but they also raise the next question: what evidence would make that identity trustworthy enough to rely upon?</strong></p><p><strong>a)</strong> <strong>Identity Alone Does Not Establish Trust</strong></p><p>Establishing an Agent identity is essential, but from a security and risk-assurance perspective, it is only the beginning of the trust conversation. Consider a few questions:</p><p>i. An Agent may have a valid enterprise identity, but is that identity being exercised by the workload we intended?</p><p>ii. An Agent may have been granted access to a set of tools, but is that access still proportionate to its purpose?</p><p>iii. If an orchestrator delegates a task to another Agent, can we demonstrate that the delegated authority has remained appropriately constrained?</p><p>iv. When an Agent is suspended, revoked, or decommissioned, can we establish that its credentials, delegated authority and downstream access have also been removed?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uv-canUPl5weQfPmEiWCtQ.png" /><figcaption>Agent Identity Establishment vs. Agent Identity Assurance</figcaption></figure><p>The distinction is important, but Agent Identity Establishment and Agent Identity Assurance should not be treated as independent or sequential disciplines. <strong>Assurance requirements should influence identity architecture from the beginning.</strong> If Risk or Security reviewers will eventually require evidence of workload binding, constrained delegation, policy decisions, credential protection, access reviews, or revocation, those requirements are better designed into the system than reconstructed after deployment.</p><p>Put simply, identity tells us who or what the Agent is expected to be; assurance asks whether we have sufficient evidence to trust how that identity is being exercised.</p><p><strong>b)</strong> <strong>What Must We Know About an Agent?</strong></p><p>Before asking whether an Agent Identity can be trusted, we need sufficient context to understand what exactly we are assuring. A name or unique identifier is necessary, but it tells a security reviewer very little about why the Agent exists, whose authority it exercises, where it runs, what it can access, or who remains accountable for it.</p><p>Emerging industry approaches are already moving toward richer representations of Agent identity.</p><p>i. The A2A protocol, for example, uses an <a href="https://a2a-protocol.org/latest/tutorials/python/3-agent-skills-and-card/">Agent Card</a> to describe an Agent’s identity, interfaces, capabilities, skills, and authentication requirements.</p><p>ii. Microsoft Entra Agent ID introduces the concept of <a href="https://learn.microsoft.com/en-us/entra/agent-id/agent-blueprint">Agent identity blueprint</a> that associates Agent identities with sponsors, owners, credentials, permissions, and auditability.</p><p>From a security and risk-assurance perspective, however, I find it useful to bring these identity attributes together with the operating context and supporting evidence required to assess them. I refer to this reviewer-oriented representation as an <strong>Agent Identity Assurance Profile</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fk1fXhjTFcGCVOusX06Slg.png" /><figcaption>Core context required to assess whether an Agent Identity can be trusted</figcaption></figure><p>The <strong>Agent Identity Assurance Profile is not intended as a new identity standard or protocol</strong>. It is an assurance lens, a way for security and risk practitioners to bring together the identity record, application context, runtime relationships, authorization information, lifecycle state, and supporting evidence before making a trust judgment.</p><h3><strong>Seven Trust Dimensions of Agent Identity Assurance</strong></h3><p>Understanding an Agent’s identity, purpose, authority, operating context, and lifecycle gives us the starting point. The next challenge is to determine how trust should be assessed. Rather than treating Agent Identity Assurance as a single control or checkpoint, I found it more useful to examine it through <strong>seven interconnected trust dimensions</strong>, each centered on a specific assurance question.</p><p>The distinction from the Agent Identity Assurance Profile is important. <strong>The profile tells us what we need to understand about the Agent; the trust dimensions tell us what we need to challenge, validate, and evidence before relying on it. </strong>The seven dimensions are:</p><p>i. <strong>Identity and Context Trust:</strong> Do we know what application and Agents are in scope, why they exist, who is accountable for them, and whether the submitted identity context is sufficiently reliable to begin the review?</p><p>ii. <strong>Authority and Delegation Trust:</strong> Who gave the Agent authority to act, how does that authority move between humans and Agents, and does it remain appropriately constrained and attributable at every hop?</p><p>iii. <strong>Effective Access Trust:</strong> What can the Agent ultimately reach through its identity, delegated authority, tools, and resources, and is that access justified, proportionate, enforced, and auditable?</p><p>iv. <strong>Workload and Runtime Trust:</strong> Can we demonstrate that the authorized logical Agent identity is being exercised by the intended and sufficiently trusted runtime workload?</p><p>v. <strong>Credential, Enforcement, and Accountability Trust:</strong> Are credentials adequately protected, are material actions subject to enforceable policy decisions, and can those actions be reconstructed across the relevant human, Agent, workload, tool, and resource?</p><p>vi. <strong>Lifecycle and Governance Trust:</strong> Does governance persist throughout the Agent’s lifecycle, including access review, suspension, revocation, descendant cleanup, credential removal, and decommissioning?</p><p>vii. <strong>Evidence-to-Decision Trust:</strong> Have the positive controls, explicit weaknesses, evidence gaps, required actions, and residual uncertainties been brought together sufficiently for an accountable human reviewer to make a defensible assurance decision?</p><p>Together, these seven dimensions provide the conceptual structure for evaluating Agent Identity trustworthiness. They are intentionally interconnected, since confidence in one dimension can depend on evidence from another. For the next step, we will translate these dimensions into a practical assurance approach that a reviewer can apply.</p><h3><strong>Putting the Seven Trust Dimensions in Context — An Enterprise Multi-Agent Scenario</strong></h3><p>Before moving from the seven trust dimensions into the assurance anchors, it helps to ground them in a common enterprise scenario. Throughout the remainder of this article, I use an <strong>illustrative multi-agent security review scenario, </strong>not as a reference architecture, but as a consistent context for examining how identity, authority, access, workload trust, lifecycle, and evidence interact. The scenario is intended to make the seven trust dimensions introduced above more concrete in an enterprise operating context.</p><p>A human <strong>Security Architect</strong> initiates a review through a <strong>Review Orchestrator</strong>, which coordinates specialist Agents for policy research, control evidence, incident triage, external-vendor advisory, and bounded evidence validation. The scenario includes both persistent and ephemeral Agents, runtime workloads, enterprise tools and sensitive resources, as well as an Agent operating across an external trust boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtbKfD5zPXz_Ci3Zg04DTA.png" /><figcaption>Illustrative enterprise multi-agent scenario showing delegated authority, runtime workloads, trust boundaries, and access to enterprise tools and sensitive resources (Hand-drawn and AI-enhanced)</figcaption></figure><p>What matters for assurance is not simply that these entities exist, but the relationships between them: <strong>who authorized the Agent, what authority can be delegated, what each Agent can ultimately access, which workload is exercising its identity, where trust boundaries are crossed, and what evidence can reconstruct those actions.</strong></p><p>Importantly, the scenario does not assume that every relationship shown in the illustration is trustworthy. Some relationships may be strongly evidenced, others may only be asserted, and some may expose weaknesses or missing evidence that require further review. This provides a common context for applying the <strong>seven assurance anchors</strong> that follow.</p><h3><strong>Applying the Model: The Seven Assurance Anchors</strong></h3><p>This is where the article moves from conceptual framing into practical assurance review. Before proceeding, it is useful to keep three related ideas distinct:</p><p><strong>·</strong> <strong>Agent Identity Assurance Profile</strong> — Establishes the context we need to understand about an Agent across six areas: identity and accountability, purpose and autonomy, authority and access, runtime and trust context, lifecycle, and evidence and traceability.</p><p>· <strong>Seven Trust Dimensions</strong> — Define the aspects of that context whose trustworthiness needs to be evaluated.</p><p>· <strong>Seven Assurance Anchors</strong> — Operationalize those dimensions by examining specific assurance questions, supporting evidence, controls, gaps, and required outcomes.</p><p>Building on that context, each anchor follows a similar rhythm:</p><p><strong>i) assurance question → ii) evidence examined → iii) trustworthiness interpretation → iv) remaining uncertainty → v) what must be demonstrated.</strong></p><p><strong>Anchor 1 — Identity Context: Do We Know What We Are Trusting?</strong></p><p>Anchor 1 operationalizes Identity Context Trust. It does not establish the Agent Identity; it determines whether the identity and surrounding system context are sufficiently understood and evidenced to begin assurance.</p><p><strong>i) Assurance Question</strong></p><p>What application, Agents, workloads, tools, resources, and trust boundaries are in scope and is the available identity evidence reliable enough to begin the review?</p><p><strong>ii) Evidence Examined</strong></p><p>Establish the context map: participating Agents, accountable owners, purpose and persistence, associated workloads, intended tools and resources, and internal or external trust boundaries. Evidence should also be distinguished as asserted, derived, or missing rather than treated uniformly.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>The evidence should allow the reviewer to understand who and what is being assessed, who is accountable, and where the material trust relationships exist. A recorded entity or relationship establishes context; it does not, by itself, demonstrate that the corresponding control is effective.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>Incomplete inventory, unclear ownership, stale or unsupported relationships, missing workload context, or insufficient evidence around external trust boundaries should remain visible as assurance gaps are not converted into assumptions.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>The review scope is sufficiently complete, accountable relationships are identifiable, material entities and trust boundaries are visible, and the supporting evidence is reliable enough to proceed with deeper assurance.</p><p>With the identity context established, the next assurance question is: who is authorized to act for whom?</p><p><strong>Anchor 2 — Authority &amp; Delegation: Who Is Allowed to Act for Whom?</strong></p><p>Anchor 2 operationalizes Authority &amp; Delegation Trust. It examines whether authority originates from an accountable principal, remains appropriately constrained as it moves between Agents, and can be traced through the complete delegation chain.</p><p><strong>i) Assurance Question</strong></p><p>Can we demonstrate who authorized each Agent to act, what authority was delegated at every hop, and whether downstream authority remains within the limits of the delegating principal?</p><p><strong>ii) Evidence Examined</strong></p><p>Review the human-to-orchestrator, orchestrator-to-specialist, and parent-to-child delegation relationships together with the underlying grants, scopes, approvals, expiry or delegation-depth limits, and allow/deny decisions.</p><p>The delegation hierarchy should be interpreted separately from workload binding and effective resource access; each represents a different assurance relationship.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>A reconstructable authority graph shows how delegation is intended to flow, but the graph alone does not prove that every grant is approved or enforced. Confidence increases when each delegation is attributable, appropriately scoped, and no downstream Agent can receive authority beyond the delegator’s permitted ceiling.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>Missing approval, scope-attenuation, expiry, delegation-depth, or runtime-enforcement evidence should remain visible as assurance gaps. In the illustrative external-vendor path, a scope-widening request is recorded as denied, but the reviewer must still verify that the denial is technically enforced.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>Every material delegation can be traced to an accountable originator, downstream authority does not exceed upstream authority, applicable constraints are evidenced, invalid scope widening is prevented, and the resulting authorization decisions are auditable.</p><p>With delegated authority understood, the next assurance question becomes: what can that authority ultimately reach?</p><p><strong>Anchor 3 — Effective Access: What Can the Agent Ultimately Reach?</strong></p><p>Anchor 3 operationalizes Effective Access Trust. It moves beyond assigned or delegated authority to determine what an Agent can ultimately reach through its scopes, tools, and downstream resources and whether that reach remains appropriate to its intended purpose.</p><p><strong>i) Assurance Question</strong></p><p>What can each Agent ultimately access or perform through its identity, delegated authority, scopes, tools, and resources and is that effective access justified, constrained, and enforceable?</p><p><strong>ii) Evidence Examined</strong></p><p>Trace the Agent → scope → tool → resource paths and compare intended access with the available grants, policies, tool permissions, resource controls, and authorization decisions. Distinguish evidenced effective access from candidate access paths where authorization or enforcement evidence remains incomplete.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>A modeled access path shows what an Agent may be able to reach; it does not by itself prove that access is authorized or correctly enforced. Assurance comes from understanding the Agent’s aggregate reach and establishing that sensitive or high-impact paths remain proportionate to its purpose.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>Missing authorization decisions, tool-level enforcement, resource restrictions, or evidence supporting sensitive and cross-boundary access should remain visible as assurance gaps. Credential or runtime weaknesses may increase the significance of an access path, but are examined more directly in subsequent anchors.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>Material access paths are attributable to the appropriate Agent, justified by its purpose, constrained to the required tools and resources, and supported by sufficient authorization and enforcement evidence to understand the resulting exposure.</p><p>With effective reach understood, the next assurance question is whether the intended workload is actually exercising the Agent Identity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nv26gwFbxEzBrGEEniv5EA.png" /><figcaption>Assurance requires connecting authority, execution, access and evidence and not collapsing them into single identity claim</figcaption></figure><p>The flow deliberately separates delegated authority, workload execution, policy enforcement, effective access, and supporting evidence. It’s important to understand</p><p>· <strong>Policy Enforcement Point (PEP)</strong> intercepts and enforces an access request and</p><p>· <strong>Policy Decision Point (PDP) </strong>evaluates the applicable policy and context and returns the authorization decision.</p><p>This distinction matters for assurance: evidence that a policy exists or a decision was made is not the same as evidence that the decision was enforced on the resulting Agent action.</p><p><strong>Anchor 4 — Workload Trust: Is the Intended Workload Exercising the Agent Identity?</strong></p><p>Anchor 4 operationalizes Workload Trust. It examines whether the logical Agent Identity being reviewed can be reliably connected to the runtime workload actually exercising that identity.</p><p><strong>i) Assurance Question</strong></p><p>Can we demonstrate that the intended and authorized workload is the one exercising the Agent Identity?</p><p><strong>ii) Evidence Examined</strong></p><p>Review the Agent-to-workload binding, workload identity, runtime or environment claims, attestation evidence, trust domain, and the credentials or identity assertions used to establish that relationship. Where appropriate, this may include workload-identity mechanisms such as <a href="https://spiffe.io/docs/latest/spiffe-about/overview/">SPIFFE</a>/<a href="https://spiffe.io/docs/latest/spire-about/spire-concepts/">SPIRE</a> or equivalent enterprise or federated evidence.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>Knowing the logical Agent and knowing that a workload exists are not sufficient on their own. Confidence increases when there is verifiable evidence connecting that Agent Identity to the expected runtime and trust context.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>Where workload binding or attestation evidence is missing, the reviewer cannot reliably establish which runtime is exercising the identity. This remains an assurance gap rather than evidence that the Agent or workload is compromised. For externally operated Agents, equivalent federation or third-party attestation may provide the required assurance without direct visibility into the vendor’s runtime environment.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>The logical Agent Identity is verifiably bound to an intended and trusted workload, and the evidence is sufficient to establish the runtime actor and applicable trust domain.</p><p>With the acting workload established, the next assurance question is whether its credentials, policy decisions, and resulting actions remain protected and accountable.</p><p><strong>Anchor 5 — Credential, Policy &amp; Accountability: Are Agent Actions Protected, Enforced, and Traceable?</strong></p><p>Anchor 5 operationalizes Credential, Policy &amp; Accountability Trust. It examines whether the credentials supporting an Agent Identity are adequately protected, whether material actions are subject to enforceable policy decisions, and whether those actions can be reconstructed and attributed.</p><p><strong>i) Assurance Question</strong></p><p>Are the Agent’s credentials appropriately protected, are material actions evaluated and enforced by policy, and can the resulting activity be reliably traced?</p><p><strong>ii) Evidence Examined</strong></p><p>Review credential type, storage, exposure, scope, rotation and revocation evidence; authorization requests and policy decisions; enforcement records; and audit evidence connecting the relevant human, Agent, workload, tool, and resource.</p><p>In this flow, the Policy Enforcement Point (PEP) enforces the access request, while the Policy Decision Point (PDP) evaluates the applicable policy and context and returns the authorization decision.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>A credential record does not demonstrate that the credential is securely handled, and a policy decision does not prove that it was enforced. Confidence increases when credential protections, policy decisions, enforcement outcomes, and resulting Agent actions can be connected through consistent evidence.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>In the illustrative evidence, possible model-context exposure of an external-vendor shared credential is explicitly reported as a weakness; whether equivalent exposure exists in production and whether compensating controls are effective would still require independent validation. Missing evidence of brokering, vaulting, rotation, revocation, enforcement, or downstream action should remain visible as separate assurance gaps.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>Credentials are appropriately protected and scoped, material actions are supported by evidenced authorization and enforcement decisions, and the resulting activity can be reconstructed across the relevant identity, workload, tool, resource, and accountable principal.</p><p>With individual actions protected and attributable, the next assurance question is whether those controls remain effective throughout the Agent’s lifecycle.</p><p><strong>Anchor 6 — Lifecycle Governance: Does Trust Persist Throughout the Agent Lifecycle?</strong></p><p>Anchor 6 operationalizes Lifecycle Governance Trust. It examines whether ownership, access, credentials, and delegated relationships remain governed as an Agent changes state from creation and operation through suspension, revocation, and eventual decommissioning.</p><p><strong>i) Assurance Question</strong></p><p>Can we demonstrate that the Agent remains appropriately owned, reviewed, and governed throughout its lifecycle and that access and authority are removed when they are no longer required?</p><p><strong>ii) Evidence Examined</strong></p><p>Review accountable ownership, lifecycle state, access-review history, expiry or time-to-live controls, suspension and revocation records, credential removal, downstream access cleanup, and decommissioning evidence. Persistent and ephemeral Agents should be evaluated according to their intended lifecycle characteristics.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>An Agent being registered or active does not demonstrate that its lifecycle remains governed. Confidence increases when ownership remains current, access is periodically reassessed, ephemeral identities expire as intended, and lifecycle events propagate to credentials, delegated authority, and dependent access.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>Missing ownership or lifecycle evidence should remain an assurance gap rather than be interpreted as proof that no owner or control exists. Similarly, an unevidenced revocation or decommissioning stage does not establish that downstream credentials, delegated authority, or child Agents were actually removed.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>The Agent has accountable lifecycle ownership, continued access is periodically justified, temporary identities expire appropriately, and suspension, revocation, or decommissioning results in demonstrable removal of associated credentials, authority, and downstream access.</p><p>With lifecycle governance established, the final assurance question is whether the accumulated evidence is sufficient for an accountable human to make a defensible trust decision</p><p><strong>Anchor 7 — Human Assurance &amp; Decision: Is the Evidence Sufficient for a Defensible Decision?</strong></p><p>Anchor 7 operationalizes Human Assurance &amp; Decision Trust. It brings together the evidence developed across the preceding anchors so that an accountable reviewer can make an informed assurance judgment.</p><p><strong>i) Assurance Question</strong></p><p>Do the available controls, weaknesses, evidence gaps, required actions, and residual uncertainties provide a sufficient basis for an accountable human to make a defensible assurance decision?</p><p><strong>ii) Evidence Examined</strong></p><p>Review the accumulated positive controls, explicit weaknesses, missing or uncertain evidence, remediation actions, conditions, ownership, and residual uncertainty across the complete assurance review.</p><p><strong>iii) Trustworthiness Interpretation</strong></p><p>The assurance posture helps organize and prioritize what the evidence indicates; it should not be treated as the assurance decision itself. The reviewer must interpret the evidence in its operating context and determine whether the remaining uncertainty is acceptable.</p><p><strong>iv) Remaining Uncertainty</strong></p><p>Unresolved evidence gaps, unvalidated control effectiveness, outstanding actions, or residual risks should remain visible at the point of decision. Machine-generated findings or counts should not be interpreted as security scores, maturity ratings, or substitutes for reviewer judgment.</p><p><strong>v) What Must Be Demonstrated</strong></p><p>The reviewer can distinguish what has been evidenced from what remains uncertain, understand the material conditions and residual risks, identify accountable actions where required, and record a traceable human assurance decision.</p><p>The objective is not to eliminate uncertainty, but to make the basis for trust and the uncertainty that remains explicit and accountable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FgTfAy0LU81JXR5BB_4uBg.png" /><figcaption>Evidence informs the assurance posture, but the final trust decision remains an accountable human judgment.</figcaption></figure><h3><strong>Toward Assured Enterprise Trust</strong></h3><p>As AI Agents take on more enterprise actions, <strong>identity is necessary, but trust must be evidenced</strong>. Knowing which Agent is acting is only the starting point. Practitioners also need to understand who authorized it, what authority it carries, which workload is exercising that identity, what it can ultimately reach, and whether its actions remain governed and traceable.</p><p>The model explored in this article connects three complementary layers: the <strong>Agent Identity Assurance Profile</strong> establishes what we need to understand; the <strong>Seven Trust Dimensions</strong> identify what needs to be validated; and the <strong>Seven Assurance Anchors</strong> provide a practical way to examine the supporting evidence, weaknesses, gaps, and uncertainty. It complements and not replaces the identity and authorization foundations being advanced through <a href="https://www.nccoe.nist.gov/projects/software-and-ai-agent-identity-and-authorization">NIST/NCCoE</a>, <a href="https://openid.net/new-whitepaper-tackles-ai-agent-identity-challenges/">OpenID Foundation</a>, and <a href="https://openid.net/new-whitepaper-tackles-ai-agent-identity-challenges/">CSA</a>.</p><p><strong>Actionable next step for practitioners</strong>: build assurance requirements into Agent architectures early. Establish accountable ownership, preserve delegation and policy decisions, connect Agent Identity to its runtime workload, make effective access and trust boundaries visible, retain lifecycle and audit evidence, and keep the ultimate assurance decision accountable to a human reviewer.</p><p>The goal is not to manufacture certainty or automate certification. It is to make what is known, what is not yet evidenced, and what requires action explicit enough to support a defensible enterprise decision.</p><p>As Agent Identity standards continue to evolve, <strong>Agent Identity Assurance can provide the connecting discipline between identity mechanisms and Trustworthy AI adoption at enterprise scale.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c42a1140e326" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/agent-identity-assurance-a-foundational-element-for-trustworthy-ai-adoption-in-enterprise-c42a1140e326">Agent Identity Assurance — A Foundational Element for Trustworthy AI Adoption in Enterprise</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge Graphs for Operationalizing Information Risk — From Siloed Artifacts to Connected…]]></title>
            <link>https://generativeai.pub/knowledge-graphs-for-operationalizing-information-risk-from-siloed-artifacts-to-connected-8d7f24d4d2de?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d7f24d4d2de</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Tue, 21 Jul 2026 00:41:35 GMT</pubDate>
            <atom:updated>2026-07-21T00:52:11.337Z</atom:updated>
            <content:encoded><![CDATA[<h3>Knowledge Graphs for Operationalizing Information Risk — From Siloed Artifacts to Connected Assurance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GvbGukID6ekg0iejh41Bsg.png" /><figcaption>Image generated by AI</figcaption></figure><blockquote>“Learning gives creativity. Creativity leads to thinking. Thinking provides knowledge. Knowledge makes you great.” — Widely attributed to Dr. A.P.J. Abdul Kalam, a celebrated scientist and former President of India.</blockquote><p>I had the opportunity to see Dr. Kalam in person during my university years, and this reflection on learning and knowledge has stayed with me ever since. It also captures the journey behind this applied research initiative: learning from established security practices, questioning how they work together, and exploring whether the knowledge produced by one activity can create greater value across the information risk lifecycle.</p><p>Information risk practices continuously generate valuable knowledge through risk registers, threat models, control assessments, compliance mappings, privacy reviews, vulnerability findings and evidence requests. Yet these outputs are often created and consumed within individual activities, teams or systems. Their value may remain confined to the artifact for which they were originally produced.</p><p><a href="https://owasp.org/www-community/Threat_Modeling">Threat modeling</a> provides a useful example of this challenge. I have been involved in threat modeling for many years, and I have often observed that the final report is treated as a reference document rather than an actionable input to downstream security validation. <strong>The challenge, therefore, is not necessarily a lack of security information. It is the fragmentation of security knowledge.</strong> This work explores whether knowledge graphs can transform threat-modeling outputs into connected, reusable assurance knowledge for penetration testing and potentially for other information risk activities.</p><h3>Why Connected Knowledge?</h3><p>Information risk artifacts already contain valuable knowledge. A threat model may identify threats, affected components, data flows, weaknesses, expected controls, assumptions, and evidence requirements. A risk register may capture business impact and ownership, while a control assessment or vulnerability report may provide additional validation. The problem is that these insights are often distributed across separate documents, tools, and teams.</p><p>The relationships among them usually remain implicit. A practitioner may need to manually determine which weakness contributes to an attack scenario, which components and data flows are affected, which controls are expected to mitigate the risk, and what evidence is required to validate those controls.</p><p><a href="https://neo4j.com/blog/knowledge-graph/what-is-knowledge-graph/">Knowledge graphs</a> make these relationships explicit.</p><p><strong>Threat → Attack Scenario → Component and Data Flow → Weakness → Control Gap → Evidence Requirement</strong></p><p>This connected representation does not create new security knowledge by itself. Instead, it preserves context and makes existing knowledge easier to trace, query, and reuse. The same information can then be viewed differently depending on the practitioner’s objective.</p><h3>Who Benefits from Connected Knowledge?</h3><p><strong>a) Threat modelers</strong> can trace identified threats to affected components, data flows, weaknesses, controls, and unresolved evidence requirements.</p><p><strong>b) Penetration testers</strong> can use the same context to define focused test scenarios, prioritize attack paths, and validate whether expected controls are implemented effectively.</p><p><strong>c) Assurance practitioners</strong> can connect risks and control gaps to evidence requirements, findings, and applicable standards.</p><p><strong>d) Risk leaders</strong> can identify recurring weaknesses, highly affected components, and scenarios requiring priority review or validation.</p><p>The value, therefore, lies not in creating another security artifact, but in allowing different practitioners to navigate and reuse the same assurance knowledge according to the decisions they need to make.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*xirrbPJE3dw2XXgXePZoxA.png" /><figcaption>From siloed information risk artifacts to connected assurance knowledge and practitioner action</figcaption></figure><h3>From Assurance Artifact to Knowledge Graph</h3><p>The implementation began with a multi-agent threat-modeling capability built around fictional reference architecture. It analyzed application components, data flows, identities, assets, trust assumptions, and security context to identify candidate threats and control gaps.</p><p>The output was not limited to a conventional report. It also produced a structured, machine-readable assurance artifact containing information such as:</p><ul><li>threats and attack scenarios;</li><li>affected components and data flows;</li><li>weaknesses and expected controls;</li><li>assumptions and open questions;</li><li>control gaps and evidence requirements;</li><li>review actions and relevant standards mappings.</li></ul><p>This distinction was important. <strong>A human-readable report explains the analysis, while a structured artifact makes it possible for another system to interpret, connect, and reuse the underlying knowledge</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*JcostKdpA1V3O3a4YZONlQ.png" /><figcaption>Representative excerpt from the structured assurance artifact for Threat T3</figcaption></figure><h4>a) From Structure to Meaning</h4><p>Structured data alone does not make a knowledge graph. The application still needs to understand what each entity represents and how it relates to the others.</p><p>To support this, I applied three related <a href="https://www.semanticarts.com/semantic-ontology-the-basics/">concepts</a>:</p><ul><li><strong>Semantics</strong> defined what terms, properties, and relationships meant.</li><li><strong>Classification taxonomy and controlled vocabularies</strong> organized threats, weaknesses, controls, evidence, and components into <a href="https://neo4j.com/blog/knowledge-graph/taxonomy-vs-ontology-vs-knowledge-graph">consistent categories</a>.</li><li><strong>A conceptual domain ontology</strong> defined how these concepts could relate.</li></ul><p>For example, a threat may affect a component, contribute to an attack scenario, involve a weakness, expose a control gap, require evidence, or map to relevant security guidance. The purpose was practical rather than academic. It ensured that the same concept was interpreted consistently and that relationships normally buried inside a report became explicit.</p><p>The ontology was implemented as a conceptual property-graph model using labels, properties, and relationship types. It was not a complete formal ontology with machine-enforced cardinality, domain and range constraints, or <a href="https://www.w3.org/TR/owl2-primer/">OWL/SHACL</a> validation. Some deterministic validation and stable-identifier rules were applied, while stronger formal enforcement remains an area for future work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*mc151vHZ3WN_PzkJgCrM8A.png" /><figcaption>The property-graph model represented assurance concepts as nodes, their connections as typed relationships, and contextual information as properties</figcaption></figure><p>Some examples of the meaning of relationships include:</p><p>· AFFECTS: a threat affects a component.</p><p>· USES_FLOW: an attack scenario uses a data flow.</p><p>· INVOLVES_WEAKNESS: a scenario involves a weakness.</p><p>· NEEDS_EVIDENCE: a control gap requires evidence.</p><h4>b) Constructing the Graph through Agentic Engineering</h4><p>I built a standalone knowledge-graph application using Agentic engineering principles. The objective was not simply to draw connections between report entries. The application performed a sequence of structured tasks:</p><ol><li>Parse the assurance artifact.</li><li>Normalize entity names and formats.</li><li>Assign stable identifiers.</li><li>Apply deterministic semantic-enrichment rules.</li><li>Create nodes, properties, and relationships.</li><li>Generate <a href="https://neo4j.com/docs/cypher-manual/current/introduction/">Neo4j Cypher queries</a>.</li><li>Produce reviewer-oriented summaries and reusable query patterns.</li></ol><p><a href="https://www.anthropic.com/engineering/building-effective-agents">Agentic engineering</a> made this architectural prototype faster and more practical to implement. It helped orchestrate multi-step processing, structured state management, semantic enrichment, and query generation. It did not make knowledge graphs possible; rather, it reduced the effort required to prototype the workflows surrounding them.</p><p>The application also provided different views of the same assurance state:</p><ul><li>a security-insight overview;</li><li>a graph explorer;</li><li>attack-scenario coverage;</li><li>Cypher and CSV exports;</li><li>and a library of curated, read-only queries for practitioner use.</li></ul><p>These views were intended to help practitioners move from a high-level assurance posture to the underlying nodes, relationships, assumptions, and evidence requirements.</p><h4>c) Exploring the Graph in Neo4j Aura</h4><p>The generated Cypher was manually executed in Neo4j Aura, where the resulting graph could be explored visually and through targeted queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*k_dqBJPCdKNgvki4JzhTxw.png" /><figcaption>Neo4j Aura visualization of the assurance knowledge graph, showing connected threats, components, controls, weaknesses, data flows, and review actions</figcaption></figure><p>A whole-graph visualization can appear complex at first glance, but its value is not in viewing every node simultaneously. The value comes from asking focused questions of the graph:</p><ul><li>Which threats affect a particular component?</li><li>Which attack scenarios depend on the same data flow?</li><li>Which scenarios involve control gaps or missing evidence?</li><li>Which components are affected by the greatest number of threats?</li><li>Which high-priority scenarios should be considered for security validation?</li></ul><p>The prototype generated curated, read-only Cypher patterns for questions such as these.</p><pre>MATCH (t:Threat)-[:AFFECTS]-&gt;(c:Component)<br>RETURN c.name AS component,<br>       count(DISTINCT t) AS threat_count<br>ORDER BY threat_count DESC<br>LIMIT 10;</pre><p>The implementation flow can therefore be summarized as:</p><p><strong>Structured assurance artifact → normalization → semantic enrichment → conceptual graph model → Cypher generation → exploration in Neo4j Aura</strong></p><p>Once this assurance information was represented as connected knowledge, the next question was whether the graph could help translate a specific design-time threat hypothesis into a more actionable penetration-testing scope.</p><h3>From Threat Hypothesis to Penetration-Test Scope</h3><p>To examine whether the knowledge graph could support downstream security validation, I selected one threat from the synthetic threat model:</p><p><strong>Threat T3: Excessive access to CRM capabilities</strong></p><p>The threat was connected to a more specific attack scenario:</p><p><strong>Attack Scenario AS3: A lower-privileged user, agent, or tool context invokes CRM actions through the tool-execution path.</strong></p><p>This distinction is important. A threat describes the broader security concern, while an attack scenario provides a more concrete hypothesis that can be validated. In this case, the concern was not simply “CRM access risk.” It was a <strong>tool-invocation privilege risk</strong>. The Tool Registry and Tool Execution Service could potentially enable access to CRM functions beyond the intended authorization boundaries. The knowledge graph preserved the path connecting this hypothesis to the application architecture:</p><p><strong>Agent Orchestrator → Tool Execution Service → CRM API</strong></p><p>It also connected the scenario to:</p><ul><li>the relevant data flows;</li><li>the weakness <strong>Excessive Tool Privilege</strong>;</li><li>an identified control gap;</li><li>assumed controls such as authentication, authorization, least privilege, access review, and audit logging;</li><li>an evidence requirement to confirm whether those controls were implemented;</li><li>and an inferred mapping to applicable security guidance.</li></ul><h4>a) Moving Beyond Generic Testing</h4><p>Without this context, a penetration tester may begin with broad CRM API testing. The graph provided a more focused starting point: validate the authorization and privilege boundaries across the tool-invocation path.</p><p>This translated the original threat into practical testing questions:</p><p>· Can a lower-privileged user/agent/tool context invoke CRM-capable tools?</p><p>· Are authorization checks enforced before Tool Execution Service calls the CRM API?</p><p>· Are least-privilege scopes applied separately to each tool and action?</p><p>· Are denied invocation attempts logged and monitored?</p><p>· Is there sufficient evidence that access to CRM-capable tools is periodically reviewed?</p><p>The knowledge graph did not automatically perform penetration testing or determine whether the threat was exploitable. Instead, it helped <strong>transform a design-time hypothesis into a traceable test scope containing the affected components, flows, weakness, expected controls, and required evidence.</strong></p><p>This establishes a clearer relationship between the two activities:</p><ol><li>Threat modeling identifies what could go wrong.</li><li>Penetration testing gathers evidence to validate whether it can occur in the implemented system.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*52lTGw4d42fEjAtiC4DmPg.png" /><figcaption>Threat T3 connected to the application components included within the proposed penetration-test scope.</figcaption></figure><h4>b) Prioritizing Validation Across the Graph</h4><p>The same graph queries could also examine multiple attack scenarios collectively. For example, practitioners could identify:</p><ul><li>high-priority scenarios with unresolved control gaps;</li><li>components affected by several threats;</li><li>commonly reused data flows;</li><li>scenarios with missing or unconfirmed evidence;</li><li>and weaknesses recurring across different parts of architecture.</li></ul><p>This broader view is important because penetration-test planning is rarely based on one threat alone. <strong>The graph can help practitioners move from individual findings toward a risk-informed view of where validation effort may be most valuable</strong>.</p><p>The prototype demonstrated the ability to create this traceability and prioritization context. Whether it reduces planning time, improves test coverage, or decreases clarification effort would need to be evaluated through practitioner-led pilots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*9AE9nLJziabA_ufjgycZAQ.png" /><figcaption>Connected assurance path translating Threat T3 into an actionable security-validation context</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*UX4kL4Y3coZ4OgVq1g6bwQ.png" /><figcaption>Knowledge-graph-derived handoff containing the context required to develop a focused pentest plan</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MPiWibWk1z-fAHjXVDFj3w.png" /><figcaption>High-priority penetration-test candidates identified through scenario priority, control gaps, affected flows, and validation status.</figcaption></figure><p>Together, these views demonstrate how connected assurance knowledge can support both focused threat validation and cross-scenario prioritization.</p><h3>What the Knowledge Graph Revealed</h3><p>The architectural prototype showed that the value of a knowledge graph does not come from visualization alone. Its value lies in preserving the relationships that are often implicit or difficult to navigate within conventional assurance artifacts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*V9nsFSQA49TjsKWuCMSN5g.png" /><figcaption>How the knowledge graph made assurance knowledge more contextual, queryable, and actionable</figcaption></figure><p>The graph also demonstrated that prioritization need not depend on severity alone. A scenario becomes more meaningful when its priority is considered alongside affected components, relevant flows, weaknesses, control gaps, and evidence status. These relationships provide a richer basis for deciding where security-validation effort may be most valuable.</p><p><strong>The architectural prototype therefore demonstrated several capabilities: threat-to-test traceability, path-based scoping, evidence identification, and cross-scenario prioritization.</strong> Potential benefits such as faster test planning, broader coverage, and reduced clarification effort would still require validation through practitioner-led pilots.</p><p>The most important insight was that the knowledge graph did not create another standalone security artifact. It created a reusable assurance layer through which existing security knowledge could be explored, connected, and applied to different information risk decisions.</p><h3>Looking Ahead: From Connected Knowledge to Context Graphs</h3><p>This technical exploration demonstrated that the value of a knowledge graph lies not merely in visualizing security information, but in preserving how assurance knowledge flows across activities. Threat modeling identified a hypothesis; the graph connected it to components, flows, weaknesses, controls, and evidence; and penetration testing received a more contextual validation scope. In that sense, the work also provided a form of <strong>value-stream mapping for security knowledge</strong>, showing where information is created, how it is transferred, and where context may otherwise be lost.</p><p>This was intentionally learning-oriented. A production implementation would require stronger entity resolution, provenance, versioning, graph-quality validation, formal constraints, validated standards mappings, and automated but controlled integration with graph infrastructure. Its operational benefits would also need to be evaluated through practitioner pilots rather than assumed.</p><p>A valuable next step would be to close the feedback loop:</p><p><strong>Threat hypothesis → validation activity → evidence → security decision → updated assurance knowledge</strong></p><p>The model could then expand beyond threat modeling to include risk registers, control assessments, vulnerability findings, privacy reviews, compliance mappings, and remediation outcomes.</p><p>Over time, this could <strong>evolve from a knowledge graph into a </strong><a href="https://neo4j.com/blog/developer/from-recall-to-reasoning-how-context-graphs-upgrade-an-agents-brain/"><strong>context graph</strong></a>, one that preserves not only connected security concepts, but also when evidence was collected, why decisions were made, which assumptions changed, and how risks evolved. Such context could provide more relevant grounding for future agentic workflows while keeping practitioners responsible for validation and decision-making.</p><p>The broader learning is that information-risk teams may already possess substantial knowledge. The opportunity is to connect that knowledge, so it remains traceable, reusable, and valuable across the assurance lifecycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*tBi4CanquU7fY4w9.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*R7SnLji-2Po_9Cx8.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d7f24d4d2de" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/knowledge-graphs-for-operationalizing-information-risk-from-siloed-artifacts-to-connected-8d7f24d4d2de">Knowledge Graphs for Operationalizing Information Risk — From Siloed Artifacts to Connected…</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Responsible Multi-Agent Systems — Towards a Trustworthy Ecosystem]]></title>
            <link>https://generativeai.pub/responsible-multi-agent-systems-towards-a-trustworthy-ecosystem-cb79c282bdd8?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb79c282bdd8</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[responsible-ai]]></category>
            <category><![CDATA[human-in-the-loop]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Wed, 13 Aug 2025 15:05:54 GMT</pubDate>
            <atom:updated>2025-08-13T15:05:54.363Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Responsible Multi-Agent Systems — Towards a Trustworthy Ecosystem</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*yAbxUl5zL7lCkIlHV_-SjA.png" /><figcaption>Image generated by AI</figcaption></figure><p>Intelligent Agents are highly effective at solving well-defined tasks individually. However, their full potential is realized when they operate collectively within a Multi-Agent System (MAS). In my <a href="https://medium.datadriveninvestor.com/multi-agent-systems-in-action-the-architecture-behind-intelligence-e474807fd8a9">previous article</a> I have discussed in detail the <strong>Memory and intelligence in Agentic-AI along with the Architectural foundations of MAS</strong>. Through collaboration, Agents can address broader, more complex goals that are beyond the reach of any single agent. While collaboration is a cornerstone of MAS, understanding the full spectrum of Agent attributes is essential to designing trustworthy, robust multi-agent ecosystems.</p><h3><strong>Attributes of Intelligent Agents</strong></h3><p>The section outlines the key attributes of Intelligent Agents to operate effectively in a Multi-Agent scenario. Based on academic research and real-time implementations, the following are the fundamental and essential attributes.</p><p>1. <strong>Autonomy: </strong>An Intelligent Agent operates with a high degree of independence, controlling its own actions and internal state without requiring constant human intervention. However, when it comes to completing system goals, the <strong>Orchestrator Agent</strong> distributes the task to be handled by individual Agents, and the decision-making is done by them.</p><p>2. <strong>Communication:</strong> Agents communicate with other Agents using protocols in a Multi-Agent scenario for a variety of reasons that include exchanging information, coordinating actions, negotiating goals/tasks, etc. This Inter-Agent communication also facilitates interoperability among heterogeneous agents.</p><p>3. <strong>Collaboration &amp; Competition:</strong> Although these are two different attributes, many real-time systems involve a mix of both. The collaboration is mainly around how Agents can cooperate to achieve shared goals. Agents could also compete for access to tools or compute resources especially in decentralized or loosely governed systems.</p><p>4. <strong>Behavior Dynamics: </strong>An Agent’s behavior dynamics refer to how it acts over time in response to the environment and in pursuit of its goals. This could be reactive or proactive based on any patterns or observations regarding the Agents’ actions as they interact with other Agents. It is important to mention that complex emergent behaviors could arise from these combined reactive and proactive qualities in certain scenarios.</p><p>5. <strong>Adaptability:</strong> Agents in a MAS are configured to learn about the environment to improve their performance and update their strategies. The <strong>cognitive capability of Agents</strong> plays an important role including how they observe and interact for future decisions. This attribute ensures how Agents operate effectively in dynamic or uncertain environments.</p><p>6. <strong>Goal-orientation:</strong> Intelligent agents are typically goal-driven as they are designed to achieve specific objectives or fulfill certain tasks. This attribute also helps us understand the roles and responsibilities of the individual Agents and how they are tracked along with their contribution to overall system objectives.</p><p>7. <strong>Agent interoperability:</strong> This attribute deals with the design measures considered when Agents from one MAS interacts with Agents from another — This includes the communication protocols, access controls, data sharing and resource consumption and so on.</p><p>8. <strong>Agent capabilities:</strong> Agents could be designed with certain capabilities that set them apart from other Agents within the MAS. There are Orchestrator Agents, Compliance Agents or Agents with Administrative capabilities — This many include deploying new Agents, rotate secrets or even manage workflow definitions, etc.</p><p>9. <strong>Learning capability:</strong> Learning is closely tied to the Adaptability of Agent but it would be good to get answers to a few elements of the Memory including what gets stored in the Memory? does the Agent have long-term memory capability? does the Agent have Context awareness, is the Agent configured with experienced-based adaptation, etc.</p><p>10. <strong>Scalability: </strong>This attribute is specific to the overall MAS as it allows new Agents with specific roles to join or leave without disrupting the whole MAS. Agents register using directory services so others can discover them dynamically to interact with them. Scalability is also closely tied to adaptability as the environment or requirements change, Agents or their configurations can be updated independently.</p><p>The Internal AI capabilities are foundational, they govern how the Agent thinks, acts, and evolves over time. In contrast, the external attributes become critical when Agents operate in Multi-Agent environments, enabling ecosystem-level trust and coordination.</p><p>The attributes mentioned above form the core characteristics of Agents, most prominently observed in Multi-Agent scenarios. In addition to these, <strong>there are other attributes that are more domain-specific or context-dependent</strong>. These include aspects such as agent heterogeneity, hierarchy, rationality, etc. As we have seen, the profile of an Intelligent Agent in a real MAS is thus a combination of Agent’s</p><p>· Internal AI capabilities — autonomy, reasoning, learning</p><p>· External framework within which it operates — communication standards, assigned roles, and oversight mechanisms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*alaBYUPCOoX1XBIpdxVCgg.png" /><figcaption>Agent attributes function within a MAS (Image by the author)</figcaption></figure><p>To capture these perspectives at a glance, the figure above serves as a helpful visual for understanding how the <strong>Agent attributes function within a MAS</strong>. On the top of the figure, we will see the attributes that are specific to Agent’s Internal Capabilities, and on the bottom are the attributes that enable the Agent to participate in a Multi-Agent interaction. The Agent attributes are driven by the various components on the Intelligent Agent that is represented in the dotted box</p><h3><strong>Aligning Agent Attributes with Responsible AI Principles</strong></h3><p>A Multi-Agent System is comprised of many Intelligent Agents with varying complexities and developed for performing specialized tasks. As these Agents coordinate to complete the assigned goal, it is essential that they operate effectively and also responsibly. What does this mean? — As Agents are powered by the Deep Learning algorithms and AI Models in their backend, it is important that these <strong>Agents are designed with Responsible AI principles to ensure safe outcomes that are aligned with human values.</strong></p><p>To get a better view of this, we will quickly understand <a href="https://www.ibm.com/think/topics/responsible-ai">Responsible AI (RAI)</a> and the principles that define RAI. As defined by IBM, Responsible Artificial Intelligence (AI) is a set of principles that help guide the design, development, deployment and use of AI, building trust in AI solutions that have the potential to empower organizations and their stakeholders. Commonly adopted RAI principles in industry include <strong>Fairness, Safety, Transparency, Robustness, Explainability, Reliability,</strong> <strong>Inclusiveness,</strong> and <strong>Accountability</strong>. Together, these principles help ensure that AI systems act in ways that uphold ethical and operational standards.</p><p>Given that intelligent agents express their behaviors through the attributes we’ve already discussed, embedding RAI principles into their design is critical. This involves mapping each RAI principle to different stages of the agent’s lifecycle and its core attributes. Below is a mapping of key RAI principles to specific Agent attributes, illustrating how these <strong>RAI principles can be operationalized when architecting Intelligent Agents and deploying them in MAS</strong>.</p><h4><strong>Fairness</strong></h4><p>· <strong>Architecture Patterns:</strong> Agents competing for resources in MAS should introduce <a href="https://arxiv.org/pdf/2502.07254">fairness constraints</a> so that resources are equitably distributed.</p><p>· <strong>Agent Attributes:</strong> Collaboration, Orchestration</p><h4><strong>Safety</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents must operate safely without causing unintended harm and be robust against failures, using guardrails</p><p>· <strong>Agent Attributes:</strong> Communication, Scalability</p><h4><strong>Transparency</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents making decisions in the MAS should be <a href="https://arxiv.org/pdf/2311.13148">transparent</a> including the thinking process involved</p><p>· <strong>Agent Attributes:</strong> Behavior Dynamics</p><h4><strong>Accountability</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents executing tasks or collaborating to accomplish common goals should be logged and continuously monitored.</p><p>· <strong>Agent Attributes:</strong> Autonomy, Collaboration</p><h4><strong>Privacy</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents be designed with <a href="https://gdpr-info.eu/issues/privacy-by-design/">Privacy by design</a> measures. Memory components should be privacy-protected, storing only required info.</p><p>· <strong>Agent Attributes:</strong> Learning Capability</p><h4><strong>Robustness</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents must be resilient to adversarial attacks and fully recover to resume normal operations.</p><p>· <strong>Agent Attributes:</strong> Adaptability, Scalability</p><h4><strong>Explainability</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents should explain their task completions (and <a href="https://ai.meta.com/research/cicero/">reasoning involved in Multi-Agent</a> conversations) in a way humans understand.</p><p>· <strong>Agent Attributes:</strong> Goal Orientation, Agent Capabilities</p><h4><strong>Inclusiveness</strong></h4><p>· <strong>Architecture Patterns: </strong>Agents must uphold service quality and policy protection when working with under-represented Agents outside their MAS</p><p>· <strong>Agent Attributes:</strong> Orchestrator, Adaptability</p><p>This is an illustrative example of how RAI principles can be integrated into the architecture of intelligent agents by configuring the appropriate Agent attributes. Many of these implementation strategies are already being adopted by leading technology firms.</p><p>For instance, <a href="https://www.microsoft.com/en-us/ai/responsible-ai">Microsoft</a> mandates <strong>fairness testing during AI deployment to ensure equitable treatment and prevent bias</strong>. From a transparency standpoint, Agents are increasingly deployed with mechanisms such as<strong> transparency logs and traceable decision records to enable auditability</strong>. As organizations across various industry verticals continue to adopt Intelligent Agents, it becomes increasingly important that RAI principles are systematically embedded. We can expect to see more domain-specific implementations and innovative applications of these principles as adoption grows.</p><h3><strong>The Importance of Human oversight in MAS</strong></h3><p>While we have explored the key principles of Responsible AI, one crucial dimension that warrants special attention is <strong>human oversight</strong>, particularly when deploying and interacting with these intelligent AI systems. While there has been significant work in this space, one research paper that profoundly influenced my thinking is the <a href="https://arxiv.org/pdf/2109.13916">Unsolved Problems in ML Safety</a>. The paper underscores the need to <strong>align AI system objectives with human values</strong>. We will apply those concepts to the world of Intelligent Agents and MAS:</p><p>i) In any ecosystem of intelligent agents, a core design challenge is balancing <strong>Agent autonomy with human oversight</strong>. As defined by Anthropic — <a href="https://www.anthropic.com/news/our-framework-for-developing-safe-and-trustworthy-agents">Principles for Trustworthy Agents</a>; Agents must be able to work autonomously and their independent operation is exactly what makes them valuable. But humans should retain control over how their goals are pursued, particularly before high-stakes decisions are made.</p><p>ii) Additionally, <strong>Agents exhibit</strong> <strong>emergent and unpredictable behavior</strong> when collaborating in a Multi-Agent scenario. These behavioral patterns call for setting up <a href="https://www.salesforce.com/blog/responsibly-manage-multi-agent-systems/">Governance for Multi-Agent System</a> and augmenting that with mechanisms to oversee the Agent’s interactions and collective behavior. The concept of <strong>Human-in-the-Loop</strong> is not only a foundational mandate from RAI but is also a strategic value-add enabling agents to learn from human feedback and refine their actions across iterative cycles..</p><p>iii) Researchers have proposed advanced techniques that involve <strong>human-centered design approach with active dynamic moderation</strong>. This research is an enhancement to the traditional human oversight and instead dynamically intervenes to drive <a href="https://arxiv.org/pdf/2502.01714">Multi-Agent Systems toward safe outcomes</a>.</p><p>Meaningful human oversight is indispensable for establishing trust in Multi-Agent Systems. To adhere to ethical standards, autonomous Intelligent Agents must be subject to human review particularly because humans possess the contextual understanding needed to make informed decisions, especially in high-risk scenarios. In practice, <strong>Responsible AI teams are embedding human oversight at multiple stages of Agent lifecycle including techniques like Red-teaming and Agent monitoring</strong>. By doing so, they reinforce the key RAI principle — Accountability and Safety, ensuring that autonomy within Multi-Agent Systems never implies the absence of human control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*2OCjgW_FGEJHyl0O.png" /><figcaption>BCG’s RAI approach — Published in Medium</figcaption></figure><p>I want to recall a powerful idea shared by the <strong>Responsible AI experts at BCG</strong>, well before the advent of GenAI and the Agentic applications that we are seeing today. The figure (seen above) emphasizes the importance of <a href="https://medium.com/bcggamma/responsible-ai-leading-by-example-c25a8a0a98ea">keeping humans at the center of AI</a>, empowering them and preserving the authority and well-being of those who develop, deploy, and use these systems. This <strong>principle of human-centered AI serves as the connective thread across all other RAI principles</strong>. As we now promote Agentic interactions within Multi-Agent Systems, I believe this foundational idea remains just as relevant and will continue to guide AI practitioners in building trustworthy ecosystems.</p><p>So far, we have seen the key attributes of the Intelligent Agents and how they contribute to the Multi-Agent scenario including these their alignment with the RAI principles. We have also seen why human oversight is essential in improving the trustworthiness of the entire ecosystem. While we have covered the foundational RAI principles and their application in MAS, some of these critical domains require deeper exploration. To gain greater confidence in the trustworthiness of Multi-Agent Systems, it’s essential to examine the <strong>architectural considerations around Security and Privacy</strong>. In my next article, I will dive into these areas sharing insights from my experience working with MAS, including requirements implementation, emerging threats, and defense strategies to ensure that Multi-Agent ecosystems are not only intelligent and collaborative, but also <strong>reliable and resilient</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ZzfY8_3v2IAqcnti.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*XQC8c0rzWDA_Wl2B.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb79c282bdd8" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/responsible-multi-agent-systems-towards-a-trustworthy-ecosystem-cb79c282bdd8">Responsible Multi-Agent Systems — Towards a Trustworthy Ecosystem</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Agent Systems in Action: The Architecture behind Intelligence]]></title>
            <link>https://medium.datadriveninvestor.com/multi-agent-systems-in-action-the-architecture-behind-intelligence-e474807fd8a9?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/e474807fd8a9</guid>
            <category><![CDATA[responsible-ai]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-security]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[intelligent-agent]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 01:23:03 GMT</pubDate>
            <atom:updated>2025-08-15T08:38:38.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*FZGr-p51NjUY-tJhJhdkXQ.png" /><figcaption>Image generated by AI</figcaption></figure><p>Issac Asimov, the great science fiction author presented the <a href="https://en.wikipedia.org/wiki/Three_Laws_of_Robotics">Three Laws of Robotics</a> in his fictional stories on the ethical behavior of robots. These laws have also influenced thought on the ethics of artificial intelligence. <strong>One key principle in AI ethics is Interpretability</strong>. In the context of AI Agents, interpretability means making their inner workings and decision-making processes transparent so that users can understand and trust them and intervene before any harmful behavior occurs.</p><p>Many of you are already exploring AI Agents in your work or researching their impact across industries. I share that fascination: what excites me most is the intelligence behind these Agents and how they’re designed and engineered. That curiosity led me to delve deeper into Intelligent-Agent architectures and, critically, to explore how multiple such Agents collaborate within a Multi‑Agent System (MAS). Before we examine MAS architectures and recent advancements, it’s essential to recognize that an <strong>AI Agent’s intelligence is the core enabler of its autonomy and the foundation for the coordination and effectiveness we see in these systems</strong>. Let’s begin by analyzing the Agents themselves and the nature of their intelligence.</p><h3><strong>Intelligence in Agents</strong></h3><p>An AI Agent’s intelligence comes from its cognitive capabilities and the components in its architecture to leverage these capabilities. This is a <a href="https://www.ibm.com/think/topics/components-of-ai-agents">great article from IBM</a> that helps us understand the essential components involved in building the Agent’s intelligence architecture — The Agents components involve the use of:</p><p><strong>1.</strong> <strong>Pretrained Language Models</strong></p><p>To enable natural language understanding and reasoning. This helps Agents perceive and interpret unstructured text environments.</p><p><strong>2.</strong> <strong>Reinforcement Learning Algorithms</strong></p><p>Enable Agents to learn from interaction and feedback, optimizing behavior over time. This is essentially valuable for goal-oriented tasks.</p><p><strong>3.</strong> <strong>Knowledge Graphs</strong></p><p>Provide a structured, symbolic memory for facts, relationships, and contextual reasoning and helps in explaining logic-based tasks.</p><p><strong>4.</strong> <strong>Vector Databases</strong></p><p>Support retrieval-augmented generation and context injection by storing and fetching semantically relevant embeddings (i.e. unstructured memory).</p><p><strong>5.</strong> <strong>Rule Engines &amp; Chain-of-Thought Reasoning</strong></p><p>Allow for explicit reasoning, conditional logic, and step-wise problem-solving, combining symbolic and neural reasoning.</p><p><strong>6.</strong> <strong>Planning Module / Goal Management</strong></p><p>Agents need a planner or goal manager to break down tasks, prioritize subgoals, and choose when to reason, retrieve, or act. This is implemented via a ReAct loop, Tree-of-Thoughts, or custom policy/planner Agent.</p><p>This diagram below provides a high-level view of the components that enable Agent intelligence. Their usage varies according to each Agent’s purpose and objectives. Active research in industry continues to advance Agent architectures, and these cognitive capabilities will only evolve further over time.</p><h3><strong>Inside the Intelligent Agents</strong></h3><p>Architecture focuses on building the right structure for the task, but Agents only acquire and develop intelligence once learning is enabled. This process involves selecting the appropriate algorithms, engineering domain knowledge, implementing continuous learning methods, and managing updates, among other steps. Agents learn by performing tasks and gaining knowledge through experience. I have created a data‑flow diagram that illustrates how these components interact and the sequence of actions involved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8hEQMPFnGs2VVWQIYMUp_A.png" /><figcaption>High level flow of Intelligence development in Intelligent Agents</figcaption></figure><p>I’m sure my sketch helps clarify the flow. However, I’d like to add a few points. Once the input is interpreted by the language model, the next step is <strong>Agent tasking </strong>where tasks are prioritized and assigned to specific agents. This determines each Agent’s subsequent actions. After an Agent decides on its course, it combines outputs based on its reasoning and executes the task.</p><p>As these Agents are used in the context of <strong>Multi-Agent collaboration</strong>, it is essential that they adapt to other Agent’s strategy. Additionally, the feedback that these Agents get could also improve future actions and task executions. This continuous learning from the user and its environment is made possible with the concept called <strong>Meta learning</strong>, that helps with Agent adaptability and tool usage.</p><p>So far, we have only looked at the intelligence aspects of AI Agents, but there are other <a href="https://aws.amazon.com/what-is/ai-agents/">components in the architecture in these Intelligent Agents</a>, the technical challenges involved at the time of deployment, and importantly the different types of AI Agents. I encourage you to learn about these before we dive into how intelligent agents work together in a Multi‑Agent System (MAS) and examine that architecture next.</p><h3><strong>Architecting Multi-Agent Systems</strong></h3><p>One way to think of a Multi‑Agent System is like a project team: each professional brings specialized skills to a particular task, but they must all collaborate and coordinate toward the shared goal of delivering the project. Similarly, in a Multi‑Agent System, intelligent agents interact and work together to achieve a common objective.</p><p><strong>Architecting a Multi-Agent System (MAS) is not just about deploying a group of Intelligent Agents but it’s about defining how these agents are built, coordinated, and deployed in real-world environments.</strong> A robust architecture is essential to govern the lifecycle of these Agents, orchestrate collaboration, ensure interoperability, and maintain alignment across the system as tasks and environments evolve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69V3LG6JChaA5qJsVtslSg.png" /><figcaption>High-level view of Multi-Agent Systems Architecture</figcaption></figure><p>Now let’s discuss the architectural pattern evident in modern MAS implementations. As seen in the figure above, there are three architectural planes:</p><p><strong>1. Design Plane — Defining the Blueprint</strong></p><p>The Design Plane captures the static, pre-runtime configuration of Agents. At this stage, practitioners use an <strong>Agent Builder</strong> to define roles, capabilities, tool access, and memory requirements. These specifications are encapsulated into an <strong>Agent Descriptor</strong>; a declarative artifact used to generate and register the agent at runtime.</p><p><strong>2. Control Plane — Orchestrating the System</strong></p><p>Once defined, Agents are loaded into the <strong>Agent Registry</strong>, which is a catalog of available Agents and their versions. The <strong>Agent Orchestrator</strong> loads the configurations of the Agents, registers the Agent with the Agent Registry. It is responsible for interpreting workflows, assigning tasks, and routing communications. Each Agent is mapped and dispatched dynamically depending on task relevance.</p><p><strong>3. Runtime Plane — Autonomous Execution and Collaboration</strong></p><p>Once dispatched, the Agents are instantiated as services. Each <strong>Agent Service</strong> interacts with external APIs, shared <strong>Memory</strong>, and other Agents (via <strong>A2A</strong> or Agent-to-Agent protocols). These runtime agents are capable of decision-making, context retention, and coordination through a defined interface, such as a <strong>Model Context Protocol (MCP)</strong>.</p><p>As mentioned in the section that discusses the architecture for Intelligent Agents, there are many patterns and advancements observed in the architecture of MAS implementations, largely from the research carried out in the Industry. Let’s see a few of them in the following section.</p><h3><strong>Advancements in MAS Architectures</strong></h3><p><a href="https://github.com/openai/swarm">The OpenAI Swarm framework</a> defines multiple specialist Agents. These are modular components with explicit handoff mechanisms for <strong>passing control and information with other Agents</strong>. By sharing context or partial results, these agents can chain their capabilities fluidly, making coordination lightweight, highly controllable, and easily testable. Swarm has since evolved into the production‑ready <a href="https://github.com/openai/openai-agents-python">OpenAI Agents SDK</a>, which builds on and replaces the original framework.</p><p>Anthropic has adopted a Multi‑Agent architecture in Claude’s “Research” mode, using an <strong>orchestrator–worker pattern</strong>. In this setup, a top‑level Lead Agent devises the overall solution and then spawns multiple Subagents to perform searches or subtasks in parallel. One particularly innovative feature is the <strong>Persistent Agent Memory</strong>: before launching its Subagents, the Lead agent records its high‑level plan in a memory buffer, ensuring that context is never lost even if token limits are reached. This memory system supports long‑horizon reasoning and is <a href="https://www.anthropic.com/engineering/built-multi-agent-research-system">critical for autonomous coordination</a>.</p><p>There have been several other exciting advances in MAS architecture. Google DeepMind has demonstrated the <a href="https://deepmind.google/discover/blog/emergent-bartering-behaviour-in-multi-agent-reinforcement-learning/">capabilities of decentralized multi-agent systems</a>, Meta AI is innovating with multi‑agent reinforcement learning, and academics are investing heavily in this area as well. One research focus is on M<strong>emory‑augmented Multi‑Agent Systems</strong>, that <a href="https://arxiv.org/pdf/2507.07957">defines different memory types</a> alongside a multi‑agent controller that manages real‑time memory updates and retrievals. Researchers have proposed architecture like <strong>Shared Recurrent Memory Transformers (SRMT)</strong> that enables agents to <a href="https://arxiv.org/pdf/2501.13200">share part of their internal state</a> with each other during decision-making.</p><p>In addition to memory, researchers are innovating around controllers and coordination mechanisms, adaptive interfaces for tool use, meta‑learning capabilities, hierarchical structures for agent communication, and more.</p><p>There are already impressive real‑world implementations of Multi‑Agent Systems, the software industry has embraced this technology and is putting business cases into action. While exploring those examples is beyond this article’s scope, it’s clear that Multi‑Agent Systems are on the cusp of becoming mainstream. <strong>The critical question now is whether our ecosystem can keep pace: </strong>Can we scale for performance, maintain rigorous QA, and extend all the security and compliance practices we have relied on for traditional software? As Agents gain autonomy and make decisions on our behalf, embedding robust risk and compliance controls will be essential. In my next article, I’ll dive into exactly how to integrate those controls and ensure these systems remain both powerful and trustworthy.</p><p>Your Business — On AutoPilot with <em>DDImedia AI Assistant</em><br>(<a href="https://waitlist.ddimedia.ai/join-the-waitlist-aie">Join Our Waitlist</a>)</p><p>Visit us at <a href="https://www.datadriveninvestor.com/"><em>DataDrivenInvestor.com</em></a></p><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/"><em>here</em></a>.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e474807fd8a9" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/multi-agent-systems-in-action-the-architecture-behind-intelligence-e474807fd8a9">Multi-Agent Systems in Action: The Architecture behind Intelligence</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing AI Systems — Defensive Strategies]]></title>
            <link>https://medium.datadriveninvestor.com/securing-ai-systems-defensive-strategies-6b1fd6ad33fa?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b1fd6ad33fa</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Tue, 06 Jun 2023 20:37:12 GMT</pubDate>
            <atom:updated>2023-06-07T14:57:17.981Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Securing AI Systems — Defensive Strategies</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NjDkbNpC5HF38pgFOE_pZA.jpeg" /></figure><p>In the first paper on “Securing AI Systems — <a href="https://medium.datadriveninvestor.com/securing-ai-systems-risks-in-ai-based-solutions-dc3077bdf6b3?source=friends_link&amp;sk=23ef0e895fba0c8c60f86238bab884c4">Risks in AI based Solutions</a>” we got an overview of the <strong>Risk landscape</strong> in the solutions built using AI technologies. The risks were analyzed from both the security and safety aspects of AI usage. Also, the impact arising from regulatory violations were studied including the importance of getting aligned with AI principles. We took a deeper dive into the <strong>Adversarial Machine Learning</strong> (AML) Attacks in the second paper (Securing AI Systems — <a href="https://medium.datadriveninvestor.com/securing-ai-systems-researching-on-security-threats-3c14834d71ec?source=friends_link&amp;sk=34e855732b1cb62d37b0541d64b2df57">Researching on Security Threats</a>) by emphasizing on the importance of doing research in enumerating the security threats and ways to proactively discover them in the Model internals. Although we discussed about the role of research in the Adversarial AI space, it is implied that performing continuous research with other segments of risks (eg: AI Safety or AI Principles) are helpful to understand the advancements happening there. We will continue the same rhythm of researching as we deal with the defensive strategies that are applied to <strong>safeguard AI solutions</strong>.</p><h4><strong>Analyzing Security Incidents</strong></h4><p>Strong processes are required to effectively implement the defensive controls for the security risks identified. This means using a combination of process engineering tools to identify risks and successfully mitigate them. One of the ways to identify these risks is by using the concept of FMEA (<strong>Failure Mode and Effect Analysis</strong>). FMEA is a process analysis tool to determine the components that are failing in the system, including their causes and effects. This in turn helps mitigate the risks by applying the right defenses. Applying this FMEA principle also helps in analyzing the security incidents in AI solutions to understand their failure modes.</p><p>Apparently, Microsoft and Harvard University have worked on the <a href="https://arxiv.org/ftp/arxiv/papers/1911/1911.11034.pdf">Failure Modes in Machine Learning</a>. Their research focusses on how AI based solutions fail, whether by the hand of an adversary or due to the inherent design of a system. Based on their study, are two types of failures:</p><ol><li><strong>Intentional failures</strong> where the failure is caused by an active adversary attempting to subvert the system to attain their goals — either to misclassify the result, infer private training data, or to steal the underlying algorithm.</li><li><strong>Unintentional failures</strong> where the failure is because an ML system produces a formally correct but completely unsafe outcome.</li></ol><p>As you could notice from their work, many of the adversarial attacks that we have briefly discussed earlier including the perturbation attack, model poisoning, model stealing as well exploiting software dependencies are a result of the attacker’s ability to misalign the model’s objectives. All these falls under the Intentional failure type. It is highly recommend going through their document as it details the different attributes of Intentional Failures including the attack scenario, examples, the compromise impacting the core tenets of security, the attacker’s methods and numerous other research papers that are being referred.</p><p>Interestingly, as we dig deeper on the Failure modes in AI, a research firm named <a href="https://www.bnh.ai/">BNH.AI</a> that specializes in the privacy, fairness, security, and transparency of AI and data has completed a thorough analysis of AI incidents. They have surveyed about <strong>169 publicly reported AI incidents</strong> and the inferences are that, 95% of these incidents were unintentional failures which includes algorithmic discrimination, lack of transparency and accountability, privacy violations and others. This means only about <strong>5% of AI incidents were caused by adversarial attacks</strong> or the Intentional failure type. The figure below gives a snapshot of these issues and is taken from the blog post on <a href="https://www.quantcast.com/blog/bias-in-ai-understand-the-implications-for-marketing-and-advertising/">Bias in AI</a> where the study made by BNH.AI has been summarized including the impact and implications of these AI incidents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4w_HG80BboYJnNNwHxsMg.png" /></figure><h4><strong>Scope of Defensive Controls</strong></h4><p>As we discuss about the defensive strategies, we will keep our focus only on the <strong>risks emerging out of these Intentional failure </strong>categories and the defenses to be planned to prevent them. This is because</p><ol><li>As practitioners building AI solutions, it is important to <strong>know our enemies</strong> and the weaknesses existing in our system that could be exploited by these adversaries. Hence it is essential to build solutions with the right countermeasures.</li><li>Secondly, the variations in the attack patterns, innovative methods involved in crafting <strong>adversarial examples</strong> pose a huge challenge and we need to constantly defend the boundaries from potential threats caused by malicious actors.</li><li>Importantly, it is required that Solution providers should do their <strong>due diligence</strong> and be <strong>continuously compliant</strong> with international laws and regulations by being resilient to security attacks on the target AI systems.</li></ol><p>Having said all these, as part of the concluding section, we will also brief the best practices that needs to be performed including the industry initiatives that are ongoing to prevent against Unintentional failures in AI systems. So, let’s look at some of the control mechanisms that help build the integrity in our AI models.</p><h4><strong>Adversarial Robustness</strong></h4><p>Adversarial Robustness<strong> </strong>is all about building the machine learning models with security, privacy and regulatory principles that will ensure the <strong>integrity of the ML models</strong> and the <strong>resiliency of complete AI solutions</strong>. So, essentially it deals with the implementation of countermeasures against all the Adversarial Machine Learning (AML) attacks.</p><p>So, what are those countermeasures? “A picture is worth a thousand words” — The figure below portrays the taxonomy of the adversarial defenses in machine learning models. Based on the paper published in <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=9153891">IEEE Reviews</a>, the team of researchers have categorized Adversarial Robustness into three classes (as seen in the figure).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7_0TMzQIbkHrTu8vf6vWw.png" /></figure><h4><strong>AML Defenses — Implementation Experience</strong></h4><p>As much as we are interested in learning these remediation measures, we should also understand the practical challenges faced by data scientists in applying these remedies and the evaluation techniques for applying the right defenses. To begin with, we will briefly discuss the various taxonomies of countermeasures and how they are being approached.</p><p><strong>A) Approaching Countermeasures</strong></p><p>In the figure above, we observe three classes of defense methods that are designed to either <strong>modify the model/data </strong>on its parameters and its features or defense methods that focus on having <strong>additional models </strong>integrated to secure the mainstream models. This is one perspective of how the defense controls are discussed but as we leverage advanced algorithms like <strong>Deep Reinforcement Learning</strong> (DRL) that are being used by a lot of AI systems to achieve high performance, we need to think about adversarial defenses that are specific to attacks against the DRL models. The work published by <a href="https://arxiv.org/pdf/2001.09684.pdf">Journal of IEEE Transactions on Artificial Intelligence</a> has both the Adversarial attacks and taxonomy of defenses for DRL models (figure below) discussed in detail, that is worth looking at.</p><p>Finally, in addition to implementing the security controls to the models, going by the principles of Adversarial Robustness, we need to invest in <strong>Privacy Preserving</strong> techniques as well. These are nicely explained in the paper on <a href="https://ieeexplore.ieee.org/document/9294026">Privacy and Security Issues in Deep Learning: A Survey</a>. The techniques include Differential Privacy, Homomorphic Encryption, Secure Multi-party Computation and Trusted Execution Environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3CN6mEEgS0N9z2Dnf0d9w.png" /></figure><p><strong>B) Challenges and Lessons Learnt</strong></p><p>It is great to see the numerous approaches published for adversarial defenses focusing on the realms of security, privacy, and advanced algorithms. However, not all these approaches are feasible and have some challenges that we should know beforehand. For example, as mentioned in <a href="https://jerryzli.github.io/robust-ml-fall19/lec10.pdf">Empirical defenses for adversarial examples</a>, <strong>Adversarial training</strong> based defenses (where the model is re-trained by augmenting adversarial examples in the training set) is observed to be very slow when implemented. Also, research shows that accuracy of <strong>robust models implementing adversarial training drops substantially </strong>than that of non-robust models.</p><p>Few other research papers have also summarized their experience on the adversarial countermeasures including the one on <a href="https://arxiv.org/pdf/2303.06302.pdf">Adversarial Attacks and Defenses in Machine Learning-Powered Networks</a>: A Contemporary Survey. Among the many lessons learnt, the authors highlight that it is generally hard to apply a defense method that is effective on one model or dataset, to other complex models or datasets. Hence, they infer that <strong>Scalability and transferability of the defense approach is a major concern</strong>. It is recommended to go over the papers cited above for more in depth knowledge.</p><p><strong>C) Evaluating Recommendations</strong></p><p>So, one question that every practitioner has while implementing these Adversarial countermeasures are “<strong>how to apply the right one</strong>?” with many unknown challenges in them. A joint effort made by Google Brain, MIT and University of Tübingen on their paper <a href="https://github.com/evaluating-adversarial-robustness/adv-eval-paper">On Evaluating Adversarial Robustness</a>, provides practical advice for evaluating defenses that are intended to be robust to adversarial examples. Some of the principles to perform these defense evaluations include:</p><ol><li>Implement <strong>threat model</strong> specifically centered around the attacker centric approach so that the defense controls planned are most effective.</li><li>Evaluate by applying <strong>diverse set of attacks. </strong>This includes testing the robustness to random noise, evaluating the attack hyperparameters, etc.</li><li>Perform <strong>ablation analysis</strong> by removing a combination of defense components and verifying that the attack succeeds on a similar but undefended model. This clarifies the point of combining multiple defense strategies which would help in model robustness.</li></ol><p>The paper has the complete list of evaluation recommendations that are being categorized as shortcomings, that need to be avoided when evaluating defenses.</p><p>In the last few sections discussing the countermeasures, we have been focusing on methods that prevent Adversarial attacks on the models. But, as we know, an AI based solution also has the application components built around the model to be leveraged by the business or integrated with another system. Hence, it is equally important to plan and implement the application layer security on top of the model. <a href="https://owasp.org/">Application Security</a> is a well-known domain to majority of us, so in this paper, we only wanted to highlight the importance of the subject and the same kind of analysis must be done to work on those countermeasures.</p><h4><strong>Major Roles in AI Programs</strong></h4><p><a href="http://www.bsa.org/">BSA | The Software Alliance</a> which is an advocate for the global software industry, have discussed in their publication about the distinction between <strong>AI Developers</strong> and <strong>AI Deployers</strong>. According to their analysis, organizations building AI systems can act as AI Developer and/or AI Deployer. It is essential to distinguish between these two roles so that legal frameworks could accurately assign obligations to the company based on its role in the AI ecosystem. As a result, <a href="https://www.bsa.org/files/policy-filings/03162023aidevdep.pdf">companies are better able to fulfill the legal requirements and better protect consumers</a>. The figure below discusses the two different roles (and the corresponding examples) that companies take in building AI systems. Going by this idea, we will define the security measures to be considered and accountabilities of these AI roles (AI Developer and AI Deployer).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*kIbvCmGph44oVrEJFni_mQ.png" /></figure><p>The defense mechanisms described in the sections above are largely aligned with the <strong>AI Developers</strong>. This means security techniques and principles that need to be implemented by</p><p><strong>a) Data Scientists</strong> working on data preparation and exploration, and internal and external data acquisitions</p><p><strong>b) AI/ML Scientists</strong> tasked with building AI models and evaluating them against different metrics.</p><p>Additionally, there are <strong>AI Deployers</strong> that use these AI systems (i.e.) companies that work towards productionalizing the model by integrating them with the CI pipelines to operationalize, monitor, and evaluate them for its performance. These are done by <strong>AI/ML Engineers.</strong> We haven’t looked at the defense strategies specific to this aspect yet and we will examine them now, including the security activities that they need to put their attention on.</p><h4><strong>Security Accountabilities &amp; its Importance for Companies using AI Systems</strong></h4><p>So, in the remainder of this paper, we will focus on the defensive strategies that AI Deployers (the companies that are using the AI Models) must follow. There are <strong>strong motives and valuable takeaways </strong>that are imperative for AI Deployers to implement the defense strategies. Sharing a few of those, based on the evidence observed from the research artifacts and the rapid developments happening in the AI community:</p><p><strong>A) Shared Accountability Model</strong></p><p>We saw in the previous section, that companies playing both AI Developer and AI Deployer roles have legal frameworks assigned to them and they must satisfy those requirements. This means AI Developers practice <strong>Due Care </strong>to build-in security mechanisms in the models and the companies that are AI Deployers must do the <strong>Due Diligence</strong> by properly investigating all possible weaknesses on the AI models including the risk mitigation before offering the service to clients.</p><p>All these efforts prevent security breaches from happening and prepares us to recover the AI systems from possible failures. Overall, this synergy of working together by the two AI roles helps build <strong>Trustworthy AI Systems</strong>. Additionally, NIST, a leading standard recommends <a href="https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf">accountability to be shared across the actors in the AI Lifecycle</a><strong> </strong>and requires every actor in the AI Lifecycle to play their part to ensure the security of the complete AI system to be very effective. As you could see from the figure below, many of these actors are part of the AI Deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3DA_0UY2uiK0qtrYAIIIWw.png" /></figure><p><strong>B) AI Assurance Ecosystem</strong></p><p>In addition to understanding how the accountability in AI Lifecycle has got to be planned, the NIST’s Risk Management Framework has many other helpful information, that is worth referring to. Their AI RMF document also highlights the importance of <strong>Test, Evaluation, Verification, and Validation</strong> (TEVV) processes throughout an AI lifecycle and generalizes the operational context of an AI system. The TEVV tasks are given for every phase of the AI lifecycle.</p><p>Although these tasks are collaboratively carried out by many actors (as given in the figure), they certainly need the help of <strong>AI Assurance tools</strong> to operate efficiently and achieve automation. There are a few promising AI Assurance solutions, but many of the commercial vendors are just warming up to tackle the very many attacks and failures including the threats emerging from the <strong>Generative AI</strong> models. This results in companies that are using the AI models (AI Deployers) to gear up and come up with their homegrown tools/solutions/strategies to assist the AI actors in the TEVV processes described above.</p><p><strong>C) The Evolving Risk Landscape</strong></p><p>With the advent of the Generative AI Models, more and more companies are leveraging them for developing their business. Enterprise applications benefit from the integration with Large Language Models (LLM) and Image-generation models by improving performance and enhancing the user experience. Apparently, a venture capital firm a16z was one among the first to share a high-level view of the <a href="https://a16z.com/2023/01/19/who-owns-the-generative-ai-platform/">Generative AI technology stack</a> in their publication. The figure below clearly shows the rise of Model providers and applications built on top of them.</p><p>So, we could infer that the AI Deployers that are consuming the <strong>AI Models as a Service, </strong>see extraordinary value in them but there are numerous challenges here. In addition to the potential threats from the AI Models and the software built around it, there are <strong>safety hazards</strong> and <a href="https://hbr.org/2023/04/generative-ai-has-an-intellectual-property-problem">intellectual property infringement</a> that need to be addressed as well. With all these unknown challenges and dynamic attack surface, companies using the AI Models must take accountability and work towards the implementation of defensive strategies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UgzogPJR1KUnWwsnd69JoA.jpeg" /></figure><h4><strong>Establishing Security competencies for AI Programs</strong></h4><p>We discussed many compelling reasons for AI Deployers to invest their efforts on the defensive controls. But determining the countermeasures may not be as straight forward as we guess. Just like the companies that built these models, the companies that are leveraging them have several different elements to consider for the defenses implementation to be successful. This is because the project using the AI Model might have varying deployment scenarios, uncertainties with technological and platform integration, budget constraints at program level, changing dynamics of the vendor/stakeholder relationship and many other businesses level pain points that impacts the productiveness of the defensive controls.</p><p>Considering these challenges, we will delve more into the <strong>strategic defenses</strong> as opposed to the objective-typed controls, so it is valuable for all of us under all circumstances. Based on that point, we will focus on the security competencies that are essential for AI Programs. It has been proven that having these <strong>competencies will evaluate the effectiveness of the defenses</strong> built into the model by the AI Developers. The upcoming sections are centered around the takeaways for AI Deployers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xldE19HVC7qOBxP1PIm6bQ.png" /></figure><p>a) <strong>AI Red Teaming</strong></p><p>“The best defense is a good offense” — Hiring Red-teaming experts helps enterprise strengthen the resiliency of the overall AI solutions by realizing the effect of Adversarial attacks that are performed using <strong>real-world simulations</strong>. As the adoption of GenAI and LLMs are on the rise, red-teaming efforts become vital to evaluate the safety of the models for the presence of harmful content and unethical output, in addition to the security and privacy aspects. <a href="https://www.anthropic.com/">Anthropic</a>, an AI start up firm focusing on AI Safety and reliability in their research work have discussed the types of harms discovered by red teaming. The picture above is a <a href="https://arxiv.org/pdf/2209.07858.pdf">visualization of the red team attacks</a> that shows the clusters of attack types. When it comes to the planning and procedures involved, I would recommend going through the resource from Microsoft that briefs the approach to be taken in <a href="https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/red-teaming">red teaming an LLM</a>. Although our example focused on LLMs and Generative Models, red teaming activities would benefit solutions developed and deployed using Discriminative models as well.</p><p>b) <strong>Threat Modeling</strong></p><p>Another favorite subject for many security professionals who come from the traditional software security world. Threat Modeling is extremely valuable for companies that buy AI-ML solutions and deploy them. The snapshot below gives an overview of the methods followed to <a href="https://arxiv.org/pdf/1905.12762.pdf">Threat Model an AI system to get the attacker’s point of view</a>. But as we attempt to address the <strong>potential threats in the complete system</strong>, we need to approach threat modeling from multiple dimensions. This includes the threats emerging from the design patterns followed during model operation scenarios, interoperability with existing systems, software technology stack, data ingestion methods, etc. Additionally, there are also threats arising out of process lapse, non-conformance aspects, regulatory violation, amongst others. Hence, it is essential to take a <strong>Combinatorial approach to Threat Modeling.</strong> If the principles are correctly applied, Threat Modeling activity adds a lot of value to AI Deployers including evaluating the defenses, prioritizing them, understanding the root cause of the threats, establishing the feedback loops with AI Developers, and importantly contributes to the maturity of the overall <strong>AI Lifecycle’s risk posture</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*to1UbfRssHfe_XuyFr934g.png" /></figure><p>c) <strong>Compliance Audits</strong></p><p>Compliance is a common practice that applies to all the roles across the AI Lifecycle. Specifically, from an AI Deployers’ standpoint, there are few aspects of compliance that are of foremost value. The first among them is the <strong>Data Privacy Laws</strong> — As AI based applications require the users to input the PII and PHI data, the auditors must investigate the processing of personal data and weather it is going to be shared with external entities. This could be evaluated by doing a <strong>Data Protection Impact Assessment</strong> (DPIA) and there are many resources, guidelines, and templates that are recommended by <a href="https://iapp.org/resources/topics/privacy-impact-assessment-2/">IAPP</a> and <a href="https://gdpr.eu/data-protection-impact-assessment-template/">GDPR</a> to get more insights. Secondly, there are <strong>AI Standards</strong>, where we are getting to witness countries regulating the use of AI. <a href="https://artificialintelligenceact.eu/the-act/">The AI Act</a>, proposed by the European commission and the <a href="https://ised-isde.canada.ca/site/innovation-better-canada/en/artificial-intelligence-and-data-act">AI and Data Act</a> from the Government of Canada are some of the examples. There are also <strong>Government Protocols</strong> and <strong>AI Regulations for Industry Verticals</strong> that require the AI Deployers to comply with the mandates given by the authorities. Hence, it is strongly advised to have a team of Internal auditors diligently assess the AI based solutions.</p><h4><strong>Continuous Compliance</strong> <strong>and Best Practices</strong></h4><p>We discussed many strategies, competencies, compliance measures, countermeasures that are specific to adversarial attacks in the previous sections (and there are numerous others too), but the effectiveness of defenses will be long-lasting only when performed continuously. This means doing a threat modeling for every milestone release, periodically auditing programs for regulatory adherence, and so on. All these efforts result in AI solutions being continuously compliant. One way of accomplishing this is by implementing the <strong>Principles of MLSecOps</strong>, where the security process gets embedded into the complete AI Lifecycle, thereby securing the data pipeline, models deployed, and the software stack built. Hence this practice is followed from the inception stages till the sunsetting and performed repeatedly for every release.</p><p>In addition to the processes and practices, <strong>Security Champions</strong> acknowledge the value of investing their time in shaping up the culture of the organization to successfully accomplish their objectives by educating and empowering the product teams. This would mean rolling out training programs and awareness sessions on the secure use of technologies to prevent <strong>Insiders risks</strong> from happening. Based on our learnings from the recent security incidents, where GenAI based code generation and chat applications’ improper use have resulted in sensitive data being leaked out. Hence alongside technical controls, <strong>administrative controls</strong> need to be carefully planned and executed.</p><h4><strong>Industry Initiatives and Ongoing Efforts</strong></h4><p>Unlike the initial days of software development, where reliability, trust, and regulations were an afterthought, the AI community has adopted and prioritized <strong>Trustworthy AI</strong> by embedding security and privacy principles in the AI model development efforts at an early stage. Today, we witness a lot of entities highlighting the benefits of Trustworthy AI and many organizations are in the process of diligently implementing its key pillars namely Security, Privacy Accountability, Transparency, Explainability and others. The Linux Foundation has done an excellent job of capturing the <a href="https://lfaidata.foundation/projects/trusted-ai/">Principles of Trusted AI</a>. There are other organizations that have done a fair job of defining the execution strategies of these <strong>AI Principles</strong>. On a similar mission to help practitioners build Responsible AI systems is the <a href="https://www.responsible.ai/">Responsible AI Institute</a> that has published many artifacts including the how-to’s that will be very valuable in building the AI solutions with the appropriate laws, regulations, and best practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YEjJ2-KM_E9PX1ZBbwwLnw.jpeg" /></figure><h4><strong>Conclusion</strong></h4><p>We looked at many strategies and approaches, but where do we get started? As quoted by the well-known American writer, Mark Twain, “The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks and starting on the first one.” Likewise, we approach the defenses to AI Systems by breaking up the complex strategies, performing prototypes to diligently analyze the working patterns, and that in turn will help us determine the defensive measures lasting effect.</p><p>Throughout the series, we noticed one attribute contributing in a significant way in guiding the AI Security practice — <strong>Research and Development</strong>. The R&amp;D community has played a huge role in building offensive and defensive principles of Securing AI system. A lot of research papers were referenced in all the three articles. It is highly recommended for every AI practitioner to invest more time and resources in researching on the latest trends, security attacks, defensive tools and importantly keeping a constant watch on the regulatory updates. All these helps them to be better prepared for the future and always manage uncertainties, to build safe and secure AI Solutions.</p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b1fd6ad33fa" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/securing-ai-systems-defensive-strategies-6b1fd6ad33fa">Securing AI Systems — Defensive Strategies</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing AI Systems — Researching on Security Threats]]></title>
            <link>https://medium.datadriveninvestor.com/securing-ai-systems-researching-on-security-threats-3c14834d71ec?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c14834d71ec</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Tue, 21 Feb 2023 23:37:54 GMT</pubDate>
            <atom:updated>2023-02-23T09:43:00.053Z</atom:updated>
            <content:encoded><![CDATA[<h3>Securing AI Systems — Researching on Security Threats</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uy2VC1A7LA_V2W4FP6hPLQ.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/noel_bauza-2019050/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1185474">Noel Bauza</a> from <a href="https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1185474">Pixabay</a></figcaption></figure><p>“Research is to see what everybody else has seen and to think what nobody else has thought”, a powerful quote from a renowned Hungarian biochemist <a href="https://en.wikipedia.org/wiki/Albert_Szent-Gy%C3%B6rgyi">Albert Szent-Györgyi</a> — the man behind the discovery of Vitamin C and the components and reactions of the citric acid cycle. That quote reminds us of the importance of recognizing the work done by researchers and at the same time, exploring further to innovate for better insights and get more creative. The extensive learnings from the existing knowledgebase helps many of the researchers and scientists to improve on the works of others and build new knowledge. Such is the impact created by Research, and this Research is an integral element with evolving technologies like Artificial Intelligence where many innovations are happening. In the last decade, we have seen rapid developments in the field of AI which comes from the betterment of the existing knowledge and also, different and creative thinking.</p><p>Advancements in algorithms, architectures, and computing abilities in AI have not only helped in developing cutting-edge solutions but also introduced new patterns of risks to everyone involved in the complete chain. In this article, we will look at those new patterns, particularly, from the perspective of <strong>Adversarial AI</strong> that are helpful in launching the <strong>security threats</strong>. We have briefly looked at these concepts, in the previous paper and now we will do a deeper analysis of this segment. Again, it is the research community that has contributed heavily to this space in showing the various possibilities and the impact of breaking into AI systems.</p><h4><strong>Researching on Security Threats</strong></h4><p>There are five segments of risks<strong> </strong>that have we have dealt with in <a href="https://medium.com/datadriveninvestor/securing-ai-systems-risks-in-ai-based-solutions-dc3077bdf6b3">first paper</a>. They are neat and straight-forward. For example, we know the impact of regulatory violations — the articles, rules in each of these regulations (eg: GDPR or FINRA) are nicely detailed to help us adhere to the requirements while developing AI based solutions. The same applies to other types of risks namely <strong>AI Safety</strong> and <strong>AI Principles.</strong> Although continuous developments and improvements are happening there, the basic concepts are well laid out, and the AI practitioners have sufficient guidance to implement them. However, there are a few other areas of risk that are not easily referenceable, and we could better implement them only through exploration and experience.</p><p>Specifically, the innovations and advancements in security threats could be better learnt from the research papers written on the topic of <strong>Adversarial attacks in ML</strong>, where the mindset of the adversaries and attacking patterns are investigated and experimented from different dimensions. To make it easier to for us to understand the length and breadth of the research involved in learning security threats, we will put them into two categories — <strong>Completeness</strong> and <strong>Comprehensiveness</strong>. In the following sections, we will explore them in detail. As much as we appreciate the research efforts, we need to understand the pragmatism involved in the implementation including its applicability in real-world scenarios. We will deal with these in a separate section called <strong>Capability</strong>. Now let’s get started…</p><p><strong>1)</strong> <strong>Completeness</strong></p><p>The term “Completeness” when it comes to researching about risks in AI solution means that the objective of practitioners should be to go deeper and do everything possible within their scope while assessing potential threats. To understand this further, we had looked at the risks in the <strong>Software Security</strong> part in the last paper. There, we had learnt about the attacker entry points that are relatively easier to observe and act upon (eg: when the models are exposed as an API) but there are few other attack methods that we are aware of but do not give <strong>attention to the depth involved</strong>, from the perspective of the patterns used, or the cascading effect that they create, etc.</p><p>A good example of this type is the possibility of exploiting the serialization process. [Serialization implementation using <strong>Pickle library</strong> is detailed in the <a href="https://docs.python.org/3/library/pickle.html">python documentation</a>]. <strong>Serialization</strong> involves converting the model into a byte stream format so that it is easier perform operations like storing and loading the models as they are productionalized. There are many references where researchers have shown the possibility of <strong>injecting malicious code in pickle files</strong> and at the time of deserializing, the code injected could bring in adverse functioning of the overall solution by disturbing the <strong>model integrity</strong>. Our assessments must give more attention to these implementation logics as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJYWHs6Cs2c0hAkNwWp54Q.png" /><figcaption>Security warning about pickle library from Python documentation</figcaption></figure><p><strong>2)</strong> <strong>Comprehensiveness</strong></p><p>If Completeness refers to the entirety of the research, comprehensiveness refers to completeness over a broader scope. As AI practitioners, we need to think about the many components in the complete AI lifecycle and the potential security threats to each of them. All these components have not only gone through many advancements but also introduce new ways of threats. In the following sections, we will study the security threats involved in the advancements of these AI components that are undervalued.</p><p><strong>a)</strong> <strong>Architectural Patterns</strong></p><p>Many <strong>Neural Network</strong> architectures have been proposed in the last few years. Computer Vision and Text processing domains are great examples where many advancements have happened. One among them is the emergence of <a href="https://en.wikipedia.org/wiki/Transfer_learning">Transfer Learning</a>. As mentioned earlier, there are not only value adds, but also security threats associated with this concept. There is a possibility of weight poisoning attack while data scientists implement Transfer Learning.</p><p>This paper on <a href="https://arxiv.org/abs/2004.06660">Weight Poisoning Attacks on Pre-trained Models</a> explores the likelihood of threats associated in using <strong>pre-trained weights</strong> from untrustworthy sources, and how the adversaries could inject vulnerabilities in them, leading to <strong>weight poisoning</strong> attacks. Leveraging pre-trained weights improves the computation efforts and data scientists could use this directly in their model to <strong>fine-tune the target task</strong> (eg: classification). But the vulnerabilities in them could be triggered and exploited by attackers leading to misclassification.</p><p>To explain this attack at a high-level, the authors of this paper have used a combination of regularization method and an initialization procedure to build a concept called <strong>RIPPLES</strong> that work behind the scenes in successfully executing the weight poisoning attacks. A picture depicting how this initialization procedure called <strong>Embedding Surgery</strong> is triggered and the way it impacts the Embedding matrix is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*fYvDi97sm1Zfe9RIqQ4hbw.png" /></figure><p><strong>b)</strong> <strong>Learning Algorithms:</strong></p><p>One more key area where advancements have been happening are with the Learning Algorithms. We have seen in our examples so far, the use of Supervised Learning, that works on labelled data. But there are Unsupervised, Semi-supervised and Reinforcement Learning algorithms as well. Many of the Adversarial ML attacks are predominant in <strong>labelled datasets</strong>. For instance, many researchers have successfully shown the data poisoning attacks on Supervised Learning Algorithms. But there are possibilities of launching backdoor attacks on models implementing <strong>Semi Supervised Learning (SSL)</strong> algorithms by adversarially poisoning the unlabeled data.</p><p>As mentioned in this paper on <a href="https://ojs.aaai.org/index.php/AAAI/article/view/17266">Deep Hidden Backdoor Attack on Semi-supervised Learning via Adversarial Perturbation</a>. The authors have proposed the idea of <strong>DeHiB — Deep Hidden Backdoor attack</strong> that could be performed using a combination of adversarial perturbations and trigger patterns to misclassify the model outcome. Based on the experiments done on SSL algorithms, research scientists who have authored the paper on <a href="https://arxiv.org/abs/2211.00453">The Perils of Learning From Unlabeled Data: Backdoor Attacks on Semi-supervised Learning</a>, believe that backdoor <strong>poisoning attacks on unlabeled data</strong> could be performed by <strong>adversaries with limited knowledge</strong> but have a severe impact. Strongly encourage everyone to read this paper to get a deeper understanding of the attack patterns in SSL algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*2xi8RoVfcJtPDqwETpthkA.png" /></figure><p><strong>c) Computation Methods:</strong></p><p>“Let’s think outside the box”. Well, Attackers do not necessarily use the adversarial examples to attack the Machine learning models. Some of the threats prevailing in the environment could be exploited as well and they are relatively easier. But there are few other attacks that require quite a lot of engineering efforts and one such example is the <strong>Memory exploitation</strong> performed on Deep Learning frameworks.</p><p>Popular frameworks like <strong>TensorFlow</strong> (there are other <a href="https://viso.ai/deep-learning/deep-learning-frameworks/">popular frameworks</a> too) for building AI models have been proven vulnerable. TensorFlow leverages <strong>GPU to run deep learning</strong> algorithms because of large number of computations and the high-performance requirements. However, Research scientists have come up with an exploitation technique called <strong>Code warpping</strong>. As per their research published in <a href="https://dl.acm.org/doi/abs/10.1016/j.cose.2020.102115">Mind control attack: Undermining deep learning with GPU memory exploitation</a>, the goal is to <strong>hijack the control flow of the GPU</strong> to achieve the arbitrary code execution ability by exploiting GPU function vulnerabilities. The outcome of this attack could help modify/reduce the <strong>prediction accuracy</strong>. The attack pre-requisites including the major steps of the process are nicely depicted in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-kXUSqar93kWgExj6NkdQ.png" /></figure><p><strong>3)</strong> <strong>Capability</strong></p><p>We appreciate the research involved in Adversarial AI. However, we need to be very practical when it comes to applying these ideas in real-world scenarios. Let us ask a couple of questions here to better understand the capability of Adversarial Attacks. “How much engineering effort does it take to generate these <strong>adversarial examples</strong>?”, “how much of cost does it take for the <strong>adversarial training</strong>?”. It is essential to ask these questions since a lot of times, the research work is proved with certain <strong>assumptions</strong>. There are hypothetical scenarios, pre-configured targets that are clearly mentioned in the research observations made. Hence the inferences made are <strong>more empirical and less practical</strong>. As practitioners try to adopt these to their <strong>real time scenarios</strong>, we need to be cognizant about the process and resource constraints including the outcome and benefits to all the stakeholders involved.</p><p><strong>a)</strong> <strong>Complexity of Adversarial Attacks</strong></p><p>The research paper on <a href="https://arxiv.org/abs/1809.04790">Adversarial Examples: Opportunity and Challenges</a> clearly describes the constraints involved in Adversarial Examples and how to evaluate them as they detail the <strong>Cause, Characteristics and Evaluation Metrics</strong>. Based on what the researchers have observed, the cost of constructing these adversarial examples for adversarial training is expensive. Secondly, the Adversarial Examples are not going to yield a higher <strong>success rate, </strong>even after meticulously planning them. Additionally, the researchers from this paper say that careful considerations need to be given when constructing the <strong>adversarial perturbations</strong> to achieve a fine balance between developing them and the human visual system so that they are not easily distinguishable by human eyes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*rB3x4jV1yYuLjx5cS95BIg.png" /></figure><p>The graph above is a great way to understand the complexity of the attack, given the abilities and goals of the attacker. If the goals of the attacker are to perform <strong>targeted misclassification</strong>, then the complexity of performing an attack is more. Of course, there are a few other information about metrics and other useful observations made in this paper that is worth a read.</p><p><strong>b) Applicability of Adversarial Attacks in Data Processing Methods</strong></p><p>Many of us would have come across the research work on Adversarial attacks focusing on scenarios where the adversary could observe the complete sample and then add perturbations at some point in the sample. Those attacks work well when the <strong>static data</strong> has been stored over a period and later inputted to the model. But today, we have many live tweet streaming applications, logs and traces of web apps that are monitored in real time, and hence the ML models are designed to work with these <strong>streaming data</strong>. To attack these models that works with streaming data, researchers have proposed a <strong>Real-Time Adversarial Attack</strong> <strong>Scheme.</strong></p><p>As mentioned in this paper <a href="https://arxiv.org/abs/1905.13399">Real-Time Adversarial Attacks</a>, the target system takes streaming input; only past data points can be observed, and adversarial perturbation can only be added to future data points. The use of <strong>adversarial perturbation generator</strong> continuously leverages the observed data to approximate an optimal adversarial perturbation for future data points (Figure describing the high-level steps below.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*JKet8uyv4Zf5fJlQ5KH2cw.png" /></figure><h4><strong>What’s Next</strong></h4><p>We have looked at a different perspectives of security risks — the threats from Adversarial ML, including the length and breadth of research required and most importantly, the constraints and its applicability in the real world. Of course, the evolution and advancements are a never-ending thing with AI and so is the risk landscape, where we will get to experience many trends and new patterns. But whatever we have seen so far, gives us a better picture of the present risk landscape in AI. In our next article, we will focus more on the steps to be taken by organization and be prepared to face the challenges posed by the AI risks to build reliable solutions.</p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c14834d71ec" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/securing-ai-systems-researching-on-security-threats-3c14834d71ec">Securing AI Systems — Researching on Security Threats</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing AI Systems — Risks in AI based Solutions]]></title>
            <link>https://medium.datadriveninvestor.com/securing-ai-systems-risks-in-ai-based-solutions-dc3077bdf6b3?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc3077bdf6b3</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regulation]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Tue, 31 Jan 2023 21:51:39 GMT</pubDate>
            <atom:updated>2023-02-02T10:15:07.327Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Securing AI Systems — Risks in AI based Solutions</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KId4bdHh_V79y9zUQ8Kz5A.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/obsidianphotography-398924/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=433071">Obsidain Photography</a> from <a href="https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=433071">Pixabay</a></figcaption></figure><p>“With great power comes great responsibility” — An adage from the Spiderman comic books — This simply means, if you have the power to do it, do it for the good of others. This saying is certainly most suited to the Leadership roles. As one might see, even within our own organization, many executives and senior leadership working on critical job functions, their work is so impactful that goes beyond their role to serve the best needs of the company and customers, and not just their own interests. Not just at the senior level, but this simple yet powerful principle needs to be followed by every practitioner in the industry as we build products and solutions, consumed by many across the globe. These products must be always reliable, and the outcome must be trustworthy. It must do good to the end-user and uphold integrity and not just solve the business requirements for which the product is designed.</p><p>As we speak about these products, many of them are built using emerging technologies and platforms, particularly the use of Artificial Intelligence (AI) in product development is on the rise. There are a range of benefits that we could accomplish, importantly, enhancing the customer experience, automating tasks that are repetitive, and much more that we all are already familiar with. As much as we are delighted with the benefits AI brings to the table, we should also be cautious about the business risks involved in the operating model and the legal implications of not being on track with the standard/recommended practices and regulatory requirements. This is where, we should also consider the Information security threats and cyber attacks on the AI Solutions, which is a huge cause of the overall business risks. Information Security in AI is a subject that is super important not only for infosec professionals, but everyone involved in using AI in some capacity (be it the product devs using AI as part of their solutions built or for end-users consuming them).</p><p>This paper, comprised of 3 parts is a small effort taken to align the business leaders, solution developers and all other AI users to get better equipped by scaling up their resources to build reliable and resilient AI solutions. In the first part of this paper, we will focus on the risks in AI based solutions and in the subsequent parts, we will go over some of the concepts that every practitioner should invest their efforts on to safeguard against the potential risks involved in AI.</p><h3><strong>Risks in AI based Solutions</strong></h3><p>Based on the developments happening in this space and the research that has been done by many entities across academics and the industry, there are 5 major categories of risks identified in AI based solutions. Let us begin by discussing about the risks from Adversarial AI.</p><p><strong>a) Adversarial AI</strong></p><p>Adversarial AI is defined as the techniques followed by users with malicious intent to defame the AI systems and their outcome. The goal could be to break the integrity of the model or even make it reveal sensitive information. There are a few categories of Adversarial attacks in Machine Learning. One among them is the <strong>Evasion Attacks</strong>, where the attacker’s intent is to force the AI system into doing a wrong prediction. A common application of these is against the Computer Vision solutions where the model is tasked with an image classification challenge and a successful attack, results in misclassifying the given image with a high confidence score.</p><p>The attacking capability of Evasion attacks could be performed using <strong>White-box or Black box</strong> approaches. Some of the evasion attack methods that have been performed in the Image models include the ZOO attack, Boundary attack, One-Pixel attack, etc. Evasion attacks are just one of the attack categories, but there are other categories of Adversarial attacks too, including data poisoning, model inversion, model stealing attacks, etc, each of them impacting one or all the <strong>three tenets of security</strong> — Confidentiality, Availability, and Integrity. To get a very detailed and thorough understanding of the Adversarial AI attacks possible on ML systems, please refer the paper on <a href="https://arxiv.org/ftp/arxiv/papers/2112/2112.02797.pdf">ML Attack Models</a>: Adversarial Attacks and Data Poisoning Attacks. Also, the document on <a href="https://learn.microsoft.com/en-us/security/engineering/failure-modes-in-machine-learning">Failure modes in Machine Learning </a>has captured the essence well.</p><p>Adversarial attacks and Software Security (discussed later) are essential for the Information Security professionals to know about the attack surface of AI. We will do a deeper analysis and discuss the defensive measures in the subsequent parts of this Article.</p><p><strong>b) AI Safety</strong></p><p>The <a href="https://cset.georgetown.edu/">Center for Security and Emerging Technology</a> (CSET), is an initiative formed with the primary goal of studying about security impacts of emerging technologies. In their published research work, they have defined <a href="https://cset.georgetown.edu/publication/key-concepts-in-ai-safety-an-overview/"><strong>AI Safety</strong></a> as “An area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably”. Based on their work, they identified three areas of Safety issues in AI. <strong>Robustness Failure</strong> — A type of AI accident that prevails in many AI based solutions. Basically, the cause of this issue happens when the AI models are not trained on all sort of input scenarios that they might see in production environment. If the data set inputted for the model has never considered the scenario which the AI system might face in production, then the (say) predictive analytics might go wrong leading to misclassification.</p><p>Secondly, AI safety issues are also observed because of <strong>Assurance Failure</strong>, where the end-users do not understand decisions taken by the AI system, thereby not validating the outcome. The AI Admins’ lack of monitoring of the internal details leads to these types of failures. The last among the AI Safety issue is the <strong>Specification Failure</strong>, which is a result of the misalignment in the approach taken by the AI models to meet the goals specified by the designer. Not carefully crafting the desired behavior to be accomplished as models meet the end goal, leads to this category of AI Safety issues. At the time of writing this article, the authors have discussed about the YouTube’s video<strong> recommendation algorithms</strong> as a classic example of Specification failures. It should be noted that these types of AI Safety issues are not just applicable to the Software Systems but predominantly observed in the world of <strong>cyber-physical</strong> systems as well.</p><p><strong>c) Non-conformance to AI Principles</strong></p><p>At first, let us understand a bit about the AI principles. There are a few organizations that have extensively talked about this subject, but most of them have identified some of the common principles namely Accountability, Transparency, Privacy, Explainability and Security. So, what are the risks involved in not following these AI Principles? Let’s take the case of Transparency and we will understand the threats here by studying the concept of <strong>AI bias</strong>.</p><p>One of the implicit elements of machine learning models is bias. However, a potential threat is observed only when the outcome of the model favors a particular class and this could be a race, gender or even a segment of geography or community. All these biased models and data affects the company’s reputation. <a href="https://www.gartner.com/en/newsroom/press-releases/2018-02-13-gartner-says-nearly-half-of-cios-are-planning-to-deploy-artificial-intelligence"><strong>Gartner </strong>prediction</a> says that through 2022, 85 percent of AI projects will deliver erroneous outcomes due to bias in data, algorithms, or the teams responsible for managing them. One among the many steps taken to <a href="https://research.aimultiple.com/ai-bias/">fix the Model bias</a> includes <a href="https://www.forbes.com/sites/forbestechcouncil/2022/06/21/preventing-harmful-ai-bias-with-fairness-through-awareness/?sh=582349146c83">establishing transparency</a> throughout the business process. The <strong>risk of unfairness</strong> from the algorithm and data must be examined to align with AI Principle of Transparency. Non-conformance to other AI principles could result in serious threats as well. To get deeper understanding of all these concepts including <strong>Fairness, Accountability and Transparency</strong>, refer to the <a href="https://www.fatml.org/">fatml.org</a>. It has got excellent collection of resources for us to deep dive and gain better insights including the risks associated because of non-adherence.</p><p>Although Non-conformance to AI Principles have been called out as a separate risk category because of the direct impact to the business, many regulations have also referred to them as part of their requirements. Therefore, the risk and the implications coming from regulations are briefed in a separate section, coming up the next.</p><p><strong>d) Regulatory violations</strong></p><p>Program audits and Regulatory experts demand that solutions developed are compliant to recommended standards. Although there are many controls that are planned, there are few lapses observed even in traditional applications. This is very much possible even in AI based solutions as well. Let’s consider the <strong>financial industry</strong>. They are one of the big adopters of any technology and that is true with the adoption of AI as well. Most entities in the financial domain are governed by strong regulations and one among them is <strong>FINRA</strong>. They help regulate member brokerage firms and exchange markets to protect America’s investors by making sure the broker-dealer industry operates fairly and honestly.</p><p>One of the <a href="https://www.finra.org/rules-guidance/key-topics/fintech/report/artificial-intelligence-in-the-securities-industry/key-challenges">FINRA Rules</a> recommends supervision to ensure financial data security. This means, at the time of supervising activities related to AI applications, GRC professionals or auditors must understand how those AI applications function, how their outputs are derived, and whether actions taken pursuant to those outputs are in line with the firm’s legal and <strong>compliance requirements</strong>. Although this is a regulatory requirement, it aligns with the AI principle — Explainability. <a href="https://www.ibm.com/watson/explainable-ai">Explainable artificial intelligence (XAI)</a> as defined by IBM (11) is “a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms. Explainable AI is used to describe an AI model, its expected impact, and potential biases.” Hence, <strong>Model Explainability</strong> is a must with AI applications that uses features like autonomous decision making to get the confidence of auditors and be FINRA compliant.</p><p>Based on the Industry for which the AI solution is developed, we must consider other regulatory requirements and the risks from Non-compliance and violations. For instance, GDPR, CCPA and other <strong>Privacy regulations</strong> emphasize on the importance of <a href="https://iapp.org/news/a/privacy-and-responsible-ai/">Privacy in AI Applications</a>.</p><p><strong>e) Software Security</strong></p><p>We have been spending the time understanding the risks involved in data, algorithms and model development. But Data Scientists knew that most of their time goes into maintaining and monitoring the ML models deployed. <strong>Productionalizng ML Models </strong>is done using a variety of methods — one of the simple methods is to develop a server application and expose this model as an API. As the API is exposed to the public domain, we must be aware of a range of <a href="https://owasp.org/www-project-api-security/">security threats</a> including injection attacks, authorization issues and sensitive data exposure that are impacting the AI based solution. Like <strong>API security</strong>, we must be geared up with defensive security measures if the deployment involves containerizing dockers and using <strong>Kubernetes clusters</strong>.</p><p>Most importantly, we will also need to understand about the <strong>Supply chain issues</strong> involved. Many Data Scientists and AI Engineers make use of open-source libraries, and tools that help them build, monitor the models faster. We need to embed security into this process so that security attacks are prevented from happening. Most recently, <strong>PyTorch</strong>, an open-source ML framework used by many practitioners, was compromised. A<a href="https://pytorch.org/blog/compromised-nightly-dependency/"> malicious dependency package (torchtriton) </a>that was uploaded to the Python Package Index (PyPI) code repository with the same package name as the one shipped on the PyTorch nightly package index. <strong>This malicious upload</strong> affects dependencies for packages that are hosted on public package indices.</p><p>There are few other stages in the entire process where data security attacks are possible. For example, in the <strong>data pipeline </strong>— right from the stage where the data is ingested and how they are handled before being processed by the algorithm and model. Also, there are scenarios, where AI models could be integrated as part of a software product that could be accessed by users through a <strong>Web portal. </strong>In this case,<strong> </strong>web applications must be fully assessed to prevent threats impacting the availability of the overall AI solution.</p><h3><strong>What’s Upcoming?</strong></h3><p>In Part 2 of this paper, we will do a deeper analysis of the security risks including the defensive measures and their effectiveness. We will also look at how the Risk landscape is changing along with the rapid evolution in the AI space.</p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc3077bdf6b3" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/securing-ai-systems-risks-in-ai-based-solutions-dc3077bdf6b3">Securing AI Systems — Risks in AI based Solutions</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolving Threat Modeling]]></title>
            <link>https://infosecwriteups.com/evolving-threat-modeling-e83a9e3c5e6b?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/e83a9e3c5e6b</guid>
            <category><![CDATA[threat-modeling]]></category>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[application-security]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Tue, 06 Apr 2021 15:12:31 GMT</pubDate>
            <atom:updated>2021-04-07T18:17:20.299Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-EKmdGbsz0TxuY7FenWGQ.jpeg" /><figcaption>Image by Gerd Altmann from Pixabay</figcaption></figure><h3><strong>Shift-left Security</strong></h3><p>The security industry is constantly evolving. This is very much evident from the earnest efforts taken by the security teams in the enterprises on shift-left activities in securing their products. Especially, increased importance is given to Threat Modeling by security professionals. There is a plethora of resources which talk about what, why, when, where and how of this particular activity. Numerous Threat Modeling approaches exist today that can be applied for various types of outcomes. Additionally, new techniques and tools are emerging to perform Threat modeling for different scenarios.</p><h3><strong>The Perpetual struggle</strong></h3><p>Enterprises and security professionals could leverage from a wide variety of these different options for the specific requirements they are handling. It is to be noted that though there are many benefits and these approaches have been proven to be useful in many complex scenarios, there are many common challenges observed too. These challenges pose a risk to the adoption of threat modeling activity in the long run and eventually its value is not perceived well by the business teams. Thus, security professionals have a constant struggle in helping the teams realize the value of this activity. If you have been practicing Threat models, I am sure you will feel the impact of the aforementioned point. Of course, there are many challenges observed in the entire threat modeling process.</p><p>Additionally, there is a strong perception in the industry that Threat modeling activity needs to have a diagram to discover all the applicable threats. Threat modelers know that this is certainly not the case with all the scenarios especially when it comes to the DevSecOps methodology. All these challenges, the unilateral mindset involved in carrying out the threat modeling process and the need to address them as we practice threat modeling in modern methodologies requires an elaborative study with the main goal of evolving threat models.</p><h3><strong>Research on Evolving Threat Modeling</strong></h3><p>The series of discussions and the outcome of an elaborative study made by practioners and researchers together from diverse backgrounds with the intent of evolving threat modeling is what I would like to share with you all. The paper is embedded along with this article for a detailed reference. However, there are a few key areas that I would like to summarize.</p><ol><li>Many of the threat modeling techniques having good implementation are affected by challenges. The primary challenges around the scalability and quality front have been identified and the relevant anti-patterns have been captured.</li><li>The challenges in the traditional practices of threat modeling, especially the diagrammatic approach have been studied in detail and how to address them to fit to DevSecOps needs is another core area that has been discussed in detail.</li><li>One of the core outcomes of threat model is capturing the security risk. The risk captured should be meaningful and impactful to the stakeholders. We have attempted to address this including how various functions in the Enterprise can benefit from using Threat Modeling as a central tool for Security Risk Management.</li><li>Finally, we have defined a Maturity Model for Threat Modeling, which is the crux of this paper. “One size does not fit all”. Threat modeling at enterprises is performed at different levels of depth by professionals. Based on this, they are integrated with different tools and processes. Accordingly, we have identified maturity levels to prepare organizations to experience the right outcome at every level.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Fembed_code%2Fkey%2Fvv0buuKcl1gstW&amp;display_name=SlideShare&amp;url=https%3A%2F%2Fwww.slideshare.net%2Fsecret%2Fvv0buuKcl1gstW&amp;image=https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2Fevolvingthreatmodelingforagilityandbusinessvalue-210405023159-thumbnail-4.jpg%3Fcb%3D1617632978&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=slideshare" width="342" height="291" frameborder="0" scrolling="no"><a href="https://medium.com/media/85b315086052d68a91636a84029cbcf8/href">https://medium.com/media/85b315086052d68a91636a84029cbcf8/href</a></iframe><h3><strong>Final Thoughts</strong></h3><p>Overall, the paper has gone through a substantial amount of work to show how Threat models could be integrated with the business and can be made easily adoptable by enterprises following modern methods like DevSecOps. The work done in integrating it as a central part of Risk Management and the Business processes and the ideas in representing this as a Maturity Model, shows how threat models could be evolved. Finally, with many challenging demands coming from the future, threat models will continue to evolve, and continuous efforts must be invested into this process by researchers and practioners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e83a9e3c5e6b" width="1" height="1" alt=""><hr><p><a href="https://infosecwriteups.com/evolving-threat-modeling-e83a9e3c5e6b">Evolving Threat Modeling</a> was originally published in <a href="https://infosecwriteups.com/">InfoSec Write-ups</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrity of Self-Sovereign Identity Solutions]]></title>
            <link>https://medium.datadriveninvestor.com/integrity-of-self-sovereign-identity-5dfc792bad7b?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dfc792bad7b</guid>
            <category><![CDATA[self-sovereign-identity]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[hyperledger-indy]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Fri, 26 Feb 2021 15:16:51 GMT</pubDate>
            <atom:updated>2021-02-26T15:59:07.708Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFt0TNWSrXG_8MY7zOv9Ng.jpeg" /><figcaption>Image by Anemone123 from Pixabay</figcaption></figure><p><strong>“Trust, but Verify”</strong> a popular quote that was frequently used by the former American President, Ronald Reagan when discussing U.S. relations with the Soviet Union. Trust is such a valuable factor in any ongoing relationship and is even more important when it comes to establishing the identity itself that is seen as a fundamental aspect to begin and develop any relationship. Identity cannot be easily proved without the sharing of personal data by the parties involved but doing so might lead to compromising the <strong>user’s</strong> <strong>privacy</strong>. The advent of technologies solves this underlying security problem by enabling trust when establishing connection between the parties, at the same time preserving the individual’s privacy. This is accomplished by the concept of “<strong>Self-Sovereign Identity</strong>” (SSI). Enterprises and Governments are gradually adopting this identity technology; however, these SSI solutions need to follow several best practices and extensive assessment before a successful deployment. Hence a correct implementation and comprehensive verification of the overall solution are very important and that is the crux of this paper.</p><h4><strong>Identity and its Evolution</strong></h4><p>There are many detailed references that discusses about the digital identity and its evolution. One of the best references to understand them is the blog written by Christopher Allen on “<strong>The Path to Self-Sovereign Identity</strong>”. Not just the generations in identity, but the blog also briefs about the principles of Self-Sovereign identity, which we will look into shortly. Throughout this paper, we are referring to the digital identity. A <strong>digital identity</strong> is information on an entity used by computer systems to represent an external agent. That agent may be a person, organization, application, or device (ISO/IEC 24760–1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXETHmV3mJz88P8BSb5q_Q.png" /><figcaption>The Evolution in Identity Management Solutions</figcaption></figure><p>The figure above is a good snapshot of the evolution in the Identity Management (IdM) Solutions. <strong>Centralized identity</strong> is where it all started in which the centralized authorities helped in proving the authenticity of the information coming from different entities. There was plethora of benefits but there were challenges too. The power associated with these authorities including other issues like security and scalability made the researchers come up with the <strong>Federated identity</strong> solution where the user’s identity could be shared with different service providers, however most of the issues associated with centralization still remained including the privacy risks since the identity is still controlled by a central authority.</p><p><strong>User-centric identity</strong> solutions seemed to be a better alternative in addressing the challenges with centralization by putting the user in the center and giving more control to the user on their identity, but it still required the user to select the existing powerful identity organizations that involves the user’s personal data for the identity management. <strong>Self-Sovereign identity</strong> (SSI) looked much more promising than the other three. The key characteristics of user-centric approach were properly defined, and a full sovereign control of that identity is determined by the individual who owns the identity. Finally, a true decentralization of trust could be achieved without involving any central authority in the identity process.</p><h4><strong>Core concepts in the SSI Solution</strong></h4><p>We need to understand the core concepts that make up an SSI solution. Let us start with the definition of an important concept, the <strong>Decentralized Identifier</strong> also called DID. DID establishes the certainty of an individual’s ID without revealing their Personally identifiable information (PII) data. DIDs do not require registration with a central authority and are cryptographically verifiable so that the DID owner can prove the ownership of their identity. These DIDs are used in a <strong>Verifiable Credential</strong> (VC) which is a digital claim used by the holders. It is the issuer that issues a credential that can be cryptographically verified making it a VC. Verifiable credentials can be used to build <strong>Verifiable Presentations</strong> (VP). Presentations are the data from the VC issued by issuer and shared with verifier but when this information is tamper-evident such that the authorship of data can be trusted after a cryptographic verification, then it is called a Verifiable Presentation. Another concept is the <strong>Wallets</strong> that are used by the holders to store all the private information and they are digitally signed. Data schemas also referred to as <strong>Schema</strong> in short are useful while enforcing a specific structure on a given collection of data and are issued and verified by the respective parties.</p><p>Finally, from a concept perspective, it is important to understand the <strong>Verifiable Data Registry</strong> (VDR) as it maintains the identifiers and schemas. There are many types of VDR that can be used including a distributed ledger. <strong>Blockchain</strong> is a type of distributed ledger. The SSI concept could be confidently implemented with the introduction of Blockchain. Choosing Blockchain as the VDR helps with a number of benefits for the parties involved including immutability, finality, security, the distributed consensus on the state of the ledger, etc. The Self-Sovereign Identities (SSI) are recorded as DIDs in the Blockchain. Also, it is good to understand and learn about the actors involved. Starting with <strong>Issuers</strong>, who are the qualified authorities that attests identity data. <strong>Holders</strong> have the authority to operate the identity data and can selectively share the identity data with the required service providers. <strong>Verifiers</strong> are the service providers who have an already established trust with the Issuer of the identity and based on that trust, the verifiers provide the service to the holders. Based on the role of the Issuer, there is an optional <strong>Examiner</strong> role. The primary responsibilities of the Examiner include gathering, examining, and validating the data to perform attestation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjOwd3_niV-vA1fBOzLDfQ.png" /></figure><h4><strong>Implementation of SSI Solutions</strong></h4><p>“A picture is worth a thousand words”. The figure above describes the core actors and the information flow in the credential verification process, implemented in a <strong>Permissioned implementation of SSI solution</strong>. There are two major implementations of SSI. The Permissioned and Permissionless implementations. There are a few other SSI solutions as well but the adoption of uPort and Sovrin have been on the rise.</p><p>· The <strong>uPort SSI solution</strong> is a Permissionless blockchain implementation, built on the Ethereum.</p><p>· The <strong>Sovrin SSI solution</strong> is a Permissioned blockchain implementation, built on Hyperledger Indy.</p><p>Based on the type of implementation, there are many components in each of the solution that has got be understood and analyzed in greater detail. Considering the agenda of this paper, we are not getting into the details of the implementation and its types.</p><h4><strong>Integrity of SSI Solutions</strong></h4><p>What is Integrity? Going by the fundamental definition in the English language, “Integrity is the practice of being honest and showing a consistent and uncompromising adherence to strong moral and ethical principles and values.” This integrity is extremely important when developing any software solutions and this <strong>integrity must be practiced by Self-Sovereign Identity Solutions </strong>as well. How? — Developers should make sure that there is absolute Integrity in following the principles, techniques and approaches that are recommended for SSI solutions. Additionally, verification and auditing aspects should make sure that these SSI solutions are completely adhering to the recommended best practices. From that perspective, there are 4 different areas that we can think about to measure the Integrity of the SSI solutions.</p><p><strong>a)</strong> <strong>Principles of SSI</strong></p><p>There are several resources which detail the laws of identity and principles of Self Sovereign Identity. These SSI principles fall into the major categories of <strong>core properties, user controllability, and security aspects</strong>. All these principles help in safeguarding the individual’s privacy and implement the concept of decentralized identity. It is important that these aspects must be completely verified in SSI solutions once implemented.</p><p>Let us study couple of examples. The first example is from the perspective of user controllability, where the <strong>users have complete control </strong>of their identities. That means, the decision to share the identity data to a particular entity is up to the individual. The SSI system must properly implement this principle by getting the <strong>consent of the user</strong> while sharing their data with another organization. It could be in the form of a notification that makes a user accept or reject such a kind of identity being shared or other different ways by which the consent is clearly sought from the user.</p><p>Secondly, from the perspective of the security category of SSI principles, <strong>persistence</strong> is an important quality to have in the SSI solutions. Going by the definition of persistence principle, identity could be disposed by the user, if they wish to do so and claims could be modified. This is often misunderstood; <strong>identity must be disposed but not deleted.</strong> Also, when identity is assigned, claims are assigned along with it. It should be noted that the identity and claims are separate. This means, the users must be able to successfully <strong>modify the associated claims</strong> but not the identity itself. So, there is a firm separation between claim and identity. Changing the claims helps in changing the identity’s associations. That way the user is able to dispose an identity when there is a need. All the above aspects must be clearly understood and designed in the SSI solution and thoroughly verified to make sure that the identity and claim are not tied forever.</p><p>Every principle of the SSI must be thoroughly looked into using the same approach. There might be possibilities where the principles could have been implemented but not fully or a weak implementation leading to compromise of privacy or not implementing the concept of decentralization properly.</p><p><strong>b)</strong> <strong>Components of SSI</strong></p><p>The components used in an SSI solution are completely dependent on the type of implementation of SSI. Based on the business requirements, one might choose to go with uPort or Sovrin SSI solution (the two most widely adopted by the industry). Now each of this solution has its own architecture and so the technological components used in them are different. For example, the use of <strong>Smart Contracts</strong> is specific to Ethereum based solutions which are followed in the Permissionless SSI implementations. When smart contracts are used, (say) when registering an identity, the code must be thoroughly tested for functionality or security bugs. Same is the case when protocols like the <strong>Inter-Planetary File System </strong>(IPFS) are used for storing and sharing data in a distributed file system, where a complete assessment is highly recommended.</p><p>There are some other components that are common to both the SSI implementations. Some of them being the use of the <strong>Wallet</strong> for storage of PIIs, keys, etc, use of <strong>Agents</strong>, that are a software program used to interact with another entity or <strong>DIDcomm</strong> the messaging protocol, used by entities for exchanging credentials.</p><p>When we apply the lenses of Integrity to DIDcomm, then, all the <strong>properties of DIDcomm</strong> must be carefully implemented — During any kind of peer-to-peer message exchange, DIDcomm implementation should follow all the <strong>security</strong> and <strong>privacy</strong> considerations, also it should be completely flexible to work on any underlying infrastructure, thereby making it <strong>interoperable</strong>. Also, DIDcomm specifications would be mostly implemented in json (though there could be other implementations as well), a through security review of the json protocol during any agent-to-agent communication is a must. All these are just a snapshot of what one needs to do for DIDcomm, but one must follow the complete guidelines recommended by the DIDcomm messaging specification from the <strong>identity foundation</strong>. Similar to how we analyzed the DIDcomm protocol, we have to apply the best practice during the designing and implementation of other components like the Wallet, Agent, the off-chain databases, etc in the overall SSI solution.</p><p><strong>c)</strong> <strong>Lifecycle of SSI</strong></p><p>There are three important <strong>SSI objects</strong> that form the foundation for a success identification of an individual in an SSI system — They are the key, identifier, and credential.<strong> </strong>It is essential to understand the lifecycle of these SSI objects and ensure they are implemented properly. The first SSI object is the key and <strong>Lifecycle of key</strong> deals with many aspects right from the creation to storage and deletion of the key. At first, this looks to be a very generic one with any security system, but the key pairs have an important role in the creation of the identifier itself. The user makes use of their public key for registering a DID on the Blockchain. This could be verified from the <strong>DID Document Object</strong> (DDO) that has all these details including the public key, the DID, claims describing the entity, etc.</p><p>It is important to note that, from a secure communication perspective, a user does not have only one public key, but the user creates a new public key for every party that they are interacting with so as to form a bilateral relationship. This means a new DID Document is created for every new entity that they establish a connection with. If an entity decides to end such a relationship with another, then the identity would be disposed. This can be accomplished by revoking the DID, or not updating the DID whenever the public key is rotated, etc. All these deal with the <strong>lifecycle of the identifier</strong>. The complete lifecycle aspects right from creation to revocation including all security and cryptographic principles must be well implemented. The third SSI object which is the <strong>lifecycle of credentials</strong>, must also be implemented by following the principles recommended for it.</p><p><strong>d)</strong> <strong>Regulations for SSI</strong></p><p>Solutions that implement SSI are an amalgamation of many technological components. The SSI implementations have some recommended regulations to be adhered to both from the perspective of the overall solution and individual components used to build them. Let us see an example in both the categories. When it comes to an individual component, we have seen that <strong>Blockchain</strong> has a major role in the infrastructure of the SSI system. The International Standards Organization (ISO) has been working on a standard for Blockchain and Distributed Ledger Technologies (DLT) which is the <strong>ISO/TC 307</strong>. A large part of this is under development but in future we will have a clearer picture on the mandatory requirements to be followed. Many aspects of the deployment and configurations of the interoperability, security, privacy, identity aspects, and the smart contracts of Blockchain will have a regulatory requirement.</p><p>From an overall SSI solutions standpoint, there are a few regulations that we may have to consider namely the <strong>data protection requirements, privacy related requirements</strong>, etc. The privacy regulations could vary based on the geography of the customer base, we have Personal Information Protection and Electronic Documents Act (PIPEDA) in Canada, California Consumer Privacy Act (CCPA) for California, etc. The data protection requirements would deal with the <strong>uniqueness of the electronic signatures</strong> of the individual using it, the association between the electronic signature and electronic document, etc. Global Data Protection Regulation (GDPR) is one that mandates these requirements. Some of the other requirements that GDPR mandates include accessing the individual’s personal data<strong> </strong>only after proper consent, individual’s <strong>right to erase the data</strong> when they are no longer needed for any processing purposes, etc. It is recommended that these regulatory requirements are incorporated into the design and configured, in addition to performing a <strong>regulatory audit</strong> which verifies if the solution is adhering to all of the mandatory points.</p><h4><strong>Final thoughts</strong></h4><p>The SSI solution could be a subset of a bigger solution interoperating with other existing solutions. These may include registration services, admin services, interoperable with other platforms, APIs that are being exposed, the web and cloud platforms where these services are launched, and many others. The future of SSI based solutions are going to be exciting with more and more enterprises developing these solutions, specific regulations being identified for easy adoption by the businesses and consumers and different variants in the solution implementations. But with all these developments, it is important that the complete solution is well designed by incorporating all the requirements, fully reviewed, and verified so that the integrity of the solution is intact and truly self-sovereign.</p><h4><strong>References</strong></h4><p>Some of the links which I have referred to during the course of writing this paper and other useful article to explore this topic.</p><p>1. <a href="https://identity.foundation/didcomm-messaging/spec/">https://identity.foundation/didcomm-messaging/spec/</a></p><p>2. <a href="https://www.w3.org/TR/did-core/">https://www.w3.org/TR/did-core/</a></p><p>3. <a href="https://www.w3.org/TR/vc-data-model/">https://www.w3.org/TR/vc-data-model/</a></p><p>4. <a href="https://www.windley.com/archives/2020/11/didcomm_and_the_self-sovereign_internet.shtml">https://www.windley.com/archives/2020/11/didcomm_and_the_self-sovereign_internet.shtml</a></p><p>5. <a href="http://www.lifewithalacrity.com/2016/04/the-path-to-self-soverereign-identity.html">http://www.lifewithalacrity.com/2016/04/the-path-to-self-soverereign-identity.html</a></p><p>6. “Design-Pattern-as-a-Service for Blockchain-Based Self-Sovereign Identity”, Article in IEEE Software · May 2020</p><p>7. “Self-Sovereign Identity Specifications: Govern Your Identity Through Your Digital Wallet using Blockchain Technology”, Article in 2020 8th IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (Mobile Cloud)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dfc792bad7b" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/integrity-of-self-sovereign-identity-5dfc792bad7b">Integrity of Self-Sovereign Identity Solutions</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning and Its Disruptions in the Information Security]]></title>
            <link>https://medium.com/swlh/machine-learning-and-its-disruptions-in-the-information-security-ac01e2168da0?source=rss-b95d52bf9dde------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac01e2168da0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Arun Prabhakar]]></dc:creator>
            <pubDate>Wed, 11 Nov 2020 17:36:37 GMT</pubDate>
            <atom:updated>2020-11-18T02:53:26.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ej6UvvuA565z-g1eZiCRw.jpeg" /></figure><p>One of the lovely things that excites a child is playing with the bubbles. The limitless fun and joy that children experience is quite entertaining and are memorable too. One such picture is what we see here with many beautiful bubbles, formed by the right mix of soap and water. I thought this photo is a good way to resonate and understand the impact of the disruptive effect of Machine Learning on Information Security. How is this picture analogous to the topic that we have? Well, let us see how. We all know how Machine learning has been disrupting every domain in the Information Technology Sector. InfoSec disciplines are no exception to this and there are many practioners and enterprises who are early adopters of this subject. And yes, a right mix of these two technologies have seen many emerging concepts and practices that are born, addressing several business challenges. I thought it would be good to share some of these and how Machine learning has been applied to solve such challenges in Information Security.</p><h3><strong>RegTech</strong></h3><p>RegTech also known as Regulatory technology is an emerging technological solution and mostly benefits the professionals in the audit and compliance domain. The primary focus of RegTech is to help businesses comply with regulations efficiently and thereby better the existing regulatory processes. There are many technologies that help in the making of RegTech solutions and in that ML and AI have a significant role. Though RegTech solutions are commonly used across the Information Technology space, the Information Security practioners leverage it for addressing many of the challenges in the domains of <strong>Risk Management, Identity Management </strong>and<strong> Compliance</strong> <strong>Monitoring</strong> and <strong>Reporting</strong>.</p><p>A common business scenario where RegTech solutions could become very helpful for the compliance professionals is when they must perform a regulatory audit that has most of its documents and requirements running into pages. It is a highly difficult process to go over these lengthy documents and identify actions for non-compliance when performed using traditional methods. Data from the past will have to be considered including relevant attributes while taking the decision about non-compliance and legal implications. This is not only a time-consuming activity but also adds more complexity to the whole process.</p><p>When RegTech solutions are applied to address this challenge, they make use of many <strong>Deep learning</strong> algorithms that can help in the processing of large volume of regulatory requirements. We can design many models using ensemble techniques and by leveraging <strong>Natural Language Processing</strong> (NLP) algorithms to help with text classifications, text extraction and come up with meaningful insights about the data so that the regulatory professionals can carry out their tasks more efficiently and it works out very effectively for organizations to complete the compliance audit in a relatively short time. RegTech solutions are commonly used by many of the FinTech companies in addressing the challenges coming from <strong>Know Your Customer</strong> (KYC)checks and <strong>Anti-Money Laundering</strong> (AML). The adoption of this in other domains are strongly on the rise.</p><h3><strong>Threat Intelligence</strong></h3><p>Threat intelligence is the concept that gives information about the context of threats, the motivation of threat agents, assets in our organization that are of interest to threat agents and the defensive mechanisms that needs to be taken in advance. Threat intelligence is considered by many as a component within Security Intelligence. Going by the definition given by <strong>Gartner research</strong>, “Threat intelligence is evidence-based knowledge, including context, mechanisms, indicators, implications, and action-oriented advice, about an existing or emerging menace or hazard to assets that can be used to inform decisions regarding the subject’s response to that menace or hazard.”</p><p>There are many disciplines within Security that deals with huge volume of data, this is especially true with the SOC analysts, DLP professionals and Firewall administrators who are at the<strong> frontline of defense</strong>. It is important for them to deal with the right characteristics of incoming data for further analysis or for integrating with other tools and not spend much time to segregate the information and the noise of the data coming from several different sources. This is when the <strong>data engineering </strong>techniques<strong> </strong>augmented with machine learning algorithms are very helpful. Many of the data cleaning, data analysis, data transformation and data visualization methods are not only a pre-requisite for the ML algorithms but also helps nail down the problem and focus on the objective. This is followed by the feature selection and feature extraction techniques that will help in building effective models and leads to <strong>discovering hidden patterns</strong>. Additionally, this also helps identify similar data so that the admins can cluster them for learning more insights.</p><p>There are also times when the <strong>Threat and Vulnerability management</strong> professionals use security tools for application assessment that may report high number of false alerts whereas using tools that are built by leveraging threat intelligence will help generate only the relevant threats. Models developed using machine learning algorithms are configured to report only threats that match a high degree of accuracy or any equivalent classification metrics that are configured, by analyzing different qualities of the past data. The concept of threat intelligence is very broad with many types of it implemented (from strategic to operational) to serve different purposes. There are many <strong>threat intelligence platforms</strong> also called TIPs that implement different concepts of Machine learning to build models that supports its complete lifecycle so that the process is repeatable.</p><h3><strong>Zero Trust</strong></h3><p>The term was first coined by <strong>Forrester Research</strong> about a decade ago and we have seen this concept being chosen by a lot of enterprises in the recent years. Zero Trust is a security concept that simply means not trusting anyone by default, irrespective of the entities accessing the organizational network from inside or outside of it. Zero Trust models makes it mandatory for any company approved devices, employees, partners, vendors who are considered insiders and customers to third parties accessing the network from outside to go through a thorough identity check and maintain strict access control procedures before accessing the resources within the network.</p><p>At the outset, the core of Zero Trust architectures involves using the fundamental security principles of <strong>identity, access control, and authentication</strong>. The model requires a well-defined process to be successfully executed. However, there are many implementation-level challenges, and it requires the use of advanced technologies to overcome them. The most common challenge is when giving access to entities for resources access within the network. There are so many parameters that are to be considered before letting the person access the network including user privileges, groups associated to, job type, time of day, location, history of commands executed, passwords changed in the recent past and many others. This aspect repeats every time for the same person for every access attempt and for any given user at any point in time. Therefore, any product implementing Zero Trust would have to handle tremendous amount of data.</p><p>As a Data scientist, the first step that I would do when I am assigned with such a task, after working on all the <strong>data wrangling</strong> techniques is that I will be using a classification algorithm since the task is to approve or deny access or to take the required action. Secondly, the target class is the one that got to be determined, hence I would use a <strong>Supervised learning</strong> algorithm to approach this. Finally, since we are dealing with many parameters, (one of the example) I can think about is by solving this is using <strong>Decision Tree classifier</strong>, where the target class can be predicted by involving all the attributes discussed above to take the right decision at every level in a sequential manner. Those implementing Zero Trust concept in the product do understand the value in applying ML as they learn continuously and adapt to the evolving. Context plays a vital role in making Zero Trust possible since we are dealing with the behavior of users and devices and this can be achieved by having AI/ML incorporated as we are dealing with lots and lots of data.</p><p>There are many IDaaS (Identity as a Service) providers that achieve Zero Trust Security in their product or offering by combining the power of Next-Gen Access and Machine learning techniques. <strong>Identity and Access Management</strong> (IDAM) professionals are hugely benefitted by this approach.</p><h3><strong>Security 4.0</strong></h3><p>We are all very familiar with the term Industry 4.0 which is playing a key role in the ongoing massive evolution in the manufacturing industry. There are about a dozen technologies that are being used to achieve the transformation using <strong>Industry 4.0</strong>. Cyber security is certainly one among them and plays a noteworthy role. Industry 4.0 is used to build many types of security solutions as well. There are three important technologies that are being used from a developmental and support viewpoint. They are<strong> Industrial IOT, Machine learning and Cyber security</strong>. We all know the role of Internet of Things that acts as the platform for all these devices to communicate with each other. Since there are many devices and humans involved it generates a lot of data, Machine learning and analytics are leveraged to extract and build the required intelligence. Finally, to design such systems with the highest level of security standards and regulations incorporated, cyber security technologies and tools are required.</p><p>Security Architects play a major role in designing these critical infrastructures especially those that deal with safety and security. They must be exceptionally skilled at using IOT and Machine learning when developing solutions and even while assessing such systems. <strong>Security 4.0</strong> is a practice where specialized security services are offered to secure products developed using Industry 4.0, also for security professionals to get involved, end-to-end in the designing and building of safety and security solutions in the cyber physical domain.</p><p>Remember the threat landscape of these systems including the defense mechanisms required are completely different from the traditional ones. Let us consider an intrusion detection and access monitoring system having a requirement to perform <strong>face verification</strong> of individuals gaining access, the Security Architect must be able to finalize which <strong>similarity learning</strong> algorithm works better for the given scenario and accordingly determine the <strong>classification metrics</strong> that would be relevant to the business including the right levels of threshold to determine access based on the regulatory requirements around security labels so that right course of action is taken. Security 4.0 is an emerging practice in product manufacturing companies.</p><h3><strong>Others</strong></h3><p>There are other areas of security where machine learning has been found to be very impactful. One of them is SOAR, which is <strong>Security Orchestration Automation Response</strong>. SOAR helps many elements involved in the security incident and response process collaborate, to streamline the process and achieve a better outcome. SOAR solutions are combined with SIEM tools in such a way that the former focuses on orchestrating and automating different third party-tools while the latter helps collate and analyze the data to generate alerts precisely. Machine Learning is a critical component of SOAR that helps at every stage of the process like in analyzing incidents, triaging efforts, or when a remediation is suggested, or for that matter when an escalation is made.</p><p><strong>User Entity Behavior Analytics</strong> is another area where we are witnessing many new developments. UEBA primarily helps with the discovery of insider attacks. It uses the concept of Machine learning to understand the actions of (trustworthy) users within the network, find out anomalies in their routine behavior that may lead to a possibility of threat. UEBA works alongside Security Incident and Event Management (SIEM) and Data Loss Prevention (DLP) tools.</p><h3><strong>Final thoughts</strong></h3><p>This may not be all. This is just a collection of some of the common and emerging practices in the Information Security domain. The business cases which I have discussed are the elementary ones to make it easier for anyone to follow along. They provide a bird’s eye view without getting into technicalities. There are many complicated scenarios where researchers are joining hands to build great platforms and products. Though, the benefits are many, there are challenges as well. There are challenges for engineers/consultants from execution purview, challenges faced by business from an adoption standpoint but to address them we have <strong>Security Data Scientists. </strong>Ohh, who are they? Well, in short, they are the people skilled at Security, Data science and Machine learning, working to build many of these practices. What they resolve, what are their responsibilities, how they are benefitting the business, what are the challenges they face … well, that would be a separate blog altogether.</p><p>For now, let’s predict how long these bubbles travel along, do they grow bigger, do they merge with other bubbles or do they even burst. Just like the kids having fun with them, for any technology practioners, it is going to be a fantastic time working on the applications of Machine Learning in Information Security domain, not just from an execution and usage standpoint but also in researching, contributing to the development of new practices and importantly sharing with the wider community.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac01e2168da0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/machine-learning-and-its-disruptions-in-the-information-security-ac01e2168da0">Machine Learning and Its Disruptions in the Information Security</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>