<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arnab Ray on Medium]]></title>
        <description><![CDATA[Stories by Arnab Ray on Medium]]></description>
        <link>https://medium.com/@arnabray?source=rss-a76f276b5ffb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y3hnuVCbYUeitO1LzbCK_g@2x.jpeg</url>
            <title>Stories by Arnab Ray on Medium</title>
            <link>https://medium.com/@arnabray?source=rss-a76f276b5ffb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:36:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arnabray/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cybersecurity for AI/ML Systems]]></title>
            <link>https://medium.com/@arnabray/cybersecurity-for-ai-ml-systems-b4d73ac7fcc4?source=rss-a76f276b5ffb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4d73ac7fcc4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Arnab Ray]]></dc:creator>
            <pubDate>Mon, 04 Oct 2021 23:54:55 GMT</pubDate>
            <atom:updated>2021-10-06T16:19:24.444Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometimes in order to understand what something is, it’s easier to start with what it is not. So, before we understand the unique aspects of securing an AI/ML system, we should first define what we mean by a non-AI/ML system. I am going to assume that the reader knows the basics of cybersecurity for a non-AI/ML system, which is why defining a non-AI/ML system is a good place to start. Once I do that, I can then provide some light on the “delta” between non-AI/ML systems and AI-ML systems when it comes to thinking about threats, vulnerabilities, and protective cybersecurity controls.</p><p>Let us start by considering a human designing a system. Let’s call the human designer, inspired by the Matrix universe, “The Architect”. The Architect is assumed to be benign, in that he does not want his designed system to fail or to be attacked. The system development process follows a sequence of refinements — -first there is what the Architect has, in his mind, the “idealization” of the system he wants to design, what students of Philosophy would recognize as similar to the Platonic notion of a “form”, the “perfect” realization of the Architect’s idea. This then becomes “user needs”, which is then refined to “system requirements”, which then is refined to “sub-system requirements” (e.g. hardware, electrical, mechanical, software requirements), when is then refined to design, and then finally to implementation. The implementation captures, in a perfect system, the initial idea of the Architect exactly as the idea is, nothing less and nothing more. Since the Architect is a human being, we can say that <em>the full specification of the system behavior has been defined by a human</em>.</p><p>We used the word “perfect system” in the previous paragraph. Of course, nothing in the world is perfect. Systems do not always behave in ways the humans who designed them thought they would. These deviations, as we all know, are called defects or “bugs”. Consider that old joke on the defensiveness of system developers; “It’s not a bug, it’s a feature” — -what the developer is saying is that the behavior is not un-intended, it was part of the developer’s intent all along, hence what you just called a bug is anything but.</p><p>In AI/ML systems, and this is where it diverges from non AI/ML systems, human beings let the system decide, to an extent, what the behavior of the system should be. How the system behaves is thus <em>no longer fully characterized by any specification that was defined by human beings</em>. One way of thinking about AI/ML systems is that here the machine is writing the specification of its own behavior, allowing for the specification of system behavior that a human mind cannot even define.</p><p>A story will make this clear. AlphaGo was a ML system for playing a game called Go and it was trained by humans. It comprehensively defeated the world’s best human player of Go. Then, AlphaGo continued to learn by playing games against itself. It was then pitted against another more advanced ML system, AlphaGoZero, which trained itself (i.e. no human intervention), starting from a blank slate or <em>tabula rasa</em>. AlphaGoZero defeated AlpaGo in a series of 100 Go Games — -100 to 0 ! Now, here below, is <a href="https://www.theatlantic.com/technology/archive/2017/10/alphago-zero-the-ai-that-taught-itself-go/543450/">an extract from an article in the Atlantic.</a></p><blockquote>Since May, experts have been painstakingly analyzing the 55 machine-versus-machine games. And their descriptions of AlphaGo’s moves often seem to keep circling back to the same several words: Amazing. Strange. Alien.</blockquote><blockquote>“They’re how I imagine games from far in the future,” Shi Yue, a top Go player from China, has told the press. A Go enthusiast named Jonathan Hop who’s been reviewing the games on YouTube calls the AlphaGo-versus-AlphaGo face-offs “Go from an alternate dimension.” From all accounts, one gets the sense that an alien civilization has dropped a cryptic guidebook in our midst: a manual that’s brilliant — or at least, the parts of it we can understand.</blockquote><p>So, what we see here is that a AI/ML system devices its own algorithms and its strategies in a way that human beings can not only just not think up of, but cannot even comprehend when presented to them. While this opens up exciting new applications that push the limits of what we know to be possible, it also raises various philosophical, moral, ethical questions (which are outside the scope of this post), and leads to one very specific system behavior conundrum (which is in-scope).</p><p>And it is this. Since the behavior of the system is no longer constrained by what a human being fully specified a-priori, what is correct behavior and what is incorrect? “It’s not a bug, it’s a feature” is no longer a tired old joke, but a fairly concise encapsulation of this dilemma — — when the system’s behavior changes over time, following algorithms and strategies no human brain could have devised, how do we know whether a new, unexpected behavior is just a bug or is indeed a feature, a feature defined not by a human being, but by the non-human intelligence of the system itself?</p><p>This above is the root cause for the cybersecurity “uniqueness” of AI/ML systems.</p><p>In non AI/ML systems as long as the attacker cannot compromise system integrity (e.g. change code on persistent storage, or get their own code running in memory or seed vulnerabilities during the system development process in hardware or software or make the developer use a system component that was compromised already), they cannot force the system to do something that the Architect did not intend it to. Since, the system perfectly captures the Architect’s intent, and since the Architect is benign (remember that’s how I started the example), the system is too. And so as long as the integrity of the system is protected, the system remains secure.</p><p>In AI/ML systems however, because the system can legitimately change its own behavior, the integrity of the full range of system behavior cannot be assumed to be “locked down” as it is for non AI/ML systems. All the best cybersecurity protections for non AI/ML systems still will not be able to protect the system from threats that target the part of system behavior that is allowed to change (Those who have seen the Matrix movies know that Agent Smith attained awareness, and hence was no longer part of the Architect’s specification of the Matrix, and that explains the third movie in the trilogy).</p><p>So how do attackers compromise non AI/ML systems? Through the exploitation of vulnerabilities. Why do vulnerabilities exist? Because, no system is ever a perfect representation of the human designer’s intent. Any system, even one that perfectly satisfies all requirements, allows for more behavior than what the designer intended. If these deviations from the ideal behavior can be exploited by an attacker, well there you have it, these become the vulnerabilities.</p><p>One way of detecting these vulnerabilities before a non AI/ML system goes into production is to do penetration testing. Here the “red team” member uses tools, techniques and strategies used by an attacker to simulate the kind of analysis the system will be subject to by a sophisticated attacker, and vulnerabilities discovered, before the system goes out into the world. In AI/ML systems, when the system’s behavior can legitimately change, how do we ensure that vulnerabilities are not introduced during operations, vulnerabilities that were not there when the penetration test was conducted? In other words, how do we know whether a deviation from expected behavior observed in a AI/ML system is a result of an attack (one that exploits a system bug) or just the AI/ML system doing something a human being could not imagine it doing (a feature that the system “learned”) ?</p><p>So now, let’s talk about threats specific to AI/ML systems.</p><p>An important class of threats to AI/ML systems work by exploiting blind spots and imprecisions of the core AI/ML mechanisms that drive its intelligence. These core AI/ML mechanisms, the so-called “smarts” of the system, the part that learns and adapts, that part is called the model of the system. There are YouTube videos of ML-based image classification systems identifying a pig as a baboon or a cat as an avocado, based on an attacker introducing targeted noise in the source image. Malware developers work hard to “escape” ML-based malware detection algorithms by ensuring their malware is engineered to have properties that will trigger the malware tool’s model to classify it as “benign”. These threats where the attacker adjusts the input to the model to make it do something that is to the attacker’s advantage are called evasion threats.</p><p>Evasion threats work in three ways:</p><ol><li><em>By reducing confidence in the result of the model</em>: Here, the correct behavior is as follows: an input “x” would be classified by the model as belonging to class M with confidence C. The attacker changes “x” to “y” (usually a small change that would pass other tampering controls) such the confidence in the output of the model changes to C’. If the output of the model is to drive some action, reducing the confidence of its result could prevent that action from happening. As an example, consider a AI/ML medical system that does automated image analysis for cancer detection. The model is provided a certain image of a colonoscopy with a tumor. It identifies the presence of a tumor with a confidence of 90%. Now the attacker adds noise to the colonoscopy image, keeping the input identical to the human eye. However, the confidence in the output of the model goes down to, let’s say 10%. If the system triggers an alert to a physician at 90%, that might not be triggered at 10%.</li><li><em>By misclassifying an input</em>: Here, the correct behavior is that an input “x” would be classified by the model as belonging to class M with confidence C. The attacker changes “x” to “y’” such that the output of the model classifies x as belonging to class M’ with confidence C’. As an example, consider a AI/ML system that detects weapons through analysis of X-ray images at a security checkpoint. An attacker, who knows the vulnerability in the model, pastes certain attachments to the weapon in order for the AI/ML system to classify the weapon as something over than a weapon.</li><li><em>By misclassifying an input into a class that the attacker wants the input to be classified into</em>: Here, the correct behavior is that an input “x” would be classified by the model as belonging to class M with confidence C. The attacker changes “x” to “y’” such that the output of the model classifies x as belonging to class M’ with confidence C’, w<em>ith M’ being a class that is determined by the attacker.</em> In the previous example, the attacker wanted to ensure that the weapon is not identified as such, what it is identified as (a toy, a jar of apricots) is not a matter of concern. Here, not only is the attacker aiming to evade classification, but the attacker also wants the classifier to classify the input in a class of their choice. For example, consider a malware detection engine which analyzes binaries and tags them into three classes — -benign, malware and suspicious. The attacker wants to modify binaries such that genuine malware gets tagged as benign. The attacker does not want the system to tag their malware as suspicious perhaps because those binaries tagged as suspicious are quarantined and not allowed to execute, whereas those tagged as benign are allowed to immediately run on the target system.</li></ol><p>Evasion threats change model behavior by modifying inputs to the model. Another important class of threats to AI/ML systems is to compromise the model itself, in order to make the system do things the attacker wants it to. This can be done in two ways.</p><ol><li><em>Backdooring the model</em>: Roughly analogous to seeding backdoors in the code or in the hardware for non AI/ML system development, model backdooring consists of replacing the benign model M with an attacker modified model M’, which is nearly identical to M, but with respect to very specific inputs, M’s takes decisions that are advantageous to the attacker. Unlike model evasion, here the input is not being changed, it’s the model itself that has been. An example of backdooring would be to add rules to an decision engine such that in certain situations, malicious rules fires, overriding other rules like “Do no harm”.</li><li><em>Poisoning the model</em>: Here the inputs to the model are being altered, as it is done in evasion, but the purpose is not to make the model do something it should not (as it is for evasion), but for it to learn something it should not. An example of model poisoning would be providing a set of malware files to a malware-detection tool while it is learning, except instead of labeling them as malware they are labeled as benign.</li></ol><p>The threats described so far have all worked on making the AI/ML system do something it was not supposed to. Not all threats violate the integrity of the AI/ML system though, some of them exist purely to find out what the underlying model is. A trained model of a AL/ML system is a very important piece of intellectual property in today’s world, the business models of some of the world’s most successful corporations (think search engines, social media sites, fraud detection for insurance companies and banks) depend vitally on high-precision AI/ML systems.</p><p><em>Model theft</em> consists of reverse-engineering a trained model in totality (e.g. let’s find out how Facebook decides a post is misinformation) or extracting information enough to reconstruct the model, not to the fidelity of the original, but acceptable enough for the purpose of the attacker. This is roughly analogous to the classic plaintext-ciphertext attacks we read about in Security 101, in that by providing a large number of inputs and observing the outputs, the attacker is trying to extract a secret or get some information about it. While model stealing is an activity worth doing on its own, it is also used as part of the reconnaissance phase of any attack that will later use evasion. In order to game the model, the attacker needs to get as much information about it to craft malicious input, right? If I am going to be crafting fake news items for the purpose of spreading mis-information on Facebook, my first activity would be to try to reverse-engineer the model used by Facebook to automatically tag a news item as fake. Once I have this model, or one that is close to it, can I then start finding out vulnerabilities and crafting evasion threats.</p><p>So what are the protections against threats specific to AI/ML systems?</p><ol><li><em>Guard the model</em>: As I just said, model theft is the first stage of evasion attacks. AI/ML systems operating online and accessed over the Internet should be protected by cloud/on-premise infrastructure, platform and application controls (just like a non AI/ML system should), accessed only through Application Programming Interfaces (API)s, ideally through registered and authenticated interfaces, with limits being applied to API calls. The requests should also be monitored, such that specific patterns of leaching are detected and cut-off. When AI/ML systems are on end-points (e.g. drones, medical embedded devices), such protections are usually more difficult to enforce — -the attacker can procure the end-point, and retrieve the model off the flash storage. That’s why AI/ML end-point devices should protect their model through encryption, with encryption keys being stored in secure enclaves, and multiple levels of flash protection (disc, file-system and application level).</li><li><em>Guard the training process:</em> AI/ML systems can be trained offline or online. In offline training, the model is trained during system development. Once training is complete, the model is “locked”, i.e. it can no longer modify itself. The system is then deployed. It can of course collect data, but the data is not used to change the model during operations. In the next iteration of system development (version i+1 of the product), the model is “unlocked”, trained again with the new data, and “locked” again, and the (i+1) th version of the system deployed. Training here takes place in a trusted, controlled dev environment, and the newly learned behavior is tested, before the system is used to operate. Offline training provide a certain degree of security protection— -access to the training environment can be restricted by least privilege, the training data can be reviewed and obtained only from trusted sources, and whatever new learned behavior the model exhibits can be tested to check for adverse behavior, including behavioral fuzzing. Many regulated products like medical devices typically use AI/ML in this restricted mode, because regulators would hesitate to approve devices whose behavior can change autonomously after approval. However, online learning, where the AI/ML system learns “on the job”, changing its behavior dynamically, is the use case in which the full potential of AI/ML systems is unlocked; where, by definition, the training process cannot be guarded in the way it can be done in offline training.</li><li><em>Guard the source of data</em>: Where possible, cryptographically authenticate sources of data that are ingested by the model. In certain applications within closed eco-systems, for example within networks where every node authenticates to the other using signatures from a trusted authority, and no unauthenticated communication is allowed, one can presume that if the authenticated nodes are benign, then the AI/ML system is protected. This “if” is a big “if”, even in case of authenticated data sources, because authenticated users can be threat agents. For example, consider a AI/ML system that detects tumors based on X-Rays sent to it by different authenticated clinics. What is the guarantee that an authenticated user in one of those clinics is not launching evasion attacks by sending altered X-Ray images? Twitter can authenticate users, link new accounts to phone numbers to guard against normal users creating many accounts, but how can it protect itself from attacks launched by nation states, where account farms (endless racks of mobile phones, actual or emulated) are used to create millions of accounts, which can then try to steal proprietary models or launch evasion attacks at scale (e.g. to get Twitter to trend topics of their choice, which their model should have classified as “spam” but classifies as “trending”)?</li><li><em>Put AI/ML Systems in safety cages</em>: In use-cases of AI/ML systems with safety implications, safety cages are controls imposed outside of the model to ensure that any “bad behavior” that is learned does not lead to harm. Safety cages could be as simple as human oversight, where a human being reviewing the result of a model before a decision is taken based on it. In evasion attacks of the kind where the image of a cat is superposed with noise such that it gets classified as an avocado (true story), human oversight is the simplest way to detect the attack and prevent any wrong decisions being taken on the basis of a “gamed” result. But then again, doesn’t that defeat, to an extent, the very intent of AI/ML systems, if a human being has to review results? Safety cages can be more complex though, like restricting range of movement of a AI/ML controlled robotic arm, such that intelligent action is allowed only in certain benign zones of operation. Of course, the success of such approaches depend on being able to define benign zones, which is not always possible. Then there is the problem of making safety cages too rigid, at which point of time the model becomes useless (an early version of a smart prototype car I rode would detect a tree on the side of the road, and if it was close enough to the curb, the tree would be in the safety cage of a detected obstacle in front, and the would stop). Or you can go the other extreme, and make safety cages too permissive, but then unsafe behavior may be allowed (i.e. a tree that is in front would not make the car stop).</li><li><em>Use resilient AI/ML algorithms: </em>There is a lot of research (aka many many PhD theses) going on in academia on “adversarial ML”, which is about developing robust learning methods that have multiple mathematical protections against evasion attacks. The best way to intuitively think about these protections (and I know it is a very imprecise and trivial characterization, but bear with me) is that if the attacker can identify some mathematical aspect of the model that they can “game”, another mathematical aspect of the model will catch it or prevent it from happening. AI/ML models are characterized by “risk” — -how likely is it that it will get its intelligence wrong? Adversarial risk is the adaptation of risk in a cybersecurity context — -how likely is it, when there is a motivated attacker trying their best (or their worst, depending on your perspective) will a system will get its intelligence wrong?</li></ol><p>Well that’s it for now. Adversarial ML is a fast developing field of study, with multiple applications in multiple domains, one of the most important of which is in medical devices and healthcare, where my personal interests are. What makes it especially interesting is that in a regulated industry like healthcare, it is not just about making your system secure but in convincing regulators, customers and the community in general that the system is secure. As AI/ML systems evolve in the coming years, I will personally be interested in advances in proofs of adversarial robustness of AI/ML systems, and how it shapes the security assurance argument.</p><p>For now, thank you for reading. And if all of this has piqued your interest, and you want to know more about medical device cybersecurity for non-AI/ML systems, may I point you to my book “<a href="https://www.amazon.com/Cybersecurity-Connected-Medical-Devices-Arnab/dp/0128182628">Cybersecurity for Connected Medical Devices</a>” published by Elsevier Publications?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*DUg_1rPKq4ouPX-SSKVt0Q@2x.jpeg" /><figcaption>That’s My Book</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4d73ac7fcc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>