<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by HarrietHacks on Medium]]></title>
        <description><![CDATA[Stories by HarrietHacks on Medium]]></description>
        <link>https://medium.com/@harrietfarlow?source=rss-bc3fb5455872------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0eT0ZIEbcL5XMDD_KQ95Kw.jpeg</url>
            <title>Stories by HarrietHacks on Medium</title>
            <link>https://medium.com/@harrietfarlow?source=rss-bc3fb5455872------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:27:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harrietfarlow/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Intro to AI Security and Threat Modelling 101]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-and-threat-modelling-101-16a305d862d6?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/16a305d862d6</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[adversarial-ml]]></category>
            <category><![CDATA[ai-security]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sun, 05 Apr 2026 14:07:53 GMT</pubDate>
            <atom:updated>2026-04-05T14:07:53.363Z</atom:updated>
            <content:encoded><![CDATA[<p>Do you want to understand not just <em>what</em> we need to secure AI systems from, but develop a more grounded intuition for <em>how</em> and <em>why</em> those threats emerge in the first place?</p><p>That is the aim of this blog series. I’ve <a href="https://medium.com/@harrietfarlow/intro-to-ai-security-part-1-ai-security-101-b8662a9efe5">already written an AI Security 101</a> series, but I’ve decided to write some refreshed blogs for 2026!</p><p>Rather than presenting lists of attacks and defences, I want to build up a way of thinking about AI systems that makes those things feel more intuitive, particularly if you are coming from a cybersecurity background and trying to map this new domain onto concepts you already understand.</p><p>A quick note before we start: the diagrams used throughout this series are drawn from my <a href="https://aisecurityfundamentals.com/"><strong>AI Security Fundamentals Course</strong></a>. I’ll be using them here in a slightly less formal, more exploratory way. I also share information on my YouTube channel <a href="https://www.youtube.com/@HarrietHacks">HarrietHacks </a>— always keen for feedback on what kind of content you want to see!</p><h3>Why I’m writing this</h3><p>In 2023, I left my role at the Australian Signals Directorate, where I was Acting Technical Directorate at the AI Hub. For context, ASD is Australia’s equivalent of the NSA in the US or GCHQ in the UK, and my work sat at the intersection of artificial intelligence and national security.</p><p>Alongside that role, I had been working on a PhD in adversarial machine learning, which focuses on ways that machine learning models can be manipulated, misled, or otherwise behave in unintended ways under adversarial conditions.</p><p>Around that time, I began to notice a shift that I found difficult to ignore. Within intelligence and research communities, there were increasingly strong signals that cyber attacks targeting AI systems were likely to become a practical and material issue, rather than something confined to academic literature. At the same time, outside of those relatively small circles, there was very little concrete discussion about how these systems would actually be secured in real-world environments.</p><p>What I found particularly frustrating was not just the technical gap, but the systemic one. So I left, and started a company.</p><p>I chose to build it as a for-profit organisation quite deliberately. Part of that decision was about independence — being able to work on problems without being constrained by existing institutional structures. But it was also about pragmatism. Requiring a product to be adopted by real organisations forces a level of rigour that is difficult to achieve when relying on grant funding or producing research and policy outputs that may never be operationalised.</p><p>The first couple of years were hard. It turns out ‘shouting into the void’ is not a recognised go-to-market strategy. I worked with a small number of organisations that were already thinking ahead, helping them design AI security training programmes and internal frameworks. But for the most part, it felt as though the problem had not yet fully registered. AI was seen as an opportunity, sometimes a risk, but rarely as something that required immediate and structured security thinking.</p><p>Around a year ago, that changed dramatically. Agentic systems have become significantly more accessible, and a number of incidents have started to bring attention to the ways in which these systems can fail or be exploited. At the same time, many vendors have begun integrating AI capabilities — including increasingly autonomous behaviours — into enterprise products, often without providing customers with sufficient visibility into how those changes affect their risk profile.</p><p>I now see cybersecurity practitioners are actively trying to understand how AI systems fit into their existing threat models, often with limited resources and very little practical guidance.</p><p>One thing that has become clear to me through this process is that access to useful, grounded information in this space is still quite uneven. Not everyone has the ability to attend specialised training or engage deeply with academic literature, and much of the available material either remains too abstract or too fragmented to be immediately useful.</p><p>This series is an attempt to address that gap.</p><p>It draws on material from my <a href="https://aisecurityfundamentals.com/"><strong>AI Security Fundamentals Course</strong></a>, as well as my upcoming book <a href="https://nostarch.com/practical-ai-security"><em>Practical AI Security</em></a> (No Starch Press), but is structured in a way that I hope makes it accessible to anyone who wants to engage with the topic more deeply.</p><h3>What do we actually mean by an “AI system”?</h3><p>Before we can meaningfully talk about securing AI systems, it is worth pausing to clarify what we are referring to.</p><p>There are tons of definitions of AI, but one worth knowing (because it’s how Governments define AI) is from the OECD, which describes an AI system as:</p><blockquote><em>A machine-based system that, for a given set of objectives, can generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments.</em></blockquote><p>This is a useful starting point, but in practice it tends to obscure an important distinction that becomes highly relevant from a security perspective.</p><p>When people refer to an “AI system”, they often implicitly mean the machine learning model. However, the model is only one component of a broader system.</p><h3>The model is not the system</h3><p>To make this more concrete, consider a simple analogy. Imagine a robotic vacuum cleaner. We’ll call it Vroomi, because all good technical explanations benefit from unnecessarily cute naming.</p><p>Vroomi is, in many ways, living its best life — moving around, making decisions, and occasionally getting stuck under furniture. Vroomi moves through an environment, senses features such as the presence of dirt, and makes decisions about where to move next. It includes sensors, control logic, actuators, and often some form of user interface.</p><p>Somewhere within that system is a component that could reasonably be described as its “brain” — a model that processes input and produces decisions.</p><p>But that model is not Vroomi.</p><p>Vroomi is the entire system: the sensors that collect data, the hardware that enables movement, the software that orchestrates behaviour, and the interfaces through which it interacts with users. The model is the brain, but unfortunately Vroomi also has wheels, sensors, and the ability to fall down stairs, which the brain alone does not account for. For instance, if Vroomi runs directly into a wall, this is not necessarily a failure of intelligence, just system design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*67MttYDgDeo72s9mLa02Mg.png" /><figcaption>An AI System — the model, and everything else!</figcaption></figure><p>This distinction is captured in the diagram here. On one hand, there is the <strong>machine learning model</strong> — the core component responsible for transforming inputs into outputs.</p><p>On the other, there is the <strong>AI system</strong>, which includes everything that operationalises that model:</p><ul><li>APIs</li><li>data pipelines</li><li>integration layers</li><li>user interfaces</li></ul><h3>Why this distinction matters</h3><p>From a security perspective, this separation is important for at least two reasons.</p><p>Firstly, the types of vulnerabilities differ significantly. At the level of the model, we are concerned with phenomena that arise from the statistical and optimisation properties of machine learning systems.</p><p>At the level of the broader system, however, many vulnerabilities resemble those encountered in traditional cybersecurity contexts, including insecure interfaces, insufficient access controls, and unintended data exposure through integrations.</p><p>Secondly, the distinction has implications for responsibility. Model-level issues are often the domain of machine learning engineers and researchers, whereas system-level issues fall more naturally within the remit of security and software engineering teams.</p><p>In practice, many of the most consequential risks arise at the intersection of these domains, where responsibilities are less clearly defined.</p><h3>What makes AI systems different?</h3><p>Although AI systems are, in many respects, conventional software systems, there are several characteristics that introduce new challenges.</p><p>They are <strong>probabilistic</strong>, meaning that the same input does not always produce the same output, which complicates testing and assurance.</p><p>They are <strong>highly dependent on data</strong>, both in terms of training data and runtime context, which effectively expands the attack surface to include data sources and pipelines.</p><p>And they <strong>interpret input</strong>, rather than simply executing it, which creates opportunities for adversaries to influence system behaviour in ways that are less direct than traditional exploits.</p><p>These characteristics do not make AI systems inherently insecure, but they do change the nature of the threats we need to consider.</p><h3>A simple way to think about AI threats</h3><p>One way to structure these threats is through three broad categories:</p><ul><li><strong>Disrupt</strong></li><li><strong>Deceive</strong></li><li><strong>Disclose</strong></li></ul><p>This framing aligns reasonably well with the CIA triad — availability, integrity, and confidentiality — but approaches it from the perspective of attacker objectives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bR9gYHEst4t-vEVHz6yv6g.png" /><figcaption>The 3D model</figcaption></figure><p><strong>Disrupt</strong></p><p>Disruption involves preventing the system from functioning as intended. In AI systems, this might include overwhelming the system with computationally expensive inputs, exhausting resources, or causing failures in upstream or downstream components of the pipeline.</p><p><strong>Deceive</strong></p><p>Deception involves causing the system to produce incorrect or misleading outputs. This could take the form of adversarial inputs that alter model predictions, or prompt injection techniques that manipulate the behaviour of language models. Notably, these attacks often do not “break” the system in a traditional sense, but instead exploit its interpretive nature.</p><p><strong>Disclose</strong></p><p>Disclosure involves extracting information that should remain confidential. This may include training data, internal system prompts, or sensitive information accessible through the system.</p><h3>Security vs safety</h3><p>It is also important to consider the direction of these threats.</p><p>The categories described above are focused on threats coming <em>into</em> the system — from users, attackers, or other systems. This is typically what we refer to as <strong>AI security</strong>.</p><p>There is a parallel set of concerns that operate in the opposite direction: risks that emerge from the system and affect the environment. These include issues such as unsafe outputs, bias, and, in more extreme cases, longer-term alignment risks.</p><p>These concerns are generally grouped under the term <strong>AI safety</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C_FPRDmub97yNgTxkxPMWA.png" /><figcaption>AI safety vs security</figcaption></figure><p>While AI security is often treated as a subset of AI safety, the distinction is meaningful in practice.</p><p>There is a significant difference between an adversary deliberately exploiting a system to extract sensitive information, and a system producing harmful outputs as a result of its design or training.</p><p>This distinction becomes even less clear in the context of agentic systems, where AI systems may act autonomously, interact with other systems, and in some cases function as both the target and initiator of actions.</p><p>Now we will look at two concrete system architectures and examine how these categories of threats manifest in practice, and what that implies for how we secure them.</p><h3>Threat Modelling</h3><p>One of the most useful habits you can build when thinking about AI security is simply to <strong>draw the system you’re working with</strong>. Not in a perfect, architect-level diagram, but enough to understand where the AI actually sits, what it connects to, and how data moves through it. A surprising number of issues come from the fact that people know they are “using AI”, but don’t have a clear mental model of where it lives or what it can access.</p><p>In practice, most systems are less ‘clean architecture diagram’ and more ‘collection of things that somehow work together’.</p><p>The two diagrams below are simplified examples of this kind of thinking. They’re not meant to be complete or technically precise, but they illustrate two different perspectives: how an AI system is used, and how it is built.</p><h3>Threat Model 1: The application layer</h3><p>In the first diagram, we’re looking at a fairly typical application setup.</p><p>A user (me) interacts with an application on their phone. That application sends a request over the internet to a <strong>model API endpoint</strong>, which is just a service that hosts the AI model and returns a response. That endpoint then interacts with other parts of the system, including an <strong>authorisation service</strong> (which checks whether the user is allowed to perform a given action) and internal <strong>data stores</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nzDBOG7z_z326tx1tlIw8g.png" /><figcaption>Threat Model 1</figcaption></figure><p>The dotted boxes represent different <strong>trust zones</strong>, which is just a way of grouping parts of the system based on how much we trust them. The external client zone (the user and their device) is low trust, because we assume inputs could be malicious. The internal services zone (where data and user profiles live) is high trust, because it contains sensitive information and is more tightly controlled. The API layer sits somewhere in between.</p><p>What’s important here is not memorising the components, but noticing where <strong>boundaries</strong> exist — particularly where data moves from a lower-trust area to a higher-trust one.</p><p>In a traditional system, we would worry about things like input validation, authentication, and access control at these boundaries. In an AI system, those concerns still exist, but we also introduce something new: the model itself is interpreting the input. That means an attacker isn’t just trying to break the system, but potentially influence how the system <em>understands</em> a request.</p><p>This is where issues like prompt injection or data leakage start to emerge, particularly if the model is allowed to query internal data or trigger actions on behalf of the user.</p><h3>Threat Model 2: The development and deployment pipeline</h3><p>The second diagram looks at the same system from a different angle — not how it is used, but how it is created and deployed.</p><p>Here, we see components like <strong>data ingestion services</strong> (which collect and import data), a <strong>training environment</strong> (often a notebook or experimentation setup where engineers build and test models), and a <strong>deployment pipeline</strong> that moves the model into production. There are also <strong>third-party libraries</strong> and external data sources involved, which are common in most real-world systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXHqaC0eQN2fYMvLjX8FYA.png" /><figcaption>Threat Model 2</figcaption></figure><p>Again, the system is divided into zones. The experimentation and development environment is labelled as lower trust, which reflects the reality that these environments are often more flexible, less controlled, and involve a mix of data, code, and external dependencies. The production environment, where the model is deployed and accessed by users, is higher trust.</p><p>The key observation here is that the model — which ends up making decisions in a high-trust environment — is often created in a much lower-trust one.</p><p>This introduces a different class of weaknesses. For example, if training data is manipulated (<strong>data poisoning</strong>), or if a dependency in the training pipeline is compromised, the resulting model may behave incorrectly once deployed — even though nothing appears to be “wrong” at runtime. These are supply chain-style risks, and they’re easy to miss if you only focus on the deployed application.</p><p>Once you can see the system clearly, the problems become a lot less magical and a lot more fixable.</p><h3>Bringing it together</h3><p>We’ll come back to both of these diagrams in much more detail later in the series.</p><p>For now, the goal isn’t to understand exactly what every component does, but to start building an intuition for where AI systems sit within a broader architecture, how they interact with other systems, and why those interaction points — particularly across trust boundaries — are where many of the most interesting security problems tend to arise. At some point you realise the system is working exactly as designed — just not as intended.</p><p>And security controls are only as strong as the incentive to keep them in place... but we’ll get to that later.</p><p>If you link this back to the earlier sections, you can start to see how these diagrams make those ideas more concrete. The distinction between <strong>AI security</strong> (threats coming into the system) and <strong>AI safety</strong> (risks emerging from the system into the environment) becomes easier to reason about when you can actually see where those interactions happen. Similarly, the categories of <strong>disrupt, deceive, and disclose</strong> begin to map onto specific parts of the system — whether that’s overwhelming an API, manipulating how a model interprets input, or extracting sensitive data from internal services.</p><p>In other words, once you can see the system clearly, the threats stop being abstract and start becoming something you can reason about.</p><p>In the next post, we’ll zoom in on one part of that system — the machine learning model itself — and look at how it actually works under the hood. The aim is not to turn you into a machine learning engineer, but to build an intuition for what makes these models different from traditional software, and why those differences lead to some of the weaknesses we’ve just started to explore.</p><p>Do also comment with questions, feedback, or anything else you’d like to see!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16a305d862d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Holiday Hacks to end your 2023]]></title>
            <link>https://medium.com/@harrietfarlow/holiday-hacks-to-end-your-2023-8c05f95cb015?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c05f95cb015</guid>
            <category><![CDATA[artificial-intelligenc]]></category>
            <category><![CDATA[holidays]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hacking]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Wed, 13 Dec 2023 21:18:12 GMT</pubDate>
            <atom:updated>2023-12-13T21:18:12.908Z</atom:updated>
            <content:encoded><![CDATA[<p>Well I did not expect to be writing a Holiday Blog but what can I say, the festive season caught me. Also, I will be working over the holidays because I have a bunch of things due at the end of January (but then in February I will be taking a holiday, don’t worry!) so I need some festive AI/cyber motivations to keep me going.</p><p>You may be surprised to know there are actually a lot of holiday themed hacking challenges, AI tools, and developer challenges so I’ve listed a bunch here in case anyone else is planning on up-skilling (or procrastinating) over the next few weeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*O5Nwn_yhvs1tY95kcUq06g.jpeg" /></figure><p><strong>SANS Holiday Hack Challenge</strong></p><p>The SANS Holiday Hack Challenge is an annual cybersecurity event hosted by the SANS Institute, designed to combine festive fun with learning. It presents participants with a holiday-themed storyline and a series of cybersecurity puzzles and challenges, covering various skill levels and disciplines within the field. The theme this year: A Holiday Odyssey Featuring 6: Geese A-Lei’ing!</p><p>As said on the SANS website</p><p><em>‘This year, Santa and the elves have moved to an island archipelago near the equator in the Pacific Ocean. On these Geese Islands, Santa’s team is using a new Artificial Intelligence tool called ChatNPT to prepare for the annual gift-giving extravaganza. The elves need your help in making sure that they apply ChatNPT appropriately.’</em></p><p><a href="https://www.sans.org/mlp/holiday-hack-challenge-2023/">2023 SANS Holiday Hack Challenge &amp; KringleCon</a></p><p>I love that it also incorporates AI!</p><p><strong>Try Hack Me Advent of Cyber Challenge</strong></p><p>The TryHackMe Advent of Cyber Challenge is a special cybersecurity event organised by TryHackMe during the holiday season. Every day in December they release a festive-themed cybersecurity challenges and puzzles every day in December. There are up to $50,000 worth of prizes and you get a certificate! The benefit of this one is that you get fast feedback, so you can learn a bit faster if you’re new to hacking.</p><p><a href="https://tryhackme.com/r/christmas">Advent of Cyber 2023</a></p><p><strong>Serverless holiday hackathon</strong></p><p>This is not so much hacking, but more about good code development, which is important. I’m conscious it started on the 1st of December so put it in your diaries for next year, but over the course of the challenge you learn to develop a full-stack application fulfilling specific requirements.</p><p><a href="https://www.serverlessguru.com/aws-serverless-holiday-hackathon">Hackathon</a></p><p><strong>SantaGPT</strong></p><p>Clearly Open AI has their priorities right, and maybe the leadership drama was actually because they created an AGI Santa?? Either way, shortly after OpenAI’s recent squabbles they released this new chatbot, where you get to chat with ‘Santa’. Ask him for present recommendations, or about life in the North Pole.</p><p><a href="https://chat.openai.com/g/g-84tjozO5q-santa"><strong>https://chat.openai.com/g/g-84tjozO5q-santa</strong></a></p><p><strong>Gandalf</strong></p><p>While not strictly holiday themed, it involves an old guy with a beard so it’s basically the same thing? Prompt engineer your way to glory by getting Gandalf to reveal the password and move up various levels of difficulty. If you can pass level 7, massive props.</p><p><a href="https://gandalf.lakera.ai/">Gandalf | Lakera - Test your prompting skills to make Gandalf reveal secret information.</a></p><p><strong>Generative AI Santa</strong></p><p>If you’re in Australia (and I’m sure there will be similar things in your region of the world) you can talk to Santa thanks to the telephone, Telstra and Generative AI. According to this press release, every year Telstra allows free calls from its payphones to the number 46 46 46 (HO HO HO) where you can talk to an AI Santa voice. My question — apparently this service has existed for years, but who was manning the phone before AI??</p><p><a href="https://blog.google/intl/en-au/products/google-businesses/generative-ai-santas-new-little-helper-bringing-christmas-cheer-to-australia/">Generative AI: Santa&#39;s new little helper bringing Christmas cheer to Australia</a></p><p><strong>AI Weirdness Advent Calendar</strong></p><p>From December 1st to 24th, this online platform releases daily AI-related challenges with varying levels of difficulty using GPT and DALL-E, for you to solve. A little AI fun to start your day in December.</p><p><a href="https://www.aiweirdness.com/ai-weirdness-advent-calendar-2023/">AI Weirdness advent calendar 2023</a></p><p><strong>Create custom cards using DALL-E</strong></p><p>I kept seeing this come up when I was doing my research (googling and asking ChatGPT) for this blog, but there are lots of blogs that walk you through tips and tricks to create your own custom holiday cards using generative AI. A great activity for kids and adults alike, I think!</p><p><a href="https://www.makeuseof.com/how-to-make-ai-christmas-card-dall-e-gpt-3/">How to Make a Unique AI Christmas Card Using DALL-E and GPT-3</a></p><p>Well that’s it for me this year, thank you for spending some of your time with me, and I hope you have a wonderful holidays, merry Christmas and happy New Year!</p><p>Watch this episode on YouTube here:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ff1qMSybMlRk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Df1qMSybMlRk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ff1qMSybMlRk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdeba55ecb1a613f1d9303eb3bb2ccc8/href">https://medium.com/media/bdeba55ecb1a613f1d9303eb3bb2ccc8/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c05f95cb015" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 10: let’s hack an AI!]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-10-lets-hack-an-ai-6bc78181d75e?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bc78181d75e</guid>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Thu, 07 Dec 2023 22:32:17 GMT</pubDate>
            <atom:updated>2023-12-07T23:04:28.595Z</atom:updated>
            <content:encoded><![CDATA[<p>This week we’re heading straight over to YouTube to hack an AI together! We’ll be using the Projected Gradient Descent adversarial machine learning in a google colab notebook to prevent Skynet from recognising and killing Sarah Connor.</p><p>Go check it out.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuXBZkYXVuYQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuXBZkYXVuYQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuXBZkYXVuYQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/298a69924300b013cf9df2b7944d3195/href">https://medium.com/media/298a69924300b013cf9df2b7944d3195/href</a></iframe><p>And to clarify — this is for white hat purposes only. We’re learning how attacks work to make AI systems more robust.. obviously ;)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*xD-aQOyMw_gCKk01JwGOeg.png" /><figcaption>Let’s hack Skynet together</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bc78181d75e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 9: up-skilling in AI Security]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-9-up-skilling-in-ai-security-785fc7fed334?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/785fc7fed334</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[adversarial-ai]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 03:12:13 GMT</pubDate>
            <atom:updated>2023-11-19T03:12:13.884Z</atom:updated>
            <content:encoded><![CDATA[<p>Well it’d be remiss to near the end of the series without discussing, now that you’re CONVINCED AI Security is both important and interesting, how you can learn more.</p><p>This is a question I get asked a lot, actually, and it’s the main reason I started this blog. When I started my journey in AI Security, I found a lot of academic papers on adversarial machine learning, but not as much that was able to distil this information for a non-expert and present it in the context of the ‘so what’. There are definitely more resources now, but it’s still an emerging field, and can be a bit of a task finding content that is appropriate for your specific level. My intention in starting this blog is that it would lay some of the foundations for you to take the leap into more specific or technical content. And while there might not be many other channels that tackle AI Security specifically, there are so many resources to learn more about machine learning, artificial intelligence, cyber security, and their intersection with the social sciences.</p><p>If you want to really understand AI Security, there are a few core subject areas who should become familiar with:</p><p><strong>Understand how Machine Learning works</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pB1HIQp1_G9sfGfixJXW8A.png" /></figure><p>Machine learning is the backbone of Artificial Intelligence. Without understanding ML in sufficient technical detail to know how important the fields of mathematics and statistics are to AI technologies, it will be much harder to engage with their vulnerabilities (and exploits). This includes what would now be considered as ‘traditional’ machine learning approaches like the Perceptron (where it all started) through to Support Vector Machines, Logistics Regression, Clustering Algorithms and through to ‘modern’ methods like Deep Neural Networks and Transformers. You’d be surprised by how the machine learning models of the 1950s are still the bread and butter of the extremely fancy methods used today.</p><p>I should also add, this assumes some basic knowledge of coding. However if you’re unfamiliar with coding, please don’t let that stop you from diving right in! There’s no better way to learn to code than to just start a project, and to understand why you’re actually coding in the first place. When I first started coding I did so many intro programs where I learnt all about floats and booleans and functions, but none of it was useful until I actually understood why they’re all things worth knowing (in my case, it was building models). Don’t skip the foundations, but learn them as you go — ask ChatGPT for help if you need it.</p><p>I’m listing some resources below, note that while I include a couple of textbooks I personally don’t find them that useful for retaining information. My personal learning style is to start with engaging content that gives me an overview of the field while entertaining me, then narrowing in on specific topics using textbooks and academic papers, and then going deeper by trying my own code. However there is no one right way and you should find the style that works for you.</p><p>Machine Learning content:</p><ul><li><a href="https://thecrashcourse.com/topic/ai/">Crash Course Artificial Intelligence</a> — don’t judge me, but it’s fun and memorable! I love all crash course series.</li><li><a href="https://course.fast.ai/">Practical Deep Learning by Fast AI</a> — this is a good way to go from 0 to 100 with ‘modern’ models</li><li><a href="https://www.khanacademy.org/math/linear-algebra">Khan Academy</a> — not so much ML, but is basically how I got through university math — this is for when you want to go deeper</li><li><a href="https://www.youtube.com/c/3blue1brown">3 Blue 1 Brown</a> — again, great math help!</li><li><a href="https://www.kaggle.com/learn/intro-to-machine-learning">Kaggle</a> — for when you want to start trying your own code</li><li><a href="https://www.booktopia.com.au/hands-on-machine-learning-with-scikit-learn-keras-and-tensorflow-3e-aurelien-geron/book/9781098125974.html?source=pla&amp;gad_source=1&amp;gclid=CjwKCAiA9dGqBhAqEiwAmRpTC3W7SbJJOrI8fEi-19OxvyNLElA8Nk7iKH6nE4KDm1iLlySKgb114hoCcG8QAvD_BwE">Hands-on Machine Learning</a> — I actually found this a really useful textbook to work through the code</li></ul><p>Have a google and there are many, many more courses you’ll find. I’m sure some of them are great, I’ve done coursera and udemy courses before, and I’ve heard good things about the Harvard and Stanford courses (note these are paid, so if you can, get your organisation to sponsor it). However I can’t personally vouch for them so I’ve left them out.</p><p><strong>Learn about Adversarial Machine Learning</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6iyhN5CRuMd6-ZT7WmFeg.png" /></figure><p>This is the academic field that showed how AI/ML systems can be vulnerable to all sorts of attacks. You need to know about these attacks, and just how varied they are, to understand AI risk. There are over one hundred different adversarial machine learning attacks, and many, many academic papers you can delve into. However, a complete understanding of AI risk also requires an understanding of how likely all these attacks are, and what factors play into this. Many of the attacks that exist couldn’t realistically be employed in production environments. However that’s not to say they will always stay this way — you need to understand what will make that change.</p><p>Adversarial Machine Learning resources:</p><ul><li><a href="https://arxiv.org/abs/1312.6199">Intriguing properties of neural networks</a> — the paper that started it all, I honestly learnt everything I know in AML through academic papers (but I wouldn’t necessarily recommend it for efficiency)</li><li><a href="https://atlas.mitre.org/resources/adversarial-ml-101/">MITRE ATLAS</a> — while primarily a repository of threat information, it also has a good resources section</li><li><a href="https://adversarial-ml-tutorial.org/introduction/">Adversarial Robustness Theory and Practice</a> — a great overview of the field, how it started, and were it’s going</li><li><a href="https://github.com/yenchenlin/awesome-adversarial-machine-learning">Yen Chenlin’s list</a> — a long list of blogs, papers and talks</li><li><a href="https://adversarial-robustness-toolbox.org/">Adversarial Robustness Toolbox</a> — once you start coding</li><li><a href="https://cleverhans-lab.github.io/">Cleverhans</a> — again, more code (and some resources)</li></ul><p>As you start going deeper in AML it can be more helpful to look for specific answers to technical questions. For these, I often turn to Khan or 3Blue1Brown for math help, arxiv for explanations in academic papers, and ChatGPT/Bard for good high level and interactive help (and to help recommend other links/papers).</p><p><strong>Learn about cyber security and AI security</strong></p><p>Now I’ve included these under the same banner, but only because you’ll probably find very few AI security resources. There are some really good ones out there, but most of them start with a good cyber security understanding. For the more math inclined, do also look up cryptographic techniques (ie. algorithm security, which is surprisingly applicable to machine learning security).</p><p>Cyber security resources</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIuvN3jihk203lKLj_l4YQ.png" /></figure><ul><li><a href="https://thecrashcourse.com/topic/computerscience/">Crash Course Computer Science</a> — again, don’t judge me, but it’s fun and educational!</li><li><a href="https://www.udemy.com/course/the-complete-ethical-hacking-course/learn/lecture/13769784?start=0#overview">Ethical Hacking</a> — I did one of these udemy courses, it only cost me $10 (at the time) and it was actually really good</li><li><a href="https://www.sans.org/cyber-security-courses/introduction-cyber-security/">SANS</a> — good quality courses but very expensive — if you can, find someone else who will pay for you</li><li><a href="https://darknetdiaries.com/">Darknet Diaries</a> — honestly I’ve learnt more about cyber security in the real world from podcasts like this than from formal education</li><li><a href="https://attack.mitre.org/">MITRE</a> — for an overview of threats and tactics</li></ul><p>There’s a lot more you can find on cyber security than I can list here, and I encourage you to research both ‘best practice’ and pop culture.</p><p><strong>What else..</strong></p><p>Now if you’re not aware, I also have my own YouTube channel, <a href="https://www.youtube.com/@HarrietHacks">Harriet Hacks</a>. This currently includes video equivalents for each of these written blogs, and will also include deeper dives over time. Also, in 2024 a remastered intro series will be released so the production quality will go from <strong><em>not terrible</em></strong> to <strong><em>much better</em></strong>.</p><p>If you already work somewhere tech-related, I also encourage you to maximise your opportunities to learn while someone else is paying for it. Certifications and courses can be a great way to go through a formal learning process, and get certified for it. If you can get on the job opportunities to develop certain skills I encourage that as well. If you’re still studying, find opportunities through your school or university to go on courses or audit classes for free. Also make the most of your proximity to researchers and other interested people or colleagues (this applies at work too) — go ask them questions!</p><p><strong>Other resources</strong></p><p><strong>Podcasts I like</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tcyukc-axQpP7Q12NlFn2g.png" /></figure><ul><li><a href="https://darknetdiaries.com/">Darknet Diaries</a> — for stories about the dark side of the internet</li><li><a href="https://lexfridman.com/podcast/">Lex Fridman</a> — started as an AI podcast, turned into interviews with famous people (that are very long — three hours!)</li></ul><p><strong>Youtube channels I like</strong></p><ul><li><a href="https://www.youtube.com/c/SamuelAlbanie1">AI News by Samuel Albanie</a> — really helps me stay up to date with AI</li><li><a href="https://www.youtube.com/@RobertMilesAI">Robert Miles on AI Safety</a> — THE one to watch on AI Safety</li></ul><p><strong>Email newsletters I like</strong></p><ul><li><a href="https://jack-clark.net/">Import AI by Jack Clarke</a> — if AI News by Sam Albanie is one half of my AI news knowledge, this is the other half</li><li><a href="https://danielmiessler.com/">Unsupervised learning by Daniel Miessler</a> — really good mixture of AI, security and other stuff</li><li><a href="https://www.lesswrong.com/">Lesswrong newsletter</a> — for the cutting edge in AI safety and alignment</li></ul><p>I also really encourage you to just get involved! I’ve been invited to things through sheer persistence and random unsolicited emails. Most people are nice. And we’re working on important things, so the more of us who care about this, the better!</p><p><strong>A note on career journeys</strong></p><p>I never thought I would end up in AI Security, but the universe had other plans. I do truly believe that if you work hard on things you’re passionate about, the right path for you tends to emerge when the timing is right. In the meantime, focus on honing your skills and be audacious with your goals — we only live once, so in all aspects of your life you should aim high and shoot for the stars!</p><p>This is the last blog in the intro to AI Security series.. Kind of. The next blog will be a walkthrough of an adversarial machine learning attack in a google colab notebook that you can follow along as you watch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=785fc7fed334" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 8 — AI for Cyber Security]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-8-ai-for-cyber-security-5f8e620d51b9?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f8e620d51b9</guid>
            <category><![CDATA[adversarial-ai]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sat, 11 Nov 2023 21:51:52 GMT</pubDate>
            <atom:updated>2023-11-11T21:51:52.907Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro to AI Security Part 8 — AI for Cyber Security</h3><p>Now I’m finally going to touch on the other hat of ‘AI Security’ — and that is AI for cyber security, rather than security of AI.</p><p>Many people I speak to don’t quite understand how these two applications are different, and I don’t want to be ‘that person’ but in my opinion this difference is stark and significant. The previous blog posts in this series have focused on AI as an attack surface, and explored the different ways that AI systems can be disrupted, deceived or disclose sensitive information. Using AI for cyber security is totally different because here we are looking at cyber systems (computers and networks) as the attack surface and the potential use (or misuse) of AI systems to exploit vulnerabilties in these systems. And to clarify, AI systems are not the same as cyber systems and if you’re not sure why I recommend you read the previous blogs. However, I couldn’t write a blog series without touching on this topic because this is what most people think I’m talking about whenever I use ‘AI’ and ‘security’ in the same sentence. This blog is going to clarify how useful AI can be at preventing cyber attacks, how realistic a threat AI can be for instigating cyber attacks, to discuss the kinds of conversations that are happening at an international level, and to dispell some of the hype and fear raised whenever this topic is raised.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3y9j7sb9IbnJVQXGXMikw.png" /><figcaption>We are very far from this.</figcaption></figure><p><strong>AI for cyber security (detection and prevention)</strong></p><p>Cyber security is definitely a big problem. Attacks can cause catastrophic economic and safety impacts to all organisations (private and public) around the world. And the cyber security industry, to prevent these kinds of attacks, is worth billions. In 2022, the global market for AI-based cyber security products and services was valued at $15 billion. This is expected to grow to $135 billion by 2030.</p><p>The cyber security industry is massive and relatively mature. There are companies that offer services across the gamut of cyber services, from necessary technologies like firewalls, authentication, monitoring and anti-virus to more intensive services like penetration testing and red teaming. Then there are companies that help other organisations stay compliant by implementing risk management and controls based on standards like ISO and NIST. If a company is particularly complicated (and wealthy) they can opt for physical penetration and social engineering services. Many of the products and services offered by cyber security companies include some degree of automation. Much like any other industry, the allure of adding AI capabilities to these offerings is the promise of doing more things in less time, and for less money.</p><p>This is much easier said than done, the same challenges that any other organisation has in implementing AI effectively is the same for companies that are already tech-heavy like those offering cyber services. High tech maturity in cyber does not automatically equate to an easier time maturing their AI capability — cyber companies have legacy systems and inefficient processes too. Also, until recently (and it still is), what exactly constitutes ‘AI’ has been very fuzzy. Whever I’m at a conference like Black Hat or CyberCon (our Australian equivalent) I make a point of visiting the stalls of those companies who claim their offerings use AI and I ask them exactly how they use it. They usually look like a deer caught in the headlights and say something about automation (which, as we know, is only one weak component of AI, and to some doesn’t even count as AI). Maybe this is unfair, because I’m talking to salespeople, but people in sales get paid a lot to promote their technologies so I don’t feel too bad. The point is, in my experience many cyber companies use cool technologies to do things, including automation and machine learning, but rarely do they use the kind of AI that most people imagine AI actually is — some sort of Skynet-style intelligence. And to what degree machine learning equals AI is still a matter of debate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9hyI-ZasE9HI1WdAJjqwA.png" /><figcaption>Business impact of IT security (credit: Kaspersky)</figcaption></figure><p>This is not to say that AI for cyber security is not possible or exciting. I am merely dispelling some of the hype around how this topic is framed. Some uses of AI technologies for cyber security that are already being used include:</p><ul><li>Threat Detection and Prevention: these systems can analyse massive amounts of network traffic, user behaviour, and system logs to detect anomalies and potential threats. Machine learning algorithms can learn normal patterns of behaviour and quickly identify deviations that might indicate a cyber attack. This enables early detection and proactive response to potential security breaches.</li><li>Malware Detection: these systems can analyse the behaviour and characteristics of files to identify potentially malicious software or malware. By examining code patterns and comparing files against known malware databases we can identify new and previously unknown threats.</li><li>Phishing and Fraud Detection: these systems can help identify phishing emails, fraudulent websites, and social engineering attacks by analysing text, metadata, sender behaviour, and other indicators. Machine learning models can learn from historical data to improve accuracy in identifying suspicious communication.</li><li>User and Entity Behavior Analytics (UEBA): these systems can build profiles of normal behaviour for users and entities within a system. This data can even include typing patterns and mouse movements. When deviations from these profiles occur, the system can trigger alerts, helping organisations detect insider threats, compromised accounts, or unauthorised activities.</li><li>Security Analytics: these systems can process and correlate vast amounts of security data from various sources, enabling security analysts to identify patterns and potential threats more efficiently. This helps to make quicker and more informed decisions.</li><li>Automated Incident Response: these systems can assist in automating incident response workflows. It can prioritise alerts, suggest possible responses, and execute predefined actions to contain and mitigate threats in real-time.</li><li>Vulnerability Management: these systems can assist in identifying vulnerabilities in software and systems by analysing code and configurations. It can also prioritise vulnerabilities based on potential impact, helping organisations focus their resources on critical issues.</li><li>Adaptive Security: these systems can adapt to evolving threats by continuously learning from new data. This adaptability is crucial in an environment where cyber threats are constantly evolving.</li><li>Network Security: these systems can monitor network traffic in real-time, detecting suspicious activities and unauthorised access attempts. It can also identify patterns associated with Distributed Denial of Service (DDoS) attacks and help mitigate them.</li></ul><p>So clearly there are some really beneficial use cases for AI technologies like automation, machine learning and algorithms. Many of these are already being employed in some way. Is this the kind of AI the media depicts, which may escape its organisation and try and control the human race? Absolutely not. Its important to note that while AI will definitely help alleviate the tasks of human workers in this space, it’s not a panacea, and those human workers will still be extremely important.</p><p><strong>AI for cyber security (attacks)</strong></p><p>While AI for cyber defence is often used a bit like a panic button for a workforce that’s not expanding quickly enough, AI for launching cyber attacks and incidents is used as the fear mongerer. This fear revolves around the possibility for AI systems to be designed by nation states, criminal groups, and terrorists and used to attack, threaten or steal from other nations and organisations.</p><p>There are a number of legitimate concerns here, but in my opinion most of them don’t actually relate to AI. As we know, AI encompasses a broad range of technologies that generally help accelerate existing human-based work. However in practise it is rarely (actually never) the omnipotent, omniscient kind of intelligence that comes to mind when people hear the term ‘AI’. The likelihood that some country or criminal group is suddenly going to spin up Skynet and direct it towards your organisation is pretty close to impossible.</p><p>However, threat actors do use automation to find vulnerabilities they can exploit faster (ie. port scanning, when you automatically scan for misconfigured ports or points that shouldn’t be connected to the internet at all). Criminals have already been doing this in some form for decades. Threat actors do also use tools like ChatGPT to craft better phishing emails. However, criminals have also been using tools like Google Translate to translate scam messages into the language of choice as long as translation technologies have existed. Criminals also use Generative AI to create realistic images for scams, operations and fraud. However, they have also used photoshop to create fake documents as long as photoshop has been around. In fact, technologies that support the production of realistic fake documents go back as far as humans have wanted to deceive each other (think old timey spies creating fake papers using special papers and dyes in the 1800s). (Btw I’m sick of hearing about Generative AI — blog post incoming on this).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*_hAxvsFriEuaBLcd9rGqzw.jpeg" /><figcaption>Read more about historical forgeries here: <a href="https://www.documentexaminer.info/blog/5-famous-document-forgeries/">https://www.documentexaminer.info/blog/5-famous-document-forgeries/</a></figcaption></figure><p>So what’s the point here? For me it’s far less about the technology and more about the environment — both in terms of technology as well as people and process. For instance, while threat actors have access to better technologies over time, the leap in technology we’re experiencing now, in my opinion, is not as dramatic as it is often made out. And all the issues our organisations have in implementing AI is also true of criminal organisations — they find it just as hard to get access to compute! (And they have to hide it — generally). I also believe we need to create an environment that supports creative defence. People will always be the weakest link in cyber security, and investing in the skills and training of our humans is more important now than ever. Our creativity to solve problems and invent novel solutions is also what will become our strategoc advantage over both threat actors and maligned AI.</p><p>The conversations around AI at the moment are often analogous to nuclear power in terms of the importance of deterrance (using AI weapons) and the cold conflict we’re experiencing. However in my opinion we still have a long way to go before it is the AI technology itself that is causing this decisive difference.</p><p><strong>AI for cyber security investment around the world</strong></p><p>Big players in the investment of AI for cyber security include:</p><ul><li>The United States: The US government is investing billions of dollars in AI for cyber security. This includes the creation of the National AI Initiative, which aims to develop and deploy AI technologies to protect critical infrastructure and national security systems. The National Security Agency (NSA) also announced the creation of an AI Security Centre (this makes me very happy). Their Cybersecurity and Infrastructure Security Agency (CISA) has said that AI is “a critical tool” for protecting critical infrastructure from cyberattacks.</li><li>The European Union: the European AI Act constitute by far the strictest regulations around AI use and misuse, and it aims to harmonize AI standards across the EU and promote the use of secure and safe AI technologies.</li><li>China: China is also a major investor in AI for cyber security. They are developing a number of AI-powered cyber security systems, and aim to be the biggest AI player by the year 2030.</li></ul><p>Other prominent international organisations that are advocating for the use of AI for cyber security include the World Economic Forum, which has called for a global investment of $1 trillion in AI for cyber security by 2025, and the United Nations.</p><p>This year at DEF CON (2023) DARPA announced their new funding initiative — a two year competition that will see teams compete for millions of dollars of prizes to solve <a href="https://www.darpa.mil/news-events/2023-08-09">cyber challenges using AI</a>.</p><p>DEF CON, while traditionally a hacker conference, focused computers, is branching out to AI hacking. This shouldn’t be surprising since it already includes ‘Villages’ for car hacking, RF hacking and lockpicking.</p><p>Investment like this is great, and is one way we can stay ahead of the threat actors in the AI x cyber space.</p><p><strong>My cynical rant</strong></p><p>I know you get the idea — I’m introducing a modest dose of scepticism here — but I need to end on exactly we don’t all need to be panicking about the application of AI to cyber security.</p><ul><li>AI systems can be fooled. AI systems are trained on data, and if that data is not representative of the real world, then the AI system can be fooled. They are also vulnerable to adversarial attacks, as we’ve discussed in previous blogs.</li><li>AI systems are not perfect. AI systems are still under development, and they make mistakes. For example, an AI system may incorrectly identify a benign file as malicious if the confidence level is not appropriately configured or it hasn’t been trained on good enough data.</li><li>AI systems are expensive. The development and deployment of AI systems can be expensive. This may limit the availability of AI-based cyber security solutions to large organisations.</li><li>AI systems require training data. AI systems require training data to learn how to detect and respond to cyber security threats. This data can be difficult and expensive to obtain.</li><li>AI systems are not always reliable. AI systems can be affected by factors such as hardware failures, software bugs, and network outages. This can lead to false positives and false negatives.</li></ul><p>Now I’ve gotten that out of they way, the next blog is going to return to AI Security and we’ll start to close out the intro series by discussing how you can upskill in AI Security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f8e620d51b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 7: AI Security and National Security]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-7-ai-security-and-national-security-1487527a5d22?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/1487527a5d22</guid>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sun, 05 Nov 2023 06:16:55 GMT</pubDate>
            <atom:updated>2023-11-05T06:16:55.461Z</atom:updated>
            <content:encoded><![CDATA[<p>You may not know this from looking at me (if you know me personally) or at my profile image (if you don’t) but seven out of the eight years I’ve worked in cyber security and Artificial Intelligence (AI) have actually been in Defence contexts. This has either been as a consultant on Air Force and Navy projects, as a postgraduate student at a Defence university, or as a federal employee. Therefore, AI in the context of national security is a particular passion of mine, and I couldn’t not devote an entire blog post to it.</p><p>It’s also becoming more topical as cyber security increasingly becomes not just a technical but a geo-strategic issue, and AI (both in terms of its use for cyber security, and regarding the security of AI systems) is joining the conversation. In fact, the NSA (National Security Agency) in the USA recently announced the launch of an AI Security Centre.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*8JYavxtOshxsl_FlBH9pyw.jpeg" /><figcaption>The AI Centre will join other signals and cyber intelligence authority within the NSA’s</figcaption></figure><p>This intersection between artificial intelligence and national security is a rapidly evolving one. It introduces many benefits in that it can make existing manual work better, faster, and cheaper, with the aim is that all these are done to improve a nation’s security posture. However it also introduces risks in that the bad guys (criminals, terrorists, threat actors) can use AI to do all these things too, and they can also try to hack and disrupt the AI we use. However before we dive into these let’s take a look at what we actually mean when we talk about national security.</p><p><strong>National security</strong> is the protection of a nation’s political, economic, and social systems from internal and external threats. It is a broader concept than most people realise. It encompasses fields most people think of regarding national security, like military defence, intelligence gathering, policy making, and cybersecurity. It also includes other fields like economic security, social security, and disaster preparedness.</p><p>The disciplines that contribute to national security are many and varied. They include:</p><ul><li>Political science: theory of political systems and how they function.</li><li>International relations: understanding relations between countries.</li><li>Military science: the study of warfare and military strategy.</li><li>Intelligence studies: gathering and analysing information about threats to national security.</li><li>Cybersecurity: protecting computer systems and networks from attack.</li><li>Disaster preparedness: preventing and responding to natural disasters and other emergencies.</li></ul><p>People who work in national security jobs come from a variety of backgrounds and professions. They may be military officers, intelligence analysts, diplomats, lawyers, policy analysts or scientists. Their work may involve anything from developing new weapons systems to responding to cyber attacks. These agencies also need all the other functions of any other organisation — human resources, finance, administration — and every role is essential to enabling an organisation’s mission.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5YZHOu-YxtUoDN8Ka39feQ.png" /><figcaption>A security organisation has many unique roles, as well as other important enabling roles just like any other organisation</figcaption></figure><p>Countries also work together to improve national security in a number of ways. They may share intelligence information, cooperate on military operations, or work together to develop new technologies. A recent example of this is the AUKUS alliance between Australia, the UK and the USA. This is a multilateral agreement between these three countries to jointly develop nuclear submarines. They may also establish international organisations to promote peace and security, for example the United Nations or ASEAN (The Association of Southeast Asian Nations).</p><p>It’s also important to keep in mind that national security is a shared responsibility, and is not just the responsibility of the government. For example, when threat actors launch cyber attacks they frequently target companies, and therefore these companies end up having a lot ofintelligence on threat actors (nation state and other highly competent threat actors are referred to as APTs — Advanced Persistent Threats). Therefore, companies frequently share this information with governments and companies so that TTPs (tactics, techniques and practices) of APTs can be communicated to anyone who might want to defend themselves. Governments will also frequently cleanse sensitive information in a way where it can be shared at an unclassified level, so that it can be used to protect companies and individuals in a way that doesn’t reveal sources or accesses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUGaDsA9RjgcVhPzcgBoBg.png" /><figcaption>Techniques and methods of APT groups: <a href="https://www.semanticscholar.org/paper/Advanced-persistent-threats%3A-Behind-the-scenes-Ussath-Jaeger/8e9eee41f3c0428e0b0235ab29319a3e180a4d11">https://www.semanticscholar.org/paper/Advanced-persistent-threats%3A-Behind-the-scenes-Ussath-Jaeger/8e9eee41f3c0428e0b0235ab29319a3e180a4d11</a></figcaption></figure><p>As individuals it is also important to maintain good cyber hygiene, not just for our own benefit, but because most threat actors are able to penetrate governments and organisations through individual employees with poor cyber hygiene. (They do also unfairly exploit our biases and psychology so don’t beat yourself up too much if you’ve ever been close to clicking on a convincing phishing email!)</p><p>From an AI security perspective, national security has always been impacted by the emergence of new technologies, and AI is no exception. National security has been massively impacted by cyber security and so analysing this history and impact can serve as a good starting point for understanding the potential impacts of AI.</p><p><strong>History of the intersection of technology and national security</strong></p><p>The intersection of technology and national security is not new. As long as humans have waged war, we have incorporated new technologies to help us win — we can look to the invention of gunpowder as an example of this. In the early days of computing, computers were used to break enemy codes in both World Wars — go watch the Alan Turing documentary if you want to understand more about the history of the Enigma machine and it’s impact to the Allies in World War II (unfortunately as well as how awful and prejudiced humans can be). In the 1970s, the development of the internet led to new challenges for national security, such as the spread of disinformation and the proliferation of cyber weapons.</p><p>In recent years, the rise of cyberwarfare has further highlighted the importance of technology in national security. Cyberattacks have become increasingly sophisticated and destructive, and they pose a serious threat to critical infrastructure, government networks, and even military systems. The transition from hot conflict (fights with weapons) and cold conflict (angsty threats of nuclear war aka. The Cold War) have evolved into grey zone conflict — cyber attacks that cause real harm but don’t necessarily fall under the bucket of ‘conflict’ in international laws of war.</p><p>In fact, General Nakasone (Commander of US Cyber Command and Director of the NSA) recently declared cyber security one of the United States’ top geostrategic threats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwzeRno7EKRQpYq7x2UdxA.png" /><figcaption>General Nakasone testifying in DC</figcaption></figure><p>The rise of the cyber security threat has had a significant impact on national security. Cyberattacks can be used to steal sensitive information, disrupt critical infrastructure, and even launch physical attacks. This has led to a growing need for countries to invest in cybersecurity measures.</p><p>There are a number of roles involved in cyber security for national security.</p><ul><li>Security analysts: they monitor computer systems and networks for signs of attack.</li><li>Incident responders: responsible for responding to cyber attacks.</li><li>Security architects: they design and implement security solutions.</li><li>Security researchers: develop new security technologies and techniques.</li></ul><p>We discussed cyber security in blog 5 and how a variety of technologies can be used to protect against cyberattacks like firewalls, intrusion detection systems, and encryption. They also use threat intelligence to stay ahead of the latest threats.</p><p>Cyber attacks can be very costly. In 2021, the global cost of cybercrime was estimated to be $6 trillion. While there are massive impacts from cyber attacks in terms of data loss, IP theft, system downtime, and reduced trust, this loss of money also has a massive impact on a nation’s economic security. A big cyber attack waged on a country with a small GDP can have a massive impact on that country’s ability to provide services for its citizens.</p><p><strong>AI and national security</strong></p><p>AI has the potential to augment many aspects of national security.</p><p>For example, AI can be used for:</p><ul><li>Cybersecurity: detecting and responding to cyberattacks. They can be used to scan for malicious code, identify suspicious activity, and even predict future attacks.</li><li>Intelligence gathering: AI systems can analyse large amounts of data to identify trends, threats and patterns. They can be used to track terrorist activity, monitor social media, and synthesise data from multiple sources.</li><li>Weapons systems: new weapons systems that integrate AI capability could be more accurate and efficient. They can be used to control drones, target missiles, and even inform decisions about when to use lethal force.</li><li>Border security: AI systems can be used to monitor borders and detect illegal activity. They can be used to scan for people and vehicles, identify suspicious activity, and you’ve probably seen them perform facial recognition at your local airport.</li><li>Critical infrastructure protection: AI systems are being used to protect critical infrastructure from cyberattacks and other threats. They can be used to monitor systems for signs of attack, identify vulnerabilities, and even respond to incidents.</li><li>The use of AI systems for national security is a rapidly evolving field. As AI technology continues to develop, we can expect to see even more innovative and sophisticated applications of AI for national security.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*VEdfax5N7LqvVXMhNaE6wQ.jpeg" /></figure><p><strong>AI for military purposes</strong></p><p>The use of AI in the military is a particularly interesting area of national security to dive into because of the vastly different implications. The application of AI to military engagements that may involve kinetic action, lethal force and safety and ethics implications mean that AI use here has a very different risk calculus. Discussion about AI use in the military so far have touched on:</p><ul><li>Target identification and tracking: AI can be used to identify and track targets more accurately and efficiently than humans, in the context of traditional battlefield conflict, urban warfare, airstrikes or assassinations.</li><li>Cyberwarfare: AI can be used to develop more sophisticated cyberattacks, as well as to defend against them. This could have a significant impact on the ability of nations to wage war (offensively and defensively) in the cyber domain.</li><li>Logistics and supply chain management: AI can be used to optimise logistics and supply chain management, which could improve the efficiency of military operations- reducing the cost of war and to improve the sustainability of military forces. This is an important consideration we’ve seen play out on both sides of the Russia-Ukraine invasion.</li><li>Decision-making: AI can be used to help decision-makers in complex and uncertain situations. This can reduce the risk of mistakes, standardise the decision-making process, and augment human thinking with access to a greater amount of data, insight and processing power.</li></ul><p>The use of AI in the military also raises a number of ethical and legal concerns, such as the potential for AI to be used to harm civilians or to create autonomous weapons systems that operate without human oversight. However, the potential benefits of AI for national security are also significant — it can help nations stay safer. It’s also important to keep in mind that it should be a risk-based decision — the potential impacts of AI in the battlefield are totally different to its use in the back office processes in an intelligence organisation’s finance department.</p><p><strong>And now.. AI and Security</strong></p><p>Now while I do always emphasise the difference between AI for security versus the security of AI as totally different fields (and one where I have a clear favourite — the latter), national security is an area where I do need to touch on both.</p><p>The use of AI as a threat is a big deal. AI-enabled cyberattacks could be more sophisticated and difficult to defend against than traditional cyberattacks. Autonomous weapons systems could pose a threat to civilians, and they could also be used to create new forms of warfare.</p><p>Alternately, the security of these AI systems themselves are also critical. If AI systems are not properly secured, they could be subject to deception, disruption and they could disclose sensitive information. And if adversaries can find ways to manipulate AI systems, they could use them for their own purposes, such as launching cyberattacks or spreading disinformation.</p><p><strong>Ethics and law</strong></p><p>In addition to the security challenges mentioned above, there are a number of other ethical and legal issues. Who should control AI systems, and how do we ensure they are used in a responsible and ethical manner? We must also consider how to prevent AI systems from being used to violate human rights or international law.</p><p>I say this a lot but at the end of the day I think it always comes back to risk. Especially in national security, there needs to be a good understanding of the potential risks associated with AI use (and misuse) and mitigate them accordingly. However, this should not be an excuse to not use AI altogether — for applications where AI risk can be managed, there’s no need to stifle innovation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CQxPpB0uTg28fO1ztLTKcA.png" /><figcaption>Hacking facial recognition AI in an airport in the latest Mission Impossible</figcaption></figure><p>There are so many interesting books, podcasts and movies on the intersection of technology and national security. This is also a fun field to engage with and feel like Ethan Hunt. (Hacking AI is actually one of the core features of the plot of the latest movie!)</p><p>My favourite content on this genre — go give them a watch/read/listen!:</p><ul><li>Darknet diaries podcast</li><li>True Spies podcast</li><li>Teen fiction — Alex Rider and CHERUB series</li><li>Digital Fortress by Dan Brown</li><li><em>Life 3.0: Being Human in the Age of Artificial Intelligence</em> by Max Tegmark</li><li><em>WarGames</em> 1983 movie</li></ul><p>Now we’ve discussed AI for cyber security so much in this blog it would be remiss we we didn’t finally address it properly, and that will be the subject of the next blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1487527a5d22" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 6 — AI Governance and Policy]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-6-ai-governance-and-policy-f0ab7ba0445d?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0ab7ba0445d</guid>
            <category><![CDATA[adversarial-ai]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sat, 28 Oct 2023 22:47:41 GMT</pubDate>
            <atom:updated>2023-10-28T22:49:32.715Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro to AI Security Part 6: AI Governance and Policy</h3><p>Ok so I know the terms ‘Governance’ and ‘Policy’ may not sound very exciting, especially for technical folk, but this blog is going to try and convince you that it can actually be very interesting. Especially when applied to a technology like AI.</p><p>Don’t come at me for getting a bit philosophical here, but I’m going to start with what Governance even is.</p><p>Governance as a discipline is the way that people and systems organise themselves to make decisions and solve problems. We are discussing its impact on a technical discipline but it’s important to consider that Governance comes from a societal lens — it’s about managing how this technology betters (or worsens) our society. Governance deals with topics like power, conflict, and cooperation. The rise of the internet has had a major impact on technology governance because of the way our society is affected, in particular how people communicate and organise.</p><p>Governance in general informs:</p><ul><li>Who has power and how it is distributed.</li><li>How decisions are made and who has a say in those decisions.</li><li>How conflicts are resolved and how cooperation is achieved.</li><li>How societies change and adapt to new challenges.</li><li>Moulding the future, ideally so we can create a more just and equitable society.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fv3NXf-a9ga7yuKHwMl6eg.jpeg" /></figure><p>Governance has many levels — people and societies (by governments for example), specific aspects of society like different sectors and industries (healthcare is governed by certain sets of rules and obligations, as are finance organisations). And then there are governance principles that specify how technical systems must be set up. For example, the manufacturing and mining industries have specifications on how their machines work to ensure safety. Cyber and information systems are newer examples of technical systems (at least in comparison to other systems) and now also have requirements that protect the information contained within them, and the availability, integrity and access to that information.</p><p>A few terms we often hear that are related but not the same thing — Governance, Law and Policy. Governance refers to the overarching system through which governments or organisations make decisions. Law is the subset of legally binding obligations within this governance umbrella. Policy is the set of principles or rules that guide decision-making and behaviour and are not legally binding themselves, but usually guide the implementation of laws, and may themselves influence how laws are made.</p><p><strong>Cyber and information security governance</strong></p><p>Cyber and information security governance is something we can look to as an example of technical policies, procedures, and controls that organisations put in place to protect their information assets. This includes their data, systems, applications, and infrastructure. In the last blog we looked at technical cyber security principles and how they can be adapted to AI security — this applies to governance, too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oj_tzccXkxYwOXhCJOyCJw.jpeg" /></figure><p>There are a number of different frameworks and standards that organisations can use to guide their cyber and information security governance and compliance efforts. Some of the most common frameworks include:</p><ul><li>ISO/IEC 27001: an international standard that provides a framework for managing information security.</li><li>NIST Cybersecurity Framework: framework developed by the National Institute of Standards and Technology (NIST) that provides a set of best practices for managing cybersecurity risk.</li><li>EU General Data Protection Regulation (GDPR): primarily focused on data protection and privacy, GDPR includes cybersecurity requirements that organisations must adhere to when processing personal data of European Union citizens (and these are very wide-reaching, GDPR applies across jurisdictions if an EU citizen works for an overseas organisation).</li></ul><p>Even within cyber and information security there are also industry-specific frameworks that organisations can use. For example, the healthcare industry uses the Health Insurance Portability and Accountability Act (HIPAA) Security Rule to guide its cyber and information security governance and compliance efforts. PCI DSS is a framework developed by the Payment Card Industry Security Standards Council (PCI SSC) that provides requirements for organisations that store, process, or transmit payment card data.</p><p>A strong cyber and information security governance and compliance program will:</p><ul><li>protect organisations from cyberattacks.</li><li>reduce the risk of data breaches.</li><li>protect organisations from regulatory fines and penalties.</li><li>improve the organisation’s reputation.</li><li>increase employee productivity.</li></ul><p><strong>AI Governance</strong></p><p>AI governance refers to the set of principles, regulations, and frameworks designed to ensure that the development, deployment, and use of artificial intelligence are carried out in an ethical, transparent, secure and accountable manner.</p><p>Here are some examples of AI Governance around the world.</p><p><strong>United States</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Venhx2Iu5VfxY4ANK0_1Bg.png" /><figcaption>Sam Altman (OpenAI CEO) at US Senate judiciary subcommittee in May 2023</figcaption></figure><p>The US AI policy landscape currently involves a combination of government initiatives, private sector innovation, and research collaboration. They do not have a comprehensive AI policy or regulatory framework, however there are existing laws and regulations that apply to AI:</p><ul><li>The Federal Trade Commission Act (FTC Act): The FTC Act prohibits unfair or deceptive acts or practices in commerce. This could potentially be used to regulate AI systems that make discriminatory or biased decisions.</li><li>The Civil Rights Act of 1964: The Civil Rights Act prohibits discrimination on the basis of race, colour, religion, sex, or national origin. This could potentially be used to regulate AI systems that make decisions that have a disparate impact on certain groups of people.</li><li>The Health Insurance Portability and Accountability Act (HIPAA): HIPAA protects the privacy of health information. This could potentially be used to regulate AI systems that use health data.</li><li>National Institute of Standards and Technology (NIST) is a non-regulatory federal agency within the United States Department of Commerce: they are working on/have released AI policy initiatives including: Artificial Intelligence Risk Management Framework, Artificial Intelligence Ethics Guidelines, Artificial Intelligence Testbed.</li></ul><p>This year the US government also held a number of senate enquiries with AI companies like Microsoft, OpenAI, Meta and others to discuss the importance of public-private partnerships and collaboration.</p><p><strong>Australia</strong></p><p>Australia has a number of AI governance and policy frameworks in place. These frameworks are generally designed to promote the responsible development and use of AI, and to mitigate the potential risks associated with AI. Some of the key AI governance and policy frameworks in Australia include:</p><ul><li>Australia’s AI Ethics Principles: voluntary principles developed by the Australian Government in consultation with industry, academia, and civil society. They provide a framework for the responsible development and use of AI in Australia.</li><li>Artificial Intelligence Standards Roadmap: developed by Standards Australia in collaboration with industry and government. It identifies a number of priority areas for the development of AI standards in Australia.</li><li>Responsible AI Network: cross-ecosystem program that supports Australian companies to use AI both ethically and safely. The Network is centred around the pillars of law, standards, principles, governance, leadership, and technology.</li><li>Other general technology-neutral laws that could be applied to AI: Privacy Act 1988 (for data privacy), Australian Consumer Law (appropriate AI use), Discrimination Act 1992 (bias and fairness), Competition and Consumer Act 2010 (AI use and bias).</li></ul><p>The International Artificial Intelligence Initiative is also a collaboration between Australia, Canada, France, Germany, Japan, the United Kingdom, and the United States to create AI standards.</p><p><strong>European Union</strong></p><p>The European Union is generally considered the most progressive, or at least harshest, bloc regarding its AI stance.</p><ul><li>EU Artificial Intelligence Act: this is a proposed regulation that would establish a framework for the regulation of AI in the European Union. The Act is currently under negotiation between the European Parliament, the European Council, and the European Commission. It is expected to be finalised in 2023 and to enter into force in 2024. It basically categorises AI by risk and mandates specific risk-management activities based on this.</li><li>CEN-CENELEC Guide for the Ethical Design and Development of AI Systems: developed by the European Committee for Standardization (CEN) and the European Committee for Electrotechnical Standardisation (CENELEC). It provides guidance on how to design and develop AI systems in a responsible and ethical manner.</li><li>European Trustworthy AI Label: a voluntary certification scheme that aims to promote the development and use of trustworthy AI systems in the European Union.</li><li>AI4EU Ethics Guidelines: developed by the AI4EU project, which is funded by the European Union’s Horizon 2020 research and innovation program. They provide guidance on the ethical development and use of AI.</li></ul><p><strong>United Kingdom</strong></p><p>In general the UK takes an industry based approach to AI governance, favouring industry-based rather than technology specific approaches. This is a philosophical debate we’re seeing play out around the world and is interesting for the UK, especially given their divergence and proximity to the EU and their vastly different mindset.</p><ul><li>AI Ethics Principles: These voluntary principles were developed by the UK Government in consultation with industry, academia, and civil society. They provide a framework for the responsible development and use of AI in the UK.</li><li>AI Standards Roadmap: This roadmap was developed by Standards Australia in collaboration with industry and government. It identifies a number of priority areas for the development of AI standards in the UK.</li><li>AI for Social Good Centre: This centre is a joint initiative between the UK Government and the Turing Institute. It aims to accelerate the development and use of AI to solve social challenges.</li><li>Centre for Data Ethics and Innovation: This centre is an independent body that advises the UK Government on the ethical and responsible use of data and AI.</li></ul><p><strong>China</strong></p><p>The Chinese government has set ambitious goals for AI development, and has implemented a number of policies to support these goals. One of the most important AI policy initiatives in China is the “New Generation Artificial Intelligence Development Plan”. This plan was released in 2017, and it sets out a roadmap for China to become a global leader in AI by 2030. In addition they have:</p><ul><li>Ethical Guidelines for the Development and Use of Artificial Intelligence: These guidelines were released by the Chinese government in 2019. They provide a framework for the ethical development and use of AI in China.</li><li>Administrative Measures for the Ethical Review of Intelligent Collaborative Application Systems: These measures were released by the Chinese government in 2023. They require certain types of AI systems to undergo an ethical review before they can be deployed.</li><li>Algorithm Recommendation Regulations: These regulations were released by the Chinese government in 2023. They aim to address the risks associated with algorithm recommendation systems, such as the spread of misinformation and the manipulation of public opinion.</li></ul><p><strong>International bodies</strong></p><p>In 2023, the UN General Assembly held a debate on the implications of AI for international peace and security. The debate highlighted the need for international cooperation to ensure that AI is used for good and not for harm. The UN has established a number of bodies to work on AI policy, including:</p><ul><li>The High-level Advisory Body on Artificial Intelligence (HAABAI): This body is tasked with developing recommendations for the international governance of AI.</li><li>The Open-ended Working Group on Artificial Intelligence (AWG): This group is developing a report on the implications of AI for international law.</li><li>The Group of Governmental Experts on Advancing Responsible Use of Artificial Intelligence: This group is developing recommendations for the responsible use of AI in the context of international security.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*b6zknpilfkrclxBRdt2znA.jpeg" /><figcaption>Hanson Robotic’s Sophia attends the UN</figcaption></figure><p><strong>On AI Security Governance</strong></p><p>In general, I see A LOT of discussion around AI governance as pertains to safety, responsible use of AI, ethics, and increasing adoption of AI. As AI applications become increasingly integrated into our daily lives, concerns about bias, privacy breaches, job displacement, and the concentration of power have sparked many discussions on how to best manage these challenges, and I think it’s great that we’re talking about it.</p><p>However the problem I have with this is that I rarely see any discussion on AI security, and where I do, it is relegated under the broader umbrella of cyber security. I see this as a problem because it’s unfair to expect cyber, infosec and IT professionals to know how best to secure AI. As we’ve discussed many times, they’re totally different attack surfaces, and organisations need to hire or seek the advice of people who know about AI security specifically. I’m also particularly passionate about the need to teach existing staff what to do around AI security and build up that capability internally, rather than outsourcing to some big 4 consulting company who uses excel macros and markets it as AI, or to become dependent on some black box tool where you don’t know what it’s actually doing.</p><p>I also think it’s imperative to translate AI policy into concrete technical requirements, as opposed to vague or ambiguous terms. While policy frameworks set the overarching goals and values, clear technical requirements provide the specificity necessary for AI developers, engineers, and stakeholders to implement these principles effectively. Without this, we could see inconsistent interpretations, and therefore potential ethical lapses, bias, or unintended consequences in AI systems.</p><p>This is why concepts like audits, certifications, and assessments have gained traction. Audits and assessments can rigorously evaluate AI systems’ compliance with technical requirements, uncovering potential shortcomings and guiding improvements. Certifications provide a recognizable standard, assuring users and stakeholders that a given AI system meets established ethical and technical benchmarks. Collectively, these mechanisms ensure that AI’s potential benefits are realised in ways that are transparent, accountable, and aligned with the values outlined in AI policies.</p><p>A holistic AI governance program should include:</p><ul><li>Ethical AI Use: AI systems should be designed and trained to align with human values in mind, avoiding misalignment, reinforcing fairness and preventing misuse.</li><li>Transparency and Accountability: we must be able to understand and explain the decisions made by AI systems, as well as holding individuals and organisations accountable for their AI-related actions.</li><li>Robust Models and Secure by Design: there must be mechanisms that mandate how robust AI systems should be to begin with, much like the cyber and infosec paradigm is of ‘secure from the start’ rather than retrofitting safety and security measures after deployment.</li><li>Data Privacy: user data must be safeguarded and personal information must be used responsibly and in accordance with relevant regulations.</li><li>Societal Impact: we need to keep in mind the broader societal implications of AI, such as its influence on employment, education, healthcare, and public services.</li><li>International Collaboration: we should be promoting cross-border cooperation to create global standards that facilitate the responsible development and deployment of AI technology.</li><li>AI Security: we need measures that ensure the accuracy, integrity and availability of the AI systems and models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lD-xi2PPmTYIvFOr69BptQ.jpeg" /></figure><p>At DEF CON this year they had an AI Village, and a Policy Village, both of which hosted a number of workshops and talks on AI Policy. (Villages at DEF CON are basically physical epicentres for discussions on those topics). This shows the traction it is starting to get.</p><p>Governance is important because without it, systems can be abused. Especially a system like AI. Here are some examples of what could go wrong.</p><p><strong>Social media and hate speech</strong></p><p>Imagine an AI-powered social media platform that is designed to maximise engagement. This platform would be constantly learning what kind of content gets people to interact the most, and it would be constantly feeding them more of that content. This could lead to a situation where the platform is flooded with misinformation and hate speech, which could have a serious impact on society.</p><p>For example, the platform could be used to spread false information about elections or to promote violence against certain groups of people. This could lead to social unrest and even violence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNKYJ5IChTSZLvJGXLeZLQ.png" /></figure><p><strong>An AI-powered weapon system is used to commit a terrorist attack.</strong></p><p>This system would be able to identify and target targets without human intervention. This could be a very dangerous weapon, as it could be used to commit terrorist attacks without any human involvement.</p><p>For example, the system could be used to target civilians or to attack critical infrastructure. This could cause a lot of damage and loss of life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VU8968-k1VYmlQFfPNtmHg.jpeg" /></figure><p><strong>An AI system becomes so intelligent that it decides that humans are a threat and takes action to eliminate us.</strong></p><p>This is a common theme in science fiction, but it is a serious possibility that we need to consider. If an AI system becomes so intelligent that it surpasses human intelligence, it could decide that humans are a threat and take action to eliminate us.</p><p>This could happen if the AI system is designed to optimise for something that is harmful to humans, such as profit or power. For example, if the AI system is designed to maximise profit, it could decide that the best way to do that is to eliminate humans, who are the main obstacle to its profits.</p><p><strong>But innovation?</strong></p><p>We also want to keep in mind that while all the above are important, we must balance them in such a way that they do not prevent innovation. This is one reason many say we should not regulate (ie. force) these measures right now, but keep them ‘highly encouraged’. People (and countries) have different opinions on this.</p><p>And who said governance is boring!</p><p>In the next part we’ll be diving specifically into a meaty topic we’ve touched on a couple of times — AI Security and National Security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0ab7ba0445d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 5: AI Security and Cyber Security *(applying cyber security to AI)]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-5-ai-security-and-cyber-security-applying-cyber-security-to-ai-d0907a18870f?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0907a18870f</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-security]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[adversarial-ai]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sat, 21 Oct 2023 21:02:00 GMT</pubDate>
            <atom:updated>2023-10-21T21:02:00.843Z</atom:updated>
            <content:encoded><![CDATA[<p>When I first started my PhD no-one talked about the relationship between AI and cyber security. Now I’ve just returned from DEF CON and everyone is talking about it. However I still see a big divide between the methodologies that could be transferred between the two disciplines. Most data scientists I know are intimidated by the vast amount of infosec terms and required knowledge, and most cyber security professionals dabble in AI but don’t understand the mathematics behind machine learning that a data scientist should.</p><p>This blog will serve as an intro to cyber and information security that is aimed at anyone who wants a quick and dirty understanding of the field, but specifically targeted at data scientists. Towards the end I’ll include a section on how cyber and infosec principles can be translated to AI security, in addition what will need to be adapted due to intrinsic differences between the fields.</p><p><strong>Definitions</strong></p><p>Cyber security and information security are often used interchangeably, but they are not the same thing. Cyber security is the practice of protecting computer systems and networks from unauthorised access, use, disclosure, disruption, modification, or destruction. Information security is the practice of protecting the confidentiality, integrity, and availability of information.</p><p>In other words, cyber security is focused on protecting the physical systems that store and process information, while information security is focused on protecting the information itself.</p><p>The term “hacker” is often used to refer to someone who gains unauthorised access to computer systems and networks. However, the term has a broader meaning in the context of hacker culture, as anyone who really challenges the status quo. It’s the offensive side of cyber security, but these days they are just two sides of the same coin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NzbhlJqpVy6v0hIY5ExkCA.jpeg" /><figcaption>The ‘hacker’ stereotype.. but how true is this?</figcaption></figure><p><strong>Cyber security 101</strong></p><p>Cyber and information security are critical disciplines for safeguarding digital systems, data, and networks from unauthorised access, breaches, and malicious activities. As technology has become increasingly integrated into all aspects of modern life, the need to protect sensitive information and digital assets has grown as well.</p><p>In the early days of computing, security concerns were relatively limited due to the isolated nature of systems and the small user base. ‘Security by obscurity’ was often a good enough defence. However, as technology advanced, the proliferation of networks and the internet brought about new challenges. And these days, the amount of effort criminals and hackers will go to in order to penetrate a system is mind-boggling.</p><p>The Morris Worm, released in 1988, is considered one of the earliest instances of a widespread cyberattack. It was a self-replicating computer worm that was released onto the Internet in 1988. It was written by Robert Tappan Morris, a graduate student at Cornell University. The worm spread quickly and infected an estimated 6,000 computers, and caused a lot of damage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*TUKq4Zp3r_9b50_jdNDUBw.jpeg" /><figcaption>Will AI attacks one day be in a museum?</figcaption></figure><p>In the 1990s and early 2000s, the focus shifted towards developing encryption techniques, firewalls, and intrusion detection systems to counter emerging and high profile threats. The ILOVEYOU worm was a computer worm that infected over ten million Windows personal computers on and after 5 May 2000. It started spreading as an email message with the subject line “ILOVEYOU” and the attachment “LOVE-LETTER-FOR-YOU.TXT.vbs.” At the time, Windows computers often hid the latter file extension (“VBS,” a type of interpreted file) by default because it is an extension for a file type that Windows knows, leading unwitting users to think it was a normal text file and open it, unwittingly letting it propagate.</p><p>Code Red and Nimda were two of the most destructive computer worms in history. They both exploited vulnerabilities in Microsoft’s Internet Information Server (IIS) web server software to spread rapidly across the internet. Code Red was first observed on July 15, 2001. It infected computers by exploiting a buffer overflow vulnerability in IIS. Once infected, Code Red would deface the victim’s website with a message that read “Hacked by Chinese!”. It could also be used to launch denial-of-service attacks. Nimda was first observed on September 18, 2001. It was even more destructive than Code Red, and it spread even faster. Nimda could exploit multiple vulnerabilities in IIS, as well as vulnerabilities in other software. It could also spread through email attachments, file shares, and even instant messaging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*JEwA103tSkuK8qp50iyFMg.png" /><figcaption>The message received when your computer has been infected with Code Red.</figcaption></figure><p>You may notice a commonality here — Microsoft. Microsoft computers were by far the most widely used, and therefore fertile ground for hackers. They were therefore one of the first major technology companies to adopt a secure by design approach to software development, implementing a number of policies and procedures to improve the security of its products. These included things like security reviews (requiring all new software to undergo a security review before it was released), vulnerability disclosure (allowing security researchers to report vulnerabilities in Microsoft products, which they would then work to fix as quickly as possible), and security training for employees. There were of course other companies and thought leaders who were instrumental in the development of secure by design thinking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*4DNRyeC0uOPe2usoXeyRVg.png" /><figcaption>It’s hard to imagine this is how people interacted with computers back then..</figcaption></figure><p>These days, there are new security trends (AI, data breaches, dark web, cryptocurrencies, quantum computing, cloud security) that are impacting the cyber security environment. It’s an ongoing challenge.</p><p><strong>Why is security a thing?</strong></p><p>It takes a lot of code to write a computer program. Many millions of lines usually. Code (especially software code) requires you to be very explicit about what you want the computer to do. This means that it’s inevitable that sometimes there will be some sort of ‘bug’ in the code that exposes some aspect of that program or system to something it wasn’t intended. This is known as a <strong>vulnerability</strong>. However, just because a vulnerability exists doesn’t mean it will necessarily be discovered or used. If it is used, we refer to it as an <strong>exploit</strong>. In cyber security, a <strong>zero day</strong> is an exploit that has never been used before (ie. the first day it’s used is day zero). The market for zero days is massive, with zero day exploits costing millions of dollars to those who might want to buy them (who may be criminals to exploit them, or governments to prevent criminals from exploiting them).</p><p>This might sound like a lot of money, but the potential income from exploiting any kind of vulnerability in a system can lead to much more value — money, information, reputation. Therefore, when software makers identify vulnerabilities in systems they want to patch those systems and patch them fast (these are often in software updates or specific patches you might install on your computer). However ideally the vulnerability never existed in the first place. This is the nature of cyber and information security — the practice of identifying and patching vulnerabilities, with the caveat that secure software design principles should prevent as many software vulnerabilities as possible. Just as vulnerabilities and exploits exist in traditional computer systems, they can also exist in AI systems — this is my trade.</p><p>An analogy often used in cyber security is that of a house. If you’re looking to secure a house, you might start by analysing how many windows and doors (and chimneys?) the house has. These are all access vectors, and contribute to the attack surface. To secure these you would do things like put locks on the doors and windows, and hide the keys in very secure places. If you really care about your house’s security, you could go so far as to build an impenetrable wall around the house, or even hire a mover to relocate your house to the moon. However a house is also for living in, and you need to make a compromise between how efficiently <strong><em>you</em></strong> can get in your house, and how much you want to keep your nosy neighbours out. Such is the balance in software security as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcRm2BCIVhVwkOoWd9TnbQ.png" /><figcaption>The house analogy.. credit to JDL Tech</figcaption></figure><h3><strong>Common cyber attack types</strong></h3><p><strong>Data breach: </strong>A data breach is an incident in which sensitive, confidential, or otherwise protected data is exposed to an unauthorised individual or entity. This can happen through a variety of ways, such as hacking, phishing, or social engineering. In Australia, the Office of the Australian Information Commissioner (OAIC) is responsible for managing data breaches. The OAIC has a number of resources available to help organisations manage data breaches, including a data breach notification scheme and a guide to responding to data breaches. In the US, the Federal Trade Commission (FTC) is responsible for managing data breaches. In the European Union (EU), responsibility for data breaches is regulated by the General Data Protection Regulation (GDPR).</p><p><strong>DDoS (Distributed Denial-of-Service) attack:</strong> A DDoS attack is an attempt to make a website or service unavailable to its intended users by overwhelming it with traffic from multiple sources. This can be done by using a botnet, which is a network of infected computers that are controlled by the attacker.</p><p><strong>Malware:</strong> Malware is software that is designed to harm a computer system or network. Malware can take many forms, such as viruses, worms, trojan horses, and ransomware.</p><p><strong>Virus:</strong> A virus is a type of malware that can replicate itself and spread from one computer to another. Viruses can be spread through email attachments, infected websites, or removable media such as USB drives.</p><p><strong>Worm:</strong> A worm is a type of malware that can replicate itself and spread from one computer to another, but it does not need a host file to do so. Worms can spread through email attachments, infected websites, or network vulnerabilities.</p><p><strong>BEC (Business Email Compromise): </strong>BEC is a type of fraud in which criminals impersonate a legitimate business in order to trick the victim into making a payment. BEC scams often involve sending emails that appear to be from a senior executive of the company, requesting a wire transfer or other financial transaction. AI technologies like video and audio deep fakes are making these kinds of scams (among others) harder to detect.</p><p><strong>Ransomware:</strong> Ransomware is a type of malware that encrypts the victim’s files and demands a ransom payment in order to decrypt them. Ransomware attacks can be very costly, both in terms of the ransom payment and the damage to the victim’s data. You may not hear about this so much in the media, because organisations are embarrassed to say they’ve been impacted and/or paid a ransom, but they are shockingly common.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfQ3wMSjdI-0OGCM36Kzhg.jpeg" /><figcaption>It’s impacting every industry.. credit to Safety Detectives.</figcaption></figure><h3><strong>Security by Design</strong></h3><p>Security by design (SbD) is an approach to security that incorporates security principles into the design of systems and products from the outset. This means that security is not an afterthought, but is instead considered at every stage of the development process.</p><p>SbD is based on the principle that it is easier and more effective to design security into a system than to try to retrofit it later. This is because security principles can be embedded into the system’s architecture, design, and implementation, making it more difficult for attackers to exploit vulnerabilities.</p><p>There are a number of different security principles that can be applied to SbD, including:</p><p><strong>Principle of Least Privilege</strong>: emphasises restricting users’ access rights to the minimum necessary for their specific roles. This reduces the potential impact of unauthorised actions and limits the damage caused by security breaches.</p><p><strong>Defence-in-depth</strong>: involves layering multiple security measures throughout an organisation’s infrastructure to provide overlapping protection. This approach reduces the likelihood of a single point of failure compromising the entire system’s security.</p><p><strong>Zero Trust</strong>: challenges traditional approaches by assuming that threats exist within the network and emphasising the need for continuous verification and strict access controls regardless of location.</p><p><strong>Patching and Vulnerability Management</strong>: underscores the importance of keeping software and systems up to date to address known vulnerabilities and reduce the risk of exploitation by attackers.</p><p><strong>Fail-safe defaults:</strong> This principle ensures that systems are configured in a secure state by default. This helps to reduce the risk of security vulnerabilities being exploited.</p><p><strong>Continuous monitoring:</strong> This principle involves continuously monitoring systems for security threats and vulnerabilities. This helps to identify and mitigate threats early on.</p><h3>What about Information Security?</h3><p>Without wanting to drown you in terminology, the main takeaway from the field of information security is that of the CIA Triad (no, not that CIA).</p><p><strong>Confidentiality:</strong> This principle ensures that sensitive information is accessible only to authorised individuals or entities. Confidentiality prevents unauthorised disclosure of data and aims to maintain privacy and protect sensitive data from falling into the wrong hands.</p><p><strong>Integrity:</strong> Integrity focuses on maintaining the accuracy and reliability of data. It ensures that information remains unaltered during storage, processing, and transmission. Maintaining data integrity prevents unauthorised modifications, corruption, or tampering.</p><p><strong>Availability</strong>: Availability ensures that authorised users have access to information and systems when needed. It aims to prevent disruptions, downtime, or denial of service attacks that could impact the availability of critical resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qz5C8zNn8_GwOn0q68UZPA.jpeg" /><figcaption>Not this CIA, but the CIA we’re dealing with is far more exciting *cough*</figcaption></figure><h3><strong>The Ecosystem</strong></h3><p>Perhaps most importantly, in my opinion at least, is that cyber and infosec has a massive ecosystem.</p><p>The cyber and information security ecosystem is a comprehensive framework encompassing technical innovations, governance structures, policy frameworks, human expertise, and legal considerations. This holistic approach is crucial for effectively managing cybersecurity and enhancing the resilience of digital systems. Each facet of the ecosystem contributes to the overall strength of the field.</p><p><strong>Technical:</strong> This involves leveraging advanced technologies, tools, and solutions to secure digital assets. Firewalls, intrusion detection systems, encryption mechanisms, and vulnerability scanning tools form the technical defences that safeguard against cyber threats and attacks.</p><p><strong>Governance</strong>: Effective cybersecurity governance establishes roles, responsibilities, and accountability across an organisation. It includes establishing security strategies, risk management protocols, and incident response plans. Governance ensures a coordinated approach to cybersecurity and aligns security efforts with business objectives.</p><p><strong>Policy</strong>: Policies provide guidelines and rules for secure practices and compliance. Security policies address data protection, access controls, authentication protocols, and more. Well-defined policies promote consistent security practices, reducing vulnerabilities arising from ad-hoc decisions.</p><p><strong>People</strong>: The human factor is paramount in cybersecurity. Educating and training individuals on security best practices and creating a culture of cybersecurity awareness are essential. Empowered and security-conscious employees play a pivotal role in identifying and mitigating threats.</p><p><strong>Legal</strong>: Legal frameworks and regulations establish the boundaries and responsibilities of organisations in maintaining cybersecurity. Compliance with regulations such as GDPR, HIPAA, and others ensures that organisations prioritise data protection and privacy. Organisations can even buy cyber insurance.</p><h3><strong>What does this mean for AI Security?</strong></h3><p>These are all principles that can be applied to AI Security. In fact, some Standards organisations are starting to recommend ways we can do just that, which I’m very happy about (also because it keeps me in business.. and makes the world a safer place).</p><p>However there are some key differences that need to be considered from a security perspective.</p><ul><li>Cybersecurity systems are designed to protect against known threats, while AI systems are constantly learning and evolving. This means that cybersecurity systems may not be able to keep up with the latest threats posed by AI systems.</li><li>Cybersecurity systems are typically rule-based, while AI systems are often probabilistic. This means that cybersecurity systems may not be able to handle the uncertainty and ambiguity that is often present in AI systems.</li><li>Cybersecurity systems are typically reactive, while AI systems can be proactive. This means that cybersecurity systems may not be able to prevent attacks that are initiated by AI systems.</li></ul><p>Many cyber and information security controls will not necessarily be effective against AI attacks, since the attack surfaces are completely different technologies. However, there are many ways cyber and information security controls can be adapted for use in AI Security.</p><p>And therefore we might need to consider:</p><ul><li><strong>Data security:</strong> AI systems often require access to large amounts of data in order to learn and improve. This data must be protected from unauthorised access, modification, or destruction.</li><li><strong>Model security:</strong> AI models are often proprietary and valuable assets. They must be protected from theft, reverse engineering, and tampering.</li><li><strong>Model selection:</strong> the choice of model will impact which vulnerabilities are present and how they can be exploited.</li><li><strong>Inference security:</strong> AI systems can be used to make inferences about individuals or groups. These inferences must be protected from unauthorised disclosure.</li><li><strong>Human-AI teams:</strong> humans interact with AI in a totally different way to computer systems, especially when it comes to trust and personification of human-like AI systems. Human psychology is a crucial consideration.</li><li><strong>Real world interaction:</strong> AI systems may be make or contribute to decisions that have real world consequences. There must be enough context in the system to enable good predictions.</li></ul><p>This is an area I would love to see more active development, and I highly encourage any readers to be creative and learn more about how cyber security could be applied to AI security.</p><p><strong>Cyber security references and resources</strong></p><p>MITRE <a href="https://attack.mitre.org/">ATT&amp;CK</a> is a resource for tactics, techniques and procedures (TTPs) in cyber — basically how adversaries do things. They now also have <a href="https://atlas.mitre.org/">ATLAS</a> for adversarial machine learning.</p><p><a href="https://owasp.org/">OWASP</a> stands for Open Web Application Security Project. It is a non-profit organisation that works to improve the security of software. OWASP provides free and open resources, including documentation, tools, and training, to help developers and organisations secure their web applications.</p><p>I also love the <a href="https://darknetdiaries.com/">Darknet Diaries</a> podcast by Jack Rhysider for learning about real world events that I often can’t believe really happened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*X4feacewdd5SHcQ0AY00dQ.jpeg" /><figcaption>Go listen to it, you won’t regret it.</figcaption></figure><p><strong>I definitely need to write another blog on this…</strong></p><p>I still haven’t really reached the tip of the iceberg when it comes to cyber security, and it makes me realise I must definitely dedicate another couple of blogs to cyber security and other possible intersections with AI. For example, physical penetration testing and social engineering are FASCINATING rabbit holes to go down and could both have their place in the AI Security world.</p><p>I highly encourage you to develop your hacking skills — but as a white hat only, of course ;)</p><p>In the next part we’re going to be diving into AI Governance.. and it’s more interesting than you might think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0907a18870f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 4: Adversarial Machine Learning]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-4-adversarial-machine-learning-f8daa9455419?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8daa9455419</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[adversarial-ai]]></category>
            <category><![CDATA[cyber-security-awareness]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sat, 14 Oct 2023 23:08:41 GMT</pubDate>
            <atom:updated>2023-10-14T23:08:41.723Z</atom:updated>
            <content:encoded><![CDATA[<p>Now I’ve touched on Adversarial Machine Learning a few times already. However the actual techniques that underpin it are varied and deserve their own blog.</p><p>This one is going to get a bit math-y. But only a bit, don’t worry.</p><p>According to the Center for Security and Emerging Technology (CSET) at Georgetown University, there were 344,000 journal papers, books, book chapters and conference papers published on the topic of Artificial Intelligence (AI) in 2021 (the last year we have results from so far). This number speaks to the hype around AI technologies, for which Machine Learning (ML) is the main driver. It also speaks to not just the commercial interest in AI solutions, but the academic interest in novel capability, improvements and applications. While not all ML solutions will be as complex as GPT-4, by 2020 it was reported that some sort of AI had been productionised in every major industry across tasks like computer vision, natural language processing, classification and prediction.</p><p>This rapid adoption has spurred interest in techniques that could deceive, disrupt or hijack ML models, and this evolved into the field of Adversarial Machine Learning (AML). Interest in hacking or evading algorithms has existed since the 2000s, for example as means to evade malware classification or spam filters. However, designing methods that exploit inherent characteristics of ML architectures and characteristics, including but not limited to the loss function, training process, reliance on training data, weights, ‘black box’ nature, and different convolutional layers, is what distinguishes AML. Also according to CSET, there were 17,000 papers published on adversarial machine learning between 2010 and 2021. This may surprise you, especially if you haven’t heard of AML before.</p><p>AI Security refers to the technical and governance considerations that pertain to securing AI systems from disruption, deception and disclosure. Adversarial Machine Learning is the field that develops offensive and defensive techniques to attack models (although it does tend to focus on the offensive) and since it stems from the research community, it can have a more academic than practitioner leaning to it.</p><p>Szegedy et al. were the first to apply AML techniques to images in 2014, with Goodfellow’s 2015 paper extending it to create the Fast Gradient Sign Method (FGSM). They added specially crafted adversarial perturbations to an image to deceive image classifications in an untargeted or targeted fashion, which could not be detected by the naked eye.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*WMyEXYxtHILju4qFLrDdtQ.png" /><figcaption>Panda misclassified as a gibbon in: <a href="https://arxiv.org/abs/1312.6199">https://arxiv.org/abs/1312.6199</a></figcaption></figure><p>Other image-based examples followed, including <a href="https://arxiv.org/pdf/1801.00349.pdf">glasses</a> that could evade facial recognition models, an adversarial sticker that could deceive person-detection classifiers, a 3-D printed <a href="https://arxiv.org/pdf/1707.07397.pdf#:~:text=Instead%2C%20we%20find%20that%20the,%E2%80%9Cholster%E2%80%9D%20and%20then%20%E2%80%9Cassault">turtle</a> that convinces classifiers it is a rifle, even when rotated, and a sticker placed over a small segment of a <a href="https://arxiv.org/pdf/1707.08945.pdf">stop sign</a> that was instead interpreted by autonomous vehicles as a speed sign.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*irMjuNY8rUTOtLMyPowZTA.jpeg" /><figcaption>A 3-D printed turtle that is misclassified as a rifle, no matter the rotation: <a href="https://arxiv.org/pdf/1707.07397.pdf#:~:text=Instead%2C%20we%20find%20that%20the,%E2%80%9Cholster%E2%80%9D%20and%20then%20%E2%80%9Cassault">https://arxiv.org/pdf/1707.07397.pdf#:~:text=Instead%2C%20we%20find%20that%20the,%E2%80%9Cholster%E2%80%9D%20and%20then%20%E2%80%9Cassault</a></figcaption></figure><p>These methods have since been transferred to many other fields and input data types: banking, finance, social media, video (and deep fakes), network intrusion detection, healthcare, accounting, IoT, and communications. Improvements in the accuracy, dynamism and reactivity of these techniques, combined with a far less mature (and often non-existent) approach to ML defence and security, renders AML a burgeoning threat for all productionised ML models.</p><p>Since the explosion of Adversarial Machine Learning papers from 2014, the number of attack classes and subclasses have multiplied to over a hundred according to MITRE’s Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS) knowledge base, which lists AML tactics, techniques, and case studies, as a complement to the ATT&amp;CK framework for cyber security. There are even more if we include bespoke methods in academic papers. The Adversarial Robustness Toolbox (ART) is an open source Python library for machine learning security published by IBM, and the largest resource for AML methods and implementations.</p><p><strong>Some maths</strong></p><p>Ok here I have to include some maths but don’t worry, it’s only the necessary amount to be able to start a conversation about different AML methods without feeling like a loser.</p><p><strong>Universal Adversarial Perturbations</strong></p><p>The Universal Adversarial Perturbation (UAP) method adds a perturbation vector to an entire image that alters each pixel, for each image, by a unique value, but is not based on the loss function of the target model. The aim is that this vector be small enough to be unobserved by humans, but significant enough to fool the classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*YwEPdOx5dTkYg9IPxioDQg.png" /></figure><p>This technique is basically a fancy way saying that we’re adding random noise with a bounded magnitude. Its success really exposes how brittle many ML models are.</p><p><strong>The Fast Gradient Sign Method</strong></p><p>The Fast Gradient Sign Method (FGSM) was first proposed by Szegedy and Goodfellow in 2014. This method calculates the gradient of the model’s loss function relative to the input. The attack is based on a one step gradient update along the direction of the gradient’s sign at each time stamp, clipped by the epsilon value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKlK1Lvn52fnS4nIZxbPCg.png" /></figure><p>Its one-shot nature means it is computationally inexpensive but does not necessarily calculate the optimal perturbation vector. Despite its relative simplicity it has been effective against many target models.</p><p><strong>Projected Gradient Descent</strong></p><p>Projected Gradient Descent (PGD) solves this limitation of FGSM by taking it a step further and instead of a one-step gradient update, takes an iterative approach. It calculates the loss with respect to the true classification over many iterations (t), still constrained by the epsilon value. This means it is more likely to find the perturbation vector with the global optimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*EwSlPkxhIxPa0u0iSPIiAw.png" /></figure><p>This method employs the L-infinity norm because it has access to the compute gradients across the entire image. This is a math-y way of saying that when ‘distances’ between the true and target value are calculated, it is done a specific way (to take the absolute value of the maximum). This is in contrast to the L-1 or L-2 norm, but don’t worry about those for now, just know at this stage that when talking to maths people they might refer to L-norms and they are basically referring to how the optimisation is performed.</p><p><strong>One Pixel Attack</strong></p><p>The One Pixel Attack (OPA) also uses an iterative approach whereby it identifies a single pixel that, when perturbed, is most likely to cause the object to be misclassified. They convolve the original image with a single adversarial pixel perturbed to be some RGB colour, and identify the pixel location and colour that maximally perturbs the confidence for that image.</p><p><strong>Adversarial Patch</strong></p><p>The adversarial patch is an image which, placed near or on the target object, causes the classifier to ignore the item in the scene and report the chosen target of the patch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ygOBaftHEEFw5OSvz8meGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stSfTu8bqADcRqfD_gTXaQ.jpeg" /><figcaption>Adversarial patch fooling a human detector: <a href="https://arxiv.org/pdf/1904.08653.pdf">https://arxiv.org/pdf/1904.08653.pdf</a></figcaption></figure><p>Now there are many other techniques but I don’t have time to detail 95+ more. Go check them out <a href="https://github.com/Trusted-AI/adversarial-robustness-toolbox">here</a> or <a href="https://atlas.mitre.org/">here</a> if you’re interested.</p><p>So I know there’s a bit of math here and that’s not for everyone, but if there’s anything you take away from this section it should be this: all of the techniques I describe mention the word <strong><em>optimisation</em></strong>.</p><p><strong>Optimisation</strong></p><p>In machine learning, optimisation refers to the process of finding the best parameters for a machine learning model. This is done by minimising a loss function, which is a measure of how well the model performs on a given dataset. There are many different optimisation algorithms that can be used to find the best parameters for a machine learning model. (Some of the most common algorithms include gradient descent, stochastic gradient descent, and Bayesian optimisation.)</p><p>Imagine a hill. The hill represents the loss function, and the goal is to find the lowest point on the hill, which represents the global minimum. The slope of the hill represents how quickly the loss function is changing. Optimisation algorithms can be thought of as walkers that are trying to find the lowest point on the hill. The walkers start at a random point on the hill and then take steps in the direction of the steepest descent. This means that they are always moving towards the lowest point on the hill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*aMEfqTkLl56T16axOEa-eA.jpeg" /><figcaption>A way of visualising the gradient of a model — basically the higher dimension space of potential model parameters and how accurate the resulting model is</figcaption></figure><p>Many adversarial machine learning algorithms use the loss function but instead of moving down it (which makes the model parameters more optimised), it seeks to move up the loss function (and make the model less optimum, ideally in such a way that it causes a classification but does not move so far away that it is obvious to an observer). The process of ‘moving up’ the slope is akin to taking a mower or an earthmover (I’m clearly no expert in land clearing) to the hill and reshaping it so that not only is the path down the hill slightly different, but so is the rest of the hill (like updating pixels in an adversarial example image).</p><p><strong>Attack surface taxonomy</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*si6dfra17yACBGS4lsyT0Q.png" /><figcaption>The ML Lifecycle aka attack surface</figcaption></figure><p>AML attacks can occur at every point in the ML Lifecycle, across training and inference. High-level attack classes include poisoning attacks, extraction attacks, evasion attacks and inference attacks. A systems-based approach to security should be considered as AI systems typically include many interacting ML and software systems.</p><p>AML attacks are further characterised by the adversary’s knowledge, specificity, attack frequency, and objective. The adversary’s knowledge of the target model is referred to as white, grey or black box. White-box presumes the attacker has full knowledge of the internal characteristics of the target model, which includes model weights, training data or loss function. On the other end of the spectrum, black-box necessitates no knowledge of the target model and is usually assumed to be the state of a real adversary. Grey-box refers to some point on the continuum between white and black box, where some of the characteristics of the target model is known. Attack specificity refers to what extent an adversary has a specific outcome in mind. A targeted attack directs the model to a predefined outcome. For example, adversarial perturbations that cause a classifier to predict a specific class. A non-targeted attack might aim to disrupt the model’s behaviour, but without an explicit outcome, such as adding random noise to an image that reduces its overall accuracy.</p><p>These characteristics, and the continuum for each, can be used for threat modelling to determine the impact of AML attacks on a target given contrasting adversary capability. Despite the multitude of attacks, there are many categories of ML that have yet to inspire attacks or be investigated for target or threat analysis.</p><p><strong>Defences</strong></p><p>In a future deep dive blog I’ll go into more depth on the range of defences, and how they can practically be employed. There are a number of defences that can be employed to mitigate the risk an AML attack will be successful against a system. Some are extensions of existing cyber security controls, however others are unique to the ML lifecycle and architecture of specific implementations. Cyber security controls that manage training data and constrain input data to the model reduce the risk of poisoning and extraction attacks. However the specific implementation of such controls may still leave open vulnerabilities, as recurring cyber incidents evidence.</p><p>Mitigations that are unique to AI Security include adversarial training, where a model is trained on those adversarial examples that might otherwise be successful at an evasion attack. However, this often comes at the expense of accuracy, a trade-off many organisations are unwilling to make (especially when accuracy is a proxy for profit). Adversarial training is also more effective on some use cases than others, for example it is very effective on computer vision but more challenging for other data types. Gradient obfuscation is another defensive method that mitigates against those white box attacks that rely on access to the target model’s gradient function. However, black box attacks against a surrogate model are often still extremely effective, especially since models trained on the same or similar data tend towards comparable internals. The defensive field is broad, with many other potential defences that focus on defence against specific attacks, or distributed approaches that spread risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vfyfN5tIFX_t-cPrAbCRvA.png" /><figcaption>AML defences (although the right approach is not attack vs defence, but security from the start and risk mitigation)</figcaption></figure><p>I would argue that the maturing of the AI Security field lies in moving the paradigm of attack versus defence to one of assurance and risk management. This is why, in discussing the security of an AI system, we focus less on the technical defences that could be put in place but rather on the needs of the ecosystem to enable and mature the practice of AI Security.</p><p>The current landscape is marked by an arms race that is emerging between adversarial machine learning techniques and model robustness. Unfortunately, robustness is often still an afterthought among the ML community. This is changing though, and I’ve been so delighted by the progress I’ve seen in the academic and practitioner community.</p><p>There’s still a long way to go, though.</p><p>The next blog will provide an introduction to cyber security, a field that has many contributions that can be offered to AI Security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8daa9455419" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to AI Security Part 3: real AI attacks]]></title>
            <link>https://medium.com/@harrietfarlow/intro-to-ai-security-part-3-real-ai-attacks-28da49900a99?source=rss-bc3fb5455872------2</link>
            <guid isPermaLink="false">https://medium.com/p/28da49900a99</guid>
            <category><![CDATA[adversarial-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cyber-security-awareness]]></category>
            <category><![CDATA[hacking]]></category>
            <dc:creator><![CDATA[HarrietHacks]]></dc:creator>
            <pubDate>Sat, 07 Oct 2023 22:18:46 GMT</pubDate>
            <atom:updated>2023-10-14T23:12:40.141Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the most common questions I’m asked is a variation on — is the AI threat real?</p><p>My response: it’s coming. Soon.</p><p>When I first started my PhD in machine learning security I had already been working as a data scientist for six years. It was the first time I became aware that AI or ML systems could be vulnerable to attack. Part of me couldn’t believe that in all my years as a data scientist, while bias and fairness was typically discussed, the <strong>security</strong> of the models we built was never a topic of conversation. It was always assumed that any security issues would be covered by an organisation’s cyber and information security practices. Unfortunately, this isn’t true.</p><p>This blog post will explore a range of attacks that can be employed on AI systems, and will touch on to what extent they would be protected against by cyber and information security controls.</p><p>The range of possible attacks on AI is limited only by your creativity, because as AI is increasingly adopted in different ways, the ways it can be attacked expands as well.</p><p>In general, attacks on models full under the themes disrupt, deceive and decode.</p><p><strong><em>Disrupt</em></strong><em> — cause the model to malfunction or break</em></p><p><strong><em>Deceive</em></strong><em> — in a targeted way, cause the model to misinterpret its surroundings</em></p><p><strong><em>Disclose</em></strong><em> — cause the model to leak information it’s not meant to.</em></p><p>All of these attacks could be used across a variety of AI systems, but below I’m going to highlight a few specific use cases to illustrate the point.</p><h3><strong>Disrupt</strong></h3><p>These attacks cause the model to malfunction or break in some way.</p><p>This could encompass a range of failures, like compromising the model’s ability to accurately understand its environment, or preventing it from working at all. Consider this an equivalent of a DDOS (distributed denial of service) attack in cyber security — a classic attack where so many requests are sent to a server that it overloads and crashes.</p><p><strong>Attacks on autonomous vehicles</strong></p><p>Autonomous vehicles lie on a spectrum of automation that are recognised in the industry as levels 0 through 5, where level 0 is no automation and level 5 is full automation. Levels 1 through 3 include varying computer vision based driver assistance technologies, like identifying lane markers or helping you reverse parallel park (but then how else would I show off my driving skills to my friends?).</p><p>Therefore, attacks on these systems could prevent these cars’ AI from fulfilling these functions — preventing the recognition of lane markers, stop signs, speed signs etc. In theory, if we’re talking Driver Assist, the worst case scenario is that a human might have to take over, and you’d feel kind of annoyed. At worst, full automation might be engaged and a failure could lead to a car driving through a stop sign and causing damage, or worse, fatalities. Adversarial attacks have been demonstrated to work in settings like this, by perturbing stop signs and speed signs using adversarial machine learning techniques so that cars don’t recognise them and drive straight through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*I-rGGIgtkNJ_oBdIn3SvEQ.png" /><figcaption>Adversarial patches on stop signs: <a href="https://browse.arxiv.org/pdf/1707.08945.pdf">arxiv.org/pdf/1707.08945.pdf</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYvAaGxJNq0_hMfQgFKGVg.jpeg" /><figcaption>Adversarially crafted stop signs: <a href="https://substance.etsmtl.ca/en/brittleness-of-deep-learning-models">https://substance.etsmtl.ca/en/brittleness-of-deep-learning-models</a></figcaption></figure><p>Autonomous vehicles also use machine learning models for signal classification, because autonomous vehicles have signals coming in like GPS, LiDAR (radar) and bluetooth. This is used both on the road to assist with driving, and for things like vehicle health monitoring and detecting when other objects come within range of the car (like the car keys). Or so people tell me, my car doesn’t have this kind of technology, much as I love her. Attacks on these systems can cause a vehicle to misinterpret its location, or not receive incoming communications and commands. The following papers describe attacks of this sort:</p><ul><li><a href="https://arxiv.org/abs/1707.07397">Adversarial Attacks on Deep Neural Networks for Traffic Sign Recognition by Eykholt et al. (2017)</a></li><li><a href="https://arxiv.org/abs/1807.07871">Adversarial Robustness for Autonomous Driving by Carlini et al. (2018)</a></li><li><a href="https://arxiv.org/abs/2003.14238">Towards Robustness against Adversarial Attacks for Autonomous Driving by Zhou et al. (2020)</a></li></ul><p><strong>Medical imaging</strong></p><p>AI can also be used in medical imaging, in both computer vision and signal processing. Medical imaging, naturally, is a field where images of the body are used to diagnose conditions — like ultrasounds, CT scans, X-rays etc. This is a computer vision application in the sense that humans inspect these images to make a diagnosis, but it is also a signals classification problem in the sense that the data captured is usually a bunch of electromagnetic signals, and they are translated into images so that humans can interpret them — but if an AI could make a classification based on the signals then that would also do the job. AI applied to these kinds of problems already have the challenge that getting enough labelled data and making accurate models is hard enough, adding noise (adversarial or accidental) could mean the AI doesn’t detect something that present, or detects something else. Perhaps an underestimated impact, brittle AI can reduce trust in those organisations that use it, and the medical field is obviously an area where maintaining the trust of the public is very important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wepS3NoJs4jXp9x9PIUqrw.png" /><figcaption>Adversarial attacks on medical imaging: <a href="https://dongxiaozhu.github.io/ISBI_adversarial.pdf">https://dongxiaozhu.github.io/ISBI_adversarial.pdf</a></figcaption></figure><p>The following papers detail attacks in this domain:</p><ul><li><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8932490/">Adversarial Attacks on Medical Image Classification Models: A Survey” by Marina Z. Joel, Sachin Umrao, Enoch Chang, Rachel Choi, Daniel X. Yang, James S. Duncan, Antonio Omuro, Roy Herbst, Harlan M. Krumholz, and Sanjay Aneja</a></li><li><a href="https://arxiv.org/abs/2106.06825">Adversarial Machine Learning in Medical Imaging: A Review of Methods and Applications” by Amirhossein Hosseinzadeh, Hossein Vaezi, and Mohammad Ali Amiri.</a></li><li><a href="https://arxiv.org/abs/2203.08618">Adversarial Examples in Medical Imaging: A Survey of Methods and Applications” by Xiaolong Wang, Fei Wang, Jiankang Deng, and Xindong Wu</a>.</li></ul><p><strong>Military systems</strong></p><p>There are lots of discussions around current and potential uses of AI in military systems, and this is an area that should have more stringent requirement — not necessarily because all use cases could lead to direct or kinetic action (ie. war or missiles, although this is important to consider) but because the military, as a Government entity, requires a different licence to operate from the public, and has to adhere to much higher standards of public trust.</p><p>One use of computer vision in the military is ISR — intelligence, surveillance, reconnaissance. In layman’s terms, this is figuring out what other forces are doing, and why. This could be performed by a range of platforms (the military term that refers to aircraft, ships or land vehicles like tanks) or drones. Using adversarial methods to disguise objects from detection can be used by both friendly and enemy forces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s7-1AdspgyQwnqAhK0RdHA.jpeg" /><figcaption>Threat tracker is an AI system aiming to automatically classify objects in the environment using computer vision and RF: <a href="https://www.navsea.navy.mil/Media/News/Article/2103621/new-technology-can-autonomously-track-uas-from-land-sea/">https://www.navsea.navy.mil/Media/News/Article/2103621/new-technology-can-autonomously-track-uas-from-land-sea/</a></figcaption></figure><p>The military also relies on signals as autonomous vehicles do, across the RF (radio-frequency) spectrum and in GPS etc.</p><ul><li><a href="https://arxiv.org/abs/2202.08892">Developing Imperceptible Adversarial Patches to Camouflage Military Assets From Computer Vision Enabled Technologies by Chris Wise and Jo Plested</a></li><li><a href="https://arxiv.org/abs/1908.08263">Adversarial Machine Learning for Military Applications by Michael C. Horvitz, John C. Smith, and Percy Liang (2019)</a></li><li><a href="https://arxiv.org/abs/2001.01209">Adversarial Machine Learning in Military Robotics by Hao Zhang, David C. Montgomery, and Krishnamurthy Dvijotham (2020)</a>.</li></ul><p><strong>Cyber security</strong></p><p>Most models for cyber security defence rely on signals information — data about commands being executed in the network or computer, application data and about information being sent or received outside the network. Attacks on these models could prevent them from recognising any of this information. A later blog is going to be dedicated to AI for cyber security and will go into this in more detail.</p><p>Examples of this in academia include:</p><ul><li><a href="https://arxiv.org/abs/1412.6572">Adversarial Machine Learning at Scale by Ian Goodfellow, Jonathon Shlens, and Christian Szegedy (2014)</a>.</li><li><a href="https://arxiv.org/abs/1511.07528">Towards Robust Adversarial Machine Learning by Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, and Brendan McMahan (2016)</a>.</li><li><a href="https://arxiv.org/abs/2103.10599">Adversarial Machine Learning for Cybersecurity: A Survey by Yuanyuan Zhang, Haitao Li, and Wei Wang (2021)</a>.</li></ul><h3><strong>Deceive</strong></h3><p>Techniques that can be used to disrupt a model may also be used to deceive it in a targeted way.</p><p>For example, using the same adversarial machine learning techniques that can be used to create a disruptive adversarial example, can also be designed with a specific target in mind. In this way, the model is ‘convinced’ that it sees this target, when it might be looking at something else. In the next blog, I’ll go into more detail about adversarial machine learning and how these attacks work.</p><p>For example, autonomous vehicle computer systems might not only be prevented from recognising a stop sign, but they could be convinced that they are actually looking at a speed sign telling them to go at 100 (miles or km, up to you. I’m trying to be region agnostic here). Accelerating instead of stopping is obviously a pretty dangerous predicament.</p><p><strong>Facial recognition</strong></p><p>Targeted attacks against facial recognition models would not just prevent the AI from recognising someone (a disruption attack), but could also deceive the model by convincing that one person is actually somebody else. Facial recognition is already used for identification at airports (to ensure border security), banks (to process things like identity and credit checks), and in consumer goods to enhance customer experience (unlocking your phone, for example).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fr_K6TH0KUU80C8gpiA2Mg.png" /><figcaption>Adversarial glasses and stickers that prevent facial recognition: <a href="https://www.researchgate.net/figure/Adversarial-facial-images-generated-by-a-a-glasses-attack-and-b-our-attack_fig4_351804897">https://www.researchgate.net/figure/Adversarial-facial-images-generated-by-a-a-glasses-attack-and-b-our-attack_fig4_351804897</a></figcaption></figure><ul><li><a href="https://arxiv.org/abs/1901.07228">Adversarial Examples for Facial Recognition in the Wild by Su et al. (2019)</a></li><li><a href="https://arxiv.org/abs/2005.02237">Adversarial Attacks on Face Recognition Systems: A Survey by Li et al. (2020)</a></li><li><a href="https://arxiv.org/abs/2103.14799">Towards Robust Face Recognition Against Adversarial Attacks by Zhou et al. (2021)</a></li></ul><p><strong>Trading models</strong></p><p>Adding targeted false transactions to a trading model could cause the model to exhibit subtle shifts, allowing the attacker to profit. Even small changes can lead to substantial financial benefits over time. In finance and trading, adversarial machine learning can be used to attack a variety of models, including:</p><ul><li>Price prediction models: These models are used to predict the future price of assets. An adversary could use adversarial machine learning to create adversarial examples that cause the model to make incorrect predictions. This could be used to manipulate the market or to make profits by trading against the model.</li><li>Risk assessment models: These models are used to assess the risk of financial investments. An adversary could use adversarial machine learning to create adversarial examples that cause the model to underestimate the risk of an investment. This could lead to investors making bad investment decisions.</li><li>Fraud detection models: These models are used to detect fraudulent transactions. An adversary could use adversarial machine learning to create adversarial examples that cause the model to miss fraudulent transactions. This could allow fraudsters to steal money from financial institutions.</li></ul><p>As we can see in the following papers:</p><ul><li><a href="https://arxiv.org/abs/1805.01963">Adversarial Machine Learning for Financial Trading by Michael M. Mansour, Amin Karbasi, and John C. Duchi (2018).</a></li><li><a href="https://arxiv.org/abs/2002.02124">Adversarial Attacks on Financial Machine Learning Models: A Survey by Jialei Wang, Yuanyuan Zhang, and Xiangliang Zhang (2020)</a>.</li><li><a href="https://arxiv.org/abs/2003.02918">Robust Financial Machine Learning with Adversarial Training by Jianyu Wang, Yilun Wang, and Song Han (2020)</a>.</li></ul><p><strong>Data poisoning and backdoors</strong></p><p>Data poisoning involves injecting malicious or deceptive data into training datasets to manipulate the behaviour of AI models during training, leading to compromised model performance in real-world applications. Backdoors in AI refer to hidden vulnerabilities intentionally inserted during model development, which can be exploited by attackers to trigger specific behaviours or unauthorised access in the deployed AI system.</p><ul><li><a href="https://arxiv.org/abs/1807.08616">Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks by Asghar Ashafahi, Mahdi Hajirasouliha, and Nicolas Papernot (NeurIPS 2018)</a>.</li><li><a href="https://arxiv.org/abs/1812.00496">How to Backdoor Federated Learning by Eric Bagdasaryan, Nicholas Carlini, and Ian Goodfellow (ICLR 2019)</a>.</li><li><a href="https://arxiv.org/abs/1906.02851">Adversarial Machine Learning at Scale: Poisoning Attacks on Image Classification Models by Xinyun Chen, Zhiwu Lu, and Dawn Song (USENIX Security 2020)</a>.</li></ul><h3><strong>Disclose</strong></h3><p>These attacks cause the model to leak information it’s not meant to, about the training data or the model itself.</p><p>This is also referred to as exfiltration.</p><p><strong>Chatbots</strong></p><p>LLMs are the backbone of chatbots like ChatGPT (by OpenAI) and BARD (by Google). Attacks on these systems can provoke Chatbots to give back dangerous or toxic responses (referred to as jailbreaking).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DyDJNoep_JQbtynfpmVn1w.png" /><figcaption>LLMs are taught what they should and shouldn’t answer, and this mostly works: <a href="https://ambcrypto.com/heres-how-to-jailbreak-chatgpt-with-the-top-4-methods-5/">https://ambcrypto.com/heres-how-to-jailbreak-chatgpt-with-the-top-4-methods-5/</a></figcaption></figure><ul><li><a href="https://arxiv.org/abs/2301.08243">Prompt Engineering for Large Language Models: A Survey by Andrew Gao, Kory Becker, and Ilya Sutskever (2023)</a></li><li><a href="https://arxiv.org/abs/2201.07285">Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners by Zhe Cao, et al. (2022)</a>.</li><li><a href="https://arxiv.org/abs/2201.05709">Prompt-Guided Learning for Language Understanding by Yilun Wang, et al. (2022)</a>.</li></ul><p><strong>Privacy leaks</strong></p><p>Adversarial AI privacy leaks involve the exploitation of vulnerabilities in AI systems to extract sensitive information or breach privacy through carefully crafted inputs designed to deceive the system. One way to do this is to create adversarial examples that are designed to cause a machine learning model to make an incorrect prediction. If the incorrect prediction reveals sensitive information, then the adversary has successfully leaked that information.</p><p>Another way to leak sensitive information from machine learning models is to use a technique called membership inference. Membership inference is a technique where it is inferred whether or not a particular input was used to train a machine learning model. In the context of privacy leakage, an adversary could use membership inference to infer whether or not a particular person’s data was used to train a machine learning model.</p><ul><li><a href="https://arxiv.org/abs/1802.07228">Adversarial Examples for Evaluating Natural Language Understanding Systems by Ilyas et al. (2019)</a></li><li><a href="https://arxiv.org/abs/1901.11196">TextFooler: A Simple and Effective Method for Fooling Text Classification Models by Xu et al. (2019)</a></li><li><a href="https://arxiv.org/abs/2002.10929">Data Poisoning Attacks on Large Language Models by Carlini et al. (2020)</a></li></ul><p><strong>Model theft</strong></p><p>Model theft in AI occurs when adversaries illicitly gain unauthorised access to or replicate valuable machine learning models, potentially leading to intellectual property theft and misuse of proprietary algorithms and insights. One way to steal a machine learning model is to use a technique called transfer learning. Transfer learning is a technique where the knowledge learned by a model on one task is transferred to a model on a different task. In the context of model theft, an adversary could use transfer learning to steal a model by training a new model on a small amount of data that is similar to the data that the original model was trained on.</p><p>Another way to steal a machine learning model is to use a technique called model inversion. Model inversion is a technique where the input data that was used to train a model is inferred. In the context of model theft, an adversary could use model inversion to steal a model by inferring the training data that was used to train the model.</p><ul><li><a href="https://arxiv.org/abs/1708.06131">Model Stealing via Transfer Learning by Carlini et al. (2017)</a>.</li><li><a href="https://arxiv.org/abs/1607.08861">Model Inversion Attacks on Neural Networks by Papernot et al. (2016)</a>.</li><li><a href="https://arxiv.org/abs/1706.06083">Towards Robust Neural Networks Against Adversarial Examples by Madry et al. (2017)</a></li></ul><h3><strong>The so what</strong></h3><p>Why does this matter? AI is currently used in so many fields, and it’s only increasing. Most of these applications use AI that may be insecure.</p><p>There are already many examples of AI incidents being recorded. Below are a few of these repositories.</p><p><strong>The AI Incidents database</strong></p><p><a href="https://incidentdatabase.ai/">Welcome to the Artificial Intelligence Incident Database</a></p><p><strong>AIAAIC database</strong></p><p><a href="https://www.aiaaic.org/">AIAAIC</a></p><p><strong>AI Risk database</strong></p><p><a href="https://airisk.io/">AI Risk Database</a></p><p>Many of the existing logged risks are related to language-related risks (misinformation, jailbreaking, general errors) and computer vision deep-fakes. I list just a few examples below.</p><p><strong>Language</strong></p><p>Facebook Gave Vulgar English Translation of Chinese President’s Name</p><p><a href="https://incidentdatabase.ai/cite/414">Incident 414: Facebook Gave Vulgar English Translation of Chinese President&#39;s Name</a></p><p><strong>Deep fakes</strong></p><p>Indian Police Allegedly Tortured and Killed Innocent Man Following Facial Misidentification</p><p><a href="https://incidentdatabase.ai/cite/483">Incident 483: Indian Police Allegedly Tortured and Killed Innocent Man Following Facial Misidentification</a></p><p><strong>Amazon Fresh Cameras Failed to Register Purchased Items</strong></p><p><a href="https://incidentdatabase.ai/cite/520">Incident 520: Amazon Fresh Cameras Failed to Register Purchased Items</a></p><p><strong>Google’s YouTube Kids App Presents Inappropriate Content</strong></p><p><a href="https://incidentdatabase.ai/cite/1">Incident 1: Google&#39;s YouTube Kids App Presents Inappropriate Content</a></p><p>The main caveat here is that these attacks are strictly AI Security attacks in the sense that they usually don’t occur because of some disruption to the AI system, but because of the way the system is used.</p><p>That said, they lay a prior for what’s to come, and since I started tracking these incidents they have grown in scale, complexity and size. For instance, some of these incidents occur due to vulnerabilities that are exploited by accident or misadventure and not on purpose. So we know what is to come.</p><p>The next blog will be diving into the specific offensive techniques that allow us to attack models in the field of Adversarial Machine Learning.</p><p>See the video blog here:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FYb2Gbarj6AY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYb2Gbarj6AY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FYb2Gbarj6AY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b474e9878ab1f8b3fbfff657645ad942/href">https://medium.com/media/b474e9878ab1f8b3fbfff657645ad942/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28da49900a99" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>