<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ilmoi on Medium]]></title>
        <description><![CDATA[Stories by ilmoi on Medium]]></description>
        <link>https://medium.com/@ilmoi?source=rss-b7592f20afda------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1ttczyUg838PoaosXw3gkQ.png</url>
            <title>Stories by ilmoi on Medium</title>
            <link>https://medium.com/@ilmoi?source=rss-b7592f20afda------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:17:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ilmoi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Relational Database in a Single Picture]]></title>
            <link>https://ilmoi.medium.com/relational-database-in-a-single-picture-dcc5685a4105?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcc5685a4105</guid>
            <category><![CDATA[relational-databases]]></category>
            <category><![CDATA[cs186]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[dbms]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Fri, 14 Aug 2020 10:09:26 GMT</pubDate>
            <atom:updated>2020-08-14T10:09:26.683Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I completed Berkeley’s amazing <a href="https://cs186berkeley.net/">CS186 course</a> on Database Management Systems (DBMS).</p><p>I followed online lectures by Prof Hellerstein <a href="https://www.youtube.com/user/CS186Berkeley/playlists">on Youtube</a> — and self-sat a few exams from CS186’s <a href="https://cs186berkeley.net/resources/">resources page</a>.</p><p>Firstly, if you’re considering (even considering!) learning databases —I’d say <strong>do it</strong>. It’s a technology that powers most of today’s economy and it’s not going away. In addition, compared to some of the other CS courses (I’m following <a href="https://teachyourselfcs.com/">TeachYourselfCS</a>) I found it to be one of the most practical. So yes, do it:)</p><p>Secondly, if you’re a visual thinker like myself then perhaps you’ll find the below useful. As I went through the course I filled in a spatial diagram of all the various parts of a RDBMS (R = relational, which is the type most of the course is about). It really helped me bring all the concepts together in one place — and perhaps will help you too.</p><p>Png over here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5K4CMQRbaGVnjn8t4IeXYg.png" /></figure><p>And you can find the raw draw.io file in my <a href="https://github.com/ilmoi/cs186-diagram">github repo</a>. Feel free to edit &amp; re-use.</p><p>Enjoy:)</p><p>PS if you spot any mistakes just make a PR on the repo or let me know. Nobody’s perfect!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcc5685a4105" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Matrix Phylogeny for Machine Learning]]></title>
            <link>https://medium.com/data-science/matrix-phylogeny-for-machine-learning-7a9e65b8f8aa?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a9e65b8f8aa</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Thu, 12 Dec 2019 00:12:28 GMT</pubDate>
            <atom:updated>2019-12-12T00:12:28.104Z</atom:updated>
            <content:encoded><![CDATA[<h4>Linear algebra lingo, visualized</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e91txnZK8SEfDZ-bm_RWcQ.jpeg" /><figcaption><a href="https://uwaterloo.ca/astereae-lab/research/goldenrods/intuitive-phylogeny-solidago-0">Source.</a></figcaption></figure><p>If you’re like me, you think in pictures, not words.</p><p>As I was teaching myself math for ML, I kept being bombarded with various linear algebra terms: <em>Symmetric</em> matrix this, <em>Normal</em> matrix that, <em>Regular</em> matrix this and that.</p><p>At some point, I gave in and did my favorite thing in the world — made a diagram of how it all fits together:) I’m going to post it here in case anyone else needs help with disambiguation, and is a visual person like me.</p><p>Without further ado, here we go:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LeqCzjx6GMoenUMWD3zxHQ.png" /><figcaption>Right click and “save image as” to download.</figcaption></figure><h3>README</h3><p>The diagram is to be read top to bottom. I built it for Real, mainly square matrices.</p><p>One of the first and key separations I make is I break down matrices into two types based on a)their eigenvalues/vectors (“eig”) and b)their determinant behavior (“det”). That gives us two separate branches to explore (left and right, respectively).</p><p>Black (vertical) arrows between boxes indicate hereditary relationships. Any box is <em>necessarily</em> a subtype of the box right above it, and <em>necessarily</em> inherits any properties of its parent(s). For example, every Normal matrix is necessarily Diagonalizable and every Symmetric matrix is necessarily Normal. This also means that every Symmetric matrix has both <em>n</em> linearly independent (L.I.) eigenvectors (inherited from Diagonalizable) and commutes with its conjugate transpose (inherited from Normal).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*Sq5IIj9ihM3IcmstHPl9PQ.png" /></figure><p>Colored (non-vertical) arrows fill in the relationship gaps for matrix-types that don’t otherwise sit under another. For example, because Orthogonal matrices sit in a different branch to say Normal and Symmetric, following just the black vertical arrows alone is insufficient to understand how they relate to each other. Are all Normal matrices Orthogonal? Are all Orthogonal matrices Normal?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*C2b1_4aQAvDxTaeZZ8w-8g.png" /></figure><p>I didn’t bother to fill in every possible relationship, just the ones that felt important. By the way, if you’re more of a table person, this was the table I filled out prior to drawing the arrows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPV8TvzfRUwv5pB4NlOkGA.png" /><figcaption>(matrices on top defined <em>through matrices on the left)</em></figcaption></figure><p>Next, colors. Blue text inside the boxes is the best / most succinct definition I could find for each matrix type. Sometimes I also included useful to know properties in blue, eg “λ are Real” (eigenvalues are Real) or “∃ SVD” (there exists a diagonal form that can be reached through <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">SVD</a>).</p><p>Box colors:</p><ul><li>Gray boxes = terminal and not very interesting ones.</li><li>Blue boxes = connected through spectral theorem (see at the bottom of the diagram).</li><li>Thin orange boxes = Complex equivalents (the diagram is about Real matrices, but I couldn’t help to include a few popular Complex versions).</li><li>White boxes = normal, don’t belong to the above.</li></ul><p>Arrow colors:</p><ul><li>Red = two-way “not necessarily” relationship</li><li>Orange = one-way “not necessarily” relationship</li><li>Green = special case of rotational matrices being both defective and not (over Reals and over Complex numbers, respectively)</li></ul><p>And a few final things:</p><ul><li>Numbers in bottom right corners represent example matrices for each type.</li><li>I’ve included the spectral theorem at the bottom and listed how it changes depending on whether A = just Diagonalizable, Normal, or Symmetric. It should be pretty self-explanatory.</li><li>I’ve also included the 5 rules for Positive-Definite matrices (stolen from Gilbert Strang’s awesome YT lecture <a href="https://www.youtube.com/watch?v=ojUQk_GNQbQ">here</a>), because why not.</li></ul><h3>Living document</h3><p>The diagram is as complete as I could make it, but I’m sure I’ve missed things (and maybe even gotten some stuff wrong 🤭🤭🤭).</p><p>Treat this as a living document — if you comment below with any ideas/fixes, I will add the changes and update the post.</p><p>Happy learning!</p><p><em>Shoutout to the folk behind </em><a href="https://mml-book.github.io/"><em>Mathematics for Machine Learning</em></a><em> — it’s an amazing self-study resource if you’re interested in some of the math behind ML in more detail than MOOCs normally go into. Full disclosure: I stole the general layout of the tree from there:)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a9e65b8f8aa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/matrix-phylogeny-for-machine-learning-7a9e65b8f8aa">Matrix Phylogeny for Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Poisoning attacks on Machine Learning]]></title>
            <link>https://medium.com/data-science/poisoning-attacks-on-machine-learning-1ff247c254db?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ff247c254db</guid>
            <category><![CDATA[security-for-ml]]></category>
            <category><![CDATA[data-poison]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[poisoning-attack]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Mon, 15 Jul 2019 00:29:33 GMT</pubDate>
            <atom:updated>2019-07-15T00:29:33.270Z</atom:updated>
            <content:encoded><![CDATA[<h4>Security for Machine Learning</h4><h4>A 15-year old security problem that’s making a comeback</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5T2oEa-b_5hH8-lAKaHBg.png" /><figcaption><a href="http://sceneofthetitans.co.uk/img/1822-1.jpg">Source.</a></figcaption></figure><p>I am really excited for machine learning. It’s a fascinating piece of technology that truly brings science fiction to reality. However, <a href="https://medium.com/@iljamoisejevs/what-everyone-forgets-about-machine-learning-974752543849">like I wrote previously</a>, machine learning doesn’t come without its own problems (in the form of security vulnerabilities) — and it’s pretty important we <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">start thinking about them</a>.</p><p>Today we’ll discuss <strong>poisoning attacks </strong>— an attack type that takes advantage of your ML model during training (as opposed to <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">inference</a>).</p><p>Let’s get started.</p><h3>Spam or Ham?</h3><p>Despite what all the hype might lead you to believe, poisoning attacks are nothing new. In fact, as soon as machine learning started to be seriously used in security — cybercrooks started looking for ways to get around it. Hence the first examples of poisoning attacks date as far back as <a href="https://www.semanticscholar.org/paper/Adversarial-classification-Dalvi-Domingos/1baee9d28db8b5ad89758972b58e1fbdd2aee681">2004</a> and <a href="https://www.semanticscholar.org/paper/Good-Word-Attacks-on-Statistical-Spam-Filters-Lowd-Meek/16358a75a3a6561d042e6874d128d82f5b0bd4b3">2005</a>, where they were done to evade spam classifiers.</p><p>But what is a poisoning attack, exactly?</p><p>A poisoning attack happens when the adversary is able to inject bad data into your model’s training pool, and hence get it to learn something it shouldn’t. The most common result of a poisoning attack is that the model’s boundary shifts in some way, such as here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*kefmvhT1e0wrrBT-fSq9Qg.png" /><figcaption><a href="https://arxiv.org/pdf/1904.06292.pdf">Source.</a></figcaption></figure><p>Poisoning attacks <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">come in two flavors</a> — those targeting your ML’s <strong>availability</strong>, and those targeting its <strong>integrity</strong> (also known as “<strong>backdoor</strong>” attacks).</p><p>The first attacks were of the availability type. Such attacks aim to inject so much bad data into your system that whatever boundary your model learns basically becomes useless. Previous work has been done on <a href="https://people.eecs.berkeley.edu/~tygar/papers/SML/Spam_filter.pdf">Bayesian networks</a>, <a href="https://arxiv.org/abs/1206.6389">SVMs</a>, and more recently — on neural networks. For example, <a href="https://arxiv.org/abs/1706.03691">Steinhardt</a> reported that, even under strong defenses, a 3% training data set poisoning leads to 11% drop in accuracy. Others proposed <a href="https://arxiv.org/abs/1708.08689">back-gradient approaches</a> for generating poisons and even used an <a href="https://arxiv.org/abs/1703.01340">eutoencoder as an attack generator</a>.</p><p>Next <a href="https://arxiv.org/abs/1708.06733">came</a> <a href="https://arxiv.org/abs/1712.05526">backdoor</a> <a href="https://pdfs.semanticscholar.org/5f25/7ca18a92c3595db3bda3224927ec494003a5.pdf">attacks</a>. These are much more sophisticated and in fact want to leave your classifier functioning exactly like it should — with just one exception: a backdoor. A backdoor is a type of input that the model’s designer is not aware of, but that the attacker can leverage to get the ML system to do what they want. For example, imagine an attacker teaches a malware classifier that if a certain string is present in the file, that file should always be classed as benign. Now the attacker can compose any malware they want and as long as they insert that string into their file somewhere — they’re good to go. You can start to imagine what consequences such an attack might have.</p><p>Finally, as transfer learning emerged as a popular way to train models with limited datasets, attackers realized they can transfer poison together with the rest of the learning. For example, <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8685687">this</a> is a really interesting paper that examined poisoning of pre-trained models, including in a real-world scenario with a US street sign classifier that learnt to recognize stop signs as speed limits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DFQQX0bgd5cUlC0iy4YCFw.png" /><figcaption><a href="https://ieeexplore.ieee.org/document/8685687?denied=">Source.</a></figcaption></figure><p>As of today, poisoning attacks have been studied against <a href="https://nds2.ccs.neu.edu/papers/aml_aisec2014.pdf">sentiment analysis</a>, <a href="https://arxiv.org/abs/1811.09985">malware clustering</a>, <a href="https://arxiv.org/abs/1804.07933">malware detection</a>, <a href="http://www0.cs.ucl.ac.uk/staff/B.Karp/polygraph-oakland2005.pdf">worm signature detection</a>, <a href="http://people.ischool.berkeley.edu/~tygar/papers/SML/IMC.2009.pdf">DoS attack detection</a>, <a href="https://link.springer.com/chapter/10.1007/11856214_5">intrusion detection</a>, <a href="https://link.springer.com/chapter/10.1007/11856214_4">more intrusion detection</a>, <a href="https://link.springer.com/chapter/10.1007/978-3-540-74320-0_13">even more intrusion detection</a>, and my personal favorite: <a href="https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist">social media chatbots</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*-yG0pJ41uC3KSCVTtjd2eg.png" /><figcaption><a href="https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist">Source.</a></figcaption></figure><h3>Attacker Capabilities and Poisoning Attacks</h3><p>How much the attacker knows about your system — before they launch the attack — <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">is important</a>. When thinking about information access, it is common to distinguish between two types of adversaries: WhiteBox (knows the internals of your model) and BlackBox (doesn’t).</p><p>When we talk about <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion attacks</a>, that knowledge is what mainly defines how powerful of an attack the adversary can mount.</p><p>In poisoning attacks, however, there is a second just as important dimension that defines the attacker’s capability — <strong>adversarial access </strong>— or how deep they can get their hands into your system. Just like information access, adversarial access comes in levels (from most dangerous to least):</p><ul><li>Logic corruption</li><li>Data manipulation</li><li>Data injection</li><li>Transfer learning</li></ul><p>Let’s go one by one.</p><p><strong>Logic corruption</strong> is the most dangerous scenario. Logic corruption happens when the attacker can change the algorithm and the way it learns. At this stage the machine learning part stops to matter, really, because the attacker can simply encode any logic they want. You might just as well have been using a bunch of if statements.</p><p>To help you visualize what logic corruption looks like, here’s an extract from a recent paper where they add a “backdoor detector” to the neural network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWeoi1DvipxlQD_77AttSA.png" /><figcaption><a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8685687">This BadNet</a> augments the benign network with a parallel network that detects the presence of a trigger and a merging layer that produces an attacker chosen mis-classification when a backdoor trigger is detected.</figcaption></figure><p>Next up is <strong>data modification</strong>. Here the attacker doesn’t have access to the algorithm itself, but can change / add to / remove from the training data.</p><p>One thing they could do is <strong>manipulate labels</strong>. For example, they could randomly draw new labels for a part of the training pool or try to optimize them to cause maximum disruption. The attack is well suited if the goal is availability compromise — but becomes more challenging if the attacker wants to install a backdoor. Also, because of its “limited” perturbation space (you can only change labels to a fixed number of other labels) the attack requires the attacker to be able to alter a high proportion of all training data (eg 40% <a href="https://arxiv.org/abs/1206.6389">here</a>, with random label flipping, 30% with heuristics).</p><p>A more sophisticated attacker might <strong>manipulate the input</strong> itself — for example perturb it to shift the classification boundary, <a href="https://arxiv.org/abs/1811.09985">change cluster distance</a>, or add an invisible watermarks that can later be used to “backdoor” into the model. The perturbation act should remind you of <a href="https://medium.com/@iljamoisejevs/part-3-evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion attacks,</a> and indeed it’s often done in similar ways by optimizing the loss function in the other direction (gradient ascent). Such attacks are similar in how they benefit from extra information available to the attacker (whitebox, blackbox, etc).</p><p>Manipulating the input is a more sophisticated attack not only because it’s more powerful — but also because it has a more realistic threat model behind it. Think about a scenario where an AV (anti-virus) product sits on multiple endpoints and continuously collects data for future re-training. It’s easy for the adversary to insert any files they like — but they have no control over the labelling process, which is done either automatically or manually by a human on the other end. Thus if they’re able to insert benign-looking malicious files, they’ve just installed a backdoor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*rCOqVFMUwjV_NVX8.jpg" /><figcaption><a href="https://cdn.arstechnica.net/wp-content/uploads/2019/02/backdoor-800x533.jpg">Source.</a></figcaption></figure><p><strong>Data injection</strong> is similar to data manipulation, except, like the name suggests, it’s limited to addition. If the attacker is able to inject new data into the training pool that still makes them a very powerful adversary. For instance, <a href="https://www.utdallas.edu/~muratk/courses/dmsec_files/ieee-sp-06.pdf">Perdisci</a> prevented Polygraph, a worm signature generation tool, from learning meaningful signatures by inserting perturbations in worm traffic flows.</p><p>Finally, the weakest level of adversarial access is <strong>transfer learning</strong>. Without repeating what we’ve already discussed above, transfer learning is the weakest of the four because, naturally, as the network goes through its second training, its original memories (including the poison) <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8685687">get diluted</a>.</p><p>Final note on capability — when we talk about poisoning the training data pool, one natural question that comes to mind is — well, how much of it can we poison? Because arguably if we could modify all of it — we could basically get the machine learning model to learn whatever we want. The answer — at least from research literature — seems to be aiming for <a href="https://arxiv.org/pdf/1804.00308.pdf">less than 20%</a>. Beyond that the threat model starts to sound unrealistic.</p><h3>Defenses Against Data Poisoning</h3><p>Similar to <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion attacks</a>, when it comes to defenses, we’re in a hard spot. Methods exist but none guarantee robustness in 100% of cases.</p><p>The most common type of defenses is <strong>outlier detection</strong>, also knows as <strong>“data sanitization”</strong> and <strong>“anomaly detection”</strong>. The idea is simple — when poisoning a machine learning system the attacker is by definition injecting something into the training pool that is <em>very</em> different to what it should include — and hence we <a href="https://cs.gmu.edu/~astavrou/courses/ISA_674_F12/Sanitizing_Anomaly_Detection.pdf">should</a> <a href="https://arxiv.org/abs/1811.00741">be able to</a> <a href="https://arxiv.org/abs/1802.03041">detect</a> <a href="https://arxiv.org/abs/1706.03691">that</a>.</p><p>The challenge is quantifying <em>“very”</em>. Sometimes the poison injected is indeed from a different data distribution and can be easily isolated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oMaH-Sz5iMvcKg0k8fypgw.png" /><figcaption><a href="https://arxiv.org/pdf/1811.00741.pdf">Source.</a></figcaption></figure><p>In other cases, the attacker might generate poisoning points that are very <a href="https://arxiv.org/pdf/1811.00741.">similar to the true data distribution</a> (called “inliers”) but that still successfully mislead the model.</p><p>Another scenario where outlier detection breaks down is when poison was injected before filtering rules were created. In that case outliers stop being outliers:)</p><p>An interesting twist on anomaly detection is <strong>micromodels</strong>. The Micromodels defense was proposed for cleaning training data for <a href="https://www.covert.io/research-papers/security/Casting%20out%20demons%20-%20Sanitizing%20training%20data%20for%20anomaly%20sensors.pdf">network intrusion detectors</a>. The defense trains classifiers on non-overlapping epochs of the training set (micromodels) and evaluates them on the training set. By using a majority voting of the micromodels, training instances are marked as either safe or suspicious. Intuition is that attacks have relatively low duration and they could only affect a few micromodels at a time.</p><p>The second most common type of defenses is to analyze the impact of newly added training samples on <strong>model’s accuracy</strong>. The idea is that if a collected input is poisonous, it will destroy the model’s accuracy on the test set, and by doing a sandboxed run with the new sample before adding it to the production training pool we can spot that. For example, <a href="https://people.eecs.berkeley.edu/~tygar/papers/SML/misleading.learners.pdf">reject on negative impact (RONI)</a> and its cousin <a href="https://arxiv.org/pdf/1803.06975.pdf">target-aware RONI (tRONI)</a> are detection methods that rely on this property.</p><p>Some other interesting defenses I’ve seen include:</p><ul><li><a href="https://arxiv.org/abs/1902.06531">STRIP</a> = intentionally perturb the inputs and observe variance in predictions vs unperturbed. If not enough variance = malicious.</li><li><a href="http://pages.cs.wisc.edu/~jerryzhu/machineteaching/pub/Mei2015Machine.pdf">Human in the loop</a> = it’s known that poisoning outliers leads to bigger boundary shifts, so why not focus human (presumably security analyst) attention on only such cases?</li><li><a href="https://arxiv.org/pdf/1804.00308.pdf">TRIM</a> = model learns in iterations from “trimmed” versions of MSEs (regression).</li></ul><h3>Evasion and Poisoning — Two Sides of the Same Coin</h3><p>Remember in the beginning I said that most poisoning attacks work by shifting the classifier’s boundary? What do you think the other kind do?</p><p>The answer is they “pull” the poisoned example across an existing boundary, instead of shifting it. Which is exactly what the authors of <a href="https://arxiv.org/pdf/1804.00792.pdf">this paper</a> managed to achieve:</p><ol><li>They wanted to poison the classifier to classify A as B</li><li>So they took a bunch of Bs, perturbed them until the classifier thought they were As, then still labelled them as Bs, and inserted into the training pool</li><li>They thus poisoned the classifier with adversarial examples of Bs</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*vEiJVpGd8ypKr02TZLCN_g.png" /><figcaption><a href="https://arxiv.org/pdf/1804.00792.pdf">Source.</a></figcaption></figure><p>The result, as the authors intended, was such that the classifier started recognizing As as Bs. But here’s the most interesting part: what’s another name for step (3) above?</p><p>…</p><p>If you’ve read my post on <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion attacks</a> then you will hopefully recognize that it’s <strong>adversarial training</strong>, the most common way to defend models against adversarial examples. This leaves us with an unsettling conclusion: <strong>by doing a really good job of protecting yourself against evasion attacks — you might actually be opening the door to poisoning.</strong></p><p>How’s that for a cliff-hanger?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ff247c254db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/poisoning-attacks-on-machine-learning-1ff247c254db">Poisoning attacks on Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What everyone forgets about Machine Learning]]></title>
            <link>https://medium.com/data-science/what-everyone-forgets-about-machine-learning-974752543849?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/974752543849</guid>
            <category><![CDATA[security-for-ml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Sun, 14 Jul 2019 22:20:33 GMT</pubDate>
            <atom:updated>2019-07-15T21:44:41.813Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/security-for-ml">Security for Machine Learning</a></h4><h4>Shiny new capabilities don’t come for free</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qVxT7SVyLvd9Q3jDP78seQ.png" /><figcaption><a href="http://blog.lenovo.com/en/blog/inventing-x1-security/">Source.</a></figcaption></figure><p>We live in crazy times. I remember watching Star Wars as a kid and wondering how long it would take until we’d actually have speaking robots in our kitchen. Not very long — as it turned out. Fewer than 10 years in fact.</p><p>AI and more specifically machine learning is really bringing science fiction to reality — there just is no other way to put it. Every time I flip through Tech Review or TechCrunch I’m blown away by the kind of things we can now “casually” do.</p><p><a href="http://rfpose.csail.mit.edu/">Seeing through walls?</a> Easy. <a href="https://news.mit.edu/2015/visual-microphone-identifies-structural-defects-0521">Guessing materials’ physical properties from video?</a> Done. <a href="https://www.sigmobile.org/mobicom/2015/papers/p142-liuA.pdf">Estimating keystrokes from keyboard sound?</a> Piece of cake. And how about generating realistic looking <a href="https://arxiv.org/abs/1812.04948">faces</a>, <a href="https://carolineec.github.io/everybody_dance_now/">bodies</a>, or <a href="https://boingboing.net/2019/03/15/digital-lit.html">poems</a>? Or <a href="https://ai.googleblog.com/2017/04/teaching-machines-to-draw.html">teaching machines to draw</a>? Or a game of <a href="https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/">StarCraft</a>?</p><p>Oh, and have you seen these things roaming around?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZjvugcMv6jbwoAdD.jpg" /><figcaption><a href="https://www.theverge.com/2018/8/16/17693866/ford-self-driving-car-safety-report-dot">Source.</a></figcaption></figure><p>Madness.</p><p>Now if you actually go talk to people working in AI/ML — you’ll probably get one of two responses. Either they are beyond excited about what AI/ML can do and are working on the next big vision / NLP / Reinforcement Learning problem — or — they are absolutely terrified about what we stupid humans are building and believe that soon Artificial General Intelligence will convert humanity into a useless <a href="https://wiki.lesswrong.com/wiki/Paperclip_maximizer">mass of paperclips</a>. That feels to me like the general split of the community today — 50% of people think AI is our future, 50% think it’s our demise.</p><p>I want to offer a third — perhaps a more mundane — perspective on what AI and machine learning are: <strong>a new attack surface for adversaries.</strong></p><p>Let’s explore.</p><h3>The Yin Yang of New Inventions</h3><p>Whenever a new invention comes out, most people tend to think about the new and amazing capabilities that the invention enables. But, where there is light, there must be shadow, and so new capabilities inadvertently come packaged with new <strong>vulnerabilities</strong> for hackers to exploit. And exploit them they do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*PNOey8P3y4aqp5tZ-QaXHA.png" /></figure><p>Let’s take a history lesson and revisit the PC market. The first PC (Altair 8800) was released in 1975 followed by a series of innovations in the next 10 years culminating with the now too familiar-looking, moused “Apple Macintosh” in 1984. What followed was an explosive adoption wave that continued through all of 1990s and well into the 2000s:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*pN1-7tnEt01sFaN7DdXHfg.png" /><figcaption>Source: <a href="http://www.retrocomputing.net/info/siti/total_share.html">retrocomputing</a> + wiki.</figcaption></figure><p>Unbeknown to most users, however, a similar explosion was taking place in the malicious software or “malware” market.</p><p>In 1989 Robert Morris experimented with Unix sendmail and built a self-replicating worm that it sent out into the internet. What started as a mere experiment ended up becoming the first DoS attack, causing damages estimated between $100,000 and $10,000,000, and slowing the entire internet down to a crawl for several days (unthinkable now of course).</p><p>This was followed by the first ransomware attack in 1989, the first Linux virus (“Staog”) in 1996, and the first AOL Trojan in 1998.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*nDeyaOglGCQO22mJPescFg.png" /><figcaption>Malware stats source: <a href="https://en.wikipedia.org/wiki/Antivirus_software">av-test (through wiki)</a>.</figcaption></figure><p>Later, same happened in mobile: the 2007 iPhone moment followed by explosive growth in smartphone adoption:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*59H72dYFPpXFPC1iVVPx7g.png" /><figcaption>Source: <a href="https://www.protocol.gr/2019/04/21/number-of-smartphones-sold-to-end-users-worldwide-from-2007-to-2018/">statista</a>.</figcaption></figure><p>…followed by explosive growth in mobile malware:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*3RF85IRnn88NWZ8DUe10kg.png" /><figcaption>Malware stats source: <a href="https://www.av-test.org/en/statistics/malware/">av-test</a>.</figcaption></figure><p>Now — what about machine learning?</p><p>Despite all the buzz, <a href="https://speakerdeck.com/mmcventures/the-state-of-ai-2019-divergence?slide=9">productization of machine learning is still nascent</a>. A lot of the really cutting-edge work is still confined to research labs and Universities — but even looking at research, we can start to see some of the same trends emerge.</p><p>Machine learning research paper count by year and by area:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*wiVVK8AEbhDb2I2wXl0d-w.png" /><figcaption>Source: <a href="https://arxiv.org/search/advanced?advanced=&amp;terms-0-operator=AND&amp;terms-0-term=&quot;reinforcement+learning&quot;&amp;terms-0-field=all&amp;terms-1-operator=AND&amp;terms-1-term=&quot;machine+learning&quot;&amp;terms-1-field=all&amp;classification-computer_science=y&amp;classification-mathematics=y&amp;classification-physics_archives=all&amp;classification-statistics=y&amp;classification-include_cross_list=exclude&amp;date-filter_by=specific_year&amp;date-year=2018&amp;date-from_date=&amp;date-to_date=&amp;date-date_type=submitted_date_first&amp;abstracts=show&amp;size=200&amp;order=-announced_date_first">arxiv</a>.</figcaption></figure><p>…vs <strong>“adversarial machine learning”</strong> (ML’s version of malware) research paper count:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*kN7mrffA1khEayU4Xvv0xQ.png" /><figcaption>Source: <a href="https://arxiv.org/search/advanced?advanced=&amp;terms-0-operator=AND&amp;terms-0-term=adversarial&amp;terms-0-field=all&amp;terms-1-operator=AND&amp;terms-1-term=%22machine+learning%22&amp;terms-1-field=all&amp;classification-computer_science=y&amp;classification-physics_archives=all&amp;classification-statistics=y&amp;classification-include_cross_list=exclude&amp;date-filter_by=specific_year&amp;date-year=2018&amp;date-from_date=&amp;date-to_date=&amp;date-date_type=submitted_date_first&amp;abstracts=show&amp;size=200&amp;order=-announced_date_first">arxiv</a>.</figcaption></figure><p>So things are coming. Time to panic?</p><h3>The Business of Security</h3><p>Not so fast. The good news is that as PCs took over our everyday lives and hackers got hackin’, yet another market developed in parallel — that of security solutions.</p><p>The first anti-virus product was developed in 1987 by Andreas Lüning and Kai Figge for the Atari ST platform. That same year McAffee, NOD, Flu Shot and Anti4us all come to life — and so did many more in the following two decades:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*VMuj2Af3YHRCCWx1qTshZw.png" /><figcaption>Companies source: wiki + news + <a href="https://crunchbase.com/">crunchbase</a>.</figcaption></figure><p>Soon VCs realized how big cybersecurity was going to be, and capital started to flow in:</p><ul><li><a href="https://symantecweblog.blogspot.com/2007/12/history-of-symantic.html">Kleiner Perkins invests $3m in Symantec</a></li><li><a href="https://www.summitpartners.com/companies/mcafee">McAffee raises from Summit Partners</a></li><li><a href="https://www.crunchbase.com/funding_round/bitdefender-series-unknown--c53718a6">BitDefender raises $7m</a></li></ul><p>…followed by multi-million dollar acquisitions:</p><ul><li><a href="https://web.archive.org/web/20101030025249/http://www.pcworld.com/article/7883/network_associates_plans_to_integrate_dr_solomons_tools.html">McAffee buys Dr Solomon’s for $700m</a></li><li><a href="https://www.wsj.com/articles/SB964713859399871269">Symantec Agrees to Buy Axent In $768.8 Million Stock Deal</a></li><li><a href="https://web.archive.org/web/20090426010633/http://www.microsoft.com/presspass/press/2003/jun03/06-10GeCadPR.mspx">Microsoft to Acquire Antivirus Technology From GeCAD Software</a></li></ul><p>Mobile, following rapid growth in mobile malware, saw a similar explosion in security players:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*kRlbR3AUkcuWCRbrmBCW3w.png" /><figcaption>Companies source: wiki + news + <a href="https://crunchbase.com/">crunchbase</a>.</figcaption></figure><p>…security dollars:</p><ul><li><a href="https://techcrunch.com/2012/06/19/bluebox-raises-9-5-million-from-andreessen-horowitz-others-for-its-stealthy-disruptive-mobile-security-solution/">Bluebox Raises $9.5M From Andreessen Horowitz</a></li><li><a href="https://techcrunch.com/2012/12/04/france-telecom-invests-up-to-20m-in-lookout-preloads-its-mobile-security-solutions-on-to-android-handsets/">France Telecom Invests Up To $20M In Lookout</a></li><li><a href="https://techcrunch.com/2013/12/20/zimperium-raises-8m-for-mobile-security-that-turns-the-tables-on-attackers/">Zimperium Raises $8M For Mobile Security</a></li></ul><p>…and, eventually, security acquisitions:</p><ul><li><a href="https://www.geekwire.com/2012/mobile-security-startup-mobilisafe-acquired-rapid7/">Mobile security startup Mobilisafe acquired by Rapid7</a></li><li><a href="https://techcrunch.com/2012/07/27/apple-buys-samsungs-android-security-partner-authentec-for-365m/">Apple Buys Samsung’s Android Security Partner AuthenTec For $356M</a></li><li><a href="https://techcrunch.com/2014/09/03/avg-buys-location-labs/">AVG Buys Mobile Security Company, Location Labs for $220M</a></li></ul><p>And so what about machine learning?</p><h3>Does ML Even Need Security?</h3><p>At some point in the past I used to run anti-fraud and <a href="https://medium.com/@iljamoisejevs/why-the-future-of-finance-is-decentralized-a58ab38e6800">anti-money-laundering</a> for one of the biggest British fintechs — <a href="https://gocardless.com/">GoCardless</a>. My team oversaw $10bn+ in transaction volume a year and we were constantly battling to keep crooks out of GC’s circulatory system. Naturally — at some point we succumbed to the hype and decided to try machine learning.</p><p>To my surprise at the time, it worked. In fact, it worked great. Moving from traditional heuristics we managed to reduce money lost to fraud by 80% and improved the detection rate for accounts suspected of money laundering by as much as 20x.</p><p>There was just one problem.</p><p>We were deploying machine learning in what I consider “critical” capacity. We handed the algorithm a job at which it was not allowed to fail — as if it did — we’d either lose a ton of money or get our financial license revoked. For me as the Product Manager directly responsible for GC’s security — neither sounded particularly exciting.</p><p>So I needed to know how and when ML could fail. How could our model be exploited? Where did its inherent vulnerabilities lie? How do I know if GoCardless is under attack?</p><p>After too many late nights spent reading ML papers and sniffing around the darkweb — I finally found what I was looking for. I learnt about <a href="https://medium.com/@iljamoisejevs/poisoning-attacks-on-machine-learning-1ff247c254db">poisoning attacks</a> on ML, where the attacker is able to influence the model’s thinking by injecting corrupted data during training. I discovered <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">adversarial examples</a>, and how models could easily be misled by carefully perturbed inputs at test time. Finally, I learnt about <a href="https://medium.com/@iljamoisejevs/privacy-attacks-on-machine-learning-a1a25e474276">privacy attacks</a> and that neither the underlying data nor the model itself were really that private.</p><p>Then, I found this…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*tG8jVem3_2UkmfwUNO9h8A.png" /><figcaption>Source: <a href="https://www.technologyreview.com/f/613254/hackers-trick-teslas-autopilot-into-veering-towards-oncoming-traffic/">MIT tech review</a>.</figcaption></figure><p>…and I was terrified.</p><p>By the end of 2019 <a href="https://speakerdeck.com/mmcventures/the-state-of-ai-2019-divergence?slide=9">1/3rd</a> of all enterprises will have deployed machine learning. That’s 1/3rd of all products you, me, and our friends and loved ones use on a daily basis — completely naked before any attacker who knows the slightest thing about how ML works.</p><p>Yes, machine learning needs security.</p><h3>First Baby Steps</h3><p>ML security is a very nascent field — basically non-existent as of today. If there was one thing I learnt from my research above, it was that anyone who didn’t have a math PhD would have a hard time figuring how to make their ML secure (there are practically no solutions today, just math-heavy research papers).</p><p>Given how much of our lives we’re about to entrust to algorithms — I think it’s our responsibility — yours, mine, and the entire ML community’s to make sure security is not left behind as an afterthought. There’s a ton we can do today to build more robust ML models — as I explain in my posts on <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion</a>, <a href="https://medium.com/@iljamoisejevs/poisoning-attacks-on-machine-learning-1ff247c254db">poisoning</a>, and <a href="https://medium.com/@iljamoisejevs/privacy-attacks-on-machine-learning-a1a25e474276">privacy</a> attacks. But more importantly, we need to go through a mindset shift — away from “accuracy at all costs” to a more balanced accuracy vs robustness approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*YOOLbU-gcBYQ8jcFW956Gg.png" /><figcaption><a href="https://arxiv.org/pdf/1712.03141.pdf">Source.</a> C1 and C2 are two models. C1 clearly starts off as less accurate, but as the attack strength increases, it also does a better job at withstanding it. Would you pick C1 or C2 as your ML model?</figcaption></figure><p>This post and the ones above are my attempt at taking the first baby steps towards a more robust ML future. If you found it insightful — be sure to share the insight forward.</p><p>Stay safe &amp; secure everyone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=974752543849" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-everyone-forgets-about-machine-learning-974752543849">What everyone forgets about Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Will my Machine Learning be attacked?]]></title>
            <link>https://medium.com/data-science/will-my-machine-learning-be-attacked-6295707625d8?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/6295707625d8</guid>
            <category><![CDATA[threat-model]]></category>
            <category><![CDATA[security-for-ml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Sun, 14 Jul 2019 22:17:16 GMT</pubDate>
            <atom:updated>2020-08-14T10:21:18.837Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/security-for-ml">Security for Machine Learning</a></h4><h3>Will my Machine Learning System be attacked?</h3><h4>Unpacking the ML security threat model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*tikP7VM-W3H6SSBJD6qQyw.gif" /><figcaption><a href="https://i.imgur.com/A1iax04.gif">Source.</a></figcaption></figure><p>Machine learning is truly bringing fiction to reality. But the unfortunate truth of any new invention (ML included) — is that <a href="https://medium.com/@iljamoisejevs/what-everyone-forgets-about-machine-learning-974752543849">new capabilities come with new vulnerabilities for attackers to exploit</a>. So if you’re the CISO or the PM responsible for security in your organization — how should you be thinking about these new vulnerabilities? Will your ML actually be attacked? And if so, how and when could a breach happen?</p><p>The answer to the above is an <strong>ML security threat model </strong>— a structured framework that lays out all the possible threat vectors on your ML system.</p><p>Machine learning is still very new and its threat vectors are poorly understood. Nevertheless, in this post we’ll try our best to think through them, and see if we can come out with a solid framework for thinking about ML security by its end.</p><p>Let’s give it a shot.</p><h3>The Threat Model Pie</h3><p>At <a href="http://calypsoai.com/">Calypso</a>, when we think about threat models, we like to think about three separate pieces of the pie:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*6m2KeohLzsTl78yXSsmCQw.png" /></figure><p><strong>WHO will attack you? </strong>It all starts with the WHO. The attacker. The adversary. The other side. You can make the job of defining the WHO super easy or super hard. On one hand you could just say “a hacker”. On the other — what information does that actually give you to work with? “A hacker who understand how machine learning works” might be better. “A hacker who understand how machine learning works and with a maths background” is better still. The more detail you can put around the WHO, the better.</p><p><strong>WHY will they attack you?</strong> WHY is super closely related to WHO (why I discuss the two together below). The idea behind the WHY is simple — they could hack you, but why would they? There has to be something in there for them. A bounty. A reward. Again — defining the WHY — can be as simple as saying “why not” (not recommended) — and as hard as saying “to violate the confidentiality and later sell the data extracted from my ML system to a entity X” (much better!).</p><p><strong>HOW will they attack you? </strong>This is the technical part. Now that you know WHO and WHY will attack you — the next question is HOW will they go about it? If we’re talking about machine learning — will they attack your ML during training or inference? How much information about your system will they have? One really interesting but also challenging aspect of thinking through the HOW is alternatives. They might be <em>able</em> to hack your ML, but if it’s easier to hack your data pipeline, will they just do that?</p><p>Let’s get into more detail.</p><h3>WHO+WHY</h3><p>The WHO. Broadly speaking, I like to think of adversaries as coming in levels of “seriousness”. In my head it goes something like this:</p><ol><li>Smart researcher / engineer fooling around with a commercial ML system (potentially for actual research — such as the attack on <a href="http://clarifai.com/">Clarifai.com</a> <a href="https://arxiv.org/abs/1611.02770">here</a>)</li><li>White-hat participating in a bounty program</li><li>Penntester / red teamer conducting a test</li><li>Hacktivist attacking a commercial ML system to prove a point</li><li>Black-hat attacking a commercial ML system to earn a financial reward (whether through actually deploying the exploit or selling it on the darknet markets)</li><li>Organized black-hat group attacking a commercial ML system (think <a href="https://en.wikipedia.org/wiki/Anonymous_(group)">Anonymous</a>, <a href="https://en.wikipedia.org/wiki/The_Shadow_Brokers">The Shadow Brokers</a>, and <a href="https://en.wikipedia.org/wiki/Legion_of_Doom_(hacking)">Legion of Doom</a>)</li><li>State sponsored organizations (at this point it’s basically cyberwarfare)</li></ol><p>It’s important to think about how serious your adversary is because that defines the kind of knowledge and tools that they might have access to — and hence the level at which your defenses should be.</p><p>Now the WHY. WHY, in my head, breaks down into two questions:</p><ol><li>What is their goal?</li><li>What is their motivation?</li></ol><p>When answering the <strong>goal</strong> question, I like to use the CIA triad:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*RQeBvqquET8lHRsy.png" /><figcaption><a href="https://blogs.ucl.ac.uk/infosec/files/2017/08/The-CIA-triad-goals-of-confidentiality-integrity-and-availability-for-information-security-600x351-300x176.png">Source.</a></figcaption></figure><p>The three elements of the triad — confidentiality, integrity, availability — refer to the three pillars of security. If you can protect all three — you’ve got security covered.</p><p><strong>Confidentiality (or </strong><a href="https://medium.com/@iljamoisejevs/privacy-attacks-on-machine-learning-a1a25e474276"><strong>privacy</strong></a><strong>) attacks</strong> aim to extract sensitive information from your ML system. For example an attacker might want to infer whether a particular datapoint (eg you) were part of a particular training dataset (eg <a href="https://arxiv.org/abs/1610.05820">hospital discharge</a>).</p><p><strong>Integrity attacks</strong> make your ML model make a mistake, but — importantly — do so quietly. For example an attacker might want your classifier to take a malicious file for benign, without affecting its overall performance, so you never notice. Within integrity, the attacker can have a number of sub-goals (from hard to easy):</p><ul><li><strong>Source/target misclassification</strong> — the attacker wants a specific class (“malicious”) to be classified as a specific other class (“benign”).</li><li><strong>Targeted misclassification</strong> — the attacker wants a specific class (“stop” sign) to be classified as any other class (eg “speed limit 60” or “speed limit 45” or “dog” or “human” or whatever. As long as there’s no stopping).</li><li><strong>Misclassification</strong> — any class into any class. Gets into the realm of availability attacks (below).</li><li><strong>Confidence reduction</strong> — the attacker wants your model to lose confidence (useful when it’s working against a threshold as eg in the case of fraud scores).</li></ul><p><strong>Availability attacks </strong>aim to take your ML system down entirely. For example, if enough bad data is inserted into the training pool, the boundary that your model learns is basically garbage and the model becomes useless. It’s the DOS attack of the machine learning world.</p><p>Next to the attacker’s goal sits attacker’s <strong>motivation</strong>. This one is much more subjective and sort of intertwines with the WHO. If you’re a hacktivist group by definition you’re trying to make a statement of some sort of and if you’re a black-hat you probably want financial gain.</p><h3>HOW</h3><p>The technical part. HOW can someone actually go about breaking your model? (I’m going to focus on ML-specific vulnerabilities here and exclude traditional cybersecurity concerns like DDOS on your ML server.)</p><p>There are four dimensions we can use to classify the HOW of ML attacks:</p><ol><li>Timing (training / inference)</li><li>Capability (whitebox / blackbox / greybox)</li><li>Limitations (perturbation distance / functionality / domain / frequency of re-training)</li><li>Alternatives</li></ol><p><strong>Timing</strong> — this refers to where in the ML deployment pipeline the attack happens. There are two broad choices here — training time and inference time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kdKI3HzMzTOAsqTnEkLPAQ.png" /><figcaption><a href="https://www.sqlservercentral.com/articles/understanding-machine-learning">Source</a>.</figcaption></figure><ol><li>An attack at <strong>training</strong> time means the attacker is able to influence the training dataset (unlimited power, but hard to do and extra limitations).</li><li>An attack at <strong>inference</strong> time means the attacker is only able to perturb the immediate input (may or may not be powerful, depending on the model, but much easier to perform as only need a way to inject modified input).</li></ol><p><strong>Capability</strong> — this refers to the attacker’s knowledge of the internals of the ML system. Specifically:</p><ul><li><strong>WhiteBox</strong> attacks assume the attacker knows the distribution of the underlying data (potentially access to parts of it), the architecture of the model, the optimization algorithm used, and the weights and biases.</li><li><strong>BlackBox</strong> attacks assume the attacker doesn’t know anything about the ML system (so none of the above things). They can be of two types: <strong>hard label</strong> (when the attacker only received the predicted label from the classifier) and <strong>confidence</strong> (when the attacker received the predicted label as well as the confidence score from the classifier).</li><li><strong>GreyBox</strong> attacks sit somewhere in the middle. For example, the attacker might know what the model looks like but not what the underlying data does — or vice versa.</li></ul><p>You might also come across a term like <strong>NoBox</strong> in scientific literature. NoBox refers to an attack on the surrogate model, that the attacker rebuilt based on their (however limited) understanding of the targeted ML system. I don’t think it makes sense to classify this separately, as once the attacker builds a surrogate model it effectively becomes a WhiteBox attack.</p><p>Rule of thumb: more knowledge = better for the attacker, worse for us (to learn more see my post on <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion attacks</a>).</p><p><strong>Limitations</strong> refers to certain rules that restrict what the attacker can do. These are really ML system specific. For example:</p><ul><li>in images it is common to restrict the perturbation space to a “distance” metric, typically picked out of <a href="https://medium.com/@montjoile/l0-norm-l1-norm-l2-norm-l-infinity-norm-7a7d18a4f40c">Li, L1, or L2 norms</a>. There’s an interesting argument here btw as to whether that actually makes sense(read this <a href="https://arxiv.org/pdf/1807.06732.pdf">33-page whitepaper</a> to learn more).</li><li>in malware the attacker can only perturb the file in certain places and in certain ways, otherwise it will lose its malicious functionality or break alltogether.</li><li>in systems deployed on physical devices (satellites, cars, drones, surveillance cameras) the attacker may be restricted to modifying the input in the physical domain.</li><li>in attacks at training time the attacker requires two things: a) that the system continuously re-trains based on new data (otherwise they can’t inject bad data) and b) that the system takes in data from external sources, preferably without human in the loop to approve it.</li><li>in privacy attacks, in general the attacker requires a public endpoint with no query restrictions and that outputs a confidence score. The confidence score is particularly restrictive — eg most anti-virus products just tell your if they think the file is “malicious” or “benign”, with no further detail.</li></ul><p><strong>Alternatives</strong> — this is the final dimension I wanted to briefly mention. I once heard someone who spent 25 years in security describe attackers as “electricity”. They always take the path of least resistance.</p><p>It’s an important — and I found often overlooked — criterion that can easily make or break a threat model. Is attacking the ML component actually the easiest way for the attacker to get what they want? Eg if the goal is privacy violation — will they go through the hassle of building <a href="https://arxiv.org/abs/1610.05820">shadow models</a> on your classifier to get the data out, or are there <a href="https://techcrunch.com/2018/09/28/everything-you-need-to-know-about-facebooks-data-breach-affecting-50m-users/">easier bugs</a> to exploit?</p><h3>Bringing it all Together</h3><p>I’ve given you a lot (6 months of our research and thinking in fact!). Let’s bring it all together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*063PsrW9ddj6W3iG0OS0kg.png" /></figure><p>Now, let’s sprinkle some nomenclature.</p><p>In Adversarial ML (the academic side of ML security) it‘s common to take “Goal” from the WHY dimension and “Timing” from the HOW dimension, to name the attacks. The output looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1_XK5Lf9Ti13Wox7JsdbQ.png" /></figure><ol><li><strong>Evasion attacks (also called “adversarial examples”) </strong>are definitely the most popular type. They happen at inference time and take advantage of ML’s inherent vulnerabilities (or are they “features”? 🤔 — read my post on <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion attacks</a> to find out).</li><li><strong>Poisoning attacks</strong> happen at training time and can have one of two goals — integrity or availability. An attacker can insert a few carefully selected examples into your training pool that build a “backdoor” into it (integrity) — or they can insert so much bad data that the model’s boundary basically becomes useless (availability). Learn all there is about poisoning machine learning <a href="https://medium.com/@iljamoisejevs/poisoning-attacks-on-machine-learning-1ff247c254db">right here</a>.</li><li><strong>Privacy attacks</strong> are perhaps the least studied, but especially today, an <a href="https://techcrunch.com/2019/04/25/facebook-privacy-investigations/">extremely relevant threat</a>. Here the attacker doesn’t want to interfere with the workings of your machine learning model — and instead they want to extract private, likely sensitive information out of it. Read about privacy vulnerabilities and ways to fix them in my <a href="https://medium.com/@iljamoisejevs/privacy-attacks-on-machine-learning-a1a25e474276">full write-up here</a>.</li></ol><p>And that’s it! By now you, hopefully, have some structure in your head for how to think about the vulnerabilities of your ML system.</p><p>Stay safe &amp; secure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6295707625d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/will-my-machine-learning-be-attacked-6295707625d8">Will my Machine Learning be attacked?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evasion attacks on Machine Learning (or “Adversarial Examples”)]]></title>
            <link>https://medium.com/data-science/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/12f2283e06a1</guid>
            <category><![CDATA[security-for-ml]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[adversarial-attack]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[adversarial-example]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Sun, 14 Jul 2019 22:17:09 GMT</pubDate>
            <atom:updated>2019-07-15T00:18:09.136Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/security-for-ml">Security for Machine Learning</a></h4><h4>Your ML model is easier to fool than you think</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*NU_cyXN7tW3lrQxu8hOM2A.gif" /><figcaption><a href="https://thumbs.gfycat.com/CrazySpanishCero-mobile.jpg">Source.</a></figcaption></figure><p>Machine learning is exciting. However, just like any new technology or invention, not only does ML enable <a href="https://medium.com/@iljamoisejevs/what-everyone-forgets-about-machine-learning-974752543849">new amazing capabilities — but also, unfortunately, new vulnerabilities</a>.</p><p>Previously I’ve discussed how to <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">think about these vulnerabilities in a structured way</a> (or how to develop a “threat model” for your ML). This time I’d like to dive deep into how your ML system can be exploited during inference time through what is known as an <strong>evasion attack</strong>.</p><p>With no time to waste, let’s get started.</p><h3>Evading Machine Learning</h3><p>An evasion attack happens when the network is fed an <strong>“adversarial example” </strong>— a carefully perturbed input that looks and feels exactly the same as its untampered copy to a human — but that completely throws off the classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*zlxO7NuxK6ZijZtXnRl46Q.png" /><figcaption><a href="https://arxiv.org/abs/1412.6572">Source.</a></figcaption></figure><p>Despite all the hype around adversarial examples being a “new” phenomenon — they’re not actually that new. <a href="https://arxiv.org/pdf/1712.03141.pdf">A paper</a> by one of the leading names in Adversarial ML, Battista Biggio, pointed out that the field of attacking machine learning dates back as far as 2004. Back then adversarial examples were studied in the context of spam filtering, showing that linear classifiers could be easily tricked by few carefully-crafted changes in the content of spam emails.</p><p>Full timeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*FFEfdahrPd1_bgRBWyb1tg.png" /></figure><p>The “newness” of adversarial examples really came from them being discovered in the context of deep learning. In 2013 Christian Szegedy of Google AI was working on understanding how neural nets “think” but instead discovered an <a href="https://arxiv.org/abs/1312.6199">intriguing property</a> that all of them seem to possess — being easily fooled by small perturbations. Given how big deep learning has become — adversarial example and the field of Adversarial ML have ended up right in the spotlight.</p><p>When talking about adversarial examples I will mainly use computer vision papers — but really they’re equally applicable to any type of machine learning system, from complicated to not. Some of my personal favorites below:</p><p>Image: <a href="https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf">fooling face detection with adversarial glasses</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*2iu8l5mXzO3qQviDjtsHGA.png" /></figure><p>Video: <a href="https://github.com/advboxes/AdvBox/blob/master/applications/StealthTshirt/README.md">stealth T-shirt at DEFCON</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*81Y79JNN_GKLXerB96U7lQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*4nWHyX_68-jdw_uEcZeELQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*GfOmU1c7EG24f7lceu8Lmg.gif" /></figure><p>Audio: <a href="https://nicholas.carlini.com/code/audio_adversarial_examples/">speech-to-text transcription.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*g7FFNmv_uyzoCxG3eIe4Zw.png" /></figure><p>Language: <a href="https://www.semanticscholar.org/paper/Generating-Natural-Language-Adversarial-Examples-Alzantot-Sharma/c68fbc1f4aa72d30974f8a3071054e3b227137fd">sentiment analysis and textual entailment</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBqWAy5LYs0Srsa0wRq0ww.png" /></figure><p>Reinforcement learning: <a href="https://openai.com/blog/faulty-reward-functions/">boat racing</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KgTVE3i8XkNvY_40vqWF5g.png" /></figure><p>Malware: <a href="https://arxiv.org/pdf/1801.08917.pdf">evading a malware classifier using RL.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*_wefj4C3Ev-8HzfcZ_lMVA.png" /></figure><h3>Why do Adversarial Examples Exist?</h3><p>It’s a good question. As of today there is no consensus in the community as to why that might be. A number of hypotheses exist.</p><p>The first and original hypothesis trying to explain adversarial examples came from <a href="https://arxiv.org/abs/1312.6199">Szegedy’s own paper</a>, where they argued that they exist due to the presence of low-probability “pockets” in the manifold (ie too much non-linearity) and poor regularization of networks.</p><p>Later an opposing theory emerged, pioneered by <a href="https://arxiv.org/abs/1412.6572">Goodfellow</a>, arguing that in fact adversarial examples occurred due to <em>too much</em> linearity in modern machine learning and especially deep learning systems. Goodfellow argued that activations functions like ReLU and Sigmoid are basically straight lines in the middle (where, it just so happens we prefer to keep our gradients to prevent them from exploding or vanishing). And so really inside a neural net you have a ton of linear functions, all perpetuating one another’s input, all in the same direction. If you then add tiny perturbations to some of the inputs (a few pixels here and there) that accumulates into massive difference on the other end of the network and it spits out gibberish.</p><p>The third and perhaps most commonly adopted hypothesis today is the <a href="https://arxiv.org/abs/1608.07690">tilted boundary</a>. In a nutshell, the authors argue that because the model never fits the data perfectly (otherwise test set accuracy would have always been 100%) — there will always be adversarial pockets of inputs that exist between the boundary of the classifier and the actual sub-manifold of sampled data. They also empirically debunk the previous two approaches, so if you are technical this paper is worth a read.</p><p>Then there’s a few others, very mathy explanations:</p><ul><li><a href="https://arxiv.org/abs/1804.11285">Lack of sufficient training data</a></li><li><a href="https://arxiv.org/abs/1805.10204">Computational intractability of ever building robust classifiers</a></li></ul><p>And finally there’s my personal favorite. <a href="https://arxiv.org/abs/1905.02175">A recent paper from MIT</a> argues that adversarial examples are not a bug — they’re a feature of how neural networks see the world. Just because we humans are limited to 3 dimensions and can’t distinguish noise patterns from one another doesn’t mean those noise patterns are not good features. Our eyes are just bad sensors. So what we’re really dealing with here is a pattern recognition machine more sophisticated than ourselves — and instead of calling patterns that we don’t understand “adversarial examples” we should just accept there’s more to the world than meets the eye (get it?).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*tVrkIVYP7gF8vdkuizCk-Q.png" /><figcaption><a href="https://arxiv.org/abs/1905.02175">Source.</a></figcaption></figure><p>If the entire paper is too long — I highly recommend this <a href="http://gradientscience.org/adv/">11min blog summary</a>.</p><p>That paper, btw, uniquely explains another really interesting phenomenon related to adversarial examples — their transferability. <a href="https://arxiv.org/abs/1605.07277">Previous work</a> pointed out that adversarial example easily transfer between both different models and different datasets. That’s right — you could have trained a model on dogs vs cats, but then can fool a completely different model trained on hotdogs vs not hotdogs with the <em>same adversarial noise</em> added to the input. Because the MIT paper points out that noise is actually a feature, it makes sense that other images in the image realm (be they cats, dogs, hotdogs or not hotdogs) all share it. Fascinating stuff.</p><h3>Attacker Capabilities and Adversarial Examples</h3><p>Attacker’s knowledge of the target system (or their “<a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">capability</a>”) is important. The more they know about your model and how its built — the easier it is for them to mount an attack on it.</p><p>At <a href="https://calypsoai.com/">Calypso</a> we’ve broken down evasion attacks into five separate classes:</p><ol><li>Those that use gradients</li><li>Those that use confidence scores</li><li>Those that use hard labels</li><li>Those that use surrogate models</li><li>Brute-force attacks</li></ol><p><strong>Gradient-based attacks</strong> by definition require access to the model’s gradients and are therefore a type of <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">WhiteBox attacks</a>. These are hands down the most powerful, because the attacker can use their detailed understanding of how the model thinks (its gradients) to mathematically optimize the attack.</p><p>This is also the technique that works best for attacking hardened models. In fact, <a href="https://arxiv.org/abs/1705.07263">research</a> has shown that if the attacker has access to the model’s gradients — they will <em>always</em> be able to craft a new set of adversarial examples to fool the model. This is why adversarial examples are such a big deal — <a href="https://en.wikipedia.org/wiki/Security_through_obscurity">security through obscurity</a> aside it’s actually pretty hard to defend against them (more on this later).</p><p>The three most powerful gradient-based attacks as of today are:</p><ul><li><a href="https://arxiv.org/abs/1709.04114">EAD</a> (L1 norm)</li><li><a href="https://arxiv.org/abs/1705.07263">C&amp;W</a> (L2 norm)</li><li><a href="https://arxiv.org/abs/1706.06083">Madry</a> (Li norm)</li></ul><p><strong>Confidence score</strong> attacks use the outputted classification confidence to estimate the gradients of the model, and then perform similar smart optimization to gradient-based attacks above. This approach doesn’t require the attacker to know anything about the model and hence is of the <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">BlackBox type</a>.</p><p>The three most powerful confidence-based attacks as of today are:</p><ul><li><a href="https://arxiv.org/abs/1708.03999">ZOO</a></li><li><a href="https://arxiv.org/abs/1802.05666">SPSA</a></li><li><a href="https://arxiv.org/abs/1804.08598">NES</a></li></ul><p><strong>Hard label</strong> attacks rely solely on the label outputted by the model (“cat”, “dog”, “hotdog”) and don’t require the confidence scores. This makes the attack more dumb — but arguably more realistic (how many public endpoints do you know of that output a classification score?). The most powerful attack in this category remains the <a href="https://arxiv.org/abs/1712.04248">Boundary Attack</a>.</p><p><strong>Surrogate model</strong> attacks are very similar to gradient-based attacks, except they require an extra step. When the adversary doesn’t have access to the model’s internals but still wants to mount a WhiteBox attack, they can try to first rebuild the target’s model on their machine. They have a few options:</p><ul><li>If the target model is available as an oracle, the attacker can learn it through repeatedly querying the endpoint and observing input-output pairs (I talk more about such “reverse-engineering” attacks <a href="https://medium.com/@iljamoisejevs/privacy-attacks-on-machine-learning-a1a25e474276">here</a>)</li><li>If the target model is applied to a standard classification task (eg face detection)— the attacker can probably just guess the architecture and data the model is trained on, and use that to build a copy</li><li>If the attacker has no information at all, because of the transferability property we discussed earlier they could simply take any off-the-shelf image classifier and produce imperfect — but functional adversarial examples</li></ul><p>Finally, there are <strong>brute-force</strong> attacks. These attacks use no optimization at all to generate adversarial examples and instead resort to simple things like:</p><ul><li><a href="https://arxiv.org/abs/1712.02779">Randomly rotating / translating the image</a></li><li><a href="https://arxiv.org/abs/1807.01697">Applying common perturbations</a></li><li><a href="https://arxiv.org/abs/1901.10513">Adding Gaussian noise with large SD</a></li></ul><h3>How to Defend Against Adversarial Examples?</h3><p>In short — it’s hard. We’re really between a rock and a hard place here. Let’s examine the two broad categories of defenses — empirical defenses and formal methods.</p><p>Let’s start with <strong>formal methods</strong> because they are quicker to discuss. Those of you who worked in industries like chip design or aerospace &amp; defense will be familiar with formal methods. For everyone else — formal methods are a mathematical technique used to guarantee the robustness of software / hardware systems. In most of software, if you design or build the thing wrong it might lead to some downtime and a few angry customer reviews — but it probably won’t kill anyone. In some industries, however, such flaws are unacceptable. You can’t manufacture 100m chips only to realize some part of the chip is flawed — and you can’t put a plane in the air unless you’ve mathematically verified every component works as intended.</p><p>The way formal methods work, very simply, is by attempting every possible scenario and seeing how it plays out. In the world of evasion attacks that means trying to generate every possible adversarial example within a certain radius of perturbation. As an example, imagine you have an image with just two grayscale pixels — let’s say 180 and 80. Then let’s say you decide on a perturbation range of 3 in each direction. That gives you (3+1+3)² or 49 combinations to try out — and if you wanted to formally verify none of those are adversarial you would have to put each one through your model and see what comes out there other end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*SIc5V05FKez2nxCG4p7TLA.png" /></figure><p>49 combinations for an image with 2 dimensions and no color. What about a 1000 x 1000px colored image with same perturbation range of 3 in each direction? That’s (3+1+3)^(3*1000<em>*</em>1000) combinations to check for (excel refused to produce a number)!</p><p>Hopefully you can quickly see the problem with formal methods — they’re not cheap and often completely intractable from a computational perspective. In fact, as applied to neural networks, state of the art formal method techniques today can’t verify a network more than a few layers deep. So for now it’s a worthy — but elusive aspiration.</p><p>If you did want to read more into formal methods for neural networks, here are some good papers:</p><ul><li><a href="https://arxiv.org/abs/1702.01135">Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks</a></li><li><a href="https://arxiv.org/abs/1610.06940">Safety Verification of Deep Neural Networks</a></li><li><a href="https://arxiv.org/abs/1711.07356">Evaluating Robustness of Neural Networks with Mixed Integer Programming</a></li></ul><p>Otherwise let’s move on to <strong>empirical defenses.</strong> Empirical defenses are, as the name suggests, relying on experiments to demonstrate the effectiveness of a defense. Eg you might attack a model twice — first a normal, undefended version, then a hardened version — and observe how easy each is to fool (hopefully the hardened model performs better). So where formal methods are trying to compute every possible scenario and in doing so verify no adversarial examples exist — empirical methods take the approach of “you see it works, why do the math”.</p><p>Let’s go through some of the more popular types:</p><ol><li>Adversarial training</li><li>Gradient masking</li><li>Input modification</li><li>Detection</li><li>Extra class</li></ol><p><strong>Adversarial training</strong> — hands down the most talked-about and arguably most effective defense today. During adversarial training the defender retrains the model with adversarial examples included in the training pool, but labelled with correct labels. This teaches the model to ignore the noise and only learn from “robust” features.</p><p>The problem with adversarial training is that it only defends your model against the same attacks used to craft the examples originally included in the training pool. Thus if someone then mounts an attack optimized using a different algorithm, or mounts an adaptive attack (ie a whitebox attack on a defended model) — they are able to fool the classifier anew, as if no defense was in place.</p><p>You might think that, well, you could keep retraining your model including newly forged adversarial examples over and over again — but at some point you’ve inserted so much fake data into the training set that the boundary the model learns basically becomes useless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCXq0OArxz4nvcLH8p2VNw.gif" /></figure><p>That said, if the goal is to simply make it harder for the attacker to bypass the classifier — then adversarial training is a solid choice. There’s also a lot of creative approaches to actually doing it:</p><ul><li><a href="https://arxiv.org/abs/1705.07204">Ensemble AT</a></li><li><a href="https://arxiv.org/abs/1708.02582">Cascade AT</a></li><li><a href="https://arxiv.org/abs/1706.06083">Robust optimization approach to AT</a></li><li><a href="https://arxiv.org/abs/1811.07457">AT via spectral normalization</a></li></ul><p>Next, <strong>gradient masking.</strong> It’s basically a non-defense. For a while the Adversarial ML community thought that because gradients are needed to compute powerful attacks on models, hiding the gradients should solve the issue. <a href="https://arxiv.org/abs/1511.04508">Defensive distillation</a> was very popular — but quickly <a href="https://arxiv.org/pdf/1607.04311.pdf">proved to be false</a>.</p><p>The reason gradient masking doesn’t work as a defense tactic is because of adversarial examples’ transferability property. Even if you succeed at hiding the gradients of your model — the attacker can still build a surrogate, attack it, then transfer the examples.</p><p><strong>Input modification</strong> happens when an input, before being passed to the model, is in some way “cleaned” to get rid of adversarial noise. Examples include all sorts of denoising solutions (<a href="https://arxiv.org/abs/1412.5068">autoencoders</a>, <a href="https://arxiv.org/abs/1712.02976">high level representational denoisers</a>), <a href="https://arxiv.org/abs/1704.01155">color bit depth reduction, smoothing</a>, <a href="https://arxiv.org/abs/1805.06605">reforming GANs</a>, <a href="https://arxiv.org/abs/1608.00853">JPEG compression</a>, <a href="https://arxiv.org/abs/1511.06292">foveation</a>, <a href="https://arxiv.org/abs/1801.08926">pixel deflection</a>, <a href="https://arxiv.org/abs/1803.10840">general basis function transformations</a>, and many others.</p><p>Next, <strong>detection</strong>. Some detection methods are closely related to input modification — as once an input has been cleaned its prediction can be compared with the original prediction, and if the two are far apart it’s likely that the input has been tampered with. Here are <a href="https://arxiv.org/abs/1711.05929">a few</a> <a href="https://arxiv.org/abs/1705.09064">examples</a>.</p><p>Others are different. Eg <a href="https://arxiv.org/abs/1702.04267">here</a> they actually train a separate detection network whose only job is to decide if an input is adversarial. Then there is a number of detection defenses that examine raw statistics computed at various points in the input feeding / prediction process:</p><ul><li>On <a href="https://arxiv.org/abs/1702.06280">input itself</a></li><li>On <a href="https://arxiv.org/abs/1612.07767">convolutional filters</a></li><li>On <a href="https://arxiv.org/abs/1704.00103">ReLU activations</a></li><li>On <a href="https://arxiv.org/abs/1803.06373">logits</a></li></ul><p>In general input modification &amp; detection methods are great because they can be applied to an already trained model and don’t require the data scientist to go back to square 0. We are big fans of these kinds of defenses here at <a href="http://calypsoai.com/">Calypso</a>.</p><p>Finally, there is the <strong>extra (NULL) class</strong> approach. The idea is simple. Classifiers are trained on a very particular data distribution and by definition are clueless when taken outside the bounds of that. So instead of forcing the classifier to guess the label when it clearly doesn’t know what it is — why not give it the option of abstaining. That’s what the NULL class enables (<a href="https://arxiv.org/abs/1703.04318">paper</a>).</p><p>All in all empirical defenses are imperfect — but they work, and sometimes only take a few lines of code. Yes, admittedly, they are a never ending cat-and-mouse game with the attacker and new ones are often <a href="https://nicholas.carlini.com/papers/2018_icml_obfuscatedgradients.pdf">quickly broken</a> — but if you think about attackers like electricity (always take the path of least resistance) — our job is to just make their lives <em>just </em>hard enough.</p><h3>Closing Thoughts</h3><p>Phew! This has bee a long post — but hopefully you found it interesting and useful.</p><p>Final note, if you’re an engineer rather than a mathematician and prefer to learn through doing — <strong>Ethical Institute for AI &amp; Machine Learning </strong>has an excellent collection of open source tools related to <a href="https://github.com/EthicalML/awesome-production-machine-learning#adversarial-robustness-libraries">adversarial robustness</a>. Go ahead and play around.</p><p>Otherwise, if you’re done with evasion attacks but want to know more about ML security — I recommend reading my other posts on the <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">ML security threat model</a> and <a href="https://medium.com/@iljamoisejevs/poisoning-attacks-on-machine-learning-1ff247c254db">poisoning</a> and <a href="https://medium.com/@iljamoisejevs/privacy-attacks-on-machine-learning-a1a25e474276">privacy</a> attacks.</p><p>Stay safe &amp; secure everyone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12f2283e06a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">Evasion attacks on Machine Learning (or “Adversarial Examples”)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Privacy attacks on Machine Learning]]></title>
            <link>https://medium.com/data-science/privacy-attacks-on-machine-learning-a1a25e474276?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1a25e474276</guid>
            <category><![CDATA[security-for-ml]]></category>
            <category><![CDATA[privacy-attack]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-extraction]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Sun, 14 Jul 2019 22:16:53 GMT</pubDate>
            <atom:updated>2019-07-15T00:17:49.637Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/security-for-ml">Security for Machine Learning</a></h4><h4>Or how to extract your training data &amp; your model from your ML endpoint</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*k-3DcEweZAEV1CZW" /></figure><p>Machine learning is <a href="https://www.youtube.com/watch?v=hPKJBXkyTKM">amazing</a>, and I think we can all agree that soon ML will be a standard component of every tech product you and me use on a daily basis.</p><p>The challenge, however, is that with all the shiny new capabilities that ML promises also comes a host of vulnerabilities, available for attackers to exploit.</p><p>Some of these vulnerabilities can be leveraged by attackers to mess with your model’s thinking and to get it to output an incorrect prediction — like in the case of <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion</a> and <a href="https://medium.com/@iljamoisejevs/poisoning-attacks-on-machine-learning-1ff247c254db">poisoning</a> attacks. Others, however, allow the attacker to extract sensitive information from your model, such as the underlying data or the model itself.</p><p>We call such attacks <strong>privacy attacks</strong> on machine learning — and that’s exactly what we’ll be talking about in today’s post.</p><h3>The Data</h3><p>Needless to say if you’re trying to <a href="https://www.wired.com/2012/06/google-x-neural-network/">recognize cats in YouTube videos</a> then data privacy is probably not a big concern for you. But once you start applying machine learning in industries like banking and healthcare, controlling access to and dissemination of underlying training data becomes a real problem.</p><p>But first, let’s take a step back and think about what machine learning really is. Like what exactly are all these machines “learning”? Well the hope is that after seeing some amount of training data the machine is able to spot certain patterns in that data. But — crucially — not only is it able to spot them inside that same dataset — but it can also generalize its “learning” to other, previously unseen datasets.</p><p>The generalization point is important. If we wanted the machine to just learn the training dataset without generalization we could have it simply memorize every example and every label and later use something like a lookup table to extract the knowledge. The reason we’re jumping through all the extra hoops with machine learning is because we want it to learn the general state of the world, not just the dataset at hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7-WjnjuRoz-FnE_A6qaNw.png" /></figure><p>Now here’s where privacy comes in. Depending on how we build and train our model, sometimes it’s still possible to reverse engineer what the machine saw during training from its memories (parameters). That opens up a few avenues for privacy attacks.</p><p>The first, and simplest one is <strong>membership inference attack</strong>. This is when the attacker has a data point at hand and wants to know if it belonged to the original dataset. Imagine, for example, if someone wanted to find out if your name was in some sensitive medical list?</p><p>This is exactly what the authors of <a href="https://arxiv.org/abs/1610.05820">this study</a> did. They devised a technique that used “shadow” models built on the original, target model to decide if a patient was “in” or “out” of a hospital discharge dataset. They proved the attack to be successful on commercial “ML as a service” classifiers from both Google and Amazon.</p><p>Here’s a few more:</p><ul><li><a href="https://arxiv.org/pdf/1705.07663.pdf">Here</a> the attackers used GANs to perform attacks on generative models. They were 100% successful at inferring which samples were used to train the target model with WhiteBox attacks, and 80% with BlackBox.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*nxpPrIapN5TuJXQNY4Ln-w.png" /><figcaption>Real samples vs samples from the original generator vs samples the attackers were able to reconstruct using GANs.</figcaption></figure><ul><li><a href="https://arxiv.org/pdf/1811.00513.pdf">Here</a>, in the spirit of data-protection regulations like GDPR, the authors devised a way to check if a specific user’s text data was used to train a bunch of language models (word prediction, dialog generation).</li><li><a href="https://arxiv.org/pdf/1807.09173.pdf">Here</a> the authors examined the threat of an adversarial insider actor in a federated ML system. They concluded the threat is real and in fact discovered that the more different the datasets between various users, the easier it is to mount an attack.</li><li><a href="https://arxiv.org/abs/1708.06145">Here</a> the authors performed an attack on aggregate location data to determine if a particular user was part of the aggregates.</li><li>Finally, in a slightly different twist, <a href="https://arxiv.org/abs/1306.4447">here</a> the authors were able to successfully infer valuable statistical information about the underlying training data.</li></ul><p>To understand more about membership inference attacks, the authors <a href="https://arxiv.org/pdf/1807.09173.pdf">here</a> did a pretty in-depth investigation about what drives them. A few of the interesting discoveries they had:</p><ul><li>The data you train your model on matters a lot. For example, having more classes makes your model more vulnerable because each class takes up a smaller region in the latent space and there is less uninformed space.</li><li>The algorithm you choose matters a lot too. In general, algorithms whose decision boundaries are unlikely to be drastically impacted by a particular instance are more resilient (choose Bayesian not decision trees!).</li><li>WhiteBox knowledge matters but is not the be all and end all. For example, with as much as 10% of data replaced by noise the attackers still mounted successful attacks.</li><li>Shadow models don’t have to be trained in the same way as the target model. In fact, for 5 out of the 7 datasets authors tested, the highest accuracy was reported when the attack data generation model was of a different type than the target model.</li><li>Membership inference attacks are just as transferable as <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">adversarial examples</a></li></ul><p>Ok so inferring if someone is “in” or “out” is interesting. But you’re probably wondering if it’s possible to actually reconstruct some of the underlying data, given just BlackBox access to a classifier?</p><p>The answer is yes (sort of). Such attacks (called <strong>“data extraction”</strong> or <strong>“model inversion”</strong>) try to extract an average representation of each of the classes the model was trained on. They’re unable to extract individual instances, but the results tend to be pretty scary.</p><p>A few examples:</p><ul><li><a href="https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-fredrikson-privacy.pdf">Here</a> the authors extracted patients’ genomic information from an ML model trained to predict medical dosage (although <a href="https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-14.md">unclear</a> if it was due to model memorization or auxiliary information that they used).</li><li><a href="https://www.cs.cmu.edu/~mfredrik/papers/fjr2015ccs.pdf">Here</a> the authors extracted specific face images from an ML model trained to do face recognition (remember what I said above re average representations? well here 1 class = 1 face image).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*pjCYLoJPRUSX2VpD8Q9fnQ.png" /></figure><ul><li><a href="https://arxiv.org/pdf/1802.08232.pdf">Here</a> the authors extracted specific credit card numbers and social security numbers from a text generator trained on private data (they looked at edge cases or what they call <strong>“unintended memorization”</strong>).</li></ul><h3>The Model</h3><p>The second valuable part of any machine learning system is the model itself — and there’s a bunch of reasons someone might want to steal it (perform <strong>“model extraction”</strong>).</p><p>The first and most obvious — because the model is valuable. Imagine your data science team worked around the clock for 6 months to build a masterpiece that differentiates you from all the competition. How would you feel if someone managed to download the whole thing by just sending a few queries to your API?</p><p>The second — perhaps somewhat less obvious but arguably more dangerous — because they want to launch a <a href="https://medium.com/@iljamoisejevs/will-my-machine-learning-be-attacked-6295707625d8">WhiteBox evasion attack</a> on you. Evasion attacks crafted with access to model’s gradients are <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">much more powerful</a>, and so if the adversary can download your model first they will have a much higher chance of successfully breaching your system.</p><p>There hasn’t been as much work done on model extraction unfortunately, but in <a href="https://arxiv.org/abs/1609.02943">this 2016 paper</a> the authors managed to extract model parameters by observing input-output pairs. Caveat: they relied on having access to confidence scores, which may or may not be available in a realistic BlackBox setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*4zt80M3-TWWCImy6PHNb-w.png" /></figure><p>Here are a few <a href="https://arxiv.org/abs/1802.05351">other</a> <a href="https://arxiv.org/abs/1711.01768">works</a> if you’re curious.</p><h3>Defenses</h3><p>Just like with <a href="https://medium.com/@iljamoisejevs/part-3-evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion</a> and <a href="https://medium.com/@iljamoisejevs/part-4-poisoning-attacks-on-machine-learning-1ff247c254db">poisoning</a> attacks there are things you can do to make your models more secure.</p><p>Let’s start with the obvious: <strong>API hardening</strong>. Most of the papers I linked above rely on having unrestricted access to the target model’s API (with confidence scores), which makes it the easiest place to start. You could:</p><ul><li>Not expose the API at all</li><li>Expose only hard labels and not the confidence scores</li><li>If you have to expose confidence scores, you could reduce the dimensionality. Eg instead of showing 64% you could bucket the scores into 3 or 4 buckets and just show “high”</li></ul><p>Next up is <strong>data sanitization</strong>. Another pretty obvious point — if you’re training a model on a host of users’ messages and don’t want to leak accidental credit card numbers —just filter them out. You could try looking for all 16-digit numbers in the body of text, or filtering out all numbers alltogether.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*RPVtnkcqMmeRoLL5.jpg" /></figure><p>A little more involved is <strong>model hardening</strong>. Remember the findings from <a href="https://arxiv.org/pdf/1807.09173.pdf">this</a> paper we discussed above? Some of those we could leverage to make our machine learning more robust:</p><ul><li>Model choice — eg Bayesian models are more robust than Decision Trees</li><li>Fit control — in general overfitting makes it easier to extract data out of your model, so using regularization is a good idea</li><li>Knowledge control — keeping the model and its development restricted to a small group of people will help prevent leaking knowledge useful to the attacker</li></ul><p>Next up is <strong>detection</strong>. In order to extract any data from the model the attackers will have to query it multiple times, probably 1000s if not 10s of 1000s of times. <a href="https://ieeexplore.ieee.org/document/8682578">The patterns</a> of their queries are likely to be anomalous compared to what your actual users would do, and hence make good anomaly detection material.</p><p>Finally there is <strong>differential privacy</strong>. If you’re new to the game, differential privacy is a theoretical framework that aims to provide a formal guarantee on robustness. More specifically it aims to prove that two models differing by exactly 1 sample will produce similar predictions (making it impossible to infer that sample).</p><p>In laymen terms, differential privacy is all about injecting<strong> noise</strong> (or <strong>“randomness”</strong>) into your machine learning system. There’s a number of ways you could do it:</p><ul><li>Perturb <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42852.pdf">user’s input</a> into a common training pool (eg when a user sends data to a server x% is replaced with random numbers)</li><li>Perturb <a href="https://arxiv.org/pdf/1807.09173.pdf">the underlying data</a></li><li>Perturb <a href="https://arxiv.org/abs/1607.00133">the parameters of the model</a> (eg inject noise into the parameter update process)</li><li>Perturb <a href="http://www.jmlr.org/papers/volume12/chaudhuri11a/chaudhuri11a.pdf">the loss function</a> (similar to eg L2 regularization)</li><li>Perturb output during prediction (eg to x% of users your API sends random noise)</li></ul><p>The challenge with differential privacy is that it’s not free. The more protection you want, the more “privacy budget” you allocate, the more your model’s performance suffers. For example <a href="http://www.tdp.cat/issues16/tdp.a289a17.pdf">here</a> in order to protect against membership inference attacks the authors had to suffer a training accuracy loss from 94.4% to 24.7%. The model basically became useless. <a href="https://arxiv.org/pdf/1712.09136.pdf">Here’s</a> an interesting alternative to differential privacy, for situations like this where DP itself is too expensive.</p><h3>Closing Thoughts</h3><p>Especially as of recent, <a href="https://techcrunch.com/2019/04/25/facebook-privacy-investigations/">privacy is a big topic</a>. As more of our lives are entrusted to algorithms it’s important that we ensure they are both secure (against causative attacks like <a href="https://medium.com/@iljamoisejevs/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1">evasion</a> or <a href="https://medium.com/@iljamoisejevs/poisoning-attacks-on-machine-learning-1ff247c254db">poisoning</a>)and preserve privacy. Hopefully this post has given you a flavor of what the problem is — and what the potential solutions to it might look like.</p><p>Stay safe &amp; secure everyone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1a25e474276" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/privacy-attacks-on-machine-learning-a1a25e474276">Privacy attacks on Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why the future of finance is decentralized]]></title>
            <link>https://ilmoi.medium.com/why-the-future-of-finance-is-decentralized-a58ab38e6800?source=rss-b7592f20afda------2</link>
            <guid isPermaLink="false">https://medium.com/p/a58ab38e6800</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[money-laundering]]></category>
            <category><![CDATA[decentralization]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[ilmoi]]></dc:creator>
            <pubDate>Wed, 19 Dec 2018 16:01:02 GMT</pubDate>
            <atom:updated>2019-01-01T14:44:21.717Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*D4celqP-p20n6naYlXc8Pw.jpeg" /></figure><p>One of my favorite <a href="https://tokeneconomy.co/visions-of-ether-590858bf848e">narratives</a> in the Ethereum space today is that of open or decentralized finance. These things are still very new and I guess everyone sees them slightly differently, but for me there are 3 big promises open finance holds:</p><ol><li>Remove intermediates &gt; reduce cost &gt; new financial products now possible that weren’t before (eg targeted at low income segments of the population)</li><li>Build financial products in an open-sourced way &gt; more experimentation and competition &gt; better suite of products for all of us</li><li>Single immutable and transparent ledger used by the entire industry &gt; reduce opportunity for <a href="https://www.bloomberg.com/opinion/articles/2017-02-17/dole-food-had-too-many-shares">fraud and silly mistakes</a></li></ol><p>The question then is — will it happen?</p><p>In this essay I argue it will, and for a reason far more mundane or boring than you might first think.</p><p>Ready to hear it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*w3v9VU8jWEjFRiKGXq7Cxg.gif" /></figure><p>Anti-money laundering or “AML” compliance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*WtIALzUmDfDcujtybowDcA.gif" /></figure><p>I know you can’t wait to hear more — so let’s dig in.</p><h3>AML 101</h3><p>At the highest level AML works something like this:</p><p><em>Financial Action Task Force (FATF) ⇒ national legislation ⇒ regulation ⇒ company policy ⇒ product ⇒ reporting to local financial intelligence unit (FIU)</em></p><p><a href="https://en.wikipedia.org/wiki/Financial_Action_Task_Force_on_Money_Laundering">FATF</a> is the holy oracle of AML. It’s an inter-governmental body founded in 1989 to develop policies against money laundering. FATF has its own bible, called the <a href="http://www.fatf-gafi.org/publications/fatfrecommendations/documents/the40recommendationspublishedoctober2004.html">40 recommendations of AML</a>, where it describes in the typical impossible-to-comprehend legal language what a country must do to prevent money laundering from happening on its soil.</p><p>Most countries follow the recommendations religiously and implement them into legislation and eventually regulation. Those that don’t — end up on FATF’s “<a href="http://www.fatf-gafi.org/countries/#high-risk">name and shame</a>” list, which <a href="https://economictimes.indiatimes.com/news/international/world-news/fatf-grey-listing-to-take-its-toll-on-pakistan-economy/articleshow/63071017.cms">can be damaging</a>.</p><p>Next come the financial institutions (FIs). Internal lawyers interpret the regulation as they see fit (following something called a “risk based approach” where companies try to identify what risks they do and don’t face) and produce an internal policy document. That then gets picked up by product teams and transcribed into actual code.</p><p>The ultimate goal is to prevent, detect, and report any incidents of money laundering.</p><p>Prevention (as much as possible) is accomplished through identification of customers during onboarding. Basic-most form of this is proof of ID + address, but if the customer seems shady, a company might ask for additional documents, like proof of funds. This is also what the industry calls Know Your Customer or “KYC”.</p><p>Detection happens through monitoring. Typically it’s a mix of rules and supervised and unsupervised ML to detect bad actors. Sometimes you’re looking for a specific pattern (eg <a href="https://en.wikipedia.org/wiki/Structuring">structuring</a>) — but more often you’re just looking for anomalies in your customers’ behaviors. Once an account gets flagged — it goes to a compliance analyst, who investigates the case and passes judgement on it.</p><p>Reporting is the final stage. When an internal investigation is finished and if money laundering is suspected, a suspicious activity report or “SAR” is filed with the local FIU, such as the NCA in the UK or FinCEN in the US.</p><p>End of today’s class.</p><h3>So what’s wrong</h3><p>Now that you’re AMLiterate, let’s dig into problems. There are 2.</p><p>(1) AML is crazy expensive. The industry spends around $100bn (<a href="https://risk.lexisnexis.com/about-us/press-room/press-release/20181010-true-cost-aml">US</a> + <a href="https://risk.lexisnexis.com/global/-/media/files/corporations-and-non-profits/research/true-cost-of-aml-compliance-europe-survey-report-pdf.pdf">Europe</a> +<a href="https://www.lexisnexis.com/risk/intl/en/resources/research/true-cost-of-aml-compliance-apac-survey-report.pdf">Asia</a>) on AML compliance per year, with onboarding costs per customer hanging around <a href="https://www.ft.com/content/97fb6f58-959d-11e8-b67b-b8205561c3fe">$300–450</a>. Think about it — you haven’t even started providing your service and you’re already in the red for $450 for every new customer that you win (as if your CAC just grew by that fixed amount). And that, by the way, is if you do everything right and don’t get fined later for <a href="https://www.marketwatch.com/story/banks-have-been-fined-a-staggering-243-billion-since-the-financial-crisis-2018-02-20">billions of $$</a>.</p><p>(2) despite all the billions spent on it — AML doesn’t really work. The figure that blows my mind every time I think about it is 1%. That’s the estimated amount of all money laundered that we actually detect and catch as an industry (a <a href="https://www.unodc.org/unodc/en/press/releases/2011/October/unodc-estimates-that-criminals-may-have-laundered-usdollar-1.6-trillion-in-2009.html">high estimate</a>, mind you). I literally can’t think of a job where the ratio between $$ spent and desired effect achieved is so poor (if you can — drop a comment below, I’m curious).</p><p>So why might this be the case?</p><p>In the past I ran an AML product team for a fintech in London called GoCardless, and I had a lot of time to think about the question. My conclusion goes something like this:</p><p>Remember we talked about onboarding and monitoring?</p><p>The way I see it, onboarding is the first line of defense and is really there to deter lazy criminals who can’t be bothered to follow the <a href="https://www.wikihow.com/Make-a-Fake-ID">simple 9 step guide to faking your ID</a> — or at least watching the <a href="https://www.youtube.com/watch?v=DncrsCLIfD4https://www.youtube.com/watch?v=DncrsCLIfD4">YouTube Photoshop video</a>. From my experience at GC — this group includes small check fraudsters, not sophisticated money launderers.</p><p>Monitoring, on the other hand, is the real deal. If done correctly any suspicious activity should be flagged and should lead to an in-depth investigation of the merchant’s account. Again from my experience at GC — these deep-dives typically uncover the truth. In fact, I’m struggling to think of a single case where we investigated the merchant and decided they’re fine — and turned out to be wrong.</p><p>So if monitoring is the crux of it — and we’re only catching 1% of all money laundered — there’s probably something wrong with it?</p><p>Probably:)</p><p>Let’s think about what we do in monitoring:</p><ol><li>We establish someone’s identity</li><li>We look at their transaction history</li><li>We put the two together and ask the question of: “given who they say they are — does what they’re doing make sense?” Eg if someone signed up as a small landlord in London — is it reasonable that they’re collecting $50k in monthly payments from customers in the Middle East?</li></ol><p>Sounds simple enough. And the best thing — we already have all the data to answer the question. At the end of the day, we collected (1) during KYC and (2) is just a matter of SELECT * FROM transaction_history WHERE merchant_id = 12345.</p><p>Right?</p><p>Wrong. By putting my outstanding creative skills to work, let me illustrate my point:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VHQyHW6UybikHZsXTCGLPg.png" /></figure><p>Do you see the problem?</p><p>Because financial services are so plentiful and money launderers are not exactly stupid — nobody will launder all of their funds through one provider. A more likely scenario is that they will break up their stolen fortune into many small packages and route them through as many different channels of the financial web as possible.</p><p>Which for us, good guys trying to detect suspicious activity, basically means we’re left with no (2) from the above equation. We really have no clue where the funds came from and where they’re ending up.</p><p>No wonder the 1% success rate.</p><h3>A better way</h3><p>So how do we fix this?</p><p>Well, captain obvious might suggest we try collaborating with each other by sharing data — in the hope that one day we are able to uncover the full web of transactions for any given merchant. If we succeed, it’s a total win-win. We as FIs all become smarter about who we serve and don’t serve, keeping us well out of trouble — and the society benefits from an effective AML defense system, hopefully more effective than the current 1%.</p><p>When I first had this idea I was intrigued enough to go and speak to some other Compliance Heads around London, to see what they’d make of the idea. Unsurprisingly, most of them have thought about collaborating before. And yet somehow none of them actually did.</p><p>It seemed strange and I kept going at it, until it started to dawn on me why:</p><p>(1) Data mapping between FIs is very, very hard. Probably not a surprise to anyone — but every FI stores their data differently (many don’t have it one place and some don’t even seem to know where parts of it are). Connecting 2 FIs is hard enough and would require custom integration work — connecting <em>n</em> FIs starts to sound like an impossible task.</p><p>(2) Trust levels between FIs are very, very low = ie nobody wants to share data. This one felt a little paradoxical at first (everyone shares the same goal of catching criminals, why not?) — but as I spoke to more people it started to make more sense. Many FIs see transaction data as their ultimate competitive edge and others simply worry about <a href="https://eugdpr.org/">privacy</a> and <a href="https://www.cms-lawnow.com/aczone/money-laundering/offences-under-poca/tipping-off-offences">pissing off the regulator</a>. So the idea of giving up data is <em>very</em> painful.</p><p>(3) Who’s going to run and control the network? There was a real sense of concern from companies I spoke to about who’d be in charge, and why trust them. That’s only fair — firstly they need to know who they’re entrusting their data to and secondly the last thing they want is to help build something that will then lock them in and extract rent.</p><p>So that’s why.</p><p>See — I thought — if only there was a way to get all the FIs to adopt a shared set of standards for their data, and to somehow pull all that data together without revealing customers’ identities or putting it all in one party’s hands…</p><p>…wait a second…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*26OtJ385Dmcqm_0qw_MPPg.gif" /></figure><p>…perhaps we could use a blockchain?</p><p>Besides <a href="https://techcrunch.com/2018/03/04/icos-delivered-at-least-3-5x-more-capital-to-blockchain-startups-than-vc-since-2017/">forever replacing VCs</a>, <a href="https://www.theatlantic.com/technology/archive/2017/05/blockchain-of-command/528543/">overthrowing governments</a> and <a href="https://techcrunch.com/2018/03/21/bitcoin-jack-dorsey-quote-single-currency/">becoming the world’s sole future currency</a>, is there a world where blockchain has another (perhaps a little more mundane) application of helping fight money laundering?</p><p>Well let’s take a look:</p><ol><li>it’s a single shared ledger that everyone connects and writes to, meaning interoperability is baked into the system and data mapping is not a thing,</li><li>it’s pseudo-anonymous, meaning you can see the full transaction web without having to share sensitive data like names and dates of births to resolve entities, and</li><li>it’s secured through trustless consensus, meaning you’re not relying on any one party to run the show and set the rules.</li></ol><p>Jeez — it sure sounds like it might work.</p><h3>Theory vs practice</h3><p>History is littered with examples of technologies that made perfect sense in theory, but <a href="https://blogs.cornell.edu/info2040/2016/11/26/why-qwerty-still-reigns/">never actually happened in practice</a> (or <a href="https://jalopnik.com/why-electric-cars-ruled-the-roads-100-years-ago-1771719651">happened much later</a>)<strong>.</strong></p><p>This could very well happen again in this scenario — but I’m going to argue it won’t. In fact, I’m going to argue that once the industry realizes the economic value of moving to decentralized rails, traditional finance will quickly deflate to a peculiar fringe industry, if not disappear altogether.</p><p>Let’s go back to the total industry spend on AML that I mentioned earlier — $100bn / year. Where is all that money going?</p><p>From my experience at GC, and from speaking to others in the industry, today it’s probably split equally between onboarding and monitoring. But that’s changing rapidly — as more data on individuals and companies comes online, onboarding becomes increasingly automated. At GC, for example, we were able to do away with some 80% of manual identity checks by writing some smart code.</p><p>Monitoring, on the other hand, is much harder to automate.</p><p>Why is that? For the same reason that we discussed before — because it doesn’t work very well. Monitoring systems only see a tiny fraction of all of the merchant’s transactions, and as a result <a href="https://www.reuters.com/article/bc-finreg-laundering-detecting/anti-money-laundering-controls-failing-to-detect-terrorists-cartels-and-sanctioned-states-idUSKCN1GP2NV">produce a sea of false positive alerts</a><strong>.</strong> What do you think FIs do with those alerts (hint below)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6FpWIhYt7bJqWLcTD-u1Ow.png" /></figure><p>That’s right — they throw more bodies at them. And human bodies are what (another hint below)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l0cv7pczl2CFeceQdWIsgQ.png" /></figure><p>Expensive. Anecdotally, I’ve spoken to someone at a top 5 US bank and she told me they spend $3bn / year on alert-resolving compliance analysts ALONE. That’s right — just the staffing costs and just for false alert resolution.</p><p>Back to blockchain then, if every FI’s monitoring system saw the entire web of transactions instead of a tiny slither, I think it’s reasonable to expect that we’d see a meaningful uplift in accuracy. It doesn’t matter what you’re running — hardcoded rules or supervised or unsupervised ML, the below always holds:</p><p><em>more data ⇒ smarter decisions ⇒ fewer and higher quality alerts ⇒ smaller and CHEAPER teams</em></p><p>Exciting. But there’s more.</p><p>Inefficient monitoring is expensive and is definitely a big contributor to the mind-boggling $100bn/year AML spend. But there’s another, much more profound reason for why policing finance for money laundering is costing us so much money:</p><p>Duplication of effort.</p><p>Everyone is doing the same work over and again: the same fraudster or launderer might get flagged at 3 different FIs — they will all conduct individual investigations, which are most likely 90% duplicates — and each submit a separate report to the FIU. That’s 180% of effort wasted — both by the FIs and the FIU.</p><p>See, in traditional financial, where everyone operates in a walled garden, there’s really no other choice. You have to police your backyard and report on any crimes that happen there — because, well, if not you, then who? But in decentralized finance, where there are no walls — only a single web of transactions visible to everyone… this model makes no sense.</p><p>Instead, what I think we’ll see, is the emergence of specialized entities who police the entire network, and offer “AML as a service” to its participants. Think about how when you purchase a property — you hire a security guard to look after it. In the future, if you run a DeBank — you hire an AMLaaS company to keep you safe just the same. Or, to push it to its logical extreme, AML might one day just become a public utility — like gas or water or electricity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYQpsVwj-loq94OtUBDhew.png" /></figure><p>To understand just how transformative this would be, let’s throw some dummy numbers around. If smarter algorithms might drive AML compliance costs down by call it 20–30–50% — then deduplication of effort, given how many participants there are in the market, will probably put us well &lt;10%. In absolute terms, that’s $90bn in annual cost savings.</p><p>$90bn.</p><p>Just let that sink in. And what do you think the industry will do once it realizes this?</p><h3>Conclusion</h3><p>AML is a tough business. You’re up against some very smart people, sitting on a lot of cash, and willing to pay some hefty premiums to get that cash cleaned.</p><p>The approach to date has been to get FIs to each monitor their backyard and report on any wrongdoings. As we’ve seen, it’s crazy expensive and not very effective — but with today’s siloed infrastructure it’s the best we can do.</p><p>Decentralization has the power to change all that. By having a single web of transactions visible to everyone we can:</p><ol><li>run much smarter detection algorithms, and</li><li>deduplicate the monitoring efforts, outsourcing them to few specialized entities (public or private)</li></ol><p>The result, hopefully, is a much more efficient and effective AML protection layer that benefits everyone — from FIs to the regulator, to the everyday man.</p><p>It’s an exciting future — and I can’t wait to see it unfold.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a58ab38e6800" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>