<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by shaistha fathima on Medium]]></title>
        <description><![CDATA[Stories by shaistha fathima on Medium]]></description>
        <link>https://medium.com/@shaistha24?source=rss-ead8136ed0bf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7olH0bJNEOHup9jt.</url>
            <title>Stories by shaistha fathima on Medium</title>
            <link>https://medium.com/@shaistha24?source=rss-ead8136ed0bf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:10:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shaistha24/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can Cat eat chicken feet?]]></title>
            <link>https://medium.com/@shaistha24/can-cat-eat-chicken-feet-781a37d2e3c7?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/781a37d2e3c7</guid>
            <category><![CDATA[pets]]></category>
            <category><![CDATA[cat-food]]></category>
            <category><![CDATA[cats]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Sun, 31 Oct 2021 16:49:34 GMT</pubDate>
            <atom:updated>2021-11-02T16:22:27.647Z</atom:updated>
            <content:encoded><![CDATA[<p>Chicken feet good or bad for your cat? How to feed chicken feet to cats?…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*xT0wIgWQ-bZlmmLTpnc4iQ.jpeg" /><figcaption>Source — Google Images</figcaption></figure><p>A simple answer is YES!!</p><p>I have been feeding my cats with chicken feet for nearly an year now and you know what, they love it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3iHgLUt8n8QcqBQpkH8LLg.jpeg" /><figcaption>Source — Google images</figcaption></figure><p>Many of you may have concerns such as:</p><ul><li>Chicken feet has bone, so it might choke my cat!</li><li>They might not be able to chew it…</li><li>Should you feed raw or cooked feet?</li><li>What is the nutritional value? … and more.</li></ul><p>Coming back. If you might be wondering as to why have I suddenly decided to write cat related article while I usually write technical ones. The answer is, I have 5 cats… Yes 5!!! And I have been trying to find what kind of food can be fed to our purry friends. This was something I was curious about and think many other cat parents would like to know more about it too.</p><p>I have already answered if your cat can eat chicken feet above, now the question is …</p><h4>How to feed chicken feet to the cats?</h4><p>Usually it is best to feed them raw as the bone is very soft and can be chewed by our feline friends. I prefer to crush it or mince it with <strong>the meat cleaver.</strong></p><p>Though the adult cats can eat it easily. My adult cats can chew on the mid-sized chicken feet easily but the bigger ones can be quite annoying for them, as it takes time to punch holes into it and crush it themselves.</p><p>Here’s <strong><em>how I prepare the chicken feet for my cats</em></strong><em>…</em></p><ol><li><strong>Soak, Scrub and wash the feet </strong>— Chicken feet are often very dirty and might also have chicken poop sticking to it. You really want to clean it off!</li></ol><p>2. <strong>Clip off the nails </strong>— Though raw nails are soft but it should not be fed to the cats as it may choke them.</p><p>3. <strong>Refrigerate!</strong> — Wipe the feet dry and put them in a box or a packet to refrigerate. This way you can store it for a week or two.</p><p><strong><em>Feeding the feet to the cat…</em></strong></p><ol><li>Heat the feet with some water in a bowl.</li><li><em>It is Important </em>to— Only heat it enough to reach normal temperature i.e., warm it up!</li><li>Do not over-cook as this will make the bone bristle and can hurt your cat.</li><li>Crush it or mince it using a cleaver.</li><li>Give a whole crushed or minced feet to adult cat but chop it into pieces if given to the kittens.</li></ol><p><strong><em>Note: </em></strong><em>You may also cook and grind it into a paste and mix it with rice or liver or meat and give it to your cat as a wet meal! My cats hate rice and chicken feet paste can be sticky, we they hate too. You might want to make a small portion for testing before turning it into batches.</em></p><h4>Benefits</h4><ol><li>Chicken feet is mostly skin , cartilages, tendons, and bones. And are good source of collagen which helps relieve joint pain.</li><li>Promote heart health.</li><li>Can be used as treats.</li><li>Help clean their teeth as they munch on it.</li></ol><p>Most importantly, cats love it!!!</p><h4>How many can you feed them?</h4><p>Depending on the cat weight and size it can be fed anywhere between 1–4 feet. I usually, give 2 feet to adult and 1 to kitten but, sometime if they really want to eat it 4 is also OK for adult cats and 2 for kittens!</p><p>I have asked the vet and he said it is OK to feed them 4 in a day if not on daily basis!</p><h4><strong>Caution</strong></h4><p>It is important to discard or chop off the area with calluses on the feet. Or if it smells like ammonia or looks sticky!!</p><p>Do not leave the feet soaked for too long as it can spoil easily. Always scrub and clean off the chicken droppings.</p><p>Do not peel the skin, as it is edible.</p><h4>Conclusion</h4><p>Yes, you can feed your cat with chicken feet but remember everything should be done in moderation. Chicken feet is usually a treat or snack and not a meal, so don’t forget to feed your cat with proper meal after sometime.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=781a37d2e3c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Privacy Basics Series Conclusion and Important list of resources.]]></title>
            <link>https://medium.com/@shaistha24/differential-privacy-basics-series-conclusion-and-important-list-of-resources-2057a0d0d2be?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/2057a0d0d2be</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[differential-privacy]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Thu, 01 Oct 2020 21:27:59 GMT</pubDate>
            <atom:updated>2020-10-01T21:27:59.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*Z5cRFHwVib0vkrMy.jpg" /></figure><p><strong><em>Summary: This the sixth and the FINAL blog post of “Differential Privacy Basics Series” — summarizing all of the previous blog posts. For more posts like these on Differential Privacy follow</em></strong><a href="https://twitter.com/shaistha24"><strong><em> Shaistha Fathima</em></strong></a><strong><em> on twitter.</em></strong></p><h3>Differential Privacy Basics Series</h3><ul><li><a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049"><strong><em>What is Differential Privacy?</em></strong></a></li><li><a href="https://medium.com/@shaistha24/global-vs-local-differential-privacy-56b45eb22168"><strong><em>Global vs Local Differential Privacy</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-definition-bbd638106242"><strong><em>Differential Privacy Definition</em></strong></a></li><li><a href="https://medium.com/@shaistha24/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837"><strong><em>Query “Sensitivity” types and effects on Differential Privacy Mechanism</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-noise-adding-mechanisms-ede242dcbb2e"><strong><em>Differential Privacy — Noise adding Mechanisms</em></strong></a></li></ul><p>Before we head towards the conclusion — let’s have a look at some of the properties of Differential Privacy.</p><h3>Qualitative Properties of Differential Privacy (DP)</h3><ul><li><strong><em>Protection against arbitrary risks</em></strong>, moving beyond protection against re-identification.</li><li><strong><em>Automatic neutralization of linkage attacks</em></strong>, including all those attempted with all past, present,and future datasets and other forms and sources of auxiliary information.</li></ul><blockquote><strong><em>Linking attacks: </em></strong><em>A </em>linking attack<em> involves combining </em>auxiliary data<em> with </em>de-identified data<em> to </em>re-identify<em> individuals. In the simplest case, a linking attack can be performed via a </em>join<em> of two tables containing these datasets.</em></blockquote><blockquote><strong>Simple linking attacks are surprisingly effective:</strong></blockquote><blockquote><em>1- Just a single data point is sufficient to narrow things down to a few records.</em></blockquote><blockquote><em>2- The narrowed-down set of records helps suggest additional auxiliary data which might be helpful.</em></blockquote><blockquote><em>3- Two data points are often good enough to re-identify a huge fraction of the population in a particular dataset.</em></blockquote><blockquote><em>4- Three data points (gender, ZIP code, date of birth) uniquely identify 87% of people in the US.</em></blockquote><ul><li><strong><em>Quantification of privacy loss</em></strong> : Differential privacy is not a binary concept, and has a measure of privacy loss. This permits comparisons among different techniques:</li></ul><p>(i) For a fixed bound on privacy loss, which technique provides better accuracy?</p><p>(ii) For a fixed accuracy, which technique provides better privacy?</p><ul><li><strong><em>Composition</em></strong>: The quantification of loss also permits the analysis and control of cumulative privacy loss over multiple computations. Understanding the behavior of differentially private mechanisms under composition enables the design and analysis of complex differentially private algorithms from simpler differentially private building blocks.</li><li><strong><em>Group Privacy</em></strong>: DP permits the analysis and control of privacy loss incurred by groups, such as families.</li><li><strong><em>Closure Under Post-Processing</em></strong>: DP is immune to post-processing. A data analyst, without additional knowledge about the private database, cannot compute a function of the output of a differentially private algorithm M and make it less differentially private. That is, a data analyst cannot increase privacy loss, either under the formal definition or even in any intuitive sense, simply by sitting in a corner and thinking about the output of the algorithm, no matter what auxiliary information is available.</li></ul><h3>Granularity of Privacy- A final remark on DP definition.</h3><p><em>(Note: Granularity with respect to data means, the level of detail in a set of data.)</em></p><p>Differential privacy promises that the behavior of an algorithm will be roughly unchanged even if a single entry in the database is modified. <strong><em>But what constitutes a single entry in the database?</em></strong></p><p>For example a <strong><em>database that takes the form of a graph</em></strong>. Such a database might encode a social network: each individual i∈[n] is represented by a vertex in the graph, and friendships between individuals are represented by edges.</p><p>This brings us to two situations:</p><p>(i) <strong><em>DP at a level of granularity corresponding to individuals.</em></strong></p><p>This would require DP algorithms be insensitive to the addition or removal of any vertex from the graph. This gives a <em>strong privacy guarantee</em>, <em>but might in fact be stronger than we need.</em></p><p>The addition or removal of a single vertex could after all add or remove up to n edges in the graph. Depending on what it is we hope to learn from the graph, insensitivity to n edge removals might be an impossible constraint to meet.</p><p>(ii) <strong><em>DP at a level of granularity corresponding to edge.</em></strong></p><p>This would require DP algorithms to be insensitive <em>only</em> to the addition or removal of single, or small numbers of, edges from the graph. This is of course a weaker guarantee, but might still be sufficient for some purposes.</p><p>That is, if we promise ε-differential privacy at the level of a single edge, then no data analyst should be able to conclude anything about the existence of any subset of (1/ε) edges in the graph.</p><p>In some circumstances, <strong><em>large groups of social contacts might not be considered sensitive information.</em></strong> For example, an individual might not feel the need to hide the fact that the majority of his contacts are with individuals in his city or workplace,because where he lives and where he works are public information.</p><p>Similarly, there might be a <strong><em>small number of social contacts whose existence is highly sensitive. </em></strong>For example, a prospective new employer, or an intimate friend. <em>In this case, edge privacy should be sufficient to protect sensitive information, while still allowing a fuller analysis of the data than vertex privacy.</em></p><p><strong>Edge privacy will protect such an individual’s sensitive information provided that he has fewer than (1/ε) such friends.</strong></p><p>Another example, a <strong><em>differentially private movie recommendation system</em></strong> can be designed to protect the data in the training set at the “event” level of single movies, hiding the viewing/rating of any single movie but not, say, hiding an individual’s enthusiasm for cowboy westerns or gore, or at the “user” level of an individual’s entire viewing and rating history.</p><h3>Summarizing — answer to what, why, when, where , how?</h3><p>(Note: These are based on my current understanding, please post a comment if you would like to have a discussion on any of it)</p><h3>What is Differential Privacy?</h3><p>Differential Privacy is a System or Framework proposed for better data privacy. It is <strong>not a property of databases, but a property of queries</strong>. The intuition behind it is that we bound how much the output can change if we change the data of a single individual in the database.</p><p>That is, if the effect of adding or removing an individual’s data is high on the output of the query then it means that the data has high sensitivity, and the chances of an adversary being able to analyze it and find some auxiliary information is high. In other words, the privacy is compromised!</p><p>In order to avoid data leak, we add a controlled amount of statistical noise to obscure the data contributions from individuals in the data set.</p><p>When training a AI model, noise is added while ensuring that the model still gains insight into the overall population, and thus provides predictions that are accurate enough to be useful. At the same time making it tough for the adversary to make any sense from the data queried!</p><h3>Why do we use Differential Privacy?</h3><p>In the current world, privacy is one of the major concerns. With all the data science and AI models being implemented the chances of user privacy leak have increased.</p><p>Sometimes, AI models can memorize details about the data they’ve trained on and could ‘leak’ these details later on. Differential privacy is a framework (using math) for measuring this leakage and reducing the possibility of it happening.</p><h3>When and Where can we use Differential Privacy?</h3><p><a href="https://blog.openmined.org/use-cases-of-differential-privacy/">Use cases of differential privacy</a></p><h3>How can we use Differential Privacy?</h3><p>PATE analysis is one of the approaches for the implementation of DP.</p><p><a href="https://arxiv.org/abs/1610.05755">PATE approach</a> at providing differential privacy to machine learning is based on a simple intuition: if two different classifiers, trained on two different datasets with no training examples in common, agree on how to classify a new input example, then that decision does not reveal information about any single training example. The decision could have been made with or without any single training example, because both the model trained with that example and the model trained without that example reached the same conclusion.</p><p>For better theoretical understanding and explanation: <a href="http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html">Privacy and machine learning: two unexpected allies?</a></p><p>For practical code example: <a href="https://zumrutmuftuoglu.github.io/OM-Study-Group/pate-on-covid/">Detecting an Invisible Enemy With Invisible Data!</a></p><h3>Some Great Resources!!</h3><ul><li><a href="https://privacytools.seas.harvard.edu/files/privacytools/files/pdf_02.pdf">Exposed! A Survey of Attacks on Private Data</a></li><li><a href="https://privacytools.seas.harvard.edu/files/privacytools/files/pedagogical-document-dp_new.pdf">Differential Privacy:A Primer for a Non-technical Audience∗</a></li><li><a href="http://pramodv.ece.illinois.edu/pubs/GV16_2.pdf">Optimal Noise Adding Mechanisms for Approximate Differential Privacy</a></li><li><a href="https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf">Differential privacy and machine learning: Calculating sensitivity with generated data sets</a></li><li><a href="http://www.diva-portal.org/smash/get/diva2:1113852/FULLTEXT01.pdf">A Case Study on Differential Privacy</a></li><li><a href="https://research.kudelskisecurity.com/2020/03/11/differential-privacy-a-comparison-of-libraries/">Differential privacy: a comparison of libraries</a> — <a href="https://github.com/IBM/differential-privacy-library">IBM/differential-privacy-library</a> (Python) vs <a href="https://github.com/google/differential-privacy">google/differential-privacy</a> (C++) vs <a href="https://github.com/brubinstein/diffpriv">brubinstein/diffpriv</a> (R)</li><li><a href="https://desfontain.es/privacy/differential-privacy-awesomeness.html">Why differential privacy is awesome</a></li><li><a href="https://win-vector.com/2015/10/02/a-simpler-explanation-of-differential-privacy/">A Simpler Explanation of Differential Privacy</a></li></ul><h4><strong>Books</strong></h4><ul><li><a href="https://github.com/ZumrutMuftuoglu/OM-Study-Group/blob/master/privacybook.pdf">The Algorithmic Foundations of Differential Privacy</a></li><li><a href="https://github.com/ZumrutMuftuoglu/OM-Study-Group/blob/master/Differential%20Privacy%20from%20Theory%20to%20Pratice.pdf">Differential Privacy from Theory to Pratice</a></li><li><a href="https://github.com/ZumrutMuftuoglu/OM-Study-Group/blob/master/Mathematical%20Foundations%20of%20DP.pdf">Mathematical Foundations of DP</a></li></ul><h4><strong>Comic</strong></h4><ul><li><a href="https://journalistsresource.org/studies/government/comic-differential-privacy-2020-census/">A brief introduction to differential privacy: A data protection plan for the 2020 census</a></li><li><a href="https://ivyclare.github.io/comic/pate/">PATE Framework</a></li></ul><h4><strong>Videos</strong></h4><ul><li><a href="https://www.youtube.com/playlist?list=PLW0RMTGWCiOXS94e5H-0RcnIsgjEbL2v0">5 part Lecture on Differential Privacy by Cynthia Dwork</a></li><li><a href="https://www.youtube.com/watch?v=_g83jr1LChA">Differential Privacy and the People’s Data</a></li><li><a href="https://www.youtube.com/watch?v=4zrU54VIK6k">Privacy preserving AI — Lecture by Andrew Trask</a></li><li><a href="https://www.youtube.com/watch?v=H9zcRfmUo8s">Hacking Deep Learning: Differential Privacy and Collaborative Learning- Anand Sarwate</a></li><li><a href="https://www.youtube.com/watch?v=ZUsW_4GdEK8">A short tutorial on differential privacy: Dr Borja Balle, Amazon Research</a></li><li><a href="https://www.youtube.com/watch?v=Rj8FkJZgFtk&amp;t=707s">Differential Privacy for Growing Databases</a></li></ul><blockquote>Now this is not related but you might find it interesting — <a href="https://changelog.com/podcasts">Podcast by The Changelog</a> especially on <a href="https://changelog.com/practicalai">Practical AI</a></blockquote><h3>Overall References for this series:</h3><ul><li><a href="https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf">https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf</a></li><li><a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf">https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf</a></li><li><a href="http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html">http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html</a></li><li><a href="https://www.seas.upenn.edu/~cis399/files/lecture/l21.pdf">https://www.seas.upenn.edu/~cis399/files/lecture/l21.pdf</a></li><li><a href="https://github.com/ZumrutMuftuoglu/OM-Study-Group/blob/master/privacybook.pdf">https://github.com/ZumrutMuftuoglu/OM-Study-Group/blob/master/privacybook.pdf</a></li></ul><p>Thanks for following through till the end of this series. Feel free to post any comments or start a discussion about differential privacy concepts. You may also check the other series I have written before this :</p><ul><li><a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea?source=---------9------------------"><strong><em>Introduction to “Tensors”</em></strong></a></li><li><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2057a0d0d2be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Privacy — Noise adding Mechanisms]]></title>
            <link>https://becominghuman.ai/differential-privacy-noise-adding-mechanisms-ede242dcbb2e?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ede242dcbb2e</guid>
            <category><![CDATA[series]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[differential-privacy]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Thu, 01 Oct 2020 21:04:18 GMT</pubDate>
            <atom:updated>2020-10-07T14:10:14.776Z</atom:updated>
            <content:encoded><![CDATA[<h3>Differential Privacy — Noise adding Mechanisms</h3><blockquote>Laplace Mechanism, Exponential Mechanism and Gaussian Mechanism</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*Ij-FmZrKg7RLzQ_P" /></figure><p><strong><em>Summary: Sometimes knowing the basics is important! This the fifth blog post of “Differential Privacy Basics Series” covering the general noise adding mechanism used in Differential Privacy. For more posts like these on Differential Privacy follow</em></strong><a href="https://twitter.com/shaistha24"><strong><em> Shaistha Fathima</em></strong></a><strong><em> on twitter.</em></strong></p><h3>Differential Privacy Basics Series</h3><ul><li><a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049"><strong><em>What is Differential Privacy?</em></strong></a></li><li><a href="https://medium.com/@shaistha24/global-vs-local-differential-privacy-56b45eb22168"><strong><em>Global vs Local Differential Privacy</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-definition-bbd638106242"><strong><em>Differential Privacy Definition</em></strong></a></li><li><a href="https://medium.com/@shaistha24/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837"><strong><em>Query “Sensitivity” types and effects on Differential Privacy Mechanism</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-noise-adding-mechanisms-ede242dcbb2e"><strong><em>Differential Privacy — Noise adding Mechanisms</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-basics-series-conclusion-and-important-list-of-resources-2057a0d0d2be"><strong><em>Differential Privacy Basics Series Conclusion and Important list of resources.</em></strong></a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WfggxjD0QZrias7y1Hji4Q.png" /><figcaption>Quick recap.</figcaption></figure><h3>(1) Laplace Mechanism</h3><p>Numeric queries, functions such as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/0*25fQix48MILd9V0I.png" /></figure><p>are one of the most fundamental types of database queries. These queries map <a href="https://becominghuman.ai/how-can-we-improve-the-quality-of-our-data-61b55c9063d1">databases</a> to k real numbers. One of the important parameters that will determine just how accurately we can answer such queries is their ℓ1 sensitivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*i90fyd7MPHgxRXUN.png" /></figure><p><strong><em>The ℓ1 sensitivity of a function gives an upper bound on how much we must perturb its output to preserve privacy</em></strong>, i.e., The ℓ1 sensitivity of a function<strong><em> f captures the magnitude by which a </em>single individual’s </strong><a href="http://becominghuman.ai/"><strong>data</strong></a><strong> can change the function f in the worst case</strong>, and therefore, intuitively, the uncertainty in the response that we must introduce in order to hide the participation of a single individual.</p><figure><a href="https://aijobsboard.com/?fbclid=IwAR0GcsLs8A6CFM-fynrZMM3sTgo_Zpbto2CjglJm0Dmi6otC6YWc7CDFpQk"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dAlwXfVrtYspdCzmC8iADw.png" /></a><figcaption>Artificial Intelligence Jobs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*hqAPj2ua_eOYN3iS.png" /></figure><blockquote><strong>The Laplace distribution is a symmetric version of the exponential distribution. Adds noise from a symmetric continuous distribution to true answer.</strong></blockquote><p>The <strong><em>Laplace mechanism </em></strong>will simply compute<strong><em> f</em></strong>, and perturb <strong><em>each coordinate</em></strong> with noise drawn from the Laplace distribution. <strong><em>The scale of the noise will be calibrated to the [(sensitivity of f (query))/ε], where δ is always equal to 0.</em></strong></p><blockquote>Noise is scaled to<em> 1/ε</em>, that is, by adding noise drawn from Lap(1/ε). The expected distortion, or error, is 1/ε, independent of the size of the database.</blockquote><p>The Laplace mechanism preserves (ε,0)-differential privacy or ε-differentially private.</p><blockquote>This corresponds to the intuition that the <strong>more sensitive</strong> the query, and the <strong>stronger</strong> the<strong> desired guarantee</strong>, the <strong>more “noise” is needed</strong> to achieve that guarantee.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/0*1m7HMSKrG4GZrt1J.png" /></figure><h3><strong>Another way to look at it is,</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8sKbN1k7Q92joJWF.png" /></figure><blockquote><strong>The <em>sensitivity</em> of a function <em>f </em>is the amount<em> f</em>’s output changes when its input changes by 1.</strong></blockquote><p><strong><em>Example: Counting queries</em> always have a sensitivity of 1</strong>: if a query counts the number of rows in the <a href="https://becominghuman.ai/roadmap-to-data-scientist-b1dcb17896e7">dataset</a> with a particular property, and then we modify exactly one row of the dataset, then the query’s output can change by at most 1.</p><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/fundamentals-of-ai-ml-for-product-managers-64279e9d3681">1. Fundamentals of AI, ML and Deep Learning for Product Managers</a></blockquote><blockquote><a href="https://becominghuman.ai/the-unfortunate-power-of-deep-learning-3c06aef76e63">2. The Unfortunate Power of Deep Learning</a></blockquote><blockquote><a href="https://becominghuman.ai/graph-neural-network-for-3d-object-detection-in-a-point-cloud-309ba40897b2">3. Graph Neural Network for 3D Object Detection in a Point Cloud</a></blockquote><blockquote><a href="https://becominghuman.ai/know-the-biggest-notable-difference-between-ai-vs-machine-learning-cc1d7d18976a">4. Know the biggest Notable difference between AI vs. Machine Learning</a></blockquote><p>Thus we can achieve differential privacy for our example query by using the Laplace mechanism with sensitivity 1 and an ε of our choosing. For now, let’s pick ε = 0.1. We can sample from the Laplace distribution using Numpy’s random.laplace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DQMSHPP5l97KAKr-.png" /></figure><p>Considerable care must be taken when programming real-valued mechanisms, such as the Laplace mechanism, due to subtleties in the implementation of floating point numbers. Otherwise differential privacy can be destroyed, as outputs with non-zero probability on a database <em>x</em>, may, because of rounding, have zero probability on adjacent databases <em>y</em>. This is just one way in which the implementation of floating point requires scrutiny in the context of differential privacy, and it is not unique.</p><h4><strong><em>Drawbacks</em> :</strong></h4><p>(i) It is only really good for low sensitivity queries. (usually with L1 sensitivity)</p><p>(ii) Need large epsilon values (aka privacy budget) if you are going to fire off a bunch of queries. (Large epsilon values produce results that are less accurate in order to achieve the privacy guarantee).</p><p>(iii) Provides solution to handle numeric queries, but cannot be applied to the non-numeric valued queries like “what is the most common nationality in this room”: Chinese/Indian/American…</p><h3>Need for Exponential Mechanism when Laplace Mechanism already existed:</h3><ul><li>Laplace could be used for numeric queries only, while, exponential can be applied for numerical or categorical functional query output.</li><li>Most of the differential privacy revolved around <strong>real-valued functions</strong> which have <strong>relatively low sensitivity</strong> to change in the <a href="https://becominghuman.ai/data-science-simplified-principles-and-process-b06304d63308">data</a> of a single individual and whose usefulness is not hampered by small additive perturbations. <em>A natural question is what happens in the situation when one wants to preserve more general sets of properties?</em> The exponential mechanism helps to extend the notion of differential privacy to address these issues.</li><li><em>(Laplace and Gaussian) are focused on numerical answers, and add noise directly to the answer itself.</em> What if we want to return a precise answer (i.e. no added noise), but still preserve differential privacy? One solution is the exponential mechanism, which allows <strong>selecting the “best” element </strong>from a set while preserving differential privacy.</li></ul><h4><strong>Example:</strong></h4><ul><li>The Laplace Mechanism gives a <strong>general purpose way of adding noise</strong> to satisfy differential privacy <em>assuming that computing f accurately is the best measure of what we want to extract from our data.</em></li><li>BUT , if <strong><em>input: x</em></strong>= A database of training</li></ul><p><strong><em>dataGoal/Output: y</em></strong>= A neural network that minimizes the training error on x</p><ul><li>The neural network <strong>returned </strong>is defined by a series of weights.</li><li>If we were to apply the Laplace Mechanism to this function, <em>Laplace noise would be added to the weights before returning the network</em>.</li><li>However, <em>even small fluctuations</em> in weights in a neural network may <em>severely impact the performance</em> of that network.</li><li>Therefore, the <em>returned network (with added noise)</em> will likely <strong><em>behave very differently</em></strong> than the <em>initial network found (before adding noise)</em> that minimized error, and thus would have a <strong><em>unpredictably higher error</em></strong> than the minimal error network we desired.</li><li>Thus, this presents us with motivation to create another mechanism to satisfy Differential Privacy in such cases where the Laplace Mechanism fails.</li></ul><h3>(2) Exponential Mechanism</h3><p>The analyst defines which element is the <strong>“best”</strong> by specifying a <strong><em>scoring function that outputs a score</em></strong> for each element in the set, and also defines the set of things to pick from.</p><p>The mechanism provides differential privacy by approximately maximizing the score of the element it returns — in other words, to satisfy differential privacy, the exponential mechanism sometimes returns an element from the set which does not have the highest score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r3RHcd4k673z_p-k.png" /></figure><p>Rather than adding noise to the output of a function <strong><em>f</em></strong>, the exponential mechanism draws an output <strong><em>o</em></strong> from a probability distribution. Given a parameter, an input <strong><em>x</em></strong>, and a utility function <strong><em>u</em></strong> with generalized sensitivity <strong><em>∆u</em></strong>, we draw an output <strong><em>o</em></strong> from the following distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/0*WNQh2n3T-xD3xxyy.png" /></figure><p>The biggest practical difference between the exponential mechanism and the Laplace mechanism is that the <strong><em>output of the exponential mechanism is always a member of the set R</em></strong>. This is extremely useful when <em>selecting an item from a finite set, when a noisy answer would not make sense.</em></p><h4><strong>For example:</strong></h4><p>We might want to <strong><em>pick a date</em></strong> for a big meeting, which uses each participant’s personal calendar to maximize the number of participants without a conflict, while providing differential privacy for the calendars.</p><p><em>Adding noise to a date doesn’t make much sense:</em> it might turn a Friday into a Saturday, and increase the number of conflicts significantly. <strong><em>The exponential mechanism is perfect for problems like this one: it selects a date without a noise.</em></strong></p><h4><strong>The exponential mechanism is interesting for several reasons:</strong></h4><ul><li>The <strong>privacy cost</strong> of the mechanism is<em> just epsilon</em>, regardless of the size of R .</li><li>It theoretically works for <strong>both finite and infinite</strong> sets R, but it can be really challenging to build a practical implementation which samples from the appropriate probability distribution when R is infinite.</li><li>It represents a<em> “fundamental mechanism” </em>of ε-differential privacy: all other ε-differentially private mechanisms can be defined in terms of the exponential mechanism with the appropriate definition of the scoring function u.</li></ul><blockquote><strong><em>Why is the exponential mechanism so much better?</em> </strong>Because <em>it releases less information</em>. The exponential mechanism releases <em>only</em> the identity of the element with the maximum noisy score — <em>not</em> the score itself, or the scores of any other element.</blockquote><h4><strong>Drawbacks:</strong></h4><ul><li><strong><em>The exponential mechanism is extremely general </em></strong>— it’s generally possible to redefine any ε-differentially private mechanism in terms of a carefully chosen definition of the scoring function <em>u. </em>If we can analyze the sensitivity of this scoring function, then the proof of differential privacy comes for free.</li><li>On the other hand, applying the general analysis of the exponential mechanism sometimes <strong><em>comes at the cost of looser bounds</em></strong>, and mechanisms defined in terms of the exponential mechanism are <strong><em>often very difficult to implement</em></strong>.</li><li>The exponential mechanism is <strong><em>often used to prove theoretical lower bounds</em></strong> (by showing that a differentially private algorithm <em>exists</em>), but practical algorithms often replicate the same behavior using some other approach (such as noisy max approach using laplace mechanism).</li></ul><h3>(3) Gaussian Mechanism</h3><p>The Gaussian mechanism is an <em>alternative to the Laplace mechanism</em>, which adds Gaussian noise instead of Laplacian noise. The Gaussian mechanism does <em>not</em> satisfy pure ε -differential privacy, but does <em>satisfy (ε, δ)-differential privacy</em>.</p><p>According to the Gaussian mechanism, for a function f(x) which returns a number, the following definition of F(x) satisfies (ε, δ) -differential privacy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s3s5IjMKaIfntP6P.png" /></figure><p>For real-valued functions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*CnRnJhbcBchODdqQy-I19Q.png" /></figure><p>we can use the Gaussian mechanism in exactly the same way as we do the Laplace mechanism, and it’s easy to compare what happens under both mechanisms for a given value of ε.</p><h4><strong>For example:</strong></h4><pre>%matplotlib inline<br>import matplotlib.pyplot as plt<br>plt.style.use(&#39;seaborn-whitegrid&#39;)<br>import pandas as pd<br>import numpy as np</pre><pre>epsilon = 1<br>vals_laplace = [np.random.laplace(loc=0, scale=1/epsilon) for x in range(100000)]</pre><pre>delta = 10e-5<br>sigma = np.sqrt(2 * np.log(1.25 / delta)) * 1 / epsilon<br>vals_gauss = [np.random.normal(loc=0, scale=sigma) for x in range(100000)]</pre><pre>plt.hist(vals_laplace, bins=50, label=&#39;Laplace&#39;)<br>plt.hist(vals_gauss, bins=50, alpha=.7, label=&#39;Gaussian&#39;);<br>plt.legend();</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*mm6sRWN8Mg1YjGy6.png" /></figure><p>Here, we graph the empirical probability density function of the Laplace and Gaussian mechanisms for ε =1, with δ=10^-5 for the Gaussian mechanism.</p><blockquote>Compared to the Laplace mechanism, the <em>plot for the Gaussian mechanism looks “squished.”</em> <em>Differentially private outputs which are far from the true answer are much more likely using the Gaussian mechanism than they are under the Laplace mechanism</em> (which, by comparison, looks extremely “pointy”).</blockquote><h4><strong>Gaussian mechanism also has <em>two major drawbacks</em> :</strong></h4><p>(i) It requires the use of the the relaxed (ε, δ)-differential privacy definition,</p><p>(ii) It’s less accurate than the Laplace mechanism.</p><p><strong>So, Why would we want to use it?</strong></p><p>In the above example, we have only considered real-valued functions (i.e. the function’s output is always a single real number). Such functions are of the form</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/116/1*jGBg9cF2qC6lUIXjiwnsEg.png" /></figure><p>Both the Laplace and Gaussian mechanism, however, can be extended to <em>vector-valued</em> functions of the form</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/1*mm0QYAzX7f2Op0z-Zu9dVA.png" /></figure><p>This return vectors of real numbers. <em>For example,</em> we can think of histograms as vector-valued functions, which return a vector whose elements consist of histogram bin counts.</p><p>Both the Laplace and Gaussian mechanisms can be extended to vector-valued functions. However, there’s a <strong>key difference between these two extensions</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*bry9bnsc3HhxUaGC.png" /></figure><p>(i) <em>The vector-valued </em><strong><em>Laplace</em></strong><em> mechanism </em><strong><em>requires</em></strong><em> the </em><strong><em>use of L1 sensitivity</em></strong><em>, while the vector-valued </em><strong><em>Gaussian</em></strong><em> mechanism allows the use of </em><strong><em>either L1 or L2 </em></strong><em>sensitivity.</em></p><p>(ii) <em>This is a major strength of the Gaussian mechanism. For applications in which</em><strong><em> L2 sensitivity is much lower than L1 sensitivity, the Gaussian mechanism allows adding much less noise</em>.</strong></p><p>That’s it folks! In the next blog and the LAST blog of this series I will finally answer some of the basic questions on What, why, how , etc followed by list of resources to learn from.</p><p>Till then you may also check out my other beginner friendly Series:</p><ul><li><a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea?source=---------9------------------"><strong><em>Introduction to “Tensors”</em></strong></a></li><li><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)</em></strong></a></li></ul><h3>References:</h3><ul><li><a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf">https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf</a></li><li><a href="https://www.seas.upenn.edu/~cis399/files/lecture/l21.pdf">https://www.seas.upenn.edu/~cis399/files/lecture/l21.pdf</a></li><li><a href="https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf">https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf</a></li></ul><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*2lvCls4yjxVMfZSR" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ede242dcbb2e" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/differential-privacy-noise-adding-mechanisms-ede242dcbb2e">Differential Privacy — Noise adding Mechanisms</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Sensitivity” in Differential Privacy]]></title>
            <link>https://becominghuman.ai/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/c94fd14b9837</guid>
            <category><![CDATA[differential-privacy]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[machine-intelligence]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Tue, 15 Sep 2020 17:44:12 GMT</pubDate>
            <atom:updated>2020-10-08T13:49:03.811Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong><em>Query “Sensitivity” types and effects on Differential Privacy Mechanism</em></strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SuIXjNpD429JKXz4.jpg" /></figure><p><strong><em>Summary: Sometimes knowing the basics is important! This the fourth blog post of “Differential Privacy Basics Series” covering Sensitivity of the Query and its effects on Differential Privacy. Thank you </em></strong><a href="https://twitter.com/jacobmerryman"><strong><em>Jacob Merryman</em></strong></a><strong><em> for the amazing graphic used in this blog post. For more posts like these on Differential Privacy follow</em></strong><a href="https://twitter.com/shaistha24"><strong><em> Shaistha Fathima</em></strong></a><strong><em> on twitter.</em></strong></p><h3>Differential Privacy Basics Series</h3><ul><li><a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049"><strong><em>What is Differential Privacy?</em></strong></a></li><li><a href="https://medium.com/@shaistha24/global-vs-local-differential-privacy-56b45eb22168"><strong><em>Global vs Local Differential Privacy</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-definition-bbd638106242"><strong><em>Differential Privacy Definition</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-noise-adding-mechanisms-ede242dcbb2e"><strong><em>Differential Privacy — Noise adding Mechanisms</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-basics-series-conclusion-and-important-list-of-resources-2057a0d0d2be"><strong><em>Differential Privacy Basics Series Conclusion and Important list of resources.</em></strong></a></li></ul><p>We have seen that Differential Privacy(DP) adds statistical noise to the <a href="https://becominghuman.ai/becoming-a-data-scientist-data-analyst-financial-analyst-and-research-analyst-d9ec01a9d11d">data</a> for better privacy, but, <strong><em>How much noise should we add? </em></strong>This depends on the below 4 factors:</p><ul><li>Sensitivity of the Query.</li><li>Desired Epsilon.</li><li>Desired delta.</li><li>Type of Noise to be added. (most commonly used — Gaussian Noise or Laplacian Noise, among which Laplacian noise is easy to use)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*868jU7kKbirrljLECRfMQg.png" /></figure><p>In this blog post we will focus on the SENSITIVITY. To know about the Epsilon and Delta, please look at the (Part-3) Differential Privacy Definition of this series.</p><figure><a href="https://aijobsboard.com/?fbclid=IwAR0GcsLs8A6CFM-fynrZMM3sTgo_Zpbto2CjglJm0Dmi6otC6YWc7CDFpQk"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dAlwXfVrtYspdCzmC8iADw.png" /></a><figcaption>Machine Learning Jobs</figcaption></figure><h3>Sensitivity</h3><p><em>Sensitivity parametrize the amount i.e., how much </em><strong><em>noise</em></strong><em> perturbation is required in the DP mechanism.</em> To determine the sensitivity, <strong>the maximum of possible change in the result </strong>needs to be calculated.</p><p><strong><em>Generally sensitivity refers to the impact a change in the underlying data set can have on the result of the query.</em></strong> Let xA, xB be any data set from all possible data set of X differing in <em>at most one element</em>. Then the sensitivity is calculated with the following equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*X6aMoVaJsyqEO3Ll.png" /></figure><p><strong><em>where ‖.‖1 is theL1-norm distance between data sets differing at most one element.</em></strong></p><p><em>The sensitivity of a query can be stated both in the global and local context:</em></p><ul><li><strong><em>Global sensitivity </em></strong><em>refers to the consideration of </em><strong><em>all possible data sets</em></strong><em> differing in at most one element.</em></li><li><em>Whereas, </em><strong><em>local sensitivity</em></strong><em> is the change in </em><strong><em>one data set</em></strong><em> with differing at most one element.</em></li></ul><h3>Global Sensitivity</h3><p><em>Global sensitivity refers to the </em><strong><em>maximum difference of the output </em></strong><em>a query function can result in </em><strong><em>when one change is made to any data set</em></strong><em>.</em></p><p>Global sensitivity determines the <strong>magnitude of the noise</strong> needed in order to meet the ε-DP requirements.</p><p><em>Given a query function </em><strong><em>f</em></strong><em> that is operating on a dataset </em><strong><em>D</em></strong><em> and producing the maximum result difference for all data sets (D1,D2) with at most one different entry will be:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*shrwTa8cUph2EoNR.png" /></figure><p><strong><em>where ‖.‖1 is the L1-norm distance between datasets differing at most one element, max is the maximum result of f(D1)−f(D2) for all datasets D1,D2.</em></strong></p><blockquote><strong>Note:</strong> <em>Global sensitivity is the maximum differences in output with consideration of all possible datasets and is therefore only </em><strong><em>dependent on the query and not the dataset.</em></strong></blockquote><p>This has wide consequences for the utility of certain queries. This can be demonstrated by a simple query that takes the <strong>sum</strong> of any dataset where every entry can in theory contribute any arbitrary amount. <em>The largest difference an output can result in from any query is infinite </em>due to the fact that there is no upper bound on any single entry, <em>thus the global sensitivity for the sum mechanism is infinite.</em></p><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/fundamentals-of-ai-ml-for-product-managers-64279e9d3681">1. Fundamentals of AI, ML and Deep Learning for Product Managers</a></blockquote><blockquote><a href="https://becominghuman.ai/the-unfortunate-power-of-deep-learning-3c06aef76e63">2. The Unfortunate Power of Deep Learning</a></blockquote><blockquote><a href="https://becominghuman.ai/graph-neural-network-for-3d-object-detection-in-a-point-cloud-309ba40897b2">3. Graph Neural Network for 3D Object Detection in a Point Cloud</a></blockquote><blockquote><a href="https://becominghuman.ai/know-the-biggest-notable-difference-between-ai-vs-machine-learning-cc1d7d18976a">4. Know the biggest Notable difference between AI vs. Machine Learning</a></blockquote><p><em>However the global sensitivity does not exclude the possibility to generate accurate information for sum queries. </em><strong><em>Bounds</em></strong><em> can be introduced into the query mechanism for the dataset, that </em><strong><em>effectively limits the datasets ability to store values</em></strong><em> greater than a predetermined threshold.</em></p><p>The dataset must continuously be modified in order to guarantee no value exceeds the threshold, the global sensitivity would not be infinite and only depend on the query function and the threshold. Thus, it can be said that <strong><em>Global sensitivity is the minimum sensitivity needed for a query to cover all possible datasets.</em></strong></p><blockquote><em>In the process of releasing data while using queries such as </em><strong><em>count </em></strong><em>or </em><strong><em>sum </em></strong><em>that has</em><strong><em> low sensitivity work well with global sensitivity</em></strong><em>. We could take the case of count query as an example which has GS(f) = 1 that is smaller than the true answer. However, when it comes to queries like median, average the global sensitivity is much higher.</em></blockquote><p><strong>Another way of looking at it is,</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oYFd3S6cM9gIKHvP.png" /></figure><p>Here, <em>d(x, x’)</em> represents the <em>distance</em> between two datasets <em>x </em>and <em>x’</em> and we say that two datasets are <em>neighbors</em> if their distance is 1 or less. How this distance is defined has a huge effect on the definition of privacy we obtain.</p><ul><li>The distance between two datasets should be equal to 1 (i.e. the datasets are neighbors) if they differ in the data of exactly one individual.</li><li>This idea is easy to formalize in some contexts (e.g. in the US Census, each individual submits a single response containing their data) but extremely challenging in others (e.g. location trajectories, social networks, and time-series data).</li></ul><p>Formally, the distance is encoded as a <em>symmetric difference</em> between the two datasets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*t5HfNRyAjjgE5yEj.png" /></figure><p><strong><em>This particular definition has several interesting and important implications, for example:</em></strong></p><ul><li>If <em>x’</em> is constructed from <em>x</em> by <em>adding one row</em>, then <em>d(x, x’)</em> = 1</li><li>If <em>x’</em> is constructed from <em>x</em> by <em>removing one row</em>, then <em>d(x, x’)</em> = 1</li><li>If <em>x’</em> is constructed from <em>x</em> by <em>modifying one row</em>, then <em>d(x, x’)</em> = 2</li></ul><p>In other words, adding or removing a row results in a neighboring dataset at distance 1, whereas modifying a row results in a dataset at a distance 2.</p><p>This particular definition of distance results in what is typically called <em>unbounded differential privacy</em>. Many other definitions are possible, including one called <em>bounded differential privacy</em> in which modifying a single row in a dataset <em>does</em> result in a neighboring dataset.</p><p>The definition of global sensitivity says that for <em>any two</em> neighboring datasets x and x’, the difference between <em>f(x)</em> and<em> f(x’) </em>is at most <em>GS(f)</em>. <strong>This measure of sensitivity is called “global” because it is independent of the actual dataset being queried, it holds true for <em>any</em> choice of neighboring <em>x and x’</em>.</strong></p><h3>Other Important Concepts</h3><h4><strong>L1 and L2 Norms</strong></h4><p>The <strong><em>L1 norm</em></strong> of a vector V of length k is defined as (i.e. it’s the sum of the vector’s elements).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*W1k3lpWHc6N8hNmc4RtRgg.jpeg" /></figure><p><strong><em>Example:</em></strong> In 2-dimensional space, the L1 norm of the difference between two vectors yields the “Manhattan distance” between them.</p><p>The <strong><em>L2 norm</em></strong> of a vector V of length k is defined as (i.e. the square root of the sum of the squares).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*LUz7IXBOR5DuXc3Co0U6tA.jpeg" /></figure><p><strong><em>Example: </em></strong>In 2-dimensional space, this is the “Euclidian distance,” and it’s always less than or equal to the L1 distance.</p><h4><strong>L1 VS L2 Sensitivities</strong></h4><blockquote><em>The L1 sensitivity of a vector-valued function is equal to the </em><strong><em>sum of the </em>element wise<em> sensitivities.</em></strong><em> For example, if we define a vector-valued function </em>f <em>that returns a length-</em>k<em> vector of 1-sensitive results, then the </em>L1<em> sensitivity of</em> f <em>is </em>k<em>.</em></blockquote><blockquote><em>The L2 sensitivity of a vector-valued function is a vector-valued function </em>f <em>returning a length-k vector of 1-sensitive results has L2 sensitivity of </em><strong>√k<em> </em></strong><em>. </em><strong><em>For long vectors, the L2 sensitivity will obviously be much lower than the L1 sensitivity!</em></strong><em> </em><strong><em>For some applications, like </em></strong><a href="http://becominghuman.ai/"><strong><em>machine learning</em></strong></a><strong><em> </em></strong><em>algorithms (which sometimes return vectors with thousands of elements), </em><strong><em>L2 sensitivity is </em>significantly<em> lower than L1 sensitivity.</em></strong></blockquote><h3>Local Sensitivity</h3><p>Local sensitivity attempts to calculate the sensitivity for a local dataset, where the possible<strong><em> changes are bound by the local data set and not the universe of all </em></strong><a href="https://becominghuman.ai/how-can-we-improve-the-quality-of-our-data-61b55c9063d1"><strong><em>data</em></strong></a><strong><em> sets.</em></strong></p><p><em>Given a query function </em><strong><em>f</em></strong><em> that is operating on a data set D1, the local sensitivity is then the maximum differences that one change in D1 can produce:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/0*1Iav69RRhd6z9GPG.png" /></figure><p>D1 is the known dataset and D2 is another dataset with at most one different element (relative to the data set D1).</p><blockquote><em>For queries such as </em><strong><em>count </em></strong><em>or </em><strong><em>range</em></strong><em>, the local sensitivity is identical to the global sensitivity.</em></blockquote><p><strong>Note:</strong> The feasibility of replacing the global with the local sensitivity have been proven to be <strong><em>problematic and additional limitations are needed </em></strong>in order to satisfy the conditions of ε-Differential Privacy. It is seen that the magnitude of the noise could leak information due to the fact the <em>amount of noise reveals information about the data set to adversaries </em>as from the formula, every differentially private algorithm must add a noise at least as large as the local sensitivity. <strong>Local sensitivity is the minimum sensitivity needed for a query to cover one specific data set.</strong></p><p>You may also check this — <a href="http://www.diva-portal.org/smash/get/diva2:1113852/FULLTEXT01.pdf">A Case Study on Differential Privacy — Sensitivity (pg-19</a>)</p><p>Similarly, <strong><em>another way of looking at it is</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*Y_jdZaTpVKL6tm9v.png" /></figure><p>Global sensitivity considers <em>any</em> two neighboring datasets, but since we’re going to run our differentially private mechanisms on an <em>actual</em> dataset — shouldn’t we consider neighbors of <em>that</em> dataset?</p><blockquote><em>The intuition behind </em>local sensitivity <em>is to</em> <strong><em>fix one of the two datasets</em></strong><em> to be the </em><strong>actual<em> dataset being queried</em></strong><em>, and consider all of its neighbors.Global sensitivity considers </em>any<em> two neighboring datasets, but since we’re going to run our differentially private mechanisms on an </em>actual<em> dataset — shouldn’t we consider neighbors of </em>that<em> dataset?</em></blockquote><p>Local sensitivity is a function of both the query <em>(f) </em>and the <em>actual</em> dataset <em>(x)</em>. Unlike in the case of global sensitivity, we <strong>can’t</strong> talk about the local sensitivity of a function <strong><em>without also considering the dataset at which that local sensitivity occurs</em></strong>.</p><blockquote><em>Local sensitivity allows us to place </em><strong><em>finite bounds</em></strong><em> on the sensitivity of some functions whose global sensitivity is difficult to bound. This is because </em><strong>local sensitivity measure is defined in terms of the actual dataset’s size, which is not possible under global sensitivity.</strong></blockquote><p><strong>But, </strong>the <strong>issue with local sensitivity</strong> is, because it itself depends on the dataset, if the analyst knows the local sensitivity of a query <em>at a particular dataset</em>, then the analyst may be able to infer some information about the dataset. It’s therefore <em>not possible</em> to use local sensitivity directly to achieve differential privacy.</p><p>Moreover, keeping the local sensitivity secret from the analyst <em>doesn’t help either</em>. It’s possible to determine the scale of the noise from just a few query answers, and the analyst can use this value to infer the local sensitivity. <strong><em>Differential privacy is designed to protect the output of f(x) — not of the sensitivity measure used in its definition.</em></strong></p><p>To solve this, <strong><em>Propose-test-release and Smooth Sensitivity like approaches</em></strong> have been proposed for <em>safely using local sensitivity</em>, which is beyond the scope of this blog post, but if you are interested to know more about it — <a href="https://github.com/jnear/cs295-data-privacy/blob/master/notes/7%20-%20Local%20Sensitivity.ipynb">check these notes shared by Professor Joseph Near </a>.</p><p><strong>Now, you might be wondering why not just go with Global sensitivity? Why waste time and effort on local sensitivity and on creating the better safer approaches of it?</strong></p><p>Global sensitivity considers <em>any</em> two neighboring datasets, i.e., <em>Global sensitivity is the minimum sensitivity needed for a </em><strong><em>query to cover all possible datasets</em></strong><em>.</em> But since we will run our differentially private mechanisms on an <em>actual</em> dataset we might want to consider neighbors of <em>that</em> dataset! Thus, <em>Local sensitivity is the minimum sensitivity needed for a </em><strong><em>query to cover one specific data set</em></strong><em>.</em></p><p>That’s it folks! In the next blog we will look deeper into the Noise adding mechanisms for the Differential Privacy.</p><p>Till then you may also check out my other beginner friendly Series:</p><ul><li><a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea?source=---------9------------------"><strong><em>Introduction to “Tensors”</em></strong></a></li><li><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)</em></strong></a></li></ul><h4>References:</h4><ul><li><a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf">https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf</a></li><li><a href="https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf">https://kth.diva-portal.org/smash/get/diva2:1112478/FULLTEXT01.pdf</a></li></ul><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*2lvCls4yjxVMfZSR" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c94fd14b9837" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837">“Sensitivity” in Differential Privacy</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Privacy Definition]]></title>
            <link>https://medium.com/@shaistha24/differential-privacy-definition-bbd638106242?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd638106242</guid>
            <category><![CDATA[differential-privacy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[series]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Tue, 15 Sep 2020 05:31:01 GMT</pubDate>
            <atom:updated>2020-10-03T20:02:17.146Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*4TVH-W74Sxv7fREo.jpg" /></figure><p><strong><em>Summary: Sometimes knowing the basics is important! This third blog post of “Differential Privacy Basics Series”, covering a quick and easy introduction to Differential Privacy Definition- Epsilon, delta, etc. For more posts like these on Differential Privacy follow</em></strong><a href="https://twitter.com/shaistha24"><strong><em> Shaistha Fathima</em></strong></a><strong><em> on twitter.</em></strong></p><h3>Differential Privacy Basics Series</h3><ul><li><a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049"><strong><em>What is Differential Privacy?</em></strong></a></li><li><a href="https://medium.com/@shaistha24/global-vs-local-differential-privacy-56b45eb22168"><strong><em>Global vs Local Differential Privacy</em></strong></a></li><li><a href="https://medium.com/@shaistha24/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837"><strong><em>Query “Sensitivity” types and effects on Differential Privacy Mechanism</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-noise-adding-mechanisms-ede242dcbb2e"><strong><em>Differential Privacy — Noise adding Mechanisms</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-basics-series-conclusion-and-important-list-of-resources-2057a0d0d2be"><strong><em>Differential Privacy Basics Series Conclusion and Important list of resources.</em></strong></a></li></ul><p><strong>What is Privacy?</strong></p><p>Everyone has different opinion on what privacy means, but in terms of data, it could be,<strong> <em>the right to control how information about you is being used, processed, stored, or shared.</em></strong></p><p>In today’s information realm, an extensive sensitive personal information is being possessed by the daily services we tend to use, such as search engines, mobile services, on-line social activity and so on. The loss of data privacy is imminent since the guarantee of data privacy incorporates controlling access to information, controlling the flow or purpose of information. Thus, the need for Differential Privacy (DP) arises with the hope of a better and robust data privacy.</p><h3>What is Differential Privacy?</h3><p>As per the most accepted DP definition by <em>Cynthia Dwork</em> in her book <a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf"><em>Algorithmic Foundations of Differential Privacy</em></a></p><blockquote>Differential Privacy describes a promise, made by a data holder, or curator, to a data subject (owner), and the promise is like this:<em> </em><strong><em>“You will not be affected adversely or otherwise, by allowing your data to be used in any study or analysis, no matter what other studies, datasets or information sources are available”.</em></strong></blockquote><h3>Formal Definition of Differential Privacy:</h3><p>Cynthia Dwork’s Formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*005l3W0Nj3GaCvuH.png" /></figure><p>Where,</p><p><strong>M:</strong> Randomized algorithm i.e query(db) + noise or query(db + noise).</p><p><strong>S: </strong>All potential output of M that could be predicted.</p><p><strong>x:</strong> Entries in the database. (i.e, N)</p><p><strong>y:</strong> Entries in parallel database (i.e., N-1)</p><p><strong>ε:</strong> The maximum distance between a query on database (x) and the same query on database (y).</p><p><strong>δ: </strong>Probability of information accidentally being leaked.</p><p><strong><em>NOTE: This definition is valid for ONE QUERY in the database and not for multiple queries.</em></strong></p><h4><strong>Key Points to be Noted:</strong></h4><ul><li>This DP definition does not create DP, instead it is a measure of “How much privacy is afforded by a query?”</li><li>It gives the comparison between running a query M on database (x) and a parallel database (y) (One entry less than the original database).</li><li>The measure by which these two probability of random distribution of full database(x) and the parallel database(y) can differ is given by Epsilon (ε) and delta (δ).</li></ul><h3>(1) Epsilon (ε):</h3><p>It is the <em>maximum distance</em> between a query on database (x) and the <em>same</em> query on database (y). That is, its a <strong><em>metric of privacy loss</em></strong> at a differential change in data (i.e., adding or removing 1 entry). Also known as the <strong><em>privacy parameter</em></strong> or the <strong><em>privacy budget</em></strong><em>.</em></p><p><strong>(i) When ε is small</strong></p><p>(ε,0)-differential privacy asserts that for all pairs of adjacent databases x, y and all outputs M, an adversary cannot distinguish which is the true database on the basis of observing the output. When ε is small, failing to be (ε,0)-differentially private is not necessarily alarming — for example, the mechanism may be (2ε,0)-differentially private.</p><blockquote>All Small Epsilons(ε) Are Alike, i.e., t<em>he nature of the privacy guarantees with differing but small epsilons are quite similar.</em></blockquote><p><strong>(ii) When ε is large</strong></p><p>Failure to be (15,0)-differentially private merely says that there exists neighboring databases and an output M, for which the ratio of probabilities of observing M conditioned on the database being, respectively, x or y, is large.</p><p>An output M, might be very unlikely (this is addressed by (ε, δ)-differential privacy); for databases x and y might be terribly contrived and unlikely to occur in the “real world”; the adversary may not have the right auxiliary information to recognize that a revealing output has occurred; or may not know enough about the database(s) to determine the value of their symmetric difference.</p><h3>A smaller ε will yield better privacy but less accurate response.</h3><blockquote><strong><em>Small values</em></strong><em> of ε require to provide </em><strong>very<em> similar outputs</em></strong><em> when given </em><strong><em>similar inputs</em></strong><em>, and therefore </em><strong><em>provide higher levels of privacy</em></strong><em>; </em><strong><em>large values </em></strong><em>of ε allow </em><strong><em>less similarity</em></strong><em> in the outputs, and therefore provide </em><strong><em>less privacy</em></strong><em>.</em></blockquote><p>Simply put, as ε is the <em>maximum distance</em> between a query on database (x) and the <em>same</em> query on database (y). When not much differential distance is present between the two query i.e., ε is small, and if, the inputs of the queries are very similar then the outputs will be very similar too. Therefore, the adversary may not be able to analyze it and get the right auxiliary information out of it. Thus, providing higher chances or level of privacy when compared to when ε is a large value.</p><blockquote><em>Thus, much as a weak cryptosystem may leak anything from only the least significant bit of a message to the complete decryption key, the failure to be (ε,0)- or (ε, δ)-differentially private may range from effectively meaningless privacy breaches to complete revelation of the entire database.</em></blockquote><p>Two types of <em>ε </em>are seen between two differentially private databases,</p><p>(i) <em>Epsilon Zero (ε = 0)</em> : The OUTPUT of a query for all parallel databases is the same value as the full (original) database. Often, if sensitivity = 0,then ε =0 too.</p><p>(ii) <em>Epsilon One (ε = 1)</em> : Maximum distance between all the queries would be 1 OR the maximum distance between 2 random distributions M(x) and M(y) is 1.</p><blockquote><em>It can also be stated that </em>ε or the privacy budget <em>is an </em><strong>individuals limit of how much privacy they are ok with leaking.</strong></blockquote><p>The concept arises from the issue that even though no single operation being done on data reveals who the subjects might be, <strong><em>by combining the data from multiple trials, one might be able to figure out details about an individual</em></strong>. As a result, <em>every person has a “privacy budget”</em><strong><em>.</em></strong></p><p>Each time an individual is included in a differential privacy calculation, it eats away at their privacy budget (by <em>ε</em> ). Most calculations in Machine Learning can be done to give a differential privacy guarantee and a utility guarantee. This means that all these calculations can also be accounted for in the privacy budget.</p><h3>Real World Examples</h3><p>1. <strong>Apple and privacy budgets:</strong></p><p>A recent issue arose with apple as news came out that they were using differential privacy in order to maintain their users’ privacy. They were running operations of the app data they were receiving from their devices and made sure to use values that would ensure the privacy budget for the users would not be exceeded. However, they broke the standard by resetting the privacy budget every day.</p><p>Since the privacy budget is a value that is used cumulatively for life, this became a problem. Users were now no longer guaranteed privacy because no one was keeping track of their overall privacy budgets, they just knew that no single day’s operations would result in a breach of privacy for a user.</p><p>2. <strong>2020 Census:</strong></p><p>The 2020 Census announced that it would be using differential privacy in order to maintain the privacy of the American people. Specifically, they will do so by releasing all statistics in a deferentially private way. It has not yet been determined how epsilon and randomization of the data will occur. This is considered a big step forward because previous methods for maintaining privacy were dependent on a bunch of tricks that caused the data to be convoluted.</p><p>This decision faces some backlash from the leaders of academic sub fields devoted to interpreting the census. These people believe that the increased privacy provided by the new methods is not worth the trade-off for accuracy. A key point to learn from this example is that society needs to make the decision of which point in the privacy-accuracy spectrum we want to beat.</p><h3>(2) Delta (δ):</h3><p>It is the <em>probability of information accidentally being leaked</em>. If δ= 0, we say that output M is ε-differentially private.</p><p>Typically we are <strong><em>interested in values of δ that are less than the inverse of any polynomial in the size of the database.</em></strong></p><p>(i) <em>The values of δ on the order of 1/∥x∥1 are </em><strong><em>very dangerous</em></strong><em>:</em> They <strong>permit “preserving privacy”</strong> by publishing the complete records of a small number of database participants — precisely the <strong>“just a few” philosophy</strong>, where only the people considered to have greater privacy concerns are protected.</p><p>(ii) When δ is negligible or equal to zero: It is called ε-differential privacy or (ε,0)-differential privacy.</p><blockquote><strong><em>Note: </em></strong><em>ε is independent of the size of database, where as in case of </em>δ, the chances of privacy leak might increase with the size of the database. <em>Hence, ideally we would want to set δ value to be less than the inverse of the size of database.</em></blockquote><h3>This brings us to question — (ε,0) or ε -differential privacy VS (ε, δ)-differential privacy?</h3><blockquote><strong><em>(ε,0)-differential privacy</em></strong><em> ensures that, for every run of the mechanism M(x), the output observed is (almost) equally likely to be observed one very neighboring database, simultaneously.</em></blockquote><p>Where as,</p><blockquote><strong><em>(ε, δ)-differential privacy</em></strong><em> says that for every pair of neighboring databases x, y, it is extremely unlikely that,ex post facto the observed value M(x) will be much more or much less likely to be generated when the database is x than when the database is y. It </em><strong><em>ensures</em></strong><em> that for all adjacent x, y, the </em><strong><em>absolute value of the privacy loss will be bounded by ε with probability at least 1−δ.</em></strong></blockquote><p><strong>So, what does it mean?</strong></p><p>In case of, <strong>(ε,0)-differential privacy or ε-differential privacy , </strong>where <em>δ =0, i.e., probability of data leak δ is to be zero. </em>Thus, a deferentially private data set with distance of 1 or less will have very similar output.</p><p>Where as, in case of <strong>(ε, δ)-differential privacy, </strong>the chances of data leaks are possible, maybe because of higher sensitivity values or large database size with large rows, etc. Thus, if the value of ε is not small, then there is less likely of the outputs being the same.</p><p>We usually go for ε-differential privacy based mechanisms — like Laplace mechanism — not only because of its simplicity but also, because in real life chances of delta being high, i.e., data leak, is pretty low! (ε, δ)-differential privacy usually works well with large databases where chances of privacy leak is high and might not also have low sensitivity.</p><p>That’s all folks! In the next post of this series we will look into types of noise adding Mechanisms in Differential Privacy and their comparison.</p><p>Till then you may also check out my other beginner friendly Series:</p><ul><li><a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea?source=---------9------------------"><strong><em>Introduction to “Tensors”</em></strong></a></li><li><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd638106242" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Global vs Local Differential Privacy]]></title>
            <link>https://medium.com/@shaistha24/global-vs-local-differential-privacy-56b45eb22168?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/56b45eb22168</guid>
            <category><![CDATA[series]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[differential-privacy]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Mon, 14 Sep 2020 05:31:01 GMT</pubDate>
            <atom:updated>2020-10-03T20:03:18.643Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>Global vs Local Differential Privacy(DP) followed by real world implementation examples</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*qk0HP9x8H3pp1dF_.jpg" /></figure><p><strong><em>Summary: We learn best with toy code we can play with and diving into the documentations. This is second blog post of “Differential Privacy Basics Series”, covering a quick and easy explanation to Global vs Local Differential Privacy(DP) followed by real world implementation examples. Thank you </em></strong><a href="https://twitter.com/jacobmerryman"><strong><em>Jacob Merryman</em></strong></a><strong><em> for the amazing graphic used in this blog post. For more posts like these on Differential Privacy follow</em></strong><a href="https://twitter.com/shaistha24"><strong><em> Shaistha Fathima</em></strong></a><strong><em> on twitter.</em></strong></p><h3>Differential Privacy Basics Series</h3><ul><li><a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049"><strong><em>What is Differential Privacy?</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-definition-bbd638106242"><strong><em>Differential Privacy Definition</em></strong></a></li><li><a href="https://medium.com/@shaistha24/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837"><strong><em>Query “Sensitivity” types and effects on Differential Privacy Mechanism</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-noise-adding-mechanisms-ede242dcbb2e"><strong><em>Differential Privacy — Noise adding Mechanisms</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-basics-series-conclusion-and-important-list-of-resources-2057a0d0d2be"><strong><em>Differential Privacy Basics Series Conclusion and Important list of resources.</em></strong></a></li></ul><p>If you haven’t read the <a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049">First Blog post</a> of this series, I strongly recommend you to read it before getting started with this one!</p><p><strong>A quick recap of Differential Privacy…</strong></p><p>As per the most accepted definition of Differential Privacy (DP) given by <em>Cynthia Dwork’s work</em> in the book <a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf"><em>Algorithmic Foundations of Differential Privacy</em></a><em>.</em></p><blockquote>Differential Privacy describes a promise, made by a data holder, or curator, to a data subject (owner), and the promise is like this:<em> </em><strong><em>“You will not be affected adversely or otherwise, by allowing your data to be used in any study or analysis, no matter what other studies, datasets or information sources are available”.</em></strong></blockquote><p>In a nutshell, differential privacy ensures that an adversary should not be able to reliably infer whether or not a particular individual is participating in the database query, even with unbounded computational power and access to every entry in the database except for that particular individual’s data.</p><p>DP works by adding statistical noise to the data (either to their inputs or to the output). Based on where the noise is added, DP is classified into two types — <strong>Local Differential Privacy </strong>and<strong> Global Differential Privacy.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sw1pbWNxJBIPVFEf.png" /></figure><h3>Local Differential Privacy</h3><p>Local DP <em>adds noise to the individual (input) data points.</em> One of the simplest example would be, by Abhishek Bhowmick in Udacity’s <a href="https://www.udacity.com/course/secure-and-private-ai--ud185">Secure and Private AI Course</a>:</p><p>Suppose we want to know the <em>average</em> amount of money an individual holds in his/her pocket to be able to buy an online course? Now chances are, many might not want to give out the exact amount! So, what do we do?!</p><p>This is where Local DP comes in, instead of asking the exact amount, we ask the individuals to <em>add any random value (noise)</em> in the range of -100 to +100 to the amount they hold in their pockets and give us just the resultant <strong>sum</strong> of it. That is if ‘X’ had 30$ in his/her pocket by adding a random number say -10 to it, (30 + (-10) ), they give us <em>just the result</em>, which is 20$ in this case. Thus, preserving their individual privacy.</p><p>Here, the noise can be added directly to the Database or the individuals can add noise to their own datasets before putting it into the Database as seen in the above example.</p><p>Another example would be of the probability of coin flips using — Randomized Response, a technique used in social science when trying to learn about the high level trends for a taboo behavior such as this example, when not sure if the results are all true or may be skewed! For example a survey on — “Did you ever steal in your life?”</p><p>Assuming a simple YES or NO answer for the above question, we follow the below steps:</p><ul><li>Step 1: Flip a coin 2 times.</li><li>Step 2: If the <em>first</em> coin flip is <strong><em>heads</em></strong>, then take the answer (Yes/No) honestly.</li><li>Step 3: Else if the first coin flip is <strong><em>tails</em></strong> , answer according to the second coin flip! Now the individual has to answer ‘Yes’ if it lands on heads and ‘No’ if it lands on tails. We aren’t really giving the person a choice here. This is where we add ‘randomness’.</li></ul><p>Here is a link to understand the probability with coin flips better! — <a href="https://towardsdatascience.com/how-your-data-is-secured-by-a-coin-toss-c933f9e13d4a">How your data is secured(by a coin toss)</a></p><p>Coming back to our example, think about the probability of a person responding with <strong><em>‘Yes’ </em></strong>even though that person actually <strong><em>hasn’t</em></strong> stolen anything in their life!</p><p>For this to happen, the second coin toss has to occur and it has to land on heads. That’s because if the first toss was heads, the person would’ve told that they did <strong>not </strong>steal in their entire life. But if the first coin toss is tails <strong>and </strong>the second coin toss is heads the person will respond<strong> Yes </strong>even though the true response is <strong>No.</strong></p><p>Thus,<strong> </strong>with the introduction of noise or randomness, each person is now protected with “plausible deniability”. Each person has a high degree of protection. Furthermore, we can recover the underlying statistics with some accuracy, as the “true statistics” are simply averaged with a 50% probability. Thus, if we collect a bunch of samples and it turns out that 60% of people answered yes, then we know that the TRUE distribution is actually centered around 70%, because 70% averaged with 50% (coin flip) is 60% which is the result we obtained.</p><p>However, this comes at the cost as we are using averaged values i.e, gained privacy but have lost some accuracy. <strong><em>The greater the privacy protection (plausible deniability) the less accurate the results.</em></strong></p><p><strong><em>Advantages?! — </em></strong>Doing this, the data curator / central aggregator, the person who is aggregating the dataset, does not know the actual value, and thus privacy is protected. <strong><em>The user does not have to trust the data curator or the database owner (Bob) to use his/her data responsibly.</em></strong></p><p>But, since each user must add noise to their own data, <strong>the total noise is much larger</strong>. And typically would need many more users to get useful results. To mitigate this problem, practical applications often use high values of epsilon (<em>ε)</em>.</p><h3>Some interesting use cases of Local DP:</h3><ul><li><a href="https://research.google/pubs/pub42852/"><strong><em>RAPPOR</em></strong></a>, where Google used local differential privacy to collect data from users, like other running processes and Chrome home pages.</li><li><a href="https://machinelearning.apple.com/2017/12/06/learning-with-privacy-at-scale.html"><strong><em>Private Count Mean Sketch</em></strong></a> (and variances) where Apple used local differential privacy to collect emoji usage data, word usage and other information from iPhone users (iOS keyboard).</li><li><a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0207639"><strong><em>Privacy-preserving aggregation of personal health data streams</em></strong></a><strong><em> </em></strong>paper, develops a novel mechanism for privacy-preserving collection of personal health data streams that is characterized as temporal data collected at fixed intervals by leveraging local differential privacy (Local DP)</li></ul><h3>Global Differential Privacy</h3><p>In case of Global DP, <em>noise is added to the outputs(query) of the database i.e, noise is added </em>only <em>once</em>, at the end of the process before sharing it with the third party. <strong>Noise is added by the data curator/ central aggregator (Mike) to the output of a query of the database. </strong>Doing this, the data curator protects user privacy from people who are querying the database.</p><p>Data curator or the database owner (Mike) is <strong>trustworthy</strong>. The database has all the private data and a <em>data curator has access to the real raw data.</em></p><p><strong><em>Advantages?!</em></strong> — Accuracy!! Don’t need to add a lot of noise to get valuable results with a low epsilon (<em>ε)</em>.</p><p>But<strong>, each user has to trust the data curator</strong> enough to share data with it. That might be difficult: the aggregator can be an untrusted company or government. Also, with the global model, all the data is collected in one place. It increases the risk of catastrophic failure, for example if the aggregator gets hacked and leaks all the data.</p><blockquote><strong><em>Note:</em></strong><em> If the database owner is trustworthy, the only difference between the local and global DP is that the global DP leads to </em><strong>more accurate results</strong><em> </em><strong>with the same level of privacy protection.</strong><em> However, this requires a database owner to be trustworthy. That is the database owner/ data curator should add noise properly and protect the privacy of the user.</em></blockquote><h3>An interesting use case of Global DP:</h3><ul><li><a href="https://www.census.gov/newsroom/blogs/random-samplings/2019/02/census_bureau_adopts.html"><strong><em>Census Bureau Adopts Cutting Edge Privacy Protections for 2020 Census</em></strong></a> i.e, the US Census will use differential privacy to anonymize the data before publication.</li></ul><p>In conclusion, <strong>When should you opt for Local DP over Global DP?</strong></p><p>The answer to this is would be, <strong>Local DP </strong>is great for <strong>individual responses</strong> to surveys and for anytime you have private information that cannot leave devices like mobile phones. That is, when you <strong>have more user privacy concerns and cannot trust the data curator completely</strong>.</p><p>But, if the data curator is trusted Global DP is a better choice with a little better results.</p><p><strong>Distributed Differential privacy</strong> is also a type of DP but is fairly new in its practical implementation. <em>It helps solve the problem of answering queries about private data that is spread across multiple databases.</em> Distributed DP is beyond the scope of this blog, but, if you are interested you may read <a href="http://arjunnarayan.com/publications/dissertation.pdf">DISTRIBUTED DIFFERENTIAL PRIVACY AND APPLICATIONS</a> to understand it better!</p><p>That’s all folks! In the next post of this series we will look into the Definition of Differential privacy and some core concepts.</p><p>Till then you may also check out my other beginner friendly Series:</p><ul><li><a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea?source=---------9------------------"><strong><em>Introduction to “Tensors”</em></strong></a></li><li><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56b45eb22168" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Differential Privacy?]]></title>
            <link>https://becominghuman.ai/what-is-differential-privacy-1fd7bf507049?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fd7bf507049</guid>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[differential-privacy]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Sun, 13 Sep 2020 10:23:31 GMT</pubDate>
            <atom:updated>2020-10-01T21:30:09.603Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Introduction to DP with Randomized response example.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*Za9vRNWh7SFYvgiB.jpg" /></figure><p><strong><em>Summary: Sometimes knowing the basics is important! This beginner friendly blog post covers a quick and easy introduction to Differential Privacy and is part of “Differential Privacy Basics Series”. Thank you </em></strong><a href="https://twitter.com/jacobmerryman"><strong><em>Jacob Merryman</em></strong></a><strong><em> for the amazing graphic used in this blog post. For more posts like these on Differential Privacy follow</em></strong><a href="https://twitter.com/shaistha24"><strong><em> Shaistha Fathima</em></strong></a><strong><em> and </em></strong><a href="https://twitter.com/openminedorg"><strong><em>OpenMined</em></strong></a><strong><em> on twitter.</em></strong></p><h3><strong><em>Differential Privacy Basics Series</em></strong></h3><ul><li><a href="https://medium.com/@shaistha24/what-is-differential-privacy-1fd7bf507049"><strong><em>What is Differential Privacy?</em></strong></a></li><li><a href="https://medium.com/@shaistha24/global-vs-local-differential-privacy-56b45eb22168"><strong><em>Global vs Local Differential Privacy</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-definition-bbd638106242"><strong><em>Differential Privacy Definition</em></strong></a></li><li><a href="https://medium.com/@shaistha24/query-sensitivity-types-and-effects-on-differential-privacy-mechanism-c94fd14b9837"><strong><em>Query “Sensitivity” types and effects on Differential Privacy Mechanism</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-noise-adding-mechanisms-ede242dcbb2e"><strong><em>Differential Privacy — Noise adding Mechanisms</em></strong></a></li><li><a href="https://medium.com/@shaistha24/differential-privacy-basics-series-conclusion-and-important-list-of-resources-2057a0d0d2be"><strong><em>Differential Privacy Basics Series Conclusion and Important list of resources.</em></strong></a></li></ul><p>Before we dive deeper into Differential Privacy (DP) and answer the 4 W’s and an H (What, Where, When, Why and How), few of the most important questions that you must ask yourself are… What is Privacy? Should we really care about it? How does it matter?…</p><h3>What is Privacy?</h3><p>As per Wikipedia’s definition,</p><blockquote><em>Privacy is the ability of an individual or group to seclude themselves or information about themselves and thereby express themselves selectively.</em></blockquote><p>To put it simply, privacy is an individual’s right to withhold some of their data which they deem to be private and share the ones they are comfortable with.</p><p>Coming to… <strong><em>Should we really care about it? How does it matter?</em></strong></p><p>In this digital age, data privacy has always been a concern for some of us, to the extent that people are paranoid enough to use a made-up names instead of their own! Even if you are not that paranoid, say you are comfortable with sharing your <em>real</em> name with an unknown person as part of your introduction, you <strong><em>might not</em></strong> feel comfortable with sharing some of your other details like birth date, hangout places you love to go to, your hobbies, etc. This is where privacy comes in, the data YOU would want to keep PRIVATE!</p><blockquote><em>Thus, </em><strong><em>Privacy can also be said to be </em></strong><em>the right to control how information about you is used, processed, stored, or shared.</em></blockquote><p>For a better understanding of — why privacy matters? OR Privacy vs Security? I would recommend you to take a short read to the below blog posts:</p><ul><li><a href="https://kromtech.com/blog/security-center/what-is-privacy">What is Privacy? in terms of security.</a></li><li><a href="https://teachprivacy.com/10-reasons-privacy-matters/">10 Reasons Why Privacy Matters</a></li><li><a href="https://www.scu.edu/ethics/focus-areas/internet-ethics/resources/why-we-care-about-privacy/">Why We Care about Privacy</a>? <em>(Longer read!)</em></li><li><a href="https://www.globalsign.com/en/blog/what-is-the-difference-between-privacy-and-security">Privacy vs. Security</a></li><li><a href="http://www.differencebetween.net/technology/internet/difference-between-security-and-privacy/">Difference Between Security and Privacy</a></li></ul><p>Coming to the main question…</p><figure><a href="https://aijobsboard.com/?fbclid=IwAR0GcsLs8A6CFM-fynrZMM3sTgo_Zpbto2CjglJm0Dmi6otC6YWc7CDFpQk"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dAlwXfVrtYspdCzmC8iADw.png" /></a><figcaption>Artificial Intelligence Jobs</figcaption></figure><h3>What is Differential Privacy (DP)?</h3><p><em>Wikipedia definition:</em></p><blockquote><em>Differential privacy is a </em><strong><em>system</em></strong><em> for publicly sharing information about a dataset by describing the patterns of groups within the dataset while withholding information about individuals in the dataset.</em></blockquote><p><strong><em>Note:</em></strong><em> Differential Privacy is not an algorithm but a System or Framework described for better data privacy!</em></p><p>One of the easiest examples to understand DP concerning the above definition is, the one stated by Abhishek Bhowmick (Lead, ML Privacy, CORE ML | Apple) in his interview in the Udacity’s <a href="https://www.udacity.com/course/secure-and-private-ai--ud185">Secure and Private AI Course</a>:</p><p><em>(Note: I will be using this course material as a reference throughout the post)</em></p><p>Suppose we want to know the <em>average</em> amount of money an individual holds in his/her pocket to be able to buy an online course? Now chances are, many might not want to give out the exact amount! So, what do we do?!</p><p>This is where DP comes in, instead of asking the exact amount, we ask the individuals to <em>add any random value (noise)</em> in the range of -100 to +100 to the amount they hold in their pockets and give us just the resultant <strong>sum</strong> of it. That is if ‘X’ had 30$ in his/her pocket by adding a random number say -10 to it, (30 + (-10) ), they give us <em>just the result</em>, which is 20$ in this case. Thus, preserving their individual privacy.</p><blockquote><em>To protect the data privacy obtained from the different potential individual, we add noise(the random number like in the above example) to the data to make it more private and secure! DP works by adding statistical noise to the data (either to their inputs or the output).</em></blockquote><p><strong>But</strong>, this brings us to another question — <strong><em>How is it useful to us if all we get are some random numbers?</em></strong></p><p>The answer to this is <strong><em>Law of Large Numbers</em></strong>:</p><blockquote><em>The law of large numbers, in probability and statistics, states that as a sample size grows, its mean gets closer to the average of the whole population.</em></blockquote><p>When a sufficiently large number of individuals give their resultant sum values. It is seen that when the average of these statistically collected data is taken, the noise cancels out and the average obtained is near to the true average (average of the data without adding noise (random number)). <em>Thus, we now have data on the “average amount an individual hold in their pocket”, at the same time preserving their privacy.</em></p><p><strong>Key Takeaways:</strong></p><ul><li>The law of large numbers states that an observed sample average from a large sample will be close to the true population average and that it will get closer, the larger the sample.</li><li>The law of large numbers does not guarantee that a given sample, especially a small sample, will reflect the true population characteristics or that a sample which does not reflect the true population will be balanced by a subsequent sample.</li></ul><p>For better understanding of the Law of Large Numbers, refer to the following:</p><ul><li><a href="https://www.khanacademy.org/math/statistics-probability/random-variables-stats-library/expected-value-lib/v/law-of-large-numbers">Law of large numbers — Khan Academy</a></li><li><a href="https://www.udacity.com/course/secure-and-private-ai--ud185">Secure and Private AI</a> <em>(for python code implementation)</em></li><li><a href="https://machinelearningmastery.com/a-gentle-introduction-to-the-law-of-large-numbers-in-machine-learning/">A Gentle Introduction to the Law of Large Numbers in Machine Learning</a></li><li><a href="https://kambria.io/blog/the-importance-of-the-law-of-large-numbers-in-ai/">The Importance of the Law of Large Numbers in AI</a></li></ul><p><strong><em>Another way of looking at DP</em></strong> is a definition by <em>Cynthia Dwork</em> in her book <a href="https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf"><em>Algorithmic Foundations of Differential Privacy</em></a></p><blockquote>Differential Privacy describes a promise, made by a data holder, or curator, to a data subject (owner), and the promise is like this:<em> </em><strong><em>“You will not be affected adversely or otherwise, by allowing your data to be used in any study or analysis, no matter what other studies, datasets or information sources are available”.</em></strong></blockquote><p>It sounds like a well thought definition but maybe more of a fantasy — as <em>De-anonymization of datasets may happen!</em></p><p>This may lead to a question — <strong><em>How do we know if the privacy of a person in the dataset is protected or not? </em></strong>For example, a database with (1) patients and their cancer status and their information OR (2) a coin flip and their (heads or tails) response!</p><p>The<em> key query</em> for the DP in such cases would be,” If we remove a person from the database and the query does not change, then that person’s privacy is fully protected”. To put it simply, <em>when querying a database, if I remove someone from the database, would the output of the query be any different?</em></p><p><strong><em>How do we check this? </em></strong>By creating a parallel database with one less entry (N-1) compared to the original database entries (N).</p><p>Lets take a simple example of coin flips, if the first coin flip is heads say Yes (1) and if its tails then answer as per the second coin flip. So, our database will be made of 0’s and 1’s i.e., a binary dataset. The easiest query that can be thought of with this binary dataset is “Sum Query”. The Sum query will add all the 1’s in the database and give a result.</p><p>Assuming, <strong>D</strong> is the original <a href="https://becominghuman.ai/cheat-sheets-for-ai-neural-networks-machine-learning-deep-learning-big-data-678c51b4b463">database</a> with N entries and <strong>D’</strong> is the parallel database with N-1 entries. On running the sum query on each of them, if <strong>sum(D) != sum(D’)</strong>, it means the <em>output query actually is conditioned directly on the information from a lot of people in D database!</em> It shows <strong><em>non-zero sensitivity</em></strong>, as the outputs are different.</p><blockquote><em>Sensitivity is the maximum amount that a query changes when removing an individual from the database.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oDxFjXaGpacnjbZq.png" /></figure><p>A good explanation of the above in terms of practice would be as quoted below from the<em> blog post by Nicolas Papernot and Ian Goodfellow — </em><a href="http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html">Privacy and machine learning: two unexpected allies?</a></p><blockquote><em>Differential privacy is a framework for evaluating the guarantees provided by a mechanism that was designed to protect privacy. Invented by Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam Smith [DMNS06], it addresses a lot of the limitations of previous approaches like k-anonymity. The basic idea is to </em><strong><em>randomize</em></strong><em> part of the mechanism’s behavior to provide privacy. In our case, the mechanism considered is always a learning algorithm, but the differential privacy framework can be applied to study any algorithm.</em></blockquote><blockquote><em>The intuition for introducing randomness to a learning algorithm is to make it hard to tell which behavioral aspects of the model defined by the learned parameters came from randomness and which came from the training </em><a href="https://becominghuman.ai/how-can-we-improve-the-quality-of-our-data-61b55c9063d1"><em>data</em></a><em>. Without randomness, we would be able to ask questions like: “What parameters does the learning algorithm choose when we train it on this specific dataset?” With randomness in the learning algorithm, we instead ask questions like: “What is the probability that the learning algorithm will choose parameters in this set of possible parameters, when we train it on this specific dataset?”</em></blockquote><blockquote><em>We use a version of differential privacy which requires that the probability of learning any particular set of parameters stays roughly the same if we change a single training example in the training set. This could mean to add a training example, remove a training example, or change the values within one training example. The intuition is that if a single patient (Jane Smith) does not affect the outcome of learning, then that </em><strong><em>patient’s records cannot be memorized and her privacy is respected.</em></strong><em> We often refer to this probability as the </em>privacy budget<em>. </em><strong><em>Smaller privacy budgets correspond to stronger privacy guarantees.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/0*wFX86gEvxIiU_eCG.png" /></figure><blockquote><em>In the above illustration, we achieve </em><strong><em>differential privacy when the adversary is not able to distinguish the answers </em></strong><em>produced by the randomized algorithm based on the data of two of the three users from the answers returned by the same algorithm based on the data of all three users.</em></blockquote><p>Few good reads to understand De-anonymization of datasets are as follows:</p><ul><li><a href="https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/">Why ‘Anonymous’ Data Sometimes Isn’t</a></li><li><a href="https://arxiv.org/pdf/cs/0610105.pdf">Robust De-anonymization of Large Datasets(How to Break Anonymity of the Netflix Prize Dataset)</a></li><li><a href="https://www.cs.princeton.edu/~arvindn/publications/de-anonymization-retrospective.pdf">Robust de-anonymization of large sparse datasets: a decade later</a></li><li><a href="https://www.andrew.cmu.edu/user/divyasha/dss-post12.pdf">Provable De-anonymization of Large Datasetswith Sparse Dimensions</a></li><li><a href="https://www.sciencedirect.com/science/article/pii/S0022000014000683">De-anonymization attack on geolocated data</a></li></ul><blockquote><strong><em>Always Remember — </em></strong><em>Differential privacy is </em><strong><em>not a property of databases, but a property of queries</em></strong><em>. The intuition behind differential privacy is that we bound how much the output can change if we change the data of a single individual in the database.</em></blockquote><h3>So, What DP ‘does’ promises?</h3><p>Differential privacy promises to protect individuals from any additional harm that they might face due to their <a href="https://becominghuman.ai/machine-learning-concepts-every-data-scientist-should-know-4d70bed18a8a">data</a> being in the private database <em>x</em> that they would not have faced had their data not been part of <em>x</em>.</p><p>By promising a guarantee of ε-differential privacy, a data analyst can promise an individual that his expected future utility <strong><em>will not be harmed by more than an exp(ε)≈(1+ε) factor.</em></strong> Note that this promise holds <em>independently</em> of the individual is utility function ui, and holds <em>simultaneously</em> for multiple individuals who may have completely different utility functions.</p><h3>What DP ‘does not’ promise?</h3><ul><li>Differential privacy <strong>does not guarantee</strong> that what one believes to be one’s <strong>secrets will remain secret.</strong> That is, it promises to make the data differentially private and not disclose it BUT not to protect it from attackers! <strong>Ex:</strong> Differential attack is one of the most common form of privacy attack.</li><li>It <strong>merely ensure</strong>s that one’s participation in a survey <strong>will not in itself be disclosed</strong>, nor will participation lead to disclosure of any specifics that one has contributed to the survey.</li></ul><p><strong>Do not confuse Security with Privacy</strong>, while <em>Security</em> controls <em>“WHO”</em> can access the data, <em>Privacy</em> is more about <em>“When” and “WHAT”</em> can they access .i.e, <strong><em>“You can’t have privacy without security, but you can have security without privacy.”</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*saotLXHdrppQlEf-U1yfFQ.png" /><figcaption>Security VS privacy</figcaption></figure><p><strong><em>For example</em></strong>, everyone is familiar with the term “Login Authentication and Authorization” right? Here, authentication is about “Who” can access the data and thus, can come under Security, whereas, authorization is all about “What”, “When” and How much of data is accessible to that specific user, thus, can come under Privacy.</p><p>That’s all folks! In the next post of this series we will look into the Types of Differential Privacy — Local VS Global DP, with some real world examples.</p><p>Till then you may also check out my other beginner friendly Series:</p><ul><li><a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea?source=---------9------------------"><strong><em>Introduction to “Tensors”</em></strong></a></li><li><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)</em></strong></a></li></ul><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*2lvCls4yjxVMfZSR" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fd7bf507049" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/what-is-differential-privacy-1fd7bf507049">What is Differential Privacy?</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to choose a Pre-Trained model!]]></title>
            <link>https://medium.com/@shaistha24/how-to-choose-a-pre-trained-model-afd3dbd75b45?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/afd3dbd75b45</guid>
            <category><![CDATA[openvino]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[model-selection]]></category>
            <category><![CDATA[intel]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Fri, 21 Feb 2020 22:59:41 GMT</pubDate>
            <atom:updated>2020-02-21T22:59:41.274Z</atom:updated>
            <content:encoded><![CDATA[<p>Choosing one of the Free Pre-Trained model’s from OpenVINO toolkit list for Edge Deployment. (<a href="https://docs.openvinotoolkit.org/latest/_models_intel_index.html">Link</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVAOuSmrd1VCKirVS6RzCg.jpeg" /><figcaption>Making the right choice!</figcaption></figure><p>This post is about “Choosing a (Free Version) of Pre-Trained model’s provided by OpenVINO Toolkit for learning or demo purposes.” Just so you know, OpenVINO Toolkit provides TWO types of Pre-trained models — <strong>Free Version </strong>(These can be deployed directed to edge with minimum model optimization steps, <em>No further tweaking or improvisation can be done!</em>) and the <strong>Public Version </strong>(<em>Further changes can be made for model improvisations</em>).</p><p>There are few questions you <strong><em>must</em></strong> <em>ask yourself</em> for the selection of good Pre-Trained model:</p><ul><li>What are the desired OUTPUTS?</li><li>What kind of INPUTS do you expect?</li><li>Does the Pre-Trained Model support such input requirements?</li><li>What is the model accuracy and other specifications?</li><li>Compare with other models available — choose the best fit!</li></ul><p>There might be a time when one model can serve the purpose outside of their desired or intended tasks as well. For example,</p><ul><li>Traffic Light Optimization Model — Detect People, Vehicles and bikes.</li><li>Asses Traffic Level in Retail Aisles Model— Pedestrian Detection.</li><li>Delivery Robot Model — Identify roadside objects.</li><li>Monitor Form When Working Out — Human Pose Estimation</li></ul><p>And many more, these are some of the Brainstorming applications you might think of which are outside of the model’s intended use-cases.</p><p>But, for the sake of simplicity, I will be choosing one of the models from the given list for “Person Detection” by answering the above questions.</p><p><strong><em>What are the desired OUTPUTS?</em></strong></p><p>For practicing OpenVINO Toolkit Pre-Trained Model deployment on edge, I hope to detect if a person is present in the given image or not? If a person is present, then the model should create a bounding box around them.</p><p><strong><em>What kind of INPUTS do you expect?</em></strong></p><p>Again for simplicity and practical knowledge, I would like to test it on “Mobile camera clicks”. So, the person may be sitting, standing or doing anything in the image.</p><p><strong><em>Does the Pre-Trained Model support such input requirements?</em></strong></p><p>As per my input image conditions i.e., Person Detection in as many positions as possible, such as standing, sitting, lying down,etc “<a href="https://docs.openvinotoolkit.org/latest/_models_intel_person_detection_action_recognition_0006_description_person_detection_action_recognition_0006.html"><strong>person-detection-action-recognition-0006</strong></a>” seems to be a good fit as it has :</p><ul><li>Pose coverage — sitting, writing, raising_hand, standing, turned around, lie on the desk.</li></ul><p><strong><em>What is the model accuracy and other specifications?</em></strong></p><ul><li>Detector AP (internal test set 2) — 90.70%</li><li>Accuracy (internal test set 2) — 80.74%</li></ul><p>Both the AP and the accuracy are pretty good for this model and should successfully give desirable results. Some of the other specifications which makes it a good choice are:</p><ul><li>Support of occluded pedestrians — YES</li><li>Occlusion coverage &lt;50%</li></ul><p><strong><em>Model Description:</em> </strong>Based on the RMNet backbone that includes <em>depth-wise convolutions </em>to <em>reduce the amount of computations</em> for the 3x3 convolution block. <em>The first SSD head from 1/8 and 1/16 scale feature maps has four clustered prior boxes and outputs detected persons (two class detector).</em> The second SSD-based head predicts actions of the detected persons.</p><p>In my case, this is the only model which fits to my desired requirements, but, if you have two or more models to choose from, the main focus should be the output, does it give the desired output for the given input types? If yes, then move on to model accuracy and so on.</p><h4>Other Brainstorming Questions you might ask yourself for better understanding or just for FUN!</h4><p><strong><em>Which model network architecture was used in your selected pre-trained model and what do you think could be the reason to choose that particular network architecture for the model?</em></strong></p><p>For example, in case of “<a href="https://docs.openvinotoolkit.org/latest/_models_intel_person_detection_action_recognition_0006_description_person_detection_action_recognition_0006.html"><strong>person-detection-action-recognition-0006</strong></a><strong>”, </strong>it includes RMNet backbone that includes <em>depth-wise convolutions </em>to <em>reduce the amount of computations</em> for the 3x3 convolution block.</p><blockquote>RMNet is custom backbone designed for the fast but still accurate inference. The RMNet architecture is inspired by ResNet and MobileNet architectures.</blockquote><p>For better understanding and use case of RMNet read this paper — <a href="https://arxiv.org/pdf/1812.02465.pdf">Fast and Accurate Person Re-Identificationwith RMNet</a></p><p><strong><em>Which other networks could be used for this exact purpose and why do you think so?</em></strong></p><p>SSD Mobilenet V1 COCO Model, Faster RCN Inception V2 COCO Model, Deep Convolution Neural Networks with AlexNet,etc.</p><p>You read this <a href="https://medium.com/@madhawavidanapathirana/real-time-human-detection-in-computer-vision-part-2-c7eda27115c6"><strong><em>post</em></strong></a> for better understanding!</p><p>That’s it! You have now successfully chosen a Pre-Trained model for practice and are ready to the next step of Pre-Processing and Model Optimization!</p><p>Happy Learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afd3dbd75b45" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Not too deep INTRO to Neural Network and Perceptron!]]></title>
            <link>https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-3-6db79028e56d?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/6db79028e56d</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[perceptron]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[secure-and-private-ai]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Wed, 11 Sep 2019 18:22:50 GMT</pubDate>
            <atom:updated>2020-09-12T21:43:03.627Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ew7zWBitKO4udoUaxnFYzw.png" /></figure><p><strong><em>Summary: This the third blog of the beginner friendly series — Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN) with a gentle introduction to neural network and perceptron. For more posts like these on Machine Learning Basics follow me here or on Twitter — </em></strong><a href="https://twitter.com/shaistha24"><strong><em>Shaistha Fathima.</em></strong></a></p><h3>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN) Series</h3><p>Hey everyone! Hope you have read the previous two posts of this series, if not, please have a look at it before going ahead. Here is the link to take a look at the previous two posts of this series <strong><em>(Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN))</em></strong><em> Topics</em> :</p><ul><li><strong><em>1-</em></strong><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em> Linear Regression — Ordinary Least Square Method</em></strong></a></li><li><strong><em>2- </em></strong><a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-2-a4105935201c"><strong><em>Ordinary least square method (P5.JS code) and Gradient Descent Intro</em></strong></a></li></ul><p>So… all this time I have been saying “Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN)”<em> </em>but <em>what does neural network mean and how is it made?</em></p><p>Before we move any further I would like you to have a basic understanding of what is a neural network ?, not too deep, just the intro…</p><h3>Not too deep INTRO to Neural Network!</h3><p>For this I will be answering 4 basic questions for you:</p><ul><li>What is neural network?</li><li>Where can we use it?</li><li>How is it made?</li><li>Why should we bother using it?</li></ul><h4>What is neural network?</h4><p>As defined in<a href="https://www.investopedia.com/terms/n/neuralnetwork.asp"> Investopedia</a> :</p><blockquote>A neural network is a series of algorithms that endeavors to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates. Neural networks can adapt to changing input; so the network generates the best possible result without needing to redesign the output criteria.</blockquote><p>To put it simply, Neural networks are a series of algorithms that mimic the operations of a human brain to recognize relationships between vast amounts of data.</p><p>“Neuron” is the fundamental unit of the neural network i.e., combination of neurons forms a neural network. Here, we are referring these neuron as “<strong><em>Perceptron</em></strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vn7VIPb_JpwEX7PAftjYsw.png" /><figcaption>Perceptron with basic example say detecting if the student has passed or failed after processing through their marks! (many inputs for one output!)</figcaption></figure><h4>Where can we use it?</h4><p>Neural networks are used in deep learning to model complex patterns and prediction problems. Simplest examples are handwriting letter prediction, object detection, etc.</p><blockquote>Neural networks are broadly used, with applications for financial operations, enterprise planning, trading, business analytics and product maintenance. Neural networks have also gained widespread adoption in business applications such as forecasting and marketing research solutions, fraud detection and <a href="https://www.investopedia.com/terms/r/risk-assessment.asp">risk assessment</a>.</blockquote><h4>How is it made?</h4><p>Neural networks are composed of layers of computational units called <em>neurons </em>(Perceptrons), with connections in different layers. These networks transform data until they can classify it as an output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*xYCDb7atOVEXG8OvtnYGmQ.jpeg" /><figcaption>This example uses 2 layers to process the data and get the desired output.</figcaption></figure><p><strong><em>Each balls in the hidden layer represents a perceptron!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*WNxN2ArLaGt0-Rm3tzWw1g.jpeg" /><figcaption>Each neuron multiplies an initial value by some weight, sums results with other values coming into the same neuron, adjusts the resulting number by the neuron’s bias, and then normalizes the output with an activation function.</figcaption></figure><p>For now you may just remember that Neural Networks are the means of doing machine learning, in which a computer learns to perform some task by analyzing training examples. Usually, the examples have been<em> labeled in advance</em>. An object recognition system, for instance, might be fed thousands of labeled images of cars, houses, coffee cups, and so on, and it would find visual patterns in the images that consistently correlate with particular labels.</p><p>So a key feature is an <em>iterative learning process</em> in which records (rows) are presented to the network one at a time, and the weights associated with the input values are adjusted each time. After all, cases are presented, the process is often repeated. During this learning phase, the network trains by adjusting the weights to predict the correct <em>class label </em>of input samples.</p><h4>Why should we bother using it?</h4><p>With the amount of computational power needed by the company almost doubled every 2 years, many companies now use powerful hardware&#39;s to perform tasks with large data-sets in just a little amount of time using deep learning! <em>Nvidia is the market leader when it comes to GPUs and now Google has come into this space as well, releasing TPUs(specialized </em><a href="https://en.wikipedia.org/wiki/Application-specific_integrated_circuit"><em>ASICs</em></a><em> for Deep Learning) in recent times, making it easier to train Deep Neural networks, requiring so much computational power.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-nBN_9akb1_O29cgKrcFA.jpeg" /></figure><p>We use a Neural network especially when we have <strong><em>a lot of data to process </em></strong>requiring high computational power and when <strong><em>accuracy matters the most to you</em></strong>.<strong> </strong>For Example, <strong>Cancer Detection</strong>. You cannot mess around with accuracy here if you want this to be used in actual medical applications.</p><p>The main advantage of Neural Network lies in their ability to outperform nearly every traditional Machine Learning algorithms, but just like everything else, they too have their disadvantages. Read this post by Rahul Bhatia on <a href="https://medium.com/datadriveninvestor/when-not-to-use-neural-networks-89fb50622429"><em>When not to use Neural Networks</em></a>.</p><p><strong><em>The only think to keep in mind while using a neural network is, the more data you give it to process the better the accuracy and result will be!</em></strong></p><h3>Perceptron</h3><p>Like I have said before, perceptrons are what we call the <em>neurons</em> of the neural networks, meaning they are like the building blocks of the neural networks! Sometimes, also referred to as the Simplest form of Neural Network or Single Neural Network.</p><p>A perceptron takes several binary inputs, <em>x</em>1,<em>x</em>2,x3,…, and produces a single binary output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/0*q5EJX8hUxp1QVTxi.png" /><figcaption>In this example, the perceptron has three inputs, <em>x</em>1,<em>x</em>2,<em>x</em>3. In general it could have more or fewer inputs.</figcaption></figure><p>From the below image of basic perceptron structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vn7VIPb_JpwEX7PAftjYsw.png" /></figure><p>Perceptron is a processing unit taking in input values and giving out the output after processing it. It helps in decision boundary making and finding the line! (from the previous example in post 2 on number of students pass based on gradient descent!)And yes, <em>perceptron also uses the concept of gradient descent!</em></p><p>In this context, the perceptron follows these steps:</p><ol><li>Multiply all the inputs by their weights <strong><em>w</em></strong>,<strong><em> </em></strong>real numbers that express how important the corresponding inputs are to the output.</li><li>Add them together referred as <strong><em>weighted sum: ∑ wj xj. </em></strong><em>( W.x in the below equation)</em></li><li>Apply the <strong><em>activation function</em></strong>, in other words, determine whether the weighted sum is greater than a <em>threshold value, </em>where -threshold is equivalent to <em>bias, </em>and assign 1<em> </em>or 0 as an output<em>.</em></li></ol><p>We can also write the perceptron function in the following terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*z8PbTkvChJLwjoPQIssv3w.png" /><figcaption>Notes: b is the bias and is equivalent to -threshold, w.x is the dot product of w, a vector which component is the weights, and x, a vector consisting of the inputs.</figcaption></figure><p>From the above equation and process you might have a few questions, lets answer each of them one by one…</p><p><strong><em>Why should you find weighted sum?</em></strong></p><p>As said above one perceptron is used to find out one definite result, though it may take in many input values in the end, it just processed those values and finds out the relevant output. To help get a result, here a Boolean value of 1 or 0, summation of all the dot products of weights and inputs was introduced. You will understand it better with the later example.</p><p><strong><em>What is an activation function and why must we use it?</em></strong></p><p><strong><em>Activation Function </em></strong><em>decides, whether a neuron should be activated or not by calculating</em><strong><em> weighted sum</em></strong><em> and further </em><strong><em>adding bias</em></strong><em> with it. The purpose of the activation function is to </em><strong><em>introduce non-linearity</em></strong><em> into the output of a neuron.</em></p><p>In a neural network, we would update the weights and biases of the neurons on the basis of the error at the output. This process is known as <em>back-propagation</em>. Activation functions make the back-propagation possible since the gradients are supplied along with the error to update the weights and biases.</p><p><strong><em>Why do we need Non-linear activation functions ?</em></strong><br> A neural network without an activation function is essentially just a linear regression model. The activation function does the non-linear transformation to the input making it capable to learn and perform more complex tasks.</p><p>There are many types of activation functions which we will discuss in the next post, but, if you are interested you might check these links here:</p><ul><li><a href="https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0">Understanding Activation Functions in Neural Networks</a></li><li><a href="https://towardsdatascience.com/activation-functions-and-its-types-which-is-better-a9a5310cc8f">Activation functions and it’s types-Which is better?</a></li></ul><h3>Example to understand the perceptron function:</h3><p>One of the examples quoted from <a href="http://neuralnetworksanddeeplearning.com/index.html">Neural Networks and Deep Learning</a></p><blockquote>Let me give an example. It’s not a very realistic example, but it’s easy to understand, and we’ll soon get to more realistic examples. Suppose the weekend is coming up, and you’ve heard that there’s going to be a cheese festival in your city. You like cheese, and are trying to decide whether or not to go to the festival. You might make your decision by weighing up three factors:</blockquote><blockquote>Is the weather good?</blockquote><blockquote>Does your boyfriend or girlfriend want to accompany you?</blockquote><blockquote>Is the festival near public transit? (You don’t own a car).</blockquote><blockquote>We can represent these three factors by corresponding binary variables <em>x</em>1,<em>x</em>2, and <em>x</em>3. For instance, we’d have <em>x</em>1=1 if the weather is good, and <em>x</em>1=0 if the weather is bad. Similarly, <em>x</em>2=1 if your boyfriend or girlfriend wants to go, and <em>x</em>2=0 if not. And similarly again for <em>x</em>3 and public transit.</blockquote><blockquote>Now, suppose you absolutely adore cheese, so much so that you’re happy to go to the festival even if your boyfriend or girlfriend is uninterested and the festival is hard to get to. But perhaps you really loathe bad weather, and there’s no way you’d go to the festival if the weather is bad. You can use perceptrons to model this kind of decision-making.</blockquote><blockquote>One way to do this is to choose a weight <em>w</em>1=6 for the weather, and <em>w</em>2=2 and <em>w</em>3=2 for the other conditions. The larger value of <em>w</em>1 indicates that the weather matters a lot to you, much more than whether your boyfriend or girlfriend joins you, or the nearness of public transit. Finally, suppose you choose a threshold of 5 for the perceptron. With these choices, the perceptron implements the desired decision-making model, outputting 1 whenever the weather is good, and 0 whenever the weather is bad. It makes no difference to the output whether your boyfriend or girlfriend wants to go, or whether public transit is nearby.</blockquote><blockquote><strong>By varying the weights and the threshold, we can get different models of decision-making</strong>. For example, suppose we instead chose a threshold of 3. Then the perceptron would decide that you should go to the festival whenever the weather was good <em>or</em> when both the festival was near public transit <em>and</em> your boyfriend or girlfriend was willing to join you. In other words, it’d be a different model of decision-making. Dropping the threshold means you’re more willing to go to the festival.</blockquote><blockquote>Obviously, the perceptron isn’t a complete model of human decision-making! But what the example illustrates is how a perceptron can weigh up different kinds of evidence in order to make decisions. And it should seem plausible that a complex network of perceptrons could make quite subtle decisions:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*Ky62FHspAUVYQCR0.png" /></figure><blockquote>In this network, the first column of perceptrons — what we’ll call the first <em>layer</em> of perceptrons — is making three very simple decisions, by weighing the input evidence. What about the perceptrons in the second layer? Each of those perceptrons is making a decision by weighing up the results from the first layer of decision-making. In this way a perceptron in the second layer can make a decision at a more complex and more abstract level than perceptrons in the first layer. And even more complex decisions can be made by the perceptron in the third layer. In this way, a many-layer network of perceptrons can engage in sophisticated decision making.</blockquote><h3>Conclusion</h3><p>Hope with the above example you could understand the importance and use of weights and bias. In the next post we will dig deeper into perceptron and try to understand the types of activation functions! You are always free to ping me here or reach out at <a href="https://twitter.com/shaistha24">twitter</a> for any queries.</p><p><strong>Resources you might want to refer to:</strong></p><ul><li><a href="https://towardsdatascience.com/understanding-neural-networks-19020b758230"><em>Understanding Neural Networks</em></a></li><li><a href="http://Neural Networks and Deep Learning is a free online book"><em>Neural Networks and Deep Learning , a free online book</em></a></li><li><a href="https://blog.statsbot.co/neural-networks-for-beginners-d99f2235efca"><em>Neural Networks for Beginners: Popular Types and Applications</em></a></li><li><a href="https://towardsdatascience.com/understanding-neural-networks-what-how-and-why-18ec703ebd31"><em>Understanding Neural Networks: What, How and Why?</em></a></li><li><a href="https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53"><em>What the Hell is Perceptron?</em></a></li></ul><p>You may also check my other series on <a href="https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53"><strong><em>Introduction to “Tensors”</em></strong></a> if interested!</p><p>Till then stay tuned and happy learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6db79028e56d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ordinary least square method (P5.JS code) and Intro to Gradient Descent]]></title>
            <link>https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-2-a4105935201c?source=rss-ead8136ed0bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4105935201c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[shaistha fathima]]></dc:creator>
            <pubDate>Thu, 29 Aug 2019 18:23:30 GMT</pubDate>
            <atom:updated>2020-09-12T21:25:10.826Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qwKMVNbU-gQIW4eIHDlzSA.png" /></figure><p><strong><em>Summary: I learn best by playing around with code and believe in sharing knowledge. This the second blog of the new beginner friendly series — Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN) with a code example for Ordinary Least Square Method and introduction to Gradient Descent. For more posts like these on Machine Learning Basics follow me here or on Twitter — </em></strong><a href="https://twitter.com/shaistha24"><strong><em>Shaistha Fathima.</em></strong></a></p><h3>Basic Concepts You Should Know Before Starting with the “Neural Networks” (NN) Series</h3><p>Hello everyone! Hope you have already read the first post of this series! If you have not read it yet, you could just read it <a href="https://medium.com/@shaistha24/basic-concepts-you-should-know-before-starting-with-the-neural-networks-nn-1-e317ab6e2b28"><strong><em>here</em></strong></a>, and come back! I strongly suggest you to have a look at it.</p><p>By now you know the working of Ordinary Least Square Method! To put it simply, a line is created and made to move to the position most suitable for dividing max of the red points from the blue points like a partition, and help predict the position for the next point! And, don’t forget it <em>“works on Linear Data Only”.</em></p><p>I think its better to show you, after all practical experience is the best experience!</p><p><strong><em>Lets begin…</em></strong></p><h4>Linear Regression Example:</h4><p>I am going to use p5.js to show how line moves as per the points. <strong><em>p5.js</em></strong><em> is a JavaScript library used for creative coding. It is based on processing which is a creative coding environment. The main focus of processing is to make it possible for beginners to learn how to program interactive, graphical applications, to make a programming language more user-friendly by visualizing it. The interesting part is that processing is not only for developers but also artists, designers, researchers and those who want to enjoy making arts.</em></p><blockquote>Note: To know more about p5.js you may refer to the<a href="https://p5js.org/"> <strong>official website</strong></a><strong> </strong>or this medium post by <a href="https://medium.com/@mhiratsuka?source=post_page-----b88f9fee0ab2----------------------">mhiratsuka</a> on <a href="https://medium.com/@mhiratsuka/what-is-p5-js-and-how-to-use-it-b88f9fee0ab2"><strong>What is p5.js and How to use it?</strong></a></blockquote><ol><li>Setup: Creating canvas and giving it a background</li></ol><pre>//var data, list with all points<br>var data = []</pre><pre>// setup<br>function setup() {<br>  createCanvas(400, 400); //width and height = 400 <br>  background(51);<br>}</pre><p>2. Create a point and store the data</p><pre>function mousePressed(){<br>// map is used to grab the location of the mouse click wrt x and y<br>// map(value, start1, stop1, start2, stop2)</pre><pre>  var x = map (mouseX, 0, width, 0, 1);<br>  var y = map (mouseY, 0, height, 1, 0);</pre><pre>// creating point on click<br>  var point = createVector(x, y);</pre><pre>// store in data<br>  data.push(point);<br>}</pre><p>3. Display the point</p><pre>function draw() {<br>  background(51);</pre><pre>//get the point and draw it<br>  for(var i =0; i&lt;data.length; i++){<br>    var x = map(data[i].x, 0, 1, 0, width);<br>    var y = map(data[i].y, 0,1,height, 0);<br>    fill(255);  // white color<br>    stroke(255);  <br>    ellipse(x,y,8,8);  // circle<br>  }<br>  </pre><pre>// drawing the line wrt to the points<br>  if(data.length &gt; 1) {<br>    linearRegression();<br>    drawLine();<br>  }<br>}</pre><p>4. Draw Line</p><pre>var m = 1;<br>var b = 0;</pre><pre>function drawLine(){<br>  var x1 = 0;<br>  var y1 = m * x1 + b;  // eq for straight line<br>  var x2 = 1;<br>  var y2 = m* x2 + b;<br>  <br>  <br>  x1 = map(x1, 0, 1,0, width);      // map the points<br>  y1 = map(y1, 0, 1, height, 0);<br>  x2 = map(x2, 0, 1,0, width);<br>  y2 = map(y2, 0, 1, height, 0);<br>  <br>  stroke(255, 0, 255);  //color <br>  line(x1, y1, x2, y2); //create a line<br>}</pre><p>5. Linear Regression</p><pre>// calculating the slope(m) and Y-intercept(b) </pre><pre>function linearRegression(){<br>  var xsum = 0;<br>  var ysum = 0;<br>  <br>  for( var i=0; i&lt;data.length; i++){<br>    xsum += data[i].x;<br>    ysum += data[i].y;<br>  }<br>  <br>//when too many points are in use, mean value is taken</pre><pre>var xmean = xsum/data.length;     <br>  var ymean = ysum/data.length;    <br>  <br>  var num = 0;<br>  var den = 0;<br>  <br>  for(var i = 0; i&lt;data.length; i++){<br>    var x = data[i].x;<br>    var y = data[i].y;<br>    num += (x-xmean) * (y-ymean);<br>    den += (x-xmean) * (x-xmean);<br>  }</pre><pre>/* slope = summation of(x-xmean) * (y-ymean) by summation of<br> (x-xmean) * (x-xmean) */</pre><pre>  m = num/den; <br>  b = ymean-m*xmean;<br>}</pre><h4><strong>Result:</strong></h4><blockquote>You may run this code <a href="https://editor.p5js.org/shaistha24/sketches/Rq6ZcVKH3"><strong>here</strong></a> on the p5.js code editor and experience it yourself</blockquote><p>Initially with just two points,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*pZzz4l3mMZ8iobR3Qbc7SA.png" /></figure><p>when more points are added,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*JxUkvIa3b0RKkKaO41xNzg.png" /><figcaption>Moves closer to the dense points !</figcaption></figure><p><strong><em>Quick Recap:</em></strong></p><p>Now you know that, <strong><em>Linear regression</em></strong> is used to find the relationship between two variables by fitting a line (linear equation) to observed data. Where, one is predictor or independent variable and other is response or dependent variable</p><p><strong><em>Example:</em></strong> a modeler might want to relate the weights of individuals to their heights using a linear regression model.</p><p>We also know that, the simplest method for fitting a regression line is <strong><em>ordinary least-squares method</em></strong>, which, finds the best-fitting line for the observed data by minimizing the sum of the squares of the vertical deviations (distance of point from the line, d1, d2,.. remember!) from each data point to the line (if a point lies on the fitted line exactly, then its vertical deviation is 0). Because the deviations are first squared,and then summed, there are no cancellations between positive and negative values.</p><h3>Gradient Descent</h3><p>If I were to define it, then, <em>Gradient descent is an optimization algorithm used to find the values of parameters (coefficients) of a function (f) that minimizes a cost function (cost).</em></p><p><strong><em>When do we use it?</em></strong></p><p>Gradient descent is best used when the parameters cannot be calculated analytically (e.g. using linear algebra) and must be searched for by an optimization algorithm. When its more complex than a simple 2D data we have discussed above.</p><h4>What is a cost function?</h4><p>This will get a bit technical, so please bear with me.</p><p>To put it simply, a <strong><em>cost function is a measure of “how good” a neural network did with respect to it’s given training sample and the expected output.</em></strong></p><p><strong><em>Example: </em></strong>In A Machine Learning Model</p><ul><li>Consider a bunch of data points in a 2 D space. Assume that the data is related to the height and weight of a group of students. We are trying to predict some kind of relationship between these quantities so that we could predict the weight of some new students afterwards. This is essentially a simple example of a supervised Machine Learning technique.</li><li>Let us now draw an arbitrary line in space that passes through some of these data points. The equation of this straight line would be <strong>Y = mX + b</strong> where <strong>m</strong> is the slope and <strong>b</strong> is its intercept on the Y-axis.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxgow_gPdDI2cp6Ys4TpmA.png" /></figure><h4>Predictions</h4><p>Given a known set of inputs and their corresponding outputs, A machine learning model tries to make some predictions for a new set of inputs.( just like predicting if Student 3 will be accepted or not!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*e9ts1gijLPxSHY6wF28MGQ.png" /></figure><p>The Error would be the difference between the two predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/0*hXw-c2gGTItTJsBA.png" /></figure><p>This relates to the idea of a <strong>Cost function or Loss function.</strong></p><h4>Cost Function</h4><p>A <strong><em>Cost Function/Loss Function</em></strong> evaluates the performance of our Machine Learning Algorithm. The<strong> Loss function</strong> computes the <strong><em>error for a single </em></strong>training example while the <strong>Cost function</strong> is the <strong><em>average of the loss functions</em></strong> for all the training examples. Henceforth, I shall be using both the terms interchangeably.</p><blockquote><em>A Cost function basically tells us ‘ how good ’ our model is at making predictions for a given value of m and b.</em></blockquote><p>Let’s say, there are a total of “N’ points in the dataset and for all those ‘N’ data points we want to minimize the error. So the Cost function would be the total squared error i.e</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/0*cxRHLqTr062MCD4v.png" /></figure><p>Cost function for N data points</p><blockquote><strong>Why do we take the squared differences and simply not the absolute differences?</strong> Because the squared differences make it easier to derive a regression line. To find that line we need to compute the first derivative of the Cost function, and it is much harder to compute the derivative of absolute values than squared values. Also, the squared differences increase the error distance, thus, making the bad predictions more pronounced than the good ones.</blockquote><p>A <strong>derivative</strong> is a term that comes from calculus and is calculated as the slope of the graph at a particular point.</p><h4>Minimizing the Cost Function</h4><blockquote><em>The goal of any Machine Learning Algorithm is to minimize the Cost Function.</em></blockquote><p>Why do you ask? That’s because a <strong><em>lower error between the actual and the predicted values signifies that the algorithm has done a good job in learning</em></strong>. Since we want the lowest error value, we want those‘ <strong>m’</strong> and ‘<strong>b’</strong> values which give the smallest possible error.</p><p>Gradient Descent Algorithm helps us to make these decisions efficiently and effectively with the use of derivatives.</p><p>In the equation, y = mX+b, ‘m’ and ‘b’ are its parameters. During the training process, there will be a small change in their values. Let that small change be denoted by δ<strong>. </strong>The value of parameters will be updated as <strong>m=m-δm</strong><strong> </strong>and<strong> b=b-δb </strong>respectively. Our aim here is to find those values of m and b in y = mx+b , for which the error is minimum i.e values which minimize the cost function.</p><p>Rewriting the cost function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*ZLM77ci6Y4bpvWmq.png" /></figure><blockquote><em>The idea is that by being able to compute the derivative/slope of the function, we can find the minimum of a function.</em></blockquote><h3><strong>Gradient descent with simple example</strong></h3><p>Imagine you’re standing somewhere on a mountain. You want to get as low as possible and as fast as possible, so you decide to follow these steps:</p><ul><li>You check your current altitude, your altitude a step north, a step south, a step east, and a step west. Using this, you figure out which direction you should step to reduce your altitude as much as possible in this step.</li><li>Repeat until stepping in any direction will cause you to go up again.</li></ul><p>Now, this is a wonderfully simple algorithm. You just have to figure out which direction the mountain is sloping where you are (this is the gradient) and take a step (this is your descent). This algorithm is great because it will quickly get you to a low point.</p><p>From our strategy, we would always have to walk in the direction that points the most downhill, and keep walking until it’s impossible to walk downhill any further.It’s not a bad strategy — it should work pretty well most of the time.</p><p><strong><em>However, it has the weakness that it does not guarantee that it will find the lowest point out of the entire search area.</em></strong></p><p><strong><em>Example: </em></strong>The <em>main downside to this strategy is that you might end up in a small hole, where if you could only walk uphill just a little bit, you could reach a much lower point.</em></p><p><strong><em>Example: </em></strong>If you found a mountain lake, then you could cause your gradient descent to stop while you’re still high on the mountain. There are much lower areas that you could get to in relatively few steps, but your method will not lead you there. (There are extra things you can do that modify a gradient descent to make it less likely to get stuck in a small local minimum).</p><p><strong>So, why or when do we use it?…</strong></p><p>This algorithm is great for anything that requires finding a local minimum, especially if it is really difficult to find the solution analytically (i.e. by solving equations for exact solutions; for example, by taking derivatives and setting them equal to zero).</p><p><strong><em>Here is one real world example I could find online!</em></strong></p><blockquote>“<strong>Inverse kinematics</strong>,” which is the process by which you figure out what angle each of the joints on an arm need to be in order to get the hand to wind up at the right place (sorry if this is too complex of an example; I’m trying to make it as layman friendly as possible). It turns out that it’s quite easy to take the angles of the joints and figure out where that will put the hand (this is “forward kinematics”), but it’s fairly challenging to do the reverse. This is an important thing to be able to do when working with robotics.</blockquote><blockquote>One of the standard solutions here is to use a gradient descent. You can “Check your current altitude” by using forward kinematics to find the hand’s position for the current joint angles. You can then “take a step north/south, etc” by adding or subtracting a little bit to each of the joint’s angles and recalculating the position. If the position is closer to your goal then that is “downhill;” farther is “uphill.” From there you increment the joints in the direction that takes the hand the biggest step towards the goal and you repeat until your hand gets where it needs to go. The cool thing about this approach is that it will make the hand take a nice straight path through space even if this means that the joints are doing weird things to get it there.</blockquote><h3>Conclusion</h3><p>In the next post of this series, we will look into what a neural network is by answering a few simple questions, just an intro not too deep and learn about the building blocks of neural networks called Perceptron. As always you can always ping me for any doubts or reach out to me on <a href="https://twitter.com/shaistha24"><em>twitter</em></a>.</p><p>You may also take a look at my other series on “<a href="https://medium.com/secure-and-private-ai-writing-challenge/introduction-to-tensors-1-de7dded35fea"><strong><em>Introduction to Tensors</em></strong></a>” if interested!</p><p>Till then, stay tuned and happy learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4105935201c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>