<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Patricia Thaine on Medium]]></title>
        <description><![CDATA[Stories by Patricia Thaine on Medium]]></description>
        <link>https://medium.com/@patriciathaine?source=rss-afe93926edf5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*vRWy-WxPKJ3sbWAI.</url>
            <title>Stories by Patricia Thaine on Medium</title>
            <link>https://medium.com/@patriciathaine?source=rss-afe93926edf5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:12:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@patriciathaine/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Talking with Dr. Ann Cavoukian, Privacy by Design inventor]]></title>
            <link>https://medium.com/@patriciathaine/talking-with-dr-ann-cavoukian-privacy-by-design-inventor-6943c01001af?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6943c01001af</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[data-protection]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[privacy-by-design]]></category>
            <category><![CDATA[anonymization]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 16:02:07 GMT</pubDate>
            <atom:updated>2021-09-06T16:02:07.587Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discussing developer responsibility, Bill C-11, positive consent, and the importance of Privacy by Design</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H9sB1bvVjWQ6ypA0FDyzKQ.jpeg" /><figcaption>Source: <a href="https://unsplash.com/photos/95YRwf6CNw8">https://unsplash.com/photos/95YRwf6CNw8</a></figcaption></figure><h3>Key Points:</h3><ul><li>Privacy by Design is such an revolutionary concept because it requires <em>proactive</em> action towards protecting user privacy, rather than action after a company gets slapped on the wrist by a lawsuit.</li><li>Positive consent for every purpose of data collection is crucial in Privacy by Design, with the second principle being Privacy by Default. Privacy by Design is required to reach GDPR compliance and it is win-win: it builds trust with users like nothing else and it can increase the quality of the data collected.</li><li>Creating a data map for your organization is the first step towards understanding what you need to get consent for and how to best protect their privacy.</li><li>There is an overwhelming demand for privacy coming from consumers.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FX6sqgYKGd9g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DX6sqgYKGd9g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FX6sqgYKGd9g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ba7b889a045acb011df8cff5182970c/href">https://medium.com/media/5ba7b889a045acb011df8cff5182970c/href</a></iframe><h3>Transcript:</h3><h4>Bio</h4><p><strong>Patricia:</strong> I’m really really excited to have Ann Cavoukian with us today, who is the three-term Privacy Commissioner of Ontario previously and the investor of Privacy by Design. She also received so many awards, including in leadership for privacy. She was named one of the top 25 women of influence in Canada. Named among the top 10 women in data security and privacy, and also named as one of the Power 50 by Canadian Business, and also the top 100 leaders in identity, and recently she was also awarded the Meritorious Service medal by the Governor General of Canada. So it is such an honour and pleasure to have our advisor and friend here today. Ann, thank you for joining us.</p><p><strong>Ann:</strong> My pleasure. It’s always a pleasure working with you, Patricia. So it’s truly my pleasure.</p><p><strong>Patricia:</strong> Really excited to dive right in and ask you as many questions as we have time for. There will be a Q&amp;A at the end, so please put in your questions in the chat.</p><h4>What is Privacy by Design and how can organizations embed it into their workflows?</h4><p><strong>Patricia:</strong> The first question for you, Ann, is: could you tell us what is Privacy by Design and how can organizations embed it into their workflows?</p><p><strong>Ann:</strong> Privacy by Design is all about being proactive. It’s a model of prevention, much like a medical model of prevention. You want to prevent the privacy harms from arising. See, when I was first appointed as Privacy Commissioner, this was a new concept, because lawyers — and of course, I’m not a lawyer, I’m a psychologist — but lawyers traditionally apply the law once there has been a problem and there’s a data breach or a privacy infraction, you apply the law and get a wonderful remedy. That’s invaluable. But I wanted more than that. I wanted to prevent the privacy harms from arising. So I wanted a model that could be embedded into one’s operations, into the design, and bake it into the code. So I developed <a href="https://www.ipc.on.ca/wp-content/uploads/resources/7foundationalprinciples.pdf">Privacy by Design</a> literally at my kitchen table over three nights and then I took it to work and tried to convince the lawyers, which I did, as to how it could complement regulatory compliance, which is after-the-fact. And so I always encourage companies — startups especially, because it’s so easy just starting up — to get this going. You want to ensure that privacy-protective measures are present throughout your entire operation from the get go. Right from the beginning, when you first start data collection. The primary purpose of the data collection. Do you have positive consent? And then do you restrict your use of the data to that purpose? Later on, if you need additional consent, you can go back if you need additional consent for secondary use. But again, you need <a href="https://iapp.org/resources/article/consent-2/">positive consent</a>. This goes such a long way to building <em>trust</em>. That’s what I’ve been told by companies who have been certified for Privacy by Design. It builds trust like no other. And right now, there’s such a trust deficit. This builds enormous trusted business relationships.</p><blockquote><strong>I wanted a model that could be embedded into one’s operations, into the design, and bake it into the code. So I developed Privacy by Design literally at my kitchen table over three nights and then I took it to work and tried to convince the lawyers.</strong></blockquote><p><strong>Patricia:</strong> Absolutely love that. And I think we’ve observed that a lot in the new businesses that are popping up and really putting privacy at the forefront.</p><h4>What do people often get wrong when integrating privacy into their workflows?</h4><p><strong>Patricia:</strong> A really important question is: what have you found that people get <em>wrong</em> when they’re integrating privacy into their workflows?</p><p><strong>Ann:</strong> You know, they think about it certainly at the beginning, usually. So they start from a good place, but one thing they don’t seem to grasp is that you don’t just get consent for the initial, primary purpose of the data collection and then use the information in a dozen different ways that flow throughout your organization. This is the problem. You need to get consent for every particular use of the data that differentiates itself from the primary purpose of the data collection. So I would say to companies “do you have a data map?” and they always look at me and say “huh?” and I would say “that’s the most valuable thing you can do right from the beginning, because then that maps all of the data throughout your organization and will alert you if you need to obtain additional consent for subsequent purposes that the data are used for that are different from the initial primary purpose of the data collection.” That’s absolutely critical. So if you have a data map, it’s an easy way to map the flow of the personal information throughout your organization and determine if additional consent is required, and then you can obtain it.</p><p><strong>Patricia:</strong> That’s very valuable advice. How does an organization not become chaotic without a data map like that?</p><p><strong>Ann:</strong> Good question.</p><h4>There’s a lot of confusion around the word anonymization. What are some dangers of misusing privacy enhancing technologies?</h4><p><strong>Patricia:</strong> We’ve noticed that there’s a lot of confusion around the world around anonymization, right? And we’ve seen it being misused in so many different contexts, in so many different newspaper articles. Can you tell us what some dangers are of misusing privacy enhancing technologies?</p><p><strong>Ann:</strong> The goal is anonymization of data, so that you can use the data for a variety of other purposes and obtain value from it without jeopardizing the privacy of the individuals to whom the data relates. But I always say: look, don’t strive for perfection. Just do your best to move forward and try to achieve anonymization. And the reason I say that is when we talk about de-identifying data, you can’t just strip a name or a number, you have to use strong de-identification protocols combined with a risk of re-identification framework that can minimize the risk or re-identification to less that .05%, which is less that the odds of being hit by lightning if you go outside when it’s raining out. You know, damn good odds. But trying to get perfection, I would avoid that. Try to minimize the likelihood of re-identification, and then you effectively have de-identified data which you can treat as anonymized data. And you can use that data for a variety of other purposes in terms of data utility, which can have enormous value to a company without risking the privacy of the individuals involved, because you’ve gone through great lengths to strongly de-identify, anonymize, the data. That makes it a win-win. Privacy by Design is all about win-win. Let’s get rid of the zero-sum mindset of either-or, win-lose, privacy vs. security, or privacy vs. data utility. That’s nonsense. You need both. And it’s never privacy that wins over the other interests, nor should it be, but I sure as heck am not going to have it lose out all the time. So if you make a strong effort to strongly de-identify the data, reaching the goal of anonymization, then you free yourself and you free the data for enormous data utility that can arise afterward.</p><blockquote><strong><em>[I]f you make a strong effort to strongly de-identify the data, reaching the goal of anonymization, then you free yourself and you free the data for enormous data utility that can arise afterward.</em></strong></blockquote><h4>What are some practical applications for de-identification?</h4><p><strong>Patricia: </strong>That leads perfectly into what we want to ask you next, which is: what are some practical applications for this de-identified data, because there is that extra misconception that, when you have anonymous data, it’s no longer useful for anything.</p><p><strong>Ann:</strong> That’s nonsense. Let me talk about smart cities. And you may know that I was retained by Sidewalk Labs when they had the contract to develop the smart city here in Toronto, which is where I live. So I wanted a smart city of privacy, not a smart city of surveillance, and at the time, when they retained me, that’s what they wanted too. They said they wanted me to embed Privacy by Design into the smart city that they envisioned. And after I studied it considerably, I said what we have to do is we have to de-identify data at source, meaning right at the time it’s collected. Because, if you do that, then you strip the personal identifiers right at the time the data are collected. And the reason you need to do that in a smart city is because the tech is <em>everywhere</em>. You can’t just decide here or there. No, it’s everywhere. You risk privacy everywhere. So, you make a policy: I’m going to strip all the data personal identifiers right at the time when it’s collected. De-identify data at source. Then you have massive value from the data you’ve collected for a variety of smart city applications, involving transit, involving the flur of people on highways, and there’s so much you can do with the data. And Sidewalk labs agreed to that from the outset, but then they reneged later on. And when they reneged, I resigned the following morning. Now the good news is, lots of other people have approached me. So I’m not working with a wonderful group called Innovate Cities, and what they’re doing is trying to develop a framework for how you build smart cities in a way that protects privacy. And the way they got me to work with them is they said: we are <em>insisting</em> on Privacy by Design right from the outset. Always de-identify data at source — we heard you loud and clear, that’s what we want. And I said: “Are you sure? Because look, I always look under the hood. You can’t just say that to me. Walk the talk.” And they have assured me that again and again. So I’m delighted. And then another group, I just got contacted from San Francisco and a company, which I can’t name at the moment, in Michigan, and they want to develop a small Smart City application in the Detroit area to lift it up. So they approached me and they want Privacy by Design embedded. Anyways, Smart Cities is one example, because you’re going to have so much collection of data which is very important. You must strip it of privacy risks first.</p><p><strong>Patricia:</strong> That’s a great example, and I’m glad that we’ll have very obvious examples, more and more coming, of de-identified data being super useful to our day-to-day lives.</p><h4>What responsibility do developers have when it comes to privacy? And how does that compare to the rest of an organization?</h4><p><strong>Patricia:</strong> One huge question, because developers are just getting familiarized with Privacy by Design, privacy enhancing technologies is still not a well explained field in universities, and we’re just starting to get that little bit of ramp up in the education developers need in order to be able to integrate this into their own projects. So what do you think, for now, are the responsibilities of developers when it comes to privacy and how does that compare with the responsibilities of the rest of the organization?</p><p><strong>Ann:</strong> It all flows together. If you don’t address privacy issues as a developer, you will fail. And I don’t mean to be extreme here, but if you have personally identifiable data that you are using throughout whatever it is you’re doing in your organization and you don’t address privacy concerns, you will invariably have a data breach or a privacy infraction. And these days, the concern for privacy is enormous. I’ve been in this business well over 20 years. I’ve never seen it — in all kinds of privacy opinion polls, research polls, privacy has just been ranked at the 90th percentile consistently. 90% of those interviewed are concerned about their privacy. 92% are concerned about loss of control over their information. And privacy is all about control: personal control over the use and disclosure of your information. And I point companies who are considering adopting Privacy by Design to what happens if you don’t do it. You will have a data breach or privacy infraction. And these days they are no longer lawsuits, they are class action lawsuits that can cost companies in the millions of dollars. I’m not exaggerating. Target, about 4–5 years ago <a href="https://www.usatoday.com/story/money/2017/05/23/target-pay-185m-2013-data-breach-affected-consumers/102063932/">had a massive data breach</a> in the United States. The CEO of Target resigned. Target had just opened up several stores in Canada for the first time, a few years ago. And we were very excited to have them here. But once there was this massive data breach, as I said, the CEO resigned in the United States, the President of Target Canada was fired, in months <a href="https://www.cbc.ca/news/business/target-canada-to-close-all-stores-by-april-12-1.3018677">all of the Target stores in Canada were closed</a> — gone. People lost confidence. There was just no more business there. So it can cost you enormously if you don’t address privacy. Don’t just do it because it’s the right thing to do. I always tell companies and governments, “just because you have custody and control over someone’s personal information doesn’t mean it belongs to you. It belongs to the data subject. So give them the ability to gain access to their data and be transparent, give them the right of access. When you do that, you may think you’re just doing it for them. No, you’re doing it for yourself. Because you will build trust. And as I’ve said, there’s such a trust deficit right now. It will build trusted business relationships with your customers, which is what all companies need.” It is in the company’s best interest, the developer’s best interest, to embed privacy-protective measures into the design of their operations.</p><blockquote><strong><em>[J]ust because you have custody and control over someone’s personal information doesn’t mean it belongs to you. It belongs to the data subject. So give them the ability to gain access to their data and be transparent, give them the right of access. When you do that, you may think you’re just doing it for them. No, you’re doing it for yourself. Because you will build trust.</em></strong></blockquote><p><strong>Patricia:</strong> Love that.</p><h4>Why is it scary to many organizations to tug on the thread of privacy?</h4><p><strong>Patricia:</strong> Given that developers should have that mandate, the organizations should have that mandate, and it’s very clear that trust is at the center of maintaining good customer relationships and how much privacy can help with that, there’s still — we’ve observed — a hesitancy towards integrating privacy. We’re wondering what you’ve observed in terms of why it’s so scary for many organizations to start thinking about integrating Privacy by Design.</p><p><strong>Ann:</strong> If people haven’t been exposed to Privacy by Design, which is like 99% of the population, they don’t know that it involves positive-sum not zero-sum, meaning privacy and data utility no matter the business they’re in. The way privacy tends to be portrayed is either-or zero-sum. You’ve got to protect people’s privacy and that’s going to impact your business negatively and that’s too bad. That’s nonsense! But that’s the zero-sum mindset that’s out there. I meet with so many companies and I go into the board room and I see all of these people sitting around the table kind of not very happy to see me and they’ve got to do it and they don’t want me there. And I say, “Give me 20 minutes. Give me 20 minutes to present Privacy by Design to you. And if you’re still reluctant to do it, I’ll leave. I’m happy to leave. But I want to show you how it can build a competitive advantage for your company over the other guys who don’t do it.” and they look at me and go, “Oh, you mean there’s some kind of positive associated with privacy?” I say, “Yes! It’s all about the positive! That’s what I want to present to you! And that’s what I want to show you, that instead of being afraid of the privacy issue, you will want to embrace it, because it will bring you many positive returns and it will increase so many factors.” One of the seven foundational principles of Privacy by Design is to give access to customers of their own personal information. Companies who have done this have come back to me and said, “I love this, because come back to us and tell us ‘that’s not correct — this was the fact two years ago, but that’s changed, here’s what’s going on now,” so they say, “They correct the information we have. It increases the quality of our data.” They love this. That’s just one of the examples of the benefits.</p><blockquote><strong><em>One of the seven foundational principles of Privacy by Design is to give access to customers of their own personal information. Companies who have done this have come back to me and said, “I love this, because come back to us and tell us ‘that’s not correct — this was the fact two years ago, but that’s changed, here’s what’s going on now,” so they say, “They correct the information we have. It increases the quality of our data.”</em></strong></blockquote><p><strong>Patricia:</strong> Beautiful example.</p><h4>Who should be responsible for enacting and measuring privacy within an organization?</h4><p><strong>Patricia:</strong> We touched upon this a little bit: developers should have a mandate. But who else in an organization do you think should be responsible for enacting and also measuring privacy?</p><p><strong>Ann:</strong> You have to have something the equivalent of a Chief Privacy Officer. If you’re a smaller organization you may not want to go that level, but you need an individual who’s charged with the responsibility of protecting privacy throughout your organization. Which includes looking under the hood, measuring privacy and how effectively it’s being monitored and extended to your customers. This is so important. When I talk to boards, I always tell them to make this individual report directly to the CEO, because if you report through a manager who reports to someone else, managers are afraid of ticking off the CEO and saying, “oh, we’re not doing very well on privacy.” So get them to report directly to the CEO who will understand that protecting privacy and offering the strongest privacy to your customers will give you positive returns again and again, you will gain a competitive advantage. It’s a total positive. But you need to give the Chief Privacy Officer the ability to enact this in some meaningful way so that if he or she sees that it’s not being operated in this department that he feels comfortable reporting on that and saying, “look, we need to upgrade this, it’s just not working here.” They need to be able to speak truthfully and measurement often involves some negative features. You have to prepare yourself for that and understand that ultimately, you will gain by doing so.</p><p><strong>Patricia:</strong> Perfect.</p><h4>What should we expect from upcoming privacy legislation around the world?</h4><p><strong>Patricia:</strong> So all of that said, legislations are moving more and more towards including Privacy by Design. Notably, the CPPA in Canada doesn’t seem to be including it. WOuld love to get your perspective on that and would love to get your perspective on how privacy regulations are moving across the world.</p><p><strong>Ann:</strong> Don’t get me on about Bill C-11 — the upgrade to our Federal private sector regulation — it just makes me gag. First of all, the GDPR, the General Data Protection Regulation in the European Union, came into effect in 2018. Everybody had been waiting for this forever and, to my delight, they included Privacy by Design and Privacy as the Default, which is the second foundational principle of Privacy by Design, which is a game changer. I was delighted. And laws all around the world have been upgrading to meet the requirements of the GDPR so that they could achieve essential equivalence and continue to engage in trade and commerce with the EU without fear of reprisal. Before the GDPR, our Federal private sector legislation was essentially equivalent with what they had before, but no longer! So, in 2017, just before it came into effect, our wonderful Federal Privacy Commissioner, <a href="https://pm.gc.ca/en/news/news-releases/2021/05/27/prime-minister-announces-nomination-daniel-therrien-reappointment">Daniel Therrien</a>, said to the government, “look, we’ve got to upgrade our law! We’re no longer essentially equivalent with the GDPR. We need to upgrade it. <a href="https://www.priv.gc.ca/en/privacy-topics/privacy-laws-in-canada/the-personal-information-protection-and-electronic-documents-act-pipeda/">PIPEDA</a> is dated from the early 2000’s, and we need to add Privacy by Design into the upgrade because that’s what they’ve done in the EU and, after all, it was created by a Canadian Ann Cavoukian. We’ve got to add that here.” In response to that, in 2018, the Federal Government put out a paper called <a href="https://www.ourcommons.ca/Content/Committee/421/ETHI/Reports/RP9690701/ethirp12/ethirp12-e.pdf">Towards Privacy by Design</a>. It was all about how they intended to upgrade PIPEDA and include Privacy by Design: yay! Wouldn’t you think “yay”? No! So here comes Bill C-11 — the upgrade to PIPEDA — is there any reference to Privacy by Design in it? No! Not whatsoever. It is just appalling to me. And there’s a whole bunch of other problems with it, which I can go into afterwards, but let me just stop here and say what we should expect from privacy legislation is what they’re doing around the rest of the world, which is embedding Privacy by Design into their upgrades, because they recognize that we’ve got to get proactive. We can’t just have things kick in after-the-fact. We have to get in front of this and have privacy protective measures embedded into our operations, baked into the code, into all of our design.</p><p><strong>Patricia:</strong> Completely agree. So curious to pick your brain later to see why that may have happened. We are at question period. We have 5 minutes for questions if people want to drop their questions on the right side.</p><p><strong>Ann:</strong> Patricia, if I can just remind everyone who is listening: privacy forms the foundation of freedom. You cannot have free and open societies without a solid foundation of privacy. Just because surveillance is mounting means nothing. It’s like a chess game: point — counter point. And there’s a huge movement towards decentralization now. There’s a <a href="https://identity.foundation/">Decentralized Identify Foundation</a> that was created last year. It’s growing dramatically. All of the major companies are included in it: Microsoft, IBM, etc. We are making gains. We never give up.</p><h3><strong>Q &amp; A:</strong></h3><h4>Why wasn’t Privacy by Design included in Bill C-11?</h4><p><strong>Ann: </strong>Gee. Let’s ask the Federal Government. I have not one clue. I’m not kidding. And there’s a number of other problems with it. The good thing was, I kept saying to the government, “You’ve got to give the Federal Commissioner order-making power.” I had order-making power when I was Privacy Commissioner of Ontario for 3 terms. It makes a big difference. It’s a stick, which I rarely used, but the other side knows you have it, so they engage in informal resolution with you and you get wonderful outcomes. Federal Commissioners never had it. So I said, “You’ve got to give it to them.” So what do they do? They gave them order-making power in Bill C-11: yay. But then what do they do? They create this tribunal that people can go to to appeal the order. If they don’t like the order, they go to a tribunal instead of going to court, which is what happens with all of the other commissioners in the provinces. This tribunal only has to have 1 privacy person on it. The other people can consist of I don’t know who, lawyers, whatever. And the commissioner’s decisions can be easily undone by going to this stupid tribunal that doesn’t exist anywhere else. Not in Canada, not elsewhere. So where they got this model from, I have no idea. And there are so many other problems as well.</p><h4>Do you expect upcoming regulation to mandate Privacy by Design beyond what GDPR and CCPA prescribe (penalizing issues after-the-fact: reactive vs. proactive)?</h4><p><strong>Ann:</strong> What I’ve seen is that ever since the GDPR included Privacy by Design and Privacy as the Default, new laws that are being created like in Brazil and other jurisdictions are including Privacy by Design.I’m delighted with that. Let me just explain Privacy as the Default for a moment. There are seven foundational principles of Privacy by Design. The second one is Privacy as the default setting. And it is <em>so</em> important. It’s a true game changer, because what it does is say to people, “You don’t have to ask for privacy. You don’t have to search through all of the Terms of Service and raid through all of the legalese in the Privacy Policy to find the opt-out clause that says ‘Do not use my personal information for any purpose other than the intended, primary purpose of the data collection’.” It takes way too long to do that. We know that people aren’t doing that. Life is too short. But people want privacy like never before. As I mentioned, 90% of them, easily, are very concerned about their privacy. Privacy as the Default flips that on its head and it says to people, “You don’t have to ask for privacy. We give it to you automatically. It’s the default setting. It exists throughout our operations. We can only collect your personal information for the primary purpose of the data collection, for which we need your positive consent. Beyond that, we can’t use your information for anything else. And if a later use arises — a secondary use that we would like to use your information for — we need to come back to you and seek your positive consent.” This is such a game changer. It builds trust like no other. People love it. Companies love it. It’s a win-win.</p><h4>One of the most useful aspects of data is the associations between different types of data. How do we preserve these associations when we anonymize the data?</h4><p><strong>Ann:</strong> You can anonymize the data and still keep the characteristics of the data linked to it. So you’re not gonna know that this data relates to Ann Cavoukians. But you can ask for information relating to the individual: what kind of work I do? Are they a professional? Do they own a home? Whatever the interest is. You can still have questions linked to the individual, but not ones that reveal the identity of the individual. And this has been perfected. <a href="https://research.uottawa.ca/people/el-emam-khaled">Professor Khaled El Emam</a>, who is at the University of Ottawa and a brilliant individual in terms of the de-identification of data. He does this beautifully and now he’s also expanding the area of synthetic data, which is growing enormously, where you can have synthetic data that is comparable to the personally identifiable data, but minus any identifiers of any kind. And it’s synthetic, so it can be used for dozens of purposes and linkages that you might be interested in. So you don’t lose out on it at all, you just love the personal identifiers.</p><h4>How do you think the perspective of privacy is different between Canada, thh U.S., and Europe? I know that, in Europe, it’s a human rights issue and in the U.S. it’s more of a consumer issue. Where does Canada stand?</h4><p><strong>Ann: </strong>Probably in between. Perhaps a little closer to Europe. But, increasingly, in the United States, you’re getting more and more privacy-related laws that are amazing and that come to approximate Canada and the EU in terms of the variety of areas which they’re addressing. For example, facial recognition is terrible from a privacy perspective. There are so many false positives associated with facial recognition. Of course, it takes place without your consent, without your knowledge. Nothing. In the U.K., where it abounds: 81% error rate. A study from Detroit a few months ago: 96% error rate. I mean, why are we doing this? A number of states in the United States have put an outright ban on facial recognition, which is brilliant and wonderful. We haven’t done that in Canada yet. So there are areas where, in the United States, they are advancing perhaps ahead of us. And they are trying to come a little closer to adopting a more European model, so stay tuned.</p><p><strong>Patricia</strong>: Thank you so much, Ann! Rohit says, “Thank you for the great discussion!” That ends our webinar. Always a pleasure. I know you’re super busy, so very grateful to you for making time for us.</p><p><strong>Ann: </strong>Oh, Patricia, it’s always my pleasure working with you and with all of your team and getting this message out there. Thank you very much.</p><p><strong>Patricia</strong>: Thank you so much</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6943c01001af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anonymized data is useless: fact or fiction?]]></title>
            <link>https://medium.com/data-science/anonymized-data-is-useless-fact-or-fiction-b8ab087d5c1e?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8ab087d5c1e</guid>
            <category><![CDATA[anonymization]]></category>
            <category><![CDATA[thoughts-and-theory]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Tue, 17 Aug 2021 18:22:20 GMT</pubDate>
            <atom:updated>2021-08-19T21:24:20.375Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/thoughts-and-theory">Thoughts and Theory</a></h4><h3><strong>Anonymized Data Is Useless: Fact or Fiction?</strong></h3><h4>Digging into unstructured data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*p3KJhH4O3vRk28K5" /><figcaption>Source: <a href="https://unsplash.com/photos/qm1zk-2oWD4">https://unsplash.com/photos/qm1zk-2oWD4</a></figcaption></figure><p><strong>“When is anonymization useful?</strong>” is a tricky question, because the answer is highly data-type- and task-dependent. Anonymized datasets are being used for academic research, industrial research, and real-world products in numerous areas, with clinical research often at the vanguard due to the high level of sensitivity and utility of the data. A <a href="https://www.nist.gov/system/files/documents/itl/iad/vug/2016-03-07-De-IDv5-2.pdf">2016 NIST presentation</a> mentions several other use cases in which anonymized data are useful, including:</p><ul><li>Improving driving solutions for directions and traffic data.</li><li>Pothole alerts.</li><li>Releasing educational records.</li><li>Voluntary safety reports submitted to the Federal Aviation Administration.</li></ul><p>While there have been years of research on proper methods for structured data anonymization (especially in the medical domain), research on unstructured data anonymization is just starting to ramp up. In this post we’ll dive into the research happening in speech, images/video, and text anonymization spaces.</p><h3><strong>Speech</strong></h3><p>For speech, anonymization means:</p><p>(1) Making a speaker’s voice unrecognizable (e.g., using the methodology proposed in <a href="https://www.eurecom.fr/fr/publication/5909/download/sec-publi-5909.pdf">Speaker Anonymization Using X-vector and Neural Waveform Models</a>) and</p><p>(2) Removing direct and quasi identifiers from the speech by either bleeping them out or replacing them (i.e., pseudonymization).</p><p>Quick reminder if you haven’t read “<a href="https://towardsdatascience.com/demystifying-de-identification-f89c977a1be5">Demystifying De-identification</a>” or “<a href="https://towardsdatascience.com/data-anonymization-perspectives-from-a-former-skeptic-f35790a2042a">Data Anonymization: Perspectives from a Former Skeptic</a>” that direct identifiers are entities that directly identify an individual (full name, exact location, social security number, etc.) and that quasi-identifiers are entities that can identify an individual with exponential likelihood when combined together (age, approximate location, spoken languages, etc.).</p><p>If speech technology and privacy is your thing, take a look at the <a href="https://www.voiceprivacychallenge.org/">VoicePrivacy initiative</a> and at the <a href="https://www.spsc-sig.org/about">ISCA Special Interest Group on Security and Privacy in Speech Communication</a>, which brings together professionals from a variety of backgrounds (from signal processing to law) to discuss privacy in speech technologies.</p><h3>Images &amp; Video</h3><p>Anonymization in images and video is a complicated task, given the variance in identifiable information. While fully and properly blurring out whole human bodies in the pictures might do the trick for certain constrained use cases, there can still be re-identification risk from name tags on backpacks, differentiated lunchboxes, a house in the background, etc. Nevertheless, anonymization for these media has often just meant removing or replacing faces, which means it is limited to a particular part of the body rather than the reduction of re-identification risk to almost zero (e.g., face anonymization — see, for instance, <a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Maximov_CIAGAN_Conditional_Identity_Anonymization_Generative_Adversarial_Networks_CVPR_2020_paper.html">CIAGAN: Conditional Identity Anonymization Generative Adversarial Networks</a>). This is a start, but considering that companies like Palantir Technologies can recognize people by their tattoos, removing or replacing a part of the body can often only really be called redaction, not anonymization.</p><p>That said, there are numerous machine learning tasks that make use of images and video without personal data or in which personal data is superfluous and could be removed/replaced without detriment to the task, including:</p><ul><li><a href="https://www.progressivegardening.com/agricultural-meteorology/image-classification.html">Agricultural meteorology</a></li><li><a href="https://github.com/imamun93/animal-image-classifications">Animal image classification</a></li><li><a href="https://www.frontiersin.org/articles/10.3389/frai.2020.534696/full">Satellite imagery comprehension</a></li><li><a href="https://github.com/ahmetozlu/vehicle_counting_tensorflow">Vehicle counting</a></li></ul><p>Just take the example provided in that vehicle counting GitHub repo for vehicle counting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2m8uUnBNDDOWA37H" /><figcaption>Image source: <a href="https://github.com/ahmetozlu/vehicle_counting_tensorflow">https://github.com/ahmetozlu/vehicle_counting_tensorflow</a> (MIT license)</figcaption></figure><p>It’s clear that neither license plates nor people’s faces play a role in the task. And if we’re concerned about unique car colours being too telling, even a black and white video could do nicely, as would counting the vehicles on the edge (e.g., directly on the camera, before the data hits any servers).</p><p>Take this other image as an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9o93JlQLSiV3SOUF" /><figcaption>Source: <a href="https://unsplash.com/photos/EJnhyLTbPH8">https://unsplash.com/photos/EJnhyLTbPH8</a></figcaption></figure><p>What can you detect in the image?</p><ul><li>Number of houses</li><li>Type of farmland</li><li>Relief</li><li>Weather</li></ul><p>There is a lot of information available about the terrain and there are plenty of similar images that can be used for determining ecosystem health, whether there are weeds growing in a crop, etc. Not to mention that lots of anonymous video feeds can be used as a partial training set for self-driving cars.</p><h3>Text</h3><p>Last but not least, let’s consider text anonymization. There has been some initial research on re-identification risk scores for text, including our work in the Journal of Data Protection and Privacy titled <a href="https://www.ingentaconnect.com/content/hsp/jdpp/2020/00000003/00000003/art00008">Reasoning about unstructured data de-identification</a> (email me if you have a hard time accessing the paper). While proper anonymization for the purposes of data publication has required an expert to go over the data and calculate the risk of re-identification, we can say that automatically redacting text has a huge role to play in greatly improving data security through data minimization (i.e., reducing the amount of personal data you collect to just the essentials). Note that there have been tests conducted on the effectiveness of statistical and rule-based systems to automatically de-identify medical text corpora (three of these studies summarized <a href="https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.8053.pdf">here</a>). These tests have to be redone to account for the vast improvements in statistical natural language processing systems from the past three years.</p><p>Anecdotally, let me give you a quick example of how much information a single anonymized email can carry:</p><p>“Hi [NAME],</p><p>Apologies, it had ended up in my spam!</p><p>I’m booked at [TIME] tomorrow, but [TIME] would work. I’ll send an updated invite for that time. Please let me know if that doesn’t work for you.</p><p>Thank you,</p><p>[NAME]”</p><p>Any idea who wrote that? It’s impossible to know unless you were the recipient or author.</p><p>But what useful information can you gather from this email?</p><ul><li>A call is being rescheduled to tomorrow</li><li>The sender is polite (says please and thank you)</li><li>The recipient’s previous email ended up in a spam folder when it shouldn’t have!</li></ul><p>What can, say, an email service provider use this information for? Well, it would be great if they could make sure that this recipient’s emails never end up in a spam folder again.</p><p>I have so many examples of these. From being able to identify how a person feels about a particular product, to determining which topics were covered in a conversation and determining a consumer’s sentiment over a chat or phone call.</p><h3><strong>Anonymous Data Is</strong></h3><p>It has taken time and ample research for the community to gain a greater understanding of what it means for data to be anonymous and useful. The process of iterating over a technology and understanding its limitations was felt in the field of cryptography just as it is in differential privacy and in anonymization. We no longer use DES to encrypt our data, but rather AES. And chances are that, in the next decade, we will have to rely more on lattice-based cryptography rather than RSA. As we find limitations in a technology, we do not throw the baby out with the bathwater, but rather look to gain a deeper understanding of what went wrong, innovate upon it, and make it stronger, more useful, and more accessible.</p><p><strong>Acknowledgements<br></strong>Thank you to John Stocks and Pieter Luitjens for their feedback on earlier drafts of this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8ab087d5c1e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/anonymized-data-is-useless-fact-or-fiction-b8ab087d5c1e">Anonymized data is useless: fact or fiction?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Anonymization: Perspectives from a Former Skeptic]]></title>
            <link>https://medium.com/data-science/data-anonymization-perspectives-from-a-former-skeptic-f35790a2042a?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f35790a2042a</guid>
            <category><![CDATA[data-protection]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[privacy-protection]]></category>
            <category><![CDATA[anonymization]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Fri, 04 Jun 2021 14:39:53 GMT</pubDate>
            <atom:updated>2021-06-08T19:07:00.911Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/notes-from-industry">Notes from Industry</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*COC6lNvSZ45Fz7pUvwxo_w.jpeg" /><figcaption>Source: <a href="https://unsplash.com/photos/1tnS_BVy9Jk">https://unsplash.com/photos/1tnS_BVy9Jk</a></figcaption></figure><p>In <a href="https://towardsdatascience.com/demystifying-de-identification-f89c977a1be5?source=your_stories_page-------------------------------------">a previous post</a>, I went over the definitions of anonymization, de-identification, redaction, pseudonymization, and tokenization. It’s a good place to start if you don’t know the difference between those terms.</p><p>Most relevant, the definition of anonymization is “[t]he process in which individually identifiable data is altered in such a way that it no longer can be related back to a given individual” (<a href="https://iapp.org/resources/glossary/">IAPP Glossary of Privacy Terms</a>). In practice, anonymization really means that there is a *negligible* chance that the altered data can be related back to a given individual.</p><p>When I first started working with privacy enhancing technologies, I lived and breathed <a href="https://private-ai.ca/2018/12/26/homomorphic-encryption-for-beginners-a-practical-guide-part-1/">homomorphic encryption</a>. I had tremendous mistrust of anonymization and thought anyone who believed otherwise simply did not know the facts. After all, there’s the famous quote “anonymized data isn’t” from Cynthia Dwork, known for her breakthrough research in differential privacy.</p><p>How many times have you seen headlines like the one below (<a href="https://www.darkdaily.com/researchers-easily-reidentify-deidentified-patient-records-with-95-accuracy-privacy-protection-of-patient-test-records-a-concern-for-clinical-laboratories/">Dark Daily</a>)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ag4jdO0OF2W8FuTa" /></figure><p><strong>So anonymization doesn’t work, right?</strong></p><p>I genuinely thought so. I believed the headlines and experts suggesting that anonymization was ineffective. As part of my PhD, I decided to look in detail at some of these re-identification attacks, at the data that had been re-identified, and at the percentage of records re-identified in the datasets:</p><p>“We found that patients can be re-identified, without decryption, through a process of linking the unencrypted parts of the record with known information about the individual such as medical procedures and year of birth” (<a href="https://www.darkdaily.com/researchers-easily-reidentify-deidentified-patient-records-with-95-accuracy-privacy-protection-of-patient-test-records-a-concern-for-clinical-laboratories/">Dark Daily</a>)</p><p>or</p><p>“A 2016 study found a 42.8% risk of matching anesthesia records to the Texas Inpatient Public Use Data File for 2013 using data such as Age, Sex, Hospital, and Year.” (<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6371259/">Re-Identification Risk in HIPAA De-Identified Datasets: The MVA Attack</a>)</p><p>and</p><p>“[Sweeney] was then able to uniquely identify the medical record of Governor Weld. She did this by noting that only 6 voters in Cambridge had the same birth date as the governor, only 3 of those were male, and only one of those lived in the same ZIP code as him.” (<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6371259/">Re-Identification Risk in HIPAA De-Identified Datasets: The MVA Attack</a>)</p><p>I uncovered that the true problem with the datasets mentioned in these headlines is that they were never truly anonymous in the first place: they included quasi-identifiers (e.g., year of birth, age, sex, approximate location) which were published without regard for population and dataset statistics. So it’s not that the datasets were fully anonymized and that it was easy to re-identify them. The truth is that these datasets were mislabeled as anonymous. <em>They never deserved that designation in the first place.</em></p><p>This is partially due to the learning curve the community went through to understand what needs to be done in order to completely anonymize a dataset. At first, it was believed that simply removing direct identifiers (full names, social security numbers, etc.) would do the trick. It took some stumbling to discover exactly how quasi-identifiers affect the likelihood of re-identifying an individual within a dataset.</p><p>These points are confirmed in “<a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0028071">A Systematic Review of Re-Identification Attacks on Health Data</a>” by El Emam, K., Jonker, E., Arbuckle, L. and Malin, B., which covers the misinterpretations of the severity or validity of a re-identification attacks. In it, the authors point out that only one re-identification attack has been successfully performed on data truly anonymized according to existing standards (see K in the table below), and even this one attack had a re-identification risk of 2/15,000 — well below HIPAA Safe Harbor’s <a href="https://www.ncvhs.hhs.gov/wp-content/uploads/2016/04/BARTH-JONES.pdf">acceptable risk threshold for re-identification of 0.04%</a>.</p><p>They summarize their findings in the following table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*01cnQwIqk3M9dD19" /><figcaption>Image: List of re-identification attacks over de-identified datasets. Only two out of the 14 attacks were on datasets that were properly anonymized. Only one of those attacks (<strong>K</strong>) has the re-identification verified. 2 out of 15,000 records were re-identified. Thank you to Professor Khaled El Emam for allowing me to reproduce this table.</figcaption></figure><p>Sources 57 and 58 from that table are not easy to find online and are not listed on the University of Toronto’s library catalogue, so digging into the weakness that lead to the re-identification of those 2 out of 15,000 records is not easy to do. For reference, those two papers are:</p><ul><li>57. Kwok P, Davern M, Hair E, Lafky D. Harder than you think: a case study of re-identification risk of HIPAA-compliant records. Chicago: NORC at The University of Chicago. Abstract #302255; 2011. [<a href="https://scholar.google.com/scholar_lookup?title=Harder+than+you+think:+a+case+study+of+re-identification+risk+of+HIPAA-compliant+records&amp;author=P+Kwok&amp;author=M+Davern&amp;author=E+Hair&amp;author=D+Lafky&amp;publication_year=2011&amp;">Google Scholar</a>]</li><li>58. Lafky D. The Safe Harbor method of de-identification: An empirical test. 2010. Fourth National HIPAA Summit West.</li></ul><p>It’s thanks to re-identification attacks by researchers and journalists, as well as concepts like k-anonymity, l-diversity, and t-closeness that the privacy community began to understand what anonymization means under a different lens. These concepts take into account what the likelihood of re-identification is based on an individual’s quasi-identifiers and how those can be optimally aggregated in order to minimize re-identification risk. One ideal paper to read in order to understand these three concepts, including their limitations and strengths, is the 2007 paper titled <a href="https://www.cs.purdue.edu/homes/ninghui/papers/t_closeness_icde07.pdf">t-Closeness: Privacy Beyond k-anonymity and l-diversity</a>, by Ninghui Li, Tiancheng Li, and Suresh Venkatasubramanian.</p><p>Anonymization methods have been used successfully on a number of clinical trial datasets with the purpose of sharing data for research. Four recent examples are cited in the 2015 paper titled <a href="https://www.bmj.com/content/bmj/350/bmj.h1139.full.pdf?casa_token=NwqT3F-i9xkAAAAA:U_T2t8ZaB1xWBgDOH7QbgQAuwMXJ6FehY07q_C0AztDejEDxp08awbjyWeOlMLOl14lV-W0z1OVjmw">Anonymising and sharing individual patient data</a>, by Khaled El Emam, Sam Rodgers, and Bradley Malin; namely:</p><ul><li>The <a href="https://www.clinicalstudydatarequest.com/">GlaxoSmithKline trials repository</a></li><li>The <a href="https://www.projectdatasphere.org/">Data Sphere project</a></li><li>The <a href="https://yoda.yale.edu/">Yale University Open Data Access project</a></li><li>The <a href="https://www.immport.org/home">Immport Immunology Database and Analysis Portal</a></li></ul><p>In sum, anonymization is far from easy. It takes years of expertise and experience to even properly conceptualize how quasi-identifiers can affect re-identification risk. And the term is often completely misused by journalists and companies alike.</p><p><strong>Acknowledgements</strong></p><p>Thank you to John Stocks and Pieter Luitjens for their feedback on earlier drafts of this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f35790a2042a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-anonymization-perspectives-from-a-former-skeptic-f35790a2042a">Data Anonymization: Perspectives from a Former Skeptic</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying De-identification]]></title>
            <link>https://medium.com/data-science/demystifying-de-identification-f89c977a1be5?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f89c977a1be5</guid>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[data-protection]]></category>
            <category><![CDATA[anonymization]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[editors-pick]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Tue, 06 Apr 2021 04:00:46 GMT</pubDate>
            <atom:updated>2021-04-10T18:35:35.668Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding key tech for data protection regulation compliance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*N8KGmYhzqG2TEQKs" /><figcaption>Source: <a href="https://unsplash.com/photos/B2SRwbYWLuI">https://unsplash.com/photos/B2SRwbYWLuI</a></figcaption></figure><p>Data anonymization, de-identification, redaction, pseudonymization, and tokenization are key technologies for satisfying data protection regulations such as the GDPR and the incoming CPRA. But distinguishing between anonymization, de-identification, redaction, pseudonymization, and tokenization is more complicated than it seems: there’s enough confusion and misinformation out there to bamboozle even the most seasoned data scientist.</p><h4><strong>Anonymization vs de-identification vs redaction vs pseudonymization vs tokenization</strong></h4><p>Getting these definitions right is a crucial step towards making the right decisions for protecting and using data most effectively. We’ll look at the definitions of each and give a running practical example to visually show what each of these looks like.</p><p>Re-identification risk is often intertwined in discussions surrounding these terms. <em>Re-identification</em> refers to the act of determining the identity of an individual who has directly identifying information (e.g., full name, social security number) or quasi-identifying information (e.g., age, approximate address) in a dataset. Quasi-identifiers, when combined together, lead to an exponential risk of re-identification. In some cases, the ability to re-identify is desired by the de-identifying body, in which case another layer of risk lies with the potential of a malicious party to access the data linking direct and quasi-identifiers with the data they were replaced with.</p><h4><strong>Anonymization</strong></h4><p><em>Anonymization </em>means removing personally identifiable information and quasi-identifiable information (i.e., data that, when combined with other information, can lead to re-identification; e.g., age, approximate address, etc.) from a dataset in such a way that individuals become permanently unidentifiable. See <a href="https://www.gdprsummary.com/anonymization-and-gdpr/">Anonymization and GDPR compliance; an overview</a> to find out how it fits into the GDPR.</p><p>Anonymization is often the preferred method for making structured medical datasets safe for sharing. What about unstructured data? Can it be anonymized? While it’s trickier to do so, the answer is yes. Here’s a sample email that’s been properly anonymized:</p><p>“Hi [NAME],</p><p>Apologies, it had ended up in my spam!</p><p>I’m booked at [TIME] tomorrow, but [TIME] would work. I’ll send an updated invite for that time. Please let me know if that doesn’t work for you.</p><p>Thank you,</p><p>[NAME]”</p><h4><strong>De-identification</strong></h4><p><em>De-identification</em> also requires the removal of personal and quasi-identifier, but through a process that enables the original data to be reconnected to the de-identified result. See <a href="https://www.educause.edu/focus-areas-and-initiatives/policy-and-security/cybersecurity-program/resources/information-security-guide/toolkits/guidelines-for-data-deidentification-or-anonymization">Guidelines for Data De-identification or Anonymization</a>.</p><p>In practice, de-identification is often used to describe the process of removing direct identifiers (full name, address, SSN, etc.) and sometimes quasi-identifiers (age, gender, etc.) but with fewer guarantees that data can’t be linked back to an individual than anonymization has, though it’s also sometimes used as a term that encompasses anonymization as well as pseudonymization.</p><p>What’s the point, you might wonder, of de-identifying data if you can’t guarantee that they’re anonymized? That depends on the use case.</p><p>Take the anonymized email above and change it to:</p><p>“Hi [NAME_1],</p><p>Apologies, it had ended up in my spam!</p><p>I’m booked at [TIME_1] tomorrow, but [TIME_2] would work. I’ll send an updated invite for that time. Please let me know if that doesn’t work for you.</p><p>Thank you,</p><p>[NAME_2]”</p><p>And now suppose a company decided to encrypt the direct and potential identifiers associated with NAME_1, NAME_2, TIME_1, and TIME_2 and store them separately in case they ever needed to re-identify the email (perhaps for potential use in court). That email would no longer count as anonymized, because the identifiers could be linked back to it.</p><p>But that doesn’t mean that privacy has been compromised. For example, if analytics and machine learning teams are using de-identified emails (as opposed to the plaintext originals), they are actually doing their users and their company a great service. It’s possible to gain insights while mitigating user privacy risk, and also minimizing the security risk of sending personal data to another part of an organization where it might become untraceable.</p><h4><strong>Redaction</strong></h4><p>To add to the confusion between anonymization and de-identification, the term redaction is also often used erroneously. According to the <a href="https://iapp.org/resources/glossary/#redaction">International Association of Privacy Professionals’ glossary</a>, redaction is “[t]he practice of identifying and removing or blocking information from documents being produced […]”.</p><p>Redaction plays an interesting role with respect to de-identification. Redaction doesn’t necessarily involve the full removal of personal data, but rather the selective removal of particularly sensitive information. One example is the removal of credit card numbers from customer service conversations. If emails, call transcripts, or chat logs get leaked with questions about how to use a vacuum cleaner… that’s probably not very damaging. But if there are credit card numbers in those chat logs and they haven’t been removed? HUGE deal.</p><h4><strong>Pseudonymization</strong></h4><p>Now there’s one term in particular that’s often misinterpreted; namely, pseudonymization. Pseudonymization means the replacement of certain data (e.g. a name, address, etc.) with fake data that is often represented as being linked to the original data. This has left a rather big hole to fill for a term that means replacing information with fake data that is not linked to the original data. We are going to coin the phrase <strong><em>natural pseudonymization without linkage</em></strong> to define this.</p><p>There are numerous advantages to natural pseudonymization without linkage over de-identification as represented above. For one, data becomes more friendly for machine learning training and inference. In the example below, the PII has been replaced with fake data in italics:</p><p>“Hi <em>Kate</em>,</p><p>Apologies, it had ended up in my spam!</p><p>I’m booked at <em>5pm</em> tomorrow, but <em>9am</em> would work. I’ll send an updated invite for that time. Please let me know if that doesn’t work for you.</p><p>Thank you,</p><p><em>Linda</em>”</p><p>Additionally, whatever personal or quasi-identifiable information was accidentally left over becomes like a needle in a haystack to distinguish from the fake data (or more like a specific piece of hay in a haystack); e.g, suppose “Linda” had been accidentally missed when de-identifying the email above:</p><p>“Hi [NAME_1],</p><p>Apologies, it had ended up in my spam!</p><p>I’m booked at [TIME_1] tomorrow, but [TIME_2] would work. I’ll send an updated invite for that time. Please let me know if that doesn’t work for you.</p><p>Thank you,</p><p>Linda”</p><p>At Private AI, we’ve spent a significant amount of time to figure out how to do automatic pseudonymization properly. Here’s a hint: dictionary lookup doesn’t work. We’ve had to create our own transformer model architecture (a type of machine learning model built for natural language processing) in order to generate realistic words and numbers in a contextualized, non-deterministic way. Careful selection of training data was paramount to generating realistic replacements, among other tricks of the trade.</p><p>Tokenization is also often referred to as a type of pseudonymization.</p><h4><strong>Tokenization</strong></h4><p>A final term that’s often used for specifically characterizing the type of token that replaces certain data is <em>tokenization</em>. That is, replacing personal data with a random token. Often, a link is maintained between the original data and the token (e.g., for payment processing on websites). Tokens can, for example, be generated by one-way functions (e.g., salted hashes) or can be completely random numbers. Some types of tokenization can even be reversible if they rely on encryption, for example, in which case only the decryption key needs to be stored, rather than the link between every piece of personal data and their replacement.</p><p>Let’s tokenize the direct and quasi-identifiable information in our running email example:</p><p>“Hi 748331D230BF99D9A39ED0E6C6668CDD,</p><p>Apologies, it had ended up in my spam!</p><p>I’m booked at 3388E06178D0634FC03FFBDECCE677F8 tomorrow, but F6F7755D5141D5B7308DF2516AA9A82C would work. I’ll send an updated invite for that time. Please let me know if that doesn’t work for you.</p><p>Thank you,</p><p>CE5A40345609B81A5E7C973C1F3D93EB”</p><p>While tokenization has been particularly useful for payment processing, it is unlikely to be a winner for unstructured data protection given its relative lack of relevant contextual information compared to, say, natural pseudonymization without linkage.</p><h4><strong>Putting it all together</strong></h4><p>Despite direct and quasi-identifiers being removed in one way or another through each of anonymization, de-identification, redaction, pseudonymization, and tokenization, they’re all remarkably effective at maintaining the contextual information of the original data. Stay tuned for our next post on how anonymization is misunderstood.</p><p>In the meantime, here’s a handy table to guide your decision-making:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8G555BqxtP2yFXGSLl3oKA.png" /></figure><p><strong>Acknowledgements</strong></p><p>Thank you to John Stocks and Pieter Luitjens for their feedback on earlier drafts of this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f89c977a1be5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/demystifying-de-identification-f89c977a1be5">Demystifying De-identification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cybersecurity and Privacy]]></title>
            <link>https://medium.com/privacy-preserving-data-processing/cybersecurity-and-privacy-161961b5243a?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/161961b5243a</guid>
            <category><![CDATA[data-protection]]></category>
            <category><![CDATA[privacy-by-design]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Mon, 14 Dec 2020 21:25:27 GMT</pubDate>
            <atom:updated>2020-12-18T17:53:11.489Z</atom:updated>
            <content:encoded><![CDATA[<h3>Complements for a more secure Internet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yOfFavzaAlI75Xax6kFAIg.jpeg" /></figure><p>There exists a vibrant ecosystem of specialized security tools. The sad truth is that it is almost impossible to reach 100% invulnerability.</p><p>Until recently, one of the main tools that organizations had in their toolbox were symmetric and asymmetric encryption to encrypt data at rest and data in transit. That left a glaring vulnerability: <em>what about data in use?</em> While antivirus software, threat detection tools, and access controls can help prevent unauthorized access to data, they are limited in their ability to protect organizations against social engineering or insider threats. In addition to the infrastructures that organizations are already used to having, there are new key requirements that can only be satisfied through the use of advanced privacy enhancing technologies (PETs) that give users more control over their data and their privacy.</p><p>PETs are necessary not only for data privacy but more generally for data protection. As defined in the GDPR, “<em>data protection</em> means keeping data safe from unauthorized access” whereas “<em>data privacy</em> means empowering your users to make their own decisions about who can process their data and for what purpose.” [<a href="https://gdpr.eu/data-privacy/">GDPR</a>]</p><p>Ensuring data privacy within an enterprise software is therefore dependent on proper data protection measures being in place. Data cannot be protected without an appropriate cybersecurity infrastructure being set up. Such infrastructure usually includes encryption at rest and encryption in transit, strict access controls, network management and monitoring, phishing attack detection, employee training programs,among others.</p><p>In particular, organizations need to know where sensitive data are stored and how they are used. This is where privacy tech comes full circle to ensure stronger cybersecurity by helping:</p><ol><li>keep track of which parts of a network are most vulnerable to data leaks, and</li><li>limit the exposure of sensitive data or personally identifiable &amp; quasi-identifiable information during processing time.</li></ol><p>What we can do as engineers is to continue building up the ecosystem which makes access to personal and sensitive data harder and harder for malicious actors to exploit.</p><p>Fairly quickly, it is becoming obvious that cybersecurity, privacy, and data protection need to be intimately intertwined in order to reach the highest possible levels of user protection.</p><h3><strong>More Fine-Grained Access Control</strong></h3><p>Access control is obviously central to keeping information safe. But unless you’re dealing with structured data alone, determining which information is sensitive and which is not becomes quite tricky. Enter AI-based personally identifiable information (PII) and quasi-identifiable information detection for unstructured data. Using Natural Language Processing and Computer Vision, it is now possible to detect PII in unstructured files with 99%+ accuracy.</p><p>You can imagine a database which returns filtered queries to machine learning engineers who are training a sentiment analysis model on chat logs. Or a database which security personnel use to review security footage for potential theft (only re-identifying the video feed once they’ve spotted a culprit). There often is no need for personal data to be present in order to get value out of a particular set of data. So why even bother sharing it, especially with so many employees working from home and using insecure networks? The more times personal data are replicated, the more difficult they become to track and keep safe.</p><p>AI-based unstructured data redaction is a key component to the next generation of fine-grained access. The very same AI system used for personal data redaction can also be used to locate where personal data are stored within a network and can thus be used to prevent data loss.</p><h3><strong>Personal Data Tracking for Data Loss Prevention and Network Management</strong></h3><p>Data Loss Prevention requires intimate knowledge about information movement: one cannot protect sensitive data without even knowing if and where it exists, and recognizing personal data across a number of unstructured data types is central to data protection and measuring risk.</p><p>Currently, most network management companies and data loss prevention companies don’t have the capabilities to accurately quantify the risk of sensitive data exposure within unstructured data using context-based data discovery. Unstructured data makes up about 80% of total data produced (see <a href="https://www.ibm.com/blogs/watson/2016/05/biggest-data-challenges-might-not-even-know/"><em>The biggest data challenges that you might not even know you have</em></a>) and as data protection and data privacy regulations multiply they are becoming an ever-growing risk to companies who have few insights as to what personal data they are storing and where. It is therefore extremely difficult for companies to keep proper track of any personal data that might be subject to access to information requests or requests to be forgotten. This is becoming an insurmountable task for humans to deal with. Enter scalable, accurate, and easy-to-deploy AI-based PII detection.</p><p>Beyond personal data detection and redaction there are numerous other methods which can be combined within existing cybersecurity infrastructure to improve data privacy and protection for different use cases.</p><h3><strong>Limiting Unencrypted Data Processing</strong></h3><p>For certain use cases (like processing credit card numbers or social security numbers), access to information must be strictly limited. Homomorphic Encryption and Secure Multiparty Computation, for example, allow for data to be encrypted/garbled throughout the entire computational process by producing unencrypted/ungarbled output only for the right parties to see.</p><p>Another technology which falls under this category is Zero-Knowledge Proofs, which is used when one party wants to prove to another (say a validator) that they have certain information without the validator finding out what that information is. Think authentication systems, for example, where your face is used to log you into a system which only has access to an encrypted version of your face and needs to compare it to their set of securely stored values.</p><p>Processing data while encrypted or garbled is a great choice when an operation is repeatable and the data does not have to be seen in order to be processed. Especially when an organization that requires quantum-safe security (banks, insurance companies, etc.) also has the budget to hire an internal team of experts or to pay a company for a bespoke solution.</p><h3><strong>Data Processing on the Edge</strong></h3><p>Processing data directly on an edge device (ie. IoT, phone, browser) can greatly limit risk exposure — in fact, it’s probably the safest approach an organization can take when processing data. Granted, it is not always the most practical. One major limitation is when data needs to be aggregated in order to provide value, like traffic data processing. The edge is truly great for deploying algorithms that are deterministic and use user-specific inputs, or have alternative relevant data to learn from in order to provide accurate probabilistic user-specific outputs. There is quite a bit of work going into edge machine learning, and both Apple and Google have such algorithms deployed on their devices to improve user experience in a privacy-preserving way. For now, however, it takes a specialized team to create edge machine learning algorithms that are truly privacy-preserving and also improve model accuracy. Thanks to organizations like <a href="https://www.openmined.org/">OpenMined</a>, Google, and Apple, this will get easier as time goes on.</p><h3><strong>Which Privacy Technique Is Right For Me?</strong></h3><p>Which privacy technique you choose to integrate within your infrastructure is entirely use-case dependent. There is really no one-size-fits-all technology. Once you have determined which technologies are optimal for your use case (this <a href="https://www.private-ai.ca/PETs_Decision_Tree.svg">PETs decision tree</a> can help), you can narrow down your choice even further by gaining a better understanding of your threat model (see, for example, <a href="https://docs.microsoft.com/en-us/azure/security/develop/threat-modeling-tool-threats">Microsoft’s STRIDE model</a>) and what kind of adversaries you’re expecting to dealing with (see <a href="http://mahdiz.com/crypto/basics/page-1.html?i=1">here</a> for some good definitions).</p><h3><strong>Further Reading</strong></h3><p><a href="https://us.norton.com/internetsecurity-privacy-privacy-vs-security-whats-the-difference.html">Privacy vs. security: what’s the difference?</a> (Symantec)</p><p><a href="https://private-ai.ca/2018/05/22/a-brief-overview-of-privacy-preserving-software-methods/">A Brief Overview of Privacy-Preserving Software Methods</a></p><p><a href="https://docs.google.com/document/d/12hysdr2H95XVOjs0PEMU3jQSnu6qLvgqzEupmZ28V80/edit">Homomorphic Encryption for Beginners: A Practical Guide</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=161961b5243a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/privacy-preserving-data-processing/cybersecurity-and-privacy-161961b5243a">Cybersecurity and Privacy</a> was originally published in <a href="https://medium.com/privacy-preserving-data-processing">Privacy-Preserving Data Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Privacy Enhancing Technologies Decision Tree (v2)]]></title>
            <link>https://medium.com/privacy-preserving-data-processing/privacy-enhancing-technologies-decision-tree-v2-e33c3842294f?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e33c3842294f</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[homomorphic-encryption]]></category>
            <category><![CDATA[differential-privacy]]></category>
            <category><![CDATA[data-protection]]></category>
            <category><![CDATA[de-identification]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Mon, 19 Oct 2020 00:59:04 GMT</pubDate>
            <atom:updated>2020-12-17T23:41:25.551Z</atom:updated>
            <content:encoded><![CDATA[<p>Medium does not allow for large, high quality images.<strong><br>For crisp version, please see:</strong> <a href="http://www.private-ai.ca/PETs_Decision_Tree.svg">www.private-ai.ca/PETs_Decision_Tree.svg</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7YpY92lwpAbzyba_-maJA.jpeg" /><figcaption><strong>For crisp version, please see:</strong> <a href="http://www.private-ai.ca/PETs_Decision_Tree.svg">www.private-ai.ca/PETs_Decision_Tree.svg</a></figcaption></figure><p><strong>For crisp version of the decision tree, please see:</strong> <a href="http://www.private-ai.ca/PETs_Decision_Tree.svg">www.private-ai.ca/PETs_Decision_Tree.svg</a></p><p>For an overview of many of the technologies mentioned in the decision tree, please see:</p><p><a href="https://medium.com/privacy-preserving-natural-language-processing/a-brief-overview-of-data-privacy-preserving-software-methods-46db9913bd78">https://medium.com/privacy-preserving-natural-language-processing/a-brief-overview-of-data-privacy-preserving-software-methods-46db9913bd78</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e33c3842294f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/privacy-preserving-data-processing/privacy-enhancing-technologies-decision-tree-v2-e33c3842294f">Privacy Enhancing Technologies Decision Tree (v2)</a> was originally published in <a href="https://medium.com/privacy-preserving-data-processing">Privacy-Preserving Data Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Perfectly Privacy-Preserving AI]]></title>
            <link>https://medium.com/data-science/perfectly-privacy-preserving-ai-c14698f322f5?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c14698f322f5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Wed, 01 Jan 2020 23:46:51 GMT</pubDate>
            <atom:updated>2020-10-08T13:58:17.849Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/data-privacy">Data Privacy</a></h4><h4>What is it and how do we achieve it?</h4><p>Data privacy has been called “<a href="https://www.forbes.com/sites/marymeehan/2019/11/26/data-privacy-will-be-the-most-important-issue-in-the-next-decade/#3211e2821882">the most important issue in the next decade</a>,” and has taken center stage thanks to legislation like the European Union’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Companies, developers, and researchers are <a href="https://www.theverge.com/2019/12/31/21039228/california-ccpa-facebook-microsoft-gdpr-privacy-law-consumer-data-regulation">scrambling to keep up with the requirements</a>. In particular, “<a href="https://www.ipc.on.ca/wp-content/uploads/resources/7foundationalprinciples.pdf">Privacy by Design</a>” is integral to the GDPR and will likely only gain in popularity this decade. When using privacy preserving techniques, legislation suddenly becomes less daunting, as does ensuring data security which is central to maintaining user trust.</p><p>Data privacy is a central issue to training and testing AI models, especially ones that train and infer on sensitive data. Yet, to our knowledge, there have been no guides published regarding what it means to have perfectly privacy-preserving AI. We introduce the four pillars required to achieve perfectly privacy-preserving AI and discuss various technologies that can help address each of the pillars. We back our claims up with relatively new research in the quickly growing subfield of privacy-preserving machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hnm8Txe9NPX3TylnmwzQfw.png" /><figcaption>The Four Pillars or Perfectly-Privacy Preserving AI</figcaption></figure><h3>The Four Pillars of Perfectly-Privacy Preserving AI</h3><p>During our research, we identified four pillars of privacy-preserving machine learning. These are:</p><ol><li><strong>Training Data Privacy</strong>: The guarantee that a malicious actor will not be able to reverse-engineer the training data.</li><li><strong>Input Privacy</strong>: The guarantee that a user’s input data cannot be observed by other parties, including the model creator.</li><li><strong>Output Privacy</strong>: The guarantee that the output of a model is not visible by anyone except for the user whose data is being inferred upon.</li><li><strong>Model Privacy</strong>: The guarantee that the model cannot be stolen by a malicious party.</li></ol><p>While 1–3 deal with protecting data creators, 4 is meant to protect the model creator.</p><h3>Training data privacy</h3><p>While it may be slightly more difficult to gather information about training data and model weights than it is from plaintext (the technical term for unencrypted) input and output data, recent research has demonstrated that reconstructing training data and reverse-engineering models is not as huge challenge as one would hope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*te8Nl1ivC1szfGSzpedj-A.png" /><figcaption>Source: <a href="https://xkcd.com/2169/">https://xkcd.com/2169/</a></figcaption></figure><p><strong><em>Evidence</em></strong></p><p>In [1], Carlini and Wagner calculate just how quickly generative sequence models (e.g., character language models) can memorize rare information within a training set. Carlini and Wagner train a character language model on the Penn Treebank with a “secret” inserted into it once: “the random number is ooooooooo” where ooooooooo is a meant to be a (fake) social security number. They show how the exposure of a secret which they hide within their copy of the Penn Treebank Dataset (PTD). They train a character language model on 5% of the PTD and calculate the network’s amount of memorization. Memorization peaks when the test set loss is lowest. This coincides with peak exposure of the secret.</p><p><strong><em>Metrics</em></strong></p><p>So how can we quantify how likely it is that a secret can be reverse-engineered from model outputs? [1] develops a metric known as <em>exposure</em>:</p><p><em>Given a canary s[r], a model with parameters θ, and the randomness space R, the exposure s[r] is</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*RjurXvT4vAS-neIL" /></figure><p>and the rank is the index at which the true secret (or canary) is among all possible secrets given the model’s perplexity for the inputs. The smaller the index, the greater the likelihood that the sequence appears in the training data, so the goal is to minimize the exposure of a secret, which is something that Carlini and Wagner. achieve by using differentially private gradient descent (see <strong><em>Solutions </em></strong>below).</p><p>Another exposure metric is presented in [2], in which the authors calculate how much information can be leaked from a latent representation of private data sent over an insecure channel. While this paper falls more into the category of input data privacy analysis, it’s still worth looking at the metric proposed to compare it with the one presented in [1]. In fact, they propose two privacy metrics. One for demographic variables such as sentiment analysis and blog post topic classification, and one for named entities such as news topic classification. Their privacy metrics are:</p><ol><li>Demographic variables: “1 − <em>X</em>, where <em>X</em> is the average of the accuracy of the attacker on the prediction of gender and age,”</li><li>Named entities: “1−<em>F</em>, where <em>F</em> is an F-score computed over the set of binary variables in <strong>z</strong> that indicate the presence of named entities in the input example,” where “<strong>z</strong> is a vector of private information contained in a [natural language text].”</li></ol><p>When looking at the evidence, it’s important to keep in mind that this sub-field of AI (privacy-preserving AI) is brand-spanking new, so there are likely a lot of potential exploits that either have not been analyzed or even haven’t been thought of yet.</p><p><strong><em>Solutions</em></strong></p><p>There are two main proposed solutions for the problem of training data memorization which not only guarantee privacy, but also improve the generalizability of machine learning models:</p><ol><li>Differentially Private Stochastic Gradient Descent (DPSGD) [3, 4]: While differential privacy was originally created to allow one to make generalizations about a dataset without revealing any personal information about any individual within the dataset, the theory has been adapted to preserve training data privacy within deep learning systems.<br>For a thorough discussion on the use of differential privacy in machine learning, please read<a href="https://medium.com/privacy-preserving-natural-language-processing/differentially-private-natural-language-processing-4f18912c5de0"> this interview</a> with Dr. Parinaz Sobhani, Director of Machine Learning at Georgian Partners, one of Canada’s leading Venture Capital firms.</li><li>Papernot’s PATE [5]: Professor Papernot created PATE as a more intuitive alternative to DPSGD. PATE can be thought of as an ensemble approach &amp; works by training multiple models on iid subsets of the dataset. At inference, if the majority of the models agree on the output, then the output doesn’t reveal any private information about the training data and can therefore be shared.</li></ol><h3>Input and output privacy</h3><p>Input user data and resulting model outputs inferred from that data should not be visible to any parties except for the user in order to comply with the four pillars of perfectly privacy-preserving AI. Preserving user data privacy is not only beneficial for the users themselves, but also for the companies processing potentially sensitive information. Privacy goes hand in hand with security. Having proper security in place means that data leaks are much less likely to occur, leading to the ideal scenario: no loss of user trust and no fines for improper data management.</p><p><strong><em>Evidence</em></strong></p><p>This is important to ensure that private data do not:</p><ul><li>get misused (e.g.,<a href="https://www.nytimes.com/interactive/2019/12/19/opinion/location-tracking-cell-phone.html"> location tracking as reported in the NYT</a>)</li><li>fall into the wrong hands due to, say, a hack, or</li><li>get used for tasks that a user had either not expected or explicitly consented to (e.g.,<a href="https://www.independent.co.uk/life-style/gadgets-and-tech/news/amazon-alexa-echo-listening-spy-security-a8865056.html"> Amazon admits employees listen to Alexa conversations</a>).</li></ul><p>While it is standard for data to be encrypted in transit and (if a company is responsible) at rest as well, data is vulnerable when it is decrypted for processing.</p><p><strong><em>Solutions</em></strong></p><ol><li>Homomorphic Encryption: homomorphic encryption allows for non-polynomial operations on encrypted data. For machine learning, this means training &amp; inference can be performed directly on encrypted data. Homomorphic encryption has successfully been applied to random forests, naive Bayes, and logistic regression [6]. [7] designed low-degree polynomial algorithms that classify encrypted data. More recently, there have been adaptations of deep learning models to the encrypted domain [8, 9, 10].<br>See<a href="https://medium.com/privacy-preserving-natural-language-processing/homomorphic-encryption-for-beginners-a-practical-guide-part-1-b8f26d03a98a"> this post</a> for an introduction to homomorphic encryption.</li><li>Secure Multiparty Computation (MPC): the idea behind MPC is that two or more parties’ who do not trust each other can transform their inputs into “nonsense” which gets sent into a function whose output is only sensical when the correct number of inputs are used. Among other applications, MPC has been used for genomic diagnosis using the genomic data owned by different hospitals [11], and linear regression, logistic regression, and neural networks for classifying MNIST images [12]. [11] is a prime example of the kind of progress that can be made by having access to sensitive data if privacy is guaranteed. There are a number of tasks which cannot be accomplished with machine learning given to the lack of data required to train classification and generative models. Not because the data isn’t out there, but because the sensitive nature of the information means that it cannot be shared or sometimes even collected, spanning medical data or even speaker-specific metadata which might help improve automatic speech recognition systems (e.g., age group, location, first language).<br>A great introduction to MPC can be found<a href="https://securecomputation.org/"> here</a>. You can find Dr. Morten Dahl’s Private Deep Learning with MPC tutorial<a href="https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/"> here</a>.</li><li>Federated Learning: federated learning is basically on-device machine learning. It is only truly made private when combined with differentially private training (see DPSGD in the previous section) and MPC for secure model aggregation [13], so the data that was used to train a model cannot be reverse engineered from the weight updates output out of a single phone. In practice, Google has deployed federated learning on Gboard (see their <a href="https://ai.googleblog.com/2017/04/federated-learning-collaborative.html">blog post</a> about it) and Apple introduced federated learning support in <a href="https://developer.apple.com/documentation/coreml">CoreML3</a>.</li></ol><h3>Model privacy</h3><p>AI models can be companies’ bread and butter, many of which provide predictive capabilities to developers through APIs or, more recently, through downloadable software. Model privacy is the last of the four pillars that must be considered and is also core to both user and company interests. Companies will have little motivation to provide interesting products and spend money on improving AI capabilities if their competitors can easily copy their models (an act which is not straightforward to investigate).</p><p><strong><em>Evidence</em></strong></p><p>Machine learning models form the core product &amp; IP of many companies, so having a model stolen is a severe threat and can have significant negative business implications. A model can be stolen outright or can be reverse-engineered based on its outputs [14].</p><p><strong><em>Solutions</em></strong></p><ol><li>There has been some work on applying differential privacy to model outputs to prevent model inversion attacks. Differential privacy usually means compromising model accuracy; however, [15] presents a method that does not sacrifice accuracy in exchange for privacy.</li><li>Homomorphic encryption can be used not only to preserve input and output privacy, but also model privacy, if one chooses to encrypt a model in the cloud. This comes at significant computational cost, however, and does not prevent model inversion attacks.</li></ol><h3>Satisfying All Four Pillars</h3><p>As can be seen from the previous sections, there is no blanket technology that will cover all privacy problems. Rather, to have perfectly privacy-preserving AI (something that both the research community and industry have yet to achieve), one must combine technologies:</p><ul><li>Homomorphic Encryption + Differential Privacy</li><li>Secure Multiparty Computation + Differential Privacy</li><li>Federated Learning + Differential Privacy + Secure Multiparty Computation</li><li>Homomorphic Encryption + PATE</li><li>Secure Multiparty Computation + PATE</li><li>Federated Learning + PATE + Homomorphic Encryption</li></ul><p>Other combinations also exist, including some with alternative technologies that do not have robust mathematical guarantees yet; namely, (1) secure enclaves (e.g., Intel SGX) which allow for computations to be performed without even the system kernel having access, (2) data de-identification, and (3) data synthesis.</p><p>For now, perfectly privacy-preserving AI is still a research problem, but there are a few tools that can address some of the most urgent privacy needs.</p><h3>Privacy-Preserving Machine Learning Tools</h3><ul><li><a href="https://github.com/tensorflow/privacy">Differential privacy in Tensorflow</a></li><li><a href="https://github.com/OpenMined/PySyft">MPC and Federated Learning in PyTorch</a></li><li><a href="https://github.com/mpc-msri/EzPC">MPC in Tensorflow</a></li><li><a href="https://developer.apple.com/documentation/coreml">On-device Machine Learning with CoreML3</a></li></ul><h3>Acknowledgments</h3><p>Many thanks to Pieter Luitjens and Dr. Siavash Kazemian for their feedback on earlier drafts of this post and Professor Gerald Penn for his contributions to this work.</p><h3>References</h3><p>[1] Carlini, Nicholas, et al., <em>The Secret Sharer: Evaluating and testing unintended memorization in neural networks (2019)</em>, 28th USENIX Security Symposium (USENIX Security 19).</p><p>[2] Coavoux, Maximin, Shashi Narayan, and Shay B. Cohen, <em>Privacy-preserving neural representations of text</em> (2018), arXiv preprint arXiv:1808.09408.</p><p>[3] Song, Shuang, Kamalika Chaudhuri, and Anand D. Sarwate, <em>Stochastic gradient descent with differentially private updates</em> (2013)<em>,</em> IEEE Global Conference on Signal and Information Processing.</p><p>[4] Wu, X., Li, F., Kumar, A., Chaudhuri, K., Jha, S., and Naughton, J., <em>Bolt-on differential privacy for scalable stochastic gradient descent-based analytics </em>(2017), In Proceedings of the 2017 ACM International Conference on Management of Data, 1307–1322. ACM.</p><p>[5] Papernot, Nicolas, et al., <em>Scalable private learning with PATE</em> (2018), arXiv preprint arXiv:1802.08908 (2018).</p><p>[6] Aslett, Louis JM, Pedro M. Esperança, and Chris C. Holmes, <em>Encrypted statistical machine learning: new privacy preserving methods </em>(2015), arXiv preprint arXiv:1508.06845.</p><p>[7] Graepel, Thore, et al., <em>Machine Learning on Encrypted Data</em> (2012), ICISC 2012, LNCS 7839.</p><p>[8] Hesamifard, Ehsan, Hassan Takabi, and Mehdi Ghasemi, <em>CryptoDL: Deep neural networks over encrypted data </em>(2017), arXiv preprint arXiv:1711.05189.</p><p>[9] Hesamifard, Ehsan, et al., <em>Privacy-preserving machine learning as a service</em> (2018), Proceedings on Privacy Enhancing Technologies.</p><p>[10] Gilad-Bachrach, Ran, et al., <em>CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy</em> (2016), International Conference on Machine Learning.</p><p>[11] Jagadeesh, Karthik A., et al., <em>Deriving genomic diagnoses without revealing patient genomes </em>(2017), Science 357.6352.</p><p>[12] Mohassel, Payman, and Yupeng Zhang, <em>SecureML: A system for scalable privacy-preserving machine learning</em> (2017), 2017 IEEE Symposium on Security and Privacy (SP).</p><p>[13] Bonawitz, Keith, et al., <em>Practical secure aggregation for privacy-preserving machine learning </em>(2017), Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security.</p><p>[14] Tramèr, Florian, et al., <em>Stealing machine learning models via prediction apis</em> (2016), 25th USENIX Security Symposium (USENIX Security 16).</p><p>[15] Wang, Yue, Cheng Si, and Xintao Wu, <em>Regression model fitting under differential privacy and model inversion attack</em> (2015), Twenty-Fourth International Joint Conference on Artificial Intelligence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c14698f322f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/perfectly-privacy-preserving-ai-c14698f322f5">Perfectly Privacy-Preserving AI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Homomorphic Encryption for Beginners: A Practical Guide (Part 2: The Fourier Transform)]]></title>
            <link>https://medium.com/privacy-preserving-natural-language-processing/homomorphic-encryption-for-beginners-a-practical-guide-part-2-the-fourier-transform-77bcaf9a1756?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/77bcaf9a1756</guid>
            <category><![CDATA[cryptography]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[signal-processing]]></category>
            <category><![CDATA[homomorphic-encryption]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 21:29:37 GMT</pubDate>
            <atom:updated>2019-09-26T13:45:52.495Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/privacy-preserving-natural-language-processing/homomorphic-encryption-for-beginners-a-practical-guide-part-1-b8f26d03a98a">Part 1</a> of this guide, we covered many of the basics of homomorphic encryption. In Part 2, we will discuss a practical application of homomorphic encryption to privacy-preserving signal processing. For those of you who have no interest in signal processing, please stay tuned for Part 3 of the guide, which should be useful to a more general audience.</p><p>The goal of this post is twofold: 1) demonstrating that signal processing in the encrypted domain can actually be quite practical and 2) providing an overview of some of the code written for Professor Gerald Penn and my paper for Interspeech 2019 titled “<a href="https://www.isca-speech.org/archive/Interspeech_2019/pdfs/1136.pdf">Extracting Mel-Frequency and Bark-Frequency Cepstral Coefficients from Encrypted Signals</a>.”</p><p><strong>Real Discrete Fourier Transform</strong></p><p>The Fourier Transform (FT) is one of the most fundamental operations in signal processing. It takes a signal from the time domain to the frequency domain. If you happen to need a primer to FT, one of the clearest explanations that I’ve found so far is in “<a href="https://books.google.ca/books/about/A_Student_s_Guide_to_Waves.html?id=ioeFBwAAQBAJ&amp;printsec=frontcover&amp;source=kp_read_button&amp;redir_esc=y#v=onepage&amp;q&amp;f=false">A Student’s Guide to Waves</a>.” If you can’t get your hands on that book, there exists a copious amount of descriptions online.</p><p>Now, for those who are already familiar with FT, you might be wondering how we can possibly convert the following formula to the encrypted domain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/0*D1HxNFx-Pvrno_nV.png" /><figcaption>Source: <a href="https://betterexplained.com/articles/an-interactive-guide-to-the-fourier-transform/">https://betterexplained.com/articles/an-interactive-guide-to-the-fourier-transform/</a></figcaption></figure><p>Well, we won’t be using the standard FT formula, since it would mean having to deal with the additional complexity of accounting for imaginary numbers within encrypted calculations. Rather, we will use the Real Discrete Fourier Transform’s (RDFT) formulae, which are very nicely described in [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*xSSp25EOX1RTE_Sen6m-cA.png" /><figcaption>Source: <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=1164632">https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=1164632</a></figcaption></figure><p>where <em>N</em> is the number of samples, <em>x(·)</em> is the vector representing the signal, and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*tkpgenrwKLc6vxOt0FqZdA.png" /><figcaption>Source: <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=1164632">https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=1164632</a></figcaption></figure><p>To calculate the RDFT, we add the Discrete Cosine Transform (DCT) with the Discrete Sine Transform (DST). For simplicity, we refer to the encrypted version of values ∗ as E(∗).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*g7pspAd8XlBoTkSaaZidwg.png" /></figure><p>Luckily for us, <em>cos(2πnk/N)</em> and <em>sin(2πmk/N)</em> can be precomputed and do not need to be encrypted prior to multiplying the values with the encrypted signal. We prepare these values for use with the B/FV homomorphic encryption scheme from the <a href="https://git.njit.edu/palisade/PALISADE">PALISADE Lattice Cryptography Library</a>. Note that we do need to choose a precision at which to round the values by which we will multiply our signal. We choose the number of decimal places to be retained based on a desired level of accuracy φ, then multiply the data by 10^φ (we use ^ to mean “to-the-power-of”) and round to the nearest integer.</p><p>Let’s work out the precomputations first. For the DCT, we have to calculate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*OnzmcnQklMPma9y_tGwRjA.png" /></figure><p>We code up the highlighted section as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b47998248a0cb4d724a757a970a6036c/href">https://medium.com/media/b47998248a0cb4d724a757a970a6036c/href</a></iframe><p>We perform almost identical calculations for the DST.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*unqYz3YVX9A3LoXKYCpvbQ.png" /></figure><p>We code up the highlighted section as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e78de3bd69fd20f935e0e8a6c82d790/href">https://medium.com/media/9e78de3bd69fd20f935e0e8a6c82d790/href</a></iframe><p>I hope that these precomputations are fairly self-explanatory.</p><p>Next, we need to encrypt the signal. For our experiments, we used an audio recording of a telephone conversation from the Switchboard corpus. The kind of signal is only relevant here insofar as knowing the sample rate, which is 8kHz. In case we ever need to perform division, we create a fraction struct to store encrypted numbers. While this won’t come in handy for the RDFT calculations, it is necessary for calculating Cepstral Coefficients (which is the topic of the paper this code was originally written for).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9871eaa556e35a83b7f46a0cdc54ab8a/href">https://medium.com/media/9871eaa556e35a83b7f46a0cdc54ab8a/href</a></iframe><p>We encrypt the signal as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fcdf6bb096273f7905d31cd438b8548/href">https://medium.com/media/5fcdf6bb096273f7905d31cd438b8548/href</a></iframe><p>Now we finally have the tools necessary to calculate the RDFT:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b7f5faf278358e1a96e7dacc1f789e9/href">https://medium.com/media/7b7f5faf278358e1a96e7dacc1f789e9/href</a></iframe><p><strong>Performance</strong></p><p>For a 128-bit security level, it takes 46.5235 s to calculate the RDFTs of 100 frames, given a sample rate of 16, a frame length of 25, and a shift length of 10 on an Intel Core i-7–8650U CPU @ 1.90GHz and a 16GB RAM.</p><p>These computations are, of course, parallelizable. If we were to run this code on a more powerful device and were even to adapt it to GPUs, it might become quite practical to perform encrypted FTs on the cloud.</p><p><strong>What can we use this for?</strong></p><p>Often, healthcare is the first sector that comes to mind when discussing sensitive data collection and processing. Audio recordings of a patient’s voice can actually contain a lot of information that’s useful for diagnosing disease (shout out to <a href="https://winterlightlabs.com/">WinterLight Labs</a>, <a href="http://www.cs.toronto.edu/~frank/">Professor Frank Rudzicz</a>, and <a href="http://www.cs.toronto.edu/~kfraser/">Dr. Katie Fraser</a> who have done great work on using speech processing to detect cognitive impairments like dementia).</p><p>One method for detecting pathological voices is to calculate jitter, which measures whether a gesture is sustained over a short period of time [2].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*CyITOvFaX0RW0vZccSZuBw.png" /></figure><p>T_i ( _ used to mean subscript) are the extracted fundamental frequency (F_0) period lengths and N is the number of extracted F_0 periods [3].</p><p>Aside from speech, images are another type of signal which can contain ample sensitive information. The Fourier Transform is often used in <a href="https://homepages.inf.ed.ac.uk/rbf/HIPR2/fourier.htm">image processing</a> (e.g., for image filtering).</p><p><strong>Acknowledgements</strong></p><p>This work was supported by an RBC Graduate Fellowship and the BRAIN Ontario Research Fund.</p><p>Feel free to email me at pthaineATcsDOTutorontoDOTedu if you would like to chat about encrypted signal processing.</p><h3>References</h3><p>[1] O. Ersoy, “Real Discrete Fourier Transform,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 33, no. 4, pp. 880–882, 1985.</p><p>[2] P. Perrot and G. Chollet, “Helping the Forensic Research Institute of the French Gendarmerie to Identify a Suspect in the Presence of Voice Disguise or Voice Forgery,” in Forensic Speaker Recognition. Springer, 2012, pp. 469–503.</p><p>[3] M. Farrus, J. Hernando, and P. Ejarque, “Jitter and Shimmer Measurements for Speaker Recognition,” in Eighth Annual Conference<br>ofthe International Speech Communication Association, 2007.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77bcaf9a1756" width="1" height="1" alt=""><hr><p><a href="https://medium.com/privacy-preserving-natural-language-processing/homomorphic-encryption-for-beginners-a-practical-guide-part-2-the-fourier-transform-77bcaf9a1756">Homomorphic Encryption for Beginners: A Practical Guide (Part 2: The Fourier Transform)</a> was originally published in <a href="https://medium.com/privacy-preserving-natural-language-processing">Privacy-Preserving Natural Language Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differentially Private Natural Language Processing]]></title>
            <link>https://medium.com/privacy-preserving-natural-language-processing/differentially-private-natural-language-processing-4f18912c5de0?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f18912c5de0</guid>
            <category><![CDATA[differential-privacy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[privacy-by-design]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Mon, 28 Jan 2019 11:01:01 GMT</pubDate>
            <atom:updated>2019-01-28T12:21:33.078Z</atom:updated>
            <content:encoded><![CDATA[<p>A conversation with Dr. Parinaz Sobhani on reverse engineering data and on integrating differential privacy into machine learning models</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_jJN_so3o1rsRWhQ-cYsbQ.jpeg" /></figure><p><a href="https://www.linkedin.com/in/parinaz-sobhani-51694262/"><em>Dr. Parinaz Sobhani</em></a><em> is the Director of Machine Learning on the Georgian Impact team and is responsible for leading the development of cutting-edge machine learning solutions. Dr. Sobhani has more than 10 years of experience developing and designing new models and algorithms for various artificial intelligence tasks. Prior to joining Georgian Partners, she worked at Microsoft Research where she developed end-to-end neural machine translation models. Previous to this she worked for the National Research Council in Canada, where she designed and developed deep neural network models for natural language understanding and sentiment analysis. She holds a Ph.D. in machine learning and natural language processing from the University of Ottawa with a research focus on opinion mining in social media.</em></p><p><strong>Patricia: </strong>Thank you very much for agreeing to be interviewed.</p><p><strong>Dr. Sobhani: </strong>You’re very welcome.</p><p><strong>Patricia: </strong>How about we start with the basic question of why we should be interested in privacy-preserving natural language processing in the first place?</p><p><strong>Dr. Sobhani: </strong>That’s a very good question. For the very same reasons that we should be interested in privacy-preserving AI in general. One of the biggest concerns of using machine learning for applications that make use sensitive data is privacy. It isn’t a surprise to us that machine learning models are very good at memorizing training data. And in NLP, because we often make use of generative models, the risk of being able to predict the training data from the model’s outputs is quite high. Any data used to train language models and machine translation models can be reverse engineered, basically giving users of those models access to a part of the training data. If we are using a machine learning model for healthcare diagnoses, for example, we don’t want the model to memorize any of the individual patients’ information that it was built with. Our ultimate goal is to extract general patterns rather than specific facts about individuals. But machine learning models have the proven ability to memorize individual data points that can be simply reverse engineered by adversaries using malicious inputs.</p><p><strong>Patricia: </strong>How does differential privacy fit into making data processing more secure?</p><p><strong>Dr. Sobhani: </strong>Differential privacy can be used to make machine learning models private. One of the most popular approaches in the literature is introduced in “<a href="https://arxiv.org/pdf/1607.00133.pdf">Deep Learning with Differential Privacy</a>.” It is applicable to any learning approach that is based on stochastic gradient descent. Some of the other approaches might have more constraints. For example, we used “<a href="https://arxiv.org/pdf/1606.04722.pdf">Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics</a>” in one of our papers (“<a href="https://arxiv.org/pdf/1811.04911.pdf">Boosting Model Performance through Differentially Private model Aggregation</a>”) which is only applicable to convex optimization. Because we are using these machine learning algorithms for NLP applications, we can apply the same kind of approaches to make them private. My hope is that we will see more research in this area, because all of the differential privacy papers I’ve seen so far have been based on processing structured data or image data. Like text and speech, image data are also unstructured. However, the models used to process them are different and the privacy protection challenges are different.</p><p><strong>Patricia: </strong>What other approaches can be used to make data processing more secure?</p><p><strong>Dr. Sobhani: </strong>I am encouraged to see so many other technologies that have been developed to make privacy-preserving data mining a possibility. One such approach is de-identification. Khaled El Emam and his research group have been working on this problem for several years. But still, we don’t know all the limitations of de-identification methods. Normally, if you want to use de-identification methods for NLP tasks, multiple steps need to be addressed. You need to extract identifiers from text, which is not easy. For example, you will want to remove all addresses, names, and so on. The first step for this is to use a named entity recognizer. This in itself is a difficult task. The best named entity recognizers are around 90% accurate. So you have a 10% error rate and we are normally evaluating these named entity recognizers by comparing them with human annotated text. We don’t have 100% inter-annotator agreement for this sort of text annotation. There are all of these sources of noise when it comes to evaluating named entity recognition systems and, depending on the nature of the text, recognizing named entities might be really difficult. For example, in the case of clinical trial reports or diagnosis reports, the grammatical structure might vary, there might be typos, word repetitions, or other problems with the text.<br>After you have extracted the named entities from a text, you have more structured data to deal with. Then you can apply de-identification algorithms. At the end of the day, when you are using de-identification algorithms, it is based on certain assumptions, including the viability of external datasets. It is so hard to maintain any initial assumptions. Everything is changing and there is so much information out there that we don’t know how to leverage yet and how to connect with the data made accessible to malicious parties through data leaks. Verizon, for example, recently published a <a href="http://www.verizonenterprise.com/resources/protected_health_information_data_breach_report_en_xg.pdf">report analyzing medical data breaches in 27 countries</a>, which revealed that there have been more than 500 data breach incidents leaking medical records. Leaked information can be cross-referenced with public datasets.<br>De-identification or anonymization that masks what we consider to be personally identifiable information might not be sufficient and does not provide any privacy guarantees without certain assumptions about the availability of external and auxiliary sources of information. Whereas, techniques such as homomorphic encryption and differential privacy provide more formal guarantees with strong mathematical foundations. Privacy guarantees garner the trust of individuals leading them to more willingly share data and, in turn, to benefit from the algorithmic improvements that personalization provides.</p><p><strong>Patricia: </strong>So, in order to be able to use differential privacy you have to know what task you want to do beforehand; whereas, theoretically, data de-identification would allow for datasets of unstructured text to be made public without knowing which tasks the data will be used for. Do you think that the best approach would be for researchers and developers to propose differentially private algorithms to the owners of the datasets they’d like to train their models on and hope the owners agree? Or is there some more practical way to go about training models on data that aren’t publicly available?</p><p><strong>Dr. Sobhani: </strong>That’s a very good question. The main benefit of differential privacy is that it gives you mathematical guarantees about privacy. But of course it has its own limitations as well. It is a probabilistic framework to measure the level or privacy of a mechanism (or an algorithm/function) that uses data to provide an answer based on some computation on that data. The main principle of preserving privacy is to introduce randomness to the function so that the final answer does not depend on any individual data points. In the case of preserving privacy for machine learning models, any learned parameters (weights) of the final system should not have a huge dependency to presence or absence of any individual user’s data that was leveraged to train the model. We can choose to add noise to the data, to the model, or to the output of the model. Which one we choose to add noise to, how we add the noise, and how much noise we add all depends on the type of function that we are interested in using. There has been some research on adding noise to the input data. In that case, the magnitude of the noise has to be higher, because your assumption is going to be that the data can be fed into <em>any</em> function. What I have seen in the literature so far is that that kills the utility of the data. An interesting alternative was presented in a paper called “<a href="https://openreview.net/pdf?id=rJv4XWZA-">Generating Differentially Private Datasets Using GANs</a>,” where the authors trained a generative model on their input data, then made that generative model differentially private, generated data using that model and then made that generated data publicly available. I’m a bit skeptical about this approach because I believe we’d need lots and lots of data to be able to train the generative model to produce useful output data. Also, choosing the right magnitude for the noise is not an insignificant task. The quality of the output data is certainly not going to be the same as the quality of the original dataset. If we were to try to use this method for healthcare applications, we might not have enough input data made available to us in order to be able to build a reliable generative model. But on paper, it is at least theoretically possible to use differential privacy to create a useful privacy-preserving public dataset.</p><p><strong>Patricia: </strong>So they are basically creating a synthetic dataset.</p><p><strong>Dr. Sobhani:</strong> Exactly. Based on applying differential privacy to a generative model.</p><p><strong>Patricia: </strong>Okay. And they didn’t provide evaluations of the quality of the output data?</p><p><strong>Dr. Sobhani: </strong>Well, in the paper, they apply their method to image data and then used their own metrics to measure the quality of the synthesized images.</p><p><strong>Patricia: </strong>Riveting! If a researcher or developer wants to start using differential privacy, where should they start?</p><p><strong>Dr. Sobhani: </strong>I’d recommend taking a look at a paper published in SIGMOD 2017 called “<a href="https://dl.acm.org/citation.cfm?doid=3035918.3054779">Differential Privacy in the Wild: A Tutorial on Current Practices &amp; Open Challenges</a>.” There’s also a blog post by a couple of the best researchers in the area (Nicolas Papernot and Ian Goodfellow) titled “<a href="http://www.cleverhans.io/privacy/2018/04/29/privacy-and-machine-learning.html">Privacy and machine learning: two unexpected allies?</a>” And I’d definitely recommend “<a href="https://www.microsoft.com/en-us/research/publication/the-promise-of-differential-privacy-a-tutorial-on-algorithmic-techniques/?from=http%3A%2F%2Fresearch.microsoft.com%2Fapps%2Fpubs%2Fdefault.aspx%3Fid%3D155617">The Promise of Differential Privacy. A Tutorial on Algorithmic Techniques</a>” by Cynthia Dwork, who is famously known for being the co-inventor of differential privacy. With regards to coding, I am so encouraged to see many open source packages available for differential privacy. Researchers working in this area started to publish their code along with their papers. <a href="https://github.com/tensorflow/privacy">One very good package has been published by Google for Tensorflow</a>. It contains a comprehensive number of mechanisms that can be applied to various deep learning methods and other machine learning methods. When we started working in this area, there weren’t as many open source packages. We identified one paper whose method we wanted to use, we contacted the authors, and they told us that they couldn’t make the code publicly available. We had to write our own code and it wasn’t easy. When we talk about ε-differential privacy, ε is actually the measure of the privacy level of the algorithm. You can think of it similarly to accuracy or F1-score, which are metrics used to measure the performance of a machine learning model. It’s much easier to test how well a machine learning model performs than to check the privacy level of the model. What I means is that there isn’t a list of approaches that you can use to verify the privacy level of your model and get a stamp of approval. Bugs in your code can also make your model non-private in practice, when all the math points to it being privacy-preserving. Or there might be a number of requirements that need to be met in order to have all of the required privacy guarantees and it’s so easy to miss one of them. One of our struggles was understanding all of the requirements. It took us a while to determine whether the privacy was provably guaranteed. That is why I am so encouraged by the new open source packages, all of the new papers coming out, and even the workshops that are focused solely on differential privacy. Even if a newcomer doesn’t manage to publish their paper in one of the main conferences, they still have an opportunity to connect with the community through one of these workshops and discuss their work.</p><p><strong>Patricia: </strong>Do the packages that are available give you an understanding of what the privacy guarantee is based on the amount of noise that you insert at different points in your model, or do you have to calculate that yourself?</p><p><strong>Dr. Sobhani: </strong>The way we calculate the privacy level is based on ε and ε is the input to your algorithm. There are some standards for selecting ε. For example, people often make ε equal to 2. You can use this value as an input in these packages to get a model that has 2ε-differential privacy.</p><p><strong>Patricia: </strong>Which pitfalls would you recommend keeping an eye out for when you are using differential privacy for machine learning?</p><p><strong>Dr. Sobhani: </strong>One<strong> </strong>of the biggest challenges I’ve seen people have was actually due to poor communication. Because of the mathematical foundations of differential privacy and because the privacy guarantees are probabilistic, it is harder to communicate the applications and functionality to people with non-technical backgrounds. I’ve often been in discussions with developers who were having a difficult time conveying how this technology works to their company’s customers. It isn’t easy to understand. There should be more work done on simplifying the language and on making the concepts comprehensible to professionals with very different backgrounds.</p><p><strong>Patricia: </strong>What would be a solution to this communication problem?</p><p><strong>Dr. Sobhani: </strong>Educating product managers would be key. Product managers are the ones who are usually responsible for communicating with customers. Scientists who work on differential privacy could describe its mathematical foundations, its privacy guarantees, and their own research to product managers without getting into too many details about the math. One way of doing this is by using concrete examples. Also, as we see more examples of differentially private approaches in production, like the ones Google, Apple, and Amazon have in production right now, it will help increase the levels of trust in these technologies. Because at the beginning of any technological adoption, people are normally skeptical about whether it works and they aren’t too sure about what it could be used for. On top of making it easier to understand the concepts, it is our responsibility as scientists to develop testing frameworks for differential privacy. That’s another challenge, because it’s hard to actually measure the privacy guarantees after you’ve shipped your model.</p><p><strong>Patricia:</strong> Are there any papers that discuss privacy frameworks?</p><p><strong>Dr. Sobhani:</strong> I know that a couple of researchers at the University of Toronto have started working in this research area, including Aleksander Nikolov. They know it’s a gap that might limit the applications of differential privacy and that we have to find ways to fill it as soon as possible. There is also a recent publication addressing this issue called “<a href="http://www.cse.psu.edu/~dbz5017/pub/ccs18.pdf">Detecting Violations of Differential Privacy</a>” by researchers from Pennsylvania State University.</p><p><strong>Patricia:</strong> Given what you know about the pitfalls and the challenges that you’ve faced and given the tasks that you are aware of that could use differential privacy for privacy preservation, what wish list do you have for the research community working on applied differential privacy?</p><p><strong>Dr. Sobhani: </strong>It would be great if there were more of a focus on the NLP side of things. Right now, most of the papers have used MNIST for their experiments. My expectation is that the community will start to use more diverse datasets and more real-world datasets where they have to deal with the challenges faced by industry. For example, more imbalanced datasets, sparse datasets, or unstructure data that is more complex than MNIST. I expect to see more realistic experiments.</p><p><strong>Patricia: </strong>And which venues would you recommend looking at if you’re interested in reading papers about differential privacy applied to machine learning or if you’re interested in publishing on the topic of differential privacy applied to natural language processing?</p><p><strong>Dr. Sobhani: </strong>There’s the Privacy Preserving Machine Learning workshop at NeurIPS, the IEEE Symposium on Security and Privacy, and the AAAI Symposium on Privacy-Enhancing Artificial Intelligence and Language Technologies, to name a few.</p><p><strong>Patricia: </strong>Thank you so much for your time, Parinaz.</p><p><strong>Dr. Sobhani: </strong>My pleasure.</p><p>You can follow Dr. Parinaz Sobhani on Twitter <a href="https://twitter.com/PariAIML">@PariAIML</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f18912c5de0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/privacy-preserving-natural-language-processing/differentially-private-natural-language-processing-4f18912c5de0">Differentially Private Natural Language Processing</a> was originally published in <a href="https://medium.com/privacy-preserving-natural-language-processing">Privacy-Preserving Natural Language Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which privacy-preserving method should I use??]]></title>
            <link>https://medium.com/privacy-preserving-natural-language-processing/which-privacy-preserving-method-should-i-use-b9f0dc41cc46?source=rss-afe93926edf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9f0dc41cc46</guid>
            <category><![CDATA[privacy-by-design]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patricia Thaine]]></dc:creator>
            <pubDate>Fri, 04 Jan 2019 21:07:08 GMT</pubDate>
            <atom:updated>2019-01-04T21:27:23.551Z</atom:updated>
            <content:encoded><![CDATA[<h4>Which privacy-preserving method should I use?? A tentative decision tree for the privacy-conscious programmer.</h4><p>After getting asked over and over again about which privacy-preserving methods to use in which situations, I decided to make a decision tree to help researchers, architects, and programmers with the process of creating privacy-preserving algorithms.</p><p>This tree is still tentative, and I have yet to incorporate certain excellent suggestions I’ve been given (e.g., Prof. Sergey Gorbunov pointed out that the topmost nodes should ask about expected threat models).</p><p>I welcome community input and suggestions for improvements. Of course, any suggestions that are integrated into future iterations will be appropriately credited.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gAdwWoADvl-354oKAeMRxw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9f0dc41cc46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/privacy-preserving-natural-language-processing/which-privacy-preserving-method-should-i-use-b9f0dc41cc46">Which privacy-preserving method should I use??</a> was originally published in <a href="https://medium.com/privacy-preserving-natural-language-processing">Privacy-Preserving Natural Language Processing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>