<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yasmine Djebrouni on Medium]]></title>
        <description><![CDATA[Stories by Yasmine Djebrouni on Medium]]></description>
        <link>https://medium.com/@yasminedj?source=rss-fe8f5532ccbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Yasmine Djebrouni on Medium</title>
            <link>https://medium.com/@yasminedj?source=rss-fe8f5532ccbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:01:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yasminedj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Law of Large Numbers is Wise]]></title>
            <link>https://medium.com/@yasminedj/the-law-of-large-numbers-is-wise-d4304be8b727?source=rss-fe8f5532ccbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4304be8b727</guid>
            <category><![CDATA[analogy]]></category>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[metaphor]]></category>
            <category><![CDATA[motivation]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Yasmine Djebrouni]]></dc:creator>
            <pubDate>Mon, 04 Sep 2023 07:30:41 GMT</pubDate>
            <atom:updated>2025-07-31T10:52:57.520Z</atom:updated>
            <content:encoded><![CDATA[<h4>Contemplating the connection between the Law of Large Numbers (LLN) and the concept of perseverance.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yg-laG_BvTFebDxJ.jpg" /><figcaption>A large number of migrating snow geese (Chen caerulescens) fly over a frozen lake at sunrise in Lancaster County, Pennsylvania, USA</figcaption></figure><p>The Law of Large Numbers (LLN) is a big idea in statistics. It tells us that when we gather a lot of data, the average of the numbers we see start to match what we expect, like the average of all the numbers. This reminds us that having a big amount of data is really important for making sure our conclusions are accurate and trustworthy.</p><p>In simple math terms, LLN means that if you have a sample of random numbers with a finite average, and you keep collecting more and more of them, their average will be super close to the true average of all the numbers. The Law of Large Numbers (LLN) can be mathematically formulated as follows:</p><p>Let X₁, X₂, X₃, … be a sequence of independent and identically distributed random variables with finite population mean (expected value) μ and finite population variance σ². Then, the Law of Large Numbers states that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*4Y8nWqrC86jI3uV80Zj-mw.png" /></figure><p>In this formulation, The expression “a.s.” on the left arrow denotes almost sure convergence, which means that the sample mean converges to the population mean with probability 1 as the sample size (n) goes to infinity</p><p>Interestingly, there’s a connection between these ideas and the enduring strength of perseverance. Perseverance means never giving up, even when you face difficulties. It’s like trying over and over again because the more you try, the closer you get to what you want. Now, let’s explore how this relates to the Law of Large Numbers.</p><h4><strong>Doing Things Over and Over</strong></h4><p>The LLN emphasizes the power of repetition and iteration. With each trial, the sample grows, providing a more comprehensive representation of the underlying phenomenon. Similarly, when you try something multiple times, stay disciplined, and don’t give up, you learn and get better at it. The more you practice, the better you get at what you do.</p><h4>Finding Your Potential</h4><p>Just as statistical convergence reveals the true nature of a phenomenon, perseverance helps you uncover your full potential. If you keep trying and never quit, you can achieve your goals.</p><h4>Failures can happen</h4><p>LLN knows that not every trial will yield the expected result. The same goes for life — you’ll face setbacks and failures.</p><h4>Dealing with Surprises</h4><p>LLN works better with bigger sets of data because it’s more accurate and it mitigates the effects of randomness, paralleling your ability to handle unexpected challenges when you persist. It’s akin to building resilience against surprises.</p><h4>Contrast with LLN</h4><p>The difference between LLN and perseverance is interesting.</p><p>LLN is all about math and probabilities. It deals with separate trials and how they eventually get very close to a specific expected value. It’s like predicting outcomes in a controlled math world.</p><p>On the other hand, perseverance is about the real world and how people keep going in the face of challenges. It’s not about separate, isolated efforts but about how each step connects to the next in a continuous journey. Success in perseverance isn’t just about numbers; it’s about a person’s determination, ability to adapt and learn from mistakes.</p><p>LLN makes us think things will become more predictable as we collect more data, like knowing the exact outcome. But in the real world, perseverance thrives on the unknown. It understands that the path to success isn’t a smooth line; it’s more like a winding road with unexpected twists and turns.</p><p>LLN is all about numbers, while perseverance is about the humans.</p><p>So, as we reflect on the analogy between LLN and perseverance, let’s not lose sight of this fundamental contrast. The contrast between the predictability of numbers and the unpredictability of human potential, between mathematical models and the beautiful chaos of life.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4304be8b727" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fairness in Machine Learning: Why Machine Learning Needs Good Role Models]]></title>
            <link>https://medium.com/@yasminedj/machine-learning-unfairness-and-kids-whats-the-story-85c11bc34c62?source=rss-fe8f5532ccbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/85c11bc34c62</guid>
            <category><![CDATA[bias-in-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bias]]></category>
            <dc:creator><![CDATA[Yasmine Djebrouni]]></dc:creator>
            <pubDate>Tue, 18 Oct 2022 06:57:25 GMT</pubDate>
            <atom:updated>2023-03-14T18:57:13.989Z</atom:updated>
            <content:encoded><![CDATA[<h4>Machine learning can go mean, racist, and sexist...</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_vYmMmQenersEAPhcvHvPQ.jpeg" /></figure><p>Even before they speak their first words, human babies develop mental models about people and objects. As a result of this ability, humans can live socially, cooperate (or compete) with others, and interact with the environment.</p><p>Babies become kids. As they grow up, they begin to use the learned models to speak and act on their own. Sometimes they are out of control, say mean things, make bad decisions at their level, and cause harm to others.</p><p>The same goes for Machine Learning (ML) models.</p><h4>How Do ML Models Learn to Be Biased?</h4><p>Today, machine learning is used everywhere in our daily lives. In the healthcare industry, banking, recruiting, recommendation systems, and so on, it’s used a lot. The reason for this is that machine learning is a compelling technology for automating decision-making.</p><p>By learning patterns based on data collected from our reality, machine learning can predict the future and guide decision-making.</p><p>There is, however, a problem.</p><p>As I mentioned, machine learning uses data collected from the real world to learn.<strong> </strong>Neither data nor reality are perfect. There are several flaws and biases in them:</p><ul><li>The sampling and collection of data are not fair; some populations are underrepresented, such as women in STEM-related data =&gt; <em>Sampling bias.</em></li><li>We label our data in an unfair and subjective manner. What’s funny or beautiful for the person who labels the data is not funny or beautiful for others =&gt; <em>Label bias</em>.</li><li>As a result of historical and social prejudices, our data reflect a biased reality against certain groups of people because of their color, gender, religion, etc. =&gt; <em>Prejudicial bias</em>.</li></ul><p>The list of possible biases present in data is even longer.</p><blockquote>Because it reflects the messed up reality, data is messed up.</blockquote><p>And guess what! In a way, machine learning algorithms are like kids. They will eventually go out of control if we don’t control where they learn from.</p><p>When we feed biased data to an ML model, the latter will learn to be biased. Once confronted with new data, the ML model’s decisions will be unfair, sexist, racist, etc. Take, for example, the task of accepting/rejecting people’s job applications using ML. If the ML Model is trained based on some unfair existing data, where people from some minority groups were never accepted (because the recruiting procedure was unfair and biased), the machine learning model will reject similar applications in the future.</p><h4>Examples of Real-World Biased ML Models</h4><p>It has been reported that real-life ML has been unethical and unfair in several instances. Amazon’s ML recruiting tool has been found to be sexist [1]. Some facial recognition systems fail to recognize the faces of certain groups [2]. As a final example, machine learning was used by the US justice system to send people to jail, and their ML model did so based on the individuals&#39; race [3].</p><h4>The Impact of Biased ML Models on Society</h4><p>Today, ML is used in critical and sensitive domains, to hire people, grant loans, detect diseases, etc. Thus, the fairness of machine learning models is a significant issue for societies. By learning bias from training data, unfair models generalize it to new individuals and perpetuate prejudice. By using unfair machine learning models, the already existing gap between majorities and minorities is widened further. The unprivileged groups will continue to live unprivileged lives because of traits like race, gender, religion, etc., which is against moral/ethical values.</p><p>We must ensure that machine learning models are fair before deploying them for a better and fairer society. But to do so, someone must define rigorously what’s fair and what’s not.</p><p>In the next article, I will write about the different fairness notions considered in machine learning literature. See you soon.</p><h3>References</h3><p>[1] Dastin, Jeffrey (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Available online: <a href="https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G">https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G</a> [Accessed 17/10/ 2022].</p><p>[2] Najibi, Alex (2020). Racial Discrimination in Face Recognition Technology. Available online: <a href="https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/">https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/</a> [Accessed 17/10/ 2022].</p><p>[3] Angwin, Julia. Larson, Jeff (2016). Machine Bias. Available online: <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing</a> [Accessed 17/10/ 2022].</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85c11bc34c62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simulation-to-Reality Gap in Federated Learning — Part 2]]></title>
            <link>https://medium.com/@yasminedj/simulation-to-reality-gap-in-federated-learning-part-2-cbd1d2b419c2?source=rss-fe8f5532ccbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbd1d2b419c2</guid>
            <dc:creator><![CDATA[Yasmine Djebrouni]]></dc:creator>
            <pubDate>Tue, 12 Apr 2022 10:15:10 GMT</pubDate>
            <atom:updated>2023-02-27T13:54:41.270Z</atom:updated>
            <content:encoded><![CDATA[<h3>Simulation-to-Reality Gap in Federated Learning — Part 2</h3><h4>Training, validation, and test in federated learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*uIBTDML8XCEXTC2L.jpg" /></figure><p>In the first part of this series (<a href="https://medium.com/@yasminedj/simulation-to-reality-gap-in-federated-learning-6a415e4fd42e">here</a>), I discussed the simulation-to-reality gap in federated learning (FL) and its causes. I explained how FL is simulated by researchers and raised some questions about how FL is done in real production systems. In this part, I try to answer the raised question by explaining how training, validation, and testing are implemented in real commercial systems today. I note that I consider a standard federated learning system with one central server.</p><p>If you are very new to federated learning, I have something for you: A fairy tail about the emergence of FL :)</p><p><a href="https://medium.com/@yasminedj98/what-is-federated-learning-fl-motivation-and-concept-fd03d7b06a2d">What Is Federated Learning (FL)? Plot Twist: A Fairy Tale</a></p><h3>Federated Learning in Real Life</h3><p>Similar to classical machine learning, federated learning consists of several phases, namely a training phase, a validation phase, and a testing phase. However, in FL, the data is distributed on different devices, and the learning process is distributed among the different data owners, called also FL clients.</p><p>During the training phase, clients contribute to the global goal by learning local models trained on their local data. The validation process takes place during training to validate the architecture of the trained model. Finally, the testing phase verifies the performance of the model before final deployment to the clients. FL is communication efficient, as learning data is not transmitted over the network during the FL process, but also privacy friendly, as the clients do not send their training data, which may contain private information, to other clients or to the server.</p><h3>Training Phase</h3><p>The training process in real FL systems takes place through several rounds under the direction of a central server. The clients may be referred to as clients, devices, or data owners. They can be mobile devices, banks, or even hospitals. Each client holds a set of private labeled data. Data is either labeled according to the activity of the owner, e.g: clicks history in mobile data, or by requesting the client to manually label his data, e.g: a hospital that labels its patients&#39; instances according to their medical records.</p><p>Training begins when an FL developer sends a training job to the server. The training job consists of the model to be trained, the initial hyper-parameters and their values, and some selection criteria for the clients. The selection criteria are used by the server to select the clients to participate in the training from those who are available on the network. FL developers can mention several criteria. Examples of criteria when the clients are mobile devices are a high battery level and an un-metered network connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*01aUeCLyj9q8disC-w4dKQ.png" /><figcaption>One round of federated learning training</figcaption></figure><p>At the beginning of each round, the server selects the clients that will participate in that learning round. The server sends the current version of the global model to the selected clients. In the first round, the server sends a randomly initialized model or a pre-trained model. Then, each participating client trains the received model on its local data and sends its local model updates to the server. Finally, the server performs a weighted aggregation of the received clients’ model updates through FedAvg[3] or other aggregation methods, to produce a new version of the global model. The process continues for several rounds until the global model converges. The process is shown in the following figure.</p><h3>Validation and Tuning</h3><p>Hyper-parameter tuning is similar to centralized training, but FL additionally involves 1) the number of training rounds and 2) the number of local iterations, which may cost lots of resources (communication bandwidth, computing power, energy, etc.). <a href="https://www.automl.org/">AutoML</a> is a tool that can reduce the number of attempts of hyper-parameter tuning. A common practice in FL hyper-parameter tuning according to FL developers is to keep observing the training curve for a given hyper-parameters setting, and terminate the training as soon as the accuracy gets worse than previous runs, without the need to wait for the end of the training.</p><p>Hyper-parameter tuning can thus be done during model training. Every few rounds, proxy data stored on the server as validation data can be used to validate the model architecture and select values for basic hyper-parameters [4]. The proxy data can be data synthetically generated using private data generation methods or a collection of donated instances of real-world data owners. The validation data, other than tuning hyper-parameters, can serve to monitor the learning process when the training is long. One can validate/test its model performance against some validation data during training rounds, without having to wait for the training to end.</p><h3>Test Phase</h3><p>In this phase, the performance of the global model is tested against new data that it has never seen before, to measure how well it can generalize.</p><p>One common case is when the FL service provider has a global “public” test dataset that doesn’t contain any private information. It can be a collection of donated data or data synthetically generated by existing private data generation techniques. The evaluation in this scenario can run on the server side and its results analysed by the FL developers.</p><p>In other cases, FL developers need to test the performance of the model on real data sets, i.e., real edge devices or clients. This may be the case to validate the trained model before deployment, as in [4, 5], or to circumvent biased/non-representative proxy data used in training and validation, as described in [6]. To do so, the server selects a group of clients and sends them the final global model to run the test phase with their real data. After running the test, the clients send the performance of the final model on their data to the server, where the developers can analyse it. When the test is run on the clients, it is usually called: Live Test, Live Inference or On-Device Inference.</p><p>Another way of testing is fully decentralized testing, where the test happens on the clients, without any interference from the server. Actually, an organization normally migrates its ML training pipeline from centralized training on a single data silo (local dataset) to a federated manner that incorporates many other data silos. This allows the organization to improve its own model. In such a case, they already have the test datasets. So they just want to see if FL helps to improve the accuracy or generalization ability. The evaluation is run at the local client/organization side and, in contrast to previous cases, nothing is communicated to the server. Fully decentralized testing is also used in a more complex setting: Personalized FL, where again each client has its own test dataset. In that case, clients intentionally fine-tune the received global model to fit their data. Such personalized models will be tested against the local test data of the clients.</p><p>Here is the end of this series. I will be very happy to discuss anything from what I said in the comments section :)!</p><h3>References</h3><p>[1] Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., … &amp; Ramage, D. (2018). Federated learning for mobile keyboard prediction. <em>arXiv preprint arXiv:1811.03604</em>.</p><p>[2] Lai, F., Zhu, X., Madhyastha, H. V., &amp; Chowdhury, M. (2021). Oort: Efficient federated learning via guided participant selection. In <em>15th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 21)</em> (pp. 19–35).</p><p>[3] McMahan, B., Moore, E., Ramage, D., Hampson, S., &amp; y Arcas, B. A. (2017, April). Communication-efficient learning of deep networks from decentralized data. In <em>Artificial intelligence and statistics</em> (pp. 1273–1282). PMLR.</p><p>[4] Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., … &amp; Beaufays, F. (2018). Applied federated learning: Improving google keyboard query suggestions. <em>arXiv preprint arXiv:1812.02903</em>.</p><p>[5] Wang, K., Mathews, R., Kiddon, C., Eichner, H., Beaufays, F., &amp; Ramage, D. (2019). Federated evaluation of on-device personalization. <em>arXiv preprint arXiv:1910.10252</em>.</p><p>[6] Augenstein, S., McMahan, H. B., Ramage, D., Ramaswamy, S., Kairouz, P., Chen, M., &amp; Mathews, R. (2019). Generative models for effective ML on private, decentralized datasets. <em>arXiv preprint arXiv:1911.06679</em>.</p><h3>Useful Links</h3><p><a href="https://fedml.ai/">FedML | The Federated Learning/Analytics and Edge AI Platform</a></p><p><a href="https://doc.fedml.ai/">https://doc.fedml.ai/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbd1d2b419c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simulation-to-Reality Gap in Federated Learning — Part 1]]></title>
            <link>https://medium.com/@yasminedj/simulation-to-reality-gap-in-federated-learning-6a415e4fd42e?source=rss-fe8f5532ccbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a415e4fd42e</guid>
            <category><![CDATA[simulation]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[reality-gap]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[federated-learning]]></category>
            <dc:creator><![CDATA[Yasmine Djebrouni]]></dc:creator>
            <pubDate>Mon, 11 Apr 2022 18:10:24 GMT</pubDate>
            <atom:updated>2022-10-17T12:37:00.286Z</atom:updated>
            <content:encoded><![CDATA[<h3>Simulation-to-Reality Gap in Federated Learning — Part 1</h3><h4>A walk through some “existential” questions about training, validation, and test in federated learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*-dvOIOFSfn1RZi9y.jpg" /></figure><h3>Introduction</h3><p><strong>Simulation-to-reality gap</strong>, or reality gap, is a term that refers to the difficulty of transferring simulated experiences into the real world. It also refers to the difference between the promised potential of scientific and technical development and actual performance or use in practice. In Federated Learning (FL), this gap is significant. Moreover, many FL developers and researchers, especially beginners, get to know a lot about how to run an FL simulation, but not how to do real federated learning.</p><p>Before I go into more detail about the gap between simulation and reality in FL, you can learn about the concept of FL and its motivation at the following link.</p><p><a href="https://medium.com/@yasminedj98/what-is-federated-learning-fl-motivation-and-concept-fd03d7b06a2d">What Is Federated Learning (FL)? Plot Twist: A Fairy Tale</a></p><p>FL tutorials and research papers are becoming more widespread. However, most of them are based on simulation. Beginners stumble over FL simulations more than real settings, making the real FL less familiar and the simulation, shall I say, more simulated?</p><p>After reading a lot of FL works, and once I had to look more closely to set the hyper-parameters for my first FL model, train it, validate it, and test it all by myself, some FL “existential” questions popped into my head. I was a real beginner then, and I was not the only one asking questions which reassured me. After some research, I decided to share the questions and the answers I found for them in this article, for all FL beginners out there (how pretentious this seems?👀)</p><blockquote>The closer you look, the less you see</blockquote><p>A big part of the discrepancy between simulation and reality in FL is caused by the virtual deployment of FL in the simulation which is very different from reality: hundreds of heterogeneous embedded devices with non-IID (heterogeneous distributions of) data. This makes the simulation performance (in terms of convergence time, consumed bandwidth, model quality, etc) far from the real performance. In these series, however, I do not want to focus on the performance gap, but on the process gap. More specifically, I will focus on the way by which training, validation, and testing are performed in FL, in simulation vs. reality.</p><h3>Federated Learning in Simulation</h3><p>The reason why FL simulation is prevalent is mainly real FL systems are expensive to set up (up to hundreds of data owners, real network, geographic distribution, data and hardware heterogeneity, etc.). Only a number of papers, usually published by large companies like Google, deploy their FL process (or a part of it, like inference) on real production systems. An example of those works is [1] where the authors presented an application of FL for mobile keyboard predictions.</p><p>Let us put on the cape of an FL developer who relies on simulations to build an FL model. The first thing is setting up the data. The learning labeled data is all stored in one dataset. Some famous datasets used in ML research are also used in FL research, such as MNIST. Data is then distributed to shards in a non-IID way, to simulate the data heterogeneity in an FL environment. A shard represents one client&#39;s data, with all clients running on the same machine. To build validation and test datasets, either portions of the data are cut from the global dataset before being distributed to shards, or portions of the data are cut from each virtual client to be combined and form validation/test sets. The data remaining on the virtual clients represent the training data. Once the data is set up, training can start. After the training, the model performance is tested on the test set. Research papers lack details on hyper-parameter/model validation. At best, the hyper-parameters used for local training are mentioned.</p><p>Although the three phases, namely training, validation, and test, exist in FL, the majority of research papers focus on the training phase. Moreover, Research papers completely ignore the role of the FL developer who is supposed to monitor the training, validate the model and launch the test. [2] is an exception to these papers.</p><p>The process described above raises some intuitive questions that we can easily answer in the context of machine learning, but that are difficult to answer in the context of real FL.</p><ul><li>In real FL, data is already distributed and private across different clients. One cannot extract sets for validation and testing as in simulation. Who performs those actions in real FL systems, is it the server or the clients? and which data is used for the purpose?</li><li>In real FL, data labeling is a problem. It cannot be done manually as in FL since we do not have access to the training data. How is data labeled in supervised FL?</li><li>And finally, how is the tuning really done in real FL scenarios?</li></ul><p>In the next part of this article, I will try to answer the raised questions, relying on information and details I got from some FL developers and/or found in research articles published by Google, where the settings of real-life FL systems were very much taken into account.</p><blockquote>To be continued</blockquote><p>!EDIT! Don’t miss part 2, available from <a href="https://medium.com/@yasminedj/simulation-to-reality-gap-in-federated-learning-part-2-cbd1d2b419c2">here</a>!</p><h3>References</h3><p>[1] Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., … &amp; Ramage, D. (2018). Federated learning for mobile keyboard prediction. <em>arXiv preprint arXiv:1811.03604</em>.</p><p>[2] Lai, F., Zhu, X., Madhyastha, H. V., &amp; Chowdhury, M. (2021). Oort: Efficient federated learning via guided participant selection. In <em>15th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 21)</em> (pp. 19–35).</p><h3>Useful Links</h3><p><a href="https://fedml.ai/">FedML | The Federated Learning/Analytics and Edge AI Platform</a></p><p><a href="https://doc.fedml.ai/">https://doc.fedml.ai/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a415e4fd42e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is Federated Learning (FL)? Plot Twist: A Fairy Tale]]></title>
            <link>https://medium.com/@yasminedj/what-is-federated-learning-fl-motivation-and-concept-fd03d7b06a2d?source=rss-fe8f5532ccbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd03d7b06a2d</guid>
            <category><![CDATA[privacy-by-design]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[federated-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Yasmine Djebrouni]]></dc:creator>
            <pubDate>Fri, 14 Jan 2022 23:31:44 GMT</pubDate>
            <atom:updated>2022-10-17T12:31:35.265Z</atom:updated>
            <content:encoded><![CDATA[<h4>About Machine Learning (ML), Big Data, Distributed ML and FL</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P_U4O9rLsjP-3scYnGrwmw.jpeg" /></figure><blockquote>Once upon a time, there was only machine learning.</blockquote><h3><strong>From Machine Learning to Distributed Machine Learning</strong></h3><p><strong>Machine Learning (ML)</strong> is a set of methods used to extract knowledge from data and assist organizations in their decision-making processes, such as product recommendations for customers, disease diagnosis for patients, and object recognition for mobile device users. In its beginnings, machine learning was initially applied to small datasets that fit on single-node systems. In the last seven years, however, we started witnessing Big Data. Huge amounts of data have been generated and stored everywhere and in various forms, such as computer logs, social media accounts, health data, etc. These huge datasets, usually distributed across different locations and devices, hold great potential that can be discovered through ML techniques.</p><p>Machine learning is a data-driven technique. The more high-quality data that is fed into an ML system, the richer the knowledge source becomes and the more the system can learn. Machine learning is therefore ideal for exploiting the hidden patterns of Big Data. However, the size of Big Data has made the ML training phase impractical on traditional single-node platforms. Training times of ML models started to become very long. In some cases, a centralized solution was not even an option because the data was too large to be stored on a single machine. Some examples of such Big Data are transactional data of large companies stored in different locations [1], or astronomical data too large to be centralized on systems with only one node [2]. Even the use of accelerators such as GPUs and FPGAs could not help.</p><blockquote>Till one day…</blockquote><p>To ensure high predictive quality and high performance, and to make ML a viable solution, solution developers were oriented towards running ML algorithms on distributed systems, taking advantage of parallelization and high storage bandwidth. This has led to the emergence of <strong>Distributed Machine Learning (DML)</strong>.</p><h3>From Distributed to Federated Learning</h3><p>In DML systems, the training process is distributed among several nodes. To facilitate the development of DML solutions, several DML libraries have been proposed, including MLlib, BigDL, and TensorFlow. These libraries allow users to easily run their ML algorithms on Big Data platforms such as Spark and Hadoop.</p><p>DML systems have brought many benefits to the ML community, including faster executions and the ability to analyze more data. However, despite their decentralization, traditional DML approaches still rely on some centralization. To learn from different data sets distributed across different locations, DML first aggregates all data in a central location (a data center in the cloud) and then distributes the aggregated data to nodes in that central location. For example, a bank collects data from different branches in different locations in a country and then distributes both the collected data and the training calculations to nodes designated for data analysis and located in main bank office. Data transfers to the central location incur unacceptably high costs, especially if the data is geographically dispersed, but the confidentiality and security of the data is also at great risk. All data transferred over the network is exposed to disclosure, attacks, and cyber risks.</p><blockquote>Server server on the cloud… show me the data of all the crowd</blockquote><p>The most serious data breaches of the 21st century include Equifax (with 143 million customers affected in 2017) [3] and Marriot (with 500 million customers affected in 2018) [4]. To address privacy issues, <strong>Federated Learning (FL)</strong> has been proposed.</p><h3>Federated Learning (FL) &amp; Some Applications</h3><p>Federated Learning (FL) is a distributed machine learning that first appeared in 2017 in the work of McMahan [5]. It allows a set of data owners connected through a network to learn collaboratively a global shared model without sharing their data.</p><p>According to the FL paradigm, the learning process is usually orchestrated by a central server located at the center of the network and accessible to different data owners. The terminology may differ in federated learning systems. For example, the central server may be referred to as the coordinator, while the data owners may be referred to as parties or clients.</p><p>In FL, Parties can be mobile devices that collaborate to develop a user-centric recommendation model or geo-distributed bank branches that aim to learn a shared predictor of clients who are able to pay their loans or even hospitals that want to learn a global image classifier to apply on their patients&#39; medical images.</p><p>Similar to classical machine learning, federated learning consists of several phases, namely a training phase, a validation phase, and a testing phase. During the training phase, parties create local models trained on their local data and send them to the server, which aggregates the obtained models and computes a global model. The validation phase takes place during training to validate the architecture of the trained global model. Finally, the testing phase verifies the performance of the global model before its final deployment on parties.</p><p>Compared to DML, FL is characterized by being:</p><p>(1) <strong>Communication-efficient,</strong> as no large amounts of data are transmitted over the network;</p><p>(2) <strong>Privacy-preserving,</strong> as data owners do not send their raw data, which may contain private information, to other data owners or to the cloud.</p><p>At the end of the FL process, and when FL developers are satisfied with their global model, the server sends the global model to the various parties that may or may not have participated in the training. There it is used to make predictions, such as predicting the next word you will type and magically displaying it in front of you, or deciding which video to show you next in your feed!</p><blockquote>And they all lived happily ever after ~</blockquote><p>We have almost reached the end of the article. To be fair, the last line was added mainly because it rhymes well with the beginning ;). Data scientists, product recommendation specialists, bank owners, etc., are happy with FL, but realistically we can not say that anyone has reached the Happy Ending yet. There are still some challenges that FL has to overcome, mainly because of the heterogeneous data and hardware of the parties that usually use federated learning. I will discuss this in another article.</p><h3>Key takeaways</h3><ul><li>Machine learning techniques (ML) were initially applied to small datasets that fit single-node systems.</li><li>With the advent of Big Data, distributed machine learning (DML) has been proposed to run ML on distributed systems, allowing ML to benefit from parallelization and high storage bandwidth and scale to large datasets.</li><li>Classical DML relies on some degree of centralization. To learn from different datasets distributed across different locations, DML first aggregates all data in a centralised location, exposing sensitive data to disclosure, attacks, and cyber risks.</li><li>Federated Learning (FL) is a paradigm of distributed machine learning. It allows a set of data owners connected by a network to learn a global shared model together without disclosing their data.</li></ul><p>!EDIT! You can find a detailed description of the federated learning training, validation, and test here: <a href="https://medium.com/@yasminedj/simulation-to-reality-gap-in-federated-learning-part-2-cbd1d2b419c2">Simulation-To-Reality Gap in Federated Learning — Part 2</a>.</p><h3>References</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P_U4O9rLsjP-3scYnGrwmw.jpeg" /></figure><p>[1] <a href="https://www.researchgate.net/publication/220490151_Accelerating_Large-scale_Data_Exploration_through_Data_Diffusion">https://www.researchgate.net/publication/220490151_Accelerating_Large-scale_Data_Exploration_through_Data_Diffusion</a><br>[2] Raicu, I., Foster, I., Szalay, A. &amp; Turcu, G. (2006). Astroportal: A science gateway<br>for large-scale astronomy data analysis, In Teragrid conference.<br>[3] Gressin, S. (2017). The equifax data breach: What to do. Federal Trade Commission, 8.<br>[4] Sanger, D. E., Perlroth, N., Thrush, G., &amp; Rappeport, A. (2018). Marriott Data Breach Traced to Chinese Hackers. The New York Times, A1-L.<br>[5] McMahan, B., Moore, E., Ramage, D., Hampson, S., &amp; y Arcas, B. A. (2017, April). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273–1282). PMLR.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd03d7b06a2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>