<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kamal Chouhbi on Medium]]></title>
        <description><![CDATA[Stories by Kamal Chouhbi on Medium]]></description>
        <link>https://medium.com/@chouhbik?source=rss-22f56f58cc0b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*dOYKOKixC3TvWiKndwLu4w.jpeg</url>
            <title>Stories by Kamal Chouhbi on Medium</title>
            <link>https://medium.com/@chouhbik?source=rss-22f56f58cc0b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:39:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chouhbik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hippocratic Oath for Data Scientists]]></title>
            <link>https://medium.com/data-science/hippocratic-oath-for-data-scientists-407d2db15a78?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/407d2db15a78</guid>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ethics]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Tue, 06 Oct 2020 15:12:05 GMT</pubDate>
            <atom:updated>2020-10-06T15:12:05.148Z</atom:updated>
            <content:encoded><![CDATA[<h4>Opinion</h4><h4>The ethical checklist that every data scientist must follow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sP2ERWQfyRMRC4gm6IRMXQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@nci?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">National Cancer Institute</a> on <a href="https://unsplash.com/s/photos/data-scientist?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The combined power of data and information technology is advancing innovation in nearly every area of ​​human enterprise.</p><p>The economic statistics are clear: according to <a href="https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/the-age-of-analytics-competing-in-a-data-driven-world"><strong><em>McKinsey Global Institute</em></strong></a>, the global volume of data doubles every three years. This exponential increase is mainly linked to the rise of digital platforms, wireless sensors and the billions of smartphones in circulation. However, in such a flow of data, respect for morals and the human person can be endangered.</p><p><strong><em>In fact, Data Science reached the point at which an ethical pledge is deemed necessary.</em></strong></p><p>Justifying the need to focus on <strong><em>Data Science ethics</em></strong> goes beyond just observing these opportunities and challenges, as the practice of Data Science challenges our perception of what it means to <em>“be human.”</em></p><p>While universities quickly implemented Data Science programs focused on statistics, Machine Learning and Data Engineering, few programs address the broader societal concerns of Data Science. Fewer still are those who analyze how responsible data practices can be conditioned and even encouraged.</p><p>Although the impact of information technology on the organization of private and public enterprises has been widely debated over the past four decades, the impact of Data Science on the management of enterprises has been the subject of much less attention.</p><p>By focusing on Data Science, these contextual concerns bring to light their own ethical considerations. If data is never objective, to what extent does management need to understand the context in which the data was collected, analyzed and transmitted?</p><p>Likewise, with algorithms becoming more and more widespread and complex, to what extent should data scientists understand their own assumptions and their limits?</p><blockquote>“In medicine, you learn about ethics from day one. In data science, it’s not very commun. Actually, it has to be there from day one and at the forefront of your mind in every step you take.”</blockquote><p>Many organizations started to employ data scientists who have a solid understanding of how to analyze data and gain maximum insight to make decisions based on algorithms. When analyzing this data, the data scientist faces ethical challenges such as data collection bias, algorithmic bias, explanation results, confidentiality, etc. In order to formulate and produce morally good solutions, he needs to stick to some ethical rules that will guide him</p><p>How then can we make a data scientist more ethical, what could inspire a data scientist to do ethical work or what moral laws or rules have been established to inspire an ethical environment?</p><p>To date, there has been no law or rule to inspire an ethical environment for data scientists. However, as a member of the data science community and a fresh data science consultant, I tried to establish several principles of ethics that will guide me and inform the good data science practice. I also wrote a draft referring to the Hippocratic oath for Data Science that I will share with you in this article.</p><h3>Fundamental principles:</h3><p>For every professional data scientist who is brought to collect, store, analyze data and design algorithms, there is a necessity to evaluate the impact of what his work can have on individuals and on society as a whole.</p><p>This is why he needs to respect the following three principles:</p><h4>1. Responsibility and Neutrality</h4><p>Every data scientist has to assume his responsibilities in the event of breaches or conflicts of interest and he must alert if any illegal acts related to data are observed. He must also exercise his professional activity respecting the privacy and dignity of people in all their dimensions.</p><h4>2. Transparency</h4><p>As a data scientist, I have the right to inform all stakeholders in an understandable and precise manner about the purposes, modalities and potential implications of my use of data.</p><h4>3. Equity</h4><p>I will always ensure that individuals or groups are not discriminated on the basis of illegal or illegitimate criteria, directly or indirectly, related to my data work.</p><h3>Hippocratic oaths:</h3><p>In this section, you will find some promises about your behavior and your actions that will help you advance your data science career.</p><h4>I work in a “data” oriented project</h4><p>I am committed to …</p><ul><li>question the purpose of the project and its legality.</li><li>do my best to ensure that my working conditions, in particular the working time allotted to me, allow me to carry out honest and as scientific work as possible with the data.</li><li>ensure that the metrics to be optimized are relevant and do not lead the project to have a negative social and environmental impact.</li></ul><h4>I collect and store data</h4><p>I am committed to …</p><ul><li>communicate, or otherwise remind the competent teams, the need to communicate to the people concerned, the use that will be made of their data, in the most clear, explicit and transparent way possible.</li><li>ensure that the consent of the people whose data I collect is obtained under conditions that are fair, explicit and transparent to them. In the event of subsequent changes to the conditions of use, ensure that these changes are also communicated to them clearly and effectively.</li><li>ensure that the data for which I am responsible is managed and stored securely.</li><li>never collect or use unnecessarily personal and/or sensitive data.</li><li>avoid collecting data that I can reasonably believe will not be useful for the needs of the project.</li><li>that the data I collect or have available is correct and that I fully understand its meaning. To do this, I undertake to trace as much as possible the origin and the creation process of the data, as well as any modifications to it.</li></ul><h4>I prepare and explore the data</h4><p>I am committed to …</p><ul><li>not to create characteristic data (“features”) which would be equivalent to sensitive personal data if their use could lead to illegal or illegitimate discriminatory effects (examples: postal code, last name, …).</li><li>take care, when eliminating or imputing missing values/outliers, not to introduce additional biases that would lead to partial or false results.</li><li>justify and document my cleaning without any selection bias.</li></ul><h4>I design and apply algorithmic models</h4><p>I am committed to …</p><ul><li>provide an explanation of the results of the algorithmic model to the persons concerned.</li><li>measure bias and variance to control the accuracy and dispersion of the result and document the error metrics used.</li><li>determine the best compromise between performance and interpretability on all the models available and, as much as possible, opt for the models that are simplest to explain to the people concerned (a high-performance model will reduce the risks of error while an interpretable model will better justify the model results).</li><li>predict and prevent possible drifts overtime of the model with respect to the data, so as to avoid the appearance of additional biases.</li><li>set up and test several models without stopping at the first model and settings that seem to be good.</li></ul><h4>I present the results to stakeholders</h4><p>I am committed to …</p><ul><li>give the alert if I notice a fraudulent, illegal, illegitimate or discriminatory use of the results.</li><li>communicate my approach and my results (or their absence) to my client and my team by ensuring the most exact understanding possible and by opting for the most meaningful visualizations and explanations.</li><li>specify the usual precautions to be taken with these results and their interpretation.</li></ul><h4>I finish the project</h4><p>I am committed to …</p><ul><li>anticipate the uses that could be made of my work in the long term and do my best to keep over it a right of review and a possibility of action.</li><li>ensure that the data is not kept longer than what is necessary for the application as defined during the project.</li><li>document the data and their processing as much as possible in order to guarantee their explainability and reproducibility.</li></ul><h3>Conclusion</h3><p>Ethics is undoubtedly important, and in a perfect world would form part of any education. This oath of ethics will help to create an ethical environment, as great power is said to come with great responsibility.</p><p>Data scientists need such an oath, just like medical doctors who swear to act only in their patients’ best interests. Being an ethical data practitioner will go a long way in paving the way for a safe social involvement and integration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=407d2db15a78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/hippocratic-oath-for-data-scientists-407d2db15a78">Hippocratic Oath for Data Scientists</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preprocessing Data: Feature Scaling]]></title>
            <link>https://medium.com/data-science/preprocessing-data-feature-scaling-cc28c508e8af?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc28c508e8af</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Wed, 08 Jul 2020 17:11:33 GMT</pubDate>
            <atom:updated>2020-07-08T17:11:33.223Z</atom:updated>
            <content:encoded><![CDATA[<h4>Some of the important Feature Scaling techniques with sklearn</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omkDxD8WNxlkfCOGl9hQ3Q.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@adonyi-gabor-604571?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Adonyi Gábor</a> from <a href="https://www.pexels.com/photo/assorted-type-and-size-utility-cutters-on-clear-and-green-olfa-measuring-tool-near-adhesive-tape-rolls-1409216/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>In general, Feature Scaling is a necessary or even essential step in updating the characteristics of every Machine Learning model. <em>Why is it that important?</em> Well, simply because each algorithm we use for prediction or clustering hides some mathematical formulas behind it. These mathematical formulas hardly appreciate the variations in the scale of values ​​between each feature, and that’s clearly visible when it comes to gradient descent!</p><p>In fact, unscaled data lead to difficulties in visualizations and, more importantly, they can degrade the predictive performance of many machine learning algorithms. This type of data can also slow down the convergence of many gradient-based estimators or maybe prevent it at all.</p><p>Indeed many estimators are designed with the assumption that all features vary on comparable scales. In particular, gradient-based estimators often assume that the training data is already standardized (centered features with unit variances). A notable exception are decision tree-based estimators that are robust to arbitrary scaling of the data.</p><p>Let’s take an example: Imagine that you are working on house price prediction, you will have features of the type: price, surface, number of rooms, etc. Of course, the scales of values ​​of this dataframe are totally different according to the features. However, you will have to process them using the same algorithm. This is where Feature Scaling is necessary! your algorithm will indeed have to mix prices of [0… 100,000] $, areas of [10… 500] m2, numbers of rooms ranging from [1 .. 10] rooms. Scaling, therefore, consists in putting this data at the same level.</p><blockquote>If you don’t apply Feature Scaling wisely you will observe slow learning and reduced performance.</blockquote><p>Fortunately, <a href="https://scikit-learn.org/stable/"><strong>Scikit-Learn</strong></a> will help us do the job once again, but before using any technique we have to understand how each one works.</p><p>Basically, <strong>Scikit-Learn</strong> (<a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing">sklearn.preprocessing</a>) provides several scaling techniques, we will review 4:</p><ul><li>StandardScaler</li><li>MinMaxScaler</li><li>MaxAbsScaler</li><li>RobustScaler</li></ul><p>First of all, we are going to create random datasets as well as some graph functions which will help us to better understand the effects of the different techniques mentioned above.</p><p>Here is the Python code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a35b1a237fbdc11122c0e92c31303a0/href">https://medium.com/media/2a35b1a237fbdc11122c0e92c31303a0/href</a></iframe><p>In this code, apart from the trace functions, we create 6 datasets in a single DataFrame (Pandas).</p><p>Let’s already look at what our datasets look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*9-7XXMOr0nUltn9Jg_A6WA.png" /></figure><h4>StandardScaler</h4><p>We will start our little tour (not exhaustive) of scaling techniques by probably the least risky: <strong>StandardScaler ().</strong></p><p>This technique assumes that data is normally distributed. The function will recalculate each characteristic so that the data gets centered around 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/154/1*CXF1tD86Nsf9Xl4vWSMRow.jpeg" /><figcaption>Standardization formula statistics</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/868f1246eb1dae72057f3110c1718192/href">https://medium.com/media/868f1246eb1dae72057f3110c1718192/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*r3iOtVYKoQTIpphXuOzvfw.png" /></figure><p>So the standardization removes the mean and scales the data to unit variance. However, the outliers still have an influence when computing the empirical mean and standard deviation.</p><h4>MinMaxScaler</h4><p>This technique transforms each feature (x) by adapting it on a given range (by default [0, 1]). It is possible to change this range via the parameters feature_range = tuple (min, max). To make it simple, here is the transformation formula for each feature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*VyeD8Gd9uBsx7-WvBqy91A.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35ed9fc29d26ace6181353d84589170a/href">https://medium.com/media/35ed9fc29d26ace6181353d84589170a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*bssNfrg7qHX_flK_6k2uFA.png" /></figure><p>If this technique is probably the best known, it works especially well for cases where the distribution is not Gaussian or when the Standard Deviation is small. However, MinMaxScaler () is very sensitive to outliers. In this case, we quickly switch to the last technique: RobustScaler ().</p><h4>MaxAbsScaler</h4><p>This scaling technique is useful when the distribution of values ​​is sparse and you have quite a few integers. The absolute values are mapped in the range [0, 1]. Indeed, on positive only data, this scaler behaves similarly to MinMaxScaler () and therefore also suffers from the presence of large outliers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f257141076adff0efe3985d3a92d831/href">https://medium.com/media/6f257141076adff0efe3985d3a92d831/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*eszpSAqOKPOwwG_XRa0qgw.png" /></figure><h4>RobustScaler</h4><p>The RobustScaler () technique uses the same principle of scaling as MinMaxScaler (). However, it uses the interquartile range instead of the min-max, which makes it more reliable with regard to outliers. Here is the formula for re-working the features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/0*SWerwlrP1YT9BpS3.jpg" /></figure><ul><li>Q1 (x): 1st quantile / 25%</li><li>Q3 (x): 3rd quantile / 75%</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2eb8559a2e32a5de49baadc2cac7a62/href">https://medium.com/media/f2eb8559a2e32a5de49baadc2cac7a62/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*Eqo4qiHNVVWxTvBbwOEPYw.png" /></figure><p>Unlike the previous scalers, this scaler uses some centering and scaling statistics which are based on percentiles. Therefore they are not influenced by very large marginal outliers. Consequently, the resulting range of the transformed feature values is larger than for the previous scalers and, more importantly, are approximately similar.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*_CudcR5gky8hEgcEEJYr5A.png" /></figure><p>Let’s just summarize the Feature Scaling techniques we just encountered:</p><ul><li>Scaling features to a range, often between zero and one, can be achieved using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler"><strong>MinMaxScaler</strong></a> or <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler"><strong>MaxAbsScaler</strong></a>.</li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler"><strong>MaxAbsScaler</strong></a> was specifically designed for scaling sparse data, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler"><strong>RobustScaler</strong></a> cannot be fitted to sparse inputs, but you can use the transform method on sparse inputs.</li><li>If your data contains many outliers, scaling using the mean and variance of the data is likely to not work very well. In this case, you need to use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler"><strong>RobustScaler</strong></a> instead.</li></ul><p>The Jupyter Notebook on which this article is based can be found <a href="https://github.com/chouhbik/FeaturesScaling/blob/master/Feature%20Scaling.ipynb">here</a>.</p><h3>References</h3><ul><li>Compare the effect of different scalers on data with outliers. <a href="https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html">https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html</a></li><li>Preprocessing data. <a href="https://scikit-learn.org/stable/modules/preprocessing.html#">https://scikit-learn.org/stable/modules/preprocessing.html</a></li><li>Should I normalize/standardize/rescale the data? <a href="http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-16.html">http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-16.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc28c508e8af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/preprocessing-data-feature-scaling-cc28c508e8af">Preprocessing Data: Feature Scaling</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should We Install The Corona Contact Tracing Mobile App ?]]></title>
            <link>https://codeburst.io/should-we-install-the-corona-contact-tracing-mobile-app-15581e56de67?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/15581e56de67</guid>
            <category><![CDATA[coronavirus]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[app-development]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 12:11:54 GMT</pubDate>
            <atom:updated>2020-06-08T14:47:02.040Z</atom:updated>
            <content:encoded><![CDATA[<h3>Should We Install The <strong>Corona Contact Tracing Mobile App ?</strong></h3><h4>What is it exactly, how does it work and why privacy groups are concerned ?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TfJeqeBG4DKLvfo-isvydA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Markus Winkler</a> on <a href="https://unsplash.com/s/photos/covid-tracing-app?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>To end containment without losing control of the <strong>Covid-19</strong> epidemic, several governments are considering contact tracing, including through an app. But the effectiveness of this tool is <strong>still debatable</strong> and <strong>controversial</strong>, as the pandemic reaches an unprecedented scale.</p><blockquote>Do we need a smartphone app to stem the Covid-19 epidemic? Or should we just question the sick to identify the people they may have infected? In short, what is the best option for implementing what is known as contact tracing?</blockquote><p>A few years ago, during the epidemics of <strong>SARS</strong> and <strong>MERS</strong> (the two coronaviruses at the start of the 21st century), the second method had proven its worth. Logically, it is therefore one of the options recommended by epidemiologists to replace containment measures. <strong>Singapore</strong>, <strong>South Korea<em> </em></strong>and <strong>Northern Ireland</strong> in particular have already implemented it, and many other countries are considering it.</p><p>This tool is based on a fairly simple principle: <strong><em>to break a chain, you just have to drop a link.</em></strong> Concretely, it is a question of finding all the people having been exposed to infected individuals, and isolating them from the rest of the population. So if some of them, called <strong><em>contact cases</em></strong>, prove to be infected, they can no longer transmit the virus if they are isolated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQ2nZ6KpEkDjRidLV7fokA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@matilda-wormwood?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Matilda Wormwood</a> from <a href="https://www.pexels.com/photo/person-in-a-protective-suit-disinfecting-inside-the-house-4099260/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>The problem is that contact tracing has never been used in such a context: the number of patients and the discreet transmissibility of <strong>Covid-19</strong> are higher comparing to <strong>SARS</strong> or <strong>MERS</strong>. Therefore, it remains difficult to prejudge its effectiveness. However, many epidemiological proposals and models emphasize its high potential, and propose two main strategies: a classic method, similar to what has always been done, and a new approach, based on a smartphone application. Two strategies that, perhaps, can be combined to make it one.</p><p>In this context, most countries are planning or have already implemented mobile applications to identify potential <strong>Covid-19</strong> patients and those who have come into contact with them, as part of their deconfinement strategy. This solution is most often developed under the impetus of the government.</p><p>Several states already offer their entire population to use this type of application. This is the case in <strong>France</strong> (<a href="https://play.google.com/store/apps/details?id=fr.gouv.android.stopcovid">StopCovid</a>), <strong>Spain</strong> (<a href="https://play.google.com/store/apps/details?id=cat.gencat.mobi.StopCovid19Cat&amp;hl=en">STOP COVID19 CAT</a>), <strong>Germany</strong> (<a href="https://www.coronawarn.app/en/">Corona-Warn-App</a>, <a href="https://www.ito-app.org/">ITO</a>), <strong>Italy</strong> (<a href="https://play.google.com/store/apps/details?id=it.ministerodellasalute.immuni">Immuni</a>), <strong>Switzerland</strong> (<a href="https://play.google.com/store/apps/details?id=ch.admin.bag.dp3t&amp;hl=en">SwissCovid</a>), <strong>Australia</strong> (<a href="https://play.google.com/store/apps/details?id=au.gov.health.covidsafe&amp;hl=en">COVIDSafe</a>), <strong>Malaysia</strong> (<a href="https://play.google.com/store/apps/details?id=my.gov.onegovappstore.mytrace&amp;hl=en">MyTrace</a>), <strong>Morocco</strong> (<a href="https://play.google.com/store/apps/details?id=covid.trace.morocco&amp;hl=en">Wiqaytna</a>) and many others.</p><p>Some of these applications gather tracking information from <strong>Bluetooth</strong> and <strong>GPS</strong> signals, but most of them only use the <strong>Bluetooth</strong> comminication protocol like their singaporean predecessor app <a href="https://www.tracetogether.gov.sg/"><strong>TraceTogether</strong></a>.</p><p>In fact,<strong> Singapore</strong> launched on March 20 <strong>TraceTogether</strong>, an application which, once installed, allows you to automatically know the people that the owner of the smartphone has met, having the application themselves.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fbuj8ZTRtJes%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dbuj8ZTRtJes&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fbuj8ZTRtJes%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e49f24935333535e0316eddd67d55168/href">https://medium.com/media/e49f24935333535e0316eddd67d55168/href</a></iframe><p>Its developers explain that the app is particularly useful when infected people can no longer remember who they have been in contact with for a long time. The city-state has also decided to offer the application in open source for interested states. So, let’s have a clearer and deeper understanding of this new tracking solution.</p><h3>Corona contact-tracing app, what is it?</h3><p>It is a smartphone application, usually compatible with <strong>iOS</strong> and <strong>Android</strong>. It is based on the <strong>Bluetooth</strong> communications protocol (which you already use to connect your phone to an interactive speaker, a smartwatch or a wireless headset, for example). And it is used to determine if you have been in contact with another person using the <strong>BLE (Bluetooth Low Energy), </strong>which<strong> </strong>is intended to provide considerably reduced power consumption.</p><p>By collecting contact information, the app not only allows to back up a path of infection if a user was declared infected, but also to warn all people crossed by contaminated people by advising them to be tested and to isolate themselves.</p><p>Concretely, the app will allow its users to be warned if they have encountered a person with coronavirus. To function, the <strong>Bluetooth</strong> must be constantly activated on the smartphone. It is obviously possible to deactivate the application, which runs in the background, at any time from the interface.</p><h3>What is the purpose of the app?</h3><p>The app will have a complementary role in the fight against the <strong>Covid-19</strong> and its purpose is twofold: Save time by identifying contact cases faster, and alert these anonymous contacts (people encountered in public transport or at the supermarket for example).</p><p>The first objective is to determine the path of contamination of the coronavirus. The path of contamination is the path traveled by the virus thanks to the infected individuals. These, with or without symptoms, transmit the virus to people close to them. And so on.</p><p>The second objective, which pursues the mission of the first, is to warn the user if he has been in contact with a person who was been confirmed that he is carrying the virus. If this is the case, the application invites the user to stay at home, isolated, so as not to infect other people. With this device, the government hopes to break the chains of contamination. Also, patients will be taken care of more quickly, since their contamination will be known further upstream.</p><h3>How does the app work?</h3><p>In most existing contact tracking applications, the operation of the solution is divided into two parts. The first component is the detection of other citizens in a close area. The second part is the rapprochement between a person declared sick (or carrying the virus) and a healthy person.</p><p>Let’s look at the first part first. The application, once activated, searches for the phone of people nearby geographically. A contact is saved when the person is nearby for at least 10–15 minutes (the time interval can vary from an app to another) and if he has a smartphone equipped with the application.</p><blockquote>We are talking about 60% or 70% of users for such a solution to be effective: is it not illusory to reach such levels of use?s</blockquote><p>For a contact to be saved in the application, the maximum distance must be 1.5 meters ( for the french app <a href="https://play.google.com/store/apps/details?id=fr.gouv.android.stopcovid"><strong>StopCovid</strong></a>). According to the company responsible for designing the Bluetooth part of the application, the calculation of distances using Bluetooth is fairly accurate up to 2 meters, which is sufficient to record a contact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*Kaxnauw9SvaGSc3fgG-qcA.png" /></figure><p>Most of the contact-tracing apps, that were launched by several governments, are mainly divided into three tabs:</p><ul><li><strong>Protect:</strong> which enables / disables tracking with a single click,</li><li><strong>Declare:</strong> for users tested positive for Covid-19,</li><li><strong>Share:</strong> Unsurprisingly, the government is asking StopCovid users to share the app with their loved ones.</li></ul><p>Once activated, the application runs in the background and constantly requests the <strong>Bluetooth</strong> connection of the smartphone. This means that the application will affect the battery autonomy. But the <strong>Bluetooth</strong> protocol remains very precise when it is used at a distance of 2 meters and it works even in closed places, unlike <strong>GPS</strong> and <strong>GSM</strong> networks.</p><p>Actually, it is normal and desirable that there is a scientific debate on the best way to design such an application, offering all the guarantees of security and protection of privacy. The choice of the deployed architecture is ultimately left to the government, which must arbitrate between a <strong>centralised</strong> and <strong>decentralised</strong> architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ZQqCmQAFOVkEfe3gtRMhQ.png" /><figcaption>Source: <a href="https://www.bbc.com/news/technology-52650576">https://www.bbc.com/news/technology-52650576</a></figcaption></figure><p>The architecture of the app includes centralized elements <strong><em>(servers)</em></strong> and decentralized elements <strong><em>(smartphones)</em></strong>.</p><p>The majority considers that the so-called centralized architecture offers more guarantees and security. It helps the server to collect the list of people tested positive (even anonymously) and that this list does not circulate.</p><h3>What data does the application collect?</h3><p>The data collected by the application is anonymous. When two smartphones equipped with the app connect to each other using the <strong>Bluetooth</strong> protocol and confirm the proximity of their owners, they exchange information. Each application issues a random identifier. This identifier is saved with the time. Geolocation is not saved, because it is not useful for determining a contamination path. Also, the app does not need a permanent <strong>GPS</strong> or <strong>GSM</strong> connection.</p><p>The contacts made (associated with anonymous random identifiers) are transmitted to a secure server when an <strong>Internet</strong> connection is available. It’s this server that will look for people on a contamination route when it’s confirmed that someone is carrying the virus. No information on the identity of the patient will be transmitted to the recipients of the alerts. We are talking about encrypted data: there is no name that appears, and your personal information remains completely protected.</p><h3>Does the app consume a lot of battery?</h3><p>One of the recurring questions is the battery consumption. In order to be able to detect nearby smartphones, the application must constantly search using the <strong>Bluetooth</strong> protocol. Using this protocol continuously affects the battery but not very much. It remains to be seen how much.</p><p>There may be a higher consumption which leads to an additional recharge. Current technical developments make it possible to limit the impact of this problem by using the <strong>Bluetooth Low Energy</strong>. Compared to <strong>Bluetooth</strong>, <strong>BLE</strong> provides limited bandwidth for <strong>10</strong> <strong>times</strong> less energy consumption. The impact on the battery is estimated to be between <strong>0.5%</strong> and <strong>3.5%</strong>.</p><h3>Is it mandatory to use it?</h3><p>No, most of the existing apps are not mandatory to install. You choose if you want to install it. Of course, the more users there are, the more effective the application is. This choice is logical, because part of the population will not be able to use it because they don’t have smartphones, even if it was made compulsory.</p><h3><strong>Why is the app so controversial?</strong></h3><p>In general, the app is controversial for two main reasons. The first is the doubt about its real effectiveness in alerting and warning citizens. The second is the protection of personal data. These are two important points raised by many experts and associations.</p><p>Other questions were asked by some IT security experts: First, is the information transmitted between smartphones anonymous? Is the exchange protocol encrypted to prevent information theft? Is the backup of contacts well protected? Is the application itself secure enough? And what about the server that will store all of the information to cross it and highlight the contamination paths?</p><p>In most cases, the government ensures the publication of the source code to make it possible to note that the commitments made on the protection of personal data are respected and to confront the application with the scientific community to identify possible flaws.</p><h3>The effectiveness of the app remains questionable…</h3><p>The effectiveness of the application is questioned for several reasons. If truth be told, the whole system is based on volunteering, a concept that has been mentioned many times. A person must volunteer to install and use the application. If the number of voluntary users is too low, the whole system will be ineffective. For this to work, it is necessary that at least <strong>60%</strong> to <strong>70%</strong> of the population is equipped.</p><p>Even in <strong>Singapore</strong>, the ultra connected smart city, with a very strong digital culture, where the rate of mobile phone equipment is very high, the government announces just over a million downloads: for a population of less than six million inhabitants, so we are far from the target.</p><p>And then if you download the application only one day before learning that you are infected it will not be useful at all ! Basically the more time passes before installing the app, the less the solution is efficient.</p><p>Moreover, the <strong>Bluetooth</strong> protocol was not designed to calculate distances between two devices. Technology adaptation is therefore necessary. But that does not guarantee that it will not generate false positives (or that it will not forget to register a contact). In some phones, the <strong>Bluetooth</strong> signal is so strong that it is able to pass through walls. That means it can record someone who was actually in an apartment downstairs or next to you — someone you never had any contact with. So there is a risk to multiply alerts … for nothing.</p><p>Conversely, the app can induce a false sense of security: if you never receive notifications, you have the impression that there is no longer a health risk. So you stop respecting sanitary measures!</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vl0IYLATkwbP4gGjWIpahg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@cottonbro?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">cottonbro</a> from <a href="https://www.pexels.com/photo/man-holding-a-warning-sign-about-coronavirus-3952235/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>Will this new contact tracing solution be a success? The application, designed to detect coronavirus patients in order to contain the epidemic, is at the center of the debate. Its effectiveness as well as the protection of private data are pointed out.</p><p>The contact tracing application is part of a process of <strong>transparency</strong>. This is why it is an application whose source code and documentation are available. This will allow the application to be improved and to verify that the commitments made are respected. In addition, it will allow other countries that do not have the capacity to develop such technology, to be able to develop a solution on this basis.</p><p>I can say that the app can be very <strong>useful</strong>. The identification of the contacts can warn those who have been around you if you test positive for <strong>Covid-19</strong>. People you have met before (in public transport, shops, public places for example), but whose identity is not known for you.</p><p>Therefore, each download of the app is an additional opportunity to prevent and be notified in the event of contact with an infected person.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15581e56de67" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/should-we-install-the-corona-contact-tracing-mobile-app-15581e56de67">Should We Install The Corona Contact Tracing Mobile App ?</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s Time to Familiarize Yourself With NoSQL Databases More Than Ever]]></title>
            <link>https://medium.com/data-science/its-time-to-familiarize-yourself-with-nosql-databases-more-than-ever-5fb1f65c22b1?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fb1f65c22b1</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Fri, 27 Mar 2020 17:03:24 GMT</pubDate>
            <atom:updated>2020-03-27T20:59:43.255Z</atom:updated>
            <content:encoded><![CDATA[<h4>SQL vs NoSQL: the difficult choice of the right database (and how to do it right)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6ANwgV_R6OUzN7DnxVwxg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@kaleidico?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kaleidico</a> on <a href="https://unsplash.com/s/photos/data?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Data is at the heart of the information system. The efficiency of its organization and its operation is a major concern for any company. At the time of <strong>Big Data</strong>, the business expertise and understanding the existing technical solutions is very essential. The technological landscape is evolving very quickly and companies must at the same time continue to assess and choose the databases that will meet their future needs and support their growth.</p><p>Relational databases have been used for decades to store data, and they still represent a viable solution for many use cases. <strong>NoSQL</strong> databases were created in response to the limitations of relational database technology. Compared to relational databases, <strong>NoSQL</strong> databases are more scalable and offer higher performance, and their data model corrects several weaknesses of the relational model. In this article, I will try to throw light on the challenges, benefits and process of migrating from a <strong>RDBMS</strong> to <strong>NoSQL</strong>.</p><p>In general, <strong>NoSQL</strong> databases were designed to solve the problems of data processing in volume, multi-sources and multi-formats, in <strong>Big Data</strong> environments. They offer a new approach to meet volume requirements<br>and new types of data.</p><p>Nowadays, the number of <strong>NoSQL</strong> databases has become more and more important. It is imperative to know their differences to adopt the right technology for the right application.</p><h3>1. INTRODUCTION</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*73e3UUYS_SsBYZfLvdOcfQ.png" /></figure><h4>- SQL :</h4><p><strong>SQL</strong> is the acronym for <strong><em>S</em></strong><em>tructured </em><strong><em>Q</em></strong><em>uery </em><strong><em>L</em></strong><em>anguage</em>. It has been known for a long time to all IT engineers who have had to quickly search for information in large relational databases <strong>(DBMS)</strong>. <strong>SQL</strong> is widely used today because it is one of the most structured and fastest database organization and querying devices; it comes in different names which show that there are various adaptations, such as Oracle’s <strong>MySQL</strong> and Microsoft’s <strong>SQL Server</strong>. In addition, <strong>SQL</strong> is the most recommended option by many companies due to its predefined structure and schemas.</p><h4><strong>- NoSQL :</strong></h4><p>The acronym <strong>“NoSQL”</strong> has two interpretations which are not clear-cut today:</p><ul><li>For some it is <strong><em>“No SQL”</em></strong>, that is to say, the use of another query language different from <strong>SQL</strong>.</li><li>For the others, it is <strong><em>“Not Only SQL”</em></strong>, but that is also to say, the combined use of <strong>SQL</strong> with other information retrieval tools.</li></ul><p>So the term relates as much to technical characteristics as to a historic generation of <strong>DBMS</strong> that emerged around the <strong>2010s</strong>. The main problem that led to inventing <strong>NoSQL</strong> was to solve the fact that the same database on a website could be used at the same time worldwide by millions of users; the typical problem of a company like <strong>Amazon</strong>…</p><p>What we are therefore trying to achieve with <strong>NoSQL</strong> is to reduce the complexity of the query language and to simplify the architecture of the database. These databases consist of <em>column-oriented</em>, <em>document-oriented</em>, <em>graph-oriented</em> and <em>key / value-oriented</em> data. The <strong>NoSQL</strong> family is made up of various products, each product has a unique set of functionalities.</p><h4>- Main Differences :</h4><ul><li><strong>SQL</strong> databases have a predefined schema while <strong>NoSQL databases</strong> have a dynamic schema for unstructured data.</li><li><strong>SQL</strong> databases are scalable vertically, while <strong>NoSQL databases</strong> are scalable horizontally. <strong>SQL</strong> databases are scaled by increasing the power of the hardware like <strong>CPU</strong>, <strong>RAM</strong> or <strong>SSD</strong>. The databases <strong>NoSQL</strong> are scaled by increasing the number of data servers to reduce the load. It’s like adding more floors to the same building versus adding more buildings to the neighborhood.</li><li><strong>SQL</strong> databases are table-based databases while <strong>NoSQL databases</strong> are databases based on key-value pairs. This means that <strong>SQL</strong> databases represent data in the form of tables composed of a certain number of rows that represents the data, while <strong>NoSQL databases</strong> are the collection of key-value pairs, of documents, of graphical databases, etc.</li><li><strong>SQL</strong> databases use <strong>SQL <em>(structured query language)</em></strong> to define and manipulate data, which is very powerful. In the <strong>NoSQL database</strong>, queries focus on document collection. Sometimes it is also called <strong>UnQL <em>(Unstructured Query Language)</em></strong>. The syntax for using <strong>UnQL</strong> varies greatly between the different existing <strong>NoSQL databases</strong>.</li></ul><h3>2. HISTORICAL ELEMENTS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KR6YEm2yVPMqqIasHVPxuQ.jpeg" /><figcaption>Source : Photo by <a href="https://unsplash.com/@nci?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">National Cancer Institute</a> on <a href="https://unsplash.com/s/photos/old-computer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>Historical domination of relational DBMS</h4><ul><li>The relational <strong>DBMS</strong> created in the <strong>1970s</strong> has gradually imposed itself until it became the very widely dominant database paradigm in the early <strong>1990s</strong>.</li><li>During the <strong>1990s</strong>, many salespeople in logistics companies started using it to store their business data. Actually, they had neither a mouse nor a user interface to search for certain information stored on servers often distant and connected by specialized lines: they were used to enter <strong>SQL</strong> commands with the keyboard and were able to retrieve relevant information about the availability of a particular product or raw material in seconds.</li><li>Several other database models have emerged, like the <a href="https://www.wikiwand.com/en/Object_database">DBMS Object-oriented</a> , <a href="https://www.wikiwand.com/en/Hierarchical_database_model">Hierarchical DBMS</a> , <a href="https://www.wikiwand.com/en/Object-relational_database">Object-relational DBMS</a> , but their use has been very limited.</li><li>Starting from the <strong>2000s,</strong> with the development of large internet companies <strong>(Amazon, eBay, Google …),</strong> huge amounts of <strong>unstructured data</strong> appeared and was growing far more rapidly than <strong>structured data </strong>that it did not fit the relational schemas of the <strong>RDBMS</strong> anymore. There was also a development of <a href="https://www.wikiwand.com/en/Computer_cluster">clustered computing</a>, thus the domination of the relational model was questioned because it had some prohibitive limits for these new practices.</li></ul><h4>Pioneers of the NoSQL model</h4><p>The large web companies had to process very large volumes of data, that is why they were the first to face the intrinsic limitations of traditional relational <strong>DBMS</strong>. These systems, based on a strict application of <a href="https://database.guide/what-is-acid-in-databases/"><strong>ACID</strong></a> properties (atomicity, consistency, isolation, durability) and generally designed to run on single computers, quickly posed problems of scalability.</p><p>In order to meet these limits, these companies have started to develop their own database management systems which can operate on distributed hardware architectures and which can process a large volume of data :</p><ul><li>Google ( <a href="https://cloud.google.com/bigtable/">BigTable</a> ),</li><li>Amazon ( <a href="https://aws.amazon.com/dynamodb/">DynamoDB</a> ),</li><li>LinkedIn ( <a href="https://www.project-voldemort.com/voldemort/">Voldemort</a> ),</li><li>Facebook ( <a href="http://cassandra.apache.org/">Cassandra</a> then <a href="https://hbase.apache.org/">HBase</a> ),</li><li>Baidu ( <a href="https://hypertable.org/">Hypertable</a> )</li></ul><p>Performance remains good with the ramp-up by simply multiplying the number of servers, a reasonable solution with the drop in costs, in particular, if revenues grow with the development of the activity.</p><h3>3. Popular Databases</h3><p>There are several popular databases systems available; both paid and free. In order to choose the right management system for you or your organization, it is important to have an idea about what exists in the market. Check out the list below of 5 popular<strong> SQL</strong> &amp; <strong>NoSQL</strong> databases.</p><h4>- SQL Databases Products:</h4><p><strong>1- MySql:</strong></p><p>It’s available for free and it offers a lot of functionality even for a free database engine.</p><figure><img alt="MySQL" src="https://cdn-images-1.medium.com/max/1024/0*Hh48VuINJuvG5I5T.png" /><figcaption>Source : <a href="https://www.mysql.com/">https://www.mysql.com/</a></figcaption></figure><p><strong>2- Oracle:</strong></p><p>Oracle database management tools are incredibly robust with the latest innovations and features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*guEsd8l2BJxilogU.png" /><figcaption>Source : <a href="https://www.oracle.com/database/">https://www.oracle.com/database/</a></figcaption></figure><p><strong>3- Postgres:</strong></p><p>This database management engine is scalable and can handle terabytes of data, with a variety of predefined functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*X4qVi5a3jKhPi733.jpeg" /><figcaption>Source : <a href="https://www.postgresql.org/">https://www.postgresql.org/</a></figcaption></figure><p><strong>4- SQL Server:</strong></p><p>It is very fast and stable. It works very well with other Microsoft products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*18d2lYUTJ0rMtgNG.png" /><figcaption>Source : <a href="https://www.microsoft.com/en-us/sql-server/sql-server-2019">https://www.microsoft.com/en-us/sql-server/sql-server-2019</a></figcaption></figure><p><strong>5- SQLite:</strong></p><p>A SQLite database is very light and quick to set up, it can also be used to store data in a real database on a smartphone application (iPhone or Android).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qdJk6XrIsYPjkN5-.png" /><figcaption>Source : <a href="https://www.sqlite.org/index.html">https://www.sqlite.org/index.html</a></figcaption></figure><h4>- NoSQL Databases Products:</h4><p><strong>1- MongoDB:</strong></p><p>MongoDB is a flexible/reliable database that will draw you to the <strong>NoSQL</strong> world. Its management and maintenance are very easy and fast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c19ArFDcdzw23Xvh.png" /><figcaption>Source : <a href="https://www.mongodb.com/">https://www.mongodb.com/</a></figcaption></figure><p><strong>2- Redis:</strong></p><p>It is incredibly simple and straightforward to use. You can download Redis and start using it during the next five minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kOIfmYy3KceiE0Tl.png" /><figcaption>Source : <a href="https://redis.io/">https://redis.io/</a></figcaption></figure><p><strong>3- Cassandra:</strong></p><p>The linear scalability provided by Cassandra, allowing to easily scale up/down the cluster by simply adding/removing the servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JSttB9h6BR2s7Vlv.png" /><figcaption>Source : <a href="http://cassandra.apache.org/">http://cassandra.apache.org/</a></figcaption></figure><p><strong>4- Hbase:</strong></p><p>It is a column oriented database that helps improve the query performance and aggregations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DLmbrVepo9SnXds7.png" /><figcaption>Source: <a href="https://hbase.apache.org/">https://hbase.apache.org/</a></figcaption></figure><p><strong>5- CouchDb:</strong></p><p>Thanks to its ability to store serialized (JSON formatted) unstructured data and its Restful HTTP API, CouchDB is highly desirable for Web and Mobile Applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*41thIfJNXrOwuHX4.png" /><figcaption>Source : <a href="https://couchdb.apache.org/">https://couchdb.apache.org/</a></figcaption></figure><h3>4. NoSQL Database Designs</h3><p>The main characteristics of <strong>NoSQL DBMS</strong> are to allow the manipulation of large volumes of data and to allow horizontal scalability. However, the difficulty, which most of the companies face today, is to target the most adequate technology in order to respond to their problems and their application.</p><p>Addressing this difficulty begins with a good understanding of the different types of <strong>NoSQL</strong> databases. There is a common myth that all <strong>NoSQL</strong> databases are created equal — this is not true! In fact, these databases can be classified into four categories: <strong>document-oriented</strong> databases, <strong>key/value</strong> databases<strong>, column</strong> databases and <strong>graph-oriented</strong> databases. They all have one thing in common: the support of more flexible and dynamic models than those produced with traditional relational databases.</p><p>In fact, each of these categories has its own attributes and limitations. There is no preferable database for solving all problems. You must select a database according to the needs of your project.</p><blockquote>You have to wonder what kind of data will be manipulated and how the applications will ultimately use it.</blockquote><h4><strong>- Document-oriented</strong> databases: Mixed Structure</h4><p>A document-oriented NoSQL database stores and extracts data as a key/value pair, but the value part is stored as a document. The document is stored in <strong>JSON</strong> or <strong>XML</strong> formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zu7WL6AonSg_Ap5dYcABXw.png" /></figure><blockquote><a href="https://www.mongodb.com/"><strong>MongoDB</strong></a>, <a href="https://couchdb.apache.org/"><strong>Apache CouchDB</strong></a>, <a href="https://www.marklogic.com/"><strong>MarkLogic</strong></a> are document-oriented databases.</blockquote><h4><strong>- Key / value</strong> databases:</h4><p>A key-value oriented database has a large hash of keys and values. It represents the simplest form of <strong>NoSQL</strong> databases. They associate unique keys with values ​​in data, with the objective of greatly enhancing the performance of applications based on relatively simple datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jqaGa_wm33d5d4PCUDXIIg.png" /></figure><blockquote><a href="https://redis.io/"><strong>Redis</strong></a><strong>, </strong><a href="https://riak.com/"><strong>Riak</strong></a><strong>, </strong><a href="https://memcached.org/"><strong>Memcached</strong></a> and <a href="https://www.aerospike.com/docs/guide/kvs.html"><strong>Aerospike</strong></a> are examples of key-value databases.</blockquote><h4><strong>- Column</strong> databases:</h4><p>Column databases hold data in tables that have a very large number of columns. Each storage block contains data from a single column and each column is treated separately. They provide high performance on aggregate queries such as COUNT, SUM, AVG, MAX, etc., because the data is readily available in a column.</p><blockquote><a href="https://hbase.apache.org/"><strong>HBase</strong></a>, <a href="http://cassandra.apache.org/"><strong>Cassandra</strong></a> and <a href="https://accumulo.apache.org/"><strong>Accumulo</strong></a> are examples of column-based databases.</blockquote><h4><strong>- Graph-oriented</strong> databases:</h4><p>A graph-based database is a network database that stores data elements in “graph” structures and make it possible to create associations between the nodes, ultimately serving as a basis for recommendation engines or social networks.</p><p>We can get a lot of information from a graph database. For example, graph technology can be used to identify relationships between different people through their interests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pTNeaEcncpeUa_8n.png" /><figcaption>Twitter Graph | Source : <a href="https://neo4j.com/blog/oscon-twitter-graph/">https://neo4j.com/blog/oscon-twitter-graph/</a></figcaption></figure><blockquote><a href="https://neo4j.com/"><strong>Neo4J</strong></a><strong>,</strong><a href="https://www.objectivity.com/products/infinitegraph/"><strong> Infinite Graph</strong></a> and <a href="https://github.com/twitter-archive/flockdb"><strong>FlockDB</strong></a> are examples of graph-oriented databases.</blockquote><h3>4. 5 Criteria for choosing the adequate type of database for your application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eBrz6NVuA6eUE6Kk.png" /></figure><blockquote>How do we decide which type of Database is the best fit for a project ? <em>Here is a checklist that you can use :</em></blockquote><ul><li><strong>The type of data to be stored</strong> : <strong>SQL</strong> databases are not suitable for hierarchical data storage. However, the <strong>NoSQL</strong> database is better suited for hierarchical data storage because it follows the key-value pair method or graph method. <strong>NoSQL</strong> databases are highly preferred for large data sets.</li><li><strong>Complex queries</strong> : <strong>SQL</strong> databases are perfectly suited to the environment requiring many queries, while <strong>NoSQL</strong> databases are not suitable for complex queries. So the queries in <strong>NoSQL </strong>are not as powerful as the <strong>SQL</strong> query language.</li><li><strong>Scalability</strong> : In most situations, <strong>SQL</strong> databases are vertically scalable. You can manage the increased load by increasing the processor, RAM, SSD, etc. on a single server. On the other hand, <strong>NoSQL</strong> databases are scalable horizontally. You can simply add a few additional servers easily to your <strong>NoSQL</strong> database infrastructure to handle heavy traffic. So you can choose the type of database that will suits you depending on your equipments.</li><li><strong>Highly transactional applications</strong> : <strong>SQL</strong> databases are better suited to transactional type applications for intensive use, because they are more stable and promise atomicity as well as data integrity. Although you can use <strong>NoSQL</strong> for transactional purposes, it is still not comparable but can be used for complex transactional applications.</li><li><strong>Properties</strong> : <strong>SQL</strong> databases emphasize <strong>ACID</strong> properties (Atomicity, Consistency, Isolation, Durability), while the <strong>NoSQL</strong> database follows the Brewers <strong>CAP</strong> theorem (Consistency, Availability and Partition Tolerance).</li></ul><h3>5. Switch from RDBMS to NoSQL</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfZml0LL5CS3r0ny5iPqjQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@scienceinhd?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Science in HD</a> on <a href="https://unsplash.com/s/photos/server?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>No matter which <strong>NoSQL</strong> database design you choose, migrating your data into it can present some serious challenges. The design of a data model in <strong>NoSQL</strong> has additional complexity because it requires knowing the end use of the data. Knowing that your application will handle <strong><em>Bills</em></strong> and <strong><em>Customers</em></strong> is no longer sufficient. You must now know how this data will be displayed to the end user. Therefore, data modeling in a <strong>NoSQL</strong> database requires real technical expertise in addition to in-depth knowledge of the end-customer use.</p><h4>Is it time to replace SQL with a NoSQL solution ?</h4><p>In my opinion, this is a hard question! Because in most cases, it’s not about replacing <strong>SQL</strong> with a <strong>NoSQL </strong>solution, but rather, making a transition from one to the other, if the application and the use case reveal the need for change. In general, this transition will be driven by the need for flexibility and scalability when it comes to building modern web and mobile applications.</p><p>Typically, many companies try to support the load in their web applications, so they choose to simply add web servers behind a load balancer to support more users. Undoubtedly, the ability to scale up is a fundamental competitive advantage in the increasingly important world of cloud computing, in which instances of virtual machines can be easily added or removed so as to meet the fluctuating demands.</p><p><strong>Relational databases (RDBMS)</strong> do not allow simple scaling and do not provide a flexible data model. Managing more users means adding a larger server and large servers are very complex and expensive, unlike low-cost hardware,<strong> “commodity hardware”</strong> and cloud architectures. Organizations are starting to see performance issues with their relational databases for existing or new applications. Especially as the number of users increases daily, they realize that the need for a faster and more flexible database has become very important. It’s time to move to <strong>NoSQL</strong>!</p><h4>What would be the main steps required for the transition from SQL to NoSQL?</h4><p>Applications / Projects can vary greatly depending on each organization, so the transition will depend on your use case. Here are some general guidelines for the transition:</p><p><strong>1- Understand the key requirements for your application:</strong></p><p>Here are some requirements corresponding to the need to have a <strong>NoSQL</strong> database:</p><ul><li><strong>Rapid application development:</strong> Changing market needs &amp; Continuing modification of data</li><li><strong>Scalability</strong></li><li><strong>Constant performance: </strong>Low response time for a better user experience</li><li><strong>Operating reliability: </strong>High availability to manage errors with minimal impact on the application &amp; Integrated monitoring APIs for better maintenance</li></ul><p><strong>2- Understand the different types of NoSQL offers:</strong></p><p>As I’ve said above, there are different types of <strong>NoSQL</strong> database management systems.</p><p>For instance, the document-oriented <strong>NoSQL</strong> databases — with <strong>Couchbase</strong> and <strong>MongoDB </strong>which are the two best known and most widely adopted examples.</p><p>Also,<strong> Cassandra</strong>, for example, may be a solution that you can use for data analysis given its columnar model. <strong>Neo4j</strong>, a graph database, can be the perfect database for applications that need to store relationships between entities.</p><p><strong>3- Build a prototype:</strong></p><p>Once you have narrowed down the possible choices for the type of database, try to develop a prototype integrating the main characteristics of your application. This prototype will help you evaluate the response times, the performance in terms of throughput and the capacity to scale up easily.</p><p><strong>4- Document modeling and development:</strong></p><p>For document-oriented databases, spend a few days on modeling your data starting from fixed tabular diagrams to arrive at flexible document models.</p><p><strong>5- Deployment then production:</strong></p><p>Operational stability is a very important aspect for interactive web applications. Test and re-test your deployment as you would for applications that normally use traditional <strong>RDBMS</strong> systems.</p><p><strong>6- Keep up with the latest trends:</strong></p><p>Today, there is a plethora of quality training that offers hands-on courses for training on <strong>NoSQL</strong>. The best way to ensure a successful implementation of <strong>NoSQL</strong> is to be up to date on the latest versions.</p><p>Don’t worry, you will find it easy to adopt certain <strong>NoSQL</strong> technologies, especially if you are familiar with <strong>JSON</strong> for document format. Developers who use <strong>SQL</strong> extensively may need to adapt and learn document modeling approaches. Rethinking how data can be structured logically using documents rather than normalizing data into a fixed database schema becomes an important aspect.</p><p>Here are links to some interesting courses:</p><ul><li><a href="https://www.udemy.com/topic/nosql/?ranMID=39197&amp;ranEAID=vedj0cWlu2Y&amp;ranSiteID=vedj0cWlu2Y-hOFDzyX.OTsY17shP_9CHw&amp;LSNPUBID=vedj0cWlu2Y">NoSQL Courses &amp; Tutorials (Udemy)</a></li><li><a href="https://www.coursera.org/learn/nosql-databases?ranMID=40328&amp;ranEAID=vedj0cWlu2Y&amp;ranSiteID=vedj0cWlu2Y-DHHjdVvR8bfQdeOnPQg.nQ&amp;siteID=vedj0cWlu2Y-DHHjdVvR8bfQdeOnPQg.nQ&amp;utm_content=10&amp;utm_medium=partners&amp;utm_source=linkshare&amp;utm_campaign=vedj0cWlu2Y">NoSQL Systems (Coursera)</a></li></ul><h3>6. Conclusion</h3><p>The objective of this article was therefore to present the main differences that exist to help you make the right decision and shape the future of your information system (or your simple application).</p><p>We have seen that <strong>SQL</strong> and <strong>NoSQL</strong> databases ultimately do almost the same thing (store data) but in different ways. The choice of a database management system <strong>(DBMS)</strong> is therefore an important and structuring moment for any data project. Of course, it is always possible to choose an option and then switch to another later. But a little bit of conceptual analysis and thinking at the start of a project will make you save time and money.</p><p>The market today is full of <strong>NoSQL</strong> databases — I think we are faced with two or three of them every day, because there are a lot of advantages for a developer to switch to <strong>NoSQL</strong>. A more flexible data model and freedom from rigid schemas is a big advantage. You can also see significantly improved performance and the ability to scale horizontally.</p><p>But most <strong>NoSQL</strong> products are still in the early stages of the product cycle. For features such as complex joins, developers may prefer using a traditional <strong>RDBMS</strong>. And for some projects, a <strong><em>hybrid approach</em></strong> might be the best choice.</p><p>To conclude, each company will have its own preferences depending on the requirements of the project. So determine your needs and the database that judiciously provides integrated support for the development of your project.</p><blockquote>Congratulations if you managed to get here. Thanks for reading, I hope you’ve liked it. For personal contact or discussion, feel free to reach out to me on <a href="https://www.linkedin.com/in/kamal-chouhbi/"><strong>LinkedIn</strong></a><strong>.</strong></blockquote><h3>References :</h3><ul><li><a href="https://www.xplenty.com/blog/the-sql-vs-nosql-difference/">SQL vs. NoSQL: How Are They Different and What Are the Best SQL and NoSQL Database Systems?</a></li><li><a href="https://www.techrepublic.com/blog/the-enterprise-cloud/migrating-from-a-relational-to-a-nosql-cloud-database/">Migrating from a relational to a NoSQL cloud database</a></li><li><a href="https://www.youtube.com/watch?v=ZS_kXvOeQ5Y&amp;t=411s">SQL vs NoSQL or MySQL vs MongoDB</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fb1f65c22b1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/its-time-to-familiarize-yourself-with-nosql-databases-more-than-ever-5fb1f65c22b1">It’s Time to Familiarize Yourself With NoSQL Databases More Than Ever</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Coronavirus (COVID-19): 3 Tips to Ensure Efficiency in Teleworking]]></title>
            <link>https://medium.com/swlh/coronavirus-covid-19-3-tips-to-ensure-efficiency-in-teleworking-b50e7f23e22e?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b50e7f23e22e</guid>
            <category><![CDATA[work]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[telework]]></category>
            <category><![CDATA[remote-working]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Wed, 18 Mar 2020 18:01:08 GMT</pubDate>
            <atom:updated>2020-03-18T22:22:04.143Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQUBHNpXpOYF790_bsdmKg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@andrewtneel?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Andrew Neel</a> on <a href="https://unsplash.com/s/photos/remote-working?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>When the government asks to avoid crowded places, using telework is the only solution to maintain business continuity in the face of the <strong>COVID-19</strong> pandemic.</p><p>When many cases of contamination by the new coronavirus have been detected during the last week, our daily life has totally changed. Traffic is minimal, shops are closed, and the streets are almost deserted. The inhabitants are obliged to respect the isolation rules imposed by their authorities, the same for businesses &amp; entreprises.</p><p>Telework on the one hand protects the health of fellow citizens and on the other hand allows business continuity for the company. Many companies have negotiated telework agreements, but few employees are prepared to work from home. For them, this is a new and questioning practice.</p><p>Fortunately, thanks to the <strong>Internet</strong> and technological advances, working remotely has become easier. Many companies have decided to respond favorably by offering their <strong><em>cloud service</em> (SaaS)</strong> telework offer for free.</p><p>But teleworking also has some drawbacks, especially in the context where the other members of the family are at home all day. I find myself obliged to participate in video meetings, follow the developments of <strong>Covid-19</strong>, chat online with my friends, do the shopping, do the cooking and cleaning, etc.</p><blockquote>In fact, it’s more stressful than usual.</blockquote><p>Here are some tips to gain efficiency while working from home:</p><h3>1. Keep a Daily Schedule</h3><p>It is not recommended to start working as soon as you get out of bed. Better to keep a morning routine, like showering, getting dressed, having breakfast, having coffee in front of your computer before you start working. Personally, the ritual at the beginning of the day helps me a lot to get in condition.</p><p>During the day, the hours can be more flexible, for example, you can work at lunchtime and take a nap at 3 p.m, but it is important to have a ritual, at least in the morning. You also need to take breaks, as you would at the office with the coffee break: you can take a quick shower or read a book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*3EWCdxv3L0ScUrYFzqMJiw.gif" /><figcaption>Source: <a href="https://gph.is/2apaVwT">https://gph.is/2apaVwT</a></figcaption></figure><p><strong>Time Blocking</strong> is also a better way to approach our day and get things done. It’s a time management approach that focuses on your daily structure instead of trying to predict how long each task will take. So the purpose of time management is enabling people to get more and better work done in less time and that means that you have to be productive for extended periods of time.</p><blockquote>Why we should time block ? Isn’t To-Do-List good enough to get things done?</blockquote><p>Unfortunately the answer is <strong>No! </strong>To-Do-Lists are great and practical but you don’t want them to run your life. The time blocking on the other hand gives you a better visibility on a big picture, flexibility without the risk of losing things in a shuffle and the satisfaction of knowing that you made most out of your day.</p><p>Read more about <strong>Time Blocking</strong> :</p><ul><li><a href="https://blog.rescuetime.com/time-blocking-101/">Time blocking 101: A step-by-step guide to getting the most from your daily schedule</a></li><li><a href="https://zapier.com/blog/best-time-blocking-app/">The 5 Best Time Blocking Apps in 2019</a></li></ul><h3>2. Create a Space Dedicated to Work</h3><p>Make sure the place you select is used exclusively for working/studying and nothing else. If you have the possibility, it is important to arrange a space entirely dedicated to work, which you can enter and exit as if you enter and leave the office. While working from home, you may be tempted to watch TV or listen to loud music. So, having a dedicated place can help you focus more and avoid distractions. It will create a real physical break with your private life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8f4g_fewDD7A-Cn9ayv2g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@carlheyerdahl?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Carl Heyerdahl</a> on <a href="https://unsplash.com/s/photos/work-home?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>When you’re working from home, you need calm and focus. Unless you work very early in the morning or very late in the evening, you cannot be both the person who watches over your kids, little brothers or little sisters, and the one who works. They usually make noise and try to interrupt you. If they stay at home, which is certainly the case with the epidemic, try to wear headphones in order to isolate yourself from noise and close the door of your room dedicated to work.</p><p>Moreover, when you’re at home, you tend to work on your laptop from your couch or bed. <strong>That’s a bad idea, </strong>because you are more likely to slip into the path of least resistance which is usually taking a nap or binge watching a show on Netflix. It is better to keep a sitting position like the one we take at the office, to get up regularly so as not to remain too static.</p><h3>3. Find the Best Tools to Communicate</h3><p>One of the aspects on which companies must act, even beyond the implementation of telework, is communication. An investment in means of communication up to the current digitalization of companies is necessary. For the time being, companies have not yet all adopted the employee management / steering component via collaborative tools.</p><p>While teleworking, you have to keep chatting with your colleagues, via corporate messaging. We can chat, talk about light topics, make jokes, keep the link, and close the messaging when we need to be focused. For meetings, we use videoconferencing, for not too critical emergencies, SMS. The telephone is more for big emergencies or complex subjects.</p><p>As an employer, you shouldn’t put pressure on your employees by harassing them with calls or emails. It is better to operate by assigning tasks, so that the person organizes his time as he can. As an employee, you don’t play dead, when you have a task to do, just try to fix a deadline and stick to it.</p><p>Here are some sites, tools and applications that I use frequently. You may find them useful for organizing your work and for communicating with your colleagues :</p><h4>Slack : Where work happens</h4><p>When you don’t have direct access to your colleagues, the idea is to avoid multiplying emails. Even if they remain a major tool at work, some team instant messaging applications can reduce email exchanges and speed up the transfer of messages.</p><p><a href="https://slack.com/intl/en-ma/"><strong>Slack</strong></a> allows you to create organized group conversations (other team members must be registered too). The conversations are archived which makes it easy to look for a specific subject later.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhvtW5_w-yfY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhvtW5_w-yfY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhvtW5_w-yfY%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b4310e0a638a23c7107832cad6470f8/href">https://medium.com/media/1b4310e0a638a23c7107832cad6470f8/href</a></iframe><p>A limited and free version exists for small teams, other packages ranging from <em>6.67 $ US / month</em> to <em>12.50 $ US / month</em> are available for larger companies or to have more developed options.</p><h4>TeamViewer : The new standard for remote control and remote support</h4><p>It’s a tool to take control of a machine remotely if you need to troubleshoot one of your employees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*evzzOabNMxd2YHJw.png" /><figcaption>Source : <a href="https://www.teamviewer.com/en/info/screen-sharing/">https://www.teamviewer.com/en/info/screen-sharing/</a></figcaption></figure><p>Find the right license that fits your needs <a href="https://www.teamviewer.com/en/buy-now/">here</a>.</p><h4>Trello : Organize anything with anyone, anywhere!</h4><p>Alone or in a team, <a href="https://trello.com/"><strong>Trello</strong></a> allows you to create projects by distributing tasks, by adding attachments visible to other collaborators and also allows to couple with other applications like <strong>Slack, Google Drive or DropBox</strong>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJ2jXzLEosp4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJ2jXzLEosp4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJ2jXzLEosp4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c34bdcd6b00511b8b32cf897e34b9cc2/href">https://medium.com/media/c34bdcd6b00511b8b32cf897e34b9cc2/href</a></iframe><h4><a href="https://klaxoon.com/">Klaxoon</a> : The Meeting Revolution</h4><p>A provision allowing tables where everyone can put their ideas on a project. Ideal when all members of the same team are not present or only a few. It’s free during the first 3 months.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBngsUlaMOOA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBngsUlaMOOA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBngsUlaMOOA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4af2c8ff6a983ffb68d896a8962bb6d0/href">https://medium.com/media/4af2c8ff6a983ffb68d896a8962bb6d0/href</a></iframe><h4>WeTransfer : To Share Files</h4><p><a href="https://wetransfer.com/"><strong>WeTransfer</strong></a><strong> </strong>is a tool for exchanging files that are too large to fit in a simple — e-mail attachment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*j6LAf7wFxYHhurclAnffcw.jpeg" /><figcaption>Source : <a href="http://ohmycode.ru/brand/there-is-no-i-in-transfer.html">http://ohmycode.ru/brand/there-is-no-i-in-transfer.html</a></figcaption></figure><h4>Skype, Google Hangouts and Zoom</h4><p>For me, these are the best three existing<strong><em> </em></strong>tools which allow you to have meetings by webcam.</p><blockquote><strong>The best video conferencing solution for work</strong></blockquote><p><a href="https://zoom.us/"><strong>Zoom</strong></a> offers videoconferencing, audio conferencing, chat and webinar functionality on mobile devices, desktops and meeting room systems. Its free version can host up to <strong>100</strong> participants and organize unlimited individual meetings. However, group meetings are limited to <strong>40</strong> minutes. <a href="https://zoom.us/"><strong>Zoom</strong></a> also offers paid versions for small teams from <strong>14.99 $</strong> per month and for small and medium businesses at <strong>19.99 $</strong> per month.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBGH1y70zSkg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBGH1y70zSkg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBGH1y70zSkg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6cb24d8c6fcceba1414558037ebf6c77/href">https://medium.com/media/6cb24d8c6fcceba1414558037ebf6c77/href</a></iframe><blockquote><strong>The best free video conferencing solution</strong></blockquote><p><a href="https://hangouts.google.com/"><strong>Google Hangouts</strong></a><strong> </strong>is a video conferencing application built into <a href="https://gsuite.google.com/"><strong>G Suite</strong> </a>that lets people join meetings via a web link that they can open on their laptop, mobile app, or a phone line. Because of Coronavirus, the <a href="https://gsuite.google.com/products/meet/"><strong>Google Hangouts Meet</strong></a> video conferencing service is now free for all <a href="https://gsuite.google.com/"><strong>G Suite</strong></a> users until <strong><em>July 1, 2020</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*XTPQzAKPdHHXXAYSrF-1Nw.jpeg" /><figcaption>Source : <a href="https://www.theandroidsoul.com/google-hangouts-meet-apk-download/">https://www.theandroidsoul.com/google-hangouts-meet-apk-download/</a></figcaption></figure><blockquote><strong>The best solution for group calls</strong></blockquote><p><a href="https://www.skype.com/en/"><strong>Microsoft Skype</strong></a> is available for free for Android, iOS, Windows, and Mac. The application allows you to make video and audio calls and also has a messaging function. Its interface is user-friendly and can make up to <strong>50 </strong>audio calls (the number of video calls depends on the device used, according to <strong>Microsoft</strong>). You can also record, save and share your video calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/0*v8dXU5bW9Hsa3xt7.png" /><figcaption>Source : <a href="https://www.datacom.co.nz/Case-Studies/Software/Kensington-Swan.aspx">https://www.datacom.co.nz/Case-Studies/Software/Kensington-Swan.aspx</a></figcaption></figure><h3>Conclusion</h3><p>Working from home gives you the opportunity to work in optimal conditions, since you have access to all the amenities of your home. Take the opportunity to work on tasks that require concentration.</p><p>Not going to the office also saves you possible transport time. Time you can use for example to prepare balanced meals, rather than the traditional sandwich swallowed on the go, or to play sports.</p><blockquote>Due to telework, the productivity of the company is considerably affected, but it is certain that health takes precedence over all other interests!</blockquote><p>I tried to give you 3 tips that will help you be efficient while working from home. If you have another useful tip, don’t hesitate to share it with us in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b50e7f23e22e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/coronavirus-covid-19-3-tips-to-ensure-efficiency-in-teleworking-b50e7f23e22e">Coronavirus (COVID-19): 3 Tips to Ensure Efficiency in Teleworking</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Thought I Was Mastering Python Until I Discovered These Tricks]]></title>
            <link>https://medium.com/data-science/i-thought-i-was-mastering-python-until-i-discovered-these-tricks-e40d9c71f4e2?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e40d9c71f4e2</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Fri, 06 Mar 2020 01:43:36 GMT</pubDate>
            <atom:updated>2020-03-06T14:25:28.103Z</atom:updated>
            <content:encoded><![CDATA[<h4>Python best practices and tips that will make you code quickly and efficiently</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0zc5FpwQKOm5d7bIbozXCA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@alvaroreyes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alvaro Reyes</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Python is one of the most popular programming languages ​​for beginners, making it the most widely taught language in schools around the world.</p><p>However, learning Python is not an easy thing. To get started, you first need to find the best online way to get there, which is difficult in itself. There are thousands of different Python courses and tutorials, all claiming to be the best.</p><p>True, practice alone is not perfect, but perfect practice is. This means that you need to make sure that you are always following the best coding practices (commenting on your code, using correct syntax, etc.), otherwise you will likely end up adopting bad habits that could harm your future lines of code.</p><blockquote><em>“A universal convention supplies all of maintainability, clarity, consistency, and a foundation for good programming habits too. What it doesn’t do is insist that you follow it against your will. That’s Python!”</em></blockquote><blockquote><em>— Tim Peters on comp.lang.python, 2001–06–16</em></blockquote><p>In this article, I’m going to give my top 10 tips to help you code in Python quickly and efficiently.</p><h3>1. Readability is important</h3><blockquote>Programs must be written for people to read, and only incidentally for machines to execute.</blockquote><blockquote><a href="https://www.azquotes.com/author/38260-Hal_Abelson"><strong>Hal Abelson</strong></a></blockquote><p>First of all, try to make your programs easy to read by following some programming conventions. A programming convention is one that a seasoned programmer follows when writing his or her code. There’s no quicker way to reveal that you’re a <strong><em>“newbie”</em></strong> than by ignoring conventions. Some of these conventions are specific to <strong>Python</strong>; others are used by computer programmers in all languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*kjVAg20iyGR70YDZ3pd6qg.jpeg" /></figure><p>Essentially, readability is the characteristic which specifies how easy another person can understand some parts of your code (and not you!).</p><p>As an example, I was not used to write with vertical alignment and to align the function’s parameters with opening delimiter.</p><pre># No, to avoid: <br>func = long_function_name(var_one, var_two,<br>       var_three, var_four)</pre><pre>#Yes, <br>func = long_function_name(var_one, var_two,<br>                          var_three, var_four)</pre><p>Look at other examples in the <a href="https://www.python.org/dev/peps/pep-0008/">Style Guide for Python Code</a> and decide what looks best.</p><p>Another important thing that we do very often is resembling programs that we have seen or written before, which is why our exposure to readable programs is important in learning programming.</p><h3>2. Avoid unuseful conditions</h3><p>Often, a long <strong><em>if &amp; elif &amp; …. &amp; else</em></strong> conditions is the sign of code that needs refactoring, these conditions make your code lengthy and really hard to interpret. Sometimes they can easily be replaced, for example, I used to do the following:</p><pre>def f():<br>    if condition:<br>    	return True<br>    else:<br>    	return False</pre><p>This is just dumb! The function is returning a boolean, so why even use if blocks in the first place? The correct what of doing this would be:</p><pre>def f():<br>	return condition</pre><p>In a <a href="https://www.hackerrank.com/dashboard">Hackerrank</a> challenge, you are given the year and you have to write a function to check if the year is leap or not. In the Gregorian calendar three criteria must be taken into account to identify leap years:</p><ul><li>The year can be evenly divided by 4, is a leap year, unless:</li><li>The year can be evenly divided by 100, it is NOT a leap year, unless:</li><li>The year is also evenly divisible by 400. Then it is a leap year.</li></ul><p>So in this challenge, forget about <strong><em>ifs</em></strong> and <strong><em>elses</em></strong> and just do the following:</p><pre>def is_leap(year):<br>    return year % 4 == 0 and (year % 400 == 0 or year % 100 != 0)</pre><h3>3. Adequate use of Whitespace</h3><ul><li>Never mix tabs and spaces</li><li>A line break between functions</li><li>Two line breaks between classes</li><li>Add a space after <strong>“,”</strong> in dictionaries, lists, tuples, arguments in a list of arguments and after <strong>“:”</strong> in dictionaries but not before.</li><li>Put spaces around assignments and comparisons (except for arguments in a list)</li><li>No space for opening / closing parentheses or just before a list of arguments.</li></ul><pre>def function(key, value=0):<br>    &quot;&quot;&quot;Return a dictionary and a list...&quot;&quot;&quot;<br>    d = {key: value}<br>    l = [key, value]<br>    return d, l</pre><h3>4. Docstrings and Comments</h3><ul><li>Docstrings = How to use the code</li><li>Comments = Why (rational) and how the code works</li></ul><p>The docstrings explain how to use the code :</p><ul><li>Explain the purpose of a function even if it seems obvious to you because it will not necessarily seem obvious to another person later.</li><li>Describe the expected parameters, the returned values ​​and the exceptions raised.</li><li>If the method is strongly coupled to a single caller, mention the calling function.</li></ul><p>The comments explain what are for the maintainers of your code. Examples including notes for yourself, such as:</p><blockquote># !!! BUG: …</blockquote><blockquote># !!! FIX: This is a hack</blockquote><blockquote># ??? Why is this here?</blockquote><p>It is on your responsibility to write good docstrings and good comments, so always keep them up to date! When making changes, make sure the comments and docstrings are consistent with the code.</p><p>You will find a detailed <strong>PEP</strong> dedicated for Doctsrings : <a href="https://www.python.org/dev/peps/pep-0257/"><strong>“Docstring Conventions”</strong></a></p><h3>5. Variables and Assignment</h3><p>In other programming languages :</p><pre>c = a<br>a = b<br>b = c</pre><p>In Python, it’s better to use the assignment in one line code :</p><pre>b, a = a, b</pre><p>You may have already seen it but do you know how it works?</p><ul><li>The comma is the syntax for building the tuple.</li><li>A tuple is created on the right (tuple packing).</li><li>A tuple is the target on the left (tuple unpacking).</li></ul><p>Other examples:</p><pre>&gt;&gt;&gt; user =[&#39;Jan&#39;, &#39;Gomez&#39;, &#39;+1-888-222-1546&#39;]<br>&gt;&gt;&gt; name, title, phone = user<br>&gt;&gt;&gt; name<br>&#39;Jan&#39;<br>&gt;&gt;&gt; title<br>&#39;Gomez&#39;<br>&gt;&gt;&gt; phone<br>&#39;+1-888-222-1546&#39;</pre><p>Useful in loops on structured data (the variable <strong>user</strong> above has been kept):</p><pre>&gt;&gt;&gt; people = [user, [&#39;German&#39;, &#39;GBT&#39;, &#39;unlisted&#39;]]<br>&gt;&gt;&gt; for (name, title, phone) in people:<br>...      print (name, phone)<br>...<br>Jan +1-888-222-1546<br>German unlisted</pre><p>It is also possible to do the opposite way, just make sure you have the same structure on the right and on the left:</p><pre>&gt;&gt;&gt; jan, (gname, gtitle, gphone) = people<br>&gt;&gt;&gt; gname<br>&#39;German&#39;<br>&gt;&gt;&gt; gtitle<br>&#39;GBT&#39;<br>&gt;&gt;&gt; gphone<br>&#39;unlisted&#39;<br>&gt;&gt;&gt; jan<br>[&#39;Jan&#39;, &#39;Gomez&#39;, &#39;+1-888-222-1546&#39;]</pre><h3>6. List Concatenation &amp; Join</h3><p>Let’s start with a list of strings:</p><pre>colors = [&#39;red&#39;, &#39;blue&#39;, &#39;green&#39;, &#39;yellow&#39;]</pre><p>We want to concatenate these chains together to create a long one. Particularly when the number of substrings is large, avoid doing this :</p><pre>result = &#39;&#39;<br>for s in colors:<br>    result += s</pre><p>It is very slow. It uses a lot of memory and performance. The sum will add up, store, and then move on to each intermediate step.</p><p>Instead, do this:</p><pre>result = &#39;&#39;.join(colors)</pre><p>The <strong>join ()</strong> method makes the entire copy in one pass. When you only process a few strings, it makes no difference. But get into the habit of building your chains optimally, because with hundreds or thousands strings, it will truly make a difference.</p><p>Here are some techniques for using the <strong>join ()</strong> method. If you want a space as a separator:</p><pre>result = &#39; &#39;.join(colors)</pre><p>or a comma and a space:</p><pre>result = &#39;, &#39;.join(colors)</pre><p>To make a grammatically correct sentence, we want commas between each value except the last one, where we prefer an “or”. The syntax for splitting a list does the rest. The <strong>[: -1]</strong> returns everything except the last value, which we can concatenate with our commas.</p><pre>colors = [&#39;red&#39;, &#39;blue&#39;, &#39;green&#39;, &#39;yellow&#39;]<br>print (&#39;Choose&#39;, &#39;, &#39;.join(colors[:-1]), \<br>      &#39;or&#39;, colors[-1])</pre><pre>&gt;&gt; Choose red, blue, green or yellow</pre><h3>7. Test true conditions</h3><p>It is elegant and quick to take advantage of Python with regard to Boolean values:</p><pre># Do this :     # And not this :<br>if x:             if x == True:<br>   pass                  pass</pre><pre># Do this :     # And not this :<br>if items:         if len(items) != 0:<br>    pass                pass</pre><pre># and especially not that :<br>        if items != []:<br>               pass</pre><h3>8. Use enumerate when it’s possible</h3><p>The <strong><em>enumerate</em></strong> function takes a list and returns pairs (index, item):</p><pre>items = [&#39;zero&#39;, &#39;one&#39;, &#39;two&#39;, &#39;three&#39;]<br>&gt;&gt;&gt; print list(enumerate(items))<br>[(0, &#39;zero&#39;), (1, &#39;one&#39;), (2, &#39;two&#39;), (3, &#39;three&#39;)]</pre><p>It is necessary to use a list to display the results because <strong><em>enumerate</em></strong> is a lazy function, generating one item (a pair) at a time, only when requested. A for loop requires such a mechanism. <strong><em>Print</em></strong> does not take one result at a time but must be in possession of the entire message to be displayed. We therefore automatically converted the generator to a list before using print.</p><p>So, using the loop below is much better:</p><pre>for (index, item) in enumerate(items):<br>    print (index, item)</pre><pre># compared to :              # And :<br>index = 0                     for i in range(len(items)):<br>for item in items:                    print (i, items[i])<br>    print (index, item)<br>    index += 1</pre><p>The version with <strong>enumerate</strong> is shorter and simpler than the two other versions. An example showing that the enumerate function returns an iterator (a generator is a kind of iterator)</p><h3>9. List Comprehension</h3><p>The traditional way with <strong>for</strong> and<strong> if</strong>:</p><pre>new_list = []<br>for item in a_list:<br>    if condition(item):<br>        new_list.append(fn(item))</pre><p>Using a list comprehension:</p><pre>new_list = [fn(item) for item in a_list if condition(item)]</pre><p>The listcomps are clear and direct. You can have several <strong><em>for loops</em></strong> and <strong><em>if conditions</em></strong> within the same listcomp, but beyond two or three, or if the conditions are complex, I suggest you use the usual <strong><em>for loop</em></strong>.</p><p>For example, the list of squares from 0 to 9:</p><pre>&gt;&gt;&gt; [n ** 2 for n in range(10)]<br>[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]</pre><p>The list of odd numbers within the previous list:</p><pre>&gt;&gt;&gt; [n ** 2 for n in range(10) if n % 2]<br>[1, 9, 25, 49, 81]</pre><p>Another example:</p><pre>&gt;&gt;&gt;   [(x, y) for x in (1, 2, 3, 4) if x % 2 == 0 <br>       for y in [&#39;a&#39;, &#39;b&#39;] if y == &#39;b&#39;]<br>[(2, &#39;b&#39;), (4, &#39;b&#39;)]</pre><h3>10. Generator Expressions</h3><p>Let’s sum the squares of numbers less than 100:</p><pre># With a loop :<br>total = 0<br>for num in range(1, 101):<br>    total += num * num</pre><p>We can also use the <strong><em>sum</em></strong> function which does the job faster for us by building the right sequence.</p><pre># With a list comprehension :<br>total = sum([num * num for num in range(1, 101)])</pre><pre># With a generator expression :<br>total = sum(num * num for num in xrange(1, 101))</pre><p>The generator expressions are like list comprehensions, except in their calculation, they are lazy. Listcomps calculate the entire result in a single pass, to store it in a list. Generator expressions calculate one value at a time, when necessary. This is particularly useful when the sequence is very long and the generated list is only an intermediate step and not the final result.</p><p>For example if we have to sum the squares of several billion integers, we will reach a saturation of the memory with a list comprehension, but the generator expressions will not have any problem. Well it will take a while though!</p><pre>total = sum(num * num for num in range(1, 1000000000))</pre><p>The difference in syntax is that listcomps have square brackets, while generator expressions do not. Generator expressions sometimes require parentheses, so you should always use them.</p><p>In short :</p><ul><li>Use a <strong>list comprehension</strong> when the expected result is the list.</li><li>Use a <strong>generator expression</strong> when the list is only an intermediate result.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*1uaf8qjQBqR_hXPt.jpg" /><figcaption>Source : <a href="https://www.azquotes.com/quote/669106">https://www.azquotes.com/quote/669106</a></figcaption></figure><h3>Conclusion</h3><p>In this article, I have presented some of my best tips for learning to program in Python. If you really want to become a programmer or add a coding skill to your skills, learning Python is a great place to start. Look for high quality Python training online and start to find out how to program in Python. I recommend that you learn the basics with an interactive course before moving on to more difficult concepts.</p><p>You should not speed up the learning process too much, or you may miss important information. Take notes and make sure to review them regularly and try to practice writing code as often as possible.</p><p>Connect with colleagues who are learning like you and don’t be afraid to ask questions when you have them. Helping others when they have problems can be a great review, and working with someone else’s code is a great way to learn new things.</p><p>If you do all of this, nothing can stop you! So what are you waiting for? Start programming in Python now!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e40d9c71f4e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/i-thought-i-was-mastering-python-until-i-discovered-these-tricks-e40d9c71f4e2">I Thought I Was Mastering Python Until I Discovered These Tricks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter Deep-Dive Analysis: US Presidential Election]]></title>
            <link>https://medium.com/analytics-vidhya/twitter-deep-dive-analysis-us-presidential-election-d6b32acc449b?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6b32acc449b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[elections]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Mon, 10 Feb 2020 08:24:41 GMT</pubDate>
            <atom:updated>2020-02-10T08:24:41.435Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Method for Predicting the Winner of the next USA Presidential Elections in 2020 using Data extracted from Twitter</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kxU-Kk5FRVsbADJxDWgrhw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@historyhd?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">History in HD</a> on <a href="https://unsplash.com/s/photos/trump-election?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>General Introduction :</h3><p>Lately, the data science community has turned its interest in analyzing web data, such as blog posts or social networks’ users’ activity as an alternative way to make an accurate prediction for the upcoming <strong><em>US Presidential Election </em></strong>on <strong>November 2020</strong>.</p><p>Furthermore, traditional polls are too costly, while online information has become easy to obtain and freely available. One of the social media that allows researchers to use their data is <strong>Twitter</strong>. It has more than <strong><em>200 million</em></strong> visitors on a monthly basis and <strong><em>500 million</em></strong> messages daily. The advantages of using tweets from <strong>Twitter</strong> as a data source:</p><ul><li>First, the number of tweets is very huge and they are available to the public.</li><li>Second, tweets contain the opinion of people including their political view.</li></ul><p>So <strong>Twitter</strong> has become the first place voters go to seek accurate information and breaking news from political candidates during the presidential campaign. The company<strong> </strong>understood the significance of this responsibility and its teams started building new tools for people who use <strong>Twitter</strong> to identify original sources and authentic information.</p><p>In fact, <strong>Twitter</strong> decided to bring back candidate labels for the <strong>2020 US Elections</strong>, the labels will appear on a candidate’s profile page, as well as any tweets and retweets they share. They will tell you the office the individual is running for, as well as in what state and district. <strong><em>(</em></strong><a href="https://www.theverge.com/2019/12/12/21011960/twitter-election-labels-2020-candidates-house-senate-gubernatorial-races"><strong><em>Read more</em></strong></a><strong><em>)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdotKWxNGBrBOR5I37SvUw.png" /><figcaption>Image: Twitter</figcaption></figure><h3>Case Study :</h3><p>One of the major battlegrounds of the 2016 presidential election was <strong>Twitter</strong>. It was one of the most publicized <em>US Presidential Election</em> in the history and <strong>Twitter</strong> had played an increasingly prominent role in it.</p><p>Now let’s focus on the cases of the two then-front runners for the <em>2016 U.S </em>presidential election: <a href="https://twitter.com/realDonaldTrump?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor"><strong>Donald Trump</strong></a><strong> and </strong><a href="https://twitter.com/HillaryClinton?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor"><strong>Hillary Clinton</strong></a>, the present study examines differences in the ways in which they self-present and communicate with voters through their <strong>Twitter</strong> account.</p><p>In line with the findings from previous studies on how women candidates tweet, this complete analysis help me find that <a href="https://twitter.com/HillaryClinton?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor"><strong>Hillary Clinton</strong></a> out tweeted the Republican nominee, attacked him more often than he did her and tweeted about political issues far more often.</p><p>By the way, I’ve conducted another analysis on how candidates for the <strong>US Congress</strong> use Twitter, you can have a look at the article <a href="https://towardsdatascience.com/are-you-democrat-or-republican-let-your-tweets-define-you-4aa4cadf4bea">here</a>.</p><h3><strong>Tweets Collection :</strong></h3><p>This dataset contains the tweets related to the 2016 United States presidential election. They were collected between <em>July 13, 2016</em> and <em>November 10, 2016 </em>from the <strong>Twitter API</strong> using <a href="https://gwu-libraries.github.io/sfm-ui/">Social Feed Manager</a>. It is an open source software that harvests social media data and web resources from <strong>Twitter</strong>, <strong>Tumblr </strong>and <strong>Flickr</strong>. I highly recommand using <strong>SFM</strong> to collect data and this is how I dit :</p><h4>1- Define and organize collections</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xoq4LVxUGRy5lZ-Z.png" /></figure><h4>2- Add collections with various harvest types</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/0*DcPWc8RPtYi8uyvQ.png" /></figure><h4>3- Add seeds</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*veYuyNNvhh6uw8c0.png" /></figure><h4>4- Turn it on, <strong>SFM</strong> will harvest the data on an on-going basis or according to the schedule you specify</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XSxOhYkbS0-NDnCd.png" /></figure><h4>5- Export a collection to a spreadsheet</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5U9oTNP2HmCiHa7Y90hFvg.jpeg" /></figure><p>You can also feed data into your own processing pipeline from the command line. Actually, no specific expertise is needed to use <strong>SFM</strong>, but it is helpful to read the <a href="https://sfm.readthedocs.org/">SFM documentation</a> and be familiar with social media APIs.</p><h3>Exploratory Data Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*asU1IYTOur2w_hp_wZx8Iw.png" /></figure><p>I used a Tweets dataset which contain <strong>~3000</strong> recent tweets from <strong>Hillary Clinton</strong> and <strong>Donald Trump</strong>, the two major-party presidential nominees.</p><p>So we have approximately the same number of tweets for the two candidates. We can say that our data set is well balanced.</p><h4>1. Tweets over time</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*xVz4bqjp7k1mmIGLAoreFg.png" /></figure><p>We can see that <strong>Trump</strong> started tweeting since the beginning of the year, and also the influx of <strong>Clinton</strong> tweets starting around <strong><em>July 16</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*oAgI1S1L_M-EyyedahQxUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*meYsKTHQr9lOcNE4exiVbQ.png" /></figure><h4>2. When do candidates tweet?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*x-uv_lUis81gj0YvmXSxPQ.png" /></figure><p>Moreover it’s interesting to see the hourly tweets tendency, canditates tend to communicate more in the afternoon and the night .</p><h4>3. Percentage of retweets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*aa5bsidj_-XtSWZD3D_Zfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*DmHFojOWVaGmVms0pxSooQ.png" /></figure><p>From this we can say that <strong>Donald Trump</strong> has used a little a bit more twitter than <strong>Hillary Clinton</strong> but <strong>Clinton’s</strong> tweets have been more retweeted than <strong>Trump’s</strong> one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*isY32D3E5kxNpe249sa-mA.png" /></figure><h4>4. Tweets Language</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*PuVzd-DlD-Jplin9A6Txxw.png" /><figcaption>Tweets languages in %</figcaption></figure><p>As we can see, most of the tweets used by the two candidates were written in <strong>English</strong>. But we have more than 100 tweets in <strong>Spanish</strong> :</p><pre>HillaryClinton     102<br>realDonaldTrump      3<br>Name: handle, dtype: int64</pre><p>This does make sense because <strong>Clinton</strong>’s campaign has made efforts to mobilize <strong><em>Latino</em></strong> voters who will be critical in the general election. In addition, she has a strong network of <strong><em>Latino</em></strong> leaders and activists.</p><h4>5. Original authors of retweets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*8esBfLUg2buWy0UFFJ2bsA.png" /></figure><h3>Data Analysis :</h3><p><strong>Extract Trump’s and Clinton”s Tweets</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62f231960a31a39654454ea593a7be81/href">https://medium.com/media/62f231960a31a39654454ea593a7be81/href</a></iframe><h4>1. Word Clouds</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14152995b30a4c37619ae29e88109876/href">https://medium.com/media/14152995b30a4c37619ae29e88109876/href</a></iframe><ul><li>Hillary Clinton’s Word Cloud :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*DY93SBrDGSTQQ8P72JjnOw.png" /></figure><ul><li>Donald Trump’s Word Cloud :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*TJoSUEbxlSkqh7mhRf7ktQ.png" /></figure><blockquote>Hillary Clinton spent significantly more time on political issues than Trump, who barely mentioned any issues on Twitter.</blockquote><h4>2. Mentions :</h4><p><strong>Extract mentions from tweets</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21dfc8eaefc18b8a293cb57058b790b8/href">https://medium.com/media/21dfc8eaefc18b8a293cb57058b790b8/href</a></iframe><p><strong>Graph mentions most used by candidates (top 10)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*ooB7gW4U7WsKrTu-kWY_mA.png" /></figure><p>While <strong>Hillary Clinton</strong> only mentions other <strong>Twitter</strong> users in every fifth tweet, most of the user mentions are for <strong>@POTUS</strong> (+125 mentions) then we have the <strong>@realDonaldTrump</strong> who is more than any other <strong>Twitter</strong> user mentioned on her account, including <strong>@TimKaine</strong> and <strong>@JoeBiden</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*3DRPTlDorcpME2y3OkNSEQ.png" /></figure><p><strong>Donald Trump</strong>, on the other hand, does not mention <strong>@HillaryClinton’s</strong> Twitter handle. <strong>Donald Trump</strong> consistently mentions other Twitter users in two-thirds of his tweets, often mentioning his own account <strong>@realDonaldTrump</strong> (+280 mentions), but also <strong>@FoxNews</strong> (+70 mentions), <strong>@CNN</strong> (+60) and Fox News anchor <strong>@MegynKelly</strong> (+52 mentions) for their allegedly biased and unfair reporting.</p><p>Seems like one thing both candidates had in common was their frequent mention of <strong>Trump</strong>. Surprinsingly, while <strong>Trump</strong> was <strong>Hillary</strong>’s 2nd most mentioned account, she was not on <strong>Trump</strong>’s list.</p><blockquote>It remains to be seen which strategy is better: engaging with and mentioning your opponent’s Twitter account or simply ignoring it.</blockquote><h4>3. Hashtags Wars:</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7bf7d6d68e78254e53d820516ef4627/href">https://medium.com/media/f7bf7d6d68e78254e53d820516ef4627/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*BqfmpD4gEYW6Mg-bab88dA.png" /></figure><p><strong>Hillary Clinton</strong> is more sparing, using hashtags in only <strong>18%</strong> of her tweets; most commonly <strong>#DemsInPhilly</strong> and <strong>#DebateNight</strong>. She has also used hashtags to react to her opponent during the <strong><em>Republican National Convention </em></strong>(<strong>#RNCinCLE</strong> used 35 times).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*6BQoBXp4ZeMoBvRhEm30Og.png" /></figure><p><strong>Donald Trump</strong> includes a hashtag in almost every other tweet, including <strong>#Trump</strong>, used <strong><em>+340</em></strong> times, and <strong>#MakeAmericaGreatAgain</strong>, used <strong><em>246</em></strong> times.</p><blockquote>It seems <strong>Trump</strong> was much more fond of hashtags than Hillary, using his favorite hashtag (#Trump) almost 9x more than she used her favorite (#DemsInPhilly)</blockquote><p><strong>Network of hashtags</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d98c3eb77e1cb40bd2089a8e5d232fe0/href">https://medium.com/media/d98c3eb77e1cb40bd2089a8e5d232fe0/href</a></iframe><p><strong>Account to Hashtags : </strong><a href="http://twitter.com/realDonaldTrump"><strong>@realDonaldTrump</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*I0tD9csGjnyB5iDhvLuXug.png" /></figure><p><strong>Account to Hashtags : </strong><a href="http://twitter.com/FoxNews">@FoxNews</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*KZ0NSFCrHY85x59zRaIKtw.png" /></figure><h3>Sentiment Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*htrxRmLzyQ-BB12_.png" /></figure><p>I tried to categorize the tweet column into <strong><em>Positive</em></strong> and <strong><em>Negative</em></strong> sentiments using <a href="https://textblob.readthedocs.io/en/dev/"><strong>TextBlob</strong></a>. The package contains several sentiment lexicons in the sentiments dataset.</p><p><a href="https://textblob.readthedocs.io/en/dev/"><strong>TextBlob</strong></a> aims to provide access to common text-processing operations through a familiar interface. You can treat <a href="https://textblob.readthedocs.io/en/dev/api_reference.html#textblob.blob.TextBlob"><strong>TextBlob</strong></a> objects as if they were Python strings that learned how to do <strong><em>Natural Language Processing</em></strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a00ee540086525296566d0b87d5b987/href">https://medium.com/media/2a00ee540086525296566d0b87d5b987/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*5HgOJ3uH6vRtIFbShIMghQ.png" /></figure><p>Unlike what we could have think, most of the tweets are <strong>postive</strong> and <strong>neutral</strong>. Now I’ll try to focus on each candidate separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCpYWDntO9AANFEdxSbZ8g.png" /></figure><p>It’s really interesting to see that <strong>Donald Trump</strong> has a really large number of positive tweets, their represent nearly the double of his neutral or negative tweets.</p><h3>Impersonating the candidates</h3><p>Thanks to <a href="https://github.com/jsvine/markovify"><strong>Markovify</strong></a>, we can use a <strong>Markov Chain</strong> <strong>Model</strong> to create synthetic tweets based on an existing library of actual tweets. Right now, its main use is for building <strong>Markov models</strong> of large corpora of text and generating random sentences from that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8052886f7242cb4557cb73bc8257c1e/href">https://medium.com/media/f8052886f7242cb4557cb73bc8257c1e/href</a></iframe><pre>Hillary<br>- Watch President Obama &amp;amp; Vice President @JoeBiden had the keys to our national debt.<br>- There’s no doubt about it—Donald Trump is unfit to be Commander-in-Chief.</pre><pre>Donald<br>- Rowanne Brewer, the most prominently depicted woman in the Republican Party can unify!<br>- I would like to express my warmest regards, best wishes and condolences to Dwyane Wade and his family, on the East coast.</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67f83b42c2e534648fe023c914211147/href">https://medium.com/media/67f83b42c2e534648fe023c914211147/href</a></iframe><pre>Hillary<br>- They don’t just want us to solve them.<br>- They are trying to scam America the clean energy superpower of the iceberg.</pre><pre>Donald<br>- We are superior and have a clue.<br>- We said no and she went with Obama - and I will beat Hillary!</pre><blockquote>Sometimes this works and it helps point the fake tweets in a certain direction.</blockquote><h3>Predicting who said it: Trump or Clinton</h3><p>In this last section I’ll attempt to build models to correctly predict the author of a given tweet. To identify each author, I’ll create a bag of words containing the most-common words of the 2 authors combined. Bag of words is a list of the most common words of a given source of text. This set later becomes the basis for feature engineering.</p><p>I started with applying a <strong>Multinomial Naive Bayes</strong> Algorithm as a classifier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3378bc9d1cc65a071a5184853255990/href">https://medium.com/media/b3378bc9d1cc65a071a5184853255990/href</a></iframe><pre>Accuracy on training set:  96.34%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*Kn9hfpipLor3Bm-Kk9uJ9Q.png" /><figcaption>Confusion Matrix on the training set</figcaption></figure><p>Then I decided to improve the model by using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html"><strong>Pipeline</strong></a><strong> </strong>and<a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html"> <strong>Cross Validation</strong></a><strong> :</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/896794b7aa356c6e6bb67fa878329251/href">https://medium.com/media/896794b7aa356c6e6bb67fa878329251/href</a></iframe><pre>[0.9244 0.9264 0.9399 0.9147 0.9264 0.9186 0.905  0.8951 0.9261 0.9047] <br><br>Mean score: 0.9181311287268004<br>Stdev: 0.012670848631606169</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*9nYQWcErHJIDGZ49iC-9lA.png" /><figcaption>Confusion Matrix on the testing set</figcaption></figure><p>So this time we got an accuracy on the testing set of <strong>92%.</strong> We can say that the NLP Powered Model has been successful in effectively classifying <strong>92%</strong> unknown (Test Set) examples correctly. In other words, 92% of tweets are identified correctly that it belongs to which author among the two.</p><p>Now let’s test our prediction model and see what</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebde129c948cf3960ef2d5d6fae8d849/href">https://medium.com/media/ebde129c948cf3960ef2d5d6fae8d849/href</a></iframe><pre>Tweet #1: &#39; We have to build an economy that works for everyone, not just those at the top. &#39;  <br> <br> I&#39;m about <strong>89%</strong> sure this was tweeted by <strong>HillaryClinton</strong></pre><pre>Tweet #2: &#39; My lawyers want to sue the failing @nytimes so badly for irresponsible intent. &#39;  <br> <br> I&#39;m about <strong>88%</strong> sure this was tweeted by <strong>realDonaldTrump</strong></pre><h3>Sum-Up :</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/0*piH1pXx9a0x4VfQI.jpg" /></figure><p>The concept behind this study is to collect tweets using <strong>Twitter’s API</strong> and apply different algorithms in order to classify them and find trends in what candidates are saying about a specific topic. We were able to show how the social media, more precisely <strong>Twitter</strong> can be used to make prediction of future outcome such as election, to extract the sentiment or views of people who are likely to vote in the general election or have an influence on those who will vote, and <strong>Sentiment Analysis</strong>, to classify their sentiment.</p><p>So the goal was to gather tweets that refer to the elections and more specifically to the two main candidates of the US presidential elections: <strong>Hillary Clinton</strong> and <strong>Donald J. Trump</strong>.</p><h3>Further Work:</h3><p>Although, we still can apply our study on the future candidates of the presidential election in <strong>2020</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PcE114_zebI2c75pY3rYVQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@element5digital?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Element5 Digital</a> on <a href="https://unsplash.com/s/photos/election-2020?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The approach is by using selected data just weeks before the election. The prediction could be derived by comparing the number of tweets mentioning each candidate or by comparing the number of tweets that has positive sentiments towards each candidate. As we saw in this article, the number of tweets mentioning a party reflects the election result and the <strong>Sentiment Analysis</strong> approach from <strong>Twitter</strong> can reduce the error of the prediction result.</p><p>We can other methods like utilizing interaction information between potential voters and the candidates in each state separately. We can also create trend line from the changes in follower of the candidates and the size of candidates’ network (followers on Twitter and friends on Facebook).</p><blockquote>What you should keep in mind is that despite the huge size of information that we can extract from social media, it still has small effect on the election results. Therefore, it only makes a difference in a closely contested election.</blockquote><h3>References :</h3><ul><li>Social Feed Manager : h<a href="https://sfm.readthedocs.io/en/latest/index.html">ttps://sfm.readthedocs.io/en/latest/index.html</a></li><li>TextBlob : <a href="https://textblob.readthedocs.io/en/dev/index.html">https://textblob.readthedocs.io/en/dev/index.html</a></li><li>Generating WordClouds in Python: <a href="https://www.datacamp.com/community/tutorials/wordcloud-python">https://www.datacamp.com/community/tutorials/wordcloud-python</a></li><li>Markovify : <a href="https://github.com/jsvine/markovify">https://github.com/jsvine/markovify</a></li><li><strong>How Facebook, Google and Twitter ‘embeds’ helped Trump in 2016 : </strong><a href="https://www.politico.com/story/2017/10/26/facebook-google-twitter-trump-244191">https://www.politico.com/story/2017/10/26/facebook-google-twitter-trump-244191</a></li><li><strong>Election Update: The How-Full-Is-This-Glass Election : </strong><a href="https://fivethirtyeight.com/features/election-update-the-how-full-is-this-glass-election/">https://fivethirtyeight.com/features/election-update-the-how-full-is-this-glass-election/</a></li></ul><p>Congratulations if you managed to get here. Thanks for reading, I hope you’ve liked it. For personal contact or discussion on Machine Learning, feel free to reach out to me on <a href="https://www.linkedin.com/in/kamal-chouhbi/"><strong>LinkedIn</strong></a><strong> </strong>and don’t forget to follow me on <a href="https://github.com/chouhbik"><strong>GitHub</strong></a> and <a href="https://medium.com/@chouhbik"><strong>Medium</strong></a><strong>.</strong></p><p><a href="https://github.com/chouhbik">chouhbik - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6b32acc449b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/twitter-deep-dive-analysis-us-presidential-election-d6b32acc449b">Twitter Deep-Dive Analysis: US Presidential Election</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks Help Upscale Conversion of Famous 1896 Video to 4K Quality]]></title>
            <link>https://medium.com/data-science/neural-networks-help-upscale-conversion-of-famous-1896-video-to-4k-quality-d2c3617310fe?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2c3617310fe</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[video-editing]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Sat, 08 Feb 2020 10:45:31 GMT</pubDate>
            <atom:updated>2020-02-08T14:57:57.577Z</atom:updated>
            <content:encoded><![CDATA[<h4>AI makes it possible to reproduce in 4K ‘The arrival of a train at La Ciotat station’ by the Lumière brothers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*E9KjPrQ3v7AuT9qyajz4Pw.gif" /><figcaption>Arrival of a Train at La Ciotat (The Lumière Brothers, 1896)</figcaption></figure><p><strong>Neural networks</strong> are becoming more sophisticated over time. They are already allowing scientists to reconstruct human thoughts from brain waves in real time and many other things.</p><p>In <strong>1896</strong>, the <a href="https://en.wikipedia.org/wiki/Auguste_and_Louis_Lumi%C3%A8re"><strong>Lumière brothers</strong></a><strong>’</strong> film <strong><em>“The arrival of a train at La Ciotat station”</em></strong>, lasting <strong>50</strong> seconds, aroused the emotion of spectators and consecrated the first steps in cinema. It remains <strong>an emblematic symbol of the work of the Lumière brothers</strong> and more generally of the beginnings of the <strong>7th art</strong>.</p><p>Recently, a man named <a href="https://www.youtube.com/channel/UCD8J_xbbBuGobmw_N5ga3MA"><strong>Denis Shiryaev</strong></a> used Neural Networks to scale the <strong>1896</strong> film to <strong>4K</strong> resolution using <a href="https://topazlabs.com/"><strong>Topaz Labs’</strong></a> commercial <a href="https://topazlabs.com/gigapixel-ai/"><strong>Gigapixel AI</strong></a> and <a href="https://sites.google.com/view/wenbobao/dain"><strong>DAIN</strong></a> image editing applications. He not only increased the footage to <strong>4K</strong>, but also increased the frame rate to <strong>60</strong> frames per second.</p><p><strong>Shiryaev</strong> used <strong>Gigapixel AI</strong> to render the <strong>50</strong>-second mute short film <strong><em>“The Arrival of a Train at La Ciotat Station”</em></strong> from <strong>1896</strong> in <strong>4K</strong> format with very realistic images. The original film has very blurred quality and very low resolution. After publishing his work, the result obtained were surprising. His new video seems very well to meet the video standards of the <strong><em>21st</em></strong> century.</p><p>Despite some imperfections, such as image jumps or blurring of certain details, <strong>the fruit of this experience is simply impressive</strong> . To complete his work, <strong>Shiryaev</strong> also added <strong>sound</strong> to this silent film.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3RYNThid23g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3RYNThid23g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3RYNThid23g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f964d25019d0e617539bab6a8e034f9e/href">https://medium.com/media/f964d25019d0e617539bab6a8e034f9e/href</a></iframe><p>These images were obtained thanks to a form of artificial neural networks: <strong>Generative Adversarial Networks</strong>, or <strong>GANs</strong>. This technique has been popularized in recent months, in particular by the emergence of the famous Kaggle competition <a href="https://www.kaggle.com/c/deepfake-detection-challenge/overview"><strong>“DeepFake Detection Challenge”</strong></a> and artistic creations generated by <strong>AI</strong>.</p><p>As a reminder, the <strong>GAN</strong> method consists <strong>in opposing two neural networks</strong> . The first generates an object based on data on which it was trained, while the second has the role of determining the authenticity of the object. The first network tries again and again, until its product is perfect enough to deceive the second. In this way, the program learns itself to generate the most likely image possible.</p><p><a href="https://topazlabs.com/gigapixel-ai/"><strong>Gigapixel AI</strong></a> software uses a proprietary interpolation algorithm which analyzes the image, recognizes details and structures and completes the image. It can make images sharper and clearer, even after they have been enlarged by <strong><em>600%</em></strong>.</p><p><a href="https://sites.google.com/view/wenbobao/dain"><strong>DAIN</strong></a><strong> <em>(Depth-Aware Video Frame Interpolation)</em></strong>, on the other hand, imagines and inserts frames between the keyframes of an existing video clip. In other words, <strong>DAIN</strong> analyzes and maps video clips, then inserts the fill images generated between the existing images. It’s the same concept as the motion smoothing function of <strong><em>4K TVs</em></strong>. To achieve the same result in this <strong>1896</strong> video, <strong>Shiryaev</strong> added enough images to increase the rate to <strong>60 FPS</strong>. The <strong>DAIN</strong> therefore automatically generated then added<strong> 36</strong> additional images per second in addition to the <strong>24</strong> images originally recorded.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMT-70ni4Ddo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMT-70ni4Ddo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMT-70ni4Ddo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea30a9dbdb1c81468210d2ff4b59d6c9/href">https://medium.com/media/ea30a9dbdb1c81468210d2ff4b59d6c9/href</a></iframe><p>The improved version of <strong>Shiryaev’s</strong> <strong><em>“Arrival of a train at La Ciotat station” </em></strong>looks like a recent film made using a <strong>smartphone </strong>or <strong>GoPro</strong>. The comparison with the original video is <strong>quite amazing, especially in the fluidity </strong>won in the new version. The reflections of the sun on the moving train and the women’s jewelry on the platform are particularly well transcribed and constitute perhaps one of the most remarkable improvements.</p><p>Neural networks can do the coloring, too, using the same basic technique: start with a bunch of color photos, convert them to black and white, and then train a neural network to reconstruct the color originals. <strong>Denis Shiryaev</strong> even had the task of adding colors to his improved version of the <strong>“<em>Arrival of a train at La Ciotat station”</em></strong> on the <a href="https://youtu.be/EqbOhqXHL7E"><strong><em>DeOldify Videos YouTube channel</em></strong></a> .</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEqbOhqXHL7E%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEqbOhqXHL7E&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEqbOhqXHL7E%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5538f0b1db6cc28e6a40f9188256024c/href">https://medium.com/media/5538f0b1db6cc28e6a40f9188256024c/href</a></iframe><p>However, people believe that <strong>Shiryaev’s</strong> technique is very well thought out and very promising, since it could well inspire other people in love with silent cinema.</p><p>The neural network algorithms are very powerful and are improved every year. The possibilities for use are very numerous and the television world may well be one of the biggest beneficiaries.</p><blockquote>“ Show the neural network a low-resolution image of a face and it will figure out that it’s a face and fill in the right details for the subject’s eyes, nose, and mouth.” says Topaz Labs.</blockquote><p>Unfortunately, these useful technologies are sometimes used for malicious purposes, they can be used for example to make “deepfakes” and allow identity theft in a video.</p><p>So, there is cause for concern about the misuse that can be made of it, but technology can just as easily be a way of using <strong>AI</strong> to update old cinema and revolutionize the industry itself.</p><h3>References:</h3><ul><li><a href="https://gizmodo.com/neural-networks-upscale-film-from-1896-to-4k-make-it-l-1841449875">https://gizmodo.com/neural-networks-upscale-film-from-1896-to-4k-make-it-l-1841449875</a></li><li><a href="https://arstechnica.com/science/2020/02/someone-used-neural-networks-to-upscale-a-famous-1896-video-to-4k-quality/">https://arstechnica.com/science/2020/02/someone-used-neural-networks-to-upscale-a-famous-1896-video-to-4k-quality/</a></li><li><a href="https://www.youtube.com/watch?v=3RYNThid23g&amp;feature=share&amp;fbclid=IwAR1pJWZhxp86_5EqEGhZflb5fU3g0wvKEDd2eOSdwEzPdGKDlhKhI5NzLWU">https://www.youtube.com/watch?v=3RYNThid23g&amp;feature=share&amp;fbclid=IwAR1pJWZhxp86_5EqEGhZflb5fU3g0wvKEDd2eOSdwEzPdGKDlhKhI5NzLWU</a></li><li><a href="https://sites.google.com/view/wenbobao/dain">https://sites.google.com/view/wenbobao/dain</a></li><li><a href="https://topazlabs.com/gigapixel-ai/">https://topazlabs.com/gigapixel-ai/</a></li></ul><p>Congratulations if you managed to get here. Thanks for reading, I hope you’ve liked it. For personal contact or discussion on Machine Learning, feel free to reach out to me on <a href="https://www.linkedin.com/in/kamal-chouhbi/"><strong>LinkedIn</strong></a><strong> </strong>and don’t forget to follow me on <a href="https://github.com/chouhbik"><strong>GitHub</strong></a> and <a href="https://medium.com/@chouhbik"><strong>Medium</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2c3617310fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/neural-networks-help-upscale-conversion-of-famous-1896-video-to-4k-quality-d2c3617310fe">Neural Networks Help Upscale Conversion of Famous 1896 Video to 4K Quality</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are you Democrat or Republican? Let your tweets define you …]]></title>
            <link>https://medium.com/data-science/are-you-democrat-or-republican-let-your-tweets-define-you-4aa4cadf4bea?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4aa4cadf4bea</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[us-elections]]></category>
            <category><![CDATA[democrats-republicans]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Wed, 05 Feb 2020 15:32:38 GMT</pubDate>
            <atom:updated>2020-02-05T19:53:56.783Z</atom:updated>
            <content:encoded><![CDATA[<h3>Are you Democrat or Republican? Let your tweets define you …</h3><h4>It feels like magic when you analyze hundreds of tweets in just a few seconds and automatically get information such as topic, opinion and sentiment.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8vdMh8bUU9YjWUuBnKvIw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mcoswalt?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Maria Oswalt</a> on <a href="https://unsplash.com/s/photos/republican-democrat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Lately, I had a conversation about politics and the next <strong>US Presidential Election of 2020</strong>. What caught my attention is that people get super polarized about it. They started to say: <strong><em>“You sound just like a Democrat!”</em></strong> or <strong><em>“Are you a anti-Republican?”</em></strong>. And by force, you start to have the arguments to defend a position. So I quickly went and took some online tests trying to answer the question: <strong>“Are you a Democrat or a Republican?”</strong></p><p>Unfortunately, I didn’t feel convinced, so I wanted to find a new way and thankfully we have the world of <strong>Twitter</strong> as our dataset!</p><p>In this study, I will try to find answers for the questions below:</p><ul><li>What are the words (terms, hashtags…) <strong>Democrats</strong> and <strong>Republicans</strong> use the most in their tweets?</li><li>Who gets more re-tweets and likes?</li><li>Can we do a <strong>Sentiment Analysis</strong> on the extracted tweets?</li><li>Can we use <strong>Machine Learning</strong> algorithms to train a model and determine if the tweet was written by a <strong>Democrat</strong> or a <strong>Republican</strong> candidate?</li><li>Will a <strong>Neural Network (RNN)</strong> help me decide if my <strong>Twitter</strong> account is more <strong>Democratic/Republican</strong>?</li></ul><h3>Data Collection :</h3><p>To start we need to figure out what <strong>Twitter</strong> accounts are <strong>Democrat</strong> or <strong>Republicans</strong>. <a href="http://www.tweetcongress.org/tweeters"><strong><em>TweetCongress.org</em></strong></a>, a directory of members of Congress on Twitter, lists <strong>101 Republicans</strong> on the site and just <strong>57 Democrats</strong>. Each party is mostly held up by a few <strong>Twitter</strong> superstars (like <strong>Democratic </strong><a href="https://twitter.com/clairecmc?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor"><strong>Sen. Claire McCaskill</strong></a> and <strong>Republican </strong><a href="https://twitter.com/senjohnmccain?lang=en"><strong>Sen. John McCain</strong></a>), but congressional <strong>Republicans</strong> overall have more followers and tweet more often.</p><p>Another option is to look directly at the <a href="https://twitter.com/TheDemocrats/lists/house-democrats/members">@TheDemocrats</a> and <a href="https://twitter.com/HouseGOP/lists/house-republicans/members">@HouseGOP </a>member lists on Twitter. After exploring the lists, you can see that we can pull all of the <strong>fullname</strong> and <strong>username</strong> class elements. We will also save the URL of the avatar image and keep it for later.</p><p>I will use <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/"><strong>Beautiful Soup</strong></a> to extract the handles. It is a Python library for parsing HTML and XML documents. When the HTML or XML document is poorly formed (for example if it lacks closing tags), <strong>Beautiful Soup</strong> offers a heuristic-based approach in order to reconstruct the syntax tree without generating errors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15735fdbef6d75a312632ae8abbf19a0/href">https://medium.com/media/15735fdbef6d75a312632ae8abbf19a0/href</a></iframe><p>After extracting the handles, I export the results to <strong>.csv</strong> file, which looks like this :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNPbnT46xiL1VJGMuYjyuw.jpeg" /></figure><p>Then , I need to create a dataset containing the tweets of these members related to the <strong>2018 U.S. Congressional Election</strong>. I collected the tweets between<strong> <em>January 22, 2018</em></strong> and <em>J</em><strong><em>anuary 3, 2019</em></strong> from the <strong>Twitter API</strong> using <a href="https://gwu-libraries.github.io/sfm-ui/"><strong>Social Feed Manager</strong></a>. See each collection’s <a href="https://github.com/chouhbik/Sentiment-Analysis-of-Tweets/blob/master/election-filter-README.txt">README</a> for dates of collection, accounts, and hashtags used in queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HeMfMQ9EjxtHeLErZEY36A.png" /><figcaption>Data Collection Method</figcaption></figure><h3>Exploratory Data Analysis (EDA):</h3><p>The dataset contains <strong>86,460</strong> tweets related to the 2018 U.S. Congressional Election (<strong>44392</strong> for Republican and <strong>42068</strong> for Democrat). We can say that we have roughly similar proportions of Democrats &amp; Republicans in the training dataset.</p><p>I am going to start with a quick exploration of tweets from <strong>Democrat</strong> and <strong>Republican</strong> lawmakers in the <strong>US</strong>. The main purpose of this initial analysis will be to visualize hashtags used by both of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*SbMn8ooZg272rYUHuhUTjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*7et_PwUyf4aHZLvLvxUfvA.png" /></figure><p>This is getting interesting. We can now see the <strong>Republican</strong> party <em>(in red) </em>touting their policies (e.g., <strong>taxcutsandjobsact</strong>). For those less familiar with US politics, the <strong>Republican</strong> party controls the presidency and both chambers of the <strong>Congress</strong>, and are therefore able to implement their agenda relatively unrestrained. On the other side, for the <strong>Democrat</strong> party, we see some clear bitterness (e.g., <strong>goptaxscam</strong>).</p><p>In addition, I will explore the words used by the different parties. This will be achieved through cleaning the tweets, parsing them into <strong>vectors of words</strong>, removing common <strong>stopwords</strong>, aggregating words by count, and plotting the most frequent results using <strong>barplots</strong>. The choice of the stopwords can be critical sometimes. I thought maybe it’s better to choose the <strong>STOPWORDS </strong>from the <strong><em>“wordcloud”</em></strong> library because they contain an average list of all of them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fef4d03f40fd3185db74fd6a476bde2/href">https://medium.com/media/0fef4d03f40fd3185db74fd6a476bde2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*iL_ooVyfgmvmRAW67Smc8g.png" /></figure><p>For example, the word <strong>“today”</strong> (3850 times) is used most in the tweets sent by democrat people. Then we find <strong>“trump”</strong> (2502 times) and <strong>“american”</strong> (2053 times).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*emZCQtc9Oy3SHHqwlLSD_Q.png" /></figure><p>If we look at the tweets of people who have a republican opinion, the word <strong>“today”</strong> (4883 times) is in the first place again, <strong>“tax”</strong> is in the second place and <strong>“great”</strong> in the third place.</p><p>I also categorized the word frequencies into 5 categories :</p><ul><li>If a word is used less than <strong>50</strong>, it is in the <strong>Very Low</strong> group.</li><li>If it’s used between <strong>50</strong> and <strong>200</strong> times, it is in the <strong>Low</strong> group.</li><li>If it’s used between <strong>200</strong> and <strong>750</strong> times, it is in the <strong>Medium</strong> group.</li><li>If it’s used between <strong>750</strong> and <strong>1500</strong> times, it is in the <strong>High</strong> group.</li><li>If usage of a word is greater than <strong>1500</strong>, it is in the <strong>Very High</strong> group.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQ0_UGSID3UGmNNLj9nMIg.png" /><figcaption>Created with <a href="https://plot.ly/">plotly</a></figcaption></figure><h4>Interpretation :</h4><p>I tried to plot a matix in which the <strong>‘X’</strong> axis refers to the word usage frequency of <strong>Republican tweets</strong> and <strong>‘Y’</strong> axis refers to the same thing but related to the <strong>Democrats</strong>.</p><p>For example, the word <strong>“taxreform”</strong> is used by Republican <strong>966</strong> times, so it’s placed in their high category. However, Democrats used that word only <strong>12</strong> times. An other example is <strong>“gun”</strong> word. Its used <strong>876</strong> times in Democrat tweets and <strong>117</strong> times in Republican tweets.</p><p>Also there is no words, when Republicans used very highly and Democrats used at the medium, low or very low level. The same as for Democrats. The words : <strong>“today”, “american”, “great, “house”, “year”, “family”, “day” and “thank”</strong> are used very highly by both <strong>Democrats</strong> and <strong>Republicans</strong>.</p><h4>Dispersion Plots :</h4><p>I have selected some words in the data set in order to produce a plot showing the distribution of the words through the text. These words are <strong>“vote”, “democracy”, “freedom”, “america”, “american”, “tax”, “trump”</strong> and <strong>“clinton”</strong>.</p><p><strong>Democrat Tweets Plot</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*qYdcYtkSx8BazjYK5HDqkA.png" /></figure><p><strong>Republican Tweets Plot</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*SkH77LOs2XtSw-_oH4OyOQ.png" /></figure><h3>Sentiment Analysis</h3><p>Here, I will try to categorize the tweets into organized groups or parties using different text classifiers, in addition to <strong>Sentiment Analysis</strong> for understanding if a given tweet is talking positively or negatively about a given subject.</p><p>So I began with converting a collection of tweets for both parties to a matrix of token counts, using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html"><strong>CountVectorizer</strong></a><strong> </strong>function from<strong> </strong><a href="https://scikit-learn.org/stable/index.html"><strong><em>sklearn</em></strong></a><strong>. </strong>Then I applied ML algorithms for classification<strong>: </strong><a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"><strong>RandomForest</strong></a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"><strong>LogisticRegression</strong></a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html"><strong>Multi-NB</strong></a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html"><strong>DecisionTrees</strong></a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html"><strong>AdaBoost</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*WUFo6QbmfGuW8Zx17WjH9A.png" /></figure><p>After that, I tried to determine the attitude or the emotion of each party, i.e., whether it is positive or negative or neutral. So I used the famous Python library <a href="https://textblob.readthedocs.io/en/dev/"><strong>TextBlob</strong></a><strong> </strong>which returns 2 important indicators:</p><ul><li>The <strong>polarity</strong> score is a float within the range <strong>[-1.0, 1.0]</strong>. And it represents emotions expressed in a sentence. <strong>1 </strong>means <strong>positive</strong> statement and<strong> -1</strong> means a <strong>negative</strong> statement.</li><li>The <strong>subjectivity</strong> is a float within the range <strong>[0.0, 1.0]</strong> where <strong>0.0</strong> is very <strong>objective</strong> and <strong>1.0</strong> is very <strong>subjective</strong>. And it refers to personal opinion, emotion or judgment.</li></ul><pre><strong>Democrat_Tweets :</strong><br>Sentiment(polarity=0.16538325109722254, subjectivity=0.4649093199451984)</pre><pre><strong>Republican_Tweets:<br></strong>Sentiment(polarity=0.19837398561739605, subjectivity=0.4590746992419168)</pre><h3>Simple neural network classification</h3><p>Now let’s try to answer the last question “if my <strong>Twitter</strong> account is more <strong>Democratic/Republican” </strong>by using a simple neural network<strong>. </strong>I will develop a <strong>LSTM</strong> and <strong>Convolutional Neural Network</strong> model for <strong>Sequence Classification</strong> using the <a href="https://keras.io/"><strong>Keras</strong></a> deep learning library.</p><p>Let’s start off by importing the classes and functions required for this model and initializing the random number generator to a constant value (7 for example) to ensure that the results can be reproduced.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91ee59468699423861b74e8bad1049aa/href">https://medium.com/media/91ee59468699423861b74e8bad1049aa/href</a></iframe><p>Then, we split the dataset, stratified according to the Party variable so as to have approximately similar proportions of <strong>Democrats</strong> &amp; <strong>Republicans</strong> in the <strong><em>training &amp; test </em></strong>sets separately as in the whole dataset. The set will be based on those <strong><em>Twitter handles</em></strong> because we don’t want any individual tweeter to appear in both <strong><em>training &amp; test</em></strong> sets.</p><p>Actually, the tweets don’t have the same length in terms of words, but same length vectors is required to perform the computation in <strong>Keras</strong>. That’s why we need to truncate and pad the input tweets so that they are all the same length for modeling. The model will learn that the zero values carry no information.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44012c8cd831caa657c1e92d9655e7e1/href">https://medium.com/media/44012c8cd831caa657c1e92d9655e7e1/href</a></iframe><p>We can now define, compile and fit our LSTM model :</p><ul><li>The first layer is the <strong>Embedded</strong> layer that uses 32 length vectors to represent each word.</li><li>We add a one-dimensional <strong>CNN</strong> and a <strong>max pooling</strong> layer after the <strong>Embedding</strong> layer which then feed the consolidated features to the <strong>LSTM</strong>.</li><li>The next layer is the <strong>LSTM</strong> layer with <strong>100</strong> memory units (smart neurons).</li><li>Finally, because this is a classification problem we use a <strong>Dense</strong> output layer with a single neuron and a <strong>sigmoid</strong> activation function to make <strong>0</strong> or <strong>1</strong> predictions for the two classes <strong>Democrat/Republican</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*idQDUgdVefo0alOce3H9eQ.jpeg" /></figure><p>I used <strong>epochs=10</strong> and <strong>batch_size=64</strong> for fitting the model. Once fit, we estimate the performance of the model on unseen tweets:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd5b1f6d4c9bd28d811642fd35f20b5b/href">https://medium.com/media/dd5b1f6d4c9bd28d811642fd35f20b5b/href</a></iframe><p>I got an overall accuracy of <strong>74,52%. </strong>It’s not so unusual for a single tweet to be mis-categorized, so perhaps we should consider the overall categorization of each person’s tweets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0913793867f4ad34b62c8a78f77de8a4/href">https://medium.com/media/0913793867f4ad34b62c8a78f77de8a4/href</a></iframe><p>With this adjustment, we get <strong>~93%</strong> accuracy.</p><h3>Conclusion :</h3><p>Text classification is not only fun, but it’s also a powerful tool for extracting value from unstructured data. <strong>Twitter</strong> is a very popular social media platform used by millions of users, and the published tweets can sometimes be used to see what the public’s opinion is regarding a certain problematic. It feels like magic when you analyze hundreds of tweets in just a few seconds and automatically get information such as topic, opinion, sentiment…</p><p>After this study, I found out that we can make classification on people to see if they are <strong>Demcorat</strong> or <strong>Republican</strong>, and the results were somehow accurate. If sentiment analysis algorithms were improved , we could use them to predict next presidential election results this year.</p><p>Even though, the developed model still have <strong>2</strong> main hurdles :</p><ul><li><strong>Sarcasm</strong> is another weakness of all the <strong>Sentiment Analysis Algorithms</strong>, and our model also fail to detect <strong>sarcasm</strong> at this stage.</li><li>Since <strong>tweets</strong> can be posted by anyone and can contain spelling &amp; grammar mistakes, the misspelt key words might be incorrectly analyzed by our algorithm.</li></ul><h4>Notebook :</h4><p><a href="https://github.com/chouhbik/Sentiment-Analysis-of-Tweets/blob/master/Tweets%20Analysis%20DemvsRep.ipynb">chouhbik/Sentiment-Analysis-of-Tweets</a></p><h3>References :</h3><ul><li><strong>TweetCongress.org : </strong><a href="http://www.tweetcongress.org/tweeters">http://www.tweetcongress.org/tweeters</a></li><li><strong>Beautiful Soup : </strong><a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">https://www.crummy.com/software/BeautifulSoup/bs4/doc</a></li><li><strong>Social Feed Manager : </strong><a href="https://gwu-libraries.github.io/sfm-ui/">https://gwu-libraries.github.io/sfm-ui/</a></li><li><strong>Scikit-learn :</strong> <a href="https://scikit-learn.org/stable/index.html">https://scikit-learn.org/stable/index.html</a></li></ul><p>Congratulations if you managed to get here. Thanks for reading, I hope you’ve liked it. For personal contact or discussion on Machine Learning, feel free to reach out to me on <a href="https://www.linkedin.com/in/kamal-chouhbi/"><strong>LinkedIn</strong></a><strong> </strong>and don’t forget to follow me on <a href="https://github.com/chouhbik"><strong>GitHub</strong></a> and <a href="https://medium.com/@chouhbik"><strong>Medium</strong></a><strong>.</strong></p><p><a href="https://github.com/chouhbik">chouhbik - Overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4aa4cadf4bea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/are-you-democrat-or-republican-let-your-tweets-define-you-4aa4cadf4bea">Are you Democrat or Republican? Let your tweets define you …</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is like to be a Data Scientist with a passion for Gaming …]]></title>
            <link>https://medium.com/data-science/what-is-like-to-be-a-data-scientist-with-a-passion-for-gaming-43c067ad6415?source=rss-22f56f58cc0b------2</link>
            <guid isPermaLink="false">https://medium.com/p/43c067ad6415</guid>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[league-of-legends]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kamal Chouhbi]]></dc:creator>
            <pubDate>Sun, 02 Feb 2020 13:06:43 GMT</pubDate>
            <atom:updated>2020-03-04T13:58:17.901Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is it like to be a Data Scientist with a passion for Gaming …</h3><h4>League of Legends : Create a model to predict winning teams based on performances</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGdgAm6AsfLtyBMrVkP4Bw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rxspawn?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Florian Olivo</a> on <a href="https://unsplash.com/s/photos/gaming?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Summary</h3><p>During 2 months, I did a short data project on <a href="https://euw.leagueoflegends.com/en-gb/"><strong>League of Legends</strong></a> since I am a big fan of the game and its use of data. The project is focused on the question of whether or not we can create a model to predict winning teams based on performances. I tried to use logistic regression and other models to predict the match outcomes. Features were extracted from the data that <a href="https://developer.riotgames.com/"><strong>Riot Games API</strong></a><strong> </strong>exposes–including champions picked for the game, player role information, and mastery levels for the players’ champions (<em>pre-game </em>knowledge) as well as <em>in-game</em> player statistics. I found out that using only <em>prematch</em> knowledge (champions, masteries, roles, spells) from the very start is only a weak predictor of the match’s outcome, but using <em>in-game</em> statistics, the model becomes a strong predictor.</p><h3>Introduction</h3><h4>1- An Introduction to Data &amp; Analytics in the Gaming Industry:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dFTW1q-8JJsp7fFxolVEGg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brolin?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Anthony Brolin</a> on <a href="https://unsplash.com/s/photos/esport?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Just as they did with traditional sports, the collection, analysis, and use of all kinds of data are starting to change the way that online competitive games are played and concepted. Nowadays, as the <strong>E-sports</strong> have become more and more popular, data analyses on <strong>E-sports</strong> games have also become much more common.</p><p>In this project, I chose the famous online game <a href="http://leagueoflegends.com/"><strong>League of Legends (LoL)</strong></a>, developed by <a href="https://www.riotgames.com/en"><strong>Riot Games</strong></a> in 2009, as a case study. The main reason is that I had played and enjoyed the game since its 5th season and would like to bring a more informed view to this project than I would with another case study. An other important motivational point was the <a href="https://developer.riotgames.com/"><strong>Riot Games’ API</strong></a>, which allows the public to easily make queries and pull information from their servers.</p><h4>2- Related Work</h4><p>As <strong>League of Legends</strong> is a popular game, there are already a couple of applications that exist in the data analysis sphere. Websites like <a href="https://www.leagueofgraphs.com/"><strong>League of Graphs </strong></a>and <a href="https://www.metasrc.com/5v5"><strong>MetaSrc</strong> </a>collect basic stats and information from the <strong>League of Legends API</strong> and use it to construct simple analyses like what is the most popular champion and what is the win rate of two champions when they play together... What these existing applications do not currently do is the prediction of a currently ongoing game with all of the features of that game in mind, and that’s what the project is attempting to fill.</p><h3>Background on League of Legends as a Game</h3><h4>1- Game Overview :</h4><p><strong>League of Legends</strong> is a <strong><em>Multiplayer Online Battleground Arena (MOBA)</em></strong> style game developed by <a href="https://www.riotgames.com/en"><strong>Riot Games</strong></a>. Each game consists of <strong>two</strong> teams with <strong>5</strong> players each. Each team starts the game at opposite sides of the arena and the goal is to overcome obstacles (turrets, inhibitors and enemy players) in order to destroy the enemy base (<em>Nexus)</em>. There are 5 standard positions for the team players divided across the map: <strong>Top, Middle, Jungle, Attack Damage Carry(ADC) and Support</strong>. Each match usually lasts between twenty minutes and an hour. The players can choose a unique champion from over 140 champions with different characteristics and abilities. The game represents also a flourishing competitive scene with millions in tournaments prize pools as well as online viewers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4PNyTztUQtT6qsuL-Kn2yw.jpeg" /><figcaption>Link: <a href="https://www.pinterest.com/pin/579697783257078372/?lp=true">https://www.pinterest.com/pin/579697783257078372/?lp=true</a></figcaption></figure><p>With so many parameters within each game, it is impossible to say that one factor is going to determine the success or failure of any particular player. Also, with such a large community behind <strong>League of Legends</strong>, predicting match outcomes for casual players and tournament games would be interesting and valuable for both players and fans.</p><h4>2- Game Info :</h4><p><strong>League of Legends</strong> is an intricate game that requires knowledge of many elements to be played proficiently. In the game, two teams of five players each battle against each other on a battlefield called <strong><em>Summoner’s Rift</em></strong>. There are other battlefields with different objectives and layouts, but they will not be studied in our project. <strong><em>Summoner’s Rift</em></strong> is comprised of each team’s base, three lanes, and the jungle. The goal of League of Legends is to destroy the opposing team’s Nexus, a structure located in the enemy base.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*GbYVvvwj94ehJC1pw8qhjA.png" /><figcaption>Link : shorturl.at/ktAK1</figcaption></figure><p>The <strong><em>gold differential</em></strong> is an example of a raw statistic that can provide a sense of the state of a match at a glance. Usually when a team has more gold than the other team, it means their champions are carrying more powerful items and are more likely to win if they get into a team fight. Combined with individual <strong><em>KDA (Kills, Deaths, Assists)</em></strong> numbers and the overall number of kills a team has accrued, it’s an easy way to let the viewers know who is winning. If we have a look at the LCS broadcast, <strong><em>gold differential</em></strong> is one of the first numbers we see at the top of the screen. But statistics like these don’t tell the whole story…</p><h3>Data Collection</h3><p>I tried to extract data from the public <strong>Riot Games API</strong> , which allows collection of data about summoners (the LoL term for players), champions, and past matches. Then I gathered it all in a file with <strong><em>.xlsx</em></strong> format containing all the data available for each game played during the 2017 season.</p><blockquote>Remark : The downloaded dataset contained more than 75 variables, some of them were not meaningful. So I decided to get rid of those variables that won’t be helpful in our analysis.</blockquote><p>I uploaded the data on Kaggle, feel free to download it <a href="https://www.kaggle.com/kamalchouhbi/league-of-legends-match-data-2017">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8Qp1tDjN_nw1FFCApOrDQ.jpeg" /></figure><p>So, I decided to delete 25 variables that won’t be helpful in our prediction. Now, we have a dataset in <strong><em>.csv</em></strong> format which contains 50 variables.</p><p>In order to interpret the downloaded match data file, you can find <a href="https://oracleselixir.com/match-data/match-data-dictionary/">here</a> a dictionary containing the variable names and their descriptions.</p><p>Here is a preview of the dataset we are analyzing using the software Tableau Desktop:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDqXTy8uley8OyWpDN2gMw.jpeg" /><figcaption>Preview of the file : 2017clean.csv</figcaption></figure><h3>Exploratory Data Analysis (EDA) :</h3><h4>a- League of Legends’ Evolution :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*eP1EM6J2zDpn-emigRZX4w.jpeg" /></figure><p>First of all, I tried to illustrate the impressive growth of the game by comparing the number of competitive games in our data with the previous year 2016.</p><p>In 2016,<strong> 2429</strong> games took place. League continues to grow with a substantial pace, reaching the unbelievable 3117 professional competitive games in 2017.</p><p>We can also see that the leagues <strong><em>LPL, NALCS, LCK</em></strong> and <strong><em>EULCS</em></strong> have the most games with more than 350 each.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*f6K13P8rv8SwLvMwT6t2Pw.jpeg" /></figure><h4>b- Teams :</h4><p>Calculating the winning percentage is equivalent to estimating a proportion of wins in total number of games. As we can see below, there are teams who have 0% and 100% as a win ratio, and this is not coherent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USHuD1ti8qY0CwEe3xvsWQ.jpeg" /></figure><p>The problem is that some teams have played just few games, others have even played just one game. In order to overcome that problem, we need to calculate the winning percentage just for the “Most Appeared Team” who played more than 60 games in different leagues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TsevwHCTaC3HvJA4n03q3A.jpeg" /></figure><p>As we can see, <a href="https://lol.gamepedia.com/SK_Telecom_T1"><strong>SK Telecom T1</strong></a>, <a href="https://lol.gamepedia.com/Flash_Wolves"><strong>Flash Wolves</strong></a> and <a href="https://lol.gamepedia.com/Samsung_Galaxy"><strong>Samsung Galaxy</strong></a> possess the highest win rate. I first calculated the wins &amp; loses, then I calculated the win ratio for each team individually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*jbQO8DUkoornSxbF4d2_KA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xzvt9kkT5EyCCCfOD-95bQ.jpeg" /></figure><p>Thus, we conclude that:</p><ul><li>The average <strong>Win Ratio</strong> for a <em>Pro Team</em> is <strong>45%</strong>.</li><li><a href="https://lol.gamepedia.com/SK_Telecom_T1"><strong>SK Telecom T1</strong></a> has the highest <strong>Win Ratio</strong>. Out of <strong>139</strong> games they managed to win <strong>97</strong> leading to an outstanding <strong>69,78%</strong> <strong>Win Ratio</strong></li><li><a href="https://lol.gamepedia.com/Flash_Wolves"><strong>Flash Wolves</strong></a> have <strong>93</strong> games winning <strong>64 (68,82%)</strong></li><li><a href="https://lol.gamepedia.com/Samsung_Galaxy"><strong>Sumsung Galaxy</strong></a> comes third in winrate. They managed to win <strong>80</strong> games leading to a <strong>67,23% Win Ratio</strong></li></ul><h4>c- Players :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*b3FSugSG7Br54m5XhN5SIw.jpeg" /></figure><ul><li><a href="https://lol.gamepedia.com/Mystic"><strong>Mystic</strong></a><strong>(598), </strong><a href="https://lol.gamepedia.com/Bjergsen"><strong>Bjergsen</strong></a><strong>(577) </strong>and<strong> </strong><a href="https://lol.gamepedia.com/Xiye"><strong>Xiye</strong></a><strong>(541) </strong>are the players with the most kills in competitive League.</li><li><a href="https://lol.gamepedia.com/Biofrost"><strong>Biofrost</strong></a><strong>(1282), </strong><a href="https://lol.gamepedia.com/Wolf_(Lee_Jae-wan)"><strong>Wolf</strong></a><strong>(1099) </strong>and<strong> </strong><a href="https://lol.gamepedia.com/Condi"><strong>Condi</strong></a><strong>(1004) </strong>are the players with the most assists in competitive League.</li></ul><p>Another important factor in this game is t<em>he ratio of kills</em>, deaths and assists <strong>KDA (kills, deaths, assists)</strong>. Formula:</p><blockquote><strong><em>KDA Ratio = (K+A) / Max(1,D)</em></strong> Where: K = Kills | D = Deaths | A = Assists</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9tE_AMpX5sQkC2K6Z8D4w.jpeg" /></figure><ul><li>The average <strong>KDA</strong> for a pro player is <strong>3,74</strong></li><li><a href="https://lol.gamepedia.com/Jensen"><strong>Jensen</strong></a> &amp; <a href="https://lol.gamepedia.com/Bjergsen"><strong>Bjergsen</strong></a> are 2 of the players with the highest <strong>KDA</strong> in competitive Leagues, having <strong>7,78</strong> &amp; <strong>7,34</strong> respectively</li></ul><h4>d- Gold and Damage</h4><p>Meanwhile, I also tried to use the linear regression method to see the correlation between <strong>Gold earned</strong> and <strong>Damage made</strong>. As shown below, there is a strong linear correlation between <strong>Gold</strong> and <strong>Damage</strong>, which means that players make good use of the gold earned in the games, and convert the gold into damage to the enemies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*IWxo27MpqArARm-gSBKxUQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*DyCjXhPolUySjjbHkwMPGg.jpeg" /></figure><p>In addition, I studied other aspects of the <strong>League of Legends </strong>games, but I don’t want to explain them here so as to avoid writing a long article. Maybe I can write other articles in the future to present the whole study. Now let’s get to the most important part!</p><h3>Winner Prediction :</h3><blockquote>Can we predict a game’s result?</blockquote><p>Actually, using only prematch knowledge (champions, masteries, roles, spells) from the very start is only a weak predictor of match outcome, but using in-game statistics, the model becomes a strong predictor. That’s why I will try to build our model based on in-game statistics.</p><p>In fact, we’re not sure which of our variables are useful in predicting a result outcome. It’s often helpful to build bivariate models, which are models that predict the outcome using a single independent variable. But the problem is which of the 28 variables is a significant predictor of the Result variable in a bivariate logistic regression model? We need to build different models so as to find the best one with the highest accuracy, or we can use the correlation function to get an idea about the variables that are highly correlated with the variable Result.</p><p><strong>First, I began with excluding the variables referring to stats at the end of the game, because it would be too late to predict a win in that case. Also, I decided omit the variables related to individual stats since it is a team game and one player can’t win by himself.</strong></p><p>I applied <strong>Logistic Regression</strong> and <strong>Random Forest</strong> on the same dataset. So I got variable importance <em>(absolute coefficient for logistic regression)</em> and <em>(variable importance for random forest)</em>. I tried to combine the two to get a final variable importance. After many attempts, I managed to build a final model <strong>(Stacking : Logistic Regression &amp; Random Froest)</strong> with the significant variables below:</p><ul><li><strong>gdat10 :</strong> Gold difference at 10:00</li><li><strong>gdat15 :</strong> Gold difference at 15:00</li><li><strong>fb :</strong> First blood kill (1 yes, 0 no)</li><li><strong>teamkill,s :</strong> Total kills by team</li><li><strong>teamdeaths :</strong> Total deaths by team</li><li><strong>ft :</strong> First tower of game killed (1 yes, 0 no)</li><li><strong>teamtowerkills : </strong>Total towers killed by team</li><li><strong>opptowerkills : </strong>Total towers killed by opposing team</li><li><strong>fd :</strong> First dragon of game killed (1 yes, 0 no)</li><li><strong>teamdragkills : </strong>Total dragons killed by team</li><li><strong>oppdragkills : </strong>Total dragons killed by opposing team</li></ul><h4>Confusion Matrix Explained</h4><ul><li><strong><em>Correct :</em></strong> Predicted Red Team to win and actually Won : 6318 times</li><li><strong><em>Correct :</em></strong> Predicted Blue Team to win and actually Won : 6351 times</li><li><strong><em>Wrong:</em></strong> Predicted Red Team to win but Red team Lost : 95 times</li><li><strong><em>Wrong:</em></strong> Predicted Blue Team to win but Blue team Lost : 106 times</li></ul><blockquote>Accuracy = 98.43 %</blockquote><h3>R Shiny Application:</h3><p><a href="https://shiny.rstudio.com/"><strong>Shiny</strong></a> is an <strong>R</strong> package that makes it easy to build interactive web apps straight from <strong>R</strong>. I decided to build a web application containing two part:</p><ul><li>First part using our best prediction model which has 98% as accuracy; you can fill a form and get the percentage of your win chance.</li><li>Second one using <strong>Riot Games Api</strong> to compare <strong>League of Legends</strong> champion statistics. Two champions can be compared at a time, and patch version is also selectable.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDt58OxeDsLqA1Erqi5y0g.gif" /></figure><h3><strong>Discussion</strong>:</h3><p>In this study, I used data related to one patch of the game because Riot Games attempts to keep champion win rates as close to 50%. Any win rate higher than 50% implies that the champion is giving the player an advantage while any win rate lower than 50% implies the champion is giving the player a disadvantage.</p><p>In order to maintain balance, <strong>Riot</strong> decides to respectively <strong><em>nerf</em></strong> and <strong><em>buff</em></strong> these champions. That means there is a strong correlation between the number of changes in the patch and the win rate if the team select the same champions.</p><h3>Conclusion:</h3><p>We hope that we’ll be able to look at more current data in the future, but this is a great demonstration of how data can power anyone from <strong>normal players</strong>, to <strong>Riot Games</strong> <strong>professional broadcasts</strong>, and even <strong>teams</strong> looking for ways to best analyze the strengths and weaknesses of enemy teams to make informed decisions.</p><p>None of the existing sites, that mine data through the <strong>Riot API </strong>and release interesting statistics, can effectively predict the winner team during a game. This study takes on this task using different variables that are related to both teams stats in order to predict the winner.</p><p>I wanted to mention that the <a href="https://developer.riotgames.com/"><strong>Riot API</strong></a><strong> </strong>also contains much more information which we ignore but could use for many other purposes.</p><h3>References:</h3><ul><li>League of Legends : Game API ==&gt; <a href="https://developer.riotgames.com/">https://developer.riotgames.com</a></li><li>Match Data Dictionary ==&gt; <a href="https://oracleselixir.com/match-data/match-data-dictionary/">https://oracleselixir.com/match-data/match-data-dictionary/</a></li><li>Doublelift Analysis 2018 ==&gt; <a href="https://rpubs.com/kail/dlift">https://rpubs.com/kail/dlift</a></li><li>Kaggle : League of Legends Ranked Games ==&gt; <a href="https://www.kaggle.com/jaytegge/league-of-legends-data-analysis/data">https://www.kaggle.com/jaytegge/league-of-legends-data-analysis/data</a></li><li>League of Analytics ==&gt; <a href="http://league-analytics.com/">http://league-analytics.com</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43c067ad6415" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-is-like-to-be-a-data-scientist-with-a-passion-for-gaming-43c067ad6415">What is like to be a Data Scientist with a passion for Gaming …</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>