<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thomas George Thomas on Medium]]></title>
        <description><![CDATA[Stories by Thomas George Thomas on Medium]]></description>
        <link>https://medium.com/@thomas-george-thomas?source=rss-45ff032abfb6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*x29fr7JVAe6j21uk67ZN7w.jpeg</url>
            <title>Stories by Thomas George Thomas on Medium</title>
            <link>https://medium.com/@thomas-george-thomas?source=rss-45ff032abfb6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:05:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thomas-george-thomas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Make GitHub recognize SQL files]]></title>
            <link>https://thomas-george-thomas.medium.com/make-github-recognize-sql-files-59637674650c?source=rss-45ff032abfb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/59637674650c</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Thomas George Thomas]]></dc:creator>
            <pubDate>Sat, 19 Jun 2021 18:47:10 GMT</pubDate>
            <atom:updated>2025-01-22T22:52:59.778Z</atom:updated>
            <content:encoded><![CDATA[<h3>Recognizing SQL files on GitHub</h3><h4>Taking a closer look at how GitHub classifies different languages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XPwjg5X_3DTSCMYC" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the Data world, it is not uncommon to commit database files and SQL queries. This could be for collaborating within teams, maintaining a queries library or even as simple as tracking for CI/CD purposes.</p><p>While committing your SQL files, it must be noted that GitHub doesn’t recognize <em>.sql</em> files automatically. The library responsible for this is the <strong>Linguist </strong>on GitHub.</p><p><a href="https://github.com/github/linguist">GitHub - github/linguist: Language Savant. If your repository&#39;s language is being reported incorrectly, send us a pull request!</a></p><p>The linguist is an extremely powerful tool that classifies your repository to show the languages used and provides syntax highlighting. This is great when it comes to traditional programming languages.</p><h4>So How does the Linguist work?</h4><blockquote>Linguist takes the list of languages it knows from <a href="https://github.com/github/linguist/blob/master/lib/linguist/languages.yml">languages.yml</a> and uses a number of methods to try and determine the language used by each file, and the overall repository breakdown.</blockquote><p>The linguist basically runs a low-priority background program that analyzes the repository and caches the results for the lifetime of the repository and is only updated when the repository is updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*odVANuyokxrwcIl_" /><figcaption>Photo by <a href="https://unsplash.com/@mbaumi?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Structured Query Language</strong> or <strong>SQL </strong>is a powerful language used worldwide to access and manipulate databases.</p><blockquote><a href="https://en.wikipedia.org/wiki/SQL"><strong>According to the Wikipedia of SQL</strong></a><strong>,</strong> <strong>SQL</strong> ( “sequel”; <strong>Structured Query Language</strong>) is a domain-specific language used in programming and designed for managing data held in a relational database management system (RDBMS), or for stream processing in a relational data stream management system (RDSMS). It is particularly useful in handling structured data, i.e. data incorporating relations among entities and variables.</blockquote><p>Unfortunately, SQL is one of the languages that the linguist considers a<a href="https://github.com/github/linguist/blob/master/docs/how-linguist-works.md">s a data language</a>. What this means is that the GitHub linguist excludes <em>.sql</em> files while determining the languages used in the repository.</p><h3>So How do you overcome this hurdle?</h3><p>We override the linguist and make <em>.sql</em> files detectable in the repository. We accomplish this by adding a <em>.gitattributes</em> file to the root of the repository.</p><blockquote>A .gitattributes file is a simple text file that gives attributes to pathnames.</blockquote><p>The code in the .gittributes file looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a829803c0a17f41183a499f27a066664/href">https://medium.com/media/a829803c0a17f41183a499f27a066664/href</a></iframe><p>As the code says, we make the .sql files detectable and tell the linguist to treat the SQL language as text. This in turn makes the language stats bar recognize the <em>.sql</em> files correctly and gives them syntax highlighting.</p><p>Example of my repository:</p><p><a href="https://github.com/Thomas-George-T/HackerRank-SQL-Challenges-Solutions">https://github.com/Thomas-George-T/HackerRank-SQL-Challenges-Solutions</a></p><p>I sure hope that I saved you countless hours researching why GitHub doesn’t recognize SQL files and how you can make GitHub recognize SQL.</p><h4>References.. Read more...</h4><p><a href="https://github.com/github/linguist/issues/3666">.gitattibutes seems to be ignored for &#39;sql&#39; language · Issue #3666 · github/linguist</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59637674650c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Patient treatment costs using Machine Learning.]]></title>
            <link>https://medium.com/analytics-vidhya/predicting-patient-treatment-costs-using-machine-learning-1bfc3732ed13?source=rss-45ff032abfb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bfc3732ed13</guid>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[simple-linear-regression]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Thomas George Thomas]]></dc:creator>
            <pubDate>Sun, 21 Mar 2021 15:49:20 GMT</pubDate>
            <atom:updated>2021-07-07T13:10:38.467Z</atom:updated>
            <content:encoded><![CDATA[<h4>Regression and EDA on personal health data to determine factors contributing to treatment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jRsvYbHQLZgNyxVw" /><figcaption>Photo by <a href="https://unsplash.com/@hikendal?utm_source=medium&amp;utm_medium=referral">Kendal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Linear regression is one of the most important algorithms under the supervised learning category in Machine Learning. It is also the simplest and commonly used model for predictive analysis. Using this we explore the personal health dataset and predict treatment and insurance costs.</p><h4>What is a Linear Regression?</h4><p>In the simplest terms, when a relationship between the target and one or more predictors is linear, it is a linear regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/154/1*D_JMa8BffDX2koYcVo_3oQ.png" /></figure><p>Where the target (y) is considered the dependent variable while one predictor (x) is considered to be the independent variable. b0 and b1 are the intercept and the slope respectively.</p><h4>Why Linear Regression?</h4><ol><li>Linear regression can be used to determine the strength of the effect that the predictor variable(s) have on the target variable.</li><li>It helps us understand how much the target variable changes with changes in the predictor variable(s).</li><li>Most importantly, linear regression can be used to get future estimates and help predict trends accurately.</li></ol><h4>Types of Linear Regression</h4><p>Linear Regression can be broadly classified into two categories:</p><ol><li><strong>Simple Linear Regression:</strong> Equation where there is one dependent (target) variable and exactly one independent (predictor) variable.</li><li><strong>Multiple Linear Regression:</strong> Equation where there are one dependent (target) variable and two or more independent (predictor) variables.</li></ol><p>In this use case, We explore Simple linear regression in detail.</p><h3>Data Description</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z_Ze0HQKfyrtqKCy" /><figcaption>Photo by <a href="https://unsplash.com/@swimstaralex?utm_source=medium&amp;utm_medium=referral">Alexander Sinn</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For predicting health insurance costs, We utilize <a href="https://www.kaggle.com/mirichoi0218/insurance">Miri Choi’s Medical Cost Personal Datasets</a> hosted on Kaggle. The column descriptions look like this:</p><ul><li><em>age</em>: age of the primary beneficiary</li><li><em>sex</em>: insurance contractor gender, female, male</li><li><em>bmi</em>: Body mass index, providing an understanding of the body, weights that are relatively high or low relative to height.</li><li><em>children</em>: Number of children covered by health insurance / Number of dependents</li><li><em>smoker</em>: Yes/No</li><li><em>region</em>: the beneficiary’s residential area in the US.</li><li><em>charges</em>: Individual medical costs billed by health insurance</li></ul><p>Now, getting our hands dirty with data</p><h3>Acquiring the Data</h3><p>Once we download the CSV data, we can import it using read_csv. We then use head() to sample the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*Aez7tmz7cKSAcueg.JPG" /><figcaption>Viewing the sample data | Image by Author</figcaption></figure><h3>Preparing the Data</h3><p>We try to identify numerical and categorical data. We proceed to collect basic descriptive stats using describe(). We try to understand what the data looks like and what it is trying to tell us.</p><pre>data.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*pY2VUIRt1zYN3aw2.JPG" /><figcaption>Viewing descriptive statistics of the data | Image by Author</figcaption></figure><blockquote>dataframe.info() is also good to give us a concise summary of the data that it is holding.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dMJ5S7iU477qwWRr" /><figcaption>Photo by <a href="https://unsplash.com/@kalenemsley?utm_source=medium&amp;utm_medium=referral">Kalen Emsley</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Exploratory Data Analysis (EDA)</h3><p>EDA is the analytical process in data science where the main characteristics are drawn and summarized by investigating and analyzing the data sets. We take a closer look at all the involved steps as follows:</p><h3>Feature Engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E1nhCj8BqZyxKON_" /><figcaption>Photo by <a href="https://unsplash.com/@agk42?utm_source=medium&amp;utm_medium=referral">Alex Knight</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Feature engineering is the process of extracting raw data using domain knowledge to improve model performance.</em></p><blockquote><em>Features in Machine learning essentially mean columns.</em></blockquote><p>From our prior step of descriptive analysis, we can see that the data is made up of two forms: <strong><em>Numerical </em></strong>and <strong><em>Categorical</em></strong>. Partitioning the features accordingly into numerical and categorical:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*KQ_PYA4KBcA-Kza6.JPG" /><figcaption>Differentiating numerical and categorical features | Image by Author</figcaption></figure><p>Building a model with categorical data is hard but not impossible. For simplicity, we proceed to convert categorical data into numerical data. For this purpose, we use the <strong><em>One hot encoding</em></strong> technique.</p><p>One hot encoding is a technique where we replace the categorical data with binary digits. The categorical column is split into the same number of columns as the values. The respective column is then given a ‘1’ or a ‘0’ corresponding to the values.</p><p>we use one-hot encoding by using get_dummies()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/0*IFtniYTCbJ65ksh4.JPG" /><figcaption>One hot encoding on Categorical data | Image by Author</figcaption></figure><p>After preparing our data, we are ready for the next step. We need to pick the important ‘<em>features</em>’ that will have an impact on the target variable. The best way to do that is to try and find the correlation(s) between the different <em>features</em>. This can be achieved by using data.corr()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CZ6aGgeZxaWQ5t3W.JPG" /><figcaption>Exploring the correlation between the features | Image by Author</figcaption></figure><p>Visualizing using a heat map to better explore the trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*rO43DXFMMAnT_ogg.JPG" /><figcaption>Heatmap showing the correlation between the features | Image by Author</figcaption></figure><p>From this we can see the following observations:</p><ol><li>Strong correlation between <em>charges </em>and <em>smoker_yes</em>.</li><li>Weak correlation between <em>charges </em>and <em>age</em>.</li><li>Weak correlation between <em>charges </em>and <em>BMI</em>.</li><li>Weak correlation between <em>BMI </em>and <em>region_southeast</em>.</li></ol><p><em>Since the values for the weak correlations are less than 0.5, we can term them as insignificant and drop them.</em></p><blockquote>Remember: Correlation doesn’t imply Causation</blockquote><p>Here we can see that there is exactly one predictor for our target (<em>charges</em>) variable. This makes for a good use case for simple linear regression.</p><p>Exploring the correlation and the trend between charges and smoker_yes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*kJxmFcbkuPoeHbLT.JPG" /><figcaption>Graph showing the varying trend for treatment charges of patients | Image by Author</figcaption></figure><p>From the graph, the treatment charges of patients range from a minimum of 1122 for a significant number of patients and a maximum of 63770 for a few patients.</p><h3>Building the Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5MVM2FafvPPAcDUP.JPG" /><figcaption>Training the linear regression model | Image by Author</figcaption></figure><p>We begin to predict the values of the patient charges using the other features. We build a simple linear regression model after importing the package sklearn.linear_model. We split the data set into training and test set. A good idea is to split 30% of the dataset for testing using test_size=0.3 We take the predictor variable without the charges column and the target variable as charges. We proceed to fit the linear regression model for the test and training set using fit(). This part is called <em>Model fitting</em>. We check the prediction score of both the training and test set using score(). It comes out to be 79%, which is pretty decent I would say.</p><h3>Evaluating the Model</h3><p>To evaluate our linear regression model, we use <strong>R²</strong> (also known as the <em>coefficient of determination</em>) and M<strong>ean Squared Error</strong> as our metrics.</p><p>R² is a statistical measure of how close predicted data is with respect to the regression line. It is expressed in terms of percentages and lies between 0% and 100%. <em>Generally speaking, the higher the percentage of R² is, the better the model.</em></p><p>Mean Squared Error (MSE) is the average squared difference between the predicted values and the actual values. <em>The lower the MSE the better the model fits.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/0*_Fmm8i5LQAzb3egZ.JPG" /><figcaption>Model Evaluation results | Image by Author</figcaption></figure><p>From the figure, MSE is closer to 0 and R² on the test data is a whopping 79%!</p><h3>Conclusion</h3><p>Our evaluation metrics of R² and mean squared error of both training and test data are closely matching and are consistent with being a good fit with the regression line. This is enough to conclude our model is appropriate to predict patient charges based on their personal health data.</p><p>I hope that I was able to explain linear regression and the related concepts of EDA, feature engineering and selection, R² and Mean Squared error through this use case effectively. Personal health data is a great example of where linear regression simply works! Thank you for reading!</p><h3>References</h3><ol><li>My code, <a href="https://github.com/Thomas-George-T/Regression-on-Personal-Health-Data">Regression on Personal Health Data</a> (2020), GitHub</li><li>Miri Choi, <a href="https://www.kaggle.com/mirichoi0218/insurance">Medical Cost Personal Datasets</a> (2013), Kaggle</li><li>Statistics Solutions, <a href="https://www.statisticssolutions.com/what-is-linear-regression/?__cf_chl_jschl_tk__=778c40adeeeb9bf683ebc6f59d3fd6e925af2469-1616173220-0-AQgJM-siD_HFStc1RQO1KsxNMX4hzW3PfBJ3kX_QL-Ibjesk6l8qXHfYiyhJd96zPrDL2dacFyp4Kv7b32a2z9nXPH5CbkcFxSrLmeWD6u0n83XCWVI_QCOyePzH_VCgllquORtE3uul0BuuEWPIqfFQMKdyCVx6Q0dz638Qt2AxwRLZvWOThhKIiUhd30M0VNStU6yWw54ToVZMAAlbMGlyXRr-PDLfkbde2b0AtMMFRYRPXEhPtr76NGqQ7l8wiyPxlrAN51Mte1L8QsnBlLllWqtuuCfdYlnwBVzRhFjMIOhdwnVzWSozoSUqSonpra2GG-yhGj953MNDFyXGdft5-UB92qDm83M5BfXddBUf">What is Linear Regression</a> (2013)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bfc3732ed13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/predicting-patient-treatment-costs-using-machine-learning-1bfc3732ed13">Predicting Patient treatment costs using Machine Learning.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[File Processing: A Comparative Analytics Study]]></title>
            <link>https://medium.com/geekculture/file-processing-a-comparative-analytics-study-e21b4693b70c?source=rss-45ff032abfb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e21b4693b70c</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thomas George Thomas]]></dc:creator>
            <pubDate>Sat, 06 Feb 2021 00:02:44 GMT</pubDate>
            <atom:updated>2021-06-23T12:54:05.673Z</atom:updated>
            <content:encoded><![CDATA[<h3>File Processing in Big Data Systems: Which is Quicker? Which is better?</h3><h4>A Comparative Analytics Study Benchmarking Popular Programming Languages and Execution Engines.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JAlaohX-4tbRGmg8" /><figcaption>Photo by <a href="https://unsplash.com/@sumo?utm_source=medium&amp;utm_medium=referral">PAUL SMITH</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Have you ever wondered which programming languages and execution engines are the quickest or the slowest at processing files? Are you in a dilemma as to which programming language should you code in to solve your business problem efficiently? Well look no further, here’s your answer.</p><p>We take a look at popular languages like <strong><em>Python, Java,</em></strong> and <strong><em>Scala</em></strong> and execution engines like <strong><em>Hadoop </em></strong>and <strong><em>Spark </em></strong>and see how they fare at processing files and benchmark them.</p><h3>Methodology</h3><p>We explore and conduct data analysis &amp; comparisons of the execution times taken for computing the word count of input text files varying from extremely small to extremely large sizes in various programming languages and execution engines individually.</p><p>We write sample word count programs to process these files and execute them. We then calculate the time taken to process the files individually and gather the results. Further, we collect our sample findings and observations and draw comparisons. All of the findings from individual analyses are collected and combined in a <a href="https://github.com/Thomas-George-T/File-Processing-Comparative-Analytics/blob/master/Data-Analysis.ipynb"><strong>Google Colab</strong></a> notebook where we then plot graphs using <em>matplotlib </em>and draw conclusions based on our findings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VdBjOs_0-IbrX6rZ" /><figcaption>Photo by <a href="https://unsplash.com/@wesleyphotography?utm_source=medium&amp;utm_medium=referral">Wesley Tingey</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Files</h4><p>For this experiment, we need two kinds of files. A large file that is input text only and a small text file while keeping in mind that they should be an appropriate size so that they don’t skew the performance tests that we are conducting.</p><p>Taking these limitations into consideration and that we are running a Data Science experiment <em>(of sorts).</em> We picked something relevant, For the Large text file, we chose <a href="https://norvig.com/big.txt">big.txt</a> which if you are not familiar with, is from the book <em>Beautiful Data(Segaran and Hammerbacher, 2009) </em>from the chapter <em>Natural Language Corpus Data </em>where they talk about running spell correction. How apt!</p><p>For the small file, we use the data that we have scraped from the <a href="https://en.wikipedia.org/wiki/Apache_Hadoop">Apache Hadoop Wikipedia page</a> which is another execution engine that we are benchmarking in our experiment.</p><h3>Data Analysis</h3><p>Enough talk, Let’s begin looking at Programming languages and Execution engines and how to benchmark them.</p><h4>Programming Languages</h4><p>The approach that we take is to write word count programs in the respective languages and parse both the large and small files that we have chosen. Since we repeat the same steps for Python, Scala and Python, we take the example of Python for demonstration:</p><h4>Python</h4><p>We begin our program in Python by reading the file as our first step. We use the small text file as our example:</p><pre># Reading file</pre><pre>file=open(&quot;../Input-Files/apache-hadoop-wiki.txt&quot;,&quot;r&quot;,encoding=&quot;utf-8&quot;)</pre><p>Now for our actual word count, we make use of the built-in structure <em>Dictionary</em>:</p><pre># Initializing Dictionary<br>dict = {}</pre><pre># counting number of times each word comes up in list of words (in dictionary)for word in file.read().split():<br>    dict[word] = dict.get(word, 0) + 1</pre><pre>file.close()</pre><p>We write the results that we have just computed:</p><pre>#write the file<br>fw = open(&quot;small-result-python.txt&quot;,&quot;w&quot;,encoding=&quot;utf-8&quot;)<br>fw.write(str(dict))<br>fw.close()</pre><p>A small snippet of our computed word count results:</p><pre>{&#39;Apache&#39;: 79, &#39;Hadoop&#39;: 201, &#39;From&#39;: 3, &#39;Wikipedia,&#39;: 1, &#39;the&#39;: 211, &#39;free&#39;: 1, &#39;encyclopedia&#39;: 1, &#39;Jump&#39;: 2, &#39;to&#39;: 122 }</pre><p>Now to Benchmark our program, we import the <em>time </em>package and insert hooks at the appropriate times.</p><p>So our full program looks like this:</p><pre>import time</pre><pre>start = time.time()</pre><pre># Reading filefile=open(&quot;../Input-Files/apache-hadoop-wiki.txt&quot;,&quot;r&quot;,encoding=&quot;utf-8&quot;)</pre><pre># Initializing Dictionary<br>dict = {}</pre><pre># counting number of times each word comes up in list of words (in dictionary)<br>for word in file.read().split():<br>     dict[word] = dict.get(word, 0) + 1</pre><pre>file.close()</pre><pre>#write the file<br>fw = open(&quot;small-result-python.txt&quot;,&quot;w&quot;,encoding=&quot;utf-8&quot;)<br>fw.write(str(dict))<br>fw.close()</pre><pre>end = time.time() print(&quot;Execution time :&quot;, end - start)</pre><p>Similarly, We collect the results for both the small and large file and plot a graph for all the languages individually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*6hy2tODqHA9TU12CyURaVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*wpAVZEjrDMiQ0_1Rb7MFlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*luLJF-PZQs6MGFSmnBnhIA.png" /><figcaption>(Left) Bar chart depicting processing times for Python | (Middle) Bar chart depicting processing times for Java | (Right) Bar chart depicting processing times for Scala | Images by Author.</figcaption></figure><h4>Execution Engines</h4><p>We consider Hadoop and Spark as our execution engines. We will make use of standalone mode/cluster for our experiment.</p><p>We repeat the same steps that we took for programming languages and results look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*mb-qHf0ZrUx9lc4_za_W8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*CRboVvnXCFFWbmSkxlx09A.png" /><figcaption>(left) Graph depicting processing time taken by Hadoop | (right) Graph depicting processing time taken by Spark | Images by Author</figcaption></figure><h3>Comparative Analytics</h3><p>Now that we have all the individual results for programming languages and execution engines. Let’s compare them and draw useful insights.</p><p>We use Google Colab to plot the benchmark graphs in python:</p><p>We start by importing the required libraries: <em>pandas </em>and <em>matlplotlib</em></p><pre><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br><strong>import</strong> <strong>matplotlib</strong> <strong>as</strong> <strong>mpl</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br>mpl.style.use(&#39;ggplot&#39;)</pre><p>Preparing our language dataset</p><pre><em># initialize list of languages </em><br>data_lan = [[&#39;java&#39;, 0.123, 1.179 ], [&#39;scala&#39;, 10.172, 3.528]] <br>  <br><em># Create the pandas DataFrame </em><br>df_languages = pd.DataFrame(data_lan, columns = [&#39;languages&#39;, &#39;small&#39;,&#39;big&#39;]) <br>  <br><em># print dataframe. </em><br>df_languages</pre><p>Since this is a growing experiment, and in the future, we could benchmark with more languages and execution engines, we use this method:</p><pre><em># Appending Data, Use this method in the future when trying to add more languages</em></pre><pre>df_languages = df_languages.append({&#39;languages&#39;:&#39;python&#39;,&#39;small&#39;:0.006, &#39;big&#39;:1.154}, ignore_index=<strong>True</strong>)<br>df_languages</pre><pre><em># Set the languages as index for our x axis</em><br>df_languages.set_index(&#39;languages&#39;, inplace=<strong>True</strong>)<br>df_languages</pre><p>Resultant data ready for graphing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/1*rOqTKZbYH8lVIHnxsJ-f3A.jpeg" /><figcaption>Dataset with clear X &amp; Y axis demarcations | Image by Author</figcaption></figure><p>Plotting our Data:</p><pre>ax = df_languages.plot(kind=&#39;bar&#39;,<br>                      figsize = (10,10)<br>                      )</pre><pre>plt.xlabel(&#39;Languages&#39;)<br>plt.title(&#39;Execution times for each language for both small and large files&#39;)</pre><pre>ax.set_facecolor(&#39;white&#39;)<br>ax.tick_params(axis=&#39;x&#39;, colors=&#39;black&#39;, labelsize=14)<br>ax.axhline(0, color=&#39;black&#39;)<br>ax.legend(facecolor=&#39;white&#39;,fontsize=14)<br>ax.tick_params(top=<strong>False</strong>, left=<strong>False</strong>, right=<strong>False</strong>, labelleft=<strong>False</strong>)</pre><pre><strong>for</strong> p <strong>in</strong> ax.patches:   <em>#display the percentages above the bars</em><br>    width, height = p.get_width(), p.get_height()<br>    x, y = p.get_xy() <br>    ax.annotate(&#39;<strong>{}</strong> s&#39;.format(height), (x, y + height + 0.1),fontsize=14)</pre><pre>plt.show()</pre><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*F0MXItV4FAoX-wHOUzAxRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*EuEDyyX_c76odQn-oIe1HQ.png" /><figcaption>(left) Graph comparing &amp; benchmarking the file processing time for Programming languages | (right) Graph comparing &amp; benchmarking the file processing time for Execution Engines | Images by Author</figcaption></figure><p>For programming languages, we observe that <strong>Python</strong> has the least execution time for small and large files while <strong>Scala</strong> has the largest execution time. Interestingly, <em>Scala </em>takes 7 seconds more for processing the small file rather than the larger file.</p><p>For execution engines, we observe that the <strong>Spark engine </strong>has the least execution time while <strong>Hadoop’s Mapreduce engine </strong>has the highest execution time. This is in line with the claim that Spark is 100 times quicker than Hadoop.</p><p>I hope that this experiment shed some light on which programming language or execution you should use for your next project. Overall, each programming language and execution engine have their pro’s and con’s, but with this experiment, we now know which of them have the best performance benchmarks.</p><p>This project is growing and if you would like to contribute towards it with more programming languages and execution engines, do so at <a href="https://github.com/Thomas-George-T/File-Processing-Comparative-Analytics">GitHub</a>.Thank you for reading.</p><h3>References</h3><ol><li><a href="https://norvig.com/ngrams/">Natural Language Corpus Data: Beautiful Data</a></li><li><a href="https://en.wikipedia.org/wiki/Apache_Hadoop">Apache Hadoop Wikipedia</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e21b4693b70c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/file-processing-a-comparative-analytics-study-e21b4693b70c">File Processing: A Comparative Analytics Study</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Tuning Apache Sqoop]]></title>
            <link>https://medium.com/swlh/performance-tuning-apache-sqoop-512242a58df5?source=rss-45ff032abfb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/512242a58df5</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[sqoop]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Thomas George Thomas]]></dc:creator>
            <pubDate>Sun, 06 Sep 2020 03:31:01 GMT</pubDate>
            <atom:updated>2020-09-17T03:29:19.896Z</atom:updated>
            <content:encoded><![CDATA[<h4>Six definite ways to improve efficiency and reduce load times</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YwmcoITH5Pp1t4bV" /><figcaption>Photo by <a href="https://unsplash.com/@jimmy2018?utm_source=medium&amp;utm_medium=referral">Jia Ye</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Sqoop is a tool offered by the Apache foundation that is used commonly in the Big Data world to import-export millions of records between heterogeneous relational databases (RDBMS) and Hadoop Distributed File System (HDFS). This data transfer can lead to varying load times ranging from a couple of minutes to multiple hours. This scenario is when Data engineers worldwide look under the hood to fine-tune settings. The goal of performance tuning is to get more data loaded in a shorter time, thus increasing efficiency and lessening the chance of data loss in case of network timeouts.</p><p>In General, performance tuning in Sqoop can be achieved by:</p><ul><li>Controlling Parallelism</li><li>Controlling Data Transfer Process</li></ul><h3>Controlling Parallelism</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YbXNDIKWGQERGxLm" /><figcaption>Photo by <a href="https://unsplash.com/@metastudio35?utm_source=medium&amp;utm_medium=referral">Meta Studio 35</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Sqoop works on the <em>MapReduce </em>programming model implemented in Hadoop. Sqoop imports exports data from most relational databases in parallel. The number of map tasks per job determines it’s parallelism. By controlling the parallelism, we can handle the load on our databases and hence its performance. Here are a couple of ways in Sqoop jobs to exploit parallelism:</p><h4>Changing the number of mappers</h4><p>Typical Sqoop jobs launch <strong>four </strong>mappers by default. To optimise performance, increasing the map tasks (Parallel processes) to an integer value of 8 or 16 can show an increase in performance in some databases.</p><p>By using the<em> </em>-m or <em>--num-mappers</em> parameter we can set the degree of parallelism in Sqoop. Changing the number of mappers to 10 for example:</p><pre>sqoop import  <br>--connect jdbc:mysql://mysql.example.com/sqoop \<br>--username sqoop \<br>--password sqoop \<br>--table cities \<br><strong>--num-mappers 10</strong></pre><p>A few things to keep in mind is that the number of map tasks should be less than the maximum number of parallel database connections possible. The increase in the degree of parallelism should be lesser than that which is available within your MapReduce cluster.</p><h4>Split By Query</h4><p>When performing parallel imports, Sqoop needs a criterion by which it can split the workload. Sqoop uses a <em>splitting column</em> to split the workload. <em>By default, Sqoop will identify the primary key column (if present) in a table and use it as the splitting column.</em> The low and high values for the splitting column are retrieved from the database, and the map tasks operate on evenly-sized components of the total range.</p><p>The --split-by parameter splits the column data <strong>uniformly </strong>on the basis of the number of mappers specified. The syntax for --split-by is given by:</p><pre>sqoop import  <br>--connect jdbc:mysql://mysql.example.com/sqoop \<br>--username sqoop \<br>--password sqoop \<br>--table cities \<br><strong>--split-by city_id</strong></pre><h3>Controlling Data Transfer Process</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*slTSncLr30O3zOrj" /><figcaption>Photo by <a href="https://unsplash.com/@keszthelyit?utm_source=medium&amp;utm_medium=referral">Keszthelyi Timi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A popular method of improving performance is by managing the way we import and export data. Here are a few ways:</p><h4>Batching</h4><p>Batching means that related SQL statements can be grouped into a batch when you export data.</p><blockquote>The JDBC interface exposes an API for doing batches in a prepared statement with multiple sets of values. With the <em>--batch</em> parameter, Sqoop can take advantage of this. This API is present in all JDBC drivers because it is required by the JDBC interface.</blockquote><p><em>Batching is disabled by default in Sqoop</em>. Enable JDBC batching using the --batch parameter.</p><pre>sqoop export   <br>--connect jdbc:mysql://mysql.example.com/sqoop \<br>--username sqoop \<br>--password sqoop \<br>--table cities \<br>--export-dir /data/cities \<br><strong>--batch</strong></pre><h4>Fetch Size</h4><p>The default number of records that can be imported at once is <em>1000. </em>This can be overridden by using the Fetch-size parameter which is used to specify the number of records that Sqoop can import at a time using the following syntax:</p><pre>sqoop import <br>--connect jdbc:mysql://mysql.example.com/sqoop \<br>--username sqoop \<br>--password sqoop \<br>--table cities \<br>--fetch-size=n</pre><p>Where <em>n</em> represents the number of entries that Sqoop must fetch at a time.</p><p>Based on the available memory and bandwidth, the value of the fetch-size parameter can be increased <em>w.r.t</em> the volume of data that needs to be read.</p><h4>Direct Mode</h4><p>By default, the Sqoop import process uses JDBC which provides a reasonable cross-vendor import channel support. However, some databases can achieve higher performance by using database-specific utilities since they are optimized to provide the best possible transfer speed while putting less strain on the database server.</p><p>By supplying the --direct argument, Sqoop is forced to attempt using the direct import channel. This channel may be higher performance than using JDBC.</p><pre>sqoop import \   <br>--connect jdbc:mysql://mysql.example.com/sqoop \<br>--username sqoop \<br>--password sqoop \<br>--table cities \<br><strong>--direct</strong></pre><p>There are several limitations that come with this faster import. For one, not all databases have available native utilities. This mode is not available for every supported database.</p><blockquote>Sqoop has direct support only for MySQL and PostgreSQL.</blockquote><h4>Custom Boundary Queries</h4><p>As seen before split-by uniformly distributes the data for import. If the column has non-uniform values, boundary-query can be used if we do not get the desired results while using the split-by argument alone.</p><p>Ideally, we configure the boundary-query parameter with the min(id) and max(id) along with the table name.</p><pre>sqoop import \<br>--connect jdbc:mysql://mysql.example.com/sqoop \<br>--username sqoop \<br>--password sqoop \<br>--query &#39;SELECT normcities.id, \<br>                countries.country, \<br>                normcities.city \<br>                FROM normcities \<br>                JOIN countries USING(country_id) \<br>                WHERE $CONDITIONS&#39; \<br>--split-by id \<br>--target-dir cities \<br>--boundary-query &quot;select min(id), max(id) from normcities&quot;</pre><p>Here $CONDITIONS is used internally by Sqoop and will be expanded to pick the min and max id of the cities table to split the data. By using custom values, efficient partitions are derived splitting the data resulting in performance improvement in general.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ehXVF4SFYxeJHS0e" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=medium&amp;utm_medium=referral">Tim Mossholder</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The goal of performance tuning is to achieve significant performance without compromising on resources. Hopefully using the above techniques, we should be able to improve efficiency with significantly reduced time for data transfer.</p><h3>References</h3><ul><li><a href="https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html">Sqoop User Guide (v1.4.6)</a></li><li><a href="https://community.cloudera.com/t5/Community-Articles/SQOOP-Performance-tuning/ta-p/248260">SQOOP Performance tuning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=512242a58df5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/performance-tuning-apache-sqoop-512242a58df5">Performance Tuning Apache Sqoop</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Tale of Two Cities — Clustering Neighborhoods of London and Paris]]></title>
            <link>https://medium.com/analytics-vidhya/a-tale-of-two-cities-clustering-neighborhoods-of-london-and-paris-5328f69cd8b6?source=rss-45ff032abfb6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5328f69cd8b6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Thomas George Thomas]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 10:51:04 GMT</pubDate>
            <atom:updated>2020-09-10T19:53:51.897Z</atom:updated>
            <content:encoded><![CDATA[<h4>IN-DEPTH ANALYSIS</h4><h3>A Tale of Two Cities</h3><h4>Clustering Neighborhoods of London and Paris using Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ssrm8y8ktM7SaVZV" /><figcaption>Photo by <a href="https://unsplash.com/@cyril_m?utm_source=medium&amp;utm_medium=referral">cyril mazarin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>A Tale of Two Cities, a novel written by Charles Dickens was set in London and Paris, which takes place during the French Revolution. These cities were both happening then and now. A lot has changed over the years, and we now take a look at how the cities have grown.</p><p>London and Paris are quite a popular tourist and vacation destinations for people all around the world. They are diverse and multicultural and offer a wide variety of experiences that are widely sought after. We try to group the neighbourhoods of London and Paris respectively and draw insights to what they look like now.</p><h3>Business Problem</h3><p>The aim is to help tourists choose their destinations depending on the experiences that the neighbourhoods have to offer and what they would want to have. This model also helps people make decisions if they are thinking about migrating to London or Paris or even if they wish to relocate neighbourhoods within the city. Our findings will help stakeholders make informed decisions and address any concerns they have, including the different kinds of cuisines, provision stores and what the city has to offer.</p><h3>Data Description</h3><p>We require geographical location data for both London and Paris. Postal codes in each city serve as a starting point. Using Postal codes, we can find out the neighbourhoods, boroughs, venues and their most popular venue categories.</p><h4>London</h4><p>To derive our solution, We scrape our data from <a href="https://en.wikipedia.org/wiki/List_of_areas_of_London">https://en.wikipedia.org/wiki/List_of_areas_of_London</a></p><p>This Wikipedia page has information about all the neighbourhoods; we limit it to London.</p><ol><li><em>borough</em>: Name of Neighborhood</li><li><em>town</em>: Name of the borough</li><li><em>post_code</em>: Postal codes for London.</li></ol><p>This Wikipedia page lacks information about geographical locations. To solve this problem, we use <em>ArcGIS API.</em></p><h4>ArcGIS API</h4><p>ArcGIS Online enables you to connect people, locations, and data using interactive maps. Work with smart, data-driven styles and intuitive analysis tools that deliver location intelligence. Share your insights with the world or specific groups.</p><p>More specifically, we use <em>ArcGIS </em>to get the geographical locations of the neighbourhoods of London. Adding the following columns to our initial data set prepares our data.</p><ol><li><em>latitude</em>: Latitude for Neighborhood</li><li><em>longitude</em>: Longitude for Neighborhood</li></ol><h4>Paris</h4><p>To derive our solution, We leverage JSON data available at <a href="https://www.data.gouv.fr/fr/datasets/r/e88c6fda-1d09-42a0-a069-606d3259114e">https://www.data.gouv.fr/fr/datasets/r/e88c6fda-1d09-42a0-a069-606d3259114e</a></p><p>The JSON file has data about all the neighbourhoods in France; we limit it to Paris.</p><ol><li><em>postal_code</em>: Postal codes for France</li><li><em>nom_comm</em>: Name of Neighborhoods in France</li><li><em>nom_dept</em>: Name of the boroughs, equivalent to towns in France</li><li><em>geo_point_2d</em>: Tuple containing the latitude and longitude of the Neighborhoods.</li></ol><h4>Foursquare API Data</h4><p>We will need data about different venues in different neighbourhoods of that specific borough. To gain that information, we will use “Foursquare” location information. Foursquare is a location data provider with information about all manner of venues and events within an area of interest. Such information includes venue names, locations, menus and even photos. As such, the foursquare location platform will be used as the sole data source since all the stated required information can be obtained through the API.</p><p>After finding the list of neighbourhoods, we then connect to the Foursquare API to gather information about venues inside each neighbourhood. For each neighbourhood, we have chosen the radius to be 500 meters.</p><p>The data retrieved from Foursquare contained information of venues within a specified distance of the longitude and latitude of the postcodes. The information obtained per venue as follows:</p><ol><li><em>Neighbourhood</em>: Name of the Neighborhood</li><li><em>Neighbourhood Latitude</em>: Latitude of the Neighborhood</li><li><em>Neighbourhood Longitude</em>: Longitude of the Neighborhood</li><li><em>Venue</em>: Name of the Venue</li><li><em>Venue Latitude</em>: Latitude of Venue</li><li><em>Venue Longitude</em>: Longitude of Venue</li><li><em>Venue Category</em>: Category of Venue</li></ol><p>Based on all the information collected for both London and Paris, we have sufficient data to build our model. We cluster the neighbourhoods together based on similar venue categories. We then present our observations and findings. Using this data, our stakeholders can take the necessary decision.</p><h3>Methodology</h3><p>We will be creating our model with the help of Python, so we start by importing all the required packages. The code is available on <a href="https://github.com/Thomas-George-T/A-Tale-of-Two-Cities/blob/master/Tale_of_Two_Cities_A_Data_Science_Take.ipynb">GitHub </a>to follow along.</p><pre>import pandas as pd<br>import requests<br>import numpy as np<br>import matplotlib.cm as cm<br>import matplotlib.colors as colors<br>import folium<br>from sklearn.cluster import KMeans</pre><p>Package breakdown:</p><ul><li><em>Pandas</em>: To collect and manipulate data in JSON and HTML and then data analysis</li><li><em>requests</em>: Handle HTTP requests</li><li><em>matplotlib</em>: Detailing the generated maps</li><li><em>folium</em>: Generating maps of London and Paris</li><li><em>sklearn</em>: To import K Means machine learning model.</li></ul><p>The approach taken here is to explore each of the cities individually, plot the map to show the considered neighbourhoods and then build our model by clustering all of the similar neighbourhoods together and finally plot the new map with the clustered neighbourhoods. We draw insights and then compare and discuss our findings.</p><h4>Data Collection</h4><p>In the data collection stage, we begin with collecting the required data for the cities of London and Paris. We need data that has the postal codes, neighbourhoods and boroughs specific to each of the cities.</p><p>To collect data for London, using pandas, we scrape the List of areas of London Wikipedia page to take the 2nd table:</p><pre>url_london = &quot;https:/<a href="https://colab.research.google.com/drive/11LYSVN5-SjspZvOTnz3dmOpoA7Xz4Dw6#">/en.wikipedia.org/wiki/List_of_areas_of_Lo</a>ndon&quot;<br>wiki_london_url = requests.get(url_london)<br>wiki_london_data = pd.read_html(wiki_london_url.text)<br>wiki_london_data = wiki_london_data[1]<br>wiki_london_data</pre><ol><li><em>nom_comm</em>: Name of Neighborhoods in France</li><li><em>nom_dept</em>: Name of the boroughs, equivalent to towns in France</li><li><em>geo_point_2d</em>: Tuple containing the latitude and longitude of the Neighborhoods.</li></ol><h4>Foursquare API Data</h4><p>We will need data about different venues in different neighbourhoods of that specific borough. To gain that information, we will use “Foursquare” location information. Foursquare is a location data provider with information about all manner of venues and events within an area of interest. Such information includes venue names, locations, menus and even photos. As such, the foursquare location platform will be used as the sole data source since all the stated required information can be obtained through the API.</p><p>After finding the list of neighbourhoods, we then connect to the Foursquare API to gather information about venues inside each neighbourhood. For each neighbourhood, we have chosen the radius to be 500 meters.</p><p>The data retrieved from Foursquare contained information of venues within a specified distance of the longitude and latitude of the postcodes. The information obtained per venue are as follows:</p><ol><li><em>Neighbourhood</em>: Name of the Neighborhood</li><li><em>Neighbourhood Latitude</em>: Latitude of the Neighborhood</li><li><em>Neighbourhood Longitude</em>: Longitude of the Neighborhood</li><li><em>Venue</em>: Name of the Venue</li><li><em>Venue Latitude</em>: Latitude of Venue</li><li><em>Venue Longitude</em>: Longitude of Venue</li><li><em>Venue Category</em>: Category of Venue</li></ol><p>Based on all the information collected for both London and Paris, we have sufficient data to build our model. We cluster the neighbourhoods together based on similar venue categories. We then present our observations and findings. Using this data, our stakeholders can take the necessary decision.</p><h3>Methodology</h3><p>We will be creating our model with the help of Python, so we start by importing all the required packages. The code is available on <a href="https://github.com/Thomas-George-T/A-Tale-of-Two-Cities/blob/master/Tale_of_Two_Cities_A_Data_Science_Take.ipynb">GitHub </a>to follow along.</p><pre>import pandas as pd<br>import requests<br>import numpy as np<br>import matplotlib.cm as cm<br>import matplotlib.colors as colors<br>import folium<br>from sklearn.cluster import KMeans</pre><p>Package breakdown:</p><ul><li><em>Pandas</em>: To collect and manipulate data in JSON and HTML and then data analysis</li><li><em>requests</em>: Handle HTTP requests</li><li><em>matplotlib</em>: Detailing the generated maps</li><li><em>folium</em>: Generating maps of London and Paris</li><li><em>sklearn</em>: To import K Means machine learning model.</li></ul><p>The approach taken here is to explore each of the cities individually, plot the map to show the neighbourhoods being considered and then build our model by clustering all of the similar neighbourhoods together and finally plot the new map with the clustered neighbourhoods. We draw insights and then compare and discuss our findings.</p><h4>Data Collection</h4><p>In the data collection stage, we begin with collecting the required data for the cities of London and Paris. We need data that has the postal codes, neighbourhoods and boroughs specific to each of the cities.</p><p>To collect data for London, using pandas, we scrape the List of areas of London Wikipedia page to take the 2nd table:</p><pre>url_london = &quot;https:/<a href="https://colab.research.google.com/drive/11LYSVN5-SjspZvOTnz3dmOpoA7Xz4Dw6#">/en.wikipedia.org/wiki/List_of_areas_of_Lo</a>ndon&quot;<br>wiki_london_url = requests.get(url_london)<br>wiki_london_data = pd.read_html(wiki_london_url.text)<br>wiki_london_data = wiki_london_data[1]<br>wiki_london_data</pre><p>The data looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*j8ih4c3LXWmdQg-bcgXCmw.jpeg" /><figcaption>Data for London scrapped from the Wikipedia page.</figcaption></figure><p>To collect data for Paris, we download the JSON file containing all the postal codes of France from <a href="https://www.data.gouv.fr/fr/datasets/r/e88c6fda-1d09-42a0-a069-606d3259114e">https://www.data.gouv.fr/fr/datasets/r/e88c6fda-1d09-42a0-a069-606d3259114e</a></p><p>Using Pandas, we load the table after reading the JSON file:</p><pre>!wget -q -O &#39;france-data.json&#39; https:/<a href="https://colab.research.google.com/drive/11LYSVN5-SjspZvOTnz3dmOpoA7Xz4Dw6#">/www.data.gouv.fr/fr/datasets/r/e88c6fda</a>-1d09-42a0-a069-606d3259114e<br>print(&quot;Data Downloaded!&quot;)<br>paris_raw = pd.read_json(&#39;france-data.json&#39;)<br>paris_raw.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vvf9buFk7XfAxyx5ZMoZzQ.jpeg" /><figcaption>JSON data containing postal codes of France</figcaption></figure><h4>Data Preprocessing</h4><p>For London, We replace the spaces with underscores in the title. The <em>borough</em> column has numbers within square brackets that we remove using:</p><pre>wiki_london_data.rename(columns=lambda x: x.strip().replace(&quot; &quot;, &quot;_&quot;), inplace=True)<br>wiki_london_data[&#39;borough&#39;] = wiki_london_data[&#39;borough&#39;].map(lambda x: x.rstrip(&#39;]&#39;).rstrip(&#39;0123456789&#39;).rstrip(&#39;[&#39;))</pre><p>For Paris, we break down each of the nested fields and create the dataframe that we need:</p><pre>paris_field_data = pd.DataFrame()<br>for f in paris_raw.fields:<br>    dict_new = f<br>    paris_field_data = paris_field_data.append(dict_new, ignore_index=True)<br> <br>paris_field_data.head()</pre><h4>Feature Selection</h4><p>For both of our datasets, we need only the borough, neighbourhood, postal codes and geolocations (latitude and longitude). So we end up selecting the columns that we need by:</p><pre>df1 = wiki_london_data.drop( [ wiki_london_data.columns[0], wiki_london_data.columns[4], wiki_london_data.columns[5] ], axis=1)<br> <br>df_2 = paris_field_data[[&#39;postal_code&#39;,&#39;nom_comm&#39;,&#39;nom_dept&#39;,&#39;geo_point_2d&#39;]]</pre><h4>Feature Engineering</h4><p>Both of our Datasets contain information related to all the cities in the country. We can narrow down and further process the data by selecting only the neighbourhoods of ‘London’ and ‘Paris’.</p><pre>df1 = df1[df1[&#39;town&#39;].str.contains(&#39;LONDON&#39;)]<br> <br>df_paris = df_2[df_2[&#39;nom_dept&#39;].str.contains(&#39;PARIS&#39;)].reset_index(drop=True)</pre><p>Looking over our London dataset, we can see that we don’t have the geolocation data. We need to extrapolate the missing data for our neighbourhoods. We perform this by leveraging the ArcGIS API. With the Help of ArcGIS API we can get the latitude and longitude of our London neighbourhood data.</p><pre>from arcgis.geocoding import geocode<br>from arcgis.gis import GIS<br>gis = GIS()</pre><p>Defining London ArcGIS <em>geocode </em>function to return latitude and longitude.</p><pre>def get_x_y_uk(address1):<br>   lat_coords = 0<br>   lng_coords = 0<br>   g = geocode(address=&#39;{}, London, England, GBR&#39;.format(address1))[0]<br>   lng_coords = g[&#39;location&#39;][&#39;x&#39;]<br>   lat_coords = g[&#39;location&#39;][&#39;y&#39;]<br>   return str(lat_coords) +&quot;,&quot;+ str(lng_coords)</pre><p>Passing postal codes of London to get the geographical coordinates</p><pre>coordinates_latlng_uk = geo_coordinates_uk.apply(lambda x: get_x_y_uk(x))</pre><p>We proceed with Merging our source data with the geographical co-ordinates to make our dataset ready for the next stage</p><pre>london_merged = pd.concat([df1,lat_uk.astype(float), lng_uk.astype(float)], axis=1)<br>london_merged.columns= [&#39;borough&#39;,&#39;town&#39;,&#39;post_code&#39;,&#39;latitude&#39;,&#39;longitude&#39;]<br>london_merged</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*taRsy2BGKU_L8bcvfVfbKg.jpeg" /><figcaption>The resulting feature engineered London Dataset</figcaption></figure><p>As for our Paris dataset, we don’t need to get the Geo coordinates using an external data source or collect it with the ArcGIS API call since we already have it stored in the <em>geo_point_2d</em> column as a tuple in the <em>df_paris </em>dataframe.</p><p>We just need to extract the latitude and longitude from the column:</p><pre>paris_lat = paris_latlng.apply(lambda x: x.split(&#39;,&#39;)[0])<br>paris_lat = paris_lat.apply(lambda x: x.lstrip(&#39;[&#39;))<br> <br>paris_lng = paris_latlng.apply(lambda x: x.split(&#39;,&#39;)[1])<br>paris_lng = paris_lng.apply(lambda x: x.rstrip(&#39;]&#39;))<br> <br>paris_geo_lat  = pd.DataFrame(paris_lat.astype(float))<br>paris_geo_lat.columns=[&#39;Latitude&#39;]<br> <br>paris_geo_lng = pd.DataFrame(paris_lng.astype(float))<br>paris_geo_lng.columns=[&#39;Longitude&#39;]</pre><p>We then create our Paris dataset with the required information:</p><pre>paris_combined_data = pd.concat([df_paris.drop(&#39;geo_point_2d&#39;, axis=1), paris_geo_lat, paris_geo_lng], axis=1)<br>paris_combined_data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*0BQdNFBy8KOamuNbcsEqTw.jpeg" /><figcaption>Resulting Feature engineered dataset of Paris</figcaption></figure><p><em>Note: Both the datasets have been properly processed and formatted. Since the same steps are applied to both the datasets of London and Paris, we will be discussing the code for only the London dataset for simplicity.</em></p><h3>Visualizing the Neighborhoods of London and Paris</h3><p>Now that our datasets are ready, using the Folium package, we can visualize the maps of London and Paris with the neighbourhoods that we collected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sn9ZDU3GYMdTowsN_S6R4Q.jpeg" /><figcaption>Map of London showing the neighborhoods</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3PRm0GltSTixAmOO4y5Q9g.jpeg" /><figcaption>Map of Paris showing the neighborhoods</figcaption></figure><p>Now that we have visualized the neighbourhoods, we need to find out what each neighbourhood is like and what are the common venue and venue categories within a 500 m radius.</p><p>This is where Foursquare comes into play. With the help of Foursquare we define a function which collects information pertaining to each neighbourhood including that of the name of the neighbourhood, Geo-coordinates, venue and venue categories.</p><pre>LIMIT=100<br> <br>def getNearbyVenues(names, latitudes, longitudes, radius=500):<br> <br>    venues_list=[]<br>    for name, lat, lng in zip(names, latitudes, longitudes):<br>        print(name)<br> <br>        # create the API request URL<br>        url = &#39;https:/<a href="https://colab.research.google.com/drive/11LYSVN5-SjspZvOTnz3dmOpoA7Xz4Dw6#">/api.foursquare.com/v2/venues/explore</a>?&amp;client_id={}&amp;client_secret={}&amp;v={}&amp;ll={},{}&amp;radius={}&amp;limit={}&#39;.format(<br>            CLIENT_ID, <br>            CLIENT_SECRET, <br>            VERSION, <br>            lat, <br>            lng, <br>            radius,<br>            LIMIT<br>            )<br> <br>        # make the GET request<br>        results = requests.get(url).json()[&quot;response&quot;][&#39;groups&#39;][0][&#39;items&#39;]<br> <br>        # return only relevant information for each nearby venue<br>        venues_list.append([(<br>            name, <br>            lat, <br>            lng, <br>            v[&#39;venue&#39;][&#39;name&#39;], <br>            v[&#39;venue&#39;][&#39;categories&#39;][0][&#39;name&#39;]) for v in results])<br> <br>    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])<br>    nearby_venues.columns = [&#39;Neighbourhood&#39;, <br>                  &#39;Neighbourhood Latitude&#39;, <br>                  &#39;Neighbourhood Longitude&#39;, <br>                  &#39;Venue&#39;, <br>                  &#39;Venue Category&#39;]<br> <br>    return(nearby_venues)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*JwpPnJKXqQySu8yQeQc5AA.jpeg" /><figcaption>Data collected using Foursquare API</figcaption></figure><h4>One Hot Encoding</h4><p>Since we are trying to find out what are the different kinds of venue categories present in each neighbourhood and then calculate the top 10 common venues to base our similarity on, we use the One Hot Encoding to work with our categorical data type of the venue categories. This helps to convert the categorical data into numeric data.</p><p>We won’t be using label encoding in this situation since label encoding might cause our machine learning model to have a bias or a sort of ranking which we are trying to avoid by using One Hot Encoding.</p><p>We perform one-hot encoding and then calculate the mean of the grouped venue categories for each of the neighbourhoods.</p><pre># One hot encoding<br>London_venue_cat = pd.get_dummies(venues_in_London[[&#39;Venue Category&#39;]], prefix=&quot;&quot;, prefix_sep=&quot;&quot;)<br> <br># Adding neighbourhood to the mix<br>London_venue_cat[&#39;Neighbourhood&#39;] = venues_in_London[&#39;Neighbourhood&#39;] <br> <br># moving neighborhood column to the first column<br>fixed_columns = [London_venue_cat.columns[-1]] + list(London_venue_cat.columns[:-1])<br>London_venue_cat = London_venue_cat[fixed_columns]<br> <br># Grouping and calculating the mean<br>London_grouped = London_venue_cat.groupby(&#39;Neighbourhood&#39;).mean().reset_index()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NClAAWi3CTR7JCTt9wYZQ.jpeg" /><figcaption>Data with the mean values for each of the Neighborhoods after one-hot encoding</figcaption></figure><h4>Top Venues in the Neighborhoods</h4><p>In our next step, We need to sort, rank and label the top venue categories in our neighbourhood.</p><p>Let’s define a function to get the top venue categories in the neighbourhood</p><pre>def return_most_common_venues(row, num_top_venues):<br>    row_categories = row.iloc[1:]<br>    row_categories_sorted = row_categories.sort_values(ascending=False)<br> <br>    return row_categories_sorted.index.values[0:num_top_venues]</pre><p>There are many categories, we will consider the top 10 categories to avoid data skew.</p><p>Defining a function to label them accurately</p><pre>num_top_venues = 10<br> <br>indicators = [&#39;st&#39;, &#39;nd&#39;, &#39;rd&#39;]<br> <br># create columns according to number of top venues<br>columns = [&#39;Neighbourhood&#39;]<br>for ind in np.arange(num_top_venues):<br>    try:<br>        columns.append(&#39;{}{} Most Common Venue&#39;.format(ind+1, indicators[ind]))<br>    except:<br>        columns.append(&#39;{}th Most Common Venue&#39;.format(ind+1))</pre><p>Getting the top venue categories in the neighbourhoods of London</p><pre># create a new dataframe for London<br>neighborhoods_venues_sorted_london = pd.DataFrame(columns=columns)<br>neighborhoods_venues_sorted_london[&#39;Neighbourhood&#39;] = London_grouped[&#39;Neighbourhood&#39;]<br> <br>for ind in np.arange(London_grouped.shape[0]):<br>    neighborhoods_venues_sorted_london.iloc[ind, 1:] = return_most_common_venues(London_grouped.iloc[ind, :], num_top_venues)<br> <br>neighborhoods_venues_sorted_london.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hoB9ZuuiRw-NIBOOWuP2eA.jpeg" /><figcaption>Top Common venues in each Neighborhood ranked.</figcaption></figure><h4>Model Building — K Means</h4><p>Moving on to the most exciting part — <strong>Model Building!</strong> We will be using K Means Clustering Machine learning algorithm to cluster similar neighbourhoods together. There are many different cluster sizes that we can select, we will be going with the number of clusters as 5 to keep it as optimized as possible.</p><pre># set number of clusters<br>k_num_clusters = 5<br> <br>London_grouped_clustering = London_grouped.drop(&#39;Neighbourhood&#39;, 1)<br> <br># run k-means clustering<br>kmeans_london = KMeans(n_clusters=k_num_clusters, random_state=0).fit(London_grouped_clustering)</pre><p>Our model has labelled each of the neighbourhoods, we add the label into our dataset.</p><pre>neighborhoods_venues_sorted_london.insert(0, &#39;Cluster Labels&#39;, kmeans_london.labels_ +1)</pre><p>We then join <em>London_merged dataframe </em>with our <em>neighborhood venues sorted dataframe</em> to add latitude &amp; longitude for each of the neighbourhoods to prepare it for visualization.</p><pre>london_data = london_merged<br> <br>london_data = london_data.join(neighborhoods_venues_sorted_london.set_index(&#39;Neighbourhood&#39;), on=&#39;borough&#39;)<br> <br>london_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gbZR6wlMphk4hwWPg7dbnQ.jpeg" /><figcaption>Labelled Clustered dataset of London</figcaption></figure><h4>Visualizing the clustered Neighborhoods</h4><p>Our data is processed, missing data is collected and compiled. The Model is built. All that’s remaining is to see the clustered neighbourhoods on the map. Again, we use Folium package to do so.</p><p>We drop all the NaN( Not a Number) values to prevent data skew</p><pre>london_data_nonan = london_data.dropna(subset=[&#39;Cluster Labels&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B5xllZ7F1gL6r6NrVDwaXg.jpeg" /><figcaption>Map of clustered neighbourhoods of London</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knF9CXVQkQ-gkcdjx8oI1g.jpeg" /><figcaption>Map of clustered neighbourhoods of Paris</figcaption></figure><h4>Examining our Clusters</h4><p>We could examine our clusters by expanding on our code using the Cluster Labels column:</p><p>Cluster 1</p><pre>london_data_nonan.loc[london_data_nonan[&#39;Cluster Labels&#39;] == 1, london_data_nonan.columns[[1] + list(range(5, london_data_nonan.shape[1]))]]</pre><p>Cluster 2</p><pre>london_data_nonan.loc[london_data_nonan[&#39;Cluster Labels&#39;] == 2, london_data_nonan.columns[[1] + list(range(5, london_data_nonan.shape[1]))]]</pre><p>Cluster 3</p><pre>london_data_nonan.loc[london_data_nonan[&#39;Cluster Labels&#39;] == 3, london_data_nonan.columns[[1] + list(range(5, london_data_nonan.shape[1]))]]</pre><p>Cluster 4</p><pre>london_data_nonan.loc[london_data_nonan[&#39;Cluster Labels&#39;] == 4, london_data_nonan.columns[[1] + list(range(5, london_data_nonan.shape[1]))]]</pre><p>Cluster 5</p><pre>london_data_nonan.loc[london_data_nonan[&#39;Cluster Labels&#39;] == 5, london_data_nonan.columns[[1] + list(range(5, london_data_nonan.shape[1]))]]</pre><h3>Results and Discussion</h3><p>The neighbourhoods of London are very multicultural. There are a lot of different cuisines including Indian, Italian, Turkish and Chinese. London seems to take a step further in this direction by having a lot of restaurants, bars, juice bars, coffee shops, Fish and Chips shop and Breakfast spots. It has a lot of shopping options too with that of the Flea markets, flower shops, fish markets, Fishing stores, clothing stores. The main modes of transport seem to be Buses and trains. For leisure, the neighbourhoods are set up to have lots of parks, golf courses, zoo, gyms and Historic sites. Overall, the city of London offers a multicultural, diverse and certainly entertaining experience.</p><p>Paris is relatively small in size geographically. It has a wide variety of cuisines and eateries including French, Thai, Cambodian, Asian, Chinese etc. There are a lot of hangout spots including many Restaurants and Bars. Paris has a lot of Bistros. Different means of public transport in Paris which includes buses, bikes, boats or ferries. For leisure and sightseeing, there are a lot of Plazas, Trails, Parks, Historic sites, clothing shops, Art galleries and Museums. Overall, Paris seems like the relaxing vacation spot with a mix of lakes, historic spots and a wide variety of cuisines to try out.</p><h3>Conclusion</h3><p>The purpose of this project was to explore the cities of London and Paris and see how attractive it is to potential tourists and migrants. We explored both the cities based on their postal codes and then extrapolated the common venues present in each of the neighbourhoods finally concluding with clustering similar neighbourhoods together.</p><p>We could see that each of the neighbourhoods in both the cities have a wide variety of experiences to offer which is unique in its own way. The cultural diversity is quite evident which also gives the feeling of a sense of inclusion.</p><p>Both Paris and London seem to offer a vacation stay or a romantic getaway with a lot of places to explore, beautiful landscapes, amazing food and a wide variety of culture. Overall, it’s up-to-the stakeholders to decide which experience they would prefer more and which would more to their liking.</p><p>The detailed code is available on <a href="https://github.com/Thomas-George-T/A-Tale-of-Two-Cities/blob/master/Tale_of_Two_Cities_A_Data_Science_Take.ipynb">GitHub</a>. Thanks for reading!</p><h3>References</h3><ol><li><a href="https://medium.com/@oludayo.oguntoyinbo/the-battle-of-neighbourhood-my-londons-perspective-d363163771e0">The Battle of Neighbourhood — My London’s Perspective by Dayo John</a></li><li><a href="https://www.linkedin.com/pulse/battle-neighborhoods-what-best-place-where-can-i-start-bouziane/">The Battle of neighborhoods! What is the best place where can I start my restaurant business in Paris? by Zakaria BOUZIANE</a></li><li><a href="https://foursquare.com/">Foursquare API</a></li><li><a href="https://www.arcgis.com/index.html">ArcGIS API</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5328f69cd8b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-tale-of-two-cities-clustering-neighborhoods-of-london-and-paris-5328f69cd8b6">A Tale of Two Cities — Clustering Neighborhoods of London and Paris</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>