<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Soowhan on Medium]]></title>
        <description><![CDATA[Stories by Soowhan on Medium]]></description>
        <link>https://medium.com/@thom42?source=rss-84435f9f9e77------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uUnLKfLQNnckRr448KhMUw.jpeg</url>
            <title>Stories by Soowhan on Medium</title>
            <link>https://medium.com/@thom42?source=rss-84435f9f9e77------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:15:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thom42/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pet (Cat and Dog) Adoption Speed Prediction (Kaggle PetFinderMy) Part 2. Model]]></title>
            <link>https://medium.com/@thom42/pet-cat-and-dog-adoption-speed-prediction-kaggle-petfindermy-part-2-model-a46b91a439bd?source=rss-84435f9f9e77------2</link>
            <guid isPermaLink="false">https://medium.com/p/a46b91a439bd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[pets-and-animals]]></category>
            <category><![CDATA[decision-tree-classifier]]></category>
            <dc:creator><![CDATA[Soowhan]]></dc:creator>
            <pubDate>Fri, 29 Mar 2019 00:12:53 GMT</pubDate>
            <atom:updated>2019-03-29T00:12:53.293Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey all! So, this is the modeling part, and let’s get started right away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rktQHz1uFDkDGa9c04ZBLw.jpeg" /></figure><p>Before we start anything, we could think about what kinds of ML algorithms we are going to use. To be honest, there’s no right answer to this I believe, and the best is to do every ML algorithms out there, and see which one works the best. From the first part of this project EDA, we found out that a lot of features would not work with “regression” since some do not have linear relationships. So, we can opt that out. Though, let’s say out of all the Decision Tree algorithms such as Adaboost, XGboost, or LGBM, to choose which is the best would totally depend on how the dataset looks like, and when I say this “looks like”, it’s something that we humans can’t really grasp by looking at random numbers and categories in the tabular data. (I mean, grasping this concept is basically the reason why we invented such decision tree algorithms)</p><p>Therefore, I have conducted all RandomForrest, XGBoost, and LGBM. From this, I found that XGBoost and LGBM produce the best results, especially LGBM. In this article, therefore, I will share my LGBM model. In terms of preparing the data, for image data, we are not conducting CNN ourselves from scratch, but we are using already predefined weights to extract numbers (in this case, vectors) from these images (using DenseNet21). We are essentially changing images into numbers, and to do this, we are using a model which was already trained on millions of pictures of dogs and cats.</p><p>For metadata and sentiment data, we are “parsing” them; extract features by ‘PetID’ and later on, we are going to merge all these data into one complete data frame with original tabular data features (breeds, colors, age, etc) with features from meta/sentiment data, and vectorized numbers feature from images.</p><p>Let’s start with importing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hJHPXrNNx-BdwUPYdaZ6xg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29VjZir48e8fbxujCZJcGg.png" /><figcaption>Loading core DataFrames</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cUF36sEVCDwKmuo7k1ZEBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1lQlrAd8hnVQi-8EOlvTug.png" /></figure><p><strong>Train Analysis:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-PF_-5FheADyTLa8pnz49g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyUkbi5sLBds-uJwPJPUDQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPK5fXlLKqs9nFsQ-dYcDg.png" /></figure><p><strong>Data Parsing &amp; Feature Extraction:</strong></p><p>After taking a look at the data, we know its structure and can use it to extract additional features and concatenate them with basic train/test DFs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rgbIjUaS7L5OzdqM0IqHCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWT-V8FGcMDAnIk9WhUJvg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YycqS9RO7d3L84JezO3wvw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6FBLfXhdp8CQrGheTHkfhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rpe_v9aG81ASH7BtKxdeDQ.png" /><figcaption>Data Parsing and Extracting Features Helper Fuctions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jBQo6TNmbcKQ-Xs2JZLJpQ.png" /><figcaption>The output of the test_dfs_sentiment.shape and test_dfs_metadata.shape</figcaption></figure><p><strong>Group Extracted Features by ‘PetID’:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3c9c0H9x4M72_BHJ536SXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lkcKkmf7qCP_g7UHe2L5rQ.png" /></figure><p><strong>Merge processed DFs with base train/test DF:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJzPeuYCpHWydZ4WQytccQ.png" /></figure><p><strong>Breed Mapping:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-O9s6krd6-teMxjYiMZOhA.png" /></figure><p><strong>Concatenate Train &amp; Test Set:</strong></p><p>Inspect NaN structure of the processed data: AdoptionSpeed is the target column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IoV0eOjgosuvvK3OqObivg.png" /></figure><p><strong>Extract different column types:</strong></p><ol><li>Integer columns are usually categorical features, which do not need encoding.</li><li>Float columns are numerical features.</li><li>Object columns are categorical features, which should be encoded.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iT-iayq0kDaqTDvArrAOAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHfKU3JzxvCdeJdKStFXEw.png" /></figure><p><strong>Feature Engineering:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SNNZzQoWElJh7HgGcnSr1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sMNXCmhiXLRAEZ5ah7MW3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uYMZFIZdARPNla_7MO_yhw.png" /></figure><p><strong>Train / Test split:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f32gOvIQX0JrEKwpjr-TSA.png" /></figure><p><strong>Model Training:</strong></p><p>Note — Before going to the model training, one can study the NaN structure. You can check it by putting in command “np.sum(pd.isnull(X_train/test))”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExqX3I2Z-zvF7yrJwulDTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-qAJ_dwlTLOT7v8ZebeOPg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxQVk5EOIt1d1dBHj_4qdA.png" /><figcaption>Modeling Helper Functions</figcaption></figure><p><strong>Light GBM Model:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vzkEx1DMU7BcwBa7imPlFQ.png" /></figure><p><strong>Train</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BxjT28vgmNJlim3EaCSV1A.png" /></figure><p><strong>Distribution of Trained Set</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aoTmnEgnjjorhSTlfYReHA.png" /></figure><p><em>adapted from: </em><a href="https://www.kaggle.com/myltykritik/simple-lgbm-image-features"><em>https://www.kaggle.com/myltykritik/simple-lgbm-image-features</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jUXK1wA9yRpHcdLuKeVHXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fN-DIwGUEDAZU2rdzNlohg.png" /></figure><p>Here, people ask a lot why manually adjusted coefficients. According to the Author who came up with this idea, this is due to trial and error. The OptimizedRounder function is what defines these coefficients automatically depending on the dataset. The thing with OptimizedRounder, though, is that when using it, the predicted distribution and the actual test data predictions were too different. This however wil not result in overfitting because it doesn’t even look at the training data. This is technically just binning the results of the predictions on the test data to match the distribution of the training targets. Though, for the actual practice, using OptimizedRounder function suffices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFQIX_w8WCjaVQuIulsc7g.png" /></figure><p>And that is it!</p><p>Thanks for reading.</p><p>Resources used:</p><p><a href="https://www.kaggle.com/ranjoranjan/single-xgboost-model">https://www.kaggle.com/ranjoranjan/single-xgboost-model</a><br><a href="https://www.kaggle.com/wrosinski/baselinemodeling">https://www.kaggle.com/wrosinski/baselinemodeling</a><br><a href="https://www.kaggle.com/skooch/petfinder-simple-lgbm-baseline/comments#483351">https://www.kaggle.com/skooch/petfinder-simple-lgbm-baseline/comments#483351</a><br><a href="https://www.kaggle.com/skooch/corrected-catboostregressor">https://www.kaggle.com/skooch/corrected-catboostregressor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a46b91a439bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pet (Cat and Dog) Adoption Speed Prediction (Kaggle PetFinderMy) Part 1. EDA]]></title>
            <link>https://medium.com/@thom42/pet-cat-and-dog-adoption-speed-prediction-kaggle-petfindermy-part-1-eda-631f9538cde6?source=rss-84435f9f9e77------2</link>
            <guid isPermaLink="false">https://medium.com/p/631f9538cde6</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[pets]]></category>
            <dc:creator><![CDATA[Soowhan]]></dc:creator>
            <pubDate>Thu, 28 Mar 2019 21:45:23 GMT</pubDate>
            <atom:updated>2019-03-28T21:45:23.480Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JK4hWIbWFq05XrGKwI3hpA.jpeg" /></figure><p>Hello Medium Readers. It’s been a few months since the last article about web-scraping with Python. Since then, I’ve been continuously pursuing on Machine Learning studies, especially through Kaggle. For those people who are not familiar with Kaggle, it’s an online community of data scientists and machine learners, owned by Google LLC. Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges. (Wikepedia) Recently on Kaggle, I found a competition held by PetFinder.my, a Malaysia’s leading animal welfare platform since 2008, with a database of more than 150,000 animals. Participants develop algorithms to predict the adoptability of pets. Here are two main reasons why I decided to do the project: First, my sister and cousin have cats and they are adopted, but the latest one who was adopted (her name is “pepper”) was suffering from depression because she was abandoned by a previous carer. It took some time for her to adjust to my cousin’s care, and eventually be in a happy state. After watching a bit of pain my cousin shared with me through this happening, I’ve always felt sympathy for strayed cats and dogs. Through this competition, therefore, I thought maybe I could help animals similar to Pepper out there. Second, the dataset provided for the competition seems to have all kinds of data such as tabular data, images, and text (description) data, and it can be a great opportunity to learn about conducting data analysis on such various data set. I know that, if you are reading this article already, you are also an animal lover, and I am glad that we together can help these cats and dogs find their homes as fast as possible. Let’s get to it right away!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4L1pn8GYRxCNcXGF-etUvQ.jpeg" /><figcaption>What are we doing?</figcaption></figure><p>So, what are we doing exactly in this competition? You might be thinking that ‘all I heard yet is that we are predicting the adoptability of pets.’ What does that mean? How’s the adoptability of pets decided? How does it help these pets actually be adopted faster?</p><p>The organization PetFinder.my’s core mission as a company is to improve animal welfare. In this case, PetFinder is currently experimenting with a simple AI tool called the Cuteness Meter, which ranks how cute a pet is based on qualities present in their photos. Participants are to develop algorithms to predict the adoptability of pets — specifically, how quickly is a pet adopted?</p><p><em>If successful, these algorithms can be adapted into AI tools that will guide shelters and rescuers around the world on improving their pet profiles’ appeal, reducing animal suffering and euthanization.</em></p><p>So, what defines the “quickness” of a pet being adopted?</p><p>From the main dataset, the tabular data, we have a column called “Adoption Speed” and the values vary from 0, 1, 2, 3, and 4. These number signify the following:</p><p>Contestants are required to predict this value. The value is determined by how quickly, if at all, a pet is adopted. The values are determined in the following way:</p><ul><li>0 — Pet was adopted on the same day as it was listed.</li><li>1 — Pet was adopted between 1 and 7 days (1st week) after being listed.</li><li>2 — Pet was adopted between 8 and 30 days (1st month) after being listed.</li><li>3 — Pet was adopted between 31 and 90 days (2nd &amp; 3rd month) after being listed.</li><li>4 — No adoption after 100 days of being listed. (There are no pets in this dataset that waited between 90 and 100 days).</li></ul><p>As the above, the column is basically the “target value”. What are the training values? Or, what kinds of data are there for us to study and train so that we can predict the target value?</p><ul><li>PetID — Unique hash ID of pet profile</li><li>AdoptionSpeed — Categorical speed of adoption. Lower is faster. This is the value to predict. See below section for more info.</li><li>Type — Type of animal (1 = Dog, 2 = Cat)</li><li>Name — Name of pet (Empty if not named)</li><li>Age — Age of pet when listed, in months</li><li>Breed1 — Primary breed of pet (Refer to BreedLabels dictionary)</li><li>Breed2 — Secondary breed of pet, if pet is of mixed breed (Refer to BreedLabels dictionary)</li><li>Gender — Gender of pet (1 = Male, 2 = Female, 3 = Mixed, if profile represents group of pets)</li><li>Color1 — Color 1 of pet (Refer to ColorLabels dictionary)</li><li>Color2 — Color 2 of pet (Refer to ColorLabels dictionary)</li><li>Color3 — Color 3 of pet (Refer to ColorLabels dictionary)</li><li>MaturitySize — Size at maturity (1 = Small, 2 = Medium, 3 = Large, 4 = Extra Large, 0 = Not Specified)</li><li>FurLength — Fur length (1 = Short, 2 = Medium, 3 = Long, 0 = Not Specified)</li><li>Vaccinated — Pet has been vaccinated (1 = Yes, 2 = No, 3 = Not Sure)</li><li>Dewormed — Pet has been dewormed (1 = Yes, 2 = No, 3 = Not Sure)</li><li>Sterilized — Pet has been spayed / neutered (1 = Yes, 2 = No, 3 = Not Sure)</li><li>Health — Health Condition (1 = Healthy, 2 = Minor Injury, 3 = Serious Injury, 0 = Not Specified)</li><li>Quantity — Number of pets represented in profile</li><li>Fee — Adoption fee (0 = Free)</li><li>State — State location in Malaysia (Refer to StateLabels dictionary)</li><li>RescuerID — Unique hash ID of rescuer</li><li>VideoAmt — Total uploaded videos for this pet</li><li>PhotoAmt — Total uploaded photos for this pet</li><li>Description — Profile write-up for this pet. The primary language used is English, with some in Malay or Chinese.</li></ul><p>As the title of this article suggests, I am mainly focusing on “EDA” part of this project on this article. Next article, I am going to be talking about some feature engineering and models such as XGBoost for the actual prediction. For now, we will study about the data and “learn” about the data to see what kinds of feature engineering and modeling would be adequate.</p><p>First of all, this project is Kernel-based only. This means that you, as a participant at Kaggle competition, will not be able to submit Python or R files separately, but the files will have to be written on the kernels and be able to run on the kernels. What this means and restricts the participants from doing is that participants cannot conduct more than 6 hours of training time with GPU provided Kernel. This may prohibit some really powerful models to be created, but this will also ensure that everyone has the same computational resources, making it a fair game.</p><p>Therefore, I am going to assume that everyone is using the same computational resource which is the Kaggle Kernel, and attach the codes that will run ‘on the kernel’ so that you can try them out on your own Kernel as well. (For the complete code, I will put it on my Github for readability.)</p><p><strong>Also, before we start, I want to let readers know that this is both an ensemble and my work for conclusive EDA of the project. (The competition ended yesterday. So this publication won’t hurt other competitors in terms of the competition results)</strong></p><p>Let’s first take a look at the whole dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kDplBvQDfYemu342KxFRxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FvimK_FoAdNG2PNDtpdNQQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3Y1K6bL3sEfrzm54hT0fw.png" /><figcaption>Looking at the whole Dataset. EDA code sourced by Andrew Lukyanenko</figcaption></figure><p>From the above simple info command, we can see the following:</p><ol><li>We have almost 15,000 dogs and cats in the dataset</li><li>Main dataset contains all important information about pets such as age, breed, color.</li><li>Descriptions were analyzed using Google’s Natural Language API providing sentiments and entities. I supposed we could do a similar thing ourselves, or we could even just incorporate the already weight-defined NLP libraries.</li><li>There are photos for some pets, and maybe we can do CNN with them.</li><li>Some meta-information was extracted from images, and we might be able to use them in training.</li><li>There are separate files with labels for breeds, colors, and states.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hfl2xZyHqgnVE5K5OlVBtA.png" /><figcaption>Value Counts for different Adoption Speed (Target Variable)</figcaption></figure><p>I can make this visually easier to see with colors and percentages of each class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHSoQpm51z4jOSV1D3PmMQ.png" /><figcaption>using seaborn library, we can achieve better visualization</figcaption></figure><p>Things to look at :</p><ol><li>Some pets (2.73%) are adopted immediately. Though, this is rare. Guess is that these are events by luck that some pets are just seen by some people who want any pet.</li><li>A lot of pets aren’t adopted at all (class 4) ranging up to 28%.</li><li>Not perfectly, but the classes have somewhat a linear relationship. The higher the number (0,1,2,3,4), the higher the percentage of being not adopted. So, not only multiclass classfication, but also regression might be worth to look at.</li></ol><p>But, intuitively, we need to check if the number of cats and dogs are the same in the dataset, or even similar. What is dogs have a very different graph of adoption rate than cats? Then, we will have to train them separately, and all we need to do when predicting is that if it is a dog prediction, put it in a dog prediction model and if it’s cat, then put it in a cat prediction model. To do this, we will have to check the number first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vA5scEwAQsyIyRx5awG67w.png" /><figcaption>Check the ratio of cats vs dogs</figcaption></figure><p>We see that the rate of dogs is a bit higher. Though, it wasn’t as much as we were worried. However, the significance is still there to make them a separete modeling. See this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IxKMKOrkISezlR2GSTMjIQ.png" /><figcaption>Code for creating dictionary and plot tools</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwC52f-SJ-xO6prem7fHHQ.png" /><figcaption>Cats vs Dogs in Adoption Speed</figcaption></figure><p>We see that cats are more likely to be adopted early than dogs. This could mean that people prefer cats more than dogs. However, we always have to remember the possibility of dataset being biased. It’s only 15000. If it bcomes 150,000 then it might converge to 50~50 for cats and dogs. Also, we can see that dogs are adopted more after several months (look at class 3 and 4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*4rasYDkH51PRWnf4UEAo4Q.png" /><figcaption>Name Tag</figcaption></figure><p>Let’s take a look into “Name” using WordCloud library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKIOz4wbPiguk2rYZRV1bg.png" /><figcaption>WordCloud</figcaption></figure><p>Things to look at:</p><ol><li>There are normal pet names like Angel, Tiger, Mimi, etc.</li><li>People often write “Puppies” and “Kittens” instead of their unique names.</li><li>People often write colors, or gender such as White, Black, Boy, and Girl.</li><li>Sometimes, the names are not names but descriptions about them such as Mixed, Adoption, Mix.</li><li>Some have no names. Is this important? Let’s check.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pfse5DjbMhkFKw6B6Gl3xw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwk_WYOlh2e8bOnx1Egh0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqcCF0bP6aJjdBcGlWy3WQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsoorZOA_2twSmdUIa_mLQ.png" /></figure><p>We see that only less than 10% of pets don’t have their names written down, but this is significant because they have a higher possibility of not being adopted.</p><p>Also, there are some supposedly “bad names.” For example, shorter names tend to be meaningless. Here are some examples of some names with 3 characters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vvkSWJh7pMaJYD5_HZItsg.png" /><figcaption>3 character names</figcaption></figure><p>And here are 1 or 2 character names</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c_mw8jJWeqizW8kccYzdUA.png" /><figcaption>1 or 2 character names</figcaption></figure><p>We can conclude that these have low chance of being adopted. Therefore, we could let people know that when they write in their pets’ names and they want their pets to be adopted, we can advise them that if the names are shorter than 3 letters, it may have lower chance of being adopted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6XDZHsliLu9YEutfgraMZg.jpeg" /><figcaption>Age goes by time</figcaption></figure><p>Let’s look at Age now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5d_QMZ6_kStHjJhptc3xA.png" /><figcaption>Age</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fVulkUVYzszhc3Z0O-sJtA.png" /></figure><p>A little bit of weird stuff here; there are a lot of pets with an age equal to multiples of 12. This could mean that they are not quite sure of exact age, and didn’t bother looking for it. We can also see that most pets are young — after their birth mostly.</p><p>Let’s take a look at Violinplot to better see the density.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIyBt9kefzUefly9ZMjDEw.png" /><figcaption>Violin Plot for Age</figcaption></figure><p>Now, the trends of Adoption Speed depending on Age</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rY_ScWaQrkKRFqPsCrtr7g.png" /></figure><p>Things to look at:</p><ol><li>We can see that young pets are adopted faster and most of them do end up being adopted (a sign of a strong feature).</li><li>Most pets are less than 4 months old with a huge spike at 2 months old.</li><li>As seen from the above, quite a lot of people do not bother with exact age (multiples of 12).</li><li>We could do feature engineering as binninig the age, or make some binary classification variables based on age.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4hZ8XgaTQXiX2O8Ox8D-2A.jpeg" /></figure><p>Now, let’s take a look at “Breeds.”</p><p>There is a main breed of the pet and secondary if relevant. A lot of pets don’t have secondary breed information.</p><p>For now, let’s check whether having secondary breed influences the adoption speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTm5Ml4nZ44-B3ImL9BmVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2gAfiaFNgPZ0lpx-pR2VQ.png" /></figure><p>From the graphs above, we can see that non-pure breed pets tend to be adopted more and faster, especially cats.</p><p>Now, let’s take a look into contents of breeds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w9zYLwocdLLyP-hv2jPRfA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WV0EnTprZ_As5KIlzcIy8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2M-XprHdae4N4WoN4Ea_8w.png" /><figcaption>Code snippets for breeds</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*krqd5Z7yvfWv5gj6ZmiFpg.png" /></figure><p>It looks like that not all values of these features are really breeds. Sometimes, people simply write that the dogs have a mixed breed, and for cats, some people write domestic with certain hair length.</p><p>Let’s take a look at the combinations of breed names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvWs-aXNbKxsw7nOrrDG7Q.png" /></figure><p>Things to look at:</p><ol><li>Most dogs are not pure breeds, but mixed breeds. (My first assumption was wrong!)</li><li>Sometimes, people write “mixed breed” in the first breed field, or sometimes even in both fields. And sometimes, breed is in the first field and is marked as mixed breed in the second field.</li><li>We could do some feature engineering here since the data seems to be complex, yet if done well with hair length of pets, can be a good feature.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJj-r_LbHgYhvE4sftRkJw.png" /></figure><p>Now, let’s take a look into gender.</p><p>Here, 1 = Male, 2= Female, 3 = Mixed, if profile represents a group of pets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HCQvpX6RB6dwTDXJdfy_Rw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qom0O3ucdAN2tehg9s4k8Q.png" /></figure><p>Things to look at:</p><ol><li>It’s clear that male pets are adopted faster than female. Also, it’s more noteworthy to see that having no information about the gender significantly decreases the chance of being adopted.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JC4UpIfGtbDNO6U7g60ZRA.png" /></figure><p>Now, let’s look at Colors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFnndNSNl-NFzVR1NHAGlA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xWWfTkG6KIgnHWvKU9aAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxC6vjZucJY1CPfLQsaaIg.png" /><figcaption>Color Code Snippet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*riD2NmvWMLCfzybn-SxJ3g.png" /></figure><p>Here, we can see that most common colors are black and brown. One interesting point to notice is that there are almost no gray or yellow dogs.</p><p>Let’s check whether colors influence adoption speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PF6SlSobZgGDofeawjdPIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8zO6CltxfOiHe6Vwrgw2YA.png" /></figure><p>Here, we see that there are some differences based on color, but the number of pets in most colors isn’t very high, and this could mean that this is due to randomness. (Less significant)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-O0QYvN45lmKeK0xltY1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5G1znfFQWLsl3wUOLVe0IQ.jpeg" /><figcaption>Different sizes</figcaption></figure><p>Now, let’s check maturity size.</p><p>Size at maturity (1 = Small, 2 = Medium, 3 = Large, 4 = Extra Large, 0 = Not Specified)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ciyO8C--petxrIWL0CV2-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrqhL5AmB2XpND7jS6Nc8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2OFGMXRii8lGhmoG6PhJFQ.png" /><figcaption>Code Snippet for Maturity Size</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0IuYZKTi0nbP3atlnzWXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1fZKzjVRtGqZeQ5CQEIhyw.png" /><figcaption>Result from code snippets</figcaption></figure><p>Things to look at:</p><ol><li>We can see that maturity size isn’t very important. Medium sized pets are most common and they have slightly higher chances of being not adopted.</li><li>There are almost no Extra Large pets. Maybe, when grown up, pets are probabilistically in a more stable relationship with their owners.</li><li>To check with our naked eyes, I showed examples of pictures of most common breeds for each maturity size of cats and dogs.</li><li>Then, not all data are entirely correct: sometimes short haired cats have breed with “medium hair”, not sure that all breeds are entirely correct. Some photos also have bad quality.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mh5ECBVcWBU1eJycvcz-gw.jpeg" /></figure><p>Let’s take a look at Fur Length.</p><p>Fur Length: 1 = Short, 2 = Medium, 3 = Long, 0 = Not Specified</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zSf_B3tE8kACMKT0DHyGWA.png" /></figure><p>We can see that most of the pets have a short fur length and long fur length is the least common. Also, pets with long hair tend to have a higher chance of being adopted. Though it could be due to randomness due to low count.</p><p>As we checked earlier in Breeds, some breed have hair length in their text. Let’s check if this meta-data of Breeds and actual fur (hair) length match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHjxGoaJ4ljEYE4ek2dvlA.png" /><figcaption>Code Snippet to check the Breed and Fur Length</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZrkBvoMVPeASTzlR69Plg.png" /><figcaption>WordCloud</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWgWCzZDgucQtBHSlcRcRg.png" /></figure><p>It looks like there are almost one thousand pets that have mismatch in their breeds and fur lengths. Let’s take some of them and look them ourselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WwjWaHQlz4UxvP5FduI3GQ.png" /></figure><p>Turns out that a lot of people lied about the information!</p><p>Sometimes, breed is more correct, and sometims fur length is correct. Maybe, we can also create another binary feature that checks whether these information match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kq5QXl5vUMS0vOrZAadcpg.jpeg" /></figure><p>Now, let’s take a look at Health.</p><p>There are four features showing health of the pets:</p><ul><li>Vaccinated — Pet has been vaccinated (1 = Yes, 2 = No, 3 = Not Sure)</li><li>Dewormed — Pet has been dewormed (1 = Yes, 2 = No, 3 = Not Sure)</li><li>Sterilized — Pet has been spayed / neutered (1 = Yes, 2 = No, 3 = Not Sure)</li><li>Health — Health Condition (1 = Healthy, 2 = Minor Injury, 3 = Serious Injury, 0 = Not Specified)</li></ul><p>My hypothesis is that these features are very important — most people would prefer a healthy pet. While sterilization isn’t the main concern, having a healthy and dewormed pet should have a great importance. Let’s see whether our hypothesis is correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKcL4J9afqv0rBqL6P6dtA.png" /></figure><p>Things to look at:</p><ol><li>Almost all pets are healthy. Pets with minor injuries are rare and sadly they aren’t adopted well. Number of pets with serious injuries is negligible.</li><li>It is interesting that people prefer non-vaccinated pets. Maybe they want to bring pets to vets themselves. [Update: So, I went online and searched about this issue, and found out that for cats and dogs, it is crucial for them to get vaccinated and dewormed in their first year. Otherwise, it may affect them for their lifetime. So, I thought that this could mean that if someone sees non-vaccinated pets, and vaccinated pets, he or she knows that the non-vaccinated pets need “more” help since they are in danger.]</li><li>People also prefer non-sterilized pets. Maybe they want puppies/kittens, or they do not agree with sterilizing for the convenience of pet owners.</li><li>Quite important is the fact that when there is no information about health condition, the probability of not being adopted is much higher.</li><li>So, let’s look at most popular health conditions.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9oojD1pHgRgobMCQjDtXzQ.png" /></figure><ol><li>Healthy, dewormed and non-sterilized pets tend to be adopted faster.</li><li>Completely healthy pets are more likely to be not adopted. I suppose that means that a lot of people pay attention to other characteristics.</li><li>And healthy pets with no information (not sure value) also tend to be adopted less frequently. Maybe people prefer having information, even if it could be negative.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-vjW5c1nboohbsIsdOK9VA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JVSkZ0uHcw5VkCuY6_SKDQ.jpeg" /></figure><p>Now, let’s look at Quantity.</p><p>Sometimes, there are more than one pet in one advertisement. Quantities of these groups very upto more than 20.</p><p>Let’s look at some examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tuoo208gEwnQwApc05odGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJ8hWAQ2fwQoxAIBTqwDIg.png" /></figure><p>If you look at closely, you can see that their is a huge amount of pets in some advertisements. But at the same time, the text and the quantity don’t match. (Example: row 823 — Pancho &amp; Tita — Says 2 kittens, but quantity says 12)</p><p>Let’s see if these have significance in predicting the adoption speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*urdnV_DPB8Am6n9gMON_HQ.png" /></figure><p>It looks like that the quantity is not too significant with adoption speed. This is good, because it doesn’t necessarily mean that if there are cats/dogs with kittens/puppies, they have almost about equal chances of being adopted than not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYoOLzZG8OZI5OF1NaVSpw.jpeg" /></figure><p>Now, let’s look at Fee, which is quite an interesting feature. Some pets are fore free, but some require certain fee. Let’s see if this matters much to the adoption speed. The intuition says probably yes. Money is usually really important. Though, by how much?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K317RV9G90xdgCn3MUVxog.png" /></figure><p>Most pets are free and it seems that asking for a fee slightly decreases the chance of adoption. Here, you can still see the pattern that the cats are adopted more than the dogs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zlzzHag5TbPMCOiYNcoHfA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VLKn7iufSNZPqz-NYvtCzg.png" /></figure><ol><li>It is interesting that pets with high fee tend to be adopted quite fast. (This is opposite to our hypothesis. So, the logic can go the other way.) Maybe people prefer to pay for “better” pets: healthy, trained and so on, rather than free, but “random quality” pets.</li><li>Most pets are given for free and fees are usually lower than $100.</li><li>Fees for dogs tend to be higher, though these are rare cases anyways.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eQLv-mesoWFciw4XKE1dJA.png" /></figure><p>It seems that fees and pet quantity have inversely proportional relationship. The less pets, the higher is the fee. I suppose these single pets are better trained and prepared than most others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UaUSuXIdX7Onu0G4l30Q6A.png" /></figure><p>Now, let’s take a look at “State.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XLe3oK9yQyhinqKLUG6dDA.png" /></figure><p>Unfortunately, I am not from Malaysia, and won’t be able to add in some local knowledge that might be useful here. though, I can say that top three states account for ~90 % of ads. Let’s see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r__2cz3ac_SFwdZYL5t1fg.png" /></figure><p>One thing interesting is that top-2 and top-3 states have lower rates of adoption despite the ads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*1-iTJFAXET5Puh9QK_Li4w.png" /></figure><p>Now, let’s take a look at “Rescuer.”</p><p>For Rescuer, we have unique hashes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KcPNHQTHzKQ-pXMPS-lRig.png" /></figure><p>These top 5 rescuers manged to save a lot of pets. Are these people individual or from organizations? We don’t know that, but we can check the adoption speed related to these people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l29vW0kQgAPNI37q5q1KyQ.png" /></figure><p>So, the top 1st rescuer has the best adoption rate, and by far! Though, this is not linear since top 3rd rescuer has the least.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mp66wBgvEDA2skoIdk6FPA.jpeg" /></figure><p>Now, let’s check VideoAmt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PN1w6m-kaBpeSmZ_KPnlw.png" /></figure><p>In most cases there are no videos at all. Sometimes there is one video, more than one video is quite rare. We don’t have videos and considering a huge disbalance in values I’m not sure this variable will be useful.</p><p>How about PhotoAmt?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKpaSamy4pS6oU0J5bdSPQ.png" /></figure><p>Pets can have up to 30 photos. That’s a lot. But I’m not convinced that amount of photoes has any real influence to Adoption Speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RfsmFQb-mp9c4aZIn8-ig.jpeg" /></figure><p>Now, let’s take a look at “Description.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AXTZHB3xYIPTluvLLIhC9A.png" /></figure><p>There are too many similar general words like “cat”. We need to go deeper.</p><p>Let’s use ELI5 library for prediction explanation. I’ll fit a basic vectorizer on desctriptions and build a simple Random Forest model. Then we will look at words which caused certain labels to be predicted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWODlFwCAnJt7rw8--rC-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uek4tYqJPomiDEDCQqBLEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYhTHihPHJhk2ZVGtKjUFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14M6uMZ9XSBMcIQ5BmlGOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_q6BB5_CspaUtqwkv8as2A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9iMdnrOzCCPgsSObCKUQ-w.png" /><figcaption>Code Snippets for Description and ELI5 library info</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AM15rwBfPhKo2RTr7pP1sw.png" /></figure><p>Interestingly pets with short text in ads are adopted quickly. Maybe, with longer descriptions, people think of more problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*5lhLUNJTkpkFhEnBaO141w.png" /></figure><p>Now, let’s look at the last feature, Sentiment.</p><p>We have run each pet profile’s description through Google’s Natural Language API, providing analysis on sentiment and key entities. You may optionally utilize this supplementary information for your pet description analysis. There are some descriptions that the API could not analyze. As such, there are fewer sentiment files than there are rows in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpsjYXP0yeEYIIMFYS4cvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAzpqp2OvmSOGcqfWlqj9A.png" /><figcaption>Sentiment Code Snippet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bbs3YLvSXOy4yNHZvIu1XA.png" /></figure><p>About language, English is the most common language by far, so language feature will hardly help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29YcqRt5aXJ1BWj715kLzw.png" /></figure><p>From the Sentiment, we can see that the lower the magnitude of score is, the faster pets are adopted.</p><p>Now, that concludes the EDA part of this project. Most features seem to have significance in adoption speed. Using decision trees, we will approach the prediction of adoption speed. We can also use CNN for photos and videos if applicable, but videos’ frequencies are much unstable than the photos. In the end, it’s through many different trial and error procedures that one will get to the best modeling possible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=631f9538cde6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install Linux on Chromebook and Enjoy the full potential]]></title>
            <link>https://medium.com/@thom42/install-linux-on-chromebook-and-enjoy-the-full-potential-4312a390f791?source=rss-84435f9f9e77------2</link>
            <guid isPermaLink="false">https://medium.com/p/4312a390f791</guid>
            <category><![CDATA[chromebook]]></category>
            <dc:creator><![CDATA[Soowhan]]></dc:creator>
            <pubDate>Mon, 17 Sep 2018 17:08:13 GMT</pubDate>
            <atom:updated>2018-09-17T17:08:13.948Z</atom:updated>
            <content:encoded><![CDATA[<p>In 42, Silicon Valley Coding School which I am attending at the moment, there are macs already provided for every student. Of course, it’s not for students to carry them around and take them home. That would be too luxurious. We can all use them at our labs. However, for the purpose I mentioned just now which is carrying it around, students want to code outside the lab. I am one of the few people who use Chromebook for coding outside of the lab at 42. In result, I’ve had many students coming up to me and ask how to install Linux on Chromebook, and whether it’s useful, and how does it compare to real Mac. First, let’s dive into installing it right away.</p><p>As I have installed Ubuntu on my Chromebook, in this tutorial, I will show you guys how to install Ubuntu using a tool called “Crouton.” It uses the chroot command to run Ubuntu on top of Chrome OS. The Chrome OS is already based on Linux. This is different from dual-booting. Yes, you can switch from Chrome OS to Ubuntu and vice-versa by just pressing control+shift+alt+left(or right). If you are considering about dual-booting, which is a bit slower but more conventional and traditional, then you should look into “ChrUbuntu” instead of “Crouton.”</p><p><strong>STEP 1: TURN ON DEVELOPER MODE</strong></p><p>First, back up your data somewhere such as Google Cloud. Then,</p><ol><li>Press and hold the Esc and Refresh keys together, then while holding, press the Power button. This will reboot your Chromebook into Recovery Mode.</li><li>Recovery Mode will pop up — the screen with the yellow exclamation point — then press Ctrl+D. This will bring up a prompt asking if you want to turn on Developer Mode.</li><li>Press Enter to continue, then give it some time. It’ll pop up with a new screen for a few moments, then reboot and go through the process of enabling Developer Mode. This may take a little while (about 15 minutes or so), and will wipe your local information.</li><li>When it’s done, it will return to the screen with the red exclamation point. Leave it alone until it reboots into Chrome OS.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*jUaYAen1-kWm6j6Us_WoWA.jpeg" /></figure><p><strong>STEP 2: INSTALL CROUTON</strong></p><ol><li>Download Crouton: <a href="https://github.com/dnschneid/crouton">https://github.com/dnschneid/crouton</a></li><li>From anywhere on your Chromebook, Press Ctrl+Alt+T to bring up a terminal on your Chromebook.</li><li>At the Terminal, run the following command to enter a Ubuntu shell:<br>“Shell”</li><li>Next, run the following command to install Crouton (In this case, Unity. You can do xfce and others too):<br>“sudo sh -e ~/Downloads/crouton -t unity”</li><li>When it’s done it’ll ask you for a username and password for your new Ubuntu installation, so enter them when prompted.</li><li>After it’s finished installing, run the following command to start your new desktop environment:<br>“sudo startunity”</li></ol><p>And there you go! Now you have it. As I said earlier, to move from Chrome OS to Ubuntu and vice-versa, you can use Ctrl+Alt+Shift+Back and Ctrl+Alt+Shift+Forward.</p><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4312a390f791" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping with Beautiful Soup in Python — NFL Football Data Pulling]]></title>
            <link>https://medium.com/@thom42/web-scraping-with-beautiful-soup-in-python-nfl-football-data-pulling-56bfa4a09260?source=rss-84435f9f9e77------2</link>
            <guid isPermaLink="false">https://medium.com/p/56bfa4a09260</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[nfl-data]]></category>
            <category><![CDATA[42-coding-school]]></category>
            <dc:creator><![CDATA[Soowhan]]></dc:creator>
            <pubDate>Sat, 04 Aug 2018 20:47:54 GMT</pubDate>
            <atom:updated>2018-08-04T20:47:54.429Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello everyone. This is my first Medium article, and I am super excited to share my journeys with you. I am currently shaping my coding skills as I am participating in 42 program in the US, and looking forward to work as Data Scientist in the future. Thus, a lot of my writings will be about data analysis and story-telling (and fun! hopefully). Today, I’d like to talk about how to obtain the data before doing all the data analysis. I am currently working on a project to create some of the tutorials for high school students during summer here in 42. This part is basically a tutorial meant for them, but as I was writing them, I figured this can be not only for high school students, but also for anyone who wants / needs data anywhere on web without any coding experience.</p><p>Enough of Intro. Let’s dive in right away.</p><p>First of all, I am going to assume the readers here have no coding experience before, and absolutely have no knowledge in web scraping in general. So, I will touch from 0 to 100. From nothing to pulling out data from NFL 49ers rosters web page. If you are experienced, and want to get into using beautiful soup library, you can either scroll down to the part, or find other useful articles that are suited to your level. Now, let’s start.</p><p><strong>What is Web Scraping?</strong></p><p>According to our friend, Wikipedia, web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites.</p><p><strong>Why would you want to web scrape?</strong></p><p>You might already be thinking numerous reasons for why to web-scrape. Here are some:</p><ol><li>Too much information is on the internet today. You only want to know what you want to know. Get the information organized and analyzable.</li><li>A popular use is to search for online deals like airline tickets, concerts, etc. You can write a script that could scrape a website when ticket sales go online, and use bot to purchase the best tickets.</li><li>You want to analyze the Sports data, but you don’t have a data to analyze. You can do this project and learn to collect data.</li></ol><p><strong>### NOTE:</strong> I know that such power sounds too powerful for some, which is totally natural. There are enormous benefits to web scraping, and that is very why it can also be harmful to others, if used wrong.<br>We will go through some of the reasons and cases why you should not scrape some sites you shouldn’t, at the end of the article.<br>Great Power comes with Great Responsibility.<br>We will use Python to parse (work with) HTML with help of the Beautiful Soup library. Use Python2.<br>[Python is pre-installed in OS X. Open up Terminal and type python — version. You should see your python version is 2.7.x.]</p><p><strong>By the way, before any of this Web-Scraping, what is Web anyways?</strong></p><p>The Web, or World Wide Web, is basically a system of internet servers that support specially formatted documents. The documents are formatted in a markup language called HTML (Hyper Text Markup Language).</p><p><strong>Wait a second… Why are we talking about these languages like HTML? I thought we were web-scraping. Isn’t it just all about the image on your screen and capturing them?</strong></p><p>Yes, that’s also another way to capture and store the data you want from the web. However, that’s not technically called web-scraping, but rather image-capturing. Web-scraping does not directly deal with what’s on your screen. It deals with these “specially formatted documents” that power the web page to be rendered and posted on your screen. How web-scraping happens is looking into these documents (HTML, etc) and “extract” the necessary information.</p><p><strong>For example,</strong><br>When you visit a web page, let’s say Instagram, the web browser (Chrome or Safari) makes a request to a web server (Instagram Company’s Computer). Then the server sends back files to and tells your browser how to render the page. Here are the main file types that are being sent:</p><ol><li>HTML — Contain the main content of the page</li><li>CSS — Add styling to make the page look nicer</li><li>JS — JavaScript files add interactivity to web pages</li><li>Images — image formats, such as JPG and PNG allow web pages to show pictures.</li></ol><p>Out of all these main types, we will focus on one main type: HTML. All these files are required to make a complete page rendered, but when we are scraping, we are mainly interested on HTML.</p><p><strong>Here are some basics about the HTML.</strong></p><p>If you are familiar with HTML and &lt;tags&gt;, skip this part.<br>Let’s create a file called “web_scraping_42.html”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsVgMFIeuWn6KJoVJhEDog.png" /><figcaption>web_scraping_42.html</figcaption></figure><p>Now, on your terminal, type the command: $ open web_scraping_42.html</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*MG_Y93I3uXUpob-cXN5WFQ.png" /><figcaption>This should appear on your Internet Explorer or Safari if Mac</figcaption></figure><p>Congratulations. You just created your first web page. Very simple one though.</p><p><strong>Here are some simply written details of what’s happening in the html above:</strong></p><ol><li>Every brackets &lt;&gt; are called “Tags.” Every tag starts with &lt;***&gt; then ends with &lt;/***&gt;. If either of it is missing, it will error.</li><li>The most basic tag would be &lt;html&gt; tag. This tells the web browser that everything inside of it is HTML. Then, inside of &lt;html&gt; tag, we have &lt;body&gt; tag. Then, inside of &lt;body&gt; tag, we have &lt;h1&gt; tag and &lt;p&gt; tag. &lt;h1&gt; tag is for title heading and &lt;p&gt; tag is for paragraph. Other useful tags include &lt;a&gt; for hyperlinks, &lt;table&gt; for tables, &lt;tr&gt; for table rows, &lt;td&gt; for table columns.</li><li>As you can see, tags are nested, and can go inside other tags. You can think of it as folders inside of folders. Then, in this case, if you want to find the text that says “Welcome to Web Scraping by 42”, you start opening the folders until you reach the very last folder &lt;h1&gt; or folder &lt;p&gt;.</li><li>The relationships between these tags are represented by hierarchical structure with terms “child, parent, and sibling.” <br>1. A child tag is a tag inside another tag. So, in our html above, &lt;h1&gt; and &lt;p&gt; tags are both children of the &lt;body&gt; tag. <br>2. A parent tag is a tag that contains another tag inside. &lt;body&gt; tag is a parent of the &lt;h1&gt; and &lt;p&gt;, and of course, &lt;html&gt; tag is a parent of the &lt;head&gt; tag. A sibling is a tag that is nested inside the same parent as another tag. <br>3. So, inside &lt;body&gt; tag, &lt;h1&gt; and &lt;p&gt; are children tags and also sibling tags, since they are both inside &lt;body&gt;. <br>4. If you haven’t understood this summary of hierarchical structure, try reading it again and googling to find more about examples. This is crucial to understand because the concept will be used for parsing html file.</li><li>Also, HTML tags sometimes come with “id” or “class” attributes. The id attribute specifies a unique id for an HTML tag and the value must be unique within the HTML document. The class attribute is used to define equal styles for HTML tags with the same class. You don’t have to understand them for now. We will only make use of these ids and classes to help us locate the information we want. Wait, that sounds so irresponsible. Yes. It was an irresponsible statement unless I assume that you will study and look on your own about ids and classes if you want to really know about them. This tutorial will only go down deep if it’s necessary for web scraping.</li></ol><p>In this 42 Web_scraping project, you will learn mainly two ways of scraping using beautifulsoup4 and Python. However, there are many other ways and those are for you to explore. <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">This</a> might help you.</p><p>First, you will learn how to “navigate the tree” of HTML hierarchical structures composed of tags. Second, you will learn how to use find() method to “search the tree.”</p><p><strong>Installation</strong></p><p><strong>Step 1:</strong><br>Install necessary modules</p><p>$ easy_install pip<br>$ pip install requests<br>$ pip install beautifulsoup4</p><p><strong>Step 2:</strong><br>Identify the structure of the URL and request</p><p><a href="https://www.49ers.com/team/players-roster/">49ers Roster | San Francisco 49ers - 49ers.com</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*NDuEQhD807OklWVTCYle2g.png" /><figcaption>The official source of the latest 49ers player roster and team information</figcaption></figure><p>This is the 49ers players for Rosters 2018. We can see that there is one big image on the top, then red bar, then white bar with some menus, and all these boxes below. The very bottom box is the main content that has all the statistics.</p><p>Now, let’s use ‘request’ and start collecting in Python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dbXTKQ9H128oWDGubPxr1w.png" /><figcaption>request — get</figcaption></figure><p>If you have successfully “get” the url, it will print out 200, which is the value of response.status_code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*q-urDKmQpi0rHBYfVfh4vg.png" /><figcaption>printing content</figcaption></figure><p>Then, response.content will have the HTML content of the web page.</p><p><strong>View easier with Graphics on Chrome with Inspect mode</strong></p><p>Response.content is also accessible through Inspect Option on Chrome browser. Right click on Chrome web page and click on Inspect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*tQBOqe2pGhh0J6RXCF6XkA.png" /><figcaption>Chrome with Inspect mode</figcaption></figure><p>In this mode, you can see where each line of html (on the right) is referring to on the actual page (on the left). Also, it is easier to navigate through all the &lt;tags&gt;. You can press “right” or “left” on keyboard to ‘open’ or ‘close’ the &lt;tags&gt;. Try playing with it for a minute to get comfortable.</p><p>Now, we will use the Beautiful Soup library to parse and dive into the html to extract what we want.</p><p><strong>Scenario:</strong><br>Let’s say you are a 9th grade high school football player and a 49ers Fan. You are an ambitious one as you came to 42. You want to go to the college that makes the most NFL rosters. You want to predict which college will have the most players of rosters by the time you graduate. So, you decide to collect all the necessary information…</p><p><strong>Quest:</strong><br>Print all the names of College of 49ers Rosters 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*EQY4HpIlqInLnwkGDDPdqQ.png" /><figcaption>print the red box content, names of college</figcaption></figure><p><strong>Instructions</strong></p><p><strong>Step 1:</strong><br>Get URL</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*9uKY_IYh1YfIRVdQftHx4g.png" /></figure><p><strong>Step 2:</strong><br>Open Inspect Mode for easier view on HTML with Graphics</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*tQBOqe2pGhh0J6RXCF6XkA.png" /><figcaption>Inspect mode</figcaption></figure><p><strong>Step 3:</strong><br>Search through HTML on the Inspect Mode to locate the data you want</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5AfU353zAGkoO9qKcYgVEQ.png" /><figcaption>locating the data on Inspect mode</figcaption></figure><p><strong>Step 4:</strong><br>Use the location of the data and BeautifulSoup4 library to parse and extract the data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*WaBe_ZuL6N0WcnpbDKvprA.png" /></figure><p>Now inside ‘nfl’, we have the content, parsed using the python built-in parser for html. We can look at it formatted even nicer with prettify() method on the BeautifulSoup object:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*VcOlbYklZzsbXrni6YhKSQ.png" /><figcaption>on your own, search what prettify() does</figcaption></figure><p><strong>Step 5: </strong><br>Navigate and parse through the HTML. The line that says “&lt;td&gt;Oregon&lt;/td&gt;” is where we want to get to. How to approach?</p><p><strong>Two methods:</strong></p><ol><li><strong>Parsing a page with hierarchical structure (child, parent, and sibling).</strong></li><li><strong>Use find() or find_all().</strong></li></ol><p><strong>### NOTE</strong>: “Oregon” may not be the same as on yours. Actually, the possibility of you having Oregon there is small. It’s because every time you request with above command, you will get a different result. For this instruction, we will use “Oregon”</p><p><strong>First Method:</strong><br><strong>Parsing a page with hierarchical structure (child, parent, and sibling).</strong></p><p>If you don’t remember what child, parent, and sibling hierarchical structure is, then go back to page 3 and review the paragraph. Now, from ‘nfl’ that we parsed, let’s try parsing in to third red-line here, which is &lt;div class=’d3-o-nav__wrap”&gt;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*9mwAfzsL6mkJRk2ivklVnA.png" /><figcaption>locate the names of folders (&lt;tags&gt;)</figcaption></figure><p>First, from &lt;html&gt; tag, which is the parent of all, we have to parse into body tag. We will label this nfl_body. Then, we will parse into &lt;div class=”d3–1-wrap”&gt; tag. We will label this nfl_body_div. Then,we will parse into &lt;header&gt; tag, then to &lt;div&gt; tag that we wanted initially.</p><p>Here’s the python script with explanations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*gQBH-2lVhdy88Qmg4tVutA.png" /></figure><p>Now, using similar techniques, we could parse in to the “Oregon” part. Try for yourself.</p><p><strong>Second Method:</strong><br><strong>Use find() or find_all().</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*6_a2OSL8duGVYOmjBocNMQ.png" /></figure><p>Take a close look at the html. You can find out that the &lt;main&gt; tag is a child tag of &lt;div&gt; tag. We will use this relationship to parse the content and eventually extract “Oregon” by repeating similar parsing technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*dRmst1G50PguLYIarOqPlA.png" /><figcaption>find_all()</figcaption></figure><p>The problem here is that this will print out all divs in your nfl, which is the whole script. So, that’s why you are seeing all of that.</p><p>So, let’s analyze the html to figure out how we will use find() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*eS83G4RYu5vLmePqs1CWwg.png" /></figure><p>The underlined with red are basically the parent (above folders) tags of the data we are looking for, which is in this case “Oregon”.</p><p>Let’s parse from nfl (the whole script) to &lt;main&gt; tag with id=”main-content”. Remember we talked about how we can use the attributes such as id in web-scraping?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*H9-rRW4A55cI35fFh-g9bA.png" /><figcaption>find() for main-content</figcaption></figure><p>Now, you see everything below &lt;main id=”main-content” role=”main”&gt;. It is your turn to try now. Play around and explore the usage of BeautifulSoup4 methods. <a href="http://akul.me/blog/2016/beautifulsoup-cheatsheet/">cheatsheet</a> by Akul.</p><p><strong>Try Yourself</strong></p><p>Try capturing the “Oregon” (or whatever is your “FIRST” player’s College name) and print it out on your terminal.</p><p><strong>Bonus</strong></p><p>Print out all the players’ College names on your terminal.</p><p>If you have done both questions above, then now you are ready to store (save) the data and neatly make it into CSV format, which is widely used format for analysis. Here is how to put “Try Yourself” College answer into your CSV format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GJ0duyy71zBEr_LxFsf_XA.png" /><figcaption>csv</figcaption></figure><p>Now, if you look at your current directory, it will have a file called ‘first_name.csv’.</p><p>Check it out yourself. Now it’s your turn to try.</p><p><strong>Try Yourself</strong></p><p>Store all the players’ College names into a csv file called “college.csv”</p><p><strong>Bonus</strong></p><p>Store all the players’ whole data (Name, Number, Position, Height, Weight, Age, Experience, and College) into a csv file called “roster_table.csv”</p><p>Have you successfully done all of them? Well, if you have, that means that you are ready to go any website and start scraping. If you haven’t, it’s okay. Keep trying. I am going to post up the answers after our program at 42 is done for High school students. Until then, Sorry about that.</p><p>Now, remember in the beginning of this tutorial we talked about how web scraping can harm other people. As you probably already have realized, web scraping can be used 24/7 and data privacy issue is there. Here are three rules from Google for web scraping.</p><p><strong>Scraping Rules</strong></p><ol><li>You should check a website’s Terms and Conditions before you scrape it. Be careful to read the statements about legal use of data. Usually, the data you scrape should not be used for commercial purposes.</li><li>Do not request data from the website too aggressively with your program (also known as spamming), as this may break the website. Make sure your program behaves in a reasonable manner (i.e. acts like a human). One request for one web page per second is good practice.</li><li>The layout of a website may change from time to time, so make sure to revisit the site and rewrite your code as needed.</li></ol><p>Make sure you are not violating the terms!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56bfa4a09260" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>