<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexander Beat on Medium]]></title>
        <description><![CDATA[Stories by Alexander Beat on Medium]]></description>
        <link>https://medium.com/@alexanderbeat?source=rss-1b0f832e6e57------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WMuH9kSoTvwiGq2X8KNsUw.jpeg</url>
            <title>Stories by Alexander Beat on Medium</title>
            <link>https://medium.com/@alexanderbeat?source=rss-1b0f832e6e57------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:24:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexanderbeat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Preparing Github Repositories For Portfolio Part 1: Organizing and Cleaning Repo Files on a Mac.]]></title>
            <link>https://medium.com/@alexanderbeat/preparing-github-repositories-for-portfolio-part-1-organizing-and-cleaning-repo-files-on-a-mac-6060f489f777?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/6060f489f777</guid>
            <category><![CDATA[repositories]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Fri, 23 Oct 2020 22:13:04 GMT</pubDate>
            <atom:updated>2020-10-23T22:13:04.073Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AELBZtHoELEJnElZdy0cJg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@bongkarn-thanyakij-683719?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>bongkarn thanyakij</strong></a> from <a href="https://www.pexels.com/photo/silver-laptop-in-white-wooden-table-3759098/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>I recently graduated from Flatiron’s Data Science Bootcamp and am in the process of cleaning up some repositories and projects on Github and making things more organized for a portfolio. We had some lessons going over cleaning up some of the files, so I chose some of the best parts that I found most helpful to share in this blog post. This first step in the process is more of the organization and handling of files in the repo, and then the next part will deal more with the cleanup of the actual project notebook code, maybe some work on the README.</p><h3>Update your repo name and URL</h3><p>You’ll want a new name for your repository that explains the project better, and while you’re at it, be sure to add a description. Go to the repository on Github and click on settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*c-oJI8BPukJkb43aYN0O2Q.png" /><figcaption>Click on Settings.</figcaption></figure><p>Then you’ll be able to change the repo name on the next page like shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*3nHjHLkvyTSh6zkV_W-T9g.png" /><figcaption>Type a new name and then click “Rename”.</figcaption></figure><h4>Change the description.</h4><p>Back on the main page for the repository, on the far right, you’ll select the gear icon to add a description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*jceEJkvhCk_c6-YYFqm_dQ.png" /><figcaption>Select the gear icon.</figcaption></figure><p>This will show the edit details screen and you can type in your description and add some tags too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*fZaWICbCugBJUvxn5zXt9Q.png" /></figure><h4>Checking and updating remote origin.</h4><p>Changing the repo name will also change it’s URL, so you will need to make sure your local repository on your machine connects back to the one on Github by updating the remote origin on the Terminal. Use these commands in Terminal to do so:</p><p>Check what your current remote repository is.</p><pre>git remote -v</pre><p>Updates your remote origin to the new one.</p><pre>git remote set-url origin &lt;new repo url&gt; </pre><h4>Updating broken links from URL change.</h4><p>Also note that since the URL changed, anything that links to it within your project files will need to be updated. For example, in one of my projects, I had links in the README that were being used to show images, and with the URL changed, those image links will be broken. Below you can see what I mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zx7-cOGglldgcUpGC-aKKQ.png" /><figcaption>Some examples of links used in the README code that could be broken if I were to change my repo name and URL. Be sure to update links like these in your notebooks and in your readme.</figcaption></figure><h4>Updating the Colab button.</h4><p>Another instance where this became problematic was when using Github with Colab. If you’re using Colab with Github, and you saved your Colab Notebook to Github with the option of having that “Open in Colab” button at the top of your project notebook, then that link will need to be updated as well. See the image below to know what button I’m talking about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1myIpVVNbY7Mb0PUoAAcOQ.png" /></figure><p>There’s a couple ways you could go about manually changing this though I don’t find them to be very intuitive. You can open the notebook editing screen in Github and directly edit the link to direct from your new repo URL like in this image below. You’ll find the link code towards the top of the edit window code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UvjYX_r5BYsSDO8rSxjIJg.png" /><figcaption>The link code for the “Open in Colab” button.</figcaption></figure><p>Or, another way you could fix this, is by saving the Colab notebook to Github again with the button option selected like seen here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*xBT6DDhDrvNYmjnK5pZEcw.png" /><figcaption>The file menu for your notebook in Colab. Save a copy in Github.</figcaption></figure><p>The next window shows your options where you can select the checkbox to include the link to Colab (that same button at the top of your notebook I showed you above).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*sFLpxU-YkWReDPY3u_2f4Q.png" /></figure><p>Both of these solutions are not very elegant but got the job done. If you have a much smoother or better way of handling this Colab/Github button disconnect then I would love to hear it and it would help me out a ton.</p><h3>.gitignore file.</h3><p>Something else that will help to clean up your repo is to create a .gitignore file. We had a lesson in our post grad work about cleaning up repositories and I found this one to be really helpful and had no clue what it even was at first until researching more about it. Here’s a screenshot of the example repository and you can see that there’s no .gitignore file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*gzlPskU36at6pLkakAsWqw.png" /></figure><p>This file will help by telling Github not to track certain files you may not want to include in your commits. Some examples of files to include in the .gitignore file might be like system files, caches that could cause problems when the repo is cloned to another user’s computer, or private data you don’t want to be committed to Github and available for others to clone from your repo, or large datasets/directories that you don’t want being uploaded to Github. Github will send a warning if your push is larger than 50MB, and will not be allowed if the push is larger than 100MB. You don’t necessarily need to list your data in the .gitignore file if you already know that you have it saved in a different spot and not within your local repository. But if you have your data saved within the local repo and don’t want to accidentally add it to your commits and push to Github, then listing it in your .gitignore file could be useful for you. It depends on how you keep your data organized and where you like to store it.</p><p>There are a lot of different types of .gitignore templates depending on what type of technologies or languages you’re coding with.</p><p><a href="https://github.com/github/gitignore/blob/master/Python.gitignore">This is the template from Github that we used in our lesson for Python notebooks</a></p><p><a href="https://github.com/github/gitignore">And a link to all of the other templates that Github provides.</a></p><p>Here is the command to create a .gitignore using Terminal:</p><pre>touch .gitignore</pre><p>This will open the file in your text editor once you’ve created it.</p><pre>open .gitignore</pre><p>Copy the code from the Github template and paste it into your .gitignore file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*F5LjPG4NSLPhUl7keyEBYg.png" /><figcaption>Example of what the Github .gitignore template page looks like. Copy all of that code.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*kyLx1PVwVgGDOx6tufUVmg.png" /><figcaption>Paste it into your open .gitignore file that you created.</figcaption></figure><p>I’m on a Mac so there is a file that you can add to the gitignore labeled “.DS_Store” that can be removed to declutter the repo. It’s a file that Macs use with Finder to access all your different folders and contains types of metadata and information on system files. So it can be beneficial to not include these on a public Github page. Add it to your .gitignore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*GJI_gbWH9uyzc0PeLdHCDw.png" /></figure><p>Remove any other files you don’t need in the repo. You can manually do this from your finder window, or in Terminal using the command:</p><pre>git rm &lt;filename&gt;</pre><p>Some files that could be removed could be the dataset like I mentioned previously, but make sure it’s backed up elsewhere. Also the .DS_Store file and the .ipynbcheckpoints folder can be removed from the repo.</p><p>In this example we are working with, there are random images just hanging in the repo. We can move the random images into a folder to clean it up a bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*gzlPskU36at6pLkakAsWqw.png" /></figure><p>You can make an image folder to contain those and do this manually in your Finder window, or in Terminal using the command</p><pre>mkdir images</pre><p>Then move images into the image folder manually in Finder or using Terminal command:</p><pre>git mv &lt;oldlocation/filename&gt; &lt;newlocation/filename&gt;</pre><p>Here’s an example of that command to move the file.</p><pre>git mv for-sale-header.jpg images/for-sale-header.jpg</pre><p>You may also want to consider renaming your files like your notebook to more descriptive names as well.</p><p>Afterwards, once the changes are pushed, the example repo has a much cleaner look to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*nHaOE3IuV3iRUsIIwKv4Mg.png" /></figure><p>I hope this helps you get some ideas for how to clean up your repositories a bit and create better looking work for portfolio. I’ll start going into detail on cleaning up notebooks and readme files in the next post. If you have some advice or additional insight related to this, I’d love to hear from you on what other steps you used to clean things up or reorganize. Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6060f489f777" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Satellite image classification with a convolutional neural network.]]></title>
            <link>https://medium.com/analytics-vidhya/satellite-image-classification-with-a-convolutional-neural-network-4ed4fbd05008?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ed4fbd05008</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Tue, 25 Aug 2020 20:09:07 GMT</pubDate>
            <atom:updated>2020-09-17T19:33:52.412Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T10Stykr8U8qE2FpeBNxZw.jpeg" /><figcaption><a href="https://www.pexels.com/photo/white-and-black-satellite-586063/">SpaceX</a></figcaption></figure><p>My latest project at Flatiron was to use neural networks to classify satellite image tiles. I chose to use a convolutional neural network (CNN) and create a dataset of webscraped images to train the model with. This will just be a quick rundown of what went into the project with additional links to my articles to more of the technical parts. This way, it can help to familiarize you with the topics or help to share more about my work with those who have similar interests in computer vision and machine learning.</p><p>I chose to use a CNN because I read some school lessons on computer vision about how a CNN has advantages with image classification. A CNN uses pooling layers that filter through patches of the image pixels, finding common patterns, which develop into more complex patterns in order to help determine image class. I chose to work on a computer vision project with satellite images because there are possible use cases for solutions on Earth as well as use cases on other planets. I’ve read articles about organizations looking at different geological patterns on the Mars surface in search of the possible presence of water or perhaps its pre-existence on the planet. This led me to try and build the model to recognize river delta patterns here on Earth, with the next step being to train the model and locate delta patterns on Mars. The model could also eventually be useful for looking at changes to river deltas on Earth, for possible use in agriculture, climate change or even real estate. For now, the project is ongoing as of my writing this blog post, with training and testing performed on the Earth images. The Mars images will be the next part I’ll begin after graduation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*PwtsBZghvkGZlYluG-GAhw.jpeg" /><figcaption>A land image tile.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*LLtDunkMZ7LnYDDMpYcUlQ.jpeg" /><figcaption>A river delta image tile.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ht2JKHZpNDURyZ2wUwdFqg.jpeg" /><figcaption>A Mars delta image tile.</figcaption></figure><p>In order to obtain the images for a dataset, I looked into some different API’s and webscraping with Beautiful Soup. Afterwards, I decided to use Selenium to scrape some images from an image search on Google. This method was able to scroll through the page interactively, which was necessary in order to have access to all of the images. I wrote a separate article about that process <a href="https://medium.com/analytics-vidhya/a-simple-selenium-image-scrape-from-an-interactive-google-image-search-on-mac-45d403e60d9a">here</a>. This method was useful as a starting point, in order to go through the process of building the dataset, creating the model, training, testing and just getting everything to work. The disadvantage was that there were a lot of images that were not clean, contained pieces of text or other image artifacts and overall led to less accurate results. There are some example images below so you can see what I mean. I do not claim copyright to any of the used images as they were used for an educational project for school and will remove them if anyone objects to their display on my article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*yrMNamLLYGICpoaJE-1HzA.png" /><figcaption>Notice the different text and labels on the images that will not help with training.</figcaption></figure><p>For the next part of the project I plan to clean up the current dataset, and also build another dataset from much more reliable images obtained from sources like <a href="https://api.nasa.gov/">NASA API’s</a> or <a href="https://earthengine.google.com/">Earth Engine</a>. If any readers have some other similar suggestions that they’ve found useful as a satellite image resource, I’d be happy to check it out and feel free to leave a note in the comments.</p><p>With a scraped dataset as my starting point, I set out to overcome the next challenge which was my outdated ’09 Macbook Pro. It only updates up to El Capitan, and is not able to run the necessary latest versions of Keras, the packages used for neural networks and my CNN. So I moved the project to a cloud-based Google Colab notebook and proceeded to run the rest of the model training there. I have a separate article <a href="https://medium.com/analytics-vidhya/a-quick-workflow-for-google-colab-github-and-jupyter-notebooks-on-mac-ff5d004e01f">here</a> that helps to detail the process I used to connect my Colab notebooks with Github and keep them synced with my local Jupyter notebooks. With everything correctly synced I continued to run different models and compared their results.</p><p>The current dataset is smaller, with 1,600+ images total between the training and test sets. I used image generators to perform image augmentation and expand the dataset up to 2,200+ images. One of the better models used was the CNN with an SGD optimizer that used dropout layers. If you’d like a little more detail on the process of building the CNN model, I have a separate article <a href="https://medium.com/swlh/a-simple-convolutional-neural-network-summary-for-binary-image-classification-with-keras-d58caf2f84a4">here</a>. Below is an image of the layers included in the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*uBb3LjyUG9yDR4YpdJnoyQ.png" /></figure><p>This classification report, confusion matrix and following plots will show you some of the stats on accuracy, recall, precision and loss that the model had.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*XXa01SGN0jzGbZwgJgNcIQ.png" /><figcaption>The model classification report. Class 0 = land, class 1 = delta.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*pzKHBMSslhxeY4pciU1Z7Q.png" /><figcaption>Confusion matrix of the classification.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*QGokldfqUBCEvGiPvwrHtA.png" /><figcaption>Accuracy and loss plots for training and validation data.</figcaption></figure><p>Overall, I’m glad that the model skeleton is built and is able to run and process the images, but the work is not done yet as I plan to obtain more images and move into the second phase of this project with classifying the Mars images. I’ll be sure to create another article with details on that part of the project upon completion. If this sort of topic interests you, you can find the current code for this project on my <a href="https://github.com/alexanderbeat/riverdelta-satellite-image-classification">Github here</a>, with more updates to come. Thanks for reading and let me know if you have similar ideas and would like to share your findings. Follow me <a href="https://medium.com/@alexanderjohnnybeat">here on Medium</a> or you can connect with me on <a href="https://twitter.com/alexanderjbeat">Twitter</a>, <a href="https://www.linkedin.com/in/alexanderjohnnybeat/">LinkedIn</a>, or my <a href="https://github.com/alexanderbeat">Github page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ed4fbd05008" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/satellite-image-classification-with-a-convolutional-neural-network-4ed4fbd05008">Satellite image classification with a convolutional neural network.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A simple Selenium image scrape from an interactive Google Image Search on Mac.]]></title>
            <link>https://medium.com/analytics-vidhya/a-simple-selenium-image-scrape-from-an-interactive-google-image-search-on-mac-45d403e60d9a?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/45d403e60d9a</guid>
            <category><![CDATA[google-images]]></category>
            <category><![CDATA[chromedriver]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[image-scraping]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Tue, 25 Aug 2020 13:11:12 GMT</pubDate>
            <atom:updated>2020-09-17T19:35:27.163Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WhGAx4PDQmTDWhN2QCrIPg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@darshan394?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Deva Darshan</strong></a> from <a href="https://www.pexels.com/photo/aerial-photo-of-gray-concrete-road-938582/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>I recently came across a challenge while working on my final capstone project to graduate from Flatiron’s data science program. For the capstone we were required to compile our own dataset. My project was an image classification model so I chose to scrape some images. My first attempts at using Beautiful Soup to scrape images from Google Image Search were faulty because I could not get the scrape to scroll down the page and it would only return a small set of ten images, which was far less than what I needed to train the model. Other sites recommended I look into Selenium. Not being familiar with it, and how it worked, I found myself hitting some obstacles while trying to get Selenium to work for what I needed. Here’s a little rundown of the steps I took in order to scrape the images using Selenium and hopefully it will give you some solutions as well. This is meant to be a simple way to go about it, though there are many other use cases and ways to approach this situation and mine happens to be a very “bare-bones” method. Here is my github link the the image scraper code notebook I used for my satellite image classification project: <a href="https://github.com/alexanderbeat/riverdelta-satellite-image-classification">https://github.com/alexanderbeat/riverdelta-satellite-image-classification</a></p><h3>1. Getting ChromeDriver.</h3><p>The first step will be to get your version of Chrome that you are currently running. Go to Chrome settings, then click “About Chrome” at the bottom of the sidebar. It’ll display your current version. Proceed to the <a href="https://chromedriver.chromium.org/downloads">ChromeDriver</a> link to download, which is needed to run with Selenium. It’ll tell you which version to download based on your current Chrome version. Unzip that to get the chromedriver.exe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*U5hmA5PAhjzfhUKEOCQN2g.png" /><figcaption>Click the About Chrome button in the Chrome settings sidebar.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*lS9D9ZFIwdzLNku6iFTMHQ.png" /><figcaption>The different versions of Chromedriver to download based on your Chrome browser version.</figcaption></figure><h3>2. Show hidden files — usr/local/bin</h3><p>With ChromeDriver downloaded, you will need to place it in a hidden folder. The typical way to show hidden folders in your Finder window is typing Cmd + Shift + . (dot). I’m running this on an older ’09 Mac with El Capitan so that shortcut won’t work and the Terminal will have to be used. Open Terminal and type “defaults write com.apple.Finder AppleShowAllFiles YES” and then press Enter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*srmQALSWIY17N026CHjPpg.png" /><figcaption>How the command looks in the Terminal window.</figcaption></figure><p>This should allow hidden files to appear in your finder window. If they do not appear, relaunch Finder and then they should show up. In Finder, if you click on your startup drive, there will be a hidden folder called “usr”. Go to usr/local/bin and place ChromeDriver in there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*1rLSM7yr_1cqvZsZEsf_RQ.png" /><figcaption>The hidden usr folder in your startup drive. Your startup drive may have a different name.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*S9wpb5BU2OaueZHZJw_bdQ.png" /><figcaption>local folder contained within usr.</figcaption></figure><p>Afterwards, you can hide the hidden files again by typing “defaults write com.apple.Finder AppleShowAllFiles NO” into Terminal and pressing Enter. Hidden files should no longer appear in Finder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*NrLKWgUVdfgtmx-LpFh-Aw.png" /><figcaption>The command to hide the hidden files shown in Terminal.</figcaption></figure><h3>3. Use Selenium in your notebook.</h3><h4>Set up and import.</h4><p>Now in a Jupyter notebook, you can pip install selenium, and import webdriver, then instantiate it.</p><pre>!pip install selenium</pre><pre><strong>from</strong> <strong>selenium</strong> <strong>import</strong> webdriver<br><strong>import</strong> <strong>time</strong></pre><pre><em>#instantiate the driver<br></em>driver = webdriver.Chrome(‘chromedriver’)</pre><h4>Set up your URL.</h4><p>Then type in the url you want to scrape. This will open a Chrome window of your URL.</p><pre>url = https://www.your_url.com</pre><pre>driver.get(url)</pre><h4>Scroll down the page.</h4><p>Then you’ll want to use the following code to have the driver interactively scroll down to the bottom of the page so you can access all of the images. Previously importing time will let you use a sleep count to allow for the page to load. Here are a couple of other links about this topic that I found helpful in getting the scrape to work:</p><p><a href="https://towardsdatascience.com/image-scraping-with-python-a96feda8af2d">Image Scraping with Python</a></p><p><a href="https://github.com/rmei97/shiba_vs_jindo/blob/master/image_scraper.ipynb"><em>https://github.com/rmei97/shiba_vs_jindo/blob/master/image_scraper.ipynb</em></a><em>.</em></p><pre>page_scroll_sleep = 2<br><br>    <em># Get scroll height</em><br>    last_height = driver.execute_script(&quot;return document.body.scrollHeight&quot;)<br><br>    <strong>while</strong> <strong>True</strong>:<br>        <em># Scroll down to bottom</em><br>        driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)<br><br>        <em># Wait to load page</em><br>        time.sleep(page_scroll_sleep)<br><br>        <em># Calculate new scroll height and compare with last scroll height</em><br>        new_height = driver.execute_script(&quot;return document.body.scrollHeight&quot;)<br><br>        <strong>if</strong> new_height == last_height:<br>        <em>#break #insert press load more</em><br>            <strong>try</strong>:<br>                element = driver.find_elements_by_class_name(&#39;mye4qd&#39;) <em>#returns list</em><br>                element[0].click()<br>            <strong>except</strong>:<br>                <strong>break</strong><br>        last_height = new_height</pre><p>The class “mye4qd” is for a button that appears at the bottom of the page that looks like what is shown in this image. The driver code will click this button in order to keep scrolling to the bottom of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*UWjPQbXj0XMcwkCBrn3_3g.png" /><figcaption>The class “mye4qd” of the “Show more results” button shown in Inspect.</figcaption></figure><h4>Get image class objects.</h4><p>Now that you’re scrolled to the bottom of the page and all images are loaded, this will find all of the image classes in the page.</p><pre><em># gets link list of images</em></pre><pre>image_links = driver.find_elements_by_class_name(‘rg_i.Q4LuWd’)</pre><h4>Pull image links from image objects.</h4><p>These loops below will pull the actual image source link from each image class in the “image_links” variable created above. Sometimes the links are a “src” link, and sometimes they are a “data-src” one. It helps to determine by right clicking on one of the images on the search page, and clicking “Inspect” to see what I mean. These loops will gather both types, which you can combine into a single list of you’d like, or pull images from each of them separately.</p><pre>src_links = [image_links[i].get_attribute(‘src’) <strong>for</strong> i <strong>in</strong> range(len(image_links))]</pre><pre>data_src_links = [image_links[i].get_attribute(‘data-src’) <strong>for</strong> i <strong>in</strong> range(len(image_links))]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*VGhsqmQO-xm1wYI06ihbLQ.png" /><figcaption>Right click on the image and click to inspect.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQYN8xBwqbBTtTsrZZbG2A.png" /><figcaption>This will show the “src” link for that specific image.</figcaption></figure><h4>Save images with urllib.</h4><p>The sleep times list with numpy will randomly grab each image from your list of links at different time intervals to help avoid getting blocked by a site if it thinks you’re a bot. Urllib.request will save your images in the current working directory you are using for your notebook. Tqdm is a basic progress bar that will show you the progress as the page iterates, scrapes and saves your images.</p><pre><strong>import</strong> <strong>urllib.request</strong></pre><pre><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong></pre><pre><strong>from</strong> <strong>tqdm</strong> <strong>import</strong> tqdm</pre><pre>sleeps = [1,0.5,1.5,0.7]</pre><p>The f string will number your images based on it’s index position in the list of image links and this loops through the list called “links” which contains all of the “src” and/or “data-src” links you scraped. And finally, “driver.quit()” will close the interactive Chrome window process.</p><pre><em># urllib save images into folder and renames using data_name string</em><br>    <strong>for</strong> i,link <strong>in</strong> enumerate(tqdm(links)):<br><br>        name = data_name+f&#39;<strong>{i}</strong>.jpeg&#39;<br><br>        urllib.request.urlretrieve(link, name)<br>        time.sleep(np.random.choice(sleeps))<br>        <br>    driver.quit()</pre><h3>Conclusion</h3><p>Scraping images can be troublesome if you are using the images in an unethical way, so I wouldn’t recommend this for the purpose of stealing images for any sort of monetary gain or similar activity. I merely am showing this as a way that I was able to obtain a set of images for use in an educational school project as an example. Keep that in mind since a lot of Google images tend to have a copyright on them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45d403e60d9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-simple-selenium-image-scrape-from-an-interactive-google-image-search-on-mac-45d403e60d9a">A simple Selenium image scrape from an interactive Google Image Search on Mac.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A quick workflow for Google Colab, Github and Jupyter notebooks on Mac.]]></title>
            <link>https://medium.com/analytics-vidhya/a-quick-workflow-for-google-colab-github-and-jupyter-notebooks-on-mac-ff5d004e01f?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff5d004e01f</guid>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Mon, 24 Aug 2020 00:32:41 GMT</pubDate>
            <atom:updated>2020-08-26T13:25:04.078Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ZLiRLseJ_QNCEVOqDbBECg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@goumbik?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Lukas</a> from <a href="https://www.pexels.com/photo/code-coding-computer-data-574071/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>I have an older ’09 Macbook Pro and had trouble while trying to run the latest Keras to use for my latest Flatiron school neural network projects. The older Mac only updates to El Capitan, so it’s unable to run the latest versions of Keras in Jupyter notebooks. To get around this, I have been trying to integrate the use of Google Colab with my workflow. Colab is a cloud-based notebook and has the needed updates to run Keras and has come in handy. Trying to learn the small differences between Jupyter notebooks and Colab in order to keep everything flowing smoothly and all my changes synced has been challenging. Here is a little workflow that I’ve found useful in managing my notebooks and Github repositories. It’s not the most elegant but I have found it to be useful and it gets the job done for now, while I continue to learn more about using Colab and all of it’s connections to Github.</p><h3>1. Create a repository in Github.</h3><p>If you already have a repository you want to work with, then move to step two of this process to create or open a Colab notebook. If you need to make a whole new repository to start with, go to Github and create a new one. Click the green “New” button under the repository section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*YNPM2v-GwlOHa7Ve8ZtIJw.png" /><figcaption>Click the green button to create a new repo.</figcaption></figure><p>Add a name for the repository on the next screen, and click the box to initialize a README. You can choose to add a license to the repository, though I am still learning about all of the different types there are and won’t go into detail about that in this article. Here’s a good link I found about the differences between some of them for you to check out. <a href="https://www.fastcompany.com/3014553/what-coders-should-know-about-copyright-licensing">https://www.fastcompany.com/3014553/what-coders-should-know-about-copyright-licensing</a>. Then click “create repository”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*wln7u64g1BoKmzIahr8MxA.png" /></figure><p>The next screen will be a sort of blank slate repository which shows your README file which you can leave alone for now and head over to Google Drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*evfTMCJuhAKD0_TfkLBCyQ.png" /></figure><h3>2. Create or open a Colab notebook in Google Drive.</h3><p>In Google Drive, click the “New” button, go to “More” and select “Google Colaboratory”. If Colab doesn’t show up in this drop down, you should be able to find it by clicking on “Connect more apps”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*SRkbmBMEkGX3zSebQnqv5Q.png" /><figcaption>Click New.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*BP0_uokf5Buh75RQV6Yl5w.png" /><figcaption>Go to More.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*eMT_St8stdvVnQ_HZU5DrA.png" /><figcaption>Click Google Colaboratory.</figcaption></figure><p>A new Colab notebook tab will open. This will serve as your new notebook to work with and eventually save to your Github repo. If you already had an existing repo and notebook and would like to open it in Colab, on the top menu bar click “File”, “Open notebook”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*SvAZ9RJFgdGN4nS6Q0eOAQ.png" /><figcaption>Click Open notebook.</figcaption></figure><p>A new pop up will appear. Click “Github” on the top. You may need to enter your Github credentials here to connect to Colab if you haven’t already done so in the past. If it’s connected you should see a screen like this, showing your Github username in the search bar. Make sure you have the check the box “Include private repos” in case your existing repo is not public. A dropdown menu of your repositories will be underneath and you can select the one you created. Once selected, if you had an existing notebook on the Github repository it should show up underneath and you can click on that to open it in a new Colab tab and continue to work on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*g_p4ACHoSYMa4tTJLjHbsg.png" /><figcaption>The existing notebook shows up underneath the dropdown.</figcaption></figure><h3>3. Save notebook to Github.</h3><p>Make your changes and edits to your Colab notebook. Feel free to give it a title on the top bar. Next, to save it to your repository, go to the File menu. Click “Save a copy in Github”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*q7BpNv5_0ZBts2AHL_r6fQ.png" /></figure><p>A pop up will appear. Select your repo from the dropdown list, and add a commit message for the push. Check the box “Include a link to Colaboratory”, which will add a shortcut button to the top of your notebook for quick access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*0rdrJZ-kd5w-u9ADmZlfIA.png" /></figure><p>Click “OK” and a new tab will open to your notebook commit on Github. You’ll see it has the “Open in Colab” shortcut button at the top of the notebook. Now you can click on that whenever you want to quickly open it back up in Colab and make changes. Then just repeat the save process to push back to Github. You also have the option to open the Colab notebook directly from your Google Drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xy1YsFZ6mQhRky94vFWMiA.png" /></figure><h4>Side note:</h4><p>If the Colab notebook is saved to Github, and then reopened in Colab using the shortcut button, it will not autosave changes and will need to be saved to Github from the Colab File menu in order to push changes to Github like in step three above. If you try to click save from the File menu, you’ll get a pop up box like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*u42l9HV6_gahfEgU4lys3A.png" /></figure><p>If you prefer to just have the Colab notebook autosave, then you can just work on that Colab notebook independently by opening it up from Google Drive instead of opening it from Github with the shortcut button.</p><p>No matter whether you are opening the file from Google Drive or from Github, you should still save it to Github same as shown before in step three in order to push your changes to the Github repository.</p><h3>4. Sync with local repo on desktop</h3><p>Once the notebook is saved on Github, if you’d like to also back it up on your local machine or work on it locally, use Terminal or <a href="https://desktop.github.com/">Github Desktop</a> and clone it to your computer. I can help you with that if you need to know how to clone it. Just leave a comment, otherwise I can make a separate story about it.</p><p>If changes to the notebook are made from your local machine, you can push those like normal back to Github. Want to open it back up in Colab? Go to the notebook on the Github repo and click the Colab button at the top of the notebook. It’ll open in a new Colab tab, and you’ll see all of the new changes you just made previously on your local machine.</p><p>There are definitely more elegant ways of doing this, but this is just a quick and easy way that I’ve found to be useful while coming across obstacles like not being able to run Keras in Jupyter notebooks on my local machine and Colab has proven to be a nice workaround without having to drop money on some new laptop during these difficult economic times. Hope this helps some others out there who are just getting familiar with Github, Colab and Jupyter.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff5d004e01f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-quick-workflow-for-google-colab-github-and-jupyter-notebooks-on-mac-ff5d004e01f">A quick workflow for Google Colab, Github and Jupyter notebooks on Mac.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Simple Convolutional Neural Network Summary for Binary Image Classification With Keras.]]></title>
            <link>https://medium.com/swlh/a-simple-convolutional-neural-network-summary-for-binary-image-classification-with-keras-d58caf2f84a4?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/d58caf2f84a4</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Wed, 08 Jul 2020 20:38:12 GMT</pubDate>
            <atom:updated>2020-08-03T00:43:24.833Z</atom:updated>
            <content:encoded><![CDATA[<p>Convolutional neural networks (CNN’s) are the main deep learning tool to use for image processing. I recently used a CNN for my latest student project here at Flatiron and got to have a look at how they work and how they differ from dense neural networks, in addition to how they perform better when working with images and python. In my project, I was able to classify patient x-ray images to determine whether they had pneumonia or not. There are also many other uses for image processing in the medical field and in other fields of work and study. Next, we’ll try to show the simplest, most basic breakdown of some of these steps so that you can get on your way to building a CNN for image classification with Keras.</p><p>Keras makes it very simple to build a neural network and building a CNN will seem pretty familiar compared to that. Everything is pretty similar in how the flow works. There are many more options and paths to take when building the model, along with parameters to tune, but this article is just to show a basic outline of the model. Instantiate the sequential model, add your layers, flatten, output, compile and fit/train. Done. Here are the main imports you’ll need, but know that there will be several others depending on your data prep, the problem you are trying to solve, and what your output will be for the model.</p><pre>import keras<br>from keras import layers<br>from keras import models<br>from keras import optimizers<br>from keras.models import Sequential</pre><p>First, instantiate a sequential model, then add the convolutional layer with this line of code. CNN’s work by scanning over the pixels of an image in small blocks using a filter. This is how the network begins to find specific patterns in the pixels, which then build into more complex patterns as the model trains. The main parameters you’ll want to focus on for this layer will be specifying the number of filters you want to use, the size of the filter and the image input shape. The layer accepts image input shape as three dimensional, so if you have an RGB image, it’ll be three dimensions, but if you have a grayscale image it’ll have a one in that place. See what I mean in the block below. You’ll also want to choose the type of activation function for each of these convolutional layers. In this case for binary classification, relu is very popular and works well since it outputs activations only if they go above 0, so it’s either one or the other in this case.</p><pre># Initialising the CNN</pre><pre>model = Sequential()</pre><pre># Create convolutional layer. There are 3 dimensions for input shape</pre><pre>model.add(layers.Conv2D(32, (3, 3), activation = &#39;relu&#39;, input_shape=(64 ,64, 3)))</pre><p>Your next step will be to add a pooling layer just after your convolutional layer. This is what gives your model power over a dense neural network and better speeds with processing. Maxpooling works well as an example from my model, where it takes the highest pixel value from each different group of pixels as the filter slides across the image. This basically downsizes your image to help the model learn the initial patterns in the image. Those patterns will be built into more complex ones as more filters and layers are added. The main parameters you’ll focus on for maxpooling are pool size and stride. Pool size works very similar to filter size in the convolutional layer which is used to find the patterns, but here, the pooling layer will drastically downsize your image, creating less parameters and less work for your system. Stride is the number of pixels the filter shifts across your image matrix. In Keras, if stride is not specified, it defaults to whatever is set for the pool size. So in the code below, you’ll see a tuple as the parameter and it’s being used for both pool size and stride. Here’s a link to the Keras information so you can read what I mean. <a href="https://keras.io/api/layers/pooling_layers/max_pooling2d/">https://keras.io/api/layers/pooling_layers/max_pooling2d/</a></p><pre># Pooling layer</pre><pre>model.add(layers.MaxPooling2D((2, 2)))</pre><p>You’ll find that a lot of models will repeat this alternating step of adding a convolutional layer and then a pooling layer so you’ll have more hidden layers to train with, and achieve better accuracy. A good idea when adding more layers will be to raise the number of filter parameters used as you add more layers so the model will learn more complex patterns in the images. See what I mean in this set of layers below.</p><pre># Convolutional layer<br>model.add(layers.Conv2D(32, (3, 3), activation = &#39;relu&#39;, input_shape = (64 ,64,  3)))</pre><pre># Pooling layer<br>model.add(layers.MaxPooling2D((2, 2)))</pre><pre># Adding a second convolutional layer with 64 filters<br>model.add(layers.Conv2D(64, (3, 3), activation = &#39;relu&#39;, input_shape = (64 ,64,  3)))</pre><pre># Second pooling layer<br>model.add(layers.MaxPooling2D((2, 2)))</pre><pre># Adding a third convolutional layer with 128 filters<br>model.add(layers.Conv2D(128, (3, 3), activation = &#39;relu&#39;, input_shape = (64 ,64,  3)))</pre><pre># Third pooling layer<br>model.add(layers.MaxPooling2D((2, 2)))</pre><p>The next step will be to flatten the layers and add an output layer. The layers are flattened into a 1-D array for use in the output layers. The final layers will be dense layers, as opposed to the previous convolutional ones we used. In this example the activation function is set to sigmoid on the final output layer. This function is most common for binary classification since it outputs activation values between 0 and 1.</p><pre># Flattening<br>model.add(layers.Flatten())</pre><pre># Full connection<br>model.add(layers.Dense(units = 512, activation = &#39;relu&#39;))</pre><pre>model.add(layers.Dense(units = 1, activation = &#39;sigmoid&#39;))</pre><p>Once your layers are set, you’ll compile the model. The optimizer used here is adam and is a type of gradient descent to help increase performance and speed up convergence of the training and validation data as your model trains. With binary classification tasks, set your loss function to binary cross entropy. This will help to calculate the loss for each sample as your model makes predictions. You will also want to state the metric for your model to use. In this case, I used accuracy of whether each image was correctly classified.</p><pre># Compiling the CNN</pre><pre>model.compile(loss = &#39;binary_crossentropy&#39;,<br>              optimizer = &#39;adam&#39;,<br>              metrics = [&#39;acc&#39;])</pre><p>Next will be fitting the model, and there are several main factors you’ll want to look at for this. First, you’ll enter in your training and label data. If you used an image generator, then that should return a tuple object of the train data and labels, which you can also enter in as a single parameter like shown below. Then, state the number of epochs to run the model for, and enter in the validation set of data for the model to compare accuracy and loss data to.</p><p>If you have a huge amount of data, it may be good to also include early stopping so your model doesn’t train for too long. This will also help with overfitting. Use the import below and set up your early stopping to monitor the validation loss, and use min_delta and patience parameters. For min_delta, 0 is very common. If you are unsure what number to use, it’s also a good idea to run your model for a few epochs to see what the validation loss values are coming out as, and then basing your min_delta value on that. I’ve seen some models using other values like 0.1, or others like 0.01, 0.001, etc. This will tell your model to look for a difference in change of validation loss based on that decimal amount, with a patience of however many epochs you state in the patience parameter. So if you state a patience of 3, and your validation loss doesn’t change much in 3 epochs, it will stop your model training early. The callbacks parameter in your model fitting will allow you to use the early stopping.</p><pre>from keras.callbacks import EarlyStopping<br></pre><pre># Define the callbacks for early stopping of model based on val loss change.</pre><pre>early_stopping = [EarlyStopping(monitor = &#39;val_loss&#39;, min_delta =  0.01, patience = 3)]<br></pre><pre># Fitting the CNN</pre><pre>history = model.fit(training_set, steps_per_epoch = 500, epochs = 10, callbacks = early_stopping, validation_data = val_set)</pre><p>Once the model has been trained, you can get your scores from test data predictions. Some ways to see how your model performs would be to use evaluate and enter in your test data and labels. This returns your test loss and test accuracy. Another way that I used would be to make a classification report by comparing the test labels compared to model predictions.</p><pre># Prints out test loss and accuracy<br>results_test = model.evaluate(X_test_data, y_test_labels)<br>print(results_test)</pre><pre># Creates a classification report showing your accuracy, recall, f1.<br>import sklearn.metrics as metrics</pre><pre>y_preds = model.predict_classes(X_test_data).flatten()<br>metrics.classification_report(y_test_labels, y_pred_labels)</pre><p>From there, some other possible options for visualizing that you could look into might be to create a confusion matrix or make plots of the loss and accuracy for training and validation data to see how your model performed across the different epochs.</p><p>This was more of a basic outline to build a CNN, but there are plenty of other hyperparameters you could look into and ways to modify the model to perform better and prevent overfitting. Along with that, some of the techniques described in this little summary could also be explored much more in depth as to how they work and ways you could approach the situation. For now though, hopefully this quick overview will give you a little introduction to how building a CNN is done and start you on your way to making one yourself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d58caf2f84a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-simple-convolutional-neural-network-summary-for-binary-image-classification-with-keras-d58caf2f84a4">A Simple Convolutional Neural Network Summary for Binary Image Classification With Keras.</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A simple way to help avoid data leakage in machine learning and predictive modeling.]]></title>
            <link>https://medium.com/@alexanderbeat/a-simple-way-to-help-avoid-data-leakage-in-machine-learning-and-predictive-modeling-ee840be7482a?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee840be7482a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-leak-prevention]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Sat, 23 May 2020 22:57:26 GMT</pubDate>
            <atom:updated>2020-05-23T22:57:26.282Z</atom:updated>
            <content:encoded><![CDATA[<p>Machine learning and the ability to make predictions for use in the real world can be a very helpful tactic in many different areas of knowledge and study, but it doesn’t come without its faults. Such a fault could be when achieving less than adequate results due to data leakage. I came across this interesting topic during my current studies as a data science student and wanted to share my general understanding of the problem. When working in machine learning, there are many different steps and a variety of different options to account for when importing data, cleaning it, developing models and interpreting results. Data leakage can occur at different points during that whole process. Introducing leaky data to your machine learning model will lead to inaccurate and unreliable metrics, overfitting and a lack of ability to handle real-world data.</p><p>It is more beneficial for your problem to develop a machine learning model with a nice, clean dataset as opposed to using a chunk of raw data, but that clean dataset the model was trained on could be a problem when making further predictions. This is due to leaking in the data. In this article, we’re going to mainly focus on data leakage in regards to building predictive models. Data leakage is when your cleaned dataset contains traces of the entire dataset even once you have performed a train-test-split and begin training your model. The model will be fit with the training data but that set still contains traces of the full set so your model is going to have inaccurate results and will most likely overfit or have incredibly high scores, among other problems.</p><p>The main problem we’re looking at here is how the data leakage will lead to your model achieving very high scores, being unable to make reliable predictions and is otherwise useless against real world datasets outside of the one it was trained on. This is an easy tell when testing the validation set and the results are too good to be true. The model performance may seem good but in reality it is not functioning properly based on the fact that those traces of the entire dataset, including the separated test set, exist within the training data fit to the model. There are additional reasons why data leakage is a problem such as when dealing with cyber security where data traces can be reversed, putting your data at risk, in addition to achieving overfit model scores for competition that would not work well in actually solving the problem. That one is very similar to what we are discussing here so keep that in mind if you are planning to submit for competition or host your own competition.</p><p>Your model may be leaking due to the cleaning process and handling of the data. There are several factors that could lead to leaking when beginning to work with your raw data in preparation for model fitting. There can be a high chance of a leak when imputing missing values, removing outliers or scaling. This is because when performing actions such as these, the code takes all of your points into account and will factor each one into the imputing, scaling or removal of outliers. This is how those traces come to be that I mentioned before. Leading up to your model fitting you will most likely perform a train-test split in order to train your model and then check it with the test set after. Since all of that data was cleaned and prepped together, those traces of the raw data will still exist within your test set. The simplest and most effective way of preventing this leak would be to switch up your order of events here. If you were provided a separate test set of data to begin with then you have a good position already, but if not, then it would be in your best interest to split aside a test set before you begin to perform any extensive data cleaning or preparation. So essentially you would end up with your training set, a test set, and then a final blind validation set after the fact. This way none of the preparation will affect your validation data as it was handled separately and the model will be completely blind to it. Switching up these steps will set you up for better success and model prediction power.</p><p>Once you have a separate validation set, and have prepared for cleaning, it’s important to keep in mind that there are still many other factors that could lead your model overfitting or showing inaccurate results. Examples would be the choices of how you normalize, what values you impute, how you remove your outliers, feature correlation and hyperparameter tuning to name a few. All of these major parts of data preparation can be extensive and we can discuss in later articles. For now, just having a basic understanding of data leakage and its negative effect on your model building will help out the build quality drastically and give you a much better head start into the world of building predictive machine learning models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee840be7482a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power Will Help Your P-value in Hypothesis Testing.]]></title>
            <link>https://medium.com/@alexanderbeat/power-will-help-your-p-value-in-hypothesis-testing-e74e3d10bc15?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/e74e3d10bc15</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[power]]></category>
            <category><![CDATA[p-value]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Wed, 11 Mar 2020 19:33:14 GMT</pubDate>
            <atom:updated>2020-03-11T19:33:14.253Z</atom:updated>
            <content:encoded><![CDATA[<p>While going through my studies in data science courses, I’ve become more familiar with statistical and hypothesis testing for experiments, and how using both a p-value and statistical power can help to support your claims. The p-value helps you decide whether to reject or accept your null hypothesis and assume whether your hypothesis holds any value or not. There are different factors that affect p-values such as sample or population size, along with normality of the data and variance, but more often than not, this p-value isn’t enough to go on when determining the significance of your hypothesis test. There are even times when experiments need to maintain a certain p-value as documentation in order to maintain funding for continued research. This can easily lead to groups getting involved in unethical “p-hacking” scenarios where they may use knowledge of larger sample size’s effect on raising p-value to help make an experiments process seem more significant against the hypothesis than it actually is. With instances like this coming up in the hypothesis testing process, it seems that only having a mere p-value as proof of statistical significance is lacking. This is where secondary steps and follow-up should come into play with further evidence such as effect size with Cohen’s D and looking at statistical power. In this article we’re mainly going to focus on explaining power, its relationship to type I and type II errors in hypothesis testing and how it helps add more validity to your claims.</p><p>P-values, in relation to the alpha (confidence level) you set for your experiment, can only take you so far in terms of accepting or rejecting a null hypothesis. For example, say you are working with an alpha of 0.05, a common one in experiments. This is basically like saying you’re fine with rejecting the null hypothesis if your p-value is less than 0.05 or has a 1 in 20 chance of just being a fluke and not actually significant to the sampled data. Power steps in to help back up your claims by taking a look at that p-value and giving you a probability of how sure you can be that your rejection of the null hypothesis is valid in this instance. If your experiment finds that there is a significant effect happening and one actually does exist, then a high power percentage is the probability that that significant effect showing up in your experiment is valid and not just based on random chance. Power is directly related to your alpha, the sample size of data and the effect size. Having all three of these pieces of information will get you your power percentage and there are some functions in Statsmodels, such as TTestIndPower and TTestPower, that will help you calculate this. Power is also related to type I and type II errors on whether you might be making a false-positive or false-negative in your hypothesis claims.</p><p>A type I error is a false positive when you reject the null hypothesis and make the assumption that there is a significant effect happening but there was a small chance for inaccuracy and there is actually no significant effect, only a random occurrence based on the sampling of data. The opposite of that would be a type II error where your results came back and you reject the alternative hypothesis when, in reality, there actually was a significant effect happening within the population or sample. The power probability will help you by giving you a clear percentage on whether you are making one of these errors with rejecting the null or alternative hypothesis. Effect size plays into this too, in terms of how high or low that power percentage will be, but that’s another topic we can get into further detail in the future. I am at the beginnings of understanding this concept, but for now, just know that a high power percentage, combined with a low p-value, good effect size and reasonable sample size will all help in pushing the validity of your experiment and your assumptions of the hypothesis while avoiding any type I errors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e74e3d10bc15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software Engineer or Data Scientist?]]></title>
            <link>https://medium.com/@alexanderbeat/software-engineer-or-data-scientist-3493360685d8?source=rss-1b0f832e6e57------2</link>
            <guid isPermaLink="false">https://medium.com/p/3493360685d8</guid>
            <category><![CDATA[career-change]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alexander Beat]]></dc:creator>
            <pubDate>Thu, 21 Nov 2019 21:12:32 GMT</pubDate>
            <atom:updated>2019-11-21T21:12:32.357Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a question that I’ve been contemplating for several months now, since happening across a Forbes article discussing the use of income share agreements (ISA) as a tuition system for coding bootcamps. Truth be told, I’ve been thinking of this move to tech for a while longer, since finishing my bachelor’s back in 2010. The article more so rekindled my ideas of looking into the tech world and coding as a new career option while I try to dig myself out of the metaphorical mountain towering over America that is student debt. We’ll take a look at some different career aspects, paths and ideas for choosing one title over the other, or perhaps even how they blend together.</p><p>I was originally introduced to coding in general through a couple of computer science courses required by my fine arts major back in the mid 2000’s. We were taught a few different lines of code in C++ and set loose on creating simple programs to solve small mathematics problems. It was seemingly insignificant to me at the time seeing as how my mind tended more towards the artist frame of mind, but interesting nonetheless. Coming from a family of mechanical engineer and drafting backgrounds always helped me to excel in math. The thing is, I never gave it a chance after dropping out of honors calculus my senior year of high school and moving into the art undergrad. The college computer science instructor noticed my aptitude for the logic and math and recommended I switch majors to computer science but I never did make the switch.</p><p>Now, closing in on the start of the ’20s, and almost two months into my being a student at a popular coding bootcamp in the U.S., I find myself looking back on what could have been from that decision. I find it funny how sometimes things can come full circle like that. My initial degree, like many other people in recent times, left me with a load of debt that has not been fun to tackle. After attempting to attend another school since then, with minimal financial backing or aid, I was left with even more debt. Trying to support myself and manage the debt left me with less options on how to get a start in a new career. After reading that Forbes article, learning more about ISA’s and realizing they could be my ticket to getting into a school with minimal up-front costs, I jumped on finding the best contracts and options out there. I was thinking this could be my chance to land a new career, not worry about barely making rent every month and dive back into the coding that I’ve been wanting to learn since my undergrad.</p><p>This brings me to the topic at hand. I previously mentioned that I am now almost two months into my current stint at a coding bootcamp where I am studying data science. Data science as I’ve come to find out, is a very broad range of career paths, just like software engineering, with many different paths to take. Initially led by my desire to get back into tech, and hearing of data science being one of the most sought after career fields had curiosity pushing me to the course. I cannot of course forget to mention the fact that it is an extremely lucrative field for all of those thinking of a mid-life career change. Many top-rated tech companies now look at hiring bootcamp graduates, and with the ISA trend upending tuition traditions in America, it seems like a no-brainer am I right? I chose data science because of its desirability, my interest in machine learning/AI, and the high salary figures. There are paths like data analysts, data scientists, machine learning and algorithm coding amongst others, while all still being under the broad umbrella term of data science. It can be overwhelming for someone who is new to the tech world. Hopefully this will help shed a little more light on the idea for others who may be in similar shoes to me.</p><p>During my first couple of months in the bootcamp, I’ve gotten a taste for data science basics, from learning Python syntax, using variables, functions, if statements and the like. It has similarities to other coding languages but from what I’ve heard Python is much more concise. We’ve also looked at topics like packages, simple linear regression models and the use of visualizations to show data trends for comparison. My artist brain often finds itself lost in the logic of it all, though I do find my conceptual background helps when thinking of the overview and higher level idea of what the code does. By higher level, I mean the overall concept of what the machine code is doing or what the programs are supposed to achieve. Diving into the code and the gritty nature of running code to get errors and realizing you had one missing semicolon or bracket in the wrong place, or remembering the specific package you need to import in order to run a small line of code is where I’m finding my challenges lie. We had the first major project to complete over the past couple of weeks where we were able to finally piece together much of what we had been learning since initial prep work and tech interviews in September and this was a bit of a game changer.</p><p>Leading up to the project, I was getting bogged down by many of the technical aspects and keeping up with the material on linear regressions, calculating R squared and drawing conclusions and predictions from models. But since nearing completion of the project and awaiting review, some of the pieces are starting to fall into place and the data science process is starting to take form in my head. My experience is still extremely lacking but it’s nice when you come across those little moments of understanding. Despite those faint moments of understanding, going through this initial introductory learning phase of data science and crawling through the thick of it was beginning to turn me off to the thought of this career path and my initial choice in fields. I found myself thinking about the greener pastures of software engineering.</p><p>Software engineering is a whole other beast driving the tech world through the creation of all the different amazing software applications out there we’ve come to know and love. The keyword there being creation. My artist mind was being tempted by the thought of familiarity, that I could look at switching into software engineering to find that outlet of creativity I was not sensing in data science. Software engineering and development is just that, creating the working code and programs to make sure things run and behave how we want them to. Following my temptations and teasing the idea of going down this path led me to research more of the different career options that differ between the two tech umbrellas. There were careers like web designer, video game and app development and operating systems software. These were all the things that we use on the daily in every aspect of our lives. I had barely scratched the surface though, and proceeded to discuss the decision at hand with several of my professors to get their opinions on the matter.</p><p>I reached out to several of my instructors about the ideas and decision I was facing on switching the focus of my studies while also posting on chat forums to receive other opinions. This is where things started to get interesting. Firstly, my main instructor was keen on hearing what I wanted to get out of the program and what my interests were as far as coding goes so he directed me to focus on that. Do I enjoy math? Yes, but am I an absolute rocket scientist? Perhaps in another life. For me, like I mentioned before, the math portion of data science had been bogging me down as of late but I do like creating. Additionally, I do like working with conceptual thinking, analyzing and organizing so it would seem data science has some pros and cons. Software engineering involves those higher levels of concept to plan out the phases of the working programs as well. Based on my wants for more creativity and coding, my instructor suggested I stay in data science, take what knowledge I can from it and focus on the machine learning path and creating packages of code as a sort of creative outlet. The packages, from what I understand, are small programs and functions that can be designed and then later installed to help with and run other programs. Very cool. He also mentioned I direct my attention to neural networks which are designed to behave and act like neurons in the brain. I do find myself wanting to learn more about AI and this could go hand in hand with that idea. That being said, those topics are going to be heavily math-focused and would require me to continue in my data science studies. I spoke to another head of my department about this topic and his response was unforgettable.</p><p>This whole discussion thus far was beginning to peak my curiosity more but I was and am still not at a final conclusion. The second instructor I reached out to was quick to the punch with his thoughts on the matter After hearing my ideas of switching to become a developer he dropped his opinions on me to help my decision process. “Data science has a lot of coding.” This was in response to my notions that data science was more analyst work than coding, compared to development. He then went on to say, “…and it’s the sexiest and most lucrative job in our generation.” The response was so concise, to the point and downright hilarious. I couldn’t help but crack up from this moment. I do have to agree with him whole-heartedly though, that this is a highly sought after career field as of late, and in a way, I suppose a data scientist could be considered “sexy.” There’s no doubt too that I am getting in this career field for the sake of having a more lucrative job instead of simply waiting tables the rest of my life, in addition to being able to live more comfortably. So in a way, his short, blunt response was just what I needed to hear.</p><p>I’m not trying to point out that there is no creativity in data science or no math involved in software engineering. I am new to the tech world so I am absorbing and learning all of this and merely trying to explain my observations from what I’ve experienced so far. After speaking with my instructors and with others on the group chats about this topic it’s been good to hear what others had to say. All in all, data science is still about the data, but it can be and is more than that. It is machine learning, communicating through code with a computer or machine in order to process information that would take us years to sort, and to utilize that data. Or it returns the data in the way an analyst can use to interpret and predict the next business move for a company. In software engineering they design and create, but it’s not without its own forms of analysis and hypothesis. There are higher conceptual levels and creativity involved. For me this has been an eye-opening past several months of my life and I imagine that continuing on this path will be even more expanding. Admittedly I am still a little torn about the greener pastures thing.</p><p>For now, I am still in my data science program, struggling my way through the material with a first project review this week. Fingers crossed. Ultimately, I think I will stay in the program, from what we talked about above, and for several other reasons that I didn’t go into today that would stop me from switching to development. I am trying to take each day and each newly learned topic as it comes, building upon that and making my way through this beast of a field called data science. I imagine my game plan will slightly shift throughout my journey but I will plan to pick up more developer coding languages later towards the end of my current program or perhaps afterward, once I land in a new tech career, gain a better handle on my student loan debt mountain and open up some more opportunities for myself.</p><p>If you have any further insights on the matter or just want to discuss your own experiences in data science or software development, feel free to reach out. I will be starting up a video series to document my trials, thoughts and experiences in coding bootcamp on Youtube soon so if you’d like to hear more about specifics in that regard definitely let me know and I will be sure to post updates once that is live. Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3493360685d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>