<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thomas Bacas on Medium]]></title>
        <description><![CDATA[Stories by Thomas Bacas on Medium]]></description>
        <link>https://medium.com/@thomasbacas?source=rss-639dce71b60e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*t1Sh247t23EyQ4Rp</url>
            <title>Stories by Thomas Bacas on Medium</title>
            <link>https://medium.com/@thomasbacas?source=rss-639dce71b60e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:20:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thomasbacas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Finding the middle ground for our next vacation with Django and Python]]></title>
            <link>https://medium.com/analytics-vidhya/finding-the-middle-ground-for-our-next-vacation-with-django-and-python-36158b469471?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/36158b469471</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[quarantine]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[creativity]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 02:22:37 GMT</pubDate>
            <atom:updated>2020-04-25T16:42:43.057Z</atom:updated>
            <content:encoded><![CDATA[<h4>Bringing a friends idea to life:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*wR52vd2G2mqfFORH" /></figure><p>This morning started like any other in the world of COVID-19 quarantine. I had just walked my dog Toby, started on my cup of coffee and I was running my morning queries to refresh a report I do daily for work. After my standard morning meetings I drifted off looking out the window thinking back to a time when simply going outside wasn’t seemingly dangerous. It was then I got a call from my friend Boyd for a standard check in. You see, after college all of my friends moved all over the country. I had one in Spartanburg SC, another in Cincinnati OH, two in Charleston SC and others spread out in New York, San Francisco, Boston. I asked Boyd “What’s your plans when Corona has run its course?”, “We should all get together” he said. But where? Traveling to Charleston seems simple because its where Boyd and another college roommate live, but that’s a far trip for all of us besides them. “Why don’t we just get a cabin or something in the dead middle of everyone so its fair” Boyd told me. What a great idea. Let’s find the middle ground between all of us and just find an Airbnb there…</p><p>We talked briefly about how mathematically we could determine a centroid between varying different points. I told Boyd, after work — I’m going to try this in Python to make things simple. Thus spawned my homebrew Hackathon… a few hours after work and I had stood up a functional webapp that looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4SpFRqkwP6pwL1QYaE3Hw.gif" /><figcaption>Here is a demo of my website called (tentatively) FindMid!</figcaption></figure><p>I leveraged many open source libraries and the very helpful googlemaps api to achieve the above result. It was a fun challenge to get all of the pieces working together, but in the end I was very happy with the result. And yes, before you criticize my terrible UX let me say, I am not a UX designer. I’m a data scientist and avid python programmer with a bit of Django experience. If I had it my way I’d pass this along to someone with much better HTML,CSS and JS skills.</p><p>Lets start with the libraries:</p><pre>from uszipcode import SearchEngine<br>import wikipedia<br>import gmplot</pre><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>import datetime<br>import os</pre><p><strong>uszipcode</strong>: <a href="https://github.com/seanpianka/zipcodes">https://github.com/seanpianka/zipcodes</a></p><p><em>“Zipcodes is a package for Python 3 and Python 2.7 which supports lookup and filtering of zipcode data for the U.S.”</em></p><p>I used <strong>uszipcode</strong> to pull down long/latitudes of my various scatter points. By passing city and state name, I’m able to take the zip-codes generated from the search engine and pull down respective longitude and latitude data.</p><p><strong>wikipedia</strong>: <a href="https://github.com/goldsmith/Wikipedia">https://github.com/goldsmith/Wikipedia</a></p><p><em>“Wikipedia is a Python library that makes it easy to access and parse data from Wikipedia.</em></p><p><em>Search Wikipedia, get article summaries, get data like links and images from a page, and more. Wikipedia wraps the </em><a href="https://www.mediawiki.org/wiki/API"><em>MediaWiki API</em></a><em> so you can focus on using Wikipedia data, not getting it.”</em></p><p>My utilization of the wikipedia api may not be immediately recognizable, but I used it to grab a quick summary of each midpoint town. When a town is generated I pass the city/state data to the wikipedia search to generate a one sentence summary for the website.</p><p><strong>gmplot:</strong> <a href="https://github.com/vgm64/gmplot">https://github.com/vgm64/gmplot</a></p><p><em>“Plotting data on Google Maps, the easy way. A matplotlib-like interface to generate the HTML and javascript to render all the data you’d like on top of Google Maps. Several plotting methods make creating exploratory map views effortless. Here’s a crash course:”</em></p><p>Gmplot did the majority of the heavy lifting, with my experience in matplotlib it made things simple to create and plot the various lines and points.</p><p>I started out my development process by establishing the logic for my center point. I went with a simple centroid formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/0*CZF8LtLMNelGBtvW" /></figure><p>I went with something simple for the sake of time and this seemed to get the job done. I started by populating some random numbers for my lat, long then trying the average distance to the centroid to get a sense of the results. Afterward I plugged it into the backend of my django webapp (I wont be covering the basics of starting a django webapp but I’m happy to point to some good resources)</p><pre>points =[]<br>search = SearchEngine()<br>cities = request.POST.getlist(&#39;feedurl[]&#39;)<br>states = request.POST.getlist(&#39;feedurl[1]&#39;)<br><br>for i in range(len(cities)):<br>   result = search.by_city_and_state(cities[i],states[i])<br>   points.append([result[0].lat,result[0].lng])</pre><pre>final_x = sum([x[0] for x in points])/ len(points)<br>final_y = sum([y[1] for y in points])/ len(points)</pre><p>By utilizing the request.POST.getlist() I would retrieve two populated lists of cities and states that I could use to search via uszipcode SearchEngine() function. The result of this search would contain lat/long and a variety of other data points which can be helpful but I didn’t end up using for this project. The lists would look like this:</p><pre>[[lat1,long1],[lat2,long2],[lat3,long3],...]</pre><p>With this list I can determine our centroid.<strong> GREAT!<em> </em></strong><em>but now what…</em></p><pre>result = search.by_coordinates(final_x,final_y)</pre><p>Lets plug those new data points into the SearchEngine() function again and find out where we ended up. In the demo above — that was Jonesville, NC. I’ve never been to Jonesville, but I’ve been through North Carolina and the state is beautiful. Maybe we’ll end up going there after all? Who knows. Now we have a City and a State with which I can get some information from wikipedia..</p><pre>wikipedia.summary(result[0].city+&#39;, &#39;+result[0].state, sentences=1)</pre><p>Our result from this API call gives us the short blurb under the midpoint announcement on the webpage. With the same above variables it was a simple adjustment to the Airbnb url, changing the date and City/State name to our above variables then passing that back to the django template. <em>Voila, </em>Airbnb data for the mid point in question.</p><p>With those done lets plot our points on GoogleMaps. Important note, Google requires an API key for utilizing gmaps properly. You do need to register a credit card but your API calls are free for your first 100,000 static map calls. For a small fun project for my friends it was no problem, but scaling up to commercialization this would be a consideration.</p><pre>gmap = gmplot.GoogleMapPlotter(final_x,<br>                               final_y, zoom =5, apikey=**yourkey**)</pre><pre>gmap.scatter([x[0] for x in points], [y[1] for y in points], size = 10000, color = &#39;red&#39;, marker=False)<br>gmap.scatter([final_x],[final_y], size = 10000, color = &#39;blue&#39;, marker=False)</pre><p>We instantiate our map centered around our centroid with the two lat/long points, I found that zoom 5 was appropriate for the US map but that is up to your use case. This is also where you would supply your API key.</p><p>With the gmap instantiated I began mapping our Origin/Destination points. Like before, I used list comprehension to pass the lat/longs from our list to generate our Origin points, then I passed our centroid lat/long as single item lists (the scatter function requires an iterable).</p><pre>gmap.plot([x[0] for x in points], [y[1] for y in points])<br>gmap.draw(&#39;findmid/media/map.html&#39;)</pre><p>Now we plot the lines between each origin and our destination. Gmplot does a great job making these lines, its super simple and you can customize a decent amount. The last step… we create our html file.</p><p>All of the pieces are here, let’s get these on a front-end framework. Below is the HTML template that Django calls to populate the page. Since all of the variables are empty until we post data to the frontend I could tag them directly into the HTML with little effect. The exception was the airbnb link, which I put behind a simple if statement.</p><pre>&lt;style&gt;<br>   .feed {padding: 5px 0}<br>&lt;/style&gt;<br>&lt;!-- onsubmit=&quot;return validate(this)&quot; --&gt;<br>&lt;form method=&quot;post&quot;&gt;<br>    {% csrf_token %}<br>&lt;table&gt;<br>&lt;tr&gt;<br>    &lt;td valign=top&gt;<br>        &lt;div id=&quot;newlink&quot;&gt;<br>            &lt;div class=&quot;feed&quot;&gt;<br>               &lt;input type=&quot;text&quot; name=&quot;feedurl[]&quot; size=&quot;50&quot;&gt;<br>               &lt;input type=&quot;text&quot; name=&quot;feedurl[1]&quot; size=&quot;2&quot;&gt;<br>            &lt;/div&gt;<br>            &lt;div class=&quot;feed&quot;&gt;<br>               &lt;input type=&quot;text&quot; name=&quot;feedurl[]&quot; size=&quot;50&quot;&gt;<br>               &lt;input type=&quot;text&quot; name=&quot;feedurl[1]&quot; size=&quot;2&quot;&gt;<br>            &lt;/div&gt;<br>        &lt;/div&gt;<br>    &lt;/td&gt;<br>&lt;/tr&gt;</pre><pre>&lt;/table&gt;<br>    &lt;p&gt;<br>        &lt;br&gt;<br>        &lt;input type=&quot;submit&quot; name=&quot;submit1&quot;&gt;<br>    &lt;/p&gt;<br>&lt;p id=&quot;addnew&quot;&gt;<br>    &lt;a href=&quot;javascript:add_feed()&quot;&gt;Add New &lt;/a&gt;<br>&lt;/p&gt;<br>&lt;/form&gt;<br>&lt;h1&gt;<br>{{ midpoint }}<br>&lt;/h1&gt;</pre><pre>{{ wiki }}<br>&lt;br&gt;<br>{% if  airbnb  != &#39;&#39; %}<br>&lt;a href  = &quot;{{ airbnb }}&quot;&gt; See options for stays in the are&lt;/a&gt;<br>{% endif %}<br>&lt;br&gt;<br>&lt;hr&gt;<br>{{ map | safe}}</pre><pre>&lt;!-- Template. This whole data will be added directly to working form above --&gt;<br>&lt;div id=&quot;newlinktpl&quot; style=&quot;display:none&quot;&gt;<br>    &lt;div class=&quot;feed&quot;&gt;<br>     &lt;input type=&quot;text&quot; name=&quot;feedurl[]&quot; value=&quot;&quot;  size=&quot;50&quot;&gt;<br>     &lt;input type=&quot;text&quot; name=&quot;feedurl[1]&quot; size=&quot;2&quot;&gt;<br>    &lt;/div&gt;<br>&lt;/div&gt;</pre><pre>&lt;script type=&quot;text/javascript&quot;&gt;<br>function validate(frm)<br>{<br>    var ele = frm.elements[&#39;feedurl[]&#39;];<br>    if (! ele.length)<br>    {<br>        alert(ele.value);<br>    }<br>    for(var i=0; i&lt;ele.length; i++)<br>    {<br>        alert(ele[i].value);<br>    }<br>    return true;<br>}<br>function add_feed()<br>{<br>    var div1 = document.createElement(&#39;div&#39;);<br>    // Get template data<br>    div1.innerHTML = document.getElementById(&#39;newlinktpl&#39;).innerHTML;<br>    // append to our form, so that template data<br>    //become part of form<br>    document.getElementById(&#39;newlink&#39;).appendChild(div1);<br>}<br>&lt;/script&gt;</pre><p>With that, I launched the Django server and it was functional!</p><p>Woof, what a ride. I’m trying to get this all done on the same day so its still fresh and the context of a hackathon doesn&#39;t seem fair coming back two days afterward. I’m putting pen to paper ASAP so I can channel the frantic prototyping this project became. I had a lot of fun with this project and others like this, its a great opportunity to hone some skills and pick up new ones as you go along. A lot of the tools I used today I had never seen before; GMAP, Wiki, USZIP were all new libraries to me. It speaks to what you can do with a bit a perseverance and creativity. Thanks for reading <em>(sorry for any bad grammar)</em>, I’ll be uploading the code to my public github in the next few days: <a href="https://github.com/tbacas">https://github.com/tbacas</a></p><p>-TRB</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36158b469471" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/finding-the-middle-ground-for-our-next-vacation-with-django-and-python-36158b469471">Finding the middle ground for our next vacation with Django and Python</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django Webapp]]></title>
            <link>https://medium.com/@thomasbacas/django-webapp-4359cfa174a4?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4359cfa174a4</guid>
            <category><![CDATA[web-app-development]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Mon, 12 Aug 2019 16:04:12 GMT</pubDate>
            <atom:updated>2020-05-18T16:14:49.973Z</atom:updated>
            <content:encoded><![CDATA[<p>At Red Oak Strategic our team is always looking for ways to maximize our efficiency. When we have a process that is manual, we immediately see an opportunity to develop a programatic solution to whatever is causing our bottleneck. One of our client requests is not only the results of our predictive models but also excel documents that divides our finding into actionable information. Before my project launched, our process involved taking our model outputs and manually inputting the specific values into an excel layout. <em>This was not an efficient process</em>. The team tasked me with improving this system and to develop a tool that empowers other team members to push out these reports at scale. In this post I will outline some of the steps I took in creating this webapp as well as ways you could integrate django and python in your own business to improve your manual processes.</p><h4>Stage 1: Develop Backend Capability.</h4><p>The tools to achieve this task were all familiar to me as I had worked with excel projects in the past. The primary libraries I would be using were Openpyxl, Pandas, Boto3 and eventually SQLalchemy/Pyodbc to connect to the client database. I will reference their uses later.</p><p>For the sake of understanding I will outline the data I used in this example. Our team fields a survey across ~1600 real voters. Using this survey as our train set, our model output predictions to individual responses across the entire voter universe. To produce the trended reports I needed to generate the averages of demographic groupings.</p><p>This gives an example of how our data is created. But how does it get input properly, styled and updated without effecting previous report data? Openpyxl provides the utility to make all of these specific changes possible. Assuming we have properly formatted data sets, it takes only a few coding steps to generate a new workbook or to input new information into an existing workbook and copy over styling.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b0cbf6fbbd92e1998aa2b3c9ff6359c/href">https://medium.com/media/6b0cbf6fbbd92e1998aa2b3c9ff6359c/href</a></iframe><p>Here is an example of generating a new report. We use a template workbook as our basis workbook for deploying our information from our model. We select the original ‘Sheet1’, copy its format, and input our information aswell as title the sheet with the issue. In the process of copying the sheet in openpyxl we lose the original conditional formatting, so it is input into the necessary rows here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1d30547f47432d9f19873fe736fd2a9/href">https://medium.com/media/a1d30547f47432d9f19873fe736fd2a9/href</a></iframe><p>The next process is a bit more complex, but is the backbone of the entire tool. After generating a report, we want to be able to update it with new information as it comes to us. Following a similar structure to the first function, this function looks for what information is available, inserts a new row, copies the formats of the cells next to the inserted row, and inputs the necessary information. These functions will come back around again when we move into discussing the front end and integration with Django but they are a helpful example of just how powerful Openpyxl can be when used programatically. This process takes only a few seconds and can iterate over hundreds of sheets in a workbook to update information. <strong>Imagine skipping all of that unnecessary data entry… </strong><em>ahh bliss</em><strong>.</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FCiTLZWskt7Fu%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FCiTLZWskt7Fu%2Fgiphy-downsized.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FCiTLZWskt7Fu%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="261" frameborder="0" scrolling="no"><a href="https://medium.com/media/201aff683fc1343c961c5bb7cec06212/href">https://medium.com/media/201aff683fc1343c961c5bb7cec06212/href</a></iframe><p>There are several steps to get the data to the point that it was ready to input into that data, but thats boring. Lets move into the front end development.</p><h4>Stage 2: Move Toward Front End.</h4><p>While these scripts could be run fairly easily on my local machine, it would be a major hassle to zip and import these files and all the dependencies to a co-worker’s computer. There were also aspects of the functions which could be optimized with a simple, yet effecient, dashboard. I had very little experience in front end development when I set out to start this project, but working with Django I was able to merge my experience with python seamlessly to a front end workspace.</p><p>Django has detailed and extensive documentation as well as many free tutorials online so I will not be going into the specifics of launching a django project in this post. Instead I will show you some of the tools I incorporated into my django app to allow for high speed cloud integrated report generating and updating.</p><p><strong>The File Page</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8LvnumNa4D7qudEOrLbag.gif" /></figure><p>This is the file page. A central hub for all of the files generated/updated by the report tool. These files are connected to an S3 bucket on our Red Oak Strategic AWS server. As we generate/update reports in the app they are sent directly to our S3 bucket using boto3. You can download any of these files simply by clicking on their name.</p><p>This page was built out of necessity during development, but I found it was a much needed addition to the app. It allowed me to quickly delete, upload, and download files from the s3 bucket without having to navigate to the s3 dashboard from AWS. When stress testing, the app would generate hundreds of reports which I needed to delete quick and easy. I could select delete all. Poof. All gone.</p><p>This page relies entirely on boto3. Boto allowed us to pull all of the file information from the s3 bucket as well as delete, upload and download files. Without exposing too much of our internal infrastructure, this is an example of how the Django app communicates with itself:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb4c582220da1357201da953e0b06e9a/href">https://medium.com/media/cb4c582220da1357201da953e0b06e9a/href</a></iframe><p>If the html form request is ‘post’ (any of the forms on this page) it results in the next action. The next action is determined by the name of the post variable. Let’s say for this example ‘deletefile’ was clicked. The form will designate name2 with the value from the dropdown select values in that form. If name2 is ‘all’, we loop through the entire bucket and delete <strong>ALL</strong> of the values. Otherwise we delete the selected value.</p><p>This is how simple it is to use Django to add an interface to your python scripts. With relatively simple code we generate a file dashboard that suits our needs much better than AWS’ s3 dashboard.</p><p><strong>The Generate Page</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sq3C14Rj9YrpjhvRdC7Sew.gif" /></figure><p>This page works with an updated version of the function above. It takes arguments from the selections on the app and pushes them toward the view to be processed and pushed to the S3 bucket. The tool takes a score file to populate the issues to be generated. Then the user inputs a ‘Report Name’ and ‘Round Number/Date’ to be included in the report. Once its generated it can be located in the file page, where you can download it to your local machine or move to the Update page to imput more information.</p><p>In Django we can reference form inputs directly as varibales in the backend code. This allows for a variety of creative uses of the forms on each page. We see here the ‘Issues’ are generated by importing the final score sheet, in this case: ‘finalSCdistrict7’ or the final model scores for SC District 7. That process reads the selected csv file and returns a list of columns (issues) which can be pushed to the template as we see here. Then, by selecting the issues from this checkbox list, the code can determine which issues to populate on the new document. This takes what would have been several code segments to a few clicks for the user which results in a massive time-save for our team.</p><p><strong>The Update Page</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nd0eMPDU1JTxU5R64seKCA.gif" /></figure><p>Like the generate page, the update page utilities the function above to insert information into a report. You will recognize the files at the bottom of the page as they’re all connected to s3 as opposed to local memory.</p><p>So, where is this all hosted? This current version is hosted on a ec2 where anyone from the team can immediately access the tool by navigating to the ip address of the box. Since this is for internal use we have no aspirations to find a permanent home for the tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4359cfa174a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Accurate Bond Prices with Neural Networks]]></title>
            <link>https://medium.com/@thomasbacas/predicting-accurate-bond-prices-with-neural-networks-ff46c051c25c?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff46c051c25c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bonds]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Fri, 26 Oct 2018 18:56:15 GMT</pubDate>
            <atom:updated>2018-10-26T18:56:15.042Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MKCrBb9TwUFRgf8q57f2sw.jpeg" /></figure><p>While studying finance I found it very interesting that the bulk of our education was covering the equity market. But the finance world is vast, and focus on equity is a fools errand. Working at Aegis Value Fund in Arlington VA the team there specialized in finding value in the less popular (and greatly less understood) companies of the small and micro cap flavor ($100–500m market cap). After seeing the fund succeed against all odds, I began to hunt for value in a similar way. I found that Bonds, much like the small cap stocks that Aegis had pursued, had vastly less efficient pricing compared to the equity market. ‘Unlike the stock market, bonds suffer from a substantial lack of pertinent information’.</p><p>“ By contrast, bond trades are required to be reported within 15 minutes and only those who pay for the TRACE feed can access this information. No quotes are publicly available and the best way to get a quote is to solicit multiple brokers and wait for a reply. Alternatively there are data companies that provide end of day prices, published after the market has closed and with no guarantee that the specific information sought will be included. Accurate bond pricing is also hindered by lack of liquidity. Only a fraction of TRACE eligible bonds trade on a given day, so the most recent trade price is often multiple days old. Pricing bonds based on other more liquid bonds that have similar features is common, but again limited by the presence of such bonds.”</p><p>Benchmark Solutions sought to develop a model to predict more accurate bond prices in real time. With no quotes publicly available, the best way to get a quote is to solicit multiple brokers and wait for a reply. Benchmark Solutions plans to predict accurate prices that incorporate interest rate data, trades or quotes of the bond in question, trades or quotes of other bonds or CDS of the issuer of the bond in question as well as other input sources. While this Kaggle challenge has been open for 6 years, I took it upon myself to try and develop a model that could accurately and efficiently produce current bond prices. I drew my inspiration from <a href="https://arxiv.org/pdf/1705.01142.pdf">a paper </a>written by Swetava Ganguli and Jared Dunnmon in which they pursued the same data set.</p><h4>Exploratory Data Analysis and Cleaning</h4><p>The data set included 762,678 data points, and 61 features. The only null values were attributed to the moving average spacing which I replaced with -1000 to continue my modeling. I also broke the “is_callable:” columns into binary columns using pd.get_dummiesfor better performance in my models. Beyond that the data came well prepared and cleaned.</p><p><strong>Column details:</strong></p><ol><li>id: The row id. bond_id: The unique id of a bond to aid in time series reconstruction. (This column is only present in the train data). 3736 unique bond ids, varying from 12,129 data points to 1 per id.</li><li>trade_price: The price at which the trade occurred. (This is the column to predict in the test data). Mean: 103.44, Median: 102.33, STD: 9.82.</li><li>weight: The weight of the row for evaluation purposes. This is calculated as the square root of the time since the last trade and then scaled so the mean is 1.</li><li>current_coupon: The coupon of the bond at the time of the trade.</li><li>time_to_maturity: The number of years until the bond matures at the time of the trade.</li><li>curve_based_price: A fair price estimate based on implied hazard and funding curves of the issuer of the bond.</li><li>reporting_delay: The number of seconds after the trade occured that it was reported.</li><li>trade_size: The notional amount of the trade.</li></ol><p><strong><em>Categories:</em></strong></p><ol><li>is_callable: A binary value indicating whether or not the bond is callable by the issuer.</li><li>trade_type: 2=customer sell, 3=customer buy, 4=trade between dealers. We would expect customers to get worse prices on average than dealers.</li></ol><p><strong><em>Time Series:</em></strong></p><ol><li>received_time_diff_last{1–10}: The time difference between the trade and that of the previous {1–10}.</li><li>trade_price_last{1–10}: The trade price of the last {1–10} trades.</li><li>trade_size_last{1–10}: The notional amount of the last {1–10} trades.</li><li>trade_type_last{1–10}: The trade type of the last {1–10} trades.</li><li>curve_based_price_last{1–10}: The curve based price of the last {1–10} trades.</li></ol><p>As expected of time series data, many of the pricing features were strongly correlated to counter act this Principal Component Analysis was run on the entire data set.</p><h4>Model Selection</h4><p>After seeing the results of Ganguli and Dunnmonthat I decided to assert the majority of my efforts into developing fine tuned neural networks and GLMS, both models which had produced the best results in their research. The first of these was a multi-layer Neural Network (NN) which consisted of 4 hidden layers with gradually decreasing nodes from 25&gt;15&gt;10&gt;5. This model produced impressive results while remaining very efficient. Achieving a Mean Absolute Error of .43 or able to predict a price within ¢43 error. This model ran 100 epochs in 36 minutes on a Intel core i7–7700hq processor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*YO92xkApUy6vX8U4ntLFsw.png" /></figure><p>I was very satisfied with this model but I wanted to be diligent and try out several others.</p><p>Next was an RNN-LSTM or Long Short-Term Memory Neural Network. LSTM’s are great for working with time series data as their weights are contained longer depending on time series lag. However, this can substantially slow down the processing speed by having to save these specific weights. This was the case with this model. It took well over 5 hours to reach convergence.</p><p>I move into GLMs where I replicated the OLS regression and WLS regression from Ganguli and Dunnmonthat’s prior research.</p><p><em>(Note: I used Mean Absolute Error in models while Ganguli and Dunnmonthat used Weighted Error in Prediction per Sample so our metrics cannot be equally compared.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*gHdMVCk_HXnXGrJXw-zHUg.png" /><figcaption>Weighted Error in Prediction per Sample</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*WIZhAZDx5MFxsT_S4HjKuA.gif" /><figcaption>Mean Absolute Error</figcaption></figure><p><em>I chose mean absolute error as my metric as error positive and negative are equally detrimental.</em></p><p>The results of my GLMs were similar to the results noted in the paper fast and decent, but not enough to overtake the accuracy of my RNN.</p><p>Finally I implemented a Random Forest Regression, as a final model to test how well my models reflected results from the Ganguli and Dunnmon paper. The results mirrored my expectations, relatively slow results and over fit model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fsheetsu.com%2Ftables%2F7a27cfebd7&amp;dntp=1&amp;url=https%3A%2F%2Fsheetsu.com%2Ftables%2F7a27cfebd7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=sheetsu" width="700" height="344" frameborder="0" scrolling="no"><a href="https://medium.com/media/335ca7076aa77c05626dd0c28b54478f/href">https://medium.com/media/335ca7076aa77c05626dd0c28b54478f/href</a></iframe><h4>Results</h4><p>An important aspect to these results is the consideration of speed AND accuracy. Benchmark Solutions wanted actionable information which could be fast and effective. Considering this, these are my findings:</p><ul><li>With the consideration of speed and accuracy I found that the multi-layer Neural Network (NN) produced the best results. Reaching convergence around 100 epochs in just under 36 minutes and returning the lowest MAE. An error score of 43 cents is great, but to be actionable I would like to bring this down closer to 10 cents.</li><li>LSTM NN which came close to the first NN in accuracy but took substantially longer to run, reaching convergence at just over 100 epochs taking 5 hours and 22 minutes.</li><li>OLS and WLS performed very quickly but fell short in accuracy.</li><li>Random Forest had decent results but at the cost of speed.</li></ul><h4>Next Steps</h4><p>In the testing process I was limited by my computer’s processing capability. My current processor Intel core i7–7700hq was quickly maxed out running only 1 NN. I was able to run on both my GPU and CPU which improved my testing times but was still a substantial barrier in adjusting my tests. Further improve computational power would enable faster fine tuning of the neural networks.</p><p>While this dataset was extensive it lacked substantial historical information. This could have substantially improved the results of both of my NN’s most notably the LSTM. In further implementation of my model I would seek to pursue additional data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff46c051c25c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas: the simple life for data analysis]]></title>
            <link>https://medium.com/@thomasbacas/pandas-the-simple-life-for-data-analysis-41172b425d5d?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/41172b425d5d</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Fri, 19 Oct 2018 16:33:49 GMT</pubDate>
            <atom:updated>2018-10-19T16:33:49.421Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*lx1NnEb1kxgDlkVvaOiAHA.jpeg" /></figure><p>One of the more intimidating aspects of working with data science is the size and scope of data. Seeing lines and lines of unorganized unstructured data can shake even a seasoned veteran. An obvious solution was to create a way to visualize and manipulate data more intuitively. Developer Wes McKinney of AQR Capital Management moved the wheel of progress by developing and releasing Pandas open source for the world to use. Thank goodness, because Pandas makes my life easier everyday.</p><p>Pandas is a python library used for data analysis and manipulation. Pandas makes it easy to organize information in easy to read ‘Data Frames’ which work as a backbone to the structure of the library. Along side these data frames Pandas offers a variety of helpful tools to simplify your analysis process.</p><h4>Getting started in Pandas:</h4><p>To get started in pandas requires data. Pandas offers a pd.read_ function that allows for the user to import files from a variety of different formats (CSV, JSon, HTML, Local Clipboard, Excel, HDF5, Feather, Parquet, Msgpack, Stata, SAS, SQL, etc..).</p><pre>import pandas as pd<br>df = pd.read_csv(&#39;./data.csv&#39;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gFp8gBXrxNAIIcIDmeX92A.png" /><figcaption>This is only the “head” of our data note the modifier at the end of ‘df’.</figcaption></figure><p>You can also produce a Dataframe from a dictionary:</p><pre>df = pd.DataFrame({&#39;A&#39;:[1,2,3,4], &#39;B&#39;:[5,6,7,8], &#39;C&#39;:[9,10,11,12])<br>df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/1*H6jcLhcgRdIsJK2NfwCzwg.png" /></figure><p>Lets say you want to change your index to start a number other than 0, or you want to make it something different entirely.</p><pre>df.index = [1,2,3,4]  or df.index = [&#39;red&#39;,&#39;blue&#39;,&#39;yellow&#39;,&#39;purple&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/1*p_clQr-Hv4y3ZJIXGE26Cw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*wt8W-6_KxG2xWXBKf_l6Cg.png" /></figure><h4>Summarize your Data:</h4><p>Now that you have your data formatted to your liking, lets explore and understand it further.</p><p><strong>df.describe() </strong>is a great first step to understand your data. This prints out the descriptive statistics for your data.</p><pre>df.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*WS4LZyBhuqS001RB2WHUsg.png" /></figure><p>We have found the basics about our data, but lets say we want to explore a single column. It’s as easy as referencing the column index.</p><pre>df[&#39;Column Name&#39;] or df.&#39;column name&#39;<br>#for our example lets try<br>df[&#39;A&#39;]</pre><p>This returns a dataframe of only our selected columns. You can view multiple columns by inputting a list.</p><pre>df[[&#39;A&#39;,&#39;B&#39;]]</pre><p>Interested in finding information based on a certain criteria? Filter your dataframe with Boolen expressions.</p><pre>df = df[df[&#39;A&#39;] &lt;= 2]<br>df</pre><p>We have our columns, lets investigate the values.</p><pre>df.value_counts(ascending = True)</pre><p><strong>df.value_counts()</strong> generates all of the values an their frequency in your dataframe. You can filter ascending or descending by changing ‘ ascending’ to True or False. This is great for identifying your major points or viewing trends.</p><p><strong>df.isnull().sum()</strong> this function shows all of the values for each row that are missing. Missing data is a very common issue for data scientists, being able to quickly identify where your data is missing is essential for your process. We have no missing data in our current dataframe, but we will later. This is a good function to have for the time being.</p><h4>Reshaping our Dataframe:</h4><p>We’ve generated, investigated and filtered our Dataframe, but what if we want to reshape it. There are several ways to reshape your Dataframe, but most commonly you will be combining two or more data frames together. Pandas provides many easy tool to accomplish this. Lets start with <strong>pd.concat().</strong></p><p>I went ahead and generated df2 with our code from before and reset our original dataframes index:</p><pre>df2 = pd.DataFrame({&#39;D&#39;:[4,3,2,1],&#39;E&#39;:[8,7,6,5],&#39;F&#39;:[12,11,10,9]})<br>df.reset_index(inplace = True)</pre><p>Lets say we wanted to attach df2 next to our original df . It would seem that pandas would know to combine in this way since they share an index. Lets see:</p><pre>pd.concat([df,df2])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*JBvcRHePrNpFf_JOdkLskA.png" /><figcaption>Thats alot of NaN</figcaption></figure><p>We notice right away that pandas tried to combine the dataframes together without any conflicts of axis. To fix this we must specify which axis we want to join on.</p><pre>pd.concat([df,df2], axis = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*dmO60ZMd0QBgCs5iQhgIYw.png" /></figure><p>Voila! Our data frames are combined. This is a very useful tool later on when you begin to manipulate your data with further pandas tools an pull in data from multiple sources.</p><p>Now that we have a Dataframe with Nans I can address some tips to deal with these. As I mentioned before you can quickly find how many Nans are present with df.isnull().sum(). Once you’ve done that you can either drop rows based on Nans with df.dropna() which will drop rows with ANY Nans. Or you can fill your Nans with df.fillna() with your chosen criteria.</p><pre>df.dropna(inplace = True) </pre><pre>#inplace modifier applies the drop effect to your current dataframe without you having to reinstance the variable</pre><pre>df.fillna(-9999)</pre><p>This is the end of part 1, I will cover some more tips and tricks to use with pandas in part two. Until then 👋🐼.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41172b425d5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Progress Bar: how to track your loops like a pro.]]></title>
            <link>https://medium.com/@thomasbacas/progress-bar-how-to-track-your-loops-like-a-pro-81eadaf7ee42?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/81eadaf7ee42</guid>
            <category><![CDATA[libraries]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Fri, 19 Oct 2018 15:02:49 GMT</pubDate>
            <atom:updated>2018-10-19T15:02:49.848Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*wQiemcEO-pXvTY7VxPhBrQ.jpeg" /></figure><p>I tend to use for loops for much of my code, this is a pretty common trend among data scientists. An issue I had was when I running my loops for web-scraping I couldn’t track where in the loop I was. Iterating through thousands or millions of rows and trusting that everything was working accordingly was not an effective way for me to code. I wanted to find a to track how my loop was performing without having to wait for it to complete.</p><p>I came across <a href="https://pypi.org/project/progressbar2/">progressbar2</a> a python library that solves this very is issue. progressbar is customizable and very easy to use. I have used it with almost all of my for-loops to track progress and identify errors as they occur.</p><h4>Installing and instantiate Progressbar:</h4><pre> <em>pip install progressbar2</em></pre><p>After installing progressbar you can instantiate the bar with:</p><pre>from progressbar import ProgressBar<br>bar = ProgressBar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VViWXOiHCn5oibnwku9v7g.png" /></figure><p>After loading the library, you can include the bar in the first portion of your for-loop. Simply wrap the entire range of your loop with the bar().</p><h4>Output:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wsj6dP8AHsu_o2YWIX6low.png" /></figure><p>The progress bar gives you ETA for the completion of your loop, elapsed time and number of completed loops. If the loop stops, it will keep its position so you can investigate your code for errors or conflicts.</p><p><em>Issues:</em></p><p><em>I’ve noticed that you need to re-instantiate the bar = Progressbar for each run of the loop. For whatever reason the loop being re-run is out of range. It’s just one more run of a code line to fix this, but it can be a pain.</em></p><p>Want to learn more about progressbar2? Check out the documentation here:</p><p><a href="https://progressbar-2.readthedocs.io/en/latest/">Welcome to Progress Bar&#39;s documentation! - Progress Bar 3.38.0 documentation</a></p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81eadaf7ee42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Divorcing Excel — Why I do my analysis in Python]]></title>
            <link>https://medium.datadriveninvestor.com/divorcing-excel-why-i-do-my-analysis-in-python-6438e261f7f6?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6438e261f7f6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 16:35:59 GMT</pubDate>
            <atom:updated>2018-10-19T13:36:29.038Z</atom:updated>
            <content:encoded><![CDATA[<h3>Divorcing Excel — Why I do my analysis in Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*_wp1d1F_ZZa0azZgMDj0lQ.png" /></figure><p>As a young financial analyst, I was raised on Excel. Everyone I worked with and learned from did everything in Excel. I had many questions, “Why couldn’t this be automated?”, “Why does this work like this?” “Can we speed this up?”. For the most part Excel got the job done. It was a comfortable standard that everyone could work with. But as I started working with more and more data I noticed Excel slowing down. After crashing workbooks and waiting hours for spinning wheels I had to find another way. *Enter Python*.</p><p>For all of the Excel fans out there, I do want to include that Excel and Python are not mutually exclusive. There are some aspects to Excel that are useful that python can do less efficiently depending on your skill and experience. A novice could easily enter into a excel cell and change value while they might struggle to do this in pandas. With that being said, Python greatly exceeds excel in processing capability and has a exponentially larger toolkit.</p><h4>Lets start with something simple:</h4><pre>import pandas_datareader.data as web<br>start_date = &#39;2016-9-30&#39;<br>end_date = &#39;2017-09-30&#39;</pre><pre>prices = web.DataReader(&#39;AAPL&#39;, &#39;robinhood&#39;, start_date, end_date)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*6Mh3--RBsdlSIZykPeX4uA.png" /></figure><p>In one line of code we quickly got all of the historical price data for AAPL. This took approximately 15 seconds and we are ready to perform our analysis. This gives specific customization for your historical price analysis and allows for you to filter through your data-frame for any specific date. You can also easily download this information if you need to work offline.</p><h4>Easy visualization:</h4><pre>prices[‘Close’].plot(grid=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*epZSnlkUQg2XLjXva5syIg.png" /><figcaption>With one line of code you can quickly make a time-series graph.</figcaption></figure><h4>We’ve got prices, how about Financials</h4><p>This is slightly more tricky but still doable with some code. I wrote this code for my analysis for some company in my personal portfolio. It’s flexible but not without issues and bugs.</p><pre>empty_list = [&#39;&#39;,&#39;&#39;,&#39;&#39;,&#39;&#39;,&#39;&#39;]<br>df_dict = {}<br>temp = {}</pre><pre>ticker_list=[&#39;AAPL&#39;]<br>financial_statements = [&#39;bs&#39;,&#39;cf&#39;,&#39;is&#39;]</pre><pre>for ticker in ticker_list:<br>    splittext_final = []<br>    for fs in financial_statements:<br>        new_url = &#39;<a href="http://financials.morningstar.com/ajax/ReportProcess4CSV.html?t=&#39;+str(ticker)+&#39;&amp;reportType=&#39;+str(fs)+&#39;&amp;period=12&amp;dataType=A&amp;order=asc&amp;columnYear=10&amp;number=3&#39;">http://financials.morningstar.com/ajax/ReportProcess4CSV.html?t=&#39;+str(ticker)+&#39;&amp;reportType=&#39;+str(fs)+&#39;&amp;period=12&amp;dataType=A&amp;order=asc&amp;columnYear=10&amp;number=3&#39;</a><br>        res = requests.get(new_url)<br>        <br>        soup = BeautifulSoup(res.content, &#39;lxml&#39;)<br>        text = soup.text<br>        line = re.sub(&#39;&quot;\w[^&quot;]*(,)&#39;,&#39;&#39;, text)<br>        line = line.replace(&#39;&quot;&#39;,&#39;&#39;)<br>        splittext = line.split(&#39;\n&#39;)<br>        splittext.remove(&#39;&#39;)</pre><pre>tempdf =pd.DataFrame({})<br>        splittext_final = []<br>        <br>        for each in range(len(splittext[2:])):<br>            headers = splittext[1].split(&#39;,&#39;)<br>            splittext_final.append(splittext[2:][each].split(&#39;,&#39;))<br>        #print(splittext_final)<br>        for row in range(len(splittext_final)):<br>            if len(splittext_final[row]) &lt;= 5:<br>                splittext_final[row].extend(empty_list)<br>                headers = splittext[1].split(&#39;,&#39;)<br>        for number in range(len(headers)-1):</pre><pre>templist_3= []</pre><pre>for i in range(len(splittext_final)):<br>                templist_3.append(splittext_final[i][number])</pre><pre>tempdf[headers[number]] = templist_3</pre><pre>temp[str(fs)] = tempdf</pre><pre>df_dict[str(ticker)] = temp</pre><p>This function takes a ticker and returns the Balance Sheets, Financial Statements and Cashflow Statements for past 5 years. This can be changed to quarterly in the period section of the URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*W76NA7Ou2gxK8vSzlSxJdw.png" /><figcaption>Example output</figcaption></figure><p>This is an example output from the above noted example. By changing BS&gt;IS&gt;FS you can quickly view and analyze the information for any given company. Like the prices above, you can simply df.to_csv() the data frame to export it as a CSV and open in Excel.</p><p>This is an introduction to python and a personal opinion on the utilization of Excel vs Python. I purposely did not include any analysis for the sake of brevity. These were a few basic tricks I used to help me visualize and analyze financial information but it is far from exhaustive. to include later: Adding moving averages, proper timeseries analysis ARMA, ARIMA, etc.</p><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6438e261f7f6" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/divorcing-excel-why-i-do-my-analysis-in-python-6438e261f7f6">Divorcing Excel — Why I do my analysis in Python</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running models on your GPU — the easy way]]></title>
            <link>https://medium.com/@thomasbacas/running-models-on-your-gpu-the-easy-way-9ad99ded6787?source=rss-639dce71b60e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ad99ded6787</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thomas Bacas]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 15:45:28 GMT</pubDate>
            <atom:updated>2018-10-17T15:48:42.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>Running models on your GPU — the easy way</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*nkW_mZg8Eysstwj--PIXKA.jpeg" /></figure><p>For the past couple of days I have been trying desperately to improve my computational power while running multiple neural networks for my capstone project here at GA. In my project to predict more accurate bond prices there are a variety of models I want to run, which results in essentially grid searching across a variety of different hyper parameters to improve my error. The major bottleneck in my computing power is my CPU, capping out fairly quickly as I run 1 multi-layer NN. I researched a variety of ways to transfer some of the computing power from my CPU to my GPU (Graphics Processing Unit and ease the burden of the computing. I found the idiot-proof way to do this is using Colab and Google drive. Below I will outline the steps needed to quickly run your models through Colab on your GPU.</p><h3>Starting a Colab file:</h3><p>Start by making a new folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*Wfl0qCcZgwuXVIa4bgUgCA.png" /><figcaption>Click “New” and Select “Folder”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*J7vK_5tEV9hEw_x2rTXq9w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*3CxLaXe6se4jdRCCoFKXhg.png" /><figcaption>After opening and naming your first folder (I named mine LSTMNN). Enter your folder.</figcaption></figure><p>Once you have navigated to your new folder, click ‘New’ once again and scroll down to more in the select window. If you see ‘Colaboratory’ click it to open a new Colab document. Otherwise scroll to ‘connect more apps which will open this window. In the Search Apps object type: “Colab” and install add Colaboratory to your drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*1etiGRoWHxRaXsf7sZ-wQw.png" /></figure><p>With Colab installed you’re ready to start your regular Data Science process. “Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.” All of the regular processes run in Colab just as they would within Jupyter Notebook.</p><p>Colab also works like any Googledoc, allowing shareable links to work with your team or group. It’s very accessible but is not without faults. One major issue is the speed at which Colab uploads data. Unlike the typical jupyter notebooks, which can read in csvs in a matter of seconds Colab takes several steps to upload your data which can be timely and frustrating.</p><h3>Running the notebook with GPU</h3><ol><li>Once you have created your new .ipynb navigate to Edit&gt;Notebook settings</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*Cxt9ZYf2YHtxbkhvRrSnNw.png" /></figure><p>2. A new window will pop up like the one below. Change your Hardware accelerator from None to GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*OVoV7AIAm_yS6BNLyNctXQ.png" /></figure><p>3. After this, simply hit save and you are done. Use the code below to confirm GPU usage.</p><pre>import tensorflow as tf<br>tf.test.gpu_device_name()</pre><p>‘/device:GPU:0’ should be output.</p><p>4. Load in your data</p><p>This step can be very tricky, but you first instantiate your drive (if you have uploaded your data to the same file).</p><pre>from google.colab import drive<br>drive.mount(‘/content/gdrive’)</pre><p>Then you open the file with the code below, followed by regular pd.read_csv process.</p><pre>with open(&#39;/content/gdrive/My Drive/LSTMNN/train.csv&#39;):<br>  pd.read_csv(&#39;train.csv&#39;)</pre><p>Voila! Save some computing power and time while you go about your other processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*m7jo9kcjAtoaAtJPECtC5Q.png" /></figure><p>I have been using this along side my normal jupyter notebooks to run multiple models at once. I retain the same computational power I had while running 1 model before as my GPU takes the lion share of the processing for my model in Colab. I hope this was helpful, as it was for me. Some useful links and sources below.</p><p>Source: <a href="https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d">https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d</a></p><p><a href="https://colab.research.google.com/notebooks/welcome.ipynb">https://colab.research.google.com/notebooks/welcome.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ad99ded6787" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>