<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marco Sanchez-Ayala on Medium]]></title>
        <description><![CDATA[Stories by Marco Sanchez-Ayala on Medium]]></description>
        <link>https://medium.com/@marcosanchezayala?source=rss-ec310f88b5d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*F0QZtWcfQo-Qm4dt</url>
            <title>Stories by Marco Sanchez-Ayala on Medium</title>
            <link>https://medium.com/@marcosanchezayala?source=rss-ec310f88b5d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:22:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marcosanchezayala/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Python Tutor]]></title>
            <link>https://medium.com/@marcosanchezayala/the-python-tutor-1adc76be5ff1?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1adc76be5ff1</guid>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 31 May 2020 17:37:51 GMT</pubDate>
            <atom:updated>2020-05-31T17:37:51.580Z</atom:updated>
            <content:encoded><![CDATA[<h4>A visualization of frame-by-frame code execution</h4><p>As programmers, it can often be very helpful to visualize what exactly the computer is doing under the hood. This can help us better understand things like where and why bugs arise or how we might be able to optimize the performance of our code. Luckily, there exists a free and easy-to-use web app for just that: <a href="http://www.pythontutor.com/">Python Tutor</a>.</p><p>Python Tutor’s online coding environment allows you to write code and visualize frame-by-frame how it gets executed by the computer. Don’t be fooled by the name… there’s no actual tutor! And despite the name, Python Tutor currently supports Python 2 and 3, C, C++, Java, JavaScript, TypeScript, and Ruby. There’s also a featuer allowing users to collaborate in real time to help each other. However, I’m not going to speak much to this feature since I believe the main value here is in visualizing the execution of your program.</p><p>This is an incredible tool for beginners looking to better understand how Python code is run, as well as more experienced programmers looking to refresh their understanding of how the code is executed.</p><h3>Getting started</h3><p>The home page provides a brief introduction and interactive demo, but if you’re not reading too carefully you may miss the hyperlink to the actual code editor and visualizer that reads, “<a href="http://www.pythontutor.com/visualize.html#mode=edit">Visualize your code and get live help now</a>”. Pro tip: just bookmark this page and not the home page.</p><p>Let’s see a simple example with some explanation.</p><h4>Pick a language</h4><p>Once on the code editor (linked above), we select a language. There exists support for the various languages mentioned above, but I’m going to use Python 3.6 for this demo.</p><h4>Write some code</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vnDkiS76EEa9xuXPdnljYg.png" /></figure><p>Here’s a simple example that will let us see how the computer stores lists in memory, as well as how the visualizer depicts it.</p><h4>Run the visualization</h4><p>Select “Visualize Execution” and you’ll be redirected to a page where the magic happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SgKlGX7peaKP_0_eCacM1A.gif" /></figure><p>We have a few controls at our disposal. Initially, we can click “Next &gt;” to start the visualization, “Last &gt;&gt;” to skip to the end, or drag the slider to view some point in between. The green and a red arrows point to the lines of code that have just been executed and that will be executed next, respectively.</p><p>For every line of code executed, the space on the right half of the visualizer draws out all of the frames storing variables and the objects to which those variables point (if any). If there’s any print output, then we can see that in the top right corner of the visualizer.</p><p>Lastly, we have the “Edit this code” hyperlink directly underneath the code sample. Should we encounter any bugs or simply want to tweak the code, all we do is click that edit button and then start the visualization again once we’ve made the changes we want.</p><p>The real beauty here is that the user controls how quickly they go through all the steps in the visualization. You can pause anywhere you’d like to gain a better understanding of what’s really going on.</p><h4>Another example</h4><p>The example above was very simple. There was no conditional logic, iteration, functions, classes, etc. but Python Tutor can handle all of this. Let’s look at a slightly more complicated example — a linked list. I want to define a singly linked list that I can append to and make sure each node properly references the next node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyU9Pv53d2DbTerxZPjqXw.png" /></figure><p>All of the code defining this linked list is explained in detail in my <a href="https://medium.com/swlh/introduction-to-linked-lists-353c78e5f556">previous post</a> in case you’re curious.</p><p>Let’s see what happens when we start to visualize this code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*dD3BplyBjF3cEY_x4MU8Ew.gif" /></figure><p>In the first few steps we see right away we have global variables Node and LinkedList pointing to class definitions containing attributes and methods.</p><p>Moving on, we see global variables pointing to different nodes, as well as frames depicting method calls and operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*NTj9HAZKmWCLceVrWz1G1A.gif" /></figure><p>Also quite importantly, at the end we get definitive proof that our code did what we thought it was doing: creating a singly linked list that can properly append values (evidenced by the llist global variable).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dgbAsh_xKG8AN5zU6VsMVQ.png" /></figure><h4>Live Programming Mode</h4><p>Recall above where we had the option to choose between “Visualize Execution” and “Live Programming Mode”. I haven’t gone into depth about LPM because it’s essentially the same thing as Visualize Execution except you don’t need to go to a different page to edit the code. You can be mid-visualization, catch a bug, immediately fix it, and Python Tutor will update the visualization accordingly.</p><p>Python Tutor has saved me in various situations throughout my time programming with Python. I’ve begun to dabble in other languages recently for fun, and actually realized through writing this post that Python Tutor support other languages too like JavaScript, C++, and more! So I’ll definitely be using it to better understand those as well.</p><p>I hope you’ve found something useful, and am interested to hear if there are any other cool visualization tools out there for programming!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1adc76be5ff1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Linked Lists]]></title>
            <link>https://medium.com/swlh/introduction-to-linked-lists-353c78e5f556?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/353c78e5f556</guid>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sat, 23 May 2020 20:20:52 GMT</pubDate>
            <atom:updated>2020-05-25T16:35:51.490Z</atom:updated>
            <content:encoded><![CDATA[<p>Implementation in Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZMDmpfmerdrGCMEK" /><figcaption>Photo by <a href="https://unsplash.com/@jjying?utm_source=medium&amp;utm_medium=referral">JJ Ying</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When I started this blog, I was a data science boot camp student determined to pursue a career in data analysis. My posts early on were often focused on techniques related to data analysis and visualization. Recently in my journey into tech, however, I’ve found a much deeper appreciation for coding and computer science than I have for purely data-focused applications of programming.</p><p>A few weeks ago I began shifting my focus towards better learning object-oriented programming, as you might tell from my most recent posts. This has opened many doors for me including one leading into the world of data structures, since their implementation is most easily achieved with OOP.</p><h3>Data Structures</h3><p>Data structures in computing refer to particular ways of organizing data for efficient use. Python users should be familiar with some of these simple data structures inherent in the language:</p><ul><li>Lists</li></ul><pre>[1, 2, 3]</pre><ul><li>Tuples</li></ul><pre>(1, 2, 3)</pre><ul><li>Sets</li></ul><pre>{1, 2, 3}</pre><ul><li>Dictionaries</li></ul><pre>{&#39;first&#39;: 1, &#39;second&#39;: 2, &#39;third&#39;: 3}</pre><p>We have different data structures because each one can offer unique advantages over others in terms of different tasks like inserting elements, removing elements, searching, sorting, etc.</p><p>The data structure I’d like to discuss today is one that does not come standard in Python, the linked list. Why it’s standard in some other languages but not Python is a completely different <a href="https://stackoverflow.com/questions/50310731/why-do-some-languages-like-c-and-java-have-a-built-in-linkedlist-datastructure">discussion</a>.</p><h3>Linked Lists</h3><p>A linked list is a linear collection of data elements. Each element points to the next, defining their ordering in the sequence. They can be used for many things like implementing stacks and queues, performing arithmetic operations on long integers, etc. See <a href="https://www.geeksforgeeks.org/applications-of-linked-list-data-structure/">here</a> for more examples!</p><p>There are many flavors of linked lists, but I’m going to introduce just the singly linked list. Let’s see an example and introduce some terminology.</p><pre>1 -&gt; 5 -&gt; -2 -&gt; None</pre><p>We have three elements in the linked list above. Each element is called a <em>node</em>. Each node has two properties: a value and a reference to the next node in the sequence. By “reference” I mean the arrow pointing to the next value. You can think of the above linked list as the following with each box representing a node.</p><pre>+------+ +------+ +-------+ <br>| 1 -&gt; | | 5 -&gt; | | -2 -&gt; | None<br>+------+ +------+ +-------+ </pre><p>I’m going to use the notation from the former example above, but just know that each node encompasses <strong>both</strong> a value and a reference to another node.</p><p>Notice how the third node references None. This is how we know we’ve found the end of the linked list because we eventually come to a point where there are no more nodes! Unlike Python lists, there are no indexes with which to reference each element in the linked list.</p><p>The first node of the linked list is called the <em>head</em>. We always start with the head when building our linked lists. In the example above, our head is a node with value 1 and pointing to a node with value 5.</p><p>So what? Let’s look at some things we can do with linked lists.</p><h3>Insertion</h3><p>We can insert elements at the beginning, middle, or end of our linked list. All we have to do is adjust how the nodes reference each other. Let’s add a node with value 0 into multiple places as an example.</p><p><strong>At the beginning:</strong></p><pre><strong>0</strong> <strong>-&gt;</strong> 1 -&gt; 5 -&gt; -2 -&gt; None</pre><p>This is the easiest and fastest. We just assign a new head of our list and make sure it references the old head.</p><p><strong>In the middle after node with value 5:</strong></p><pre>1 -&gt; 5 -&gt; <strong>0 -&gt;</strong> -2 -&gt; None</pre><p>This is slower because the computer has no idea <strong>where</strong> the node with value 5 is until we iterate through to find it. Thus, we look at the head. Is that value 5? No, so we look at the next node whose value is conveniently 5. Now we just insert a new node and make sure it continues the chain of references</p><p><strong>At the end:</strong></p><pre>1 -&gt; 5 -&gt; -2 -&gt; <strong>0 -&gt;</strong> None</pre><p>Another simple case, but takes the longest because we have to iterate through the entire list to find the last node and then add a new node at the end. Again, we make sure this continues our chain of references.</p><p>The important thing in all of these examples is that not only did we add a value, but we added a reference to the next node.</p><h3>Removal</h3><p>Conversely, we can remove elements from different positions in our linked list. Again, all we have to do is adjust how the nodes reference each other. We start again with the list:</p><pre>1 -&gt; 5 -&gt; -2 -&gt; None</pre><p><strong>At the beginning:</strong></p><pre>5 -&gt; -2 -&gt; None</pre><p>The fastest since we just take out the first node and do nothing else.</p><p><strong>In the middle (remove node with value 5):</strong></p><pre>1 -&gt; -2 -&gt; None</pre><p>Slower than removal at the beginning because we have to find node with value 5 to remove it.</p><p><strong>At the end:</strong></p><pre>1 -&gt; 5 -&gt; None</pre><p>The slowest because we have to find the last node to remove it.</p><h3>Implementation in Python</h3><p>Now that we have an idea of the basic building blocks of a singly linked list, let’s look at a basic implementation of this in Python.</p><p>We require two classes in order to do this: a Node class and LinkedList class that keeps track of our head and has methods to insert/remove nodes.</p><h4>Node</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b48454d6c2c40d3fb135abeeb2ca179/href">https://medium.com/media/8b48454d6c2c40d3fb135abeeb2ca179/href</a></iframe><p>Recall that a node keeps track of just two things: a value and a reference to another node. Thus, our __init__() method takes care of these two things for us. We assign the reference self.next later.</p><h4>LinkedList</h4><p>The absolute simplest implementation of this class just keeps track of the head.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/113b08fb2c1cd0e1ce52e9d5f47ea54d/href">https://medium.com/media/113b08fb2c1cd0e1ce52e9d5f47ea54d/href</a></iframe><p>We can then construct and see our linked list above with the following assignments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/daa90eef260678739f4fbd2f5fe58dff/href">https://medium.com/media/daa90eef260678739f4fbd2f5fe58dff/href</a></iframe><p>The output of the while loop is</p><pre>1<br>5<br>-2</pre><p>Great! Let’s look at how we might insert elements.</p><h4>Insertion: Beginning</h4><p>Rememeber there are three ways we can insert. Let’s start at the beginning of the linked list.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9f9086937080dad60ae64cf5074402d/href">https://medium.com/media/a9f9086937080dad60ae64cf5074402d/href</a></iframe><p>Note the self parameter due to the fact that this is actually a method within LinkedList.</p><p>We follow the same logic as explained in the visual example above. Thus, if we have</p><pre>llist: 1 -&gt; 5 -&gt; -2</pre><p>and we run</p><pre>llist.push(0)</pre><p>we will get the structure</p><pre>0 -&gt; 1 -&gt; 5 -&gt; -2 -&gt; None</pre><h4>Insertion: Middle</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9473e271368a07907b44e80d49bddfa4/href">https://medium.com/media/9473e271368a07907b44e80d49bddfa4/href</a></iframe><p>This method may be fewer steps, but you do have to already have prev_node to insert the new node. In llist_02.py we created llist, second, and third. We could insert a node with value 0 into our linked list by calling</p><pre>llist.insert_after(second, 0)</pre><p>giving us</p><pre>llist: 1 -&gt; 5 -&gt; 0 -&gt; -2 -&gt; None</pre><h4>Insertion: End</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f251b76118329ac59d3c02553b917ce5/href">https://medium.com/media/f251b76118329ac59d3c02553b917ce5/href</a></iframe><p>To use this in practice all we do is take our linked list and append!</p><pre>llist.append(0)</pre><p>Gives</p><pre>llist: 1 -&gt; 5 -&gt; -2 -&gt; 0 -&gt; None</pre><h4>Removal</h4><p>This follows the same intuition as described above and will be a nice exercise for anyone following along to try themselves. If not, I’ll have code up on my <a href="https://github.com/msanchez-ayala/sort-search/blob/master/data_structures/linked_list.py">GitHub</a> with the solutions and more.</p><h3>Final Thoughts</h3><p>We can use insertion and removal methods to carry out all sorts of algorithms on this data structure. The repo I linked to in the previous paragraph will ultimately be updated with some examples of this as I work through more problems.</p><p>Also, as I mentioned before, there are many flavors of linked lists, not just singly linked lists! We also have doubly linked lists, circular linked lists, etc. I’d highly recommend looking more into these if you’re interested.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=353c78e5f556" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/introduction-to-linked-lists-353c78e5f556">Introduction to Linked Lists</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Bike Share Simulation Using Python]]></title>
            <link>https://medium.com/swlh/building-a-bike-share-simulation-using-python-cd5834be4e1d?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd5834be4e1d</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[object-oriented]]></category>
            <category><![CDATA[simulation]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 17 May 2020 18:03:42 GMT</pubDate>
            <atom:updated>2020-05-18T15:51:39.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AOnA1JpwltZ0uwwY" /><figcaption>Photo by <a href="https://unsplash.com/@aginsbrook?utm_source=medium&amp;utm_medium=referral">Anthony Fomin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Some Background</h3><p>In my last post, I introduced a Python project aimed at implementing object-oriented programming. The goal was to create a simple bike share simulation modeled after New York City’s <a href="https://www.citibikenyc.com/">Citibike</a>, of which I am a big fan!</p><p>A bike share system can quite easily be modeled with OOP because there are just a few basic components that interact and need to be tracked:</p><ol><li>Bikes</li><li>Docks</li><li>Stations</li></ol><p>A bike share system has a fixed number of stations. Each station holds a fixed number of docks that can hold bikes that users can ride between stations. Example 1 below shows possible bike movement across a simple implementation of my simulation’s bike share system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*npVLRErcQR466C29TBnaUA.png" /><figcaption>Example 1</figcaption></figure><p>We have three stations, each with a single dock, and one bike total. Bike 0 can be checked out of Station 0 and moved to Station 1 or 2 (assuming that there are no joy rides resulting in a bike being checked out of and into the same station). The time at which the bike is checked out and checked in is recorded, and the trip duration is used to calculate the price of that ride. Once at Station 1 or 2, the bike can be checked out again and moved to one of the open stations.</p><p>Bikes can only be checked into stations with empty docks. Example 2 shows a case where Bike 0 can only move to Station 1 because Station 2 is occupied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CG57a3s0H_pHJpxw4RcLMQ.png" /><figcaption>Example 2</figcaption></figure><p>In reality, the system will have more stations, docks, and bikes. The base implementation of my bike share simulation has by default 9 stations of equal size with a total of 135 docks and 80 bikes.</p><p>In my previous post, I mentioned that I’d share some code, but I think there may be more value in instead explaining my process and organization since I’d like to document my process here more than anything. After all, the code itself is available on <a href="https://github.com/msanchez-ayala/bike-share-sim">GitHub</a> and will make much more sense after the following explanation.</p><h3>Planning</h3><p>I’m a firm believer of having at least some outline of a project before jumping in, so the very first thing I did was to plan out the major pieces of this simulation and how they would interact. It wouldn’t make much sense jumping into the logic of the simulation itself right away, so I started with the core components of a bike share system.</p><p>As described above, there are really only 3 pieces here: bikes, docks, and stations. Because I knew there will ultimately be several of each that all behave similarly to their counterparts, these three entities became excellent candidates for having their own classes. It’s worth noting though that I intentionally leave out customers because they’re not necessary to keep track of the ride lengths and prices.</p><p>Without any coding, I started out by writing down what each class would need in order for a system full of these objects to capture a complete picture of a bike share system. First I considered stations, then docks, and finally bikes.</p><h4>Station</h4><ul><li>Unique identifier: to keep track of which stations are being checked in or out of.</li><li>Location: a coordinate pair to be able to calculate distances (and thus trip duration) between stations.</li><li>Size: the fixed number of docks this station contains.</li><li>Docks: one or more docks that can hold bikes.</li><li>Activity log: to keep track of which bikes come and go at what times, and how much was paid for rides.</li></ul><h4>Dock</h4><ul><li>Unique identifier: to keep track of which dock within the current station is being checked in or out of. Only needs to be unique within this station. In other words, each station starts with Dock 0 as in Examples 1 and 2 above.</li><li>Bike: a Bike object waiting to be checked out or None.</li><li>Check in/check out methods: a way to record the times and locations at which bikes enter/exit this dock.</li></ul><h4>Bike</h4><ul><li>Unique identifier: to keep track of which bike in the system is moving around.</li><li>Rate: the price that this bike costs to use for different amounts of time.</li><li>Condition: to keep track of the wear and tear. Originally, I thought this would be cool to implement, but I decided there would be too much to incorporate since the idea was that after a certain number of uses, the bike would need to be repaired. It would be an unnecessary amount of work given that my goal is really just to build a simulation using OOP, not to model a bike share system perfectly.</li></ul><p>Lastly, I knew there would need to be some way to put these pieces together: a Simulation class.</p><h4>Simulation</h4><p>This class needs to instantiate a given number of Stations with enough Docks to fit a predetermined number of Bikes and have extra room for people to be able to check in bikes at other Stations. Simulation needs to then run a loop that simulates the passing of time with each iteration representing one minute. Every minute, this loop needs to</p><ol><li>Determine if any bikes will be checked out, determine where it would be, and execute the check out(s).</li><li>Determine if any bikes need to be checked in, determine if the destination is available, and execute the check in(s) if possible. If there are no open docks at the destination, then the bike needs to be sent to another station to park.</li><li>Keep track of any bikes “in transit”.</li></ol><p>At the end of the simulation, I wanted this class to output some statistics like the total number of completed trips, the total revenue for the bike share company, the average trip duration, etc.</p><h3>Implementation</h3><p>To actually implement this, I first created a repository with the following empty files for my models.</p><h4>Models</h4><pre>bike-share-sim<br>|-- sim<br>|   |-- __init__.py<br>|   |-- bike.py<br>|   |-- dock.py<br>|   |-- station.py<br>|-- test<br>|   |-- __init__.py<br>|   |-- test_bike.py<br>|   |-- test_dock.py<br>|   |-- test_station.py</pre><p>I started by first implementing bike.py, as it’s the simplest of the three models and also is a building block needed for the other two. As I developed bike.py I also used the unittest package to develop test_bike.py to make sure every method in bike.py was working as expected. I repeated this process with Docks and Stations. Soon thereafter, I had all the pieces I needed to actually create a simulation.</p><h4>Simulation</h4><p>This was the most difficult part of the project because its implementation involved figuring out how to simulate people coming and checking out bikes at different times. There are many approaches, but after some research I decided to use a Poisson process because this would allow to predict bike checkouts every minute given a predetermined average checkout rate that can be modified to suit the needs of the simulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*s8eRfy_cmpWBW5Wpo73Nnw.png" /></figure><p>Every minute, I calculate the probability of K checkouts occurring, P(K). I use lambda = 4 rides/min based on some research I did into DC’s Q1 bike share data (accessible from Mode Analytics’ platform). However, picking a K value every minute is where I had to get creative.</p><p>I decided that the upper half of a normal distribution with mu = 1 and sigma= 1would be appropriate to model the number of people considering renting a bike at any given minute of the day for a system with 130 bikes. It seemed reasonable to me that every minute, generally speaking there is a chance that 1 or maybe 2 people could possibly try to check out a bike (although possible 3 or even 4 people could check one out). I say half the distribution because I only take the distribution for K &gt;= 1 to avoid calculating the probability that 0 or a negative amount of people check out bikes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*QJT72eKYjSwoizdzdm4B3A.png" /></figure><p>Thus, every minute of the simulation I randomly pick a K from the distribution above and plug it into the Poisson probability function to calculate the probability (<strong><em>as a percentage</em></strong>) that K people will actually check out a bike that minute. Lastly, I pick a random number from 1–100 every minute. If random number &lt;= P(K), then at minute t, K people attempt to check out bikes from the system.</p><p>I say “attempt” because there is the possibility that all bikes are currently in use, but that almost never happens.</p><p>The implementation of this class required several (repeated) constants to be defined, which I put into a separate module, consts.py for easy access and modification.</p><h3>Putting Everything Together</h3><p>The last thing I wanted to do was to make the whole simulation callable from the command line. Thus, in the root directory of this project, I created __main__.py that simply instantiates a Simulation object with a given duration in minutes to carry out the simulation.</p><h4>Sample Run</h4><p>The simulation instantiates the following bike share system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxBh1WasePAOk0EuHsejtQ.png" /></figure><p>There are 9 stations at evenly spaced out locations, each with 15 docks. 80 bikes are spread evenly throughout the stations. The simulation runs for 120 minutes and generates a summarized activity log for every minute of the simulation. Here is an example.</p><blockquote>Minute: 33</blockquote><blockquote>— — Customer tried to check in a bike</blockquote><blockquote>— — Bike checked into Station: 2 Dock: 14</blockquote><blockquote>— — — —</blockquote><blockquote>— — Customer tried to check in a bike</blockquote><blockquote>— — Bike checked into Station: 5 Dock: 1</blockquote><blockquote>— — — —</blockquote><p>At the end of the simulation, we get a summary message such as this:</p><blockquote>SIMULATION IS OVER AFTER 120 MINUTES</blockquote><blockquote>There were 19 rides</blockquote><blockquote>Total revenue was $67.5</blockquote><blockquote>The average price per ride was $3.55</blockquote><blockquote>The average ride length was 16 minutes</blockquote><h3>Next Steps</h3><p>It would be nice to tinker with this more to be able to have some command line input of simulation duration and number of bikes. Also, a visual component would be really nice as well. Perhaps a simple GUI could be built to visualize the simulation. Lastly, I’d like to incorporate other real bike share system features such as different types of bikes (such as ElectricBike and thus perhaps ElectricDock), wear and tear, and different models of the same type of bike.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd5834be4e1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/building-a-bike-share-simulation-using-python-cd5834be4e1d">Building A Bike Share Simulation Using Python</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OOP Practice in Python: Bike Share Simulator]]></title>
            <link>https://medium.com/swlh/oop-practice-in-python-bike-share-simulator-4717bd12b365?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4717bd12b365</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[bike-sharing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[object-oriented]]></category>
            <category><![CDATA[bikeshare]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 10 May 2020 18:13:20 GMT</pubDate>
            <atom:updated>2020-05-11T15:12:28.824Z</atom:updated>
            <content:encoded><![CDATA[<p>Most of my coding experience so far as a student of data science and engineering has revolved around procedural programming. For example, I would write a Python module with several functions that call one another in a logical order within a data pipeline to extract, transform, and then load data. One process would simply trigger another. The only objects I use are those from other packages, which ends up working out completely fine for that particular application!</p><p>This is just one approach to programming, however, and it is important in many roles to know how to build object-oriented system. I aspire to build more with my programming, and thus have begun a deep-dive into the world of object-oriented programming (OOP). My reintroduction to OOP consists of two phases:</p><ol><li>Relearning OOP and filling in the holes in my knowledge.</li><li>Implementing my own OOP project from scratch.</li></ol><h3>What is OOP?</h3><p>OOP is a paradigm that allows programmers to implement real-world entities with code such as inheritance, polymorphism, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*yox45V3KzQM6AAOKJF8zwQ.jpeg" /><figcaption><a href="https://www.geeksforgeeks.org/object-oriented-programming-oops-concept-in-java/">geeksforgeeks</a></figcaption></figure><p>I had some exposure to OOP in an intro CS class I took in college several years ago, but I really needed to brush up on my skills in order to properly implement anything. I thus turned to YouTube, as I often do to learn things.</p><h4>Corey Schafer</h4><p>Quick plug for Corey Schafer, who has an incredible YouTube channel covering many programming topics in detail! Most, if not all, of his content is organized into playlists, including his <a href="https://www.youtube.com/playlist?list=PL-osiE80TeTsqhIuOqKhwlXsIBIdSeYtc">OOP playlist</a>. I used this to go over most of the programming I use below.</p><h3>Bike Share Simulator</h3><p>I recently did a really interesting data analysis assessment using a 2012 DC bike share data set on Mode.com. The assessment got me thinking about all of the things one needs to keep track of to properly manage such a system. Given my OOP kick and the fact that the whole system is made up of just a few simple pieces, I figured this would be a great first project.</p><h4>Bike Share Systems</h4><p>Bike shares consist of multiple stations throughout a city. Each station contains a fixed number of docks that can each house a single bike.</p><p>The number of stations depends on many factors, but there should probably be more stations where there is greatest demand for the service. The number of docks within each station also likely follows a similar criteria, although this varies by bike share company. I would imagine that not all docks are initially filled with bikes because then users wouldn’t be able to park bikes anywhere once they check one out.</p><p>The consumer pays based on how long they ride. Classic bikes have a base rate of perhaps $3 for the first 30 minutes. Trips longer than 30 minutes incur an additional small fee (~$0.10) per minute of use. Some bike share systems offer electric bikes at a higher cost.</p><h4>Design</h4><p>This provides an excellent starting point for my project. To keep things simple, I limit the bike share system to its main components:</p><ol><li>Bike (of which there can be two varieties: classic and electric)</li><li>Dock</li><li>Station</li></ol><p>Each component will be its own class as there will need to be multiple stations, each of which tracking the activity of its docks, which each need to check bikes in and out. All stations are fundamentally the same with the exception of location and size. All docks are the same, and all bikes are the same with the exception of the type of bike used. I omit customers as a class because one doesn’t really need to keep track of the customers to run the simulation.</p><h4>Simplest Simulation</h4><p>The simplest case of the simulation with the pieces above is one with just a single station, a single dock, and a single bike. The bike has only one option, which is to leave and then come back at some later time. Every time the bike moves, the dock records the time and trip id, and the bike registers some wear and tear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*kujzuZK9pCNc-hzMuUtYEw.png" /></figure><p>The station has the ability to pull all of the records from each dock to compile a master list of all the activity at that station. In theory, when there are multiple stations and multiple docks at each station, this allows the simulation to compile all of the stations’ data into a database (I’ll use PostgreSQL) that can be used to see where and when each trip starts and ends.</p><p>That’s all for now. I’ll cover the implementation of this system with code snippets in my next post. In the meantime, you can check out the <a href="https://github.com/msanchez-ayala/bike-share-sim">GitHub repo</a> to see my progress!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4717bd12b365" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/oop-practice-in-python-bike-share-simulator-4717bd12b365">OOP Practice in Python: Bike Share Simulator</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL-Inspired Querying of Spreadsheets in Google Sheets]]></title>
            <link>https://medium.com/swlh/sql-inspired-querying-of-spreadsheets-in-google-sheets-a81620042310?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a81620042310</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google-sheets]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 03 May 2020 20:10:46 GMT</pubDate>
            <atom:updated>2020-05-05T07:06:01.452Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*QlxnSctXw10a4_Y0uq5JQQ.jpeg" /></figure><p>The first day of my data science bootcamp last October, the head instructor said, “You will never use Excel again.” In this world though, the vast majority of people still use spreadsheets to keep track of and process data.</p><p>I was recently helping a friend with some work-related data processing in Google Sheets. He wanted to aggregate his data to calculate some statistics, but really only knew how to do this by aggregating everything one-by-one. He could do it pretty easily on Google Sheets, but it would take him a long time and would be very prone to errors.</p><p>His problem would be so easily solved if he could just use SQL to query the data in such a way:</p><pre>SELECT<br>  A,<br>  SUM(C)<br>FROM<br>  A2:D60<br>GROUP BY<br>  A</pre><p>It turns out you can!</p><h3>The Query Function</h3><p>Perhaps the most powerful Google Sheets function, as it lets you execute SQL-like queries on data as if it were stored in an actual database. This function takes in three parameters.</p><pre>QUERY(data, query, [headers])</pre><ul><li>data: The cell range to query e.g. A2:D60.</li><li>query: The query to execute against data. This is written in <a href="https://developers.google.com/chart/interactive/docs/querylanguage">the Google Visualization API Query Language</a> e.g. &quot;select A, SUM(C) group by A&quot;. Note that there is no FROM clause because this has been defined in data.</li><li>headers: This is an optional argument that denotes the number of header rows at the top of data. If omitted or set to -1, then Sheets infers this number.</li></ul><p>For example, to reproduce our hypothetical SQL query above we would type the following into a cell in our Google Sheets spreadsheet.</p><pre>=QUERY(A2:D60, &quot;select A, SUM(C) group by A&quot;)</pre><p>After pressing enter, we will see the result of our query generated by Google Sheets. This assumes data exists in columns A and C for rows 2–60, and that C is numerical (and thus can be summed).</p><h3>Google’s Query Language</h3><p>According to the documentation:</p><blockquote>The query language provides the ability to send data manipulation and formatting requests to the data source, and ensure that the returned data structure and contents match the expected structure.</blockquote><p>It turns out this language was born from the need to format data in a way appropriate for visualizing data. That’s why it’s called the Google Visualization API Query Language. I’ll refer to it as Google Query Language (GQL) from here on out for simplicity.</p><p>GQL is quite similar to SQL, so if you’re familiar with SQL then this should all be straightforward. Below is a <strong>very brief</strong> overview of how the language works. I recommend checking out <a href="https://developers.google.com/chart/interactive/docs/querylanguage#setting-the-query-from-javascript">the docs</a> or my example at the bottom of this post if you have questions.</p><h4>Allowed Clauses</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KE2MUJoao0xFeQFN49URwA.png" /><figcaption>(from <a href="https://developers.google.com/chart/interactive/docs/querylanguage#data-manipulation-functions">documentation</a>)</figcaption></figure><h4>Aggregation Functions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HybK-1fouH_FTSyxb-9Dbw.png" /><figcaption>(from <a href="https://developers.google.com/chart/interactive/docs/querylanguage#data-manipulation-functions">documentation</a>)</figcaption></figure><blockquote>Aggregation functions can be used in select, order by, label, format clauses. They <em>cannot </em>appear in where, group by, pivot, limit, offset, or options clauses. (GQL Documentation)</blockquote><h4>The Rest</h4><p>The above should get you through most simple queries. Check <a href="https://developers.google.com/chart/interactive/docs/querylanguage#language-elements">the documentation</a> for things like scalar functions and language elements.</p><h3>How to Implement</h3><p>Let’s see this in action! Here’s an excerpt of the data I’m working with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*e-alc6dEGrAQuFwPavENaw.png" /></figure><p>I have the above categorical and numerical information for a little over 1300 salsa and bachata tracks.</p><p>Let’s find the average song duration by artist. My query function would look like this:</p><pre>=QUERY(A1:E1365, &quot;select B, AVG(D) group by B&quot;)</pre><p><strong>Note:</strong> I reference the columns by their column letter rather than by their actual column name. But other than that, the syntax pretty much follows SQL exactly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*b96RYnj0HVF_xNt3XBqwGA.gif" /></figure><p>You can do any type of queries that’s allowed by the clauses and functions discussed above. You can even now query this new table created from the first query!</p><h4>Storing a Query in Another Cell</h4><p>When you have a bunch of queries like this, it can be hard to remember what your query actually looks like. I like to keep track of this by writing out the query in the cell directly above where I want my result to go. Then I reference that cell in my query function like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*fGyKwXKbsS85fpTcmIXZWA.gif" /></figure><p>The query is written in cell G2. My query function then references G2 through the query parameter. The result looks a little cluttered but you can style it however you want. Some light formatting results in the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*3dOMP-TERS6LsFJ6XuQtLQ.png" /></figure><p>GQL is quite flexible and powerful for quick work in a spreadsheet that needs to be sent around a company to non-technical users, for instance. I was really excited to discover this feature in Google Sheets, but have not seen the same functionality from Excel or Numbers. I wonder what other secrets Google has waiting to be uncovered…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a81620042310" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/sql-inspired-querying-of-spreadsheets-in-google-sheets-a81620042310">SQL-Inspired Querying of Spreadsheets in Google Sheets</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Travel Time with Google Maps: Part 2]]></title>
            <link>https://medium.com/@marcosanchezayala/optimizing-travel-time-with-google-maps-part-2-f5c97a16b150?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5c97a16b150</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 16:55:51 GMT</pubDate>
            <atom:updated>2020-04-24T16:55:51.860Z</atom:updated>
            <content:encoded><![CDATA[<h4>ETL Pipeline and Dashboard</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*DyT1B9qn-65gJhiyncHFVQ.gif" /></figure><h3>Introduction</h3><p>My girlfriend and I live in two different NYC boroughs, making travel between our apartments a real pain. Although Uber/Lyft are not too expensive at off-peak times, my method of choice is public transit since I already have a monthly unlimited pass.</p><p>It usually takes around an hour to get between our apartments, given that I typically only make the trip on the weekends. I started this project to figure out when we can get between our apartments the quickest via public transportation, namely subway and bus.</p><p>The objective of this project is to collect the “best trip” information from the Google Maps API every 5 minutes for one week from 1) my apartment to my girlfriend’s and 2) her apartment to mine. I then use this information to find the best times to travel between our apartments via public transit.</p><p>In <a href="https://medium.com/@marcosanchezayala/optimizing-transit-travel-time-with-google-maps-part-1-d2fca97a9e3b">Part 1</a> of this post I introduced my underlying theory, experimental method, and assumptions necessary to make it run. Since writing that post, I’ve explored a number of alternatives to some of the methods I was considering then. Namely, I’ve opted to leverage my Google Cloud Platform Free Tier membership to spin up a Google Compute Engine VM instance and a Storage bucket for this project. But most other assumptions and limitations still stand.</p><p>I have built an ETL pipeline and a simple dashboard to visualize the results using Python and SQL. I’m going to walk through the ideas behind how I implemented this, as well as some discussion of why I made certain decisions. Lastly, I’ll also share some of my biggest challenges and future ambitions with the project.</p><h3>Outcome</h3><p>The dashboard I built displays some simple queries from a PostgreSQL database containing all my Google Maps API trips data. It compares trip durations between two locations, A and B (I won’t say which is which!), for different time breakdowns. This is a very basic implementation of the app, which I plan to improve upon for speed and better access to key metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Fmw2FWMX3bKLkASR6U5MrA.gif" /></figure><p>We can already see that the quickest average travel times (~45min) in either direction for any given day of the week are about 8–9 am and 4–7 pm. Furthermore, trips earlier in the week seem to be quicker than those later in the week, and traveling on weekdays generally seems faster than on weekends.</p><p>This is important information to know because it reveals that travel times are quickest around peak transit hours. This makes sense, as MTA probably adds more train service during hours that commuters are making their way to/from work. It is unfortunate though for my girlfriend and me, as we also will probably traveling to/from work at those hours. It seems that on average, we are stuck not taking the optimal travel times.</p><h3>Data Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJB3cTOlL8ylNdHuiuYVYw.jpeg" /><figcaption>Data Pipeline. Made with <a href="https://www.lucidchart.com/">Lucidchart</a></figcaption></figure><ol><li>Call Google Maps API for trip instructions between two locations for both directions of the trip (JSON) every 5 minutes.</li><li>Store each JSON in a Google Cloud Storage bucket acting as a data lake.</li><li>Pull down all data from the GCS bucket to my local machine.</li><li>Run a Docker container hosting PostgreSQL.</li><li>Create and populate a PostgreSQL database using the star schema shown below.</li><li>Run the Dash app, which pulls data from the PostgreSQL database.</li></ol><h3>Implementation</h3><p>I’ve written a number of Python modules to carry out the pipeline outlined above. You can see them all in the <a href="https://github.com/msanchez-ayala/google_maps">GitHub repo</a>.</p><h4>Collecting Data from Google Maps</h4><p>First, I wrote <strong>data_collection.py</strong>, which calls the Google Maps API for public transit directions between the two locations. Each individual trip is parsed for only information relevant to this study, and is saved as a JSON file in a public Google Cloud Storage bucket. Each individual trip JSON will look like this:</p><pre>{<br>  &quot;start_location&quot;: {&quot;lat&quot;: latitude, &quot;lng&quot;: longitude},<br>  &quot;start_location_id&quot;: &quot;A&quot;,<br>  &quot;departure_time&quot;: 1586809354,<br>  &quot;arrival_time&quot;: 1586812163,<br>  &quot;duration&quot;: 46,<br>  &quot;steps&quot;: [<br>    {&quot;step&quot;: 1, &quot;distance&quot;: 1490, &quot;html_instructions&quot;: &quot;Subway<br>     towards 8 Av&quot;, &quot;line_name&quot;: &quot;L&quot;}, <br>    {&quot;step&quot;: 2, &quot;distance&quot;: 4403, &quot;html_instructions&quot;: &quot;Subway <br>     towards Court Sq - 23 St&quot;, &quot;line_name&quot;: &quot;G&quot;}, {&quot;step&quot;: 3, <br>     &quot;distance&quot;: 9957, &quot;html_instructions&quot;: &quot;Subway towards Jamaica <br>     Center - Parsons/Archer&quot;, &quot;line_name&quot;: &quot;E&quot;}<br>  ]<br>}</pre><p>Data collection for this study was run entirely on a GCE VM instance that runs the required scripts every 5 minutes for 1 week. I scheduled a cron job to actually run a bash script <strong>data_collection.sh</strong>, which activates the virtual environment, defines a Google authorization environment variable, and executes <strong>data_collection.py</strong>. This is necessary because of GCP authorization restrictions, which dictate that we need to set the environment variable every time the cron job runs.</p><p>I chose to do this on a VM rather than my own machine because I was encountering significant authorization issues setting up a cron job to execute Python files on my mac. Furthermore, any cron job on my laptop would stop if it goes to sleep for any reason. I could set the screen to never sleep, but it’s possible I mistakenly close the laptop for a few hours without realizing. In order to avoid any of those issues, I just set up a free VM with my GCP free tier account! It all runs smoothly.</p><h4>ETL and Dashboard</h4><p>To initiate the ETL pipeline and start the Dash app, I first run the Docker container that will host PostgreSQL. Then, I run a bash script, <strong>app.sh</strong>, which carries out the etl. It executes the following commands in order:</p><ol><li><strong>download_storage.py</strong>, which is essentially the “extract” step of the ETL pipeline. It downloads all contents of the GCS bucket locally (~11 MB). This step can be slow (maybe a couple minutes) because we’re pulling each JSON individually.</li><li><strong>create_tables.py</strong>, which connects to the Docker container and creates the database and all tables we’ll need to populate.</li><li><strong>etl.py</strong>, which populates the database by transforming/loading the data acquired by <strong>download_storage.py</strong>.</li><li><strong>app.py</strong>, which is a Dash app to visualize some simple queries on data in our database.</li></ol><p><strong>Note:</strong> <strong>app.sh </strong>in the final repo has the first step commented out — it’s optional. I wanted to speed up the time my app takes to run, so I removed this step. Instead, I compressed all the data into a zip file that resides in the project repo and can be uncompressed by the user. Steps 2–4 will run properly.</p><p>However, if the user wishes to collect their own data, they have a couple options that are discussed in detail in the main project <a href="https://github.com/msanchez-ayala/google_maps/blob/master/README.md">README</a>. These are:</p><ol><li>Download files straight to their machine and skip the cloud step. Users would have to run the data collection script locally. They wouldn’t have to worry about <strong>download_storage.py</strong> because there’s no Storage bucket to download from!</li><li>Set up their own VM and Storage bucket like I did.</li></ol><h3>Database Schema</h3><p>Recall the JSON file we’re using has the format:</p><pre>{<br>  &quot;start_location&quot;: dict with coordinates,<br>  &quot;start_location_id&quot;: str,<br>  &quot;departure_time&quot;: timestamp,<br>  &quot;arrival_time&quot;: timestamp,<br>  &quot;duration&quot;: int,<br>  &quot;steps&quot;: list of individual steps<br>}</pre><p>I’ve translated these six fields into the following database schema. Transformations are accomplished in <strong>etl.py </strong>by a series of Python Datetime manipulations on the departure timestamp. Other fields are kept as is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZXXuWB5w8GIa5XmWLAy9Q.jpeg" /></figure><p>We have one fact table, trips, and three dimension tables, steps, locations, and time. Joining trips and time can answer most simple questions we have. For instance, what times of day will be quickest for travel? We only need the two aforementioned tables. However, we still have other tables such as steps that could tell us some stats about perhaps which train/bus lines are most common in each route. We could also find the times of day that will require fewest transfers. There are many possibilities!</p><h3>Challenges</h3><p>This was my first time setting up an end-to-end app like this that can be shared with other users. In fact, it was my first time ever using tools like Compute Engine and Storage. It was a huge challenge figuring out how to set up the environments correctly to orchestrate and harmonize all of the processes.</p><p>That being said, I a lot of fun learning everything and know that this experience will help greatly speed up any subsequent projects I develop that involve the same or similar tools. As an old music teacher of mine used to say, “this will never be as difficult or painful as it the first time you do it.”</p><h3>Further Directions</h3><p>This project has lots of potential. I’ve limited the scope severely so as to so as to produce something in a reasonable amount of time. Here are some ideas for expansion that I’d like to consider:</p><ul><li>Add to other modes of transportation (driving, walking, biking)</li><li>Try other locations. For instance, it could be used to find the optimal commute time.</li><li>There are some machine learning applications too such as combining more data sets such as weather and seeing how that affects ride times.</li><li>Ideally, I would set up a service to run all the ETL and app in the cloud so that the user doesn’t need to do anything themselves unless they want to conduct their own study. This would likely require more financial resources though, so I’ll hold off for now.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5c97a16b150" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL Views: What They Are and How to Make Them]]></title>
            <link>https://medium.datadriveninvestor.com/sql-views-what-they-are-and-how-to-make-62229b2f0fd7?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/62229b2f0fd7</guid>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 19 Apr 2020 23:51:21 GMT</pubDate>
            <atom:updated>2020-04-21T09:31:20.076Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*1iZZ4O2S52_p5YpvZbCntQ.jpeg" /><figcaption>“Views” - Drake album cover</figcaption></figure><h3>Motivation</h3><p>Consider the physical schema outlined below (check out <a href="https://medium.com/@marcosanchezayala/data-modeling-the-star-schema-c37e7652e206">my article</a> on star schemas if you’re unfamiliar with them!):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*IGdP28VX3cXhjKD-I6mGWA.png" /><figcaption><a href="https://www.guru99.com/star-snowflake-data-warehousing.html">https://www.guru99.com/star-snowflake-data-warehousing.html</a></figcaption></figure><p>If I’m interested in knowing the total revenue for each product in the current year, I would have to run this query (or something similar) every time I want to check it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e6a62b714365e37b9dfa2465e94ce7f/href">https://medium.com/media/4e6a62b714365e37b9dfa2465e94ce7f/href</a></iframe><p>It would be nice if SQL could just remember this query since it matters a lot to me. That’s where views come in!</p><h3>What is a View?</h3><p>Views are an incredibly useful SQL tool that unfortunately have nothing to do with Canadian rapper Drake (though I’m happy to discuss him at another time). They allow us to easily manage and query aggregated information from base tables. They come in a couple flavors: views and materialized views, both of which will I’ll discuss in this post.</p><p>On a more granular level, views are virtual tables computed dynamically from a stored query on tables in the database. By “virtual” table, we mean the view does not actually become part of the physical schema. The query defining the view is actually run <strong>each</strong> <strong>time</strong> the view is referenced in another query. Materialized views, however, are an exception.</p><h3>Why views and not tables?</h3><p>You could argue it would make sense to just create a new table in the database containing the desired aggregations, but you would be responsible for keeping that table synchronized. This could involve lots of code and testing, and still even allow for the potential of holes after everything.</p><p>Views offer the following advantages over tables:</p><ul><li>Views are automatically synchronized with the underlying tables used to make them.</li><li>They can represent a subset of the underlying data and thus can be used to limit the degree of exposure of the underlying information to certain users.</li><li>They take very little space to store since the database stores only the definition of a view and not a copy of all the data in it.</li><li>They are incredibly simple to create.</li></ul><p><a href="https://www.datadriveninvestor.com/2020/02/19/five-data-science-and-machine-learning-trends-that-will-define-job-prospects-in-2020/">Five Data Science and Machine Learning Trends That Will Define Job Prospects in 2020 | DataDrivenInvestor</a></p><h3>How to create a view</h3><p>The basic syntax is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20c91a7757ecb757057a7d9743994921/href">https://medium.com/media/20c91a7757ecb757057a7d9743994921/href</a></iframe><p>We basically just state that we want to create a view, give it a name, and then follow it with a typical SELECT statement.</p><p>In the case of the example at the beginning of this post, we would create the view like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ea649e2d55c3b119be8aee841cdc875/href">https://medium.com/media/8ea649e2d55c3b119be8aee841cdc875/href</a></iframe><p>We can then access the data within that view using a simple SELECT statement.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ecc43b21dc5124e3dc897f3150cad9d/href">https://medium.com/media/8ecc43b21dc5124e3dc897f3150cad9d/href</a></iframe><p>Every time you write a query referencing a view, the database management system (DBMS) executes it as a subquery. In the case above, product_revenue is substituted with the SELECT query within the view definition.</p><p>You could also add other conditions to the above query as you would with any SELECT statement.</p><h3>How to alter or remove a view</h3><p>Altering a view consists of just <strong>ALTER VIEW</strong> view_name (alter command).</p><p>Removing a view is as simple as <strong>DROP VIEW [IF EXISTS] </strong>view_name. We can add in IF EXISTS because trying to drop a non-existent view will throw an error.</p><h3>Additional notes on views</h3><p>Views can be created from other views. This makes database normalization above second normal form much simpler.</p><p>Rows in a view lack ordering, just as the rows in its base tables do. Thus, an ORDER BY clause would be useless in a CREATE VIEW statement. In fact, the SQL standard doesn’t even allow you to use an ORDER BY clause in the CREATE VIEW statement, just as it does with in a CREATE TABLE statement.</p><p>Views can be either read-only or updatable. The idea is that only updatable views will allow the database system to reverse map the view schema to the schema of the underlying tables, allowing for INSERT, UPDATE, and DELETE operations. Different DBMSs (e.g Oracle, MySQL, and PostgreSQL) have different implementations of views. I highly recommend checking out the appropriate documentation to make sure you’re doing everything specific to your DBMS.</p><p>We can also create <strong>materialized views</strong>, which are pre-executed, non-virtual views that are commonly used in data warehousing and business intelligence applications. They work in pretty much the same way as normal views, but their accuracy depends on the frequency of trigger mechanisms behind its updates. Again, the implementation an exact functionality depends on your DBMS, so go check your documentation!</p><p>I’ve spoken with several data engineers over the past months who have raved about views and materialized views. They’re clearly being used all the time! I’m still looking for my first data job, so I haven’t had a chance to use them myself, but I figured I’d take a stab at learning the theory behind them at least. They cleraly offer many advantages to normal tables, and are easy to make. I definitely plan to use them in my next project as practice!</p><h3>Sources</h3><ul><li><a href="https://www.postgresql.org/docs/12/sql-createview.html">PostgreSQL documentation</a></li><li><a href="https://en.wikipedia.org/wiki/View_(SQL)">Views (SQL) - Wikipedia</a></li><li><a href="https://dba.stackexchange.com/questions/23280/why-not-use-a-table-instead-of-a-materialized-view">Awesome Stack Overflow Answer</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62229b2f0fd7" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/sql-views-what-they-are-and-how-to-make-62229b2f0fd7">SQL Views: What They Are and How to Make Them</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Transit Travel Time with Google Maps: Part 1]]></title>
            <link>https://medium.com/@marcosanchezayala/optimizing-transit-travel-time-with-google-maps-part-1-d2fca97a9e3b?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2fca97a9e3b</guid>
            <category><![CDATA[google-maps]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 12 Apr 2020 21:30:11 GMT</pubDate>
            <atom:updated>2020-04-16T15:14:21.714Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing Travel Time with Google Maps: Part 1</h3><h4>Experiment Design</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wLHnOSwvCDrZtbao" /><figcaption>Photo by <a href="https://unsplash.com/@hjkp?utm_source=medium&amp;utm_medium=referral">henry perks</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>My girlfriend and I live in two different NYC boroughs, making travel between our apartments a real pain. Although Uber/Lyft are not too expensive at off-peak times, my method of choice is public transit since I already pay for a monthly unlimited pass.</p><p>It usually takes around an hour to get between our apartments, given that I typically only make the trip on weekends. I generally come and go at random times, and had never given much thought as to which times of day would be quickest for travel. Eager to put some new ETL skills I gained through Udacity’s Data Engineering nanodegree program into practice, I set out to study when we can get between our apartments the quickest via public transportation, namely subway and bus.</p><h4>Hypothesis</h4><p>I predict the quickest travel times between our apartments will be during rush hour times because train availability is maximized in these times. However, it would be super inconvenient for us if that really is the best time to travel between our apartments because it would never make sense for us travel to each other’s place at 8 am on a Tuesday, for instance. Thus, I hope this hypothesis is wrong but it does seem to make the most sense as of right now.</p><h4>First Thoughts</h4><p>Initially in designing this experiment I thought I would want to access the MTA’s API, since they run NYC’s public transit and would probably have the most accurate bus and train information. However, the trip between our apartments usually requires multiple transfers and a bit of walking. Since I’m interested in capturing all the different transfers, as well as the door-to-door time, it makes sense to instead use a service like Google Maps which easily does all of this. Part of their API allows for visualization of such trips along with other cool functionality, so it made sense to go with this instead.</p><p>I quckly found a handy Python driver to access Google Maps’s API called <a href="https://pypi.org/project/googlemaps/1.0.2/">googlemaps</a>. After a little exploring, I found that all of the information I would need for this study could indeed be found with Google Maps data. I just have to make a couple assumptions about what I’m gathering.</p><h4>Experimental Method</h4><p>It makes most logical sense to me now set up an experiment that follows this sequence of steps:</p><ol><li>Call the Google Maps API for public transit directions between the two locations every 5 minutes. Each set of directions should be parsed and then saved locally as a JSON file. I can afford to save this locally for now since there won’t be much data and I don’t want to pay for storage online. Eventually I could batch upload everything to S3 or Cloud Storage to minimize the number of transactions.</li><li>After a week when all data is collected, run an ETL script to push the information from the JSON files to a PostgreSQL database. I plan to host it in a Docker container.</li><li>Display sample queries from the database in a simple dashboard using Dash by Plotly.</li></ol><p>From the visualizations in the dashboard we’ll be able to answer pretty much any questions we might have about the data, namely when is it fastest to travel between our apartments?</p><h4>Assumptions</h4><ol><li><strong>The data is accurate.</strong> While I believe Google has a great service and the most up-to-date information they can gather, the NYC subway is unpredictable. You never know when delays will strike. I use Google Maps every single day getting from place A to B in NYC using public transit, and can say that the trip duration estimates are usually correct within a minute or two with reliable train lines. For simplicity sake, this study assumes that the trip durations are always accurate.</li><li><strong>Sufficient “best trips” can be captured with a 5-minute API call interval.</strong> There are two motivations for this. First, the trains around me usually don’t even run any quicker than in 3 minute intervals so it shouldn’t make a big difference to not capture every single possible trip. Second, I have a finite amount of Google Cloud credits, so I chose to not call Google’s API at a smaller time interval like every minute.</li></ol><h4>Implementation</h4><p>I’m developing this project using Python and you can see it on <a href="https://github.com/msanchez-ayala/google_maps">GitHub</a>! Here’s what’s inside.</p><ol><li>The first module I wrote was <strong>directions.py</strong>, where I define the Directions class that handles connections to the Google Maps API, parsing information, and exporting to JSON.</li><li>Next, <strong>data_collection.py</strong> instantiates the Directions object for trips between the two locations and stores the JSON files in a subdirectory.</li><li><strong>sql_queries.py </strong>contains all queries to handle DROP, CREATE, and INSERT statements for all of the tables that will be stored within PostgreSQL.</li><li><strong>create_tables.py</strong> calls <strong>sql_queries.py</strong> to make all the tables we’ll need.</li><li><strong>etl.py</strong> extracts the JSON data and transforms it into the form that is then loaded into PostgreSQL.</li><li><strong>app.py </strong>will become the Dash app that will display the dashboard.</li></ol><h4>Next Steps</h4><p>Many things need to happen!</p><ul><li>I need to keep developing <strong>directions.py</strong>, which currently is not in its most modular form. It works fine now, but also will need to be tested for best usage.</li><li>I haven’t yet set up the scheduler to run <strong>data_collection.py</strong> every 5 minutes. I’ll see if I can do this with cron or some other scheduler.</li><li><strong>etl.py </strong>needs to be rewritten to do what is stated above.</li><li><strong>app.py</strong> needs to be fully implemented.</li></ul><h4>Extensions</h4><p>I’m really excited to finish the above next steps in the coming days! I’ll be able to extend this project to more than just public transit. Google Maps, as we know, also works for walking, driving, and biking directions. So I could easily extend this project to do the same. Collecting all that data would allow someone to perform all sorts of predictive analytics on travel times in the future.</p><p>I plan to write some follow up posts to show off the product once it’s up and running to discuss how it works and its limitations. Let me know if you have any ideas or want to contribute!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2fca97a9e3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Modeling: The Star Schema]]></title>
            <link>https://medium.com/@marcosanchezayala/data-modeling-the-star-schema-c37e7652e206?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c37e7652e206</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[star-schema]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 05 Apr 2020 23:31:43 GMT</pubDate>
            <atom:updated>2020-04-08T18:22:50.978Z</atom:updated>
            <content:encoded><![CDATA[<p>Data modeling is a crucial step in modern data workflows, as its purpose is to organize raw data into convenient and efficient forms. Data analysts and scientists will find their jobs much easier if a usable dataset is readily accessible. Quicker analytics and predictions will then lead to faster insight for business decisions.</p><p>The first step to modeling is often to normalize the data, which is a process of organization that increases database flexibility by reducing inconsistent dependencies and redundancy. I’d suggest <a href="https://en.wikipedia.org/wiki/Database_normalization">reading up</a> on this and/or looking up some videos if you’re unfamiliar! The problem with a normalized database is that any truly interesting insights from the data will require many JOINS, which can significantly slow down the speed of our query as the size of our database increases. For instance, looking at the schema below, most tables are not directly related. This means that to connect information from two tables like Orders and Location we’d need a minimum of 4 JOINS (Orders -&gt; Employment -&gt; Person -&gt; Phone_Number -&gt; Location is one way to arrive there)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*b6gU9z0GAjQ10TG0hBxPkQ.png" /><figcaption><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-relational-modeling.html">https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-relational-modeling.html</a></figcaption></figure><p>Now what if we need data from even MORE tables after that 4-part JOIN? It would be madness. Not to mention, it would be an absolute headache to just write the query without any errors.</p><p>Furthermore, real-life databases can have far more tables than just the ones shown in the example above. As you can imagine, it becomes increasingly difficult to even understand the relationships between tables as our schema grows.</p><h3>The Star Schema</h3><p>One solution to this problem is to perform a denormalization step of data modeling to create a simpler and easy-to-understand schema optimized for ceratin queries. The process of creating a star schema involves distilling down our full schema into just relevant features for a particular analytic prupose. The general structure of the star schema is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUhA6fbkoEtFoiIVXHzSlg.png" /><figcaption><a href="https://docs.microsoft.com/en-us/power-bi/guidance/star-schema">https://docs.microsoft.com/en-us/power-bi/guidance/star-schema</a></figcaption></figure><p>The star schema consists of two types of tables:</p><ol><li>Facts: Metrics of a business process. These are generally numeric and additive (e.g. amount of an invoice or the number of invoices), or quantitative. The fact table also contain keys pointing to relevant dimension tables. There is just one fact table at the center of the star schema.</li><li>Dimensions: The where, when, what, etc. (e.g. date/time, locations, goods sold). These typically contain qualitative information. There are multiple dimension tables in the schema, all of which are related to the fact table.</li></ol><h4>Advantages</h4><ul><li>A simplified schema means that we won’t have to write confusingly long queries every time we want some information from our database.</li><li>We optimized for reads. Now that we can write fewer JOINS, our results will be returned more quickly.</li><li>Also, it will business logic for reporting. We won’t have to explain to stakeholders all the crazy joins that went into making the schema, just maybe .</li></ul><h4><strong>Disadvantages</strong></h4><ul><li>Denormalizing our data means that data anomalies could arise from one-off inserts or updates. In practice, star schemas are implemented via “trickle feeds” or batch processing to compensate for this issue.</li><li>We have limited analytical flexibility. A star schema is generally designed for a particular purpose. Since we have fewer features in the star schema than in the full database, we are restricted to just what this star schema contains.</li></ul><h4>Example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*Th43tG5qbAGKTZzr7oITNg.png" /><figcaption><a href="https://www.guru99.com/star-snowflake-data-warehousing.html">https://www.guru99.com/star-snowflake-data-warehousing.html</a></figcaption></figure><p>Let’s consider a database of sales for a store. We have a fact table, Revenue, at the center of the schema, and four dimension tables.</p><p>The fact table is comprised of a composite primary key, which is the combination of the dimension table primary keys. Fact table non-primary keys Units_Sold and Revenue are the facts we’re interested in, and the dimensions such as Product_Name and Name (branch name) allow us to understand more information about the goods sold.</p><p>For instance, the following query would allow us to calculate the total revenue by product in the year 2010:</p><pre><strong>SELECT</strong><br>  p.Product_Name AS product,<br>  <strong>SUM</strong>(r.Revenue) AS total_revenue<br><strong>FROM</strong><br>  Revenue r<br><strong>JOIN<br>  </strong>Product p <strong>ON</strong> (r.Model_ID = p.Model_ID)<br><strong>JOIN<br>  </strong>DateDim d <strong>ON</strong> (r.Date_ID = d.Date_ID)<br><strong>WHERE<br>  </strong>d.Year = 2010<strong><br>GROUP BY<br> </strong> p.Product_ID</pre><p>The star schema is widely used and incredibly useful for business applications. It helps us speed up queries that we may run often and clean up what could otherwise be very messy queries, among other things.</p><p>There are other schemas such as the snowflake and galaxy schemas that are simple extensions of the star schema. If you like the star schema, I recommend checking the others out too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c37e7652e206" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Online Learning in the Age of Coronavirus: Data Engineering]]></title>
            <link>https://medium.com/@marcosanchezayala/online-learning-in-the-age-of-coronavirus-data-engineering-276b698b5f00?source=rss-ec310f88b5d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/276b698b5f00</guid>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[datacamp]]></category>
            <dc:creator><![CDATA[Marco Sanchez-Ayala]]></dc:creator>
            <pubDate>Sun, 29 Mar 2020 22:42:01 GMT</pubDate>
            <atom:updated>2020-03-29T22:42:01.292Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_Kl6fLrWKYcnsPOVqHyCQ.jpeg" /><figcaption><a href="https://analyticsindiamag.com/data-engineering-trending-now/">https://analyticsindiamag.com/data-engineering-trending-now/</a></figcaption></figure><p>Recently, I’ve developed quite an interest in data engineering. I’ve found the silver lining amidst the coronavirus pandemic to be that I get a unique chance to invest as much time as I want into learning. Although it’s difficult to stay home (self isolate!!) every day, I’m learning many new practical skills that I feel will propel my career forward once we emerge back from these troubling times.</p><h4>Why data engineering?</h4><p>I’m a recent data science bootcamp graduate who is actively job searching. I graduated from college in 2018, so the combination of my skillset straight out of the bootcamp and total professional experience (which is little) makes me eligible for entry-level data analyst positions. Professionals in the industry have warned me to be skeptical of data analyst jobs that will mainly have me doing point-and-click analysis in BI tools like Looker or Tableau, as those roles will not develop my data science abilities.</p><p>As I spoke with more and more data scientists, I often found myself telling them that my favorite part of my data science projects was setting up my ETL pipeline. I love getting my hands dirty in the code to extract data from an online source, primarily via webscraping or an API. I relish the moment I finally fix a bug that had been preventing me from progressing for the past couple hours. I am pretty OCD about keeping the code neatly organized, legible, and well-documented.</p><p>You may be familiar with some form of the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3RTDD5SF0hbIkTh85ksrMw.png" /><figcaption>Created by <a href="https://medium.com/@mrogati">Monica Rogati</a></figcaption></figure><p>Although I didn’t quite understand at first, my real interests lie in data engineering. I went through a data science bootcamp to figure out I would rather be a data engineer. Kind of funny when I say it like that!</p><p>It makes sense though. Even during my bootcamp, I was never really too fond of machine learning, statistical testing, or AI. While I certainly appreciate the power of predictive modeling, I also realize that I lack much of the mathematical foundation to truly understand and fine tune predictive algorithms. I could learn it, but I’d prefer to pursue something that excites me. This is 100% the case with data engineering.</p><h4>Resources</h4><p>While many of the skills I learned in my data science bootcamp are transferable, there are many technologies to learn. For instance, data engineers need to be well-versed in ETL tools, cloud infrastructure services, database administration, etc. I’m able to generally pick up new things pretty quickly, but the world of data engineering is pretty daunting given the number of tools in the data engineer’s stack.</p><p>I thus sought out online resources to aid in this process, and found two wonderful and free (during a trial) sites. These are Datacamp and Udacity. Let’s look at these more in detail.</p><h3>Datacamp</h3><h4>Overview</h4><p>This is a hands-off paid subscription service designed to teach data science to anyone. What I mean by that is that there are several suggested curriculums laid out for you, but you can go through at your own pace to pick and choose whatever content you want to learn in whatever order. It’s nice if you’re interested in diving deep or simply supplementing a couple skills here and there</p><p>Datacamp is a wonderful resource. I would highly recommend it to anyone looking to supplement their data science knowledge. It covers different topics such as programming fundamentals (in R, Python, and Scala), math, data visualization, machine learning, Git, software engineering fundamentals, etc.</p><p><strong>$$$: </strong>You can get a 3-month free trial of Datacamp if you sign up for the <a href="https://education.github.com/pack">GitHub student pack</a>. For those who are no longer students or who can no longer reap the benefits of their university email addresses, one can also get a 2-month free trial through the instructions in this <a href="https://www.presentslide.in/2019/09/free-datacamp-subscription-courses.html">link</a>.</p><p><strong>Organization:</strong> I consider the basic building block of their content to be a “course”, which is generally a 4-chapter learning module. Each chapter takes around an hour to complete, and usually consists of an introduction video followed by a mix of instructional videos, informational pages of text, and interactive exercises that build progressively throughout the chapter.</p><p>Multiple courses are grouped together into tracks, which Datacamp puts together to present material in a logical and progressive manner. There are two types of tracks, skills tracks and career tracks. I chose the Data Engineer career track, which currently has 75 hours of content. Incredible!</p><p>There are many tracks, and I suggest you look into them yourself to check it out. The first chapter of most courses are free for exploring even without a subscription! This allows you to poke around before committing to anything.</p><h4>Outcomes</h4><p>I took a handful of courses within the data engineering track over my first weekend with my trial. I took Introduction to Data Engineering, Software Engineering for Data Scientists in Python, Introduction to Git, Introduction to PySpark, and JOINS in PostgreSQL.</p><p>My goal was to get my feet wet into the world of data engineering, and I feel like I accomplished quite a lot. Each course exposes the user to real-word applications of the tools. I was able to learn things like setting up an ETL pipeline using PySpark, automating with Airflow, familiarizing myself with PostgreSQL, and much more.</p><p>I was very entertained throughout the whole thing, which I think is a testament to the strong presentation of the material that Datacamp puts together. The interactive nature of the exercises is nice too. I have done a little bit of work on Code Academy. It felt kind of like that but just more focused to data science topics. Furthermore, I think the material is split up into very digestible chunks, so as not to throw too much at you at once without allowing you to practice.</p><p>There are MANY things I could keep learning on this site, but given that there are many other resources out there to learn, I decided to shop around a bit. This led me to Udacity, which a fellow ex-bootcamper recently told me about.</p><h3>Udacity</h3><h4>Overview (Udacity, free coronavirus trial)</h4><p>Udacity is another paid subscription service. Unlike Datacamp, you enroll in Nanodegree programs, which are similar to Datacamp’s “tracks”, but are more structured and include actual projects for you to submit for feedback from actual humans. It’s a strong platform.</p><p><strong>$$$:</strong> They have several programs, but I chose the Data Engineering nanodegree. They say that this program should take roughly 5 months if working 5–10 hours per week. I only have one month to use for free before paying something like $400/month for the service, which I certainly do not have (especially now). If you do the math, this means they recommend anywhere from 100–200 hours total. Thus, I could theoretically complete the nanodegree putting in between 25–50 hours per week. Luckily, I don’t have a job and coronavirus is keeping me at home while I continue my search. It might be possible! I’ll write a blog post later updating on my status…</p><p><strong>Organization:</strong> The curriculum is laid out into four sections plus a capstone project: Data modeling, cloud data warehouses, data lakes with Spark, and data pipelines with Airflow. I’m super excited to get through everything I can. Each section is broken up into several modules and at least one project to practice all the skills learned. You submit the projects upon completion and get some feedback, which is a really nice feature that Datacamp does not offer.</p><p>They also have this thing called Knowledge, which is like a forum for students to ask each other questions which can be answered by Udacity instructors or other students. I’ve found this pretty helpful, although most of my questions have been answered through Google.</p><p>The main value to me in this program is the fact that there are projects I can complete and show to a prospective employer. My professional experience in the industry is nonexistent, so I think that giving solid proof that I can build something is the best way to demonstrate to employers how quickly I can adapt and learn new technologies.</p><h4>Outcomes</h4><p>After just a few days, I’m close to finishing the data modeling section. I’m working on a data modeling project to build an ETL pipeline for mock music streaming data into a PostgreSQL database. Students have the option to work in a Udacity-hosted workspace or to develop the code locally and submit a GitHub repo.</p><p>I opted for the challenge of developing locally. There won’t be training wheels forever, and I wanted to gain real-world experience developing my code locally and then making available to others with Docker. Additionally, I’m keeping in mind software engineering best practices and trying to minimize dependence on Jupyter Notebooks, which were heavily emphasized in my bootcamp.</p><p>My goal is to get through the four sections in this program. If I don’t get to the capstone project then oh well. I can make my own capstone project and ask a more experienced data engineer to give me feedback on it later.</p><p>I really like Datacamp and Udacity. I highly recommend them to anyone looking to do something productive during these coronavirus times, especially since there are ways to start out with them for free. Plus, these are just two of MANY online learning platforms out there. I’m sure if you look around, you’ll be able to find other free trials or heavily discounted subscriptions in light of the coronavirus.</p><p>Udacity</p><p>Datacamp</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=276b698b5f00" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>