<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by J.P. Rinfret on Medium]]></title>
        <description><![CDATA[Stories by J.P. Rinfret on Medium]]></description>
        <link>https://medium.com/@jprinfret?source=rss-3b361e3c4389------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*qFJfG8leNsqXeLvFiEX5fg.jpeg</url>
            <title>Stories by J.P. Rinfret on Medium</title>
            <link>https://medium.com/@jprinfret?source=rss-3b361e3c4389------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:12:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jprinfret/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Monty Hall Problem Solved]]></title>
            <link>https://levelup.gitconnected.com/the-monty-hall-problem-solved-ec4f3753a682?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec4f3753a682</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[monty-hall]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Tue, 25 Feb 2020 00:40:31 GMT</pubDate>
            <atom:updated>2020-02-27T01:59:46.750Z</atom:updated>
            <content:encoded><![CDATA[<p>Using Python to generate ten-thousand random simulations of the classic game show.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lpcs2xMjYqQ3KfIS" /><figcaption>Photo by <a href="https://unsplash.com/@mitchel3uo?utm_source=medium&amp;utm_medium=referral">Mitchell Luo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I was recently watching the movie <em>21</em> and came across the famous scene in which the main character Ben, played by Jim Sturgess, explains the classic Monty Hall problem to his MIT professor and soon to be card counting couch Micky Rosa. The scene can be found <a href="https://www.youtube.com/watch?v=cXqDIFUB7YU">here</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcXqDIFUB7YU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcXqDIFUB7YU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcXqDIFUB7YU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba621cbda37f9da89a8d5140db46e61d/href">https://medium.com/media/ba621cbda37f9da89a8d5140db46e61d/href</a></iframe><h3>The Problem</h3><p>You are a contestant on a game show. On the stage in front of you are three doors. Behind two of the doors are goats, behind the remaining door is a Ferrari.</p><p>At the start of the show, the devilishly handsome host asks you to pick a door (let’s say you pick door 2). The host then opens one of the other doors (1 or 3) and, unsurprisingly, reveals a goat. Since the host knows the location of the Ferrari a goat is revealed at this point every single round. The host then asks you if you would like to stay with your original pick (door 2) or switch to the remaining door.</p><p>Is it in your best interest to stay with your original guess or to switch? Is the host using reverse psychology to influence your guess (since he knows the location). Do you even care?</p><h3>The Solution</h3><p>Like all good probability questions, we should first think about it logically. A lot of people (incorrectly) assume that you have a 50/50 chance of winning because technically you are randomly picking one out of the two remaining doors in the second round (the door you picked and the door the host did not reveal). However, I prefer a more disciplined approach; let’s examine the probabilities of winning for each approach (switch vs. stay) and see if we can get a probability greater than 50% to render the 50/50 argument once and for all useless.</p><p>What are the odds of winning at the onset of the game show, if you stayed with your original pick of door 2? Well, since there are two goats and one Ferrari (and I am assuming that walking away with a Ferrari over a goat is considering winning — unless of course paying taxes on a brand new Ferrari does not excite you) the probability of winning is 1/3 and losing is 2/3.</p><p>Now what if we switched every single time you were given the chance? In order to win, you would first have to pick a goat and then switch your pick to the Ferrari (if you switched away from the Ferrari you would obviously lose). In other words, if you would were to pick a goat at the onset you would switch to the Ferrari and win (remember the host always eliminates the other goat so you can never switch from one goat to the other goat!). So by this logic, what is the probability of originally picking a goat? That is right, 2/3.</p><p>Okay great, we have logically reasoned that the probability of winning if you were to switch is 2/3, which is greater than the probability of winning if you were to stay with your original guess each time (1/3).</p><p>Does this logic hold if we were to simulate multiple random environments where we either switch or stay? Let’s write some code to find out.</p><h3>The Code</h3><p>First, we will use Numpy to generate the random environments (or trials) and guesses and Matplotlib and Seaborn to visualize the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUTfQspAHXfNNvF_KPmUUw.png" /></figure><p>Next, we will create stay and switch containers. These lists will hold 1s and 0s depending on a win or loss, respectively. Also, let’s create our doors while we are at it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MI-Hrtz-q_t_KBcwq3gSIQ.png" /></figure><p>Now for the fun part.</p><p>For each “environment”, we will consider ten-thousand of them, we need to randomly place the Ferrari (“ferrari_door”) and randomly assign an original guess for the contestant (“contestant_door”). Note, we do not need to randomly assign the goats because the goats are automatically behind the two doors that the Ferrari is not (“goat_doors”).</p><p>Next, we need a list of doors that the host could choose from for his big reveal (“possible_reveal_doors”). We also know that the ferrari_door is not one of the elements of possible_reveal_door (since the host will never reveal the Ferrari’s location). Possible_reveal_doors can be of length 2, option A, or length 1, option B. For option A to occur, the contestant_door must equal the ferrari_door. Since which door the host reveals (“reveal_door”) ultimately doesn’t matter, as long as it is a goat, we will randomly select one of the doors for the host to reveal. For option B to occur, the contestant_door must be one of the original goat_doors (because you cannot switch to a goat since the host will reveal the remaining goat). Since there is only one element in this list, we grab it.</p><p>Finally, let’s define a “switch_door,” which represents the remaining door (not the original contestant_door and not the reveal_door). This switch_door is defined for each environment regardless of whether or not our strategy is to switch or stay.</p><p>So now let’s focus in on our strategy to win the game show. If we blindly switch every environment and the switch_door equals the ferrari_door, then we win. Each time this happens we will append a 1 to the switch container and a 0 to the stay container. If we blindly stay every environment and the switch_door does not equal the ferrari_door, then we lose. Each time this happens we will append a 0 to the switch container and a 1 to the stay container. Once we cycle through each environment, we will have a switch container with ten-thousand elements and a stay container with ten-thousand elements.</p><p>Let’s examine the code for this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*39PBVjfB2yF2jAO8nVMKzw.png" /></figure><p>Now we must calculate the probability of winning for each strategy. This is simply (Number of Wins)/(Total Environments). Since every time each strategy won we appended a 1 (and every time it lost we appended a 0) to the respective container, the above numerator equals the sum of each list. So the probability of winning equals sum(container)/len(container) or simply the mean value of each list. That code is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tcYJqTSCyFLAYLosF6K1pw.png" /></figure><p>As you can see, the probability of Switch winning equals 66.68% (about 2/3) and the probability of Stay winning equals 33.32% (about 1/3)</p><h3>The Visualization</h3><p>The below block of code attempts to plot the probability of winning after each consecutive environment. This probability rapidly approaches our theoretical probability of 2/3 for Switch and 1/3 for Stay (less than 500 trials).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IaqVoJQd7zSn1yNHZzD35g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*yutOzJz3K0_CtZyx9oivsQ.png" /></figure><h3>The Summary</h3><p>As you can see, we have simulated an environment where we randomly assigned the Ferrari behind one of three doors, had our contestant randomly select one of the three doors, had the host reveal a door hiding a goat (and never a Ferrari), and then assigned a 0 or 1 to the appropriate container depending on whether or not the switch strategy would have won or the stay strategy would have won. We then did this ten-thousand times and then calculated the probability of winning if we consistently stayed or switched.</p><p>So there you have it. If you are disciplined in your approach and switch your guess every single time, your probability of winning rapidly approaches 2/3.</p><p>Please visit my <a href="https://github.com/jprinfret/monty_hall_problem">GitHub</a> for the full jupyter notebook. As always, I welcome your comments or questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec4f3753a682" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/the-monty-hall-problem-solved-ec4f3753a682">The Monty Hall Problem Solved</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The MySQL Functions Used in (Almost) Every ETL Pipeline]]></title>
            <link>https://medium.com/data-science/the-mysql-functions-used-in-almost-every-etl-pipeline-d2007d6f5086?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2007d6f5086</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Mon, 17 Feb 2020 17:31:24 GMT</pubDate>
            <atom:updated>2020-02-26T15:34:59.088Z</atom:updated>
            <content:encoded><![CDATA[<h4>Commit these functions to memory.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EHrfI-gbW4_RFv7P" /><figcaption>Photo by <a href="https://unsplash.com/@realaxer?utm_source=medium&amp;utm_medium=referral">tian kuan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction to an ETL Pipeline</h3><p>What does “ETL” stand for? To answer this, let’s consider the 80/20 rule that every Data Scientist knows: a data scientist’s time is allocated 80% to <strong>E</strong>xtracting, <strong>T</strong>ransforming, and <strong>L</strong>oading data and 20% to simultaneously watching The Office re-runs and Amazon Prime’s Jack Ryan as to juxtapose John Krasinski’s incredible range.</p><p>In this blog, we will examine some of the most frequently used MySQL functions, five to be exact and in no particular order, that every SQL beginner should have committed to memory.</p><h3>The Pipeline</h3><p>Since the purpose of this blog is to focus on the frequently used SQL commands, we will only be focusing on the data extraction process of the pipeline.</p><p>To kick us off, let’s import Pandas and Sqlite3 and define a “cursor.” The cursor object allows us to call the .execute() method to perform SQL commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*PfBz7ZE-bA2QjtSTRh-qPg.png" /><figcaption>The cursor object allows us to call the .execute() method to perform SQL commandas</figcaption></figure><p>Now that we are connected to our database, which is a local file called “data.sqlite”, we can start extracting the data. But first, let’s examine the database schema:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*lbGKGnZ5v7BxPsQwxpBkCw.png" /><figcaption>Our databases’ schema</figcaption></figure><p>As we can see, we have multiple tables some of which have one-to-one relationships and some have one-to-many relationships (more on that <a href="https://www.youtube.com/watch?v=lDnL1gwCE0o">here</a> and <a href="https://www.youtube.com/watch?v=KjA2LhT4TRU">here</a>).</p><h3>Function 1: SELECT</h3><p>The <strong>SELECT</strong> function is what allows us to grab the desired data. Let’s say we wanted every column in the orderdetails table. We could write a query like the below:</p><pre>query = &quot;&quot;&quot;SELECT od.orderNumber,<br>                  productCode,<br>                  quantityOrdered,<br>                  priceEach,<br>                  orderLineNumber<br>            FROM orderdetails;&quot;&quot;&quot;</pre><p>However, instead of writing out each column we can simply use an asterisk to grab every column.</p><pre>query = &quot;&quot;&quot;SELECT * FROM ordertails;&quot;&quot;&quot;</pre><p>Now if your data contained duplicate entries, you can call <strong>DISTINCT</strong> to grab only unique data points before putting the data into a dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*6vCJHQzitkYuykOuq-EAeQ.png" /><figcaption>A SELECT DISTINCT query example</figcaption></figure><h4>Aggregator Functions: COUNT(), SUM(), MIN(), MAX(), AVG()</h4><p>Notice that the first four rows of the above dataframe have the same orderNumber. We can see that each orderNumber contains multiple productCodes, each with their own quantity and price. Let’s say for each order we wanted to know the total amount the customer spent — notice we do not have a total column for each product. This is where aggregator functions can come in handy. Note, when using aggregator functions, it is very import to always include the <strong>GROUP BY</strong> function (more on this later) so that it is known what to aggregate the data on, in this case orderNumber.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*NPTAwTyaCQZbet82HDZ2zQ.png" /><figcaption>The aggregator function SUM() and alias function AS query example</figcaption></figure><p>Using the <strong>SUM()</strong> function allowed us to create a new column (using the alias function <strong>AS</strong>) called “orderTotal,” and we can see that the first five orders were for $10,222.83, $10,549.01, $5,494.78, $50,218.95, and $40,206.20. Quite useful.</p><h3>Function 2: JOIN</h3><p>The <strong>JOIN</strong> command is used to select multiple rows from different tables using a common column (a.k.a. the foreign key). We will focus on the four most common join clauses: outer, inner, left, and right. Note that you must define the common column with either <strong>ON</strong> or <strong>USING</strong>. The syntax for <strong>ON</strong> would be … ON table1.common_column_name = table2.common_column_name whereas the syntax for USING is simply … USING (common_column_name).</p><h4>OUTER JOIN</h4><p>An <strong>OUTER JOIN </strong>returns data where the is a match in either the left or right table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*eMdkvb2v1DSjlcTawB3Feg.gif" /><figcaption>Image courtesy of: <a href="https://www.w3schools.com/sql/sql_join.asp">https://www.w3schools.com/sql/sql_join.asp</a></figcaption></figure><h4>INNER JOIN</h4><p>An <strong>INNER JOIN</strong> returns data where the is a match in both the left and right table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*rE514WQ54gQXmsqfPoewkQ.gif" /><figcaption>Image courtesy of: <a href="https://www.w3schools.com/sql/sql_join.asp">https://www.w3schools.com/sql/sql_join.asp</a></figcaption></figure><h4><strong>LEFT JOIN</strong></h4><p>A <strong>LEFT JOIN</strong> returns all records from the left table plus the matching rows from the right table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*NRJd8DqOt5GqEoRHMZC5gw.gif" /><figcaption>Image courtesy of: <a href="https://www.w3schools.com/sql/sql_join.asp">https://www.w3schools.com/sql/sql_join.asp</a></figcaption></figure><h4>RIGHT JOIN</h4><p>A <strong>RIGHT JOIN</strong> returns all records from the right table plus the matching rows from the left table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*Ahh8nZMACTI0uWwSkoa-pA.gif" /><figcaption>Image courtesy of: <a href="https://www.w3schools.com/sql/sql_join.asp">https://www.w3schools.com/sql/sql_join.asp</a></figcaption></figure><p>Now let’s walk through an example. Using our previous schema, let’s say we wanted the employee number, first name, last name, and zip code of every employee. We could perform a simple <strong>INNER JOIN</strong> of the employees table and offices table using officeCode as our foreign key (this is just one of many ways to do this).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*fVM1iao5Gs2fOMHU3xW5Xw.png" /><figcaption>An INNER JOIN example query</figcaption></figure><p>Note, when using joins you have to specify which table you are pulling the columns from. This can be done using the syntax table_name.column_name in the <strong>SELECT</strong> clause. This can be tedious so a more common approach is to assign a nickname for each table. The above code labels the employees table as “e” and the offices table as “o” to avoid typing out the full table name for each column needed.</p><h3>Function 3: WHERE</h3><p>The <strong>WHERE</strong> function is used when we want to set a certain conditional on the data we want. Additionally, you can use <strong>WHERE</strong> along with <strong>AND, OR, NOT, LIKE</strong>, and <strong>BETWEEN </strong>to set specific requirements for the data. Let’s look at a couple of quick examples (these queries are not using the original schema and are purely illustrative).</p><h4>AND, OR, and NOT</h4><pre>#all males that weigh more than 200lbs<br>query = &quot;&quot;&quot; SELECT patient_id, height, weight, age, sex, region<br>            FROM patients<br>            WHERE sex = &#39;M&#39; AND weight &gt; 200;&quot;&quot;&quot;</pre><pre>#all patients that weigh more than 200 or are older than 50<br>query = &quot;&quot;&quot; SELECT patient_id, height, weight, age, sex, region<br>            FROM patients<br>            WHERE weight &gt; 200 OR age &gt; 50;&quot;&quot;&quot;</pre><pre>#all patients not from the Northeast<br>query = &quot;&quot;&quot; SELECT patient_id, height, weight, age, sex, region<br>            FROM patients<br>            WHERE region NOT &#39;NE&#39;;&quot;&quot;&quot;</pre><h4>LIKE</h4><pre>#all patients whose patient_id starts with A100<br>query = &quot;&quot;&quot; SELECT patient_id, height, weight, age, sex, region<br>            FROM patients<br>            WHERE patient_id LIKE &#39;A100%&#39;;&quot;&quot;&quot;</pre><h4>BETWEEN</h4><pre>#all patients who weigh between 195 and 255 lbs<br>query = &quot;&quot;&quot; SELECT patient_id, height, weight, age, sex, region<br>            FROM patients<br>            WHERE weight BETWEEN 195 and 255;&quot;&quot;&quot;</pre><h3>FUNCTION 4: ORDER BY</h3><p>The <strong>ORDER BY</strong> command organizes our data by a specified column in ascending (default) or descending order. Let’s revisit our original schema for a quick example.</p><h4><strong>ASCENDING (ASC)</strong></h4><p>Let’s grab customerName, creditLimit, addressLine1, city, state, postalCode, and country from the customers table. We will order the results alphabetically by state and then by city. Finally, we will only consider the U.S. customers who have a creditLimit of more than $25,000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*2c3dvgh5DD0DyktBGpv1Bg.png" /><figcaption>An ORDER BY ascending query example</figcaption></figure><h4>DESCENDING (DESC)</h4><p>Let’s query the exact same data except this time we will sort the rows by creditLimit and only look at the top 5 creditLimits (using the <strong>LIMIT</strong> clause)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*-c-z2iFs4BR8xlQQevvunQ.png" /><figcaption>An ORDER BY descending query example</figcaption></figure><h3>Function 5: GROUP BY</h3><p>As mentioned earlier, the <strong>GROUP BY </strong>function is typically used with aggregator functions in order to group the “results” data by one or more columns. Below is a simple example of how we can group by one column in order to see an aggregator function applied to a different column. Specifically, let’s say we wanted to see which five states, U.S. only, we do the most business in. This could involve grouping the data by states and summing the orderTotal column we calculated earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*uKAzvbcYkxuQCzZ9Q6oc-Q.png" /><figcaption>A GROUP BY and LIMIT query example</figcaption></figure><p>Note that in the above code you can pass 1, 2, etc. instead of the column_name. This specifies the location of the column (relative to the <strong>SELECT </strong>cause) to group the results on. So in the above example, we could have said “GROUP BY 1” instead of “GROUP BY state” since state is in the 1 position of the <strong>SELECT</strong> clause</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*oRZMU6OIXLi1I1-AkoHoJQ.png" /><figcaption>A GROUP BY 1 query example</figcaption></figure><p>Another important function that is commonly used with the <strong>GROUP BY </strong>function is <strong>HAVING. </strong>The <strong>HAVING </strong>function is similar to the <strong>WHERE </strong>function in that it creates a condition that needs to be satisfied for a row to be included in the results, but it is different in that the <strong>HAVING </strong>clause assigns the conditional directly to the <strong>GROUP BY</strong> command. This might be a tad bit confusing so let’s try using some code to clarify this point.</p><p>The below code pulls for us the customerName, customerNumber, productName, productCode, and totalOrdered for any customer who has ordered 10 or more of the products cumulative. Note that the <strong>HAVING </strong>clause acts similarly to the <strong>WHERE </strong>clause here but only applies to the GROUP BY 1 (customerName), 4 (productCode).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*pF9tKB_aiggEIKBR28Zmag.png" /><figcaption>A GROUP BY and HAVING query example</figcaption></figure><h3>One Last Tip: Subqueries!</h3><p>Subqueries are basically queries within queries. They can be used in many ways to solve many different problems. But ultimately, you want use subqueries when your conditionals get complicated and would require multiple <strong>JOIN </strong>commands and/or multiple <strong>WHERE </strong>commands.</p><p>For example, what if we wanted to get the firstName, lastName, and city of every employee in an office with less than 5 employees. One way of handling this would be to first query a results set of offices with less than 5 employees and then only consider the firstName, lastName, and city of the employees whose officeCodes is in the results set. Let’s see what the code looks like for that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*q2sgl-oMMDgRCReatQY9OA.png" /><figcaption>A subquery example</figcaption></figure><h3>Cheat Sheet</h3><p>In this blog we covered the 5 most frequently used MySQL functions in a typical ETL Pipeline, along with all of their corresponding clauses/commands. So let’s quickly recap.</p><p>Function 1: SELECT, SELECT DISTINCT, SELECT + an aggregator function and AS (alias for what the column should be called in the output)</p><p>Function 2: JOIN (OUTER/INNER/LEFT/RIGHT) + ON or USING</p><p>Function 3: WHERE + AND/OR/NOT, LIKE, BETWEEN</p><p>Function 4: ORDER BY + ASC (default) or DESC</p><p>Function 5: GROUP BY + HAVING</p><p>Subqueries: a query within a query</p><p>I highly encourage everyone reading this blog to do <a href="https://www.w3schools.com/sql/default.asp">further reading</a>. What do you think? Are there any honorable mentions?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2007d6f5086" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-mysql-functions-used-in-almost-every-etl-pipeline-d2007d6f5086">The MySQL Functions Used in (Almost) Every ETL Pipeline</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s Not Gambling; It’s Data Science.]]></title>
            <link>https://medium.com/@jprinfret/its-not-gambling-it-s-data-science-897971f63572?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/897971f63572</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[golf]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Tue, 28 Jan 2020 04:06:14 GMT</pubDate>
            <atom:updated>2020-03-09T22:38:50.430Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using machine learning to bet on a basket of golfers as to mimic the risk/return profile of a Venture Capital fund.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iKvIaL2ClZ4Wvf05" /><figcaption>Photo by <a href="https://unsplash.com/@will0629?utm_source=medium&amp;utm_medium=referral">Will Porada</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The Masters Tournament is the first major tournament of every PGA Tour season, and it is by far and away my favorite sporting event of the year. But what really makes the Masters so entertaining is that it is the only major that is played on the same course every year (some argue that this is why THE PLAYERS has yet to be officially named the fifth major, but I digress). The course IS the tournament and the fans know it.</p><p>With this huge variable turned constant already solved for us, I set out to use my machine learning skills, as many of us sports gambling fans do, to figure out a way to gather data and predict the winner for the upcoming 2020 Masters Tournament.</p><h3>Project Summary</h3><p>As it turns out, that is really hard to do! If you don’t believe me just ask anyone who has tried to predict an anomaly (at about 1:100 this isn’t exactly an anomaly, it’s just more of an imbalance, but you get my point: it is hard to do).</p><p>But after a few five-star performances (on expert) at the Guitar Hero arcade console it hit me: I didn’t actually need to predict the winner. Well, I did but not exactly how I originally thought. Instead, what I needed to do was predict a subset of golfers and hope that the winner is one of those golfers. This seemed like a much easier task, especially considering that in golf the difference between first, second, third, and even tenth can be just one lucky bounce.</p><p>With a database of full season aggregate statistics dating back to 2004, I attempted to use two-class classification to predict which golfers would make the cut. Then with that subset in hand, I used regression to predict each golfers final score. Grabbing the 10 lowest scores for each year, I compared my predicted top 10 (“the prediction set”) to the actual top 10 and analyzed the IRR if I were to bet equal money on each golfer each year.</p><p>The best classification model was a Random Forest Classifier, which had a F1 Score of 76%. The best regression model was a Random Forest Regressor with a RMSE of 5 strokes (total across all 4 rounds). Using both models together correctly captured 5 of the last 7 Champions in the prediction set, equating to an IRR of about 150%. However, these results were not achieved without some feature engineering (like all good machine learning projects). Additionally, I used SMOTE to handle the class imbalance of the training set (67% / 33%) and StandardScaler to scale the data. I also incorporated PCA into the machine learning pipeline but ultimately the best model included all of the 11 features I originally considered.</p><p>Additionally, I wanted to make sure that the model was not just predicting the 10 best golfers each year (as that is never the actual top 10 in any tournament). It was not, and that strategy would only have captured 2 of the last 7 Champions. Further, I wanted to make sure that the model was not predicting the same top 10 that Vegas bookmakers were predicting (because then why would you need your own model?). Again, it was not, and that method only captured 3 of the last 7 Champions. Both of these strategies are deeply out of the money.</p><p>Finally, I wanted to address why I only went back 7 years. When back testing results you can only use data up until that point to make predictions (e.g., you can only use data prior to 12/31/13 to make your 2014 Tournament predictions). Otherwise, you would be incorporating the actual results in your predictions, which we know to be describing and not predicting. Because of this, the available data — which was aggregate season long data — became very limited very quickly. Given I was working with 11 features, I wanted to cut it off arbitrarily at a minimum of about 500 data points.</p><p>Here are links to the projects <a href="https://github.com/jprinfret/masters_tournament_classifier/blob/master/README.md">GitHub</a> and <a href="https://docs.google.com/presentation/d/1nH3akRXU6ILs_raoP0fZK6bcODCH0-iZTfAy8SXOVi8/edit#slide=id.g6da5e17a54_0_145">Executive Overview</a>.</p><h3><strong>Why 10 golfers?</strong></h3><p>The main idea to betting on a basket of golfers is derived (some what) from Venture Capital. In an overly simplified form, the idea behind early stage investing is to allocate capital across portfolio companies with the understanding that you’ll have a combination of strikeouts and home runs — hopefully your home runs make up for (and then some!) your losses.</p><p>So why not bet on more than 10 golfers? Simply put, I didn’t want to be right (i.e., the champion is in the prediction set) and lose money. For example, this year’s betting favorite is Brooks Koepka. His odds are currently +900 to win. This means that a $100 bet on Brooks to win yields $900 in winnings (plus you get back your original $100). If you bet $100 on 10 golfers, Brooks being one of them and the Champion, you will lose $900 on the other 9 golfers and win $900 on Brooks. This is the breakeven scenario. You can easily see that if you were to bet on more than 10 golfers you would lose money if Brooks won; in fact, the number of golfers you bet on should be a function of the lowest odds in the prediction set to avoid being right and losing money, but I decided to use a prediction set of 10 each year for the sake of comparing the year-over-year results.</p><h3>Expected Value</h3><p>Let’s further analyze the 2020 Masters Tournament by considering current betting odds and two scenarios: (A) the Champion is not in the prediction set and (B) the Champion is in the prediction set. We will also assume a 50% success rate for capturing the champion in the prediction set (a conservative success rate compared to back tested results). For scenario B we will assume that each of the 10 golfers in the prediction set has a 10% chance of being the Champion (this is not the case, and we can calculate each individual’s probability using his odds, but we will over simplify yet again).</p><p>Total Risk: $1,000 (10 golfers x $100 bet on each golfer)</p><p>Scenario A’s expected value: 50% x $0 = $0. This is the scenario where the prediction set does not contain the Champion, and we lose our entire bet.</p><p>Scenario B’s expected value: 50% x [(10% + $1,800) + (10% + $2,000) + (10% + $1,000) + (10% + $900) + (10% + $1,400) + (10% + $4,000) + (10% + $1,800) + (10% + $4,000) + (10% + $2,000) + (10% + $5,000)] = $1,195. This is the scenario where the Champion is one of the 10 golfers in the prediction set. Above are the current payouts for each those golfers (see slide 12 of the executive overview).</p><p>Total Expected Value is therefore $0 + $1,195 =$1,195 — a 19.5% return. But again, this is the conservative case. If we use our current track record of 5/7, our expected value increases to about $1,700 or a 70% return. We do not have negative expected value until we assume that scenario B will only occur 42% of the time with the current odds.</p><h3>Conclusion</h3><p>Neither this blog nor this project were not meant to serve as gambling advice; please don’t act on it. Rather, the point of all of this was to shed light on a different way to consider risk management and cash flow. It also served as another interesting way to use the various machine learning algorithms (classification first, then regression, followed by back tested analysis).</p><p>Please let me know your thoughts on either my logic, code, or results.</p><p>And finally I will leave you with this quote made popular by Mark Twain…“There are three kinds of lies: lies, damned lies, and statistics.” So always be weary of thinking you have the edge when it comes to gambling.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=897971f63572" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Latent Dirichlet Allocation (LDA)]]></title>
            <link>https://medium.com/swlh/latent-dirichlet-allocation-lda-eff969bda284?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/eff969bda284</guid>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Wed, 18 Dec 2019 02:26:46 GMT</pubDate>
            <atom:updated>2019-12-18T06:35:11.538Z</atom:updated>
            <content:encoded><![CDATA[<h3>Latent Dirichlet Allocation</h3><p>What the heck is LDA and how is it used for topic modeling?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bd9CaNARkrHsVmL6" /><figcaption>Photo by <a href="https://unsplash.com/@danielcgold?utm_source=medium&amp;utm_medium=referral">Dan Gold</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Humans have at least two advantages over computers: we understand context and can pick up on the emotions attached to text (“K.”). Computers on the other hand do none of these things particularly well without a little help. So how does a computer cluster words into topics like it does in say sentiment analysis and topic models?</p><p>Thats where LDA comes in handy. By assuming documents are nothing more than a probability distribution of topics, and topics are nothing more than a probability distribution of words, LDA calculates a probability that a document is mostly this topic or mostly that topic (e.g., Document N is 77% topic 1, 10% topic 2, 8% topic 5, and 5% topic 7) based on the words it contains.</p><p>Basically what this boils down to is this: documents with similar topics have a higher probability of using similar groups of words.</p><p>But before we can take a look at how this done, we most let go of all preconceived notions for how we humans draft text documents in order to understand how a computer <em>thinks</em> we draft text documents.</p><h3>Plate Notation</h3><p>As defined by Wikipedia:</p><blockquote>In Bayesian inference, <strong>plate notation</strong> is a method of representing variables that repeat in a graphical model. Instead of drawing each repeated variable individually, a plate or rectangle is used to group variables into a subgraph that repeat together, and a number is drawn on the plate to represent the number of repetitions of the subgraph in the plate. The assumptions are that the subgraph is duplicated that many times, the variables in the subgraph are indexed by the repetition number, and any links that cross a plate boundary are replicated once for each subgraph repetition.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*rW0u9Kx9HJBVdqRQQgkwoA.png" /><figcaption>Plate Notation for Latent Dirichlet Allocation</figcaption></figure><p>Let’s define each model parameter:</p><ul><li><strong>M </strong>is the total number of documents within the corpus</li><li><strong>N </strong>is the total number of words in a document</li><li><strong>α </strong>is the latent per-document topic probability distribution. Visually, α points into M but does not penetrate N. This symbolizes that α is the same across all of M (the documents) and describes a sub-set of the documents (i.e., the topics). A high value for <strong>α </strong>implies each document is likely to contain a mixture of most of the topics</li><li><strong>β</strong> is the latent per-topic word probability distribution. Visually, <strong>β</strong> penetrates both <strong>M</strong> and <strong>N</strong>, making it a word describing parameter at the topic level. A high value for <strong>β</strong> implies each topic is likely to contain a mixture of most of the words</li><li>𝚹 is the topic distribution per document M (𝚹 is contained by <strong>M</strong>). You can compare the word distributions of topics in document 1 to document 2 by comparing 𝚹₁ and 𝚹₂</li><li><strong>Z</strong> is used to notate each topic assignment as a mixture of words. Z₁₂ is the topic for the 2nd word in document 1.</li><li><strong>W </strong>is used to denote each specific word. W₁₂ would be the 2nd word in document 1.</li></ul><p>The goal of LDA is to use <strong>N</strong>, <strong>W</strong>, <strong>Z</strong>, and 𝚹 to describe <strong>α </strong>and<strong> β.</strong></p><p>Simply put, according to LDA, if we wanted to create a new document <strong>X</strong>, we would first determine the number of words in document <strong>X</strong> (which is <strong>N</strong>), choose a topic mixture for <strong>X</strong> over a fixed number of topics (i.e., set 𝚹), and then come up with the words of <strong>X</strong> by:</p><ul><li>picking each word’s topic (<strong>Z</strong>) based <strong>X</strong>’s topic distribution (𝚹)</li><li>picking a word (<strong>W</strong>) based on that word’s assigned topic (<strong>Z</strong>)</li></ul><p>Once we have each word (<strong>W</strong>) and the topic of each word (<strong>Z</strong>), as well as the actual distribution of the topics in each document (𝚹), we can calculate the word distribution of each topic (<strong>β</strong>) and the topic distribution of each document (<strong>α)</strong>. When viewed at the corpus level, we have our probability distributions per-document and per-topic.</p><p>Of course, this is not how an actual human being drafts a text document, though it has proven to be a useful generative process for topic modeling.</p><h3>Let’s put numbers to an example</h3><p>Say we have a group of newspaper articles that we assume to be only about sports, politics, and tech. Each of these topics are describe by the following words:</p><ul><li><strong>Sports:</strong> game, score, win, lose, home, away</li><li><strong>Politics</strong>: Republicans, Democrats, Trump, Pelosi, bipartisan</li><li><strong>Tech:</strong> data, computer, machine, gadget, wearables</li></ul><p>LDA would assume that these articles were drafted by first choosing the length of the article (a 1,000 word column would have N=1,000). Then we would determine the topic mix of our article, or 𝚹. Let’s say our one-thousand word column is 50% politics, 20% sports, and 30% tech. That means we would choose 50 words from politics, 20 words from sports, and 30 words from tech. Finally, as we are writing our article, we would chose a word (<strong>W</strong>) with topic (<strong>Z</strong>) to met this 50/20/30 split based on the distribution defined above (𝚹). If we were to fill the December 17th edition of this newspaper with various articles following the same creation algorithm, we could easily compute the newspaper’s latent word probability distribution for each topic (<strong>β</strong>), as well as the newspaper’s topic distribution for each article (<strong>α)</strong>.</p><p>Remember syntax, context, emotion, etc. are lost on a computer! While we would never be able to comprehend this newspaper, a computer would have no problem modeling out the per-topic word probability distribution and the per-document topic probability distribution.</p><h3>Working Backwards</h3><p>Humans actual draft text documents in the opposite direction of LDA. We write a collection of words (phrases) that have context and meaning, but also follow basic rules or syntax. So, in order to use LDA to solve for <strong>α </strong>and<strong> β </strong>of an article written by a human (with context, syntax, emotion, etc.), we will follow the creation algorithm in reverse order.</p><p>First, let’s randomly assign each word W in each document X to one of K pre-determined topics (i.e., set Z for every W in every document X). Again, LDA assumes that every corpus has a latent per-document probabilistic topic distribution and a latent per-topic probabilistic word distribution. In other words, we assume that each document in a corpus can be mostly one of say 7 topics.</p><p>For each document X, we will assume that all randomly generated topic assignments Z for each word W are correct except for the current word W we are analyzing.</p><p>In order to find the correct topic assignment Z for this word W, we will need to calculate two probabilities:</p><ul><li>The proportion of words in document X that are currently randomly assigned to topic Z, this is 𝚹 or P(Z|X) = P(Z∩X) / P(X). In other words, what is the topic distribution of document X</li><li>The proportion of topic assignments over all documents that contain word W, this is <strong>β </strong>or<strong> </strong>P(W|Z) = P(W∩Z) / P(Z)</li></ul><p>We can then assign word W and new topic assignment Z based on this per-document topic probability distribution relative to all documents that contain W, or 𝚹 * <strong>β = α</strong>. In other words, we are assuming that the current topic assignment of a word is wrong and using the inherent features of the document to assign the “correct” topic to a word based on probability.</p><p>If we iterate through each word W and repeat this process enough times, we should reach a steady state where all assignments make sense based on the assumed latent probability distributions and model parameters (i.e., <strong>α</strong> will approach the true latent value of <strong>α</strong>).</p><p>And that my friends, is LDA!</p><h3>Credit Where Credit is Due</h3><p>I highly recommend everyone watch Scott Sullivan’s YouTube video (<a href="https://www.youtube.com/watch?v=DWJYZq_fQ2A">here</a>) on LDA, especially his summary of core assumptions and conclusions. His video was absolute the basis for this blog and the foundation of my understanding of LDA.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eff969bda284" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/latent-dirichlet-allocation-lda-eff969bda284">Latent Dirichlet Allocation (LDA)</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hyperparameter Cheat Sheet]]></title>
            <link>https://medium.com/swlh/the-hyperparameter-cheat-sheet-770f1fed32ff?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/770f1fed32ff</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[statistical-learning]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Mon, 16 Dec 2019 22:07:40 GMT</pubDate>
            <atom:updated>2020-01-27T19:15:47.401Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick guide to hyperparameter tuning utilizing Scikit Learn’s GridSearchCV, and the bias/variance trade-off</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*i_fLdbs7b2Ls222Ck48g0g.jpeg" /></figure><p>Before we dive in, let’s start with a quick definition. In machine learning, a hyperparameter (sometimes called a tuning or training parameter) is defined as any parameter whose value is set/chosen at the onset of the learning process. Whereas other parameter values are computed during training.</p><p>In this blog, we will discuss some of the important hyperparameters involved in the following machine learning classifiers: K-Nearest Neighbors, Decision Trees and Random Forests, AdaBoost and Gradient Boost, and Support Vector Machines. Specifically, I will focus on the hyperparameters that tend to have the greatest effect on the bias-variance tradeoff. Please note that the below lists are by no means meant to be exhaustive, and I encourage everyone to research each parameter on their own.</p><p>This blog assumes a basic understanding of each classifier, so we will skip a theory overview and mostly dive right into the tuning process utilizing Scikit Learn.</p><h3>K-Nearest Neighbors (KNN)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LxxRhoRy1ENygOzF" /><figcaption>Photo by <a href="https://unsplash.com/@woodpecker65?utm_source=medium&amp;utm_medium=referral">Christian Stahl</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the KNN classifier (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html">documentation</a>), a data point is labeled based on its proximity to its neighbors. But how many neighbors should be considered in the classification?</p><h4><strong>N_neighbors (K)</strong></h4><p>Simply put, K is the number of neighbors that defines an unlabeled datapoint’s classification boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qvcq5GdVX0hRc0JqHVcVtA.gif" /><figcaption>At K=3, the black datapoint is labeled as red since red has blue outnumbered 2:1</figcaption></figure><p>K takes in a range of integers (default = 5), finds the K-nearest neighbors, calculates the distance from each unlabeled point to those K-neighbors. How distance is calculated is defined by the metrics parameter explained below.</p><p><em>Important Note: K tends to be odd to avoid ties (i.e., if K = 4, this could result in a 2 Yes and 2 No, which would confuse the classifier).</em></p><p><strong>Bias-Variance Tradeoff:</strong> in general, the smaller the K, the tighter the fit (of the model). Therefore, by decreasing K, you are decreasing bias and increasing variance, which leads to a more complex model.</p><h4>Other Parameters for Consideration</h4><ul><li><strong>Leaf_size</strong> determines how many observations are captured in each leaf of either the BallTree of KDTree algorithms, which ultimately make the classification. The default equals 30. You can tune leaf_size by passing in a range of integers, like n_neighbors, to find the optimal leaf size. It is important to note that leaf_size can have a serious effect on run time and memory usage. Because of this, you tend not to run it on leaf_sizes smaller than 30 (smaller leafs equates to more leafs).</li><li><strong>Weights</strong> is the function that weights the data when making a prediction. “Uniform” is an equal weighted function, while “distance” weights the points by the inverse of their distance (i.e., location matters!). Utilizing the “distance” function will result in closer data points having a more significant influence on the classification.</li><li><strong>Metric </strong>can be set to various distance metrics (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html">see here</a>) like Manhattan, Euclidean, Minkowski, or weighted Minkowski (default is “minkowski” with a p=2, which is the Euclidean distance). Which metric you choose is heavily dependent on what question you are trying to answer.</li><li>Note: we will skip the algorithm hyperparameter because it is preferred to set this parameter to “auto” so that the computer tells you which tree algorithm is best. I would recommend reading sklearn’s documentation for more information on the differences between the BallTree and KDTree algorithms.</li></ul><h4>Sample K optimizing code block (assumes F1 is best scoring metric</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*0ryncRR2oDVdpt7sflAO5w.png" /><figcaption>Optimizing for K</figcaption></figure><h3>Decision Trees and Random Forests</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZcnCXEnEihxYSXxq" /><figcaption>Photo by <a href="https://unsplash.com/@garri?utm_source=medium&amp;utm_medium=referral">Vladislav Babienko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When building a Decision Tree (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html">documentation</a>) and/or Random Forest (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html">documentation</a>), there are many important hyperparameters to be considered. Some of these are found in both classifiers while some are just specific to forests.</p><p>Let’s take a look at the hyperparameters that are most likely to have the largest effect on bias and variance.</p><ul><li><strong>N_estimators</strong> (only used in Random Forests) is the number of decision trees used in making the forest (default = 100). Generally speaking, the more uncorrelated trees in our forest, the closer their individual errors get to averaging out. However, more does not mean better since this can have an exponential effect on computation costs. After a certain point, there exists statistical evidence of diminishing returns. <strong>Bias-Variance Tradeoff: </strong>in theory, the more trees, the more overfit the model (low bias). However, when coupled with bagging, we need not worry.</li><li><strong>Max_depth</strong> is an integer that sets the maximum depth of the tree. The default is None, which means the nodes are expanded until all the leaves are pure (i.e., all the data belongs to a single class) or until all leaves contain less than the min_samples_split, which we will define next. <strong>Bias-Variance Tradeoff: </strong>increasing the max_depth leads to overfitting (low bias)</li><li><strong>Min_samples_split</strong> is the minimum number of samples required to split an internal node. <strong>Bias-Variance Tradeoff: </strong>the higher the minimum, the more “clustered” the decision will be, which could lead to underfitting (high bias).</li><li><strong>Min_samples_leaf </strong>defines the minimum number of samples needed at each leaf. The default input here is 1. <strong>Bias-Variance Tradeoff:</strong> similar to min_samples_split, if you do not allow the model to split (say because your min_samples_lear parameter is set too high) your model could be over generalizing the training data (high bias).</li></ul><h4>Other Parameters for Consideration</h4><ul><li><strong>Criterion</strong> measures the quality of the split and receives either “gini”, for Gini impurity (default), or “entropy”, for information gain. Gini impurity is the probability of <em>incorrectly</em> classifying a randomly chosen datapoint if it were labeled according to the class distribution of the dataset. Entropy is a measure of chaos in your data set. If a split in the dataset results in lower entropy, then you have gained information (i.e., your data has become more decision useful) and the split is worthy of the additional computational costs.</li></ul><h4>GridSearchCV Optimization for Random Forests</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*9gtGUpIsMdlTA8HLv-lTdQ.png" /><figcaption>A sample block of code illustrating how to tune the Random Forest Classifier</figcaption></figure><h3>AdaBoost and Gradient Boosting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WxNWy8bnH9QefnG9" /><figcaption>Photo by <a href="https://unsplash.com/@m_bonkosky?utm_source=medium&amp;utm_medium=referral">Michelle Bonkosky</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>By utilizing weak learners (aka “stumps”), boosting algos like AdaBoost (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html">documentation</a>) and Gradient Boosting (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html">documentation</a>) focus on what the model misclassifies. By overweighting these misclassified data points, the model focuses on what it got wrong in order to learn how to get them right.</p><p>Similar to Decision Trees and Random Forests, we will focus on the bias-variance tradeoff usual suspects.</p><ul><li><strong>N_estimators</strong> is the maximum number of estimators at which boosting is terminated. If a perfect fit is reached, the algo is stopped. The default here is 50. <strong>Bias-Variance Tradeoff: </strong>the higher the number of estimators in your model the lower the bias.</li></ul><h4>Other Important Parameters</h4><ul><li><strong>Learning_rate</strong> is the rate at which we are adjusting the weights of our model with respect to the loss gradient. In layman’s terms: the lower the learning_rate, the slower we travel along the slope of the loss function. <strong>Important note: </strong>there is a trade-off between learning_rate and n_estimators as a tiny learning_rate and a large n_estimators will not necessarily improve results relative to the large computational costs.</li><li><strong>Base_estimator (AdaBoost) / Loss (Gradient Boosting) </strong>is the base estimator from which the boosted ensemble is built. For AdaBoost the default value is None, which equates to a Decision Tree Classifier with max depth of 1 (a stump). For Gradient Boosting the default value is deviance, which equates to Logistic Regression. If “exponential” is passed, the AdaBoost algorithm is used.</li></ul><h4>GridSearchCV Optimization for AdaBoost</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*fKeb3TWqgh9oHnEGBzqlEw.png" /><figcaption>A sample block of code illustrating how to tune the AdaBoost Classifier</figcaption></figure><h3>Support Vector Machines (SVM)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6s2094aIuz6qmWNh" /><figcaption>Photo by <a href="https://unsplash.com/@lalonchera?utm_source=medium&amp;utm_medium=referral">lalo Hernandez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This classifier, which is formally defined by a separating hyperplane (let’s take a minute to appreciate how awesome the word hyperplane is), has many tuning parameters to consider, but we will only focus on three: C, Kernel, and Gamma.</p><ul><li><strong>C </strong>is the regularization parameter. As the documentation notes, the strength of regularization is inversely proportional to C. Basically, this parameter tells the model how much you want to avoid being wrong. You can think of the inverse of C as your total error budget (summed across all training points), with a lower C value allowing for more error than a higher value of C. <strong>Bias-Variance Tradeoff: </strong>as previously mentioned, a lower C value allows for more error, which translates to higher bias.</li><li><strong>Gamma </strong>determines how far the scope of influence of a single training points reaches. A low gamma value allows for points far away from the hyperplane to be considered in its calculation, whereas a high gamma value prioritizes proximity. <strong>Bias-Variance Tradeoff: </strong>think of gamma as inversely related to K in KNN<strong>, </strong>the higher the gamma, the tighter the fit (low bias).</li></ul><h4>A Very Important Hyperparameter</h4><ul><li><strong>Kernel </strong>specifies which kernel should be used. Some of the acceptable strings are “linear”, “poly”, and “rbf”. Linear uses linear algebra to solve for the hyperplane, while poly uses a polynomial to solve for the hyperplane in a higher dimension (see <a href="https://en.wikipedia.org/wiki/Kernel_method#Mathematics:_the_kernel_trick">Kernel Trick</a>). RBF, or the radial basis function kernel, uses the distance between the input and some fixed point (either the origin or some of fixed point c) to make a classification assumption. More information on the Radial Basis Function can be found <a href="https://en.wikipedia.org/wiki/Radial_basis_function">here</a>.</li></ul><p>GridSearchCV Optimization for SVM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*yOVEtHohCP0S_KptRZ3C1A.png" /><figcaption>A sample block of code illustrating how to tune the Support Vector Classifier</figcaption></figure><h3>Final Word</h3><p>We have seen multiple was to train a model using sklearn, specifically GridSearchCV. However, it is very, very important to keep in mind the bias-variance tradeoff, as well as the tradeoff between computational costs and scoring metrics. Ideally, we want a model with low bias and low variance to limit overall error, but is it really worth the extra run-time, memory, etc. for only slight improvements? I will let you answer that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=770f1fed32ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/the-hyperparameter-cheat-sheet-770f1fed32ff">The Hyperparameter Cheat Sheet</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Better Understanding of Beta]]></title>
            <link>https://medium.com/swlh/a-better-understanding-of-beta-8e20466f3f35?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e20466f3f35</guid>
            <category><![CDATA[equity-market-tips]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[portfolio-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Fri, 22 Nov 2019 06:10:32 GMT</pubDate>
            <atom:updated>2019-12-18T06:35:00.153Z</atom:updated>
            <content:encoded><![CDATA[<h4>Utilizing basic Python packages and linear regression modules to analyze and visualize equity beta</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fglET5fBaTOvD3fT" /><figcaption>Photo by <a href="https://unsplash.com/@ishant_mishra54?utm_source=medium&amp;utm_medium=referral">Ishant Mishra</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Finance is littered with fancy sounding greek letters like alpha, beta, delta, rho, etc. Personally, my favorite is Theta (the concept of “Theta Bleed” in the derivatives’ universe is fascinating, and I encourage you all to read more about it <a href="https://www.investopedia.com/terms/t/theta.asp">here</a>). But fear not my friends…despite the appearance of complexity, these symbols are all easily explained.</p><p>This blog takes a deeper dive into equity beta by utilizing the wonderful analytical and visualization packages available in Python; specifically, we will explore how to leverage the Pandas, Numpy, Scikit-learn and Matplotlib packages to perform regression analysis.</p><h3>What is Equity Beta?</h3><p>Simply put, a stock’s beta can be thought of as a measure of the volatility of a stock’s performance relative to the market. In other words, beta is how much we can expect a stock to increase (or decrease, if negative) for every 1% increase in the underlying equity index (for the sake of simplicity, I will use the S&amp;P 500 (“the market”) as our underlying index). In other, other words, beta is the <em>slope coefficient of the best fit line</em> if we were to regress said stock on the market.</p><p>Let’s see an example.</p><h3>Regressing a historically high beta stock and a historically low beta stock on the market</h3><p>We will use twenty years worth of monthly performance data to regress NVIDIA, a historically high beta stock, and Verizon, a historically low beta stock, on to the S&amp;P 500.</p><p>First, let’s import our packages.</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>%matplotlib inline<br>from sklearn.linear_model import LinearRegression</pre><p>Check for linearity.</p><pre>df_spy[&#39;pct_return&#39;].plot.kde(color=&#39;red&#39;, label=&#39;S&amp;P 500 Index&#39;);<br>df_nvda[&#39;pct_return&#39;].plot.kde(color=&#39;green&#39;, label=&#39;NVIDIA Corp&#39;);<br>df_vz[&#39;pct_return&#39;].plot.kde(color=&#39;blue&#39;, label=&#39;Verizon Inc&#39;);</pre><pre>plt.title(&#39;Check for Linearity&#39;);<br>plt.legend(loc=&#39;upper right&#39;, fontsize=&#39;large&#39;);<br>plt.grid(True, which=&#39;both&#39;);<br>plt.axvline(x=0, color=&#39;black&#39;, linestyle=&#39;:&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*jFTJrhIOugX6xVB5BiIwXg.png" /><figcaption>A density distribution with means centered around 0</figcaption></figure><p>Looks like we have three some-what normally distributed density distributions with μ = 0 and variance = σ² (aka some constant with which we do not care about for the purposes of this blog.).</p><p>Now let’s use Scikit-learn’s .LinearRegression() and .fit() to calculate the lines of best fit. It is important to note that you have to reshape your data using .reshape(-1,1) so that the arrays are properly formatted for the .fit() call. See the below block of code as an example — I reshaped the data into a (240 x 1) array to better fit LinearRegression().fit().</p><pre>#Build data frames and drop unnecessary columns<br>df_spy = pd.DataFrame(pd.read_csv(&#39;spy.csv&#39;)).drop([&#39;High&#39;, &#39;Low&#39;, &#39;Adj Close&#39;, &#39;Volume&#39;], axis=1)<br>df_nvda = pd.DataFrame(pd.read_csv(&#39;nvda.csv&#39;)).drop([&#39;High&#39;, &#39;Low&#39;, &#39;Adj Close&#39;, &#39;Volume&#39;], axis=1)<br>df_vz = pd.DataFrame(pd.read_csv(&#39;vz.csv&#39;)).drop([&#39;High&#39;, &#39;Low&#39;, &#39;Adj Close&#39;, &#39;Volume&#39;], axis=1)</pre><pre>#Get performance data as arrays<br>spy = np.array(df_spy[&#39;pct_return&#39;])<br>nvda = np.array(df_nvda[&#39;pct_return&#39;])<br>vz = np.array(df_vz[&#39;pct_return&#39;])</pre><pre>#Define regression lines<br>reg_spy = LinearRegression().fit(spy.reshape(-1,1), spy.reshape(-1,1))<br>reg_nvda = LinearRegression().fit(spy.reshape(-1,1), nvda.reshape(-1,1))<br>reg_vz = LinearRegression().fit(spy.reshape(-1,1), vz.reshape(-1,1))</pre><p>As a brief aside, if we were analyzing portfolios instead of individual securities, the intercept coefficient, or alpha, would be very meaningful. Alpha is a measure of the risk-adjusted outperformance relative to the market (i.e., how does the portfolio perform when the S&amp;P 500 is 0%). Many use alpha as one measure of a manager’s skill, but I digress…</p><p>We can use .coef_ and .intercept_ to pull our slope and intercept coefficients (i.e., beta and alpha).</p><pre>#Calculate intercept coefficient (aka Alpha)<br>A_spy = round(float(reg_spy.intercept_), 3)<br>A_nvda = round(float(reg_nvda.intercept_), 3)<br>A_vz = round(float(reg_vz.intercept_), 3)</pre><pre>#Calculate slope coefficient (aka Beta)<br>B_spy = round(float(reg_spy.coef_), 1)<br>B_nvda = round(float(reg_nvda.coef_), 1)<br>B_vz = round(float(reg_vz.coef_), 1)</pre><p>Hurrah, we have calculated a beta coefficient of 2.2 and for NVIDIA and a beta coefficient of 0.6 for Verizon.</p><p>Finally, lets graph our scatter plots. We will set our independent variable as the S&amp;P 500 performance array and the dependent variable as the NVIDIA and VZ performance arrays.</p><pre>#Plot data and regression lines<br>fig = plt.figure(figsize=(18,6));<br>fig.suptitle(&#39;Monthly Performance: 10/31/1999 - 10/31/2019&#39;, fontsize=22);</pre><pre>#nvda<br>ax1 = fig.add_subplot(121);</pre><pre>ax1.scatter(spy, nvda, color=&#39;blue&#39;, alpha=0.3);</pre><pre>ax1.plot(spy.reshape(-1,1), reg_spy.predict(spy.reshape(-1,1)), color=&#39;red&#39;, linewidth=2, label=f&quot;S&amp;P 500 Index&quot;);<br>ax1.plot(spy.reshape(-1,1), reg_nvda.predict(spy.reshape(-1,1)), color=&#39;blue&#39;, linewidth=2, label=f&quot;NVIDIA Corp&quot;);</pre><pre>ax1.set_title(f&quot;High Beta: {B_nvda}&quot;, {&#39;fontsize&#39;: 16});<br>ax1.set_ylabel(&#39;Percent Return of Individual Securities&#39;, {&#39;fontsize&#39;: 16});<br>ax1.set_xlabel(&#39;Percent Return of the S&amp;P 500 Index&#39;, {&#39;fontsize&#39;: 16}, labelpad=10);<br>ax1.set_xlim(-0.1, 0.1);<br>ax1.set_ylim(-0.5, 0.5);<br>ax1.set_yticks([-0.5, -0.25, 0, 0.25, 0.5]);<br>ax1.grid(True, which=&#39;both&#39;);<br>ax1.axhline(y=0, color=&#39;black&#39;, linestyle=&#39;:&#39;);<br>ax1.axvline(x=0, color=&#39;black&#39;, linestyle=&#39;:&#39;);<br>ax1.legend(loc=&#39;upper right&#39;, fontsize=&#39;large&#39;);</pre><pre>#vz<br>ax2 = fig.add_subplot(122);</pre><pre>ax2.scatter(spy, vz, color=&#39;green&#39;, alpha=0.3);</pre><pre>ax2.plot(spy.reshape(-1,1), reg_spy.predict(spy.reshape(-1,1)), color=&#39;red&#39;, linewidth=2, label=f&quot;S&amp;P 500 Index&quot;);<br>ax2.plot(spy.reshape(-1,1), reg_vz.predict(spy.reshape(-1,1)), color=&#39;green&#39;, linewidth=2, label=f&quot;Verizon Inc&quot;);</pre><pre>ax2.set_title(f&quot;Low Beta: {B_vz}&quot;, {&#39;fontsize&#39;: 16});<br>ax2.set_xlim(-0.1, 0.1);<br>ax2.set_ylim(-0.5, 0.5);<br>ax2.set_yticks([-0.5, -0.25, 0, 0.25, 0.5]);<br>ax2.grid(True, which=&#39;both&#39;);<br>ax2.axhline(y=0, color=&#39;black&#39;, linestyle=&#39;:&#39;);<br>ax2.axvline(x=0, color=&#39;black&#39;, linestyle=&#39;:&#39;);<br>ax2.legend(loc=&#39;upper right&#39;, fontsize=&#39;large&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HxU2j-A64NsodDQHuNQieA.png" /></figure><p>Now let’s focus on our attention to the regression models themselves and determine their appropriateness. As we already know, the regression line’s goal is to reduce the mean sum of squared errors (MSE). The formula for MSE can be found <a href="https://en.wikipedia.org/wiki/Mean_squared_error">here</a>. Our models produced a MSE of 0.025 for NVIDIA and 0.031 for Verizon. Pretty, pretty, pretty good (insert Larry David gif).</p><p>However, our model does a poor job of explaining the variation in the performance of our underlying security (the dependent variable) as predicted by the stock market (the independent variable). Or in other words, our NVIDIA and Verizon models have tiny R² values (<a href="https://en.wikipedia.org/wiki/Coefficient_of_determination">see here</a>) of 23% and 5%, respectively (i.e., 23% of the variation in the performance of NVDA can be explained by performance of the market). So our model is garbage, right?</p><p>Not so fast…let’s think about this for a second. Normally regressions are run to make predictions. However, in our model, it doesn’t make sense to try to predict an underlying stocks performance using the market’s performance as an independent variable because the market’s performance is determined by the underlying stocks’ performance (and not vice-versa — fundamentalists rejoice and technicians mourn)! So why run this regression?</p><p>We perform these types of regressions because, despite its aforementioned shortcomings, beta, as a measure of a stock’s volatility relative to the market, is an important data point in and of itself (see <a href="https://en.wikipedia.org/wiki/Capital_asset_pricing_model">Capital Asset Pricing Model</a>).</p><p>So do we care about R² here? (Hint: No.)</p><h3>Final Word (albeit rushed)</h3><p>Beta is the slope of the regression line when an underlying security is regressed on the market. It can be thought of as a measure of a stock’s volatility relative to the market (a beta greater than 1.0 implies the stock is more volatile and risky). Beta can be especially powerful, when combined with equity market <a href="https://en.wikipedia.org/wiki/Risk_premium">risk premium</a>, in performing risk management and expected return calculations. However, like all indicators/metrics it is not the end all be all. Similar to the conclusion of my earlier blog on <a href="https://medium.com/@johnpaul.rinfret/moving-averages-a-powerful-tool-in-data-science-67e3b25ce5db">moving averages</a>, it is our job as data scientists to understand all of the tools in our toolkit before making any recommendations (and beta is one of those tools).</p><p>Thank you for listening to my τεδ talk.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e20466f3f35" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-better-understanding-of-beta-8e20466f3f35">A Better Understanding of Beta</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginner’s Guide to Data Science in the Portfolio Management Process]]></title>
            <link>https://medium.com/data-science/a-beginners-guide-to-data-science-in-the-portfolio-management-process-56d559a3d39?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/56d559a3d39</guid>
            <category><![CDATA[investment-management]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[portfolio-management]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Wed, 06 Nov 2019 15:51:46 GMT</pubDate>
            <atom:updated>2020-09-14T21:16:05.773Z</atom:updated>
            <content:encoded><![CDATA[<h4>Portfolio optimization made simple with a deeper understanding of the statistical computations at work.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oeg6SVKZ5jwzXMHx" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A prospective client calls with $10,000,000 to invest. How does a portfolio manager/advisor go about determining the appropriate mix of stocks, bonds, cash, gold, bitcoin, or whatever? How does this manger ensure that what she is doing outperforms or — more importantly — is a more appropriate fit for her client than the results achieved by a monkey throwing darts at a dartboard?</p><p>These are questions that Data Science can help answer!</p><p>At the onset of the Portfolio Management Process, portfolio managers need to understand their clients’ objectives and constraints in order to construct an appropriate asset allocation. Reviewing a client’s objectives and constraints includes understanding her risk/return appetite, liquidity needs, investment time horizon, tax profile (asset location can be a powerful tool!), unique circumstances, and any current legal issues. With a solid understanding of these objectives and constraints, portfolio construction can begin.</p><p>An analysis of how categorical inputs like liquidity needs (none, current, future, etc.), time horizon (short-term, long-term, etc.), or taxes (pays taxes or does not pay taxes) blend together in the decision making process is beyond the scope of this blog. Rather, my goal in writing this blog is to explore the critical statical computations, such as mean, covariance, correlation, and standard deviation, that go hand-in-hand with proper portfolio construction/optimization.</p><h3>A Two Asset Portfolio Example</h3><p>Consider two stocks: A and B. Managed portfolio Z is constructed by purchasing $60,000 worth of A and $40,000 worth of B. How can we use what we know about A and B to make predictions on the future performance of Z? Well ignoring risk for a second, this is quite simple.</p><p>Let’s define the expected return of A and B (denoted E(A) and E(B), respectively) as A and B’s population means. In other words, E(A) equals the average return of A since A’s initial public offering.</p><p>How can we use this information to predict our managed portfolio’s expected return knowing the weighting of our portfolio is 60% A and 40% B? The formula is:</p><blockquote>E(Z) = WA*E(A) + WB*E(B); where WX is asset X’s percent weight in the portfolio.</blockquote><p>For illustrative purposes, we will assume E(A) = 15% and E(B) = 17.5%, giving us an expected return for our managed portfolio of 16%.</p><p>Now let’s focus on risk; we will use standard deviation as a proxy for the portfolio’s risk. To understand standard deviation, we first need to understand what a correlation coefficient is. The correlation coefficient is a measure of the strength of a linear relationship between two data sets (in our example A and B’s performance), and it ranges from -1 to 1 inclusive. A correlation coefficient of 1 means that a perfectly positive linear relationship exists (as A increases, B will also increase), and a coefficient of -1 means that a perfectly inverse linear relationship exists (as A increase, B will decrease). A correlation value somewhere in the middle, indicts the strength (or lack there of) of the linear relationship with the sign indicating the direction.</p><p>So, how is the correlation coefficient calculated? If we denote the correlation coefficient as p, then p is calculated as:</p><blockquote>p(x,y) = cov(x,y) / (std_x * std_y)</blockquote><p>It is worth quickly mentioning that cov(x,y) is short hand for the covariance between random variables x and y. Covariance, depending on sign, shows the tendency in the linear relationship between x and y (i.e., the variables show similar behavior). By dividing by (std_x*std_y), we standardize the covariance in an effort to make it comparable (i.e., we care more about the strength of the linear relationship rather than the tendency of the relationship).</p><p>Once we have our p, solving for our portfolio’s standard deviation (by taking the square root of its variance) is quite simple. We will denote our portfolio’s standard deviation as rZ, so the portfolio’s variance is:</p><blockquote>r²Z = Wx²*r²x + Wy²*r²y + 2(Wx*rx)(Wy*ry)p(x,y)</blockquote><p>For illustrative purposes, we will assume the standard deviation of A is 20%, the standard deviation of B is 10%, and the correlation of A and B is 0.85. Using our formula for rZ above, we calculate an expected standard deviation for our managed portfolio of 15.5%.</p><h3>What If You Have More Than Two Assets?</h3><p>Buy a computer. Its hard to do by hand. That being said, we should understand what a computer program is doing under the hood.</p><p>Below is the methodology behind generating portfolios on the efficient frontier, which Investopedia defines as:</p><blockquote>…the set of optimal portfolios of risky assets that offer the highest expected return for a defined level of risk or the lowest risk for a given level of expected return.</blockquote><h4>Let’s first define our <em>input</em> parameters:</h4><ol><li><strong>stock_price_data</strong> = m X n (read “m by n”) matrix of m number of closing prices for n number of stocks</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*eTxDZqfhDdYkYYfQmFZiAw.png" /><figcaption>Matrix by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><p>2. <strong>avg_stock_price</strong> = 1 X n matrix of average prices for each stock.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*mKzLRQvWNU-fw3chCHmIDA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*-RnwL5ldMSETKVwgm_mSZA.png" /><figcaption>Matrix by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><p>3. <strong>stocks_demeaned</strong> = m X n matrix of the individual stock prices minus the average price for that stock. This gives us a measure of how far each data point is from its mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*CiBRsLs4uFWYOvYFa0UijQ.png" /><figcaption>Matrix by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><p>4. <strong>covariance_matrix</strong> = m X n matrix that is the product of the stocks_demeaned matrix multiplied by its <a href="https://en.wikipedia.org/wiki/Transpose">transpose</a>. We only care about the diagonal line (from top left to bottom right) of the resulting matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*DwM-RwAzoyFxk5DffNfphw.png" /><figcaption>Matrix by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><p><em>Note: the diagonal line from top left to bottom right of the covariance matrix is representative of the variance of the individual stocks, as illustrated below.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*vAQ2agPzpOuh4cagB0DPMQ.png" /><figcaption>Matrix by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><p>5. <strong>asset_weights</strong> = n X 1 matrix of the allocation weights for each risky asset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/1*lkYTr1AelBG8fNyNeDtjfg.png" /><figcaption>Matrix by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><h4>Calculating a Portfolio’s Expected Return and Expected Risk</h4><p>With the inputs we defined above, we can calculate expected portfolio return and expected portfolio variance doing some matrix multiplication using the following formulas (z = portfolio):</p><blockquote><em>E(z) = M*W</em></blockquote><blockquote><em>r²z = Wt * (Covariance Matrix) * W; where Wt = the transpose of W</em></blockquote><blockquote><em>rz = sqrt(r²z)</em></blockquote><p>Now that we have calculated the expected return and expected standard deviation of our portfolio, we can generate tens of thousands portfolios by changing the individual weights in matrix W, and plotting these portfolios on a scatter plot, like the one below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*9jHHUcCgzz2jVt8HGY2TnQ.png" /><figcaption>Graph by <a href="https://www.linkedin.com/in/varun-divakar-b862a667/">Varun Divakar</a></figcaption></figure><p>Using the above scatter plot, a portfolio manager can easily obtain the portfolio weightings that best aligns with her client’s risk/return objectives. Finally, it should be noted that an active manager seeking the best risk-adjusted returns should focus on the portfolio that is highlighted above by the red star (the one with the highest <a href="https://www.investopedia.com/terms/s/sharperatio.asp">Sharpe ratio</a>).</p><h3>Final Word</h3><p>Portfolio construction/optimization can seem daunting at first. However, with a solid understanding of the mathematics, one can comfortably understand what makes for an appropriate allocation.</p><p><strong><em>Note from Towards Data Science’s editors:</em></strong><em> While we allow independent authors to publish articles in accordance with our </em><a href="https://towardsdatascience.com/questions-96667b06af5"><em>rules and guidelines</em></a><em>, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our </em><a href="https://towardsdatascience.com/readers-terms-b5d780a700a4"><em>Reader Terms</em></a><em> for details.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56d559a3d39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-beginners-guide-to-data-science-in-the-portfolio-management-process-56d559a3d39">A Beginner’s Guide to Data Science in the Portfolio Management Process</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving Averages: A Powerful Tool in Data Science]]></title>
            <link>https://medium.com/@jprinfret/moving-averages-a-powerful-tool-in-data-science-67e3b25ce5db?source=rss-3b361e3c4389------2</link>
            <guid isPermaLink="false">https://medium.com/p/67e3b25ce5db</guid>
            <category><![CDATA[trend-analysis]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[moving-average-crossover]]></category>
            <category><![CDATA[technical-analysis]]></category>
            <dc:creator><![CDATA[J.P. Rinfret]]></dc:creator>
            <pubDate>Sat, 19 Oct 2019 16:09:42 GMT</pubDate>
            <atom:updated>2019-10-19T22:08:24.864Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are moving averages and how can Data Scientists use them to spot the trend?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*h6J7GMdlAini6-QWL9ilJg.jpeg" /></figure><p>When trying to predict a trend, whether it is in retail, finance, hospitality, etc., there are many powerful tools backed by complex mathematical theories and daunting equations. That being said, would you be surprised to hear that one of the easiest indicators to understand is also one of the most powerful?</p><p>Meet moving averages.</p><p>Let’s stick our toes in the water. A simple Wikipedia search on moving averages yields the following result:</p><blockquote>“A moving average (aka. rolling average or running average) is a calculation to analyze data points by creating series of averages of different subsets of the full data set”</blockquote><p>In layman’s terms, moving averages “smooth” your data to make trend spotting more obvious.</p><p>It is important to note that the length of the subset used in calculating a moving average is fixed. For instance, if time (specifically days) is your independent variable and you’re interested in calculating a rolling 20-day moving average, your subset does not grow to 21 elements one day later. Rather, your subset drops the oldest element and gains a new more recent element — all the while remaining at 20 elements.</p><h3><strong>Different types of moving averages</strong></h3><p>Let’s now discuss three different moving averages: Simple Moving Average (SMA), Weighted Moving Average (WMA), and Exponential Moving Average (EMA).</p><p>A simple moving average is, simply, the arithmetic mean of your subset. Period.</p><p>The weighted moving average follows the same idea of the simple moving average but weights each element of the subset using a linearly increasing (or decreasing) factor. Once the elements are weighted, the weighted moving average returns the simple moving average of the new subset. For example, calculating a <em>X</em>-day weighted moving average could assign a weighting factor of <em>x </em>to the most recent data point, <em>x</em>-1 to next most recent data point, all the way down to 1 for the oldest data point. This method puts a <em>higher value </em>on more recent data.</p><p>Similarly to the weighted moving average, the exponential moving average weights each element of the subset. However, the exponential moving average uses an exponentially increasing (or decreasing) weighting factor (see <a href="https://en.wikipedia.org/wiki/Exponential_smoothing">exponential smoothing</a>). This method puts the <em>highest value</em> on more recent data points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*ZPBBUpnWVwU24aY2IUjVBA.png" /></figure><h3>Using moving averages</h3><p>While moving averages can be used by data scientists in many, many ways (e.g., forecasting seasonality at hotel chains, inventory management at retail warehouses, or real-time patient data analysis in healthcare), the remainder of this blog will focus on the basics of technical analysis as it relates to stock trading.</p><p>Moving averages are at the foundation of some of the most complicated trading strategies utilized on the street today. For now, we will keep it simple and explore a relatively common stock price trend indicator called a <em>crossover</em>. To explain what a crossover is, let us consider a <em>M</em>-period moving average, a <em>N</em>-period moving average (let’s assume <em>M </em>&gt; <em>N</em>), an independent variable equal to time (delineated by the same periods used to define the moving averages), and a dependent variable equal to the price of the underlying security we are trying to analyze. A crossover occurs when <em>N</em>’s rolling average price is greater than (or less than) <em>M</em>’s rolling average price. Colloquially, this is called “crossing above (below)”.</p><p>A <em>golden cross</em> occurs when the shorter-term moving average (a common short-term moving average in technical analysis is 50-days) crosses above a long-term moving average (a common long-term moving average is 200-days). This is an indication that a stock’s price should increase in the future since the short-term trend has more momentum (i.e., is stronger than) the longer-term trend. A <em>death cross</em> is the opposite: the shorter-term moving average crosses below the longer-term moving average, indicating a loss of positive momentum or a lack of buying support.</p><p>The below chart is Apple, Inc.’s ($AAPL) daily share price from July 2012 to March 2014. Each vertical bar, or candlestick, represents the daily range of the share price. The top of the wick, the thin vertical line that stretches through the rectangle, is the maximum daily price and the bottom of the wick is the minimum daily price. The top and bottom of the colored rectangles represent the opening and closing prices of that day. If the box is colored black then your opening price was less than the closing price (the stock traded up during the day’s session). If the box is colored red then your opening price was greater than your closing price (the stock traded down).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*KzVUKVmMGsdv51saYHIJWA.jpeg" /></figure><p>As the above chart indicates, in early December 2012 the 50-day moving average crossed below the 200-day moving, representing a <em>death cross,</em> giving traders a sell signal. If you were long AAPL and sold your shares, you protected your embedded gains. If you didn’t own AAPL but proceed to short the stock at ~$550 (i.e., you borrowed AAPL shares from another shareholder and sold them in the market for ~$550/share), you could have covered your short position (i.e., you bought back an equal amount of shares you sold earlier at ~$550/share) anywhere from ~$375 to ~$450 between mid-April 2013 and mid-September 2013. This represents a 18–32% profit (less commissions and borrowing costs) once you returned the shares to the original lender. Additionally, the above chart indicates that a <em>golden cross</em> occurred mid-September 2013. This gave traders a buy signal around $450. Only a few months later, in March 2014, AAPL was trading at ~$530 (an 18% profit less commissions on the purchase).</p><p>The above is just one of many examples. It also happens to be one example in which the crossovers indicated profitable trades (can you say cherry picking?). There are plenty of examples of false negatives out there, and I encourage you to explore them on your own. Just remember that stop-losses were invented for a reason.</p><h3>Do I dare say in conclusion?</h3><p>As we have seen, moving averages are a simple yet effective tool that every data scientist should have in his or her toolbox. But moving averages alone aren’t enough. An effective forecaster knows how to combine moving averages with other statistical trend indicators (see <a href="https://towardsdatascience.com/data-science-simplified-part-5-multivariate-regression-models-7684b0489015">here</a>, <a href="https://towardsdatascience.com/predicting-what-song-phish-will-play-next-with-deep-learning-947ccce3824d">here</a>, and <a href="https://towardsdatascience.com/machine-learning-predicting-house-prices-with-regression-2388bb876a6f">here</a>) in order to produce accurate results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67e3b25ce5db" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>