<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bobby Lindsey on Medium]]></title>
        <description><![CDATA[Stories by Bobby Lindsey on Medium]]></description>
        <link>https://medium.com/@bobbylindsey?source=rss-14c7e73f0fe5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k6BCTbmK5KipoIIsck0GQw.jpeg</url>
            <title>Stories by Bobby Lindsey on Medium</title>
            <link>https://medium.com/@bobbylindsey?source=rss-14c7e73f0fe5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:58:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bobbylindsey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Guide to Git for Data Scientists]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/a-guide-to-git-for-data-scientists-fd68bc1c729?source=rss-14c7e73f0fe5------2"><img src="https://cdn-images-1.medium.com/max/1800/0*S6YFBo78Dvsq6iAe.jpg" width="1800"></a></p><p class="medium-feed-snippet">You don&#x2019;t have to be terrified of git anymore.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/a-guide-to-git-for-data-scientists-fd68bc1c729?source=rss-14c7e73f0fe5------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/a-guide-to-git-for-data-scientists-fd68bc1c729?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd68bc1c729</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[getting-started]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Wed, 07 Oct 2020 19:00:38 GMT</pubDate>
            <atom:updated>2020-11-26T00:44:03.432Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Understanding Maximum Likelihood Estimation]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/understanding-maximum-likelihood-estimation-fa495a03017a?source=rss-14c7e73f0fe5------2"><img src="https://cdn-images-1.medium.com/max/1400/0*vdXs5jCpWU_f2g88.png" width="1400"></a></p><p class="medium-feed-snippet">Let&#x2019;s say you collect some data from some distribution. As you might know, each distribution is just a function with some inputs. If you&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/data-science/understanding-maximum-likelihood-estimation-fa495a03017a?source=rss-14c7e73f0fe5------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/understanding-maximum-likelihood-estimation-fa495a03017a?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa495a03017a</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Wed, 06 Nov 2019 15:10:58 GMT</pubDate>
            <atom:updated>2020-11-15T22:01:04.854Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Understanding how the perceptron really works]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/understanding-how-the-perceptron-really-works-234b1b7cf44b?source=rss-14c7e73f0fe5------2"><img src="https://cdn-images-1.medium.com/max/1280/0*ZK8Dtx88dnTXramM.png" width="1280"></a></p><p class="medium-feed-snippet">A deep dive into the fundamentals of the perceptron model with implementations in Python</p><p class="medium-feed-link"><a href="https://medium.com/data-science/understanding-how-the-perceptron-really-works-234b1b7cf44b?source=rss-14c7e73f0fe5------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/understanding-how-the-perceptron-really-works-234b1b7cf44b?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/234b1b7cf44b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Mon, 07 Oct 2019 01:41:46 GMT</pubDate>
            <atom:updated>2020-11-15T21:55:00.167Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A Practical Look at Vectors and Your Data]]></title>
            <link>https://medium.com/data-science/a-practical-look-at-vectors-and-your-data-95bde21b37d1?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/95bde21b37d1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Tue, 05 Mar 2019 05:34:54 GMT</pubDate>
            <atom:updated>2020-11-15T21:57:29.692Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcdVqPn-s6wDUjW4vDvrzQ.png" /><figcaption>A vector in the night sky</figcaption></figure><p><a href="https://bobbywlindsey.ck.page/5dca5d4310">Want to get better at data science?</a></p><p>I remember a feeling of utter confusion when I first learned vector spaces in my first course of Linear Algebra. What’s a space? And what are these vectors, really? Lists? Functions? Kittens? And how the hell does that relate to all this data that I analyze for some scientific endeavor? In this article, I attempt to explain the topic of vectors, vector spaces, and how they relate to data in a way that my former self would have appreciated.</p><h3>The Utility of the Vector</h3><p>Imagine a two-by-two graph as you might have seen in school. A <strong><em>vector</em></strong> is an object that’s typically seen in the form of an ordered list of elements, like <em>(4, 3)</em>, that lives in a vector space and is commonly represented as an arrow whose tail starts at the origin, <em>(0, 0)</em>, and ends at some other point, like <em>(4, 3)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*tqz-F5ea8HXv4QES" /><figcaption>The vector (4, 3) represented as an arrow</figcaption></figure><p>This arrow has a couple properties that are interesting to note: it has a direction and a length. A vector can represent many real-world objects like the wind, the throwing of a football, and the velocity of your car. It can even represent features of an animal, the quantities of your grocery list, or a row in your spreadsheet of data.</p><h3>A Space for Your Vectors</h3><p>But how do you know if your object or ordered list is a vector? Well, it must belong to a set called a <strong><em>vector space</em></strong>. A vector space can be any set of elements (this could be lists, functions, or other objects) but these elements must follow some rules:</p><ul><li>Adding any two elements in the set results in another element that’s already in the set</li><li>Multiplying any elements in the set by some number (called a scalar) results in another element that’s already in the set</li><li>All the elements are <a href="https://en.wikipedia.org/wiki/Associative_property">associative</a>, <a href="https://en.wikipedia.org/wiki/Commutative_property">commutative</a>, and scalars are <a href="https://en.wikipedia.org/wiki/Distributive_property">distributive</a> with respect to element addition</li><li>There’s an element in the set such that adding it to any other element doesn’t change its value</li><li>There’s some number (called a scalar) such that multiplying it by any other element doesn’t change the element’s value</li><li>Any element in the set has some element that can be added to it which results in an element of 0s (called the zero vector)</li></ul><p>If all the elements in your set follow the rules above, then congratulations! Your elements are called vectors and the set they belong to is called a vector space. But break any one of these rules, and you’re set is <em>not</em> a vector space and the elements inside your set <em>aren’t</em> vectors.</p><p>In mathematical jargon, these rules translate to the following:</p><ul><li>All elements are closed under addition</li><li>All elements are closed under scalar multiplication</li><li>All elements are associative, commutative, and scalars are distributive with respect to element addition</li><li>An additive identity exists for every element</li><li>A multiplicative identity exists for every element</li><li>An additive inverse exists for every element</li></ul><h3>A Tangible Vector Space</h3><p>Consider the set you already know and love, <em>R²</em>. <em>R²</em> is just <em>R × R</em> which is the set of all possible two-dimensional lists represented by the following set notation <em>{(a, b) : a, b in R}</em>. You can visualize <em>R²</em> below (obviously the <em>x</em> and <em>y</em> axes limits do not stop at ten but instead approach infinity):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bX_jfSvuv0wRO90Z" /><figcaption>R² visualized as the Cartesian Plane</figcaption></figure><p>Is the set <em>R²</em> a vector space? Well if it were, it would need to follow all the rules of a vector space that were mentioned above. Consider the first rule: adding any two elements in the set results in another element that’s already in the set. Does <em>R²</em> follow this rule? Well, for example, take any two elements in <em>R²</em>, like <em>(2, 3)</em> and <em>(-1, -4)</em>. When you add them together, do you get an element that’s also in <em>R²</em>? Yes! In fact, you get <em>(1, -1)</em> which does indeed live in the set <em>R²</em>. This is true for <em>any</em> two elements you add in <em>R²</em> — you’ll always get something back that also belongs to <em>R²</em>.</p><p>So you’ve verified the first rule! But in order for <em>R²</em> to be called a vector space, you must verify that it follows all the rules; which it does.</p><p>In short, there are many sets that follow the rules above so naturally you give it a name. And now when someone talks about some arbitrary set, you know that if the elements of the set follow the rules of a vector space, then the set must be a vector space.</p><h3>Fields</h3><p>Another note to make is about these scalars with which you can multiply vectors. Scalars are elements that belong to a set called a <strong><em>field</em></strong> which has the same rules as a vector space, but with the bonus of having a multiplicative inverse as well.</p><p>Since a field is a set that has all the rules a vector space has, then a field is also a vector space. You’ve actually been using fields all along, like the set of real numbers or the set of complex numbers. Any time you draw a plot on a two-dimensional grid, you’re actually drawing on a two-dimensional field.</p><h3>Subspaces</h3><p>In practice, you might think that it’s tedious to check every rule against a set to see if the set is a vector space. Well, you’re right. So that’s why it’s far easier to identify a set as a <em>subset</em> of a vector space you already know (like the set of real numbers) and prove that this subset is nonempty and that it’s closed under the same operations as the vector space (i.e. under addition and scalar multiplication).</p><p>If you’re able to do this, then your subset is called a <strong><em>subspace</em></strong> which also happens to be a vector space in and of itself (again, to see for yourself, take a subspace and verify that it has all the properties of a vector space).</p><p>So if you want to prove that a set is a vector space, try to prove that it’s a subspace instead. Since subspaces are vector spaces in their own right, you’ll have successfully shown that a set is a vector space.</p><h3>Real-World Data</h3><p>Now all this theory is not very helpful if you can’t apply it. So consider the first few rows of the classic Iris dataset, which is a dataset containing samples of three different species of the Iris flower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*zM8V_OoHz3fBY7yVR7QDJw.png" /><figcaption>Some rows of the Iris dataset</figcaption></figure><p>As you can see from the header, the features of each sample are:</p><ul><li>sepal length</li><li>sepal width</li><li>petal length</li><li>petal width</li></ul><p>Each row of features can be viewed as an ordered list. The first list would be <em>(5.1, 3.5, 1.4, 0.2)</em>, the second <em>(4.9 , 3, 1.4, 0.2)</em>, and so on. But are these ordered lists vectors? Well, each entry in the lists is a real number and thus belongs to the set of real numbers, <em>R</em>. And since each of these ordered lists are four-dimensional, then they live in <em>R⁴</em>. Since <em>R⁴</em> is a vector space, then these ordered lists can be called vectors.</p><p>Notice that each entry in these vectors represents a dimension in <em>R⁴</em> where each dimension corresponds to a feature in the dataset (like sepal length, sepal width, etc…). That is, each feature in your data can be considered a random variable. And since they’re random variables, we can do some descriptive statistics like find their means and standard deviations.</p><p>As you’ve already represented each row in the table as a vector, you can calculate the means and standard deviations of each random variable in one fell swoop. This is the power of vectors.</p><p>Pretend that the rows/vectors <em>(5.1, 3.5, 1.4, 0.2)</em> and <em>(4.9 , 3, 1.4, 0.2)</em> were all the data you had in the table above. Using two of the vector space rules, scalar multiplication and addition, we can easily calculate the means of every random variable we have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*czWwtb78bQ01om6kzF3NCQ.png" /><figcaption>Scaling and adding vectors to find the mean</figcaption></figure><p>So the mean of sepal length is <em>5</em>, the mean of sepal width is <em>3.25</em>, and so on. Representing your data as a set of vectors is not just aesthetically pleasing to look at, but also more performant in calculations since computers are optimized for computations involving vectors (turns out you can replace a lot of for loops with vector and matrix operations instead).</p><h3>Conclusion</h3><p>In the end, because you can represent your data as vectors which belong to a set with special rules called a vector space, many of the awesome things you do with your data like: linear feature transformations, standardizations, and dimensionality-reduction techniques can be justified by the rules of vector spaces.</p><p>And not only are vector spaces foundational to any work involving data, it constitutes the bedrock of linear algebra which is central to almost all areas of mathematics. Even if the phenomenon you’re studying is nonlinear, linear algebra is the go-to tool to use as a first-order approximation.</p><p>In short, vectors are tightly intertwined with your data and are very useful! So next time you’re importing data into a dataframe and performing a bunch of operations, remember that your computer is treating your data as a set of vectors and is happily applying transformations in a performant way. And all thanks to these little guys called vectors.</p><p><em>If you like what I’ve written here, make sure to check out my </em><a href="https://bobbywlindsey.com/"><em>personal blog</em></a><em> where I have articles that aren’t available on Medium.</em></p><p><em>Originally published at </em><a href="https://www.bobbywlindsey.com/2019/03/05/practical-look-at-vectors-and-your-data/"><em>bobbywlindsey.com</em></a><em> on March 5, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95bde21b37d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-practical-look-at-vectors-and-your-data-95bde21b37d1">A Practical Look at Vectors and Your Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Doing Good Science is Hard and How to Do it Better]]></title>
            <link>https://medium.com/data-science/why-doing-good-science-is-hard-and-how-to-do-it-better-61f977730891?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/61f977730891</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 22:32:33 GMT</pubDate>
            <atom:updated>2020-11-15T21:56:46.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ykhSZlqEdH-zompUvoVxMA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/Kr8Tc8Rugdk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Steve Johnson</a> on <a href="https://unsplash.com/search/photos/difficult?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><a href="https://bobbywlindsey.ck.page/5dca5d4310">Want to get better at data science?</a></p><p>Doing good science is hard and a lot of experiments fail. Although the scientific method helps to reduce uncertainty and lead to discoveries, its path is full of potholes. In this post, you’ll learn about common p-value misinterpretations, p-hacking, and the problem with performing multiple hypothesis tests. Of course, not only are the problems presented, but their potential solutions as well. By the end of the post, you should have a good idea of some of the pitfalls of hypothesis testing, how to avoid them, and an appreciation for why doing good science is so hard.</p><h3>P-Value Misinterpretations</h3><p>There are many ways to misinterpret a p-value. By definition, a p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming the null hypothesis is true.</p><p>What the p-value is <em>not</em>:</p><ul><li>A measure of the size of the effect or the strength of the evidence</li><li>The chance that the intervention is effective</li><li>A statement that the null hypothesis is true or false</li><li>A statement that the alternative hypothesis is true or false</li></ul><p>If you want to measure the strength of the evidence or size of effect, then you need to calculate the effect size. This can be done with Pearson’s r correlation, standardized difference of means, or other methods. Reporting the effect size in your research is suggested since p-values will tell you the likelihood that experimental results differ from chance expectations but not the relative magnitude of the experimental treatment or the size of the experimental effect.</p><p>P-values also don’t tell you the chance that the intervention is effective but calculating precision does, and base rates influence this calculation. If the base rate for the intervention is low, this opens the door to many opportunities for false positives even if a hypothesis test shows a statistically significant result. For example, if the chance the intervention is effective is 65%, then there is <em>still</em> only a 65% chance that the intervention was actually effective while leaving a false discovery rate of 35%. Neglecting the impact of base rates is known as the <a href="http://neglecting/%20the%20impact%20of%20base%20rates%20on%20hypothesis%20tests%20is%20known%20as%20the%20base%20rate%20fallacy">base rate fallacy</a> and happens more often than you think.</p><p>Lastly, p-values also can’t tell you whether a hypothesis is true or false. Statistics is an inferential framework and there’s no way to know for sure if some hypothesis is true or not. Remember, <a href="https://bobbywlindsey.com/math/2019/02/13/how-to-really-prove-something/"><em>there’s no such thing as proof in science</em></a>.</p><h3>The P-Hacking Problem</h3><p>As a scientist, one of your degrees of freedom when setting up a hypothesis test is deciding which variables to include in the data you test. Your hypothesis will, to a degree, influence which variables you might include in the data and after testing the hypothesis with those variables, you might get a p-value greater than 5%.</p><p>At this point, you might be tempted to try different variables in your data and retest. But if you try enough combinations of variables and test each scenario, you’re likely to get a p-value of 5% or less as demonstrated by this app in <a href="https://fivethirtyeight.com/features/science-isnt-broken/#part1">this fivethirtyeight blog post</a>. It’s called <em>p-hacking</em>, and it can allow you to achieve a p-value of 5% or less under competing alternative hypotheses.</p><p>There are at least a few problems with this:</p><ul><li>Since you can get a statistically significant p-value under competing alternative hypotheses as a result of the data you choose to include in testing, p-hacking doesn’t help you get closer to the truth of the thing you’re studying. Even worse, if such results are published and the research makes its way into conventional wisdom, it’ll be <a href="https://xkcd.com/386/">difficult to remove</a>.</li><li>As the number of hypothesis tests performed increases, the rate of false positives (i.e. erroneously calling a null finding significant) increases.</li><li>You might be falling victim to confirmation bias by ignoring the results of other hypothesis tests performed and only considering results of the tests that align with your beliefs.</li><li>Since many journals require a p-value of 5% or less for publication, it creates an incentive for you to p-hack your way to this 5% threshold creating not only an ethical dilemma but also lower quality research.</li></ul><h3>Addressing P-Hacking</h3><p>To help mitigate p-hacking, you should disclose the number of hypotheses explored during the study, all data collection decisions, all statistical analyses conducted, and all p-values computed. If you performed multiple hypothesis tests without a strong basis for expecting the result to be statistically significant, as can happen in genomics where genotypes for millions of genetic markers can be measured and tested, you should verify that there was some sort of control for the family-wise error rate or false discovery rate (as discussed in the next section). Otherwise, the study might not be meaningful.</p><p>It might also be a good idea to report the power of the hypothesis test. That is, report <em>1 — the probability of not rejecting the null hypothesis when it’s false</em>. Keep in mind that power can be influenced by the sample size, significance level, variability in the dataset, and if the true parameter is far from the parameter assumed by the null hypothesis. In short, the greater the sample size, the greater the power. The greater the significance level, the greater the power. The lower the variability in the dataset, the greater the power. And the further away the true parameter is from the parameter assumed by the null hypothesis, the greater the power.</p><h3>Testing Multiple Hypotheses with the Bonferroni Correction</h3><p>Since the probability of false positives increases as the number of hypothesis tests performed increases, it is necessary to try and control this. As such, you might want to control the probability of one or more false positives out of all hypothesis tests conducted. This is sometimes called the <em>family-wise error rate</em>.</p><p>One way to control for this is to set the significance level to <em>α/n</em> where <em>n</em> is the number of hypothesis tests. This kind of correction is called the <a href="https://en.wikipedia.org/wiki/Bonferroni_correction">Bonferroni correction</a> and ensures that the family-wise error rate is less than or equal to <em>α</em>.</p><p>However, this correction can be too strict especially if you’re performing many hypothesis tests. The reason is since you’re controlling for the family-wise error rate, you also might be missing some true positives that existed at a higher significance level. Clearly there’s a balance to be struck between increasing the power of the hypothesis test (i.e. increasing the probability of rejecting the null hypothesis when the alternative hypothesis is true) and controlling for false positives.</p><h3>Testing Multiple Hypotheses with the Benjamini-Hochberg Procedure</h3><p>Instead of trying to control for the family-wise error rate, you can instead try to control for the false discovery rate which is the proportion of all the hypothesis tests identified as having statistically significant results that actually don’t have statistically significant results. In other words, the false discovery rate is equal to <em>FP/(FP + TP)</em>.</p><p>Controlling for the false discovery rate should help you identify as many hypothesis tests with statistically significant results as possible, but still try to keep a relatively low proportion of false positives. Like <em>α</em> which controls the false positive rate, we similarly use another significance level, <em>β</em>, which controls the false discovery rate.</p><p>The procedure you can use to control the false discovery rate is called the <a href="https://en.wikipedia.org/wiki/False_discovery_rate">Benjamini-Hochberg procedure</a>. You first choose a <em>β</em>, the significance level for the false discovery rate. Then calculate the p-values for all null hypothesis tests performed and sort from lowest to highest with <em>i</em> being the index of the p-value in the list. Now find the index, <em>k</em>, of the largest p-value such that it’s less than or equal to<em> i/m*β</em> where <em>m</em> is the number of null hypothesis tests performed. All null hypothesis tests with p-value index <em>i ≤ k</em> are considered statistically significant by the Benjamini-Hochberg procedure.</p><h3>Conclusion</h3><p>As you can see, doing good science does not just involve performing a null hypothesis test and publishing your findings when you get a p-value less than or equal to 5%. There are ways to misinterpret p-values, to tweak data to get the right p-value for a hypothesis that you’re convinced of, and to perform enough tests with different samples of data until you get the desired p-value.</p><p>But now that you you’re aware of the potholes and are armed with some ways to avoid them, I hope it helps you improve the quality of your research and get you closer to the truth.</p><p><em>If you like what I’ve written here, make sure to check out my </em><a href="https://www.bobbywlindsey.com/"><em>personal blog</em></a><em> where I have articles that aren’t available on Medium.</em></p><h3>References</h3><ul><li><a href="https://blogs.plos.org/absolutely-maybe/2016/04/25/5-tips-for-avoiding-p-value-potholes/">5 Tips For Avoiding p-Value Potholes</a></li><li><a href="https://newonlinecourses.science.psu.edu/stat414/node/304/">Definition of Power</a></li><li><a href="http://web9.uits.uconn.edu/lundquis/ASA%20statement%20on%20p%20values.pdf">The American Statistical Association’s Statement on p-Values</a></li><li><a href="https://royalsocietypublishing.org/doi/full/10.1098/rsos.140216">An Investigation of the False Discovery Rate and the Misinterpretation of p-values</a></li><li><a href="https://stats.stackexchange.com/questions/336455/fpr-false-positive-rate-vs-fdr-false-discovery-rate">False Positive Rate vs False Discovery Rate</a></li><li><a href="http://www.math.tau.ac.il/~ybenja/MyPapers/benjamini_hochberg1995.pdf">Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing</a></li><li><a href="https://www.khanacademy.org/math/ap-statistics/tests-significance-ap/error-probabilities-power/v/introduction-to-power-in-significance-tests">Introduction to Power in Significance Tests — Khan Academy</a></li><li><a href="https://en.wikipedia.org/wiki/Sensitivity_and_specificity">Sensitivity and Specificity — Wikipedia</a></li><li><a href="https://www.statisticsdonewrong.com/p-value.html">The p-Value and the Base Rate Fallacy — Statistics Done Wrong</a></li><li><a href="http://www.bwgriffin.com/gsu/courses/edur9131/content/Effect_Sizes_pdf5.pdf">How to Calculate Effect Sizes from Published Research: A simplified Methodology</a></li><li><a href="https://en.wikipedia.org/wiki/Multiple_comparisons_problem">Multiple Comparisons Problem — Wikipedia</a></li></ul><p><em>Originally published at </em><a href="https://www.bobbywlindsey.com/2019/02/25/good-science-is-hard/"><em>bobbywlindsey.com</em></a><em> on February 25, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61f977730891" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-doing-good-science-is-hard-and-how-to-do-it-better-61f977730891">Why Doing Good Science is Hard and How to Do it Better</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essentials of Hypothesis Testing and the Mistakes to Avoid]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/essentials-of-hypothesis-testing-and-the-mistakes-to-avoid-b50079d35ee5?source=rss-14c7e73f0fe5------2"><img src="https://cdn-images-1.medium.com/max/1920/1*kcR6UIjR2g6rvmRZsFgOEg.jpeg" width="1920"></a></p><p class="medium-feed-snippet">Hypothesis testing is the bedrock of the scientific method and by implication, scientific progress. It allows you to investigate a thing&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/data-science/essentials-of-hypothesis-testing-and-the-mistakes-to-avoid-b50079d35ee5?source=rss-14c7e73f0fe5------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/essentials-of-hypothesis-testing-and-the-mistakes-to-avoid-b50079d35ee5?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b50079d35ee5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[science-education]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Tue, 19 Feb 2019 20:57:00 GMT</pubDate>
            <atom:updated>2020-11-15T21:57:48.293Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Really Prove Something]]></title>
            <link>https://medium.com/@bobbylindsey/how-to-really-prove-something-ec38ace4e41d?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec38ace4e41d</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Wed, 13 Feb 2019 16:24:23 GMT</pubDate>
            <atom:updated>2020-11-15T21:58:02.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQZBP2f1zfjG3zfLtVqHXg.jpeg" /></figure><p><a href="https://bobbywlindsey.ck.page/5dca5d4310">Want to get better at data science?</a></p><p>“Prove it” is a phrase thrown around with glee. But rarely are you really challenging a friend to provide a rigorous proof of some statement you disagree with. In science, there is no proof. Instead, the scientific method encourages you to form a hypothesis, collect data, and test that hypothesis. Repeating this process advances you closer and closer to the truth, but nowhere in this process can you say you’ve <em>proven</em> a hypothesis. This would be impossible as you would have to exhaustively collect all data in the known universe to see if your hypothesis still holds and even then, your measurements will always be imprecise. The scientific body of knowledge will always be subject to revision as more data is collected and better measuring instruments are built.</p><p>On the other hand, mathematics is built from the ground up using a set of simple statements called axioms. These axioms are statements composed of sets, functions, and logical symbols. I’ve always found this a beautiful fact of mathematics; that it’s nothing more than some sets, functions, and logic. Of course this means that now you can build the whole of mathematics and ensure that every new statement is coherent with previous ones. This naturally endows anyone looking to prove a mathematical statement with the ability to:</p><ul><li>prove it with 100% certainty</li><li>disprove it with 100% certainty</li><li>prove that it is not provable with 100% certainty (ref. <a href="https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems">Gödel’s incompleteness theorems</a>)</li></ul><p>So how do you prove a mathematical statement? There are generally four methods:</p><ol><li>Implication</li><li>Contradiction</li><li>Contrapositive</li><li>Induction</li></ol><h3>Proof by Implication</h3><p>Proof by implication (also sometimes called proof by direct implication) is a method where you start with one fact, and use that fact to imply other facts. Through this chain of implications, you reach some conclusion. More simply, you’re saying “if A, then B”.</p><p>To drive this home, say you’re trying to prove that if <em>m</em> and <em>n</em> are perfect squares, then <em>mn</em> is a perfect square.</p><p>You first start off with the fact you know <em>m</em> and <em>n</em> are perfect squares. What does this mean? Well, <em>m = x²</em> and <em>n = y²</em> where <em>x</em> and <em>y</em> are some integers. Using that fact, you can show that <em>mn = x²y² = (xy)²</em>, thus showing that <em>mn</em> is also a perfect square.</p><p>In practice, proof by implication is a common first-approach to proving some statement. By working forward from A and backward from B, you eventually bridge the gap and create a chain of implications.</p><h3>Proof by Contradiction</h3><p>But sometimes that gap can’t be bridged. In these cases, it might be helpful to consider proving “if A, then B” by reaching a contradiction. The reasoning is as follows: if you’re trying to prove “if A, then B” and this statement is indeed true, then there must be a reason why B cannot be false. Your goal is to find that reason by seeking a contradiction. To do this, you assume A is true and (not B) is true, then work forward by implication. The hope is that you reach some contradiction and if you do, then you’ve successfully proven “if A, then B”.</p><p>You should try this with an example. Let <em>n</em> be an integer. If <em>n²</em> is even, then <em>n</em> is even. If you were to try and prove this using implication, you might say that since <em>n²</em> is even, then there exists some integer <em>m</em> such that <em>n² = 2m</em>. And since <em>n</em> is even, then there exists some integer <em>k</em> such that <em>n = 2k</em>. But how do you get the square root of <em>2m</em> to look like <em>2k</em>? This is where you can use the contradiction method.</p><p>Suppose <em>n</em> is an integer and <em>n²</em> is even <strong>and</strong> <em>n</em> is <em>odd</em>. Then there exists some integer <em>k</em> such that <em>n = 2k + 1</em>. This would imply that</p><p><em>n² = (2k + 1)² = 4k² + 4k + 1 = 2(2k² + 2k) + 1</em></p><p>If you let <em>z = 2k² + 2k</em> then <em>n² = 2z + 1</em>, an odd number. But wait! This is a contradiction since you had assumed <em>n²</em> was even. So “if <em>n²</em> is even, then <em>n</em> is even” must be true.</p><p>A proof by contradiction isn’t the only method you can try if implication doesn’t work. Sometimes you can just turn implication on its head.</p><h3>Proof by Contrapositive</h3><p>You’re still trying to prove “if A, then B” but it’s not working out. You might instead try “if not B, then not A”. In fact, this statement is the logical equivalent to “if A, then B” and is called its <em>contrapositive</em>. If you’re not familiar with why it’s logically equivalent, it happens to be a result of boolean logic which is usually represented as a <a href="https://en.wikipedia.org/wiki/Truth_table">truth table</a>. However, consider the example statement “if it’s raining outside, then the grass is wet”. The contrapositive of this statement would be “if the grass is <em>not</em> wet, then it’s <em>not</em> raining outside”. Both statements are logically equivalent to each other. And once the statement you’re trying to prove is in its contrapositive form, you can then use implication to complete the proof.</p><p>Take the statement we recently proved via contradiction :</p><blockquote><em>Let </em>n<em> be an integer. If </em>n²<em> is even, then </em>n<em> is even.</em></blockquote><p>Now take the contrapositive of this statement:</p><blockquote><em>Let </em>n<em> be an integer. If </em>n<em> is odd, then </em>n²<em> is odd.</em></blockquote><p>If <em>n</em> is odd, then there is some integer <em>k</em> such that <em>n = 2k + 1</em>. So by implication,</p><p><em>n² = (2k + 1)² = 4k² + 4k + 1 = 2(2k² + 2k) + 1</em></p><p>If you let <em>z = 2k² + 2k</em> then <em>n² = 2z + 1</em> which is an odd number. And the proof is complete.</p><p>Another way to think about a proof by contrapositive is that you suppose A and (not B) are both true and work forward to prove that A is false (i.e. not A). In essence, you’re really doing a proof by contradiction but this time you know what contradiction you’re looking for.</p><h3>Proof by Induction</h3><p>A proof by induction is a bit like falling dominos. The first domino falls. Then when any domino falls, the next one falls. Proof by induction is similar. You first start by proving the base case, <em>n = 1</em>. Then you assume the statement is true for <em>k</em> and show that it’s also true for <em>k + 1</em>. Once you’ve done that, then you’ve officially proven the statement for all <em>n</em>.</p><p>It’s now mathematical lore that in the late 1700s while in elementary school, Carl Friedrich Gauss was given busy work by his math teacher. He was told to sum the numbers from 1 to 100. After some clever observations, he found that he could easily calculate the total with the following formula, <em>n(n + 1)/2</em>. This is what you’ll prove by induction, that the sum of <em>n</em> positive integers is equal to the function <em>n(n + 1)/2</em>.</p><p>First you prove the base case where <em>n = 1</em>. Does the sum of 1 equal <em>f(n) = n(n + 1)/2</em>? Plugging in 1 for <em>n</em>, it most assuredly does. So the base case is proven. Now assume <em>f(k) = n(n + 1)/2</em> to be true. You want to show that <em>f(k + 1)</em> is also true. Well <em>f(k + 1)</em> is just the sum of the first <em>k + 1</em> positive integers</p><p><em>1 + 2 + …+ k + (k + 1)</em></p><p>Well this is equal to the sum of the first <em>k</em> positive integers plus <em>k + 1</em>, giving you <em>(k(k + 1)/2) + (k + 1)</em>. If you use some basic algebra to apply a common denominator and factor out some terms, you’ll end up with <em>(k + 1)((k + 1) + 1)/2</em>. Taking a closer look at that expression, you’ll see that it resembles our initial function, <em>n(n + 1)/2</em>. And now the proof is complete!</p><p>To wrap you head around what you just did, you assumed the function was true for <em>k</em> and proved it was also true for <em>k+1</em>. Since you proved it was true for <em>f(1)</em> (our base case), then you know it’s true for <em>f(2)</em>. And since you know it’s true for <em>f(2)</em>, then it’s true for <em>f(3)</em>, and so on… So by induction, you’ve proven the function must be true for all of <em>n</em> (i.e. <em>f(n) = n(n + 1)/2</em> is true).</p><h3>Logical Symbols in Statements</h3><p>With the above proof methods, you’re ready to tackle most proofs out there. However, in some statements you might see phrases (or their logical symbol equivalents) like “there exists”, “for all”, or “unique”. Statements that contain these phrases are still tackled with the four proof methods, but there are some helpful guidelines for addressing each one. For simplicity, assume you’re still thinking about the statement, “if A, then B”.</p><h4>Existential Crisis</h4><p>If you see that B has the phrase “there exists”, do the following:</p><ol><li>Suppose A is true</li><li>Guess or construct the object having the certain property and show that the something happens</li><li>Conclude the desired object exists</li></ol><h4>Me Too</h4><p>If you see that B has the term “for all” or “for every”, do the following:</p><ol><li>Suppose A is true</li><li>Choose an object with the certain property</li><li>Use implication to conclude that the something happens</li></ol><h4>The Only One</h4><p>If you see that B has the word “unique”, do the following:</p><ol><li>Suppose A is true and that two such objects exist</li><li>Use implication to conclude that the two objects are equal thus showing there’s really only one</li></ol><h4>This or That</h4><p>If B has the form “C or D”:</p><ol><li>Suppose A is true and (not C) is true</li><li>Use implication to prove that D is true</li></ol><p>Or you could also:</p><ol><li>Suppose A is true and (not D) is true</li><li>Use implication to prove that C is true</li></ol><h3>Conclusion</h3><p>Through the use of making iron-clad logical deductions, you now know how to <em>really </em>prove something. Hedging your bets is no longer required. If you want to learn more about how to do proofs, Daniel Solow has a really good book called <a href="https://www.amazon.com/How-Read-Proofs-Introduction-Mathematical/dp/1118164024/ref=sr_1_1?ie=UTF8&amp;qid=1550073433&amp;sr=8-1&amp;keywords=how+to+read+and+do+proofs">How to Read and Do Proofs</a>. Jeremy Kun also has some good primers on the methods of proof available at <a href="https://jeremykun.com/primers/">his blog</a>.</p><p><em>If you like what I’ve written here, make sure to check out my </em><a href="https://bobbywlindsey.com/"><em>personal blog</em></a><em> where I have articles that aren’t available on Medium.</em></p><p><em>Originally published at </em><a href="https://www.bobbywlindsey.com/2019/02/13/how-to-really-prove-something/"><em>bobbywlindsey.com</em></a><em> on February 13, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec38ace4e41d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker for Data Scientists]]></title>
            <link>https://medium.com/data-science/docker-for-data-scientists-5732501f0ba4?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5732501f0ba4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[application-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Tue, 21 Aug 2018 12:59:43 GMT</pubDate>
            <atom:updated>2020-11-15T22:01:31.795Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ycw8qm-LUp8OTpdz.jpg" /></figure><p><a href="https://bobbywlindsey.ck.page/5dca5d4310">Want to get better at data science?</a></p><p>Docker is hot in the developer world and although data scientists aren’t strictly software developers, Docker has some very useful features for everything from data exploration and modeling to deployment. And since major services like AWS support Docker containers, it’s even easier to implement Continuous Integration Continuous Delivery with Docker. In this post, I’ll show you how to use Docker in a data science context.</p><h3>What is Docker?</h3><p>It’s a software container platform that provides an isolated container for us to have everything we need for our experiments to run. Essentially, it’s a light-weight VM that’s built from a script that can be version controlled; so we can now version control our data science environment! Developers use Docker when collaborating on code with coworkers and they also use it to build agile software delivery pipelines to ship new features faster. Any of this sound familiar?</p><h3>Docker Terminology</h3><p>I have a mathematics background, so it’s hard to avoid definitions!</p><p><strong>Containers</strong>: very small user-level virtualization that helps you build, install, and run your code</p><p><strong>Images</strong>: a snapshot of your container</p><p><strong>Dockerfile</strong>: a yaml-based file that’s used to build your image; this is what we can version control</p><p><strong>Dockerhub</strong>: GitHub for your Docker images; you can set up Dockerhub to automatically build an image anytime you update your Dockerfile in GitHub</p><h3>Why Docker is So Awesome for Data Science</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*9BwzdciPT-AtC8eC.png" /><figcaption>Source: <a href="http://www.developermemes.com/2013/12/13/worked-fine-dev-ops-problem-now/">developermemes</a></figcaption></figure><p>Ever heard these comments from your coworkers?</p><ul><li>“Not sure why it’s not working on your computer, it’s working on mine.”</li><li>“It’s a pain to install everything from scratch for Linux, Windows, and MacOS, and trying to build the same environment for each OS.”</li><li>“Can’t install the package that you used, can you help me out?”</li><li>“I need more compute power. I could use AWS but it’ll take so long just to install all those packages and configure settings just like I have it on my machine.”</li></ul><p>For the most part, these concerns are easily resolved by Docker. The exception at the moment of posting is GPU support for Docker images, which only run on Linux machines. Other than that, you’re golden.</p><h3>Docker for Python and Jupyter Notebook</h3><p>Check out this <a href="https://github.com/bobbywlindsey/docker-data-science/blob/master/Dockerfile">Dockerfile</a>.</p><pre># reference: <a href="https://hub.docker.com/_/ubuntu/">https://hub.docker.com/_/ubuntu/</a><br>FROM ubuntu:16.04</pre><pre># Adds metadata to the image as a key value pair example LABEL version=&quot;1.0&quot;<br>LABEL maintainer=&quot;Your Name &lt;some_email@domain.com&gt;&quot;</pre><pre># Set environment variables<br>ENV LANG=C.UTF-8 LC_ALL=C.UTF-8</pre><pre># Create empty directory to attach volume<br>RUN mkdir ~/GitProjects</pre><pre># Install Ubuntu packages<br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    wget \<br>    bzip2 \<br>    ca-certificates \<br>    build-essential \<br>    curl \<br>    git-core \<br>    htop \<br>    pkg-config \<br>    unzip \<br>    unrar \<br>    tree \<br>    freetds-dev</pre><pre># Clean up<br>RUN apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/*</pre><pre># Install Jupyter config<br>RUN mkdir ~/.ssh &amp;&amp; touch ~/.ssh/known_hosts<br>RUN ssh-keygen -F github.com || ssh-keyscan github.com &gt;&gt; ~/.ssh/known_hosts<br>RUN git clone <a href="https://github.com/bobbywlindsey/dotfiles.git">https://github.com/bobbywlindsey/dotfiles.git</a><br>RUN mkdir ~/.jupyter<br>RUN cp /dotfiles/jupyter_configs/jupyter_notebook_config.py ~/.jupyter/<br>RUN rm -rf /dotfiles</pre><pre># Install Anaconda<br>RUN echo &#39;export PATH=/opt/conda/bin:$PATH&#39; &gt; /etc/profile.d/conda.sh<br>RUN wget --quiet <a href="https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh">https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh</a> -O ~/anaconda.sh<br>RUN /bin/bash ~/anaconda.sh -b -p /opt/conda<br>RUN rm ~/anaconda.sh</pre><pre># Set path to conda<br>ENV PATH /opt/conda/bin:$PATH</pre><pre># Update Anaconda<br>RUN conda update conda &amp;&amp; conda update anaconda &amp;&amp; conda update --all</pre><pre># Install Jupyter theme<br>RUN pip install msgpack jupyterthemes<br>RUN jt -t grade3</pre><pre># Install other Python packages<br>RUN conda install pymssql<br>RUN pip install SQLAlchemy \<br>    missingno \<br>    json_tricks \<br>    bcolz \<br>    gensim \<br>    elasticsearch \<br>    psycopg2-binary</pre><pre># Configure access to Jupyter<br>WORKDIR /root/GitProjects<br>EXPOSE 8888<br>CMD jupyter lab --no-browser --ip=0.0.0.0 --allow-root --NotebookApp.token=&#39;data-science&#39;</pre><p>If you’ve ever installed packages in Ubuntu, this should look very familiar. In short, this Dockerfile is a script to automatically build and setup a light-weight version of Ubuntu with all the necessary Ubuntu packages and Python libraries needed to do [my] data science exploration with Jupyter Notebooks. The best part is that this will run the same way whether I’m on MacOS, Linux, or Windows — no need to code separate install scripts and third-party tools to have the same environment in each operating system.</p><p>To build a Docker image from this Dockerfile, all you need to do is execute</p><pre>docker build -t bobbywlindsey/docker-data-science .</pre><p>in the command line and Bob’s your uncle. To run the image, you have two options — you can either run the image interactively (which means you’ll see the output of your Jupyter Notebook server in real time) or in detached mode (where you can drop into the image’s terminal and play around).</p><p>To run the image interactively on Windows, execute</p><pre>docker run -it -v ~/GitProjects:/root/GitProjects --network=host -i bobbywlindsey/docker-data-science</pre><p>Otherwise,</p><pre>docker run -it -v ~/GitProjects:/root/GitProjects -p 8888:8888 -i bobbywlindsey/docker-data-science</pre><p>To run the image in detached mode for linux:</p><pre>docker run -d --name data-science -v ~/GitProjects:/root/GitProjects --network=host -i bobbywlindsey/docker-data-science<br>docker exec -it data-science bash</pre><p>or for MacOS and Windows:</p><pre>docker run -d --name data-science -v ~/GitProjects:/root/GitProjects -p 8888:8888 -i bobbywlindsey/docker-data-science<br>docker exec -it data-science bash</pre><p>Not too bad! I realize those run commands might be a bit much to type up, so there’s a couple options that I see. You can either alias those commands or you can use a docker-compose file instead.</p><h3>Using Multiple Containers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*eHa3BEk94LAD5TSz.jpg" /><figcaption><a href="https://www.google.com/url?sa=i&amp;source=images&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwi4trnPp6TcAhVQKawKHfayAKwQjxx6BAgBEAI&amp;url=https%3A%2F%2Fhub.docker.com%2Fr%2Fmonitoringartist%2Fdocker-killer%2F&amp;psig=AOvVaw23qhvR2c-y1jCPiPCErl4a&amp;ust=1531854164638537">Source</a></figcaption></figure><p>I won’t go into <a href="https://github.com/docker/compose">Docker Compose</a> much here, but as an example, this a docker-compose file I have to run a Docker image used for a <a href="https://github.com/bobbywlindsey/docker-jekyll">Jekyll blog</a>:</p><pre>version: &#39;3&#39;<br>services:<br>  site:<br>    environment:<br>      - JEKYLL_ENV=docker<br>    image: bobbywlindsey/docker-jekyll<br>    volumes:<br>      - ~/Dropbox/me/career/website-and-blog/bobbywlindsey:/root/bobbywlindsey<br>    ports:<br>      - 4000:4000<br>      - 35729:35729</pre><p>With that file, your run command then becomes:</p><pre>docker-compose run --service-ports site</pre><p>But Docker Compose is much more capable than just using it as a substitute for aliasing your run commands. Your docker-compose file can configure multiple images and by using a single command, you create and start all your services at once. For example, let’s say you build one Docker image to preprocess your data, another to model the data, and another to deploy your model as an API. You can use docker-compose to manage each image’s configurations and run them with a single command.</p><h3>Conclusion</h3><p>Even though Docker might require a learning curve for some data scientists, I believe it’s well worth the effort and it doesn’t hurt to brush up those DevOps skills. Have you used Docker for your data science efforts?</p><p><em>If you like what I’ve written here, make sure to check out my </em><a href="https://bobbywlindsey.com/"><em>personal blog</em></a><em> where I have articles that aren’t available on Medium.</em></p><p><em>Originally published at </em><a href="https://www.bobbywlindsey.com/2018/07/16/docker-for-data-scientists/"><em>bobbywlindsey.com</em></a><em> on July 16, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5732501f0ba4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/docker-for-data-scientists-5732501f0ba4">Docker for Data Scientists</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Few Useful Things to Know About Machine Learning]]></title>
            <link>https://medium.com/@bobbylindsey/a-few-useful-things-to-know-about-machine-learning-bf66906467db?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf66906467db</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Thu, 14 Jun 2018 09:35:26 GMT</pubDate>
            <atom:updated>2020-11-15T21:58:54.228Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://bobbywlindsey.ck.page/5dca5d4310">Want to get better at data science?</a></p><p><a href="https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf">A Few Useful Things to Know about Machine Learning</a> is a high-level machine learning paper written by Pedro Domingos of the computer science and engineering department at the University of Washington. His paper details some useful machine learning guidelines and the following are some highlights I took from it.</p><p>When building models, we’re after generalization so there are a few things to note:</p><ul><li>Cross-validation is a must — that is, randomly dividing your training data into <em>n</em> subsets, holding out each subset while training on the rest, validating your model on the held-out subset and then averaging the results.</li><li>Although cross-validation helps you to choose the best model parameters, your model is still biased toward the validation set. As a final unbiased test, you need to test your model on a subset of data that wasn’t used for training or validation</li><li>The objective function is only a proxy for the true goal so we may not even need a global optimum</li></ul><p>Problems with high-dimensional datasets:</p><ul><li>Choosing to increase the number of dimensions of your dataset increases the entire input space. For example, if you have 100 variables each taking a binary value, then your input space is of size 2¹⁰⁰ which is a number greater than the number of atoms in the universe. In this scenario, even if you have a million samples, there are still 2¹⁰⁰-10⁶ samples whose classes you don’t know. This makes generalization a lot harder.</li><li>The more irrelevant variables you have, the more they will dominate the relevant variables and your model effectively makes random predictions since it was trained on noise</li><li>Similarity-based reasoning that models use to learn breaks down in higher dimensions since samples in higher dimensions appear to be sparse and therefore similar in many ways</li></ul><p>Feature engineering is really important:</p><ul><li>Automate as much of it as you can</li><li>Features that look irrelevant in isolation may be relevant in combination</li><li>But brute-forcing which features have useful combinations might be intractable so use your smarts when feature engineering</li></ul><p>On models:</p><ul><li>A less sophisticated algorithm with lots of a data can beat a clever algorithm with only a moderate amount of data</li><li>Although model accuracy is important, insights and reduction in human labor are usually more important; this makes decision trees and other rule-based learners attractive</li><li>Model ensembles are awesome — this includes bagging, boosting, and stacking</li><li>There isn’t a mandatory connection between the number of parameters a model has and its tendency to overfit. SVMs can have <em>a lot</em> of parameters, but can avoid overfitting. Even though <em>sign(sin(ax))</em> only has one parameter, it can discriminate an arbitrarily large, arbitrarily labeled set of points on the <em>x </em>axis.</li><li>A model with a larger hypothesis space that tries fewer hypotheses from it is less likely to overfit than a model that tries more hypotheses from a smaller space</li><li>Simpler models should be preferred because simplicity is a virtue in its own right, not because of a hypothetical connection with accuracy</li><li>Just because a function can be represented by a model doesn’t mean the model can learn the function. This is because the number of training samples you have might not be enough and most models can only learn a tiny subset of all possible functions and these subsets are different for each model; so usually the model gets stuck in a local optima.</li></ul><p><em>Originally published at </em><a href="https://www.bobbywlindsey.com/2018/06/14/a-few-useful-things-to-know-about-machine-learning/"><em>bobbywlindsey.com</em></a><em> on June 14, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf66906467db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Remote Computing with Jupyter Notebooks]]></title>
            <link>https://medium.com/data-science/remote-computing-with-jupyter-notebooks-5b2860f761e8?source=rss-14c7e73f0fe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b2860f761e8</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[remote-working]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[computing]]></category>
            <dc:creator><![CDATA[Bobby Lindsey]]></dc:creator>
            <pubDate>Thu, 10 Aug 2017 11:56:31 GMT</pubDate>
            <atom:updated>2020-11-15T21:59:12.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QTsuaid22dh8UXcyplbB-g.jpeg" /><figcaption><a href="https://www.lynda.com/Linux-tutorials/Linux-Desktops-Remote-Access/517442-2.html">Source</a></figcaption></figure><p><a href="https://bobbywlindsey.ck.page/5dca5d4310">Want to get better at data science?</a></p><p>Awhile ago, I had AWS set up to provide me a unique URL that I could navigate to and use Jupyter Notebooks. I admired the convenience and the ability to just start a computation and close my laptop knowing full well my computations continued working away. However, using an AWS P2 instance can get very costly depending on your usage, which for me would be around $600 per month. So, I figured I could just build a computer with that kind of money which could serve as a deep learning rig along with the occasional video gaming.</p><p>This post describes the configuration setup <em>once</em> you have already built your computer and installed a flavor of Linux like Ubuntu. Turns out that the following configuration was way easier for me to setup than AWS, and with the help of aliases, connecting to my server is easier than ever. Let’s get started!</p><h3>Installation</h3><p>So first you need to install the following on your Ubuntu server:</p><ul><li><a href="https://www.continuum.io/downloads">Anaconda</a>, which will provide a lot of the default Python packages you’d need</li><li>openssh-server, which can be installed with the following: sudo apt-get install openssh-server -y</li><li>tmux, which can be installed with sudo apt-get install tmux -y</li></ul><p>If you ever need to see the status of openssh-server or restart it, type the below into your terminal:</p><pre>systemctl status ssh </pre><pre>sudo service ssh restart</pre><h3>Connecting Locally To Your Server</h3><p>To make sure most things are set up correctly, we first need to verify that you can connect to your server on your local network.</p><p>Ok! So on your server, open your sshd_config file located at /etc/ssh/sshd_config. To make changes to it, you’ll need sudo privileges. Once the file is open, you’ll need to specify what port you’ll want to use when connecting in. Whatever you choose, I highly advise not using the default port 22. Let’s say you decide to use port 22222 instead. There’s an entry in your sshd_config file called Port and you should edit it as such:</p><pre>Port 22222</pre><p>Under AllowUsers, put the username you use when logging into your server.</p><pre>AllowUsers your_username</pre><p>Next, set PasswordAuthentication to yes.</p><p>Lastly, to make sure Ubuntu won’t block incoming web traffic on port 8888, we need to adjust its iptables:</p><pre>sudo iptables -I INPUT -p tcp -s 0.0.0.0/0 --dport 8888 -j ACCEPT </pre><pre>sudo iptables -I INPUT -p tcp -s 0.0.0.0/0 --dport 443 -j ACCEPT </pre><pre>sudo netfilter-persistent save</pre><p>Now, take note of your server’s IP address. This can be found by typing ifconfig into your terminal and looking for something like: inet 192.168.1.112. Once you’ve identified your server’s IP address on your local network, it’s time to pick up your laptop and try to log into it:</p><pre>ssh your_username@192.168.1.112 -p 22222</pre><p>If you get a terminal prompt, you’re in!</p><h3>Connecting Remotely To Your Server</h3><p>Now the whole point of setting up remote computing is so that you can leave your house and remote into your server while on someone else’s network. To do this only requires a few changes (which require you to still be on your own network):</p><ul><li>If you don’t already have a set of public and private keys on your laptop, <a href="https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/">generate them</a></li><li>Copy your public key to your server: ssh-copy-id your_username@192.168.1.112 -p 22222</li><li>Identify your server’s WAN IP address by typing in your server’s terminal: curl &#39;https://api.ipify.org&#39; (note that your ISP changes this address frequently which is why I use <a href="http://amzn.to/2vnHIjk">Google Wifi</a> which allows me to check my WAN address from anywhere)</li><li>On your router, turn on port forwarding. Using our example, you need to forward port 22222 to port 22222.</li></ul><p>Now try to remote into your server using the WAN address you found!</p><pre>ssh your_username@server_wan_ip -p 22222</pre><p>If you see a prompt, well done! One last thing is to set PasswordAuthentication to no in your sshd_config file since now you’re logging in with a ssh key; that way no one can try brute-forcing your password.</p><p>You can now access your server from outside your network, go grab yourself that Starbucks coffee :).</p><h3>Starting a Remote Jupyter Notebook</h3><p>Now that all the hard work is done, you can easily use a remote Jupyter Notebook with the following steps:</p><ol><li>ssh into your server: ssh your_username@server_wan_ip -p 22222</li><li>Start a new tmux session that you can easily detch from later: tmux new -s session-name</li><li>Start jupyter-notebook without a browser: jupyter-notebook --no-browser --port=8889</li><li>Now in a new terminal on your laptop, forward your server’s port traffic to your laptop’s local port: ssh -N -L localhost:8888:localhost:8889 your_username@server_wan_ip -p 22222</li><li>In your web browser, navigate to localhost:8888/tree and you should see your Jupyter Notebooks!</li></ol><p>Now you can easily use Jupyter Notebooks on your laptop, but instead using your server’s beefy resources to do the computing.</p><p>One last thing, after having figured out the steps above, I thought I’d make the process more simple by using aliases and functions. The below are the relevant lines I added to my laptop’s .bashrc file:</p><pre>server-connect() {<br>    ssh your_username@$1 -p 22222<br>}</pre><pre>jn-connect() {<br>    ssh -N -L localhost:8888:localhost:8889 your_username@$1 -p 22222<br>}</pre><p>And the lines I added to my server’s .bashrc file:</p><pre>alias jn-remote=&quot;jupyter-notebook --ip=&#39;*&#39; --no-browser --port=8889&quot;</pre><p>Now the 5 steps above become:</p><ol><li>server-connect server_wan_ip</li><li>tmux new -s session-name</li><li>jn-remote</li><li>open new terminal and type jn-connect server_wan_ip</li><li>navigate to localhost:8888/tree</li></ol><p>And that’s it! Now you can chill with your laptop in a coffee shop while running those crazy neural nets you’re building. Best of luck!</p><p><em>If you like what I’ve written here, make sure to check out my </em><a href="https://bobbywlindsey.com/"><em>personal blog</em></a><em> where I have articles that aren’t available on Medium.</em></p><p><em>Originally published at </em><a href="https://www.bobbywlindsey.com/2017/08/10/remote-computing-with-jupyter-notebooks/"><em>bobbywlindsey.com</em></a><em> on August 10, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b2860f761e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/remote-computing-with-jupyter-notebooks-5b2860f761e8">Remote Computing with Jupyter Notebooks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>