<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Erin Hoffman on Medium]]></title>
        <description><![CDATA[Stories by Erin Hoffman on Medium]]></description>
        <link>https://medium.com/@erin_r_hoffman?source=rss-29b741b41e48------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TmufqbIoMFl_CsDseijMRA.jpeg</url>
            <title>Stories by Erin Hoffman on Medium</title>
            <link>https://medium.com/@erin_r_hoffman?source=rss-29b741b41e48------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:04:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@erin_r_hoffman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tutorial: Removing Large Files from Git]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/analytics-vidhya/tutorial-removing-large-files-from-git-78dbf4cf83a?source=rss-29b741b41e48------2"><img src="https://cdn-images-1.medium.com/max/2600/1*ZJaUmqEEnLkzs0MchZcF3g.png" width="2719"></a></p><p class="medium-feed-snippet">How to overcome the &#x201C;error: GH001: Large files detected&#x201D; error message when you&#x2019;re pushing changes to GitHub</p><p class="medium-feed-link"><a href="https://medium.com/analytics-vidhya/tutorial-removing-large-files-from-git-78dbf4cf83a?source=rss-29b741b41e48------2">Continue reading on Analytics Vidhya »</a></p></div>]]></description>
            <link>https://medium.com/analytics-vidhya/tutorial-removing-large-files-from-git-78dbf4cf83a?source=rss-29b741b41e48------2</link>
            <guid isPermaLink="false">https://medium.com/p/78dbf4cf83a</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Erin Hoffman]]></dc:creator>
            <pubDate>Wed, 09 Sep 2020 11:25:00 GMT</pubDate>
            <atom:updated>2024-02-27T09:39:12.713Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Decorators in Python: Why and How to Use Them and Write Your Own]]></title>
            <link>https://medium.com/swlh/decorators-in-python-why-and-how-to-use-them-and-write-your-own-c1da4ed9f3a9?source=rss-29b741b41e48------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1da4ed9f3a9</guid>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Erin Hoffman]]></dc:creator>
            <pubDate>Sun, 23 Aug 2020 10:01:00 GMT</pubDate>
            <atom:updated>2020-08-23T10:01:00.975Z</atom:updated>
            <content:encoded><![CDATA[<h4>Level up your ability to follow SOLID principles with an intermediate-level Python technique.</h4><figure><img alt="Photo of hands typing on a laptop, with a Python textbook sitting to the left" src="https://cdn-images-1.medium.com/max/1024/1*dddN1KmEUsx6SKpOB51mzg.jpeg" /><figcaption>Image credit: <a href="https://www.wocintechchat.com/blog/wocintechphotos">https://www.wocintechchat.com/blog/wocintechphotos</a></figcaption></figure><p>When you’re just getting started with Python, essentially everything you’re learning is foundational. There is not a lot of time or motivation to stop and ask <em>when will I use this?</em> when you’re learning about basic things like data types, conditionals, loops, and functions…because the answer is <em>as a Python programmer, you’ll use all of these things all the time! </em>Whether you are a data scientist building machine learning models, or a software developer building the backend of a website, it’s unlikely you’ll be able to make any progress at all if you don’t know the basics.</p><p>But once you understand the essential foundations of Python, it can be challenging to identify which new tools and concepts will be helpful — and possibly even necessary—in your advancement, compared to those that might just be a waste of your time.</p><p>In this story, I’ll walk you through:</p><ol><li>Review of <em>SOLID software development principles</em></li><li>Examples of “tacked-on” scenarios that are challenging to implement while following the SOLID principles</li><li>How a <em>decorator approach</em> overcomes this challenge</li><li>Using <em>Python decorators</em> written by others</li><li>Writing your own decorators in Python</li></ol><p><em>The code in this blog post can also be found in this Google Colaboratory notebook:</em></p><p><a href="https://drive.google.com/file/d/1Hd6qP1Kiv7SL2k1EjKVxPbu7QEdvVP3A/view?usp=sharing">2020_08_20_Python_Decorators_Demo.ipynb</a></p><h3>SOLID principles</h3><p>If you’re planning on contributing to “real” software projects, not just one-off scripts or personal projects, it’s a good idea to be familiar with the SOLID principles.</p><figure><img alt="Green circular logo with a check mark and the words “clean code”" src="https://cdn-images-1.medium.com/max/200/1*Pa3lmDGKd6VyHNBFoW3XKg.png" /><figcaption><a href="https://sites.google.com/site/unclebobconsultingllc/getting-a-solid-start">Uncle Bob Consulting, LLC</a></figcaption></figure><p>The SOLID principles were originally created by Robert C. Martin, AKA “Uncle Bob” of Uncle Bob Consulting. To quote from his <a href="https://sites.google.com/site/unclebobconsultingllc/getting-a-solid-start">website</a>:</p><blockquote>The principles are mental cubby-holes. They give a name to a concept so that you can talk and reason about that concept. They provide a place to hang the <em>feelings</em> we have about good and bad code. They attempt to categorize those feelings into concrete advice.</blockquote><p>The 5 SOLID principles are:</p><ul><li><strong>S</strong>ingle-Responsibility Principle</li><li><strong>O</strong>pen-Closed Principle</li><li><strong>L</strong>iskov Substitution Principle</li><li><strong>I</strong>nterface Segregation Principle</li><li><strong>D</strong>ependency Inversion Principle</li></ul><p>The two we’re going to focus on in this blog post are the single-responsibility principle and the open-closed principle. These are the two principles that are most challenging to follow in the “tacked-on” scenarios described later.</p><h4>Single-responsibility principle</h4><p>Every component of a piece of software should only be responsible for a single set of functionality. In the context of the examples in this blog, a “component” will mean a Python function.</p><h4>Open-closed principle</h4><p>Already-written software should be “closed” for modification, but “open” for extension. Meaning, we should avoid re-writing existing code whenever possible, but ideally we should be able to add extended functionality to that code.</p><h3>“Tacked-on” software scenarios</h3><p>While “SOLID” is a widely-known term in software development, “tacked-on” is just a name that I came up with for a category of software scenarios where you have a perfectly-functional software library, but now a stakeholder is asking you to “tack on” some more functionality that is related to the original functionality while simultaneously being conceptually distinct.</p><p>Some examples of “tacked-on” software scenarios are:</p><ul><li>Caching</li><li>Logging</li><li>Access control</li><li>Input validation</li><li>Tweaks to input or output format</li></ul><p>In all of these cases, you want to have the original functionality, but also every time you invoke that original functionality, you want something else to happen on top of it.</p><h4>Caching example</h4><p>Let’s focus on <strong>caching</strong> as a specific example of a “tacked-on” scenario. In general, <em>caching</em> means that you save the result of a costly operation in a <em>cache</em> (usually implemented as some form of a dictionary/hashmap) so that later you can just look up that result instead of re-doing that costly operation.</p><p>If you’ve done any technical interview algorithms practice before, you’ve likely encountered the task of optimizing the recursive <a href="https://en.wikipedia.org/wiki/Fibonacci_number">Fibonacci number</a> algorithm. We’ll use it as a stand-in for a more realistic computing task. An example implementation of this algorithm (found in the <a href="https://docs.python.org/3/library/functools.html#functools.lru_cache">Python docs</a>) is:</p><pre>def fib(n):<br>    if n &lt; 2:<br>        return n<br>    return fib(n-1) + fib(n-2)</pre><p>Let’s assume that your coworker already wrote that code as part of a larger software project, as well as this code (standing in for a more realistic unit test suite) to test the functionality:</p><pre>assert fib(25) == 75025</pre><p>The problem with this implementation is that you end up re-computing the same values repeatedly. For example, if you’re computing fib(10) that means fib(9) + fib(8). Then fib(9) =fib(8) + fib(7) and fib(8) = fib(7) + fib(6). Already you can see that fib(8) and fib(7) are being computed twice, and this repetition cascades through the rest of the recursion. How can you make this code faster?</p><p>The “known” way to optimize your Fibonacci solution is to add in caching, so that once fib(8) is calculated, you never have to calculate it again, you just have to retrieve it from the cache. But how to do that while following the SOLID principles?</p><h4>Sub-optimal caching solutions</h4><p>The most obvious/common way to address this is to make a cache outside of the function, then adapt the function to check the cache before doing the computation and add all computed values to the cache. Something like this:</p><pre>cache = {}</pre><pre>def fib(n):<br>    if n &lt; 2:<br>        return n</pre><pre>    if n in cache:<br>        return cache[n]</pre><pre>    result = fib(n-1) + fib(n-2)<br>    cache[n] = result<br>    return result</pre><p>The test suite will still pass ✅</p><pre>assert fib(25) == 75025</pre><p>But what about the SOLID principles?</p><p><strong>Single-responsibility</strong> ❌</p><p>Now instead of a function that computes Fibonacci numbers, we have a function that computes Fibonacci numbers and manages adding and retrieving things from a cache.</p><p><strong>Open-closed</strong> ❌</p><p>We did not leave the existing function “closed”. If you looked at the <a href="https://git-scm.com/docs/git-blame">Git blame</a> for the function, 3 of the original lines of the function would be the same, 1 would be removed, and 6 would be added. You’ve re-written the majority of this whole function, and if you later decide to use a different kind of cache, you would need to “open up” the function again.</p><p>Another popular implementation of this would be to avoid creating a global cache by converting the function into a method of a <strong>class</strong>, and making the cache a member variable of that class. This has the same single-responsibility ❌ and open-closed ❌ issues as the previous solution, and additionally would require re-writing of the unit test ❌ (and all code that currently expects this to be a standalone function rather than a class method).</p><h3>The decorator approach</h3><p>The decorator approach is a great way to solve this kind of problem. It goes back to the classic “Gang of Four” <em>Design Patterns</em> book published in 1994.</p><figure><img alt="“Design Patterns” book cover image" src="https://cdn-images-1.medium.com/max/370/1*nzIYoXo-a45FpBCuPeZ7kg.jpeg" /><figcaption><a href="https://www.google.com/books/edition/Design_Patterns/6oHuKQe3TjQC?hl=en&amp;gbpv=0">Design Patterns: Elements of Reusable Object-Oriented Software</a></figcaption></figure><p>While that book describes <em>“composition over inheritance” </em>in the context of designing OOP classes, we can translate it in this context to be about the composition of functions.</p><p>In general, the decorator approach means you want to “wrap” your code in this tacked-on functionality, meaning you create something that is <em>composed of </em>the original functionality, now <em>decorated </em>with the tacked-on functionality.</p><p>Because we don’t need to modify the existing code, by using a decorator approach we are able to “tack on” the new functionality while still following the single-responsibility and open-closed SOLID principles.</p><h4>Clarification</h4><p>There are many resources describing the <strong>Decorator Pattern</strong>, like <a href="https://python-patterns.guide/gang-of-four/composition-over-inheritance/#solution-3-the-decorator-pattern">this article</a>. This terminology is specific to a narrow type of object-oriented “decorator classes”, which can theoretically be implemented using Python decorators. Decorator classes are addressing the same kinds of scenarios we have been discussing, and the general idea is similar. But “decorators in Python” refers to a more high-level construct that can apply to both functions and classes. This potential confusion around the naming of Python decorators is addressed but not resolved in the <a href="https://www.python.org/dev/peps/pep-0318/#on-the-name-decorator">Python Enhancement Proposal from 2003</a>.</p><h3>Using python decorators</h3><p>Before we dive into writing our own custom Python decorator, let’s use one that has already been written for us. Conceptually this is similar to learning how to use existing functions like print() or len() before attempting to write functions for ourselves.</p><p>We’ll return to the Fibonacci numbers example from before:</p><pre>def fib(n):<br>    if n &lt; 2:<br>        return n<br>    return fib(n-1) + fib(n-2)</pre><p>Now let’s add an LRU cache from the <a href="https://docs.python.org/3/library/functools.html#functools.lru_cache">built-in </a><a href="https://docs.python.org/3/library/functools.html#functools.lru_cache">functools module</a> (built-in meaning we don’t need to install any libraries beyond base Python, we just need to import it):</p><pre>import functools</pre><pre>@functools.lru_cache(maxsize=None)<br>def fib(n):<br>    if n &lt; 2:<br>        return n<br>    return fib(n-1) + fib(n-2)</pre><p>…and that’s it! 🤯</p><p>The <strong>test suite</strong> still passes ✅</p><p><strong>Single-responsibility principle</strong> ✅</p><p>The fib function, as written, is still just a Fibonacci number function. It is not a function combining Fibonacci numbers and caching.</p><p><strong>Open-closed principle</strong> ✅</p><p>We left the original function “closed”. If someone went to look at the Git blame, it would still be fully attributed to the original author. But at the same time, we “opened” it up for extension, since now it’s using caching.</p><p><strong>Reality check:</strong> In case you are wondering if adding these two lines of code really “did anything”, I ran these two snippets using the %%timeit magic command and fib(25). The original code took 30.9 ms (30.9 milliseconds), and the code with caching took 6 µs (6 microseconds). There are 1000 microseconds in a millisecond, meaning the performance of the code with caching was over 1000x faster than the original—pretty impressive for an addition of 2 lines of code!</p><h4>So…what just happened?</h4><p>Mechanically, we just imported a module (functools), then added an @ symbol plus a function from within that module.</p><p>Conceptually, we “wrapped” the existing fib function in a cache decorator:</p><ol><li>The lru_cache decorator took in our function as an argument</li><li>It added caching functionality</li><li>It returned a new function composed of fib plus the caching logic</li><li>Finally, it re-assigned the name fib to the new function, so we could continue using the original interface</li></ol><h4>Using already-written Decorators</h4><p>Unfortunately, the Python developers do not maintain a global list of decorators available in base Python, but this GitHub repo contains a pretty good list of those decorators as well as decorators “in the wild” (i.e. located in libraries outside of Python itself):</p><p><a href="https://github.com/lord63/awesome-python-decorator">lord63/awesome-python-decorator</a></p><p>(This repo has not been updated since 2017, let me know in the comments if you have found a better or complementary resource!)</p><p><strong>Reality check:</strong> If you’re curious how often and in what context you’ll encounter decorators “in the real world”…in my work in software engineering and data science, I most frequently encounter decorators in a backend/full-stack web development context, especially <a href="https://flask.palletsprojects.com/en/1.1.x/">Flask</a> and <a href="https://plotly.com/dash/">Dash</a>. Those frameworks use decorators to allow you to write functions that will be wrapped in the full web server functionality. It is essentially impossible to create an application using either of those frameworks without using decorators. Using a Flask decorator looks something like this (the “minimal” application from the <a href="https://flask.palletsprojects.com/en/1.1.x/quickstart/#a-minimal-application">Quickstart guide</a>):</p><pre><strong>from</strong> flask <strong>import</strong> Flask<br>app = Flask<strong>(</strong>__name__<strong>)</strong><br><br>@app.route<strong>(</strong>&#39;/&#39;<strong>)</strong><br><strong>def</strong> hello_world<strong>():</strong><br>    <strong>return</strong> &#39;Hello, World!&#39;</pre><p>Using decorators that someone else has written is great! You can get some excellent functionality right “out of the box”: just import (if needed), add the @ symbol syntax, and your function is now decorated with added functionality. There are excellent caching, logging, and access control decorators already out there. It can be acceptable to leave decorators as an unopened “black box” if someone has already written a decorator that accomplishes what you need to accomplish!</p><p>But sometimes, you might want to use this technique in a context where nobody has written a decorator that does the thing you want to do. So let’s go ahead and explore how you might write a custom decorator.</p><h3>Python functions as “first-class” objects</h3><p>Before we can jump right in to writing custom decorators, we need to review how Python functions are <a href="https://en.wikipedia.org/wiki/First-class_function">“first-class” objects</a>. This is a concept that will be very familiar if you’ve used <a href="https://developer.mozilla.org/en-US/docs/Glossary/First-class_Function">JavaScript for front-end web development</a> (think <em>callbacks</em> and <em>event listeners</em>) and very foreign if you’ve only used a language like Ruby or Java before, that doesn’t treat functions this way.</p><p>Functions in Python:</p><ul><li>Can be stored in data structures</li><li><strong>Can be assigned to variables</strong></li><li><strong>Can be passed to other functions</strong></li><li><strong>Can be nested</strong></li><li>Can capture local state</li></ul><p>(Check out <a href="https://dbader.org/blog/python-first-class-functions">this tutorial</a> for more explanation and examples of these properties.)</p><p>We’re going to focus on those three bolded properties, since they are key for understanding how decorators work.</p><h4>Assigning functions to variables, and passing functions to other functions</h4><p>Here are some short/trivial examples to show what these properties mean. First, let’s set up some functions:</p><pre>def squared(num):<br>    return num ** 2</pre><pre>def cubed(num):<br>    return num ** 3</pre><pre>print(squared(5)) # prints 25</pre><pre>print(cubed(5))   # prints 125</pre><p>Pretty straightforward. We have two functions, each of which takes a single argument num and returns a polynomial transformation of num. squared squares it, and cubed cubes it.</p><p>Now to demonstrate the “first-class” functionality:</p><pre>def func_plus_two(polynomial_func, num):<br>    return polynomial_func(num) + 2</pre><pre>first_func = squared<br>second_func = cubed</pre><pre>print(func_plus_two(first_func, 5))  # prints 27</pre><pre>print(func_plus_two(second_func, 5)) # prints 127</pre><p>So, we are <strong>assigning functions to variables</strong>, in this case assigning squared to first_func and cubed to second_func. (There is no particularly useful reason for doing this, it’s just for demonstration purposes.)</p><p>Then, maybe more interestingly, we are <strong>passing those functions to another function</strong>, first passing first_func then second_func into func_plus_two. That function calls the function that was passed into it, then returns the result of the original function + 2. So, instead of 25 and 125, we get 27 and 127, when num = 5. This structure (passing a function into another function) is also called a <em>higher-order function, </em>i.e. we could say “func_plus_two is a higher-order function”.</p><h4>Nested functions</h4><p>It’s even more of a “stretch” to connect this example to something realistically useful, but bear with me, it becomes important when we actually get to writing decorators!</p><p>Consider this example of nested functions:</p><pre>def nth_degree(num, n):<br>    def squared_inner(num):<br>        return num ** 2<br>    def cubed_inner(num):<br>        return num ** 3</pre><pre>    if n == 2:<br>        return squared_inner(num)<br>    elif n == 3:<br>        return cubed_inner(num)</pre><p>Now instead of having squared and cubed as functions in the global scope, we have functions squared_inner and cubed_inner that only exist within the scope of the nth_degree function. In other words, if we tried to run squared_inner(5) in the global scope, it wouldn’t return 25, it would throw a NameError: name &#39;squared_inner&#39; is not defined. To <em>invoke</em> (i.e. call) this function, we could use code like:</p><pre>nth_degree(4, 2) # 4 squared, prints 16</pre><pre>nth_degree(2, 3) # 2 cubed, prints 8</pre><p>In the first example, we passed in an n of 2, so nth_degree called the squared_inner function on 4 and returned 16. In the second example, we passed in an n of 3, so nth_degree called the cubed_inner function on 2 and returned 8. In reality we could easily re-write this code to avoid inner functions altogether, but I hope you understand the main takeaway that a function can have another function nested inside itself, then return something based on that nested function.</p><h3>Writing our own comma-adding decorator</h3><p>Ok, now that we’ve completed the review of first-class Python functions, let’s write a ⭐ decorator ⭐️ with custom functionality!</p><p>Let’s say your boss comes to you with this task:</p><blockquote><em>“Our users are having a hard time reading these large numbers. At a glance, is 1000000 1 million or 10 million? Let’s take out the guesswork and put in some commas, like they’re used to in Comma Style format in Excel. So they see something like 1,000,000 instead of 1000000.”</em></blockquote><p>(This is inspired by a real task I had to do as a software developer, back when I worked in software consulting at <a href="https://www.crowe.com/">Crowe</a>. Accounting folks love their commas!)</p><p>You are maintaining a library with 5 functions, all of which currently returns a string representation of an integer to be displayed in the user interface. You need to update the output format of those functions while continuing to follow the single-responsibility and open-close principles.</p><p>Since you work in a <a href="https://en.wikipedia.org/wiki/Test-driven_development">test-driven</a> company, the software developer in test has already started the project by adapting the <strong>unit tests</strong> to match the new requirements.</p><p>Old unit tests:</p><pre>assert one_million() == &quot;1000000&quot;<br>assert one_billion() == &quot;1000000000&quot;<br>assert times_100(7) == &quot;700&quot;<br>assert minus_10000(150000) == &quot;140000&quot;<br>assert multiply(300, 800) == &quot;240000&quot;</pre><p>New unit tests:</p><pre>assert one_million() == &quot;1,000,000&quot;<br>assert one_billion() == &quot;1,000,000,000&quot;<br>assert times_100(7) == &quot;700&quot;<br>assert minus_10000(150000) == &quot;140,000&quot;<br>assert multiply(300, 800) == &quot;240,000&quot;</pre><p>The decorator you decide to write has this overall structure:</p><pre>def add_commas(func):<br>    def add_commas_wrapper():<br>        # call original function<br>        # add in the commas<br>        # return the result<br>    return add_commas_wrapper</pre><p>In other words, it is a wrapper function that takes in the original function as an argument, then returns an inner function that will call the original function and also add in the commas.</p><h4>Simplest version of decorator (no arguments)</h4><p>Rather than trying to write a decorator with complete functionality all at once, let’s start with something very basic, which only handles the library functions without any arguments.</p><pre>def add_commas(func):<br>    def add_commas_wrapper():<br>        original_string = func()<br>        # needs to be int for string formatting<br>        original_int = int(original_string)<br>        # we are ignoring locale, using default thousands sep<br>        return f&#39;{original_int:,}&#39;<br>    return add_commas_wrapper</pre><p>Now that we have that function, we can modify the existing one_million and one_billion functions:</p><pre>def one_million():<br>    return &quot;1000000&quot;</pre><pre>def one_billion():<br>    return &quot;1000000000&quot;</pre><pre>one_million = add_commas(one_million)<br>one_billion = add_commas(one_billion)</pre><p>Now we are passing two of the unit tests:</p><pre>assert one_million() == &quot;1,000,000&quot; ✅<br>assert one_billion() == &quot;1,000,000,000&quot; ✅<br>assert times_100(7) == &quot;700&quot; ✅ # &quot;accidentally&quot; passing, &lt;1000<br>assert minus_10000(150000) == &quot;140,000&quot; ❌ # returns &quot;140000&quot;<br>assert multiply(300, 800) == &quot;240,000&quot; ❌ # returns &quot;240000&quot;</pre><h4>Adding the syntactic sugar</h4><p>You might be wondering…what about that @ symbol we were using earlier? How is this a “decorator” like functools.lru_cache if there’s no @ symbol? Well, the @ symbol here is a form of syntactic sugar.</p><p><a href="https://en.wikipedia.org/wiki/Syntactic_sugar"><em>Syntactic sugar</em></a><em> </em>is a concept that extends beyond Python specifically. It means some kind of syntax that shortens a common bit of code, usually to make it easier to read and/or harder to mess up.</p><p>Let’s revise the previous code snippet to use decorator syntactic sugar:</p><pre>@add_commas<br>def one_million():<br>    return &quot;1000000&quot;</pre><pre>@add_commas<br>def one_billion():<br>    return &quot;1000000000&quot;</pre><p>It’s the same number of lines of code, and it’s actually doing the exact same thing as the previous snippet. But it’s a little cleaner (with fewer parentheses and repetitions of the name of the function), and someone looking at the function definitions (lines starting with def) will be immediately aware that some decoration is happening beyond what they can see in the function, in case later they’re trying to debug the function. In the previous syntax, the decoration is separated from the definition in a way that could end up being confusing, especially with more or longer functions than these trivial examples.</p><h4>Accommodating arguments</h4><p>What if we just try to add this decorator to the times_100 function?</p><pre>@add_commas<br>def times_100(num):<br>    return str(num * 100)</pre><pre>print(times_100(7))</pre><figure><img alt="Screenshot of an error message. It says that add_commas_wrapper() takes 0 positional arguments but 1 was given" src="https://cdn-images-1.medium.com/max/615/1*gMZ_SyjDsXMA1Fp2nqCS8A.png" /></figure><p>😱 what happened? Well, if we look back at the definition of add_commas_wrapper, it’s:</p><pre>def add_commas(func):<br>   <strong> def add_commas_wrapper():</strong><br>        original_string = func()<br>        # needs to be int for string formatting<br>        original_int = int(original_string)<br>        # we are ignoring locale, using default thousands sep<br>        return f&#39;{original_int:,}&#39;<br>    return add_commas_wrapper</pre><p>Just like the error message said, that function takes 0 arguments. How can we adapt it to work with the num argument of times_100 while not breaking the functionality of one_million and one_billion, which don’t take any arguments?</p><p>We could add some kind of custom logic, like the if statements in the nth_degree nested function, but that means we might need to repeatedly modify add_commas to handle different numbers of parameters. We shouldn’t need to do that, since really all this function needs to do is start with a string and put commas in that string…it shouldn’t matter how many arguments were passed in to the original function.</p><p>Fortunately there is a technique to accept an arbitrary number of arguments (i.e. <em>variable-length arguments</em>) in Python! The traditional syntax for this is *args, **kwargs, meaning 0 or more positional arguments (args) then 0 or more keyword arguments (kwargs). So that would work for one_million (0 arguments of any kind), times_100 (1 positional argument), multiply (2 positional arguments), and any other number of positional or keyword arguments.</p><p>Let’s add variable-length arguments:</p><pre>def add_commas(func):<br>   <strong> def add_commas_wrapper(*args, **kwargs):</strong><br><strong>        original_string = func(*args, **kwargs)</strong><br>        # needs to be int for string formatting<br>        original_int = int(original_string)<br>        # we are ignoring locale, using default thousands sep<br>        return f&#39;{original_int:,}&#39;<br>    return add_commas_wrapper</pre><p>And add the decorator syntax to all the functions:</p><pre><strong>@add_commas<br></strong>def one_million():<br>    return &quot;1000000&quot;</pre><pre><strong>@add_commas<br></strong>def one_billion():<br>    return &quot;1000000000&quot;</pre><pre><strong>@add_commas<br></strong>def times_100(num):<br>    return str(num * 100)</pre><pre><strong>@add_commas<br></strong>def minus_10000(num):<br>    return str(num - 10000)</pre><pre><strong>@add_commas<br></strong>def multiply(num1, num2):<br>    return str(num1 * num2)</pre><p>Now we are passing all tests!</p><pre>assert one_million() == &quot;1,000,000&quot; ✅<br>assert one_billion() == &quot;1,000,000,000&quot; ✅<br>assert times_100(7) == &quot;700&quot; ✅<br>assert minus_10000(150000) == &quot;140,000&quot; ✅<br>assert multiply(300, 800) == &quot;240,000&quot; ✅</pre><p>🎉 🎉 🎉 🎉 🎉</p><h4>One more thing</h4><p>If you want to use decorators in production, there’s one more thing you’ll want to include, although it’s irrelevant for passing these particular tests. You might have noticed with the error message earlier that it reported to be from add_commas_wrapper, not times_100. In that example, it wasn’t particularly relevant, but we could imagine another scenario when someone tries to use a float rather than an int, or does something else that breaks the logic inside of add_commas, when we would want to be able to <em>introspect </em>and see the original function causing the error, not the wrapper.</p><p>You can find more details about what’s happening and how to address it manually in <a href="https://realpython.com/primer-on-python-decorators/">this tutorial</a>, but luckily there is yet again a function inside of the built-in functools module that can handle this for us! It is called wraps, and the documentation can be found <a href="https://docs.python.org/3/library/functools.html#functools.wraps">here</a>.</p><p>The final version of our code with wraps added is:</p><pre><strong>import functools</strong></pre><pre>def add_commas(func):<br>    <strong>@functools.wraps</strong><br>    def add_commas_wrapper(*args, **kwargs):<br>        original_string = func(*args, **kwargs)<br>        # needs to be int for string formatting<br>        original_int = int(original_string)<br>        # we are ignoring locale, using default thousands sep<br>        return f&#39;{original_int:,}&#39;<br>    return add_commas_wrapper</pre><pre>@add_commas<strong><br></strong>def one_million():<br>    return &quot;1000000&quot;</pre><pre>@add_commas<strong><br></strong>def one_billion():<br>    return &quot;1000000000&quot;</pre><pre>@add_commas<strong><br></strong>def times_100(num):<br>    return str(num * 100)</pre><pre>@add_commas<strong><br></strong>def minus_10000(num):<br>    return str(num - 10000)</pre><pre>@add_commas<strong><br></strong>def multiply(num1, num2):<br>    return str(num1 * num2)</pre><h3>Recap</h3><h4>Reviewing our updated code</h4><p>The test suite passes, meaning we have successfully implemented the specified “tacked-on” functionality ✅</p><p><strong>Single-responsibility principle</strong> ✅</p><p>Our computational functions are still just doing computation (standing in for the more-sophisticated computations or queries you would do in a real application). We also have a wrapper function that adjusts the output format of the functions, and it’s not concerned with the number of arguments in the original function, it just always converts a string representation of an integer into a format with commas.</p><p><strong>Open-closed principle</strong> ✅</p><p>We did not make any changes to the contents or the call signature of the original functions. But, by adding 15 lines of additional code, we were able to “open” the code for extension to meet the new stakeholder requirements.</p><h4>Reviewing what we learned</h4><p>Some kinds of scenarios, which I call <strong>“tacked-on” scenarios</strong>, can be challenging to implement without violating the <strong>SOLID principles</strong>, particularly the single-responsibility principle and the open-closed principle. The decorator approach, and <strong>Python decorators</strong> in particular, are helpful for tackling this challenge.</p><p>In some cases, someone has <strong>already written</strong> a decorator that handles the “tacked-on” scenario you need, especially for common scenarios like caching, logging, and access control. In other cases, you need to <strong>write your own decorator</strong> for custom functionality.</p><p>In order to write your own decorator, you need to recall that <strong>Python functions are “first-class” objects</strong>, particularly that they can be assigned to variables, can be passed to other functions, and can be nested. With that knowledge in mind, and the help of <strong>syntactic sugar</strong> and functools.wraps to ensure proper introspection, you can write your own decorators to handle these “tacked-on” tasks while following the SOLID principles.</p><h4>References</h4><p>In addition to the references linked above, credit to this excellent tutorial from Dan Bader for helping me formulate the framing:</p><p><a href="https://dbader.org/blog/python-decorators">Python Decorators: A Step-By-Step Introduction - dbader.org</a></p><p>And I’ll link again to this in-depth tutorial from Real Python for anyone who wants to dig into all of the granular details:</p><p><a href="https://realpython.com/primer-on-python-decorators/">Primer on Python Decorators - Real Python</a></p><p>Check out this set of exercises if you want to get some more practice before using this technique in production:</p><p><a href="https://www.learnpython.org/en/Decorators">Decorators - Learn Python - Free Interactive Python Tutorial</a></p><p>Thanks for reading, and let me know in the comments if you know of any other notable uses of decorators or additional tutorials!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1da4ed9f3a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/decorators-in-python-why-and-how-to-use-them-and-write-your-own-c1da4ed9f3a9">Decorators in Python: Why and How to Use Them and Write Your Own</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial: Exploratory Data Analysis (EDA) with Categorical Variables]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/analytics-vidhya/tutorial-exploratory-data-analysis-eda-with-categorical-variables-6a569a3aea55?source=rss-29b741b41e48------2"><img src="https://cdn-images-1.medium.com/max/2048/1*b84yZmN470VbffbnbNQi8w.jpeg" width="2048"></a></p><p class="medium-feed-snippet">Expanding beyond scatter plots for better exploratory data analysis</p><p class="medium-feed-link"><a href="https://medium.com/analytics-vidhya/tutorial-exploratory-data-analysis-eda-with-categorical-variables-6a569a3aea55?source=rss-29b741b41e48------2">Continue reading on Analytics Vidhya »</a></p></div>]]></description>
            <link>https://medium.com/analytics-vidhya/tutorial-exploratory-data-analysis-eda-with-categorical-variables-6a569a3aea55?source=rss-29b741b41e48------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a569a3aea55</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Erin Hoffman]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 00:40:59 GMT</pubDate>
            <atom:updated>2020-07-15T14:26:48.908Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Getting Started with scikit-learn Pipelines for Machine Learning]]></title>
            <link>https://medium.com/analytics-vidhya/getting-started-with-scikit-learn-pipelines-for-machine-learning-fa88efdca3b9?source=rss-29b741b41e48------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa88efdca3b9</guid>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Erin Hoffman]]></dc:creator>
            <pubDate>Thu, 26 Mar 2020 03:15:54 GMT</pubDate>
            <atom:updated>2020-08-21T18:18:56.783Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building a pipeline from the ground up</h4><figure><img alt="A woman looks at her laptop while leaning on the glass in front of a row of computer servers" src="https://cdn-images-1.medium.com/max/1024/1*sYywBmpCmmna_JlFf-NRrg.jpeg" /><figcaption>Image credit: <a href="https://www.wocintechchat.com/blog/wocintechphotos">https://www.wocintechchat.com/blog/wocintechphotos</a></figcaption></figure><p>(All code in this post is also included in <a href="https://github.com/hoffm386/simple-sklearn-pipeline-example">this GitHub repository</a>.)</p><h3>Why Use Pipelines?</h3><p>The typical overall machine learning workflow with <a href="https://scikit-learn.org/stable/index.html">scikit-learn</a> looks something like this:</p><ol><li>Load all data into X and y</li><li>Use X and y to perform a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">train-test split</a>, creating X_train, X_test, y_train, and y_test</li><li>Fit preprocessors such as <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">StandardScaler</a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html">SimpleImputer</a> on X_train</li><li>Transform X_train using the fitted preprocessors, and perform any other preprocessing steps (such as dropping columns)</li><li>Create various models, tune hyperparameters, and pick a final model that is fit on the preprocessed X_train as well as y_train</li><li>Transform X_test using the fitted preprocessors, and perform any other preprocessing steps (such as dropping columns)</li><li>Evaluate the final model on the preprocessed X_test as well as y_test</li></ol><p>Here is an example code snippet that follows these steps, using an <a href="https://raw.githubusercontent.com/hoffm386/simple-sklearn-pipeline-example/master/antelope.csv">antelope dataset</a> (“antelope.csv”) from a <a href="https://college.cengage.com/mathematics/brase/understandable_statistics/7e/students/datasets/mlr/frames/frame.html">statistics textbook</a>. The goal is to predict the number of spring fawns based on the adult antelope population, annual precipitation, and winter severity. This is a very tiny dataset and should only be used for example purposes! This example skips any hyperparameter tuning, and simply fits a vanilla linear regression model on the preprocessed training data before evaluating it on the preprocessed testing data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/752b2725203f2b8f4a9e9f1080d0121e/href">https://medium.com/media/752b2725203f2b8f4a9e9f1080d0121e/href</a></iframe><p><strong>The train-test split is one of the most important components of a machine learning workflow.</strong> It helps a data scientist understand model performance, particularly in terms of overfitting. A proper train-test split means that we have to perform the preprocessing steps on the training data and testing data separately, so there is no “leakage” of information from the testing set into the training set.</p><p><strong>But as a software developer looking at this code, an issue stands out immediately: steps 4 and 6 are virtually identical. Whatever happened to DRY (don’t repeat yourself)?! The solution: pipelines. </strong>Pipelines are designed to avoid this problem completely. You declare the preprocessing steps once, then you can apply them as needed to X_train as well as X_test.</p><h3>First, Write the Code Without a Pipeline</h3><p>Yes, you read that correctly. Until you are a real expert using pipelines, it’s a good idea to write out the repetitive/redundant version of your code first, then refactor it to use pipelines instead. If you are hoping to write functioning pipeline code, go back and create something that resembles the code snippet above first!</p><h3>Second, Iteratively Add Preprocessing Steps</h3><p>The error messages produced by pipelines can be extremely hard to decipher! So if you add more than one step at a time and something breaks, you’ll have a very hard time figuring out what broke. A better plan is to add steps one at a time, and double-check that it still works as you go.</p><p>My general strategy is to start with whatever step has dependencies, e.g. a SimpleImputer (since other preprocessing steps might fail if there is missing data). In this example case, let’s go ahead and start with the OneHotEncoder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e98d8c89d94d1110e839c00843d88d3/href">https://medium.com/media/4e98d8c89d94d1110e839c00843d88d3/href</a></iframe><p>Let’s zoom in on some specifics here. First, <strong>fitting</strong> (#3 in the ML process). The old version was:</p><pre>ohe = OneHotEncoder(sparse=False, handle_unknown=&quot;ignore&quot;)</pre><pre>ohe.fit(X_train[[&quot;winter_severity_index&quot;]])</pre><p>The new version is:</p><pre>pipe = Pipeline(steps=[<br>    (&quot;encode_winter&quot;, ColumnTransformer(transformers=[           <br>        (&quot;ohe&quot;, OneHotEncoder(sparse=False, handle_unknown=&quot;ignore&quot;), [&quot;winter_severity_index&quot;])<br>    ], remainder=&quot;passthrough&quot;<br>))])<br>pipe.fit(X_train, y_train)</pre><p>We still have the same encoder with the same parameters, but now it’s nested inside of a ColumnTransformer, which is nested inside of a Pipeline. Instead of subsetting X_train(with [[]]) to specify which column(s) to one-hot encode, we pass that into the ColumnTransformer. (See <a href="https://towardsdatascience.com/using-columntransformer-to-combine-data-processing-steps-af383f7d5260">this post</a> from my former student Allison Honold for more details on ColumnTransformers.) Then instead of using the encoder directly, we add it as the first “step” of the Pipeline.</p><p>Second, <strong>transforming</strong> (#4 and #6 in the ML process). The old version was:</p><pre>train_winter_array = ohe.transform(X_train[[&quot;winter_severity_index&quot;]])</pre><pre>train_winter_df = pd.DataFrame(train_winter_array, index=X_train.index)</pre><pre>X_train = pd.concat([train_winter_df, X_train], axis=1)</pre><pre>X_train.drop(&quot;winter_severity_index&quot;, axis=1, inplace=True)</pre><pre>test_winter_array = ohe.transform(X_test[[&quot;winter_severity_index&quot;]])</pre><pre>test_winter_df = pd.DataFrame(test_winter_array, index=X_test.index)</pre><pre>X_test = pd.concat([test_winter_df, X_test], axis=1)</pre><pre>X_test.drop(&quot;winter_severity_index&quot;, axis=1, inplace=True)</pre><p>The new version is:</p><pre>columns_with_ohe = [0, 1, 2, 3, &quot;adult_antelope_population&quot;, &quot;annual_precipitation&quot;]</pre><pre>X_train_array = pipe.transform(X_train)</pre><pre>X_train = pd.DataFrame(X_train_array, columns=columns_with_ohe)</pre><pre>X_test_array = pipe.transform(X_test)</pre><pre>X_test = pd.DataFrame(X_test_array, columns=columns_with_ohe)</pre><p>As you can see, we are already getting some benefit from using the pipeline. We no longer have to manually concat the encoded data with the original data, or manually drop the original column.</p><p>However at this point we have a bit of a “hack”, where we are hard-coding the column names so that the later code is able to work. We need the name of the “annual_precipitation” column in order to create the “low_precipitation” column, but the one-hot encoding has removed all of the column names. Let’s continue adding the preprocessing steps to the pipeline, and be sure to do the one-hot encoding <em>after</em> the custom transformation, so we don’t need this “hack” any more.</p><h3>Third, Create Custom Transformers As Needed</h3><p>For the purpose of feature engineering, we often want to use Pandas to do something that is not a common enough task to be included as a scikit-learn preprocessor like OneHotEncoder. To make that work in a pipeline, you need to create a custom transformer class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fc4e96578a00bbff33aaa4878346519/href">https://medium.com/media/1fc4e96578a00bbff33aaa4878346519/href</a></iframe><p>Looking at the specifics again, the old version of <strong>fitting</strong> was…nothing. We weren’t using any information about the training data to perform the transformation. The old version of <strong>transforming</strong> was:</p><pre>X_train[&quot;low_precipitation&quot;] = [int(x &lt; 12) for x in X_train[&quot;annual_precipitation&quot;]]</pre><pre>X_test[&quot;low_precipitation&quot;] = [int(x &lt; 12) for x in X_test[&quot;annual_precipitation&quot;]]</pre><p>The new version of <strong>fitting and transforming </strong>is that we added a new class PrecipitationTransformer:</p><pre>class PrecipitationTransformer(BaseEstimator, TransformerMixin):<br>    def fit(self, X, y):<br>        return self<br>    def transform(self, X, y=None):<br>        X_new = X.copy()<br>        X_new[&quot;low_precipitation&quot;] = [int(x &lt; 12) for x in X_new[&quot;annual_precipitation&quot;]]<br>        return X_new</pre><p>and a single “step” in the pipeline:</p><pre>...<br>(&quot;transform_precip&quot;, PrecipitationTransformer()),<br>...</pre><p>It’s not shorter, but it does avoid repetition!</p><h3>Fourth, Add in Your Model</h3><p>Adding the model as a final step is where, I think, the pipeline really shines. You add it in just the same way that you add the preprocessing steps:</p><pre>...<br>(&quot;linreg_model&quot;, LinearRegression())<br>...</pre><p>Here is the final workflow. We have somewhat reduced the number of lines of code, but more importantly we are no longer repeating anything!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/901d7441db384503035bef0fc80d247c/href">https://medium.com/media/901d7441db384503035bef0fc80d247c/href</a></iframe><p>Check out the <a href="https://towardsdatascience.com/using-columntransformer-to-combine-data-processing-steps-af383f7d5260">blog post on ColumnTransformers</a> mentioned previously, <a href="https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html">this example</a> from scikit-learn, or <a href="https://towardsdatascience.com/custom-transformers-and-ml-data-pipelines-with-python-20ea2a7adb65">this Medium post</a> for more advanced examples.</p><p>Thanks for reading, and let me know in the comments if you have any questions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa88efdca3b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/getting-started-with-scikit-learn-pipelines-for-machine-learning-fa88efdca3b9">Getting Started with scikit-learn Pipelines for Machine Learning</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>