<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brian Mattis on Medium]]></title>
        <description><![CDATA[Stories by Brian Mattis on Medium]]></description>
        <link>https://medium.com/@brian-mattis?source=rss-47e773cabf10------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sG0WOV0HelDqwxYm1sSxFg.jpeg</url>
            <title>Stories by Brian Mattis on Medium</title>
            <link>https://medium.com/@brian-mattis?source=rss-47e773cabf10------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:53:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@brian-mattis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Productivity Through Micro-tasks]]></title>
            <link>https://medium.com/illumination/productivity-through-micro-tasks-16fe922a6ff7?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/16fe922a6ff7</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[perseverance]]></category>
            <category><![CDATA[organization]]></category>
            <category><![CDATA[planning]]></category>
            <category><![CDATA[motivation]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 20:22:38 GMT</pubDate>
            <atom:updated>2024-07-12T20:22:38.004Z</atom:updated>
            <content:encoded><![CDATA[<h4>The life hack to keep you motivated when you hit the wall</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I2MvoEZ-pC6TMa-V" /><figcaption>Photo by <a href="https://unsplash.com/@priscilladupreez?utm_source=medium&amp;utm_medium=referral">Priscilla Du Preez 🇨🇦</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’m the kind of person who always has more personal projects than time. There are just so many things I find interesting that I want to do, learn, and pursue but my spare time is never enough. At some point though, the mental to-do list becomes so long it is overwhelming, and progress on <em>everything</em> comes to a screeching halt. Keep in mind, these are projects I <em>want</em> to do, and yet I hit a limit.</p><p>The result is that I fall into a trap of mental “cotton candy”. These are things that are nice in the moment, but after a bit, just make you feel very blah. This can be playing a mindless solo video game, scrolling on the phone, or just turning on Netflix. Things where there’s not even the reward of social interaction — they are semi-enjoyable in the moment, but the feeling is fleeting because you know they don’t bring fulfillment or a sense of accomplishment.</p><p>I do believe that the best way to start something new is to — just start. I’ve heard this advice from many sources, and it’s a great one for breaking down mental barriers. But what about when you have 5–10 projects that are all stalled partway through? Forcing yourself to be on-task without strong motivation on any of them rarely brings joy.</p><h4>The Experiment</h4><p>This leads to my latest experiment. I took a look at my projects one by one, and broke them down into micro-tasks. These are little steps that are written down and signify quantifiable progress. They’re also designed to be checked off with just an hour or less of focus. They’re small enough to where you can look at the list and think “yeah, I can knock that off tonight”.</p><p>This works for two reasons. First, we get the little serotonin boost from checking that item off in our list — it sounds silly, but it can be powerful to keep you going. Next, we get to see our progress day after day as those items get checked off and we progress to the ultimate project goal. I’ll even put a date next to an item when I check it off, so I can look back a month later at how far I’ve come. I personally build my multi-project micro-task checklist in Google Sheets, and you can have nice little checkboxes that can auto-populate with a timestamp (<a href="https://www.youtube.com/watch?v=AsGyokG6ITY">here’s</a> how).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*cfzqhgz6zDDhgQH4JekJlQ.png" /><figcaption>Micro-task tracking sheet in Google Sheets — Image by Author</figcaption></figure><h4>Diversification</h4><p>Even with this, there is always the risk of burn-out. This is especially true for fun side-projects after a long day of work. Personally, most of my day job is behind a computer screen, so after a long day of that, one of the last things I want to do is spend an evening in front of a screen. The answer is variety.</p><p>At any given time I’ll make sure to have some hands-on physical projects (things like woodworking, painting, wiring an Arduino, house repairs, etc), or even just microtasks in a bigger project that aren’t performed in-front of a computer. While having too many concurrent projects can be challenging, if they are micro-tasked, at any given time you can find something on your multi-project list that sounds the most attractive/doable. The goal isn’t to completely focus on a single project, even when you’re burned out on that activity — the goal is just progress — on anything!</p><p>Check that box and you’ll feel proud of yourself for how you spent your time, and you likely learned something new, which is really the end goal in all of this, right?</p><h4>Perseverance and Recovery</h4><p>All that being said, we are still human. We all have bad days where we’re exhausted, uninspired beyond repair, and brain-dead. Some days you need “cotton candy”, and that’s okay. Don’t beat yourself up about it — Personally I just try my hardest to not give up two days in a row. If you’re still struggling on day 2, pick an easy micro-task, knock it out, and you’ll be glad you did. Progress, no matter how little, is progress. We can persevere, and our mental state will be better off for it.</p><p>The last key is simple — sleep. When you are tired, motivation and creativity are the first things to go, and no amount of caffeine can make up for it. If you find yourself hitting a wall for your own fun projects often, chances are you’re just tired. Pay it forward to your future self — go to bed and be ready to crush it tomorrow.</p><h4>Conclusion</h4><p>That’s all there is to it. I’ve been running the microtasks scheme for about 6 months and I’ve felt a significant boost in productivity, and a decrease in wasted time. Also, I’m much happier seeing a consistent progress in my projects and goals. Looking back on my micro-task list on the things I’ve accomplished in the last 6 months helps me remember how much progress I’m actually making and gives an additional motivational boost. If you’ve hit the same wall that I was fighting, give micro-tasks a try, and let me know how it goes in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16fe922a6ff7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/illumination/productivity-through-micro-tasks-16fe922a6ff7">Productivity Through Micro-tasks</a> was originally published in <a href="https://medium.com/illumination">ILLUMINATION</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Level Up Your Data Storytelling with Animated Bar Charts in Plotly]]></title>
            <link>https://medium.com/data-science/level-up-your-data-storytelling-with-animated-bar-charts-in-plotly-f9ace6d73f27?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9ace6d73f27</guid>
            <category><![CDATA[animation]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[animated-plot]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Sat, 02 Dec 2023 04:59:47 GMT</pubDate>
            <atom:updated>2023-12-04T05:36:34.711Z</atom:updated>
            <content:encoded><![CDATA[<h4>Transforming static plots into captivating narratives</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nmreYlbQAfhJT8tc" /><figcaption>Photo by <a href="https://unsplash.com/@xteemu?utm_source=medium&amp;utm_medium=referral">Teemu Paananen</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Plotly supports an excellent foundation for animated plots. I highly recommend their basic tutorial <a href="https://plotly.com/python/animations/">here</a>. However, plotly animations are primarily set up to add another dimension to the visualization — usually time. This is fantastic for adding more meaning to a plot.</p><p>Animation, however, <em>doesn’t have to be used to add complexity to a plot</em>. The ability to <strong>emphasize</strong> with animation is powerful. When we have the pivotal graph for our audience, we want to pull in their attention without explicitly shouting “This is the plot you should really pay attention to!” Our eyes are naturally drawn to things moving, and animated plots can have a way of building up anticipation. Your audience engages and mentally tries to anticipate next bar or line in real time. Imagine if key business decisions hinge on this plot — your audience waits with baited breath while the results unfold before their eyes!</p><p>To put it in perspective — imagine you’re 20 minutes into a colleague’s presentation — which graph grabs your attention? This simple (pandas-generated) graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*DSXK5SOeFt8MukQsRyAsmg.png" /><figcaption>Basic bar plot — Image by Author</figcaption></figure><p>Or this animated graph of the same data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*MwORp2yFoBPDMvTORDB2bw.gif" /><figcaption>Custom Plotly animated bar plot — Image by Author</figcaption></figure><h3>Building in Plotly</h3><p>To pull this off in Plotly, we actually have to reshuffle our data a bit. Again, Plotly animations are set up to step with some sequence (usually time). So, we’re going to stretch our data to effectively make a frame for each data point. Here we go:</p><p>First, let’s import the plotly package (install with pip if you haven’t already), and create a dummy data set to work with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c54ec8170ee27e8653f48cf1970bc3a1/href">https://medium.com/media/c54ec8170ee27e8653f48cf1970bc3a1/href</a></iframe><p>To use plotly’s animation directly, we’ll need to augment our data set into this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iv8-zExw81wQstGUYl5frw.png" /><figcaption>Data transformation for animated bars — Image by Author</figcaption></figure><p>We’ll want to effectively stack copies of our dataframe, with the number of copies being the length of our original dataframe. With each copy, we’ll want to add a Frame value to set its sequence in the animation. Where it gets tricky is that we want more bars to appear with each step of the sequence. We want to start with no bars showing on our plot, so for the first Frame we’ll set all of the values to 0. Then, on the second Frame we’ll set all of them to zero <strong><em>except</em></strong> the first data point (A: 10). This method continues until the last Frame (Frame #6 in this case) shows all of the values from our original data.</p><p>At this point we have set up individual “Frame” elements for Plotly to smoothly animate between. Note that we <strong>are not</strong> coding every frame of the animation, we simply are snapshotting the final value for each Label, one-by-one, and letting Plotly fill in the blanks. Now we just need a helper function so we can apply this transformation concept to any simple dataframe:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb90cda44e90273c417ef5773af94a59/href">https://medium.com/media/bb90cda44e90273c417ef5773af94a59/href</a></iframe><p>In this code, we start my creating our blank destination dataframe, which will be the returned output. We loop through for as many times as there are elements in the passed-in dataframe — range(len(starting_df)+1). We add 1 here, as the first Frame will have all values zeroed out so that our plot starts out empty. Within the for loop, we start with a deep copy of the original dataframe. This means that changes we make to this copy won’t propagate back to the original. On this free-and-clear copy we add a Frame value, and zero out some of the data points, depending on which loop we’re on. Finally, we can add this copy to the end of the final dataframe we’re constructing.</p><h3>Plotting</h3><p>From here we’re ready to start plotting. If we just use the stock plotly function, it looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acb83a508e5bb20338a8782d90945779/href">https://medium.com/media/acb83a508e5bb20338a8782d90945779/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v6xYKg4XBHXBSde-Zj4_CA.gif" /><figcaption>Default plotting setup — Image by Author</figcaption></figure><p>As you can see, it gets a little wonky at the get-go. This mainly has to do with how the plot autoscales the y-axis as the data appears. It’s not particularly surprising, as when it first plots (Frame=0), all of the values are zero. On top of that, the speed of the sequence may not be what you want. Maybe you want it to draw the bars a bit slower? Maybe it’d be nice to have a pause between drawing bars? These are all modifiable parameters. Best of all, with plotly, we don’t even have to have the bars grow linearly — there are a broad set transition settings you can use. To clean this plot into something I’d want to present, I’ll alter the:</p><ul><li>Y-axis range (take the max value in the dataframe and add 10%)</li><li>Figure size and aspect ratio</li><li>Duration of each Frame and the time between Frames</li><li>The animation movement itself (transition)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32eee9631fde1d89176ebb33f00a24a0/href">https://medium.com/media/32eee9631fde1d89176ebb33f00a24a0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*FYYMdO_Lgx1dx17CmbdM-w.gif" /><figcaption>Formatted animated bar plot using the “quad-out” animation — Image by Author</figcaption></figure><p>In this case, I used the “quad-out” transition, but there are bunch to chose from that can impact the visualization. Check out the documentation <a href="https://plotly.com/python/reference/layout/sliders/#layout-sliders-items-slider-transition-easing">here</a> and give a few a try — “bounce” in particular is pretty entertaining (though maybe not the most professional). There are also lots of <a href="https://plotly.com/python/reference/layout/sliders/">settings</a> you can alter to modify or remove the buttons below the plot, though depending on how you export your graph, it may not be necessary.</p><h3>Export to GIF</h3><p>Now for the bad news — there’s no built-in way to extract these animations in Plotly. You’ll have to do a work-around for now. There are packages you can use to extract to a movie file like moviepy, but this is a bit overkill if we’re looking to put this into a powerpoint or web-based customer dashboard. An animated GIF is really what we’re after here. While extracting the animation frames as pngs with Plotly’s graph_objects and using other packages to build the GIF inline is an option, it’s a bit out of scope for this post. For now, it’s easy to use a basic GIF screen-capture software on the Jupyter Notebook window. There are lots of options out there — <a href="https://www.screentogif.com/">here</a> is a free one that worked for me.</p><h4>Extra Credit</h4><p>Of course, this sequence is made to draw the bars from left to right. Should we want the opposite, Frame #1 would have a value for F (16) and A would still be 0. This may come in handy for line plots, where multiple lines can grow from different sides of the plot. 🤯</p><h3>Conclusion</h3><p>I hope you can see how adding a bit of animation can make a simple plot much more impactful to our audience. By using this relatively simple function we can quickly and easily transition our static plots into images that are dramatically more engaging. Of course, this is just the start! As followers of my channel know, I love enhancing plots with meaningful <a href="https://medium.com/towards-data-science/beautiful-bars-scaled-gradient-fill-on-bar-plots-e4da4cdae033">bar color gradients</a> and <a href="https://medium.com/towards-data-science/custom-matplotlib-colormaps-for-danger-zone-plots-62310983eb67">background transitions</a> to intuitively communicate more to our audience. More on that coming soon.</p><p>As usual, the entire code walk-through notebook can be <a href="https://github.com/bamattis/Blog/tree/main/Plotly_animation">snagged from my github</a>. <strong><em>Please follow me if you found this useful! </em></strong>Cheers, and happy coding out there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9ace6d73f27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/level-up-your-data-storytelling-with-animated-bar-charts-in-plotly-f9ace6d73f27">Level Up Your Data Storytelling with Animated Bar Charts in Plotly</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Flourish for Beautiful Animated Plots: a Tutorial]]></title>
            <link>https://brian-mattis.medium.com/using-flourish-for-beautiful-animated-plots-a-tutorial-bf648f428c62?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf648f428c62</guid>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[animated]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[flourish]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Fri, 30 Dec 2022 04:07:17 GMT</pubDate>
            <atom:updated>2022-12-30T04:07:17.455Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick introduction to creating stunning visualizations without coding</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hb_zO33WD5RWdn3e" /><figcaption>Photo by <a href="https://unsplash.com/@emrekaratas?utm_source=medium&amp;utm_medium=referral">Emre Karataş</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Animated plots are fantastic for giving your audience a sense of how things change. In a way, it helps tell a story of our data. When thinking about observing change in our data, <em>time</em> is the most obvious choice, but these visualizations can use any column variable to add another dimension to our analysis. This is extremely powerful in illustrating complex ideas and relationships to our audience — and the animated motion naturally attracts their eyes and attention.</p><p>Flourish isn’t the only game in town for animated plots —<a href="https://plotly.com/python/animations/"> Plotly</a> is the general go-to , with code very similar to Matplotlib. Surprisingly, even matplotlib can create animated plots with the PillowWriter function within matplotlib.animation , though just making simple plots is <a href="https://analyticsindiamag.com/a-hands-on-guide-to-creating-animated-plots-using-matplotlib/">pretty tricky</a>. What makes Flourish different is that it’s a no-code solution, with all knobs and dials to customize your plot within a web-based graphical user interface. It makes adjusting the aesthetics of your plot just a click away, with adjustments happening in real-time. Flourish also excels in making things look great right out of the box in most cases.</p><p>For Flourish, as in most things, the best learning is by doing. I’ve generated a racing bar chart and an animated pie chart for the sake of demonstration, and I’ll walk though the <em>how,</em> along with tips I learned along the way.</p><iframe src="https://flo.uri.sh/visualisation/11973629/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/554de6efb3a52f3d6ee7c51917a1c753/href">https://medium.com/media/554de6efb3a52f3d6ee7c51917a1c753/href</a></iframe><h4>The Setup</h4><p>First, to get started you’ll need to make a free account. It’s painless. Go <a href="https://app.flourish.studio/register?referer=true">here</a>.</p><p>Flourish is for plotting <strong>only</strong>. You’ll need to set up, clean, and format your data elsewhere. For these examples I used the public <a href="https://www.kaggle.com/datasets/zgrcemta/world-gdpgdp-gdp-per-capita-and-annual-growths?resource=download&amp;select=gdp.csv">world GDP by year</a> available on kaggle. We’ll use this to plot how the each country’s GDP grew and ranked over the last 60 years. I used pandas to clean, filter, and organize the data into the structure that Flourish needed. It’s beyond the scope of this article, but that data setup can be viewed <a href="https://github.com/bamattis/Blog/tree/main/Flourish_tutorial">here</a>.</p><p>However, before you start manipulating and setting up your own data, you need to know how Flourish wants it.</p><h3>Loading in Data</h3><h4>Tip#1: Look at the online Flourish example for a plot that you’d like to make</h4><p>Flourish provides a broad range of gorgeous <a href="https://flourish.studio/examples/">examples</a>. Once you have one in mind that you’d like to build off of, go to your project space and start a new visualization building off one of their templates. In this example we select the <strong><em>Bar Chart Race</em></strong>. Next, select the data tab at the top, and you’ll see the data organized like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYXyHYNeeKzgRKfJpXOe2A.png" /><figcaption>Loading the Flourish Bar Chart Race template — from flourish.studio</figcaption></figure><p>The data setup is dictated by the red box on the right. The example is plotting urban population by year, which has a similar setup to ours. We’ll also be increasing by year, and have a by-country breakdown. The data we’ll import will need to have columns by year, as this data does, and a column for country names at a minimum. In the red box, <strong>Categories</strong> refers to what we can have in a selectable legend to remove certain regions from our visualization. Finally, <strong>Image</strong> refers to the images associated with each bar. In this case it’s country flags, but could easily be corporate logos, head shots of celebrities or politicians, or anything, really.</p><h4>Tip #2: Flourish provides label images (for flags, at least)</h4><p>Flourish specifically provides an image set for flags, which can be added in as links or as shortcode into our table manually, as discussed <a href="https://help.flourish.studio/article/46-how-to-add-images-to-your-visualization">here</a>. However, these images can also come from any URL, or you can upload directly. In this case, we’ll merge these Flourish URLs into our GDP database in pandas rather than typing the shortcode in by hand.</p><h4>Tip #3: Getting the data pointers right gets you 90% of the way there</h4><p>Now, we’ll upload our adjusted and cleaned GDP file (github link <a href="https://github.com/bamattis/Blog/blob/main/Flourish_tutorial/df_GDP.csv">here</a>) using the ‘upload’ button on the right side of the ‘data’ screen. We’ll also want to name our project in the top left so that you can find it again later from your Flourish Projects page.</p><p>Looking at the columns, we’ll assign Label to the country name (A), the Values being the many columns of years (E-BM), the Categories to the Region (B) so that we can filter by continent, and assign the Images to the Image_URL (C) column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2epOsr2WWDFOzkYz_nGfNw.png" /><figcaption>GDP data setup —Image by Author (implemented in flourish.studio)</figcaption></figure><h3>Fine-tuning your chart</h3><p>When we first click back to the “Preview” tab along the top of the page, we see the first draft of our design. It’s… pretty good. For example, the legend is selectable, meaning we can filter out entire continents from our plot. We can also drag the timeline slider manually to see the data at any specific point in time. Before we dive into editing our formatting, one big tip:</p><h4>Tip #4: Before you adjust any value in the formatting settings, make a mental note of the parameter default value — there is no “return to default value” button to fix it later on.</h4><p>All of the rest is a matter of personal preference and style. Here’s what I did for this example:</p><p>Adding a title — no plot is complete without a title. Here we add in one in the <strong>Header</strong> section under <strong>Title</strong>, and adjust the alignment to centered.</p><p>Printing the values as billions of $ — We adjust this in the <strong>Number Formatting</strong> section, be adding “$” as the <strong>Prefix</strong>, “ B” as the <strong>Suffix</strong>, and ensuring the <strong>Decimal places</strong> are set to 0.</p><p>Fixing our labels — With the default settings, many country names are too long and get cut off. We can give them more space by adjusting the <strong>Space</strong> setting in the <strong>Labels</strong> section.</p><p>Ascending/Descending controls — I prefer adding a control bar to let our users decide if they want to sort largest-to-smallest or smallest to largest. I do that in the Controls formatting section and enable <strong>Show sort control</strong>. However, we can also set the default value in the <strong>Setup</strong> section under <strong>Sort mode</strong>.</p><p>To see what we’ve changed, here’s a quick look at our before and after, with a visual summary of the settings we’ve changed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H9EnSRXBvupq1tksU9PcPQ.png" /><figcaption>Before and after formatting fixes — Image by Author (implemented in flourish.studio)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RTmd-qcRl-eGG9lmuDJWEA.png" /><figcaption>Summary of formatting adjustments — Image by Author (implemented in flourish.studio)</figcaption></figure><p>There’s so much more you can tweak and adjust, which leads me to:</p><h4>Tip #5: Just Play! Adjust different settings and see what happens!</h4><p>Here’s a direct link to the final product shared earlier, so you can play with the sorting and timeline (<a href="https://public.flourish.studio/visualisation/11973629/">link</a>)</p><h3>Evaluation</h3><h4>Drawbacks</h4><p>The downsides of Flourish are pretty limited. One complaint is that at times some of the layout adjustment knobs don’t seem to actually change anything, as some knobs seem to be available even on plots that don’t use them. While the level control to adjust almost everything is nice, it’s sometimes challenging to see exactly which margin or spacing you’re adjusting. Luckily, after you make a few plots, you start to get a good feel for the settings hierarchy and menu options, so some of this goes away with experience.</p><p>Easily my biggest complaint refers to Tip #4 above: sometimes you’ll be modifying a parameter wildly, just to see visually what it is adjusting — but once you’re done playing, there’s no way to quickly go back to the Flourish “default” value.</p><h4>Cost</h4><p>It starts with a 100% free plan that allows for an unlimited number of projects, with no annoying time-limited “trial period”. The only catch is that all published (read: exported) projects are publicly viewable — so if you have business sensitive data, that becomes a problem. Solving this requires jumping to the first paid level — and it’s a <strong>big</strong> jump — <em>$999 a month. </em>However, it does come with quite a few nice features, including custom themes for standardizing across company projects, and supporting visualizing “live” data.</p><h4>What’s next?</h4><p>I’ve included another source file in my github to play with as well, which looks at <a href="https://github.com/bamattis/Blog/blob/main/Flourish_tutorial/gdp_per_capita.csv">GDP per capita </a>on this same <em>Bar Chart Race</em> format. It’s produces some pretty surprising results!</p><p>I’ve also built a <em>Pie Chart with Time Slider</em> as another animated example. I’ve taken the same data, but it needs to be sorted a bit differently to use this kind of plot. The reconfigured GDP data set is available in my github <a href="https://github.com/bamattis/Blog/blob/main/Flourish_tutorial/df_GDP_Pie_reorder.csv">here</a>, and the full, controllable visualization can be seen <a href="https://public.flourish.studio/visualisation/12260620/">here</a>. This one takes quite a bit more adjusting in the data loading and formatting, but it’s a fun exercise.</p><iframe src="https://flo.uri.sh/visualisation/12260620/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa03a85828b63b8104bca4f3ea33b583/href">https://medium.com/media/aa03a85828b63b8104bca4f3ea33b583/href</a></iframe><p>I’ve found Flourish to be a great graphing platform to create stunning visualizations without long lines of code. The default graphs are excellent, and Flourish provides enough formatting knobs to provide a sense of freedom in design. Beyond the plots shown here, Flourish supports many unique (and intuitive!) plots that you won’t find supported in Plotly or Matplotlib.</p><p>Thanks for reading! As always, the full project code can be found on my github page <a href="https://github.com/bamattis/Blog/tree/main/Flourish_tutorial">here</a>. If you found this article helpful, please click the Follow button!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*cMPVrPqypHPI5rmPFB4LKA.png" /><figcaption>Bar chart race comparing country GDP versus time — Image by Author (implemented in flourish.studio)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf648f428c62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Quick Tricks For Better Plots in Matplotlib]]></title>
            <link>https://medium.com/data-science/4-quick-tricks-for-better-plots-in-matplotlib-1efd82acd780?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/1efd82acd780</guid>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[arrow]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 06:48:49 GMT</pubDate>
            <atom:updated>2022-07-26T19:49:53.557Z</atom:updated>
            <content:encoded><![CDATA[<h4>Easily adding arrows, multiple axes, gradient fill, and more</h4><p>When we start learning data visualization methods with tools like matplotlib, we generally start with the simplest possible plots for ease of coding. However, as we begin sharing our visualizations with others, it’s important to customize our plots to craft our message and make everything more visually appealing.</p><p>Rather than create complex code and spending hours to make a plot look “just right,” we can drop in a couple of lines of code to make our plots more functional and help the data tell a story to our audience.</p><p>Here are 4 quick tips to add emphasis, functionality, and visual appeal to basic matplotlib line and bar plots:</p><h3>1. Emphasizing on Bar Plots with Color</h3><p>The goal is to make standalone plots that can tell a story, whether you’re there to present the slides or not. Before sharing a graph, you should be thinking <em>“what conclusions do I want my audience to draw from this?”</em> and <em>“what part of the data do I really want them to notice?”</em> To help with this, we can use <strong>lists</strong> to customize a large number of the tunable parameters that come with bar plots. What was color=&#39;blue&#39; can become color=[&#39;blue&#39;, &#39;red&#39;, ...] with each entry indicating what to do with <strong>each</strong> bar in our plot.</p><p>As an example, a simple bar plot looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e906fe191638dcc67b82be14f304b6ff/href">https://medium.com/media/e906fe191638dcc67b82be14f304b6ff/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*WdMHUNI12M7p73rqgGKRvA.png" /></figure><p>However, if we wanted to draw attention to the third bar (for example), we can do that by changing the color, edgecolor, and linewidth inputs into lists:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5fc609a04f1d4c6140cda14ae65cc97/href">https://medium.com/media/c5fc609a04f1d4c6140cda14ae65cc97/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*rcQVcK6xY2lfwLQifkYqxQ.png" /></figure><h3>2. Emphasizing on Line Plots with Arrows</h3><p>Line plots can be challenging to add emphasis. For line plots, simplest is usually best — adding an arrow pointing to a specific data point is a great way to pull the audience’s eyes. While matplotlib has a specific <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.arrow.html">arrow function</a>, it’s pretty inflexible and oddly challenging to get arrows to show up “just right”. A much simpler and quicker way to go is using the <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.annotate.html?highlight=annotate#matplotlib.axes.Axes.annotate">annotate function</a>. Here, we get lots of functionality that is pretty intuitive. Our two main variables xytext and xy tell where the arrow <strong>starts </strong>and <strong>ends</strong>, respectively. The xytext is also the end of the arrow where text will be placed, should we choose to label the arrow (but we don’t have to!).</p><p>We can adjust the arrow properties using arrowprops=dict(), wherein we can use the standard keys like color, linewidth, and more. That gets you simple, straight lines — but things can get more interesting from there. We can adjust the shape of the arrowhead with arrowstyle (full list <a href="https://matplotlib.org/stable/gallery/text_labels_and_annotations/fancyarrow_demo.html">here</a>), and also deviate from a straight arrow path. connectionstyle comes in three main flavors —</p><ul><li>arc3 — curved path where the degree of curvature is set in radians (rad). Play with positive and negative rad values to turn your curve from concave to convex. In a line plot with straight connections, adding a bit of curvature to the arrow path definitely helps it get noticed.</li><li>angle — segments of the line drawn at a specified angles ( angleA, angleB). The method is smart enough to make just about whatever angles you choose connect between your text and xy coordinates.</li><li>bar — segments are kept at 90 degree bends of one-another, though the entire arrow can be based on a common angle. By playing with fraction here we’re changing where the <strong>turn</strong> is in relation to the endpoints. Try negative values on the code below and see what happens!</li></ul><p>Given the large number of varieties of arrows you can create, it’s best to experiment with the parameters to get exactly what you’re looking for. Here’s an example with 6 different arrows applied to our plot to help visualize what is possible.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04f0bcad07ae87b50d84ef47557c410d/href">https://medium.com/media/04f0bcad07ae87b50d84ef47557c410d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*pqEXgyk31QMqWlG9vVe1Kg.png" /></figure><h3>3. Adding a Secondary Y-axis</h3><p>Overlaying multiple columns of data on a single plot can drive conclusions about the relationships between them. However, it’s rare that both columns operate on the same numerical scale, causing scaling issues where one of the lines hovers near the axis. To build a secondary y-axis, we use the twinx() function to create a secondary y-axis that shares the same x-axis. This secondary axis will automatically be positioned on the far right, and will be auto-scaled based on the data fed to this ax item. It is important to use color to help our audience understand which plot is associated with each axis. To really drive the point home, arrows can be added using the tricks we just learned.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21bf555fada928748cdad97476a7d015/href">https://medium.com/media/21bf555fada928748cdad97476a7d015/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*8CwPEKMuaGVhnAAapROFdw.png" /></figure><p>Side note: adding a secondary y-axis can also be easily done in native pandas, using the secondary_y = True attribute, yielding an almost identical result. If you’re curious, the code is <a href="https://gist.github.com/bamattis/7ff2faa50ff5d884edad37350e8667ce">here</a> and the resulting plot is <a href="https://github.com/bamattis/Blog/blob/main/MatplotLib_Tips/pandas_lines.png">here</a>.</p><h3>4. Adding Fill Color</h3><p>We can also add emphasis and general visual appeal to our line plots by filling in the region below the line. This is a quick addition using the fill_between() function where we indicate the x range that we want it to fill, and the two “lines” that we’d like to fill between. In this case, we’re wanting to fill between the data and the bottom of the plot (y=0 in this case). It’s also a good idea to set the plot area to match your data with plt.axis(), otherwise the color won’t continue to fill outside of your data range, leaving unsightly white areas in your plot.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10d217252561f199b93d4a136937eed7/href">https://medium.com/media/10d217252561f199b93d4a136937eed7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NsShnZi_9ivhkQEBujotDw.png" /><figcaption>Line plot fill with and without defining the plot area to match the data — Image by Author</figcaption></figure><p>We can push the limits a bit further — turning our fill into a color gradient. In this case, we have to get creative. imshow() is great for adding gradients, but it traditionally does it over our entire plot area. To overcome this, we add a gradient with imshow(), then use fill_between() to color everything <strong>above </strong>our line white. imshow() is a bit tricky to get off the ground, and I’ve gone into more depth of it in one of my prior articles linked <a href="https://towardsdatascience.com/custom-matplotlib-colormaps-for-danger-zone-plots-62310983eb67">here</a>. However, the simple version has us set the range of the color map we’re using in X — here we choose a vertical color gradient using the bottom 80% of the cm.Blues color map with [[0.8, 0.8],[0,0]]. (A similar horizontal gradient could be achieved like this: X=[[0,0.8][0,0.8]]).</p><p>We also set the extent of the imshow() fill, effectively setting the bounding box for this gradient image fill. We can set the values manually, or use our min() and max() functions on our data series. These can be simply augmented to extend the plot region further down. Finally, we set the upper bound of the plot area in the fill_between() function, as we now go from our data to a the top of the plot (defined here as df.b.max()+0.5 ).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f79b9d60e65bba83a97fc09417834df6/href">https://medium.com/media/f79b9d60e65bba83a97fc09417834df6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*KmB-dZ7hcRS9k7vLF9Bw7Q.png" /></figure><h3>Extra Credit</h3><p>Taking what we’ve learned, we can now combine our methods to join two line plots using a secondary axis, while also adding fill and annotating an important point with an arrow.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c25eae65e1d300653e416d72c6d693d/href">https://medium.com/media/9c25eae65e1d300653e416d72c6d693d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*RFzK2_u8QjPnakNWdK9taA.png" /></figure><h3>Conclusions</h3><p>With a few quick adjustments to our code we can add both emphasis and visual appeal to our matplotlib plots. Emphasized plots enable our data to speak for itself and can instantly drive others to focus on the data points that <strong><em>we</em></strong> think are worth paying attention to. Luckily, we can reach this goal with just a few lines of code!</p><p>As always, the entire code walk-through notebook can be <a href="https://github.com/bamattis/Blog/tree/main/MatplotLib_Tips">snagged from my github</a>. <strong><em>Please follow me if you found this useful! </em></strong>Cheers, and happy coding out there.</p><p><em>All images generated by Author</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1efd82acd780" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/4-quick-tricks-for-better-plots-in-matplotlib-1efd82acd780">4 Quick Tricks For Better Plots in Matplotlib</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beautiful Bars: Scaled Gradient Fill on Bar Plots]]></title>
            <link>https://medium.com/data-science/beautiful-bars-scaled-gradient-fill-on-bar-plots-e4da4cdae033?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4da4cdae033</guid>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[bar-plot]]></category>
            <category><![CDATA[gradient]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[matplotlib-tutorial]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Mon, 04 Jul 2022 13:51:43 GMT</pubDate>
            <atom:updated>2022-07-06T15:59:59.523Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using color to add <strong>meaning</strong> to your plots in matplotlib</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aFOh-dR-U48_IUHL" /><figcaption>Photo by <a href="https://unsplash.com/@denmychan?utm_source=medium&amp;utm_medium=referral">Denise Chan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Bar plots are one of the most basic and useful ways to convey data to our audience. While there are more data-dense plots like box plots, violin plots, and heat-maps, no plot is as universally understood by non-technical audiences like the bar plot.</p><p>On the flip-side, bar plots really are <em>boring</em>. They’re also very simplistic in that they don’t give the audience any intuition on whether the values shown are good, bad, or otherwise.</p><p>In this article, we’re going to start by spicing up the basic bar plot with gradient color fill to add a little flair. Then, we’re going to show you how to <strong>scale</strong> the gradient fill so that the individual bars don’t traverse the entire color range, naturally emphasizing the <strong><em>differences</em></strong> between values. After all, isn’t this usually the point we’re trying to convey with bar plots?</p><p>Finally, we’re going to see how to apply custom color maps where we can specifically define where the color changes, so we can illustrate where certain bars are indicating problems and further draw the audience’s eyes to the most important points. Let’s get started!</p><p>First, we’ll import the standard packages, as well as some ones specific to the gradient color maps we’ll be generating later on. Next, we’ll create a simple dataframe for our data, along with a very bland bar plot.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44c1f6c31fcf5b41093b236ed0e6efbf/href">https://medium.com/media/44c1f6c31fcf5b41093b236ed0e6efbf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*eCd2am3bSfyM5r26bhI7qw.png" /><figcaption>The most boring of bar plots… 😴</figcaption></figure><p>Now, simply changing the color of the bars isn’t hard ( ax.bar(df.a, df.b, color = &#39;orange&#39;) for example), but you didn’t come here for that, did you?</p><h4>Basic Gradient Fill (for style, not substance)</h4><p>To perform the fill, we going to create a basic bar plot and pass it into a custom function as a variable. From here, we’re going to pick it apart, bar by bar, and apply the ax.imshow() function to apply colormaps (cmaps for short).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0e22a200663ea1c03c20dfc6c6fa227/href">https://medium.com/media/f0e22a200663ea1c03c20dfc6c6fa227/href</a></iframe><p>One of the most challenging parts of using imshow() is setting up the first variable it needs (“X” in the documentation). Here, we want to progress through our selected color map linearly, so we’re going to create a linear array of 256 elements between 0 and 1 with np.linspace(). It’s important to note the .T in the linspace() code below — that sets up the gradient to be in the right orientation for a vertical gradient. Try taking it off and seeing what happens!</p><p>Next, we reestablish ax within the function so that we can extract the x and y limits of our total plot region. We extract those limits with get_xlim() and get_ylim(). Finally, we apply these with the ax.axis() function. Although it seems silly, these steps are necessary to tell matplotlib to not fill the entire plot area with our color map.</p><p>Finally, we step through each bar of our bar plot in a for loop. We first must set the bar’s facecolor() to “none”, otherwise the base color will be placed on top of the gradient fill we add. Alternatively, this line can be skipped if we increase the zorder in the imshow() function — but more on that later. We then extract the features of the bar, and then use imshow() to apply our gradient fill over that extent (the bar x, y coordinates).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*P9SgPN0Kk0uEWaa1lX0FLw.png" /><figcaption>Uniform gradient fill using default cmap</figcaption></figure><h4>Advanced Gradient Fill</h4><p>While the above plot certainly looks prettier, it certainly isn’t telling us anything about the data that we didn’t already know. When we’re showing bar plots, the highest points are usually more meaningful, good or bad. <em>We really want to draw our audience’s eye to differences in the bar heights.</em></p><p>We can do this as a fairly simple upgrade to our previous function. Before we set the grad variable at the top, so all bars would traverse between 0 and 256 colors on the color map. To scale our colors, what we really want is for the <strong><em>largest</em></strong> bar to get the full color range, and the other bars to traverse a subset of that. To do this, we move the grad call into our bar for loop. Each time it is called, it generates an array of 256 numbers, but we modify the step size. So, for the largest bar, grad goes from 0 to 1 (with 256 steps), but for all other bars grad doesn’t make it to 1. The step size is scaled by the height of that particular bar ( h ), versus the height of the max bar ( max(ydata) ). To make this happen, you may have also noticed that we now pass in ydata as part of the function call. We’ve also passed in a cmap name, so we can choose our gradient cmap from the function call.</p><p>One more addition — in the imshow() function we have to turn off norm, otherwise it would undo our magic with grad and auto-scale it to be between 0 and 1 for all inputs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a984c04cca94bc2842538860b2c58ce9/href">https://medium.com/media/a984c04cca94bc2842538860b2c58ce9/href</a></iframe><p>We can try this with different color maps (cmaps) to see what best suits our data’s story:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qEhgmU0PAiGdhIQjFF4sTQ.png" /><figcaption>Using ‘viridis’, ‘hot’ and ‘cool’ cmaps, respectively</figcaption></figure><p>In these plots we see that only the taller bars are able to reach certain colors. This helps drive our audience’s eye. <em>The difference is subtle, but important.</em> In the side-by-side comparison shown below, this new scaled gradient fill (left) only reaches into the purple color range on bars 3 and 5, giving them a higher gravity. In this case, we can make our audience think “is it bad if a bar gets to purple?”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DD7ev9jCn8XFtsrSfljUSw.png" /><figcaption>Comparing scaled gradient fill to unscaled gradient fill (Image by Author)</figcaption></figure><h4>Refining The Plot</h4><p>Before we share this plot with an audience, we need to do some general visual clean-up. We’re going to add grid lines as well as borders to our bars to keep them from blending into the background (since some colors maps can be pretty light). The only difference between this and traditional bar plots is that a little extra care has to be taken with zorder, which controls which elements are in-front of other elements. Since we do the bar coloring within a function, but after the bars are fundamentally generated, we add zorder like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aeeeaa9fbc192c936250b22ce934d9b7/href">https://medium.com/media/aeeeaa9fbc192c936250b22ce934d9b7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*3Db2T3hofj-TPmWh9L-VZQ.png" /><figcaption>Scaled gradients in ‘YlOrRd’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*EotscvxlKBcB0BPG8GgO6g.png" /><figcaption>Scaled gradients in ‘viridis_r’</figcaption></figure><h4>Fully Customized Gradients (extra credit)</h4><p>Let’s get picky. What if we want our bars to change to a certain color at a certain value to better reflect when things need our audience’s attention. Do this, we need to construct custom color maps. <a href="https://towardsdatascience.com/custom-matplotlib-colormaps-for-danger-zone-plots-62310983eb67">I covered this in-depth in my last post</a>, showing how to color maps work, and how to functionalize building color maps that transition between 3 colors. Even better, these functions enabled us to choose the colors, choose where the transitions happen, and how blended the transitions are. Rather than step through the code to make the cmaps shown below, I’ll jump to giving you a taste of how they can look on our gradient fill plots. I’ll use the same scaled gradient fill code above, but now just plug in custom cmaps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09de9b312d5fe32d7f9fbb6c6aef95bc/href">https://medium.com/media/09de9b312d5fe32d7f9fbb6c6aef95bc/href</a></iframe><p>In the code above, we choose to transition from the first to second color at 3.5 (with a blend of 1), and to transition to the third color at 5.5 (also with a blend of 1). The message we’re giving to our audience is that values above 3.5 should have some cause for concern and that values above 5.5 are outright dangerous. See if you get that same message just by looking at this plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*yR6j5_qJUeS-T5HfB7FBUw.png" /><figcaption>3-level custom cmap applied to our bars</figcaption></figure><h4>Summary</h4><p>Bar plots are usually very vanilla, but we can use gradients fills to spice them up. Even better, we can use <em>scaled gradients</em> and <em>custom scaled gradients </em>to teach our audience how to interpret the data. The good news — you now have the tools to make beautiful (and meaningful!) bar plots. The bad news — standard bar plots will now look more plain and boring to you than ever before (scroll up to the first bar plot and cringe 😬).</p><p><strong><em>Thanks for reading, and if you found this helpful, please follow me!</em></strong></p><p>As always, the entire code walk-through notebook can be <a href="https://github.com/bamattis/Blog/tree/main/Gradient_Bars">snagged from my github</a>. Cheers, and happy coding out there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4da4cdae033" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/beautiful-bars-scaled-gradient-fill-on-bar-plots-e4da4cdae033">Beautiful Bars: Scaled Gradient Fill on Bar Plots</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Matplotlib Colormaps for Danger-Zone Plots]]></title>
            <link>https://medium.com/data-science/custom-matplotlib-colormaps-for-danger-zone-plots-62310983eb67?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/62310983eb67</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gradient]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Fri, 03 Jun 2022 03:44:48 GMT</pubDate>
            <atom:updated>2022-06-03T03:44:48.741Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building precise background color gradients for more meaningful plots</h4><figure><img alt="Colorful flower" src="https://cdn-images-1.medium.com/max/1024/0*OEen0jVadkSTwKAW" /><figcaption>Photo by <a href="https://unsplash.com/@sharonp?utm_source=medium&amp;utm_medium=referral">Sharon Pittaway</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Context — it’s everything. After we sort, organize, and interpret our data, we make descriptive visualizations to tell the story of the data. When we’re using the plots to understand the data ourselves, it’s all well and good — we understand the context of our plot. However, if our goal is to inform others and drive decision making in a broader audience, we want plots that don’t just share the data, but clearly drive a conclusion. Your audience simply won’t know the data as well as you, so it’s up to you to guide their thought processes. This usually entails plot elements like trend-lines or color coding, but even then you’re assuming a lot from your audience — is an upward trend a good or bad thing? How will they know?</p><p>Here, we’re going to describe how to create custom matplotlib cmap objects (color maps) to create “danger-zone” plots. These create plots that are instantly understood by technical and non-technical audiences alike to show when some data is trending in a way that’s “bad” for the organization. Compare the two plots below — without knowing anything about what this data is, you’d know from the plot on the right that there’s something “bad” about months 2 and 3, just by the addition of some custom background color gradients.</p><figure><img alt="Left plot has no background color. Right plot fades from white to yellow to red" src="https://cdn-images-1.medium.com/max/864/1*Vfh2QB0LpA46M00I6fMvUw.png" /><figcaption>Traditional bar plot vs. a “danger-zone” bar plot — Image by Author</figcaption></figure><p>We also don’t always control how our data is shared. What may start as a presentation to internal and external customers (where you can explain the data trends and insights) may turn into your figures being sent by email without deeper context. If the plot can explain itself <strong>and</strong> indicate conclusions one <em>should</em> draw, it makes it less mentally taxing on those seeing it for the first time. It also helps ensure that others draw the same conclusions that you did.</p><h3>What makes a colormap (cmap)?</h3><p>Before we go about making custom cmaps, let’s dive into the basics of how they work. The built-in matplotlib cmaps are color gradient groups that fit into one of four categories: sequential, diverging, cyclic, and qualitative. There‘s a solid visual repository for your options in matplotlib’s documentation <a href="https://matplotlib.org/stable/tutorials/colors/colormaps.html">here</a>, but here’s a quick taste:</p><figure><img alt="Four different color-map examples" src="https://cdn-images-1.medium.com/max/460/1*hV_jOMguGS2MWM1MwkxrUg.png" /><figcaption>Examples of different pre-loaded cmap types — Image by Author</figcaption></figure><p>When we fetch a cmap object, python generates a 256-element list of colors used to define the colormap. Each of the colors is internally represented as a 4-element array of Red-Green-Blue-Alpha (RGBA), where alpha is best thought of as the “transparency” of the color. So, for example, if we pull the &#39;Reds&#39; cmap object, and investigate the 100th color (out of 256), we see this:</p><figure><img alt="Interrogating the color components of a cmap" src="https://cdn-images-1.medium.com/max/613/1*D_JaugP62MqDEVw_vdBjRw.png" /><figcaption>Investigating the ‘Reds’ cmap</figcaption></figure><p>Of course, none of the pre-made cmaps are going to perfectly fit with the specific colors and transition points we want to make a proper ‘danger plot’. But with this kind of direct access to a cmap’s core elements, re-writing rows to get what we want becomes and interesting proposition. This is certainly one way to do it (which I include in my <a href="https://github.com/bamattis/Blog/tree/main/Danger_Plots">github notes</a>), but it’s harder than it needs to be, especially when trying to manually blend between colors.</p><p>As a more automated path, we’ll use LineasrSegmentedColormap() from matplotlib. But before we get there, we’ll need a lightweight function to view the colormaps we create before we apply them to our plots:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b929eb27cc003c7162b116f7fea6284b/href">https://medium.com/media/b929eb27cc003c7162b116f7fea6284b/href</a></iframe><figure><img alt="gradient from light red to dark red" src="https://cdn-images-1.medium.com/max/461/1*ItplS4lG92n3i2F9ggfBzA.png" /><figcaption>Reds color gradient plot — Image by Author</figcaption></figure><h3>Building Custom CMAPs</h3><p>The LinearSegmentedColormap() function allows us to give a list of colors, with a list of “nodes”. The “nodes” refers from points on the colormap (scaled from 0 to 1) where the listed colors reside — outside of that, it blends with it’s neighbors. So, in the example below, the 256-element cmap has element 1(node 0.0) as pure green (rgba= 0,1,0,1), element 128 (0.5) as pure yellow (1,1,0,1) and element 256(1.0) as pure red (1,0,0,1).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb593baf40d2e1e8d029db506c5bd21d/href">https://medium.com/media/cb593baf40d2e1e8d029db506c5bd21d/href</a></iframe><figure><img alt="Gradient from green to yellow to red" src="https://cdn-images-1.medium.com/max/447/1*zb0SNzznh0ES0-m-2nGnhg.png" /><figcaption>3-element cmap — Image by Author</figcaption></figure><p>Ultimately, we’re going to design a function where we tailor where colors transition. By adding more nodes we can effectively do this. If two neighboring nodes are the same color, then the space between them will automatically be the same color. Take a look at the following example, where we add some repetitive colors:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18e21842d406bccd46cea8a11db562aa/href">https://medium.com/media/18e21842d406bccd46cea8a11db562aa/href</a></iframe><figure><img alt="Gradient from green to yellow to red" src="https://cdn-images-1.medium.com/max/455/1*RywGQwjPNkubPZFWJ-G36A.png" /><figcaption>5-element cmap — Image by Author</figcaption></figure><p>Ah! Much tighter transitions! In this case, we transition from green to yellow between 40% and 50%, and from yellow to red between 60%-70%. Great! This in itself could be used to manually create the cmaps needed for your own custom-shaded danger plots. However, when we’re dealing with our data, we’re more familiar with the <strong><em>value</em></strong> that we want the cmap to change color, instead of the % of the total axis length. Let’s functionalize this for our plots.</p><h3>Two-Color and Three-Color Custom CMAPs</h3><p>Using this same technique, we’ll build 2-color blends and 3-color blends. These functions both take in the colors we want to use — keep in mind that LinearSegmentedColormap() is flexible enough to take in colors as names (‘Red’), rgba ([1,0,0,1]), or even hex (#ff0000). In fact, here’s a simple <a href="https://www.google.com/search?channel=nus5&amp;client=firefox-b-1-d&amp;q=hex+color+picker">color-picker tool</a> to help you get just the color you want. These functions also take in the value for the transition between colors, the width of that transition, as well as the plot axis limits (min and max) so that the transition values can be scaled appropriately.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25ddba5da24048a2de34304224ca6c3e/href">https://medium.com/media/25ddba5da24048a2de34304224ca6c3e/href</a></iframe><figure><img alt="Gradient from white to red, with a specific transition point at 50%" src="https://cdn-images-1.medium.com/max/507/1*lqe9jBqZI05HJf1MVegVPA.png" /><figcaption>Targeted transition cmap — Image by Author</figcaption></figure><p>In the function call, we pretend we have a plot with a min of 0 and max of 10, with the transition happening right at 5, with a width of 2. Sure enough, the transition occurs halfway through the 256-element generated cmap.</p><p>Expanding this concept to a 3-color function, we do much of the same, just with a little more complexity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39da345831081337556dd4d2f1f6bb93/href">https://medium.com/media/39da345831081337556dd4d2f1f6bb93/href</a></iframe><figure><img alt="Gradient from green to yellow to red, with sharp transitions" src="https://cdn-images-1.medium.com/max/663/1*Pux5JyhoqFWk_HgZ6mhIoQ.png" /><figcaption>3-color targeted transition cmap — Image by Author</figcaption></figure><h3>Applying to Plots</h3><p>First, let’s generate some dummy data to use for our plots. Here we have a dataframe with the number of patients seen in a hospital per month:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/199234589091ef9a6713dbdb26da5218/href">https://medium.com/media/199234589091ef9a6713dbdb26da5218/href</a></iframe><p>To add any gradient cmap to the background of a matplotlib plot, we use imshow(), which displays data as an image. The most challenging variable to pass is the first one (“X”). This maps a location on the plot to an element in our cmap. Since we want to step across our custom cmap linearly, we’ll use np.linspace() to create a linear array of 256 elements between 0 and 1. We’ll also want to pass in the extent parameter to let imshow() know the limits of the plot that we’re stretching our cmap across. Finally, note that ylim() gives an output list of (y min, y max), so we use bracket notation to access each as the limits when calling our two_color_cmap() function.</p><p>Applying to our data and setting an arbitrary transition point, we get this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d6c1fd1f4fa71ee7e9884815ab16227/href">https://medium.com/media/8d6c1fd1f4fa71ee7e9884815ab16227/href</a></iframe><figure><img alt="plot with background transitioning from white to red" src="https://cdn-images-1.medium.com/max/455/1*TE1yN-aqA0xd3K9FXzxAAg.png" /><figcaption>2-color Danger-Zone plot - Image by Author</figcaption></figure><p>Or, if want to create a ‘warning’ region before we get into a red ‘danger’ zone, we can use our three-color function to add in some yellow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e38a64b8a2cf0a499d315c7c0c6c37a/href">https://medium.com/media/6e38a64b8a2cf0a499d315c7c0c6c37a/href</a></iframe><figure><img alt="Danger-Zone plot with background transitioning from white to yellow to red" src="https://cdn-images-1.medium.com/max/452/1*3I1NLRe0oXVSema1NRONgA.png" /><figcaption>3-color Danger-Zone plot — Image by Author</figcaption></figure><p>Awesome! Now our audience’s eyes are drawn to Month 2, where there were an unacceptably high number of patients.</p><h4>Horizontal Bar Charts</h4><p>These custom cmaps also support horizontal plots. We only have to make a few tweaks to our plot code. The most important is how we change the “X” variable in imshow(). In this case, we remove the transpose .T which turns grad from a set of 256 1-element arrays to a single 256-element array. Investigate the grad variable with print(grad) to understand the subtle differences on how we make this work. Finally, note that np.linspace() now goes from 0-&gt;1. If we ever wanted to flip how our cmap is displayed, we can turn this back into 1-&gt;0 like we used previously.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e3c533eec5c6aa9d999e81c9520029d/href">https://medium.com/media/9e3c533eec5c6aa9d999e81c9520029d/href</a></iframe><figure><img alt="horizontal Danger-Zone plot with background transitioning from white to yellow to red" src="https://cdn-images-1.medium.com/max/341/1*k72jvtQMDDMXBLS52iC8KQ.png" /><figcaption>Horizontal 3-color Danger-Zone plot — Image by Author</figcaption></figure><h4>Applying to Line Plots</h4><p>There are many situations where we want to highlight an acceptable ‘range’ for our data. In such cases, both high and low values indicate problems, but in the middle there is a happy medium. For these situations line plots make a more telling visual. Using the same data and the same three_color_cmap() function we made, we can make a line plots with our custom cmap.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1976385a99a36d4ac93cc2fa806ffaf4/href">https://medium.com/media/1976385a99a36d4ac93cc2fa806ffaf4/href</a></iframe><figure><img alt="Line Danger-zone plots with gradients from red to white to red" src="https://cdn-images-1.medium.com/max/1024/1*5DZ3-Zc8iw7pUH3bwQ4t2A.png" /><figcaption>Lineplots with modified cmap transition widths — Image by Author</figcaption></figure><p>On the left is one of our standard ‘blended’ color maps, but our custom cmap functions work perfectly well for zero-transition cmaps. The zero-transition cmap on the right would be more useful if the values for failure are very specifically known and there is no “wiggle-room”. This is particularly useful for <a href="https://www.simplilearn.com/spc-charts-article">statistical process control (SPC)</a> plots where known upper and lower control limits exist.</p><h3>Conclusions</h3><p>Besides being visually appealing, Danger Plots add context to our plots. They enable our data to speak for itself and can instantly empower anyone to identify data points representing when something has gone wrong. With the framework of these cmap-generating functions, the sky is the limit — make your own and join as many colors as you want! As you’ve seen, once our setup functions in place,they’re simple to implement, so you’re not burning calories generating the perfect plot.</p><p>As usual, the entire code walk-through notebook can be <a href="https://github.com/bamattis/Blog/tree/main/Danger_Plots">snagged from my github</a>. <strong><em>Please follow me if you found this useful! </em></strong>Cheers, and happy coding out there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62310983eb67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/custom-matplotlib-colormaps-for-danger-zone-plots-62310983eb67">Custom Matplotlib Colormaps for Danger-Zone Plots</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Cleaning: Automatically Removing Bad Data]]></title>
            <link>https://medium.com/data-science/data-cleaning-automatically-removing-bad-data-c4274c21e299?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4274c21e299</guid>
            <category><![CDATA[data-type]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[bad-data]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Thu, 07 Apr 2022 13:49:39 GMT</pubDate>
            <atom:updated>2022-04-07T13:49:39.422Z</atom:updated>
            <content:encoded><![CDATA[<h4>Purging wrong data-type entries from numeric and character columns</h4><figure><img alt="Vacuum cleaner sucking up legos" src="https://cdn-images-1.medium.com/max/1024/0*4hdsPMuU6E5M8654" /><figcaption>Photo by <a href="https://unsplash.com/@norevisions?utm_source=medium&amp;utm_medium=referral">No Revisions</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Cleaning data is almost always one of the first steps you need to take after importing your dataset. Pandas has lots of great functions for cleaning, with functions like isnull(), dropna(), drop_duplicates(), and many more. However, there’s two major situations that aren’t covered:</p><ul><li>A would-be numeric column is littered with strings and booleans</li><li>An aspiring character column has sporadic numeric values and booleans</li></ul><p>This bad data can be specifically challenging to isolate and remove — here I’ll show you the tools to do just that. But first, there’s another great reason to get this bad data removed before doing any further analysis… <em>the object type</em>.</p><h4>The Object Type</h4><p>While pandas typically does a pretty good job of deciphering the data type of our columns during import, it almost always gets tripped up when bad data introduces elements of a very different data type. The end result is that the column’s data type ( dtype) gets set to the dreaded <strong><em>object</em></strong>.</p><p>Why is that so bad? For starters, it will cause headaches when sorting, plotting, and filtering data, and merging data from another dataframe. To make things worse, objects consume <strong><em>a lot</em></strong> of compute resources, making everything else you do with your data — including EDA and machine learning — much slower. And of course, to fix a column’s data type, we first need all of the data to be consistent with that new data type, otherwise we’ll get errors during conversion. This means purging all of the data of the wrong type that doesn’t belong.</p><p>Unfortunately, there simply are no go-to functions for cleaning up mixed-type data in Pandas, which usually leads to a series of sorting methods to tease out their locations and removing them one-by-one. Instead, with some fairly simple custom functions, we can automate the entire process!</p><h4>Setup</h4><p>To give us some data to play with, here’s a quick dataframe with repeating elements, with the first column representing our would-be numeric column, and the second column representing our would-be character column — but both have a healthy dose of bad data of different dtypes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3499ebdb906aee66b4ea303ffbb7b44a/href">https://medium.com/media/3499ebdb906aee66b4ea303ffbb7b44a/href</a></iframe><p>Inspecting df.head() and df.dtypes we see our data, along with the dreaded <strong>object</strong> dtype for both columns. Also note how in this object form, we’re unable to detect NaNs directly — something we’ll need to pay attention to when cleaning our data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*blswmbP8dinPjYi-o0guAQ.png" /></figure><h3>Purging chars and booleans from a numeric column</h3><p>First, let’s create a simple function to list the contents and the row numbers of all of the rows in num_col that contain non-numeric values. An important note here is that we’re choosing to not report null values (NaNs). This leaves it up to the user later on to remove these with dropna() , or convert them to some other value as desired once everything else is cleaned up.</p><p>To enable this, we go through each row and attempt to turn the value into a float. Since NaN is considered a float value, these rows get left alone. Meanwhile, if the value isn’t a number or an NaN, the float() function returns a ValueError. In our function that triggers it to print the offending value and the row number.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48ec538fb6800b22fa42f63889a7f0d9/href">https://medium.com/media/48ec538fb6800b22fa42f63889a7f0d9/href</a></iframe><p>If we use this on our existing dataframe, it immediately captures the two non-float values that are repeated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*tNbw08_ri08CpgeO2u28qA.png" /></figure><p>Great! We now know where all of the non-numeric values are hiding in our dataframe. After this inspection, we can draft a very similar function to fully remove these misfit rows. Once we find them, we now can simply drop the row — and of course we can always customize the line after ValueError to instead assign it to another (numeric) value we prefer instead.</p><p>After we loop through the dataframe column, we then use astype() to attempt to force the entire column into the float dtype. Why? Because when we call float() in the try:, we’re doing just that… trying. It’s not doing any permanent changes. If anything went wrong, the astype() function will throw errors letting us know where it ran into problems. This statement is also turning those non-null &#39;nan&#39; entries into proper NaNs we can detect. Finally, we reset the index so that when we use this function on other columns, the index will be sequential and work with our cnt counter.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14907b0f9b0ebb6decd5c68c5485cd10/href">https://medium.com/media/14907b0f9b0ebb6decd5c68c5485cd10/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e7e4c7c3dacf0898eb5a3b25049df5e/href">https://medium.com/media/5e7e4c7c3dacf0898eb5a3b25049df5e/href</a></iframe><p>After running our function, we can check the dtype of our column, and also run char_finder() one last time to make sure no wrong-type values show up. Voila!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*Jxoo1sD6Lc46B4KB_uU_xg.png" /></figure><h3>Removing numbers and booleans from a character column</h3><p>The flip side of the coin can be a little trickier. <em>Technically</em> ‘8’ is a character… <em>technically</em> ‘True’ is a word… but they probably don’t belong in our character column. One sneaky challenge we have here is that although we entered in a boolean True value and 24.5 into our name_col when we created it, upon Panda’s conversion of the column to the object, it actually treats everything like strings ( &#39;True’ and &#39;24.5&#39;) . As a result, like our earlier function, float() is actually testing float(&#39;True&#39;), which results in a ValueError (p.s. float(True) equals 1.0). Shown here, our &#39;True&#39; element isn’t considered an instance of type bool… again, because it thinks it’s a string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*yRiNtEX2dk1xdOcl42p66w.png" /></figure><p>To overcome this, we’ll have to put in extra logic to explicitly test for &#39;True&#39; and &#39;False&#39;, as functions like bool() aren’t as particular as numeric functions like float() and will give a True value for <strong>any</strong> character.</p><p>Again, we’re also taking special care to leave any NaNs in place. To enable this we do a nested int(float()) conversion. The inner float() acts as before, attempting to turn our strings like ‘24.5’ into 24.5. int() will then convert floats like 24.5 to 24, which is fine for this checking algorithm — but NaNs are not integers and will throw a ValueError. This is good! We want NaNs to get sent to the ValueError section, as non-errors (i.e. numbers) are flagged in the bottom else: statement. With that in mind, our number/boolean finder function and results look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f48d5601eaa2222760a7f21a0cc42b2/href">https://medium.com/media/1f48d5601eaa2222760a7f21a0cc42b2/href</a></iframe><p>We’ve identified all of the numeric and boolean values! Before we make the ‘fixer’ function, remember that our NaNs are still as strings in this column. We’ll need to handle that explicitly in our elif statement to manually turn them back into NaNs. We can see this reflected in our output as nan is now represented as &lt;NA&gt;, and df.isnull().sum() is able to detect them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/317c3c3a909b4a97c182cc84f72c05f5/href">https://medium.com/media/317c3c3a909b4a97c182cc84f72c05f5/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc342ec72480c919872bca96e7811c50/href">https://medium.com/media/dc342ec72480c919872bca96e7811c50/href</a></iframe><p>At this point, our two dataframe columns are effectively cleaned of all wrong-type data as shown below. We’ve also converted them into the proper data types, which will allow for all further cleaning, EDA, or machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*8eLHTKcuG2562SLHHaRUUA.png" /></figure><p>Now, we would just need to decide what to do with the NaNs (now that we can detect them!)— if they make up a large number of the rows, we may consider dropping the entire column — but otherwise we typically use df.dropna() to simply eliminate all NaN rows.</p><h3>Extra Credit — speed analysis</h3><p>Usually, for loops are about the last thing we want to do when manipulating dataframes. For loops can be significantly computationally slow. As a thought exercise, these same functions can be implemented using panda’s apply() function, and can then use a more vectorized implementations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2163c92fbcae2f18a30ec57e803d1163/href">https://medium.com/media/2163c92fbcae2f18a30ec57e803d1163/href</a></iframe><p>However, if we inspect the timing (after resetting the df between each, of course), we find that our original solution is pretty fast, even for a dataframe with 42,000 rows. Even more surprising, it performs more than 2X faster than this apply() solution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2da9515d2f70900d0a17c7669d5b1ad8/href">https://medium.com/media/2da9515d2f70900d0a17c7669d5b1ad8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*cIKb4GqlfcJlkbV98Gobaw.png" /></figure><p>While other vectorizing solutions certainly exist, I’ve found a few milliseconds to be very manageable for cleaning each column.</p><h3>Summary</h3><p>All of the code shown here, as well as a functions-only jupyter notebook can be <a href="https://github.com/bamattis/Blog/tree/main/DataCleaning">snagged from my github</a>. I’ve also included char_fixer_nan() and num_fixer_nan() that converts ALL bad data to NaNs, in case that is more useful for your purposes. These functions are also simple to import into any of your notebooks once you save them locally using %run ColumnCleaner.ipynb.</p><p>Hunting and purging wrong-type data isn’t fun, and these functions were born out of necessity. I really hope this helps everyone with speeding up the cleaning process for particularly dirty datasets. <strong><em>Please follow me if you found this useful! </em></strong>Cheers, and happy coding out there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4274c21e299" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-cleaning-automatically-removing-bad-data-c4274c21e299">Data Cleaning: Automatically Removing Bad Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building and Exporting Python Logs in Jupyter Notebooks]]></title>
            <link>https://medium.com/data-science/building-and-exporting-python-logs-in-jupyter-notebooks-87b6d7a86c4?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/87b6d7a86c4</guid>
            <category><![CDATA[log-files]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[logs]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Fri, 10 Dec 2021 07:54:45 GMT</pubDate>
            <atom:updated>2021-12-10T17:43:48.050Z</atom:updated>
            <content:encoded><![CDATA[<h4>A how-to on capturing and exporting log files</h4><figure><img alt="Pile of logs" src="https://cdn-images-1.medium.com/max/1024/0*3ucxC9K1vb2xslwl" /><figcaption>Photo by <a href="https://unsplash.com/@rgrzybowski?utm_source=medium&amp;utm_medium=referral">Radek Grzybowski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Logging is an important capability within Python to monitor the health of our code. In a way, log files share a lot of the same concepts as the debugger. We can use logs to place breadcrumbs in our code to help determine what’s going on inside. This is especially useful when our script crashes for an unknown reason — if we leave enough breadcrumbs behind we can track the path the code took, and we can focus our attention there. Typically the python debugger is useful when we’re first testing out our code, but at some point we consider our code “working” before we’ve tested every possible input case. If we use logging in the background, we’ll have a great idea of where we went wrong if something breaks down the road. <em>In particular, if we use logging correctly, we can see the exact unique situation that broke our “working” code.</em></p><p>Logs become even more critical when others use our code. When a script fails on another user’s machine, it can be challenging to replicate the error — but the logs can guide us down the path the script followed just before it failed on the user’s system. This is exceptionally useful in creating code that manages all of the “edge” cases (why do you think software always asks for permission to send crash data when you install it?!)</p><p>Finally, there’s another awesome use for logs — as trackers of the code development cycle. Imagine that within the first week of release to other users your python script generates 200 errors, 500 warnings, and various other logging flags. As the code is refined and becomes more robust through iteration, we can see the rates of the different issues decrease over time. This gives us an idea of where we are in our development cycle and where to focus our future efforts by tackling the most common errors first. So let’s get started:</p><h3>Python Logging Basics</h3><p>Logging in Python when running scripts from the command line is very well documented, but we need to start with the basics before diving into the tricks and tools we can use. First, when we send messages to the logger, we choose which category the information fits into. These are organized by increasing severity, from NotSet to Critical as shown below. What’s powerful about this hierarchy is that when we output the log data, we can select which <strong>level </strong>to report. In this case, the logger will capture everything at that level and more severe. For example, if we were to chose a level of Warning, then the logger would capture all Warning, Error, and Critical messages. Complete documentation on the standards of when to use each level are found <a href="https://docs.python.org/3/howto/logging.html#logging-basic-tutorial">here</a>.</p><figure><img alt="Logging levels graphic" src="https://cdn-images-1.medium.com/max/731/1*a-qkXz1NaBqXgjksa7hEZA.png" /><figcaption>Photo by the Author — Logging levels</figcaption></figure><p>To build this in Python with the logging messages being posted to the console, it looks like this:</p><pre>import logging</pre><pre>#create a logger<br>logger = logging.getLogger()</pre><pre># to control the reporting level<br>logger.setLevel(logging.WARNING)</pre><pre>#send two messages to the logger<br>logging.info(&quot;This is an INFO message&quot;)<br>logging.warning(&quot;This is a WARNING message&quot;)</pre><p>As the level is set to logging.WARNING we won’t see the info-level message. Setting the level is useful as during first coding we’ll send breadcrumbs in DEBUG and INFO to make sure we’re doing things right. Then, when we think we no longer need the breadcrumbs, we can eliminate them from the log and focus on functionality problems in warnings, errors, and criticals. This can work hand-in-hand with the debugger.</p><p>To output to an actual log file instead of the command line, we need to set up the logger using basicConfig() where we can also set the level, like this:</p><pre>import logging<br>logging.basicConfig(filename=&#39;my_first_log.log&#39;, level=logging.INFO)</pre><p><strong>Unfortunately, this method won’t work when operating in Jupyter Notebooks.</strong> At all. The problem lies in that at start-up, a Jupyter Notebook launches an IPython session that has already initiated logging. As a result, logging.basicConfig() won’t work.</p><h3>Logging in Jupyter Notebooks</h3><p>Logging is also particularly useful in Jupyter Notebooks, as the debugger can have some serious <a href="https://stackoverflow.com/questions/46495269/debuggers-not-acting-properly-on-jupyter-notebooks">functionality issues</a>. These debugger issues are avoided by using the enhanced debugger in JupyterLab— but if you’re constrained to Jupyter Notebooks, then logging gives you the flexibility to do what you need to handle a broader range of debug tasks.</p><h4>Save Log to a File</h4><p>To export log messages to an external log file in Jupyter, we’ll need to set up a FileHandler and attach it to the logger:</p><pre>import logging<br>logger = logging.getLogger()<br>fhandler = logging.FileHandler(filename=&#39;test_log.log&#39;, mode=&#39;a&#39;)<br>logger.addHandler(fhandler)<br>logging.warning(&#39;This is a warning message&#39;)</pre><p>It’s important to note the mode selection in the FileHandler setup:</p><ul><li><strong>‘a’ </strong>— append. All new messages are tacked onto the end of the log file</li><li>‘<strong>w’ </strong>— write. The log file is cleaned out and started fresh each time you run the python script</li></ul><p>If all went well, we should have a new ‘test_log.log’ file in the same directory as our Jupyter Notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*AIoSb6bJewN4IjG-51fM4g.png" /></figure><h4>Formatting Outputs</h4><p>We can also improve and customize our log outputs to contain more than the text we dictate in the logging messages. By using logging.Formatter() we can add in a host of fields to help us understand more about the messages, such as when the message was generated and by at which line in the code. Here are some of my favorites, with the complete list <a href="https://docs.python.org/3/library/logging.html">here</a>:</p><ul><li>%(asctime)s — the date/time of the entry</li><li>%(levelname)s — the logging level (so you don’t have to spell that out in your error)</li><li>%(name)s — name of the logger used (as in, which file generated this message)</li><li>%(lineno)s — the line number that generated the message</li><li>%(message)s — of course! The message itself!</li></ul><p>To attach our formatting to the logger, we add it to the file handler like this:</p><pre>formatter = logging.Formatter(&#39;%(asctime)s - %(name)s - %(lineno)s - %(levelname)s - %(message)s&#39;)<br>fhandler.setFormatter(formatter)</pre><h4>Exporting Formatted Output to the Console</h4><p>We can also add another handler to show this formatting in the console as too. Here, we access the console through a StreamHandler() using sys.stdout and attach it to the logger:</p><pre>import sys<br>consoleHandler = logging.StreamHandler(sys.stdout)<br>consoleHandler.setFormatter(formatter)<br>logger.addHandler(consoleHandler)</pre><p>It gets even better! We can also independently adjust the levels outputted by <strong>each handler</strong>. For example, we can set our console to report down to the debug level like this: consoleHandler.setLevel(logging.DEBUG)</p><p><em>Important note</em> <em>— The main logger level must be </em><strong><em>equal to or lower than</em></strong><em> the level given to the handlers, otherwise it won’t work, as the high-level logger level sets the floor.</em></p><h4>Variable Values</h4><p>We can also pass more than static messages to our logger. Often, if our code is returning warnings or errors, we’ll want to know what user values triggered the event. Luckily, we can simply parse variable values into our messages like common strings using .format().</p><pre>var1 = 1<br>var2 = &#39;cat&#39;<br>logging.debug(&#39;var1: {}; var2: {}&#39;.format(var1, var2))</pre><h4>Putting it all together</h4><p>With what we’ve learned so far, we can initiate a new logger in Jupyter Notebooks, adjust the reporting level, adjust the formatting, output to file, output to the console, and adjust the level of each output. Here’s a simple snapshot of what it looks like in action, with the console at DEBUG and the export file at INFO:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*x84BzsdsgHWEiE2DRtCceQ.png" /><figcaption>Logging in a Jupyter Notebook</figcaption></figure><p>And here’s what we see in that file_out.log file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*J9M7vCG-tfD1YInlduZnFw.png" /><figcaption>Output log file, showing only INFO and above</figcaption></figure><p>And while it’s nice to have the ability to export the log to a file, sometimes we want to be a bit lazy and not switch over to a text editor to read our output file. To see the log file in Jupyter Notebook, we can just read and print the contents like this:</p><pre>with open(&quot;file_out.log&quot;) as log:<br>    print(log.read())</pre><h3>Hints and Tips:</h3><p>Because the logger is automatically set up as part of IPython when we start a notebook, certain configurations conflict with some of the functionality that exists for python files run in the console. One in particular is the use of the getLogger(__name__) functionality. When our python scripts become more complex, we will often break some functions into separate script files that are later imported as separate modules. We can pass all log messages to a common log file, with the %(name)s formatter telling us which python script generated the message. There’s even a tutorial on how to do this in python running from the <a href="https://docs.python.org/3/howto/logging-cookbook.html">console</a>. Unfortunately, this capability is not immediately compatible in Jupyter Notebooks. I haven’t found a solution yet… but for now know that using __name__ will prevent anything from being printed to the log file or the formatted console handler.</p><p>As you get deeper into python logs, you’ll find that we can even export log files though email. This is very advantageous — after our code is deployed we can monitor its health from afar. Even better, it only takes a couple of lines of code to add the SMTPHandler. Example code can be found <a href="https://www.python4networkengineers.com/posts/python-intermediate/log_to_console_file_and_smtp_with_logging_module/">here</a>.</p><p>Also, with the ability to set the level for each handler, we can create multiple log outputs simultaneously that report and different levels. For example, we may want one log file that captures every event down to the debug level, but we may also want one that is filtered to only report warning and above — this gives us a simple file to monitor for major problems, and a separate log file of everything for when we want to dive in deeper.</p><h3>Conclusions</h3><p>Logs offer a great way to maintain the quality and functionality of our code, especially when we release our python scripts for others to use. It can help flag when our code to acts unexpectedly and leaves a trail of breadcrumbs to show how and why things broke. It’s extraordinarily challenging during development to anticipate <strong><em>every</em></strong> use case and <strong><em>every</em></strong> user input that our scripts are going to receive — logging gives us a way to continually iterate and improve the robustness of our code.</p><p>There are a couple of differences when implementing logs in Jupyter Notebooks, but once you understand the modifications you can successfully generate truly useful logs. Thanks for reading, and as always, all of the code shown above can be accessed on my Github <a href="https://github.com/bamattis/Blog/tree/main/Logs">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87b6d7a86c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-and-exporting-python-logs-in-jupyter-notebooks-87b6d7a86c4">Building and Exporting Python Logs in Jupyter Notebooks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science in Semiconductor Process Yield]]></title>
            <link>https://medium.com/data-science/data-science-in-semiconductor-process-yield-a6da6a37253c?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6da6a37253c</guid>
            <category><![CDATA[semiconductors]]></category>
            <category><![CDATA[modelling]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cleanroom]]></category>
            <category><![CDATA[yield]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Sun, 17 Oct 2021 04:56:51 GMT</pubDate>
            <atom:updated>2021-10-19T03:53:08.755Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Machine Learning to improve fab throughput and profitability</h4><figure><img alt="Silicon chips on a wafer" src="https://cdn-images-1.medium.com/max/1024/0*ewOnOUWY4CMo48PS" /><figcaption>Photo by <a href="https://unsplash.com/@viazavier?utm_source=medium&amp;utm_medium=referral">Laura Ockel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this stage of the pandemic, demand for all kinds of semiconductor chips has exploded and semiconductor fabs are <a href="https://www.cnbc.com/2021/10/15/chip-shortage-could-persist-for-another-2-3-years-hisense-president.html">struggling to keep up</a>. Traditional IC customers are now competing with the demands of new compute applications (such as AI and autonomous driving) to get their custom-design wafers fabricated. Unfortunately, increasing the wafer capacity of semiconductor fabs to react with new waves of demand is very expensive and challenging. Once a decision is made to increase a fab’s capacity, it can take months to purchase new tools, install them in the fab, and qualify them for production.</p><p>In cases like these, a better option is to improve the the yield — that is, the ratio of “good” die to total die per wafer — to get more chips out of existing capacity. Increasing the yield also directly increases fab <a href="http://web.pdx.edu/~webercm/documents/2004%20Weber%20Yield%20Learning.pdf">profit margins</a>, with even small enhancements in yield driving massive profits — so, let’s just increase the die yield! Easier said than done.</p><p>Before the rise of data science in the semiconductor industry, device engineers would methodically hunt through available data searching for any clues or correlations to the yield. As humans, we like problems where a result can be attributed to a single factor. We’d love to have a strong correlation with yield and one specific process parameter — be that a film deposition rate, an etch rate, a lithography dimension tolerance, or any number of 1000s of process parameters. Unfortunately, this is almost never the case. The processes used to make modern semiconductor chips compound on one-another constantly.</p><p>New breed of device and yield engineer is needed to work with ever increasing process complexity and volumes of metrology data— a deep understanding of device physics and fab processes must be combined with a data science toolbelt to find the true yield triggers.</p><h3>Variables Overlapping</h3><p>To help understand the problem, consider an Aluminum routing layer. In this simple process we have an Aluminum deposition step, lithography for patterning, a plasma etch to define the features, followed by cleans. If we were to have a simple yield failure, such as neighboring lines being electrically shorting, the two most likely candidates would be that the Aluminum was deposited thicker than expected (preventing the etch from fully separating the lines), or the rate of the etch was slower than expected. However, in most process controlled fabs, the most often answer is that both processes were within the bounds of the statistical process control (SPC) system, and are therefore operating within the acceptable range. So what gives? In this case, the confluence of the deposition being a <em>little</em> thicker than normal and the etch being a <em>little</em> slower than normal can be a yield killer. Extracting which <strong>combination</strong> of variables contribute to failure is a problem perfectly set up for machine learning. We’re actually pretty lucky if our problem is as simple as this Aluminum routing case — it usually gets much worse.</p><figure><img alt="Silicon wafers loading into a processing tool" src="https://cdn-images-1.medium.com/max/1024/0*oZtiUq8aiWr3ozj1" /><figcaption>Photo by <a href="https://unsplash.com/@maxence_pira?utm_source=medium&amp;utm_medium=referral">Maxence Pira</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Consider a new IC design running on an advanced-node (16nm or less) process. Physical failures like electrical shorts or electrical opens are simple. However, devices can fail to yield because the circuit timing path prevents the chip from performing it’s desired function. Maybe the turn-on voltage of the transistors is off target? Maybe the junction capacitance is high, increasing the RC product and introducing delay? In a semiconductor process flow containing hundreds of process steps, how would we go about finding the root cause?</p><h3>Plan of attack</h3><p>The benefit of a data science approach is the ability to evaluate the impacts of many variables simultaneously. In this case, we can incorporate four main types of variables:</p><ul><li><strong><em>Inline metrology</em></strong>: physical measurements made upon the wafers as they processes, including things like film thicknesses and densities, lithographic size accuracy and alignment, contaminant particle defects, and many others.</li><li><strong><em>End-of-line electrical test</em></strong>: Data from the discrete scribeline devices (devices printed in the streets between die) tested at end-of-line, prior to dicing. The individual devices in the scribe are designed to tease out certain process dependencies and offer unique process insights (like <a href="https://www.researchgate.net/publication/228608938_Cross-bridge_Kelvin_resistor_CBKR_structures_for_measurement_of_low_contact_resistances">Kelvin contact resistors</a>).</li><li><strong><em>Tool process qualification runs</em></strong>: Tools regularly run a set process (identical to or very similar to the production process) on test wafers that can be directly measured. The results such as deposition rates, etch rates, or non-uniformity are tracked in a statistical process control (SPC) system and have associated date stamps.</li><li><strong><em>Product circuit performance</em></strong>: Depending on the chip and types of test equipment needed to interrogate it, the product die are tested at end-of-line (before dicing), after dicing, or post-packaging. Regardless, this performance or yield is what we’re trying to optimize as our dependent variable. This can either be on a pass/fail criteria (<em>classification</em>), performance binning (<em>also classification</em>), or raw performance (<em>regression</em>).</li></ul><p>Using all four variable types together gives the engineer significant capability in taking the outcome (measured performance) and drilling down to the physical root cause, empowering engineers to directly impact yield through process tool adjustments.</p><h3>Excursion versus Optimization</h3><p>Where to go next really depends on what our main goals are. Did a given wafer or lot dramatically decrease in die yield? In that case, we know that whatever happened, happened on a wafer level. As we play detective, this drastically simplifies our data preparation process. Here, we’re looking for wafer-level shifts in variables, meaning we can probably average values from all four variables types by wafer and still find the smoking gun. A simple check using functions like varImp() in <a href="https://www.rdocumentation.org/packages/caret/versions/6.0-90/topics/varImp">Caret</a> (R) or feature_importances_in <a href="https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html">Scikit-learn</a> (Python) can quickly give insights to which variables have direct impacts on yield.</p><p>Instead, if we have a fairly stable process, but on each wafer only 70% of the die are functional (or high enough performance to be sell-able), we have an opportunity to optimize. At a high level, are the failed die randomly spread across the wafer, or are they all grouped in one physical region of the wafer? This alone can tell us if the failures are defect-based (such as a particle landing on the wafer mid-process), or process non-uniformity based, allowing us to filter many of the potential variables.</p><p>It’s important to note that wafer-level processes tend to have unique non-uniformity profiles. Many will be radial, but gradient profiles are common, and even sombrero-shaped profiles can occur. How these tool process profiles match up with our yield is a fantastic tool. But the human eye can only pick out so much if multiple processes overlay, and again it’s data science to the rescue.</p><figure><img alt="Wafer contour plots showing within-wafer variation profiles" src="https://cdn-images-1.medium.com/max/1024/1*_4Ohs1g_x_GfNWfM1pxtIQ.png" /><figcaption>Photo by the author — Common wafer-level process profiles</figcaption></figure><p>Before we get there, however, we have more data preparation to do for wafer-level yield engineering. While we may have end-of-line scribeline test data and metrology data that associate with each product die, finding a similar association with tool-qualification (a.k.a. “qual”) data is tricky, as those are often measured on unpatterned wafers with evenly distributed 9, 13, or 39-point maps with radial coordinates. For our data we’ll need to map each die to the physically closest measurement, which usually includes the following steps:</p><ol><li>In our qual data we’ll translate (radius, theta) coordinates to (x,y) Cartesian coordinates and ensure the data collection date is in a proper datetime format.</li><li>Populate our product data set with the processing date/time that the lot ran through <em>each process</em> that has associated qual data.</li><li>For a given lot/wafer we’ll iterate over our qual data to filter down to the run with the closest possible date, and find the measurement <a href="https://www.w3resource.com/python-exercises/python-basic-exercise-40.php">nearest</a> to the product die coordinates.</li></ol><p>Once our data is combined in this fashion, we have a very content-rich variable data set, and we can automate the process of looking for simple 1:1 correlations between any variable and our dependent variable (product die performance). However, to really get down to the interlinking of our variables we’ll need to dive into creating machine learning models.</p><h3>Modelling</h3><p>In this step, the goal isn’t to create the highest accuracy model — the goal is to create a model that does a reasonable job of predicting yield but is also <strong>easily interpreted</strong> by the yield engineer. Here we’ll want to stick to algorithms like Decision trees, Linear/logistic regression, Lasso regression, or Naive Bayes. Once we find an appropriate algorithm, we can dive into the <a href="https://mljar.com/blog/visualize-decision-tree/">visualizations</a> to learn how our model “thinks” and drive actions in the fab to improve yield.</p><h3>Predictive Yield</h3><p>The next big leap comes from the predictive power of data science. What if we could to predict wafer yield before it even finishes processing? This kind of information could drive stakeholders to make important lot decisions such as:</p><ul><li>Triggering a rework process (recovering and repeating a certain step)</li><li>Provide a milestone report to customers that a lot is in good health</li><li>Scrapping a lot to cut losses and prevent future tool time on a “dead” lot</li><li>Starting a “make-up” lot to replace a low-yielding lot to minimize the delivery delay to the customer.</li></ul><p>Unlike in the yield improvement steps, here we’re looking to have the most accurate model possible. Interpretability is a distant “nice-to have”, so we are able to explore a much broader range of models. Once we have a model selected, we can also use tools like Variable Importance and Recursive Feature Elimination to see what data is really driving our yield detectability. With that data in hand we can go as far as eliminating some existing metrology data collection to further reduce costs and process cycle times.</p><p>With a good enough model, we can also do virtual process windowing — we can alter the acceptable SPC range for a process to see how broadening or tightening control limits can impact yield. Done iteratively, the control limits for every process in the fab can be directly tied to wafer yield. This impacts how preventative maintenance is performed, how new tools are introduced to the line, and even the frequency of qual procedures — all of which impact fab costs.</p><p>So there we have it! Using the power of data science we can reshape how we investigate yield excursions, improve overall line yield, and predict wafer yield before it even finishes processing. In the hands of semiconductor device engineers who understand the physics of fabrication, these predictive modelling capabilities have dramatic impacts on wafer process yield and fab profits. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6da6a37253c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-science-in-semiconductor-process-yield-a6da6a37253c">Data Science in Semiconductor Process Yield</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Forecasting in R with Holt-Winters]]></title>
            <link>https://medium.com/data-science/time-series-forecasting-in-r-with-holt-winters-16ef9ebdb6c0?source=rss-47e773cabf10------2</link>
            <guid isPermaLink="false">https://medium.com/p/16ef9ebdb6c0</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[holt-winters]]></category>
            <dc:creator><![CDATA[Brian Mattis]]></dc:creator>
            <pubDate>Fri, 23 Jul 2021 13:39:08 GMT</pubDate>
            <atom:updated>2022-11-15T04:26:53.410Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>A template for seasonal forecasting</h4><figure><img alt="leaves on a string of various colors" src="https://cdn-images-1.medium.com/max/1024/0*GtKrja-iPlW6Yact" /><figcaption>Photo by <a href="https://unsplash.com/@chrislawton?utm_source=medium&amp;utm_medium=referral">Chris Lawton</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Time series forecasting is a great way to predict future events when we only have historical data to guide us. We can use it to predict things like the weather, stock markets, or even <a href="https://www.kaggle.com/ronaldocr9803/icecream-consumption-usa">ice cream consumption</a>. In this article I’ll guide you though time series setup, creating fits to the data, predicting the future, and model evaluation using the ubiquitous Holt-Winters forecasting.</p><p>For this example, we’ll be using a dataset that tracks candy production in the US by month, available on <a href="https://www.kaggle.com/rtatman/us-candy-production-by-month">kaggle</a>. With Halloween and Christmas towards the end of the year, followed closely by those New Year’s resolutions, it’s not surprising that candy consumption and production can have very strong seasonal components. However, the beauty of the time series forecasting shown here is that it can isolate seasonal components to reveal year-over-year trends that could indicate metrics like annual growth rates.</p><h4>Data Setup</h4><p>First, let’s import this data set and take a quick look at what we have:</p><pre>df &lt;- read.csv(&#39;Data/candy_production.csv&#39;, header=TRUE)<br>head(df)</pre><p>Here we see two columns — we have a “date-like” observation_date with values like 1972-01-01 , and a second column with the candy production value. Simple enough. As we don’t actually have a date-type column, it’s never a bad idea to add one (though not required for performing time series!!). If we ever want to filter or subset data, it certainly does make life a lot easier. To do this, we look at the date values to extract the format it’s in. In this case it’s year-month-day (YYYY-MM-DD). Here’s a quick cheat sheet for the conversion so you don’t have to scour the internet:</p><pre>#Symbol  Meaning                Example<br># %d      day as a number (0-31)  01-31<br># %a      abbreviated weekday     Mon<br># %A      abbreviated weekday     Monday<br># %m      month (00-12)          00-12<br># %b      abbreviated month       Jan<br># %B      unabbreviated month     January<br># %y      2-digit year            07<br># %Y      4-digit year            2007</pre><p>So for us, let’s create a new column that is of type “Date” like this:</p><pre>df$Date &lt;- as.Date(df$observation_date, &quot;%Y-%m-%d&quot;)</pre><p>Should we get into the case where our time series uses hours/minutes/seconds or time zones, we’d need to convert into a <a href="https://www.rdocumentation.org/packages/tis/versions/1.38/topics/POSIXct">POSIXct</a>, but that’s not needed here.</p><h4>Converting to a time series</h4><p>This is a very simple step, but also one of the easiest to forget. Before we do anything regarding forecasting, we need to tell R that this data is a time series. To do this, we make a time series object. There are two critical inputs we must give the function — <strong>frequency </strong>and <strong>start</strong>.</p><p>First, we should accept that the function that creates a time series object isn’t smart. Despite our data set having very readable date-like values, we’ll need to manually tell it the <strong>frequency</strong> of our readings — that is, how many rows of data you have in a year. In this case, we have one data point per month, so frequency=12. For <strong>start</strong>, if we’re not interested in attaching specific dates with our results, then we can just enter “1”, but since we usually do we’ll indicate the year and month of the first data point as start=c(1972,1).</p><pre>dfts &lt;- ts(df$IPG3113N, frequency=12, start=c(1972,1))<br>dfts</pre><p>Inspecting dfts shows how the data is now stacked in years and months.</p><h4>Decomposing</h4><p>Immediately we can “decompose” the time series — which in this case means separating out the 3 main components that make up the time series:</p><ul><li>trend: the long-term trends in the data</li><li>seasonal: the repeated seasonal signal adder</li><li>random: the “left-over” components that aren’t expected from the seasonality or trend components.</li></ul><p>We can easily extract these components and plot them with:</p><pre>components_dfts &lt;- decompose(dfts)<br>plot(components_dfts)</pre><figure><img alt="time series decomposition" src="https://cdn-images-1.medium.com/max/682/1*7LQZtV728NE3KXmWbOtv7Q.png" /></figure><p>As we look at the decomposition components, we can visually see how they can add up to our “observed” value (our real values). It’s also very important to inspect the scales of each section to see which one is more dominant. For example, if “random” has a range significantly larger than seasonal or trend, this data is going to be very challenging to accurately forecast later on.</p><h4>Fitting with Holt-Winters</h4><p>Before we predict values in the future, we’ll need to create a fit to the data. In the most basic method, we can simply call the Holt-Winters function and let R figure out the tuning parameters on it’s own. We also have the opportunity to tune the fit manually by setting tuning variables:</p><ul><li>alpha: the “base value”. Higher alpha puts more weight on the most recent observations.</li><li>beta: the “trend value”. Higher beta means the trend slope is more dependent on recent trend slopes.</li><li>gamma: the “seasonal component”. Higher gamma puts more weighting on the most recent seasonal cycles.</li></ul><p>Here we can make two fits, and plot them versus the raw data to see the quality of the fits. I highly recommend playing with the alpha, beta, and gamma values to see how the fit changes.</p><pre>HW1 &lt;- HoltWinters(dfts)</pre><pre># Custom HoltWinters fitting<br>HW2 &lt;- HoltWinters(dfts, alpha=0.2, beta=0.1, gamma=0.1)</pre><pre>#Visually evaluate the fits<br>plot(dfts, ylab=&quot;candy production&quot;, xlim=c(2013,2018))<br>lines(HW1$fitted[,1], lty=2, col=&quot;blue&quot;)<br>lines(HW2$fitted[,1], lty=2, col=&quot;red&quot;)</pre><figure><img alt="HoltWinters auto and manual fits" src="https://cdn-images-1.medium.com/max/1024/1*jgoAivyV_LanUgA7vG_2aw.png" /><figcaption>HoltWinters fits to our data</figcaption></figure><h4>Predictions</h4><p>Both fits look to follow our data quite well, so now it’s time to see how they do predicting future candy production. Using the predict function we’ll need to specify how many data points we’ll predict into the future. Here, we’ll use a value of 24 to project 2 years into the future (remember, this is a <em>monthly</em> time series). We’d also like some concept of the “error bars” associated with the prediction to give us an idea of our prediction confidence. To do this, we set prediction.interval=TRUE and level to the <a href="https://www.simplypsychology.org/confidence-interval.html">confidence interval</a> (0.95 selected here). Once again, we’ll plot showing the tail of our existing data and the new predictions:</p><pre>HW1.pred &lt;- predict(HW1, 24, prediction.interval = TRUE, level=0.95)</pre><pre>#Visually evaluate the prediction<br>plot(dfts, ylab=&quot;candy production&quot;, xlim=c(2008.5,2020))<br>lines(HW1$fitted[,1], lty=2, col=&quot;blue&quot;)<br>lines(HW1.pred[,1], col=&quot;red&quot;)<br>lines(HW1.pred[,2], lty=2, col=&quot;orange&quot;)<br>lines(HW1.pred[,3], lty=2, col=&quot;orange&quot;)</pre><figure><img alt="HoltWinters prediction" src="https://cdn-images-1.medium.com/max/1024/1*p5rp76R1XCXLJNEyq2StIA.png" /><figcaption>Predictions for our HW1 seasonally-additive fit</figcaption></figure><h4>Seasonality Tuning</h4><p>When we make fits, we also have the option to tune the behavior of the seasonality component. The standard Holt-Winters fits use an <strong>additive</strong> seasonality — where it assumes that the amplitude of any seasonality components are relatively constant throughout the series. However, if we use <strong>multiplicative</strong> seasonality, we allow the seasonal variations (amplitude) to grow with the overall level of the data. To see how that works in our candy production case, we’ll create a new fit, predict into the future, and compare to our <strong>additive</strong> fit of HW1.</p><pre>HW3 &lt;- HoltWinters(dfts, seasonal = &quot;multiplicative&quot;)<br>HW3.pred &lt;- predict(HW3, 24, prediction.interval = TRUE, level=0.95)<br>plot(dfts, ylab=&quot;candy production&quot;, xlim=c(2008.5,2020))<br>lines(HW3$fitted[,1], lty=2, col=&quot;blue&quot;)<br>lines(HW3.pred[,1], col=&quot;red&quot;)<br>lines(HW3.pred[,2], lty=2, col=&quot;orange&quot;)<br>lines(HW3.pred[,3], lty=2, col=&quot;orange&quot;)</pre><figure><img alt="Multiplicative forecast — wide confidence bands" src="https://cdn-images-1.medium.com/max/892/1*ahXHPQKsNAi1ngBZYN8S5g.png" /><figcaption>Predictions for our HW3 seasonality-multiplicative fit</figcaption></figure><p>As we can see, the prediction looks pretty similar to our HW1 result, but the confidence intervals spread wildly outward. For this data set, <strong>multiplicative</strong> fitting doesn’t appear to be the way to go.</p><h3>The forecast library</h3><p>Those new to time series forecasting may find it’s common to see examples across the web that use forecast instead of predict to model future values. So what’s the difference? Put simply, forecast is a wrapper for predict that allows for more confidence intervals, makes plotting easier, and gives us tools to evaluate the quality of our predictions. Using our HW1 Holt-Winters fit from before, we can use forecast to make new predictions and include both 80% and 95% confidence intervals. Here’s how we do it:</p><pre>library(forecast)<br>HW1_for &lt;- forecast(HW1, h=24, level=c(80,95))</pre><pre>#visualize our predictions:<br>plot(HW1_for, xlim=c(2008.5, 2020))<br>lines(HW1_for$fitted, lty=2, col=&quot;purple&quot;)</pre><figure><img alt="HoltWinters using the forecast package" src="https://cdn-images-1.medium.com/max/1024/1*As6ZagOt5Tb26RIEcIM2aA.png" /></figure><h4>Forecast Evaluation</h4><p>forecast also calculates the quality of our predictions by compiling the observed values minus the predicted values for each data point. These are added to our forecast model as $residuals . To best evaluate the smoothing functions we used in our model, we want to check that there are no correlations between forecast errors. Simply put, if neighboring points in our fits continually miss the observed values in a similar fashion, our main fit line is not reactive enough to the changes in the data. To capture this, we use the acf function to evaluate the correlation of fit residuals between points of various temporal separations in the time series (lag). Ideally, for non-zero lag, the ACF bars are within the blue range bars shown below. It’s important to use na.action=na.pass because the last value of $residuals is always NA, and the function will otherwise error-out.</p><p>A Ljung-Box test can also indicate the presence of these correlations. As long as we score a p-value &gt; 0.05, there is a 95% chance the residuals are independent. Finally, it’s useful to check the histogram of the residuals to ensure a normal distribution. If the residuals are heavily skewed, then our model may be consistently overshooting in one direction.</p><pre>acf(HW1_for$residuals, lag.max=20, na.action=na.pass)<br>Box.test(HW1_for$residuals, lag=20, type=&quot;Ljung-Box&quot;)<br>hist(HW1_for$residuals)</pre><figure><img alt="Residual evaluation" src="https://cdn-images-1.medium.com/max/1024/1*jVuYOOMN1A0ueSgEKdMNyA.png" /><figcaption>Evaluating $residuals</figcaption></figure><h4>Conclusions</h4><p>As you can see, Holt-Winters forecasting is a powerful tool for predicting future data in a time series. Many other time series forecasting models exist (ARIMA, TBATS, etc), but almost all follow this same basic code sequence.</p><p>You may have also noticed that the code embedded in this article makes a great template for forecasting just about any data — just keep an eye on frequency and start when converting into a time series, and you should be all set! To save you some time and hassle, the source code can be found on my github:</p><p><a href="https://github.com/bamattis/Blog/tree/main/R/TimeSeries">Blog/R/TimeSeries at main · bamattis/Blog</a></p><p><strong><em>Please follow me if you found this useful! </em></strong>Cheers, and happy coding out there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16ef9ebdb6c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/time-series-forecasting-in-r-with-holt-winters-16ef9ebdb6c0">Time Series Forecasting in R with Holt-Winters</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>