<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis Casanave on Medium]]></title>
        <description><![CDATA[Stories by Louis Casanave on Medium]]></description>
        <link>https://medium.com/@ls.casanave?source=rss-a25bdaa6a5ad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fZzUFwDxw4ytP9Numz0LdA.jpeg</url>
            <title>Stories by Louis Casanave on Medium</title>
            <link>https://medium.com/@ls.casanave?source=rss-a25bdaa6a5ad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:38:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ls.casanave/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Learned to Stop Worrying and Love Data]]></title>
            <link>https://medium.com/codex/how-i-learned-to-stop-worrying-and-love-data-a1a95388d116?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1a95388d116</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[code-newbie]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[learning-zone]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Sat, 26 Oct 2024 08:51:40 GMT</pubDate>
            <atom:updated>2024-10-27T03:17:12.114Z</atom:updated>
            <content:encoded><![CDATA[<h4>My Best Advice to Those Looking to Break Into the World of Data</h4><p>My story is not uncommon, although the details vary quite a bit from person to person. I am certainly not alone in having new health problems after surviving long-haul COVID, nor am I alone in needing a full-remote job, or needing a profession that paid me a living wage, promptly after the worst of the pandemic. <a href="https://medium.com/pythons-gurus/why-everybody-tried-to-work-in-tech-during-and-after-covid-7524befb73e9">I am indeed, one of a cohort</a>. We decided to learn a code language to survive in a changing world. Older technologists may say we “flooded the market” and those who turned to academia for learning web development, data engineering, cybersecurity, or another discipline within technology, may look down on those who are self-taught or code bootcamp graduates. None of these trends surprised me when I entered the tech job market, around the same time Elon Musk bought Twitter. (Not a forgiving time for a budding data enthusiast.)</p><p>What did surprise me, however, is the number of my peers who took on this huge undertaking, and seemed determined to chip away at their goals in misery. I can’t tell you how many folks I met at code boot camp who seemed resigned to their fate, memorizing the minutia of machine learning algorithms with all the enthusiasm of someone getting a flu shot over and over. As if the struggle to learn logic and syntax was designed to be unpleasantly consumed; bore with a strict upper lip.</p><p>I took a different tactic, challenging myself from a place of joy. When I leaned into playful curiosity, I learned more from a qualitative perspective, and my understanding was of higher quality. I also protected my mental health from imposter syndrome while learning rigorously, which can be a serious deterrent for new technologists.</p><p>To learn Python and SQL quickly and immersively, leaving my comfort zone behind was necessary. <a href="https://medium.com/middle-pause/between-the-comfort-zone-and-the-panic-zone-is-the-just-right-learning-zone-2d263200cfaf">Research shows that our learning zones are separate from our comfort zones.</a> In the immortal words of Jake the Dog from Adventure Time, “sucking at something is the first step to being sorta good at something.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*TpPWFvzkafXorSA56LkdZA.jpeg" /><figcaption>Vkjzh45ls325dkf321vgl, CC BY-SA 4.0 &lt;<a href="https://creativecommons.org/licenses/by-sa/4.0">https://creativecommons.org/licenses/by-sa/4.0</a>&gt;, via Wikimedia Commons</figcaption></figure><p>I agree with these learning models and accept them as truth. I had to step away from the comfort of my expertise in other skills to build a new code-specific knowledge base. It wasn’t comfortable. It was challenging, intensive, and had a steep learning curve. However, it is a common misconception that to embrace one’s learning zone, one must therefore be unhappy in one’s discomfort.</p><p>When can someone be happy and outside of their comfort zone? It is in play, experimentation, and joy when the motivation to continue learning and the mechanisms of immersive learning are aligned. I needed to go back to kindergarten and remember that in learning anything I must give myself time and space to play with the information. I began to revel in my process and care about the projects I was working on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*phlJ6WBeTGotydJL" /><figcaption>Photo by <a href="https://unsplash.com/@lastnameeaster?utm_source=medium&amp;utm_medium=referral">La-Rel Easter</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Indeed, when I was interviewing with a major American bank through their neurodiversity hiring program, I learned from the hiring manager conducting my interviews that I was selected for my passion for teaching code. Not my coding skills — but moreover, my drive to improve my coding skills illustrated my employability.</p><p>In my Data Science Code Bootcamp at Flatiron School, I remember many of my peers tried to invent portfolio projects that would attract financial institutions. One peer thought churn analysis was a must-have, to demonstrate their worth as an analyst, and another thought that an algorithm to try and predict stock prices would ensure the interest of fintech companies. When I was onboarded as an employee at the bank, none of the projects in my portfolio had anything to do with making more money. I was much more interested in how code could help protect people in crisis after natural disasters, or how a data visualization could democratize NYPD Stop and Frisk data and make a more just world.</p><p>When I spoke about my projects in my interview, I remember my desperation for a new job, like an unkind alarm blaring in my imagination, went almost silent in the midst of my data storytelling. It’s very difficult to be misinterpreted as disinterested or unengaged when geeking-out about a special interest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jo3IXOnXQ6uHz4rg" /><figcaption>Photo by <a href="https://unsplash.com/@mstyazhkin?utm_source=medium&amp;utm_medium=referral">Maks Styazhkin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In my technical interviews, joy allowed me to thrive and show my skills. It was not dedication, perseverance, or even technical know-how that proved my worth. With joy, I easily recalled the syntax for merging two data sets during a technical interview requiring a left join, the memory accessible especially because I remembered how unstoppable I felt when I finally understood the difference between a left and inner join. With joy, I took questions like trivia and was uninhibited, curious and confident when approaching a puzzle. With joy, I could gamify an interview’s structure in my imagination, turning a code challenge in front of a hiring manager into a collaborative effort. After all, how better to show my value as a team member then by teaming up with my interviewer?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uzkpk4POVbeBNmfW" /><figcaption>Photo by <a href="https://unsplash.com/@joshua_hoehne?utm_source=medium&amp;utm_medium=referral">Joshua Hoehne</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I will admit, that my interviews remained nerve-wracking and left me emotionally drained. However, I was able to make the switch in my head, and make interviews more akin to a Dungeon and Dragon’s session zero, when the playing group (aka, the party,) decides what kinds of boundaries they’re interested in, and if the proposed story is something of interest to me as a player.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5Y_YDVmr0iQe0uPd" /><figcaption>Photo by <a href="https://unsplash.com/@ianfajardo?utm_source=medium&amp;utm_medium=referral">Ian Fajardo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Knowing if I should ask for a different salary figure, or let the hiring manager know if the job opportunity was not for me, became far less challenging. It became second nature to speak with hiring managers not as antagonists to be defeated, but as team members, with whom, for however long, I could find a mutual “win”.</p><p>Therefore it is my best advice to all who are attempting to learn to code, and to myself as I continue to hone and sharpen my technical skills, to embrace code from a perspective of joy and playfulness. It is in play that we can make the most of our learning, and give ourselves our best, most wholistic chance to be technologists. It is easy to have imposter syndrome and to expect failure, but there is no substitute for being passionate about your work. When I approached data from joy, my career change into data became inevitable. I stopped worrying about if I could be a data engineer. The important thing was to learn to love data, and in loving data, the rest followed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1a95388d116" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/how-i-learned-to-stop-worrying-and-love-data-a1a95388d116">How I Learned to Stop Worrying and Love Data</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pumpkin Spice Time Series Analysis]]></title>
            <link>https://medium.com/data-science/pumpkin-spice-time-series-analysis-afd8daaedce1?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/afd8daaedce1</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[seasonality]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Tue, 24 Oct 2023 17:01:09 GMT</pubDate>
            <atom:updated>2023-10-24T17:01:09.308Z</atom:updated>
            <content:encoded><![CDATA[<h4>Throw on your comfiest lo-fi, grab an oversized sweater, your favorite hot beverage, and let’s python.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7xCvTtQpbNZd9ZjC" /><figcaption>Photo by <a href="https://unsplash.com/@nate_dumlao?utm_source=medium&amp;utm_medium=referral">Nathan Dumlao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It’s that time again in the northern hemisphere — a time for apples, pumpkins, and various configurations of cinnamon, nutmeg, ginger, allspice, and cloves. And as the grocery isles start getting ready for Halloween, Thanksgiving, and the winter holidays, it’s a great time to dust off my statistical modeling skills. Hold onto your seasoned lattes, and let’s do some function-oriented seasonal modeling. <a href="https://github.com/casanave/Pumpkin_Spice/tree/main">The full code notebook can be found here.</a></p><h4>Hypothesis:</h4><p>Pumpkin Spice’s popularity as a Google searched term in the USA will have strong seasonality since it’s associated with American Fall Holidays and seasonal food dishes.</p><h4>Null hypothesis:</h4><p>Using last week’s or last year’s data will be more predictive of this week’s level of popularity for the search term “pumpkin spice.”</p><h4>Data:</h4><p><a href="https://github.com/casanave/Pumpkin_Spice/blob/main/pumpkin_spice_5_years.csv">The last 5 years of data from Google Trends, pulled on the 7th of October, 2023. </a>[1]</p><h3>Iterative Modeling Method:</h3><ul><li>Make a naive model where last week’s/last year’s data is this week’s prediction. Specifically, it’s not enough for my final model to be accurate or inaccurate in a void. My final model must outperform using historical data as a direct prediction.</li><li>The train test split will give me two sets of data, one for the algorithm to learn from. The other is for me to test how well my algorithm performed.</li><li>Seasonal decomposition will give me a rough idea of how predictable my data is by trying to separate the yearly overall trend from the seasonal patterns and the noise. A smaller scale of noise will imply that more of the data can be captured in an algorithm.</li><li>A series of statistical tests to determine if the data is stationary. If the data is not stationary, I’ll need to take a first difference (run a time-delta function where each time interval’s data only shows the difference from the previous time interval’s data. This will force the data to become stationary.)</li><li>Make some SARIMA models, using inferences from autocorrelation plots for the moving average term, and inferences from partial auto-correlation plots for the autoregressive term. SARIMA is a go-to for time series modeling and I’ll be trying ACF and PACF inferencing before I try a brute-force approach with Auto Arima.</li><li>Try using Auto Arima, which will iterate through many terms and select the best combination of terms. I want to experiment to learn if the parameters it gives me for a SARIMA model yield a better-performing model.</li><li>Try ETS models, using inference from the seasonal decomposition as to whether x is additive or multiplicative over time. ETS models focus more heavily on seasonality and overall trend than SARIMA family models do, and may give me an edge when capturing the relationship pumpkin spice has to time.</li></ul><h4>Performance plotting KPIs:</h4><ul><li>Try using the MAPE score because it&#39;s an industry standard in many workplaces, and folks may be used to it. It’s easy to understand.</li><li><a href="https://towardsdatascience.com/forecast-kpi-rmse-mae-mape-bias-cdc5703d242d">Try using the RMSE score because it’s more useful.</a></li><li>Plot predictions against the test data and visually check for performance.</li></ul><h3>Eyeballing the data:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R2j1MpKydPGWuFooMGEhvQ.png" /><figcaption>Image by the author.</figcaption></figure><p>As we can see from the above plot, this data shows strong potential for seasonal modeling. There’s a clear spike in the second half of each year, with a taper and another spike before a drop down into our baseline.</p><p>However, each year’s primary spike is larger each year besides 2021, which makes sense, given the pandemic, when folks may not have had celebrating the season on their minds.</p><h3>Imports:</h3><p>Note: These imports appear differently in the notebook itself, as in the notebook I’m relying on seasonal_mod.py which has a lot of my imports baked in.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7d2ae8a828d7771f3dc55cabc07da24/href">https://medium.com/media/e7d2ae8a828d7771f3dc55cabc07da24/href</a></iframe><p>These are the libraries I used to make the code notebook. I went for statsmodels instead of scikit-learn for their time series packages, I like statsmodels better for most linear regression problems.</p><h3>Function-Based Approach to the Code:</h3><p>I don’t know about you but I don’t want to write several lines of code each time I make a new model and then more code to verify. So instead I made some functions to keep my code DRY and prevent myself from making errors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc0586ceede3fd1a1e3222a8e54aed60/href">https://medium.com/media/fc0586ceede3fd1a1e3222a8e54aed60/href</a></iframe><p>These three little functions work together so I only need to run metrics_graph()with y_true and y_preds as the input and it will give me a blue line of true data and a red line of predictive data, along with the MAPE and RMSE. That will save me time and hassle.</p><h4>Using Last Year’s Data as a Benchmark for Success:</h4><p>My experience in retail management informed my decision to try last week’s data and last year’s data as a direct prediction for this year’s data. Often in retail, we used last season’s (1 unit of time ago’s) data as a direct prediction, to ensure inventory during Black Friday for example. Last week’s data didn’t perform as well as last year’s data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YLjW9YW6EMwSQAq7KtnV8w.png" /><figcaption>Image by the author.</figcaption></figure><p>Last week’s data to predict this week’s data showed a MAPE score of just over 18, with a RMSE of about 11. By comparison, last year’s data as a direct prediction to this year’s data showed a MAPE score of just about 12 with a RMSE of about 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BAzlnkwmKJT4bmmkWsSgXQ.png" /><figcaption>Image by the author.</figcaption></figure><p>Therefore I chose to compare all statistical models I built to a naive model using last year’s data. This model got the timing of the spikes and decreases more accurately than our naive weekly model, however, I still thought I could do better. The next step in modeling was doing a seasonal decomposition.</p><p>The following function helped me run my season decomposition and I’ll be keeping it as reusable code for all future modeling moving forward.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96ccd4c8e2e0b8c2c8c734f68e770d55/href">https://medium.com/media/96ccd4c8e2e0b8c2c8c734f68e770d55/href</a></iframe><p>The below shows how I used that seasonal decomposition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ZUnb84qCMWMUfi3_zlCHQ.png" /><figcaption>Image by the author.</figcaption></figure><p>The additive model had a reoccurring yearly pattern in the residuals, evidence that an additive model wasn’t able to completely decompose all the recurring patterns. It was a good reason to try a multiplicative model for the yearly spikes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKlMRBqSGbSESmOxvv2dcQ.png" /><figcaption>Image by the author.</figcaption></figure><p>Now the residuals in the multiplicative decomposition were much more promising. They were much more random and on a much smaller scale, proving that a multiplicative model would capture the data best. The residuals being so small — on a scale between 1.5 to -1, meant that there was a lot of promise in modeling.</p><p>But now I wanted a function for running SARIMA models specifically, only inputting the order. I wanted to experiment running c,t and ct versions of the SARIMA model with those orders as well since the seasonal decomposition favored a multiplicative type of model over an additive type of model. Using the c, t and ct in the trend = parameter, I was able to add multipliers to my SARIMA model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b00625d850ff48f00742b7848593ec00/href">https://medium.com/media/b00625d850ff48f00742b7848593ec00/href</a></iframe><p>I’ll skip describing the part where I looked at the AFC and PACF plots and the part where I also tried PMD auto arima to find the best terms to use in the SARIMA models. <a href="https://github.com/casanave/Pumpkin_Spice/blob/main/Pumpkin_Spice.ipynb">If you’re interested in those details, please see my full code notebook.</a></p><h4>My best SARIMA model:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUZOTYDHYRscv7-ZvJeYOw.png" /><figcaption>Image by the author.</figcaption></figure><p>So my best SARIMA model had a higher MAPE score than my naive model, nearly 29 to nearly 12, but a lower RMSE by about a unit, nearly 7 to nearly 6. My biggest problem with using this model is it really underpredicted the 2023 spike, there’s a fair amount of area between the red and blue lines from August to September of 2023. There are reasons to like it better than my yearly naive model or worse than my yearly naive model, depending on your opinions about RMSE vs MAPE. However, I wasn’t done yet. My final model was definitively better than my yearly naive model.</p><h3>Final Model:</h3><p>I used an ETS (exponential smoothing) model for my final model, which allowed me to explicitly use the seasonal parameter to make it use a multiplicative approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2W1SkEFsNYN2RWMbAc5Gw.png" /><figcaption>Image by the author.</figcaption></figure><p>Now you may be thinking “but this model has a higher MAPE score than the yearly naive model.” And you’d be correct, by about 0.3%. However, I think that’s a more than fair trade considering that I now have an RMSE of about 4 and a half instead of 7. While this model does struggle a bit more in December of 2022 than my best SARIMA model, it’s off by less area amount for that spike than the larger spike for fall of 2023, which I care more about. <a href="https://github.com/casanave/Pumpkin_Spice/blob/main/ps_predictor.sav">You can find that model here.</a></p><h3>Further Validation:</h3><p>I will wait until 10/7/2024 and do another data pull and see how the model did against last year’s data.</p><h3>Conclusion:</h3><p>To sum up, I was able to disprove the null hypothesis, my final model outperformed a naive yearly model. I’ve proved that pumpkin spice popularity on Google is very seasonal and can be predicted. Between naive, SARMA models, and ETS models, ETS was better able to capture the relationship between time and pumpkin spice popularity. The multiplicative relationship of pumpkin spice to time implies that pumpkin spice’s popularity is based on more than one independent variable besides time in the expression time * unknown_independant_var = pumpkin_spice_popularity.</p><h4>What I Learned and Future Work:</h4><p>My next step is to use some version of <a href="https://developers.facebook.com/docs/instagram-api/">Meta’s graph API</a> to look for “pumpkin spice” being used in business articles. I wonder how correlated that data will be to my Google trends data. I also learned that when the seasonal decomposition points towards a multiplicative model, I’ll reach for an ETS much sooner in my process.</p><p>Furthermore, I’m interested in automating a lot of this process. Ideally, I’d like to build a Python module where the input is a CSV directly from Google Trends and the output can be a useable model with good enough documentation that a nontechnical user could make and test their own predictive models. On the eventuality that a user would pick data that is hard to predict (IE a naive or random walk model would suit better), I hope to build the module to explain that to users. I could then collect data from an app using that module to showcase findings of seasonality across lots of untested data.</p><p>Look out for that app by pumpkin spice season of next year!</p><p>[1] Google Trends, N/A (<a href="http://www.google.com/trends">https://www.google.com/trends</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afd8daaedce1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pumpkin-spice-time-series-analysis-afd8daaedce1">Pumpkin Spice Time Series Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Data Ingestion Tools With Pandas: A Love Letter to a Job I Did Not Get]]></title>
            <link>https://medium.com/geekculture/automated-data-ingestion-tools-with-pandas-a-love-letter-to-a-job-i-did-not-get-b091ad3bbfdc?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/b091ad3bbfdc</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Fri, 24 Feb 2023 04:02:45 GMT</pubDate>
            <atom:updated>2023-02-24T20:53:39.119Z</atom:updated>
            <content:encoded><![CDATA[<p>Valentine’s Day, 2023</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U0YR3aw3H02-_RU_" /><figcaption>(Photo by <a href="https://unsplash.com/ja/@swimstaralex?utm_source=medium&amp;utm_medium=referral">Alexander Sinn</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>Last November, I graduated from Flatiron School’s Data Science Program and started the last and most important phase of my career change: getting my first job in data. And I fell hard in love with a job posting, even when trying not to become emotionally attached. C’est la vie. The job had so many features I’d been looking for:</p><ol><li>Mission Oriented</li><li>Pandas and SQL</li><li>Visualization</li><li>Team Work Culture</li></ol><p>The job had opportunities for creative solutions with R&amp;D, which made me really excited. This job even had a gift for me, like a bouquet of flowers, hidden in the posting. A bulk of the job’s responsibilities would be working on python scripts to help automate the data-ingestion process. Wow! It felt like a gift to realize I could be using Pandas on a higher level.</p><p>Being a fan-boy of function-oriented code, I was quite excited to try building Exploratory Data Analysis scripts that were generalized enough to be used across projects. It made so much sense to try and make tools for cleaning data on a larger scale. Scripts offered so much flexibility and could be used together with pipelines in the future for an automated or semi-automated workflow. If the script had enough utility and intuitiveness, it could save time and headaches for my whole team. So, I applied enthusiastically, and started scripting and preparing for an interview I was hoping to land.</p><p>Unfortunately, that job was not for me to have. However, much like Sir Thomas Wyatt when he wrote love poems about Anne Boleyn, I have all this evidence of my heart’s desire left to keep me company after my heart’s desire has settled on someone else. And while my fledgling script is still being developed and is certainly no genius work like <em>Whoso List to Hunt, I Know Where Is a Hind, </em>I’ve decided to share this script and what it can do with all of you. What we make while in love can be valuable, even when that love is not meant to be, as Wyatt evidenced.</p><p>Without (much) further emo poetry, let’s talk code. This <a href="https://github.com/casanave/Automating_EDA_w_a_Categorical_Target">module</a> was designed to help automate getting to know your data through visualizations and tables. It is not intended to produce deliverables to a stakeholder. Rather, it was designed to help speed up the process of beginning to ask the data what it needs to produce a deliverable.</p><h3>What Does This Module Do?</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.loom.com%2Fembed%2Fb4c2b1dd5e514ff4986de1bc8f94bc6f&amp;display_name=Loom&amp;url=https%3A%2F%2Fwww.loom.com%2Fshare%2Fb4c2b1dd5e514ff4986de1bc8f94bc6f&amp;image=https%3A%2F%2Fcdn.loom.com%2Fsessions%2Fthumbnails%2Fb4c2b1dd5e514ff4986de1bc8f94bc6f-00001.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=loom" width="1280" height="960" frameborder="0" scrolling="no"><a href="https://medium.com/media/d563c9a8fe422ba11c58f0a45ece56d2/href">https://medium.com/media/d563c9a8fe422ba11c58f0a45ece56d2/href</a></iframe><p><strong>In short, the script takes the user through a series of prompts. A CSV (comma-separated value file,) with a categorical target (the dependent variable or OUTPUT COLUMN of the data) is input by the user. It displays a number of useful visuals and returns a data frame for future use.</strong></p><h4><strong>The script is designed to display:</strong></h4><ol><li>the target variable distribution, and the distribution of all other categorical features</li><li>a pivot table with the target as the index and the mean of each numeric feature</li><li>histograms of each numeric feature’s distribution</li><li>bar plots of the mean of each numeric column on the x-axis, grouped by each target outcome</li></ol><p>The idea here is to give the user a quick and dirty view under the hood of the data.</p><h4><strong>These visualizations and tables can help the user ask the following questions about their data:</strong></h4><ol><li>Is my target column balanced? If not, I’m going to start thinking about resampling vs synthetic data vs how I hyper-tune my model.</li><li>Do I have a categorical feature where there are many one-off values? If so, can I combine any of them?</li><li>Are my numeric columns normally distributed (in a bell curve?) Do I need to normalize or transform any of my numeric features before modeling?</li></ol><p>When the output of the deploy function deploy_categorical_target_EDA() is saved to df with df = deploy_categorical_target_EDA() , the user can easily keep working with the data in Jupyter notebooks once this function has run its course. This makes for a much smoother workflow than going through each of these steps by hand for each categorical-target dataset you encounter.</p><p>But how does this module work? How did I build it?</p><h3>Making the Module:</h3><p>I love functions. And this module showcases one of my favorite formats with which to write any code as a bunch of tiny helper functions, organized by the larger functions that deploy them. So let’s take a look at this module, mini-function by mini-function.</p><p>You’ll notice I’ve instituted a try and except methodology with my functions. This is what allows my module to keep rolling, even when the data doesn’t have numeric dependent features but does have categorical dependent features and vice versa. I’ve asked the code to display any error message as a dictionary, simply my preferred way to read that kind of output.</p><p>First thing is first, the module handles importing Pandas as pd, Matplotlib.pyplot as plt, Seaborn as sns, and imports string.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50049c29121f02d33130a1ad4f507a79/href">https://medium.com/media/50049c29121f02d33130a1ad4f507a79/href</a></iframe><p>Now let’s take a look at the first function, input_csv():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21c578a6848b04b2d76b4fc9e7dc1aca/href">https://medium.com/media/21c578a6848b04b2d76b4fc9e7dc1aca/href</a></iframe><p>This function:</p><ol><li>asks for input, prompting the user for a CSV file and attempts to read it with Pandas, displaying the name of all the columns</li><li>asks for input, prompting the user if the columns are currently in ‘Title Case’, if so the user can enter “y” into the input and the columns will be turned into ‘snake_case’ using some string manipulation</li><li>displays df.info to show the columns, how many null values are in each column, the column’s data type, how many rows and the step size of the index</li><li>returns the df</li></ol><p>The next function is select_target():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87868c650b82616cb7b940bf1fb746a6/href">https://medium.com/media/87868c650b82616cb7b940bf1fb746a6/href</a></iframe><p>This function:</p><ol><li>asks for input, prompting the user to select a column for the target</li><li>splits y (each and every row of the target column) X (each and every row from the dependent features) and saves y as a Pandas series and X as it’s own Pandas data frame while saving the name of the target_column as target_column</li><li>displays the distribution of y by using the df[column_name].value_counts() functionality, taking it a step further by styling it to a barplot inline</li><li>returns X, y, and target_column</li></ol><p>The next function is select_cat_cols():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3688da2de2d3d03446ee60ae118b80ab/href">https://medium.com/media/3688da2de2d3d03446ee60ae118b80ab/href</a></iframe><p>This function requires X, which we have from our last function. It works by:</p><ol><li>prompting the user for the categorical columns and making that input into cat_cols as a list, it then slices out these columns and saves them as cat_df</li><li>for each column in cat_df, display the distribution using .value_counts().to_frame().style.bar()</li><li>returns cat_cols so that list can continue to be used</li></ol><p>The next function is select_num_cols():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eee83f5a8856b59e84a3b516f0c21bc1/href">https://medium.com/media/eee83f5a8856b59e84a3b516f0c21bc1/href</a></iframe><p>This function requires X, which we have from previous. It works by:</p><ol><li>prompting the user for numeric columns and saving that as a list num_columns</li><li>for each numeric column in X, it’s going to run pd.to_numeric() on that column to ensure that if it was in a string format before, now it will be numeric (the default datatype is a string in Pandas, so this circumvents that.)</li><li>it returns num_columns so we can continue to use that list in the future</li></ol><p>So far, nothing terribly interesting has happened. However, with our next series of functions, we start getting a little jazzier.</p><p>The function histogram_jam() looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a42ba4616ad631d3eaa88ac98b2853c7/href">https://medium.com/media/a42ba4616ad631d3eaa88ac98b2853c7/href</a></iframe><p>This function needs X and num_columns which we already know about. It also requires something we don’t have yet, labels_dict. This has to do with the order in which the functions are deployed. I want you to notice in the docstring, it notes that this function is deployed by another function called output_viz(). Let’s skip ahead and take a look at a few lines from output_viz() too, as shown below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/715e1029137600c04664137d8f9901d3/href">https://medium.com/media/715e1029137600c04664137d8f9901d3/href</a></iframe><p>These three lines say that basic_labels is a list containing the each column in X. title_labels are simply reverting ‘snake_case’ back into ‘Title Case.’ labels_dict therefore is a dictionary where the format is {&lt;snake_case_label&gt; : &lt;Title Case Lable&gt;, ...}</p><p>Now, looking back at histogram_jam we can see that this function:</p><ol><li>for each column in X that is also in our list num_columns, make a Seaborn histogram as save that as distributions</li><li>using the labels_dict and an f string formatting we can set distributions.set_title and distributions.set_xlabel to make sure that our histograms are using the label_dict to transform the labels on our plots</li><li>display each histogram</li><li>return nothing</li></ol><p>The next function is pivot_table_mean() and it makes and displays a pivot table from our numeric columns, needing X, num_cols, y, and target_column, which we have saved from all our previous functions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/309bc647cbf42d26f5652116a5f0f5a2/href">https://medium.com/media/309bc647cbf42d26f5652116a5f0f5a2/href</a></iframe><p>This function:</p><ol><li>makes num_cols_df which is a data frame made of each numeric column in the data set with its final column being y, our target column</li><li>it then uses the groupby() method with .mean() to aggregate each numeric column by the mean, making a row for each target outcome. The displayed result looks like this:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7D7qyrSd4CjoG7PLs_tNPg.png" /><figcaption>(Image by the author.)</figcaption></figure><p>3. it returns that pivot table, so it can be saved and used for the next function</p><p>The last mini-function in this set of functions is viz_numeric_vars_to_target() and it needs our new pivot_table, labels_dict and target_column:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfe4c28bce538dbd3e6e732f26311f4f/href">https://medium.com/media/dfe4c28bce538dbd3e6e732f26311f4f/href</a></iframe><p>What this function does is:</p><ol><li>working with our pivot_table, for each column in pivot_table, make a barplot with the index (our target categorical outcomes) as the X axis and the values of pivot_table as the y axis.</li><li>using our labels_dict as we have with the .set_title(), set_ylabel(), and set_xlabel(), we can make sure that our bar plots have the labels in the form we want</li><li>returns nothing</li></ol><p>Lastly, we have our deploy functions, which act as wrappers to organize the order in which our mini-functions pop off. Here’s the first deploy function, it contains the execution for histogram_jam(), pivot_table_mean(), and viz_numeric_vars_to_target(). It requires X, num_cols, and target_column, which we have from our mini-functions, if we’ve saved their returns as variables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/198c97dea90414993bde43069a98b79f/href">https://medium.com/media/198c97dea90414993bde43069a98b79f/href</a></iframe><p>This function operates by:</p><ol><li>making our labels_dict in the way we’ve already explored, to be ready to use by our other functions contained within this one</li><li>deploys histogram_jam(), passing in the necessary variables</li><li>deploys pivot_table_mean() with the necessary variables and saves the resulting pivot table as pivot_table</li><li>deploys viz_numeric_vars_to_target(), with the necessary variables including pivot_table as the variable we just made</li><li>returns nothing</li></ol><p>And now we have our big deploy function, which ensures all the other functions execute in their proper order and have all the variables they require to be passed in.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04d172b16597113efd34c374b4490ae2/href">https://medium.com/media/04d172b16597113efd34c374b4490ae2/href</a></iframe><p>This function is the brain of the module. It tries, in order:</p><ol><li>saving the df as the return from input_csv()</li><li>saving X, y, and target_column as the return from select_target() and uses our new df as the required variable for that function</li><li>saving num_columns as the list of numeric columns, from the return of select_numeric_cols() using X as the required variable for that function</li><li>saving cat_cols as the list of categorical features returned from select_cat_cols(), using X as its required variable for that function</li><li>deploys our middle-management function output_viz() with the required variables it needs</li><li>returns our data frame to us, to be potentially saved for further use</li></ol><p>And there you have it!</p><h3>Future Work:</h3><p>This is but my beginning foray into the world of automation for data science and analysis. Any QA tester would be able to brake this module easily. If the target has 15+ different potential outcomes, the bar graphs become difficult to read, for example. Increasing the complexity this module can handle is one obvious next step.</p><p>In the future, I’d also like to make similar modules for myself for when I’m working with time series data specifically because time series work is a favorite of mine.</p><p>I’d also like to keep building this up with functionality that can handle more cleaning tasks, like inputting missing values or looking for anomalies. I’m very motivated to keep working doing automation, debugging is very fun for me, and I enjoy the idea of eventually making a customized graphical user interphase for my own data projects.</p><p>I really must thank the company whose job posting I fell in love with, which prompted me to make this module. I’ve discovered a whole new way to think about the data process, even though I’ll not get the opportunity to use that new way of thinking with this company.</p><h3>A Tiny Bit More Emo Poetry:</h3><p>Here is my favorite poem by Sir Thomas Wyatt, about how he loved Anne Boleyn, but alas, could not pursue her since she was being courted by King Henry the Eighth. Wyatt compares Boleyn to a beautiful doe who he may not hunt for all deer in the king’s woods at that time belonged to the king.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bkFg96cgek4jXiHR" /><figcaption>(Photo by <a href="https://unsplash.com/de/@scapesbyjana?utm_source=medium&amp;utm_medium=referral">Scapes by Jana</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>While about a very different kind of falling in love, it being about falling in love with a person, not a new aspect of coding, it still helped me process my disappointment about this job that I did not get. Not having a “traditional background” for a career in data, I have sometimes felt that I was <em>of them that farthest commeth behind</em> in my search for a new opportunity. <em>Whoso List to Hunt, I Know Where is an Hind</em> remains a truly beautiful and evocative poem, where we can feel the anguish of Sir Thomas Wyatt still to this day. And I must thank Sir Thomas Wyatt for helping me process my love for the job that was not for me to have, even though he would have no idea what a python module would entail.</p><p><em>Whoso list to hunt, I know where is an hind,</em></p><p><em>But as for me, </em>hélas<em>, I may no more.</em></p><p><em>The vain travail hath wearied me so sore,</em></p><p><em>I am of them that farthest cometh behind.</em></p><p><em>Yet may I by no means my wearied mind</em></p><p><em>Draw from the deer, but as she fleeth afore</em></p><p><em>Fainting I follow. I leave off therefore,</em></p><p><em>Sithens in a net I seek to hold the wind.</em></p><p><em>Who list her hunt, I put him out of doubt,</em></p><p><em>As well as I may spend his time in vain.</em></p><p><em>And graven with diamonds in letters plain</em></p><p><em>There is written, her fair neck round about:</em></p><p>Noli me tangere<em>, for Caesar’s I am,</em></p><p><em>And wild for to hold, though I seem tame.</em></p><p><em>-Sir Thomas Wyatt, </em><strong><em>Whoso List to Hunt, I Know Where is an Hind</em></strong>[1]</p><p><strong>Happy Belated Valentines Day, Data Lovers.</strong></p><p>[1] T. Wyatt, <a href="https://www.poetryfoundation.org/poems/45593/whoso-list-to-hunt-i-know-where-is-an-hind">Whoso List to Hunt, I Know Where is an Hind</a>, (unknown), Poetry Foundation</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b091ad3bbfdc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/automated-data-ingestion-tools-with-pandas-a-love-letter-to-a-job-i-did-not-get-b091ad3bbfdc">Automated Data Ingestion Tools With Pandas: A Love Letter to a Job I Did Not Get</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EDA with Tableau: 2021 NYPD Stop-and-Frisks by Demographics]]></title>
            <link>https://medium.com/data-science/eda-with-tableau-2021-nypd-stop-and-frisks-by-demographics-e5e88eb43939?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5e88eb43939</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[nyc]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Wed, 07 Sep 2022 16:11:41 GMT</pubDate>
            <atom:updated>2022-11-12T15:35:53.301Z</atom:updated>
            <content:encoded><![CDATA[<h4>The case for making accessible dashboards when working with public data —</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Cxqj5MZSkCpWc-HVkcelYQ.jpeg" /><figcaption>(<a href="https://www.flickr.com/photos/25246917@N04/7391499704">“Silent march to end stop and frisk and racial profiling</a>” by <a href="https://www.flickr.com/photos/25246917@N04">longislandwins</a> is licensed under <a href="https://creativecommons.org/licenses/by/2.0/?ref=openverse">CC BY 2.0</a>.)</figcaption></figure><p>The above image was taken in a silent march to end stop-and-frisk and racial profiling, and it was taken on June 17th, 2012, one year before the practice of stop-and-frisk was deemed unconstitutional. The public has a vested interest in exploring NYPD Stop-and-Frisk data. This last year in NYC, there were just under nine thousand stops made.</p><p>But what is a stop-and-frisk?</p><h4>A NYPD Stop-and-Frisk Overveiw:</h4><p>Dictionary.com defines stop-and-frisk as <em>“</em><a href="https://www.dictionary.com/browse/stop-and-frisk#:~:text=a%20policy%20that%20permits%20a,suspected%20of%20concealing%20a%20weapon."><em>the policing practice of stopping a person briefly in order to search them for weapons or prohibited items</em></a><em>.”</em></p><p>NYPD has started calling their program “Stop, Question and Frisk” this last year, as a reflection of Mayor Adam’s attitude on the practice of stop-and-frisk, a former NYPD Captain.</p><blockquote>“I was continually pressed on my position on the policing procedure known as ‘stop and frisk’ — which is actually in law enforcement known as ‘stop, question and frisk’ — and why I believed that, if used properly, it could reduce crime without infringing on personal liberties and human rights,” — Mayor Adams <em>[1]</em></blockquote><p>Why this is important to New York City’s history of policing is because if you Google “stop and frisk + unconstitutional”, you see the following first, from Wikipedia:</p><blockquote><a href="https://en.wikipedia.org/wiki/Stop-and-frisk_in_New_York_City#:~:text=In%20Floyd%20v.,where%20such%20stops%20are%20authorized.">In Floyd v. City of New York, decided on August 12, 2013, US District Court Judge Shira Scheindlin ruled that stop-and-frisk had been used in an unconstitutional manner and directed the police to adopt a written policy to specify where such stops are authorized.</a></blockquote><p><a href="https://www.nyclu.org/en/stop-and-frisk-data">The NYCLU has been clear on its stance on NYPD’s stop-and-frisk program</a>.[4]</p><p>From the NYCLU:</p><blockquote><strong>Annual Stop-and-Frisk Numbers:</strong></blockquote><blockquote>An analysis by the NYCLU revealed that innocent New Yorkers have been subjected to police stops and street interrogations more than 5 million times since 2002, and that Black and Latinx communities continue to be the overwhelming target of these tactics. At the height of stop-and-frisk in 2011 under the Bloomberg administration, over 685,000 people were stopped. Nearly <strong>9 out of 10 stopped-and-frisked New Yorkers have been completely innocent.</strong></blockquote><blockquote>…In <strong>2021</strong>, 8,947 stops were recorded.<br>5,422 were innocent (61 percent).<br>5,404 were Black (60 percent).<br>2,457 were Latinx (27 percent).<br>732 were white (8 percent).<br>192 were Asian / Pacific Islander (2 percent)<br>71 were Middle Eastern/Southwest Asian (1 percent)</blockquote><p>However, there’s a problem stopping most New Yorkers from approaching stop-and-frisk data. As I’ve mentioned in my <a href="https://towardsdatascience.com/using-folium-on-police-data-3207e505c649">previous article </a>about NYPD Stop-and-Frisk Data, this information is made available to the public but it isn’t accessible to the public. The files published by NYPD only exist in their raw form, they aren’t summarized or aggregated. Each row of the file represents one record of a stop and to look at the raw files goes a bit like this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FillObkP3TLQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DillObkP3TLQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FillObkP3TLQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2bf90e9c642d79c7fe71e0ee9171757/href">https://medium.com/media/d2bf90e9c642d79c7fe71e0ee9171757/href</a></iframe><p>For data that needs to be accessible to the public, a tool like Tableau Public is an excellent match. Its point-and-click graphical user interface is intuitive for the data-savvy and data novice alike. By using Tableau Public for my EDA, the benefit of interactive visualization helped my own understanding of the data but perhaps, more importantly, I built something accessible for the average New Yorker to understand this last year’s Stop-and-Frisks.</p><p>Let’s take a look at what I was able to build. Then I’ll be working backward to show my methods so you can understand how I was able to build this dashboard and utilize it for EDA.</p><p>My EDA is going to be focused on understanding the relationships between demographics (race, age and sex) to police precincts where the stop-and-frisk happened, and the suspected crime of the person stopped. I’ll also be looking into whether or not Physical Force of any kind was used, and whether or not the stop led to an arrest.</p><h3>Results:</h3><p><a href="https://public.tableau.com/views/StopandFrisk2021NYPD/StopandFrisk2021DemographicOverview?:language=en-US&amp;:display_count=n&amp;:origin=viz_share_link">https://public.tableau.com/views/StopandFrisk2021NYPD/StopandFrisk2021DemographicOverview?:language=en-US&amp;:display_count=n&amp;:origin=viz_share_link</a></p><p>Here’s a static picture of the dashboard I made, as Medium doesn’t support embedding Tableau Public dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DmLq7PIMwcenXJ7HMn2vYg.png" /><figcaption>(Image by the author)</figcaption></figure><h4>The Data:</h4><p>NYPD posts all of its available data to <a href="https://opendata.cityofnewyork.us/">NYC Open Data</a>, and here’s their <a href="https://www1.nyc.gov/site/nypd/stats/reports-analysis/stopfrisk.page">Stop, Question, and Frisk Data</a> [2].</p><p>NYC has it’s NYPD precinct geojson file <a href="https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page">here under “School, Police, Health and Fire</a>” [3].</p><h4>The Code:</h4><p><a href="https://github.com/casanave/tableau_data">My Github Repo where I cleaned the data and prepare it for aggregation is here.</a> I won’t be going over every step I did but instead giving an overview of the choices I made.</p><ol><li>I made STOP_FRISK_DATE and STOP_FRISK_TIME into a date time column for each stop and resaved the file as a csv.gz file to save some space:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abe92d598e13ef28b33556cf896f868a/href">https://medium.com/media/abe92d598e13ef28b33556cf896f868a/href</a></iframe><p>2. I separated out the columns of interest and only kept those:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fb5dfa3a19350aca7579b1bee5c3905/href">https://medium.com/media/3fb5dfa3a19350aca7579b1bee5c3905/href</a></iframe><p>3. I replaced some of the more encoded crime descriptions with their full wordage:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b3e7f38cfbbfaf113d5644463a10f87/href">https://medium.com/media/4b3e7f38cfbbfaf113d5644463a10f87/href</a></iframe><p>4. I changed some of the racial descriptions towards more nuanced language where I could. (Business Insider does a great job of breaking down why “Middle Eastern” is dated language <a href="https://www.businessinsider.com/why-you-should-stop-calling-it-the-middle-east-2015-1">here</a>.[<a href="https://www.businessinsider.com/why-you-should-stop-calling-it-the-middle-east-2015-1">4</a>]) Changing mistakes like MALE and (null) values to UNKNOWN in the SUSPECT_RACE_DESCRIPTIONwas pretty simple. I didn’t want to change too many of these values because I found it pertinent to keep precision in how the police are racially profiling people.</p><p>However, arguments could be made to combine BLACK HISPANIC with BLACK and WHITE HISPANIC with WHITE or to make a HISPANIC or LATINXcategory onto itself. Changing ASIAN / PACIFIC ISLANDER to E. ASIAN is the only category where I lose some of this nuance in exchange for a more clipped name object for ease of use, which is something I fix once I’m in Tableau with an alias.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ede95d4eae4014cf69fd3d26440f1a8e/href">https://medium.com/media/ede95d4eae4014cf69fd3d26440f1a8e/href</a></iframe><p>5. I replaced Y with 1 and the UNKNOWNwith 0 in the columns about Physical Force Used. When I aggregate, this will tally all the Y s together.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ce2c42dae9c40667660d36de0200bbe/href">https://medium.com/media/1ce2c42dae9c40667660d36de0200bbe/href</a></iframe><p>6. For aggregation purposes and our descriptive statistics, I also made a column that tallies all counts of Physical Force used during a single encounter (row.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a548963674237f99e48741db51661ee/href">https://medium.com/media/6a548963674237f99e48741db51661ee/href</a></iframe><p>Which returns to us the following table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*o_9wha2IHZEhdov7PH7_Ow.png" /><figcaption>(Image by the author)</figcaption></figure><p>What’s very important to note about this return is out of the 8,947 rows this data represents, less than 100 of those rows had an UNKNOWN amount of Physical Force used. All the other stops in this data set had a known factor of at least one count of some kind of Physical Force used. We’ll come back to this when we get our descriptive statistics.</p><p>7. I made a boolean column to capture if Physical Force of any kind was used. Note that when I aliased this column in Tableau, FALSE became UNKNOWN :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8834a1277bf61ef7225ea971ec3f92a/href">https://medium.com/media/b8834a1277bf61ef7225ea971ec3f92a/href</a></iframe><p>8. I cast the DateTime column as the index and changed the STOP_LOCATION_PRECICNTcolumn to a string datatype since it’s categorical information and we don’t want to compute these values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a84e0f64231f2bc4578c7743c335e0e3/href">https://medium.com/media/a84e0f64231f2bc4578c7743c335e0e3/href</a></iframe><p>9. I changed some of the values in SUSPECT_REPORTED_AGE that didn’t make sense into UNKNOWN after doing a data[&#39;SUSPECT_REPORTED_AGE&#39;].value_counts() and discovering some age values that were illogical like 0 years and 120 years.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b49cf06b0af5ae3cfa807789959306ad/href">https://medium.com/media/b49cf06b0af5ae3cfa807789959306ad/href</a></iframe><p>10. I sliced out of our data the rows where the age was a known value, and converted those values into integers so we can compute them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9efd2b641ee7ca7d1eff750dc0bb1333/href">https://medium.com/media/9efd2b641ee7ca7d1eff750dc0bb1333/href</a></iframe><p>I did this because I want to make anAGE_CATEGORY categorical column based on the quartile ranges of ages. By splitting up ages by quartile ranges, I knew I could most equally split the data where the data was distributed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a0314eb657d82d4f3d55ef3e1912f59/href">https://medium.com/media/1a0314eb657d82d4f3d55ef3e1912f59/href</a></iframe><p>This bit of code returned to us the numbers that correlate to where the age bins should be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvSOGEgoL5jf1bGlYUBadA.png" /><figcaption>(Image by the author)</figcaption></figure><p>11. So I made my bins to capture ages 20 and below, 21–28, 28–38 and 38 and above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5ef488044dca68724e91930a9fcce45/href">https://medium.com/media/e5ef488044dca68724e91930a9fcce45/href</a></iframe><p>This bit of code returns to us SUSPECT_REPORTED_AGE column to compare to the AGE_CATEGORY column we just made. Let’s see if it worked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y8fj_KzlIuzZYgKiNORyUQ.png" /></figure><p>It worked! I can tell it worked because our first row, indexed at 2021-01-01 01:50:00 is listed as being 40 years old and in the Thirtyeight-Above category and so forth.</p><p>From there, I wanted to mix our age unknowns back into the data with their age category as UNKNOWN :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fea4d91efae804b923be3aa0dc90ded/href">https://medium.com/media/6fea4d91efae804b923be3aa0dc90ded/href</a></iframe><p>Which returns the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*paQ155WjixZEQqBXYx8aNg.png" /><figcaption>(Image by the author)</figcaption></figure><p>Looks like it took! From there, I wanted to get a sense of the overall distributions of ages in the AGE_CATEGORY</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f21448c356defe62b7d78dbcf3d420e7/href">https://medium.com/media/f21448c356defe62b7d78dbcf3d420e7/href</a></iframe><p>Which returned the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKOHNrYcLnJbXkV5Pl3yqQ.png" /><figcaption>(Image by the author)</figcaption></figure><p>Looks like our quantiles worked to capture the distribution evenly, and our UNKNOWN age category is just smaller than half of our other categories.</p><p>11. When I looked into the SUSPECT_SEX distributions like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2666f1560a97b714f1fbf135d84613d5/href">https://medium.com/media/2666f1560a97b714f1fbf135d84613d5/href</a></iframe><p>It returned the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*jRD_AJswhUuFCx6-u7FXYw.png" /><figcaption>(Image by the author)</figcaption></figure><p>Making FEMALE the gross minority in this data set. Having the majority of folks be an UNKNOWN gender didn’t make very much sense. I decided to change UNKNOWN to MALE with the understanding that there may have been some folks who were neither female nor male who may experience erasure in the data by handling it this way.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/180afbc8df74a9f8e910489e9ffc0b09/href">https://medium.com/media/180afbc8df74a9f8e910489e9ffc0b09/href</a></iframe><p>12. I made a column simply for aggregation purposes to count the total number of stops in any given category. I will use this column quite a bit in Tableau.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfbc493ec3f5d47b4b77c8977c27c2a6/href">https://medium.com/media/bfbc493ec3f5d47b4b77c8977c27c2a6/href</a></iframe><p>13. I made the SUSPECT_ARRESTED_FLAG column suitable for aggregation and made its counterpart column SUSPECT_NOT_ARRESTED_FLAG show the opposite values so that I can use both.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/59da1837c2cd6eda32b5672a6c254c2d/href">https://medium.com/media/59da1837c2cd6eda32b5672a6c254c2d/href</a></iframe><p>So if someone was arrested, it would show a value of 1 for SUSPECT_ARRESTED_FLAG and a value of 0 for SUSPECT_NOT_ARRESTED_FLAG which will help me out during visualization.</p><p>14. I took a look into the descriptive statistics using data.describe() and here are my findings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nWszWxnMpTYaqe037NQJJQ.png" /><figcaption>(Image by the author)</figcaption></figure><ul><li>The overall average ratio is 37% Arrested to 62% Not Arrested on Average with a standard deviation of about 50% which means that the variability is higher than the actual arrest rate.</li><li>The average ratio of firearms being drawn during stops is just under 6% of the time.</li><li>Handcuffs are being enforced about 20% of the time during stops.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8alOtSfWZsPgU0QWqbTr5g.png" /><figcaption>(Image by the author)</figcaption></figure><ul><li>OC spray wasn’t used during 2021 during stops according to the data.</li><li>Physical force “other” was used about 3% of the time during stops.</li><li>Restraint was used just under 3% of the time during stops.</li><li>Physical Force Verbal Instruction was used 90% of the time, making it a regular part of most stops.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*QCg3LfbSpQRXlzuXRDZrDg.png" /><figcaption>(Image by the author)</figcaption></figure><ul><li>Weapon Impact was hardly ever recorded as being used.</li><li>The physical force total, perhaps the most interesting metric in this set of stats is at about 1.25% meaning if we sum up all the various counts of physical force used, there were more counts of physical force used than there were stops by about a quarter. This is a very interesting statistic. With the standard deviation being at about 0.5 or a half count of a kind of physical force used, that means that the average experience was that physical force was used on a stop, and how much physical force normally varied by a half count.</li></ul><p>From here I just saved the data using data.to_csv(&#39;2021_data.csv&#39;) and proceeded to make the dashboard with Tableau Public.</p><h4>The Dashboard:</h4><p>This was my first time using Tableau Public and it took me a minute to get my bearings. Here are my generalized steps, but for more detail, I encourage you to download the dashboard itself and start playing around with it.</p><ol><li>Adding the precincts.json file as a Spacial file and the 2021_data.csv file as a text file, and defining the relationship between them as Precicnt1 = STOP_LOCATION_PRECICNT</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oaGxWr9jS-tpCs56BOWL7g.png" /><figcaption>(Image by the author)</figcaption></figure><p>2. Making the <strong>Outcome by Precinct</strong> map by dragging the Geometry onto the blank page, making Stop Location Precicnta dimension detail, (not by SUM,) allows me to scroll over each precinct and have that precinct pop. From there it was a matter of what I wanted to show by precinct. I chose to show the total stops made in that precinct and I chose to show a calculated field I made called Not-Arrested Rate , details on how I calculated that field is here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*FkXu8AJB7gr9H5F7iMotEg.png" /><figcaption>(Image by the author)</figcaption></figure><p>3. For the <strong>Suspected Crime of Stopped </strong>plot, I dragged Total Stops to the Columns and made sure it was taking the SUM. I then dragged the Suspected Crime pill over to rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zzNLq-eOOocYL7_WFg8cUg.png" /><figcaption>(Image by the author)</figcaption></figure><p>I decided to alias “Selling of Cannabis” from “Selling of Marijuana” because I wanted to destigmatize Spanish speakers. More on that from NPR[<a href="https://www.npr.org/local/309/2019/09/19/762044859/pot-weed-marijuana-what-should-we-call-it">5</a>.]</p><p>Making the numbers pop next to the bar was simple too, just dragging SUM(Total Stops) to the text box like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*q_ctswEpVYp7uFhskGP0mQ.png" /><figcaption>(Image by author)</figcaption></figure><p>4. Making <strong>Racial Disparity Over Time </strong>plot was a two-step process. First I dragged the time index column over to Columns and selected Month instead of Year. Then I dragged Total Stops over to Rows and made sure it was taking the SUM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6AmrEYtr_9-7jZFYz02zag.png" /><figcaption>(Image by the author)</figcaption></figure><p>Then I dragged the Suspect Race Description pill over to Marks and made it a color detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*pkFCKbfUQsVRllOabiPbbw.png" /><figcaption>(Image by the author)</figcaption></figure><p>5. Making the <strong>Ages</strong> plot was similar, I dragged Age Category over to columns and Total Stops over to Rows, making sure it was taking the SUM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ov0jrZrTYrE9o2lKZxp5SQ.png" /><figcaption>(Image by the author)</figcaption></figure><p>Then I likewise dragged Suspect Race Description over to the color Marks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*jkS1M9JUZPpL2dR8vZGw1Q.png" /><figcaption>(Image by the author.)</figcaption></figure><p>6. Putting it all together onto one dashboard, making each plot interact, and setting up all the filters, was a fun process of mess-around-and-find-out. To see my full process, the easiest thing would be to download the dashboard from Tableau Public and mess/find out yourself. Further resources are abundant in the Tableau community documentation: <a href="https://help.tableau.com/current/pro/desktop/en-us/filtering.htm">https://help.tableau.com/current/pro/desktop/en-us/filtering.htm</a> [6]</p><p><em>As a note: this dashboard could still stand to see some changes or improvements. I haven’t forayed into using the SETS functionality to group all precincts in the same borough together for example. Nor have I connected a way to show the stops that have a count of 2 or more kinds of Physical Force used, which may be pertinent when separating out</em> PHYSICAL_FORCE_<em>.</em></p><h3>Why is this better than doing all my EDA in python?</h3><ol><li>I can interface with the data with rapid-fire questions without being slowed down by making a ton of static tables and graphs.</li><li>I can share my findings extremely easily with the public and my stakeholders.</li></ol><p>For example, what if I just wanted to get a breakdown of stats where the Not-Arrested rate is the highest? On the map, I could easily spot precinct 75 (East New York) as the darkest red, and when I click on it, I can see the distribution breakdowns of the Suspected Crimes just in that precinct being overwhelmingly Criminal Possession of a Weapon. From here I can also see the distributions of ages by race, and the time the arrests happened by race. A single click has saved me many, many lines of code!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQY7SvL_4Vv-iXUAdTKrUQ.png" /><figcaption>(Image by the author)</figcaption></figure><p>When we use those descriptive statistics as insights into this precinct, we can understand that on average, Precinct 75 is stopping folks much younger than the average age for stops across all stops(28 years old.) We also learn that Precinct 75’s Not-Arrested rate is much higher than the average for the city, 83% compared to 67%. If we take a look at the timeline, it occurs to me to ask “What started happening in April to spike stops, especially when stopping Black folks?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PX1pOUFvbpZsaIkDX7Xvtw.png" /><figcaption>(Image by the author)</figcaption></figure><p>When we compare our graphs using the Outcome filter, we can see that out of the 330 people stopped in Precinct 75 in this last year for suspected criminal possession of a weapon, 278 of those folks were not arrested.</p><h4>Further findings:</h4><p><strong>Suspicion of Criminal Possession of a Weapon Stops That Do Not Lead to Arrest:</strong></p><p>Of the 8,947 stops made this year, 2,471 of them were under suspicion of criminal possession of a weapon and did not lead to an arrest. That’s just under <strong>27% of all stops. </strong>By looking at the Ages plot, we can see that more than a fourth of those stops were stopping folks under twenty years of age, or where their age was not listed. We can also see a majority of black folks make up these stops. And when we look over to the Racial Disaparity Over Time Plot we can see a wave that starts in September and peeks in November where the number of stops is more than double what they were in September. It begs the question, what started happening in September, and why are the police stopping young black men under suspicion of criminal possession of a weapon as a result?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3z5q8WiyJ1JJrX0-fLj-lg.png" /><figcaption>(Image by the author)</figcaption></figure><p><strong>Staten Island Stops:</strong></p><p>When we take a look at the stops that happened only within Staten Island, we get a different slice of the data. We see even more evidence of targeting younger folks for stops, (but less trend to not list the age,) and black young folks make up half of the stops in the twenty-and-under category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EfsRfIYxEoETWgOWkLTcpw.png" /><figcaption>(Image by the author.)</figcaption></figure><p>What’s most notable about the stops that happened in Staten Island is a vast majority of them used physical force. These two photos show all the stops that happened on Staten Island, and then I isolate only the stops that used Physical Force of some kind. These two photos are almost identical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YfLMJ4Ld4o4v7mEBUXlmLg.png" /><figcaption>(Image by the author)</figcaption></figure><p><strong>That’s because there were only 5 stops that happened in Staten Island this last year where it was unconfirmed if Physical Force of some kind was used or not. All the rest of the stops there involved Physical Force. </strong>In the future, I’d want to run hypothesis tests to see how different Staten Island data is from the rest of NYC, as we previously discovered that at least one count of Physical Force being used is common in 90% of stops this last year city-wide.</p><p><strong>Precinct 14’s Stops and Women:</strong></p><p>When just looking at the stops where those who were stopped were listed as Female (and bearing in mind that while most female folks ID as women, this is an oversimplification,) we can see one precinct in lower manhattan in particular where the not-arrest rate is particularly high (dark red.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PhYQPtrFKdkMHIjMrqW0sw.png" /><figcaption>(Image by the author)</figcaption></figure><p>On further investigation, we can see that this is precinct 14, which had 60 stops made up of women this last year, 90% of whom were not arrested. That’s quite a high rate compared to our 67% not-arrest rate for all stops across the city.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7-9jN4TDK5aIHrgd8vRhA.png" /><figcaption>(Image by the author.)</figcaption></figure><p>Of these 60 stops, 40 (67% or two-thirds) were under suspicion of criminal possession of a weapon. When we look to see their ages, we can more than half of these women were under 28 years old, and a majority of those who didn’t have their age listed were of East Asian or Pacific descent. When we look at our Racial Disparity Over Time plot, we can see that these stops had a big spike between October and November.</p><p>When we take an even closer look, we can see that a majority of these stops did not lead to arrests. <strong>By comparing these two snapshots we can see the difference in the Racial Disparity Over Time is made up of mostly Black women. Importantly, most of the stops that lead to the arrests of these black women didn’t happen when stops were spiking from October-November.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1qKQDRwpDgYvmAuckjdYw.png" /></figure><p>By using Tableau, I can share these findings with a much larger audience, and I was able to rapidly fire questions at the data with ease of use. Tableau has proven extremely useful in helping me explore the data, and in finding where I want to inquire the data even further.</p><p><strong>In conclusion</strong>, I’ll be incorporating Tableau into my EDA process and I encourage you to consider doing the same. It saves you time as the person asking questions of the data, and your stakeholders will thank you for being able to understand and interact with the data themselves.</p><p>[1] E. Adams, A. Wehenkel and G. Louppe, <a href="https://www.nydailynews.com/opinion/ny-oped-how-we-make-new-york-city-safe-20211128-4ekltfwlx5f4bgb3r4aegdnaiu-story.html">How we make New York City safe: Mayor-elect Eric Adams explains why we need stop and frisk and proactive policing</a> (2021), New York Daily News</p><p>[2] NYPD, <a href="https://www1.nyc.gov/site/nypd/stats/reports-analysis/stopfrisk.page">Stop, Question and Frisk Data</a> (2021)</p><p>[3] NYC, <a href="https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page">Political and Administrative Districts — Download and Metadata</a> (2021)</p><p>[4] NYCLU, <a href="https://www.nyclu.org/en/stop-and-frisk-data">Stop-And-Frisk Data</a> (2021), NYCLU</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5e88eb43939" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/eda-with-tableau-2021-nypd-stop-and-frisks-by-demographics-e5e88eb43939">EDA with Tableau: 2021 NYPD Stop-and-Frisks by Demographics</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[8 Steps Towards A More Publishable Technical Blogpost]]></title>
            <link>https://medium.com/geekculture/8-steps-towards-a-more-publishable-technical-blogpost-fc7e25736fa2?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc7e25736fa2</guid>
            <category><![CDATA[publishing]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[writing-tips]]></category>
            <category><![CDATA[technical-writing]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Thu, 26 May 2022 06:03:41 GMT</pubDate>
            <atom:updated>2022-05-26T06:03:41.695Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Protocol To Ensure Your Blog’s Excellence</h4><p>A hiring manager finds your blog post about a certain SQL aggregate function and its use-cases specifically within financial records and sends you a message over Medium to check out the new SQL-heavy position at their workplace. They think you’d be a great fit. A recruiter reads your blog post about why scikit-learn has the best-written documentation compared to other python machine learning packages, and wants to talk with you about a new ML engineer position that their boss is hiring for. An amazing technical publication wants to pick up your most recent blog post on your experiences at the latest hackathon, and in doing so your name gets mentioned in a digital room full of opportunities. That’s the dream, right?</p><p>Blogging is showcasing our skill sets as technologists, (in my case, as a data scientist.) While blogging improves your written communication skills, and how you explain your ideas to others, blogging can also have the very real consequence of getting you hired for your next job. Publishing your blog will only increase the scope of who discovers your skills.</p><p>But how to write a blog that showcases those written communication skills? With APIs, we can look up the documentation to make sure we are using the correct parameters to return the information we need. But where is the documentation for making a good technical blog?</p><p>This blog post attempts to provide some of that structure. It can be thought of as a protocol, a set of guidelines or a checklist, whatever framework you like. These steps were born out of the workshop <strong>Blogs That Will Get You Noticed </strong>that I developed and presented to Flatiron School twice now, while being a student there, as a supplement to their curriculum. I developed that workshop after my first blog post was published by CodeX. After my last blog post was accepted as an Editor’s Pick in Towards Data Science, I knew I had something to offer the greater technical blogging community here on Medium.</p><p>Without further posturing, here is that protocol, Eight Steps Towards A More Publishable Technical Blogpost.</p><h3>1) The HOOK: how do you capture your audience’s attention?</h3><p>Let me bring your attention to Medium and how previewing articles works on this site specifically. I searched Medium for “how to blog” and here are the first two results that popped: <strong>How to Blog the Post-Pandemic Way</strong> by <a href="https://medium.com/u/d573475dabea">Jason Weiland</a> and <strong>How to Blog: A guide for non-bloggers</strong> by <a href="https://medium.com/u/79ffa075b035">Haje Jan Kamps</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3AhqfiEP27NKb155kCmgzw.png" /><figcaption>Image by the author.</figcaption></figure><p>Both look interesting! If you only had time to read one how would you choose? The title, the accompanying picture, the publication, or maybe the tags, but probably you read those three lines of prose that Medium.com previewed for you, right?</p><p>A <strong><em>hook</em></strong> is something that grabs the reader&#39;s attention right away and sparks their interest so they want to keep reading. On Medium we have something equivalent to the length of a Twitter post to inspire our readers to click on our blog article. And the hook to our article is competing with the hooks of other articles for that singular click.</p><p>This idea comes from creative writing conventions but is essential to technical blogging. After all, if you were looking for a blog article about using Seaborn for example, which article would you click on first based on just the first line:</p><p><strong>A</strong>: “MatPlotLib is a python library for data visualization, a beautiful but complex monster of different syntaxes which Seaborn was built on top of, as a way to regain some sanity.”</p><p><strong>B</strong>: “MatPlotLib is a python library for data visualization. Seaborn was built on top of MatPlotLib as its own library with much more consistent syntax.”</p><p>They’re very similar and are about to discuss the very same information. So why does A sound so much more fun to read? What does B lack? When writing A, I used evocative language ie “regain some sanity.” I let more of my personality and opinions show in my writing. B, by contrast, is very dry and feels, for lack of a better word, kind of <em>blah</em> to read. A also used language that put a picture in my reader’s head, using “a beautiful but complex monster.” B by contrast is less engaging of the readers&#39; visual imagination.</p><p>Here’s an example of a great hook from popular media:</p><blockquote>“I was 12 going on 13 the first time I saw a dead human being.” — first line of the film Stand By Me, written by Richard Dreyfuss based on the novella by Stephen King</blockquote><p>This hook uses the unexpected to get the viewer’s attention. It immediately fills the viewer&#39;s head with further questions about said dead body, and why the narrator had access to one at the age of 12. Curiosity is a powerful kind of attention to capture your readers.</p><p>Another strategy for building a good hook is to start by thinking of what your reader would be most interested in, and giving it to them in the first line. I used that strategy to get you (hopefully) reading this blog post, by saying <em>a</em> <strong><em>hiring manager</em></strong><em> finds </em><strong><em>your</em></strong><em> blog post…</em> In those seven words, I am already talking directly to the reader, (<strong>you</strong>,) and potentially finding a <strong>job, </strong>something I am guessing you are very interested in.</p><p>But what do we do with that attention once we’ve earned it?</p><h3>2) The MESSAGE: what do you want your reader to know by the end of reading your blog post?</h3><p>You are going to say very many amazing and intelligent things in your blog posts. You probably want your reader to leave your blog post with more than one thought on their mind. So what constitutes a <em>message</em> and why is it important to have one?</p><p>A message here is defined as one overarching thesis, an umbrella statement that every line of your post will point to or prove true. The message could also be framed as the thing your audience is looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3vn8k0a6HS80rQYi" /><figcaption>Photo by <a href="https://unsplash.com/@austinchan?utm_source=medium&amp;utm_medium=referral">Austin Chan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The message of <em>this</em> particular blog post could be stated as <em>“if you do these eight steps when writing, your blog post will be more publishable.”</em></p><p>Having a clear message before you start writing is similar to pseudo coding a function. Here, I’ll use a simple function as an example:</p><p><em>f</em>(<em>x</em>) = <em>x</em> + <em>m</em></p><p>You are inputting <em>x </em>and the function will result in an output of <em>x</em> + whatever <em>m</em> contains.</p><p>If the reader is <em>x</em>, and the act of reading your writing is the function, then the input is your reader before reading your writing, and the output is your reader + <em>m</em> where <em>m</em> represents the message of your writing.</p><p>This might sound simplistic but thinking of your writing as a definitive value that your reader will leave with at the end of your article, will help your clarity, your focus, and your reader’s understanding.</p><h3>3) The <strong>TECHNICAL CLARITY</strong>: how well can someone understand the concepts you are explaining?</h3><p>Technical clarity is essential, especially with the rigorous amount of technical knowledge some of your blog posts attempt to summarize and illustrate. So often I find myself going to technical blogs, specifically when there is something technical that I’m struggling to understand. There are many ways to go about making your technical writing clear, and here are a few:</p><ol><li>get feedback on your writing from someone who thinks very differently than you, they may be able to point out your blind spots (where your own expertise on a subject prevents you from understanding how a novice might misunderstand that subject)</li><li>try and anticipate where something may be ambiguous and spend extra time explaining where folks may get lost</li><li>explain the same idea in a few different ways to cater to different kinds of intelligence (ie linguistically or by using analogies, visually, mathwize, etc)</li><li>try and break down complex procedures by going in a detailed linear order, as if you were explaining how a recipe worked</li><li>showcase examples of what you are trying to explain</li><li>explain jargon, acronyms, and vocabulary in plain language</li></ol><p>My biggest tip for technical clarity is to put yourself in the mind of two very specific people: an expert in your field but who is, perhaps, very drunk, very tired, or very distracted, and a very intelligent space alien who knows absolutely nothing about your field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FkeXp78P417uVuOO" /><figcaption>Photo by <a href="https://unsplash.com/es/@stephenleo1982?utm_source=medium&amp;utm_medium=referral">Stephen Leonardi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you can lead this party of two through the technical aspects of your writing, you have done an amazing job!</p><h3>3) The AUDIENCE: who was your blog article designed to be read by?</h3><p>This blog article is designed to be read by technical writers who are looking to publish their blogs, and potentially get hired. I know this because that audience, (you,) is who I keep addressing.</p><p>Now, it’s a good attitude to have, thinking that all blog articles you are writing are for a hiring manager, a recruiter, or your next boss to read. This attitude will help you not only stay motivated when writing but keep you on your best game.</p><p>However, most of the time, you’ll not be writing to them directly as the primary audience for your technical blog. That’s what cover letters are for!</p><p>Whether your blog article is for code newbies looking to execute their first line of code with a print(“hello world”), or you are looking to discuss the intricacies of pre-processing data, you can signal to your audience “hey, I see you. I’m here to talk to you specifically.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V2jvGV7mX6WKjFj5" /><figcaption>Photo by <a href="https://unsplash.com/@jannerboy62?utm_source=medium&amp;utm_medium=referral">Nick Fewings</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A very simple and direct example of choosing your audience that you might be familiar with in advertising is a list of questions about the audience where the audience can answer “yes” to each question.</p><p>For a non-technical example, if I was writing a post about how to pick out a new washer/dryer, I could use this as both my hook, and to define my audience: <em>Are you tired of going to the laundromat? Sick of the fluorescent lights, and the quarter machine rejecting your dollar bills? Are you considering buying a new Washer/Dryer? If so, this post is for you!</em></p><p>There are a lot of less direct ways to go about defining your audience in your writing, but I find that if I chose my audience before I start writing, then I will unconsciously fit my article to that audience. But, when in doubt, the sales pitch approach described above is a functional framework. The goal here is to make the reader feel seen and welcomed.</p><h3>5) The IMAGES: what will your reader see in your blog?</h3><p>(*Please note, this section is written under the assumption that you, my reader, are a primarily visual reader.)</p><p>This blog, no matter how interesting the words may be, will have a much more impactful and fascinating effect on my readers if I include relevant images. The human eye can respond to color, to composition and will often try to make a story out of the pictures in an article.</p><p>So far in this blog, I’ve used a screenshot of the return from a search on Medium, and a series of images from Upslash. I’ve introduced each image with either a phrase or an idea, so the reader can connect that image to my overall message.</p><p>There’s a reason why The New York Times tends to put a large image with their headlines. Let’s explore the picture below for a non-technical example. The image below gives us a single word: <em>cliffhanger</em>, and a picture of Donald Trump. What meaning does the word “cliffhanger” mean when paired with this image and how does that meaning change if the image isn’t provided?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0tLQxh8nl-t3bw1x" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A good image (or video) will break up the monotony of the text in your blog post, but a great image will reinforce a concept or prove what your writing is trying to say.</p><p>To prove my point, let’s do an experiment with something technical. I made this video for my post <a href="https://towardsdatascience.com/using-folium-on-police-data-3207e505c649">Using Folium on Police Data</a> last month, using data from <a href="https://opendata.cityofnewyork.us/">NYC OpenData</a> about Stop and Frisks from 2020. Hit play and close your eyes.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FillObkP3TLQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DillObkP3TLQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FillObkP3TLQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/01db2123a050d5125de74b1303132c06/href">https://medium.com/media/01db2123a050d5125de74b1303132c06/href</a></iframe><p>Pretty boring right? And it may not lend itself much to your understanding of what this data set looked like.</p><p>Now try rewatching it with your eyes open. Now you can see as I narrate, just how unwieldy that data set was to work with. Your visual understanding of the video adds meaning to my narration.</p><p>Don’t be afraid to make your own videos of your technical processes!</p><h3>6) The CODE: what does your code do or prove?</h3><p>Your code is evidence of your intelligence, your process, and your technical prowess. Adding snippets of code, by using GitHub’s Gists for example, is a surefire way to increase the tactical wow factor in whatever your blog is about. What could be more straightforward than having enough code in your blog about your code?</p><p>(<a href="https://docs.github.com/en/get-started/writing-on-github/editing-and-sharing-content-with-gists/creating-gists">Documentation of GitHub’s Gists here.</a>)</p><p>But what makes a good gist? Let’s use that recipe metaphor again. All gists and images for that matter can follow this order of operations:</p><ol><li>introduce what the gist will show like I just did above</li><li>include the gist</li><li>state in plain language what the above gist just displayed</li></ol><p>If I was writing a technical blog about how to print three strings in python. I could just put all my code in one block, like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70a3de18f938fb03d9282f6b38649be8/href">https://medium.com/media/70a3de18f938fb03d9282f6b38649be8/href</a></iframe><p>The above code defines three strings. I create the object “all_strings” where the three strings are concatenated. Then I use a print statement which will then show me the output as</p><pre>Hello World! Thanks for reading my blog, I hope you learned something useful.</pre><p>This wouldn’t be the worst thing in the world for this tiny bit of code. It’s pretty simplistic for who this blog article was designed for. But what if the audience was folks who’d never coded (in python) before?</p><p>Then I’d probably want to unpack this code by breaking it down into three gists, and comment on each step.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0423b701cf15cce90a4d0d7e24bee2fe/href">https://medium.com/media/0423b701cf15cce90a4d0d7e24bee2fe/href</a></iframe><p>The code above shows how I was able to create my three strings. Note how in string1 and string2, the last character is an empty space, since python will not automatically add empty space to the end of each string. Strings are made up of characters and are wrapped in either single-parenthesis ‘’ or double parenthesis “” syntactically.</p><p>Next, I’m going to <em>concatenate</em> these three strings which means sticking them together. Concatenation keeps the original order of each object concatenated. You can think of strings here as letters in a word. Each letter starts at the end of the last letter and the order of A + P + P + L + E turns into APPLE using this method, keeping the order intact.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b325849c5d577b06abd3395bcb5a44ba/href">https://medium.com/media/b325849c5d577b06abd3395bcb5a44ba/href</a></iframe><p>The code above shows how to do that concatenation. Finally, I’m going to print all_strings, so we can see the output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/030f9d9e418d580c6a3c8eeca0dcc983/href">https://medium.com/media/030f9d9e418d580c6a3c8eeca0dcc983/href</a></iframe><p>The above code will return:</p><pre>Hello World! Thanks for reading my blog, I hope you learned something useful.</pre><p>Keep your audience in mind when you display your code, and let your audience guide you when you are asking yourself “how much should I break down and explain my code during my blog post.”</p><h3>7) The AUTHOR: who are you and what do you bring to the article you are writing?</h3><p>How much detail you include in your technical blog about who you are is a choice you are going to have to make for yourself. It’s a personal question of boundaries.</p><p>You have an opportunity to choose how you’d like to present yourself. You can do this directly, as I did in the introduction to this article, by telling you how I developed this blog article and my publication history. You can and will do this undirectly, in every choice you make while blogging. Your cadence, your vocabulary, your message, how you chose your images and how you wrote your code, etc, are all going to paint a picture of you in the mind of your audience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n_EOrhSkBbU3EA8G" /><figcaption>Photo by <a href="https://unsplash.com/@jontyson?utm_source=medium&amp;utm_medium=referral">Jon Tyson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is something to lean into, in my opinion. I read a lot of blogs where the author’s voice will sound either like a college essay or even more impersonally, like documentation. I would encourage you to use your own voice when writing as if you were speaking to a friend, family member, or a mentor/boss with whom you have a good rapport. There are a million college-style essays to read, and documentation to go through, but there is only one you in the world, and that’s already more interesting to read.</p><h3>8) The CONCLUSION: how do you want to end your article?</h3><p>You’ve probably heard the phrase “start strong, end strong.” Your conclusion can be thought of as your reader reaching the final destination on a journey. It’s when your reader steps off the plane at the airport and thinks to themselves “I was there, and now, I’m here.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1EizsQbSMnAh16Pe" /><figcaption>Photo by <a href="https://unsplash.com/@mantashesthaven?utm_source=medium&amp;utm_medium=referral">Mantas Hesthaven</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Your conclusion is an opportunity to make sure your message hits home, to recall the journey that your reader has gone on with you, and perhaps ask new questions of your reader, so they can see this blog in a new light.</p><p>For example:</p><p>In this blog article, we read about how to capture your audience&#39;s attention, how to choose images to include in your article, and how to unpack your code for readers. We also discussed why having a clear idea about the message you want to get across, as well as a specific intended audience can help your audience find that message. Additionally, we went over strategies for providing technical clarity, and I encouraged you to speak in your own voice when writing. To conclude, I’m going to show you an example of a conclusion, and emphasize that by following these eight steps showcased, your writing will be clearer, more interesting to read, publishable, and potentially get you your next job interview. After all, you never know who will be reading your blog in the future!</p><p>Taking this a step further, if you were to write an article about how to write articles, as I have attempted, what would you do differently? Are there any tips or tricks you’d share with your readers about getting published? What are your secret methods for writing technical blogs?</p><p>Conclusions are your opportunity to leave a lasting impression and reward your reader for reading the entire article. My hope is that you leave this blog article feeling empowered to write and explain your ideas. After all, there is no shortage of genius in the Data Science community on Medium, and my personal investment is that I want to be able to understand your ideas when I read your technical blogs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc7e25736fa2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/8-steps-towards-a-more-publishable-technical-blogpost-fc7e25736fa2">8 Steps Towards A More Publishable Technical Blogpost</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Folium on Police Data]]></title>
            <link>https://medium.com/data-science/using-folium-on-police-data-3207e505c649?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/3207e505c649</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[choropleth-map]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Sun, 24 Apr 2022 17:49:48 GMT</pubDate>
            <atom:updated>2023-01-16T18:23:10.950Z</atom:updated>
            <content:encoded><![CDATA[<h4>Say YES to the MESS with Choropleths</h4><p>Not one week into my Data Science Program at Flatiron School and I was already haunted by the colloquial knowledge “<em>police data is a headache to work with. It’s </em><strong><em>messy</em></strong><em>.</em>” And even under the best circumstances, I would expect police data to be messy. The police are not trained to make data collection their first priority, nor should they be. This can be easily contrasted to the entire teams of data collectors working at Netflix or Spotify or Google — whose first priority is engineering their data to be optimized for analytics.</p><p>However, I think messy, police data is practically <em>ripe</em> for visualization, simply because it’s too messy, complex, and large for the average person to understand. These are circumstances where visualization can bring much-needed intuitive understanding. A simple choropleth can become the beginning of the story our data has for us. And this story is one that the average New Yorker deserves to have access to. And if you are an average New Yorker and care more about the findings than the code, feel free to scroll down to <strong>findings</strong>.</p><h3>NYPD and Stop-and-Frisks, a Boilerplate:</h3><p>For this project, I used 2020 NYPD stop-and-frisk data. The American Civil Liberties Union (ACLU) <a href="https://www.nyclu.org/en/stop-and-frisk-data">has already thoroughly analyzed this data</a>.[1] They found that of the <em>9,544 stops recorded, 5,791 were innocent (61 percent).</em></p><p>They also found:</p><blockquote>5,367 were Black (56 percent).<br>2,879 were Latinx (30 percent).<br>856 were white (9 percent).<br>219 were Asian / Pacific Islander (2 percent)<br>132 were Middle Eastern/Southwest Asian (1.4 percent)</blockquote><p>And why those demographics are relevant to a baseline knowledge of stop-and-frisk is because stop-and-frisks were deemed unconstitutional in 2013 as a violation of the 4th Amendment, prohibition of unreasonable searches and seizures.</p><p>If you Google “stop and frisk + unconstitutional”, you see the following first, from Wikipedia:</p><blockquote><a href="https://en.wikipedia.org/wiki/Stop-and-frisk_in_New_York_City#:~:text=In%20Floyd%20v.,where%20such%20stops%20are%20authorized.">In Floyd v. City of New York, decided on August 12, 2013, US District Court Judge Shira Scheindlin ruled that stop-and-frisk had been used in an unconstitutional manner and directed the police to adopt a written policy to specify where such stops are authorized.</a></blockquote><p>In New York City, Mayor Adams is a former police captain, and a founding member of 100 Blacks In Law Enforcement Who Care (an advocacy group that focuses on fighting injustices between the African American community and their interactions with the NYPD.)<a href="https://www.ny1.com/nyc/all-boroughs/news/2021/05/25/eric-adams-explains-why-he-supports-stop-and-frisk--when-it-s-used-legally"> Eric Adams also promised to bring a legal form of stop-and-frisks back</a>, [2] stating <em>“If you have a police department where you’re saying you can’t stop and question, that is not a responsible form of policing…”</em></p><h3>An Expensive Institution:</h3><p>In 2022, <a href="https://council.nyc.gov/budget/wp-content/uploads/sites/54/2021/05/NYPD.pdf">“The proposed NYPD expense budget for Fiscal 2022 is $5.44 billion, representing 5.5 percent of the City’s total.”</a> [3]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WBxEqR1am1O9lTEy67fjtw.png" /><figcaption>Image by the author (<a href="https://council.nyc.gov/budget/wp-content/uploads/sites/54/2021/05/NYPD.pdf">from page 2 of the linked NYC Council Budget</a>.)</figcaption></figure><p><a href="https://www.businessinsider.com/the-real-cost-of-police-nypd-actually-10-billion-year-2020-8">Business Insider estimated the real cost of NYPD’s funding was closer to 10B in 2020.</a> [4] New Yorkers’ taxes make up around 95% of the funding for NYPD, as the most heavily funded police, anywhere. It’s extremely important for the health of New York City that the average New Yorker knows about the everyday goings-on of their city’s police. But there’s a huge bottleneck of that information.</p><h3>The Accessibility Problem With Pubic Police Data:</h3><p>For transparency, NYPD makes much of its data public.[5] But in the case of <a href="https://www1.nyc.gov/site/nypd/stats/reports-analysis/stopfrisk.page">stop-and-frisk data</a>, none of that data was aggregated. Instead, each row of information represented one independent stop-and-frisk.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FillObkP3TLQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DillObkP3TLQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FillObkP3TLQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/01db2123a050d5125de74b1303132c06/href">https://medium.com/media/01db2123a050d5125de74b1303132c06/href</a></iframe><p>There’s nothing unusual about public data sets not being made for the average human to discern meaning from. It’s common practice. But it’s not doing the average New Yorker much good as is until we do something with it. Let’s put it on a map!</p><h3>New Yorkers Know Neighborhoods:</h3><p>Maybe it’s from constantly looking at subway maps?</p><p>After all, the average New Yorker can now look at this map and understand <em>“Stop-and-Frisk meant something extremely different to East Harlem, East New York, and The Bronx (marked by red arrows) than it did to Staten Island, the Financial District, and Bay Ridge (marked by green arrows.)”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6rN58PZhSmCKMBx6Gmtzw.png" /><figcaption>Image by the author.</figcaption></figure><h3>The Code:</h3><p>As a note, this blog post is <em>not</em> going to teach you anything that I didn’t learn first from <a href="https://medium.com/u/dce21577b847">Jade Adams</a> in her amazing post: <a href="https://medium.com/@jade.adams517/using-geopandas-and-folium-to-build-maps-on-python-c51c5aad5f44">“A Python Tutorial on Geomapping using Folium and GeoPandas</a>.” [6] Look at that post if you’re looking for an amazing Folium tutorial, I’m going to keep talking about how to visualize NYPD data specifically. I’ll be taking you through my steps on how I made the above map.</p><p>First, open up whatever you use to write code, I like Jupyter Notebooks. Let’s import pandas to read the excel file and folium to make the map.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c164984103e7922c2b98b344590f277/href">https://medium.com/media/9c164984103e7922c2b98b344590f277/href</a></iframe><p>The next step is to take a look at the shape and columns of the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/951ab85633e5a135d320a7229448108e/href">https://medium.com/media/951ab85633e5a135d320a7229448108e/href</a></iframe><p>The output told us that we’re working with 9544 rows, 83 columns, and the names of all the columns.</p><p>Let’s get rid of anything that’s not id or zipcode data for our purposes, just to keep things extra simple. We can always come back to this extremely complex data set later for further investigation (as well we should!) After all, this map is not the end of our process, only the beginning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/111968e9ab0bb4646f770f9e8f30425d/href">https://medium.com/media/111968e9ab0bb4646f770f9e8f30425d/href</a></iframe><p>However, now that we have this new data frame, saved as zipcode_frequency, we need to do some basic reformatting to be able to use it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a4f59cc7f090b7c0d031dae901b078f/href">https://medium.com/media/1a4f59cc7f090b7c0d031dae901b078f/href</a></iframe><p>The above code will return the following report of our columns, how many non-null values there are in that column, and the data type of that column.</p><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 9544 entries, 0 to 9543<br>Data columns (total 2 columns):<br> #   Column                  Non-Null Count  Dtype <br>---  ------                  --------------  ----- <br> 0   STOP_ID                 9544 non-null   int64 <br> 1   STOP_LOCATION_ZIP_CODE  9544 non-null   object<br>dtypes: int64(1), object(1)<br>memory usage: 149.2+ KB</pre><p>Notice how the Non-Null Count shows 9544 non-null values for both STOP_ID and STOP_LOCATION_ZIP_CODE, this shows us that our null values will not be filled by a simple zipcode_frequency.fill_na(&lt;params here&gt;). However, when we run the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9be9b87fac1f58fd01c3d27b54d664ed/href">https://medium.com/media/9be9b87fac1f58fd01c3d27b54d664ed/href</a></iframe><p>We got the following output: a count of each time the zip code was filled in, a count of the unique zip codes, the most occurring zip code, and the frequency of that most occurring zip code.</p><pre>count       9544<br>unique       179<br>top       (null)<br>freq         654<br>Name: STOP_LOCATION_ZIP_CODE, dtype: object</pre><p>This tells us that our null values are filled with the value of “(null)” in the zip code column 654 times. But what percentage of the data is that? How many times did police officers in NYC fail to record the zip code where they did record a stop-and-frisk?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22196109d87cc52ca7d07bc34b660cb7/href">https://medium.com/media/22196109d87cc52ca7d07bc34b660cb7/href</a></iframe><p>This code returns our string with its value-filled in as the percent of data we’re getting rid of.</p><pre>This data has 0.07 percent of its zipcode data missing.</pre><p><strong>Removing the nulls, vs. adding the null values to the Yes column vs. adding the null values to the No column</strong>: there is a bit of a dilemma here. In the future, I’d like to come back to this project and figure out a way to both aggregate the data, and hold on to all three, distinct values. Normally, I wouldn’t hold onto null values, but with this kind of data, null values could tell their own story (do certain zipcodes have higher tendencies for their reports to contain null values?) However, for the purposes of this blog post, simplicity and just getting an initial idea about what the data says, I’ll be removing null values from the data and keeping track of these losses.</p><h3>Aggregating the Data with Pandas:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cde53b8c8200d5b35e0ad88f47c119d/href">https://medium.com/media/0cde53b8c8200d5b35e0ad88f47c119d/href</a></iframe><p>And the return shows our aggregated data! Now our rows represent a whole zip code instead of an instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*o3VNqcuMBQKho-0K_AlWhA.png" /><figcaption>Note that STOP_LOCATION_ZIP_CODE is now the index, and STOP_ID no longer correlates to ID, as it counts all the IDs that were in that column when it got aggregated. Image by the author.</figcaption></figure><p>But we still have three more things to do before we map.</p><ol><li>We will rename STOP_ID to FREQUENCY for clarity.</li><li>We will cast our zip codes as strings, instead of numeric values. This will allow folium to assign the frequencies of the zip codes to their zip code labels.</li><li>Finally removing those &quot;(null)&quot; values from the data, with the understanding that we are introducing bias into our process, and changing what our map will mean.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94146652e7e24a84d34dc5aecbdcaefb/href">https://medium.com/media/94146652e7e24a84d34dc5aecbdcaefb/href</a></iframe><h3>Making A Choropleth Map:</h3><p>A choropleth is a geographic heat-map. Folium is a python map-making library. We require two basic things to make a choropleth: some boundaries for our map as a GeoJson file, and our data. We’ve prepared our data. I’ll be using the NYC GeoJson file I found here: <a href="https://github.com/fedhere/PUI2015_EC/blob/master/mam1612_EC/nyc-zip-code-tabulation-areas-polygons.geojson">https://github.com/fedhere/PUI2015_EC/blob/master/mam1612_EC/nyc-zip-code-tabulation-areas-polygons.geojson</a>,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47f160f979cec27f151ba305778ebcca/href">https://medium.com/media/47f160f979cec27f151ba305778ebcca/href</a></iframe><p>For more on folium choropleths, I will refer you to the folium documentation directly: <a href="https://python-visualization.github.io/folium/quickstart.html">https://python-visualization.github.io/folium/quickstart.html</a></p><p>It’s true, ideally, I’d be making these maps interactive, so that one could click on each zip code area of the map, and statistics for that zip code would pop, and I’d probably want to make these using tableau, not folium. That would increase usability for sure. There are very many ways to improve on these maps to increase the amount of understanding we can learn from them.</p><p>But I’d argue that just a quick look at the map we’ve been able to make together here, let’s take another look at it really quick, is already telling us more, by a level of magnitude than looking at the original excel file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sruu4rTUaeirotbfYAQC6w.png" /><figcaption>Image by the author.</figcaption></figure><h3>Embracing Questions to Find a Story:</h3><p>While this isn’t the dashboard of our dreams, it’s a fantastic storytelling tool because it already makes us curious with questions to ask of our data:</p><ul><li>How do these zipcodes correlate to the PHSYSICAL_FORCE_RESTRAINED_USED_FLAG column, that we have yet to explore in our data? Are there any zip codes that have a higher percentage of folks being restrained, and are they the zip codes where the most stop-and-frisks happened?</li><li>How do these zipcodes correlate to the ASK_FOR_CONSENT_FLAG column in our data? Were folks across zipcodes asked for their consent at the same rates?</li><li>What about the OFFICER_IN_UNIFORM_FLAG column? What happens to this column when we aggregate by zip code? (Were there more undercover police officers in certain zip codes?)</li><li>How would this map compare to a map with the same parameters from 2019? Were zip codes with a lot of stop-and-frisks happening there the same as the year before?</li><li>Perhaps most notably: “what is happening in East Harlem, East New York, and the Bronx?”</li></ul><p>My enthusiasm about messy data and using choropleths can be summed up to fellow data scientists, thus: <strong><em>just as we use histograms to establish a baseline understanding of frequency data, we can use simple choropleths to establish a baseline of understanding of geographical data.</em></strong> From there, we can use our choropleth to figure out where in the data we’d like to explore more.</p><h3>Consent-Asked:</h3><p>Using similar methods, I was able to get the percentage frequency of consent-asked by zip code. (As an important note, ASK_FOR_CONSENT is not measuring when consent was asked for and given in this data set, only times when consent was asked for. There are very many instances of stop-and-frisks where consent was asked for, consent was not given, and the stop-and-frisk still happened.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DN_LjPDbV3LWRMIwVDcUrw.png" /><figcaption>Using simple division to show the frequency, in percentage, of instances when NYPD asked someone for consent for a stop-and-frisk in 2020, by zip code. Image by the author.</figcaption></figure><p>Using that data I was able to make the following map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zkPNSXor4JMidTzKqzzQZg.png" /><figcaption>This map shows rates of consent-asked for by zip code. It’s great for showing that most zip codes range from 0% to 33% for their consent-asked for rate. Image by the author.</figcaption></figure><p>The above map is great for showing the real range of how consensual stop-and-frisks were in 2020, there are two small zip codes, one in east Queens and one right near Yonkers that had a 100% ask for consent rate. But what if we want to compare the consent-asked rates in these zipcodes to each other in a way that really highlights where most of the data is?</p><h3>Using Quartile Ranges in Choropleth Maps:</h3><p>The following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0d3c536e18dde5488c8459eae298c0d/href">https://medium.com/media/f0d3c536e18dde5488c8459eae298c0d/href</a></iframe><p>Will give us the following map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uukSw_Bl9klQr9Q-zjkWPg.png" /><figcaption>Now, this map is showing the same exact data but is dialed into where most of the data is spread. Image by the author.</figcaption></figure><p>It’s important to point out the legend, which shows that zipcodes must be in the top 75 percentile to have consent-asked rates of higher than 18%. It’s our duty to point this out as storytellers, lest this map is misinterpreted to show that <em>“consent-asked rates are generally high throughout NYC in 2020 since a bunch of the map is dark blue.”</em> What we should or shouldn’t consider a “high” consent-asked rate to be is a topic I won’t discuss in this blog, but intuitively “high” could be associated with “a majority of the time” and we must let our audience know this isn’t the case with this data.</p><h3>Keep Digging to Find a Story:</h3><p>When taking a close look at this map, I again want to point out some neighborhoods of interest:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTrsVSIoMrSFoJ3-3ei7MA.png" /><figcaption>Image by the author.</figcaption></figure><p><strong>The red arrows</strong>: Here are two zip codes in Staten Island that had a relatively low frequency of stop-and-frisks but relatively high consent-asked rates. Bay Ridge shows a similar pattern.</p><p><strong>The purple arrows</strong>: Here is Sunset Park, which had a relatively low consent-asked rate compared to its frequency of stop-and-frisks. East Harlem also looks to be showing that pattern.</p><p><strong>The orange arrow</strong>: This is East New York which had a relatively high frequency of stop-and-frisks along with East Harlem, but here the recorded consent-asked rates are higher than they are in East Harlem.</p><p>I won’t go into every neighborhood on this map, nor will I delve too far into my further analysis, it’s all on my GitHub. However, by looking at some maps and inspecting the data I had the following findings. Please note I’ll be using zip code and neighborhood names interchangeably, for context but this is also not the most accurate.</p><h3>Findings</h3><p><strong>High-Frequency Stop-and-Frisk Zip Codes</strong></p><h4>10029:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*vsber41nGaEDfe_pgDB4zA.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/East_Harlem#/media/File:East_Harlem_Skyline_(48200097101).jpg">Wikipedia</a>.</figcaption></figure><p>In East Harlem, the most common stop-and-frisk zip code, where 220 folks were reported stop-and-frisked in 2020, the consent-asked rate is around 11%, (in between the first and second quartile, so slightly less than the absolute average for this data set.)</p><h4>11207:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WpFyQAkLwOZxfRW2W7k_Hw.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Broadway_Junction_station#/media/File:Broadway_Junction_from_outside_vc.jpg">Wikipedia</a>.</figcaption></figure><p>East New York, the second most common stop-and-frisk zip code, percent-asked is around 20% (putting it in the 75 percentile of most consensual stop-and-frisks as well.) That’s a big difference from East Harlem’s 11% consent-asked rate, for those 169 folks stop-and-frisked.</p><h4>11206:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*N7_Vd74uyo75wQTRYFk3jg.jpeg" /><figcaption>Image from <a href="https://commons.wikimedia.org/wiki/File:Almost_bedstuy.jpg">Wikipedia</a>.</figcaption></figure><p>Bushwick, the seventh most common stop-and-frisk zip code, consent-asked is around 25%, (also putting it in the 75% percentile of most consent-asked zipcodes.) This zip code has one of the highest consent-asked rates among the zipcodes where stop-and-frisks were most frequent.</p><h4>11234:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*XibtUnOWhvVzg2MvAZnZJQ.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Mill_Basin,_Brooklyn#/media/File:Brooklyn,_Mill_Basin_(28118760587).jpg">Wikipedia</a>.</figcaption></figure><p>Flatlands/Bergen Beach, the 14th most common stop-and-frisk zip code, the consent-asked rate is at about 23%, while just next door in Canarsie, zipcode 11236, the consent-asked rate is less than half of that at about 10.5%.</p><h4>11220:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*7V8PrFNOBlNRjCx168rwHw.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Sunset_Park,_Brooklyn#/media/File:SunsetPark.jpg">Wikipedia</a>.</figcaption></figure><p>Sunset Park, the 15th most common stop-and-frisk zip code, consent-asked is the lowest% within the top 25 highest frequency zip codes, at just under 4% asked. This is a very interesting case since all the other zip codes in the top 25 most frequent zip codes have consent-asked rates much higher by about 3 times at least. The next most consent-asked zip code in the top 25 is 10002 (The Lower East Side) with a consent-asked rate of about 9%.</p><p><strong>Low-Frequency Consent-Asked Zip Codes:</strong></p><h4>11232:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*_xXyV0_0_2Gnl-XGAKso9Q.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Industry_City#/media/File:Industry_City_Dec_4,_2018_02.jpg">Wikipedia</a>.</figcaption></figure><p>This is Bush Terminal/Greenwood/Sunset Park. Of the zip codes where consent was <strong>never</strong> asked for in 2020, this is the zip code where stop-and-frisks had the highest stop-and-frisk frequency of 40 people.</p><h4>11375:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*SJAqN4mt4oD5IYguzja4ZQ.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Industry_City#/media/File:Industry_City_Dec_4,_2018_02.jpg">Wikipedia</a>.</figcaption></figure><p>Forrest Hills had a low consent-asked rate too compared to its stop-and-frisk frequency of about 30 people, similar to sunset park.</p><h4>10306:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*vdbXcsjbUKpW9MMf2ZijIg.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Midland_Beach,_Staten_Island#/media/File:Hylan_Bl_Midland_Av_td_(2018-09-19)_10.jpg">Wikipedia</a>.</figcaption></figure><p>This zip code is in Staten Island and includes Midland Beach. It had a surprising 0% consent-asked rate for the 31 people stop-and-frisked there, compared to just next door, in 10309 where 22 people were stop-and-frisked. But when we look to try to see what percentage of the folks in 10309 were asked for consent, we see a percent close to 20%.</p><h4>11210:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*DozKqa6FnFSouMlr9RHemw.jpeg" /><figcaption>Image from <a href="https://en.wikipedia.org/wiki/Flatlands,_Brooklyn#/media/File:EuropeanBUilding1238a.jpg">Wikipedia</a>.</figcaption></figure><p>For the about 30 people who were stop-and-frisked in this part of the Flatlands in 2020, <strong>none</strong> were asked if they consented.</p><h3>In Conclusion:</h3><p>I’d like to challenge my fellow Data Scientists to visualize police data specifically. I know it’s messy. I know it’s unwieldy. But fear not the null value! Instead, keep diligent care of your tracking your missing data, make a visual, and take it one column at a time. You don’t need to tell the whole story behind the data before you start to understand the data. Let your questions guide your process. Say YES to the MESS of police data with choropleths!</p><h4>References:</h4><p>[1] NYCLU, <a href="https://www.nyclu.org/sites/default/files/field_documents/20190314_nyclu_stopfrisk_singles.pdf">Stop-and-Frisk in the de Blasio Era</a> (2019), NYCLU</p><p>[2] E. Ngo, <a href="https://www.ny1.com/nyc/all-boroughs/news/2021/05/25/eric-adams-explains-why-he-supports-stop-and-frisk--when-it-s-used-legally">Eric Adams explains why he supports stop-and-frisk, when it’s used legally</a> (2021), Spectrum News</p><p>[3] New York City Council, <a href="https://council.nyc.gov/budget/wp-content/uploads/sites/54/2021/05/NYPD.pdf">Report to the Committees on Finance and Public Safety on the Fiscal 2022 Executive Budget for the New York Police Department</a> (2021)</p><p>[4] A. Narishkin, K. Hauwanga, H. Gavin, J. Chou, A. Schmitz, <a href="https://www.businessinsider.com/the-real-cost-of-police-nypd-actually-10-billion-year-2020-8">The real cost of the police, and why the NYPD’s actual price tag is $10 billion a year</a> (2020), Business Insider</p><p>[5] NYPD, <a href="https://www1.nyc.gov/site/nypd/stats/reports-analysis/stopfrisk.page">Stop, Question and Frisk Data</a> (2020)</p><p>[6] J. Adams, <a href="https://medium.com/@jade.adams517/using-geopandas-and-folium-to-build-maps-on-python-c51c5aad5f44">A Python Tutorial on Geomapping using Folium and GeoPandas</a> (2022), Medium</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3207e505c649" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-folium-on-police-data-3207e505c649">Using Folium on Police Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why The Rest of Us Need and Deserve Data Science]]></title>
            <link>https://medium.com/codex/why-the-rest-of-us-need-and-deserve-data-science-f67a805f3d2d?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/f67a805f3d2d</guid>
            <category><![CDATA[public-health]]></category>
            <category><![CDATA[racism]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[nonprofit]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 17:21:24 GMT</pubDate>
            <atom:updated>2021-09-12T14:57:52.694Z</atom:updated>
            <content:encoded><![CDATA[<p>When I decided to become a Data Scientist, it wasn’t because I heard about <a href="https://twitter.com/search?q=%23DataScientist&amp;src=typeahead_click">the job title trending on Twitter</a>. It was because I wanted to make my workplace better. I was working as a Respite Worker at the time at a <a href="https://www1.nyc.gov/site/doh/health/health-topics/crisis-emergency-services-respite-centers.page#:~:text=Crisis%20Respite%20Center%20provide%20an,can%20be%20provided%20at%20home.">Mental Health Crisis Center in New York City,</a> and was regularly working with folks who were either homeless, HIV+, disabled, a person of color, or what we refer to as within an <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1490126/">“at-risk” population</a>. For folks not in public health, that term refers to folks who have marginalized identities which put them at increased risk for worse outcomes due to structural injustice. I was regularly collecting demographic information during intake. The organization I worked for would either send this information directly to New York City for their own records, or my organization would store it internally so they could have statistics about the populations they served when applying for grants.</p><p>But I thought our internal statistics could be doing more to help the folks we were serving. Surely, since we were keeping records anyway about the frequency of marginalized populations we served, we could use that thoroughly scrubbed and anonymous data, to give the organization more utility in our services. For example, if we kept yearly statistics about how many homeless folks we served, and how many of those respite stays were from the same clients returning within the year for another stay, as was often the case, and cross-referenced return stays with how often those guests fell into other marginalized categories, we could have some important statistics about how many of our repeat clients were both HIV+ and homeless as compared to their counterparts. Having ready information about the intersections of identity our clients had, as a whole, would have been invaluable in helping my organization know which pieces of training on <a href="https://www.youtube.com/watch?v=rwqnC1fy_zc">intersectionality</a> to prioritize.</p><p>We could even predict the correlation of how someone’s identities would make them more likely to need to come back and hopefully give them more of the services they needed in their first stay to prevent the necessity of them coming back. We could also track how often we were hiring folks whose own identities correlated with the identities of the populations we were serving, since representation matters in <a href="https://npin.cdc.gov/pages/cultural-competence">cultural competency and cultural literacy</a> when delivering care. We could have used Data Science in respite to help monitor when our waitlist to receive services was either too long or too short and prevent some of the bottlenecks with algorithms to help us even out the peaks and valleys in our scheduling. There is almost no end in my imagination about how the use of statistics could be better utilized at respite to make public health provide better outcomes.</p><p>Much to my own surprise, I learned that in Data Science, this is called “business understanding.” I remember laughing when I read that term for the first time since one cannot run a public health service center like a business without harming the very people I wanted most to help. But knowing what I wanted to ask the data and have the data do for me was my introduction to Data Science. And I was enraptured with Data Science’s ability to both tell the truth and be a force for good in a world that needs help.</p><p>(Reasons why the world needs help, are everywhere. The amazing folks at <a href="https://endhomelessness.org/homelessness-in-america/homelessness-statistics/state-of-homelessness-2021/">The National Alliance to End Homelessness</a> made this map on Tableau for public use, a great implementation of big data for good.)</p><figure><a href="https://public.tableau.com/views/Map-2021_16246275433390/Regular?:language=en-US&amp;:embed=y&amp;:embed_code_version=3&amp;:loadOrderID=0&amp;:display_count=y&amp;:origin=viz_share_link"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QfLlcWtQ6N5tfvpjfRs-Bg.png" /></a></figure><h3>Science for Business, and Who Owns It</h3><p>As an aspiring Data Scientist, it’s obvious to me why Data Science is implemented in business. Data Science has become essential for trading stocks, and Data Science itself thrives in doing so. Outside of hedge funds, the giants of industry in modern times, <a href="https://help.netflix.com/en/node/100639">Netflix</a>, Amazon, Spotify, and <a href="https://ai.google/">Google</a>, all — not only implement Data Science in their respective fields but were built on Data Science from the ground up, focusing on their customer algorithms and the statistics behind their customer’s choices. This is fantastic news for Data Science advancement in those enterprises. But what about the implementation of Data Science for public well-being? Would I ever see Data Science being used in public health? Or even to make the world better, more generally?</p><p>When Frederick Banting discovered insulin in 1923, he said <a href="https://bantinghousenhs.ca/2018/12/14/insulin-patent-sold-for-1/">“insulin belongs to the world, not me.”</a> That didn’t go well for Banting’s wishes, due to controversy over who owned the patent for insulin, <a href="https://rightcarealliance.org/actions/insulin/">people are still dying from trying to ration this extremely expensive and life-saving drug.</a> My point here is that, while it has long been debated how patents should work, especially with private data and in the sciences, the benefits of science should be accessible, even at the expense of business from time to time. We may disagree there, and the issue can be circumvented to a point. But even more importantly, industry alone cannot and should not be the driving engine that moves Data Science into the future. The cutting edge of technology in Data Science is too important and too powerful to be exclusive to business. As with any science, the study is furthered only as much as it can be explored. Data Science for business would benefit if it was also building off of advancements in Data Science from non-buisness sectors and vice-versa.</p><p>Data Science, being so new, has massive application in any field you can think of, most of which are yet untapped, underfunded or underutilized. Again, think of respite, think of public health, and how our tax-dollars could be used more efficiently at the very least with a bit of Data Science implementation. Data Science can also help us explore the natural world around us, our <a href="https://www.marinedatascience.co/blog/2019/01/16/oce-package-for-oceanographical-data/">seas</a> and <a href="https://pds.nasa.gov/datasearch/data-search/">skies</a>, our universe, and is, in this very moment, leading the fight against <a href="https://covidtracking.com/">COVID-19</a>, against <a href="https://www.icr.ac.uk/our-research/centres-and-collaborations/centres-at-the-icr/department-of-data-science">cancer</a>, and <a href="https://climateanalytics.org/">climate change</a>.</p><h3>Data Science and the Truth</h3><p>Often lost in the rat race, is the idea that Data Science is the study of information and the public deserves to be informed and told the truth. Data Science is crucial in reporting. The media has a responsibility to the public to offer not only the microcosm of individual interviews but also the macrocosm, a big picture, that big data can offer. Fake news is anecdotal news, is cherry-picking data points to prove a preconceived argument, and it is everywhere.</p><p>We will never know how different the COVID-19 outbreak could have happened in New York City in March 2020, and consequently, the rest of the American continents, if cases weren’t under-reported and hush-hushed in those first few weeks. <a href="https://ag.ny.gov/sites/default/files/2021-nursinghomesreport.pdf">How would then-Governer Cuomo have been able to hide the data about nursing homes, if it was already policy for that data to be public and publically analyzed?</a> How different would our vaccination rates as a country be, if everyone got, in the mail, a single piece of visualized data, transcending our language and cultural barriers, about survival rates with and without the vaccine? How different would our country be with a restored trust in the truth itself?</p><p>The beauty of Data Science in its most poetic form is its ability to tell the truth. John Keats, the romantic poet said:</p><p><a href="https://www.poetryfoundation.org/poems/44477/ode-on-a-grecian-urn">“Beauty is truth, truth beauty, — that is all</a></p><p><a href="https://www.poetryfoundation.org/poems/44477/ode-on-a-grecian-urn">Ye know on earth, and all ye need to know.”</a></p><p>And while Keats was talking about a Grecian Urn, and the timelessness of art, we can still apply his teachings to the realm of information. Because when a Data Scientist performs their job properly, ethically, and artfully, the data practically sings stories and ballads unheard before but not untold. Good Data Science can pierce the echo chamber of social media, and cannot be censored as “critical race theory” in the way fact-based history has been, most recently.</p><p>In more long-standing methods of telling the truth, of representation, journalists have often tried to get a diverse sample of viewpoints, sometimes regardless of the population each viewpoint may represent. This is effective for selling newspapers where everyone can find someone they agree with represented. The method is a viewpoint, a counter viewpoint, and all in-between. This method does have its merits, it’s the logic that our Senate was built upon. But as our Senate needs the House of Representatives as a checks and balances system, traditional sample reporting needs data and Data Science to convey the entire scope of a story. Data Science moreover can do what our legislation structures cannot because Data Science offers rapid flexibility of data, off the binary of point and counter-point, Democrat or Republican, beyond the binary code that it was even built upon. We have not yet discovered our limits to how complex each data point in a data set can be, a much truer expression of my clients at respite, of the human condition as well as the universe itself.</p><p>That is all well, good and flowery but what does it actually mean? Well, have you heard the expression <a href="https://www.nyu.edu/washington-dc/academics/special-programs/welcome-to-chocolate-city/taxation-without-representation.html">“taxation without representation”</a> in regards to Washington DC? It refers to the fact that folks living in Washington, pay full taxes but do not benefit from the rights of representation by legislators that folks in Arlington Virginia do, right next door, for example. With the power of Data Science, however, one can and should advocate for our nation’s <a href="https://www.va.gov/HOMELESS/pit_count.asp">homeless veteran population</a> to lawmakers, especially in Maryland and Virginia, using national data sets since so many veterans are homeless and in Washington. Data Science at its best can give a voice to the voiceless, tell a story, and liberate the truth from under the rug.</p><h3>Data Science and Justice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*X6N5BuKU5Gftk_PyZimAow.png" /><figcaption>Image is borrowed from the <a href="https://www.ctdata.org/blog/black-lives-matter">Connecticut Data Collaborative,</a> I own nothing.</figcaption></figure><p>Specifically, with racial justice organizations doing the critical antiracism work that is so necessary, Data Science can turn from an oppressive force into a force for positive change when telling these unheard stories. It is my dream to work somewhere like The Center for Policing Equity, the makers of our nation’s first <a href="https://policingequity.org/what-we-do/national-justice-database">National Justice Database</a>. This database is implemented by keeping lawmakers informed and is telling a previously ignored tale of systematic racism by police in America on a national scale. Data Science here is fighting for human lives that deserve justice. Ignoring a protest can be handled with spinning public opinion, but ignoring facts is much more difficult to justify.</p><p>Data Science isn’t just important as a tool for social justice, in its ability to do good in the world, but it has an obligation and responsibility specific to anti-racist work. Researchers more qualified than myself, have already discussed thoroughly how <a href="https://www.codedbias.com/">artificial intelligence</a> and <a href="https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/">facial recognition software</a> — Data Science in some of its most sophisticated forms — is specifically harmful to Black, and Indigenous folks as well as other People Of Color. Data Science here has a unique opportunity to shine in its application. The NAACP’s Legal Defense and Education Fund is currently hiring Data Scientists as well, and I am thrilled at the prospect that someone like myself might eventually have an opportunity to do <a href="https://www.naacpldf.org/naacp-mission/">this kind of work.</a></p><p>Data Science is also critical in the mission and survival of non-profit organizations, both for funding and their operations. <a href="https://www.aidshealth.org/">The AIDS Health Organization</a> for example needs to be able to track their aid, their funders, and provide educational materials about HIV-AIDS to any individual at their countless testing sites around the world. Let us not forget about the simple power of visual data, as statistical iconographs become as popular and widespread as memes on the internet. The AIDS Health Organization, much like my job at respite, has countless opportunities where Data Science could increase the overall productivity and success rates of nearly all of its branch’s projects.</p><h3>The World Needs Data Science</h3><p>We know that business needs Data Science. Respites also need Data Science. We all deserve what Data Science can offer. It is my highest aspiration to be a part of the implementation of Data Science into fields I am passionate about, and use Data to make the changes I want to see in the world. So much progress within the field of Data Science rests within realms immediately monetizing its technology. But what about the rest of us? We all need and deserve to benefit from what Data Science can do. The tools that bring us into the future cannot exclusively remain in the profit margins.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f67a805f3d2d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/why-the-rest-of-us-need-and-deserve-data-science-f67a805f3d2d">Why The Rest of Us Need and Deserve Data Science</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Welcome to my Data Science Blog!]]></title>
            <link>https://medium.com/@ls.casanave/welcome-to-my-data-science-blog-c8843a534e6f?source=rss-a25bdaa6a5ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8843a534e6f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Louis Casanave]]></dc:creator>
            <pubDate>Sun, 05 Sep 2021 14:54:56 GMT</pubDate>
            <atom:updated>2021-09-05T15:12:33.961Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks for following, community. I owe a huge shout-out to everyone who helped with my crowdfunding campaign and made my dream of attending a Data Science Bootcamp possible. I truly could not have achieved this feat without your support.</p><p>Soon, I’ll have posts up about my journey through learning Data Science. My next blog post will be published shortly, about why I find Data Science so inspiring and important to our ever-evolving world. Please check back soon, and for now, please enjoy this visualization of my first week in my Data Science Bootcamp. Hope to see you again soon!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*BNGFQlF0xz_7ioMIL5GQ3w.png" /></figure><p>(This graph was made in Python using the Numpy, and Matplotlib libraries with the Seaborn.xkcd_palette.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8843a534e6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>