<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Misha Berrien on Medium]]></title>
        <description><![CDATA[Stories by Misha Berrien on Medium]]></description>
        <link>https://medium.com/@mishaberrien?source=rss-2c1e058445b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*1c6RQL4HuV4uIaln_CCaUg.png</url>
            <title>Stories by Misha Berrien on Medium</title>
            <link>https://medium.com/@mishaberrien?source=rss-2c1e058445b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:09:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mishaberrien/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Structure a Python-Based Data Science Project (a short tutorial for beginners)]]></title>
            <link>https://medium.com/swlh/how-to-structure-a-python-based-data-science-project-a-short-tutorial-for-beginners-7e00bff14f56?source=rss-2c1e058445b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e00bff14f56</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Misha Berrien]]></dc:creator>
            <pubDate>Wed, 24 Jul 2019 17:05:11 GMT</pubDate>
            <atom:updated>2019-09-24T16:01:57.341Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x9g-raNtNtmNFeTqchI9Ew.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Markus Spiske</a> on <a href="https://unsplash.com/search/photos/python-programming?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>The Problem<br></strong>In the middle of my last data science project, I realized that I had a real problem. After staying up late the night before to clean a particularly disorganized dataset, I woke up to discover that much of my hard work had been lost. What’s worse, I had simultaneously been working in a number of different jupyter notebooks, so the code I did have was disordered and unclear. That day, as I struggled to reproduce the code that I had written the night before, I vowed then and there to learn a way to organize my projects in a way that would ensure that this particular road bump would never happen again.</p><p><strong>The Solution<br></strong>After speaking with a number of friends working in the data science field, I discovered that I needed my own data science project template. This post is broken up into three distinct parts:</p><ul><li>First, I’ll discuss three characteristics that make a good data science project template.</li><li>Second, I’ll walk you through a short tutorial — in the form of a mini-project — in order to demonstrate how I use my project template to keep my projects organized.</li><li>Third, I’ll show you how to easily save, reuse and share your project templates.</li></ul><p>Let’s get started!</p><p><em>(Note: I chose the machine learning pipeline template found on the Data Science for Social Good website. You can find the organizations original explanation </em><a href="https://github.com/dssg/hitchhikers-guide/tree/master/sources/curriculum/0_before_you_start/pipelines-and-project-workflow"><em>here</em></a><em> along with many other useful data science resources).</em></p><h3><strong>STEP 1: THINK ABOUT THE PROBLEM YOU’RE TRYING TO SOLVE</strong></h3><p>After taking a look through my last project, I realized that I needed to solve a number of different (but related) problems:</p><ol><li><strong><em>Inconsistent file and variable names</em></strong>: During my early days of coding, I would often return to a project a few days or even weeks later and find it difficult to remember which files to work with. I also found it incredibly difficult to remember the differences between variables and data frames with vague names such as df, df_new, df_new_final and (the dreaded) df_new_final2.</li><li><strong><em>Inconsistent folder structure</em></strong>: In the past, each of my project structures contained a data folder, but that was the extent of my project structure. Even within the data folder there were often a large number of datasets with vague names in different stages between raw and clean.</li><li><strong><em>Wasted Time</em></strong>: I found that I was wasting a lot of time re-running older scripts in order to re-clean my code before moving onto the next step.</li></ol><p>What’s worse, all of these problems were compounded when I worked with other people with different file naming conventions and ideas about how to organize a project.</p><h3><strong>STEP 2: CHOOSE A PROJECT TEMPLATE (no need to reinvent the wheel)</strong></h3><p>After talking with a few data scientist — and doing a lot of independent research — I realized that I needed to come up with a consistent data science project file structure (a project template). There is no reason to reinvent the wheel at this stage. If you google ‘data science project templates’, you’ll find a number of <a href="https://drivendata.github.io/cookiecutter-data-science/#data-is-immutable">great examples</a> on the web. After reading through — and trying out — a number of project structures, I’ve found that the best ones all have a few things in common:</p><ol><li><strong><em>A clear data pipeline (starting with unchanged raw data)</em></strong>: This means that the data is transformed in stages. For example, the raw data is fed into a notebook that transforms and saves an intermediate (cleaned) dataset. The intermediate dataset is then read into a processing notebook where feature engineering takes place. The new dataset (with features) is then saved before moving onto the next stage.</li><li><a href="https://en.wikipedia.org/wiki/Separation_of_concerns"><strong><em>Separation of concerns (seperate files that do different things)</em></strong></a>: In computer science, separation of concerns (SoC) is a design principle for separating a computer program into distinct sections, so that each section addresses a separate concern.</li><li><strong><em>Standard file naming conventions</em></strong>: In order to avoid the terrible df/ df_new/df_new_final scenario listed above, it’s important to build one consistent file naming system before you start coding.</li></ol><p>Ultimately, the idea is that someone who stumbles across your project on github should be able to quickly get a sense of your file structure and be able to reproduce your results from running just a small number of scripts (for more in-depth info on the logic behind the project structure look <a href="https://github.com/dssg/hitchhikers-guide/tree/master/sources/curriculum/0_before_you_start/pipelines-and-project-workflow">here</a>).</p><p>In the end, I chose to follow the project structure laid out by the people at <a href="https://github.com/dssg/hitchhikers-guide/tree/master/sources/curriculum/0_before_you_start/pipelines-and-project-workflow">Data Science for Social Good</a>. Below is a slightly-modified schema of their system.</p><pre>├── README.md          &lt;- The top-level README for developers.<br>├── conf               &lt;- Space for credentials<br>│<br>├── data<br>│   ├── 01_raw         &lt;- Immutable input data<br>│   ├── 02_intermediate&lt;- Cleaned version of raw<br>│   ├── 03_processed   &lt;- Data used to develop models<br>│   ├── 04_models      &lt;- trained models<br>│   ├── 05_model_output&lt;- model output<br>│   └── 06_reporting   &lt;- Reports and input to frontend<br>│<br>├── docs               &lt;- Space for Sphinx documentation<br>│<br>├── notebooks          &lt;- Jupyter notebooks. Naming convention is<br>|                         date YYYYMMDD (for ordering),<br>│                         the creator&#39;s initials, and a short `-` <br>|                         delimited description.<br>│<br>├── references         &lt;- Data dictionaries, manuals, etc. <br>│<br>├── results            &lt;- Final analysis docs.<br>│<br>├── requirements.txt   &lt;- The requirements file for reproducing the <br>|                         analysis environment.<br>│<br>├── .gitignore         &lt;- Avoids uploading data, credentials, <br>|                         outputs, system files etc<br>│<br>└── src                &lt;- Source code for use in this project.<br>    ├── __init__.py    &lt;- Makes src a Python module<br>    │<br>    ├── d00_utils      &lt;- Functions used across the project<br>    │   └── remove_accents.py<br>    │<br>    ├── d01_data       &lt;- Scripts to reading and writing data etc<br>    │   └── load_data.py<br>    │<br>    ├── d02_intermediate&lt;- Scripts to transform data from raw to <br>    |   |                  intermediate<br>    │   └── create_int_payment_data.py<br>    │<br>    ├── d03_processing &lt;- Scripts to turn intermediate data into <br>    |   |                 modelling input<br>    │   └── create_master_table.py<br>    │<br>    ├── d04_modelling  &lt;- Scripts to train models and then use <br>    |   |                  trained models to make predictions. <br>    │   └── train_model.py<br>    │<br>    ├── d05_model_evaluation&lt;- Scripts that analyse model <br>    |   |                      performance and model selection.<br>    │   └── calculate_performance_metrics.py<br>    │    <br>    ├── d06_reporting  &lt;- Scripts to produce reporting tables<br>    │   └── create_rpt_payment_summary.py<br>    │<br>    └── d07_visualisation&lt;- Scripts to create frequently used plots<br>        └── visualise_patient_journey.py</pre><h3><strong>STEP 3: LEARN HOW TO USE YOUR NEW TEMPLATE</strong></h3><p>After learning the basics of organizing a data science project and choosing a project structure, I decided to test it out on a mini-project before implementing it on a larger scale.</p><p><strong><em>Libraries, packages and tools used:</em></strong></p><ul><li><a href="https://docs.python.org/3/library/os.html"><strong><em>OS Module (os)</em></strong></a>: The functions that the OS module provides allows you to interface with the underlying operating system that Python is running on</li><li><a href="https://www.tutorialsteacher.com/python/sys-module"><strong><em>System-specific parameters and functions (sys)</em></strong></a>: This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter.</li><li><strong><em>Jupyter notebooks</em></strong>: All experimental code is kept in the notebooks folder. When completed, code is put into a function and migrated to the appropriate sub-folder under ‘src’ and read back into a new notebook. Below is a sample pipeline for a simple project workflow<em>.</em></li></ul><p><strong><em>Below is a recap of the mini project I ran with a step-by-step guide.</em></strong></p><p><strong><em>Jupyter Notebook 1</em></strong>: <em>20190723-mlb-data-collection-file</em></p><p>I first wrote code to interact with the alpha vantage api and pull the raw dataset. After pulling the data I did two things:</p><p>First, I saved the raw data to the data/01_raw folder in my file system (raw data should never be edited, more on that <a href="https://drivendata.github.io/cookiecutter-data-science/#data-is-immutable">here</a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91af8fca36f3277de98ea4b953cd927f/href">https://medium.com/media/91af8fca36f3277de98ea4b953cd927f/href</a></iframe><p>Second, after testing the code used to download the data in a jupyter notebook, I then wrote a function to pull the dataset seamlessly with just a few inputs. I then saved the new function in a .py file inside the src/d00_utils folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/372a40c83faac76cbe1a9baeffc0c7bb/href">https://medium.com/media/372a40c83faac76cbe1a9baeffc0c7bb/href</a></iframe><p><strong><em>Jupyter Notebook 2</em></strong><em>: 20190723-mlb-feature-processing</em></p><p><strong><em>Note</em></strong>: <em>Since the data came in clean, I will skip the intermediate step of data cleaning (the product of going from raw to clean data would go into the data/02_intermediate folder) and go straight to feature engineering.</em></p><p>I first imported the raw data from the 01_raw data folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/393b6b916ca55d1e0c356615bee1f68f/href">https://medium.com/media/393b6b916ca55d1e0c356615bee1f68f/href</a></iframe><p>Below is the raw data structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1r6raZJTF5bnhnHYiXHoYw.png" /><figcaption>(output from msft_proc.head())</figcaption></figure><p>I then ran a (very naive) regression on the dataset in order to test the project structure. For this mini-project, I’m interested in understanding if the day of the week corresponds to the volume of Microsoft stock being traded. Before running the regression, I first changed the timestamp variable into a dummy variable denoting days of the week.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f95061a345b1f94334c73f7ce4a48a66/href">https://medium.com/media/f95061a345b1f94334c73f7ce4a48a66/href</a></iframe><p>Below is the new dataframe structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-KPfvYWg-IkbLZQ6OJlQA.png" /><figcaption>(output from msft_proc.head())</figcaption></figure><p>Like in the last example, I then wrote a function that contains all steps in the feature engineering process. I then saved the function in a .py file in the src/d00_utils folder. Afterwards, I saved the processed data in the data/03_processed folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a3823cca9151a7b6a0489c71e308682/href">https://medium.com/media/6a3823cca9151a7b6a0489c71e308682/href</a></iframe><p><strong><em>Jupyter Notebook 3</em></strong><em>: 20190723-mlb-model-fitting</em></p><p>I first read the processed dataset into the new jupyter notebook.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/894ef380b7e33ffea16e384b9bb2c61f/href">https://medium.com/media/894ef380b7e33ffea16e384b9bb2c61f/href</a></iframe><p>I then ran a simple regression on the processed dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/771127c9be6517329cc5dcd6ce4d0128/href">https://medium.com/media/771127c9be6517329cc5dcd6ce4d0128/href</a></iframe><p>After running the regression, I then chose a model to save to the data/04_models folder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66b4d2aa042b7f75362fd6fa2fcabd97/href">https://medium.com/media/66b4d2aa042b7f75362fd6fa2fcabd97/href</a></iframe><h3><strong>STEP 4: SAVE YOUR TEMPLATE (and share it with others!)</strong></h3><p>In June, github released a feature called repository templates that makes reusing (and sharing) a project file structure incredibly easy. Checkout their blog post <a href="https://github.blog/2019-06-06-generate-new-repositories-with-repository-templates/">here</a> for more details.</p><p>AND THAT’S IT (for now)! Whichever project file structure you choose, make sure to stick with it and share it with the people you work with. Your future self will thank you.</p><p>Check out my project template on github <a href="https://github.com/mishaberrien/standardize-py">here</a> and try it out for yourself!</p><p>Source: For even more detailed information check out <a href="https://drivendata.github.io/cookiecutter-data-science/#directory-structure">Cookiecutter Data Science</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e00bff14f56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-to-structure-a-python-based-data-science-project-a-short-tutorial-for-beginners-7e00bff14f56">How to Structure a Python-Based Data Science Project (a short tutorial for beginners)</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WHY DATA SCIENCE?]]></title>
            <link>https://medium.com/@mishaberrien/why-data-science-82fb4ecfe455?source=rss-2c1e058445b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/82fb4ecfe455</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Misha Berrien]]></dc:creator>
            <pubDate>Mon, 24 Jun 2019 19:49:10 GMT</pubDate>
            <atom:updated>2019-07-06T12:17:24.268Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Science in Public Policy</h3><p>I recently accepted a Data Science Fellowship at the Flatiron School in Washington D.C. For 4 months, I will be immersed in data science and machine learning, honing my skills in Python and SQL and deepening my understanding of linear algebra and statistical analysis. But, why would a policy analyst need to study data science?</p><p>Over the course of my public policy graduate program at the University of Chicago, I learned not only what public policy is and what good policy looks like, but — more significantly — I discovered a new way to organize information and think about the world.</p><p>The ability to collect and analyze data in an unbiased fashion is important when determining a course of action. Ensuring that you truly understand your data — including the biases that it reflects — becomes imperative when your decisions have the power to impact people’s lives.</p><p>My interest in data science stems from my need to ensure that — as a policy analyst — I have a way to systematically collect, process and analyze information so that I can do the most good while hurting the least number of people.</p><p><strong><em>So, what is public policy?</em></strong></p><p>Public policy — in its simplest form — is a collection of rules that govern our daily lives. A simple(ish) example of a public policy issue is the question of what members of the public are and are not allowed to do in a national park. National parks are for everyone’s enjoyment, but preferred activities vary from person to person. For example, many people enjoy riding ATVs through parks while others enjoy taking peaceful hikes. Many hikers dislikes hearing the sound of ATVs rumbling through the parks and would like the ATVs banned. The question of whether we allow the use of ATVs in national parks is a public policy issue. Whatever decision we make will govern what everyone can do within our national parks system. So, what is the correct decision? It is this question — how to choose the best policy to govern everyone — that we focused on at Harris.</p><p><strong><em>What role does data science play in public policy?</em></strong></p><p>“Social impact, down to a science.” This is the motto at the University of Chicago Harris School of Public Policy. My classmates often debated the merits of our motto. Some criticized the school’s emphasis on statistics and economics to the neglect of political theory and policy courses. According to one classmate in particular, Harris was at best attempting to “teach us empathy through math”, and at worst, trying to turn us into number crunching automatons.</p><p>I always appreciated the motto. Empathy is incredibly important when writing policy, but one cannot rely on empathy alone to decide the best course of action when deciding policy issues. In order to dive a bit deeper into this question, let’s take a look at a very simplified version of a policy issue that is commonly discussed.</p><p><strong><em>Prison Recidivisms Intervention Programs</em></strong></p><p>The difficulties that prisoners face when re-entering the community are well documented. There is a wealth of information on best-practices for helping to ensure that people that are released from prison have both a smooth reintegration into the community and avoid returning to prison.</p><p>Let’s say — for the sake of our example — that a number of policy analysts in the fictional state of Crystalport have taken a look at the research on re-entry programs in other states and have decided to implement a program to help people in their state. Because of limited resources, the state is unable to place all prisoners into reentry programs, so the analysts need to come up with criteria to decide which prisoners will be enrolled in the new program.</p><p>In our case, the policy analysts — in concert with statisticians — have built a model that predicts a “recidivism risk score” (note: this on its own is no small feat). With this model we are able to rank those from most at risk to least at risk of recidivism and separate them into three groups:</p><p><em>1.</em> <em>High Risk: The majority will return to prison without an intervention.</em></p><p><em>2.</em> <em>Medium Risk: More than 50% will return to prison without an intervention.</em></p><p><em>3.</em> <em>Low Risk: A small number will return to prison without an intervention.</em></p><p>Now, the question is who should be enrolled in the program? One common response is that the group at the highest risk for recidivism should receive the intervention. This is a reasonable response from the information that we were provided, but let’s say that the analysts dig a bit deeper into data from programs in other states that have been running these programs for a number of years, and find the additional information provided below:</p><p><em>1.</em> <em>High Risk: The majority of people in this group responds poorly to the re-entry program.</em></p><p><em>2.</em> <em>Medium Risk: The majority of people in this group responds well to the re-entry program.</em></p><p><em>3.</em> <em>Low Risk: This group is unlikely to return to prison even without the intervention.</em></p><p>Now — with the knowledge from the additional data — which group would you say should be enrolled in the reentry program? It looks as if providing the intervention to those in the medium risk group may directly help the greatest number of people avoid returning to prison, while the majority of those in the high-risk group may return to prison regardless of their participation in the reentry program.</p><p><strong><em>Closing Remarks</em></strong></p><p>So, what does this mean for the Crystalport reentry program? Should the state avoid enrolling those prisoners who are at highest risk? It depends, many programs provide services to those that are at highest risk, knowing the outcome numbers may be lower. Some programs measure success purely on the number of people that they successfully keep from returning to prison within a certain number of years. No choice is strictly correct, but, when we are making decisions that will affect people&#39;s lives, it is important that we understand the likely outcome of those choices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82fb4ecfe455" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>