<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eduardo Blancas on Medium]]></title>
        <description><![CDATA[Stories by Eduardo Blancas on Medium]]></description>
        <link>https://medium.com/@edublancas?source=rss-1ed5e037fe0e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Z-mtPCEQOiZzka6sooi5Yw.jpeg</url>
            <title>Stories by Eduardo Blancas on Medium</title>
            <link>https://medium.com/@edublancas?source=rss-1ed5e037fe0e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:20:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@edublancas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop using 0.5 as the threshold for your binary classifier]]></title>
            <link>https://medium.com/data-science/stop-using-0-5-as-the-threshold-for-your-binary-classifier-8d7168290d44?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d7168290d44</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Tue, 29 Nov 2022 19:52:18 GMT</pubDate>
            <atom:updated>2022-11-30T18:02:04.265Z</atom:updated>
            <content:encoded><![CDATA[<h4>Statistics for Machine Learning</h4><h3>Stop Using 0.5 as the Threshold for Your Binary Classifier</h3><h4>Learn how to set the optimal threshold for your Machine Learning model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loloNlP48JLf3VDhM50abQ.png" /><figcaption>Image by author, using image files from flaticon.com</figcaption></figure><p>To produce a binary response, classifiers output a real-valued score that is thresholded. For example, logistic regression outputs a probability (a value between 0.0 and 1.0); and observations with a score equal to or higher than 0.5 produce a positive binary output (many other models use the 0.5 threshold by default).</p><p>However, using the default 0.5 threshold is suboptimal. In this blog post, I&#39;ll show you how you can choose the best threshold from your binary classifier. We&#39;ll be using <a href="https://github.com/ploomber/ploomber?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">Ploomber</a> to execute our experiments in parallel and <a href="https://github.com/ploomber/sklearn-evaluation?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">sklearn-evaluation</a> to generate the plots.</p><blockquote>Hi! My name is Eduardo, and I like writing about all things data science. If you want to keep up-to-date with my content. Follow me on <a href="https://medium.com/@edublancas?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">Medium</a> or <a href="https://twitter.com/edublancas?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">Twitter</a>. Thanks for reading!</blockquote><p>Let’s continue with the example of training a logistic regression. Let’s imagine we’re working on a content moderation system, and our model should flag posts (images, videos, etc.) that contain harmful content; then, a human will take a look and decide whether the content is taken down.</p><p>The following snippet trains our classifier:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7daaa66d693cf233c09841fded81f42/href">https://medium.com/media/e7daaa66d693cf233c09841fded81f42/href</a></iframe><p>Let’s now make predictions on the test set and evaluate performance via a confusion matrix:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b54def0b42f7d5d97a71e19325faef7/href">https://medium.com/media/8b54def0b42f7d5d97a71e19325faef7/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cyLzisZ1SIZ04Vszpo9kAA.png" /><figcaption>Model’s confusion matrix. Image by author.</figcaption></figure><p>A confusion matrix summarizes the performance of our model in four regions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*nkXGIKJfR0awk0zNYOFSmw.png" /><figcaption>Confusion matrix regions. Image by author.</figcaption></figure><p>We want to get as many observations as we can (from the test set) in the upper-left and bottom-right quadrants since those are observations that our model got right. The other quadrants are model mistakes.</p><p>Changing the threshold of our model will change the values in the confusion matrix. In the previous example, we used the clf.predict function, which returns a binary response (i.e., uses 0.5 as threshold); however, we can use the clf.predict_proba function to get the raw probability and use a custom threshold:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fae9105edf34a40d653cd59b710cdd4c/href">https://medium.com/media/fae9105edf34a40d653cd59b710cdd4c/href</a></iframe><p>Let’s now make our classifier a bit more <em>aggressive</em> by setting a lower threshold (i.e., flag more posts as harmful) and create a new confusion matrix:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/207982b8aade0fe631ce459935f1fed5/href">https://medium.com/media/207982b8aade0fe631ce459935f1fed5/href</a></iframe><p>Let’s compare both matrices. The <a href="https://github.com/ploomber/sklearn-evaluation?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">sklearn-evaluation</a> library allows us to do that easily:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77ca8e3c147c3c4232674bc9adc7f9aa/href">https://medium.com/media/77ca8e3c147c3c4232674bc9adc7f9aa/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*h1ZmCzizJxP6K4_mLtXqlg.png" /><figcaption>Combined confusion matrix. Image by author.</figcaption></figure><p>The upper triangles are from our 0.5 threshold, and the lower ones are from the 0.4 threshold. A few things to notice:</p><ul><li>Both models are predicting 0 for the same number of observations (this is a coincidence). 0.5 threshold: (90 + 56 = 146). 0.4 threshold: (78 + 68 = 146)</li><li>Lowering the threshold is causing a <em>few more false nega</em>tives (68 from 56)</li><li>Lowering the threshold is <em>increasing true positives a lot</em> (154 from 92)</li></ul><p>As you can see, the tiny threshold changes hugely affected the confusion matrix. However, we’ve only analyzed two threshold values. Let’s analyze model performance across all values to understand the threshold dynamics better. But before that, let’s define new metrics we’ll use for model evaluation.</p><p>So far, we’ve evaluated our models with absolute numbers. To ease comparison and evaluation, we’ll now define two normalized metrics (they take values between 0.0 and 1.0).</p><p>Precision is the proportion of flagged observations that are events (i.e., posts that our model thinks are harmful, and they are). On the other hand, recall is the proportion of actual events that our model retrieves (i.e., from all the harmful posts, which proportion of them we’re able to detect).</p><p>You can see both definitions graphically in the following diagram (Source: <a href="https://upload.wikimedia.org/wikipedia/commons/2/26/Precisionrecall.svg">Wikipedia</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*85TwAia5OmxfvIcNNG3btA.png" /><figcaption>Precision and recall diagram. Source: Wikipedia.</figcaption></figure><p>Since both precision and recall are proportions, they are on the same zero-to-one scale. So let’s now proceed to run the experiments.</p><h3>Running experiments</h3><p>We’ll obtain the precision, recall, and other statistics along several threshold values to better understand how the threshold affects them. We’ll also repeat the experiment multiple times to measure the variability.</p><p><em>Note: the commands in this section are bash commands. Execute them in a terminal or add the </em><em>%%sh magic if using Jupyter.</em></p><p>To efficiently scale our work, we’ll run our experiments using <a href="https://www.cloud.ploomber.io/signin.html?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">Ploomber Cloud</a>. It allows us to run experiments in parallel and retrieve the results quickly.</p><p>We <a href="https://github.com/ploomber/posts/blob/master/threshold/fit.ipynb?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">created a notebook</a> that fits one model and computes the statistics for several threshold values. We’ll execute the same notebook 20 times in parallel. First, let’s download the notebook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/005fe839dc7838f4f78cc2c3da225c7f/href">https://medium.com/media/005fe839dc7838f4f78cc2c3da225c7f/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf4e9a0e549232ea1a32d790e8649592/href">https://medium.com/media/bf4e9a0e549232ea1a32d790e8649592/href</a></iframe><p>Let’s execute the notebook (the configuration in the notebook file tells Ploomber Cloud to run it 20 times in parallel):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a14375c1faab87b0d323a1a665aa98b7/href">https://medium.com/media/a14375c1faab87b0d323a1a665aa98b7/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32e6be50bdd088c7ae062c1599124ac4/href">https://medium.com/media/32e6be50bdd088c7ae062c1599124ac4/href</a></iframe><p>After a few minutes, we’ll see that our 20 experiments are finished:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b849b08ced1b05033090f2bcfd7054d9/href">https://medium.com/media/b849b08ced1b05033090f2bcfd7054d9/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abab5f0c8cb3d46e5f05070dfeef101d/href">https://medium.com/media/abab5f0c8cb3d46e5f05070dfeef101d/href</a></iframe><p>Let’s download the results from each experiment. The results are stored in .csv files:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/507fbb16b4baed86f400660430da089a/href">https://medium.com/media/507fbb16b4baed86f400660430da089a/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cae1f2485df4e59da7e60b591ff4bba/href">https://medium.com/media/0cae1f2485df4e59da7e60b591ff4bba/href</a></iframe><h3>Visualizing experiment results</h3><p>We’ll now load the results of all experiments and plot them all at once.</p><p>The left scale (zero to one) measures our three metrics: precision, recall, and F1. The F1 score is the harmonic mean between precision and recall, the best value of the F1 score is 1.0 and the worst is 0.0; F1 values both precision and recall equally so you can see it as balancing between the two. If you are working on a use case where both precision and recall are important, maximizing F1 is an approach that can help you optimize your classifier&#39;s threshold.</p><p>We also included a red curve (scale on the right side) showing the number of cases that our model flagged as harmful content. This statistic is relevant because there’s a limit on the number of events we can intervene in many real-world use cases.</p><p>Following our content moderation example, we might have an X number of people looking at the posts flagged by our model, and there&#39;s a limit on how many they can review. Hence, considering the number of flagged cases can help us better choose a threshold: there&#39;s no benefit in finding 10,000 daily cases if we can only review 5,000. And it&#39;d be wasteful for our model to only flag 100 daily cases if we have more capacity than that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5eac4ecf9b98ca06cb27654bd78cada1/href">https://medium.com/media/5eac4ecf9b98ca06cb27654bd78cada1/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d32cba2c7be4b76d10c96c5b5a411b88/href">https://medium.com/media/d32cba2c7be4b76d10c96c5b5a411b88/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6185f3a026df3e7e970a482da2eaf1ba/href">https://medium.com/media/6185f3a026df3e7e970a482da2eaf1ba/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*oQ3AOaTsFgQvlgEZMJgEuQ.png" /><figcaption>Metrics for difference threshold values. Image by author.</figcaption></figure><p>As you can see, when setting low thresholds, we have high recall (we retrieve a large proportion of actually harmful posts) but low precision (there are many non-harmful flagged posts). However, if we increase the threshold, the situation reverses: recall goes down (we missed many harmful posts), but precision is high (most flagged posts are harmful).</p><p>When choosing a threshold for our binary classifier, we must compromise on the precision or recall since no classifier is perfect. So let’s discuss how you can reason about choosing the suitable threshold.</p><h3>Plotting intervals</h3><p>The data gets noisy on the right side (larger threshold values). So, to clean it up a bit, we’ll re-create the plot, but this time, I’ll plot the 2.5%, 50%, and 97.5% percentiles instead of plotting all values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49896cf149573100470c192ad21d54f2/href">https://medium.com/media/49896cf149573100470c192ad21d54f2/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3b9ea2baa43d12324a70a8cd0fe3fd3/href">https://medium.com/media/c3b9ea2baa43d12324a70a8cd0fe3fd3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be5bb014cdc3b86ba54e172dff426a2e/href">https://medium.com/media/be5bb014cdc3b86ba54e172dff426a2e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ae933168f66595c1553ca718707b803/href">https://medium.com/media/1ae933168f66595c1553ca718707b803/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*1z7mNLDCTkGqCeUTVqYANQ.png" /><figcaption>Metrics for difference threshold values. Image by author.</figcaption></figure><h3>Selecting the optimal threshold</h3><p>When choosing a threshold, we can ask ourselves: is it more important to retrieve as many harmful posts as possible (high recall)? Or is it more important to have high certainty that the ones we flag are harmful (high precision)?</p><p>If both are equally important, one common way to optimize under these conditions is to maximize the F-1 score:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2166d50a3addd8de151e2034e4d56ee/href">https://medium.com/media/f2166d50a3addd8de151e2034e4d56ee/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8750f17f5cff481eb3d9d824347a937a/href">https://medium.com/media/8750f17f5cff481eb3d9d824347a937a/href</a></iframe><p>However, it’s hard to decide what to compromise in many situations, so incorporating some constraints will help.</p><p>Say we have ten people reviewing harmful posts, and they can check 5,000 posts together. Let’s see our metrics if we fix our threshold, so it flags approximately 5,000 posts:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21a790fdd75d434ad62fee948a40ffc3/href">https://medium.com/media/21a790fdd75d434ad62fee948a40ffc3/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0479143ac5d7638a573931fd3961a337/href">https://medium.com/media/0479143ac5d7638a573931fd3961a337/href</a></iframe><p>However, when presenting the results, we might want to show a few alternatives: the model performance under the current constraints (5,000 posts) and how better we could do if we increase the team (e.g., by doubling the size).</p><h3>Closing remarks</h3><p>The optimal threshold for your binary classifier is the one that optimizes for business outcomes and takes into account process limitations. With the processes described in this post, you’re better equipped to decide the optimal threshold for your use case.</p><p>If you have questions about this post, feel free to ask in our <a href="https://ploomber.io/community?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">Slack community</a>, which gathers hundreds of data scientists worldwide.</p><p>Also, remember to sign up for <a href="https://www.cloud.ploomber.io/signin.html?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold">Ploomber Cloud</a>! There’s a free tier! It’ll help you quickly scale up your analysis without dealing with complex cloud infrastructure.</p><h3>Packages used</h3><p>Here are the package versions we used when writing this post:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88e6208216e30880b136118d4b40ef24/href">https://medium.com/media/88e6208216e30880b136118d4b40ef24/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/698cca6b5f536752d5681e28d7709ce9/href">https://medium.com/media/698cca6b5f536752d5681e28d7709ce9/href</a></iframe><p><em>Originally published at </em><a href="https://ploomber.io/blog/threshold/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=threshold"><em>ploomber.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d7168290d44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/stop-using-0-5-as-the-threshold-for-your-binary-classifier-8d7168290d44">Stop using 0.5 as the threshold for your binary classifier</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can I Trust My Model’s Probabilities? A Deep Dive into Probability Calibration]]></title>
            <link>https://medium.com/data-science/can-i-trust-my-models-probabilities-a-deep-dive-into-probability-calibration-fc3886cfc677?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc3886cfc677</guid>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Thu, 10 Nov 2022 02:39:36 GMT</pubDate>
            <atom:updated>2022-11-10T02:39:36.276Z</atom:updated>
            <content:encoded><![CDATA[<h4>Statistics for Data Science</h4><h4>A practical guide on probability calibration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*53y4xPBFWjkpskYM" /><figcaption>Photo by <a href="https://unsplash.com/@edge2edgemedia?utm_source=medium&amp;utm_medium=referral">Edge2Edge Media</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Suppose you have a binary classifier and two observations; the model scores them as 0.6 and 0.99, respectively. Is there a higher chance that the sample with the 0.99 score belongs to the positive class? For some models, this is true, but for others it might not.</p><p>This blog post will dive deeply into probability calibration-an essential tool for every data scientist and machine learning engineer. Probability calibration allows us to ensure that higher scores from our model are more likely to belong to the positive class.</p><p>The post will provide reproducible code examples with open-source software so you can run it with your data! We’ll use <a href="https://github.com/ploomber/sklearn-evaluation?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">sklearn-evaluation</a> for plotting and <a href="https://github.com/ploomber/ploomber?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Ploomber</a> to execute our experiments in parallel.</p><blockquote>Hi! My name is Eduardo, and I like writing about all things data science. If you want to keep up-to-date with my content. Follow me on <a href="https://medium.com/@edublancas?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Medium</a> or <a href="https://twitter.com/edublancas?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Twitter</a>. Thanks for reading!</blockquote><h3><strong>What is probability calibration?</strong></h3><p>When training a binary classifier, we’re interested in finding whether a specific observation belongs to the positive class. What <em>positive class</em> means depens on the context. For example, if working on an email filter, it can mean that a particular message is spam. If working on content moderation, it can mean <em>harmful post</em>.</p><p>Using a number in a real-valued range provides more information than Yes/No answer. Fortunately, most binary classifiers can output scores (Note that here I’m using the word <em>scores</em> and not <em>probabilities</em> since the latter has a strict definition).</p><p>Let’s see an example with a Logistic regression:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffef2c634d14d9d272142ac51c26240b/href">https://medium.com/media/ffef2c634d14d9d272142ac51c26240b/href</a></iframe><p>The predict_proba function allows us to output the scores (for logistic regression&#39;s case, this are indeeded probabilities):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8746e1a7f7f7fcc494df4af2a91af353/href">https://medium.com/media/8746e1a7f7f7fcc494df4af2a91af353/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27a8447cc30896da84e6e8972ce18ff7/href">https://medium.com/media/27a8447cc30896da84e6e8972ce18ff7/href</a></iframe><p>Each row in the output represents the probability of belonging to class 0 (first column) or class 1 (second column). As expected, the rows add up to 1.</p><p>Intuitively, we expect a model to give a higher probability when it’s more confident about specific predictions. For example, if the probability of belonging to class 1 is 0.6, we might assume the model isn&#39;t as confident as with one example whose probability estimate is 0.99. This is a property exhibited by well-calibrated models.</p><p>This property is advantageous because it allows us to prioritize interventions. For example, if working on content moderation, we might have a model that classifies content as <em>not harmful</em> or <em>harmful</em>; once we obtain the predictions, we might decide to only ask the review team to check the ones flagged as <em>harmful</em>, and ignore the rest. However, teams have limited capacity, so it’d be better to only pay attention to posts with a high probability of being harmful. To do that, we could score all new posts, take the top N with the highest scores, and then hand over these posts to the review team.</p><p>However, models do not always exhibit this property, so we must ensure our model is well-calibrated if we want to prioritize predictions depending on the output probability.</p><p>Let’s see if our logistic regression is calibrated.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d599e7281c62ada6890af7d216920ed5/href">https://medium.com/media/d599e7281c62ada6890af7d216920ed5/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mfxX65_EexKvFJHa0GW3vw.png" /></figure><p>Let’s now group by probability bin and check the proportion of samples within that bin that belong to the positive class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c49a7094aca4e8e0f21a5f735eec1a2c/href">https://medium.com/media/c49a7094aca4e8e0f21a5f735eec1a2c/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48c704251f90f0338ccee88128e1696d/href">https://medium.com/media/48c704251f90f0338ccee88128e1696d/href</a></iframe><p>We can see that the model is reasonably calibrated. No sample belongs to the positive class for outputs between 0.0 and 0.1. For the rest, the proportion of actual positive class samples is close to the value boundaries. For example, for the ones between 0.3 and 0.4, 29% belong to the positive class. A logistic regression returns well-calibrated probabilities because of its <a href="https://en.wikipedia.org/wiki/Loss_functions_for_classification?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">loss function</a>.</p><p>It’s hard to evaluate the numbers in a table; this is where a calibration curve comes in, allowing us to assess calibration visually.</p><h3>What is a calibration curve?</h3><p>A calibration curve is a graphical representation of a model’s calibration. It allows us to benchmark our model against a target: a perfectly calibrated model.</p><p>A perfectly calibrated model will output a score of 0.1 when it&#39;s 10% confident that the model belongs to the positive class, 0.2 when it&#39;s 20%, and so on. So if we draw this, we&#39;d have a straight line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPeXPIMOE3NoQuxJoIXD0g.png" /><figcaption>A perfectly calibrated model. Image by author.</figcaption></figure><p>Furthermore, a calibration curve allows us to compare several models. For example, if we want to deploy a well-calibrated model into production, we might train several models and then deploy the one that is better calibrated.</p><h3>The notebook</h3><p>We’ll use a notebook to run our experiments and change the model type (e.g., logistic regression, random forest, etc.) and the dataset size. You can see the <a href="https://github.com/ploomber/posts/blob/master/calibration-curve/fit.ipynb?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">source code here</a>.</p><p>The notebook is straightforward: it generates sample data, fits a model, scores out-of-sample predictions, and saves them. After running all the experiments, we’ll download the model’s predictions and use them to plot the calibration curve along with other plots.</p><h3>Running the experiments</h3><p>To accelerate our experimentation, we’ll use <a href="https://www.cloud.ploomber.io/signin.html?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Ploomber Cloud</a>, which allows us to parametrize and run notebooks in parallel.</p><p><em>Note: the commands in this section as bash commands. Run them in a terminal or add the </em><em>%%sh magic if you execute them in Jupyter.</em></p><p>Let’s download the notebook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/149044bccc30f7054791e6978f2dc8eb/href">https://medium.com/media/149044bccc30f7054791e6978f2dc8eb/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb69f97be2abbfaca9ada07a2a0b4164/href">https://medium.com/media/cb69f97be2abbfaca9ada07a2a0b4164/href</a></iframe><p>Now, let’s run our parametrized notebook. This will trigger all our parallel experiments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed9b2ddc6fff58e2065e8184c7b5f12d/href">https://medium.com/media/ed9b2ddc6fff58e2065e8184c7b5f12d/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/106d22f77b0533dcb9669fe3367eac85/href">https://medium.com/media/106d22f77b0533dcb9669fe3367eac85/href</a></iframe><p>After a minute or so, we’ll see that all our 28 experiments are finished executing:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb32659b3700a57de77caceac350d41a/href">https://medium.com/media/bb32659b3700a57de77caceac350d41a/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6285232a0d70c4dedb4cfde592f019ed/href">https://medium.com/media/6285232a0d70c4dedb4cfde592f019ed/href</a></iframe><p>Let’s download the probability estimations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a88b65d89ec4fb453f27c37823c90a0/href">https://medium.com/media/4a88b65d89ec4fb453f27c37823c90a0/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f44b5f22cb5ec5f9c5fbe6cf25d6e69/href">https://medium.com/media/6f44b5f22cb5ec5f9c5fbe6cf25d6e69/href</a></iframe><h3>Loading output</h3><p>Each experiment stores the model’s predictions in a .parquet file. Let&#39;s load the data to generate a data frame with the model type, sample size, and path to the model&#39;s probabilities (as generated by the predict_proba method).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d89e73eec550c619e9eba56a3dd8cd87/href">https://medium.com/media/d89e73eec550c619e9eba56a3dd8cd87/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B691tF1Q0dBB68hccaEoCg.png" /></figure><p>name is the model name. n_samples is the sample size, and path is the path to the output data generated by each experiment.</p><h3>Logistic regression: a well-calibrated model</h3><p>Logistic regression is a special case since it’s well-calibrated by design given that its objective function minimizes the log-loss function.</p><p>Let’s see its calibration curve:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc3c3b62373c636770d4b867fa2df911/href">https://medium.com/media/bc3c3b62373c636770d4b867fa2df911/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db3dc1a28a2c3a08862e9e44a0f34e9e/href">https://medium.com/media/db3dc1a28a2c3a08862e9e44a0f34e9e/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*cuozszJGUS4OPLX9u0UmHg.png" /><figcaption>Logistic regression calibration curve. Image by author.</figcaption></figure><p>You can see that the probability curve closely resembles one of a perfectly calibrated model.</p><h3>The effect of the sample size</h3><p>In the previous section, we showed that logistic regression is designed to produce calibrated probabilities. But beware of the sample size. If you don’t have a large enough training set, the model might not have enough information to calibrate the probabilities. The following plot shows the calibration curves for a logistic regression model as the sample size increases:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4f6aa5bb60f6608dc01573ce6dd3427/href">https://medium.com/media/a4f6aa5bb60f6608dc01573ce6dd3427/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*Yq_uIqwQ9r2tw-1Apn6tpg.png" /><figcaption>Logistic regression calibration curve for different sample sizes. Image by author.</figcaption></figure><p>You can see that with 1,000 samples, the calibration is poor. However, once you pass 10,000 samples, more data does not significantly improve the calibration. Note that this effect depends on the dynamics of your data; you might need more or fewer data in your use case.</p><h3>Non-calibrated estimators</h3><p>While a logistic regression is designed to produce calibrated probabilities, other models do not exhibit this property. Let’s look at the calibration plot for an AdaBoost classifier:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b912ce43e458157e67c56a722bebc0e6/href">https://medium.com/media/b912ce43e458157e67c56a722bebc0e6/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*zqLroApuIMyUeIJ03QOqfQ.png" /><figcaption>Calibration curve for AdaBoost with different sample sizes. Image by author.</figcaption></figure><p>You can see that the calibration curve looks highly distorted: the fraction of positives (y-axis) is far from its corresponding mean predicted value (x-axis); furthermore, the model doesn’t even produce values along the full 0.0 to 1.0 axis.</p><p>Even at a sample size of 1000,000, the curve could be better. In upcoming sections, we’ll see how to address this problem, but for now, remember this: not all models will produce calibrated probabilities by default. In particular, maximum margin methods such as boosting (AdaBoost is one of them), SVMs, and Naive Bayes yield uncalibrated probabilities (Niculescu-Mizil and Caruana, 2005).</p><p>AdaBoost (unlike logistic regression) has a different optimization objective that does not produce calibrated probabilities. However, this does not imply an inaccurate model since classifiers are evaluated by their accuracy when creating a binary response. Let’s compare the performance of both models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/967b3c5d8897018790cfea79c1a7075e/href">https://medium.com/media/967b3c5d8897018790cfea79c1a7075e/href</a></iframe><p>Now we plot and compare the classification metrics. AdaBoost’s metrics are displayed in the upper half of each square, while Logistic regression ones are in the lower half. We’ll see that both models have similar performance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4366f986e40d9c0b1eea7f233a3c08d/href">https://medium.com/media/c4366f986e40d9c0b1eea7f233a3c08d/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7OpxVXP66GCP2xsh1HmrA.png" /><figcaption>AdaBoost and logistic regression metrics comparison. Image by author.</figcaption></figure><h3>The importance of the probabilities distribution</h3><p>Until now, we’ve solely used the calibration curve to judge whether a classifier is calibrated. However, another crucial factor to take into account is the distribution of the model’s predictions. That is, how common or rare score values are.</p><p>Let’s look at the random forest calibration curve:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf0aac2970fbdc9cc36be47d4649b930/href">https://medium.com/media/bf0aac2970fbdc9cc36be47d4649b930/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fttNhDiIB2NdEyJU_6oCDA.png" /><figcaption>Random forest vs logistic regression calibration curve. Image by author.</figcaption></figure><p>The random forest follows a similar pattern as the logistic regression: the larger the sample size, the better the calibration. Random forests are known to provide well-calibrated probabilities (Niculescu-Mizil and Caruana, 2005).</p><p>However, this is only part of the picture. First, let’s look at the distribution of the output probabilities:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f3668ac07329f561f6e66057be68913/href">https://medium.com/media/7f3668ac07329f561f6e66057be68913/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rfbOx0s0hLndOyl_nxTJjQ.png" /><figcaption>Random forest vs logistic regression distribution of probabilities. Image by author.</figcaption></figure><p>We can see that the random forest pushes the probabilities towards 0.0 and 1.0, while the probabilities from the logistic regression are less skewed. While the random forest is calibrated, there aren&#39;t many observations in the 0.2 to 0.8 region. On the other hand, the logistic regression has support all along the 0.0 to 1.0 area.</p><p>An even more extreme example is when using a single tree: we’ll see an even more skewed distribution of probabilities.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e79134d86406983288b52e4bec424d99/href">https://medium.com/media/e79134d86406983288b52e4bec424d99/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*JvyDlSffvzxazUedqGokjg.png" /><figcaption>Decision tree distribution of probabilities. Image by author.</figcaption></figure><p>Let’s look at the probability curve:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/999f328b3b2eb17891385a114a791e89/href">https://medium.com/media/999f328b3b2eb17891385a114a791e89/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*MN-AtKqjRgN0nJQEC87omQ.png" /><figcaption>Decision tree probability curves for different sample sizes. Image by author.</figcaption></figure><p>You can see that the two points we have ( 0.0, and 1.0) are calibrated (they&#39;re reasonably close to the dotted line). However, no more data exists because the model didn&#39;t output probabilities with other values.</p><h3>Calibrating a classifier</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzjuduA4cxiy3Aoek9r_2A.png" /><figcaption>Training/Calibration/Test split. Image by author.</figcaption></figure><p>There are a few techniques to calibrate classifiers. They work by using your model’s uncalibrated predictions as input for training a second model that maps the uncalibrated scores to calibrated probabilities. We must use a new set of observations to fit the second model. Otherwise, we’ll introduce bias in the model.</p><p>There are two widely used methods: Platt’s method and Isotonic regression. Platt’s method is recommended when the data is small. In contrast, Isotonic regression is better when we have enough data to prevent overfitting (Niculescu-Mizil and Caruana, 2005).</p><p>Consider that calibration won’t automatically produce a well-calibrated model. The models whose predictions can be better calibrated are boosted trees, random forests, SVMs, bagged trees, and neural networks (Niculescu-Mizil and Caruana, 2005).</p><p>Remember that calibrating a classifier adds more complexity to your development and deployment process. So before attempting to calibrate a model, ensure there aren’t more straightforward approaches to take such a better data cleaning or using logistic regression.</p><p>Let’s see how we can calibrate a classifier using a train, calibrate, and test split using Platt’s method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ce501911424b9812ca533b43430b533/href">https://medium.com/media/7ce501911424b9812ca533b43430b533/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*22MGbG1m31_LAHvskpJWIQ.png" /><figcaption>Uncalibrated vs calibrated model. Image by author.</figcaption></figure><p>Alternatively, you might use cross-validation and the test fold to evaluate and calibrate the model. Let’s see an example using cross-validation and Isotonic regression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PbhXlXrwqwXnw0k-GKcVaA.png" /><figcaption>Using cross-validation for calibration. Image by author.</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/506810ca85b6661d891df945e2d0ab55/href">https://medium.com/media/506810ca85b6661d891df945e2d0ab55/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*UbwBnVuiDg6T8VdER0Ewhw.png" /><figcaption>Uncalibrated vs calibrated model (using cross-validation). Image by author.</figcaption></figure><h3>Calibrating a multi-class model</h3><p>In the previous section, we discussed methods for calibrating a classifier (Platt’s method and Isotonic regression), which only support binary classification.</p><p>However, calibration methods can be extended to support multiple classes by following the <em>one-vs-all</em> <a href="https://scikit-learn.org/stable/modules/multiclass.html?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve#ovr-classification">strategy</a> as shown in the following example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/105a68333695f27e8a5fff743797a462/href">https://medium.com/media/105a68333695f27e8a5fff743797a462/href</a></iframe><p><strong>Console output (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2wC8B34VknRlBhBozBAlwA.png" /><figcaption>Uncalibrated vs calibrated multi-class model. Image by author.</figcaption></figure><h3>Closing remarks</h3><p>In this blog post, we took a deep dive into probability calibration, a practical tool that can help you develop better predictive models. We also discussed why some models exhibit calibrated predictions without extra steps while others need a second model to calibrate their predictions. Through some simulations, we also demonstrated the sample size’s effect and compared several models’ calibration curves.</p><p>To run our experiments in parallel, we used <a href="https://www.cloud.ploomber.io/signin.html?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Ploomber Cloud</a>, and to generate our evaluation plots, we used <a href="https://github.com/ploomber/sklearn-evaluation?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">sklearn-evaluation</a>. Ploomber Cloud has a free tier, and sklearn-evaluation is open-source, so you can grab <a href="https://github.com/ploomber/posts/blob/master/calibration-curve/fit.ipynb?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">this post in notebook format from here</a>, get an <a href="https://www.cloud.ploomber.io/signin.html?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">API Key</a> and run the code with your data.</p><p>If you have questions, feel free to join our <a href="https://ploomber.io/community?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">community</a>!</p><h3>References</h3><ul><li><a href="https://scikit-learn.org/stable/modules/calibration.html?utm_source=ploomber&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Probability calibration (scikit-learn documentation)</a></li><li><a href="https://scikit-learn.org/stable/modules/calibration.html?utm_source=ploomber&amp;utm_medium=blog&amp;utm_campaign=calibration-curve#calibrating-a-classifier">Calibrating a classifier (scikit-learn documentation)</a></li><li><a href="https://davidrosenberg.github.io/ttml2021/calibration/2.calibration.pdf?utm_source=ploomber&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Probability calibration notes from David S. Rosenberg</a></li><li><a href="https://developers.google.com/machine-learning/crash-course/classification/prediction-bias?utm_source=ploomber&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Classification: Prediction Bias</a></li><li><a href="https://www.cs.cornell.edu/~alexn/papers/calibration.icml05.crc.rev3.pdf?utm_source=ploomber&amp;utm_medium=blog&amp;utm_campaign=calibration-curve">Predicting Good Probabilities With Supervised Learning. Alexandru Niculescu-Mizil and Rich Caruana (2005)</a></li></ul><h3>Packages used</h3><p>Here are the versions we used for the code examples:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27ca3556af140adc8edf7e7c356df835/href">https://medium.com/media/27ca3556af140adc8edf7e7c356df835/href</a></iframe><p><strong>Console output (1/1):</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c55540103ffbaeabe33fa49d0051f2e7/href">https://medium.com/media/c55540103ffbaeabe33fa49d0051f2e7/href</a></iframe><p><em>Originally published at </em><a href="https://ploomber.io/blog/calibration-curve/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=calibration-curve"><em>ploomber.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc3886cfc677" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/can-i-trust-my-models-probabilities-a-deep-dive-into-probability-calibration-fc3886cfc677">Can I Trust My Model’s Probabilities? A Deep Dive into Probability Calibration</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Data Science Platform on AWS: Parallelizing Experiments (Part III)]]></title>
            <link>https://medium.com/data-science/deploying-a-data-science-platform-on-aws-parallelizing-experiments-part-iii-774d972aabba?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/774d972aabba</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 13:34:54 GMT</pubDate>
            <atom:updated>2022-11-01T13:34:54.030Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Science Cloud Infrastructure</h4><h4>A step-by-step guide to deploy a Data Science platform on AWS with open-source software</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bLa41clCN5VGOHFK" /><figcaption>Photo by <a href="https://unsplash.com/@cdr6934?utm_source=medium&amp;utm_medium=referral">Chris Ried</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In our <a href="https://towardsdatascience.com/deploying-a-data-science-platform-on-aws-running-containerized-experiments-part-ii-bef0e22bd8ae">previous post</a>, we configured Amazon ECR to push a Docker image to AWS and configured an S3 bucket to write the output of our Data Science experiments.</p><p>In this final post, we’ll show you how to use <a href="https://github.com/ploomber/ploomber">Ploomber</a> and <a href="https://github.com/ploomber/soopervisor">Soopervisor</a> to create grids of experiments that you can run in parallel on AWS Batch, and how to request resources dynamically (CPUs, RAM, and GPUs).</p><blockquote>Hi! My name is Eduardo, and I like writing about all things MLOps. If you want to keep up-to-date with my content. Follow me on <a href="https://medium.com/@edublancas">Medium</a> or <a href="https://twitter.com/edublancas">Twitter</a>. Thanks for reading!</blockquote><p>This is how our architecture looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y5FpXDk9onfm4UL0pHwCHA.png" /><figcaption>Platform’s architecture. Image by author.</figcaption></figure><h3>Authenticating with the aws CLI</h3><p>We’ll be using the aws CLI again to configure the infrastructure, so ensure you&#39;re authenticated and have enough permissions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/955c78a009bc1f59c3485e3f83edf6ca/href">https://medium.com/media/955c78a009bc1f59c3485e3f83edf6ca/href</a></iframe><h3>Checking Docker</h3><p>We’ll be using Docker for this part, so ensure it’s up and running:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e9456bfd16efadafaf6ba3ef9dd30a6/href">https://medium.com/media/1e9456bfd16efadafaf6ba3ef9dd30a6/href</a></iframe><h3>Creating an Amazon ECR repository</h3><p>First, let’s create another ECR repository to host our Docker image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0a61688ca89682dc894269d3374e9f1/href">https://medium.com/media/e0a61688ca89682dc894269d3374e9f1/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c15047d322d19f3d9fddacc27ccea28/href">https://medium.com/media/2c15047d322d19f3d9fddacc27ccea28/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0e1fee91bf93897138146fb829974cf/href">https://medium.com/media/a0e1fee91bf93897138146fb829974cf/href</a></iframe><p>Assign the REPOSITORY variable to the output of the previous command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18ce3ac2a29aa2e29c815539b7d6aa54/href">https://medium.com/media/18ce3ac2a29aa2e29c815539b7d6aa54/href</a></iframe><h3>Getting sample code</h3><p>We’ll now get a sample project. First, let’s install the required packages.</p><p><em>Note: We recommend you install them in a virtual environment.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/761b202d975649cbf5cdc88d103054df/href">https://medium.com/media/761b202d975649cbf5cdc88d103054df/href</a></iframe><p>Download the example in the grid directory:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/783f8b07f44a346a83986bf89f8d0883/href">https://medium.com/media/783f8b07f44a346a83986bf89f8d0883/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b02512f64d3b9a7f8272cb37604ce0c/href">https://medium.com/media/9b02512f64d3b9a7f8272cb37604ce0c/href</a></iframe><p>This downloaded a full project:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70e8140492fc71e3d9efe27ba19b7f68/href">https://medium.com/media/70e8140492fc71e3d9efe27ba19b7f68/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4749770a531b4841f23ae966cc8b6ec3/href">https://medium.com/media/4749770a531b4841f23ae966cc8b6ec3/href</a></iframe><p>The example we downloaded prepares some data and trains a dozen Machine Learning models in parallel, here’s a graphical representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*ws11FF-1QyH2PzAhKIzVNA.png" /><figcaption>Graphical representation of our workflow. Image by author.</figcaption></figure><p>Let’s look at the pipeline.yaml file, which specifies the tasks in our workflow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec3e2b97f25c71d45bab7efdbace790f/href">https://medium.com/media/ec3e2b97f25c71d45bab7efdbace790f/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbb5b42fa145834ebc9ba40c739dc763/href">https://medium.com/media/cbb5b42fa145834ebc9ba40c739dc763/href</a></iframe><p>The pipeline.yaml is one interface that Ploomber has to describe computational workflows (you can also declare them with Python).</p><p>The tasks section contains five entries, one per task. The first four are Python functions that process some input data ( tasks.raw.get, tasks.features.sepal, tasks.features.petal, tasks.features.features), and the last one is a script that fits a model ( scripts/fit.py).</p><p>Note the last entry is longer because it’s a grid task: it’ll use the same script and execute it multiple times with different parameters. In total, the script will be executed 12 times, but this could be a larger number.</p><p>To learn more about the pipeline.yaml file and Ploomber, <a href="https://docs.ploomber.io/en/latest/api/spec.html">check our documentation</a>.</p><p>Let’s now configure AWS Batch as our cloud environment (Kubernetes, SLURM, and Airflow are supported as well).</p><h3>Configuring the project to run on AWS</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8a180cc524908372bab693d95bc5574/href">https://medium.com/media/a8a180cc524908372bab693d95bc5574/href</a></iframe><p><strong>Output:</strong></p><pre>========================= Loading DAG =========================<br>No pipeline.aws-env.yaml found, looking for pipeline.yaml instead Found /Users/Edu/dev/ploomber.io/raw/ds-platform-part-iii/grid/pipeline.yaml.<br>Loading...<br>Adding /Users/Edu/dev/ploomber.io/raw/ds-platform-part-iii/grid/aws-env/Dockerfile...<br>============================= Done ============================<br>Fill in the configuration in the &#39;aws-env&#39; section in soopervisor.yaml then submit to AWS Batch with: soopervisor export aws-env<br>Environment added, to export it:<br>$ soopervisor export aws-env<br>To force execution of all tasks:<br>$ soopervisor export aws-env --mode force</pre><p>There are a few extra things we need to configure, to facilitate the setup, we created a script that automates these tasks depending on your AWS infrastructure, let’s download it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b2fecadcc30f79dafe72f76293afd77/href">https://medium.com/media/2b2fecadcc30f79dafe72f76293afd77/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7060015a0b47afd033402e334e005619/href">https://medium.com/media/7060015a0b47afd033402e334e005619/href</a></iframe><p>Now, set the values for the AWS Batch job queue and artifacts bucket you want to use. (If in doubt, you might want to revisit the previous tutorials: <a href="https://towardsdatascience.com/deploying-a-data-science-platform-on-aws-setting-up-aws-batch-part-i-da4a7566db7e">Part I</a>, and <a href="https://towardsdatascience.com/deploying-a-data-science-platform-on-aws-running-containerized-experiments-part-ii-bef0e22bd8ae">Part II</a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15298e614c361455f7fe5f36e6a3bbd4/href">https://medium.com/media/15298e614c361455f7fe5f36e6a3bbd4/href</a></iframe><p>Let’s generate the configuration file that specifies the job queue to use and the ECR repository to upload our code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9da768980d459690ccd2b17c9e7c182a/href">https://medium.com/media/9da768980d459690ccd2b17c9e7c182a/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c46147ea6d2e63c80f5a184363682abf/href">https://medium.com/media/c46147ea6d2e63c80f5a184363682abf/href</a></iframe><p>Now, let’s specify the S3 client so the outputs of the pipeline are uploaded to the bucket:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46fffd61bfdced3e72ae8562eab1d5d7/href">https://medium.com/media/46fffd61bfdced3e72ae8562eab1d5d7/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/945d78180b8bc021303fa711e6f84f01/href">https://medium.com/media/945d78180b8bc021303fa711e6f84f01/href</a></iframe><p>Modify the pipeline.yaml so it uses the client we created in the step above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f42f3475d8a014b01a61f8a990228dd7/href">https://medium.com/media/f42f3475d8a014b01a61f8a990228dd7/href</a></iframe><h3>Upload the project to the ECR repository</h3><p>Let’s upload our project to ECR:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cc594df42a370edbea0c58f683fad22/href">https://medium.com/media/0cc594df42a370edbea0c58f683fad22/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43d22ca6ec03917a76508b273a69d2b3/href">https://medium.com/media/43d22ca6ec03917a76508b273a69d2b3/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bf1caf4925d87ef0131094a09f246bc/href">https://medium.com/media/7bf1caf4925d87ef0131094a09f246bc/href</a></iframe><p>Ensure boto3 is installed as part of our project. We need to upload to S3:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca79c958caf6c2d06569242fab34016e/href">https://medium.com/media/ca79c958caf6c2d06569242fab34016e/href</a></iframe><h3>Execute jobs in AWS Batch</h3><p>We’re now ready to schedule our workflow! Let’s use the soopervisor export command to build the Docker image, push it to ECR and schedule the jobs on AWS Batch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c39ba641acca559a1457669ff1575ac7/href">https://medium.com/media/c39ba641acca559a1457669ff1575ac7/href</a></iframe><p>You can monitor execution in the <a href="https://us-east-1.console.aws.amazon.com/batch/home?region=us-east-1#jobs">AWS Batch console</a>. Or use the following command, just ensure you change the job name. The following command retrieves the status of the fit-random-forest-1-gini task:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbc11fdb7a2afb96b7dd460775e35640/href">https://medium.com/media/bbc11fdb7a2afb96b7dd460775e35640/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbb2930011aebaefc0986a79c8e5b03a/href">https://medium.com/media/dbb2930011aebaefc0986a79c8e5b03a/href</a></iframe><p>After a few minutes, all tasks should be executed!</p><h3>Checking output</h3><p>Let’s check the outputs in the S3 bucket:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a3d096dcc4a397aeceb6ec7a4730460/href">https://medium.com/media/7a3d096dcc4a397aeceb6ec7a4730460/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/399b41c6183e600b1f50126550f23adf/href">https://medium.com/media/399b41c6183e600b1f50126550f23adf/href</a></iframe><p>You can see there’s a combination of .pickle files (the trained models), .csv (processed data), and .html (reports generated from the training script).</p><p>Let’s download one of the reports:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e95c504275bcc3d8824676332e17373a/href">https://medium.com/media/e95c504275bcc3d8824676332e17373a/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06ac9535ce797583b7387257f19ae5ff/href">https://medium.com/media/06ac9535ce797583b7387257f19ae5ff/href</a></iframe><p>Open the report.html and you&#39;ll see the outputs of the training script!</p><h3>Requesting more resources</h3><p>Let’s take a look at the grid/soopervisor.yaml file which configures the cloud environment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ea2fc66cf96dc39795349f30b18b3a9/href">https://medium.com/media/9ea2fc66cf96dc39795349f30b18b3a9/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99d98819d66de8e4f65bb1f1325d4670/href">https://medium.com/media/99d98819d66de8e4f65bb1f1325d4670/href</a></iframe><p>The soopervisor.yaml file specifies the backend to use ( aws-batch), the resources to use by default ({memory: 16384, vcpus: 8}), the job queue, region and repository.</p><p>We can add a new section to specify per-task resources, to override the default value:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6210103631a3516e1e52cdfe6222ed8/href">https://medium.com/media/d6210103631a3516e1e52cdfe6222ed8/href</a></iframe><h3>Closing remarks</h3><p>In this final part, we showed how to create multi-step workflows, and how to parametrize a script to create a grid of experiments that can run in parallel. Now you have a scalable infrastructure to run Data Science and Machine Learning experiments!</p><p>If you need help customizing the infrastructure or want to share your feedback, please join our <a href="https://ploomber.io/community">community</a>!</p><p>To keep up-to-date with our content; follow us on <a href="https://twitter.com/ploomber">Twitter</a>, <a href="https://www.linkedin.com/company/ploomber/">LinkedIn</a>, or <a href="https://www.getrevue.co/profile/ploomber">subscribe to our newsletter</a>!</p><h3>Epilogue: Cleaning up the infrastructure</h3><p>Here’s the command you need to run to delete the ECR repository we created on this post. To delete all the infrastructure, revisit the previous tutorials.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6eba1b4b9d3e59d86b2b81848103ff30/href">https://medium.com/media/6eba1b4b9d3e59d86b2b81848103ff30/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bbe2a465b3502b7b251bacd90e9959a/href">https://medium.com/media/5bbe2a465b3502b7b251bacd90e9959a/href</a></iframe><p><em>Originally published at </em><a href="https://ploomber.io/blog/ds-platform-part-iii/"><em>ploomber.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=774d972aabba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploying-a-data-science-platform-on-aws-parallelizing-experiments-part-iii-774d972aabba">Deploying a Data Science Platform on AWS: Parallelizing Experiments (Part III)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Data Science Platform on AWS: Running containerized experiments (Part II)]]></title>
            <link>https://medium.com/data-science/deploying-a-data-science-platform-on-aws-running-containerized-experiments-part-ii-bef0e22bd8ae?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bef0e22bd8ae</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Wed, 26 Oct 2022 14:05:20 GMT</pubDate>
            <atom:updated>2022-10-26T14:05:20.156Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Science Cloud Infrastructure</h4><h4>A step-by-step guide to deploy a Data Science platform on AWS with open-source software</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fp8-TBN_xi0mcAFY" /><figcaption>Photo by <a href="https://unsplash.com/@guibolduc?utm_source=medium&amp;utm_medium=referral">Guillaume Bolduc</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In our <a href="https://towardsdatascience.com/deploying-a-data-science-platform-on-aws-setting-up-aws-batch-part-i-da4a7566db7e">previous post</a>, we saw how to configure AWS Batch and tested our infrastructure by executing a task that spinned up a container, waited for 3 seconds and shut down.</p><p>In this post, we’ll leverage the existing infrastructure, but this time, we’ll execute a more interesting example.</p><p>We’ll ship our code to AWS by building a container and storing it in <a href="https://aws.amazon.com/ecr/">Amazon ECR</a>, a service that allows us to store Docker images.</p><blockquote>If you want to keep up-to-date with my Data Science content. Follow me on <a href="https://medium.com/@edublancas">Medium</a> or <a href="https://twitter.com/edublancas">Twitter</a>. Thanks for reading!</blockquote><h3>Authenticating with the aws CLI</h3><p>We’ll be using the aws CLI again to configure the infrastructure, so ensure you&#39;re authenticated and have enough permissions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8eaf2799986a49da25b2a28b162eaa38/href">https://medium.com/media/8eaf2799986a49da25b2a28b162eaa38/href</a></iframe><h3><strong>Checking Docker</strong></h3><p>We’ll be using Docker for this part, so ensure it’s up and running:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f7b0a5aa036716d66c12c71273e1a6d/href">https://medium.com/media/6f7b0a5aa036716d66c12c71273e1a6d/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af8e10f32a18dd8753e70dc4db5da2a5/href">https://medium.com/media/af8e10f32a18dd8753e70dc4db5da2a5/href</a></iframe><h3><strong>Creating an Amazon ECR repository</strong></h3><p>We first create a repository, which will host our Docker images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bc6e84acd92de2884bbcb9d578ecf0c/href">https://medium.com/media/7bc6e84acd92de2884bbcb9d578ecf0c/href</a></iframe><p>Create ECR repository:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9c4288572e4338ec19c686197f03003/href">https://medium.com/media/a9c4288572e4338ec19c686197f03003/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50e707ee3a59c4f7abf4fafec9b2709c/href">https://medium.com/media/50e707ee3a59c4f7abf4fafec9b2709c/href</a></iframe><p>The command above will print the repository URI, assign it to the next variable since we’ll need it later:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1198297e00a0f2d25a7f24fd93c294f8/href">https://medium.com/media/1198297e00a0f2d25a7f24fd93c294f8/href</a></iframe><h3><strong>Getting some sample code</strong></h3><p>We’ll now use two open source tools ( <a href="https://github.com/ploomber/ploomberhttps://github.com/ploomber/ploomber">Ploomber</a>, and <a href="https://github.com/ploomber/soopervisorhttps://github.com/ploomber/soopervisor">Soopervisor</a>) to write our computational task, generate a Docker image, push it to ECR, and schedule a job in AWS Batch.</p><p>Let’s install the packages:</p><p><em>Note: We recommend you installing them in a virtual environment.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b509ac6223863772dbfc0d8bc4e162cf/href">https://medium.com/media/b509ac6223863772dbfc0d8bc4e162cf/href</a></iframe><p>Let’s get an example. This example trains and evaluates a Machine Learning model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/376dc967987734a0d3d37b98b333c037/href">https://medium.com/media/376dc967987734a0d3d37b98b333c037/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31676fa709a5a7725d09c6737a472838/href">https://medium.com/media/31676fa709a5a7725d09c6737a472838/href</a></iframe><p>Let’s see the files:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b022f94c2aea6ef74bf569c6b76a2728/href">https://medium.com/media/b022f94c2aea6ef74bf569c6b76a2728/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d0ed3fc7701ab26484495a81d1c0d23/href">https://medium.com/media/9d0ed3fc7701ab26484495a81d1c0d23/href</a></iframe><p>The structure is a typical <a href="https://github.com/ploomber/ploomber">Ploomber</a> project. Ploomber allows you to easily organize computational workflows as functions, scripts or notebooks and execute them locally. To learn more check out <a href="https://docs.ploomber.io/">Ploomber’s documentation</a>.</p><p>On the other hand, <a href="https://github.com/ploomber/soopervisor/">Soopervisor</a> allows you to export a Ploomber project and execute it in the cloud.</p><p>The next command will tell Soopervisor to create the necessary files so we can export to AWS Batch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0337621ee11eb4d6981e8fbd1c73ba77/href">https://medium.com/media/0337621ee11eb4d6981e8fbd1c73ba77/href</a></iframe><p><strong>Output:</strong></p><pre>===================Loading DAG===================<br>No pipeline.aws-env.yaml found, looking for pipeline.yaml instead<br>Found /Users/Edu/dev/ploomber.io/raw/ds-platform-part-ii/example/pipeline.yaml.<br>Loading... Adding /Users/Edu/dev/ploomber.io/raw/ds-platform-part-ii/example/aws-env/Dockerfile... <br>===================Done===================<br>Fill in the configuration in the &#39;aws-env&#39; section in soopervisor.yaml then submit to AWS Batch with: soopervisor export aws-env<br>Environment added, to export it: $ soopervisor export aws-env<br>To force execution of all tasks: $ soopervisor export aws-env --mode force</pre><p>soopervisor add will create a soopervisor.yaml file and a aws-batch folder.</p><p>The aws-batch folder contains a Dockerfile (which we need to create a Docker image):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b281f570714525e0c820f8678f6719e9/href">https://medium.com/media/b281f570714525e0c820f8678f6719e9/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c68b83eb7058818560f8ce92caf0b90/href">https://medium.com/media/6c68b83eb7058818560f8ce92caf0b90/href</a></iframe><p>The soopervisor.yaml file contains configuration parameters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fd7d4d09ecf466873ef7c2730a2d9e3/href">https://medium.com/media/1fd7d4d09ecf466873ef7c2730a2d9e3/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c514be17e95182c8e36a63eacaacdbc/href">https://medium.com/media/9c514be17e95182c8e36a63eacaacdbc/href</a></iframe><p>There are a few parameters we have to configure here, we created a small script to generate the configuration file:</p><ul><li>job_queue: the name of your job queue</li><li>aws_region: the region where your AWS Batch infrastructure is located</li><li>repository: the ECR repository URI</li></ul><p>Here are the values for my infrastructure (replace it with yours):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7257b02cd99efdf008e40c4617950339/href">https://medium.com/media/7257b02cd99efdf008e40c4617950339/href</a></iframe><p><em>Note: If you don’t have the job queue name, you can get it from the </em><a href="https://us-east-1.console.aws.amazon.com/batch/home?region=us-east-1#queues"><em>AWS console</em></a><em> (ensure you’re in the right region).</em></p><p>Let’s download a utility script to facilitate creating the configuration files:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/339f6f36c5a14c587dd3f2c9bb703952/href">https://medium.com/media/339f6f36c5a14c587dd3f2c9bb703952/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76eefc70b155945106fa77adc33d916a/href">https://medium.com/media/76eefc70b155945106fa77adc33d916a/href</a></iframe><p>Create the soopervisor.yaml configuration file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11ef2b672e53d7cf5bd6244c2c6aba6c/href">https://medium.com/media/11ef2b672e53d7cf5bd6244c2c6aba6c/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8596ee23a79756e3768aa22a8c91769e/href">https://medium.com/media/8596ee23a79756e3768aa22a8c91769e/href</a></iframe><p>This is how the file looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fd7d4d09ecf466873ef7c2730a2d9e3/href">https://medium.com/media/1fd7d4d09ecf466873ef7c2730a2d9e3/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b6dcda988a03772dfb79f3a487904bd/href">https://medium.com/media/2b6dcda988a03772dfb79f3a487904bd/href</a></iframe><p>Let’s now use soopervisor export to execute the command in AWS Batch. Such command will do a few things for us:</p><ul><li>Build the Docker container</li><li>Push it to the Amazon ECR repository</li><li>Submit the jobs to AWS Batch</li></ul><p>We need to install boto3 since it&#39;s a dependency to submit jobs to AWS Batch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ee7a1e17e42db45712bc8fe7c1abf8c/href">https://medium.com/media/6ee7a1e17e42db45712bc8fe7c1abf8c/href</a></iframe><p>Authenticate with Amazon ECR so we can push images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e47f2bc7d3513592a086718525e9ddb/href">https://medium.com/media/7e47f2bc7d3513592a086718525e9ddb/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/724c43615735343736731ebbad211c37/href">https://medium.com/media/724c43615735343736731ebbad211c37/href</a></iframe><p>Let’s now export the project. Bear in mind that this command will take a few minutes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3044eb039ed835ae3e045db7d2370551/href">https://medium.com/media/3044eb039ed835ae3e045db7d2370551/href</a></iframe><p>If all goes well, you’ll see something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65622d6ea21a2063da11c8c2a045deb7/href">https://medium.com/media/65622d6ea21a2063da11c8c2a045deb7/href</a></iframe><p>If you encounter issues with the soopervisor export command, or are unable to push to ECR, join our <a href="https://ploomber.io/community">community</a> and we&#39;ll help you!</p><p>Once the command finishes execution, the job will be submitted to AWS Batch. Let’s use the aws CLI to list the jobs submitted to the queue:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/401729bdf57c5220f573ac94c320088c/href">https://medium.com/media/401729bdf57c5220f573ac94c320088c/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd38ee3ab01c4a5bd537b05bf0e23a84/href">https://medium.com/media/dd38ee3ab01c4a5bd537b05bf0e23a84/href</a></iframe><p>After a a minute, you’ll see that task shows as SUCCEEDED (it&#39;ll appear as RUNNABLE, STARTING or RUNNING if it hasn&#39;t finished).</p><p>However, there is a catch: AWS Batch ran our code but shortly after, it shut down the EC2 instance, hence, we no longer have access to the output.</p><p>To fix that, we’ll add an S3 client to our project, so all outputs are stored.</p><h3><strong>Creating an S3 bucket to store outputs</strong></h3><p>Let’s first create a bucket in S3. S3 bucket names must be unique, you can run the following snippet in your terminal or choose a unique name and assign it to the BUCKET_NAME variable:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77831aaaa46ab6e6c0b96b9334ace160/href">https://medium.com/media/77831aaaa46ab6e6c0b96b9334ace160/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/983e4bd5388ed56d7a7f7e21115acd30/href">https://medium.com/media/983e4bd5388ed56d7a7f7e21115acd30/href</a></iframe><p>Create bucket:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5279f3b1922e282de811709c987b711b/href">https://medium.com/media/5279f3b1922e282de811709c987b711b/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6d88bfb0be360650a2eda87c3ebac40/href">https://medium.com/media/a6d88bfb0be360650a2eda87c3ebac40/href</a></iframe><h3><strong>Adding a client to our pipeline</strong></h3><p>Ploomber allows us to specify an S3 bucket and it’ll take care of uploading all outputs for us. We only have to create a short file. The generate.py script can create one for us:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71a4f3f33caf9686c50ef11b91959576/href">https://medium.com/media/71a4f3f33caf9686c50ef11b91959576/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f59bead69f9aaca2d4a9576177b25b66/href">https://medium.com/media/f59bead69f9aaca2d4a9576177b25b66/href</a></iframe><p>We need to configure our pipeline.yaml file so it uploads artifacts to S3. Let&#39;s use the generate.py file so it does it for us:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c25d287e4410584dac9ada85e318c0c/href">https://medium.com/media/4c25d287e4410584dac9ada85e318c0c/href</a></iframe><p>Furthermore, let’s add boto3 to our dependencies since we&#39;ll be calling it to upload artifacts to S3:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94c56f814fb36376155e4ab238216888/href">https://medium.com/media/94c56f814fb36376155e4ab238216888/href</a></iframe><h3><strong>Giving AWS Batch permissions to access the bucket</strong></h3><p>Let’s add S3 permissions to our AWS Batch tasks. Generate a policy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebd215cbbc68cc4564777c86c545c2d3/href">https://medium.com/media/ebd215cbbc68cc4564777c86c545c2d3/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c6f0f9adc13b88c8c9f0da05202a6f7/href">https://medium.com/media/9c6f0f9adc13b88c8c9f0da05202a6f7/href</a></iframe><p>Apply it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f54b438f5c48fca619b39da57176a5b6/href">https://medium.com/media/f54b438f5c48fca619b39da57176a5b6/href</a></iframe><h3><strong>Executing the workload</strong></h3><p>We’re now ready to execute our task on AWS Batch!</p><p>Let’s ensure we can push to ECR:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e47f2bc7d3513592a086718525e9ddb/href">https://medium.com/media/7e47f2bc7d3513592a086718525e9ddb/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/724c43615735343736731ebbad211c37/href">https://medium.com/media/724c43615735343736731ebbad211c37/href</a></iframe><p>Submit the task again:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7cadc714ab38ad402c1854f91ffc042/href">https://medium.com/media/c7cadc714ab38ad402c1854f91ffc042/href</a></iframe><p>Note that this time, the soopervisor export command is a lot faster, since it cached our Docker image!</p><p>Let’s check the status of the task:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/208b94347ca382b50a5cd0644f2a930e/href">https://medium.com/media/208b94347ca382b50a5cd0644f2a930e/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7c907202127682e849cdee3639d4071/href">https://medium.com/media/d7c907202127682e849cdee3639d4071/href</a></iframe><p>After a minute, you should see it as SUCCEEDED.</p><p>Check the contents of our bucket, we’ll see the task output (a .parquet file):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b22e090f9a79590948c567dc1696a05a/href">https://medium.com/media/b22e090f9a79590948c567dc1696a05a/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1b04b7670233b35510c0d252273215a/href">https://medium.com/media/c1b04b7670233b35510c0d252273215a/href</a></iframe><h3><strong>Wrapping up</strong></h3><p>In this post, we learned how to upload our code and execute it in AWS Batch via a Docker image. We also configured AWS Batch to read and write an S3 bucket. With this configuration, we can start running Data Science experiments in a scalable way without worrying about maintaining infrastructure!</p><p>In next (and final) post of this series, we’ll see how to easily generate hundreds of experiments and retrieve the results.</p><p>If you want to be the first to know when the final part comes out; follow us on <a href="https://twitter.com/ploomber">Twitter</a>, <a href="https://www.linkedin.com/company/ploomber/">LinkedIn</a>, or <a href="https://www.getrevue.co/profile/ploomber">subscribe to our newsletter</a>!</p><h3><strong>Epilogue: Cleaning up the infrastructure</strong></h3><p>If you wish to delete the infrastructure we created in this post, here are the commands.</p><p>Delete ECR repository:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/641d4d3e3212b8a2ded992adf88efc95/href">https://medium.com/media/641d4d3e3212b8a2ded992adf88efc95/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bbda6f0336c89cec8c64c97476a9fc4/href">https://medium.com/media/6bbda6f0336c89cec8c64c97476a9fc4/href</a></iframe><p>Delete S3 bucket:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3410994561ea4174f34299f4f7a07d8a/href">https://medium.com/media/3410994561ea4174f34299f4f7a07d8a/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f771518fba2da8baea21543a860e014/href">https://medium.com/media/6f771518fba2da8baea21543a860e014/href</a></iframe><p><em>Originally published at </em><a href="https://ploomber.io/blog/ds-platform-part-ii/"><em>ploomber.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bef0e22bd8ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploying-a-data-science-platform-on-aws-running-containerized-experiments-part-ii-bef0e22bd8ae">Deploying a Data Science Platform on AWS: Running containerized experiments (Part II)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Data Science Platform on AWS: Setting Up AWS Batch (Part I)]]></title>
            <link>https://medium.com/data-science/deploying-a-data-science-platform-on-aws-setting-up-aws-batch-part-i-da4a7566db7e?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/da4a7566db7e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Fri, 07 Oct 2022 07:29:17 GMT</pubDate>
            <atom:updated>2022-10-07T07:29:17.737Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Science Cloud Infrastructure</h4><h4>A step-by-step guide to deploy a Data Science platform on AWS with open-source software</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4UoX6133oj_lWsTU" /><figcaption>Your laptop isn’t enough, let’s use the cloud. Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">CHUTTERSNAP</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this series of tutorials, we’ll show you how to deploy a Data Science platform with AWS and open-source software. By the end of the series, you’ll be able to <strong>submit computational jobs to AWS scalable infrastructure with a single command.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*wpquduWTP9QQeG4PiR1xyg.png" /><figcaption>Architecture of the Data Science platform we’ll deploy. Image by author.</figcaption></figure><h3><strong>What is AWS Batch?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*R1cm1Nt7Tigqsaz8Lxsebw.png" /><figcaption>Screenshot of the AWS Batch console, showing our recent jobs. Image by author.</figcaption></figure><p>To implement our platform, we’ll be using several AWS services. However, the central one is AWS Batch.</p><p>AWS Batch is a managed service for computational jobs. It takes care of keeping a queue of jobs, spinning up EC2 instances, running our code and shutting down the instances. It scales up and down depending on how many jobs we submit. It’s a very convenient service that allows us to execute our code in a scalable fashion and to request custom resources for compute-intensive jobs (e.g., instances with many CPUs and large memory) without requiring us to maintain a cluster (no need to use Kubernetes!).</p><p>Let’s get started!</p><h3>Installing the aws CLI</h3><p>The only requirement for this tutorial is to have the AWS command-line interface installed (and <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-prereqs.html">access keys</a> with enough permissions to use the tool). Follow the <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">installation instructions</a>. If you have issues, ask for help in our <a href="https://ploomber.io/community">Slack</a>.</p><p>Verify your installation (ensure you’re running version 2):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c832ee0ef7e5f5c6a1750e7f8f2f1300/href">https://medium.com/media/c832ee0ef7e5f5c6a1750e7f8f2f1300/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/948cb309d2eb979755700b9dc362054a/href">https://medium.com/media/948cb309d2eb979755700b9dc362054a/href</a></iframe><p>Then authenticate with your access keys:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75cf1b290dd5282b9d11459bfa3e0937/href">https://medium.com/media/75cf1b290dd5282b9d11459bfa3e0937/href</a></iframe><h3>Networking configuration</h3><p>We need to create a VPC (Virtual Private Cloud) for the EC2 instances that will run our tasks, this section has all the commands you need for configuring the VPC.</p><p>Note that all AWS accounts come with a default VPC. If you want to use that one, ensure you have the subnet IDs and security group IDs you want to use and skip this section.</p><p>If you neep help, feel free to ask us anything on <a href="https://ploomber.io/community">Slack.</a></p><p>Let’s create a new VPC, and retrieve the VPC ID:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be86b43b2e3678bd7054aa0408d21f75/href">https://medium.com/media/be86b43b2e3678bd7054aa0408d21f75/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ed11ab9ad8b952ff2a079d00f1af3cb/href">https://medium.com/media/6ed11ab9ad8b952ff2a079d00f1af3cb/href</a></iframe><p>Let’s assign the ID to a variable so we can re-use it (replace the ID with yours):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10ad700e2d19f5272af525cd4b135213/href">https://medium.com/media/10ad700e2d19f5272af525cd4b135213/href</a></iframe><p>Now, let’s create a subnet and get the subnet ID:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e53fb1767a7e20743dd772d58dcb5fc/href">https://medium.com/media/2e53fb1767a7e20743dd772d58dcb5fc/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76a7ee0911396fa849962228f142a4dd/href">https://medium.com/media/76a7ee0911396fa849962228f142a4dd/href</a></iframe><p>And assign the ID to a variable (replace the ID with yours):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a367d96360d7f49696103adaecccfaf/href">https://medium.com/media/9a367d96360d7f49696103adaecccfaf/href</a></iframe><p>We need to modify the subnet’s configuration so each instance gets a public IP:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67658cac4e7ecec952a090a87db9e3cd/href">https://medium.com/media/67658cac4e7ecec952a090a87db9e3cd/href</a></iframe><p>Now, let’s configure internet access:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0cffa74bc0326426753e7f581df464e/href">https://medium.com/media/a0cffa74bc0326426753e7f581df464e/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d4758b3ae334b742ddd168637bc6390/href">https://medium.com/media/8d4758b3ae334b742ddd168637bc6390/href</a></iframe><p>Assign the gateway ID to the following variable (replace the ID with yours):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cc0d29f9becdf999aef6609750d1b6d/href">https://medium.com/media/4cc0d29f9becdf999aef6609750d1b6d/href</a></iframe><p>Let’s attach the internet gateway to our VPC:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5da420dfc25fd824c085220345a1fcc3/href">https://medium.com/media/5da420dfc25fd824c085220345a1fcc3/href</a></iframe><p>This <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-subnets-commands-example.html">documentation</a> explains in more detail the commands above.</p><p>Note that allowing internet access to your instances simplifies the networking setup. However, if you don’t want the EC2 instances to have a public IP, you can configure a NAT gateway.</p><p>Let’s now finish configuring the subnet by adding a route table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d69edc851d03a4b7232f791ab31310ac/href">https://medium.com/media/d69edc851d03a4b7232f791ab31310ac/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f308fd3bf51494e3730693049433159/href">https://medium.com/media/2f308fd3bf51494e3730693049433159/href</a></iframe><p>Assign the route table ID (replace the ID with yours):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a43fd9a0e7e19572b01189fe6c6297d/href">https://medium.com/media/3a43fd9a0e7e19572b01189fe6c6297d/href</a></iframe><p>Let’s add a route associated with our internet gateway:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4eb6e11b9309099e3a411b2d26f38c58/href">https://medium.com/media/4eb6e11b9309099e3a411b2d26f38c58/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28b7bc0d4f7e55057498d68e1b700230/href">https://medium.com/media/28b7bc0d4f7e55057498d68e1b700230/href</a></iframe><p>And associate the table to the subnet:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfa615221ffa47193131fa7ce8332bed/href">https://medium.com/media/bfa615221ffa47193131fa7ce8332bed/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80942ce7b44d6afda0971c438d1fb625/href">https://medium.com/media/80942ce7b44d6afda0971c438d1fb625/href</a></iframe><p>Finally, create a security group in our VPC:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09ee074bbce9f6622477e35f0a96a2d9/href">https://medium.com/media/09ee074bbce9f6622477e35f0a96a2d9/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a8f5a367248697ae0f5d03f54f63a0a/href">https://medium.com/media/2a8f5a367248697ae0f5d03f54f63a0a/href</a></iframe><p>And assign the security ID (replace the ID with yours):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/caa39a7d956dd3b9d532c06d92f0f90f/href">https://medium.com/media/caa39a7d956dd3b9d532c06d92f0f90f/href</a></iframe><h3>Configuring permissions</h3><p>We now need to create a role to allow AWS Batch to call ECS (another AWS service).</p><p>Download the configuration file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ec526472c2d681ccc6ae77078b26141/href">https://medium.com/media/2ec526472c2d681ccc6ae77078b26141/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c2ca57513587fad8f0ec31385c1c3fb/href">https://medium.com/media/4c2ca57513587fad8f0ec31385c1c3fb/href</a></iframe><p>Create role:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f8fb8e8eabd9d1951ee85d2582d84f4/href">https://medium.com/media/2f8fb8e8eabd9d1951ee85d2582d84f4/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a581b9a96ad3317f60d4452dffcf9423/href">https://medium.com/media/a581b9a96ad3317f60d4452dffcf9423/href</a></iframe><p>Create instance profile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f99b6844039c451b979bc3e964462019/href">https://medium.com/media/f99b6844039c451b979bc3e964462019/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b3e284e8d49ce077bd41a1bf335df16/href">https://medium.com/media/4b3e284e8d49ce077bd41a1bf335df16/href</a></iframe><p>Add role to instance profile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3313031e88eee30a365b68e902d4f799/href">https://medium.com/media/3313031e88eee30a365b68e902d4f799/href</a></iframe><p>Attach role policy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c89de7ce8c1988029c836a8352abdc8f/href">https://medium.com/media/c89de7ce8c1988029c836a8352abdc8f/href</a></iframe><h3>Configuring the compute environment</h3><p>With networking and permissions configured, we’re now ready to configure the compute environment!</p><p>In AWS Batch, a compute environment determines which instance types to use for our jobs.</p><p>We created a simple script to generate your configuration file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fdea461f88cbb20d3d12c5bf0c514ca/href">https://medium.com/media/3fdea461f88cbb20d3d12c5bf0c514ca/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4473d81bfd53d161386f488c90226e42/href">https://medium.com/media/4473d81bfd53d161386f488c90226e42/href</a></iframe><p>Run the script and pass the subnet and security group IDs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea3ff0048e81ff53e70a8ef196d991d5/href">https://medium.com/media/ea3ff0048e81ff53e70a8ef196d991d5/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22d025fa0239cd6a1052dcb81b4d06da/href">https://medium.com/media/22d025fa0239cd6a1052dcb81b4d06da/href</a></iframe><p>You may also edit the my-compute-env.json file and put your subnet IDs in the subnets list, and your security group IDs in the securityGroupIds list. If you need more customization for your compute environment, join our <a href="https://ploomber.io/community">Slack</a> and we&#39;ll help you.</p><p>Create the compute environment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83a8efd8c83db05142e470ad20e1caae/href">https://medium.com/media/83a8efd8c83db05142e470ad20e1caae/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf517af6e9ecec7a248337f8c480013e/href">https://medium.com/media/cf517af6e9ecec7a248337f8c480013e/href</a></iframe><p>To submit jobs, we need to create a job queue. The queue will receive job requests and route them to the relevant compute environment.</p><p><em>Note: give it a few seconds before running the next command, as the compute environment might take a bit to be created.</em></p><p>Download file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50f6486da1bbd269522307243713ff78/href">https://medium.com/media/50f6486da1bbd269522307243713ff78/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a803e6076d3f6fb0a0bccb124e86d79/href">https://medium.com/media/5a803e6076d3f6fb0a0bccb124e86d79/href</a></iframe><p>Create a job queue:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94e05a24793784c675545a9a201d31fc/href">https://medium.com/media/94e05a24793784c675545a9a201d31fc/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/865c65f35b9b93a84f08cf97c9b9523b/href">https://medium.com/media/865c65f35b9b93a84f08cf97c9b9523b/href</a></iframe><p>Let’s test that everything is working!</p><p>We define an example job that waits for a few seconds and finishes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2e185a7f2e43e2d263ee046fc5fc675/href">https://medium.com/media/f2e185a7f2e43e2d263ee046fc5fc675/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cc9f929a0c9e2fc527c18bb208eab43/href">https://medium.com/media/2cc9f929a0c9e2fc527c18bb208eab43/href</a></iframe><p>Let’s submit a job to the queue:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/780d97b4bb9769c4c51ca8ae81807874/href">https://medium.com/media/780d97b4bb9769c4c51ca8ae81807874/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b07f03a6b649b182668f0ff8c77f783d/href">https://medium.com/media/b07f03a6b649b182668f0ff8c77f783d/href</a></iframe><p>Let’s ensure the job is executed successfully. Copy the jobId printed when executing the command and pass it to the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/332ae39920ae33b44f5734ad519e07a8/href">https://medium.com/media/332ae39920ae33b44f5734ad519e07a8/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a80ea44187b301b376617346ed778c73/href">https://medium.com/media/a80ea44187b301b376617346ed778c73/href</a></iframe><p>The first time you run the above command, you’ll most likely see: RUNNABLE, this is normal.</p><p>AWS Batch spins up new EC2 machines and shut them down after your jobs are done. This is great because it’ll prevent idling machines that keep billing. However, since new machines spin up every time, this introduces some startup time overhead. Wait for a minute or so and run the command again, you should see STARTING, RUNNING, and SUCCEEDED shortly.</p><p>If the job is still stuck in RUNNABLE status after more than a few minutes, ask for help in our <a href="https://ploomber.io/community">community.</a></p><h3><strong>Wrapping up</strong></h3><p>In this blog post, we configured AWS Batch so we can submit computational jobs on demand. There’s no need to maintain a cluster or manually spin up and shut down EC2 instances. You’re only billed for the jobs you submit. Furthermore, AWS Batch is highly scalable, so you can submit as many jobs as you want!</p><p>In the next post, we’ll show you how to submit a custom container job to AWS Batch, and configure an S3 bucket to read input data and write results.</p><p>If you want to be the first to know when the second part comes out; follow us on <a href="https://twitter.com/ploomber">Twitter</a>, <a href="https://www.linkedin.com/company/ploomber/">LinkedIn</a>, or <a href="https://www.getrevue.co/profile/ploomber">subscribe to our newsletter</a>!</p><h3><strong>Epilogue: Cleaning up the infrastructure</strong></h3><p>There’s no billing for using AWS Batch apart from EC2 usage. However, if you want to clean up your environment, follow these steps.</p><p>Disable the AWS Batch queue and compute environments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/616047f506e2499445a4f2e99afa64dc/href">https://medium.com/media/616047f506e2499445a4f2e99afa64dc/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/865c65f35b9b93a84f08cf97c9b9523b/href">https://medium.com/media/865c65f35b9b93a84f08cf97c9b9523b/href</a></iframe><p>Update compute environment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16d8dce2eb33ebaecfb3d89e7e0a5af2/href">https://medium.com/media/16d8dce2eb33ebaecfb3d89e7e0a5af2/href</a></iframe><p><strong>Output:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf517af6e9ecec7a248337f8c480013e/href">https://medium.com/media/cf517af6e9ecec7a248337f8c480013e/href</a></iframe><p>You’ll need to wait 1–2 minutes for the queue and the compute environment to appear as DISABLED.</p><p>Delete the queue and the compute environment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a584da549dcd9b792f2b7a298085626/href">https://medium.com/media/4a584da549dcd9b792f2b7a298085626/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/891978605f6e05fe1523d5a3fa1034c9/href">https://medium.com/media/891978605f6e05fe1523d5a3fa1034c9/href</a></iframe><p>Delete the VPC and its components:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/667974fa1d5587089543cf2219f0f4c6/href">https://medium.com/media/667974fa1d5587089543cf2219f0f4c6/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/419dff901517aee9dc78153c347af809/href">https://medium.com/media/419dff901517aee9dc78153c347af809/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6507a8e6f5b7aceb384e8774f525b68/href">https://medium.com/media/e6507a8e6f5b7aceb384e8774f525b68/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75e3ee54f58ae9cfc37ec66778b9584b/href">https://medium.com/media/75e3ee54f58ae9cfc37ec66778b9584b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e31bb6ed639ffd41cb0d37fec978a5e/href">https://medium.com/media/9e31bb6ed639ffd41cb0d37fec978a5e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2a70faa1c4be52f051a39408f4821d6/href">https://medium.com/media/b2a70faa1c4be52f051a39408f4821d6/href</a></iframe><p>Delete IAM role:</p><pre>aws iam remove-role-from-instance-profile --instance-profile-name \</pre><pre>    ploomber-ecs-instance-role \</pre><pre>    --role-name ploomber-ecs-instance-role</pre><pre><br>aws iam delete-instance-profile --instance-profile-name ploomber-ecs-instance-role</pre><pre>aws iam detach-role-policy --role-name ploomber-ecs-instance-role \<br>    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role</pre><pre>aws iam delete-role --role-name ploomber-ecs-instance-role</pre><h3>About the author</h3><p>Hi! My name is Eduardo, and I like writing about all things data science. If you want to keep up-to-date with my content. Follow me on <a href="https://medium.com/@edublancas">Medium</a> or <a href="https://twitter.com/edublancas">Twitter</a>. Thanks for reading!</p><p><em>Originally published at </em><a href="https://ploomber.io/blog/ds-platform-part-i/"><em>ploomber.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da4a7566db7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploying-a-data-science-platform-on-aws-setting-up-aws-batch-part-i-da4a7566db7e">Deploying a Data Science Platform on AWS: Setting Up AWS Batch (Part I)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips and Tricks to Use Jupyter Notebooks Effectively]]></title>
            <link>https://edublancas.medium.com/tips-and-tricks-to-use-jupyter-notebooks-effectively-91c9329e792e?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/91c9329e792e</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Mon, 08 Aug 2022 21:27:37 GMT</pubDate>
            <atom:updated>2022-08-09T18:10:40.256Z</atom:updated>
            <content:encoded><![CDATA[<h4>A few things to make you 10x more productive with Jupyter.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QozOj8AT8U2oMhoBx5Gh_A.png" /></figure><p>The Jupyter Notebook is a web-based interactive computing platform, and it is usually the first tool we learn about in data science. Most of us start our learning journeys in Jupyter notebooks. They are great for learning, practicing, and experimenting.</p><p>There are several reasons why the Jupyter notebook is a highly popular tool. Here are some of them:</p><ul><li>Being able to see the code and the output together makes it easier to learn and practice.</li><li>It supports Markdown cells which are great for write-ups, preparing reports, and documenting your work.</li><li>In-line outputs including data visualizations are highly useful for exploratory data analysis.</li><li>You can run the code cell-by-cell which expedites the debugging process as well as understanding other people’s code.</li></ul><p>Although we quite often use Jupyter notebooks in our work, we do not make the most out of them and fail to discover its full potential. In this article, we will go over some tips and tricks to make more use of Jupyter notebooks. Some of these are shortcuts that can increase your efficiency.</p><p>Creating a new cell is one of the most frequently done operations while working in a Jupyter notebook so a quick way of doing this is very helpful.</p><ul><li>ESC + A creates a new cell above the current cell</li><li>ESC + B creates a new cell below the current cell</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*P0wg-rHsPenpfHmSud9ygA.png" /></figure><p>One of the great features of Jupyter notebooks is that they maintain the state of execution of each cell. In other words, cell outputs are cached. This is very useful because you do not have to execute a cell each time you want to check its output or results.</p><p>However, some outputs take too much space and make the overall content hard to follow. We can hide a cell output with “ESC + O” and unhide by pressing these keys again.</p><p>If a cell is not needed anymore, you can delete it with “ESC + D + D”.</p><p>Magic commands are built-in for the IPython kernel. They are quite useful for performing a variety of tasks. Magic commands start with the “%” syntax element. Here are some examples that will help you get more productive.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/468eb7b56d72264024c20c4718eab725/href">https://medium.com/media/468eb7b56d72264024c20c4718eab725/href</a></iframe><blockquote><strong><em>Note:</em></strong><em> If you’re looking to improve your Jupyter workflow even more, check out Ploomber’s open-source projects: </em><a href="https://github.com/ploomber/ploomber"><em>Ploomber</em></a><em> for developing modular data pipelines, </em><a href="https://github.com/ploomber/soorgeon"><em>Soorgeon</em></a><em> for refactoring and cleaning, or </em><a href="https://github.com/ploomber/nbsnapshot"><em>nbsnapshot</em></a><em> for notebook testing.</em></blockquote><p>By default, when you execute a cell that returns multiple outputs, only the last output is shown. Here is an example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f6b9029a0666af27ee8728c8d1b60e4/href">https://medium.com/media/6f6b9029a0666af27ee8728c8d1b60e4/href</a></iframe><p>It is possible to see all the outputs but we need to change a setting as below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63a227a68ab708f97a084c8f7348b15b/href">https://medium.com/media/63a227a68ab708f97a084c8f7348b15b/href</a></iframe><p>If we execute the code block above, we will see the output of both variables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3b98d5a7c24d50f71716db2dd53cf79/href">https://medium.com/media/c3b98d5a7c24d50f71716db2dd53cf79/href</a></iframe><p>There are several shortcuts that you can use in Jupyter notebooks. Here are the ones I find quite useful:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*XYmqkS34DyzHDkt3lO58Kw.png" /></figure><ul><li>ESC + M changes a cell to Markdown.</li><li>ESC + UP or ESC + K selects the cell above.</li><li>ESC + DOWN or ESC + J selects the cell below.</li><li>ESC + SHIFT + M merges selected cells. If only one cell is selected, it is merged with the cell below.</li></ul><p>Python has a rich selection of third party libraries, which simplify tasks and speed up development processes. Those libraries typically have a lot of functions and methods. We sometimes can’t remember what a function does exactly or its syntax.</p><p>In such cases, we can learn about the signature and docstring of a function inside the Jupyter notebook. We just need to add ? at the end of the function name. Here is how we can learn about the query function of Pandas library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c764d28936212e3cb5c5e2191db181e/href">https://medium.com/media/8c764d28936212e3cb5c5e2191db181e/href</a></iframe><p>We have learned about some useful tips and tricks for Jupyter notebooks. You do not have to use all of them immediately but you will see they increase your productivity and efficiency once you start using them.</p><blockquote><em>Questions? </em><a href="https://ploomber.io/community"><em>Join our growing community of Jupyter practitioners!</em></a></blockquote><p><em>Originally published at </em><a href="https://ploomber.io/blog/tips-tricks-notebooks/"><em>ploomber.io</em></a><em> by Soner Yildrim. Reposted with permission.</em></p><p><em>Header image credits: </em><a href="https://www.flaticon.com/free-icons/effective"><em>Effective icons created by Pixel perfect — Flaticon</em></a><em> and </em><a href="https://www.flaticon.com/free-icons/notebook"><em>Notebook icons created by mikan933 — Flaticon</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91c9329e792e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Snapshot Testing for Jupyter Notebooks]]></title>
            <link>https://medium.com/data-science/introducing-snapshot-testing-for-jupyter-notebooks-896512971df8?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/896512971df8</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Tue, 05 Jul 2022 19:39:08 GMT</pubDate>
            <atom:updated>2022-07-14T22:56:35.464Z</atom:updated>
            <content:encoded><![CDATA[<h4>Software Engineering for Data Science</h4><h4>nbsnapshot is an open-source package that benchmarks notebook’s outputs to detect issues automatically</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEwFOZgDdxzvjqPpXuhF6Q.png" /><figcaption>Image by author.</figcaption></figure><blockquote>If you want to keep up-to-date with my content. <strong>Follow me on </strong><a href="https://medium.com/@edublancas"><strong>Medium</strong></a><strong> or </strong><a href="https://twitter.com/edublancas"><strong>Twitter</strong></a><strong>.</strong> Thanks for reading!</blockquote><h3>Motivation</h3><p>When analyzing data in a Jupyter notebook, I unconsciously memorize “rules of thumb” to determine if my results are correct. For example, I might print some summary statistics and become skeptical of some outputs if they deviate too much from what I’ve seen historically. For more complex analysis, I often create diagnostic plots (e.g. a histogram) and check them whenever new data arrives.</p><p>I’ve done this semi-manual testing for years, and while talking to other data scientists, I realized this is common; yet, there aren’t tools available to streamline the process. So I started to ask myself if we could automate this testing process. The solution became clear to me while talking to a frontend developer, who introduced me to the idea of <a href="https://jestjs.io/docs/snapshot-testing">snapshot testing</a>.</p><p>Imagine a frontend developer authoring a UI using HTML, CSS, and Javascript. How can the developer test whether a user interface is correct? You could say that some user interfaces are better than others (if they’re easier to use), but the correctness argument does not apply here. So our best shot is to use our previous implementation as a benchmark. For example, we might raise an error if the position of a button has changed unexpectedly.</p><p>We can apply the same idea to notebook testing. Imagine you have a notebook that cleans an input dataset, computes some metric (e.g. percentage of NAs on a given column), and then stores the clean dataset for further analysis:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e080e3bb4bf086a9efd872b4e0c1d85/href">https://medium.com/media/4e080e3bb4bf086a9efd872b4e0c1d85/href</a></iframe><p>Let’s say we run this notebook today. Then, a week goes by, we run the notebook and obtain the same results. Then, two weeks go by, we run the notebook and realize that our metric looks very different from the previous two values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*94fJsmdDGeR-XDtRQUaIZQ.png" /><figcaption>The results from week 3 deviate too much from the first two weeks.</figcaption></figure><p>We don’t know what’s the cause, but we know <em>something occurred</em>. So we proceed to analyze the situation: it could be that the input data changed, or it could be that our code contains some errors. Whatever that is, it’s great that we caught this so we can investigate further.</p><p>The snapshot testing aims to facilitate this process: we’ll automatically benchmark the notebook’s outputs to detect potential issues.</p><h3>Introducing nbsnapshot</h3><p>To the best of our knowledge, <a href="https://github.com/ploomber/nbsnapshot">nbsnapshot</a> is the first implementation of Snapshot Testing for Jupyter notebooks, and we’re stoked to release it!</p><blockquote>Show your support by starring the <a href="https://github.com/ploomber/nbsnapshot">GitHub repository!</a> Thanks!</blockquote><p>The idea is simple, each time we run our notebook, we record the outputs we want to benchmark. Then, once we have at least two historical records, we start comparing new values with historical ones. Finally, we show an error if the new value deviates (by default, it uses three standard deviations as a threshold) from the recorded values. Let’s see a complete example.</p><p>First, let’s import a few packages and declare two functions that we’ll use:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5d6552b94f374c4e13cb2b363bc3d0b/href">https://medium.com/media/c5d6552b94f374c4e13cb2b363bc3d0b/href</a></iframe><p>Download a sample notebook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89f3d2ed3cdfafbbfa689307db91159b/href">https://medium.com/media/89f3d2ed3cdfafbbfa689307db91159b/href</a></iframe><p>Let’s create a fake historical file to simulate that we ran a notebook (with a single cell) 10 times. To generate a fake output, we draw values from a normal distribution centered at 10 (and a standard deviation of 1):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c802bbd41d3c39cfd5b5a9bf0edf9f3/href">https://medium.com/media/1c802bbd41d3c39cfd5b5a9bf0edf9f3/href</a></iframe><p><strong>Console output: (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*7dRbC8SkW39RlBKsV2pzWQ.png" /></figure><p>Let’s run the sample notebook ( constant.ipynb). This notebook only contains a cell that prints the number 10. Since this number is within the boundaries of our fake history, the test passes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f878e3281353daff85f42d8e9ce42faa/href">https://medium.com/media/f878e3281353daff85f42d8e9ce42faa/href</a></iframe><p><strong>Console output: (1/1):</strong></p><pre>Testing: &#39;metric&#39; - OK!</pre><p>Now, let’s overwrite the existing history and replace it with numbers drawn from a normal distribution centered at 0:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df3405ac99b7f9f353a2f6d300d4fd8e/href">https://medium.com/media/df3405ac99b7f9f353a2f6d300d4fd8e/href</a></iframe><p><strong>Console output: (1/1):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*l30KrYv4V2lb-1JeWYRP0Q.png" /></figure><p>Rerun the notebook. This time, the value in the notebook ( 10) deviates too much from the historical records. Hence, the test fails:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f878e3281353daff85f42d8e9ce42faa/href">https://medium.com/media/f878e3281353daff85f42d8e9ce42faa/href</a></iframe><p><strong>Console output: (1/1):</strong></p><pre>Testing &#39;metric&#39; - FAIL! Value is too high (10), expected one between -4.17 and 3.92</pre><p>That’s it! This simple approach lets us detect issues in our notebooks without extra effort!</p><h3>Use cases</h3><p>Here are two important use cases where snapshot testing is instrumental.</p><p><strong>Monitoring data quality</strong></p><p>Suppose you have a notebook that ingests data every week. You can use snapshot testing to measure the quality of the ingested data and throw an error if it suddenly degrades. For example, you might have a cell that prints the percentage of NAs in a column named age:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*Y4OeuA5L8LxoaBYH5a6lvg.png" /><figcaption>nbsnapshot will throw an error on Friday, since the value is unexpected.</figcaption></figure><p>You can see in the table above that from Monday to Thursday, the percentage of NAs deviated a little from 1%, but on Friday, it skyrocketed to 5%, so something must have happened.</p><p>Another example test would be to count the number of observations in the age column with negative values. This test will allow you to detect whenever corrupted values such as -2, or -10 enter the notebook.</p><p><strong>Monitoring model training performance</strong></p><p>Imagine you have a notebook that uses a training dataset as input and outputs a trained model. You may use snapshot testing to throw an error if the model performance metric (e.g. accuracy) degrades suddenly. Snapshot tests can help you detect problems such as corrupted data, model misconfiguration, or data <a href="https://en.wikipedia.org/wiki/Leakage_(machine_learning)">leakage</a>.</p><p>For example, let’s say you’ve been training models, and suddenly, the model accuracy drops:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*d91Kp1nDpHyL9v-X7qwLbA.png" /><figcaption>nbsnapshot will throw an error on the fifth experiment since performance dropped significantly.</figcaption></figure><p>Knowing about this sudden change will allow you to investigate further to find out what happened.</p><h3>Using Snapshot Testing for Continuous Integration</h3><p>To test effectively, we recommend you implement snapshot testing in a Continuous Integration manner; this implies running the tests whenever you modify the source code. For example, you might use <a href="https://github.com/features/actions">GitHub Actions</a> and run the following command on each commit to your project:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a596b72eea3e2b6b0ecb7ee4a8009df3/href">https://medium.com/media/a596b72eea3e2b6b0ecb7ee4a8009df3/href</a></iframe><p>This command will run the test whenever you push changes to your repository.</p><h3>Considerations</h3><p>nbsnapshot supports checking multiple cells in the same notebook; however, consider that the more tests you have, the higher the chance that any of your checks will fail due to a <a href="https://en.wikipedia.org/wiki/Multiple_comparisons_problem">false positive</a>. So we recommend only testing a couple of essential metrics. In cases where your outputs are deterministic, you can check as many cells as you want.</p><p>Snapshot Testing requires the historical results to be updated manually. For example, let’s say you’ve been training a model and the historical accuracy has been around 0.80 for the last ten experiments; then, you realize that those experiments had data leakage. After fixing the problem, the accuracy drops to 0.70. Hence, if you run a snapshot test, it will fail. However, in this case, the error is in the historical records, so you should delete them and replace them with the new value of 0.70. Doing this clean-up requires extra effort, but it’s worth the time savings that snapshot testing provides in the long run.</p><h3>Closing Remarks</h3><p>In this blog post, we introduced Snapshot Testing for Jupyter notebooks, a simple idea that can greatly improve the workflow of Data Scientists. Please give <a href="https://github.com/ploomber/nbsnapshot">nbsnapshot</a> a try. The library is new so let us know if you have any feature requests! A perfect complement for nbsnapshot is <a href="https://github.com/ploomber/ploomber">Ploomber</a>, our open-source framework for orchestrating notebook-based pipelines.</p><blockquote>Hi! My name is Eduardo! I like writing about all things data science and contributing to open-source projects. If you want to keep up-to-date with my content. <strong>Follow me on </strong><a href="https://medium.com/@edublancas"><strong>Medium</strong></a><strong> or </strong><a href="https://twitter.com/edublancas"><strong>Twitter</strong></a><strong>.</strong> Thanks for reading!</blockquote><p><em>Originally published at </em><a href="https://ploomber.io/blog/snapshot-testing/"><em>ploomber.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=896512971df8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/introducing-snapshot-testing-for-jupyter-notebooks-896512971df8">Introducing Snapshot Testing for Jupyter Notebooks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Jupyter to Kubernetes: Refactoring and Deploying Notebooks Using Open-Source Tools]]></title>
            <link>https://medium.com/data-science/from-jupyter-to-kubernetes-refactoring-and-deploying-notebooks-using-open-source-tools-19f99585e923?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/19f99585e923</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Thu, 23 Jun 2022 16:08:16 GMT</pubDate>
            <atom:updated>2022-07-04T22:39:28.646Z</atom:updated>
            <content:encoded><![CDATA[<h4>Software Engineering For Data Science</h4><h4>A step-by-step guide to going from a messy notebook to a pipeline running in Kubernetes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D4k68s7aQyu6lpNr" /><figcaption>Photo by <a href="https://unsplash.com/@mjessier?utm_source=medium&amp;utm_medium=referral">Myriam Jessier</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Notebooks are great for rapid iterations and prototyping but quickly get messy. After working on a notebook, my code becomes difficult to manage and unsuitable for deployment. In production, code organization is essential for maintainability (it’s much easier to improve and debug organized code than a long, messy notebook).</p><p>In this post, I’ll describe how you can <strong>use our open-source tools to cover the entire life cycle of a Data Science project</strong>: starting from a messy notebook until you have that code running in production. Let’s get started!</p><p>The first step is to clean up our notebook with automated tools; then, we’ll automatically refactor our monolithic notebook into a modular pipeline with soorgeon; after that, we&#39;ll test that our pipeline runs; and, finally, we&#39;ll deploy our pipeline to Kubernetes. The main benefit of this workflow is that all steps are fully automated, so we can return to Jupyter, iterate (or fix bugs), and deploy again effortlessly.</p><h3>Cleaning up the notebook</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-NAz3skVjKRWibihiqjqA.png" /><figcaption>Image by author.</figcaption></figure><p>The interactivity of notebooks makes it simple to try out new ideas, but it also yields messy code. While exploring data, we often rush to write code without considering readability. Lucky for us, there are tools like <a href="https://github.com/PyCQA/isort">isort</a> and <a href="https://github.com/psf/black">black</a> which allow us to easily re-format our code to improve readability. Unfortunately, these tools only work with .py files; however, soorgeon enable us to run them on notebook files (.ipynb):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b88de89128eda5668c2e11d0c02c6bc3/href">https://medium.com/media/b88de89128eda5668c2e11d0c02c6bc3/href</a></iframe><p><em>Note: If you need an example notebook to try these commands, here’s one:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f14ac2afafcb0e33b44fbd0f308754f1/href">https://medium.com/media/f14ac2afafcb0e33b44fbd0f308754f1/href</a></iframe><p>Check out the image at the beginning of this section: I introduced some extra whitespace on the left notebook. However, after applying soorgeon clean (picture on the right), we see that the extra whitespace went away. So now we can focus on writing code and apply soorgeon clean to use auto-formatting easily!</p><h3>Refactoring the notebook</h3><p>Creating analysis on a single notebook is convenient: we can move around sections and edit them easily; however, this has many drawbacks: it’s hard to collaborate and test. Organizing our analysis in multiple files will allow us to define clear boundaries so multiple pipelines can work in the project without getting into each other’s ways.</p><p>The process of going from a single notebook to a modular pipeline is time-consuming and error-prone; fortunately, soorgeon can do the heavy lifting for us:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/005acb0f3d07e9c501e39fd55e74c0cf/href">https://medium.com/media/005acb0f3d07e9c501e39fd55e74c0cf/href</a></iframe><p>Upon refactoring, we’ll see a bunch of new files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*gEllld7miWi1VPPnYA03cA.png" /><figcaption>Image by author.</figcaption></figure><p>Ploomber turns our notebook into a modularized project automatically! It generates a README.md with basic instructions and a requirements.txt (extracting package names from import statements). Furthermore, it creates a tasks/ directory with a few .ipynb files; these files come from the original notebook sections separated by markdown headings. soorgeon refactor figures out which sections depend on which ones.</p><p>If you prefer to export .py files; you can pass the --file-format option:</p><pre>soorgeon refactor nb.ipynb --file-format py</pre><p>The tasks/ directory will have .py files this time:</p><pre>  ├── README.md <br>  ├── nb.ipynb<br>  ├── pipeline.yaml<br>  ├── requirements.txt<br>  └── tasks<br>      ├── clean.py<br>      ├── linear-regression.py<br>      ├── load.py<br>      ├── random-forest-regressor.py<br>      └── train-test-split.py</pre><p>soorgeon uses Markdown headings to determine how many output tasks to generate. In our case, there are five of them. Then, soorgeon analyzes the code to resolve the dependencies among sections and adds the necessary code to pass outputs to the each task.</p><p>For example, our “Train test split” section creates a variables X, y, X_train, X_test, y_train, and y_test; and the last four variables are used by the &quot;Linear regression&quot; section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*1nAAd9crpGSfp8Hth5M4RA.png" /><figcaption>Image by author.</figcaption></figure><p>By determining input and output variables, soorgeon determines that the &quot;Linear regression&quot; section depends on the &quot;Train test split&quot; section. Furthermore, the &quot;Random Forest Regressor&quot; section also depends on the &quot;Train test split&quot; since it also uses the variables generated by the &quot;Train test split&quot; section. With this information, soorgeon builds the dependency graph.</p><h3>Testing our pipeline</h3><p>Now it’s time to ensure that our modular pipeline runs correctly. To do so, we’ll use the second package in our toolbox: ploomber. Ploomber allows us to develop and execute our pipelines locally.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9bc2a30d43309f2889cf2ccf7489849/href">https://medium.com/media/e9bc2a30d43309f2889cf2ccf7489849/href</a></iframe><p>This is the output:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/861364682dc19617a816af5c921ff8ca/href">https://medium.com/media/861364682dc19617a816af5c921ff8ca/href</a></iframe><p>ploomber offers a lot of tools to manage our pipeline; for example, we can generate a plot:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f700603e99362eeab683c1d58ba135c/href">https://medium.com/media/4f700603e99362eeab683c1d58ba135c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*0ad1cpwxaFcRIcXWTQ0OUQ.png" /><figcaption>Image by author.</figcaption></figure><p>We can see the dependency graph; there are three serial tasks: load, clean, and train-test-split. After them, we see two independent tasks: linear-regression, and random-forest-regressor. The advantage of modularizing our work is that members of our team can work independently, we can <a href="https://docs.ploomber.io/en/latest/user-guide/testing.html">test tasks</a> in isolation, and run independent tasks in <a href="https://docs.ploomber.io/en/latest/api/_modules/executors/ploomber.executors.Parallel.html">parallel</a>. With ploomber we can keep developing the pipeline with Jupyter until we&#39;re ready to deploy!</p><h3>Deployment</h3><p>To keep things simple, you may deploy your Ploomber pipeline with <a href="https://ploomber.io/blog/cron/">cron</a>, and run ploomber build on a schedule. However, in some cases, you may want to leverage existing infrastructure. We got you covered! With soopervisor, you can export your pipeline to <a href="https://soopervisor.readthedocs.io/en/latest/tutorials/airflow.html">Airflow</a>, <a href="https://soopervisor.readthedocs.io/en/latest/tutorials/aws-batch.html">AWS Batch</a>, <a href="https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html">Kubernetes</a>, <a href="https://soopervisor.readthedocs.io/en/latest/tutorials/slurm.html">SLURM</a>, or <a href="https://soopervisor.readthedocs.io/en/latest/tutorials/kubeflow.html">Kubeflow</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfea9868aed7f04bfb717d3857aeacba/href">https://medium.com/media/cfea9868aed7f04bfb717d3857aeacba/href</a></iframe><p>soopervisor add adds some files to our project, like a preconfigured Dockerfile (which we can modify if we want to). On the other hand soopervisor export takes our existing pipeline and exports it to Argo Workflows so we can run it on Kubernetes.</p><p>By changing the --backend argument in the soopervisor add command, you can switch to other supported platforms. Alternatively, you may sign up for <a href="https://docs.ploomber.io/en/latest/cloud/cloud-execution.html">free cloud service</a>, which allows you to run your notebooks in the cloud with one command.</p><h3>Final remarks</h3><p>Notebook cleaning and refactoring are time-consuming and error-prone, and we are developing tools to make this process a breeze. In this blog post, we went from having a monolithic notebook until we had a modular pipeline running in production-all of that in an automated way using open-source tools. So please let us know what features you’d like to see. <a href="https://ploomber.io/community">Join our community</a> and share your thoughts!</p><blockquote>Hi! My name is Eduardo, and I like writing about all things data science. If you want to keep up-to-date with my content. Follow me on <a href="https://medium.com/@edublancas">Medium</a> or <a href="https://twitter.com/edublancas">Twitter</a>. Thanks for reading!</blockquote><p><em>Originally published at </em><a href="https://ploomber.io/blog/notebook-to-kubernetes/"><em>ploomber.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19f99585e923" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/from-jupyter-to-kubernetes-refactoring-and-deploying-notebooks-using-open-source-tools-19f99585e923">From Jupyter to Kubernetes: Refactoring and Deploying Notebooks Using Open-Source Tools</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyze and plot 5.5M records in 20s with BigQuery and Ploomber]]></title>
            <link>https://medium.com/data-science/analyze-and-plot-5-5m-records-in-20s-with-bigquery-and-ploomber-8bf7646c031?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bf7646c031</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Mon, 23 May 2022 13:57:02 GMT</pubDate>
            <atom:updated>2022-05-23T13:57:02.543Z</atom:updated>
            <content:encoded><![CDATA[<h4>Software Engineering for Data Science</h4><h4>Develop scalable pipelines on Google Cloud using open-source software</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YY-EH2EF2ZTLH1xhp4YpnQ.png" /><figcaption>Image by author.</figcaption></figure><p>This tutorial will show how you can use Google Cloud and Ploomber to develop a scalable and production-ready pipeline.</p><p>We’ll use Google BigQuery (data warehouse) and Cloud Storage to show how we can transform big datasets with ease using SQL, plot the results with Python, and store the results in the cloud. Thanks to BigQuery scalability (we’ll use a dataset with 5.5M records!) and Ploomber’s convenience, <strong>the entire process from importing the data to the summary report is on the cloud takes less than 20 seconds!</strong></p><h3>Introduction</h3><p>Before we begin, I’ll quickly go over two Google Cloud services we use for this project. <a href="https://en.wikipedia.org/wiki/BigQuery">Google BigQuery</a> is a serverless data warehouse that allows us to analyze data at scale. In simpler terms, we can store massive datasets and query using SQL without managing servers. On the other hand, <a href="https://en.wikipedia.org/wiki/Google_Cloud_Storage">Google Cloud Storage</a> is a storage service; it is the equivalent service to Amazon S3.</p><p>Since our analysis comprises SQL and Python, we use <a href="https://github.com/ploomber/ploomber">Ploomber</a>, an open-source framework to write maintainable pipelines. It abstracts all the details, so we focus on writing the SQL and Python scripts.</p><p>Finally, the data. We’ll be using a public dataset that contains statistics of people’s names in the US over time. The dataset contains 5.5M records. Here’s what it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*td8bHLyclw91p8f3ShlM7w.png" /><figcaption>Sample data. Image by author.</figcaption></figure><p>Let’s now take a look at the pipeline’s architecture!</p><h3>Architecture overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*GdmcvKf7KGxBsqU4LpkefA.png" /><figcaption>Pipeline architecture overview. Image by author.</figcaption></figure><p>The first step is the create-table.sql script; such script runs a CREATE TABLE statement to copy a public dataset. create-view.sql and create-materialized-view.sql use the existing table and generate a view and a materialized view (their purpose is to show how we can create other types of SQL relations, we don&#39;t use the outputs).</p><p>The dump-table.sql queries the existing table, and it dumps the results into a local file. Then, the plot.py Python script uses the local data file, generates a plot, and uploads it in HTML format to Cloud Storage. The whole process may seem intimidating, but Ploomber makes this straightforward!</p><p>Let’s now configure the cloud services we’ll use!</p><h3>Setup</h3><p>We need to create a bucket in Cloud Storage and a dataset in BigQuery; the following sections explain how to do so.</p><h4>Cloud Storage</h4><p>Go to the <a href="https://console.cloud.google.com/storage">Cloud Storage</a> console (select a project or create a new one, if needed) and create a new bucket (you may use an existing one if you prefer so). In our case, we’ll create a bucket “ploomber-bucket” under the project “ploomber”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*w1NlnkKbbgdCmy3F6BkuAg.png" /><figcaption>Create a bucket in Cloud Storage. Image by author.</figcaption></figure><p>Then, enter a name (in our case “ploomber-bucket”), and click on “CREATE”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*mwXAUOYibRNg76Kku_iXWA.png" /><figcaption>Confirm creation of the Cloud Storage bucket. Image by author.</figcaption></figure><p>Let’s now configure BigQuery.</p><h4>BigQuery</h4><p>Go to the <a href="https://console.cloud.google.com/bigquery">BigQuery</a> console and create a dataset. To do so, click on the three stacked dots next to your project’s name and then click on “Create dataset”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*chXfC8aAqhczBAaqFfv_ng.png" /><figcaption>Create a dataset in BigQuery. Image by author.</figcaption></figure><p>Now, enter “my_dataset” as the Dataset ID and “us” in <em>Data location</em> (location is important since we’ll be using a public dataset located in such region), then click on “CREATE DATASET”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*FijBTH3vPg-0k646IXaPEA.png" /><figcaption>Confirm BigQuery dataset details. Image by author.</figcaption></figure><p>Google Cloud is ready now! Let’s now configure our local environment.</p><h3>Local setup</h3><p>First, let’s authenticate so we can make API calls to Google Cloud. Ensure you authenticate with an account that has enough permissions in the project to use BigQuery and Cloud Storage:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65c2fad529d90138b771b1f6e1c3c352/href">https://medium.com/media/65c2fad529d90138b771b1f6e1c3c352/href</a></iframe><p>If you have trouble, check <a href="https://cloud.google.com/sdk/gcloud/reference/auth">the docs.</a></p><p>Now, let’s install Ploomber to get the code example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/972f42019b22f930f3cbf54ab3baf32d/href">https://medium.com/media/972f42019b22f930f3cbf54ab3baf32d/href</a></iframe><p>Let’s now review the structure of the project.</p><h3>Project structure</h3><ul><li>pipeline.yaml Pipeline declaration</li><li>clients.py Functions to create BigQuery and Cloud Storage clients</li><li>requirements.txt Python dependencies</li><li>sql/ SQL scripts (executed in BigQuery)</li><li>scripts/ Python scripts (executed locally, outputs uploaded to Cloud Storage)</li></ul><p>You can look at the files in detail <a href="https://github.com/ploomber/projects/tree/master/templates/google-cloud">here.</a> For this tutorial, I’ll quickly mention a few crucial details.</p><p>pipeline.yaml is the central file in this project; Ploomber uses this file to assemble your pipeline and run it, here&#39;s what it looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1a856eb543460af7d385638fdeaed06/href">https://medium.com/media/f1a856eb543460af7d385638fdeaed06/href</a></iframe><p>Each task in the pipeline.yaml file contains two elements: the source code we want to execute and the product. You can see that we have a few SQL scripts that generate tables and views. However, the dump-table.sql creates a.parquet file. This indicates to Ploomber that it should download the results instead of storing them on BigQuery. Finally, the plot.py script contains an.html output; Ploomber will automatically run the script and store the results in the HTML file.</p><p>You might be wondering how the order is determined. Ploomber extracts references from the source code itself; for example, the create-view.sql depends on create-table.sql. If we look at the code, we&#39;ll see the reference:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cbb39107566ab9e2646bfb57104ed08/href">https://medium.com/media/0cbb39107566ab9e2646bfb57104ed08/href</a></iframe><p>There is a placeholder {{ upstream[&quot;create-table&quot;] }}, this indicates that we should run create-table.sql first. At runtime, Ploomber will replace the placeholder for the table name. We also have a second placeholder{{ product }}, this will be replaced by the value in the pipeline.yaml file.</p><p>That’s it for the pipeline.yaml. Let&#39;s review the clients.py file.</p><h3>Configure clients.py</h3><p>clients.py contains a function that returns clients to communicate with BigQuery and Cloud Storage.</p><p>For example, this is how we connect to BigQuery:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd247e5d3acacf88d7bda5d443fa9a05/href">https://medium.com/media/fd247e5d3acacf88d7bda5d443fa9a05/href</a></iframe><p>Note that we’re returning a ploomber.clients.DBAPIClient object. Ploomber wraps BigQuery&#39;s connector, so it works with other databases.</p><p>Secondly, we configure the Cloud Storage client:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee53546089d7e47ebac7cfd0286f53a7/href">https://medium.com/media/ee53546089d7e47ebac7cfd0286f53a7/href</a></iframe><p>Here, we return a ploomber.clients.GCloudStorageClient object (ensure that the bucket_name matches yours!)</p><p>Great, we’re ready to run the pipeline!</p><h3>Running the pipeline</h3><p>Ensure your terminal is open in the gcloud folder and execute the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d5e9c9889e3def69bad948df32bebc0/href">https://medium.com/media/7d5e9c9889e3def69bad948df32bebc0/href</a></iframe><p>After a few seconds of running the ploomber build command, you should see something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dc9889e4e7a558c0fcfa8e5ddbf254a/href">https://medium.com/media/1dc9889e4e7a558c0fcfa8e5ddbf254a/href</a></iframe><p>If you get an error, you most likely have a misconfiguration. Please send us a <a href="https://ploomber.io/community">message on Slack</a> so we can help you fix it!</p><p>If you open the <a href="https://console.cloud.google.com/bigquery">BigQuery</a> console, you’ll see the new tables and views:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*yCmLzR7nItUR9PhKWsMFcQ.png" /><figcaption>Tables and views created in BigQuery after running our pipeline. Image by author.</figcaption></figure><p>In the <a href="https://console.cloud.google.com/storage">Cloud Storage</a> console, you’ll see the HTML report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*d97VhS9x7nVKwg8M3ptd_Q.png" /><figcaption>Artifacts generated in Cloud Storage after running our pipeline. Image by author.</figcaption></figure><p>Finally, if you download and open the HTML file, you’ll see the plot!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*1pEQXfba-WFqivnO329d0w.png" /><figcaption>Plot generated after aggregating our 5.5M rows dataset! Image by author.</figcaption></figure><h3>Incremental builds</h3><p>It may take a few iterations to get the final analysis. This process involves making small changes to your code and rerunning the workflow. Ploomber can keep track of source code changes to accelerate iterations, so it only executes outdated scripts next time. Enabling this requires a bit of extra configuration since Ploomber needs to store your pipeline’s metadata, we already pre-configured the same workflow, so it stores the metadata in a SQLite database, you can run it with the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8edde597cdd0fbec952e21b3c8bc2841/href">https://medium.com/media/8edde597cdd0fbec952e21b3c8bc2841/href</a></iframe><p>If you run the command another time, you’ll see that it skips all tasks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eae002fecf3841d22223b47be9924099/href">https://medium.com/media/eae002fecf3841d22223b47be9924099/href</a></iframe><p>Now try changing plot.py and rerun the pipeline; you&#39;ll see that it skips most tasks!</p><h3>Closing remarks</h3><p>This tutorial showed how to build maintainable and scalable data analysis pipelines on Google Cloud. Ploomber has many other features to simplify your workflow such as parametrization (store outputs on a different each time you run the pipeline), task parallelization, and even cloud execution (in case you need more power to run your Python scripts!).</p><p>Check out our <a href="https://docs.ploomber.io/">documentation</a> to learn more, and don’t hesitate to <a href="https://ploomber.io/community">send us any questions!</a></p><p><em>Originally published at </em><a href="https://ploomber.io/blog/google-cloud/"><em>ploomber.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bf7646c031" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analyze-and-plot-5-5m-records-in-20s-with-bigquery-and-ploomber-8bf7646c031">Analyze and plot 5.5M records in 20s with BigQuery and Ploomber</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Gentle Introduction to Open-Source Contributions]]></title>
            <link>https://medium.com/data-science/a-gentle-introduction-to-open-source-contributions-4b2a8f1aa5cd?source=rss-1ed5e037fe0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b2a8f1aa5cd</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Eduardo Blancas]]></dc:creator>
            <pubDate>Tue, 10 May 2022 12:52:16 GMT</pubDate>
            <atom:updated>2022-05-10T12:52:16.231Z</atom:updated>
            <content:encoded><![CDATA[<h4>Open-source 101</h4><h4>A step-by-step guide for contributing to an open-source project</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CEEd6sd5vbLJem_H" /><figcaption>Photo by <a href="https://unsplash.com/@yancymin?utm_source=medium&amp;utm_medium=referral">Yancy Min</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As <a href="https://github.com/ploomber/ploomber">Ploomber</a> becomes the go-to tool for building pipelines from Jupyter, we’ve seen an increasing interest from our community to contribute to the project. We’re extremely honored to have such a vibrant community willing to go the extra mile, especially because many of our contributors are first-time open-source contributors. To help new open-source contributors, we’ve prepared this in-depth guide that covers all you have to know to make your first contribution. Let’s get started!</p><p><em>Note that this guide focuses on contributing to Python projects hosted on GitHub, but the guidelines are broadly applicable.</em></p><h3>Finding a GitHub issue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*YNSP1lbLL0CzQsEIQmFvAA.gif" /><figcaption>Image by author.</figcaption></figure><p>Head to the issues section once you’ve chosen a project to contribute. For example, click here to see <a href="https://github.com/ploomber/ploomber/issues">Ploomber’s issues</a>. A GitHub issue is something that the contributors need help with; the complexity and involved work varies, so if you’re contributing for the first time to a particular project, it’s a good idea to filter by <em>good first issue</em>; such issues are suitable for your first contribution, click here to see <a href="https://github.com/ploomber/ploomber/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">Ploomber’s good first issues.</a></p><p>Consider spending some time navigating the issues; the tags are a quick way of identifying what they involve. Then, if you’re interested in any of them, click on any of them and check the description. Maintainers often provide a brief description, but you can ask for more details if you’re interested.</p><p>Once you have identified an issue you want to work on, ask a maintainer to assign it to you. Once the maintainers reply and assign it to you, you can start working on it. It’s essential to ask questions before you begin your work to get on the same page with the maintainers. Remember that they will be reviewing your code.</p><p>Note that different projects have different rules, so check the CONTRIBUTING document in the repository to understand the contribution guidelines. The CONTRIBUTING file is in the root folder, but the extension might vary (common extensions are .md, and .rst). For example, this is Ploomber&#39;s <a href="https://github.com/ploomber/ploomber/blob/master/CONTRIBUTING.md">CONTRIBUTING.md</a> file.</p><p>Often, projects need help with documentation; this is easier than contributing with code since you can edit the files from GitHub’s online editor. However, documentation is crucial to any project, so don’t underestimate its importance! For example, here are the documentation issues in <a href="https://github.com/ploomber/ploomber/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22+label%3Adocumentation">Ploomber’s repository</a>.</p><h3><strong>Forking the repository</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*-HOSEKk2-ode9hAvJ8WS9Q.gif" /><figcaption>Image by author.</figcaption></figure><p>Now it’s time to fork the repository. Forking means creating a copy of the repository to your account. Repositories are identified by a username (or organization name) and the repository name. For example, Ploomber’s URL is github.com/ploomber/ploomber, which means it is under the Ploomber organization, and the repository name is also Ploomber. If the repository belongs to an individual account, it will have the same format; for example, my project sklearn-evaluation is under github.com/edublancas/sklearn-evaluation, where edublancas is my account, and sklearn-evaluation the repository name.</p><p>Once you fork a repository, GitHub will create a copy in your account. For example, if you fork Ploomber, it will create a repository under github.com/{your-username}/ploomber. After forking the repository, clone it locally:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/306f8360b7888ffce03df0c9dafd0446/href">https://medium.com/media/306f8360b7888ffce03df0c9dafd0446/href</a></iframe><p>Note that we’re cloning the fork, not the original repository! This means that any changes to the code <em>will not</em> affect the main repository, so feel free to experiment!</p><p>It is a good practice to create a new branch; you can do so with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38c7490131816dbce9f9b66415c951f3/href">https://medium.com/media/38c7490131816dbce9f9b66415c951f3/href</a></iframe><h3><strong>Setting up your development environment</strong></h3><p>Now it’s time to start coding! The CONTRIBUTING document often contains the instructions to set up the development environment. For example, in Ploomber, once you cloned the repository, you can set up the development environment with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/368f502e951b681559f94c5c06da9409/href">https://medium.com/media/368f502e951b681559f94c5c06da9409/href</a></iframe><p>The CONTRIBUTING document often specifies how to test your environment to ensure successful installation.</p><h3><strong>Coding!</strong></h3><p>Generally speaking, there are two types of code contributions: bug fixes and new features. Bug fixes involve improving something that’s already implemented but isn’t working as expected, while new features entail adding unique characteristics to the project. Our recommended process is the same in both cases: write your tests first.</p><p>A test is a short program that checks that the library behaves as expected given some input. Libraries have tests distributed across many files. And usually, they’re located in a tests/ directory at the root of the project. For example, this is <a href="https://github.com/ploomber/ploomber/tree/master/tests">Ploomber&#39;s tests directory</a>. The logic to organize tests changes; for instance, in Ploomber, we have one file per module in the project: here&#39;s the <a href="https://github.com/ploomber/ploomber/tree/master/src/ploomber/dag">dag module</a> and here&#39;s the test folder for testing the <a href="https://github.com/ploomber/ploomber/tree/master/tests/dag">dag module</a>.</p><p>Writing a test first will allow you to have a quick way of checking your implementation, plus showing your test to the maintainer allows you to get on the same page in terms of code functionality. So, as a first step, find where to add the test. Usually, you’ll be modifying an existing test file, but when in doubt, ask the maintainer.</p><p>To make things concrete, let’s say you’re working on fixing a bug in a function called sum, and the issue has the following description:</p><blockquote>There’s an error in the sum function, it should output 4 when called as sum(2, 2), but it outputs 5</blockquote><p>Often, you’ll encounter descriptions with short descriptions, but you can be proactive and start gathering information about what’s required. For example, you can use GitHub’s search bar to locate the sum function definition. By default, the search bar will look up the query in the existing repository. So, in this case, you may search for sum, or, even better, for def sum to find the definition of the function.</p><p>Once you’ve identified where you would be making code changes, locate the appropriate test file. So look in the tests/ folder and see if you can find where the maintainers are testing the sum function. If you cannot find it, it&#39;s ok to ask, but the maintainer will greatly appreciate it if you take the time to understand the codebase layout and identify the relevant files. Once you&#39;ve identified the test file, edit it locally and add a new test; essentially, you need to translate the issue description into a program; following our example, here&#39;s a simple test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfd15e767fe8a269c629e9e712593cf4/href">https://medium.com/media/bfd15e767fe8a269c629e9e712593cf4/href</a></iframe><p>First, we added an import statement to get the function that we’re testing; then, we created a test by defining a function prefixed by test_, note that we have a long name, that&#39;s ok; it&#39;s essential to have descriptive names.</p><p>A great way to write good tests is by using the <a href="https://en.wikipedia.org/wiki/Given-When-Then">Given-When-Then</a> template (or GWT). GWT breaks our test logic into three parts: given (existing conditions), when (code to test), then (expected results). The following snippet shows the same test explicitly organized using the GWT template:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/788271380d4647490a39baf2a680686a/href">https://medium.com/media/788271380d4647490a39baf2a680686a/href</a></iframe><p>However, to reduce verbosity, we often collapse the different sections, especially in a simple case, so you’d normally write something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4395e8151b02eb3ebfbb769252a36b24/href">https://medium.com/media/4395e8151b02eb3ebfbb769252a36b24/href</a></iframe><p>I recommend you explicitly organize the different GWT parts in more extensive tests. Note that our test essentially translates the issue description into code since it formally describes how things <em>should</em> work.</p><p>The assert keyword helps us establish the rule that our test will evaluate. If the statement is true, nothing will happen; if it&#39;s false, it will trigger an error, causing our test to fail. In some cases, libraries provide testing tools to simplify writing these assert statements. For example, pandas has a pandas.testing module with valuable functions to <a href="https://pandas.pydata.org/docs/reference/api/pandas.testing.assert_frame_equal.html">compare data frames</a>. A great way to learn more about how you can proceed with your test is to look at existing tests since tests in the same file tend to be similar to each other.</p><p>Since we haven’t made any changes, it’s natural that the test will fail. Most Python projects use pytest for running tests (this should be specified in the CONTRIBUTING document). To run a single test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87001f9edbe97326e7e0b40405cc4dbe/href">https://medium.com/media/87001f9edbe97326e7e0b40405cc4dbe/href</a></iframe><p>You’ll see that the test is failing, that’s expected since we haven’t fixed the bug!</p><p>In a real-world scenario, coming up with the sample test will require more work, so don’t be afraid to ask the maintainer for guidelines. Furthermore, existing tests can help give you an idea of how to test.</p><p>Once you have your (currently failing) test, it’s time to work on a fix (or a new feature if that’s the case). So go ahead and take a look at the module/function/class that requires changes; let’s say you find the following implementation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ec0cd371a91dfe5f4ec2228e65dea6a/href">https://medium.com/media/7ec0cd371a91dfe5f4ec2228e65dea6a/href</a></iframe><p>After examining the code, you notice that the + 1 seems incorrect, so you delete it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19dd40d4889f73e3cbe4651fd87a1965/href">https://medium.com/media/19dd40d4889f73e3cbe4651fd87a1965/href</a></iframe><p>Then, rerun the test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87001f9edbe97326e7e0b40405cc4dbe/href">https://medium.com/media/87001f9edbe97326e7e0b40405cc4dbe/href</a></iframe><p>This time the test will pass!</p><p>Of course, things will require more work to get your code to pass. It’s easier to understand by interacting with it, as a recommendation, you may start an interactive session inside your test by adding the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f68d1e519b88421f980127a21d73ee1/href">https://medium.com/media/9f68d1e519b88421f980127a21d73ee1/href</a></iframe><p>Alternatively, you may use the pdb module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbc759887f32ea239da2105f83bbca39/href">https://medium.com/media/dbc759887f32ea239da2105f83bbca39/href</a></iframe><p>The difference between using IPython and pdb is that the IPython behaves like a regular Python session while pdb has specific functionality for debugging code. To learn more, read <a href="https://docs.python.org/3/library/pdb.html">pdb documentation</a>.</p><p>In both IPython&#39;s and pdb&#39;s case, you need to pass the the -s flag to pytest:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d03154a85fa4c7ce15d67624befa6247/href">https://medium.com/media/d03154a85fa4c7ce15d67624befa6247/href</a></iframe><p>You’ll see that an interactive session starts. Now you can run things interactively to run some quick commands to help you understand the code. For example, you may run the function with different arguments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87f133fb2d7b21f3477403ac766b5b8b/href">https://medium.com/media/87f133fb2d7b21f3477403ac766b5b8b/href</a></iframe><p>Once you’re done experimenting, type quit and press enter to exit the session and remember to delete the from IPython import embed; embed() line (or from pdb import set_trace; set_trace() if using pdb).</p><p>Alternatively, you may want to let the test run and start a debugging session once it fails, in that case, there’s no need to add anything to the test source code. Keep it like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a23987a8960f26c2925caaffe7c58513/href">https://medium.com/media/a23987a8960f26c2925caaffe7c58513/href</a></iframe><p>And then execute the test with the --pdb flag:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e375b5ce91e5461f090c43f60b376283/href">https://medium.com/media/e375b5ce91e5461f090c43f60b376283/href</a></iframe><p>This time, the test will run, and if an exception occurs during the execution, it will start an interactive debugging session. Note that this is similar to a regular Python session, but it has a few extra things. Here’s a <a href="https://docs.python.org/3/library/pdb.html#debugger-commands">list of commands you can execute</a>. The Python debugger is a handy tool for exploring the code interactively and understanding the conditions that led to the failure: variable values, which modules are involved, etc. To exit the debugging session, type quit and press enter.</p><h3><strong>Drafting a Pull Request</strong></h3><p>As the name suggests, Pull Requests (or PR) are a <em>request</em> to <em>pull</em> your code into the original repository. When submitting one, maintainers can look at your code and decide to accept your proposed changes. However, Pull Requests are not a one-off process; if maintainers have observations, they will provide feedback and suggestions, then you can make changes to the code and ask for another review.</p><p>GitHub introduced <a href="https://github.blog/2019-02-14-introducing-draft-pull-requests/">Draft Pull Requests</a> as a way for contributors to get early feedback and ensure you’re aligned with the maintainers. Getting on the same page via comments on the GitHub issue is difficult, so the maintainers will greatly appreciate that you draft a PR so they can see your progress and provide feedback. Getting your contributions merged into the codebase is a collective effort, so expect to have these back and forth feedback sessions a couple of times (or more, depending on the complexity of what you’re contributing).</p><p>There is no need to open another pull request if you make changes; push to your fork, and the Pull Request will automatically update. In Ploomber’s case, it’s ok to ask for feedback multiple times, but make an effort to have specific questions so the maintainer can quickly help you. When in doubt, check out the CONTRIBUTING document for the project you’re contributing to, as it might contain more specific instructions.</p><p>Once you pushed to your fork, the steps for opening a draft Pull Request are as follows:</p><ol><li>Open the original (not your fork) repository on your browser</li><li>Click on the “Pull Requests” section in the navigation bar</li><li>Click on the green “New pull request” button</li><li>Click on “compare across forks”</li><li>On the right side of the selection, select your fork ({your-username}/repository-name)</li><li>Select the branch where you committed the changes</li><li>Click on the green button “Create pull request”</li><li>Add comments</li><li>Click on the green arrow next to the “Create pull request” button, and click on “Create draft pull request”</li><li>Confirm by clicking on the green “Create draft pull request” button</li></ol><p>The process is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SAijqFn2s33hftZs6y4iiQ.gif" /><figcaption>Image by author.</figcaption></figure><p>Congrats! You submitted a Pull Request! Now give some time for the maintainers to review your changes and share their feedback.</p><h3><strong>Adding more tests</strong></h3><p>Depending on how complex the contribution is, you’ll most likely need to add more than one test. As you become familiar with the code, consider what other cases are relevant to test. Following our sum example, you might want to try negative numbers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c22bb664c1e06826e914384aaeb94f3/href">https://medium.com/media/2c22bb664c1e06826e914384aaeb94f3/href</a></iframe><p>When in doubt, ask the maintainers if they can think of other relevant test cases; since they’re more familiar with the code, they’ll be in an excellent position to suggest them.</p><p>Often, test cases are pretty similar, except for their inputs and outputs. To simplify your code (and assuming the library you’re contributing to uses pytest), you can parametrize a test like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c601036526a8aef9b1f450ee8a62fffd/href">https://medium.com/media/c601036526a8aef9b1f450ee8a62fffd/href</a></iframe><p>Note that we renamed the test case since it now embeds many test cases, we may optionally add the ids argument to identify each test case:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/304b6290ec283d2e964d5174cac96f6f/href">https://medium.com/media/304b6290ec283d2e964d5174cac96f6f/href</a></iframe><p>To run both test cases:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25fa653ea7609b7af59bc1d40056bf9d/href">https://medium.com/media/25fa653ea7609b7af59bc1d40056bf9d/href</a></iframe><p>To run a specific test case, pass the ID, for example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abe30c29230873cc0eadc3ee7c2a78d7/href">https://medium.com/media/abe30c29230873cc0eadc3ee7c2a78d7/href</a></iframe><p>To learn more about pytest parametrization, <a href="https://docs.pytest.org/en/6.2.x/parametrize.html">check out the docs.</a></p><h3><strong>Testing exceptions</strong></h3><p>In our previous example, we were testing the output of a function, but this is only one type of test. In some cases, you may want to test that the code throws a meaningful error to the user. If the project you are contributing to uses pytest, you can test that particular exception is raised:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c789d48f419fa13dec2dbe5d18713e9c/href">https://medium.com/media/c789d48f419fa13dec2dbe5d18713e9c/href</a></iframe><p>Even better, you can test that a specific error message is shown like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a34731d541c4153b07638a4b12f2dec5/href">https://medium.com/media/a34731d541c4153b07638a4b12f2dec5/href</a></iframe><p>Note that we don’t need to import ValueError since it&#39;s a <a href="https://docs.python.org/3/library/exceptions.html#concrete-exceptions">built-in exception</a>; in some cases, the project may define its exceptions (typically in an exceptions.py file), so you may need to import it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60c2c795279eacbabb2034227fac934f/href">https://medium.com/media/60c2c795279eacbabb2034227fac934f/href</a></iframe><h3><strong>Submitting the Pull Request</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*jFcoS_1P8hRsU8vMgGh5hg.png" /><figcaption>Image by author.</figcaption></figure><p>After a few reviewing cycles, you’ll be ready for a final code review. Ideally, this should be fast since you already requested feedback during the drafting process. Before marking the Pull Request as <em>ready for review</em>, ensure the following:</p><p><em>1. Tests are passing locally</em></p><p>Ensure the tests you added are passing locally:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0aa262c4ad069282fec754a55cd0b3f/href">https://medium.com/media/d0aa262c4ad069282fec754a55cd0b3f/href</a></iframe><p><em>2. Check linting and formatting</em></p><p>Projects often use code linters that automatically check the code for specific rules; ensure that your code complies with the rules. (the CONTRIBUTING document should mention which linter the project uses so you can check it locally). Although to simplify things, some auto-formatters automatically make your code compliant with the linter (check the CONTRIBUTING document for more details).</p><p>If tests and the linting pass, you can git push and mark your PR as ready for review!</p><h3><strong>Continuous Integration</strong></h3><p>Your git push will trigger the repository&#39;s Continuous Integration system (or CI). CI is a fancy way of saying &quot;running all the tests on each PR.&quot; Running all the tests ensures that your code doesn&#39;t have any unintended side effects. Usually, test suites are pretty comprehensive, so it might take a few minutes to finish. If tests fail, check the logs for hints; perhaps there are some linting issues or side effects, so try to understand what&#39;s going on; if you&#39;re unable to, you may ask the maintainer.</p><p>Be patient. This process takes time. Often, maintainers volunteer their spare time to keep the project alive, so be patient if they take a bit to respond.</p><p>If tests pass, the maintainer will perform a code review, and if everything goes well, they will merge your code! On the other hand, if tests do not pass, you may click on “Details” (see the screenshot above) to check the tests. Take some time to understand why tests are failing — the maintainer will greatly appreciate it if you’re proactive and identify the root cause, fix the tests, and push your changes (pushing new changes will trigger the tests again).</p><h3><strong>Closing Remarks</strong></h3><p>Stay tuned for our next blog post, where we’ll discuss the thought process of contributing with new features and some advanced testing techniques such as using the <a href="https://docs.python.org/3/library/unittest.mock.html">unittest.mock</a> module and testing CLI applications.</p><p>If you’re up for the challenge, apply what you learned today by contributing to <a href="https://github.com/ploomber/ploomber">Ploomber</a>!</p><p><em>Originally published at </em><a href="https://ploomber.io/blog/open-source/"><em>ploomber.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b2a8f1aa5cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-gentle-introduction-to-open-source-contributions-4b2a8f1aa5cd">A Gentle Introduction to Open-Source Contributions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>