<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudhanshu Passi on Medium]]></title>
        <description><![CDATA[Stories by Sudhanshu Passi on Medium]]></description>
        <link>https://medium.com/@sdhnshu?source=rss-e571a28d699e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uLiu6St_A5s-90gBr44kQA.jpeg</url>
            <title>Stories by Sudhanshu Passi on Medium</title>
            <link>https://medium.com/@sdhnshu?source=rss-e571a28d699e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:31:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sdhnshu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Feature Engineering with Random Forests (Part2)]]></title>
            <link>https://medium.com/@sdhnshu/feature-engineering-with-random-forests-part2-160eb0356172?source=rss-e571a28d699e------2</link>
            <guid isPermaLink="false">https://medium.com/p/160eb0356172</guid>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[Sudhanshu Passi]]></dc:creator>
            <pubDate>Wed, 29 Apr 2020 11:12:31 GMT</pubDate>
            <atom:updated>2020-04-29T11:12:31.561Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@sdhnshu/feature-engineering-with-random-forests-part1-601c66dfb09b"><em>Link to Part 1 of the series</em></a></p><p>In the part 1 of the series, we looked at the random forests algorithm and figured how to get accurate predictions from the model. But the part we are forgetting is:</p><blockquote><em>The features that get into the model are as important as the model itself.</em></blockquote><p>Especially if you work at an institution, like the one making <a href="https://www.kaggle.com/c/bluebook-for-bulldozers/data">bulldozers</a>, you are trying to understand the following things about the features:</p><ul><li>What is the <strong>importance of each</strong> feature?</li><li><strong>How similar</strong> are they to each other?</li><li>How do these <strong>features interact</strong> with each other?</li><li>How to make features truly <strong>independent from time</strong>?</li></ul><p><em>Let us look at each of them using concepts from random forests.</em></p><h3>i. Feature Importance</h3><p>The only way to find the feature importance is rightly described by one of my oldest inspirations in this verse:</p><blockquote><em>You don’t know what you’ve got, until its gone — </em><a href="https://www.youtube.com/watch?v=oM-XJD4J36U"><em>Chester Bennington</em></a></blockquote><p>Technically speaking, if we <strong>randomly shuffle</strong> the values in a feature but keep everything else the same, <strong><em>how much worse</em></strong> does our model perform. The <em>worse</em> the model performs, the <em>higher</em> the importance of that feature.</p><pre># Feature Importance Intuition</pre><pre>def feature_importance(self, model, x_valid):</pre><pre>    def shuffle_col(colname, x_valid):<br>        df_temp = x_valid.copy()<br>        df_temp[colname] = np.random.permutation(<br>            df_temp[colname].values)<br>        return df_temp</pre><pre>    pred_actual = model.predict(x_valid)</pre><pre>    return [(1 - r2_score(pred_actual, model.predict(<br>        shuffle_col(col, x_valid)))) for col in x_valid.columns]</pre><p>The above function describes the functionality of feature importance and the one below is a convenient Fast.ai wrapper around the scikit learn’s optimized functionality. Let us plot 30 most important features out of the 66 we have.</p><pre>set_rf_samples(50000)</pre><pre>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>                          max_features=0.5, n_jobs=-1, <br>                          oob_score=True)<br>m.fit(X_train, y_train)</pre><pre># <a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L394-L396">rf_feat_importance</a> wrapper from Fast.ai</pre><pre>fi = rf_feat_importance(m, df_trn)<br>fi[:30].plot(&#39;cols&#39;, &#39;imp&#39;, &#39;barh&#39;, figsize=(12,7), legend=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/0*BcS_MXeq3Kv-ibQf.png" /></figure><p>We can remove the ones lower than the importance of 0.005. This leaves us with 23 features. Let us plot the importance again and score the model.</p><pre>to_keep = fi[fi.imp&gt;0.005].cols</pre><pre>df_keep = df_trn[to_keep].copy()<br>X_train, X_valid = split_vals(df_keep, n_trn)</pre><pre>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>                       max_features=0.5, n_jobs=-1, oob_score=True)<br>m.fit(X_train, y_train)<br>print_score(m)</pre><pre>fi = rf_feat_importance(m, df_keep)<br>plot_fi(fi[:30]);</pre><pre><strong>Output:</strong></pre><pre>&gt; RMSE Train:  20.720376603129303<br>&gt; RMSE Valid:  24.570217524283752<br>&gt;  R2  Train:  <strong>91</strong>.02715603822325<br>&gt;  R2  Valid:  <strong>89</strong>.21882794959521<br>&gt;    OOB    :  <strong>89</strong>.39924413165484</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*soe_Udg2GtaRkXWC.png" /></figure><p>YearMade, Coupler_System and ProductSize turn out to be the top 3 most important features and we have a validation score of <strong>89.2%</strong></p><h3>ii. Feature correlation</h3><p>Combining similar features can be done by going through all the <em>pairs</em> and clustering the ones that are <em>closest</em> to each other. This kind of <strong>hierarchical clustering</strong> works based on the <strong>ordinality</strong> of those features. Which is similar to how random forests work. While splitting a node, it only cares about the sorted order of the data and not their actual values. A common coefficient to measure for this kind of <strong>ranked correlation</strong> is <em>spearman’s ranked coefficient.</em></p><pre>from scipy.cluster import hierarchy as hc</pre><pre>corr = np.round(scipy.stats.spearmanr(df_keep).correlation, 4)</pre><pre>plt.figure(figsize=(16,10))<br>hc.dendrogram(hc.linkage(hc.distance.squareform(1-corr), <br>                         method=&#39;average&#39;), <br>              labels=df_keep.columns, orientation=&#39;left&#39;, <br>              leaf_font_size=16)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/0*8zHuirGqrQL55zdU.png" /><figcaption>Dendogram</figcaption></figure><p>The features that are correlated connect faster. With that, we can see that saleYear and saleElapsed are similar which is expected as they both are time-dependent. Hydraulics_flow and Grouser_Tracks are similar and also fiBaseModel and fiModelDesc.</p><p>If two features are similar removing one of them won’t affect the accuracy of our model, it will only make it simpler. So let us try dropping them and measuring the drop in score. Before that let’s establish a baseline.</p><pre>def get_oob(df):<br>    m = RandomForestRegressor(n_estimators=30, min_samples_leaf=5, <br>                              max_features=0.6, n_jobs=-1, <br>                              oob_score=True)<br>    x, _ = split_vals(df, n_trn)<br>    m.fit(x, y_train)<br>    return m.oob_score_ * 100</pre><pre>get_oob(df_keep)  </pre><pre><strong>Output:</strong></pre><pre>&gt; <strong>89</strong>.03929035253717   # Baseline</pre><p>Let’s drop them one by one.</p><pre>for c in [&#39;saleYear&#39;, &#39;saleElapsed&#39;, &#39;Grouser_Tracks&#39;, &#39;Hydraulics_Flow&#39;, &#39;fiModelDesc&#39;, &#39;fiBaseModel&#39;]:<br>    print(c, get_oob(df_keep.drop(c, axis=1)))</pre><pre><strong>Output:</strong></pre><pre>&gt; <strong>saleYear 88.96577294296006</strong><br>&gt; saleElapsed 88.72699565865227<br>&gt; Grouser_Tracks 89.02477486920333<br>&gt; <strong>Hydraulics_Flow 88.98468803187907</strong><br>&gt; fiModelDesc 88.93420811313398<br>&gt; <strong>fiBaseModel 88.91058697856653</strong></pre><p>Because they didn’t do much damage, saleYear, Hydraulics_Flow, and fiBaseModel shall be removed.</p><pre><br>to_drop = [&#39;saleYear&#39;, &#39;Hydraulics_Flow&#39;, &#39;fiBaseModel&#39;]<br>get_oob(df_keep.drop(to_drop, axis=1))</pre><pre><strong>Output:</strong></pre><pre>&gt; <strong>88</strong>.89865421652972</pre><p>The overall OOB score dropped a little but not significantly. Let us check how a full bootstrapped model is looking with the cleaned data. It happens in 30 seconds rather than 1 min in the last post!</p><pre>reset_rf_samples()</pre><pre>df_keep.drop(to_drop, axis=1, inplace=True)<br>X_train, X_valid = split_vals(df_keep, n_trn)</pre><pre>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)<br>%time m.fit(X_train, y_train)<br>print_score(m)</pre><pre><strong>Output:</strong></pre><pre>&gt; CPU times: user 1min 45s, sys: 922 ms, total: 1min 46s<br>&gt; Wall time: <strong>30.4 s</strong><br>&gt; RMSE Train:  12.51548564415119<br>&gt; RMSE Valid:  22.79049206209574<br>&gt;  R2  Train:  <strong>96</strong>.72636512376931<br>&gt;  R2  Valid:  <strong>90</strong>.72411452493563<br>&gt;    OOB    :  <strong>90</strong>.86147528243829</pre><h3>iii. Partial dependence</h3><p>In this section we’ll see how two features interact. But before we get into it, it would be interesting to see the interactions between specific categories from a categorical variable as well. We can one-hot encode categorical variables with less than 7 categories to do that. Let us check their feature importance.</p><pre>set_rf_samples(50000)  # Subsampled model</pre><pre>df_trn2, y_trn, nas = proc_df(df_raw, &#39;SalePrice&#39;, max_n_cat=7)  # 1hot encoded<br>X_train, X_valid = split_vals(df_trn2, n_trn)<br>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.6, n_jobs=-1)<br>m.fit(X_train, y_train);</pre><pre>fi = rf_feat_importance(m, df_trn2)<br>plot_fi(fi[:20]);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*WYgcNZUdWFGQeDOW.png" /></figure><p>Enclosure was a categorical feature but is now one-hot encoded. And Enclosure_EROPS w AC has gained the highest importance among all. Some search on the internet reveals that it stands for an <em>enclosed space with an AC</em>. Which, if in a bulldozer will surely be a deciding factor for its price.</p><p>The second most important feature is YearMade. It would be logical to see its interacts with 7th most important, saleElapsed.</p><pre>df_raw.plot(&#39;YearMade&#39;, &#39;saleElapsed&#39;, &#39;scatter&#39;, alpha=0.01, figsize=(10,8));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/0*gHFbPT5HE9JLgWRl.png" /></figure><p><em>Oh no!</em> There are some outliers in the data with YearMade=1000. This might be due to some technical error. We can remove them from our analyses for now.</p><pre>df_raw = df_raw[df_raw.YearMade&gt;1930]</pre><p>Let us see how YearMade affects our dependent variable, SalePrice by simply plotting them together.</p><pre>from plotnine import *</pre><pre>x_all = get_sample(df_raw[df_raw.YearMade&gt;1930], 500)  # sample 500 pts excluding years with wrong year=1000<br>ggplot(x_all, aes(&#39;YearMade&#39;, &#39;SalePrice&#39;))+stat_smooth(se=True, method=&#39;loess&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/0*eRH40X0btXLy10ae.png" /></figure><p>This plot shows ( YearMade + <strong>the rest of the features</strong>)&#39;s affect on SalePrice.</p><p>To <em>single out</em> how a feature affects another, we need to change one and keep the rest constant. In the example below, we sampled 500 rows and changed all values of YearMade in the dataset to a value on x, eg: 1990 and predicted a value of SalePrice. Doing so for all values of YearMade gives us the plot.</p><pre>x = get_sample(X_train[X_train.YearMade&gt;1930], 500)</pre><pre>def plot_pdp(feat, clusters=None, feat_name=None):<br>    feat_name = feat_name or feat<br>    p = pdp.pdp_isolate(m, x, feat)<br>    return pdp.pdp_plot(p, feat_name, plot_lines=True,<br>                        cluster=clusters is not None,<br>                        n_cluster_centers=clusters)<br>plot_pdp(&#39;YearMade&#39;)<br>plot_pdp(&#39;YearMade&#39;, 5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/0*vu4fB_9e_nDarbv7.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*QGbSRpn3XQMHy_Wq.png" /><figcaption><em>X: YearMade and Y: SalePrice</em></figcaption></figure><p>Similarly to find how YearMade and SaleElapsed together affect SalePrice, we can use an interaction plot.</p><pre>feats = [&#39;saleElapsed&#39;, &#39;YearMade&#39;]<br>p = pdp.pdp_interact(m, x, feats)<br>pdp.pdp_interact_plot(p, feats)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*Wzx31D2CkGFSkYcO.png" /></figure><p>We can see from the interaction plot that if YearMade is high and saleElapsed low, the SalePrice would be higher. So the age = saleYear - YearMade of the truck is important. You can add age as another feature to our dataset.</p><pre>df_raw.YearMade[df_raw.YearMade&lt;1950] = 1950<br>df_keep[&#39;age&#39;] = df_raw[&#39;age&#39;] = df_raw.saleYear-df_raw.YearMade</pre><pre>X_train, X_valid = split_vals(df_keep, n_trn)<br>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>                          max_features=0.6, n_jobs=-1)<br>m.fit(X_train, y_train)<br>plot_fi(rf_feat_importance(m, df_keep));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*DjzkQSca-kwhl0UO.png" /></figure><p>Age quickly becomes our most important feature.</p><h3>iv. Removing time dependency</h3><p>Unlike linear models or neural networks, random forests are <em>not very good at extrapolating data</em>. So <em>extrapolating over time</em> is tough. It is better, we remove the dependency on time from our features.</p><p><strong><em>Q:</em></strong><em> But how to we find the time dependent features?</em></p><blockquote><em>If the validation set were a </em><strong><em>random sample</em></strong><em> of the training set, it would be difficult to predict if a row </em><strong><em>is in</em></strong><em> the validation set.</em></blockquote><p>So if a model can successfully learn whether a value is in the validation set, then it has some temporal dependency helping it to do so. And the most important features in such a model will be the most time dependent features.</p><p>All we have to do is, make is_valid our dependent variable, and train the model. Because it is a 0/1 discrete prediction we are using a RandomForestClassifier instead of a regressor.</p><pre>df_ext = df_keep.copy()<br>df_ext[&#39;is_valid&#39;] = 1<br>df_ext.is_valid[:n_trn] = 0<br>x, y, nas = proc_df(df_ext, &#39;is_valid&#39;)</pre><pre>m = RandomForestClassifier(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)<br>m.fit(x, y);<br>m.oob_score_</pre><pre><strong>Output:</strong></pre><pre>&gt; 0.<strong>99999</strong>50140230601</pre><p>There is a very high score of prediction, thus we have features with time dependency. The validation set is not a random sample from the dataset. Let us see which features are the most important to a purely temporal dependent forest.</p><pre>fi = rf_feat_importance(m, x); fi[:10]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/0*AAeuknC1bCxFAQRq.png" /></figure><p>Our top 3 time dependent features are: SalesId, saleElapsed and MachineId. Let us remove them and see their effect on the score.</p><pre>x.drop([&#39;SalesID&#39;, &#39;saleElapsed&#39;, &#39;MachineID&#39;], axis=1, inplace=True)</pre><pre>m = RandomForestClassifier(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)<br>m.fit(x, y);<br>m.oob_score_</pre><pre><strong>Output:</strong></pre><pre>&gt; 0.<strong>97</strong>88444998441882</pre><p>Checking the importance after removing those features</p><pre>fi = rf_feat_importance(m, x); fi[:10]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*onbQPrnDELkYqKS0.png" /></figure><p>Now we see new time-dependent features pop up like age, YearMade, and saleDayofyear. Let us drop these 6 one by one, and check their negative affects. Before that let us look at our baseline.</p><pre>set_rf_samples(50000)</pre><pre>X_train, X_valid = split_vals(df_keep, n_trn)<br>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>                        max_features=0.5, n_jobs=-1, oob_score=True)<br>m.fit(X_train, y_train)<br>print_score(m)</pre><pre><strong>Output:</strong></pre><pre>&gt; RMSE Train:  20.633388115029838<br>&gt; RMSE Valid:  24.491469116919422<br>&gt;  R2  Train:  91.1023376550528<br>&gt;  R2  Valid:  89.2878252704439<br>&gt;    OOB    :  89.46118267044744</pre><p>Now let us drop them one by one.</p><pre>feats = [&#39;SalesID&#39;, &#39;saleElapsed&#39;, &#39;MachineID&#39;, &#39;age&#39;, &#39;YearMade&#39;, &#39;saleDayofyear&#39;]</pre><pre>for f in feats:<br>    df_subs = df_keep.drop(f, axis=1)<br>    X_train, X_valid = split_vals(df_subs, n_trn)<br>    m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>            max_features=0.5, n_jobs=-1, oob_score=True)<br>    m.fit(X_train, y_train)<br>    print(f)<br>    print_score(m)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/0*j5LmuOYm8Y5RPYFe.png" /></figure><p>The R² score on the validation set<em> gets better</em> by dropping SalesId, MachineId, and saleDayofyear.</p><p>With that, our features are as time-independent as could be. So, let us drop them and train a complete bootstrap model.</p><pre>df_subs = df_keep.drop([&#39;SalesID&#39;, &#39;MachineID&#39;, &#39;saleDayofyear&#39;], axis=1)</pre><pre>reset_rf_samples()</pre><pre>X_train, X_valid = split_vals(df_subs, n_trn)<br>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>               max_features=0.5, n_jobs=-1, oob_score=True)<br>%time m.fit(X_train, y_train)<br>print_score(m)</pre><pre>plot_fi(rf_feat_importance(m, X_train));</pre><pre><strong>Output:</strong></pre><pre>&gt; CPU times: user 1min 15s, sys: 991 ms, total: 1min 16s<br>&gt; Wall time: <strong>23.4 s</strong><br>&gt; RMSE Train:  13.775059373372189<br>&gt; RMSE Valid:  21.81697224146163<br>&gt;  R2  Train:  <strong>96</strong>.03428239799979<br>&gt;  R2  Valid:  <strong>91</strong>.49964757192211<br>&gt;    OOB    :  <strong>90</strong>.91069189446978</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*041BsubE6iLfLlkK.png" /></figure><p>Congratulations on achieving a <strong>91.4%</strong> validation score in <strong>23</strong> <strong>seconds</strong> on the whole dataset!</p><h3>Boost it up!</h3><p>Now that we have our features all engineered and model tuned, we can turn the number of trees <strong>all the way up </strong>to get an accuracy of 92% on the validation set.</p><pre>m = RandomForestRegressor(n_estimators=160, max_features=0.5, <br>          n_jobs=-1, oob_score=True)<br>%time m.fit(X_train, y_train)<br>print_score(m)</pre><pre><strong>Output:</strong></pre><pre>&gt; CPU times: user 5min 52s, sys: 11.4 s, total: 6min 4s<br>&gt; Wall time: <strong>1min 53s</strong><br>&gt; RMSE Train:  8.013775394813543<br>&gt; RMSE Valid:  21.09451703745362<br>&gt;  R2  Train:  <strong>98</strong>.65782495931056<br>&gt;  R2  Valid:  <strong>92</strong>.05329397797038<br>&gt;    OOB    :  <strong>91</strong>.45711650192492</pre><p>This gets us into the <strong>top 1%</strong> of the competition and it is trained in under <strong>2 mins</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*lqg4n8qMeEUkVRmM.png" /></figure><p><a href="https://medium.com/@sdhnshu/feature-engineering-with-random-forests-part1-601c66dfb09b"><strong><em>Link to Part 1 of the series</em></strong></a></p><h3>Further reading</h3><ul><li><a href="https://medium.com/@hiromi_suenaga/machine-learning-1-lesson-1-84a1dc2b5236">Fast.ai ML course notes</a></li><li><a href="https://medium.com/@ag.ds.bubble/model-interpretability-a4244d82ffb2">Model interpretability</a></li><li><a href="https://medium.com/usf-msds/intuitive-interpretation-of-random-forest-2238687cae45">Intuitive interpretation of random forests</a></li><li><a href="https://structuringtheunstructured.blogspot.com/2017/11/coloring-with-random-forests.html">Coloring with random forests</a></li><li><a href="https://medium.com/@aravanshad/gradient-boosting-versus-random-forest-cfa3fa8f0d80">Gradient Boosting vs Random Forest</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=160eb0356172" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Engineering with Random Forests (Part1)]]></title>
            <link>https://medium.com/@sdhnshu/feature-engineering-with-random-forests-part1-601c66dfb09b?source=rss-e571a28d699e------2</link>
            <guid isPermaLink="false">https://medium.com/p/601c66dfb09b</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[random-forest]]></category>
            <dc:creator><![CDATA[Sudhanshu Passi]]></dc:creator>
            <pubDate>Wed, 29 Apr 2020 10:57:57 GMT</pubDate>
            <atom:updated>2020-04-29T11:13:25.231Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@sdhnshu/feature-engineering-with-random-forests-part2-160eb0356172"><em>Link to Part 2 of the series</em></a></p><p>Making an educated guess on what <em>price</em> should you sell a product is not everyone’s cup of tea. It needs a good amount of <em>domain knowledge</em> and at least some <em>experience</em>. And if we are giving this job to an algorithm, it makes it even more complicated.</p><blockquote><em>An algorithm is computationally scalable but logically bound.</em></blockquote><p>Unless you set these <em>bounds</em> the right way, you might not get the best output from it. This is what we are going to look at in this post.</p><p>We’ll go through the first <a href="http://course18.fast.ai/lessonsml1/lesson1.html">7 lectures</a> of the Fast.ai machine learning course and learn how to predict the price of a bulldozer (SalePrice) using the <a href="https://www.kaggle.com/c/bluebook-for-bulldozers/data">data</a> available about its manufacturing and usage. In this post, we&#39;ll cover the random forests algorithm and feature engineering in the part 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/0*easpKibgbYJgCSG8.png" /><figcaption><a href="https://www.kaggle.com/c/bluebook-for-bulldozers">BlueBook for Bulldozers</a></figcaption></figure><h3>Why Random Forests?</h3><p>Logically speaking, there are <em>very few constraints</em> in the Random Forests algorithm. Thus when we use it, we are introducing <em>no biases</em> from our end. And because it is simple, it is <em>highly interpretable,</em> unlike the black box algorithms that lack this feature.</p><blockquote><em>It takes the simplicity of a decision tree but </em><strong><em>generalizes</em></strong><em> it over a forest over the dataset to give us an exploratory tool that we can use as a flashlight in the dark.</em></blockquote><p>If you want to follow along, the jupyter notebook used in this post is available at <a href="https://github.com/sdhnshu/Random-forests-and-Feature-Engineering">the github repo</a> along with the Fast.ai library <em>(need a local copy of v0.7, because it has been upgraded with major changes)</em> and an environment file to set up an Anaconda environment. Now that you have everything ready, let us jump right in.</p><h3>Preprocessing</h3><p>If you are new to Kaggle competitions, <a href="https://www.kaggle.com/getting-started/44997">here are some handy</a> first steps. If not, make sure you have gone through the <a href="https://www.kaggle.com/c/bluebook-for-bulldozers/overview">overview</a> and start looking at the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xuijDvckOqpBhTqn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s-G2Z18Bb5LGTYNd.png" /><figcaption><a href="https://www.kaggle.com/c/bluebook-for-bulldozers/data">Bluebook for Bulldozers on Kaggle</a></figcaption></figure><p>By eyeballing, we can see that there are:</p><ul><li><strong>Numerical features</strong> — measurement or count (<em>SalePrice, BladeWidth</em>)</li><li><strong>Categorical features</strong> — ordinal and nominal (TrackType, EnclosureType, ProductSize, UsageBand)</li><li><strong>DateTime features</strong> (SaleDate)</li><li><strong>Ids</strong> (MachineId, ModelId)</li></ul><p>Download the <a href="https://www.kaggle.com/c/bluebook-for-bulldozers/data">data</a> locally in a folder ‘<em>bluebook-for-bulldozers</em>’ and extract the <em>Train.zip</em>.</p><p><em>Then let us look into some preprocessing techniques like:</em></p><ul><li><strong>Log</strong> transform</li><li>Extracting data from <strong>DateTime</strong></li><li>Handling <strong>categorical</strong> features</li><li>Imputing <strong>missing</strong> values</li></ul><h3>i. Log Transform</h3><p>If you see the <a href="https://www.kaggle.com/c/bluebook-for-bulldozers/overview/evaluation">Evaluation metrics</a>, they want the <em>Root Mean Squared </em><strong><em>Log</em></strong><em> Error.</em></p><blockquote><strong>RMSE: </strong>standard deviation of our model’s prediction errors</blockquote><pre>def rmse(actual, pred): <br>    return np.sqrt(np.mean((actual-pred)**2))</pre><p>This is common for a feature like a <em>price. </em>They are more interested if you missed by 10% than if you missed by $10. If it is $10,000 item and you are off by a $1000 or if it is a $100 item and you are off by $10, they would be considered as equivalent issues.</p><pre>df_raw = pd.read_csv(&#39;bluebook-for-bulldozers/Train.csv&#39;, <br>                     parse_dates=[&#39;saledate&#39;])<br>df_raw.SalePrice = np.log(df_raw.SalePrice)</pre><p>Taking a <strong><em>log</em></strong> brings the data closer to a <strong>normal distribution</strong>. So, if the prediction errors are normally distributed, around 95% of the prediction errors would be in <em>±2*RMSE. </em>It also decreases the effect of any <em>outliers</em> in the data. RMSE is the metric we’ll mathematically optimize and try to bring it down to 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/0*x_XL8FNSnnBOagEe.png" /><figcaption>Taking a Log on the SalePrice</figcaption></figure><h3>ii. Extracting Data from DateTime</h3><p>If some of your features are DateTime varaiables, we can <em>decompose</em> it into some useful features like — <em>was it a weekend, what year was it</em>, etc. Such engineering is often neglected but is very useful to extract the <strong>temporal</strong> information from the data. At the end of the post, we’ll also see how to remove any kind of temporal dependency from the data. This helps the model to generalize across time and will help us predict the SalePrice for the next month.</p><pre>from fastai.structured import *<br>from fastai.imports import *</pre><pre>add_datepart(df_raw, &#39;saledate&#39;)</pre><p><a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L70-L124"><strong>add_datepart</strong></a><strong>: </strong>Converts a datetime into <em>Year, Month, Week, Day, Dayofweek, Dayofyear, Is_month_end, Is_month_start, Is_quarter_end, Is_quarter_start, Is_year_end, Is_year_start (and Hour, Minute, Second)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/203/0*VBpJuWcoh_FNtBo4.png" /><figcaption>Before</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*4K_9QfdrE82tANuX.png" /><figcaption>SaleDate transformed into many different features using <a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L70-L124">add_datepart</a></figcaption></figure><h3>iii. Handling Categorical Features</h3><p>Pandas can handle categorical features by maintaining a <em>number:string </em>hash.</p><pre>train_cats(df_raw)</pre><p><a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L128-L153"><strong>train_cats</strong></a><strong>:</strong> This function from the Fast.ai library converts any columns of strings to a column of categorical values, in-place. It also fills <em>empty/Nan</em> values with <em>code=-1.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*LN88EvcZzCZucYTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/0*v51i5LqgXYGxfiTr.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/0*XFb1QCF0VIk_gdKd.png" /><figcaption>Converting UsageBand into a pandas category</figcaption></figure><h3>iv. Imputing Missing Values</h3><p>It is hard to find a real-life dataset without having missing values generated due to miscellaneous reasons<em>.</em> You can neglect them by simply removing entire rows containing some missing features. <em>But there is more to it.</em> Exploring them can lead you to the issues spawning them from behind the scenes. Here’s Jeremy Howard, creator of the Fast.ai course and ex-president of Kaggle giving <a href="https://youtu.be/YSFG_W8JxBo?t=1h11m16s">an example</a>.</p><p>Instead of deleting entire rows and reducing the dataset size, we can fill them using their <em>mean, median or mode</em>. You can also use algorithms like <em>k-NN and </em><a href="https://datascienceplus.com/imputing-missing-data-with-r-mice-package/"><em>MICE</em></a> to do so. There’s <strong>no best way</strong> to fill the missing values. A technique that works on a dataset with certain missing values may not work on another.</p><pre>df, y, nas = proc_df(df_raw, &#39;SalePrice&#39;)</pre><p><a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L298-L392"><strong>proc_df</strong></a>: This Fast.ai function fills <em>empty/Nan </em>values with the <strong>median</strong> of the feature and splits the dataframe into <strong><em>dependent</em></strong><em> </em><em>y</em> and <strong><em>independent</em></strong> df variables. It also adds 1 to the categorical codes so they start from 0 instead of -1 (<em>pandas default</em>). You can choose a random subset from df_raw and can also make features 1-hot encoded.</p><p>You should now have a clean matrix of integers as your df with no empty values. We can save the df and split it into a training and validation set.</p><blockquote>We want our validation set the same properties of the test set. Thus, keep the size of the validation set the same as the test set.</blockquote><pre># Pandas dataframes saved in <a href="https://blog.rstudio.com/2016/03/29/feather/">Feather format</a></pre><pre>df.to_feather(&#39;tmp/bulldozers-clean&#39;)<br>df = pd.read_feather(&#39;tmp/bulldozers-clean&#39;)</pre><pre><br># Splitting df and y into train and validation set</pre><pre>def split_vals(a,n): return a[:n].copy(), a[n:].copy()</pre><pre>n_valid = 12000  # Same size as test set<br>n_trn = len(df) - n_valid</pre><pre>X_train, X_valid = split_vals(df, n_trn)<br>y_train, y_valid = split_vals(y, n_trn)</pre><ul><li><strong>Training set size</strong>: 389,125</li><li><strong>Validation set size</strong>: 12,000</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/0*dbY-LZKoAoQNe2Jo.png" /><figcaption>We’ve successfully processed the data. Let us get into the algorithm.</figcaption></figure><h3>Random Forest Interpretation</h3><p>A set of <a href="https://en.wikipedia.org/wiki/Decision_tree">decision trees</a> trained on a <a href="https://en.wikipedia.org/wiki/Bootstrapping_(statistics)">bootstrapped</a> dataset <em>(random sampling with replacement of the same size as the original dataset (</em>389,125<em>)) </em>is called a random forest. Scikit-learn provides a RandomForestRegressor for predicting continuous floating-point numbers and a RandomForestClassifier for discrete numbers.</p><p>Let us train a random forest regressor and take a look at its predictions.</p><pre>from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier</pre><pre>m = RandomForestRegressor(n_jobs=-1)<br>m.fit(X_train, y_train)</pre><pre>m.predict(X_valid), m.score(X_valid, y_valid), np.mean(m.predict(X_valid)), np.std(m.predict(X_valid))</pre><pre><strong>Output</strong>:</pre><pre>&gt; (array([9.17702, 9.16897, 9.25441, ..., 9.44273, 9.31048, 9.31048]),<br>&gt; 0.<strong>89</strong>13676814741088,<br>&gt; <strong>10</strong>.012604788545604,<br>&gt; 0.<strong>69</strong>32527067952876)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/0*WaUnLoDw6sK6v_Eq.png" /></figure><h3>Predictions and Feature Contributions</h3><p>The main output of a model is its predictions. They lie around <em>10.01 </em>and have a standard deviation of <em>0.69. </em>If we want to know <em>how much</em> each feature contributes to the prediction, we can do that using a <a href="https://github.com/andosa/treeinterpreter">Tree Interpreter</a>.</p><pre>from treeinterpreter import treeinterpreter as ti<br>prediction, bias, contributions = ti.predict(m, row)</pre><p>The bias (10.10561) mentioned above is the <em>mean of y</em> in the training set. That is what we&#39;ll add to the contributions[-0.00458, -0.00525, ... 0.00227, 0], to get the prediction (9.17702). Let us see them in a sorted manner.</p><pre>idxs = np.argsort(contributions[0])<br>[o for o in zip(X_valid.columns[idxs], X_valid.iloc[0][idxs], contributions[0][idxs])]<br># Feature,    Value in the row,    Contribution</pre><pre><strong>Output:</strong></pre><pre>&gt; [(&#39;ProductSize&#39;, 5, -0.8274220325463135),<br>&gt;  (&#39;saleElapsed&#39;, 1284595200, -0.06203234395696224),<br>&gt;  (&#39;fiProductClassDesc&#39;, 17, -0.05853101269602874),<br>&gt;  (&#39;fiModelDesc&#39;, 3232, -0.03869971271031911),<br>&gt;  ...<br>&gt;  (&#39;fiModelSeries&#39;, 69, 0.00821771617784126),<br>&gt;  (&#39;fiBaseModel&#39;, 1111, 0.018305760128304628),<br>&gt;  (&#39;YearMade&#39;, 1999, 0.024869657037691794),<br>&gt;  (&#39;Coupler_System&#39;, 0, 0.10542811579794246)]</pre><h3>R² Score</h3><p>An equally important output of the model is the R² score.</p><blockquote><strong>R² score: </strong>the ratio between how good your model is <strong>vs.</strong> how good a naïve mean model is (both measured using RMSE).</blockquote><pre>def r2_score(actual, pred): <br>    return 1 - rmse(actual, pred)/ rmse(actual, np.mean(actual))</pre><p>It is not the value we are optimizing for, but it lets you compare different models and get a sense of how a score of 0.8 compares to 0.9. A common misconception is that its value can only range from 0 to 1. But,</p><blockquote><em>If you predicted infinity for every row, R² = 1 −∞. So when your R² is negative, it means your model is worse than predicting the mean.</em></blockquote><p>And our model gives an R² score of <strong>89%</strong> on the validation set with a <strong>69%</strong> variance in the predictions, out of the box. We’ll treat this as our baseline.</p><h3>Decision Tree Interpretation</h3><p>Let us zoom into a single tree:</p><pre>m = RandomForestRegressor(n_estimators=1, max_depth=3, <br>                          bootstrap=False, n_jobs=-1)<br>m.fit(X_train, y_train)</pre><pre>draw_tree(m.estimators_[0], X_train, precision=3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*XoBMr4-jU-Iy7Wn6.png" /><figcaption>A decision tree plot using <a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L21-L31">draw_tree</a></figcaption></figure><p>If max_depth=3 were removed, the tree would continue to split until every node has the no. of samples=1. But now the tree is only 3 layers deep. At the root node of the tree, is the whole training set (samples=389125) and a value <em>(avg y)</em> of 10.106. Then it is split into 2 samples with size 348,393 and 40,732 at Coupler_system &lt;= 0.5.</p><h3>How Do We Find the Best Split?</h3><p>Let us consider that we are trying to split a node containing <em>dogs and cats.</em> If we split so that one node has all dogs and the other has all cats, we would see that the <strong><em>standard deviation</em></strong> of both is 0. Also the size of the child nodes matters.</p><p>Thus, we need to minimize this <strong>split score </strong>using brute force:</p><blockquote>Std_dev(y) in left child node * no. of points in the left child node<br>+<br>Std_dev(y) in right child node * no. of points in the right child node</blockquote><p>Which is mathematically equivalent to:</p><blockquote><strong>Mean squared error</strong> between each point in <strong>y</strong> and the <strong>avg y</strong></blockquote><p><em>We can control the architecture of the forest using these parameters:</em></p><ul><li>Min samples leaf</li><li>Max features</li><li>No of estimators</li></ul><h3>i. Min Samples Leaf</h3><p>You can control the negative depth of the tree using this argument. It stops dividing the nodes once there are 3 samples or less in the node. This gives a <em>performance increase</em>, as the depth of the tree has been reduced by one or two.</p><pre>m = RandomForestRegressor(n_estimators=40, n_jobs=-1, <br>                          min_samples_leaf=3)</pre><p>It makes an individual tree’s predictions <em>less accurate</em>. As in each leaf node, it <em>averages</em> those 3 data points to give a prediction. But for the same reason, it <em>generalizes better. </em>The numbers that work well are 1, 3, 5, 10, 25, but it is relative to your overall dataset size.</p><h3>ii. Max Features</h3><p>Similar to sampling rows while bootstrapping, we can <em>sample features</em> before choosing to split on it.</p><pre>m = RandomForestRegressor(n_estimators=40, n_jobs=-1, <br>                          min_samples_leaf=3, max_features=0.5)</pre><p>That’s important because, if there exists a<em> super important feature,</em> every tree would make that their <em>first split.</em> Thus your trees will have a <em>high correlation.</em> Which we do not want. There might exist <em>other important interactions</em> between features that might not be explored.</p><p>The above forest samples <em>0.5*no_of_features, </em>and then brute forces through them to find the best split. Other good values are sqrt and log2.</p><h3>iii. No of Estimators</h3><p>Boosting the no. of trees might increase the accuracy. But the graph flattens out. We can use this parameter as our last resort to stretch it out to its limits.</p><pre>from sklearn import metrics</pre><pre>preds = np.stack([tree.predict(X_valid) for tree in m.estimators_])<br>plt.plot([metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(40)]);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*K52fjVdWcfT1lNz4.png" /></figure><h3>Bagging and Subsampling</h3><p>As mentioned earlier, random forests by default are built on a <a href="https://en.wikipedia.org/wiki/Bootstrapping_(statistics)">bootstrapped</a> sample. The sample is picked <em>randomly with replacement</em> but is the <em>same size as the entire training set</em> for each tree (389,125). And the collective predictions from the trees is averaged in the end. This is what <a href="https://en.wikipedia.org/wiki/Leo_Breiman">Leo Breiman</a>, the creator of random forests describes as <em>bootstrap aggregation (</em><strong><em>bagging</em></strong><em> for short).</em></p><pre>m = RandomForestRegressor(n_estimators=40, n_jobs=-1)<br>%time m.fit(X_train, y_train)  <br>print_score(m)</pre><pre><strong>Output:</strong></pre><pre>&gt; CPU times: user 6min 50s, sys: 4.37 s, total: 6min 54s<br>&gt; Wall time: <strong>2min 15s</strong><br>&gt; RMSE Train:  7.856880831112418<br>&gt; RMSE Valid:  23.892947209645595<br>&gt;  R2  Train:  <strong>98</strong>.70986499747674<br>&gt;  R2  Valid:  <strong>89</strong>.80499551721383</pre><p>Training on so much duplicate data is unnecessary for experimentation. It raises the training time to more than 2 mins and is overfitting on our training set. And unlike 1999, when Breiman introduced the world to <em>bagging, </em>we have a lot of data to train our models on.</p><blockquote><em>So instead of sampling rows </em><strong>with</strong><em> replacement for each tree during bagging, why not sample them </em><strong>without</strong><em> replacement?</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*IiHNFeVCYVTn0WFo.jpg" /></figure><p>This is called <strong><em>subsampling.</em></strong> It is like having a <em>team of scientists</em>, working on <em>different datasets</em> of a common problem. Because their datasets have nothing in common, the system generalizes better. And given enough scientists, the team will surely go through all the data.</p><pre>set_rf_samples(20000)</pre><pre>m = RandomForestRegressor(n_estimators=40, n_jobs=-1)<br>%time m.fit(X_train, y_train)<br>print_score(m)</pre><pre><strong>Output:</strong></pre><pre>&gt; CPU times: user 40.6 s, sys: 890 ms, total: 41.5 s<br>&gt; Wall time: <strong>19.3 s</strong><br>&gt; RMSE Train:  22.767005816704206<br>&gt; RMSE Valid:  26.17177986481096<br>&gt;  R2  Train:  <strong>89</strong>.16705188450162<br>&gt;  R2  Valid:  <strong>87</strong>.7675206461405</pre><p>This function, <a href="https://github.com/fastai/fastai/blob/775dcd03f49f4f6385f19b503f954468b83afbea/old/fastai/structured.py#L398-L409">set_rf_samples</a>(20000) gives each tree a different random sample of 20,000 rows to train on. It <em>increases the variance</em> in our predictions and trains <em>faster</em> as each tree gets a smaller dataset.</p><p>This function is not available in scikit-learn, and is a workaround made by Jeremy Howard as a part of the fast.ai library. You can reset to a normal bootstrapped model by using reset_rf_samples().</p><h3>When to Use What?</h3><p>According to Breiman, there are 2 things you are trying to balance while using random forests:</p><ul><li>Each tree <em>(aka estimators)</em> is trained as well as possible.</li><li>But across estimators, the correlation between them is as low as possible.</li></ul><p><strong>Subsampling</strong> gives speed and an ability to generalize. Thus it can be used for any kind of feature engineering or experimentation. <strong>Bagging</strong> helps train the trees very well. We’ll use this after we freeze the architecture and features.</p><h3>Bonus Metric — OOB Score:</h3><p>Whether we sample <strong>with</strong> (bootstrapping) or <strong>without</strong> replacement (subsampling), not all the rows in the training set will be used for each tree. This gives random forests a <em>free validation set</em>. Calculating the R² using these<em> out of the bag (OOB) rows </em>as a validation set gives us an idea of how well is the model generalizing over the training set. It is mostly used with bagging.</p><pre><br>reset_rf_samples()</pre><pre>m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, <br>                      max_features=0.5, n_jobs=-1, oob_score=True)<br>%time m.fit(X_train, y_train) <br>print_score(m)</pre><pre><strong>Output:</strong></pre><pre>&gt; CPU times: user 3min 13s, sys: 2.6 s, total: 3min 16s<br>&gt; Wall time: 1min 6s<br>&gt; RMSE Train:  11.906626005969184<br>&gt; RMSE Valid:  22.80435877833463<br>&gt;  R2  Train:  <strong>97</strong>.03713235875033<br>&gt;  R2  Valid:  <strong>90</strong>.71282339508942<br>&gt;    OOB    :  <strong>91</strong>.19405377806355</pre><p>Be careful while using the OOB score with set_rf_samples(20000). If the training set consists of a million rows and you give each tree around 20k rows, the validation set for an OOB score would be huge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/0*Cj4l8QQ0nYREk3bC.jpg" /></figure><p>That’s enough detail about the random forests for now. In the part 2 we’ll see how this algorithm helps us with feature engineering.</p><p><a href="https://medium.com/@sdhnshu/feature-engineering-with-random-forests-part2-160eb0356172"><strong><em>Link to Part 2 of the series</em></strong></a></p><h3>Further Reading</h3><ul><li><a href="https://medium.com/@hiromi_suenaga/machine-learning-1-lesson-1-84a1dc2b5236">Fast.ai ML course notes</a></li><li><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4120293/">Log-transformation and its implications for data analysis</a></li><li><a href="https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779">6 Different Ways to Compensate for Missing Values In a Dataset</a></li><li><a href="https://machinelearningmastery.com/how-to-calculate-nonparametric-rank-correlation-in-python/">How to Calculate Nonparametric Rank Correlation in Python</a></li><li><a href="https://machinelearningmastery.com/data-leakage-machine-learning/">Data leakage in machine learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=601c66dfb09b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Affordable Deep Learning with automated AWS Spot Instances]]></title>
            <link>https://medium.com/@sdhnshu/pro-deep-learning-setup-at-90-off-e9e68f5e84ec?source=rss-e571a28d699e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9e68f5e84ec</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[spot-instances]]></category>
            <dc:creator><![CDATA[Sudhanshu Passi]]></dc:creator>
            <pubDate>Thu, 14 Feb 2019 14:39:19 GMT</pubDate>
            <atom:updated>2020-06-12T19:07:06.017Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ws2g7yqu0dYEcmwO.jpg" /><figcaption>A Portal to Jupyter</figcaption></figure><p><strong><em>Amazon Spot instances</em></strong> offer <strong><em>spare compute capacity</em></strong> available at <strong><em>steep discounts</em></strong>. You can check out the pricing <a href="https://aws.amazon.com/ec2/spot/pricing/">here</a>. The only catch is, these instances can be taken away from you anytime.</p><p>For a person who wants to focus on deep learning and not on making spot instances <strong><em>reliable</em></strong>, I’ve automated the process and set up the project below:</p><blockquote><a href="https://github.com/sdhnshu/Deep-Jupyter-Portal">https://github.com/sdhnshu/Deep-Jupyter-Portal</a></blockquote><p>Spot instances can be launched with any <em>AMI (Amazon Machine Image)</em>. We’ll be choosing an AMI with <em>all deep learning libraries</em> installed. The package includes some storage (in our case 60–75GB) and I am choosing a <em>g2.2xlarge </em>machine which has 8 core CPU, 15 GB RAM, and NVIDIA GK104 GPU with 8GB graphics memory.</p><p>In the time of my using spot instances, I’ve seen them stable for days. But we’ll setup another persistent storage (20 GB) that will never be taken away from us, later in this post. All our data will be saved in this volume.</p><h3>What would you get?</h3><p>Once your preferences are setup, with <strong>one</strong> command, you can spawn a <em>cheap AWS instance</em>. With another you can launch a <strong><em>jupyter notebook</em></strong> on it, which automatically opens up in your browser. It also automatically <em>syncs</em> back the data from the server to your machine.</p><h3>4 step installation</h3><p>We’ll be using a library called portal-gun (v 0.2.0). It can automate the reservation of a spot instance. Check out its <a href="https://portal-gun.readthedocs.io/en/stable/">documentation</a> for more details.</p><p>Clone this <a href="https://github.com/sdhnshu/Deep-Jupyter-Portal"><em>https://github.com/sdhnshu/Deep-Jupyter-Portal</em></a> and setup a python 2.7 environment. Install the <em>requirements.txt</em> in the environment. To setup the config for <em>portal-gun,</em> we need to setup four things in AWS.</p><h3>1. IAM user</h3><p>Portal-gun will need some access keys for communicating to AWS. Create a new IAM user <a href="https://console.aws.amazon.com/iam/home">here</a>, and give it programmatic access. You’ll see the following screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*brtvmLnSuqz1GQoKu4ancg.png" /><figcaption>Creating IAM user</figcaption></figure><p>To attach a policy, create a new one with the following permissions and attach it to the user. These are all the permissions portal-gun needs to access the servers.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;iam:PassRole&quot;,<br>                &quot;ec2:DescribeAccountAttributes&quot;,<br>                &quot;ec2:DescribeAvailabilityZones&quot;,<br>                &quot;ec2:DescribeSubnets&quot;,<br>                &quot;ec2:CreateVolume&quot;,<br>                &quot;ec2:ModifyVolume&quot;,<br>                &quot;ec2:AttachVolume&quot;,<br>                &quot;ec2:DetachVolume&quot;,<br>                &quot;ec2:DeleteVolume&quot;,<br>                &quot;ec2:DescribeVolumes&quot;,<br>                &quot;ec2:DescribeVolumeStatus&quot;,<br>                &quot;ec2:DescribeVolumeAttribute&quot;,<br>                &quot;ec2:DescribeVolumesModifications&quot;,<br>                &quot;ec2:RequestSpotFleet&quot;,<br>                &quot;ec2:CancelSpotFleetRequests&quot;,<br>                &quot;ec2:RequestSpotInstances&quot;,<br>                &quot;ec2:CancelSpotInstanceRequests&quot;,<br>                &quot;ec2:ModifySpotFleetRequest&quot;,<br>                &quot;ec2:ModifyInstanceAttribute&quot;,<br>                &quot;ec2:DescribeSpotFleetRequests&quot;,<br>                &quot;ec2:DescribeSpotInstanceRequests&quot;,<br>                &quot;ec2:DescribeSpotFleetInstances&quot;,<br>                &quot;ec2:DescribeSpotPriceHistory&quot;,<br>                &quot;ec2:DescribeSpotFleetRequestHistory&quot;,<br>                &quot;ec2:DescribeInstances&quot;,<br>                &quot;ec2:DescribeInstanceStatus&quot;,<br>                &quot;ec2:DescribeInstanceAttribute&quot;,<br>                &quot;ec2:CreateTags&quot;,<br>                &quot;ec2:DeleteTags&quot;,<br>                &quot;ec2:DescribeTags&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><p>You’ll get an access key and a secret key after the user is created. Make a file<em> ~/.portal-gun/config.json</em> if not present and add/replace the following lines to it.</p><pre>{<br>    &quot;aws_region&quot;: &quot;us-east-1&quot;,<br>    &quot;aws_access_key&quot;: &quot;AKIAXXXXXXXXX7F6QQ&quot;,<br>    &quot;aws_secret_key&quot;: &quot;Ji4esAyXXXXXXXXXXXXXXXXXXXXXXXXXtrQBjcg&quot;<br>}<br></pre><p>I’ll be working in the US East (N Virginia) region, because they have all the GPU machines one can think of.</p><h3>2. RSA keys</h3><p>To ssh into the server, we need to add your laptop’s RSA public key (<em>~/.ssh/id_rsa.pub</em>) to the key-pairs in the EC2 tab. Note down the key pair name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*gwxJfptEoI-APfwDPDpV_g.png" /><figcaption>RSA key pair</figcaption></figure><h3>3. Security Group</h3><p>Create a new security group using the <em>network and security</em> section on the left. Make a group with 22 (ssh) and 8888 (jupyter) ports open. This would allow communication through these ports. Note down the security group id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*X9QsJLMxbzC3jmsW7WLYvA.png" /><figcaption>create security group</figcaption></figure><h3>4. Persistent volume</h3><p>The volumes with the AMI would be destroyed when the spot instance is taken away from you. Create a volume in the <em>us-east-1a </em>region using the <em>Elastic Block Store </em>section on the left. This would be used for our permanent storage. Note down the volume id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgOYPAioOK35WLVfR6DOYw.png" /><figcaption>volumes for ec2</figcaption></figure><h3>5. AMI (optional)</h3><p>I’ve chosen the <em>deep learning AMI v21.0. </em>It has all deep learning frameworks installed with CUDA and CuDNN. Note down the AMI id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HEGhWvHprMN7pT9imtw30g.png" /><figcaption>deep learning AMIs</figcaption></figure><p><strong><em>If you are using the same AMI as I am, you can jump to the next section.</em></strong></p><p>To find out what mount points are free in the AMI, we need to first start an instance just with <em>just</em> the AMI and run ‘<em>lsblk’.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*PejpOV_FJ9QooVSSaMrjUw.png" /><figcaption>lsblk</figcaption></figure><p>Here we can see 2 disks xvda and xvdb are mounted. We can use generally use xvd(g-j) for our usage. If it says sda, sdb in your AMI, choose something within sd(g-j). I’ve chosen xvdg.</p><h3>Verifying the config file</h3><p>Open up the<em> deep_gpu.json </em>file from the repo<em>.</em> Check all marked values and change them to your needs. Put your account number in the morphed location in <em>iam_fleet_role</em>. You can find your account number <a href="https://console.aws.amazon.com/support/home">here</a> on the top left.</p><p>The <em>channels</em> block is for a real-time sync of data between your local machine and the server. Direction <em>in </em>specifies <strong><em>server to local</em></strong> and <em>out </em>stands for<strong> <em>local to server.</em></strong></p><p><em>Remember to keep both local_paths different</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*DusmttvQI49TXsoZdOK71A.png" /><figcaption>deep_gpu.json</figcaption></figure><h3>Opening the portal</h3><p>To start the portal, we use the following command from the path where <em>deep_gpu.json </em>is located.</p><pre>portal open deep_gpu</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*Po9bZb0X8gg3Qdoz39MnXw.png" /><figcaption>opening a portal</figcaption></figure><p>For me it takes around 4 minutes to find and setup the machine. To interact with the AWS instance, use the following commands.</p><pre>portal open deep_gpu<br>portal info deep_gpu<br>portal ssh deep_gpu<br>portal close deep_gpu</pre><h3>Starting the jupyter notebook</h3><pre>python start_jupyter.py</pre><p>This will run the jupyter notebook on the machine and open it in your browser. <strong><em>The jupyter service might take a while to start.</em></strong> Refresh the browser in a while.</p><p>Starting the jupyter notebook also starts the sync process and syncs both folders as anything changes in them.</p><p>After you are done working, just use the following command to terminate the spot instance.</p><pre>portal close deep_gpu</pre><p><em>Reach out to me in the comments if you have any issues.</em></p><blockquote>PS. Shout out to <a href="https://hackernoon.com/@coderik">Vadim Fedorov</a> for making the library (portal-gun). You can check out his blog post <a href="https://hackernoon.com/deep-learning-on-amazon-ec2-spot-instances-without-the-agonizing-pain-4cedf9b129c4">here</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9e68f5e84ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Asynchronous python app architecture]]></title>
            <link>https://medium.com/cowrks/asynchronous-python-app-architecture-5395d5338c4a?source=rss-e571a28d699e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5395d5338c4a</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cowrks]]></category>
            <category><![CDATA[sanic]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[asynchronous]]></category>
            <dc:creator><![CDATA[Sudhanshu Passi]]></dc:creator>
            <pubDate>Mon, 30 Jul 2018 14:19:38 GMT</pubDate>
            <atom:updated>2018-10-16T09:39:18.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y_3JGr5VRvkcAcjE.png" /></figure><p>Choosing an application architecture greatly depends on the <strong><em>scale</em></strong> you are building it for.</p><blockquote><strong>1x</strong> Monolith<br><strong>10x</strong> Micro service<br><strong>100x</strong> Asynchronous Micro service<br><strong>1000x</strong> Distributed</blockquote><p>In this post, we’ll be deep-diving into how to build an asynchronous python api with async db connections.</p><p>We’ll be using <strong><em>sanic</em></strong> (v0.7) as the api framework and <strong><em>asyncpg</em></strong> (v0.15) to connect to a postgres db. You can use any async db framework you prefer.</p><h3>How we use async at CoWrks?</h3><p>We started using sanic after we had experimented with various other api frameworks like falcon, flask, tornado and twisted. In a few months, python added asyncio to its framework. Libraries started popping up using this async feature. We found this framework when it had around 500 stars on github. And the community grew so fast and at the time of this writing it has shot up to over 10k stars. We started using it on our production servers and have yet to encounter any serious issues.</p><p>Due to that switch, we had to use async db frameworks, async file readers, async requests package and many more async packages. We have started building async libraries for our apps and are quite satisfied by the performance.</p><h3>Is python purely asynchronous?</h3><p>No. The most that we can get with python is context switching between different tasks so that the thread is being used to its optimum processing capacity. (<em>Don’t worry if you didn’t get this, we’ll come back to this later.</em>)</p><h3>How does async in python work?</h3><p>The concept of asynchronous programming was introduced into python 3.4. It was introduced very late due to a fundamental python concept — Global Interpreter Lock (GIL) which blocked it from being asynchronous. If you do want to elevate the api scale to a massive level, you might have to resort to some other approach for instance having a distributed cluster, building an api that supports pure async eg: c, cpp.</p><p>Almost all asynchronous python frameworks use python’s standard library: <strong><em>asyncio</em></strong> underneath. All of these async libraries have similar architectures.</p><h4>Event loop</h4><p>The central part of async is an <strong><em>event loop</em></strong><em>.</em> As the name suggests, it is a loop. One python event loop runs on one OS <strong><em>thread.</em></strong> So only one calculation will be executed by the processor at a time. You can add <strong><em>tasks</em></strong> to this loop and it executes them on a FIFO fashion. So it’s really important that your tasks have <strong><em>a trigger </em></strong><em>(more about it later)</em> that passes the control back to the loop when it is doing an input/output function that does not require processing power. This transfer of control is the central theme of any async framework. Once the control is back at the loop, it can execute the other tasks in the loop.</p><p>Here’s how you can create an event loop using asyncio:</p><pre>loop = asyncio.get_event_loop()<br>loop.run_until_complete(tasks())</pre><p>The command in the second line is directing the loop to run until all the tasks are complete. The above function <em>tasks</em> is an async function. We’ll explain how to define an async function later on.</p><p>The loop controls what task gets executed when, and does the context switching between different tasks.</p><h4>Defining async functions</h4><p>Here’s the syntax of async function:</p><pre><strong>async</strong> def function_name():<br>    resp = <strong>await</strong> some_other_async_function()<br>    return resp</pre><p>This function is added to an event loop to be executed as a task.</p><p>Here you can see two new things <strong><em>async</em> </strong>and<strong> <em>await</em></strong><em>. Async </em>means the function is asynchronous. When the interpreter reaches the <em>Await </em>part, it passes the control back to the event loop <strong><em>(the trigger)</em></strong> and waits for <em>some_other_async_function </em>to complete. <em>some_other_async_function </em>is a function that doesn’t require computation, it can be a input/output operation like a db fetch or a db input. So while that db query is being executed, other tasks in the event loop can be executed. And when the response comes back from the db function, it goes back to the <em>resp </em>variable and is returned.</p><p>So there’s nothing special about this function except that it can pause itself and give others the time to execute. It doesn’t <strong><em>block</em></strong> the thread when not needed.</p><p>So an async function can call another async function; it can also call a sync function. But remember that it will block the thread till it completes so it must be using some computation. Also every <em>async</em> function will be <em>await</em>ed by the one who called it.</p><h3>Sanic</h3><p>Sanic is a api framework that uses <a href="https://magic.io/blog/uvloop-blazing-fast-python-networking/"><strong>uvloop</strong></a><strong> </strong>as its event loop. Uvloop is what makes sanic blazingly fast. Its around <strong>2x faster than any nodejs</strong> async server. <strong>Almost 5x faster than tornado or twisted</strong> (python api frameworks) and has half the latency of any other servers (and is obviously around 10x faster than any sync api framework like falcon, flask, django). Uvloop is written in Cython and is built on top of a asynchronous library written in C.</p><p>Let’s check out an example:</p><pre>app = sanic.Sanic()</pre><pre>app.add_route(root_func, &#39;/&#39;, methods=[&#39;GET&#39;])<br>app.add_route(users_func, &#39;/users/&lt;uid&gt;&#39;, methods=[&#39;GET&#39;])</pre><pre>if __name__ == &quot;__main__&quot;:<br>    app.run(host=&quot;0.0.0.0&quot;, port=8000, workers=2)</pre><p>Here, <em>root_func, users_func </em>are async functions.</p><pre>async def root_func(request):<br>    resp = await do_whatever()<br>    return sanic.response.json({&#39;resp&#39;: resp})</pre><pre>async def users_func(request, uid):<br>    resp = await do_whatever()<br>    return sanic.response.json({&#39;resp&#39;: resp})</pre><p>In a sync api framework, the app can take only one request at a time, process it completely and then move on to other requests. And if that process is an i/o operation and takes a lot of time, the app will be stuck till it is completed. But here, it can be awaited and the app can work on other requests while the i/o operation returns a response.</p><p>Hence we are not executing things in parallel, just switching between tasks when free.</p><p>To add some more speed to the app, we can increase the number of workers while creating the app. But remember, a higher number of workers doesn’t necessarily mean faster speed (<em>processors are also limitedly async.</em>)</p><h3>Asyncpg</h3><p>To use asyncpg or any other async library with sanic, we need to perform one more step. Sanic does not use the default asyncio loop underneath. It uses another loop called uvloop which is a faster version of the asyncio loop. So when we define the asyncpg connection, we need to connect that task to the uvloop which is already running instead of creating another event loop and adding tasks to that. In order to do that you need access to the uvloop object. Here’s how you do it:</p><pre>app = sanic.Sanic()</pre><pre>@app.listener(&#39;before_server_start&#39;)<br>async def before_start(app, uvloop):<br>    db = await asyncpg.connect(postgresdb, loop=uvloop)</pre><pre>app.add_route(root_func, &#39;/&#39;, methods=[&#39;GET&#39;])<br>app.add_route(users_func, &#39;/users/&lt;uid&gt;&#39;, methods=[&#39;GET&#39;])</pre><pre>if __name__ == &quot;__main__&quot;:<br>    app.run(host=&quot;0.0.0.0&quot;, port=8000, workers=2)</pre><p>Now this db variable can be used with your project and is adding tasks in the uvloop. The function <em>before_start</em> is a sanic special function that gives you access to things before the server has started.</p><h3>Bonus points</h3><h4>Semaphore</h4><p>To avoid the async api from overloading due to a Denial Of Service (DOS) attack, you can add a <em>semaphore </em>to limit the number of simultaneous api requests that are computed by doing the following:</p><pre>app = sanic.Sanic()</pre><pre>@app.listener(&#39;before_server_start&#39;)<br>async def before_start(app, uvloop):<br>    sem = await asyncio.Semaphore(100, loop=uvloop)</pre><pre>app.add_route(root_func, &#39;/&#39;, methods=[&#39;GET&#39;])<br>app.add_route(users_func, &#39;/users/&lt;uid&gt;&#39;, methods=[&#39;GET&#39;])</pre><pre>if __name__ == &quot;__main__&quot;:<br>    app.run(host=&quot;0.0.0.0&quot;, port=8000, workers=2)</pre><pre>---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----</pre><pre>async def users_func(request, uid):<br>    async with sem:<br>        resp = await do_whatever()<br>        return sanic.response.json({&#39;resp&#39;: resp})</pre><pre>async def root_func(request):<br>    async with sem:<br>        resp = await do_whatever()<br>        return sanic.response.json({&#39;resp&#39;: resp})</pre><p>Here I’ve set a semaphore for 100 and attached it to the uvloop. And every api function will be called only when the semaphore lock is open.</p><h4>Async classes</h4><p>Classes with async functions are defined as:</p><pre>class Class_name:</pre><pre>def __init__(self):<br>        pass</pre><pre>async def func1(self):<br>        pass<br>   <br>    async def func2(self):<br>        pass</pre><p>They are initialized normally but the functions will be awaited because they are async.</p><pre>c = Class_name()<br>resp = await c.func1()<br>resp = await c.func2()</pre><h3>Conclusion</h3><p>Asynchronous application architecture is fairly easy to understand and is used widely in all languages especially large scaling apps. It is also a useful tool to have in your arsenal if you are looking towards upgrading your coding game. If you want to showcase your abilities and have a drive to learn, we at CoWrks are looking for you. You can apply at <a href="https://careers.cowrks.com/">https://careers.cowrks.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5395d5338c4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cowrks/asynchronous-python-app-architecture-5395d5338c4a">Asynchronous python app architecture</a> was originally published in <a href="https://medium.com/cowrks">CoWrks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>