<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Beng Chew on Medium]]></title>
        <description><![CDATA[Stories by Beng Chew on Medium]]></description>
        <link>https://medium.com/@benlc77?source=rss-c0655757995------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*AcUzI9cPBIqFVfsUxvfuWA.png</url>
            <title>Stories by Beng Chew on Medium</title>
            <link>https://medium.com/@benlc77?source=rss-c0655757995------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:19:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@benlc77/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[4 Minutes Guide to Clean Code for Data Scientists]]></title>
            <link>https://medium.com/@benlc77/4-minutes-guide-to-clean-code-for-data-scientists-73b5380e5b0a?source=rss-c0655757995------2</link>
            <guid isPermaLink="false">https://medium.com/p/73b5380e5b0a</guid>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Beng Chew]]></dc:creator>
            <pubDate>Tue, 12 Apr 2022 16:19:01 GMT</pubDate>
            <atom:updated>2022-04-12T16:19:01.157Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3H8W4VV2-C9yH4-S" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>“Hey, can we quickly build a proof-of-concept ML solution? Don’t worry so much about the code, as long as it works.”</blockquote><p>It baffles me whenever I hear data scientists are reluctant to embrace clean code practice just because of time constraints. There could be numerous reasons why clean code is important such as better code maintainability, scalability, and readability. Apart from all that, clean code is a MUST because it will be easier to collaborate<strong>.</strong></p><p>Personally, I did not practice clean code during my first data science project. It ended up having me working in silo and always hesitant to accept new suggestions from teammates because I couldn’t plug-and-play their functions.</p><p>Writing clean coding practice is not a choice. It should be implanted in you whenever you write any code. Now, here are the 7 tips that you can follow to become a “hygienic” data scientist.</p><h4><strong>1. Be descriptive in naming variables or functions</strong></h4><p>Your code should be self explanatory to your teammates without any comments. <strong>Always be very clear to what the function intention is</strong>. It is fine to have long names, but personally I try not to have more than 3 words function name. Avoid using acronym which only you will understand. Here is a general rule of thumb for naming convention.</p><ul><li><em>Variables — use noun or be + adjectives. E.g. is_valid or customer_data</em></li><li><em>Functions — use verb. E.g. generate_months</em></li><li><em>Classes — use noun. E.g. TwitterRecommendation</em></li></ul><h4>2. Be consistent with the naming</h4><p>Don’t use different naming convention for the same thing. For example,clean_data(), tidy_data()and prepare_data()all could give similar meaning. <strong>Choose one and stick to it</strong>.</p><h4>3. Every function should be doing only one ‘thing’</h4><p>No matter how neat your code is, it is still difficult to glance through big chunks of code. Small function is easier to understand, debug and test. Also, project requirements might be changing from time to time in an agile development environment. Having <strong>small and modularised codes</strong> are probably more convenient in making changes without affecting the rest of the code.</p><blockquote>Not clean code: One function doing two ‘things’</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e33b0f1846bddc8c28121220c2b38a9/href">https://medium.com/media/7e33b0f1846bddc8c28121220c2b38a9/href</a></iframe><blockquote>Clean code: One function doing one ‘thing’</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78519a6443ffd023fb0aab6b2c455c87/href">https://medium.com/media/78519a6443ffd023fb0aab6b2c455c87/href</a></iframe><h4>4. Stay “DRY” — Don’t Repeat Yourself</h4><p>If you found yourself copy and paste lines of code multiple times, it is a sign of code duplication. Aside from having longer script, code duplication script is more prone to human error. This is because whenever you change a part of the code, you need to change it at multiple places as well. Hence, it is good practice to <strong>wrap it under a function and reuse that function</strong>.</p><h4>5. Refactor your code based on high / low level abstraction</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*Vo2JOmuelK12fmZB4KolWw.png" /></figure><blockquote><strong>“Abstraction </strong>is defined as a process of handling complexity by hiding unnecessary information from the user. “— <strong>Great Learning Team</strong></blockquote><p>Before I was introduced to abstraction concept, I always not sure when I should refactor my functions or classes. Consequently, my code always had different levels of abstraction in one function and resulted to less readable code . To write cleaner code, it is important to ensure <strong>every function only contains one abstraction level</strong>.</p><blockquote>Mixed level of abstraction code — Bad</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/400629ddd07f950c9c5ae29831333064/href">https://medium.com/media/400629ddd07f950c9c5ae29831333064/href</a></iframe><blockquote>Low level of abstraction code — Good</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a42ab4c36f2cd6fa07062586e0d6bda4/href">https://medium.com/media/a42ab4c36f2cd6fa07062586e0d6bda4/href</a></iframe><blockquote>High level of abstraction code — Good</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d1a1f13d3bbbec7dae4dc2b674d9b79/href">https://medium.com/media/2d1a1f13d3bbbec7dae4dc2b674d9b79/href</a></iframe><h4>6. Avoid comment</h4><p>Yes, you see it right. Generally, you should aim to have <strong>as few comments as possible</strong>. This is because your code should be self explanatory. Having many comments will only provide redundant information which makes your code even messier. However, that does not mean you should not have any comments at all. It is okay to leave comments such as to-do notes or additional explanation about why a decision is made. Okay comment: <em>“TODO : Need to filter non active customer” </em>or <em>“Model B is selected because …”</em></p><h4>7. Include docstrings</h4><p>Docstring is a convenient element when you are sharing code with your teammates. It provides a quick understanding to them on what a function does without diving deep into the code. Also, you can use <a href="https://www.sphinx-doc.org/en/master/index.html">Sphinx</a>to automatically generate a basic scaffold for the documentation of your project. Hence, it is a good practice to add docstrings whenever possible.</p><h3>Closing thoughts —</h3><p>Even though I am emphasising the importance of clean code, it does not mean you should overly focus on writing perfect code from now on. It’s okay to start with working code. Then, what you can do is try to allocate 10 minutes every day to clean up your code. Clean code takes time. You’ll get better.</p><h3>References</h3><ol><li>Martin, R. C. (2009). <em>Clean code. A Handbook of Agile Software Craftsmanship</em>. Pearson Education.</li><li>Great Learning Team (2021, May 13). <em>Abstraction in python</em>. Retrieved April 20, 2022, from <a href="https://www.mygreatlearning.com/blog/abstraction-in-python/#:~:text=Abstraction%20in%20python%20is%20defined,oriented%20programming%20(OOP)%20languages.">greatlearning blog</a>.</li></ol><p><em>I share about my data science knowledge &amp; learning experience. Follow &amp; subscribe my blog if you want to learn more from my sharing.</em></p><p><em>Hope you enjoyed this read. Have a nice day!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73b5380e5b0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Write Clean And Scalable Code With Custom Transformers & Sklearn Pipelines]]></title>
            <link>https://medium.com/@benlc77/how-to-write-clean-and-scalable-code-with-custom-transformers-sklearn-pipelines-ecb8e53fe110?source=rss-c0655757995------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecb8e53fe110</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sklearn-pipeline]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Beng Chew]]></dc:creator>
            <pubDate>Sat, 12 Mar 2022 15:05:12 GMT</pubDate>
            <atom:updated>2022-03-12T15:05:12.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vY_tqeamfTKEb04U" /><figcaption>Photo by <a href="https://unsplash.com/@frostroomhead?utm_source=medium&amp;utm_medium=referral">Rodion Kutsaev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When I created my first Machine Learning model, I was thrilled and even jumped out of my chair. After feeling satisfied for the first 10 seconds, I returned to wrap up my work by applying the same preprocessing step on test dataset. Thinking I could finish up in 5 minutes, and the next thing I see…</p><blockquote>ValueError: Found input variables with inconsistent numbers of samples: [365, 1459]</blockquote><blockquote>ValueError: Input contains NaN, infinity or a value too large for dtype(‘float64’).</blockquote><blockquote>ValueError: Some of the variables to transform contain NaN. Check and remove those before using this transformer.</blockquote><h4>Error, error and another error!!!!</h4><p>Usually for this kind of errors, it means that the test dataset is different from the train dataset. So, I ended up troubleshooting the error, changing code, refitting model, copy-pasting the same preprocessing code and re-applying it on test dataset.</p><p>As you can see, it gets very annoying if the preprocessing steps are long. To quickly edit code, refit and transform new dataset, Pipeline will be useful in creating repeatable and customizable workflows.</p><h3>Introduction of scikit-learn pipeline</h3><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html">Pipeline </a>is a scalable framework used in scikit-learn package. It can combine multiple transformation steps (e.g. one-hot encoding, missing imputation, scaling &amp; etc.) together sequentially or in parallel without having data leakage issue. Also, you can easily pass this pipeline to Optuna, (stay tune in the next post) a hyperparameter tuning package, and find the best parameter set.</p><p>This post will use a classic regression example — <a href="https://www.kaggle.com/c/house-prices-advanced-regression-techniques">predicting house prices</a> to demonstrate 4 practical steps in using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html">Pipeline</a>, which are how to:</p><ol><li><strong>Structure a workflow systematically before writing any pipeline code</strong></li><li><strong>Create custom transformers in </strong><strong>Pipeline</strong></li><li><strong>Apply modular approach when building pipeline</strong></li><li><strong>Use same pipeline with different models to evaluate models’ performance quickly</strong></li></ol><h4>House Prices Data Introduction</h4><p>Before I dive deep into how to apply Pipeline, here are the 2 keys summary of this dataset.</p><ul><li>It has 80 features with a mixture of integer, float and categorical data type</li></ul><pre><strong>import</strong> pandas <strong>as</strong> pd<br><strong>import</strong> numpy <strong>as</strong> np</pre><pre># dataset url<br># &#39;https://www.kaggle.com/c/house-prices-advanced-regression-techniques&#39;</pre><pre>train = pd.read_csv(&quot;house-prices-advanced-regression-techniques/train.csv&quot;, index_col =&#39;Id&#39;)</pre><pre>test = pd.read_csv(&quot;house-prices-advanced-regression-techniques/test.csv&quot;, index_col =&#39;Id&#39;)</pre><pre>X = train.drop(&#39;SalePrice&#39;, axis = 1)<br>y = train[&#39;SalePrice&#39;]</pre><pre>train</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfKLZ5Mj5Dz9ZvI-ERpwpg.png" /><figcaption>Example of house prices dataset</figcaption></figure><ul><li>18 features has missing data. Out of 18 features, 5 are considered high missingness (&gt;50%) and 13 are considered low missingness (&lt;50%).</li></ul><pre>import missingno as msno</pre><pre># check pattern of missingness<br>msno.matrix(train.iloc[:,:50])<br>msno.matrix(train.iloc[:,50:])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3g1i6E2QLsz44CHBNC_UGQ.png" /><figcaption>House prices missing data example</figcaption></figure><h3>Step 1: <strong>Structure a workflow systematically before writing any pipeline code</strong></h3><p>Before you jump directly into writing pipeline code, it is important to have a “plan of attack”. I recommend you to spend some time sketching out an end-to-end Machine Learning pipeline first. Having a visual perspective of the workflow will help you self-validate in your mind whether every step makes sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*8_VsDgiTGendHAFCxiOJhA.png" /><figcaption>“Plan of attack” for house price prediction</figcaption></figure><p>In this example, the preprocessing steps are categorised into 3 following sections:-</p><h4>1) Filtering</h4><ul><li>Drop columns with more than 50% of missingness</li><li>Select numerical / category features depending on the pipeline flow</li></ul><h4>2) Missingness Treatment</h4><ul><li>Median imputation for numerical features</li><li>Most frequent imputation for categorical features</li></ul><h4>3) Feature Engineering</h4><ul><li>Create practical and predictive features using domain knowledge (e.g. adding all bathrooms). Full details check on this <a href="https://www.kaggle.com/lavanyashukla01/how-i-made-top-0-3-on-a-kaggle-competition#Feature-Engineering">link</a>.</li><li>Encode all categorical features’ items with less than 20% frequency as <strong>‘rare’</strong>.</li><li>One hot encoding all categories features at last step of preprocessing in categorical pipeline.</li></ul><h3>Step 2: Create custom transformers in pipeline</h3><p>When I first started learning pipeline, I found it difficult to put into practical use because of 2 reasons. They are:-</p><ol><li>No simple preprocessing steps like selecting or dropping columns by names</li><li>Scikit-learn transformer returns output as <em>np.array</em> which makes it hard to specify the subsequence steps</li></ol><p>Don’t worry, you won’t experience the struggle I had before once I show you how to create custom transform with examples.</p><p><strong>These are 4 configurations that I always used when creating custom transformer</strong></p><ul><li>AddBaseEstimator &amp; TransformerMixinto inherit <em>get_params </em>and <em>fit_transform </em>function. Personally, I always use <em>fit_transform </em>when I need to have a quick view on the output.</li><li>Defineself.variablesattributes so that you can easily select which columns to be applied on later</li><li>Definefitmethod. Depending on the transformation, if it doesn’t require fitting, just create a dummy fit function.</li><li>Definetransformmethod. This is used to transform original dataset to modified dataset based on your transformation method.</li></ul><p><em>Example 1: Custom transformer without requiring fit method</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca95abf15fb55d14c933ef776590d8da/href">https://medium.com/media/ca95abf15fb55d14c933ef776590d8da/href</a></iframe><p><em>Example 2: Custom transformer requiring fit method</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6ab3794467ec725522ba1d6297799a9/href">https://medium.com/media/d6ab3794467ec725522ba1d6297799a9/href</a></iframe><h3>Step 3: Apply modular approach when creating pipelines</h3><p>In practical ML modelling experiment, it usually involves many preprocessing steps. If you don’t breakdown an end-to-end pipeline into a few smaller pipelines, sometimes pipeline will get very long and complicated. Also, it can become difficult to make changes and compare results of different pipeline performance.</p><p>Over here, we’ll see how to build scikit-learn pipeline with complete example.</p><blockquote>Import packages — sklearn &amp; feature_engine</blockquote><p>Firstly, we will import the necessary packages. Apart for sklearn, we can also integrate other package likefeature_enginefunction into pipeline.</p><pre>from sklearn.base import BaseEstimator, TransformerMixin<br>from sklearn.impute import SimpleImputer<br>from sklearn.preprocessing import StandardScaler, OneHotEncoder<br>from sklearn.model_selection import train_test_split<br>from sklearn.pipeline import Pipeline, FeatureUnion<br>from sklearn.linear_model import LinearRegression<br>from sklearn.ensemble import AdaBoostRegressor, RandomForestRegressor<br>from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score<br>from sklearn.model_selection import KFold, cross_val_score</pre><pre>from feature_engine import encoding</pre><blockquote>Custom transformers</blockquote><p>Following step 1 &amp; 2, we will create 6 custom transformers in order for the pipeline to work.</p><ul><li>DropFeatureSelector — Drop columns based on names</li><li>FeatureSelector —Select columns based on names</li><li>SimpleImputerCustom— Improvise onsklearn.impute.SimpleImputer function so it returns as dataframe instead of np.array.</li><li>DomainNumFE— Feature engineering technique for numerical features based on domain knowledge</li><li>OneHotEncodercustom— Improvise onsklearn.preprocessing.OneHotEncoder function so it returns as dataframe instead of np.array.</li><li>DomainCatFE—Feature engineering technique for categorical features based on domain knowledge</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c264b1dbbaeaf9809bcf37cb03805a40/href">https://medium.com/media/c264b1dbbaeaf9809bcf37cb03805a40/href</a></iframe><blockquote>Columns name list</blockquote><pre># list for numerical columns<br>numerical_cols = [&#39;LotFrontage&#39;,&#39;MasVnrArea&#39;,&#39;GarageYrBlt&#39;,<br>    &#39;MSSubClass&#39;,&#39;LotArea&#39;,&#39;OverallQual&#39;,&#39;OverallCond&#39;,<br>    &#39;YearBuilt&#39;,&#39;YearRemodAdd&#39;,&#39;BsmtFinSF1&#39;,&#39;BsmtFinSF2&#39;,<br>    &#39;BsmtUnfSF&#39;,&#39;TotalBsmtSF&#39;,&#39;1stFlrSF&#39;,&#39;2ndFlrSF&#39;,<br>    &#39;LowQualFinSF&#39;,&#39;GrLivArea&#39;,&#39;BedroomAbvGr&#39;,&#39;TotRmsAbvGrd&#39;,<br>    &#39;GarageArea&#39;,&#39;WoodDeckSF&#39;,&#39;OpenPorchSF&#39;,&#39;EnclosedPorch&#39;,<br>    &#39;3SsnPorch&#39;,&#39;ScreenPorch&#39;,&#39;PoolArea&#39;,&#39;YrSold&#39;,&#39;MiscVal&#39;,<br>    &#39;MoSold&#39;]</pre><pre># list for one hot encoding columns<br>ohe_cols = [&#39;MSZoning&#39;, &#39;Street&#39;,  &#39;LotShape&#39;,&#39;LandContour&#39;,   <br>    &#39;Utilities&#39;,&#39;LotConfig&#39;, &#39;LandSlope&#39;, &#39;Neighborhood&#39;, <br>    &#39;Condition1&#39;, &#39;Condition2&#39;,&#39;BldgType&#39;, &#39;HouseStyle&#39;, <br>    &#39;RoofStyle&#39;,&#39;RoofMatl&#39;, &#39;Exterior1st&#39;,&#39;Exterior2nd&#39;, <br>    &#39;MasVnrType&#39;, &#39;ExterQual&#39;, &#39;ExterCond&#39;, &#39;Foundation&#39;,<br>    &#39;BsmtQual&#39;, &#39;BsmtCond&#39;, &#39;BsmtExposure&#39;, &#39;BsmtFinType1&#39;,   <br>    &#39;BsmtFinType2&#39;,&#39;Heating&#39;, &#39;HeatingQC&#39;, &#39;CentralAir&#39;, <br>    &#39;Electrical&#39;, &#39;KitchenQual&#39;,&#39;Functional&#39;, &#39;GarageType&#39;, <br>    &#39;GarageFinish&#39;, &#39;GarageQual&#39;,&#39;GarageCond&#39;, &#39;PavedDrive&#39;, <br>    &#39;SaleType&#39;, &#39;SaleCondition&#39;]</pre><pre># list for categorical columns<br>categories_cols = ohe_cols + [&#39;BsmtFullBath&#39;,&#39;BsmtHalfBath&#39;,<br>    &#39;FullBath&#39;,&#39;HalfBath&#39;,&#39;KitchenAbvGr&#39;,&#39;Fireplaces&#39;,<br>    &#39;GarageCars&#39;]</pre><blockquote>Pipeline 1 (Component A): Numerical Preprocessing Pipeline</blockquote><p>This pipeline will focus on the preprocessing steps for numerical features. As you can see, it is easy to assign preprocessing task to the features through variablesparameter.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60064bd163496fd583163eb59aa466c3/href">https://medium.com/media/60064bd163496fd583163eb59aa466c3/href</a></iframe><blockquote>Pipeline 2 (Component B): Categorical Preprocessing Pipeline</blockquote><p>This pipeline will focus on the preprocessing steps for categorical features. Interestingly, functions other than sklearnsuch asencoding.RareLabelEncoder function can be easily integrated with other custom transformer. Aside from custom transformers, scikit-learn pipeline also accepts other package functions as long as it has fit &amp; transform configuration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3873c890c6139a6e575cf26d34fda12a/href">https://medium.com/media/3873c890c6139a6e575cf26d34fda12a/href</a></iframe><blockquote>Pipeline 3 (Component A + B): Numerical &amp; Categorical Combined Pipeline</blockquote><p>FeatureUnion is used to combine numerical &amp; categorical pipeline together. In doing so, both pipelines will run in parallel instead of sequential.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/352ebc48e34f88f86c7884040ff319a6/href">https://medium.com/media/352ebc48e34f88f86c7884040ff319a6/href</a></iframe><h3>Step 4: Use<strong> Same Pipeline With Different Models To Evaluate Models’ Performance Quickly</strong></h3><p>If you have followed along until here, congratulations! This is the part where we reap the fruit of our effort. We’ll see how easy to train and evaluate 10+ (I only showed 3 in this example) different models performance in less than 50 lines of code.</p><blockquote>Pipeline 4 (Component A + B + C[m1, m2, m3]) : Numerical &amp; Categorical Combined With Different Models Pipeline</blockquote><p>By using for loop, we are able to fit, predict and evaluate 3 models performance easily.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7af9db85a4a9b1701c0d84a5837f82d2/href">https://medium.com/media/7af9db85a4a9b1701c0d84a5837f82d2/href</a></iframe><p>Judging from the R-squared values, it seems like Random Forecast(RF) is the best model with 0.8844 R-squared value. <em>FYI, the current RF model may have some sign of overfitting issue, but that’s a discussion for another day.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SKYcRLp4crIZ91_7HEm2mg.png" /></figure><h3>Conclusion</h3><p>As a data scientist, apart for getting high accuracy model, it is also important to make your code readable and reproducible. Pipeline is definitely one of the good frameworks.</p><p>Next, I will share with you how to integrate Optuna, an auto hyperparameter tuning package, with Pipeline. Stay tune!</p><p>I hope this guide help you to write better machine learning code.</p><h4>References</h4><p><a href="https://cloud.google.com/ai-platform/prediction/docs/custom-pipeline">https://cloud.google.com/ai-platform/prediction/docs/custom-pipeline</a></p><p><em>I share about my data science knowledge &amp; learning experience. Follow me if you want to learn more from my sharing.</em></p><p><em>Hope you enjoyed this read. Have a nice day!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecb8e53fe110" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How My First Kaggle Competition Changed My Data Science Learning Experience]]></title>
            <link>https://medium.com/@benlc77/how-my-first-kaggle-competition-changed-my-data-science-learning-experience-58f39f293cf5?source=rss-c0655757995------2</link>
            <guid isPermaLink="false">https://medium.com/p/58f39f293cf5</guid>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Beng Chew]]></dc:creator>
            <pubDate>Tue, 15 Feb 2022 17:20:52 GMT</pubDate>
            <atom:updated>2022-02-20T17:57:57.099Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bRzfEjMQU_Bt8Kz5" /><figcaption>Photo by <a href="https://unsplash.com/@giorgiotrovato?utm_source=medium&amp;utm_medium=referral">Giorgio Trovato</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>“Don’t waste your time on Kaggle competition. It won’t be helpful when dealing with real business problem.”</blockquote><p>Someone once told me this when I first started my data science career. As a young, clueless data scientist, I followed this advice for 2 years and had been avoiding Kaggle competitions. Recently, I stumbled upon Abhishek Thakur’s LinkedIn post promoting <a href="https://www.kaggle.com/c/song-popularity-prediction"><em>Song Popularity Prediction</em></a> Kaggle competition.</p><p>After reading his post, I had this urge to join his competition and justify if the past advice was true. So, here are my 5 keys takeaways after completing the competition.</p><ol><li><strong>Generous code sharing in community forum</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*X2i7iarzc2WSiSsqN0qPEw.png" /></figure><p>At first, I thought the overall experience would be mainly competing with everyone and finding out who has the best score. To my surprise, I am amazed by how generous the community is. Even at the beginning of competition, they already started sharing useful EDA codes, feature engineering techniques &amp; etc. I personally felt this was more like everyone working together instead of competing with each other.</p><p><strong>2. Enriched my experience in other types of ML problem</strong></p><p>My job experience mainly focused on time series forecasting &amp; optimization. It is actually a good hands-on opportunity for me to experience common issues faced in different types of ML. For example, this <strong><em>Song Popularity Prediction</em></strong> <em>(classification)</em> problem enabled me to try out resampling methods to tackle imbalanced data issue and understand how to use correct metrics to evaluate models.</p><p><strong>3. Exposed to creative feature engineering &amp; missing imputation techniques</strong></p><p><em>“At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used.”</em></p><p><em>— Prof. Pedro Domingos, University of Washington</em></p><p>With many models to choose from, many people have shifted their focus to model/algorithm development rather than data quality or feature engineering techniques. Personally, I find it more important to focus more on data quality and advanced feature engineering techniques while using simple models.</p><p>In the forum, there are tons of feature engineering and missing data imputation techniques being shared. To be honest, I was not able to try out all of those which I had bookmarked. Here are the followings techniques which I find it interesting:-</p><ul><li>Clusters models (e.g. KMeans) to generate new features</li><li>PCA, a dimensionality-reduction method to reduce the dimensionality features</li><li>Apply log, box-cox, power transformation for data scaling</li><li>LGBM imputer for missing data</li><li>Feature interaction between 2–3 variables based on domain knowledge</li><li>Polynomial feature transformation</li></ul><p><strong>4. Learnt how to write efficient data analysis &amp; plotting code</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*Lj8VI-q1JOrICHxQHDEg0Q.png" /></figure><p>As I have just recently transitioned from R to Python, I am more familiar with <em>ggplot</em> than <em>matplotlib/seaborn</em> code structure. Oftentimes, I will either encounter many code errors or end up with not so pretty plots. So, what I really like about the code sharing forum is they don’t just share code on how to build models, instead they also shared many detailed EDA notebooks with their own findings. Here are two of my personal favorites from the competition.</p><ul><li><a href="https://www.kaggle.com/dextermojo/song-popularity-eda-heads-or-tails-python-ver">Simple EDA using interactive plot</a></li><li><a href="https://www.kaggle.com/headsortails/song-popularity-eda-live-coding-fun">Detailed data analysis report</a></li></ul><p><strong>5. Learnt how to use optuna package to tune models</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*BoLj98GdE9copP3wkiJU5A.png" /></figure><p>As I was scrolling through the sharing notebooks, I found a few notebooks with an interesting term — <strong>Optuna</strong>. So, this got me curious and I decided to check out one of them. And, that’s it! I will definitely be using it for my next ML problem.</p><p>If you have not heard of it just like me, click this <a href="https://optuna.org/">link</a> to find out more. Basically, it is an open source hyperparameter optimization framework to automate hyperparameter search.</p><p><strong>Closing thoughts —</strong></p><p>No doubt, Kaggle is a great platform for beginner or intermediate data scientist to practise their coding skills.</p><p>For advanced user, I find that joining those distinguished competitions such as M5 Forecasting competition will be a challenging yet rewarding experience.</p><p><em>I share about my data science journey &amp; learning experience. Follow me if you want to learn more from my sharing.</em></p><p><em>Hope you enjoyed this read. Have a nice day!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58f39f293cf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>