<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wing Poon on Medium]]></title>
        <description><![CDATA[Stories by Wing Poon on Medium]]></description>
        <link>https://medium.com/@wpoon?source=rss-ad3edf793aa6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yXpY8WFaNH8nbEUdE5PlmQ.jpeg</url>
            <title>Stories by Wing Poon on Medium</title>
            <link>https://medium.com/@wpoon?source=rss-ad3edf793aa6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:13:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wpoon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Use Google Colab Like A Pro]]></title>
            <link>https://pub.towardsai.net/use-google-colab-like-a-pro-39a97184358d?source=rss-ad3edf793aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a97184358d</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[colab]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Wing Poon]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 22:20:52 GMT</pubDate>
            <atom:updated>2022-09-21T00:44:09.458Z</atom:updated>
            <content:encoded><![CDATA[<h4>15 tips to supercharge your productivity</h4><figure><img alt="Google Colaboratory logo on photo of notepad" src="https://cdn-images-1.medium.com/max/1024/1*gD053DApa9pKi13o9eoAOA.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/world-fly-856412/">Alexander</a> on <a href="https://pixabay.com/">Pixabay</a> (modified by author)</figcaption></figure><p>Regardless of whether you’re a Free, Pro, or Pro+ user, we all love Colab for the resources and ease of sharing it makes available to all of us. As much as we hate its restrictions, Google is to be commended for democratizing Deep Learning, especially for students from countries which would otherwise have no opportunity to participate in this ‘new electrification’ of entire industries.</p><p>Have you spent countless hours working in Colab but have yet to invest a few minutes of time to really get to know the tool and be <em>at your most productive</em>?</p><blockquote>“Time is not refundable; use it with intention.”</blockquote><h4>1. Check and Report on your GPU Allocation</h4><blockquote>“Life is like a box of chocolates. You never know what you’re gonna get.” <br> — Forest Gump</blockquote><p>I’m so sorry, we’re out of V100s, let’s set you up with our trusty old K80 with 8GB of RAM. You can try again in an hour if you’re feeling lucky! Uhm, yeah Google, thanks but no thanks.</p><pre>gpu = !nvidia-smi -L<br>print(gpu[0])<br>assert any(x in gpu[0] for x in [&#39;P100&#39;, &#39;V100&#39;])</pre><p>Jokes aside, I’m sure you’ve been burned by going through a lengthy pip install, mounting GDrive, downloading, and preparing your data, only to find out three-quarters ways through your notebook that you forgot to set your runtime to ‘GPU’? Put this assertion on the first cell of your notebook — you can thank me later.</p><h4>2. No More Authenticating “gdrive.mount( )”</h4><p>Not all Python notebooks are the same, Colab treats Jupyter Notebooks as second-class citizens. When you upload your Jupyter Notebook to GDrive, you’ll see that it appears as a blue/folder icon. When you create a new Colab notebook from scratch in GDrive (<strong>+</strong>New Button ➤ More ➤ Google Colaboratory), it’ll appear as an <em>orange icon with the Colab logo</em>.</p><p>This ‘native’ Colab notebook has special powers — namely, Google can <em>automatically</em> mount your GDrive on the Colab VM for you as soon as you open the notebook. If you open a Jupyter notebook and click on the Files icon (left sidebar), and then the Mount Drive icon (pop-out panel top-row), Colab will insert the following new cell into your Jupyter notebook:</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><p>And when you execute that cell, it’ll pop up a new browser tab and you have to go through that annoying i. Select your Account, ii. Allow and iii. Copy-and-Paste iv. Switch and Close Tabs, rigamarole.</p><p>If you have a Jupyter notebook that you frequently open and it needs GDrive access, invest thirty seconds and save yourself that constant hassle. Simply create a new (native) Colab notebook (as described above), then open your existing Jupyter notebook — <em>with Colab — </em>in another browser tab. Click: Edit ➤ Clear all outputs. Then, making sure you’re in Command mode, press &lt;SHIFT&gt; + &lt;CMD|CTRL&gt; + A , then &lt;CMD|CTRL&gt; + C. Go to your new Colab notebook and press &lt;CMD|CTRL&gt; + V . Mount your GDrive using the Mount Drive icon method and delete the old Jupyter notebook to avoid confusion.</p><h4>3. Fastest way to Copy your Data to Colab</h4><p>One way is to copy it from Google Cloud Storage. You do have to sign up for a Google Cloud Platform account, but Google offers a <a href="https://cloud.google.com/free">Free Tier</a>, which gives your 5GB, as long as you select US-WEST1, US-CENTRAL1, or US-EAST1 as your region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qwy3ElpYvmHAvadMJlE4pw.png" /></figure><p>Besides security, another benefit of using Google Storage buckets is if you’re using <a href="https://www.tensorflow.org/datasets/api_docs/python/tfds">TensorFlow Datasets</a> for your own data (if you aren’t, you really <em>should</em> … I’ll be writing an article about why <strong>Follow</strong> me to be notified), you can bypass the copying and load your datasets directly using tfds.load():</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cAnRPtMDOPvSbLCBdu5eA.png" /></figure><p>Another way is to use the wonderful gdown utility. Note that as of this writing (Mar 2022), even though gdown is pre-installed on Colab instances, you have to <em>upgrade</em> the package for it to work. You’ll need to have the file ID, which you can get by Right-Clicking on the file in Drive ➤ Get link ➤ Anyone with link, and then plucking out the ID (shown in bold below) from the provided URL:</p><p>https://drive.google.com/file/d/<strong>1sk...IzO</strong>/view?usp=sharing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gzD4QRyHTWUYgpX2PGnBsw.png" /></figure><p><strong>NOTE:</strong> Since <em>anyone</em> with the link can access your file with this file-sharing permission, use this method only for your <em>personal projects</em>!</p><h4>4. Bypass “pip install …” on Runtime (Kernel) Restarts</h4><p>If you’re like me and are always restarting and re-running the python kernel — usually because you’ve got non-idempotent code like dataset = dataset.map().batch()— save precious seconds and keep your notebooks readable and concise by not invoking pip install every time your restart the runtime, by creating a dummy file and testing for its presence:</p><pre>![ ! -f &quot;pip_installed&quot; ] &amp;&amp; pip -q install tensorflow-datasets==4.4.0 tensorflow-addons &amp;&amp; touch pip_installed</pre><p>In fact, use this same technique to avoid <em>unnecessary</em> copying and downloading of those large data files whenever you rerun your notebook:</p><pre>![ ! -d &quot;my_data&quot; ] &amp;&amp; unzip /content/drive/MyDrive/my_data.zip -d my_data</pre><h4>5. Import your own Python Modules/Packages</h4><p>If you find yourself constantly using a helper.py, or your own private Python package, e.g. to display a grid of images or chart your training losses/metrics, put them all on your GDrive in a folder called /packages and then:</p><pre>import sys<br>sys.path.append(&#39;/content/drive/MyDrive/packages/&#39;)<br>from helper import *</pre><h4>6. Copy files To Google Storage Bucket</h4><p>If you’re training on Google’s TPUs, your data has to be stored in a Google Cloud Storage Bucket. You can use the pre-installed gsutil utility to transfer files from the Colab VM’s local drive to your storage bucket if you need to run some pre-processing on data prior to kicking off each TPU training run.</p><pre>!gsutil -m cp -r /root/tensorflow_datasets/my_ds/ gs://my-bucket/</pre><h4>7. Ensure all Files have been Completely Copied to GDrive</h4><p>As I’m sure you’re well aware, Google can and will terminate your session due to inactivity. There are good reasons not to store (huge) model checkpoints during model training to your mounted GDrive. E.g. so as not to exceed Colab I/O limits, or you’re running low on your GDrive storage quota and need to make use of the ample local disk storage on your Colab instance, etc. To ensure that your final model and data is completely transferred to GDrive if you’re paranoid, call drive.flush_and_unmount() at the very end of your notebook:</p><pre>from google.colab import drive</pre><pre>model.fit(...)  # I&#39;m going to take a nap now, &lt;yawn&gt;<br>model.save(&#39;/content/drive/MyDrive/...&#39;)<br>drive.flush_and_unmount()</pre><p>Note that the completion of copying/writing files to /content/drive/MyDrive/ does not mean that all files are safely on GDrive and that you can immediately terminate your Colab instance because the transfer of data between the VM and Google’s infrastructure happens asynchronously, so performing this flushing will help ensure it’s indeed safe to disconnect.</p><h4>8. Quickly open your local Jupyter Notebook</h4><p>There’s no need to copy your <em>.ipynb</em> to GDrive and then double-click on it. Simply go to <a href="https://colab.research.google.com/">https://colab.research.google.com/</a> and then click on the Upload tab. Your uploaded notebook will reside on GDrive://Colab Notebooks/ .</p><h4>9. Use Shell Commands with Python Variables</h4><pre>OUT_DIR = &#39;./models_ckpt/&#39;<br>...<br>model.save(OUT_DIR + &#39;model1&#39;)<br>...<br>model.save(OUT_DIR + &#39;model2&#39;)<br>...<br>!rm -rf {OUT_DIR}*</pre><p>Make use of the powerful Linux commands available to you … why bother with importing the zip file and requests libraries and all that attendant code? In fact, get the best of both worlds by piping the output of a Linux command to a Python variable:</p><pre>!wget -O data.zip <a href="https://github.com/gohugoio/hugo/archive/master.zip">https://github.com/ixig/archive/data_042020.zip</a><br>!unzip -q data.zip -d ./tmp<br># &#39;wc&#39;: handy linux word and line count utility<br>result = !wc ./tmp/tweets.txt<br>lines, words, *_ = result[0].split()</pre><h4>10. Am I running in Colab or Jupyter?</h4><p>If you switch between running your notebooks on your local machine and training on Colab, you need a way to tell where that notebook is running, e.g. don’t pip install when running on your local machine. You can do this using:</p><pre>COLAB = &#39;google.colab&#39; in str(get_ipython())<br>if COLAB:<br>    !pip install ...</pre><h4>11. Message Me, Baby!</h4><p>No need to sit around waiting for your training to complete, have Colab send you a notification on your phone! First, you’ll need to follow the instructions to allow <a href="https://www.callmebot.com/blog/free-api-signal-send-messages/">CallMeBot</a> to message you on your Signal/FB/WhatsApp/Telegram app. Takes all of one minute — very simple, quick, and safe signup. Then, you can:</p><pre>import requests<br>from urllib.parse import quote_plus</pre><pre>number = &#39;...&#39;<br>api_key = &#39;...&#39;<br>message = quote_plus(&#39;Done Baby!&#39;)<br>requests.get(f&#39;https://api.callmebot.com/signal/send.php?phone={number}&amp;apikey={api_key}&amp;text={message}&#39;)</pre><h4>12. Use IPython Cell Magics</h4><p>Okay, this one is not specific to Colab Notebooks as it applies to Jupyter Notebooks as well, but here are the most useful ones to know about.</p><p>%%capture : Silence the copious, annoying outputs from executing statements in a cell. Useful for those ‘<em>pip install …</em>’, ‘<em>tfds.load(…)</em>’, and innumerable TensorFlow deprecation warnings.</p><p>%%writefile &lt;filename&gt; : Writes the text contained in the rest of the cell into a file. Useful for creating a YAML, JSON, or simple text file on-the-fly for testing.</p><p>%tensorflow_version 1.x : If you’re still stuck in the past (not judging … well, maybe just a little!), don’t ‘<em>pip install tensorflow==1.0</em>’, it’ll make a mess of dependencies, use this line magic instead, before importing TensorFlow.</p><h4>13. Dock the Terminal</h4><p>If you’re a Pro/Pro+ user, you have access to the VM via the Terminal. You can do some super-powerful things like run your own jupyter server on it (so you can have back the familiar Jupyter Notebook UI if that’s your preference). Also for wrangling files, the terminal is invaluable. If you’re a Free Tier user and want access to a terminal, check out <a href="https://blog.infuseai.io/run-a-full-tty-terminal-in-google-colab-without-colab-pro-2759b9f8a74a">colab-xterm</a>.</p><p>When you click on the Terminal icon on the left sidebar, the Terminal panel pops out on the right of the page, but that’s really difficult to use given how narrow it is and constantly having to close it so you can see your code!<strong> </strong>Solution: Dock the Terminal as a separate Tab. After opening the Terminal, click on the Ellipsis <strong>(…)</strong> ➤ Change page layout ➤ Single tabbed view.</p><h4>14. Change those Shortcuts</h4><p>Who can remember the obscure shortcuts that Google assigns?! Go to Tools ➤ Keyboard shortcuts, and make sure you assign the following shortcuts to your own <strong><em>memorable</em></strong><em> two-key </em>combinations:</p><ul><li>Restart runtime and run all cells in the notebook</li><li>Restart runtime</li><li>Run cells before the current</li><li>Run selected cell and all cells after</li></ul><p>The nice thing is that Colab remembers your changes (be sure to click on the Save button at the bottom of the pop-out) so you only need to do this once.</p><h4>15. GitHub Integration</h4><p>You can launch any notebook hosted on GitHub directly in Colab using the following URL:</p><p>https://colab.research.google.com/github/&lt;org&gt;/&lt;repo&gt;/…/&lt;xx.ipynb&gt;</p><p>If you install the <a href="https://chrome.google.com/webstore/detail/open-in-colab/iogfkhleblhcpcekbiedikdehleodpjo">Open in Colab</a> Chrome extension, and you’re presently previewing a Jupyter Notebook in GitHub, you can click on the extension’s icon (pin it to ensure easy access), on the right-side of your browser’s navigation bar, to directly open it in Colab.</p><p>Or you can just bookmark your favorite org/repo with the following URL and you’ll be prompted by a file browser whenever you click on the bookmark:</p><p>https://colab.research.google.com/github/&lt;org&gt;/&lt;repo&gt;</p><p><em>A Couple of Bonus Tips …!</em></p><h4>16. Interactive Pandas Tables</h4><pre>from google.colab import data_table<br>data_table.enable_dataframe_formatter()</pre><p>Turning this feature ON will enhance the rendering of DataFrames and allow for interactive exploration. Pagination is supported with selectable number of rows displayed per page, thus keeping your notebooks compact and readable. Columns can be sorted (globally across the entire dataframe). The most power feature is an interactive Filters dialog with RegEx matching.</p><h4>17. Mirror Cell in Tab</h4><p>If you find that you’re often scrolling back to, or re-executing, a cell, e.g. you’re using it to plot your training loss or a confusion matrix — rather than bloat your notebook with duplicate code/cells that does the same thing as you experiment with different code setups in the cells following it, you can ‘pin’ a cell by ‘mirroring’ it. Select the cell you wish to mirror, the Click: Tools ➤ Command palette ➤ Mirror cell in tab .</p><p>That cell will now pop up as a separate tab-panel in your notebook. Colab takes care of syncing changes you make to either the original cell in the notebook, or in the mirrored cell in the tab. You can re-execute the standalone cell with &lt;SHIFT|CTRL&gt; + &lt;ENTER&gt; just like any other cell.</p><h4>18. Disconnect Colab Runtime</h4><p>Being a responsible and considerate Colab user means not hogging the GPU instance after you’re done training … Google will notice and reward you with a better GPU the next time you request one! So after you’ve copied your saved model files to GDrive or Google Storage, and you’ve sent yourself a message informing you that your training is done, put the following in last cell in your notebook:</p><pre>from google.colab import runtime<br>runtime.unassign()</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a97184358d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/use-google-colab-like-a-pro-39a97184358d">Use Google Colab Like A Pro</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Engineering for Machine Learning (2/3)]]></title>
            <link>https://medium.com/data-science/feature-engineering-for-machine-learning-434c9b4912c6?source=rss-ad3edf793aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/434c9b4912c6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[feature-extraction]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[kaggle]]></category>
            <dc:creator><![CDATA[Wing Poon]]></dc:creator>
            <pubDate>Mon, 14 Mar 2022 14:34:15 GMT</pubDate>
            <atom:updated>2022-03-16T03:41:25.101Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2: Feature Generation</h4><figure><img alt="Pyramids rising up from the desert floor" src="https://cdn-images-1.medium.com/max/1024/1*A_-O9aHN3bSV0wxnSflYNw.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/thedigitalartist-202249/">Pete Linforth</a> from <a href="https://pixabay.com/">Pixabay</a></figcaption></figure><p>In this second of the three-part series on Feature Engineering (<a href="https://medium.com/@wpoon/feature-engineering-for-machine-learning-a80d3cdfede6">Part I: Data Prepreprocessing</a>), we’ll see there’s an almost infinite number of ways to build new features from existing ones, so the <em>art</em> in Feature Generation, once you’re aware of the basic techniques described below, is really in gaining an <em>intuition</em> on what to try for a given problem domain.</p><blockquote>“Good features allow a simple model to beat a complex model”</blockquote><blockquote>— Peter Norvig</blockquote><h3>2. Feature Generation</h3><p>For this article, we’ll be jointly describing both <em>Feature Extraction</em>, which generally refers to domain-specific methods of dimensionality reduction, as well as <em>Feature Generation</em>, usually accomplished via <em>i.</em> mapping existing features into a new space, <em>ii.</em> combining multiple features into a composite, <em>iii.</em> aggregating data to find patterns or <em>iv.</em> merging auxiliary data. We’ll be grouping methods by their applicability to the underlying data type.</p><h4>2.1 Date &amp; Time</h4><p>Events often exhibit <strong>periodicity</strong> or seasonality. The periodicity may manifest at more than one time-scale so, depending on your data, you may wish to decompose a timestamp column into multiple columns, such as: Minutes, Hour, Day of week, Weekday-or-Weekend, Day of Month, Month, Season or Year. Doing so will also let you use <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.groupby">pd.DataFrame.groupby()</a> to perform aggregations, which is in itself one of the most powerful ways to generate new features. One easy way to break up a timestamp is to use Pandas’ string methods, e.g.<a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.str.html?highlight=str">pd.Series.str.split()</a>:</p><pre># ‘date’ column in transactions is encoded as “01.04.2022”<br>transactions_date = transactions[‘date’].str.split(‘.’)</pre><pre># add three new columns to the dataframe<br>transactions[[‘day’, ‘month’, ‘year’]] = pd.DataFrame(transactions_date.tolist())</pre><p>Once these groupby() columns are created, to predict future sales of a product for example, you can calculate and add the following new features to your model: i. number of items sold on the same day <em>last week</em>, the ii. average daily sales from the <em>last month</em>, and iii. sales from the same month <em>last year.</em></p><p>Another very useful feature that can be generated is <strong>time deltas</strong>, such as the difference between two feature dates. For example, when predicting customer churn, the number of months since the start of service/subscription, or contract expiration, or the last call to customer support. In fraud detection, it could be days since the last credit-card transaction for this customer. In consumer spending, the days since the last bi-weekly pay period (the most common period according to the U.S. Bureau of Labor Statistics). In medical diagnosis, subtracting the date-of-birth to obtain age, then further subtracting the 20th-percentile target-positive age to obtain age in excess of disease onset.</p><p>When events occur across multiple time zones, it may be beneficial to encode the local time for each sample if their corresponding location is known. E.g. a U.S. online retailer serves customers across 4 time zones (Pacific, Mountain, Central and Eastern), and when predicting click-through rates, whether it’s 8pm or 11pm for the customer may make a difference in how they respond to recommendations or ads.</p><h4>2.2 Geolocation</h4><p>For geolocations, say if predicting home prices, one way to add new features is to supplement with auxiliary data. You can add <strong>distances</strong> to relevant landmarks: city-center, places of worship, good schools, etc. If the geolocation data has been anonymized (i.e. non-absolute coordinates), you can still perform clustering and then calculate the distance to cluster centroids.</p><p>Local minimas/maximas are also useful points of references for calculating distances from, serving as proxies for desirable or undesirable geographical entities. Yet another feature is surrounding building density, with high densities indicating popularity, confluence of transportation options, etc.</p><h4>2.3 Pricing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9gGVvTaoqllqxMzeeevBvA.png" /><figcaption>Image by <a href="https://pixabay.com/users/200degrees-2051452/">200degrees</a> from <a href="https://pixabay.com/">Pixabay</a></figcaption></figure><p>To an algorithm, there is hardly any difference between an item priced at $5.00 versus at $4.99, but we know humans aren’t rational. Therefore, if predicting sales for example, you could add a flag for prices that end with $0.49 or $0.99 (or add a column for fractional part of the price and let the decision tree figure out the patterns by itself). Along the same vein, it is well known that certain price points have psychological resonance, and serve as resistance-levels, e.g. $19, $49, $199, so add a (modulo-x) feature to help your algorithm detect them.</p><h4>2.4 Numerical</h4><p>Linear Regression isn’t limited only to linear relationships — it just needs to be linear on the independent variable, i.e. the features. So adding <strong>polynomial features</strong> will allow LinR to be much more expressive, and reduce fitting bias. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html">sklearn.preprocessing.PolynomialFeatures()</a> makes adding them a cinch, and even supports interaction features, as discussed next.</p><p>Numerical features are good candidates to add <strong>interaction features</strong> (also called interaction <strong>terms</strong>). These are formed by performing Summing/Subtractions/Divisions/Multiplications between two or more features. They are especially beneficial for tree-based models which have difficulty extracting such dependencies.</p><p>The difficulty for us, of course, is there’s an infinite number of ways to create such interactions, and only some are helpful. E.g. calculating price/sq-ft given price and floor area, average household size given census block group size and number of households, multiplying pressure by temperature to predict rate of chemical reactions.</p><p><strong>t-SNE</strong> is usually used as a visualization aid, but you can also use its projections <a href="https://zelros.medium.com/anomaly-detection-with-t-sne-211857b1cd00">as features</a>. Be warned though that t-SNE is extremely sensitive to hyperparameter (perplexity) values, so you’ll need to experiment to find what works best. If you do use t-SNE as a feature, do not use scikit-learn’s implementation (sklearn.manifold.TSNE) as it does not have a .transform() method. Instead use <a href="https://opentsne.readthedocs.io/en/latest/api/index.html">openTSNE.TSNE.transform()</a> which not only is faster, but crucially, supports embedding of new points. If projecting to only 2-dimensions is too limiting for your data, you can give <a href="https://umap-learn.readthedocs.io/en/latest/index.html"><strong>UMAP</strong></a> a try.</p><p><strong>Bucketing</strong> of continuous variables may be useful if there are multiple distinct peaks in the feature histogram. A plot of longitude versus home prices shows no discernible relationship, but plotting the histogram reveals multiple peaks, corresponding to cities that lies within longitudinal ‘bands’ — these bands can can be used as thresholds for binning continuous features into categories.</p><h4>2.5 Categorical</h4><p>Categorical interaction features are useful for Linear and k-NN models, but not tree-based models. They are formed by string-concatenating two categorical features to form a new composite, e.g. concatenating ‘sex’ ∈ {M, F} and ‘education-level’ ∈ {1, 2, 3, 4} to give ‘sex-edc’ ∈ {M1, F1, M2, …}. Good candidates are features that are frequently referenced on adjacent decision-tree nodes.</p><p>Categorical <strong>Target Encoding</strong> (also known as Likelihood or Mean Encoding) is very popular on Kaggle competitions. It is the encoding of each categorical class as some function of the target class values. Popular functions include:</p><ul><li>Mean: P(Y=1 | X_i)</li><li>Weight of Evidence (WOE): log( P(Y=1 | X_i) / P(Y=0 | X_i) )</li><li>Count: #(Y=1 | X_i)</li><li>Difference: #(Y=1 | X_i) − #(Y=0 | X_i)</li></ul><p>e.g. Mean can be calculated using pd.df.groupby(&#39;feature&#39;)[&#39;y&#39;].mean()</p><p>Decision-trees have difficulty with high categorical cardinality (many unique values) due to max-depth limitations (you can only go one of two ways at each node) — target encoding results in lower loss for the same tree-depth.</p><p>The main problem with target-encoding is risk of severe <em>overfitting</em>, or <em>data-leakage</em>. One way this can happen is when the data distribution changes between the training and validation/test sets (e.g. the target mean changes). K-fold cross-validation and additive smoothing are two ways to mitigate this risk.</p><h4>2.6 Statistics Aggregation</h4><p>This is one of the most powerful ways to create new features from existing ones. Three choices have to be made: i. what to group by, ii. how to aggregate each group, and finally, iii. on what key to join the aggregated statistics on. Here are some examples:</p><ul><li>Calculate the Min/Max/Std-Dev (aggregation) of the cost of all ads shown to a user (join-on) on the same page (group-by). Can be useful for predicting Click-Through Rate on a specific ad by providing context, i.e. more costly ads on that page will be more prominently featured and will therefore negatively impact cheaper ads placed on the same page</li><li>Number of pages the user has already viewed during a session, i.e. recommendation/ad fatigue. Number of symptoms a patient has, i.e. any one symptom may not be significant, but multiple symptoms may</li><li>For each customer over the last year, the median time between two consecutive transactions, mean number of transactions per month, mean and std-dev of charges per spending category, all can help predict fraud by informing how frequently the card is typically used, on what, for how much and the pattern of usage (ad-hoc vs regular automatic payments)</li></ul><p>Pandas’ .groupby() is indispensable for calculating these statistics, but groups may also need to be synthesized on-the-fly from numeric features, e.g. the average price/sq-ft of homes within a set radius.</p><p>Sometimes data is spread over multiple tables, in which case, supplement the main (e.g. transaction) table with insights gleaned from other tables, e.g. adding a flag in the transaction table if the merchant is associated with a recent spike in disputed transactions.</p><h4>2.7 Text</h4><p>In Part I, we preprocessed out raw text and tokenized them into small chunks of text. Now we’ll need to convert these sequences of chunks to numbers, which is the job of the vectorizer.</p><p>A Bag-of-Words (BOW, also known as Count) <strong>vectorizer</strong> counts the number of occurrences of each unique token in a document. It’s common to have options to ignore words that appear too infrequently, or too frequently. A Term Frequency (TF) vectorizer normalizes raw counts to show what proportion of document each term constitutes (i.e. vector elements sum to one). A TF-IDF vectorizer further multiplies the TF by the log of the inverse of how frequently each term appears in the document corpus.</p><p>Word and sentence <strong>embeddings</strong> (e.g. Word2vec, GloVe, fastText) provide the next level of sophistication in text feature generation. Note that due to the way that these are trained, i.e. un-supervised training, the mathematical ‘closeness’ of the resulting word embeddings reflects not semantic similarity, but rather co-occurence of words — e.g. “Love” and “Hate” occur frequently together in a sentence and will thus be mapped close together in the embedding space even though they are antonyms.</p><p>Also note that these simple embeddings are static and for each word, there is but one, and only one, corresponding embedding. As such, they are not capable of semantic disambiguation, i.e. the ‘bear’ in black bear, bear market and bear arms will all be assigned the same vector embedding.</p><p>When it comes to embeddings (such as with Word2vec), you have a choice as to whether to use <a href="https://code.google.com/archive/p/word2vec/">pretrained embeddings</a>, or have the model learn its own embeddings on your data using back-propagation. Sentences can also have embeddings, if you wish to create a single feature that summarizes the sentence, by averaging individual word vectors together, using the embedding of the [CLS] token, or using something like <a href="https://radimrehurek.com/gensim/models/doc2vec.html">Doc2vec</a>. For phrases (e.g. “frying pan”), try <a href="https://github.com/explosion/sense2vec">sense2vec</a>.</p><p>One final trick to create additional text features is to perform double-translation on each piece of text (e.g. English → Spanish → English), and then predict on both corpuses using an ensemble.</p><h4>2.8 Image</h4><figure><img alt="Grid of colored squares" src="https://cdn-images-1.medium.com/max/800/1*haomfY1EIGS6clDS99XpLQ.jpeg" /><figcaption>Image by <a href="https://www.flickr.com/photos/15363357@N00/271992932">Jan Tik</a> marked <a href="https://creativecommons.org/licenses/by/2.0/?ref=openverse">CC BY 2.0</a>.</figcaption></figure><p>Pretrained models, typically CNNs, trained on large datasets learn to represent whole images as latent representations in their final layers. By truncating the ‘head’, but keeping the ‘body’, these models can be used as <strong>feature extractors</strong>. Feed an image in and out comes an image embedding, with common output dimensions of 768 or 2048.</p><p>Not only can you pick from different model architectures (e.g. VGG16, ResNet50, MobileNet), you can also pick weights based on training the model on a <em>domain-specific subset</em> of images. Why this works is because of the No Free Lunch theorem: there is no one-size-fits-all, universally optimum solution. TensorFlow Hub offers a selection of ResNet50 models, identical architectures but trained on different subsets of ImageNet categories. Using one trained on the<a href="https://tfhub.dev/google/experts/bit/r50x1/in21k/flower/1"> flower subtree</a>, I found I was able to obtain significantly higher accuracy on the <em>Oxford102</em> dataset than using weights trained on the entire <em>ImageNet-1K </em>(i.e. relevance beats abundance)<em>.</em></p><p>For the state-of-the-art, just as Transformers revolutionized NLP through the application of attention, one can also use a pretrained Vision Transformer (<a href="https://tfhub.dev/sayakpaul/collections/vision_transformer/1">ViT</a>) for extracting (768 to 1024 dimension) vector representations of an image.</p><h4>2.9 Time-Series</h4><p>For time-series data, in addition to those already mentioned in [Sect 2.1], one can generate new features by adding one or more <strong>lag features</strong>. Simply delay or shift, e.g. <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.shift">pd.DataFrame.shift()</a>, the data back by a set amount of time. Use autocorrelation, e.g. <a href="https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.acf.html">statsmodels.tsa.stattools.acf()</a>, to determine the most predictive lag period(s).</p><p>Another technique, which has the added benefit of also detrending a time-series, is to encode the series as a <strong>difference</strong> or delta from some period back. For time-series that have a strong trend, e.g. stock prices over long timeframes, it’s better to use <strong>ratiometric</strong> or percentage change rather than absolute change, as that better reflects how prices change. Both of these transformations (difference or ratiometric) can also be applied to the target variable (have the model predict the following day’s change rather than the absolute value).</p><p><strong>Moving-window aggregations</strong> (also known as trailing indicators) can also add context. For example, the moving average of prices can be used to detect market trends. Again, using different aggregation time-spans can be helpful, e.g. traders often use 50 and 100-day SMAs. Expanding-windows can also be used, but are less common.</p><p>Then there are domain-specific features, such as adding a feature to indicate how much of a gap-up/down there is from the previous day’s close. The MACD indicator is computed using the difference between two Exponential Moving Averages (EMAs) and is used to detect when trends are accelerating. For weather-related time-series, calculate the dew point (a function of temperature and pressure).</p><p>Time-domain data can be transformed into the <strong>frequency-domain</strong>. Performing a Fourier Transform alone falls more in the category of feature extraction than feature generation, since the transformed data is no longer anchored in time, and so cannot be used alongside time-domain features, such as those described above. However, time can be incorporated back in, for example in the case of spectrograms, we can track power spectral density changes as a function of time.</p><h4>2.10 Representation Learning</h4><p>Representational Learning (RL) refers to learning latent representations using non-parametric (i.e. non-statistical) methods to <em>extract features</em>. Some examples have already been described, such as word-embeddings and CNN/ViT-based image feature-extractors.</p><p>RL has been very successfully applied in computer-vision. <a href="https://github.com/google-research/simclr">SimCLR</a> for instance, can be used to learn visual representations from unlabeled data using unsupervised learning. Then, by introducing just a small subset of labels, the learned representations can be leveraged to classify the rest of the dataset with remarkable accuracy.</p><p>Feature extractors are particularly useful for performing similarity searches. This is because once trained (or if using one that is pretrained), the extracted feature vector is invariant for a set input. In similarity search, you are comparing one sample against the entirety of your database, based on a similarity metric such as cosine similarity. By <em>pre-computing</em> all feature vectors corresponding to your raw inputs just once, you can very efficiently store and subsequently perform nearest-neighbor similarity search using a special database such as <a href="https://faiss.ai/">FAISS</a>.</p><p>This concludes Part II of this series on feature engineering. We have now amassed a profusion of features, so in Part III we’ll turn our attention to <em>Feature Selection</em>, where we look at ways to separate the wheat from the chaff to ensure we’re not overfitting on our data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=434c9b4912c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/feature-engineering-for-machine-learning-434c9b4912c6">Feature Engineering for Machine Learning (2/3)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Engineering for Machine Learning (1/3)]]></title>
            <link>https://medium.com/data-science/feature-engineering-for-machine-learning-a80d3cdfede6?source=rss-ad3edf793aa6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a80d3cdfede6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[Wing Poon]]></dc:creator>
            <pubDate>Mon, 14 Mar 2022 14:32:35 GMT</pubDate>
            <atom:updated>2022-04-08T18:47:31.535Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1: Data Preprocessing</h4><figure><img alt="Multiple gears on a page" src="https://cdn-images-1.medium.com/max/1024/1*B-FrsT65CzQM66ILlC9p-A.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/thedigitalartist-202249/">Pete Linforth</a> from <a href="https://pixabay.com/">Pixabay</a></figcaption></figure><p>The era of Deep Learning has popularized the approach of end-to-end machine learning wherein raw data goes into one end of the pipeline and predictions out the other end. This has certainly produced speedups in model inference in some domains, especially in computer-vision pipelines, as evidenced, for example, by the higher frame-rates of Single-Shot Detectors versus models that rely on region proposals followed by object-detection. The ability of sophisticated models to automatically extract features has made it possible to trade off computational resources to save on human resources.</p><p>This approach has meant that machine-learning practitioners have been increasingly churning through models rather than honing their data. But there comes a point when the easy gains have been reaped, and doubling the model size can only eke out a tiny bit of performance improvement. This is when hand-crafting features can have a better payoff.</p><blockquote>“Applied machine learning is basically feature engineering”</blockquote><blockquote>— Andrew Ng</blockquote><p>In part, the automatic vs hand-crafted features tradeoff has been made possible by the richness, high-dimensionality and abundance of visual data. When working with a paucity of data, or less feature-rich data, which is all too common for data scientists tasked with coming up with predictions based on just a dozen or so features, <em>feature engineering</em> is essential to eke and tease out all the available ‘signal’ that’s present in the limited data; as well as to overcome the limitations of popular machine-learning algorithms, for example, difficulty in separating data based on multiplicative or divisive feature interactions.</p><p>When competing in Kaggle competitions, the best teams don’t win simply on account of model selection, ensembling and hyperparameter tuning, but rather, significantly on their ability to engineer new features, sometimes seemingly out of thin air, though more often than not, that which is born of a combination of truly understanding the data (bringing domain knowledge), supplementing with auxiliary data, and the dogged, creative (more an art than a science) but tedious, trial-and-error work of constructing and testing new features to see what works.</p><p>In this multi-part series, we’ll go over the three parts of a complete Feature Engineering pipeline:</p><ol><li>Data Preprocessing</li><li>Feature Generation</li><li>Feature Selection</li></ol><p>These three steps are performed in order but sometimes there’s ambiguity as to whether a certain technique constitutes data preprocessing, feature extraction, or generation. But we’re not hung up about semantics here … instead we’ll focus on surveying a gamut of techniques any good machine-learning practitioner and data scientist can bring to bear on a project.</p><p>The intent of this series of articles is to raise awareness of these sometimes forgotten, especially in the era of deep-learning and billion-parameter models, techniques to always keep in the back of one’s mind, and knowing some library functions that can greatly facilitate their use. Describing the inner-workings of each technique would each require an article onto itself, of which many can be found here on <em>Towards Data Science</em>.</p><h3>1. Data Preprocessing</h3><h4>1.1 Data Cleaning</h4><blockquote>“Garbage In, Garbage Out.”</blockquote><p>During EDA, one of the first steps to undertake should be to check for and remove <strong>constant features</strong>. But surely the model can discover that on its own? Yes, and no. Consider a Linear Regression model where a non-zero weight has been initialized to a constant feature. This term then serves as a secondary ‘bias’ term and seems harmless enough … but <em>not</em> if that ‘constant’ term was constant only in our training data, and (unbeknownst to us) later takes on a different value in our production/test data.</p><p>Another thing to be on the lookout for is <strong>duplicated features</strong>. This may not be blatantly obvious when it comes to categorical data, as it might manifest as different labels <em>names</em> being assigned to the same attribute across different columns, e.g. One feature uses ‘XYZ’ to denote a categorical class that another feature denotes as ‘ABC’, perhaps due to the columns being culled from different databases or departments. <a href="https://pandas.pydata.org/docs/reference/api/pandas.factorize.html">pd.factorize()</a> can help identify if two features are synonymous.</p><p>Next up, <strong>redundant</strong> and <strong>highly-correlated</strong> features. Multicollinearity can cause model coefficients to be unstable and highly-sensitive to noise. Besides its negative impacts on storage and computational costs, redundant features weaken the effectiveness of other features when weight regularization is taken into account, making the model more prone to noise. <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html">pd.DataFrame.corr()</a> can be used to identify correlated features.</p><p>Duplication may occur not just across columns, but also across <em>rows</em>. Such <strong>sample duplications</strong> can cause data imbalance and/or over-fitting during training. <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.duplicated.html">pd.DataFrame.duplicated()</a> will return a series with a True value in each duplicated row beyond its first occurence.</p><h4>1.2 Data Shuffling</h4><p>It’s important to differentiate between <strong>shuffling</strong> during preprocessing versus during training.</p><p>During preprocessing it’s important to shuffle your dataset prior to splitting it into train/validation/test subsets. For small or highly-imbalanced datasets (e.g. in anomaly, fraud or disease detection), utilize the stratify on feature of <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">sklearn.model_selection.train_test_split()</a> to ensure there is consistent distribution of your minority targets across all your subsets. <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html">pd.DataFrame.sample(frac=1.0)</a> can be used to easily shuffle your dataframe.</p><p>For training, most ML frameworks will shuffle the data for you, but it’s important to understand whether it’s doing a one-time shuffle, i.e. only when loading the dataset, or if it’s doing so continually on a batch-by-batch basis. The latter is preferred for obtaining the lowest training loss, but can result in slower training since the mini-batches of data cannot be cached and reused for the next epoch.</p><h4>1.3 Data Imputation</h4><p>This is a big topic! <strong>Missing features</strong> should be thoughtfully dealt with.</p><p>Missing features may not be immediately obvious, so merely using <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isna.html">pd.DataFrame.isna.sum(axis=0)</a> may not surface all of them. Missing fields may be denoted using a special (non-null) string, or numerical value (e.g. “-”, 0 or -999). The dataset you’re using may have already been preprocessed and pre-imputed by someone. Fortunately, <strong>detecting missing features</strong> can be done by plotting the histogram of each feature — unusual outlier spikes indicate the use of special values, while a spike in the middle of the distribution is a sign that mean/median imputation has already been performed.</p><p><strong>How</strong> to impute missing fields is the next question. The most common and straightforward method is to substitute the <em>mode</em> (for categorical features), <em>mean</em> (for numerical features without a lot of outliers) or <em>median</em> (where outliers significantly skew the mean) for the missing value.</p><p>Even so, especially if you think the feature is an important one, do not blindly substitute the <strong>mean / median / mode</strong> of the <em>entire</em> dataset. To illustrate, in the <em>Titanic</em> dataset, several passengers are missing their ages. Rather than to impute all such passengers with the mean age of all passengers on the ship, we can get a more precise estimate by realizing that the mean (and median) age between passenger classes were quite different aboard the ship.</p><figure><img alt="Table showing median ages of Titanic passengers by sex and fare class" src="https://cdn-images-1.medium.com/max/662/1*wX5ZEXaWNnbA_qzx08KDJw.png" /><figcaption>Table 1: Median Passenger Age by Passenger Class and Gender</figcaption></figure><p>In fact, since there are no missing values for passenger sex in this dataset, you can go one step further when imputing and substitute the median age based on both i. fare class and ii. gender of each sample that is missing an age value.</p><p>For a time series, one should never impute missing samples using mean / median substitution, as this will invariably cause abrupt changes in the series that are unrealistic. Instead, impute using <strong>value repetition</strong>, or interpolation. Signal-processing denoising methods like median-filtering or low-pass zero-phase filtering can also be used to fill small gaps in the training data; but bear in mind that <em>non-causal</em> methods cannot be used during production unless a delayed output from the model is acceptable.</p><p>An alternative is to not impute at all and instead add a binary <strong>flag</strong> to allow the downstream learning algorithm to learn on its own how to handle such conditions. The disadvantage of doing so is that you may have to add many more such low-signal features if the missing values are spread out over many features. Note that XGBoost handles NaN <a href="https://xgboost.readthedocs.io/en/stable/faq.html">out of the box</a>, so there’s no need to add missing-data columns when using it. In general, Decision Trees can natively handle special-value label-encoding of missing values, by setting them to lower/upper extreme values that decision nodes can then easily split upon.</p><p>Another popular method is to run <strong>k-NN</strong> to impute missing values. Using a <strong>neural network</strong> for imputation is another popular route. For example, an auto-encoder can be trained to reproduce the training data with input dropout, and once trained, its outputs can be used to predict missing feature values. Using ML methods to learn imputations can be tricky however, since it’s difficult to assess how the imputation model’s hyperparameters (e.g. the value of <em>k</em>) are affecting the final model’s performance.</p><p>Regardless of the imputation method, where possible, always perform <em>Feature Generation</em> (discussed in the second part of this series) <strong>prior </strong>to data <em>imputation</em>, as this will allow for more accurate calculation of generated feature values, particularly when special-value encoding is used for imputation. Knowing that a relevant feature value is missing leaves the door open for special-handling when new features are being generated.</p><h4>1.4 Feature Encoding</h4><p><strong>Ordinal features</strong> may have integer values, but they differ from numeric features in that although ordinal values obey the transitive comparison relation, they don’t abide by the arithmetic rules of subtraction or division.</p><p>Ordinal features, such as Star Ratings, often have highly non-linear ‘true’ mappings, corresponding to a strongly-bipolar distribution. i.e. The quantitative ‘difference’ between a 4-Star vs 5-Star rating is oftentimes small, and considerably smaller than half of the difference between a 4-Star vs 2-Star rating, which can confuse linear models.</p><p>As such, linear models may benefit from a more linear <strong>remapping</strong> of ordinal value assignments, but tree-based models on the other hand are able to handle such nonlinearity intrinsically. It’s better to leave ordinal encodings as a <em>single</em> quantitative feature (i.e. ‘column’) rather than <em>dummifying</em>, as dummifying <em>decreases</em> the Signal-to-Noise Ratio of other predictive features (e.g. when parameter regularization is applied).</p><p>Handling of <strong>categorical features</strong> depends on whether your model is tree-based. Tree-based models can use <strong>label-encoding </strong>(i.e. fixed strings or integers denoting class membership) and don’t need further preprocessing. Non-tree methods require that categorical features be <strong>one-hot encoded</strong>, which can be performed using either <a href="https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html">pd.get_dummies()</a> or <a href="http://sklearn.preprocessing.onehotencoder/">sklearn.preprocessing.OneHotEncoder()</a>. Avoid dropping the first-column (i.e. don’t specify drop=’first’) unless you’re dealing with a binary category (i.e. use drop=’if_binary’), since dropping <a href="https://inmachineswetrust.com/posts/drop-first-columns/">creates more problems</a> than it solves.</p><p>An alternative to one-hot encoding is to use, or supplement with, <strong>frequency-encoding</strong>. This involves calculating the normalized frequency of the target-variable corresponding to each category. E.g. Assign a value of 0.4 to a binary categorical feature if 40% of samples in that category result in a target value of one. Doing so is helpful when the categorical feature is correlated to the target value.</p><p>Yet another way to encode categorical features is to use <strong>categorical embeddings</strong>. This is particularly suitable for high cardinality categorical features, such as ZIP codes or products. As with word embeddings, these embeddings are learned using a dense neural network. <a href="https://github.com/entron/entity-embedding-rossmann">Analysis</a> of results show that these continuous embeddings are also meaningful for clustering and visualization, while reducing overfitting.</p><p>A final point on handling categorical features is what to do if an <em>unseen</em> category is encountered in the validation/test subset. In this case, treat it the same way you would a missing feature, or assign it the reserved category of ‘Unknown’.</p><h4>1.5 Numerical Features</h4><p><strong>Scaling</strong> or <strong>normalization</strong> is not required for tree-based methods, but is otherwise essential for achieving low training loss, fast convergence, and in order for weight regularization to work properly. Choose one or the other.</p><p>Scaling <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html">sklearn.preprocessing.MinMaxScaler()</a> is fast and often all that is required for image data. Min-Max Scaling however can be significantly impacted by outliers (even just a single one!) or missing values that are encoded as values outside of the normal feature range.</p><p>Normalization <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">sklearn.preprocessing.StandardScaler()</a> is more robust to outliers as long as the proportion of outliers is small (i.e. if the outliers do not significantly skew the mean and standard-deviation).</p><p>Numerical features can often benefit from <strong>transformations</strong>. <em>Log</em> transformation, np.log(1 + x), is a very strong transformation that is particularly helpful when a feature observes a <em>power-law</em> relationship, or when there’s a long tail in the outliers distribution. <em>Square-Root</em> transformation, np.sqrt(x) is less strong, and can serve as a useful intermediate transform to try. The<em> Box-Cox </em><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox.html">scipy.stats.boxcox()</a> transform allows one to smoothly transition from a straight linear to a highly non-linear transfer function, due to the use of the lambda hyperparameter, and is often used to turn a <em>skewed</em> distribution (e.g. COVID test-positivity as a function of age) into a more <em>normal</em> distribution, which is the underlying assumption made by many traditional ML algorithms like NB, LogR and LinR. [NOTE: x &gt;= 0 is assumed for all of the abovementioned transformations].</p><h4>1.6 Geolocation Features</h4><p>Tree-based methods can sometimes benefit from <strong>rotational transformations</strong> on geolocation coordinates. For example, South-of-Market (SoMa) is a neighborhood in San Francisco that is bordered by Market Street. However, Market St. doesn’t run North-South or East-West, but rather SW-NE. Decision trees will thus have difficulty cleanly segmenting out this neighborhood (when tree depth is bounded, as is often necessary), since trees can only draw partition lines that are <em>axis-parallel</em> (i.e. to either the Longitude or Latitude axes). The difficulty with performing rotational transformations though is choosing the pivot point to rotate the coordinates around, as there may not be a globally-optimum one.</p><p>One solution is to perform a cartesian to <strong>polar-coordinate</strong> transformation. The city of Paris for example is divided into arrondissements that are laid out roughly in a circular manner around the city center. Alternatively, <em>clustering</em> can be performed and a new categorical feature created to indicate which cluster a point belongs to — after which polar coordinates can be used to encode each point’s location from its cluster centroid/nucleus.</p><h4>1.7 Time Features</h4><p>Time in a dataset may appear as UTC/GMT time rather than local-time, even when all events are exclusive to a single (and different) time-zone. Even when timestamps are given in local time, it may be beneficial to perform <strong>time-shifting</strong> on time features.</p><p>For example, ride-sharing fares tend to be higher ‘later’ in the day (especially Fri/Sat/Sun) due to a combination of higher demand and lower supply. However, as any late-night reveler knows, the partying doesn’t stop at midnight. Due to the roll over of time, it’s difficult for a model therefore to discern that 00:30H is ‘later’, or even ‘close’ to, 23:30H. However, if time-shifting is performed and the tricky 23:59H → 00:00H wrap-around happens during the time of the day when there’s the <em>least</em> amount of activity, linear models will regress better while tree-based models will require fewer branching levels.</p><h4>1.8 Text</h4><figure><img alt="A sea of text" src="https://cdn-images-1.medium.com/max/1024/1*KQNviVHrVVVw4u4cJ-g3Bg.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/geralt-9301/">Gerd Altmann</a> from <a href="https://pixabay.com/">Pixabay</a></figcaption></figure><p>Text is arguably the feature that requires the most preprocessing. We begin with transformations, of which there are many.</p><p><strong>Lowercasing</strong> is necessary for simple models, and it is the most basic form of text normalization, providing the benefits of producing a stronger and more robust signal (due to higher word frequency occurrence) while reducing vocabulary size. For more advanced pretrained <em>language models </em>(LM) that come with their own tokenizers, it’s best to let the LM perform tokenization on the <em>original</em> text, as capitalization (e.g. with <a href="https://huggingface.co/bert-base-cased">bert-base-cased</a>) may help the model perform better on <em>sentence parsing, name-entity recognition (NER) </em>and <em>part-of-speech </em>(POS) tagging.</p><p><strong>Stemming</strong> and <strong>Lemmatization</strong> provide the same benefits as lowercasing, but are more complicated and take far longer to run. <em>Stemming</em> pares down words using fixed rules which do not take into account the word’s context and usage within a sentence, e.g. university gets truncated to univers, but so does universal. <em>Lemmatization</em> takes into account <em>context</em> and requires the use of large LMs and is therefore slower than stemming. However it’s much more accurate, e.g. university and universal remain as separate root words. Given the compute resources readily available today, the authors of <a href="https://spacy.io/">Spacy</a> make the argument that accuracy matters most in production and therefore the library only supports lemmatization.</p><p><strong>Contractions Expansion</strong> is yet another form of text normalization with the same motivations. Here can’t and you’ll get expanded to can not and you will. Simple count/frequency-based models benefit from this standardization, but LMs like those in Spacy handle contractions natively through the use of <em>subword tokenization</em> (can’t gets tokenized into ca followed by n’t).</p><p><strong>Text-normalization</strong> aims to convert text and symbols to a canonical form so that models can learn better. They convert abbreviations (e.g. BTW), misspellings (e.g. ‘definately’), diacritics (e.g. café/naïve → cafe/naive), emoticons (e.g. &lt;grin&gt; | :-) | ;) | 🙂 → &lt;SMILE&gt;, ROFL | LOL | LMAO | 😆 → &lt;LAUGH&gt;), etc. Without text-normalization, many NLP models will have a rough time making sense of Twitter and social-media posts!</p><p>Next up is <strong>filtering </strong>— stopwords, punctuations, numbers, extra whitespace, emojis (when not performing sentiment analysis), HTML tags (e.g. &lt;br&gt;), HTML escape chars (e.g ‘&amp;amp;’, ‘&amp;nbsp;’), URLs, Hashtags (e.g. #blackfriday) and Mentions (e.g. @xyz)</p><p>The final preprocessing step is <strong>tokenization</strong>. Simple tokenization can be performed using python regular-expressions, but NLP frameworks provide dedicated tokenizers that are written in C/C++ or Rust that are much more performant. When using pretrained LMs, such as those from Hugging Face, it’s crucial to use the exact same tokenizer (and weights) as that which is used by the LM when it was trained.</p><h4>1.9 Images</h4><p>Preprocessing images is nowadays less common with the rise in popularity of CNNs. However, in resource-constrained hardware without a GPU, or when high frame-rates are desired, these techniques form the bread-and-butter of traditional computer-vision processing pipelines.</p><p><strong>Color-Space conversion</strong> can provide a slight performance boost with simple CNNs. <a href="https://arxiv.org/pdf/1902.00267.pdf">This study</a>, for example, found that converting the CIFAR-10 dataset to the L*a*b color-space resulted in a ~2% improvement in classification accuracy, but the best results came from using multiple color-spaces concurrently. It has been found that color-spaces with separate chrominance and luminance channels (e.g. YUV) are helpful for picture colorization and style transfer. I’ve personally found that specialized color transformations tailored for specific domains (e.g. Haematoxylin-Eosin-DAB for histopathology) can be particularly helpful in boosting model performance.</p><p><strong>Histogram equalization</strong>, especially adaptive histogram equalization (e.g. CLAHE), is often performed on medical imaging to improve visual contrast. It’s particularly helpful when illumination is uneven across the image and distinguishing features are small relative to the entire image frame, such as detecting cancerous tissue in mammography images. In retinal fundus imaging where there’s high variability in image quality during image acquisition, performing non-uniform illumination correction has been shown to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6113146/">improve</a> grading accuracy.</p><p>Traditional CV feature extraction techniques include Local Binary Patterns (<strong>LBP</strong>), Histogram of Oriented Gradients (<strong>HOG</strong>), and <strong>Gabor Filter</strong> banks. There’s been a long track-record of successfully using these feature extractors with non CNN-based models. Merging these older techniques (e.g. <a href="https://arxiv.org/pdf/1904.13204.pdf">learnable Gabor filters</a>) with standard convolutional layers can result in faster convergence and better accuracy with some datasets (e.g. D<em>ogs-vs-Cats</em>).</p><p>This concludes Part I of this series on feature engineering. In <a href="https://medium.com/@wpoon/feature-engineering-for-machine-learning-434c9b4912c6">Part II</a>, we’ll turn our attention to <em>Feature Generation</em>, where we’ll look at extracting and synthesizing brand new features. This is truly where art meets science, and is what separates a Kaggle Grandmaster from a novice!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a80d3cdfede6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/feature-engineering-for-machine-learning-a80d3cdfede6">Feature Engineering for Machine Learning (1/3)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>