<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by yonatan hadar on Medium]]></title>
        <description><![CDATA[Stories by yonatan hadar on Medium]]></description>
        <link>https://medium.com/@yonatan.hadar?source=rss-ed91e559149------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*H1jFMXDCz8AOTiTLBZfHmA.jpeg</url>
            <title>Stories by yonatan hadar on Medium</title>
            <link>https://medium.com/@yonatan.hadar?source=rss-ed91e559149------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:11:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yonatan.hadar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Should I train a model for each customer or use one model for all of my customers?]]></title>
            <link>https://medium.com/data-science/should-i-train-a-model-for-each-customer-or-use-one-model-for-all-of-my-customers-f9e8734d991?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9e8734d991</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Wed, 18 Aug 2021 13:21:58 GMT</pubDate>
            <atom:updated>2021-08-18T13:38:03.457Z</atom:updated>
            <content:encoded><![CDATA[<h3>Should I Train a Model for Each Customer or Use One Model for All of My Customers?</h3><h4><strong>Almost every data scientist that works in a software as a service company will finally come to the question in the title: </strong>Should I train a model for each customer or use one model for all of my customers?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t26k6E71DRS8Lpnc2euA4Q.jpeg" /><figcaption>People vector created by pch.vector — <a href="http://www.freepik.com/">www.freepik.com</a></figcaption></figure><p>I work as a senior data scientist at BigPanda, a SaaS company that sells software to enterprises. This means that any of our machine learning features should work properly for many different customers that each one of them has a very different data distribution and that our models should also work ASAP for our new customers that hopefully will keep growing exponentially.</p><p>This challenge is not just ours, I heard it from many of my friends that do data science in different domains such as fraud detection, e-commerce, sales forecasting, ad recommendation, and more. In this post, I will share 4 different strategies for dealing with the multi-tenancy problem in machine learning and highlight the advantages and disadvantages of these strategies. I will also share my experience from BigPanda on the approaches that I tried in our production ML models.</p><h3>Model Per Customer</h3><p>This is probably the most common approach to the multi-customer scenario, we have different customers with differences in their data distribution so maybe a model that is specified for the customer will work best. This brings more questions about how specific the model is for the customer. Do we use the same model for every customer and just train it on their data? Do we need to do hyperparameter tunning for every customer? Do we want to do feature selection? Model selection?</p><p>Answering yes to each of these questions will probably increase the model accuracy and add major complexity to the productization and maintenance of your models.</p><p><strong>Disadvantages</strong></p><ul><li><strong>Production complexity- </strong>Training and optimizing a model per customer requires lots of automation in model training, deployment, and monitoring to make sure that your system will scale to more than 10 customers.</li><li><strong>Smaller data- </strong>Some of your customers probably won’t have lots of data and their models will probably not be very accurate.</li><li><strong>Cold start- </strong>When a new customer wants to use your model he will have to wait for the model to have enough data to generate accurate predictions. This can take weeks or even months in different use cases and will impact the customer’s time to value of the product significantly.</li></ul><p><strong>Advantages</strong></p><ul><li><strong>The model is well suited for the customer- </strong>Learning on irrelevant data can hurt the accuracy of your ML model significantly. By training only on customer data you can make sure that all the training data is relevant and helps your model.</li><li><strong>Sometimes it’s the only possible choice- </strong>In many scenarios training on multiple customers can hurt your model due to different feature distribution between customers (for example if each one of your customers is in a different country or different industry) so training a model per customer is the only possible approach.</li></ul><p><strong>What to pay attention to?</strong></p><ul><li><strong>Model validation- </strong>In the research phase, you need to make sure that your model design will work on a variety of customers, try to validate the model on different customers, and make sure that the model is accurate on most of them and analyze what are the weak spots of your modeling.</li><li><strong>Automation- </strong>When training a model for each customer you will probably end up with hundreds or even thousands of models, to make this scale you must have lots of automation in your production services for model training, optimization, deployment, and monitoring.</li><li><strong>Cold start- </strong>You must analyze how to add the model for new customers. How much data do you need? What optimization needs to be done? What is the acceptance test for going live on a new customer?</li></ul><p><strong>My takeaways</strong></p><p>In one of our ML features in BigPanda, we have a model per customer. We did it because our customer’s data is very different, each customer has different attributes, different scales of data, different use cases, and different internal terms. While researching and implementing this solution we focused a lot on scaling the model to new customers by researching when can we train a model for a new customer and automating most of the model training and optimization.</p><h3>The Global Model</h3><p>The global model is the most simplistic approach, We will just ignore that we have different customers in the data and train one ML model on all the data that we have. Using this approach definitely prevents most of the disadvantages of the model per customer but it introduces new problems.</p><p><strong>Disadvantages</strong></p><ul><li><strong>Differences in data distribution- </strong>Training a model on irrelevant data will usually decrease the model’s accuracy. Many times data of different customers is very different and will hurt the model’s accuracy.</li><li><strong>Monitoring hidden issues- </strong>While monitoring the model accuracy as a whole is tempting, problems in the model may accrue only for a subset of customers that won’t impact the KPIs of the model but will definitely impact customer satisfaction. Finding and fixing these issues in a global model can be very hard.</li></ul><p><strong>Advantages</strong></p><ul><li><strong>Engineering effort- </strong>Training and optimizing only one model is much easier than many models and can drastically reduce the engineering effort in productization and maintenance.</li><li><strong>Cold start- </strong>With the global model approach we can generate predictions instantly when we have a new customer, this can really improve customer satisfaction.</li><li><strong>Larger dataset- </strong>Usually after mixing data from several customers you will have a large dataset that you can use to train complicated and powerful models.</li></ul><p><strong>What to pay attention to?</strong></p><ul><li><strong>Model validation- </strong>In the research phase you need to make sure that your model will actually work on new customers that don’t have data in the model training, this can be done with group cross-validation where you train your model on several customers and test on others.</li><li><strong>Model weak spots- </strong>The model will probably have lower accuracy on some of the customers. Identifying which customers have lower accuracy and what causes this can be very beneficial for improvements of the algorithm.</li><li><strong>Feature standardization across customers- </strong>Often, your model&#39;s features will have different distribution on different customers. This can cause a reduction in model accuracy in some of the customers and overfitting others. By normalizing the data for each customer we can reduce these differences and build a better model.</li></ul><p><strong>My takeaways</strong></p><p>In another ML feature in BigPanda, we have a global model for all of our customers. We decided on using a global model because it was very important for us that new customers will be able to use this model immediately. While researching this model we focused on making sure that we validate the model without using the same organization in train and test and on trying to engineer features that will be generic cross-customer so the model can generalize to every new customer.</p><h3>Customer Segmentation</h3><p>The next strategy is kind of a middle strategy between model per customer and global model. This approach will first split our customers into segments and next train a model for each segment. In this approach, we can enjoy some of the advantages of the global model (scalability and cold start) while using more relevant data and hopefully produce more accurate models.</p><p><strong>Disadvantages</strong></p><ul><li><strong>How to segment?</strong>- There are many ways to segment customers, we can do it by language, country, industry, and more or even use some clustering or embedding model to do the segmentation. There is also a trade-off between segments with many customers (the data will be too different) and few customers (we will have many different models) and finding the best segmentation is a hard research question.</li><li><strong>Can still have the disadvantages of other strategies- </strong>After customer segmentation, you will have fewer models than customers but you will probably still have tens of models that are also hard to deploy and maintain and requires automation and engineering effort.</li></ul><p><strong>Advantages</strong></p><ul><li><strong>Control in tradeoffs- </strong>The number of segments generally controls the tradeoff between model accuracy (training on more or less similar data) and operational complexity (number of models), in this method you can choose the point that suits you best.</li><li><strong>Best of both worlds?- </strong>In some use cases the customer segmentation can give you the best of both worlds, large datasets of relevant data and models for cold start customers with a not very large engineering effort by maintaining only a few models and not hundreds.</li></ul><p><strong>What to pay attention to?</strong></p><ul><li><strong>Segmentation methods- </strong>The method you segment your customers can be one of the most important parts of the model that will affect the accuracy of the model drastically. Make sure you test and iterate on different segmentation methods and find the one the yields the best model.</li><li><strong>Engineering effort- </strong>While using customer segmentation is much simpler than using model per user, some engineering effort in automation, deployment, and training will still be needed.</li></ul><p><strong>My takeaways</strong></p><p>I haven’t tried this method in any of our use cases but I did hear from colleges in the industry that they use customer segmentation. They mostly said that finding the best way to segment customers is a hard problem and that the data scientist must pay lots of attention to it.</p><h3>Transfer learning</h3><p>Using transfer learning is the most advanced method of the 4. In this method, we will first train a global model on all of our data with lots of data and later fine-tune the model on a specific customer&#39;s data. This method will mostly work with deep learning models that can transfer knowledge between networks.</p><p><strong>Disadvantages</strong></p><ul><li><strong>Mostly works with deep learning</strong>- Transfer learning is mostly used with deep neural networks but for many applications (especially tabular data), tree-based models like XGboost are more accurate, explainable, and lightweight. Thus sometimes you will not be able to use transfer learning with the model that you chose.</li><li><strong>Requires a nontrivial orchestration- </strong>Training a large model on all your data and then finetuning on each customer is not a simple process and will probably require building a complicated pipeline for every model retraining.</li></ul><p><strong>Advantages</strong></p><ul><li><strong>Can be very accurate- </strong>By using transfer learning we can automatically leverage the commonalities in our data in the pre-training phase and discard the non-relevant information in the fine-tuning part, this can potentially lead to a very accurate model. Models like BERT, GPT3, and SimCLR are evidence of the potential of transfer learning.</li><li><strong>Cold start- </strong>In the transfer learning scenario we can’t start giving predictions immediately like in the global model but we can train a very accurate model for a new customer with only hundreds or thousands of data points.</li></ul><p><strong>What to pay attention to?</strong></p><ul><li><strong>How to transfer the knowledge- </strong>There are many methods to do transfer learning. We can finetune the network, extract embeddings or use knowledge distillation. Each of these methods can lead to a better or worse model depends on the use case.</li><li><strong>New customers- </strong>It’s important to validate the model on customers that were not available in the pretraining phase and assess the amount of data needed for a new customer to train a good model.</li></ul><p><strong>My takeaways</strong></p><p>I didn’t use transfer learning for learning on multiple customers but I used transfer learning on a few different use cases and it showed great effectiveness in improving model performance. I believe this is a great solution to scenarios where having an accurate model with a small amount of labeled data is important.</p><h3>Conclusions</h3><p>In this post, I reviewed 4 different methods for modeling the multi-customer scenario and tried to highlight the pros and cons of every approach. The answer to the question in the headline is that it depends on the business and technical requirements of your project. How accurate should the model be? How much engineering effort are you willing to do? How important is the cold start of new customers? How different is the data of different customers? and many more questions that will help you choose the right solution for your problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9e8734d991" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/should-i-train-a-model-for-each-customer-or-use-one-model-for-all-of-my-customers-f9e8734d991">Should I train a model for each customer or use one model for all of my customers?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Data Analysis for Text Data]]></title>
            <link>https://medium.com/bigpanda-engineering/exploratory-data-analysis-for-text-data-29cf7dd54eb8?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/29cf7dd54eb8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Tue, 17 Mar 2020 12:09:39 GMT</pubDate>
            <atom:updated>2020-03-17T13:56:24.114Z</atom:updated>
            <content:encoded><![CDATA[<p>Last month I started my new data science job at BigPanda and after a few days of installations, lectures and meeting new people I finally got access to the data. As an educated data scientist that always works according to <a href="https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining">CRISP-DM</a>, I wanted to start my project with an exploratory data analysis (EDA). I usually do EDA on tabular data by using histograms, bar plots, a correlation matrix, scatter plots and other tools but none of these tools were suitable for textual data — which is the main data type at BigPanda.</p><p>In this blog post, I’m going to review several methods I used to get to know the textual data in the data exploration phase with code examples.</p><p>I will use a dataset of <a href="https://www.kaggle.com/kingburrito666/better-donald-trump-tweets">Donald Trump tweets</a> from Kaggle for demonstration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*7RuMdyq2yATszTUwrCuACw.jpeg" /></figure><h3>First of all, see some examples!</h3><p>The first thing I do in every new data exploration is looking at the data. I want to feel out some basic questions about the text: Do I need a special pre-processing method? Are there any special characteristics? If there are labels, can I understand the label from the text or do I need more knowledge on the domain? Can I already spot special patterns in the data? Is there a connection to other variables in the data?</p><p>In the Trump dataset, we can see some Twitter specifics (ie: hashtags and usernames) and many URLs that we probably want to remove.</p><h3>The Analysis</h3><p>Now, on to the good stuff!</p><p>We’ll start the analysis with very simple text data statistics like text length and word frequencies. After that, we’ll investigate more sophisticated text characteristics using unknown words and feature engineering. Finally, we’ll use a machine learning-based topic modeling method to create semantic features from the text and assess the text in the context of time series analysis.</p><h3>Basic statistics on the text</h3><h4>Text length</h4><p>The length of the samples in the dataset is very important, as it can affect how you represent your text as features for the ML models. For example, TF-IDF is usually too sparse for short texts and average Word2Vec is usually too noisy for long texts.</p><p>The length can also affect the algorithm you use. For example, LSTM networks are much better than vanilla RNN networks on long texts.</p><p>We calculate the number of words in each tweet and look at the length distribution.</p><pre>lens = data.Tweet_Text.str.split().apply(lambda x: len(x))</pre><pre>print(lens.describe())<br>lens.hist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*egwefDo-hlTVVhkDiuhJJA.png" /></figure><p>Results of this calculation establish that text length is short (average length of 17 words) but not very short (less than 10% has less than 5 words).</p><h4>Term frequencies</h4><p>In this step, I find the most frequent words in the data, extracting information about its content and topics.</p><p>When doing your analysis, remember to add domain-specific stop words, not just the known English stop words. I remove stop words from this particular analysis because they appear often but are not very informative.</p><pre>from sklearn.feature_extraction.text import CountVectorizer<br>from nltk.corpus import stopwords<br>stops =  set(stopwords.words(&#39;english&#39;)+[&#39;com&#39;])</pre><pre>co = CountVectorizer(stop_words=stops)<br>counts = co.fit_transform(data.Tweet_Text)<br>pd.DataFrame(counts.sum(axis=0),columns=co.get_feature_names()).T.sort_values(0,ascending=False).head(50)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*kN1m_9OU_2WyWztYIqrqEQ.png" /></figure><p>The top words are mostly campaign-related words, ie: Trump, Hillary, America, vote, poll, etc.</p><p>Now we can check for frequent <a href="https://en.wikipedia.org/wiki/Bigram">bi-grams</a>:</p><pre>co = CountVectorizer(ngram_range=(2,2),stop_words=stops)<br>counts = co.fit_transform(data.Tweet_Text)<br>pd.DataFrame(counts.sum(axis=0),columns=co.get_feature_names()).T.sort_values(0,ascending=False).head(50)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*Tilc7p8-uiXI4I3LbKb4Cw.png" /></figure><p>The most popular bi-grams are Trump’s special phrases, like “crooked Hillary” and “failing NYTimes”. There are also many mentions of competitors in the bi-grams (Hillary Clinton, Ted Cruz, Marco Rubio). This already gives us information about the differences in sentiment towards specific entities in the text.</p><h3>More Sophisticated Text Characteristics</h3><h4>Unknown words</h4><p>In this step, I search the text for elements like:</p><ul><li>Unknown words or words that don’t match known word lists.</li><li>Words that appear especially often.</li><li>How much of the text is slang?</li><li>Which words are Domain-specific</li><li>How many spelling mistakes appear in the text.</li></ul><p>We can also find more steps to our preprocessing that will yield a better percentage of known words. This step is critical if you want to use pre-trained embeddings or pre-trained networks. In these cases, we need to compare the text data to the dictionary of the pre-trained resource.</p><p>In this analysis, I will use an English word list from <a href="https://github.com/dwyl/english-words">https://github.com/dwyl/english-words</a> and pre-process the text by removing hashtags, usernames and non-alphanumeric symbols.</p><pre>words = pd.read_table(&#39;<a href="https://raw.githubusercontent.com/dwyl/english-words/master/words.txt&#39;">https://raw.githubusercontent.com/dwyl/english-words/master/words.txt&#39;</a>)<br>words.columns=[&#39;word&#39;]<br>words = words[&#39;word&#39;].str.lower().values.tolist()</pre><pre>data[&#39;clean_text&#39;] = data.Tweet_Text.apply(lambda x: &#39; &#39;.join([i for i in x.split(&#39; &#39;) if not (i.startswith(&#39;@&#39;) or i.startswith(&#39;#&#39;))]))<br>data[&#39;clean_text&#39;] = data.clean_text.str.lower().str.replace(&#39;[^a-zA]&#39;, &#39; &#39;)</pre><pre>non_list = {}<br>for sent in tqdm(data.clean_text.str.split().values):<br>    for token in sent:<br>        if token not in words:<br>            non_list[token] = 1 if token not in non_list else non_list[token]+1<br>            <br>pd.Series(non_list).sort_values(ascending=False).head(30)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*4fndZUZw3jlPJkxdn51F4Q.png" /></figure><p>We probably need to add ״http” and ״https״ to the stop words (domain-specific stop words) list because they are frequent but not significant. Moreover, mostly names and bigrams were united (maybe we need a smarter tokenizer). <a href="https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html">Stemming or lemmatization</a> can also help with removing more unknown words.</p><h3>Feature engineering</h3><p>It is possible to build some structure from the text by extracting features and then using the tools for structured data (histograms, bars, etc) on the features. Even though these features are very domain-specific, we can extract different types of entities or the sentiment of a sentence as features for almost every text dataset.</p><p>Here is a bar graph using hashtag frequency as features.</p><pre>data.Tweet_Text.str.extractall(r&#39;(\#\w+)&#39;)[0].value_counts().head(20).plot.bar()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*0LLEAniZGJZc8sV-G-lRXQ.png" /></figure><h3>Using machine learning models</h3><h4>Topic modeling</h4><p>Topic modeling is a group of unsupervised machine learning methods aimed to extract meaningful topics from a text corpus. Topic modeling is helpful in exploring a new text dataset and understanding the different types of text samples. We can also use regular clustering but topic modeling is more interpretable and thus better for exploration.</p><p>I use the Latent Dirichlet Allocation algorithm with 10 topics and print the most important words in every topic.</p><pre>from sklearn.decomposition import LatentDirichletAllocation, NMF</pre><pre>vectorizer = CountVectorizer(stop_words=stops)<br>model = vectorizer.fit(data.clean_text)<br>docs = vectorizer.transform(data.clean_text)</pre><pre>lda = LatentDirichletAllocation(20)<br>lda.fit(docs)</pre><pre>def print_top_words(model, feature_names, n_top_words):<br>    for topic_idx, topic in enumerate(model.components_):<br>        message = &quot;Topic #%d: &quot; % topic_idx<br>        message += &quot; &quot;.join([(feature_names[i])<br>                             for i in topic.argsort()[:-n_top_words - 1:-1]])<br>        print(message)<br>    print()<br>    <br>print_top_words(lda,vectorizer.get_feature_names(),10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*_tPqtZRgUESH_BUbCqZz7Q.png" /></figure><p>Here we see some clear topics. For example, Topic #0 about competitors, Topic #9 about rallies and Topic #2 about the campaign phrases:</p><p>Now we assess how often Trump tweets about each topic:</p><pre>data[&#39;topic&#39;]=lda.transform(docs).argmax(axis=1)<br>data.topic.value_counts(normalize=True).plot.bar()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*w_UwGNRXLduTGqwEtSSZnw.png" /></figure><p>And we can look at examples of specific topics:</p><pre>data[data.topic==0].Tweet_Text.sample(5).values</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rnN75TlVSnyRlhP_J0bmGA.png" /></figure><h4>Time series analysis</h4><p>In many applications, our text corpus is associated with a time series (ie: news, tweets, reviews, etc.). Here we investigate the connection of the text to the time series. Now that we’ve built a text structure using the topic modeling, we can look at the topic distribution on the time series. We can also use time series anomaly detection and time series clustering methods to gain more insights on the topics.</p><pre>data[&#39;Date&#39;]=pd.to_datetime(data.Date,yearfirst=True)<br>data.index = data.Date<br>for i in range(10):<br>    temp = data[data.topic==i]<br>    temp.resample(&#39;7D&#39;).size().plot()<br>    plt.title(&#39;topic %s&#39; %i)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*i0bWVIFEtgHu0UF-Njs4qA.png" /></figure><p>Topic #0 (competition) has 3 significant anomalies in May-August, there are suddenly over 30 tweets a week on the same topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*VBVFXLGUgRWm2WSg5PsUTw.png" /></figure><p>Topic #7 (radio and TV) has an interesting structure — in April it decreases drastically, gradually increasing until November. This makes sense because it represents two election periods, the republican primaries election and the US presidential election.</p><h3>Conclusions</h3><p>These methods really helped me get to know my text data, find the issues that needed attention and gain insight on how to appropriately use machine learning models on data. I hope it helps you, too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29cf7dd54eb8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigpanda-engineering/exploratory-data-analysis-for-text-data-29cf7dd54eb8">Exploratory Data Analysis for Text Data</a> was originally published in <a href="https://medium.com/bigpanda-engineering">BigPanda Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Best Tips for Agile Data Science Research]]></title>
            <link>https://medium.com/data-science/my-best-tips-for-agile-data-science-research-b40365cc979d?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/b40365cc979d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Wed, 23 Jan 2019 14:47:16 GMT</pubDate>
            <atom:updated>2019-01-23T14:47:16.553Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to bring maximum value in minimal time</h4><p>A few days ago I read two great posts on agile data science project management by <a href="https://towardsdatascience.com/data-science-project-flow-for-startups-282a93d4508d">Shay Palachy</a> and <a href="https://towardsdatascience.com/data-science-agile-cycles-my-method-for-managing-data-science-projects-in-the-hi-tech-industry-b289e8a72818">Ori Cohen</a>. These posts discuss the different phases of data science projects and how are they different from regular software projects. These posts have inspired me to write my own view on agile data science with a focus on the research methodology and not on the whole project phases.</p><p>Agile data science research is hard, how can you give a time estimation when you are not sure that your problem is solvable? how can you plan your sprint before looking at the data? You probably can’t. Agile data science requires many adjustments, In this post, I am going to share some of the best practices that work best for me for agile data science research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*JlRk82LTksgk7uDu_WCLTA.jpeg" /></figure><h3>Set the project goals</h3><p>Every machine learning project should start by defining the goals of the project. We must define what is a good result in order to know when to stop the research and move forward to the next problem. This phase is usually done with the business stakeholders.</p><p>The goal is defined by 3 questions:</p><ul><li><strong>What is the KPI that we are optimizing?</strong> This is maybe the most important question in the project, the KPI must be measurable with a test set but also as correlative as possible to the business KPI.</li><li><strong>What is the evaluation method?</strong> What is the size of the test-set? Do we need time series split/group split? Do we need an online test?</li><li><strong>What is the minimum valuable KPI?</strong> Sometimes the machine learning model will replace some simple heuristic an even 65% percent accuracy will be very valuable for the business. We need to define what is a success.</li></ul><h3>Always compare to a baseline model</h3><p>What is a good performance is a pretty hard question which is heavily based on how hard is the problem and what are the business needs. My advice is to start your modeling by building a simple baseline model, it can be a simple machine learning model with basic features or even a business rule (heuristics) like the average label in an important category. This way we can measure our performance in comparison to the baseline and monitor our improvement in the task.</p><h3>Start with a simple model</h3><p>Iterations are one of the core characteristics of agile development. In a data science project, we don’t iterate on features like the engineering team, we iterate on models. Starting with a simple model with a small number of features and making it more and more complex iteratively has many advantages. You can stop at any point when your model is good enough and save time and complexity. You know exactly how every change you made has affected the model performance and this gives you intuition for your next experiments and maybe most importantly, by adding complexity iteratively you can debug your model for bugs and data leakages much easier and faster.</p><h3>Plan sub-goals</h3><p>Planning research projects is hard because they have a very large amount of uncertainty. From my experience, it is best to plan your projects using subgoals, for example, data exploration, data cleaning, dataset building, feature engineering, and modeling are small parts of the research that you can plan at least a few weeks forward. These sub-goals can bring value on their own without the final model. For example, after data exploration, the data scientist can bring actionable insights for the business people and data set cleaning and building can help other data scientist and analysts for their own projects immediately.</p><h3>Fail fast</h3><p>Failing fast is maybe my most important point and probably the hardest to do. At each iteration, you must ask yourself what is the probability that the model performance will reach the minimum valuable KPI? I think that making the model more complex iteratively really helps in this part. Adding more features and trying more models usually gives incremental improvements. If your model performance is 70% and your minimum valuable KPI is 90% you are probably not going to get there, so, you need to stop your project and move to the next problem or change something drastic like changing your label or tagging much more data. I am not saying that you shouldn’t try to solve very hard problems, just make sure that you are not wasting time on methods that probably won’t reach your project goals.</p><h3>Move to production ASAP</h3><p>My last advice is deploying your model in production at the earliest point or a little after the point that your model is valuable. I know that maybe your final model will have totally different features and a lot of the work will be wasted. But, first, your model gives value, why wait? Secondly and more importantly, in many cases, the production has its own constraints, some features are not available at the production systems, some features are in different formats, maybe your model is to slow or uses to much RAM etc. Solving these problems early can save a lot of unrealistic modeling time.</p><p>Hope you enjoyed my post, and you’re more than welcomed to read and follow my <a href="https://medium.com/@yonatan.hadar">blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b40365cc979d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/my-best-tips-for-agile-data-science-research-b40365cc979d">My Best Tips for Agile Data Science Research</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top Examples of Why Data Science is Not Just .fit().predict()]]></title>
            <link>https://medium.com/data-science/top-examples-of-why-data-science-is-not-just-fit-predict-ce7a13ef7663?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce7a13ef7663</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 15:23:56 GMT</pubDate>
            <atom:updated>2018-12-11T15:28:34.027Z</atom:updated>
            <content:encoded><![CDATA[<h3>Top Examples of Why Data Science is Not Just .fit().predict()</h3><h4>In this post, I’m going to review some of the top concepts I learned that turned me from a technical data scientist to a good data scientist</h4><p>Two months ago, I finished my second year as a data scientist at YellowRoad so I decided to do a retrospective analysis on my projects, what did I do good? what interesting methods did I discover? what mistakes I did? and most importantly, what have I learned?</p><p>In this post, I’m going to review some of the top concepts I learned that in my opinion turned me from a technical data scientist that run models and can easily be replaced by autoML to a good data scientist that gives value in every project even if it doesn’t require state of the art research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*z1tqL_iF4peGVP6u5wl8ow.jpeg" /></figure><p><strong>1) Data Leakage</strong></p><p>According to Kaggle, data leakage is the creation of unexpected additional information in the training data, allowing a model to make unrealistically good predictions. Or a simpler definition, anything that will make your model perform better in research than in production. I had a million types of data leakage, sometimes due to a bug in the model (like entering the label as a feature), sometimes due to a bug in the dataset creation (like creating a feature that contains information about the future) and sometimes due to a bug in the business logic (like using features that cannot be used in production). The most dangerous thing about data leakage is that it is very hard to find and sometimes you discover the leakage only when you see that while your test set accuracy is 90% the production accuracy is only 60%. If you want to learn more about data leakage you can watch <a href="https://www.youtube.com/watch?v=m-tAASQA7XQ">my lightning talk</a>, starting at 11:15 (only 10 minutes:)).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*EfV6xUrzNjMOj8A29gDDlg.png" /></figure><p><strong>2) Selection Bias</strong></p><p>Selection<strong> </strong>bias is the bias introduced by the selection of individuals, groups or data for analysis in such a way that proper randomization is not achieved (from <a href="https://en.wikipedia.org/wiki/Selection_bias">Wikipedia</a>). For example, if you are building a training set using a questionnaire that you put on your Facebook wall. The people that will answer it are probably around your age, from your country or even city, with your socioeconomic properties. If you build a model based only on this data, when it will go to production it will be exposed to a much wider variety of population and the fact that your model worked on your friends doesn’t mean it will work on everyone. Sometimes selection bias is even harder than data leakage because if you could monitor the labels of all the population you would use these labels for training. Selection bias is also harder to fix, you can reweight your training set based on the whole population or remove features that differentiate the training set from the whole population, but you can’t really evaluate if these methods work. More on selection bias on the <a href="https://www.youtube.com/watch?v=3ZWCKr0vDtc">brilliant talk</a> by Lucas Bernardi.</p><p><strong>3) Production Feedback Loops</strong></p><p>In many machine learning applications, the production workflow looks like this: The machine learning model performs predictions on a user behavior, the system takes actions according to the model output, the user reacts to the system and the model re-trains based on the user behavior. This feedback loops where the model effects on its own future training set can cause some biases. For example, in a recommendation engine, we rank according to what items the user clicked but if we ranked an item first, its clicking probability is higher than an item that was ranked as number 10. This way our model will never learn new things even if some item is better for the user. To address this problem we usually use exploration, ranking an item high even if it got a low score from the model with some probability or rescaling the model labels with respect to the previous ranking of the items, if an item was clicked from the second page it’s a stronger indication than if the first item was clicked. Another example is predicting if a customer will return his loan or not. We don’t know what would have happened if we approved the loans that we declined, thus, we are creating a bias for the next training set of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*Z2-5-d1KyHEMkGbYwsKjUQ.png" /></figure><p><strong>4) Special cross-validation techniques</strong></p><p>In some cases, using a random cross-validation is not enough, because the production data will look different than the research data and we need to use special cross-validation methods. For example, in a time series dataset, there is usually a change in the data distribution due to nature or special events. If we will use a random cross-validation, the train and test data distribution will be the same and we will not know if our model is robust to this data distribution changes. Therefore, in time series we need to use a sliding window cross-validation and not a random one. More on time series cross-validation you can read in <a href="https://towardsdatascience.com/time-series-nested-cross-validation-76adba623eb9">this post</a>. Another type of special case is when you have groups in your data, for example in a medical clinical trial where you have many examples of the same patient during the trial. In order to make sure that our model will generalize to new groups we need to use group cross-validation where each fold contain all the examples for a few groups and in test time we will test on new groups.</p><p><strong>5) Choosing the “right” evaluation metric</strong></p><p>It’s a bit cliché sentence and sometimes very hard to do in real scenarios but the way you evaluate your model should be highly correlated with the business needs of your use case. I will give some examples, I had a project in an ad-tech company of building a bidder that will decide how much to pay for an impression. Our approach was building a model that will predict how much the company can earn from an impression and after, set the bid price based on the prediction. In this model, MSE was a bad evaluation metric even though it is probably the most popular metric for regression models. MSE punishes large mistakes more than small mistakes. In this model, most of the large mistakes were anomalies when the company has managed to sell the impression for a very high price. Predicting anomalies is almost impossible so probably even in these anomalies, everyone else will offer about the same bids as other impressions. If our predictions will be lower, we will still sometimes buy these anomalies but if we will offer large bids on regular impressions, we will lose a lot of money. Another example is ranking models, accuracy and MSE obviously don’t match to the business use case but even different ranking metrics can fit different use cases. If you present to the user only the top K items, Recall at K should be your evaluation metric. But if you present a long list of ranked items to the user and his probability of seeing an item reduces in lower ranks, NDCG is a very good metric.</p><p><strong>6) Start with a simple model</strong></p><p>Using the new Deep Cnn Rnn with Residual Transformer Attention of Gans with some Cyber Blockchain layers of Buzzwords is cool, even very cool but it’s not a good way to start a project. Starting with a simple model with a small number of features gives you several advantages. It is way easier to debug the model and find bugs or data leakages, you understand the model’s predictions better and know how to make it perform better and which research directions are more promising, you can fail fast if you see that there is no signal in the data and move to your next project with a little time spent and maybe the most important advantage, sometimes the simple model is just good enough for your business needs so you saved tons of research and GPU time.</p><p><strong>7) Don’t use epsilon models</strong></p><p>In a Kaggle competition, the winning model is usually an ensemble of 30 models with hundreds of features. These models are usually 0.01 points better than one good model or an ensemble of 2–3 simple models. These complex models are called epsilon models because they add only an epsilon of performance but a huge complexity to the production system. Each model or even feature is another potential place for bugs and it requires more processing time both in training and inference and more memory. In the tradeoff of cost due to complexity and performance, except special cases where the model’s accuracy is highly correlated with revenue, the simpler model wins.</p><p><strong>8) Monitor your model in production</strong></p><p>In most real-life applications, the data distribution is always changing due to business decisions, change in user behavior, changes that the model causes and more. This means that your model performance will probably decrease with time. In order to stop this phenomenon, we need to monitor the performance of the model and re-train the model or even return to the research phase when the performance decreases significantly. But, in many scenarios, we don’t have new labels (for example if we labeled our training set with Mechanical Turk) or we discover the labels after a long time (for example if we predict if a business will return its 5 years loan). In these scenarios, we should monitor the changes in the data between the training set and the production data or even monitor the changes in the distribution of the model’s predicted probabilities. <a href="https://www.youtube.com/watch?v=IqKunD0Bl5c">Here</a> is a great lecture about monitoring machine learning model in production by Jannes Klaas.</p><p><strong>Thank you for reading :)</strong></p><p>I hope this post gave you some insights and hopefully you learned from my past mistakes and won’t do them yourself in the future.</p><p>Hope you enjoyed my post, and you’re more than welcomed to read and follow my <a href="https://medium.com/@yonatan.hadar">blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce7a13ef7663" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/top-examples-of-why-data-science-is-not-just-fit-predict-ce7a13ef7663">Top Examples of Why Data Science is Not Just .fit().predict()</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learned from Applying Deep Learning for NLP Without Big Data]]></title>
            <link>https://medium.com/data-science/lessons-learned-from-applying-deep-learning-for-nlp-without-big-data-d470db4f27bf?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/d470db4f27bf</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Wed, 24 Oct 2018 19:31:34 GMT</pubDate>
            <atom:updated>2018-10-25T08:46:20.054Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*fECSk8I56fB3ymR7AK1dFA.png" /></figure><p>As a data scientist, one of your most important skills should be choosing the right modeling techniques and algorithms for your problems. A few months ago I was trying to solve a text classification problem of classifying which news articles will be relevant for my customers.</p><p>I had only a few thousand labeled examples so I started with simple classic machine learning modeling methods like Logistic regression on TF-IDF. These models that usually works well on text classification of long documents like news articles performed only slightly better than random on this task.</p><p>After investigating the mistakes of my model I found that bag of words representation is just not enough for this task and I need a model the will use a deeper semantic understanding of the documents.</p><p>Deep learning models have presented very good performance on complex tasks that require deep understanding of text like translation, question answering, summarization, natural language inference etc. so it seemed like an excellent approach but deep learning is usually trained on hundreds of thousands or even millions of labeled data points and I had a much smaller dataset.</p><p>Usually, we need big datasets for deep learning to avoid over-fitting. Deep neural networks have many many parameters thus usually if they don’t have enough data, they tend to memorize the training set and perform poorly on the test set. To avoid this phenomenon without big data we need to use special techniques.</p><p>In this post I will show some methods I found on articles, blogs, forums, Kaggle, and more resources or developed by myself in order to make deep learning work better on my task without big data. Many of these methods are based on best practices that are widely used in computer vision.</p><p>A small disclaimer: I am not a deep learning expert and this project was one of my first big projects with deep learning. Everything on this post is based on my experience and possibly will behave differently on your problems.</p><h3>Regularization</h3><p>Regularization methods are methods that are used inside the machine learning model in different ways in order to avoid over-fitting, these methods have a strong theoretical background and work in a generic manner for most problems.</p><p><strong>L1 and L2 Regularization</strong></p><p>These methods are probably the oldest and are used for many years in many machine learning models. In this method, we add the weight size to the model’s loss function that we are trying to minimize. This way the model will try to make the weights small and weights that don’t help the model significantly will be reduced to zero and won’t affect the model. This way we have a much smaller number of weights that the model can use to memorize the training set. for more explanation, you can read <a href="https://towardsdatascience.com/only-numpy-implementing-different-combination-of-l1-norm-l2-norm-l1-regularization-and-14b01a9773b">this</a> post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pv8LrA_E7Npr3FXz9UvBmg.png" /></figure><p><strong>Dropout</strong></p><p>Dropout is another newer regularization method that suggests that during training time, every node (neuron) in the neural network will be dropped (weights will be set to zero) in a probability of P. This way, the network can’t rely on specific neurons or interaction of neurons and must learn every pattern in different parts of the network. This makes the model focus on important patterns that generalize to new data.</p><p><strong>Early stopping</strong></p><p>Early stopping is an easy regularization method, just monitor your validation set performance and if you see that the validation performance stops improving, stop the training. This method is very important without big data because the model tends to start over-fitting after 5–10 epochs or even earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*fVtU6koRzJmBVi9c8HC87w.png" /></figure><p><strong>Small number of parameters</strong></p><p>If you don’t have a large dataset you should be very careful with the number of layers and the number of neurons in each layer. Also, special layers like Convolutional layers have fewer parameters than fully connected layers so it’s very helpful to use them when they fit your problem.</p><h3>Data augmentation</h3><p>Data augmentation is a method to create more training data by changing training data in a way that the label doesn’t change. In computer vision, many image transformations are used to augment a dataset like flipping, cropping, scaling, rotating and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fzUU9EBEVulBI7L_vLltPA.png" /><figcaption>Image data augmentation example</figcaption></figure><p>These transformations are good for image data but will not work for text, for example flipping a sentence like “dogs love I” is not a valid sentence and using it will make the model learn junk. Here are a few text data augmentation methods:</p><p><strong>Synonym replacement</strong></p><p>In this method, we replace random words in our text with their synonyms, for example, we will change the sentence “ I like this movie very much” to “ I love this film very much” which still has the same meaning and probably the same label. This method didn’t work for me because synonyms have very similar word vectors, so the model sees both sentences as almost the same sentence and not as an augmentation.</p><p><strong>Back translation</strong></p><p>In this method, we take our text, translate it to an intermediate language with machine translation and then translate it back to English. This method was used successfully in the Kaggle toxic comments challenge. For example, if we translate “ I like this movie very much” to Russian, we get “Мне очень нравится этот фильм”, when we translate back to English we get “I really like this movie”. The back translation method gives us both synonym replacement like the first method but also it can add or remove words and paraphrase the sentence while preserving the same meaning.</p><p><strong>Document cropping</strong></p><p>News articles are very long and while looking at the data I saw that I don’t need all the article in order to classify the document. Moreover, I saw that the articles main idea usually returns a few times. This made me think on cropping the article to several sub-documents as a data augmentation, this way I will have much more data. First, I tried to sample a few sentences from the documents and create 10 new documents. This created documents without a logical relation between sentences, so I got a bad classifier. My second attempt was splitting every article into pieces of 5 consecutive sentences. This method worked very nicely and gave me a nice boost in performance.</p><p><strong>Generative adversarial network</strong></p><p>GANs are one of the most exciting recent advances in data science, they are usually used as a generative model for image creation. <a href="https://towardsdatascience.com/generative-adversarial-networks-for-data-augmentation-experiment-design-2873d586eb59">This blog post</a> explains how to use GANs for data augmentation for image data but it probably can be used for text also.</p><h3>Transfer learning</h3><p>Transfer learning is the use of weights from a network that was trained on another problem, usually with a big dataset, on your problem. Transfer learning is sometimes used as a weight initialization to some of the layers and sometimes as a feature extractor that we don’t train anymore. In computer vision, starting with a pre-trained Imagenet model is a very common practice to solve problems but NLP doesn’t have a very big dataset like Imagenet that can be used for transfer learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*d3lHCtCkbfVambYFrYIfqQ.jpeg" /></figure><p><strong>Pre-trained word vectors</strong></p><p>NLP deep learning architectures usually start with an embedding layer that converts One hot encoded words to numerical vector representation. We can train the embedding layer from scratch but we can also use pre-trained word vectors like Word2Vec, FastText or Glove that were trained on tons of data using unsupervised learning methods or train on data from our domain. Pre-trained word vectors are very effective because they give the model context for words that are based on a lot of data and reduces the number of parameters of the model which reduces the chances for overfitting significantly. You can read more about word embedding <a href="https://www.springboard.com/blog/introduction-word-embeddings/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXNXYfAqfLUeiDXPCo130w.png" /></figure><p><strong>Pre-trained sentence vectors</strong></p><p>We can change the input for the model from words to sentences, this way we can have smaller models with a smaller number of parameters that will still be expressive enough. In order to do this, we can use pre-trained sentence encoders like Facebook’s <a href="https://github.com/facebookresearch/InferSent">InferSent</a> or Google’s <a href="https://tfhub.dev/google/universal-sentence-encoder/1">Universal sentence encoder</a>. We can also train a sentence encoder on unlabeled data from our domain using methods like skip-thought vectors or language models. You can learn more about unsupervised sentence vectors from my <a href="https://blog.myyellowroad.com/unsupervised-sentence-representation-with-deep-learning-104b90079a93">previous blog post</a>.</p><p><strong>Pre-trained language models</strong></p><p>Recent papers like <a href="https://arxiv.org/abs/1801.06146">ULMFIT</a>, <a href="https://blog.openai.com/language-unsupervised/">Open-AI transformer</a>, and <a href="https://arxiv.org/abs/1810.04805v1">BERT</a> got amazing results for many NLP tasks by pre-training a language model on a very large corpus. Language model is a task of predicting the next word in a sentence using the previous words. For me, this kind of pre-training didn’t really help in getting better results, but the articles have shown a few methods to help with better fine tuning that I haven’t tried. <a href="http://ruder.io/nlp-imagenet/">This is</a> a great blog about pre-trained language models.</p><p><strong>Pre-training with unsupervised or self-supervised learning</strong></p><p>If we have a large dataset of unlabeled data from our domain we can use an unsupervised method like Autoencoders or masked language models to pre-train our models using only the text itself. Another option that worked better for me was using self-supervision. Self-supervised models are models that the label is extracted automatically without human annotation. A great example is the Deepmoji project. In Deepmoji, the authors trained a model to predict emojis from tweets, after getting good results in the emoji prediction they used their network to pre-train a tweeter sentiment analysis model that got state of the art results. The emoji prediction and sentiment analysis are obviously very related thus it performed very well as a pre-training task. Self-supervised tasks for news data can be predicting the headline, the newspaper, number of comments, number of retweets and more. Self-supervised can be a very good way to pre-train, but it is usually hard to tell what proxy label will be connected to your real label.</p><p><strong>Pre-training with other in company networks</strong></p><p>In a lot of companies many machine learning models are built on the same dataset or similar datasets for different tasks. For example, for tweets, we can predict its topic, sentiment, number of retweets, and more. Pre-training your network with a network that is already in use can be the best thing that can be done and for my task, it gave a nice boost in performance.</p><h3>Features engineering</h3><p>I know that deep learning “killed” feature engineering and that it is a little old fashion to do it. But when you don’t have big data, helping the network to learn complex pattern with feature engineering can give a big boost in performance. For example, in my classification of news articles the author, the newspaper, the number of comments, tags, and more features can help in predicting our label.</p><p><strong>Multimodal architecture</strong></p><p>We can use multimodal architecture to combine document level features into our model. In multimodal we build two different networks, one for the text and one for features, merge their output layers (without softmax) and add a few more layers. These models are pretty hard to train because the features usually have a stronger signal than the text so the network learns mostly the feature effect.<a href="https://becominghuman.ai/neural-networks-for-algorithmic-trading-multimodal-and-multitask-deep-learning-5498e0098caf"> This </a>is a great Keras tutorial on multimodal networks. This approach improved my performance by less than one percent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*ERtepDsAYz51480F2Vdyww.png" /></figure><p><strong>Word level features</strong></p><p>Another type of feature engineering is word level features like Part of speech tagging, semantic role labeling, entity extraction and more. we can combine one hot encoded representation or an embedding of the word feature with the embedding of the word and use it as an input to the model. We can also use other word features in this method, for example in a sentiment analysis task we can take a sentiment dictionary and add another dimension to the embedding with 1 for words that we have in our dictionary and 0 for other words, this way the model can easily learn some of the words that it needs to focus on. On my task, I added dimensions with certain entities that were important and this gave me a nice boost in performance.</p><p><strong>Pre-processing as feature engineering</strong></p><p>The last feature engineering method is pre-processing the input text in a way that it will be easier for the model to learn. One example is special “stemming”, if sports is not important for our label we can change football, baseball, and tennis to the word sport, this will help the network learn that the difference between sports is not important and can reduce the number of parameters in the network. Another example is using automatic summary. As I said earlier neural networks don’t perform very well on long texts so we can run an automatic summary algorithm like “text rank” on our text and give the network only the important sentences.</p><h3>My model</h3><p>In my case, after trying different combinations of the methods I discussed in this post, the best model was a Hierarchical Attention Network from <a href="https://www.cs.cmu.edu/~hovy/papers/16HLT-hierarchical-attention-networks.pdf">this </a>article with dropout and early stopping as regularization and document cropping as data augmentation. I used pre-trained word vectors and pre-training on another task that my company did for this customer on the same data. As feature engineering, I added entity word level feature to the word embedding. These changes from the basic model gave me a boost of almost 10% accuracy that made my model from slightly better than random to a model with valuable business impact.</p><p>Deep learning with small data is still in its early stages as a research field but it looks like it’s gaining more popularity especially with pre-trained language models and I hope that researchers and practitioners will find more methods that will make deep learning valuable to every dataset.</p><p>Hope you enjoyed my post, and you’re more than welcomed to read and follow my <a href="https://medium.com/@yonatan.hadar">blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d470db4f27bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/lessons-learned-from-applying-deep-learning-for-nlp-without-big-data-d470db4f27bf">Lessons Learned from Applying Deep Learning for NLP Without Big Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unsupervised sentence representation with deep learning]]></title>
            <link>https://medium.com/yellowblog/unsupervised-sentence-representation-with-deep-learning-104b90079a93?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/104b90079a93</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Sun, 08 Apr 2018 13:24:10 GMT</pubDate>
            <atom:updated>2018-04-08T13:24:10.748Z</atom:updated>
            <content:encoded><![CDATA[<p>In recent years, the field of natural-language processing (NLP) has gained a major boost in performance thanks to the technique of representing words as a continuous vector instead of a sparse one-hot encoded vector (Word2Vec).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXNXYfAqfLUeiDXPCo130w.png" /><figcaption>Word2Vec examples</figcaption></figure><p>While Word2Vec works very well and creates nice semantics like King — Man + Woman = Queen, sometimes we are not interested in the representation of a word but in the representation of a sentence.</p><p>In this post, we will review and share code examples for several unsupervised, deep-learning methods of sentence representation. We will present their effectiveness as a pre-processing step for a specific text classification task.</p><h3>The Classification Task</h3><p>The data that we used to present the different sentence representation methods is based on 10,000 news articles that we scraped from the World Wide Web. The task is to classify each article into one of 10 possible topics (the data has topic labels, and is therefore a supervised task). For the sake of the presentation, I will use a logistic regression model, each time using a different pre-processing representation method that we will apply to the article headline.</p><h3>Baseline model — Average Word2Vec</h3><p>We will start with a very simple baseline. We will represent the headline by averaging the headline words in their Word2Vec representation. As previously mentioned, Word2Vec is a machine-learning method for representing words as vectors. The Word2Vec model is trained by predicting words close to the target word with a shallow neural network. You can read more about how the algorithm works <a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/">here</a>.</p><p>We can train our own Word2Vec with <a href="https://radimrehurek.com/gensim/">Gensim</a> but in this example we will use a Google pre-trained Word2Vec model that was built based on Google news data. After representing each word as a vector, we will represent a sentence (the headline) as an average of its words (vectors) and run logistic regression for classification of the article’s category.</p><pre>#load data and Word2vec model<br>df = pd.read_csv(&quot;news_dataset.csv&quot;)<br>data = df[[&#39;body&#39;,&#39;headline&#39;,&#39;category&#39;]]<br>w2v = gensim.models.KeyedVectors.load_word2vec_format(&#39;/GoogleNews-vectors-negative300.bin&#39;, binary=True)</pre><pre>#Build X and Y<br>x = np.random.rand(len(data),300)<br>for i in range(len(data)):<br>    k = 0<br>    non = 0<br>    values = np.zeros(300)<br>    for j in data[&#39;headline&#39;].iloc[i].split(&#39; &#39;):<br>        if j in w2v:<br>            values+= w2v[j]<br>            k+=1<br>    if k &gt; 0:<br>        x[i,:]=values/k<br>    else: non+=1<br>y = LabelEncoder().fit_transform(data[&#39;category&#39;].values)</pre><pre>msk = np.random.rand(len(data)) &lt; 0.8<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]</pre><pre>#Train the model<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>Our baseline average Word2Vec model achieved an accuracy score of <strong>68%</strong>. This is good, but let’s see if we can do better.</p><p>The average Word2Vec method has two main weaknesses: it is a bag-of-words model that doesn’t relate to the word order, and all the words are treated with the same weight. We will try to address these issues in the next methods by leveraging RNN architectures for sentence representation.</p><h3>Autoencoder</h3><p>An autoencoder is an unsupervised deep learning model that attempts to copy its input to its output. The trick of autoencoders is that the dimension of the middle-hidden layer is lower than that of the input data. Thus, the neural network must represent the input in a smart and compact way in order to reconstruct it successfully. In many cases, using the autoencoder for feature extraction has proved to be very effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*HIBRgSV2ePFtCSOF_lgPCQ.png" /></figure><p>Our autoencoder is a simple Sequnce2Sequence architecture built from an input layer followed by an embedding layer, an LSTM layer, and a softmax layer. Both the input and the output of the entire architecture are the headline, and we will use the output of the LSTM to represent the headline. After getting the representation from the encoder, we will use logistic regression to predict categories. In order to gain more data, we will train the autoencoder with all the sentences in the articles and not just the headline.</p><pre>#parse all sentences<br>sentenses = []<br>for i in data[&#39;body&#39;].values:<br>    for j in nltk.sent_tokenize(i):<br>        sentenses.append(j)</pre><pre>#preprocess for keras<br>num_words=2000<br>maxlen=20</pre><pre>tokenizer = Tokenizer(num_words = num_words, split=&#39; &#39;)<br>tokenizer.fit_on_texts(sentenses)<br>seqs = tokenizer.texts_to_sequences(sentenses)<br>pad_seqs = []<br>for i in seqs:<br>    if len(i)&gt;4:<br>        pad_seqs.append(i)<br>pad_seqs = pad_sequences(pad_seqs,maxlen)</pre><pre>#The model<br>embed_dim = 150<br>latent_dim = 128<br>batch_size = 64</pre><pre>#### Encoder Model ####<br>encoder_inputs = Input(shape=(maxlen,), name=&#39;Encoder-Input&#39;)<br>emb_layer = Embedding(num_words, embed_dim,input_length = maxlen, name=&#39;Body-Word-Embedding&#39;, mask_zero=False)<br># Word embeding for encoder (ex: Issue Body)<br>x = emb_layer(encoder_inputs)</pre><pre>state_h = GRU(latent_dim, name=&#39;Encoder-Last-GRU&#39;)(x)<br>encoder_model = Model(inputs=encoder_inputs, outputs=state_h, name=&#39;Encoder-Model&#39;)</pre><pre>seq2seq_encoder_out = encoder_model(encoder_inputs)</pre><pre>#### Decoder Model ####</pre><pre>decoded = RepeatVector(maxlen)(seq2seq_encoder_out)<br>decoder_gru = GRU(latent_dim, return_sequences=True, name=&#39;Decoder-GRU-before&#39;)<br>decoder_gru_output = decoder_gru(decoded)</pre><pre>decoder_dense = Dense(num_words, activation=&#39;softmax&#39;, name=&#39;Final-Output-Dense-before&#39;)<br>decoder_outputs = decoder_dense(decoder_gru_output)</pre><pre>#### Seq2Seq Model ####</pre><pre>#seq2seq_decoder_out = decoder_model([decoder_inputs, seq2seq_encoder_out])<br>seq2seq_Model = Model(encoder_inputs,decoder_outputs )</pre><pre>seq2seq_Model.compile(optimizer=optimizers.Nadam(lr=0.001), loss=&#39;sparse_categorical_crossentropy&#39;)</pre><pre>history = seq2seq_Model.fit(pad_seqs, np.expand_dims(pad_seqs, -1),<br>          batch_size=batch_size,<br>          epochs=5,<br>          validation_split=0.12)</pre><pre>#Feature extraction<br>headlines = tokenizer.texts_to_sequences(data[&#39;headline&#39;].values)<br>headlines = pad_sequences(headlines,maxlen=maxlen)x = encoder_model.predict(headlines)</pre><pre>#classifier<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>We achieved an accuracy score of 60%, which is actually worse than our baseline. We can probably improve this score by optimizing the hyperparameters, increasing the number of training epochs, or training the model on more data.</p><h3>Language model</h3><p>Our second method is training a language model to represent our sentences. A language model describes the probability of a text existing in a language. For example, the sentence “I like eating bananas” would be more probable than “I like eating convolutions.” We train a language model by slicing windows of n words and predicting what the next word will be in the text. You can read more about language modeling with RNN <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">here</a>. By building a language model, we gain an understanding of how “journalistic English” is built, and the model should be capable of focusing on important words in its representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*b5rfWo2gIdAJ7Jj9XF8o4g.png" /></figure><p>Our architecture is similar to the autoencoder architecture, but instead of predicting a sequence of words, we will only be predicting one word. The input will contain windows of 20 words from the articles, and the label will be the 21st word. After training the language model, we will take the headline representation from the LSTM output hidden state and run logistic regression to predict categories.</p><pre>#Building X and Y<br>num_words=2000<br>maxlen=20<br>tokenizer = Tokenizer(num_words = num_words, split=&#39; &#39;)<br>tokenizer.fit_on_texts(df[&#39;body&#39;].values)<br>seqs = tokenizer.texts_to_sequences(df[&#39;body&#39;].values)</pre><pre>seq = []<br>for i in seqs:<br>    seq+=i<br>    <br>X = []<br>Y = []<br>for i in tqdm(range(len(seq)-maxlen-1)):<br>    X.append(seq[i:i+maxlen])<br>    Y.append(seq[i+maxlen+1])</pre><pre>X = pd.DataFrame(X)<br>Y = pd.DataFrame(Y)<br>Y[0]=Y[0].astype(&#39;category&#39;)<br>Y =pd.get_dummies(Y)</pre><pre>#Buidling the network<br>embed_dim = 150<br>lstm_out = 128<br>batch_size= 128</pre><pre>model = Sequential()<br>model.add(Embedding(num_words, embed_dim,input_length = maxlen))<br>model.add(Bidirectional(LSTM(lstm_out)))<br>model.add(Dense(Y.shape[1],activation=&#39;softmax&#39;))<br>adam = Adam(lr=0.001, beta_1=0.7, beta_2=0.99, epsilon=None, decay=0.0, amsgrad=False)<br>model.compile(loss = &#39;categorical_crossentropy&#39;, optimizer=adam)<br>model.summary()</pre><pre>print(&#39;fit&#39;)<br>model.fit(X, Y, batch_size =batch_size,validation_split=0.1, epochs = 5,  verbose = 1)</pre><pre>#Feature extraction<br>headlines = tokenizer.texts_to_sequences(data[&#39;headline&#39;].values)<br>headlines = pad_sequences(headlines,maxlen=maxlen)<br>inp = model.input                                          <br>outputs = [model.layers[1].output]      <br>functor = K.function([inp]+ [K.learning_phase()], outputs )<br>x = functor([headlines, 1.])[0]</pre><pre>#classifier<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>We achieved an accuracy score of <strong>72%. </strong>This is better than our baseline, but let’s see if we can make it better still.</p><h3>Skip-Thought Vectors</h3><p>In the skip-thought <a href="https://arxiv.org/abs/1506.06726">paper</a> from 2015, the authors took the same intuition from the language model. However, in skip thought, instead of predicting the next word, we are predicting the previous and the next sentence. This gives the model more context for the sentence, thus we can build better representations of sentences. For more information about the model, check out <a href="https://medium.com/@sanyamagarwal/my-thoughts-on-skip-thoughts-a3e773605efa">this</a> great blog post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLPR8O3fT0NRAvbkx0op5A.png" /><figcaption>Example from the skip thought <a href="https://arxiv.org/abs/1506.06726">paper</a></figcaption></figure><p>We will build a Seq2Seq architecture similar to the autoencoder architecture, but with two main differences. First, we will have two output layers with LSTMs — one for the previous sentence and one for the next sentence. Second, we will use teacher forcing in the output LSTMs. This means that instead of giving the output LSTM the previous hidden state only, we will also give it the real previous word (you can see an illustration of the input in the picture above, in the bottom line of the outputs).</p><pre>#Build x and y<br>num_words=2000<br>maxlen=20</pre><pre>tokenizer = Tokenizer(num_words = num_words, split=&#39; &#39;)<br>tokenizer.fit_on_texts(sentenses)<br>seqs = tokenizer.texts_to_sequences(sentenses)<br>pad_seqs = pad_sequences(seqs,maxlen)</pre><pre>x_skip = []<br>y_before = []<br>y_after = []</pre><pre>for i in tqdm(range(1,len(seqs)-1)):<br>    if len(seqs[i])&gt;4:<br>        x_skip.append(pad_seqs[i].tolist())<br>        y_before.append(pad_seqs[i-1].tolist())<br>        y_after.append(pad_seqs[i+1].tolist())</pre><pre>x_before = np.matrix([[0]+i[:-1] for i in y_before])<br>x_after =np.matrix([[0]+i[:-1] for i in y_after])<br>x_skip = np.matrix(x_skip)<br>y_before = np.matrix(y_before)<br>y_after = np.matrix(y_after)</pre><pre>#Building the model<br><br>embed_dim = 150<br>latent_dim = 128<br>batch_size = 64</pre><pre>#### Encoder Model ####<br>encoder_inputs = Input(shape=(maxlen,), name=&#39;Encoder-Input&#39;)<br>emb_layer = Embedding(num_words, embed_dim,input_length = maxlen, name=&#39;Body-Word-Embedding&#39;, mask_zero=False)<br>x = emb_layer(encoder_inputs)</pre><pre>_, state_h = GRU(latent_dim, return_state=True, name=&#39;Encoder-Last-GRU&#39;)(x)</pre><pre>encoder_model = Model(inputs=encoder_inputs, outputs=state_h, name=&#39;Encoder-Model&#39;)</pre><pre>seq2seq_encoder_out = encoder_model(encoder_inputs)</pre><pre>#### Decoder Model ####<br>decoder_inputs_before = Input(shape=(None,), name=&#39;Decoder-Input-before&#39;)  # for teacher forcing</pre><pre>dec_emb_before = emb_layer(decoder_inputs_before)</pre><pre>decoder_gru_before = GRU(latent_dim, return_state=True, return_sequences=True, name=&#39;Decoder-GRU-before&#39;)<br>decoder_gru_output_before, _ = decoder_gru_before(dec_emb_before, initial_state=seq2seq_encoder_out)</pre><pre>decoder_dense_before = Dense(num_words, activation=&#39;softmax&#39;, name=&#39;Final-Output-Dense-before&#39;)<br>decoder_outputs_before = decoder_dense_before(decoder_gru_output_before)</pre><pre>decoder_inputs_after = Input(shape=(None,), name=&#39;Decoder-Input-after&#39;)  # for teacher forcing</pre><pre>dec_emb_after = emb_layer(decoder_inputs_after)</pre><pre>decoder_gru_after = GRU(latent_dim, return_state=True, return_sequences=True, name=&#39;Decoder-GRU-after&#39;)<br>decoder_gru_output_after, _ = decoder_gru_after(dec_emb_after, initial_state=seq2seq_encoder_out)</pre><pre>decoder_dense_after = Dense(num_words, activation=&#39;softmax&#39;, name=&#39;Final-Output-Dense-after&#39;)<br>decoder_outputs_after = decoder_dense_after(decoder_gru_output_after)</pre><pre>#### Seq2Seq Model ####</pre><pre>seq2seq_Model = Model([encoder_inputs, decoder_inputs_before,decoder_inputs_after], [decoder_outputs_before,decoder_outputs_after])</pre><pre>seq2seq_Model.compile(optimizer=optimizers.Nadam(lr=0.001), loss=&#39;sparse_categorical_crossentropy&#39;)</pre><pre>seq2seq_Model.summary()</pre><pre>history = seq2seq_Model.fit([x_skip,x_before, x_after], [np.expand_dims(y_before, -1),np.expand_dims(y_after, -1)],<br>          batch_size=batch_size,<br>          epochs=10,<br>          validation_split=0.12)</pre><pre>#Feature extraction<br>headlines = tokenizer.texts_to_sequences(data[&#39;headline&#39;].values)<br>headlines = pad_sequences(headlines,maxlen=maxlen)x = encoder_model.predict(headlines)</pre><pre>#classifier<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>We achieved an accuracy score of <strong>74%. </strong>This is the best score we’ve achieved so far!</p><h3>Conclusion</h3><p>In this post, we reviewed three unsupervised methods for creating vector representations of sentences with RNNs and presented their effectiveness in solving a supervised task. The results of the autoencoder method were worse than the results for the baseline model (possibly because of the relatively small dataset used). The language and skip-thought vector models, both of which used context for predicting words or sentences, obtained the best result.</p><p>The many options available for improving the methods we’ve presented include tuning the hyperparameters, training using more epochs, using a pre-trained embedding matrix, changing the neural network architecture, and more. In theory, such advanced tuning work might change some of the results, but I believe the basic intuition of each pre-processing method can be achieved with the examples I’ve shared above.</p><p>Hope you enjoyed my post, and you’re more than welcomed to read and follow our blog at <a href="https://blog.myyellowroad.com/">YellowBlog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=104b90079a93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/unsupervised-sentence-representation-with-deep-learning-104b90079a93">Unsupervised sentence representation with deep learning</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using categorical data in machine learning with python: from dummy variables to Deep category…]]></title>
            <link>https://medium.com/@yonatan.hadar/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-42fd0a43b009?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/42fd0a43b009</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[word2vec]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Sun, 24 Sep 2017 10:49:03 GMT</pubDate>
            <atom:updated>2017-09-24T10:49:03.122Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using categorical data in machine learning with python: from dummy variables to Deep category embedding and Cat2vec -Part 2</h3><blockquote>Part 2- Advenced methods for using categorical data in machine learning</blockquote><p>In part 1 we reviewed some Basic methods for dealing with categorical data like One hot encoding and feature hashing. Both of these methods yield a very sparse and high dimentional representation of the data. In this post we will cover more advanced methods that aim to capture the diffrent categories with a much smaller dimension.</p><h3>Method 4: Encoding categories with dataset statistics</h3><p>We saw in part 1 that the RF model performed nicely in the ordinal encoding even though it is not a very good embedding method. Now we will try to give our models a numeric representation for every category with a small number of columns but with an encoding that will put similar categories close to each other. The easiest way to do it is replace every category with the number of times that we saw it in the dataset. This way if New York and New Jersey are both big cities, they will probably both appear many times in our dataset and the model will know that they are similar.</p><pre>X_train_count = copy.copy(X_train)<br>X_test_count = copy.copy(X_test)<br>X_train_count[&quot;test&quot;]=0<br>X_test_count[&quot;test&quot;]=1<br>temp_df = pandas.concat([X_train_count,X_test_count],axis=0)<br>for i in range(temp_df.shape[1]):<br>    temp_df.iloc[:,i]=temp_df.iloc[:,i].astype(&#39;category&#39;)<br>X_train_count=temp_df[temp_df[&quot;test&quot;]==0].iloc[:,:-1]<br>X_test_count=temp_df[temp_df[&quot;test&quot;]==1].iloc[:,:-1]<br>for i in range(X_train_count.shape[1]):<br>    counts = X_train_count.iloc[:,i].value_counts()<br>    counts = counts.sort_index()<br>    counts = counts.fillna(0)<br>    counts += np.random.rand(len(counts))/1000<br>    X_train_count.iloc[:,i].cat.categories = counts<br>    X_test_count.iloc[:,i].cat.categories = counts<br>                       <br>l.fit(X_train_count,y_train)<br>y_pred = l.predict_proba(X_test_count)<br>print(log_loss(y_test,y_pred))</pre><pre>r.fit(X_train_count,y_train)<br>y_pred = r.predict_proba(X_test_count)<br>print(log_loss(y_test,y_pred))</pre><p>We got results of <strong>0.44</strong> for LR and <strong>0.411</strong> for RF. This is nice but we can try it in a bit different way. Now we will replace each category with it’s CTR rate in the training set. This way we can help the model distinguish between good and bad categories. We need to be very careful with this method because if we will use the target from the test set also, this is a major <a href="https://blog.myyellowroad.com/beware-of-data-leakage-f6c307009ad9?source=collection_home---6------0-----------">data leakage</a> that will lead to good performance on the test set but poor performance on real new data.</p><pre>X_train_ctr = copy.copy(X_train)<br>X_test_ctr = copy.copy(X_test)<br>X_train_ctr[&quot;test&quot;]=0<br>X_test_ctr[&quot;test&quot;]=1<br>temp_df = pandas.concat([X_train_ctr,X_test_ctr],axis=0)<br>for i in range(temp_df.shape[1]):<br>    temp_df.iloc[:,i]=temp_df.iloc[:,i].astype(&#39;category&#39;)<br>X_train_ctr=temp_df[temp_df[&quot;test&quot;]==0].iloc[:,:-1]<br>X_test_ctr=temp_df[temp_df[&quot;test&quot;]==1].iloc[:,:-1]<br>temp_df = pandas.concat([X_train_ctr,y_train],axis=1)<br>names = list(X_train_ctr.columns.values)<br>for i in names:<br>    means = temp_df.groupby(i)[&#39;click&#39;].mean()<br>    means = means.fillna(sum(temp_df[&#39;click&#39;])/len(temp_df[&#39;click&#39;]))<br>    means += np.random.rand(len(means))/1000<br>    X_train_ctr[i].cat.categories = means<br>    X_test_ctr[i].cat.categories = means<br>                  <br>l.fit(X_train_ctr,y_train)<br>y_pred = l.predict_proba(X_test_ctr)<br>print(log_loss(y_test,y_pred))</pre><pre>r.fit(X_train_ctr,y_train)<br>y_pred = r.predict_proba(X_test_ctr)<br>print(log_loss(y_test,y_pred))</pre><p>With this method we got a logloss of <strong>0.443</strong> with LR and <strong>0.422</strong> with RF which is worst from the ordinal encoding. Another good way of switching categories to real numbers is using some domain knowledge for example replacing a country category with the countries GDP or replacing a URL category to it’s Similar web or Alexa rank are both good strategies that will probably help your model to generalize.</p><h3>Method 5: Cat2Vec</h3><p>Problems in NLP usually have the same issues that categorical data has and Bag of words (A very common text representation) is pretty much the same as one hot encoding. NLP researchers has developed a very cool method called <a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/">Word2Vec</a> (or word embedding) in order to deal with the problems of bag of words. In Word2Vec we represent each word as a vector in a way that terms that are semantically close will be close in the vector space, Moreover, we can apply linear arithmetic for example in Word2vec space King-Man+Women= Queen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXNXYfAqfLUeiDXPCo130w.png" /></figure><p>We can say that every raw in our dataset is like a sentence and that every category is a word, shuffle the categories and maybe we will find that Ebay -US+Chaina = Aliexpress. This will probably won’t happen but hopefully we’ll get a representation that will help our models to generalize between categories.</p><pre>from gensim.models.word2vec import Word2Vec<br>from random import shuffle<br>size=6<br>window=8<br>x_w2v = copy.deepcopy(data.iloc[:,features])<br>names = list(x_w2v.columns.values)<br>for i in names:<br>    x_w2v[i]=x_w2v[i].astype(&#39;category&#39;)<br>    x_w2v[i].cat.categories = [&quot;Feature %s %s&quot; % (i,g) for g in x_w2v[i].cat.categories]<br>x_w2v = x_w2v.values.tolist()<br>for i in x_w2v:<br>    shuffle(i)<br>w2v = Word2Vec(x_w2v,size=size,window=window)</pre><pre>X_train_w2v = copy.copy(X_train)<br>X_test_w2v = copy.copy(X_test)<br>for i in names:<br>    X_train_w2v[i]=X_train_w2v[i].astype(&#39;category&#39;)<br>    X_train_w2v[i].cat.categories = [&quot;Feature %s %s&quot; % (i,g) for g in X_train_w2v[i].cat.categories]<br>for i in names:<br>    X_test_w2v[i]=X_test_w2v[i].astype(&#39;category&#39;)<br>    X_test_w2v[i].cat.categories = [&quot;Feature %s %s&quot; % (i,g) for g in X_test_w2v[i].cat.categories]<br>X_train_w2v = X_train_w2v.values<br>X_test_w2v = X_test_w2v.values<br>x_w2v_train = np.random.random((len(X_train_w2v),size*X_train_w2v.shape[1]))<br>for j in range(X_train_w2v.shape[1]):<br>    for i in range(X_train_w2v.shape[0]):<br>        if X_train_w2v[i,j] in w2v:<br>            x_w2v_train[i,j*size:(j+1)*size] = w2v[X_train_w2v[i,j]]</pre><pre>x_w2v_test = np.random.random((len(X_test_w2v),size*X_test_w2v.shape[1]))<br>for j in range(X_test_w2v.shape[1]):<br>    for i in range(X_test_w2v.shape[0]):<br>        if X_test_w2v[i,j] in w2v:<br>            x_w2v_test[i,j*size:(j+1)*size] = w2v[X_test_w2v[i,j]]</pre><pre>l.fit(x_w2v_train,y_train)<br>y_pred = l.predict_proba(x_w2v_test)<br>print(log_loss(y_test,y_pred))</pre><pre>r.fit(x_w2v_train,y_train)<br>y_pred = r.predict_proba(x_w2v_test)<br>print(log_loss(y_test,y_pred))</pre><p>With Cat2Vec we get a logloss of <strong>0.417</strong> for LR and logloss of<strong> 0.397</strong> for RF. This is very cool because we used an unsupervised method that was designed for a different problem and managed to use it with our problem. The Cat2vec has some weird behavior, if we plot the categories we will find that for example the most close category to New York is Cnn.com which is weird because we want that values from the same category will be closer to each other than values in other categories. This pitfall in our method is caused by the way that we train the Cat2Vec that assumes that all categories are similar like words that are all similar.</p><h3>Method 6: Category embedding with deep learning</h3><p>In our last method we will deal with the pitfall of Cat2vec and separately build a vector embedding to every category type. We will do it with an embedding layer in a deep neural network. Embedding layers are used to convert one hot encoded variables into vector representation but as opposed to Word2vec the layers mission is not creating a semantical embedding but it is creating an embedding that will help our prediction goal. We will use the DNN described in the paper <a href="https://arxiv.org/abs/1604.06737">Entity Embeddings of Categorical Variables</a>. After training the DNN we will use it’s category embedding as input for the LR and RF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*fwNpdhYLc0awJyaPlt3SPw.png" /><figcaption>Dnn architecture from <a href="https://arxiv.org/abs/1604.06737">Entity Embeddings of Categorical Variables</a>. The DNN is composed of a one hot encoded input layer followed by an embedding layer for each categorical feature followed by two fully connected layers and a softmax layer.</figcaption></figure><pre>from keras.models import Sequential<br>from keras.layers.core import Dense, Activation, Reshape<br>from keras.layers import Merge<br>from keras.layers.embeddings import Embedding<br>from keras.callbacks import ModelCheckpoint<br>import h5py</pre><pre>X_train_dnn = copy.copy(X_train).values<br>X_test_dnn = copy.copy(X_test).values<br>    <br>les = []<br>for i in range(X_train_dnn.shape[1]):<br>    le = preprocessing.LabelEncoder()<br>    le.fit(data.iloc[:,features].iloc[:, i])<br>    les.append(le)<br>    X_train_dnn[:, i] = le.transform(X_train_dnn[:, i])<br>    X_test_dnn[:, i] = le.transform(X_test_dnn[:, i])<br>    <br>def split_features(X):<br>    X_list = []</pre><pre>C1 = X[..., [0]]<br>    X_list.append(C1)</pre><pre>banner_pos = X[..., [1]]<br>    X_list.append(banner_pos)</pre><pre>site_id = X[..., [2]]<br>    X_list.append(site_id)<br>    <br>    site_domain = X[..., [3]]<br>    X_list.append(site_domain)</pre><pre>site_category = X[..., [4]]<br>    X_list.append(site_category)</pre><pre>app_id = X[..., [5]]<br>    X_list.append(app_id)</pre><pre>app_domain = X[..., [6]]<br>    X_list.append(app_domain)</pre><pre>app_category = X[..., [7]]<br>    X_list.append(app_category)<br>    <br>    device_id = X[..., [8]]<br>    X_list.append(device_id)</pre><pre>device_model = X[..., [9]]<br>    X_list.append(device_model)</pre><pre>device_type = X[..., [10]]<br>    X_list.append(device_type)<br>    <br>    device_conn_type = X[..., [11]]<br>    X_list.append(device_conn_type)</pre><pre>C14 = X[..., [12]]<br>    X_list.append(C14)<br>    <br>    C15 = X[..., [13]]<br>    X_list.append(C15)</pre><pre>C16 = X[..., [14]]<br>    X_list.append(C16)</pre><pre>C17 = X[..., [15]]<br>    X_list.append(C17)</pre><pre>C18 = X[..., [16]]<br>    X_list.append(C18)</pre><pre>C19 = X[..., [17]]<br>    X_list.append(C19)<br>    <br>    C20 = X[..., [18]]<br>    X_list.append(C20)</pre><pre>C21 = X[..., [19]]<br>    X_list.append(C21)</pre><pre>return X_list</pre><pre>class NN_with_EntityEmbedding(object):</pre><pre>def __init__(self, X_train, y_train, X_val, y_val):<br>        self.nb_epoch = 10<br>        self.__build_keras_model()<br>        self.fit(X_train, y_train, X_val, y_val)</pre><pre>def preprocessing(self, X):<br>        X_list = split_features(X)<br>        return X_list</pre><pre>def __build_keras_model(self):<br>        models = []</pre><pre>model_C1= Sequential()<br>        model_C1.add(Embedding(len(les[0].classes_), 3, input_length=1))<br>        model_C1.add(Reshape(target_shape=(3,)))<br>        models.append(model_C1)</pre><pre>model_banner_pos = Sequential()<br>        model_banner_pos.add(Embedding(len(les[1].classes_), 3, input_length=1))<br>        model_banner_pos.add(Reshape(target_shape=(3,)))<br>        models.append(model_banner_pos)<br>        <br>        model_site_id = Sequential()<br>        model_site_id.add(Embedding(len(les[2].classes_), 8, input_length=1))<br>        model_site_id.add(Reshape(target_shape=(8,)))<br>        models.append(model_site_id)<br>        <br>        site_domain = Sequential()<br>        site_domain.add(Embedding(len(les[3].classes_), 8, input_length=1))<br>        site_domain.add(Reshape(target_shape=(8,)))<br>        models.append(site_domain)</pre><pre>site_category = Sequential()<br>        site_category.add(Embedding(len(les[4].classes_), 3, input_length=1))<br>        site_category.add(Reshape(target_shape=(3,)))<br>        models.append(site_category)</pre><pre>app_id = Sequential()<br>        app_id.add(Embedding(len(les[5].classes_), 8, input_length=1))<br>        app_id.add(Reshape(target_shape=(8,)))<br>        models.append(app_id)</pre><pre>app_domain = Sequential()<br>        app_domain.add(Embedding(len(les[6].classes_), 4, input_length=1))<br>        app_domain.add(Reshape(target_shape=(4,)))<br>        models.append(app_domain)<br>        <br>        app_category = Sequential()<br>        app_category.add(Embedding(len(les[7].classes_), 3, input_length=1))<br>        app_category.add(Reshape(target_shape=(3,)))<br>        models.append(app_category)<br>        <br>        device_id = Sequential()<br>        device_id.add(Embedding(len(les[8].classes_), 10, input_length=1))<br>        device_id.add(Reshape(target_shape=(10,)))<br>        models.append(device_id)<br>        <br>        device_model = Sequential()<br>        device_model.add(Embedding(len(les[9].classes_), 8, input_length=1))<br>        device_model.add(Reshape(target_shape=(8,)))<br>        models.append(device_model)<br>        <br>        device_type = Sequential()<br>        device_type.add(Embedding(len(les[10].classes_), 2, input_length=1))<br>        device_type.add(Reshape(target_shape=(2,)))<br>        models.append(device_type)<br>        <br>        device_conn_type = Sequential()<br>        device_conn_type.add(Embedding(len(les[11].classes_), 2, input_length=1))<br>        device_conn_type.add(Reshape(target_shape=(2,)))<br>        models.append(device_conn_type)</pre><pre>C14 = Sequential()<br>        C14.add(Embedding(len(les[12].classes_), 8, input_length=1))<br>        C14.add(Reshape(target_shape=(8,)))<br>        models.append(C14)<br>        <br>        C15 = Sequential()<br>        C15.add(Embedding(len(les[13].classes_), 3, input_length=1))<br>        C15.add(Reshape(target_shape=(3,)))<br>        models.append(C15)<br>        <br>        C16 = Sequential()<br>        C16.add(Embedding(len(les[14].classes_), 3, input_length=1))<br>        C16.add(Reshape(target_shape=(3,)))<br>        models.append(C16)<br>        <br>        C17 = Sequential()<br>        C17.add(Embedding(len(les[15].classes_), 4, input_length=1))<br>        C17.add(Reshape(target_shape=(4,)))<br>        models.append(C17)<br>        <br>        C18 = Sequential()<br>        C18.add(Embedding(len(les[16].classes_), 2, input_length=1))<br>        C18.add(Reshape(target_shape=(2,)))<br>        models.append(C18)<br>        <br>        C19 = Sequential()<br>        C19.add(Embedding(len(les[17].classes_), 4, input_length=1))<br>        C19.add(Reshape(target_shape=(4,)))<br>        models.append(C19)<br>        <br>        C20 = Sequential()<br>        C20.add(Embedding(len(les[18].classes_), 5, input_length=1))<br>        C20.add(Reshape(target_shape=(5,)))<br>        models.append(C20)<br>        <br>        C21 = Sequential()<br>        C21.add(Embedding(len(les[19].classes_), 4, input_length=1))<br>        C21.add(Reshape(target_shape=(4,)))<br>        models.append(C21)</pre><pre>self.model = Sequential()<br>        self.model.add(Merge(models, mode=&#39;concat&#39;))<br>        self.model.add(Dense(150, kernel_initializer=&#39;uniform&#39;))<br>        self.model.add(Activation(&#39;relu&#39;))<br>        self.model.add(Dense(250, kernel_initializer=&#39;uniform&#39;))<br>        self.model.add(Activation(&#39;relu&#39;))<br>        self.model.add(Dense(1))<br>        self.model.add(Activation(&#39;sigmoid&#39;))</pre><pre>self.model.compile(loss=&#39;binary_crossentropy&#39;,<br>              optimizer=&#39;adam&#39;,<br>             metrics=[&#39;acc&#39;])<br>        <br>    def fit(self, X_train, y_train, X_val, y_val):<br>        self.model.fit(self.preprocessing(X_train), y_train,<br>                       validation_data=(self.preprocessing(X_val), y_val),<br>                       epochs=self.nb_epoch, batch_size=128,<br>                       )</pre><pre>dnn = NN_with_EntityEmbedding(X_train_dnn, y_train, X_test_dnn, y_test)   <br>weights = dnn.model.get_weights()</pre><pre>n = 0<br>for i in range(0,40,2):<br>    n+=(weights[i][0].shape)[1]<br>    <br>x_dnn_train = np.random.random((len(X_train_dnn),n))<br>start_ind=0<br>for j in range(X_train_dnn.shape[1]):<br>    mat = weights[j*2][0]<br>    dim = mat.shape[1]<br>    for i in range(X_train_dnn.shape[0]):<br>        x_dnn_train[i,start_ind:start_ind+dim]=mat[X_train_dnn[i,j]]<br>    start_ind += dim</pre><pre>x_dnn_test = np.random.random((len(X_test_dnn),n))<br>start_ind=0<br>for j in range(X_test_dnn.shape[1]):<br>    mat = weights[j*2][0]<br>    dim = mat.shape[1]<br>    for i in range(x_dnn_test.shape[0]):<br>        x_dnn_test[i,start_ind:start_ind+dim]=mat[X_test_dnn[i,j]]<br>    start_ind += dim</pre><pre>l.fit(x_dnn_train,y_train)<br>y_pred = l.predict_proba(x_dnn_test)<br>print(log_loss(y_test,y_pred))</pre><pre>r.fit(x_dnn_train,y_train)<br>y_pred = r.predict_proba(x_dnn_test)<br>print(log_loss(y_test,y_pred))</pre><p>This approach got us to a logloss of <strong>0.436</strong> for LR and <strong>0.429</strong> for RF. Category embedding is usually a good method for dealing with categorical data but it needs a lot of labeled data which is usually very expensive and a long training time.</p><h3>Conclusions</h3><p>In this post we reviewed 6 methods for dealing with categorical data from very simple methods like ordering encoding or one hot encoding to complicated methods like Category embedding and Cat2vec and tried to explain each methods pros and cons . In the dataset that we tested on, the best methods were One hot encoding with rare values and Cat2Vec but it will probably change a little with more parameter tuning of the models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*3CWflYvdulrFhRQELU2u9g.png" /><figcaption>Methods comparison</figcaption></figure><p>We have to say that from our experience diffrent methods will work better or worse on diffrent datasets, for us methods 3,4 and 6 usually gives the best results but the other methods can suprise too depends on the type of data that you are trying to model (as we saw in this example).</p><p>Hope you enjoyed my post and you are more than welcomed to read our other posts on <a href="https://blog.myyellowroad.com/">YellowBlog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42fd0a43b009" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using categorical data in machine learning with python: from dummy variables to Deep category…]]></title>
            <link>https://medium.com/@yonatan.hadar/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-66041f734512?source=rss-ed91e559149------2</link>
            <guid isPermaLink="false">https://medium.com/p/66041f734512</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Tue, 19 Sep 2017 09:01:17 GMT</pubDate>
            <atom:updated>2019-04-28T08:36:49.517Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using categorical data in machine learning with python</h3><blockquote>From dummy variables to Deep category embedding and Cat2vec — Part 1 (Basic Methods)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*I4zVi0P_jUaq2FmYvMQg1Q.jpeg" /></figure><blockquote>Intro</blockquote><p>Categorical data is very common in business datasets. For example, users are typically described by country, gender, age group etc., products are often described by product type, manufacturer, seller etc., and so on.</p><p>Categorical data is very convenient for people but very hard for most machine learning algorithms, due to several reasons:</p><ol><li>High cardinality- categorical variables may have a very large number of levels (e.g., city or URL), were most of the levels appear in a relatively small number of instances.</li><li>Many machine learning models (e.g., SVM) are algebraic, thus their input must be numerical. Using these models, categories must be transformed into numbers first before we can apply the learning algorithm.</li><li>While some ML packages might transform categorical data to numeric automatically based on some default embedding method, many other ML packages don’t support such inputs (like our beloved scikit-learn).</li><li>For the machine (and we are in <strong>machine</strong> learning :)), categorical data doesn’t contain the same context or information that we humans can easily associate and understand. For example, when looking on a feature called “City”, we humans can understand that for many business aspects New York is a similar concept to New Jersey, while New York and Tehran are much different. But for the machine, New York, New Jersey and Tehran, are just three different levels (possible values) of the same feature “City”. If we won’t represent the additional contextual information it will be impossible for the machine to generalize out of this information.</li></ol><p>In this post I will share some basic strategies of using categorical data that worked for us (at <a href="https://medium.com/u/2a6e2e6fd40f">YellowRoad</a>) in recent projects, while on <a href="https://blog.myyellowroad.com/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-42fd0a43b009">part 2</a>, I will share some more advanced methods. I will discuss on why they work and why different methods are better for different algorithms in different scenarios, while sharing my code implementation in Python.</p><p>We will cover different methods, from a simple dummy variables to complex methods like leveraging deep learning for category embedding.</p><blockquote>Evaluation</blockquote><p>We will evaluate every method on a sample of 2M rows from the Avatzo CTR prediction Kaggle challenge <a href="https://www.kaggle.com/c/avazu-ctr-prediction">dataset</a> that has many categorical features. Our evaluation metric will be <a href="https://www.kaggle.com/wiki/LogLoss">logarithmic loss</a> (like in the contest). On every feature representation I will apply Logistic Regression and Random Forest algorithms to evaluate it’s performance. I will review 6 different embedding strategies in a series of two posts (the more advanced methods will be in a following post).</p><p>As with any ML solution, we will start by creating a simple baseline method, and assess it’s performance. For that, our baseline prediction will be to predict a constant CTR based on the mean CTR proportion that was seen in the training set:</p><pre>import numpy as np<br>import pandas<br>from sklearn.metrics import log_loss<br>train_file = &quot;train.csv&quot;<br>train = pandas.read_csv(train_file)<br>msk = np.random.rand(len(train)) &lt; 0.8<br>features = [3,4,5,6,7,8,9,10,11,13,14,15,16,17,18,19,20,21,22,23]<br>X_train = train[msk].iloc[:,features]<br>X_test = train[~msk].iloc[:,features]<br>y_train = train[msk].iloc[:,1]<br>y_test = train[~msk].iloc[:,1]<br>print(log_loss(y_test,np.ones(len(y_test))*y_train.mean()))</pre><p>We get a log-loss of <strong>0.457</strong>.</p><blockquote>Embedding Methods</blockquote><h3>Method 1: Encoding to ordinal variables</h3><p>Our first method is changing every categorical feature to an ordinal one. The order will be selected randomly (for example, like the order in the dataset or in an alphabetical order). This method does not make much sense because if we are encoding New York as 1, Tehran as 2 and New Jersey as 3, our algorithm will assume that Tehran is more similar to New York than New Jersey. Let’s try this method, anyway:</p><pre>from sklearn import preprocessing<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.linear_model import LogisticRegression<br>X_train_ordinal = X_train.values<br>X_test_ordinal = X_test.values<br>les = []<br>l = LogisticRegression()<br>r = RandomForestClassifier(n_estimators=25,max_depth=10)<br>for i in range(X_train_ordinal.shape[1]):<br>    le = preprocessing.LabelEncoder()<br>    le.fit(X_train_ordinal.iloc[:,features].iloc[:, i])<br>    les.append(le)<br>    X_train_ordinal[:, i] = le.transform(X_train_ordinal[:, i])<br>    X_test_ordinal[:, i] = le.transform(X_test_ordinal[:, i])<br>l.fit(X_train_ordinal,y_train)<br>y_pred = l.predict_proba(X_test_ordinal)<br>print(log_loss(y_test,y_pred))<br>r.fit(X_train_ordinal,y_train)<br>y_pred = r.predict_proba(X_test_ordinal)<br>print(log_loss(y_test,y_pred))</pre><p>Our logistic regression got a logloss almost similar to the baseline (<strong>0.450</strong>) but the random forest made a nice improvement to <strong>0.422</strong>. how is it possible? Where in logistic regression we apply a coefficient to each explaining feature (most of which contain merely noise, due to the nonsense encoding), random noise includes an inherent feature selection mechanism. Most probably, random forest selected these features in which the encoding somehow correlated with the CTR, and used mainly these features to explain the target.</p><h3>Method 2: One hot encoding (or dummy variabales)</h3><p>Our second method is encoding each category as a one hot encoding (OHE) vector (or dummy variables). OHE is a representation method that takes each category value and turns it into a binary vector of size |i|(number of values in category i) where all columns are equal to zero besides the category column. Here is a little example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*WXpoiS7HXRC-uwJPYsy1Dg.png" /></figure><p>OHE is probably the most common method for dealing with categorical data and is also very common in NLP (Bag of word models). Let us run it.</p><pre>from sklearn.preprocessing import OneHotEncoder<br>enc = OneHotEncoder(handle_unknown=&#39;ignore&#39;)<br>enc.fit(X_train_ordinal)<br>X_train_one_hot = enc.transform(X_train_ordinal)<br>X_test_one_hot = enc.transform(X_test_ordinal)<br>l.fit(X_train_one_hot,y_train)<br>y_pred = l.predict_proba(X_test_one_hot)<br>print(log_loss(y_test,y_pred))<br>r.fit(X_train_one_hot,y_train)<br>y_pred = r.predict_proba(X_test_one_hot)<br>print(log_loss(y_test,y_pred))<br>print(X_train_one_hot.shape)</pre><p>We got a <strong>0.398</strong> logloss with logistic regression and <strong>0.445</strong> with random forest. This is a nice result but why is logistic regression better than random forest? In this method we represented our data with a huge amount of features (<a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">curse of dimensionality</a>). Having many features we need a very simple classifier in order to not overfit the data. Logistic regression is way more simple than random forest. Moreover, the sparsity of the data makes it very hard for the random forest to find good splits that will help in separating the classes.</p><p>OHE has some significant shortcomings:</p><ol><li>OHE representation produces very high dimensionality, this causes an increase in the model’s training and serving time and memory consumption.</li><li>OHE can easily cause a model to overfit the data.</li><li>OHE can’t handle categories that weren’t in the training data (like new URLs, new device types etc), this can be problematic in domains that change all the time.</li></ol><p>We can overcome some of this disadvantages by encoding all rare categories to the same features (“rare value”). This method can reduce the dimensionality drastically in some datasets with a small decrease in performance (or even an increase). Let us try it.</p><pre>import copy<br>X_train_rare = copy.copy(X_train)<br>X_test_rare = copy.copy(X_test)<br>X_train_rare[&quot;test&quot;]=0<br>X_test_rare[&quot;test&quot;]=1<br>temp_df = pandas.concat([X_train_rare,X_test_rare],axis=0)<br>names = list(X_train_rare.columns.values)<br>temp_df = pandas.concat([X_train_rare,X_test_rare],axis=0)<br>for i in names:<br>    temp_df.loc[temp_df[i].value_counts()[temp_df[i]].values &lt; 20, i] = &quot;RARE_VALUE&quot;<br>for i in range(temp_df.shape[1]):<br>    temp_df.iloc[:,i]=temp_df.iloc[:,i].astype(&#39;str&#39;)<br>X_train_rare = temp_df[temp_df[&quot;test&quot;]==&quot;0&quot;].iloc[:,:-1].values<br>X_test_rare = temp_df[temp_df[&quot;test&quot;]==&quot;1&quot;].iloc[:,:-1].values<br>for i in range(X_train_rare.shape[1]):<br>    le = preprocessing.LabelEncoder()<br>    le.fit(temp_df.iloc[:,:-1].iloc[:, i])<br>    les.append(le)<br>    X_train_rare[:, i] = le.transform(X_train_rare[:, i])<br>    X_test_rare[:, i] = le.transform(X_test_rare[:, i])<br>enc.fit(X_train_rare)<br>X_train_rare = enc.transform(X_train_rare)<br>X_test_rare = enc.transform(X_test_rare)<br>l.fit(X_train_rare,y_train)<br>y_pred = l.predict_proba(X_test_rare)<br>print(log_loss(y_test,y_pred))<br>r.fit(X_train_rare,y_train)<br>y_pred = r.predict_proba(X_test_rare)<br>print(log_loss(y_test,y_pred))<br>print(X_train_rare.shape)</pre><p>We got almost the same log lost (<strong>0.399</strong> for logistic regression and<strong> 0.433</strong> for random forest) but our number of features has decreased dramatically from ~270,000 with OHE to ~8,000!</p><h3>Method 3: Feature hashing (a.k.a the hashing trick)</h3><p>Feature hashing is a very cool technique to represent categories in a “one hot encoding style” as a sparse matrix but with a much lower dimensions. In feature hashing we apply a hashing function to the category and then represent it by its indices. for example, if we choose a dimension of 5 to represent “New York” we will calculate H(New York) mod 5 = 3 (for example) so New York representation will be (0,0,1,0,0). This technique became very common and was used in the winning solution for the Avatzo CTR prediction Kaggle challenge which we are now trying to solve.</p><p>Feature hashing has some major pros: It is low dimensional thus it is very efficient in processing time and memory, it can be computed with online learning because as opposed to one hot encoding we don’t need to go over all the data and build a dictionary of all possible categories and their mapping and it is not affected by new kinds of categories.</p><p>But it also have some cons. As we know, hashing functions sometimes have collision so if H(New York) = H(Tehran) the model can’t know what city were in the data. There are some sophisticated hashing function that try to reduce the number of collision but anyway, studies have shown that collisions usually doesn’t affect significantly on the models performance. Second shortcoming is that hashed features are not interpretable so doing things like feature importance and model debugging is very hard.</p><p>Lets implement the hashing trick:</p><pre>from sklearn.feature_extraction import FeatureHasher<br>X_train_hash = copy.copy(X_train)<br>X_test_hash = copy.copy(X_test)<br>for i in range(X_train_hash.shape[1]):<br>    X_train_hash.iloc[:,i]=X_train_hash.iloc[:,i].astype(&#39;str&#39;)<br>for i in range(X_test_hash.shape[1]):<br>    X_test_hash.iloc[:,i]=X_test_hash.iloc[:,i].astype(&#39;str&#39;)<br>h = FeatureHasher(n_features=100,input_type=&quot;string&quot;)<br>X_train_hash = h.transform(X_train_hash.values)<br>X_test_hash = h.transform(X_test_hash.values)</pre><pre>l.fit(X_train_hash,y_train)<br>y_pred = l.predict_proba(X_test_hash)<br>print(log_loss(y_test,y_pred))#0.4</pre><pre>r.fit(X_train_hash,y_train)<br>y_pred = r.predict_proba(X_test_hash)<br>print(log_loss(y_test,y_pred))</pre><p>We got a result of <strong>0.411</strong> for LR and <strong>0.426</strong> for RF with a much smaller dimension than one hot encoding of 1000 and <strong>0.402 </strong>for LR and<strong> 0.435 </strong>for RF with a dimension of 10,000. The reason for the difference between the two classifiers is probably that RF usually doesn’t work well with sparse data.</p><h3>Summary</h3><p>So far we tried 3 Basic methods for using categorical data in our machine learning models. On our dataset the One hot encoding with Logistic regression gave the best performance but due to it’s high Dimensionality, One hot encoding with Rare values is probably the best option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*EhZEQV0PQfXWmCyDJdnqZg.png" /></figure><p><a href="https://blog.myyellowroad.com/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-42fd0a43b009">On our next post</a> we will cover more advanced methods like using Cat2Vec and Category embedding with deep neural networks.</p><p>Hope you enjoyed my post and you are more than welcomed to read and follow our blog at <a href="https://blog.myyellowroad.com/">YellowBlog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66041f734512" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>