<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Francesco Tuscolano on Medium]]></title>
        <description><![CDATA[Stories by Francesco Tuscolano on Medium]]></description>
        <link>https://medium.com/@francescotuscolano?source=rss-b1e0b67d7885------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DlRaOJWg56xmgVsitoK6qA@2x.jpeg</url>
            <title>Stories by Francesco Tuscolano on Medium</title>
            <link>https://medium.com/@francescotuscolano?source=rss-b1e0b67d7885------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:23:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@francescotuscolano/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Say hi to Chi-square (χ²) for Optimal Topic Modeling]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/say-hi-to-chi-square-%CF%87%C2%B2-for-optimal-topic-modeling-1cf1d55dc7fa?source=rss-b1e0b67d7885------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cf1d55dc7fa</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[theory]]></category>
            <category><![CDATA[knime]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Francesco Tuscolano]]></dc:creator>
            <pubDate>Wed, 10 Apr 2024 05:31:15 GMT</pubDate>
            <atom:updated>2024-04-10T08:07:06.652Z</atom:updated>
            <content:encoded><![CDATA[<h4>DATA SCIENCE THEORY | TOPIC MODELING | KNIME ANALYTICS PLATFORM</h4><h4><em>Unlocking textual complexity using a no-code approach with KNIME</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d0hXHsWlZHwGNZ2a" /><figcaption><em>Photo by</em><a href="https://unsplash.com/@ninjason?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"><em> Jason Leung</em></a><em> on</em><a href="https://unsplash.com/photos/wooden-bookcase-filled-with-books-D4YrzSwyIEc?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"><em> Unsplash</em></a><em>.</em></figcaption></figure><p>In the realm of natural language processing (NLP), extracting meaningful insights from extensive textual datasets remains a challenge. Latent Dirichlet Allocation (LDA) and Structural Topic Modeling (STM) offer powerful solutions for understanding and simplifying the content of a collection of documents. However, those algorithms are influenced by a few parameters. One of particular significance is the number of topics that the researcher wants to extract, which must be predetermined.</p><p>Indeed, the essence of topic modeling lies in the ability to cluster complex textual data into coherent and interpretable themes, yet the determination of the optimal number of topics is a challenging task.</p><p>To tackle it, researchers have devised a range of indicators, each carrying its own set of strengths and weaknesses. In Blei’s seminal work (2003), the authors introduced the perplexity index, which is one among the prominent methods for evaluating topic model performance. The indicator measures essentially how well the topic model can predict unseen topics. However, while perplexity serves as a common measure in topic modeling, it is important to note its limitations. Perplexity does not directly measure accuracy; hence, a model may exhibit low perplexity yet high error rates. In essence, a model’s confidence in its predictions does not necessarily guarantee their correctness. Moreover, comparing perplexity across datasets poses challenges due to the unique word distributions inherent in each dataset, compounded by variations in model parameters. Consequently, directly comparing the performance of models trained on different datasets becomes arduous.</p><blockquote>In this article, we will explore a methodology that involves computing Chi-square statistics (χ²) to measure the <strong>goodness of fit of a topic model</strong>. This method addresses the limitations mentioned earlier.</blockquote><p>All along, we’ll rely on a low-code approach using the <a href="https://hub.knime.com/-/spaces/-/~nvkqJquBaLc9agoa/current-state/">Topic Model Goodness of Fit (χ²)</a> component in <a href="https://www.knime.com/knime-analytics-platform?utm_source=medium&amp;utm_medium=organic&amp;utm_term=Roberto&amp;utm_content=blog&amp;utm_campaign=com_journal">KNIME Analytics Platform</a>.</p><p>Let’s get started!</p><h3>Recap: LDA Basic Setup</h3><p>Topic models indeed function akin to techniques for clustering. They uncover the latent structure of the words within a collection of documents by identifying clusters that constitute topics.</p><p>More formally, we define a corpus and a document respectively as collections of documents and words:</p><p><em>Corpus = {document₁, document₂, document₃, documentₙ}</em></p><p><em>documentₙ = {word₁, word₂, word₃, wordⱼ}</em></p><p>The content of the corpus is summarized by the estimation of two matrices:</p><ul><li>The<em> </em><strong>Document-Topic Weights (DTW) </strong>matrix<strong> </strong>shows topics distribution across documents;</li><li>The <strong>Topic-Word Weights (TWW) </strong>matrix assigns weights to words within each topic, aiding human comprehension.</li></ul><p><em>DTW </em>displays the topic presence across documents, aiding quantitative exploration. This is achieved by adding <em>N</em> columns (topicₙ) that represent the probabilities of each document discussing that particular topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kC-oGAeOKjNGTNMs" /><figcaption>Document-Topic Weights (DTW).</figcaption></figure><p><em>TWW</em> enhances topic interpretation by assigning a weight to words and the topics, facilitating qualitative insights extraction. The relationship between words and topics is not exclusive; the same words might be assigned to different topics. We can determine the conditional probability of a word, represented by wordₚ , belonging to topic zₚ, using the softmax function to transform the weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hxk-Ul50SEPTtI5L" /><figcaption>Topic-Word Weights (TWW).</figcaption></figure><p>In the figure above, you can observe that the model does not directly provide the probabilities of occurrence for the words within each document. This information needs to be derived by performing a pairwise multiplication between the DTW and DTM. By doing that, we can estimate the <em>word vectors</em> according to the LDA:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vOeo2CGC6LQ5AKJW" /></figure><p>where Pr (wₚ| zₚ=k) is the probability of observing wordₚ given the presence of topic zₚ , and Pr (zₚ=k) is the probability of observing the topic zₚ, in documentⱼ.</p><p>This equation enables the quantification of the word relative frequencies following the generative process of the topic model, as Pr (wₚ| zₚ=k) can be easily derived by means of <em>TWW </em>and Pr ( zₚ=k) are the probabilities in the <em>DTW</em>.</p><h3>Why Chi-square (χ²) statistics?</h3><p>Determining the goodness of fit of a topic model involves establishing a theoretical framework to address the question: what constitutes a proficient representation of a corpus of documents by a topic model?</p><blockquote>Lewis, C. M., &amp; Grossetti, F. in “<a href="https://www.jmlr.org/papers/volume23/19-297/19-297.pdf">A Statistical Approach for Optimal Topic Model Identification</a>” (Journal of Machine Learning, 23(58), 1−20, 2022), introduced a series of parametric tests designed to assess the fit of a topic model to a given corpus. The first test indicates that a <em>k</em>-topic model accurately describes a collection of documents if the actual word frequencies (what we see in the documents) and the estimated word frequencies (calculated by means of the topic model output by the formula above) are very close.</blockquote><p>To quantify this discrepancy and assess whether the word vectors align, they devised a Chi-Square Statistics that utilizes both observed word vectors and those estimated by the Latent Dirichlet Allocation (LDA) model. Indeed, this traditional statistic serves as a similarity measure. The authors also developed an <a href="https://github.com/contefranz/OpTop">R package</a>, which is not on CRAN yet.</p><p>Remember that a Chi-Square, in its easiest formulation, is calculated as the squared difference between the expected values and the observed values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*-oxs600XrIpeEGwi" /></figure><p>Based on this principles, the authors derived a measure that represents how well the topic model represents the corpus of documents:</p><p>The “Optop” statistic measures the goodness of fit of a topic model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nytvLoEhey8tB2Be" /></figure><p>At first glance, this formula may appear daunting, but its purpose is straightforward: it computes a Chi-Square distance between the estimated word vectors and the observed relative word frequencies. Once we have computed the estimated word vectors from the LDA (the most challenging part), and the relative word frequencies (much simpler), we have all the elements needed to apply the formula to our dataset. Now, let’s delve into the equation.</p><p>The formula consists of three terms, which are computed at document level and summed up at the end at the corpus level:</p><ul><li><strong>The Important Words Component (Blue Frame)</strong>: This term represents the Chi-Squared difference between the most relevant estimated word vector and the observed word vector for document ⱼ.</li><li><strong>The Unimportant Word Component (Red Frame)</strong>: Here, the Chi-Square distance is calculated for all aggregate words that are not particularly relevant in document ⱼ.</li><li>The term “Pⱼ + 1” signifies the <strong>degrees of freedom</strong>, where “Pⱼ” represents the count of relatively important words in document ⱼ.</li></ul><p>As you may guess, to reduce the noise, less significant words are grouped into a single bin, referred to as the “uninformative component.” This component is populated when the cumulative probability distribution of the words in document ⱼ surpasses 0.8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E42LVGMm22Hw8qUs" /><figcaption>Understanding the inner workings involved in computing the OpTop statistic.</figcaption></figure><p>Summing up the χ² values by document, we obtain a measure of goodness of fit for each documentⱼ.</p><p>Furthermore, if we sum all the χ² at a corpus level, we get a unique and comprehensive measure on how well the corpus is represented by that topic model.</p><p>A lower Chi-Square value indicates a better fit, as it signifies a smaller distance between the observed and estimated word vectors, indicating that the LDA accurately represents the documents (χ² is lower-bounded to 0). However, to determine the ideal number of topics, a hypothesis test would be a more effective strategy, which we will discuss in the section below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WBU7I1YsZzPvELrq" /></figure><h3>Testing Chi-square (χ²) on real data</h3><p>To test the goodness of Chi-square statistics, we need to think of the process as a typical hyperparameter optimization in machine learning.</p><p>This means that we will loop over the number of topics we wish to extract to test which number of topics provides the best fit for our corpus. We are going to test the χ² on real data.</p><p>The first corpus comprises around 190 diverse news articles scraped from the web, while the second consists of 500 texts focused on political topics. It’s important to note that LDA is effective when dealing with documents of sufficient length. This technique may yield inconsistent results if the text lacks adequate length.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pDb1-jtL3na6Thm3" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IrNGLxIpHiQHzMMP" /><figcaption>Length of texts in Corpus 1: Articles (left) | Length of texts in Corpus 2: Politics (right).</figcaption></figure><h3>The Topic Model Goodness of Fit (χ²) component in KNIME</h3><p>To run the test, we will rely on a fully no-code approach thanks to the <a href="https://hub.knime.com/-/spaces/-/~nvkqJquBaLc9agoa/current-state/">Topic Model Goodness of Fit (χ²)</a> component available for free on the KNIME Community Hub. Additionally, in the <a href="https://hub.knime.com/francescots/spaces/Public/Optimal%20Topic%20Modeling%20Project~XmefEckFP_Slr9YR/">project folder</a> on the KNIME Hub, you will find the example workflow that is used in this test.</p><p>This component operates on the output of the LDA topic model or any other similar model that outputs the TWW and DTW matrices. It has been tested on two different corpora and the results are presented below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R_nT3mAxTBUyrqRY" /></figure><p>The plots below show the behavior of the χ² as the number of topics extracted increases. The steepness of the curve indicates that increasing the number of topics extracted enhances the goodness of fit of the topic model.</p><p>The optimal number of topics is found at a point where the curve starts to flatten out, indicating a diminishing return in model performance improvement. The authors suggest that the precise number of topics is where we reject the hypothesis that the observed and estimated word distributions for document Dj are statistically indistinguishable. However, the statistical significance of the Chi-Square test is not yet incorporated into this component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0imWMWc8aise5aRw" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-THifp-QHBD7fIUI" /><figcaption>Aggregated χ² for<strong> Corpus 1: </strong>all data points tested up to 40 with a step size of 10 thereafter (left) | Aggregated χ² for <strong>Corpus 2:</strong> all data points tested up to 40 with a step size of 10 thereafter (right).</figcaption></figure><p>The cool thing about this Chi-square is that we can also use it to optimize the other parameters like alpha (α) and beta(β).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yGFvmoUSAYasvG4O" /></figure><p><strong>Alpha (α)</strong>: It controls the sparsity of document-topic distributions. A higher alpha will result in documents being represented by a mixture of more topics, making the distribution more uniform. This parameter directly affects the DTW.</p><p><strong>Beta (β):</strong> It controls the sparsity of topic-word distributions. A higher beta will result in topics containing a mixture of more words, making the distribution more uniform. This parameter directly affects the TTW.</p><p>Alpha and Beta directly impact the generated word vectors, and their interaction and impact can be visualized in a surface plot.</p><p>In the first surface plot, we observe a minimum point at the intersection of a beta value of 0.15 and an alpha value of 0.275.</p><p>On the other hand, in the second surface plot, we find a minimum at a beta value of 0.15 and an alpha value of 0.075.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bh0W_DdduycQQxhz" /><figcaption>Optimizing Alpha and Beta — Corpus 1: Articles</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TL4-b-2NNVe0vPft" /><figcaption>Optimizing Alpha and Beta — Corpus 2: Politics</figcaption></figure><p>It’s important to select the Alpha and Beta parameters meticulously to ensure that the distribution of words, both real and estimated, closely align.</p><h3>Limitations</h3><p>The computation of Chi-square statistics stands as a useful method. However, its application to a vast datasets, consisting of thousands of lengthy documents, presents a significant computational challenge. To navigate this hurdle effectively, it’s important to use a strategic approach.</p><h4>Optimize Workflow Location</h4><p>Given the computational cost of Chi-square computation, it’s advisable to execute the process on a server or utilize platforms like <a href="https://www.knime.com/knime-business-hub">KNIME Business Hub</a>.</p><p>Alternatively, for expedited results, consider sampling the documents.</p><p>The following graph represents the computational time of the component to execute on an increasing number of lengthy documents, as you can observe, the computational time of a single Chi-square increases more than proportionally.</p><p>Currently, the component is in the experimental stage, but improving computational speed is a priority on my list of enhancements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rXbbfI5j2HR7MT_o" /><figcaption>Execution time in minutes for the component based on the number of documents.</figcaption></figure><h4>Tailoring Term Representation</h4><p>A relevant aspect to consider is the number of terms displayed in the TWW matrix. This parameter is configurable within the LDA node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*onnV066rmGQyY4Ym" /></figure><p>Although it doesn’t influence model estimation, it directly affects the number of terms available in the TWW matrix. For Chi-square computation, ideally, access to the full vocabulary is preferred. However, to maintain efficiency without compromising accuracy, setting the number of words to be &gt;=15 ensures statistical scalability. Indeed, the Chi-square looks more stable as the number of words involved increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yABBso9eWLDTbxkC" /></figure><h3>Conclusions and Benefits of Using Optop to Evaluate Your Topic Model</h3><p>In summary, incorporating Chi-square statistics offers a statistically rigorous method for assessing the effectiveness of topic models like Latent Dirichlet Allocation (LDA).</p><blockquote><em>This approach not only addresses the challenge of determining the optimal number of topics but also provides insights into model fit. The latter is a particularly difficult aspect to assess, especially within unsupervised algorithms like topic modeling.</em></blockquote><blockquote><em>Moreover, this metric offers the additional advantages of optimizing Alpha and Beta parameters, comparing models trained on different datasets, and evaluating the goodness of fit not only at the corpus level but also at the document level.</em></blockquote><p>While Chi-square statistics provide a robust statistical framework for assessing topic model performance, they primarily focus on quantitative aspects, potentially overlooking the interpretability of topics.</p><p>To maximize the utility of this metric, it’s advisable to complement its insights with those obtained from other indicators, such as Semantic Coherence, which correlates well with human judgment of topic quality. This combination allows for a more balanced evaluation, incorporating both statistical rigor and qualitative interpretability. You can compute the Semantic Coherence by making use of the <a href="https://hub.knime.com/knime/spaces/Examples/00_Components/Text%20Processing/Topic%20Scorer%20(Labs)~5_W2h2g6hBY_M0Bc/current-state">Topic Scorer KNIME Verified Component</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cf1d55dc7fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/say-hi-to-chi-square-%CF%87%C2%B2-for-optimal-topic-modeling-1cf1d55dc7fa">Say hi to Chi-square (χ²) for Optimal Topic Modeling</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dealing with sparse categorical variables in predictive modeling]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/dealing-with-sparse-categorical-variables-in-predictive-modeling-6d9548e22e7f?source=rss-b1e0b67d7885------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d9548e22e7f</guid>
            <category><![CDATA[knime]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[stories]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <dc:creator><![CDATA[Francesco Tuscolano]]></dc:creator>
            <pubDate>Fri, 04 Mar 2022 08:59:11 GMT</pubDate>
            <atom:updated>2022-03-07T14:05:43.281Z</atom:updated>
            <content:encoded><![CDATA[<h4>DATA STORIES | QUALITATIVE FEATURE ENCODING | KNIME ANALYTICS PLATFORM</h4><h4>How to use the Word2Vec model to numerically encode qualitative features while preserving ordinality</h4><h4>Traditional Feature Encoding Techniques</h4><p>One of the biggest challenges a data scientist must deal with is to find an efficient way to numerically encode qualitative features. Indeed, only numerical representation of categorical variables can be used as input of predictive models. The most known method is called <em>one-hot encoding</em>, and it works by creating dummy variables. Hence, if a qualitative column has n modalities, n columns will be added to the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*ut9j6j3dse9giaR3Yu4bng.png" /></figure><p>Even if most of the time dummy encoding is an effective and flexible way to reach a good performance, there are situations in which it would be needed to explore other methods, like “<em>Frequency encoding</em>” and “<em>Target encoding</em>”. The former aims at replacing each modality with its frequency inside the training set, while the latter replaces it with the mean value of the target variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*4FUlb22Y5qJiqzrzEvk_bQ.png" /></figure><p>The optimization of predictive models often involves an iterative process where a global minimum of a regularized error functions is minimized. This is possible thanks to gradients. Sometimes, using a continuous features space can make this process easier and generate more <em>statistical power</em>. Another popular method is called <em>“Label encoding”</em>, to each category is assigned a value from 1 to <em>N</em> (here <em>N</em> is the number of categories for the feature).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*QoLp-58OIT-pQ4wwKdxmDQ.png" /></figure><h4>Disadvantages of Traditional Feature Encoding Techniques</h4><p>Dummy encoding is an effective technique to manage qualitative variable, but when the number of categories is huge it may result in a too large number of columns. An excess of sparsity in the data can lead to:</p><ul><li>An increase of the model complexity</li><li>Overfitting</li></ul><p>On the other hand, target and frequency encodings generate only a one-dimensional representation. Furthermore, both methods do not take into consideration ordinality in the data. Is there a way to arrange qualitative variables in a way so that the similarity between elements is preserved?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cHbK60wFIHlJv3xakFCaUg.png" /></figure><p>The challenge is encoding the concatenated string in a way that a similar combination of qualitative features shows a similar encoding vector representation, if it happens, we will state that they are <em>meaningfully represented into a continuous space</em>.</p><h4>Feature Encoding using Word2Vec</h4><p>A Word2Vec model allows to assign to each word in a text a list of numerical coordinates, let’s see it in a one-dimensional space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pBJy1e_fcCYs5g2qYtp4nA.png" /></figure><p>As you can see, a similar combination of tokens has a similar vector coordinate. Word2vec is a simple neural network that is able to map words into a pre-define number of dimensions, called embeddings. It works by predicting the most probable word for each word in the vocabulary. You can find a lot of articles which explain very well its mechanism out there:</p><ul><li>I recommend the blog post <a href="https://www.knime.com/blog/word-embedding-word2vec-explained">Word Embedding: Word2Vec Explained</a> from the KNIME Blog</li></ul><p>The result is a set of coordinates associated to each word, this set of coordinates could be averaged, so that we get the average value for each embedding dimension, and the rows are placed into a new coordinates system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtbmD9Vf9hNzjxP-OuOZVA.png" /></figure><p>In the table above, we can observe that the two Porsche Carrera are extremely close in the embeddings space. The only difference is given by the fact that the first one is <em>model S,</em> while the second one is not.</p><p>Same vehicles will have the same encoding representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oj7TrF8nMURbIF8-ulJGlA.png" /></figure><p>Since the variables involved in the analysis are way too many, it is almost impossible to provide an exhaustive visual representation, but a coloured scatterplot is useful to understand what the technique is doing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5Eo_IXzv9Beg7bND1QwnA.png" /></figure><p>As a result, semantically similar car models are supposed to belong to the same context, hence you will find them closer in the embeddings space. We are visualizing only two out of five embeddings’ dimensions coloured by one out of five variables.</p><blockquote>The advantage of using Word2Vec is that entities are represented meaningfully because they have closer distance when qualitative characteristics are similar. This would not happen when using frequency encoding and target encoding, since the numerical vector depends on something else and is tied to be one-dimensional.</blockquote><p>Instead, what happens when using dummy encoding on this example dataset? The complexity skyrockets! As a result of dummy encoding, we get 1.627 from 19 columns!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*7Xkictk30sKSRnNS4mMqfg.png" /><figcaption>Number of columns generated by dummy encoding</figcaption></figure><h4>Feature Encoding using Word2Vec in KNIME</h4><p>However, to make Word2Vec model work correctly, we need to define some parameters:</p><p><strong>Minimum count</strong></p><p>It represents the minimum frequency value of words to be present in the vocabulary.</p><p><strong>Windows</strong></p><p>The maximum distance between the current word and its neighbouring words. If your neighbouring word is greater than the width, then, some neighbouring words would not be considered as being related to the current word. <em>A good value for that is the average number of tokens for each concatenated string</em>.</p><p><strong>Embeddings dimensions</strong></p><p>Define the number of embeddings dimensions to map tokens. Then, the average value for each dimension defines the coordinates of the concatenated string.</p><p><strong>Training algorithm</strong></p><p>It defines the training algorithm used, the alternatives are two: COBW (Continuous Bag of Words) and SKIP-gram. On the KNIME Hub, you can find a component which, given a set of input parameters, estimate the Word2Vec model using Python and automatically compute the average vector.</p><ul><li>Try out the <a href="https://kni.me/c/N__yooePF8qeZ9d_">Multi-column encoder with Word2Vec</a> component from the KNIME Hub</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*QkOam9Esnme8OcIGKvi1qA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*1_4FLcSTpBiK0OEsXDy1iQ.png" /></figure><p>Something really challenging is the choice of the number of embeddings dimensions to correctly represent the data. An analysis of the variability can be useful to define the correct cut off, for example, through a PCA analysis, we can inspect the cumulative variance of embeddings dimensions explained by the number of extracted components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZEk4MK7fV_8iS1FHauB83Q.png" /></figure><p>Suppose that fourteen embeddings have been extracted. With four components we capture 90% of the variability. On the other hand, the graph becomes almost flat with more than seven components. Hence, we can conclude that extracting more than seven embeddings would not add any useful information. For the purpose of this article, I choose to extract five embeddings and 2 PCA components, which capture 90% of the variability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*ZX6Hc4FK7egzU5hSf5Eagg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*yuNz9D0DsGhBrfMPGCYQ4g.png" /></figure><p>Now it looks cleaner than before! We are showing only 12 brands out of 42.</p><h4>Training a Classifier with Word2Vec-encoded Features</h4><p>Finally, we can use the Support Vector Machine algorithm to classify High Consumption vehicles and Low Consumption vehicles. The binary target variable has been created by splitting the CO2 Emission using the median value: 246. The KNIME Workflow used for the analyses can be found at the following link on the KNIME Hub:</p><ul><li><a href="https://hub.knime.com/francescots/spaces/Public/latest/Variables%20Encoding%20with%20word2vec~vrHF_i2ByIQosypK">Handling sparse categorial variables with Word2Vec</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EnX0frEANASrCm5vkkW_zw.png" /></figure><p>The advantage of extracting only two components is that we can visualize how the target variable is distributed comprehensively:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*svpmlcpBr2cDrJnjS5Sypg.png" /></figure><p>We will find the separating vectors using a Gaussian Kernel, which allows to map the two-dimensional feature space into a higher dimensional one, where we can separate classes through a linear decision surface. The <em>radial basis function</em> computes pairwise similarities between elements and project them closer in the new space as the distance is lower:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DGOt01ApWfxff-KHWmN9vw.png" /><figcaption>Image Source (Edited): <a href="https://medium.com/@zxr.nju/what-is-the-kernel-trick-why-is-it-important-98a98db0961d">https://medium.com/@zxr.nju/what-is-the-kernel-trick-why-is-it-important-98a98db0961d</a>.</figcaption></figure><p>We need to find the optimal values of gamma to define the most appropriate shape of the decision surface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jc_JFFORBE5VvKhEUDBsgQ.png" /><figcaption>Image Source: Figures from openclassroom.stanford.edu (Andrew Ng).</figcaption></figure><p>The parameter sigma corresponds to the standard deviation of the Gaussian Kernel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*vSkrBYJLz5sOcqA7A3cmmw.png" /></figure><p>A low value of sigma makes the decision surface strongly dependent on the observations closest to the boarder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QN83osJ1J4cFmMzMOTrQwg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uCys2KDqkCC2zWGPsk-wGw.png" /></figure><p>Hence, with just two principal components we reach a satisfying result of 0.87 AUC. As we have seen, dummy encoding generates a huge number of columns. Indeed, when looking at the frequency distribution of the count of unique combinations for the five embedded variables, they occur in most cases only once or twice. As a consequence, the model would have seen a low number of unique combinations as training instances, affecting the ability to learn patterns and to make good predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*5aTFVULoRY5hnUZNMZWPpQ.png" /><figcaption>Frequency of each unique combination (65% of each combination occur only once or twice).</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d9548e22e7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/dealing-with-sparse-categorical-variables-in-predictive-modeling-6d9548e22e7f">Dealing with sparse categorical variables in predictive modeling</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>