<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emilio Lapiello on Medium]]></title>
        <description><![CDATA[Stories by Emilio Lapiello on Medium]]></description>
        <link>https://medium.com/@emiliolapiello?source=rss-fac2dff7460e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*C-7NAbq9ahBLaglAx0mcXg.png</url>
            <title>Stories by Emilio Lapiello on Medium</title>
            <link>https://medium.com/@emiliolapiello?source=rss-fac2dff7460e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:45:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@emiliolapiello/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Decoding DeepSeek: Why China’s Latest AI Move Might Not Be What It Seems]]></title>
            <link>https://medium.com/@emiliolapiello/decoding-deepseek-why-chinas-latest-ai-move-might-not-be-what-it-seems-7a9ce2d634ec?source=rss-fac2dff7460e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a9ce2d634ec</guid>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Emilio Lapiello]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 23:09:34 GMT</pubDate>
            <atom:updated>2025-01-27T23:09:34.663Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DTPdCLO0Q61Q0kzi-SWCXA.jpeg" /></figure><p><em>The recent unveiling of DeepSeek’s open-source model is more than just another AI announcement — it’s a clear signal that China has found a significantly more efficient path to creating Large Language Models than OpenAI and other counterparts.</em></p><p>Back in 2019, I had the opportunity to visit several Chinese tech giants, including Tencent. What struck me then was how the common western narrative of “China catching up” was not just inaccurate — it was fundamentally misguided. Chinese tech companies weren’t following; they were charting their own course, and in many ways, they were ahead. More importantly, the integration of technology into daily life was far deeper than what we were seeing in the West.</p><p>Fast forward to 2025 and this ‘China catching up’ flaw is now evident to everyone after DeepSeek’s recent move. Their efficient model raises some intriguing questions that deserve careful consideration:</p><p>Why would DeepSeek choose this particular moment to release such news, right after OpenAI and the American Government announced their $500B AI development pledge? And perhaps more puzzling: if DeepSeek’s approach is truly groundbreaking and strategic, why release it as open source? Why essentially hand over your technological advantage to your competitors?</p><p>Here’s my perspective on what might be really happening:</p><p>By demonstrating the ability to create high-performing AI models without massive investments, DeepSeek is making a sophisticated statement. <em>They’re not just showcasing technical mastery— they’re challenging the entire approach to AI development.</em> Making it open source adds another layer to this message:</p><ul><li>“We can disrupt markets with free alternatives”</li><li>“Your $500B funding approach is unnecessary and fundamentally misguided” (I can almost hear Elon’s characteristic laugh)</li></ul><p>But there’s an even more interesting angle to consider: what if this is just the tip of their AI iceberg? No strategic player reveals their best technology without reason. This leads to two possible interpretations:</p><ol><li>They have more advanced AI systems in reserve</li><li>They view general-purpose AI models as non-strategic assets</li></ol><p>This brings us to what I believe is the real story behind DeepSeek’s move. While the West focuses on pouring massive capital into general-purpose AI development, China might have already identified a more sophisticated strategy: proving they can build these models efficiently, then openly sharing them precisely because they’re not the real prize.</p><p>The future of AI won’t be determined by who can build the biggest, most expensive general-purpose models. Instead, success will likely come from specialized models that excel at specific tasks, backed by efficient development approaches. Companies like Perplexity have already recognized this, focusing on mastering specific use cases rather than trying to build do-it-all systems.</p><p>In this light, DeepSeek’s open-source release isn’t just a technological statement — it’s a strategic message about the future of AI development. And while the we debate massive funding packages, others might be quietly mastering something far more intriguing: the art of building valuable AI.</p><p>— —</p><p>For those interested in the technical details behind DeepSeek’s architecture and their MoE (Mixture of Experts) approach, here’s an excellent analysis:</p><p><a href="https://open.substack.com/pub/cameronrwolfe/p/moe-llms?utm_source=share&amp;utm_medium=android&amp;r=2rwnm0">[link]</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a9ce2d634ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Embeddings, a building block for LLMs and GenAI]]></title>
            <link>https://medium.com/@emiliolapiello/demystifying-embeddings-a-building-block-for-llms-and-genai-407e480bbd4e?source=rss-fac2dff7460e------2</link>
            <guid isPermaLink="false">https://medium.com/p/407e480bbd4e</guid>
            <category><![CDATA[categorical-data]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Emilio Lapiello]]></dc:creator>
            <pubDate>Thu, 31 Aug 2023 19:15:46 GMT</pubDate>
            <atom:updated>2023-09-01T02:33:55.046Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Embeddings are a real powerhouse in the world of machine learning. They are one of the building blocks of the Transformer architecture, which is behind the magic of Generative AI and Large Language Models.</em></p><p><em>Yet, embeddings can be a bit of a puzzle for many. In this article, we aim to demystify embeddings by illustrating their connection to the widely recognized data transformation technique known as One-Hot encoding.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*v1WUqjEAYLaTPuNNgSTy5A.png" /></figure><h3><strong>Machines learning transformation rules</strong></h3><p>To understand embeddings — or better, <em>feature embeddings</em> — we need to take a quick step back and look at how machines learn rules.</p><p>Let’s take supervised learning for example: a computer uses an iterative procedure (i.e., algorithm) to learn rules through a feedback loop.</p><p>At each step of the loop, the algorithm calculates the error between the machine predictions and true observed values for a target variable, then updates its parameters to try and reduce this gap.</p><p>When this error minimization process is completed, the resulting model has learned rules to transform input data (i.e., features) into its best prediction for the target variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*vnd2v4DD8cRoV0OYrG131Q.png" /><figcaption>Figure 1 : Schematic view of model training with supervised learning</figcaption></figure><h3><strong>It’s all about the inputs</strong></h3><p>Let’s focus our attention on input features; transforming them into a format which can be manipulated by a computer is key to achieve any modeling results.</p><p>In general, input features belong to two types: <em>continuous</em> and <em>categorical</em>.</p><p><strong>Continuous</strong> variables are numbers. Their ordinality matters, and often the difference between two values represents a precise gap in the variable they are representing. We use this type of variable when value ranking carries information.</p><p>Examples of continuous variables are someone’s age, or a score preference ranking (e.g., 1 to 5 stars).</p><p><strong>Categorical</strong> variables are not meant to express ordinality or ranking, but only some sort of differentiation. For example, color options for a new car could be represented by a categorical variable.</p><p>Leaving aside the usual cleaning and data preparation tasks that all good data scientists would do, continuous variables can be fed into our machine learning algorithms pretty much as they are. Also, they can be represented by just one column in our dataset.</p><p>But this is not the case with categorical variables, because machines can only process numbers, so how do we deal with car colors?</p><h3><strong>Making (some) categorical variables useful: One-Hot encoding</strong></h3><p>We need to transform categorical variables into a numerical representation which does not carry along any implicit ranking or ordering.</p><p>One very well-known way to accomplish this task is to transform categorical variables using a procedure called <em>One-Hot encoding</em>.</p><p>Data scientists are familiar with this approach, but it is worth briefly summarizing it here — just in case…</p><p>One-Hot encoding a categorical variable means adding to our dataset one feature (i.e., column) for each different value our variable can assume. Then for each data observation in our dataset (i.e., rows) we set the value for all those added columns to 0, except for the one marking the value of the specific observation, which we set to 1.</p><p>Using our car color example, if we had three color options (say red, blue, white) we would add three features to our dataset (namely <em>is-red</em>, <em>is-blue</em>, <em>is-white),</em> and then set only one of these features to 1 to identify the actual car color. For a red car we will set <em>is-red</em> to 1, while <em>is-blue</em> and <em>is-white</em> features will be set to 0.</p><p><strong>What’s wrong with One-Hot encoding?</strong></p><p>One-hot encoding is extensively used in machine learning, but it has its limitations.</p><p>It turns out machines can encounter insurmountable challenges in learning patterns from data when there are too many features and not enough observations in our dataset.</p><p>Because One-Hot transformations generate as many features as the values each categorical variable can assume, when a variable cardinality is large, this process generates too many features for our model to handle.</p><p>As a result, One-Hot encoding can’t really be used to transform high cardinality variables such as zip-code, product-ids, or words within the English vocabulary.</p><p>Often data scientists end up removing high-cardinality categorical variables from their machine learning models, at the price of throwing away all the valuable information those features might carry.</p><p>In other cases, like with Natural Language Processing, it is not possible to throw away your only input, so how do we deal with that?</p><p>Wouldn’t it be helpful to solve the One-Hot encoding high-cardinality limitations, and extract all the information from categorical variables?</p><h3><strong>Making all categorical variables useful: Embeddings</strong></h3><p>Let’s take a deeper look at one-hot encoding with an example.</p><p>Say we are working on a model; we believe a variable like Store ID carries some important information and we identified one thousand stores.</p><p>If we apply one-hot encoding, we will need to add one thousand columns to represent each store id, and the columns to be added to our dataset will look like the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*zm2hiV8vFNGBO0t3tLEgRA.png" /><figcaption>Figure 2: Example of One-Hot encoding for Store ID feature</figcaption></figure><p>As we mentioned, this representation can create problems for our algorithm learning ability, particularly if we don’t have plenty of data.</p><p>Keeping the number of features under control helps our algorithm learn, but even dropping just one of the generated one-hot features would remove all the information carried by the StoreID variable.</p><p>We need a transformation that allows us to reduce the number of features representing StoreID, while retaining as much information as possible.</p><p>This transformation exists and it can be applied using a linear matrix multiplication. Linear matrix multiplications have the property of changing the dimensions of a space, and therefore we can use them to reduce the number of dimensions needed to describe the information contained in a categorical variable like StoreID.</p><p>In our example we have a One-Hot encoded matrix whose size is <strong>number of observations</strong> x <strong>number of stores (N x 1000)</strong>. Say we wanted to limit the store ID representation to just 4 dimensions.We can use a matrix multiplication to map the 1000 dimensions to only 4 by multiplying the one-hot encoded matrix by a matrix with size <strong>number of stores</strong> <strong>x dimension we want (1000 x 4)</strong>.</p><p>This multiplication results in a matrix with size <strong>number of observations x 4 (N x 4)</strong>.</p><p>We effectively mapped a one-thousand-dimensional space to a 4-dimensional one. This is a great result, because we can now add to our dataset only 4 columns instead of 1000. These 4 columns represent Store ID features usually called <em>implicit</em> (or <em>latent</em>) <em>features</em>, or <em>feature embeddings</em>.</p><p>Note that the number of resulting latent features is arbitrary and a modeler’s design choice. Also, One-Hot values can only be 1 or 0, but implicit features can assume any continuous value.</p><p>The question becomes: which 4 embedding values are the correct ones for each of the observations?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*oJ3LJF3eifko0iUjMPpCbw.png" /><figcaption>Figure 3: mapping one-hot encoding to feature embeddings</figcaption></figure><h4><strong>Machines learning Embedding values</strong></h4><p>So far, we have been able to ‘squeeze’ our one-thousand-dimensional variable StoreID into just 4 features, but we have not said anything about how feature values are determined for each observation.</p><p>It would be ideal to determine those values using an automated procedure, a machine learning procedure that learns the necessary matrix multiplication coefficients for the mapping matrix of size <strong>number of stores</strong> <strong>x dimensions we want (1000 x 4)</strong>.</p><p>If you are familiar with Neural Networks (NN) or their Deep version (DNN), you might have already realized that this automated procedure exists, it usually leverages some sort of Stochastic Gradient Descent algorithm to minimize prediction error, and it is exactly the procedure used by (D)NNs to learn the coefficients of the (D)NN matrix multiplications (also called <em>weights</em>).</p><p>Let’s get a little deeper.</p><p>Feed-forward Neural Networks are a sequence of linear matrix multiplications interspersed with non-linear functions, usually ReLU (Rectified Linear Unit — a fancy name for a function that returns the maximum between 0 and a number in input).</p><p>The non-linear function is not part of the learning process as this function is fixed. The only parameters learned are the weights in linear matrix multiplications, so let’s focus on those.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*ZHJi30-O_HuJD5bf_kP4OA.png" /><figcaption>Figure 4: Illustrative matrix multiplication in a neural network</figcaption></figure><p>In a DNN there are multiple matrix multiplications, and each of them is usually referred to as a <em>layer</em> of the DNN. The purpose of these layers is to <em>transform</em> input features into an output which minimizes our prediction error when compared to the ‘ground truth’ of a target variable.</p><p>If we plug our One-Hot-encoding-to-embeddings mapping matrix as the first transformation applied to our categorical inputs, we can leverage our DNN learning process to determine our embeddings coefficients: our mapping matrix will be learned while training our DNN, just like any other layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*CTPCmrHz05-jIQg_KzLaGg.png" /><figcaption>Figure 5: Embedding values determined as part of the NN learning process</figcaption></figure><p>These are quite powerful results: first we squeezed one thousand dimensions into 4 features so that our algorithm could learn, then we automatically determined the best value for those features to carry as much useful information as possible about our stores.</p><p>We are done, and the only thing we really did was to add one extra matrix multiplication layer to our DNN.</p><p>In case you are wondering, in modern DNN implementations there is no need to go through the One-Hot encoding step, and embeddings are implemented as their own DNN layer. But next time you feel puzzled by encountering embeddings, think about a One-Hot encoded categorical variable multiplied by a mapping matrix in the background, and you might feel some sort of relief.</p><h3><strong>Conclusions</strong></h3><p>Embeddings are powerful tools in machine learning, especially in LLMs and GenAI models. They help us deal with high-cardinality categorical variables, like words in a vocabulary.</p><p>Despite their fancy reputation, embeddings are just a clever twist on something data experts are familiar with: One-Hot encoding. Hopefully this article helped you realize they are not as puzzling as they seem!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=407e480bbd4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How good is your forecasting? Unpacking metrics to evaluate true business impact of your models]]></title>
            <link>https://medium.com/bcggamma/how-good-is-your-forecasting-unpacking-metrics-to-evaluate-true-business-impact-of-your-models-4a08b5a213c4?source=rss-fac2dff7460e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a08b5a213c4</guid>
            <category><![CDATA[measure-what-matters]]></category>
            <category><![CDATA[model-evaluation]]></category>
            <category><![CDATA[mape]]></category>
            <category><![CDATA[forecasting-models]]></category>
            <category><![CDATA[business-value]]></category>
            <dc:creator><![CDATA[Emilio Lapiello]]></dc:creator>
            <pubDate>Tue, 14 Jun 2022 21:42:47 GMT</pubDate>
            <atom:updated>2022-06-16T12:30:28.818Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wChHB3yzO245lImZUwCWVw.png" /></figure><p><em>Assessing machine learning models with granular metrics can boost business impact.</em></p><p><em>by Emilio Lapiello, Mikel Arizaleta, Andra Fehmiu, Alessandro Scaglia, Wenting (Tina) Hou</em></p><p>There are multiple machine learning metrics data scientists use to assess the accuracy of forecasting models. No single metric is considered ideal. Most, including MAPE (Mean Absolute Percentage Error), WMAPE (weighted Mean Absolute Percentage Error) , and R2, share a common approach: Using the notion of an “average” to provide feedback on model performance. MAPE, for example, describes the average absolute percentage error a model makes when comparing its predictions to actual values in a back-testing effort.</p><p>Aggregated “on-average” metrics such as MAPE help data scientists compare model performance against a benchmark. But these metrics often lack the detail needed to understand whether the model is providing predictions that are useful from a business perspective. Model predictions are much more valuable when they enhance business knowledge and help users understand how to solve the problem at hand.</p><p>The value of model predictions depends strictly on the nature of the sought-after outcome. Say you are a restaurant chain manager and want to understand hourly customer demand to optimize staffing at each of your restaurants. To have business value, your model must be accurate during demand peaks and troughs throughout the day, rather than on average. Knowing that your model’s MAPE is low, for instance, would be meaningless if you can’t determine whether the error is concentrated at high or low demand hours.</p><p>Similarly, say you sell canned soups in different flavors and want to forecast the number of boxes of each product your sales representative should be able to sell to any given store in a month. The challenge is that a typical store buys very few of some soup flavors (perhaps one or two boxes), but dozens of boxes of other ones. In this situation, an aggregated metric might be highly skewed by the quantity of the less-popular products sold — even though these sales would not have a large impact on your business overall performance. If, for example, the metric predicted that a store would buy two boxes of the product when, in fact, it bought only one, the metric would account a 100% error on this item.</p><p>Clearly there is a business need to evaluate forecasting models beyond aggregated mathematical metrics and consider the real business impact forecasting errors make.</p><p><strong>Introducing F-MAPE, a new way to evaluate the business impact of forecasting-model errors</strong></p><p>To overcome the limitations of the MAPE approach, we have developed a new way to assess model forecasting accuracy — one that we believe helps solve some of the issues that surround aggregated metrics. We call our approach Factorized-MAPE (or F-MAPE) because it unpacks or “factorizes” the summation components of MAPE. In doing so, it provides feedback on model performance that is actionable from both a business and a data science perspective and presents a clearer view of any error that might be induced from using the forecasting model.</p><p>To demonstrate the F-MAPE approach, we will use the canned soup sales forecast example referred to above. Assume we have built a model that predicts monthly sales volume for each flavor at each store. Our goal, using a back-testing exercise, is to compare the model’s predicted sales volume to the actual sales volume. In doing so, we will focus on two numbers:</p><p>1. The error (with sign) of our sales volume prediction for each product-store combination (i.e. actual volume minus predicted volume) in a specific month</p><p>2. The actual volume sold for the product-store combination in the same month</p><p>For the sake of argument, we will stipulate that the error spans from -147 to 147, and that actual sales span from 3 to 740. We can bin both dimensions and use those bins to create a table. In each cell of this table, we represent the percentage of model predictions falling into the relative range of both error and actual volume sold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4GS7K3dcg4TtF6JkX_vo3A.png" /><figcaption>Figure 1: Percentage of predictions by volume prediction error and actual volume sold</figcaption></figure><p>For example, the highlighted cell shows that 1.2% of all our predictions are for products that sold between 10 and 20 boxes and have a prediction error between 1 and 2 boxes.</p><p>From this table we can also derive a maximum percentage-error for each cell by dividing its maximum <em>error</em> range by the minimum <em>actual</em> range. For example, the highlighted cell only contains predictions with a maximum error of 20% (i.e., 2/10).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSrj2616QOsy9iznJQcQlQ.png" /><figcaption>Figure 2: Maximum error prediction by cell</figcaption></figure><p><strong>How is our model performing?</strong></p><p>If we set a model error tolerance threshold of, say, +/- 20%, then we can tell whether our prediction errors in a cell are tolerable or not: whether it is, in effect, a set of “good” or “bad” predictions. In the following table we have highlighted in green those cells that have good (i.e. tolerable) error predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*68VN_Pn4FzXj--gPbcM0sA.png" /><figcaption>Figure 3: Percentage of predictions with tolerable error (green) vs not (white)</figcaption></figure><p>Moreover, by summing all such “good” cells, we get a measure of model performance ‘accuracy’ strictly defined by business-set error tolerance, which, in this case, is 73.2%.</p><p>Notice that:</p><ul><li>Businesses can set different tolerance thresholds according to their needs</li><li>Tolerance thresholds need not be symmetric for over- and under-prediction errors</li><li>Bin sizes impact the model performance accuracy <em>summary metric </em>described above — because we are conservatively assuming all predictions in a specific cell have maximum possible error</li><li>Bin ranges can be chosen according to the required analysis resolution and detail needed</li><li>It is possible to calculate an accuracy summary metric using each individual prediction percentage error and then counting how many predictions are within our tolerance thresholds</li></ul><p>In the product store volume example, we decided not to use predictions for products that had only recently been introduced at specific stores and identified as those selling less than 3 boxes. We could also decide that our error tolerance is 20% in over-prediction, but only 10% in under-prediction in order not to encourage a loss in sales volume. With this business input, we can determine that 68.2% of the time our model predictions are within the business-set margin of error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2QA4_OXOEGr2BRv3JozeBw.png" /><figcaption>Figure 4: Percentage of predictions with tolerable error, asymmetric tolerance thresholds</figcaption></figure><p><strong>The Considerable Benefits of F-MAPE</strong></p><p>F-MAPE unpacks the “model accuracy black-box”, which leads to specific actions that both business and data science stakeholders can take. We identified more such benefits while designing and implementing the F-MAPE approach.</p><p>First, F-MAPE gives business stakeholders the ability to effectively compare the percentage of good-versus-bad predictions based on those factors that matter most to them. If, for example, the business is willing to accept overpredicting but not underpredicting, then F-MAPE tolerance thresholds can be set to account for that.</p><p>Stakeholders can also use F-MAPE to ascertain for which products and store locations the predictions are good or bad. It is possible, for instance, to use the F-MAPE matrix to extract the bad predictions and determine whether they tend to happen for specific products, stores, or any other relevant feature.</p><p>Stakeholders can exclude from the model accuracy calculation predictions that, from their business perspective, are not important. If, for example, a model prediction errs on products that sell less than 5 boxes and low sales products are not relevant, the business can simply exclude those predictions from its model accuracy assessment and get a better view of the actual value of model predictions.</p><p>From a data science perspective<em>, </em>F-MAPE makes it possible to identify potential model bias in predictions. For example, the metric enables data scientists to quickly observe and calculate if the right-hand side (with respect to 0 error) of the F-MAPE table includes more predictions than the left-hand side. Data scientists can further quantify this skewness by row and investigate potential trends in model bias.</p><p>F-MAPE also enables data scientists to focus further research and model development on improving the model in those areas where it is not performing well, rather than focusing on improving its performance on average. If the data science team detects that the model is failing in specific areas of the matrix, they can then flag those areas and build a descriptive model to automatically profile instances in which the model fails, and focus improvements on those by, for instance, using more or better data for specific products or stores.</p><p>Finally, F-MAPE allows data scientists to isolate predictions for products or stores they know are not correctly predicted by the current model, and then build separate models for those. If the matrix generates significant errors for product with large sales volumes, then the data scientist can, for instance, create a separate model for just those products.</p><p><strong>Getting to True Business Value</strong></p><p>F-MAPE provides a novel and effective way to unpack aggregated forecasting metrics and improve forecasting model performance using business goals as its compass. We believe this approach of separating and binning dimensions in accuracy metrics such as MAPE can, in general, lead to a better understanding of the business benefits of forecasting predictive models and how to improve these models to effectively drive business value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a08b5a213c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bcggamma/how-good-is-your-forecasting-unpacking-metrics-to-evaluate-true-business-impact-of-your-models-4a08b5a213c4">How good is your forecasting? Unpacking metrics to evaluate true business impact of your models</a> was originally published in <a href="https://medium.com/bcggamma">GAMMA — Part of BCG X</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Better Way? Forecasting with Embeddings]]></title>
            <link>https://medium.com/bcggamma/a-better-way-forecasting-with-embeddings-8f45e7065f2b?source=rss-fac2dff7460e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f45e7065f2b</guid>
            <category><![CDATA[deep-neural-networks]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Emilio Lapiello]]></dc:creator>
            <pubDate>Thu, 13 Jun 2019 18:30:25 GMT</pubDate>
            <atom:updated>2019-06-14T14:32:05.916Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsXjv7mTOs8mYPr3Xlu2sQ.jpeg" /></figure><blockquote><em>Written by </em><a href="https://medium.com/u/8000b8fa84ff"><em>Vince F</em></a><em>, </em><a href="https://medium.com/u/fac2dff7460e"><em>Emilio Lapiello</em></a><em> &amp; </em><a href="https://medium.com/u/e616bd356454"><em>Andrew Fowler</em></a></blockquote><p>There are a wide variety of models and tools designed to tackle time series forecasting problems: ARIMAX, exponential smoothing, Kalman filters, RNN, and LSTM, to name just a few. But the time series forecasting techniques most data scientists usually use to leverage historical data don’t always yield the level of granularity desired, and their implementation can be complicated or even impossible in certain scenarios.</p><p>There might be a better way. We find that a feedforward neural network with embeddings layers constitutes a straightforward and interesting non-recurrent deep learning architecture that provides excellent forecasting results and shows clear benefits, especially when:</p><p><strong>1. The historical data available is limited.</strong> Traditional time series methods need multiple seasonal cycles to perform optimally. In practice, that means a model must be trained on multiple years of data — which are frequently not available.</p><p><strong>2. There are many distinct, but related, time series</strong>. When generating multiple related forecasts — for example, forecasting sales at each of a retailer’s stores — traditional time series techniques (e.g., ARIMAX) need to be applied to each store independently, making it hard for a single model to leverage the data generated by all stores at once such as system trends, shopper behavior during holidays, etc.</p><p><strong>3. There are many high cardinality categorical variables</strong>. Common techniques for handling categorical variables such as dimensionality reduction or one-hot encoding can be time-consuming, computationally complex, and may require extensive model regularization.</p><p>For instance, in a multi-store example, the model can learn from a mix of exogenous and auto-regressive features. The embedding layers allow the model to learn from distinct stores’ time series at once by embedding the store IDs, or to encode categorical features in a meaningful way (e.g., holidays, weather, geography, etc.) in a lower dimensional space to extract valuable information.</p><h4><strong>An example of forecasting with embeddings</strong></h4><p>Let’s take an example of a classic sales forecasting problem for a large retailer, where we have access to weekly sales data from thousands of stores. We can easily manipulate and augment the data to obtain the following feature set:</p><ul><li>Past sales data (e.g. <em>1,2,…, n</em> weeks ago)</li><li>Time-related variables (e.g. <em>month, week of year, day of week, day of month</em>)</li><li><em>Store ID and/or department ID</em></li><li>Promotion features and sales calendar</li><li>Holidays (e.g. <em>Christmas, Labor Day</em>, etc.)</li><li>Weather data (e.g. <em>temperature, pressure, and precipitation forecast</em>)</li><li>Geographic data (e.g. <em>state, city, ZIP code, store accessibility</em>)</li></ul><p>Each data point is a pair <strong>x</strong>(i), y(i) where y(i) is the weekly sales and <strong>x</strong>(i) is the feature vector.</p><h4><strong>The embeddings</strong></h4><p>Some of the features listed above are categorical with high cardinality. For instance, <strong><em>x</em></strong><em>_storeid</em> takes as many values as there are stores in the dataset (e.g., thousands of values). Using a classic encoding method such as one-hot encoding might not be very effective, as it blows up the dimensionality of the input feature vector and greatly increases its sparsity.</p><p>One way to deal with such features is to use embeddings. As the <a href="https://www.tensorflow.org/">TensorFlow </a>team notes, “<em>an embedding…stores categorical data in a lower-dimensional vector than an indicator column.</em>” If the feature is in <strong>ℝᴺ</strong> , an embedding is simply a map from <strong>ℝᴺ → ℝᵐ </strong>where <strong><em>m</em> </strong>is much smaller than <strong><em>N</em>. </strong>The corresponding vector in <strong>ℝᵐ</strong> is called the embedded vector, and it is much denser and smaller than in its original <strong>ℝᴺ</strong> space.</p><p>For those familiar with models such as <em>word2vec</em>, it’s exactly the same concept. Note that the dimensions of the latent space in <strong>ℝᵐ</strong> is one of the model’s hyperparameters. In other words, the choice of <strong><em>m</em></strong><em> </em>will impact model performance and as such, needs to be chosen carefully.</p><p>Embedding layers’ weights and biases are initialized randomly and then learned with the rest of the neural network’s parameters via backpropagation, just like any other layer.</p><p>Their optimized weights and biases minimize the error on the task the neural network is trying to perform — in our case, to predict sales. It could be that if two holidays are <em>similar</em> in terms of how they impact sales, their embedded vector representations will be very close to one another in the latent space <strong>ℝᵐ</strong>. In some ways, the learning algorithm is allowed the flexibility to transform the input it’s given for certain features into something more meaningful in order to minimize its prediction error (i.e., loss).</p><p>Let’s take a concrete example. Assume there are only six mutually exclusive holiday states: <em>no_holiday, columbus_day, independence_day, Christmas, labor_day, new_year.</em></p><p>We can build and train an embedding layer for this feature from <strong>ℝ⁶</strong> to <strong>ℝ³</strong> such that the sparse one-hot representation of <em>independence_day</em> might be <strong><em>x_holiday </em></strong>=<strong><em> </em></strong>[0,0,1,0,0,0], but -for instance- its embedded representation as a dense vector in <strong>ℝ³</strong> is [4.2, -0.8, -1.8].</p><p>Once trained, we can extract the weights of the embedding layer and visualize the mapping, as shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*LaGjd-eLiDadcBFrrUbIGQ.png" /><figcaption>Embedded representation of the ‘holiday’ feature</figcaption></figure><p>As we can see, <em>Columbus Day</em> and <em>Independence Day</em> are <em>closer</em> and therefore more similar to one another in the latent space than <em>No holiday.</em></p><p><strong>The architecture</strong></p><p>In its simplest form, the neural network architecture used in this problem is shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CRSrInjvJpSlQgv2sZB15Q.png" /><figcaption>A basic Neural Network architecture with Embeddings</figcaption></figure><p>In considering the architecture, it’s important to keep in mind the following:</p><ul><li><strong>Not all features require embeddings</strong>; each one should be treated on a case-by-case basis (e.g., normalization, encoding, binning, etc.)</li><li><strong>The embedding features are implicitly one-hot-encoded before being embedded</strong>; that’s why they appear as one-dimensional in the input layer.</li><li><strong>Deep learning architectures tend to perform better when fed unstructured data without much feature engineering</strong>. For example, feeding in the past <em>0,1, …, n</em> weeks’ worth of sales and letting the learning algorithm build the autoregressive features it needs from the raw data (e.g., rolling mean) to minimize the loss.</li></ul><p><strong>The training</strong></p><p>The parameters <strong><em>θ </em></strong>of the neural network <strong><em>f </em></strong>are trained via the usual optimization problem, which is aimed at minimizing the cross-entropy between training data and the model distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*7GC_OOmtgMzWGv9LR76Osw.png" /></figure><p>If we let:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*c8pzs86P_Ji5dSCkX3Sbdg.png" /></figure><p>we then recover the mean squared error loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*-l9HSjFftwN7r8FincsLsg.png" /></figure><p>which yields:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*5wMIAPGJsSr87FSSGgFF6g.png" /></figure><p>The above loss is then optimized via stochastic gradient descent. For more information, you can reference the book “<a href="http://www.deeplearningbook.org/">Deep Learning</a>” by Ian Goodfellow, and Yoshua Bengio, and Aaron Courville, from where the equation was taken.</p><p><strong>Putting it to the test</strong></p><p>Using a publicly available data set of weekly sales from a large retailer, we trained and tested such architecture and obtained solid results.</p><p>As a benchmark, we used simple ARIMA(3,1,0) models fitted on the time series of every individual store/department and running a one-step forward prediction. We retrained those models every time a new data point was observed.</p><p>We also trained and tested a random forest to assess the performance of a simple off-the-shelf, non-linear model against the same dataset. The metrics below are computed on unseen test data pulled from about 180 stores/departments over a period of two and a half years:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*1gJkaL-vsgUb1jLrspGvAg.png" /></figure><p>More interestingly, the power of the embeddings are highlighted when we compute the performance metrics exclusively on the holiday data points, which tend to have erratic weekly sales behavior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*jkukewgheAeRsPyTvzaRPQ.png" /></figure><p><strong>A potentially better way</strong></p><p>Deep learning has traditionally been seen as an effective tool, but one whose application was mainly limited to specific prediction and classification tasks related to images, sound, and text data. When applied to time series and forecasting, recurrent architectures (e.g., RNN, LSTM) are the preferred ones.</p><p>But when there are numerous high cardinality categorical variables, the available historical data available is limited, and/or there are a multiple distinct but related time series, a simple feedforward architecture with embeddings can offer an easier, more effective — and more efficient — forecasting tool than traditional time series techniques.</p><p>Its efficiency becomes especially relevant when generating predictions for uncommon days such as holidays, where traditional methods tend to be more challenged, and may require dedicated models and tuning to make them work.</p><p><em>For additional insights, be sure to check out </em><a href="https://www.fast.ai/"><em>fast.ai</em></a><em>, where </em><a href="https://www.fast.ai/about/#jeremy"><em>Jeremy Howard</em></a><em> and </em><a href="https://www.fast.ai/about/#rachel"><em>Rachel Thomas</em></a><em>, along with Sylvain Gugger, write extensively about the power of deep learning. Another good resource is Howard’s </em><a href="https://www.youtube.com/watch?v=qqt3aMPB81c"><em>video series on YouTube</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f45e7065f2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bcggamma/a-better-way-forecasting-with-embeddings-8f45e7065f2b">A Better Way? Forecasting with Embeddings</a> was originally published in <a href="https://medium.com/bcggamma">GAMMA — Part of BCG X</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shuffling Paragraphs: Using Data Augmentation in NLP to Increase Accuracy]]></title>
            <link>https://medium.com/bcggamma/shuffling-paragraphs-using-data-augmentation-in-nlp-to-increase-accuracy-477388746bd9?source=rss-fac2dff7460e------2</link>
            <guid isPermaLink="false">https://medium.com/p/477388746bd9</guid>
            <category><![CDATA[dnn]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-augmentation]]></category>
            <dc:creator><![CDATA[Emilio Lapiello]]></dc:creator>
            <pubDate>Tue, 22 May 2018 13:03:47 GMT</pubDate>
            <atom:updated>2018-05-29T17:10:48.687Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0bJAOZSflvzUUGedR-baPQ.png" /></figure><p>While data augmentation is increasingly being used in machine learning to train models to classify images, when it comes to natural language processing (NLP) and text analytics, its application is still limited.</p><p>That’s partly because data augmentation is relatively new. But most importantly, it simply hasn’t been explored much in the text analytics space yet. It’s also less intuitive to do in NLP than it is with images.</p><p>But it is possible and in practice, deeply impactful. In this article, inspired by <a href="http://course.fast.ai/">fast.ai’s Practical Deep Learning for Coders</a>, a 7-week course taught by Enlitic founder and former Kaggle President and Chief Scientist <a href="http://www.fast.ai/about/#jeremy">Jeremy Howard</a>, I share my hypothesis that we can use data augmentation in NLP to make a model significantly more accurate.</p><p><strong>What is data augmentation?</strong></p><p>Let’s first consider the challenge of classifying specific images.</p><p>Say we want our model to be able to correctly classify images of cats and dogs, but we only have a limited number of labeled images of cats and dogs. How can we create more to better train our model? And how can we make sure our model doesn’t just learn to recognize specific cats and dogs from among the few examples that we do have?</p><p>To create more images, we could modify slightly the existing ones. For example, we could flip an image of a dog horizontally, shift it slightly to the left or to the right, or zoom in or out. (See Exhibit 1.) We could also combine those modifications in various ways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*jOMwvDCtJlbALRhtBDJLSg.png" /><figcaption>Exhibit 1: An image of a beautiful dog…</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*5WQq9oFZo_7gPGt6BoL2_Q.png" /><figcaption>…and an image of the same dog flipped horizontally</figcaption></figure><p>Each time we make one of those modifications, we are effectively “augmenting” our dataset by generating more examples of cats and dogs that we can feed into our model to make it “smarter” (i.e., better at classifying images of cats and dogs).</p><p>But there are limits to this approach. If we take that original dog image and flip it vertically, for example, we now have an image of an upside-down dog. And because we don’t have upside-down dogs in real life, that image would “confuse” our model.</p><p><strong>Text analytics: consider the message</strong></p><p>Text, like a picture of a dog, cannot just be indiscriminately flipped and shifted.</p><p>If we did indiscriminately flip and/or shift it, we would end up with examples that would confuse our model, not make it smarter. Think of trying to train a model to classify movie reviews using both the review text and the same text written in reverse. It would make no sense.</p><p>So, while we can use data augmentation to get more text samples and improve our NLP models, we must first consider the message the text is conveying and the format in which it’s being presented.</p><p>Let’s take an example of a document with a specific format, like a résumé.</p><p>A résumé is usually laid out in sections, such as personal information, work experience, education, interests, hobbies, etc. (See Exhibit 2.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*Mcv2JPueGzMoFBBrcWZKAw.png" /><figcaption>Exhibit 2: A typical résumé layout</figcaption></figure><p>From a content perspective, those sections are independent from each other. In other words, each section would convey the same message and meaning even if the order in which they appeared in the document changed. If we shuffled the paragraphs in a résumé, we’d get another résumé conveying the same message.</p><p>More importantly, for the purposes of training our model, each shuffled résumé would preserve the label we gave to its original version (e.g., a <em>good </em>resume).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*D2UDoMwBYDxGQfxbfGExwQ.png" /><figcaption>Exhibit 3: The same résumé, but with its sections shuffled</figcaption></figure><p>Say that we have identified 100 résumés as representing a “candidate we want to talk to.” A résumé with its paragraphs shuffled around would be still classified as representing a “candidate we want to talk to.” That’s because the order of the paragraphs on a résumé does not change the way we would classify it. Put another way, we would talk to a good candidate regardless of whether she listed her education before her work experience or vice versa.</p><p>And as I learned when I tested my hypothesis, even a rudimentary and random shuffling of résumé sentences will create thousands of “artificial” résumés that can be fed into our machine learning algorithm — and improve our classification accuracy by a striking 10 percent!</p><p>This approach could easily be applied to a real-world hiring situation. Since organizations don’t typically have thousands of résumés on hand to train a model, they could use this augmentation technique to create a machine learning classification to prioritize the efforts of their HR team, saving both time and money during the hiring process.</p><p><strong>Context + algorithm = impact</strong></p><p>As the<strong> </strong>résumé classification example makes clear, there are two components that we need to consider when using data augmentation in NLP to improve the accuracy of a machine learning model.</p><p><strong>1. The context in which the augmentation technique is used. </strong>To fully harvest the boost to our NLP model, we need to base our data augmentation design on a deep understanding of our document’s structure and content.</p><p>With the résumé, the fact that the information contained within the different sections isn’t altered by being moved around is critical. That same data augmentation technique could never be applied to a format where the meaning of the text depends on its order, such as a novel. The application of data augmentation must be specific to the context in which it’s being used.</p><p><strong>2. The augmentation technique being applied.</strong> The impact of applying data augmentation will depend on the specific technique itself. Every technique can potentially induce the machine to learn something different and in turn, generate a different impact.</p><p>Once we have a deep understanding of our document’s format and the information that it contains, we can experiment with different augmentation techniques to find the one that most meaningfully impacts our issue.</p><h3>Conclusion</h3><p>Although data augmentation for NLP and text analytics techniques are probably underdeveloped, they are worth pursuing as they can quite substantially boost our NLP model performance. To fully harvest their potential, however, we must base our data augmentation design on a deep understanding of both our document’s structure and content.</p><h3>Additional details</h3><p>This technique has been tested on a dataset of about 450 documents 55% of which are labeled “positive” and 45% are labeled “negative”.</p><p>Because of the structure and semantic of each of those documents, paragraphs can be assumed to be independent of each other.</p><p>After several training cycles, a DNN with (not pre-trained) word embeddings performed correct classification of about 68% of the documents in the test group.</p><p>The same DNN classified 75% of the documents correctly when applying the data augmentation technique discussed in this paper (~10% increase).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=477388746bd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bcggamma/shuffling-paragraphs-using-data-augmentation-in-nlp-to-increase-accuracy-477388746bd9">Shuffling Paragraphs: Using Data Augmentation in NLP to Increase Accuracy</a> was originally published in <a href="https://medium.com/bcggamma">GAMMA — Part of BCG X</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>