<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Or Herman-Saffar on Medium]]></title>
        <description><![CDATA[Stories by Or Herman-Saffar on Medium]]></description>
        <link>https://medium.com/@orherman?source=rss-76bbfa9f0d59------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*vY950LJJ2dig0lq0AeFdLg.png</url>
            <title>Stories by Or Herman-Saffar on Medium</title>
            <link>https://medium.com/@orherman?source=rss-76bbfa9f0d59------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:10:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@orherman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Approach for Choosing Number of Clusters for K-Means]]></title>
            <link>https://medium.com/data-science/an-approach-for-choosing-number-of-clusters-for-k-means-c28e614ecb2c?source=rss-76bbfa9f0d59------2</link>
            <guid isPermaLink="false">https://medium.com/p/c28e614ecb2c</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <dc:creator><![CDATA[Or Herman-Saffar]]></dc:creator>
            <pubDate>Sun, 27 Jun 2021 14:00:41 GMT</pubDate>
            <atom:updated>2021-06-29T07:25:31.896Z</atom:updated>
            <content:encoded><![CDATA[<h4>The suggested approach considers the common trade-off between the inner-distances and the number of clusters- and automatically choose the number of clusters</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gwh4ElgPDfTIF0AfGxrdxQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@franki?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Franki Chamaki</a> on <a href="https://unsplash.com/s/photos/data-clusters?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>When we use clustering algorithms, choosing the number of clusters is always a challenging task. While there are some existing approaches that can help with this task, they are usually being used separately in order to take the decision. Moreover, you will probably also need to manually choose the final number of clusters, based on the approaches’ outcome.</p><p>Here I suggest an approach that considers the common trade-off between the inner-distances and the number of clusters- and automatically choose the number of clusters.</p><h3>Existing Approaches</h3><p>In this section you can find the two most common approaches for choosing the number of clusters. Each has its own advantages and limitations.</p><h4>Silhouette Analysis</h4><p>This is a well-known approach that also provides very cool visualizations. It allows us to interpret and validate the consistency within the clusters. The full Python implementation and explanation can be found in this scikit-learn page:</p><p><a href="https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html">Selecting the number of clusters with silhouette analysis on KMeans clustering</a></p><p>This scikit-learn Python code produces the below visualization that help us understand whether the chosen number of clusters is a good choice or not.</p><p>A drawback of this approach (which is also presented in the link) is that it can produce more than one possible number of clusters, so you will need to choose by yourself from the different options.</p><h4>Inertia and Elbow Method</h4><p>Inertia is the sum of squared distance of samples to their closest cluster center. We would like this number to be as small as possible. But, if we choose K that is equal to the number of samples we will get inertia=0. This is the smallest inertia value we can achieve, however we miss our goal of clustering the data into the optimal number of clusters.<br>The value of inertia decreases as the number of clusters increase- so we will need to manually pick K while considering the trade-off between the inertia value and the number of clusters. For that, we usually use the Elbow Method- and we choose the elbow point in the inertia graph. After that point the improvement in the inertia value is not significant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*xLl_4YMENQBEGZV0BclMsg.png" /><figcaption>Inertia Graph</figcaption></figure><p>Based on this approach, the chosen K is around 8–11, but it needs to be manually picked.</p><p>Here is a very good article summarizing this approach:</p><p><a href="https://towardsdatascience.com/k-means-clustering-from-a-to-z-f6242a314e9a">K-Means Clustering: From A to Z</a></p><h3>Suggested Approach- Scaled Inertia</h3><p>The two approaches above require your manual decision for the number of clusters. Based on what I have learned from these approaches, I have developed an automatic process for choosing K- the number of clusters.</p><p>The suggested approach takes into account the inertia value for each possible K and weights it by a penalty parameter. This parameter represents the trade-off between the inertia and the number of clusters.</p><p>Instead of using the inertia by itself, we computed a weighted version of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*koIzAEI-tJ8uBavuouMFPw.png" /><figcaption>Scaled Inertia Formula</figcaption></figure><ul><li>Inertia- sum of squared distanced of samples to their closest cluster center</li><li>Alpha- manually tuned factor that gives penalty to the number of clusters</li><li>Inertia(K=1)- inertia for the basic situation in which all data points are in the same cluster</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*lOVgsMF3-AYyH5AlAtEZUA.png" /><figcaption>Scaled Inertia Graph</figcaption></figure><p>Alpha is manually tuned because as I see it, the penalty for the number of clusters is a business decision that should be incorporated into the analysis.</p><p>Using the Scaled Inertia, the chosen K is obvious and can be done automatically. In the above case K=9.</p><h3>Code</h3><p>The below functions are computing Adjusted Inertia for each K, in parallel computation. Non-parallel option is in the next code snippet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d75a606356868ccd3d2e8038a24f579/href">https://medium.com/media/3d75a606356868ccd3d2e8038a24f579/href</a></iframe><p>Non-parallel option:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f6a3578ef362594fdf0d0930494072f/href">https://medium.com/media/7f6a3578ef362594fdf0d0930494072f/href</a></iframe><h3>Examples- how to use</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35d7ce41c1ee50d8ac597ad65aefee74/href">https://medium.com/media/35d7ce41c1ee50d8ac597ad65aefee74/href</a></iframe><p>The function returns the best k and a data frame that contains all adjusted inertia values for all possible values of k:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/115/1*i4CPzZtpy-_4nlDF_WwRDA.png" /><figcaption>Returned Results</figcaption></figure><h3>Closing</h3><p>The suggested solution worked very well on various business problems I have encountered in the past. Please let me know how it worked out for you, and if you think more adjustments could be made to improve it.</p><p>Many thanks to Oshry Ben-Harush that helped me develop this approach and for <a href="https://medium.com/u/8572060fa298">Noga Gershon Barak</a> and <a href="https://medium.com/u/91a55b355044">Noam Kalush</a> for their technical feedback and for proofreading and reviewing this article.</p><h3>Additional Reading</h3><p>Good guide that goes into details on k-means algorithm and use in Python</p><p><a href="https://realpython.com/k-means-clustering-python/">K-Means Clustering in Python: A Practical Guide - Real Python</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c28e614ecb2c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-approach-for-choosing-number-of-clusters-for-k-means-c28e614ecb2c">An Approach for Choosing Number of Clusters for K-Means</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Based Cross Validation]]></title>
            <link>https://medium.com/data-science/time-based-cross-validation-d259b13d42b8?source=rss-76bbfa9f0d59------2</link>
            <guid isPermaLink="false">https://medium.com/p/d259b13d42b8</guid>
            <category><![CDATA[crossvalidation]]></category>
            <category><![CDATA[train-test-split]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <dc:creator><![CDATA[Or Herman-Saffar]]></dc:creator>
            <pubDate>Mon, 20 Jan 2020 06:33:10 GMT</pubDate>
            <atom:updated>2020-01-26T16:25:01.504Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23WsqANJv5IS4ar-LGU2sw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@curtismacnewton?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Curtis MacNewton</a> on <a href="https://unsplash.com/s/photos/time-based-validation?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4><strong>What happens when our data is not a time-series, but still have a time dimension which is very important? This is a </strong>Python solution for time-based cross-validation with all required inputs and an output matches scikit-learn methods.</h4><p>Training and evaluating machine learning models usually require a training set and a test set. In most cases, train and test splitting is done randomly by taking 20% of the data as test data, unseen by the model and using the rest for training.</p><p>When dealing with time-related and dynamically changing environments, where the characteristics of the environment change throughout time, it is best to use time-based splitting to provide statistically robust model evaluation and best simulate real-life scenarios. For this we should use time-based cross validation, a method taken from the time-series field, which forms a type of “sliding window” training approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*AXRu72CV1hdjLfODFGbMWQ.png" /><figcaption>Time based cross validation approach</figcaption></figure><p>This approach is well known in the time-series domain, where we have a signal which is a sequence taken at successive equally spaced points in time.</p><p>But what happens when our data is not a time-series, but still have a time dimension which is very important?</p><h3><strong>Example of a problem requires time-based cross-validation</strong></h3><p>We would like to predict delivery time of an order. Each record is an order, represented by set of features to create data table. We know <strong>when</strong> each order happened, and <strong>several</strong> orders could be placed on the same date. Detailed explanation of such problem could be found in my previous <a href="https://towardsdatascience.com/delivery-date-estimation-5aff1a0ff8dc">blog</a>. In this case, our intention was to train a new model based on last month orders, then apply it to predict the delivery time of next week’s orders.<br> In order to best mimic real world, we should train our models using data taken from a time period of a month, then test them on new data captured from the following week. To create robust and general models, we should use several splitting-points in time and apply time-based cross validation. Our final test results would be the weighted average over all test windows.</p><p><em>We need to pay attention to 3 important aspects:</em></p><p>1. <strong>Time-based train\test split</strong>- in each split, test indices must be higher than before.</p><p>2. We would like to choose our <strong>train\test set sizes</strong> in order to mimic real world scenarios in which we will train a model over some period and then apply it on the upcoming period. For example- train the model over last month data and apply it to predict on the upcoming week data.</p><p>3. <strong>Dates matter</strong>. For our intention, the number of records in each set does not matter. What matters is the size of the windows in terms of days. We would like to split the data so that each window will consist data from X days.</p><h3>Other partial solutions</h3><p>Scikit-learn has <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html">TimeSeriesSplit</a> method, but it has several drawbacks. Assuming our data is sorted by time, this method splits it into train\test sets in a “sliding window” approach, but it doesn’t allow us to choose the sets sizes, we can only choose how many splits we would like to have. Scikit-learn TimeSeriesSplit also assumes that there is one observation per date, and therefore does not address 2 and 3 above.</p><p>Another solution is the one suggested by Germayne and presented in his <a href="https://medium.com/eatpredlove/time-series-cross-validation-a-walk-forward-approach-in-python-8534dd1db51a">blog</a>. He explains very well the entire approach and the differences from Scikit-learn method. One of the inputs for this solution is the train set size (called initial) and test set size (called horizon), but it creates sets containing a fixed number of <strong>records</strong>. For our purpose, we should create sets contain fixed number of <strong>days</strong>. This solution does not address point 3.</p><h3>Suggested solution</h3><p>Therefore, I have written my own solution for time-based train and test splitting, that not only allow us to choose relevant set sizes, but also address the significant aspect of considering the window size in terms of days (and not in terms of records).</p><p><em>The returned CV splits works like any other scikit-learn cross validator and could be used with any of their methods.</em></p><p>Note that your data frame must have one column that contains the date for each record, as this solution leverages the dates of the data.</p><h4>Parameters:</h4><ul><li>train_period: int, default=30<br>Number of time units to include in each train set.</li><li>test_period: int, default=7<br>Number of time units to include in each test set.</li><li>freq: string, default=’days’<br>Frequency of input parameters. possible values are: days, months, years, weeks, hours, minutes, seconds.</li></ul><h4>Methods:</h4><p><strong>get_n_splits</strong>(self)<br>Returns the number of splitting iterations in the cross-validator</p><p><strong>split</strong>(self, data, validation_split_date=None, date_column=’record_date’, gap=0)<br>Returns list of tuples (train_index, test_index) similar to sklearn cross-validators.</p><ul><li>data: pandas DataFrame<br>Your data, contain one column that indicates the record’s date</li><li>validation_split_date: datetime.date <br>First date to perform the splitting on. This is the date when the first test set starts.</li><li>date_column: string<br>Date of each record</li><li>gap: int<br>For cases the test set does not come right after the train set, *gap* days are left between train and test sets</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a0094a7f893a39b0127d66853ccbc78/href">https://medium.com/media/9a0094a7f893a39b0127d66853ccbc78/href</a></iframe><h3>Examples- how to use</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f04a048a24915cfe1010658a27f455af/href">https://medium.com/media/f04a048a24915cfe1010658a27f455af/href</a></iframe><h3>Closing</h3><p>The suggested solution worked very well on various problems I have encountered in the past. Please let me know how it worked out for you, and if you think more adjustments could be made to improve this class.</p><p>Special thanks to Noga Gershon who shared with me her experience with related challenges.<br>Thanks to Noga and Idan Richman-Goshen for their awesome technical feedback and for proofreading and reviewing this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d259b13d42b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/time-based-cross-validation-d259b13d42b8">Time Based Cross Validation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delivery Date Estimation]]></title>
            <link>https://medium.com/data-science/delivery-date-estimation-5aff1a0ff8dc?source=rss-76bbfa9f0d59------2</link>
            <guid isPermaLink="false">https://medium.com/p/5aff1a0ff8dc</guid>
            <category><![CDATA[competitiveness]]></category>
            <category><![CDATA[on-time-delivery]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[train-test-split]]></category>
            <dc:creator><![CDATA[Or Herman-Saffar]]></dc:creator>
            <pubDate>Tue, 10 Sep 2019 13:47:46 GMT</pubDate>
            <atom:updated>2020-01-20T06:40:53.904Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wWleFjim3FghMgJxAHqDdw.jpeg" /></figure><h4>Technical challenges you can face when dealing with time-related and dynamically changing environments.</h4><p>At Dell, we do our best to provide our customers with an excellent experience. This also applies to meeting our product delivery deadlines. Providing the customer with an accurate estimated delivery date is not easy in a complex supply chain. Supply chain teams in large organizations can be large and heavily matrixed but with the shared goal of supplying products to customers quickly and on time. At the time of an order, we provide the customer with an estimated delivery date using the best information we have. It requires the consideration of several factors such as inventory levels, region, holidays, expected future demand, etc. These factors along with ample historical data make this problem a perfect use case for applying machine learning.</p><p>In this article we will explore some of the nuanced technical learnings as well as some challenges we faced while trying to introduce algorithmic intelligence and machine learning to this important challenge. we will also illustrate some of the approaches used to solve them.</p><p>Let’s start.</p><h3>On-time orders Vs. Competitiveness</h3><p>Estimating the delivery date is slightly tricky and a certain balance needs to be maintained to ensure the order is not late while also quoting as competitive of a lead time as possible. When a customer places an order and gets the promised delivery date, we must fulfill it and deliver the order on-time: on or before the estimated date we provide (and, in some cases, within a specific date range). An easy solution for this challenge is to provide the customer with a long delivery time. For example, if we think the product can be delivered in a week, we can provide a delivery date of a month out to minimize the risk of being late. However, if the customer can get a comparable product from a competitor in a shorter time-frame, we might lose the sale.</p><h4>There is a constant trade-off between being accurate and being competitive and, of course, we would aim to optimize both.</h4><p>The vast majority of machine learning algorithms are constructed with a symmetric cost function optimization. In this case, the cost of either direction of the error is the same- an error of being X days late or X days early is the same. In our case, there is an error perceived to be costlier: we would rather deliver the order early than be late. We encoded this business logic in the machine learning models by creating a custom asymmetric cost function that penalizes late orders more than early orders.</p><p>The graph below shows a comparison between regular cost function and asymmetric cost function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*vejH5koYbzil680FhOpNBA.png" /><figcaption>Comparison between regular cost function and customize asymmetric cost function</figcaption></figure><p>We would like to keep the late orders percentage very low, while as in any optimization problem, we would also like to minimize the cost function as much as we can to have zero loss. This way we would be able to provide the customer with an accurate and competitive delivery time.</p><h3>Choosing the target variable</h3><p>The most intuitive target variable is the order cycle time (the total number of days an order takes from order entry to final delivery). However, to accurately predict this time, we should consider two aspects:</p><ul><li><strong>Unit of Measure</strong>: Products are built and delivered during <strong>business days</strong>, thus our target variable should be computed to include business days only. This way, we remove the effect of weekends and holidays on our estimation of the delivery date.</li><li><strong>Cycle Time Definition</strong>: The total delivery time can be split into two main sections: build/preparation time (time between order entry and the order being shipped) and shipping time (time from when an order is shipped to when the customer receives the order). As shipping time is generally controlled by more general factors (shipping provider, weather, shipping type, etc). The suggestion is to use a main model for the prediction of the orders’ build/preparation time and use an additional sub-model to predict the shipping time after it leaves the factory (this model will depend on other factors that are more relevant for this specific problem).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIwE7FPlJBxlgDrWXwu8xw.png" /><figcaption>Timeline emphasizing the possible target variables</figcaption></figure><p>Considering the aspects above, we can be sure that no external effects are introduced into the model. That way, the model should be able to predict the preparation time based only on the orders’ characteristics and will not need external data sources such as holidays or shipping options.</p><h3>Time Based Train\test splitting</h3><p>Training and evaluating machine learning models usually require a training set and a test set (as well as a cross-validation set to optimize model parameters). In most cases, train and test splitting is done randomly by taking 20% of the data as test data unseen by the model and using the rest for training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*DQTd7HkxZqu8VgBc8ocZHA.png" /><figcaption>Random 80/20% split</figcaption></figure><p>When dealing with time-related and dynamically changing environments, such as a large company supply chain, where the characteristics of the environment change throughout time, it is best to use time-based splitting and even apply time-based cross-validation to provide statistically robust model evaluation. For this we use a method taken from the time-series field, which is time-series cross validation that forms a type of “sliding window” training approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*nqP9pXMPvMbeLH-S4sPMtg.png" /><figcaption>Time-based split</figcaption></figure><p>The suggested approach also helps us mimic the real-time production environment, in which the machine learning model is trained based on all orders available until this point in time and used to predict future orders’ delivery time.</p><p>To summarize, we have shared the main challenges we encountered while estimating the delivery date of an order using machine learning. Our goals were to increase accuracy and deliver our products on-time, while maintaining competitiveness delivery times. To do so, we used an asymmetric cost function to predict a chosen target variable and best-mimic the real-world behavior by using a unique train\test splitting approach.</p><p>Special thanks to:</p><p>Kyle West and Oshry Ben-Harush for the joint work on this project and for proofreading, reviewing this article and giving awesome technical feedback.</p><h3>Recommended reading</h3><p>If you would like to learn more about asymmetric cost function, I recommend reading <a href="https://towardsdatascience.com/custom-loss-functions-for-gradient-boosting-f79c1b40466d">Custom Loss Functions for Gradient Boosting</a> by Prince which helped me during the work on this project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5aff1a0ff8dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/delivery-date-estimation-5aff1a0ff8dc">Delivery Date Estimation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>