<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Taboola Engineering - Medium]]></title>
        <description><![CDATA[Taboola is a world leader in data science and machine learning and in back-end data processing at scale. We specialize in advanced personalization, deep learning and machine learning. Our engineers influence how consumers discover and consume content across the globe. - Medium]]></description>
        <link>https://medium.com/taboola-engineering?source=rss----a30fa12bd527---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Taboola Engineering - Medium</title>
            <link>https://medium.com/taboola-engineering?source=rss----a30fa12bd527---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:02:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/taboola-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Neural Networks from a Bayesian Perspective]]></title>
            <link>https://medium.com/taboola-engineering/neural-networks-from-a-bayesian-perspective-f2496ea5314d?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/f2496ea5314d</guid>
            <category><![CDATA[uncertainty]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Mon, 06 Aug 2018 12:48:36 GMT</pubDate>
            <atom:updated>2018-08-08T14:24:51.889Z</atom:updated>
            <content:encoded><![CDATA[<p>by Inbar Naor and Yoel Zeldes</p><p>Understanding what a model doesn’t know is important both from the practitioner’s perspective and for the end users of many different machine learning applications. In <a href="https://engineering.taboola.com/using-uncertainty-interpret-model">our previous blog post</a> we discussed the different types of uncertainty. We explained how we can use it to interpret and debug our models.</p><p>In this post we’ll discuss different ways to obtain uncertainty in Deep Neural Networks. Let’s start by looking at neural networks from a Bayesian perspective.</p><h3>Bayesian learning 101</h3><p>Bayesian statistics allow us to draw conclusions based on both evidence (data) and our prior knowledge about the world. This is often contrasted with frequentist statistics which only consider evidence. The prior knowledge captures our belief on which model generated the data, or what the weights of that model are. We can represent this belief using a <em>prior distribution</em> p(w) over the model’s weights.</p><p>As we collect more data we update the prior distribution and turn in into a <em>posterior distribution</em> using Bayes’ law, in a process called <em>Bayesian updating</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*TQ6L1uX5P-MzPsVA.png" /></figure><p>This equation introduces another key player in Bayesian learning — the <em>likelihood</em>, defined as p(y|x,w). This term represents how likely the data is, given the model’s weights <em>w</em>.</p><h3>Neural networks from a Bayesian perspective</h3><p>A neural network’s goal is to estimate the likelihood p(y|x,w). This is true even when you’re not explicitly doing that, e.g. <a href="https://www.jessicayung.com/mse-as-maximum-likelihood">when you minimize MSE</a>.</p><p>To find the best model weights we can use <em>Maximum Likelihood Estimation</em> (MLE):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*56M_UiM9XQp9MY5P.png" /></figure><p>Alternatively, we can use our prior knowledge, represented as a prior distribution over the weights, and maximize the posterior distribution. This approach is called <em>Maximum Aposteriori Estimation</em> (MAP)<strong>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*-fsXHwjkEVp70Oxw.png" /></figure><p>The term <em>logP(w)</em>, which represents our prior, acts as a regularization term. Choosing a Gaussian distribution with mean 0 as the prior, you’ll get the mathematical equivalence of L2 regularization.</p><p>Now that we start thinking about neural networks as probabilistic creatures, we can let the fun begin. For start, who says we have to output one set of weights at the end of the training process? What if instead of learning the model’s weights, we learn a distribution over the weights? This will allow us to estimate uncertainty over the weights. So how do we do that?</p><h3>Once you go Bayesian, you never go back</h3><p>We start again with a prior distribution over the weights and aim at finding their posterior distribution. This time, instead of optimizing the network’s weights directly we’ll average over all possible weights (referred to as marginalization).</p><p>At inference, instead of taking the single set of weights that maximized the posterior distribution (or the likelihood, if we’re working with MLE), we consider all possible weights, weighted by their probability. This is achieved using an integral:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*XpLxxo7PE7uGQsS4.png" /></figure><p><em>x</em> is a data point for which we want to infer <em>y</em>, and <em>X</em>,<em>Y</em> are training data. The first term p(y|x,w) is our good old likelihood, and the second term p(w|X,Y) is the posterior probability of the model’s weights given the data.</p><p>We can think about it as an ensemble of models weighted by the probability of each model. Indeed this is equivalent to an ensemble of infinite number of neural networks, with the same architecture but with different weights.</p><h3>Are we there yet?</h3><p>Ay, There’s the rub! Turns out that this integral is intractable in most cases. This is because the posterior probability cannot be evaluated analytically.</p><p>This problem is not unique to Bayesian Neural Networks. You would run into this problem in many cases of Bayesian learning, and many methods to overcome this have been developed over the years. We can divide these methods into two families: variational inference and sampling methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*xGsm2qW9BLY5oE3P.png" /></figure><h4>Monte Carlo sampling</h4><p>We have a problem. The posterior distribution is intractable. What if instead of computing the integral over the true distribution we’ll approximate it with the average of samples drawn from it? One way to do that is the <a href="https://towardsdatascience.com/a-zero-math-introduction-to-markov-chain-monte-carlo-methods-dcba889e0c50">Markov Chain Monte Carlo</a> — you construct a markov chain with the desired distribution as its equilibrium distribution.</p><h4>Variational Inference</h4><p>Another solution is to approximate the true intractable distribution with a different distribution from a tractable family. To measure the similarity of the two distribution we can use KL divergence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*hqsB9En8rCcjg9Ok.png" /></figure><p>Let <em>q</em> be a variational distribution parameterized by θ. We want to find the value of θ that minimizes the KL divergence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*cQFONz1DGVnn6NZ0.png" /></figure><p>Look at what we’ve got: the first term is the KL divergence between the variational distribution and the prior distribution. The second term is the likelihood with regards to <em>q</em>θ. So we’re looking for <em>q</em>θ that explains the data best, but on the other hand is as close as possible to the prior distribution. This is just another way to introduce regularization into neural networks!</p><p>Now that we have <em>q</em>θ we can use it to make predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*q_sZFK5nC6Q3GD-H.png" /></figure><p>The above formulation comes from a <a href="http://proceedings.mlr.press/v37/blundell15.html">work by DeepMind</a> in 2015. Similar ideas were presented by <a href="http://proceedings.mlr.press/v37/blundell15.html">graves</a> in 2011 and go back to <a href="http://www.cs.toronto.edu/~fritz/absps/colt93.pdf">Hinton and van Camp</a> in 1993. The <a href="https://www.youtube.com/watch?v=FD8l2vPU5FY">keynote</a> in NIPS Bayesian Deep Learning workshop had a very nice overview of how these ideas evolved over the years.</p><p>OK, but what if we don’t want to train a model from scratch? What if we have a trained model that we want to get uncertainty estimation from? Can we do that?</p><p>It turns out that if we use dropout during training, we actually can.</p><p>[caption id=”attachment_405&quot; align=”aligncenter” width=”719&quot;]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*MMYXoQVKFuFrUmRV.jpg" /></figure><p>Professional data scientists contemplating the uncertainty of their model — an illustration[/caption]</p><h4>Dropout as a mean for uncertainty</h4><p><a href="http://jmlr.org/papers/v15/srivastava14a.html">Dropout</a> is a well used practice as a regularizer. In training time, you randomly sample nodes and drop them out, that is — set their output to 0. The motivation? You don’t want to over rely on specific nodes, which might imply overfitting.</p><p>In 2016, <a href="http://proceedings.mlr.press/v48/gal16.pdf">Gal and Ghahramani</a> showed that if you apply dropout at inference time as well, you can easily get an uncertainty estimator:</p><ol><li>Infer y|x multiple times, each time sample a different set of nodes to drop out.</li><li>Average the predictions to get the final prediction E(y|x).</li><li>Calculate the sample variance of the predictions.</li></ol><p>That’s it! You got an estimate of the variance! The <a href="http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa532c1ce.html">intuition</a> behind this approach is that the training process can be thought of as training 2^m different models simultaneously — where m is the number of nodes in the network: each subset of nodes that is not dropped out defines a new model. All models share the weights of the nodes they don’t drop out. At every batch, a randomly sampled set of these models is trained.</p><p>After training, you have in your hands an ensemble of models. If you use this ensemble at inference time as described above, you get the ensemble’s uncertainty.</p><h3>Sampling methods vs Variational Inference</h3><p>In terms of the <a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff">bias-variance tradeoff</a>, variational inference has high bias because we choose the distributions family. This is a strong assumption that we’re making, and as any strong assumption, it introduces bias. However, it’s stable, with low variance.</p><p>Sampling methods on the other hand have low bias, because we don’t make assumptions about the distribution. This comes at the price of high variance, since the result is dependent on the samples we draw.</p><h3>Final thoughts</h3><p>Being able to estimate the model uncertainty is a hot topic. It’s important to be aware of it in high risk applications such as medical assistants and self-driving cars. It’s also a valuable tool to understand which data could benefit the model, so we can go and get it.</p><p>In this post we covered some of the approaches to get model uncertainty estimations. There are many more methods out there, so if you feel highly uncertain about it, go ahead and look for more data :)</p><p>In the next post we’ll show you how to use uncertainty in recommender systems, and specifically — how to tackle the <a href="https://en.wikipedia.org/wiki/Multi-armed_bandit">exploration-exploitation challenge</a>. Stay tuned.</p><p><em>This is the second post of a series related to a paper we’re presenting in a workshop in this year KDD conference:</em><a href="https://arxiv.org/abs/1711.02487"> <em>deep density networks and uncertainty in recommender systems</em></a><em>.</em></p><p><em>The first post can be found </em><a href="https://engineering.taboola.com/using-uncertainty-interpret-model"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2496ea5314d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/neural-networks-from-a-bayesian-perspective-f2496ea5314d">Neural Networks from a Bayesian Perspective</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Uncertainty to Interpret your Model]]></title>
            <link>https://medium.com/taboola-engineering/using-uncertainty-to-interpret-your-model-997ae5083d73?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/997ae5083d73</guid>
            <category><![CDATA[uncertainty]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Mon, 30 Jul 2018 11:10:02 GMT</pubDate>
            <atom:updated>2018-08-08T14:26:03.842Z</atom:updated>
            <content:encoded><![CDATA[<p>by Inbar Naor and Yoel Zeldes</p><p>As deep neural networks (DNN) become more powerful, their complexity increases. This complexity introduces new challenges, including model interpretability.</p><p>Interpretability is crucial in order to build models that are more robust and resistant to adversarial attacks. Moreover, designing a model for a new, not well researched domain is challenging and being able to interpret what the model is doing can help us in the process.</p><p>The importance of model interpretation has driven researchers to develop a <a href="https://arxiv.org/pdf/1706.07979.pdf">variety of methods</a> over the past few years and an entire <a href="http://www.interpretable-ml.org/nips2017workshop/">workshop</a> was dedicated to this subject at the NIPS conference last year. These methods include:</p><ul><li><a href="https://arxiv.org/abs/1602.04938">LIME</a>: a method to explain a model’s prediction via local linear approximation</li><li><a href="https://pdfs.semanticscholar.org/65d9/94fb778a8d9e0f632659fb33a082949a50d3.pdf">Activation Maximization</a>: a method for understanding which input patterns produce maximal model response</li><li><a href="https://distill.pub/2017/feature-visualization/">Feature Visualizations</a></li><li>Embedding a DNN’s layer into a <a href="http://www.interpretable-ml.org/nips2017workshop/papers/04.pdf">low dimensional explanation space</a></li><li><a href="https://deepmind.com/blog/cognitive-psychology/">Employing methods from cognitive psychology</a></li><li>Uncertainty estimation methods — the focus of this post</li></ul><p>Before we dive into how to use uncertainty for debugging and interpreting your models, let’s understand why uncertainty is important.</p><h4>Why should you care about uncertainty?</h4><p>One prominent example is that of high risk applications. Let’s say you’re building a model that helps doctors decide on the preferred treatment for patients. In this case we should not only care about the accuracy of the model, but also about how certain the model is of its prediction. If the uncertainty is too high, the doctor should to take this into account.</p><p>Self-driving cars are another interesting example. When the model is uncertain if there is a pedestrian on the road we could use this information to slow the car down or trigger an alert so the driver can take charge.</p><p>Uncertainty can also help us with out of data examples. If the model wasn’t trained using examples similar to the sample at hand it might be better if it’s able to say “sorry, I don’t know”. This could have prevented the embarrassing mistake Google photos had when they misclassified african americans as gorillas. Mistakes like that sometimes happen due to an insufficiently diverse training set.</p><p>The last usage of uncertainty, which is the purpose of this post, is as a tool for practitioners to debug their model. We’ll dive into this in a moment, but first, let’s talk about different types of uncertainty.</p><h4>Uncertainty Types</h4><p>There are different types of uncertainty and modeling, and each is useful for different purposes.</p><p><strong>Model uncertainty</strong>, AKA epistemic uncertainty: let’s say you have a single data point and you want to know which linear model best explains your data. There is no good way to choose between the different lines in the picture — we need more data!</p><p>[caption id=”attachment_366&quot; align=”aligncenter” width=”712&quot;]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cEEbd8Bk4lbn6h8V.png" /></figure><p>On the left: not enough data results in high uncertainty. On the right: given more data uncertainty reduces[/caption]</p><p>Epistemic uncertainty accounts for uncertainty in the model’s parameter. We are not sure which model weights describe the data best, but given more data our uncertainty decreases. This type of uncertainty is important in high risk applications and when dealing with small and sparse data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WQYlCl6w_ACfr5hN.png" /></figure><p>As an example, let’s say you want to build a model that gets a picture of an animal, and predicts if that animal will try to eat you. Let’s say you trained the model on different pictures of lions and giraffes and now it saw a zombie. Since the model wasn’t trained on pictures of zombies, the uncertainty will be high. This uncertainty is the result of the model, and given enough pictures of zombies it will decrease.</p><p><strong>Data uncertainty</strong>, or aleatoric uncertainty, captures the noise inherent in the observation. Sometimes the world itself is stochastic. Obtaining more data will not help us in that case, because the noise is inherent in the data.</p><p>To understand this point, let’s get back to our carnivorous animals model. Our model can recognize that an image contains a lion, and therefore you’re likely to be eaten. But what if that lion is not hungry right now? This time the uncertainty comes from the data. Another example is that of two snakes that look the same but while one of them is venomous, the other isn’t.</p><p>Aleatoric uncertainty is divided into two types:</p><ol><li>Homoscedastic uncertainty: uncertainty is the same for all inputs.</li><li>Heteroscedastic uncertainty: uncertainty that depends on the specific input at hand. For instance, for a model that <a href="https://arxiv.org/abs/1703.04977">predicts depth in an image</a> a featureless wall is expected to have a higher level of uncertainty than that of an image with strong vanishing lines.</li></ol><p><strong>Measurement uncertainty</strong>: another source of uncertainty is the measurement itself. When the measurement is noisy, the uncertainty increases. In the animals example the model’s confidence can be impaired if some of the pictures are taken using a bad quality camera; or if we were running away from a scary hippo and as a result we only have blurry images to work with.</p><p><strong>Noisy labels</strong>: with supervised learning we use labels to train the models. If the labels are noisy, the uncertainty increases.</p><p>There are various ways to model each type of uncertainty. These will be covered in the following posts in this series. For now, let’s assume we have a black box model that exposes the uncertainty it has regarding its predictions. How can we use it in order to debug the model?</p><p>Let’s consider one of our models in Taboola used for predicting the likelihood of a user clicking on a content recommendation, also known as CTR (Click Through Rate).</p><h4>Using uncertainty to debug your model</h4><p>The model has many categorical features <a href="https://engineering.taboola.com/using-word2vec-better-embeddings-categorical-features/">represented by embedding vectors</a>. The model might have difficulties with learning generalized embeddings for rare values. A common way to solve this is to use a special Out Of Vocabulary (OOV) embedding.</p><p>Think about the advertiser of an article. All rare advertisers share the same OOV embedding, therefore, from the point of view of the model they are essentially one advertiser. This OOV advertiser has many different items, each with different CTR. If we’d use only the advertiser as a predictor for CTR, we should get high uncertainty for OOV.</p><p>To validate the model outputs high uncertainty for OOV, we took a validation set and switched all the advertisers embeddings into OOV. Next, we inspected what was the uncertainty before and after the switch. As expected, the uncertainty increased due to the switch. The model was able to learn that given an informative advertiser it should reduce the uncertainty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pZXn3WCOCW19VwyY.png" /></figure><p>We can repeat this for different features and look for ones that result in low uncertainty when replaced with OOV embeddings. Either those features are uninformative, or something in the way we feed them to the model is not ideal.</p><p>We can even go to finer granularity: some advertisers have high variability between CTR of different items, while others have items with roughly the same CTR. We would expect the model to have higher uncertainty for advertisers of the first type. A useful analysis is therefore looking at the correlation between uncertainty and CTR variability within an advertiser. If the correlation isn’t positive, it means the model failed to learn what uncertainty to associate with each advertiser. This tool allows us to understand if something went wrong in the training process or in the model’s architecture, indicating we should further debug it.</p><p>We can perform a similar analysis and see if the uncertainty associated with a specific item decreases the more times we show it (i.e. show it to more users / in more places). Again, we expect the model to become more certain, and if it doesn’t — debug we will!</p><p>Another cool example is the title feature: unique titles with rare words should incur high model uncertainty. This is the result of the model not seeing a lot of examples from that area of all possible titles. We can look in the validation set for a group of similar titles that are rare and estimate the model’s uncertainty on these titles. Then we’ll retrain the model using one of the titles, and see if the uncertainty has been reduced for the entire group. Indeed, we can see that’s exactly what happened:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*k-rH-Td2R-PYorlW.jpg" /></figure><p>Wait a second… By exposing the model to some titles it was able to get better and be more certain about a bunch of new titles. Maybe we can use that to somehow encourage exploration of new items? Well, yes we can! More on that in a following post of the series.</p><h4>Final thoughts</h4><p>Uncertainty is a big deal in many domains. Identifying which uncertainty type is important is application specific. You can use them in a variety of ways once you know how to model them. In this post we discussed how you can use them to debug your model. In the next post we’ll talk about different ways to get uncertainty estimations from your model.</p><p><em>This is the first post of a series related to a paper we’re presenting in a workshop in this year KDD conference: </em><a href="https://arxiv.org/abs/1711.02487"><em>deep density networks and uncertainty in recommender systems</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=997ae5083d73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/using-uncertainty-to-interpret-your-model-997ae5083d73">Using Uncertainty to Interpret your Model</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hitchhiker’s Guide to Hyperparameter Tuning]]></title>
            <link>https://medium.com/taboola-engineering/the-hitchhikers-guide-to-hyperparameter-tuning-7339fa23218c?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/7339fa23218c</guid>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Thu, 14 Jun 2018 08:25:16 GMT</pubDate>
            <atom:updated>2018-08-08T14:28:58.609Z</atom:updated>
            <content:encoded><![CDATA[<p>by Yoel Zeldes</p><p>Now that more than a year has passed since our <a href="http://engineering.taboola.com/deep-learning-from-prototype-to-production/">first deep learning project</a> emerged, we have had to keep moving forward and delivering the best models we can.</p><p>Doing so has involved a lot of research, trying out different models, from as simple as <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">bag-of-words</a>, <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">LSTM</a> and <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">CNN</a>, to the more advanced <a href="https://distill.pub/2016/augmented-rnns/">attention</a>, <a href="http://publications.aston.ac.uk/373/1/NCRG_94_004.pdf">MDN</a> and <a href="http://ruder.io/multi-task/index.html">multi-task learning</a>.</p><p>Even the simplest model we tried has many hyperparameters, and tuning these might be even more important than the actual architecture we ended up using — in terms of the model’s accuracy.</p><p>Although there’s a lot of active research in the field of hyperparameter tuning (see <a href="https://arxiv.org/pdf/1607.08316.pdf">1</a>, <a href="https://arxiv.org/pdf/1705.08520.pdf">2</a>, <a href="https://arxiv.org/pdf/1703.01785.pdf">3</a>), implementing this tuning process has evaded the spotlight.</p><p>If you go around and ask people how they tune their models, their most likely answer will be “just write a script that does it for you”. Well, that’s easier said than done…</p><p>Apparently, there are a few things you should keep in mind when implementing such a script.</p><p>Here, at Taboola, we implemented a hyperparameter tuning script. Let me share with you the things we learned along the way…</p><h4>Let’s start simple</h4><p>Sometimes using scikit-learn for hyperparameter tuning might be enough — at least for personal projects.</p><p>For long term projects, when you need to keep track of the experiments you’ve performed, and the variety of different architectures you try keeps increasing, it might not suffice.</p><h4>Bare bones</h4><p>The first version of the script was simple, but encompassed most of our needs.</p><p>The requirements were:</p><h4>Easy to run</h4><p>You are going to run this script many times. So, it should be as easy as possible to specify what experiments you want to run.<br>We ended up with the following JSON format:</p><pre>{<br>	“architecture”: “lstm-attention”,<br>	“date-range”: [[“2017-01-01”, “2017-04-01”]],<br>	“parameters”: {<br>		“num_of_attentions”: [1, 2, 3],<br>		“attention_hidden_layer_size”: [32, 128, 256],<br>		“attention_regularization”: [0.1, 0.01, 0.001]<br>	}<br>}</pre><ul><li><strong>Architecture</strong>: architecture you want to tune, assuming your code supports multiple types</li><li><strong>Date-range</strong>: list of tuples, each defines a time range of data that will be used to train a model. Each experiment will be executed once per date-range</li><li><strong>Parameters</strong>: values to try for each hyperparameter. Our initial implementation only supported a finite set of values (grid search)</li></ul><p>The script randomly generates experiments out of this JSON.</p><p>We created a Jenkins job that runs the script on one of our machines with GPUs, thus freeing us from the need to use SSH.</p><h4>Enrich experiments with metrics</h4><p>Our training process generates many metrics, such as MSE, loss and training time. You can choose whichever you want, and they will show in the results.</p><h4>Save results to the cloud</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z4lSFaWVjg3zv5rk.png" /></figure><p>Results are saved as a CSV file in Google Cloud Storage, which enables us to launch the script from any machine, and watch the results from our laptops.</p><p>The results are continuously uploaded, so we don’t have to wait for all the experiments to finish to start inspecting the results.</p><p>The models themselves are also saved.</p><h4>So what have we learned so far?</h4><p>This first implementation was the most important one.</p><p>Being simple, the script didn’t do anything smart for you. It didn’t know which experiments to perform — you had to manually define the JSON input for every run.</p><p>After running the script many times, you start to understand which values work better than others. It’s one of the most important things we got from the script — a more in depth understanding of our models.</p><p>Using this script for the first time got us a big improvement — the MSE improved by more than 10%.</p><h4>Are we sure it’s real?</h4><p>When you run hundreds of experiments, the best ones usually have negligible differences. How can you know it’s statistically significant?</p><p>One way to tackle this problem, assuming you have enough data, is to train the same model on several date-ranges. If one model is better than the others in all of the date-ranges, you can be more confident it’s real.</p><p>Let’s say today is June, and you run the script. Here are the date-ranges the script will choose:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/0*hXCHDCCbSv6ehCMZ.png" /></figure><p>Note that the script chooses a new set of date-ranges if you run it on a different month. This is important, since otherwise you could accidentally overfit your models due to extensive hyperparameter tuning.</p><h4>I don’t have time for this</h4><p>Who has the time to run every experiment three times? It’s nice that you can get reliable results, but it means you’ll end up running fewer different experiments.</p><p>The next version of the script tackled this problem by supporting a new mode of operation:</p><ul><li>Only one date-range is used</li><li>The date-range contains only one month of data</li><li>Training is limited to fewer epochs</li></ul><p>But are the results correlated with what we would get if we used more data and epochs?</p><p>To answer this, we performed some experiments, each using a different amount of data ranging from one week to three months. We found out that one month had a good tradeoff between MSE and training time.</p><p>(Did you notice we used the hyperparameter tuning script to tune the hyperparameter script? How cool is that?)</p><p>To investigate what the right number of epochs would be, we analyzed the MSE on TensorBoard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_7O3NfR60boixwLq.png" /></figure><p>Each plot in the graph represents a different model trained on a different amount of data.<br>After 20 epochs all of the models have almost converged, so it’s safe to stop there.</p><h4>C’mon script, do the job for me!</h4><p>At this point we decided the script should choose the hyperparameter values for you.<br>We started with the learning rate related hyperparameters: initial learning rate, decay factor, number of epochs with no improvement for early stopping, etc.</p><p>Why learning rate?</p><ul><li>It significantly affects training time. We should first settle on a good learning rate, both in terms of accuracy and training time, before tuning all the rest</li><li><a href="https://books.google.co.il/books?id=Np9SDQAAQBAJ&amp;pg=PA417&amp;dq=%22The+learning+rate+is+perhaps+the+most+important+hyperparameter.+If+you+have+time+to+tune+only+one+hyperparameter,+tune+the+learning+rate%22&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwj-kdXZn8bWAhWEbRQKHeAbAiMQ6AEIJDAA#v=onepage&amp;q=%22The%20learning%20rate%20is%20perhaps%20the%20most%20important%20hyperparameter.%20If%20you%20have%20time%20to%20tune%20only%20one%20hyperparameter%2C%20tune%20the%20learning%20rate%22&amp;f=false">Some suggest</a> it controls the effective capacity of the model in a more complicated way than other hyperparameters, so it might be better to start with that.</li></ul><p>The new mode of the script uses hard-coded ranges of values that are reasonable for our models.</p><p>Then came the next demand:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*jfaCD1IgV-4z3G8n.jpg" /></figure><p>The next version of the script did just that: provided with an architecture name, it automatically generates experiments for you. You don’t need to specify any hyperparameter values.</p><h4>Let’s go random</h4><p>Although being worse than random search in <a href="http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf">some cases</a>, grid search is easier to analyze: every value is used by multiple experiments, so it’s easy to spot trends.</p><p>Since we had already gained intuition on what values work better, it was time to implement random search. Doing so helped the script find better hyperparameters.</p><h4>Some final thoughts</h4><p>In the research phase of any machine learning project, hyperparameter tuning can be done manually. However, when you want to take the project to the next level, it’s highly effective to automate the process. In this post I described some of the small touches we implemented into the automation process. Some might be useful for you, some might not. Drop a line in the comments if you found any other exciting things to be helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7339fa23218c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/the-hitchhikers-guide-to-hyperparameter-tuning-7339fa23218c">The Hitchhiker’s Guide to Hyperparameter Tuning</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Simple tips for boosting your Jenkins performance]]></title>
            <link>https://medium.com/taboola-engineering/5-simple-tips-for-boosting-your-jenkins-performance-277d23354e10?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/277d23354e10</guid>
            <category><![CDATA[garbage-collector]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[cg]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Wed, 30 May 2018 13:40:36 GMT</pubDate>
            <atom:updated>2018-08-08T14:29:53.649Z</atom:updated>
            <content:encoded><![CDATA[<p>by Tidhar Klein Orbach</p><p>Do you know that feeling when you’ve finished working on a feature, pushed the code, but then your CI system refuses to respond?</p><p>Lagging or slow responsiveness is very common among Jenkins users, and you can find many <a href="https://www.google.co.il/search?q=jenkins+performance+issues&amp;rlz=1C1GGRV_enIL751IL754&amp;oq=jenkins+performance+issues&amp;aqs=chrome.0.0j69i57.4117j0j7&amp;sourceid=chrome&amp;ie=UTF-8">reported issues</a> on it. Slow CI systems are frustrating, they make you develop slower and waste your time. I have worked on several Jenkins systems with various versions (1.x and 2.x), tens of slaves and hundreds of builds per day. I have managed to improve the performance of those systems using a few simple guidelines. In the following post I will share 5 tips that can make your Jenkins better, and put a smile on your developers’ faces.</p><h4>Tip 1: Minimize the amount of builds on the master node</h4><p>The master node is where the application is actually running, this is the brain of your Jenkins and, unlike a slave, it is not replaceable. So, you want to make your Jenkins master as “free” from work as you can, leaving the CPU and memory to be used for scheduling and triggering builds on slaves only. In order to do so, you can restrict your jobs to a node label, for example:</p><figure><img alt="Restrict to label" src="https://cdn-images-1.medium.com/max/1024/0*_UkxRSYBm1ad6Hvv.png" /></figure><p>In the case of pipeline jobs, put the label when allocating the node, for example:</p><pre>stage(&quot;stage 1&quot;){<br>    node(&quot;ParallelModuleBuild&quot;){<br>        sh &quot;echo \&quot;Hello ${params.NAME}\&quot; &quot;<br>    }<br>}</pre><p>In those cases, the job and node block will only run on slaves labeled with <em>ParallelModuleBuild</em>.</p><h4>Tip 2: Do not keep too much build history</h4><p>When you configure a job, you can define how many of its builds, or for how long they, will be left on the filesystem before getting deleted. This feature, called Discard Old Builds, becomes very important when you trigger many builds of that job in a short time. I have encountered cases where the history limit was too high, meaning too many builds were kept on the filesystem. In such cases, Jenkins needed to load many old builds — for example, to display them in the history widget, — and performed very slowly, especially when trying to open those job pages. Therefore, I recommend limiting the amount of builds you keep to a reasonable number.</p><figure><img alt="Discard old builds" src="https://cdn-images-1.medium.com/max/1024/0*2S_07DXguXm0nI_N.png" /></figure><h4>Tip 3: Clear old Jenkins data</h4><p>In continuation to the build data from the previous tip, another important thing to know is the old data management feature. As you probably know, Jenkins keeps the jobs and builds data on the filesystem. When you perform an action, like upgrading your core, installing or updating a plugin, the data format might change. In that case, Jenkins keeps the old data format on the file system, and loads the new format to the memory. It is very useful if you need to rollback your upgrade, but there can be cases where there is too much data that gets loaded to the memory. High memory consumption can be expressed in slow UI responsiveness and even OutOfMemory errors. To avoid such cases, it is best to open the old data management page (http://JenkinsUrl/administrativeMonitor/OldData/manage), verify that the data is not needed, and clear it.</p><figure><img alt="Manage old data" src="https://cdn-images-1.medium.com/max/1024/0*bG0aGdoqe7UcjVrX.png" /></figure><h4>Tip 4: Define the right heap size</h4><p>This tip is relevant to any Java application. A lot of the modern Java applications get started with a maximum heap size configuration. When defining the heap size, there is a very important JVM feature you should know. This feature is called UseCompressedOops, and it works on 64bit platforms, which most of us use. What it does, is to shrink the object’s pointer from 64bit to 32bit, thus saving a lot of memory. By default, this flag is enabled on heaps with sizes up to 32GB (actually a little less), and stops working on larger heaps. In order to compensate the lost space, <a href="https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/">the heap should be increased to 48GB</a>(!). So, when defining heap size, it is best to stay below 32GB. In order to check if the flag is on, you can use the following command (jinfo comes with the JDK):</p><pre>jinfo -flag UseCompressedOops &lt;pid&gt;</pre><h4>Tip 5: Tune the garbage collector</h4><p>The garbage collector is an automatic memory management process.</p><p>Its main goal is to identify unused objects in the heap and release the memory that they hold. Some of the GC actions cause the Java application to pause (remember the UI freeze?). This will mostly happen when your application has a large heap (&gt; 4GB). In those cases, GC tuning is required to shorten the pause time. After dealing with these issues in several Jenkins environments, my tip contains a few steps:</p><ul><li>Enable G1GC — this is the most modern GC implementation (default on JDK9)</li><li>Enable GC logging — this will help you monitor and tune later</li><li>Monitor GC behavior — I use <a href="http://gceasy.io/">http://gceasy.io/</a></li><li>Tune GC with <a href="http://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html">additional flags</a> as needed</li><li>Keep monitoring</li><li>Read <a href="https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-responsiveness-and-stability">this post</a> and <a href="https://jenkins.io/blog/2016/11/21/gc-tuning/">this one</a> — these are great posts about GC tuning</li></ul><h4>Now it’s your turn</h4><p>Jenkins responsiveness issues are very common, but can be dealt with. In one of the environments I worked on, I had a configuration page which took over a minute to open. After implementing the above tips, especially G1GC and old data, this page opened in a split second.</p><p>If you got this far, you are probably familiar with this issue. So, try my tips and share what worked for you and what hasn’t. There is a good chance your Jenkins will speed up too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=277d23354e10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/5-simple-tips-for-boosting-your-jenkins-performance-277d23354e10">5 Simple tips for boosting your Jenkins performance</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Word2Vec for Better Embeddings of Categorical Features]]></title>
            <link>https://medium.com/taboola-engineering/using-word2vec-for-better-embeddings-of-categorical-features-72f5247d4723?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/72f5247d4723</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Wed, 25 Apr 2018 14:08:49 GMT</pubDate>
            <atom:updated>2018-08-08T14:30:51.696Z</atom:updated>
            <content:encoded><![CDATA[<p>by Inbar Naor</p><p>Back in 2012, when neural networks regained popularity, people were excited about the possibility of training models without having to worry about feature engineering. Indeed, most of the earliest breakthroughs were in computer vision, in which raw pixels were used as input for networks.</p><p>Soon enough it turned out that if you wanted to use textual data, clickstream data, or pretty much any data with categorical features, at some point you’d have to ask yourself — how do I represent my categorical features as vectors that my network can work with?</p><p>The most popular approach is embedding layers — you add an extra layer to your network, which assigns a vector to each value of the categorical feature. During training the network learns the weights for the different layers, including those embeddings.</p><p>In this post I will show examples of when this approach will fail, introduce category2vec, an alternative method for learning embedding using a second network and will present different ways of using those embeddings in your primary network.</p><h4>So, what’s wrong with embedding layers?</h4><p>Embedding layers are trained to fit a specific task — the one the network was trained on. Sometimes that’s exactly what you want. But in other cases you might want your embeddings to capture some intuition about the domain of the problem, thus reducing the risk of overfitting. You can think of it as adding prior knowledge to your model, which helps it to generalize.</p><p>Moreover, if you have different tasks on similar data, you can use the embeddings from one task in order to improve your results on another. This is one of the major tricks in the Deep Learning toolbox. It’s called transfer learning, pretraining or multi-task learning, depending on the context. The underlying assumption is that many of the unobserved random variables explaining the data are shared across tasks. Since the embeddings try to isolate these variables, they can be reused.</p><p>Most importantly, learning the embeddings as part of the network increases the model’s complexity by adding many weights to the model, which means you’ll need much more labeled data in order to learn.</p><p>So it’s not that embedding layers are bad, but we can do better. Let’s see an example.</p><h4>Example: Click Prediction</h4><p>Taboola’s research group develops algorithms that suggest content to users, based on what they’re currently reading. We can think about it as a Click Prediction problem: given your reading history, what is the probability that you will click on each article?</p><p>To solve this problem, we <a href="http://engineering.taboola.com/deep-learning-from-prototype-to-production/">train deep learning models</a>. Naturally, we started with learning the embeddings as part of our network.</p><p>But a lot of the embeddings we got didn’t make sense.</p><h4>How do you know if your embeddings make sense?</h4><p>The simplest way is to take embeddings of several items and look at their neighbors. Are they similar in your domain? That can be pretty exhausting, and doesn’t give you the big picture. So, in addition you can reduce the dimensionality of your vectors using <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">PCA</a> or <a href="https://www.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm">t-SNE</a>, and color them by specific characteristic.</p><p>The advertiser of an item is a strong feature, and we want similar advertisers to have similar embeddings. But this is what our embeddings for different advertisers looked like, colored by the language of that advertiser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AiYijmt5QM9yE_Lj.png" /></figure><p>Ouch. Something is clearly not right. I mean, unless we assume that our users are multilingual geniuses that read one article in Spanish and then effortlessly go and read another one in Japanese, we would probably want similar advertisers in our embedding space to have content in the same language.</p><p>This made our models harder to interpret. People were upset. Some even lost sleep.</p><p>Can we do better?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/0*iWXXuYvFx65S1VYp.png" /></figure><h4>Word2Vec</h4><p>If you’ve ever heard about embeddings you’ve probably heard about <a href="https://www.tensorflow.org/tutorials/word2vec">word2vec</a>. This method represents words as high dimensional vectors, so that words that are semantically similar will have similar vectors. It comes in two flavors: Continuous Bag of Words (CBOW) and Skip-Gram. CBOW trains a network to predict a word from its context, while Skip-Gram does the exact opposite, predicting the context based on a specific target word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*y3HDuA30IALFFQgc.png" /></figure><p>Can we use the same idea to improve our advertisers’ embedding? YES WE.. well, you get the idea.</p><h4>From Word2Vec to Category2Vec</h4><p>The idea is simple: we train word2vec on users’ click history. Each “sentence” is now a set of advertisers that a user clicked on, and we try to predict a specific advertiser (“word”) based on other advertisers the user liked (“context”). The only difference is that, unlike sentences, the order is not necessarily important. We can ignore this fact, or enhance the data set with permutations of each history. You can apply this method to any kinds of categorical features with high modality, e.g, countries, cities, user ids, etc.. See more details <a href="https://arxiv.org/abs/1603.04259">here</a> and <a href="https://openreview.net/pdf?id=HyNxRZ9xg">here</a>.</p><p>So simple, yet so effective. Remember that messy visualization of embeddings we had earlier? This is what it looks like now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*ta2-L3guGpnFWYox.png" /></figure><p>Much better! All advertisers with the same language are clustered together.</p><p>Now that we have better embeddings, what can we do with them?</p><h4>Using embeddings from different models</h4><p>First, note that category2vec is just one example of a general practice: take the embeddings learned in task A and use them for task B. We could replace it with different architectures, different tasks, and in some cases, even different datasets. That’s one of the great strengths of this approach.</p><p>There are three different ways to use the new embeddings in our model:</p><ol><li>Use the new embeddings as features for the new network. For example, we can average the embeddings of all the advertisers the user clicked on, to represent the user history and use it to learn the user’s tastes. The focus of this post is Neural Networks but you can actually do it with any ML algorithm.</li><li>Initialized the weights of the embedding layer in your network, with the embeddings we just learned. It’s like telling the network — this is what I know from other tasks, now adjust it for the current task.</li><li><a href="http://ruder.io/multi-task/">Multitask Learning</a> — take two networks and train them together with parameter sharing. You can either force them to share the embeddings (<a href="http://ruder.io/multi-task/index.html#hardparametersharing">hard sharing</a>), or allow them to have slightly different weights by “punishing” the networks when their weights are too different (<a href="http://ruder.io/multi-task/index.html#softparametersharing">soft sharing</a>). This is usually done by regularizing the distance between the weights.<br>You may think about it as passing knowledge from one task to another, but also as a regularization technique: the word2vec network acts as a regularizer to the other network, and forces it to learn embeddings with characteristics that we are interested in.</li></ol><h4>OK, let’s recap.</h4><p>Sometimes we can get great results simply by taking some existing method, but applying it to something new. We used word2vec to create embeddings for advertisers and our results were much more meaningful than the ones obtained with embedding layers. As we say in Taboola — meaningful embeddings = meaningful life.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72f5247d4723" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/using-word2vec-for-better-embeddings-of-categorical-features-72f5247d4723">Using Word2Vec for Better Embeddings of Categorical Features</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Re2 Shattered My Bottleneck]]></title>
            <link>https://medium.com/taboola-engineering/how-re2-shattered-my-bottleneck-e05f27c5666c?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/e05f27c5666c</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[web-applications]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Tue, 13 Mar 2018 22:20:09 GMT</pubDate>
            <atom:updated>2018-08-08T14:31:41.323Z</atom:updated>
            <content:encoded><![CDATA[<p>by Ohad Edelstein</p><p>One pleasant morning I got to work, thinking this day couldn’t get any better. But as Murphy would have it, there was my boss walking frantically toward me.</p><p>It turned out that almost over night one of the main data pipeline systems had become a major bottleneck for the company, and a solution was needed, Fast!</p><p>Usually in a startup, let alone a company moving as fast as Taboola, these things can occur on a weekly basis.</p><p>I needed to find some quick wins to relieve some of the bottlenecks inside the system.</p><p>Luckily Re2 was there to the rescue — in this post I will share how to find the bottlenecks using Gprof2dot beautiful image rendering, and of course, what Re2 is and how to use it.</p><p>* Note that this article addresses a pain I had in a Python framework, but because there are Re2 implementations to all major languages you should still find value reading this article.</p><h4><strong>The strategy to relieve bottlenecks is pretty straightforward</strong></h4><p><strong>Step one:</strong> Recognize were it hurts the most</p><p><strong>Step two:</strong> Reduce the pain</p><p><strong>Step three:</strong> Iterate back to step one until you get the desired result</p><p>Sounds simple enough, right? Well it is, in theory.</p><p>Usually the implementation is where it gets you.</p><h4><strong>Step one: Recognize where it hurts the most</strong></h4><p>Like anyone who ever saw a guy talking about A.A. meetings on TV, will say:<strong><em> “The first step in dealing with your problem is recognizing that it exists”</em></strong>, in our case that meant measurements and finding the bottlenecks.</p><p>The system in question is a pretty straightforward ETL system written in Python, so I decided to start by looking at the transform part first.</p><p>I spent a couple of hours extracting the main part of the code into a single independent process, and when it was done I started a Jupyter notebook <a href="http://jupyter.org/">http://jupyter.org/.</a><br>Using cProfile built-in python package and gprof2dot, I ran the following code to test it -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WwGEhz4hw0GKxG63.png" /></figure><p>This is the image Gprof2dot produced:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PCzvX_msgbq5Hk4P.png" /></figure><p>A short explanation if this is the first time you encountered the Gprof2dot library: The image that I got was a profile that measured our code, and determined, for each major block, how much time our code visited that block and how much time in total each block contributed to the overall running time.</p><p>The dark blue ones are the “cold” areas, the interesting parts in our investigation.<br>The path starting from the red box, and “cooling down” all the way to the leaves are the interesting paths to investigate.</p><p>Each box has the following format:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Q5e_bWzyIQsY8PYG.png" /></figure><p>In the next image below, I looked at the leaf at the end of the path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/0*VU9l3nqtKK9uWkYF.png" /></figure><p>If you look at the upper box, you can see that it’s a method called doesTextFit in class GenericTextInTextSearch. The system spent 32.05% of it’s runtime in that code, but only 4.94% of it is in the actual code of the method, the other ~95% of the time was spent in code called from doesTextFit.<br>The 2,221,386 is the amount of times this code was called.<br>The more interesting part is that doesTextFit calls another method, where it spent ~26% of the time.<br>That was the method I decided to focus my attention on.</p><p>* Note that it might happen in the leaf box, that the <strong><em>self time</em></strong> — in the parenthesis — and the <strong><em>total time</em></strong> — above the self time — don’t add up.</p><p>This happens when the execution time is too short, causing the round-off errors to be large. Don’t worry about it.</p><p><strong><em>“Method ‘Search of’ _sre.SRE_Pattern object“</em></strong> is from Python built-in library — the Regex library <strong><em>re</em></strong>.<br>It turned out that the system that had a lot of moving parts, spent ~26% of the time on a single method on a single library call.</p><p>So now I set out to see if I could find a replacement for the Python built-in method.</p><p>It turns out there is. And I was completely surprised by it.</p><h4><strong>Step two: Reduce the pain</strong></h4><p>I came across these two articles:</p><ul><li><a href="http://www.benfrederickson.com/python-catastrophic-regular-expressions-and-the-gil/">Python, Catastrophic Regular Expressions</a></li><li><a href="https://swtch.com/~rsc/regexp/regexp1.html">Regular Expression Matching Can Be Simple And Fast</a></li></ul><p>They basically told me, I was living a lie — I always thought that Regex is the most efficient way to match a string. It turns out that it might have started like that, but somewhere along the way we sacrificed — in all the major programing languages — performance for features.</p><p>The salvation was going back to the fundamentals: State Machine Based Regex!</p><h4><strong>What’s wrong with the built in Regex engine?</strong></h4><p><strong><em>TL;DR </em></strong>Backtracking.</p><p>When the built-in Regex tries to match a string, it follows a greedy algorithm pattern. Trying to match the full string first, and when it fails trying to match minus one characters on the full string and the leftover char separately. Then on the full string minus two characters, and on the two leftover characters separately, and so on and so forth. If there is no match, the engine is destined to fail, it simply doesn’t know that before it iterates through all the different combinations.</p><p>This process is time and space consuming.</p><h4><strong>So what’s the alternative?</strong></h4><p>Introducing Ken Thompson. you can read more about him in his <a href="https://en.wikipedia.org/wiki/Ken_Thompson">wiki page</a>, but here is the TL;DR for the list of his achievements:</p><ul><li>Unix</li><li>B (programming language) — ancestor to C</li><li>UTF-8</li><li>Go (programming language)</li><li>State Machine Based Regex!</li></ul><p>The last bullet is best represented in this comparison graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*jaAX-mv-EWrpz4VI.png" /></figure><p>The left one is for Perl, but represents the major programing languages, the right one is the Thompson method. Make sure you notice that the left one is in <strong>seconds</strong> and the right one in <strong>microseconds(!)</strong>.</p><p>How did he do that? state machine.</p><p>Think of a regular expression as a state machine built from the expression you want to match.</p><p>Each char the engine encounters, propels it to make the next decision, if by the end of its journey it reaches a “final” state. Then the engine found a match.</p><p>If not, then there is no match.</p><p>The space complexity of a solution like that, is the amount of different states the regular expression has. And the time complexity is the length of the text being checked.</p><p>With that in mind, I went out to see if I could find an implementation of the Thompson engine, and I found the good people of Google with their <a href="https://github.com/google/re2">Re2 implementation</a> in C++.</p><p>To integrate Google Re2 in Python, I had to use a third party wrapper <a href="https://pypi.python.org/pypi/re2/">py-re2</a> , but for Java there is <a href="https://github.com/google/re2j">Google Re2/J project</a> or <a href="http://www.brics.dk/automaton/">Brics</a>, we in Taboola, use Brics in our Java code, you can read comparison of the two projects in this great blog: <a href="https://blog.rapid7.com/2016/03/15/brics-vs-re2j/">brics-vs-re2j</a>.</p><p>The integration in the code is that simple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*OfoPieruQc_3Zlw2.png" /></figure><p>It is important to remember that not all the functionality that exists in re, exists in re2. So what I did, and what you should do if you plan to use this solution, is to analyze all your Regex calls, and make sure that you get the expected result.</p><p>Although the graph above promised a performance boost of hundreds of percent, the performance boost I managed to get was just a ~40%-50% boost. I didn’t complain, it was not too bad for a few days work.</p><h4><strong>Step three: Iterate back to step one till you get the desired result</strong></h4><p>I never got to this part, and hopefully, if you use this engine you won’t either…</p><h4><strong>No Silver Bullet</strong></h4><p>The specific circumstances of the system i described here, is that the majority of the Regex comparisons finish without a match.<br>So it worked really well to have a new Regex engine that returns “no match” faster than the regular — backtracking — one. You will need to test it in your unique scenario to see how much performance improvement you will get.</p><p>Second thing to check is the Regex you use, if it needs Backtracking to complete it’s comparison it won’t work, and you will get Segfault.<br>But, it shouldn’t be too hard to rewrite them to a simpler syntax.</p><p>Third thing to monitor, is the state explosion problem when building deterministic automata, that can happen in some of the Re2 implementations — Brics for example but not in Google Re2. The cost is in performance.</p><p>The last thing you need to check is the return values, there might be a case that the expected outcome has changed from what you had before. For example to get a null were before you got a False.</p><p>The bottom line is that you will need to do the same evaluation process that you would have done before upgrading to a new version of a library. Things might change, so you need to be carefull.</p><p>Having said that, even if it is only a few percent improvement, the fact that it is so easy to integrate makes it a “money on the floor” scenario.</p><p><strong>Good luck</strong>, and feel free to contact me for help and guidance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e05f27c5666c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/how-re2-shattered-my-bottleneck-e05f27c5666c">How Re2 Shattered My Bottleneck</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DIY: Engineering Your Company Culture]]></title>
            <link>https://medium.com/taboola-engineering/diy-engineering-your-company-culture-3543168858bb?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/3543168858bb</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[bots]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Thu, 15 Feb 2018 13:31:47 GMT</pubDate>
            <atom:updated>2018-08-08T14:32:23.886Z</atom:updated>
            <content:encoded><![CDATA[<p>by Amir Keren</p><p>This is a tale of heroism, of overcoming obstacles and hardships. This is a tale of ingenuity, of originality and thinking outside the box. This is a tale… of how I was too lazy to go and look if someone was already playing table tennis in the game room.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*sYf_jpkd_CLed2Mf.jpg" /></figure><p><strong>Hiking Across the Office is a Drag</strong></p><p>Taboola’s Israeli office in Tel Aviv, houses about 350 people, spread over five large floors. The game room, however is smack dab in the middle of them.</p><p>In smaller companies, if you wanted to know if the game room was available, all you had to do was to look slightly over your monitor and you would have your answer.</p><p>Here, it takes 60 seconds and 110 steps, including one flight of stairs, to get from my workstation all the way to the game room — believe me, I counted. Unfortunately, due to the unbending laws of physics, it takes exactly the same time to go back, frustrated that it was already being used.</p><p>And so, after one too many of these time-wasting, context-switching, hope-shattering trips, I started asking myself one simple question — what if we could just <em>ask</em> if someone was already using the game room?</p><h4>Weapon(s) of Choice — Slack Bot, PIR Sensor and a Raspberry Pi</h4><p>Salvation came in the form of a Passive Infrared (PIR) motion sensor connected to a Raspberry Pi. The sensor data is transmitted to the Raspberry Pi, which in turn saves the data to a local DB. The Slack bot user then queries the DB, to find out if actual movement has been detected in the room recently.</p><p>The entire system looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/0*sz0zV5_55klaZdJT.png" /></figure><h4>I Sense a Disturbance in the Room</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*RdNveSlQDnkdH1M_.png" /></figure><p>A PIR sensor is just a fancy name for your run-of-the-mill motion detector, the ones you usually see mounted on walls of houses. Basically, the way it works is you have a sensor that measures infrared light, radiating from objects in its field of view. The term <em>passive</em> in this instance refers to the fact that PIR devices do not generate or radiate energy for detection purposes, they work entirely by detecting infrared radiation, emitted by or reflected from objects (find more information about PIR sensors <a href="https://learn.adafruit.com/pir-passive-infrared-proximity-motion-sensor/how-pirs-work">here</a>).</p><h4>Easy as Pi</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*tIo06P44beVJOz26.jpg" /></figure><p>Raspberry Pi is a series of small single-board computers designed to promote the teaching of basic computer science in schools and in developing countries. It achieves this goal by being highly versatile, easy to setup and most important — very, VERY affordable, starting at only $5 (!) for the most <a href="https://www.raspberrypi.org/products/raspberry-pi-zero/">basic model</a> and $35 for the <a href="https://www.raspberrypi.org/products/raspberry-pi-3-model-b/">latest model</a>.</p><p>It has a lot of cool features, but maybe the most important one is the General Purpose Input/Output (GPIO) connector. These pins are a physical interface between the Pi and the outside world. This is what is going to be used in order to connect the motion sensor output to the Raspberry Pi.</p><h4>A Tale of Two Scripts</h4><p>Raspberry Pi can run all sorts of images on its MicroSDHC card, but the most prevalent one is its own flavor of Unix — <a href="https://www.raspberrypi.org/downloads/raspbian/">Raspbian</a>, which is what I installed to run the Python interpreter and the various dependencies and tools required for this project.</p><p>The solution was divided into two separate Python scripts, which run on startup — the motion detection and the slackbot script.</p><p>A local MySQL database is used to store information about the activity in the room, meaning a list of all previous sessions and a current active one, should one exist.</p><p>This is to allow for persistency and other possible future features, such as to determine when is the optimal time to go and play, or how long someone has already been using the room.</p><p>The motion detection script listens for movement (or lack thereof) and updates the database accordingly — opens a new game session if one is not already opened, extends active sessions and closes sessions after a certain period of non-movement.</p><p>The slackbot script handles Slack integration and queries the database to provide information about the room’s status, by checking if there are currently any active open sessions.</p><h4>Halt! Who Goes There?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*rX7pGfrCd-qFCVyV.png" /></figure><p>I used an <a href="https://www.mpja.com/download/31227sc.pdf">HC-SR501</a> motion sensor, which you can get online for $1. It only has three pins that you need to connect to the Raspberry Pi — GND (Ground), VCC (5V) and OUT (Data) — follow the diagram above to connect these (see the pin layout <a href="https://pinout.xyz/">here</a>).</p><p>The motion detection script communicates with the GPIO channels by listening on the data pin — marked as OBSTACLE_PIN in the code. This is done utilizing the RPi.GPIO package, which allows you to access the various pins on the board and get their current state (see documentation <a href="https://sourceforge.net/p/raspberry-gpio-python/wiki/Home/">here</a>).</p><p>Once the motion detection script starts running it goes in an endless loop, which is divided to one second intervals, and in each iteration the sensor input is read. Querying the sensor gives a binary response — a movement was detected or not.</p><p>Since the motion detector samples motion every second, it needs to be able to determine that an actual movement occurred not only in a particular second, but several times during a short period of time, to avoid false positives, i.e. detecting movement when there was none, and false negatives, i.e. not detecting movement when it should have.</p><p>So, in every iteration a “detection frame” is examined, which is basically a time window in which the number of seconds a movement was detected is counted. The number of seconds with movement, out of the total number of seconds in the time window, is compared to a predefined threshold.</p><p>If the number exceeds the threshold, then a movement was detected in the room. This allows control of the sensitivity of the detection, in addition to being able to adjust it physically on the sensor itself.</p><p>The following code demonstrates this logic:</p><pre># The physical pin number on the board connected to the DATA output pin of the motion sensor</pre><pre>OBSTACLE_PIN = 7</pre><pre># The maximum time in seconds allowed for a session to be inactive<br>MAX_SESSION_TIME_IN_SECONDS = 60<br># The time window in seconds we are looking for consecutive movement<br>DETECTION_FRAME_LENGTH_IN_SECONDS = 10<br># The threshold that determines if a movement was detected<br>DETECTION_INCIDENTS_THRESHOLD = 0.4</pre><pre>def setup():<br>   print_log(&#39;initializing sensor&#39;)<br>   GPIO.setmode(GPIO.BOARD)<br>   GPIO.setup(OBSTACLE_PIN, GPIO.IN)<br>   print_log(&#39;initializing db&#39;)<br>   time.sleep(10)<br>   # preparing tables, cleaning existing open sessions<br>   setup_database()</pre><pre>def is_motion_detected():<br>   return GPIO.input(OBSTACLE_PIN)</pre><pre>def loop():<br>   iteration_counter = 0<br>   detection_counter = 0.0<br>   print_log(&#39;started monitoring&#39;)<br>   # start monitoring for movement<br>   while True:<br>	# if the detection window is over<br>       if (iteration_counter &gt;= DETECTION_FRAME_LENGTH_IN_SECONDS):<br>           # if we saw enough movement during that window<br>	   # i.e. seconds with movement out of total window time is larger than the threshold<br>           if (detection_counter / iteration_counter &gt;= DETECTION_INCIDENTS_THRESHOLD):<br>               handle_motion_detected()<br>           else:<br>               handle_no_motion(DETECTION_FRAME_LENGTH_IN_SECONDS, MAX_SESSION_TIME_IN_SECONDS)<br>	   # reset counters for next detection window<br>           iteration_counter = 0<br>           detection_counter = 0.0<br>       # if a movement is detected in this particular second<br>       if (is_motion_detected()):<br>           detection_counter += 1<br>	    print_log(&#39;motion &#39; + str(detection_counter) + &#39; detected&#39;)<br>       iteration_counter += 1<br>       time.sleep(1)</pre><pre>Slacking Off</pre><pre>Below is a code snippet from the slackbot script, outlining the interaction with the aptly named PingPongBot - you will need to add a new bot user to your Slack team (see <a href="https://api.slack.com/bot-users">this</a> link for more information), after which you will receive a token to be used in your client:</pre><pre># these can be obtained when creating a new slack bot user<br># see instructions on how to do that here - <a href="https://my.slack.com/services/new/bot">https://my.slack.com/services/new/bot</a><br>SLACK_BOT_TOKEN = &#39;&lt;your slack bot token here&gt;&#39;<br>SLACK_BOT_NAME = &#39;&lt;your slack bot name here&gt;&#39;</pre><pre># aux method to get the slack bot id, required for authentication<br>def get_slack_bot_id(slack_client, bot_name):<br>   if __name__ == &quot;__main__&quot;:<br>       api_call = slack_client.api_call(&#39;users.list&#39;)<br>       if api_call.get(&#39;ok&#39;):<br>           users = api_call.get(&#39;members&#39;)<br>           for user in users:<br>               if &#39;name&#39; in user and user.get(&#39;name&#39;) == bot_name:<br>                   print(&quot;Bot ID for &#39;&quot; + user[&#39;name&#39;] + &quot;&#39; is &quot; + user.get(&#39;id&#39;))<br>                   return user.get(&#39;id&#39;)<br>       else:<br>           print(&#39;could not find bot user with the name &#39; + bot_name)<br>   return None</pre><pre># set up connection to Slack client<br>slack_client = SlackClient(SLACK_BOT_TOKEN)<br>BOT_ID = get_slack_bot_id(slack_client, SLACK_BOT_NAME)<br>AT_BOT = &#39;&lt;@&#39; + BOT_ID + &#39;&gt;&#39;</pre><pre># the actual command to look for when addressing the bot in chat<br>EXAMPLE_COMMAND = &#39;free&#39;</pre><pre># the actual logic when receiving a message in the chat<br>def handle_command(command, channel):<br>   response = &#39;Hello! I am the ping pong bot! Ask me if the table is free by typing - \&quot;free?\&quot;&#39;<br>   # parse the chat message and check if it contains the desired command<br>   if EXAMPLE_COMMAND in command.lower():<br>       if is_open_session_exists():<br>           response = &#39;Sorry, someone is playing right now... Try again later!&#39;<br>       else:<br>           response = &#39;Free to play! enjoy :)&#39;<br>   # return the response in the chat according to the result from the database      <br>   slack_client.api_call(&#39;chat.postMessage&#39;, channel=channel, text=response, as_user=True)</pre><pre># aux method to receieve messages from the chat<br>def parse_slack_output(slack_rtm_output):<br>   output_list = slack_rtm_output<br>   if output_list and len(output_list) &gt; 0:<br>       for output in output_list:<br>           if output and &#39;text&#39; in output and AT_BOT in output[&#39;text&#39;]:<br>               return output[&#39;text&#39;].split(AT_BOT)[1].strip().lower(), output[&#39;channel&#39;]<br>   return None, None</pre><pre>if __name__ == &quot;__main__&quot;:<br>   READ_WEBSOCKET_DELAY = 1<br>   # establishing connection to bot<br>   if slack_client.rtm_connect():<br>       print(SLACK_BOT_NAME + &#39; connected and running&#39;)<br>       # start listening for messages<br>       while True:<br>           command, channel = parse_slack_output(slack_client.rtm_read())<br>           if command and channel:<br>               handle_command(command, channel)<br>           time.sleep(READ_WEBSOCKET_DELAY)<br>   else:<br>       print(&#39;connection failed, invalid Slack token or bot ID?&#39;)<br>You can find the full project code on my <a href="https://github.com/amirkeren/pingpong-bot">github</a> page.</pre><h4>I Love It When a Plan Comes Together</h4><pre>This is what the sensor looks like taped to the floating ceiling of our game room (the Raspberry Pi is hidden from view):</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2M3JTOsW5zxstvfN.png" /></figure><pre>And finally, this is how it looks like when you ask the PingPongBot if the game room is free:</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*3k33zjs85uL0L09x.png" /></figure><h4>Not Only for the Office</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*D4AhjvU5G7XmmLmN.jpg" /></figure><pre>Hopefully by now you realize how simple and easy it is to implement an entire motion detection system using accessible and affordable parts.</pre><pre>There are many other sensors that you can incorporate in your next project, turning your office/home into a smart one. For example, a <a href="https://www.ebay.com/sch/i.html?_odkw=pir+motion+sensor+raspberry+pi&amp;_sop=15&amp;LH_BIN=1&amp;_osacat=0&amp;_from=R40&amp;_trksid=p2045573.m570.l1313.TR0.TRC0.H0.Xtemperature+sensor+raspberry+pi.TRS0&amp;_nkw=temperature+sensor+raspberry+pi&amp;_sacat=0">temperature and humidity sensor</a> can be used as input to <a href="http://www.ibroadlink.com/rm/">control your AC</a>, or a <a href="https://www.ebay.com/sch/i.html?_from=R40&amp;_sacat=0&amp;LH_BIN=1&amp;_nkw=Rain+Detection+Module+raspberry+pi&amp;_sop=15">rain detection module</a> to let you know when to turn off your sprinklers or close the electric windows.</pre><pre>Well, writing this post really made me parched. Hmm... I wish there was a way to know if there are beers left in the kitchen...</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3543168858bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/diy-engineering-your-company-culture-3543168858bb">DIY: Engineering Your Company Culture</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Calculating Git Version]]></title>
            <link>https://medium.com/taboola-engineering/calculating-git-version-cdff67f6f07?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/cdff67f6f07</guid>
            <category><![CDATA[system]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Tue, 26 Dec 2017 13:42:54 GMT</pubDate>
            <atom:updated>2018-08-08T14:33:57.753Z</atom:updated>
            <content:encoded><![CDATA[<p>by Yuri Goikhman</p><p>Hello Git user. In this blog post I will discuss a technique for a unique version calculation for every Git commit.</p><p>You may ask why we need this, after all every commit in Git is identified by a unique sha1 hash. That’s right, let’s take 2 commits, 4bd92c9 and f5fc029, use their sha1 hash as a version and perform a simple A/B test. The test showed that 4bd92c9 is preferred to f5fc029.</p><p>If this is the case, how can we tell:</p><ul><li>Which version is newer?</li><li>If 4bd92c9 is included in f5fc029, or vice versa?</li><li>What branch they were built from?</li></ul><p>It seems we need an alternative.</p><p>The common standard for the versioning is a <a href="https://semver.org/">SemVer</a> scheme.</p><p>We will use its parts as follows:</p><ul><li>Major — manual increment</li><li>Minor — every released feature will increment the minor</li><li>Patch — will always be 0</li></ul><p>Now let’s take a look at our Git graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*Q9unI-nUEVgVy-zT.png" /></figure><p>We will give a version number for every mentioned commit:</p><ul><li><strong>M1–1.0.0:</strong> our init major version</li><li><strong>M2–1.1.0:</strong> some new feature released</li><li><strong>M3–1.2.0:</strong> feature B released</li><li><strong>M4–1.3.0:</strong> feature A released</li></ul><p>But what about A1,B1? We need to give them a version number as well in order to identify these builds for various purposes like automation test, deploying on GA environment, etc. According to the semver rules we need to mark these versions as a next minor pre-release, so we get:</p><ul><li>A1–1.1.0-A-5</li><li>B1–1.2.0-B-3</li></ul><p>Let’s put the version numbers in the graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*0GADfjgiRn2NufkM.png" /></figure><p>Now let’s automate it by using the following logic. If you are on the master branch calculate the version by:</p><pre>latest=$(git tag  -l --merged master --sort=&#39;-*authordate&#39; | head -n1)</pre><pre>latest=$(git tag  -l --merged master --sort=&#39;-*authordate&#39; | head -n1)<br>semver_parts=(${latest//./ })<br>major=${semver_parts[0]}<br>minor=${semver_parts[1]}<br>patch=${semver_parts[2]}</pre><pre>version=${major}.$((minor+1)).${patch}</pre><pre>Put the tag on a HEAD</pre><pre>If you are on the feature branch just calculate the version (without putting a tag) by:</pre><pre>latest=$(git tag  -l --merged master --sort=&#39;-*authordate&#39; | head -n1)</pre><pre>latest=$(git tag  -l --merged master --sort=&#39;-*authordate&#39; | head -n1)<br>semver_parts=(${latest//./ })<br>major=${semver_parts[0]}<br>minor=${semver_parts[1]}<br>patch=${semver_parts[2]}</pre><pre>branch=$(git rev-parse --abbrev-ref HEAD)</pre><pre>count=$(git rev-list HEAD ^${latest} --ancestry-path ${latest} --count)</pre><pre>version=${major}.${minor}.${patch}-${branch}-${count}</pre><pre>Combining all these together we get the following script:</pre><pre>#!/bin/bash</pre><pre>branch=$(git rev-parse --abbrev-ref HEAD)<br>latest=$(git tag  -l --merged master --sort=&#39;-*authordate&#39; | head -n1)</pre><pre>semver_parts=(${latest//./ })<br>major=${semver_parts[0]}<br>minor=${semver_parts[1]}<br>patch=${semver_parts[2]}</pre><pre>count=$(git rev-list HEAD ^${latest} --ancestry-path ${latest} --count)<br>version=&quot;&quot;</pre><pre>case $branch in<br>   &quot;master&quot;)<br>      version=${major}.$((minor+1)).0<br>      ;;<br>   &quot;feature/*&quot;)<br>      version=${major}.${minor}.${patch}-${branch}-${count}<br>      ;;<br>   *)<br>      &gt;&amp;2 echo &quot;unsupported branch type&quot;<br>      exit 1<br>      ;;<br>esac</pre><pre>echo ${version}<br>exit 0</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdff67f6f07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/calculating-git-version-cdff67f6f07">Calculating Git Version</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing a Homepage Widget with AB Testing]]></title>
            <link>https://medium.com/taboola-engineering/optimizing-a-homepage-widget-with-ab-testing-eef135e41221?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/eef135e41221</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[testing-cycle]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Wed, 15 Nov 2017 15:17:19 GMT</pubDate>
            <atom:updated>2018-08-08T14:35:15.790Z</atom:updated>
            <content:encoded><![CDATA[<p>by Noa Bar Or</p><p>As part of the optimization team at Taboola, we are constantly working with publishers and conducting AB tests, to find the best user interface (UI) for our widget. In doing so, we improve sponsored content (SC) revenue per mille (RPM).</p><p>When we dealt with a very large news site in India, as part of an ongoing optimization, we needed to get a bit creative.</p><p>We were already implementing our best practices for our widget on the publisher homepage — so we came up with a different solution. We tested a native UI, one that would help blend the widget into the design of the page.</p><h4>User Behaviour on the Homepage</h4><p>Unlike article pages, where users come to read an article, and then sometimes leave after finish reading it, a user who enters a homepage directly will sometimes scan it from top to bottom to find something to read. When they finish reading everything interesting a Taboola widget with additional reading material at the bottom of the homepage could be a good idea.</p><p>This behaviour of scanning the homepage until reaching the bottom of the page, can be related to our brain’s tendency to need to finish what it starts, as Russian psychologist Bluma Zeigarnik talks about in a 1927 study.</p><p>Our widget location on the client’s homepage was the 3rd row from the bottom of the page — a sometimes unattractive location that can be easily missed when scrolling down.</p><p>So, our first step was to test placing our widget at the far bottom of the page. That way users will see it when they reach the bottom. This generated a conclusive 4 percent uplift in SC RPM — but we didn’t stop there…</p><h4>AB Testing a Dark Background</h4><p>Atop the new location, there was a photo gallery surrounded by a dark background.</p><p>Since a dark background on a white page causes high contrast that attracts the eye, we decided it could be interesting to add the dark background to our widget as well. We wanted to mimic the exact UI, font style, text color, etc.</p><p>This spiked the SC RPM by 40 percent.</p><h4>Trying Out a Hero Image</h4><p>After a few months, we were suddenly faced with a new challenge — The client changed the entire homepage design in order to bring in more content cards. The dark gallery above was completely changed, and our widget was a complete stranger on the site.</p><p>Since mimicking site design had worked previously, we again A/B tested a design, like the unit that was placed above ours — a unit with one large image, commonly referred to in the industry as a Hero image, and 6 smaller images to the right. There was also a subtle gray shadow around it. This generated a 20 percent uplift.</p><figure><img alt="Variant Performance by Week Taboola" src="https://cdn-images-1.medium.com/max/976/0*wXCx9zJnYan4Hk3H.png" /></figure><h4>Size matters? Adding a control variant</h4><p>After seeing the SC RPM increase, and examining the homepage layout, we realized we had missed an important step along the way.</p><p>By mimicking the Hero unit, we increased the size of our widget by almost 50 percent. So, what caused the RPM uplift? The size, or the UI?</p><p>If we tested Taboola’s best practice widget, but with the same size as the Hero unit, would it perform better?</p><p>To measure this, we added two additional variants:</p><p>A regular 4x2 widget — almost the same size as the Hero unit — and a variation of it, with a light shadow surrounding the widget, mimicking the shadow surrounding the client’s unit. Below is an example:</p><h4><strong>Variant 1: 4x2 widget — generated 6 percent uplift</strong></h4><figure><img alt="Variant Performance by Week Taboola Dark background" src="https://cdn-images-1.medium.com/max/906/0*1xu3QKk07iThG10R.png" /></figure><h4><strong>Variant 2: 4x2 widget, with surrounding shadow to mimic the client’s unit — generated 7 percent uplift</strong></h4><figure><img alt="Variant Performance by Week Taboola Dark Background Subtle Shadow" src="https://cdn-images-1.medium.com/max/910/0*wFnb6pHvj6lvsZjd.png" /></figure><p>Both variants, though successful in results, gave a smaller uplift compared to the variant that looked most like the site’s design. This proved that it wasn’t just increasing the size of the widget that helped.</p><h4>Outcome:</h4><p>After viewing the results, the client agreed to change the previous, dark background unit to the new Hero design.</p><p>Before:</p><figure><img alt="Taboola Widget" src="https://cdn-images-1.medium.com/max/1024/0*ciQzEiM59MVOGrgN.png" /></figure><p>After:</p><figure><img alt="Taboola Widget" src="https://cdn-images-1.medium.com/max/1002/0*relaZhGxfqXwEB4b.png" /></figure><p>After keeping the old UI on 5% traffic in order to measure the long term effect of the new UI, we still saw a conclusive and positive uplift of 12%</p><figure><img alt="Winning Variant Performance Before and After" src="https://cdn-images-1.medium.com/max/947/0*B_C45XqD4IOknqkD.png" /></figure><h4>AB Testing Conclusion</h4><p>Using A\B testing to compare between two different designs is an amazing tool that can help Taboola explore and identify what works best for publishers and what should be avoided, thus improving performance.</p><p>Don’t be afraid to take chances and test new and innovative designs.</p><p>You’ll never know unless you try.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eef135e41221" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/optimizing-a-homepage-widget-with-ab-testing-eef135e41221">Optimizing a Homepage Widget with AB Testing</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Out Jenkins Based CI with Docker and Nomad]]></title>
            <link>https://medium.com/taboola-engineering/scaling-out-jenkins-based-ci-with-docker-and-nomad-e7dfa6977828?source=rss----a30fa12bd527---4</link>
            <guid isPermaLink="false">https://medium.com/p/e7dfa6977828</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[system]]></category>
            <dc:creator><![CDATA[Taboola Engineering]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 06:49:52 GMT</pubDate>
            <atom:updated>2018-08-08T14:35:58.833Z</atom:updated>
            <content:encoded><![CDATA[<p>by Anton Weiss</p><p>The existing CI processes in Taboola are quite demanding — a full product build includes 150 maven modules accounting for about 20000 unit tests. Beside running builds from master branch there are also builds for all feature branches and patch releases. All in all accumulating to more than a hundred builds per day. Some of the executed tests are pretty heavy on CPU and memory, performing resource-intensive data crunching. As you can imagine — all this requires substantial CI infra horse power. Which it definitely possesses:</p><p>Taboola’s Jenkins cluster currently has 35+ Jenkins slaves, each with 20 to 40 CPU cores and at least 100 Gb memory. Each slave runs 5 to 10 executors. All in all — a powerful CI/CD factory.</p><p>But even with all this power — there are limitations. On the day of the weekly release the volume of builds peaks and we sometimes find ourselves starving for resources. Builds line up in queues wasting valuable developer time and causing unnecessary stress.</p><p>Further enhancing the problem is the fact that some of the build slaves have specific system configuration and can’t be used for all kinds of processes. We love and nurture our pet servers but treating them as cattle is so much more productive.</p><p>So there’s room for improvement! Add to this the fact that Taboola has been growing fast and we can only anticipate the build volume to continue accelerating in the future — and finding that improvement becomes a necessity.</p><p>Taboola development infra (should we say DevOps?) team is constantly looking at ways of enhancing the overall productivity and eliminating bottlenecks in the delivery process. Smart test execution algorithms have been developed breaking down the tests into variable size chunks to optimize the load distribution.</p><p>But we didn’t want to stop there. On a quest to maximize effective resource utilization we started looking at using containers. It’s no secret that in the last couple of years they’ve become the single hottest piece of technology everyone is raving about. They bring all the benefits of workload portability, resource isolation and deterministic deployments. And yes — they are a perfect match for CI/CD — allowing to quickly spin up and tear down pristine and versioned build and test environments. Moreover — they can help with turning our Jenkins slave cluster into a real herd of cattle, where all we need from the slave nodes is to have Docker installed — all the rest of configuration will be taken care of by container images.</p><p>As with all improvement initiatives involving new tech — we started out small. Installed the Jenkins Docker plugin and set out to create the build slave image equipped with all the needed tools. The physical build slaves are managed by Foreman+Puppet — so puppet manifests were a great help in getting this configured fast.</p><p>Once the slave image was ready we uploaded it to Taboola’s Artifactory integrated Docker registry and tried running builds from Jenkins with the use of Jenkins Docker plugin. Another day of tweaking was spent trying to get the ssl certificates working but in the end we achieved great success — the build and tests were passing! Unsurprisingly it took hours to complete the full cycle as at this stage we were only using one physical Docker host for our games. As depicted in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*ku0v5lQE2M2gGUGz.png" /></figure><p>Now that we’ve proven the build could be executed in a container the time came to decide on a container scheduling solution. Container schedulers (sometimes also referred to as orchestrators) are responsible for receiving container workload tasks and distributing them across a cluster of container hosts. A number of solutions exist with the most renowned being Docker Swarm (by Docker Inc), Kubernetes (from Google) and Marathon (from Mesosphere). But we didn’t go with one of those. Instead we chose to try a tool that comes from a company we’ve learnt to rely on — Hashicorp. Taboola has been using their service discovery tool — Consul in production for quite some time, and we heard good things about their scheduler/cluster manager named Nomad. Beside coming from Hashicorp, it also boasts the following attractive features (as stated on the official website):</p><ul><li><strong>Flexible</strong> <strong>Workloads:</strong> Nomad can schedule containers, but also standalone applications and batch tasks across a cluster of hosts.</li><li><strong>Operational Simplicity: </strong>Nomad ships as a single binary, both for clients and servers, and requires no external services for coordination or storage. Nomad is distributed and highly available, and combines resource management and scheduling into a single system for simplicity.</li><li><strong>Built for Scale:</strong> Nomad was designed from the ground up to support global scale infrastructure. Nomad is distributed and highly available, using both leader election and state replication to provide availability in the face of failures. Nomad is optimistically concurrent, enabling all servers to participate in scheduling decisions which increases the total throughput and reduces latency to support demanding workloads.</li></ul><p>The decision to go with Nomad was taken in collaboration with Taboola’s production operations team. They were also looking for a cluster management solution and it was decided we will all be better off by joining our research efforts.</p><p>Additional motivation was provided by this blog post which showed us integration between Nomad and Jenkins was already taken care of: <a href="http://www.ivoverberk.nl/scalable-ci-cd-with-nomad-and-jenkins/">http://www.ivoverberk.nl/scalable-ci-cd-with-nomad-and-jenkins/</a></p><p>All we had to do is connect the dots! We’ve created a Nomad cluster, installed the Jenkins plugin and tried to schedule our first Nomad-based docker slave. Now it looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*OaE-lrx__wI4rdU3.png" /></figure><p>But of course it’s never that easy.</p><p>One of the issues we had to deal with was the fact that current builds rely on 1) shared workspaces and 2) configuration written to an NFS mount.</p><p>Nomad in its current configuration doesn’t allow mounting specific host folders into containers. This makes some sense — if you want real dynamic cluster scheduling you don’t want to depend on data sitting on the host. But that also meant we had to mount the NFS from inside the container at bringup. While fully possible in Nomad — this wasn’t doable through the Jenkins plugin.</p><p>But that’s what’s so great about open source. You need functionality — you’re free to build it!</p><p>We extended the plugin to support alternative container bringup commands, verified it was working and submitted a pull request to the official Github repo. The maintainer — Ivo Verberk, was quick to react — he merged the change, added some improvements of his own and released a new <a href="https://github.com/jenkinsci/nomad-plugin/tree/nomad-0.2">version</a>.</p><p>In the meantime we encountered a new issue. Containers weren’t getting scheduled fast enough. Sometimes Jenkins would idly wait for up to a couple of minutes before deciding to schedule a new slave. Now multiply this by 50 test jobs and you find yourself waiting for hours… The investigation of Jenkins slave scheduling strategy wasn’t easy — it is not really documented and the examples online are rare and sparse. The real breakthrough came from reading the code of <a href="https://wiki.jenkins-ci.org/display/JENKINS/Yet+Another+Docker+Plugin">Yet-Another-Docker-Plugin</a>. It revealed that there is a better way — extending Jenkins <a href="http://javadoc.jenkins.io/hudson/slaves/NodeProvisioner.Strategy.html">NodeProvisioner.Strategy</a> . A couple of days of tweaking and voilà</p><p>- we had slaves coming up in seconds, not minutes — just as one would expect from containers.</p><p>This has also been merged upstream and is now part of <a href="https://wiki.jenkins-ci.org/display/JENKINS/Nomad+Plugin">jenkins-nomad-plugin</a> release 0.3</p><p>So the end result is a cluster of 4 Nomad nodes capable of running 15–20 Jenkins slaves whenever needed.</p><p>We’re still sorting out some stability and performance issues with this setup. As mentioned — Taboola testing suite is very CPU and memory-intensive and we occasionally see some jobs strangling others when scheduled on the same node. Probably comes down to careful resource quota tuning. We still need to find the perfect balance between stability and efficient resource utilization. Of course any tips on this will be most welcome.</p><p><strong>The conclusion:</strong></p><p>Nomad and Docker are still young technologies. Together they can provide great benefits for your software delivery tooling, but there’s tweaking involved. As always with open source — you can enjoy it the most if you’re prepared to give back. We’ll be happy if you decide to use Jenkins with Nomad and benefit from our contribution to the plugin. Technology grows faster with community support — so do let us know if we can help you, or if you can help us. Building software together is much more fun. After all that’s what DevOps is all about.</p><p>Happy delivering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7dfa6977828" width="1" height="1" alt=""><hr><p><a href="https://medium.com/taboola-engineering/scaling-out-jenkins-based-ci-with-docker-and-nomad-e7dfa6977828">Scaling Out Jenkins Based CI with Docker and Nomad</a> was originally published in <a href="https://medium.com/taboola-engineering">Taboola Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>