<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[YellowBlog - Medium]]></title>
        <description><![CDATA[Blog written by YellowRoad — When Machine Learning and Big Data meet Business - Medium]]></description>
        <link>https://medium.com/yellowblog?source=rss----4981ab970712---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>YellowBlog - Medium</title>
            <link>https://medium.com/yellowblog?source=rss----4981ab970712---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:12:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/yellowblog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unsupervised sentence representation with deep learning]]></title>
            <link>https://medium.com/yellowblog/unsupervised-sentence-representation-with-deep-learning-104b90079a93?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/104b90079a93</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[yonatan hadar]]></dc:creator>
            <pubDate>Sun, 08 Apr 2018 13:24:11 GMT</pubDate>
            <atom:updated>2018-04-08T13:24:10.748Z</atom:updated>
            <content:encoded><![CDATA[<p>In recent years, the field of natural-language processing (NLP) has gained a major boost in performance thanks to the technique of representing words as a continuous vector instead of a sparse one-hot encoded vector (Word2Vec).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXNXYfAqfLUeiDXPCo130w.png" /><figcaption>Word2Vec examples</figcaption></figure><p>While Word2Vec works very well and creates nice semantics like King — Man + Woman = Queen, sometimes we are not interested in the representation of a word but in the representation of a sentence.</p><p>In this post, we will review and share code examples for several unsupervised, deep-learning methods of sentence representation. We will present their effectiveness as a pre-processing step for a specific text classification task.</p><h3>The Classification Task</h3><p>The data that we used to present the different sentence representation methods is based on 10,000 news articles that we scraped from the World Wide Web. The task is to classify each article into one of 10 possible topics (the data has topic labels, and is therefore a supervised task). For the sake of the presentation, I will use a logistic regression model, each time using a different pre-processing representation method that we will apply to the article headline.</p><h3>Baseline model — Average Word2Vec</h3><p>We will start with a very simple baseline. We will represent the headline by averaging the headline words in their Word2Vec representation. As previously mentioned, Word2Vec is a machine-learning method for representing words as vectors. The Word2Vec model is trained by predicting words close to the target word with a shallow neural network. You can read more about how the algorithm works <a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/">here</a>.</p><p>We can train our own Word2Vec with <a href="https://radimrehurek.com/gensim/">Gensim</a> but in this example we will use a Google pre-trained Word2Vec model that was built based on Google news data. After representing each word as a vector, we will represent a sentence (the headline) as an average of its words (vectors) and run logistic regression for classification of the article’s category.</p><pre>#load data and Word2vec model<br>df = pd.read_csv(&quot;news_dataset.csv&quot;)<br>data = df[[&#39;body&#39;,&#39;headline&#39;,&#39;category&#39;]]<br>w2v = gensim.models.KeyedVectors.load_word2vec_format(&#39;/GoogleNews-vectors-negative300.bin&#39;, binary=True)</pre><pre>#Build X and Y<br>x = np.random.rand(len(data),300)<br>for i in range(len(data)):<br>    k = 0<br>    non = 0<br>    values = np.zeros(300)<br>    for j in data[&#39;headline&#39;].iloc[i].split(&#39; &#39;):<br>        if j in w2v:<br>            values+= w2v[j]<br>            k+=1<br>    if k &gt; 0:<br>        x[i,:]=values/k<br>    else: non+=1<br>y = LabelEncoder().fit_transform(data[&#39;category&#39;].values)</pre><pre>msk = np.random.rand(len(data)) &lt; 0.8<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]</pre><pre>#Train the model<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>Our baseline average Word2Vec model achieved an accuracy score of <strong>68%</strong>. This is good, but let’s see if we can do better.</p><p>The average Word2Vec method has two main weaknesses: it is a bag-of-words model that doesn’t relate to the word order, and all the words are treated with the same weight. We will try to address these issues in the next methods by leveraging RNN architectures for sentence representation.</p><h3>Autoencoder</h3><p>An autoencoder is an unsupervised deep learning model that attempts to copy its input to its output. The trick of autoencoders is that the dimension of the middle-hidden layer is lower than that of the input data. Thus, the neural network must represent the input in a smart and compact way in order to reconstruct it successfully. In many cases, using the autoencoder for feature extraction has proved to be very effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*HIBRgSV2ePFtCSOF_lgPCQ.png" /></figure><p>Our autoencoder is a simple Sequnce2Sequence architecture built from an input layer followed by an embedding layer, an LSTM layer, and a softmax layer. Both the input and the output of the entire architecture are the headline, and we will use the output of the LSTM to represent the headline. After getting the representation from the encoder, we will use logistic regression to predict categories. In order to gain more data, we will train the autoencoder with all the sentences in the articles and not just the headline.</p><pre>#parse all sentences<br>sentenses = []<br>for i in data[&#39;body&#39;].values:<br>    for j in nltk.sent_tokenize(i):<br>        sentenses.append(j)</pre><pre>#preprocess for keras<br>num_words=2000<br>maxlen=20</pre><pre>tokenizer = Tokenizer(num_words = num_words, split=&#39; &#39;)<br>tokenizer.fit_on_texts(sentenses)<br>seqs = tokenizer.texts_to_sequences(sentenses)<br>pad_seqs = []<br>for i in seqs:<br>    if len(i)&gt;4:<br>        pad_seqs.append(i)<br>pad_seqs = pad_sequences(pad_seqs,maxlen)</pre><pre>#The model<br>embed_dim = 150<br>latent_dim = 128<br>batch_size = 64</pre><pre>#### Encoder Model ####<br>encoder_inputs = Input(shape=(maxlen,), name=&#39;Encoder-Input&#39;)<br>emb_layer = Embedding(num_words, embed_dim,input_length = maxlen, name=&#39;Body-Word-Embedding&#39;, mask_zero=False)<br># Word embeding for encoder (ex: Issue Body)<br>x = emb_layer(encoder_inputs)</pre><pre>state_h = GRU(latent_dim, name=&#39;Encoder-Last-GRU&#39;)(x)<br>encoder_model = Model(inputs=encoder_inputs, outputs=state_h, name=&#39;Encoder-Model&#39;)</pre><pre>seq2seq_encoder_out = encoder_model(encoder_inputs)</pre><pre>#### Decoder Model ####</pre><pre>decoded = RepeatVector(maxlen)(seq2seq_encoder_out)<br>decoder_gru = GRU(latent_dim, return_sequences=True, name=&#39;Decoder-GRU-before&#39;)<br>decoder_gru_output = decoder_gru(decoded)</pre><pre>decoder_dense = Dense(num_words, activation=&#39;softmax&#39;, name=&#39;Final-Output-Dense-before&#39;)<br>decoder_outputs = decoder_dense(decoder_gru_output)</pre><pre>#### Seq2Seq Model ####</pre><pre>#seq2seq_decoder_out = decoder_model([decoder_inputs, seq2seq_encoder_out])<br>seq2seq_Model = Model(encoder_inputs,decoder_outputs )</pre><pre>seq2seq_Model.compile(optimizer=optimizers.Nadam(lr=0.001), loss=&#39;sparse_categorical_crossentropy&#39;)</pre><pre>history = seq2seq_Model.fit(pad_seqs, np.expand_dims(pad_seqs, -1),<br>          batch_size=batch_size,<br>          epochs=5,<br>          validation_split=0.12)</pre><pre>#Feature extraction<br>headlines = tokenizer.texts_to_sequences(data[&#39;headline&#39;].values)<br>headlines = pad_sequences(headlines,maxlen=maxlen)x = encoder_model.predict(headlines)</pre><pre>#classifier<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>We achieved an accuracy score of 60%, which is actually worse than our baseline. We can probably improve this score by optimizing the hyperparameters, increasing the number of training epochs, or training the model on more data.</p><h3>Language model</h3><p>Our second method is training a language model to represent our sentences. A language model describes the probability of a text existing in a language. For example, the sentence “I like eating bananas” would be more probable than “I like eating convolutions.” We train a language model by slicing windows of n words and predicting what the next word will be in the text. You can read more about language modeling with RNN <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">here</a>. By building a language model, we gain an understanding of how “journalistic English” is built, and the model should be capable of focusing on important words in its representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*b5rfWo2gIdAJ7Jj9XF8o4g.png" /></figure><p>Our architecture is similar to the autoencoder architecture, but instead of predicting a sequence of words, we will only be predicting one word. The input will contain windows of 20 words from the articles, and the label will be the 21st word. After training the language model, we will take the headline representation from the LSTM output hidden state and run logistic regression to predict categories.</p><pre>#Building X and Y<br>num_words=2000<br>maxlen=20<br>tokenizer = Tokenizer(num_words = num_words, split=&#39; &#39;)<br>tokenizer.fit_on_texts(df[&#39;body&#39;].values)<br>seqs = tokenizer.texts_to_sequences(df[&#39;body&#39;].values)</pre><pre>seq = []<br>for i in seqs:<br>    seq+=i<br>    <br>X = []<br>Y = []<br>for i in tqdm(range(len(seq)-maxlen-1)):<br>    X.append(seq[i:i+maxlen])<br>    Y.append(seq[i+maxlen+1])</pre><pre>X = pd.DataFrame(X)<br>Y = pd.DataFrame(Y)<br>Y[0]=Y[0].astype(&#39;category&#39;)<br>Y =pd.get_dummies(Y)</pre><pre>#Buidling the network<br>embed_dim = 150<br>lstm_out = 128<br>batch_size= 128</pre><pre>model = Sequential()<br>model.add(Embedding(num_words, embed_dim,input_length = maxlen))<br>model.add(Bidirectional(LSTM(lstm_out)))<br>model.add(Dense(Y.shape[1],activation=&#39;softmax&#39;))<br>adam = Adam(lr=0.001, beta_1=0.7, beta_2=0.99, epsilon=None, decay=0.0, amsgrad=False)<br>model.compile(loss = &#39;categorical_crossentropy&#39;, optimizer=adam)<br>model.summary()</pre><pre>print(&#39;fit&#39;)<br>model.fit(X, Y, batch_size =batch_size,validation_split=0.1, epochs = 5,  verbose = 1)</pre><pre>#Feature extraction<br>headlines = tokenizer.texts_to_sequences(data[&#39;headline&#39;].values)<br>headlines = pad_sequences(headlines,maxlen=maxlen)<br>inp = model.input                                          <br>outputs = [model.layers[1].output]      <br>functor = K.function([inp]+ [K.learning_phase()], outputs )<br>x = functor([headlines, 1.])[0]</pre><pre>#classifier<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>We achieved an accuracy score of <strong>72%. </strong>This is better than our baseline, but let’s see if we can make it better still.</p><h3>Skip-Thought Vectors</h3><p>In the skip-thought <a href="https://arxiv.org/abs/1506.06726">paper</a> from 2015, the authors took the same intuition from the language model. However, in skip thought, instead of predicting the next word, we are predicting the previous and the next sentence. This gives the model more context for the sentence, thus we can build better representations of sentences. For more information about the model, check out <a href="https://medium.com/@sanyamagarwal/my-thoughts-on-skip-thoughts-a3e773605efa">this</a> great blog post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLPR8O3fT0NRAvbkx0op5A.png" /><figcaption>Example from the skip thought <a href="https://arxiv.org/abs/1506.06726">paper</a></figcaption></figure><p>We will build a Seq2Seq architecture similar to the autoencoder architecture, but with two main differences. First, we will have two output layers with LSTMs — one for the previous sentence and one for the next sentence. Second, we will use teacher forcing in the output LSTMs. This means that instead of giving the output LSTM the previous hidden state only, we will also give it the real previous word (you can see an illustration of the input in the picture above, in the bottom line of the outputs).</p><pre>#Build x and y<br>num_words=2000<br>maxlen=20</pre><pre>tokenizer = Tokenizer(num_words = num_words, split=&#39; &#39;)<br>tokenizer.fit_on_texts(sentenses)<br>seqs = tokenizer.texts_to_sequences(sentenses)<br>pad_seqs = pad_sequences(seqs,maxlen)</pre><pre>x_skip = []<br>y_before = []<br>y_after = []</pre><pre>for i in tqdm(range(1,len(seqs)-1)):<br>    if len(seqs[i])&gt;4:<br>        x_skip.append(pad_seqs[i].tolist())<br>        y_before.append(pad_seqs[i-1].tolist())<br>        y_after.append(pad_seqs[i+1].tolist())</pre><pre>x_before = np.matrix([[0]+i[:-1] for i in y_before])<br>x_after =np.matrix([[0]+i[:-1] for i in y_after])<br>x_skip = np.matrix(x_skip)<br>y_before = np.matrix(y_before)<br>y_after = np.matrix(y_after)</pre><pre>#Building the model<br><br>embed_dim = 150<br>latent_dim = 128<br>batch_size = 64</pre><pre>#### Encoder Model ####<br>encoder_inputs = Input(shape=(maxlen,), name=&#39;Encoder-Input&#39;)<br>emb_layer = Embedding(num_words, embed_dim,input_length = maxlen, name=&#39;Body-Word-Embedding&#39;, mask_zero=False)<br>x = emb_layer(encoder_inputs)</pre><pre>_, state_h = GRU(latent_dim, return_state=True, name=&#39;Encoder-Last-GRU&#39;)(x)</pre><pre>encoder_model = Model(inputs=encoder_inputs, outputs=state_h, name=&#39;Encoder-Model&#39;)</pre><pre>seq2seq_encoder_out = encoder_model(encoder_inputs)</pre><pre>#### Decoder Model ####<br>decoder_inputs_before = Input(shape=(None,), name=&#39;Decoder-Input-before&#39;)  # for teacher forcing</pre><pre>dec_emb_before = emb_layer(decoder_inputs_before)</pre><pre>decoder_gru_before = GRU(latent_dim, return_state=True, return_sequences=True, name=&#39;Decoder-GRU-before&#39;)<br>decoder_gru_output_before, _ = decoder_gru_before(dec_emb_before, initial_state=seq2seq_encoder_out)</pre><pre>decoder_dense_before = Dense(num_words, activation=&#39;softmax&#39;, name=&#39;Final-Output-Dense-before&#39;)<br>decoder_outputs_before = decoder_dense_before(decoder_gru_output_before)</pre><pre>decoder_inputs_after = Input(shape=(None,), name=&#39;Decoder-Input-after&#39;)  # for teacher forcing</pre><pre>dec_emb_after = emb_layer(decoder_inputs_after)</pre><pre>decoder_gru_after = GRU(latent_dim, return_state=True, return_sequences=True, name=&#39;Decoder-GRU-after&#39;)<br>decoder_gru_output_after, _ = decoder_gru_after(dec_emb_after, initial_state=seq2seq_encoder_out)</pre><pre>decoder_dense_after = Dense(num_words, activation=&#39;softmax&#39;, name=&#39;Final-Output-Dense-after&#39;)<br>decoder_outputs_after = decoder_dense_after(decoder_gru_output_after)</pre><pre>#### Seq2Seq Model ####</pre><pre>seq2seq_Model = Model([encoder_inputs, decoder_inputs_before,decoder_inputs_after], [decoder_outputs_before,decoder_outputs_after])</pre><pre>seq2seq_Model.compile(optimizer=optimizers.Nadam(lr=0.001), loss=&#39;sparse_categorical_crossentropy&#39;)</pre><pre>seq2seq_Model.summary()</pre><pre>history = seq2seq_Model.fit([x_skip,x_before, x_after], [np.expand_dims(y_before, -1),np.expand_dims(y_after, -1)],<br>          batch_size=batch_size,<br>          epochs=10,<br>          validation_split=0.12)</pre><pre>#Feature extraction<br>headlines = tokenizer.texts_to_sequences(data[&#39;headline&#39;].values)<br>headlines = pad_sequences(headlines,maxlen=maxlen)x = encoder_model.predict(headlines)</pre><pre>#classifier<br>X_train,y_train,X_test,y_test = x[msk],y[msk],x[~msk],y[~msk]<br>lr = LogisticRegression().fit(X_train,y_train)<br>lr.score(X_test,y_test)</pre><p>We achieved an accuracy score of <strong>74%. </strong>This is the best score we’ve achieved so far!</p><h3>Conclusion</h3><p>In this post, we reviewed three unsupervised methods for creating vector representations of sentences with RNNs and presented their effectiveness in solving a supervised task. The results of the autoencoder method were worse than the results for the baseline model (possibly because of the relatively small dataset used). The language and skip-thought vector models, both of which used context for predicting words or sentences, obtained the best result.</p><p>The many options available for improving the methods we’ve presented include tuning the hyperparameters, training using more epochs, using a pre-trained embedding matrix, changing the neural network architecture, and more. In theory, such advanced tuning work might change some of the results, but I believe the basic intuition of each pre-processing method can be achieved with the examples I’ve shared above.</p><p>Hope you enjoyed my post, and you’re more than welcomed to read and follow our blog at <a href="https://blog.myyellowroad.com/">YellowBlog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=104b90079a93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/unsupervised-sentence-representation-with-deep-learning-104b90079a93">Unsupervised sentence representation with deep learning</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating the Error of Regression Models: Some Real-Life Challenges and Practical Tips]]></title>
            <link>https://medium.com/yellowblog/evaluating-the-error-of-regression-models-some-real-life-challenges-and-practical-tips-817b921a9a62?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/817b921a9a62</guid>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[rmse]]></category>
            <category><![CDATA[aem]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mape]]></category>
            <dc:creator><![CDATA[Shahar Cohen]]></dc:creator>
            <pubDate>Mon, 02 Oct 2017 11:25:29 GMT</pubDate>
            <atom:updated>2017-10-02T11:25:28.602Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*iyZroKEGO_76KWpn955asg.png" /></figure><p>In regression, we predict (or estimate) the numerical value of an unknown quantity, denoted by <strong><em>y</em></strong>, as a function of some explaining features. The difference between the prediction and the actual value is the error. It is a numeric random variable, that might depend on the explaining features.</p><p>There are few common ways for estimating the performance and comparing several competing regression modes, which differ in their ways to measure error. Probably the most common measures are: Root Mean Square Error (RMSE), Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE). All these 3 measures are straight forward to compute and understand, however applying the right measure in real-life use cases is far from being trivial, and in most times, none of these measures cannot tell the entire story. In this post I describe some of the difficulties of evaluating the error of regression models, and suggest some practical tips on how to tackle these difficulties.</p><p><strong>The main limitations of RMSE, MAE and MAPE</strong></p><p>Here are two important limitations of RMSE, MAE and MAPE:</p><ol><li>Each of these measures is merely an average (or the square root of that average in the case of RMSE) of realizations of the test errors. The error is a numeric random variable and you cannot grasp the entire behavior of a random variable with a single aggregation of observations. The error is just a random variable, it is often a highly skewed random variable. When we predict skewed outcomes (like prices, incomes, user engagement, item sales and many many more) most probably that the error will also be skewed, meaning that in the majority of the cases the error is very small, but there are relatively rare examples that might have extremely large errors. When the error is highly skewed, the average often says nothing.</li><li>It might be the other side of the same coin, but it provides a different perspective. In average, all the observations equally contribute to the total error, regardless of the actual value of <strong><em>y</em></strong>. But error means something different, when associated to different values of <strong><em>y</em></strong>. For small values of <strong><em>y</em></strong> (e.g., near zero), we can accept relatively high percentage error, as long as the absolute error is small. For large values of <strong><em>y</em></strong> we may accept larger absolute errors, but not high percentage errors.</li></ol><p><strong>An Illustrative Example</strong></p><p>The following plot describes the predictions of a simple regression tree, on a common regression task (in which the objective is to predict sales prices).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*vDMAw-EiiSwTI8KprXSJnA.png" /></figure><p>Each sale in the plot is a single point, where the X-axis describe the actual price, and the y axis describes the prediction by the regression tree. The tree has 6 leaves that separates sales into price ranges almost perfectly. Since the major part of the sales is associated to very low prices, and due to the scale of the plot, the first 3 ranges cannot be recognized. Such sales ranges may be found, for example, in online advertisement applications.</p><p>Since the model is a regression tree, a single value (leaf average) is associated with each price level, but the range of the actual prices in each level is different. Where in the low-price sales the range is limited, the highest range describe sale with actual price range between ~1000 and 4000. Clearly this mean that most of the errors of the regression tree are expected to be quite small, with a few extreme outliers.</p><p>The following plot describes the distribution of the error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*n8NuWTlyHUdIcK6-ddOj7Q.png" /></figure><p>Most of the errors have relatively small absolute value (actually near zero in most of the cases) but there are few extreme errors. The MAE is 3.633, where the median absolute error is only 0.83. The really few large absolute errors significantly affect the overall performance. The RMSE in this case is 32.7(!!!), since it gives quadratic strength to the few extremely big errors. Clearly this number say nothing on the typical expected error.</p><p>Although there are some big errors, the model clearly differentiates well between low prices and high prices. The large errors are associated with large actual values of <strong><em>y</em></strong>. Predicting a price of 2,300 instead of actual of 1,800 may be acceptable, where predicting a price of 501, instead of an actual of 1, is unacceptable, yet both cases contribute an absolute error of 500 to the aggregation measure. This implies that measuring the percentage error might be a good idea.</p><p>The MAPE here, however, is 1,129%. Seems to be very bad. The maximal percentage error is over 3,000,000% and it is obtained when y = 3.106549e-05 and the prediction is 1.003 (an absolute error of ~1). The median percentage error is 51%, and the average percentage error when y &gt; 1 is only 32%.</p><p><strong>No single number can tell the entire story</strong></p><p><strong>Communicating performance measures to the business peers</strong></p><p>Where all the above peculiarity might be perfectly clear to data scientists (although we have met so many data scientists failing with these pitfalls), it becomes extremely hard to communicate the complex reality to business peers. No one will accept an average error of 1,000%. No one will accept mean absolute error of 3.5, knowing that most of the sales are closed with prices of less than 1. Failing to communicate the real meaning of a model performance to the business peers is the reason for a no-go decision to POCs that could have been a huge success.</p><p><strong>Practical tips on measuring the performance of regression models</strong></p><p>Here are few practical tips on measuring the performance of regression models.</p><ol><li>Really early in the process (before modeling the data, and even before preparing it for modeling) start a discussion with the business peers on what would make the project a success, business wise. Then try to translate this important business understanding into performance KPIs, and approve these KPIs with the customer. In many cases we find ourselves explaining to the customer the limitations in some suggested KPIs. This way, you don’t have surprises in the validation step, and you can use the right KPIs in the model training step.</li><li>Never use a single static aggregation as the sole performance measure.</li><li>Never use KPIs as black boxes. Remember that error is a numeric random variable, and strive to learn as much as possible on its distribution (is it symmetric? is it correlated with y? does it have extreme values? Is it correlated with any of the explaining features? Etc.). We often use the findings in the analysis of the error that allows us to improve the model (move from linear to non-linear, building an ensemble or a composition of models, and so on).</li><li>Always communicate the meaning of the performance measures to the business users, and help them use this information to tailor a better operational process (e.g., use the model only on specific pre-defined cases).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=817b921a9a62" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/evaluating-the-error-of-regression-models-some-real-life-challenges-and-practical-tips-817b921a9a62">Evaluating the Error of Regression Models: Some Real-Life Challenges and Practical Tips</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The “better than random” model]]></title>
            <link>https://medium.com/yellowblog/the-better-than-random-model-cb503712265e?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/cb503712265e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[heuristics]]></category>
            <category><![CDATA[methodology]]></category>
            <category><![CDATA[risk-management]]></category>
            <dc:creator><![CDATA[Shahar Cohen]]></dc:creator>
            <pubDate>Sun, 16 Jul 2017 14:24:47 GMT</pubDate>
            <atom:updated>2017-07-16T14:10:52.159Z</atom:updated>
            <content:encoded><![CDATA[<p>Although the last think you want to hear about a state of the art machine learning model is that it is just better than random, the way to a winning model often begins with a simple and naive solution, using the most basic modeling techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*rOKTXS0RjOA3DqbMur3QPA.jpeg" /></figure><p>In machine learning we always get new problems and need to figure out how to solve them. The solution involve research, which is a process that contains significant level of uncertainty. Even the best data science team might fail to deliver a valuable model, simply because valuable is sometimes infeasible. One effective way of hedging the risk is starting with a benchmark model. A good benchmark model have three characteristics:</p><p>i) It is simple and quick to build (quick means faster than a highly handcrafted model, in at least an order of magnitude).</p><p>ii) It has some positive contribution to the task that is being solved (even if it is still not accurate or valuable enough).</p><p>iii) It contains some hints about what is working well and what isn’t working well.</p><p>Due to the 2nd characteristic, we (at <a href="https://medium.com/u/2a6e2e6fd40f">YellowRoad</a>) sometimes call a good benchmark: the “better than random” model.</p><p>Now let’s be clear here. We will always search for a valuable model, business wise, and the “better than random” model is typically just a first baby step on the way. But, in many cases it is a very important baby step. Here are the main reasons for that:</p><ol><li>The “better than random” model is often a good Go/No-Go criterion: If you succeed in building a (simple and quick) “better than random” model, there are high chances that you can invest more, add more parameters (more data, more complexity) and get a significantly better model. It is not a guarantee, but assuming that you attained the “better than random” model easily (significantly faster than the entire project), by having it, you remove a significant amount of risk. We also sometimes see valuable machine learning solutions that were obtained after failing to quickly produce a “better than random” model. But, failing this way significantly increases the risk.</li><li>Running the “better than random” model on real data, characterizing the cases in which it works well, versus the cases in which it fails teaches us a lot about the task that is being solved and helps us on the way to a better model. That is, in many cases, the “better then random” model is not just a risk management step, but a constructive step on the way to a really successful model.</li><li>Ensemble: in machine learning it is sometimes possible to generate an ensemble of relatively weak models, and combine them into a single, surprisingly strong one. If you can generate a series of “better than random” models, you might gain a very good result by combining their sub-results.</li></ol><p>Next time when a customer or a manager asks you how good is the solution, do not answer that it is better than random. None data scientist will probably miss the joke, but let the “better than random” model be part of your working flow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb503712265e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/the-better-than-random-model-cb503712265e">The “better than random” model</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the most important step in a machine learning project?]]></title>
            <link>https://medium.com/yellowblog/what-is-the-most-important-step-in-a-machine-learning-project-3ee20a13bd20?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/3ee20a13bd20</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[methodology]]></category>
            <dc:creator><![CDATA[Shahar Cohen]]></dc:creator>
            <pubDate>Sun, 02 Jul 2017 14:43:58 GMT</pubDate>
            <atom:updated>2017-07-02T14:43:58.329Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*1JkT0ecHzExbtctMLieBsg.png" /></figure><p><a href="https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining">Cross Industry Standard Process for Data Mining - Wikipedia</a></p><p>The CRISP-DM is a common standard for machine-learning projects. Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation and Deployment. All these six steps of a machine-learning project are crucial. Quality issues in each step will directly affect the quality of the entire outcome. They are all important.</p><p>However, advising to many organizations on machine learning, and running even more such projects ourselves, we (at YellowRoad) came to a conclusion, that the most under-invested step in the process is Business Understanding. We see many companies discussing algorithms and technology, before understanding the business aspects of the task that they are solving. This is clearly not a good starting point.</p><p>We composed a series of questions that we use in any machine learning project that we get involved in, and we do not invest serious efforts in the following steps, until we have good answers to these questions. We find that practice to be extremely helpful.</p><p>Here are the questions:</p><ol><li>What are we trying to achieve, business wise? Why is it important?</li><li>What are the inputs and outputs for the task that we are trying to solve?</li><li>Given a hypothetical solution to that task, how would it affect our operations? (another way to ask this question: assuming that I have a perfect solution to your machine learning task, how will you use it?)</li><li>Do we already have the ability to act based on such solution, or do we also need to develop that ability? (if the ability is there, learn it carefully. If not, keep close contact with the team that is responsible for developing it)</li><li>How are we going to measure a suggested solution? (KPIs)</li><li>What would make it a success?</li><li>Do we have the input data available? How hard it is to extract it? Are we allowed to use it?</li><li>Are we experience in building similar solutions? Do we understand what it takes?</li><li>Do we have hard budget and timelines constraints?</li><li>Who will develop the solution? Do we have the required skills in house?</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ee20a13bd20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/what-is-the-most-important-step-in-a-machine-learning-project-3ee20a13bd20">What is the most important step in a machine learning project?</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tell us your Story?]]></title>
            <link>https://medium.com/yellowblog/whats-your-story-ddd43c7095fb?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/ddd43c7095fb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[services]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[YellowRoad]]></dc:creator>
            <pubDate>Mon, 26 Jun 2017 22:12:03 GMT</pubDate>
            <atom:updated>2017-06-29T06:22:43.454Z</atom:updated>
            <content:encoded><![CDATA[<h4>We want to hear from you…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*4Ovtyt4xDyymQnRBIKHXJg.png" /></figure><p>YellowBlog is our place, where @YellowRoad team members could share from their experience and knowledge with the community. However we are always eager to get feedbacks back from you.</p><p>We would like to hear feedbacks such as:</p><ul><li>Topics you would like us to cover in our Blog.</li><li>Any need for some quick advice.</li><li>You would like to check a potential collaboration.</li></ul><p>Feel free to respond here (in the comment section below) or contact us directly at: info@myyellowroad. We will do our best to respond quickly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddd43c7095fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/whats-your-story-ddd43c7095fb">Tell us your Story?</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beware of Data Leakage]]></title>
            <link>https://medium.com/yellowblog/beware-of-data-leakage-f6c307009ad9?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/f6c307009ad9</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[YellowRoad]]></dc:creator>
            <pubDate>Mon, 26 Jun 2017 14:45:55 GMT</pubDate>
            <atom:updated>2017-07-16T14:16:02.604Z</atom:updated>
            <content:encoded><![CDATA[<h4>Don’t say it will never happen to you</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*r3sCofWrCUm7BzP5fKeKSg.jpeg" /></figure><p>Every data scientist, I believe, knows the feeling. You are working on a supervised learning task. You learn all of the business nuances, collect data, extract features, spare an independent test set, train a model, and then test it over the test set. Boom! 99.6% of accuracy. After a split second of pure happiness and satisfaction, you realize that as tempting as such an outcome might be, something went wrong in the process. In many cases, the reason is Data Leakage.</p><p>Data leakage is the unintended existence of information in the training data that will not be available in production, allowing the machine learning algorithm to perform better in the research environment than it ever could in reality. This is a very common pitfall that many projects fall into, and it may appear in many different forms, some of which are incredibly hard to catch. The effect of data leakage on the actual performance of production models and on the total effort spent on data modeling could be huge.</p><p>In this post, we will review some of the more common data leakage patterns that you should be aware of and recommend the best practices which will help you avoid this malicious pitfall or at least eliminate its impact.</p><h4>1. Unfair inclusion of proxy variables</h4><p>In statistics, a proxy variable is a variable that has a high correlation with the target. So high, that we can use the proxy variable as a surrogate target. When there is a “legitimately” available proxy variable at hand, the supervised learning should be relatively easy. For example, using the annual expenses of a household is often a good proxy for that household’s income. If we have the expenses available, we can predict the income quite well.</p><p>However, in many cases, proxy variables are not real predicates, but some side effect of the actual target which exists in the training set only because the target values are also there. For example, let us assume that a churning customer, at a given company, must call the contact center, to churn. Naturally, churning calls tend to be longer than other calls. In this case, all of the churning customers will have a relatively long call to the contact center, and the duration of the recent call to the contact center is an unfair proxy variable. It is unfair because it is not a prior alert to the churn, but a direct outcome of that churn. In future predictions, we will not have that indication at hand at the prediction point (it becomes available only when it is too late).</p><p>When an unfair proxy affects all of the training and testing examples, it is typically easier to catch it. The impact of such a proxy might be trickier when it affects only some of the examples.</p><h4>2. Data that is available for research but will not be available for production</h4><p>Building up an initial supervised learning model is a research effort. Before building such a model you typically collect and prepare the input data. This preparation is a one-time effort (or at least an effort that you only make once in a while). As such, we often see companies gathering as much information as possible (e.g., attaching information from external sources), and running many ad hoc data enhancement efforts (e.g. manual data cleansing).</p><p>Production is a totally different environment. Data in production is fast and typically comes from production systems. It is often the case that some of the data that was used for training is not maintained in production. If that unavailable data contains significant information, its leakage to the training set will result in a poor performing production model.</p><p>A specific example of data that will not be available in production is proxy variables. For example, in the income prediction task, it might be the case that we only have the expense data for the households in the training set (thanks to local expense research) for which we also have the income, but we will not have that data for future households.</p><h4>3. Time series leakage</h4><p>Many supervised learning tasks are a time series: predicting the future based on information from the past. Analyzing a time series is challenging for many reasons — data leakage is one of them.</p><p>When modeling a time series, we sometimes see explaining features that describe the future. For example, a country’s GDP per capita can correlate with many economic measures. Using GDP to make a prediction might be tricky because the estimated GDP for a specific period typically get published sometime after that period ends. Predictions related to that period typically require some time before the period begins. So, using GDP as a predictor is simply impossible, because it is information from the future.</p><p>Another form of leakage that can occur in time series is mixing the past and the future. The training set includes records from different timestamps. We want to use past observations to predict future ones. Not vice versa. If we randomly split the input set into train and test, we are mixing the past and the future, and it might lead to wrong performance estimations.</p><h4><strong>4. Train to test leakage</strong></h4><p>The common supervised learning practice involves sparing an independent subset of data for testing. Train to test leakage occurs when some of the knowledge that is needed for training makes use of the test set as well. Let it be data standardization (or any other sort of normalization), completion of missing values, or tuning the algorithm. If the entire training flow involves supplying values to parameters, we must make sure that the data that determined those parameters is merely the training set. Otherwise, we add to the process information that is not available during prediction. Being careful with an algorithm’s parameters is a relatively common practice. We find significantly less caution in pre-processing procedures, but these pre-processing procedures might have a dramatic impact on the outcome.</p><h4><strong>5. Few levels of generalization</strong></h4><p>The objective of supervised learning is to generalize from the training samples. It is sometimes the case that there are several possible levels of generalization. For example, consider the task of gesture recognition based on some inertial sensors. The training set will include the sensory signals of several users, along with an annotation on the specific gesture that was recorded by each signal. Now, the purpose here is to generalize from these examples. There are two possible levels of generalization: a) generalizing from the specific gestures of a specific user into other gestures of the same user, and b) generalizing from the specific gestures of a specific user into general gestures of any other user. Data leakage due to a few levels of generalization occurs when we use data that is suitable for generalizations of a lower level to make generalizations in a higher level. For example, in the gesture recognition task, if we use signals by the same users in both the training and the testing, we let the personal user data affect the performance where such data will not be available.</p><h3><strong>Spotting Data Leakage</strong></h3><p>Spotting data leakage is one of the most misleading challenges. The concept seems to be clear and straightforward: in your research process, only use data for predictions that you will have at hand in the real prediction scenario. But, other than a few simple cases, leakage can take complex forms. Here are a few practices that will help you to be aware of and hopefully even avoid (or at least minimize) the impact of data leakage.</p><ul><li>Decide on a benchmark model first and assess its performance: A benchmark model is a simple yet better-than-random prediction model. Thereafter, compare the performance of every supervised learning model to that of the benchmark model. Performance measures that are simply too good to be true often imply data leakage.</li><li>Be careful with training, validation, and testing: sparing an independent subset of the data for testing and validation is a recommended practice, of course. Split your dataset with extra care. Once you let your test set affect the trained model, you have data leakage. If you need an independent set for optimizing some parameters, you must use an additional independent set (train, validation, and test).</li><li>Pay attention to the features that appear to be the most important for your prediction: whether you are using feature selection techniques, or if your model includes an inherent indication of feature importance (e.g., decision trees or random forest), search for a good (or at least reasonable) explanation as to why that feature is important. If no such explanation exists, be suspicious.</li><li>Examine several performance measures: in too many projects we see a naive definition of performance measures, such as merely accuracy (or minimum MSE). Naive performance measures may hide local data leakages. For example, let us assume that, in a classification task, we have a brutal data leakage which affects only 10% of the population. Let us further assume that the objective accuracy (without the leakage) is 80%. The total accuracy due to the leakage will be 81%. Only slightly better and might seem like a success. Plotting an ROC curve for that example might reveal a perfect section, and we should always be suspicious of perfection.</li><li>Create variation in the set of explaining features. Specifically, examine what happens if you eliminate each one of the input features. If by eliminating just a single feature from the input set you experience a dramatic reduction in performance, there is a high chance that the eliminated feature contains leakage. When possible, try to eliminate a few features at a time. However, the more features you artificially eliminate, the lower the performance that you can expect will be.</li><li>In a time series, try to learn in different time resolutions and project on to different horizons. Be suspicious towards any surprising variations in performance.</li><li>Strive to get additional validation data: the train and test set that you are using for training and testing the model will probably describe the database as it is prior to the kickoff of the project. In the time that passes from that kickoff and until you obtain a model, it is most probable that new data was obtained. On that point, the flow of data in the production system is also probably better understood. Whenever possible, spare the time to collect another validation set, and do everything you can to make this set represent reality as it is, at the prediction point.</li></ul><p>Data leakage is more common than one might think. It may affect the bottom line significantly, and spotting it can be difficult. Knowing the common leakage patterns and implementing the recommended practices above can save you from huge implementation efforts and save you from getting disappointing results in production.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6c307009ad9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/beware-of-data-leakage-f6c307009ad9">Beware of Data Leakage</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How NOT to become a Data-Driven Organization]]></title>
            <link>https://medium.com/yellowblog/how-not-to-become-a-data-driven-organization-b11ac841dbb2?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/b11ac841dbb2</guid>
            <category><![CDATA[methodology]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-driven]]></category>
            <dc:creator><![CDATA[YellowRoad]]></dc:creator>
            <pubDate>Mon, 26 Jun 2017 14:38:16 GMT</pubDate>
            <atom:updated>2017-06-26T14:47:57.169Z</atom:updated>
            <content:encoded><![CDATA[<h4>Six Soft Pitfalls to Avoid on the Way to a Data Driven Organization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*XkqGgEiyXuSbgjzTZCmLbQ.jpeg" /></figure><p>Starting a blog post by writing about the data revolution is a cliché (but I am going to start with this statement anyway). The potential of data for organizations is an undoubted and widely-agreed fact. As geeks of algorithms and technology, we at YellowRoad enjoy learning about the many advancements surrounding data. It is a crucial part of our professionalism. However, as we meet and serve more and more organizations, the one thing that becomes apparent to us is that the technical data aspects (aka algorithms and technology) are only second in importance to data-driven initiatives. It is the soft organizational aspects that eventually lead to success or failure.Now, please don’t get us wrong.</p><p>We provide technical services (cutting edge, if I may add). Doing things right, technically wise, is a must. It is challenging, and we see many pitfalls in that arena, as well. The thing is that even with the best technical practices, no success will be obtained, without the firm foundations of strong (yet soft) business aspects.</p><p>In this post, we will describe 6 soft business pitfalls that lead to failure for many organizations on their way to become data driven.</p><h4><strong>1. Lack of data vision (or, vision is a leadership responsibility, not a technical aspect)</strong></h4><p>Information systems emerged mainly for transaction processing. In transaction processing, the design of data structures and data processing procedures is merely technical. Consequently, IT divisions (mainly in enterprises, in those days) often took a technocratic perspective on systems, and management in these organizations learned that data was something technical that needed no management guidance.</p><p>Being a data-driven organization means using data to tackle business challenges of a higher level. For managers who got used to think that data is unimportant in the tactic and strategic level, the change in perspective is sometimes hard. These managers might acknowledge the potential importance of data, but they often fail in showing the right way to their organization, data wise. It is very hard to make real progress in materializing the potential of data, without a clear and sharp data vision.</p><p>We have seen C-level executives who show lots of enthusiasm towards data many times. They acknowledge the potential and say that their organizations must compete on data. Trying to coin the enthusiasm into a somewhat more practical level (i.e., how are you going to compete on data?) will sometimes reveal a surprising gap. We even hear: “Why don’t you tell me, you are the experts”. So yes, we see ourselves as trusted consultants in data strategy. But, when data has the potential to significantly affect and even transform businesses, we truly believe that it is the role of the organization’s leadership to take the call.</p><h4><strong>2. Inability to act according to the outcomes of the data</strong></h4><p>A weak data vision often creates situations in which data-driven business objectives are not planned all the way through. For example, a very common data-driven task is churn prediction (to be honest, it was more common a decade or two ago than it is today). The intuition makes sense. Recruiting a new customer costs way more than retaining the existing customers. If we can predict churn before it actually happens, and somewhat proactively retain the customers who are expected to churn, then we have an economic way of increasing the number of active customers. In many cases, however, the thing is that there is a very low certainty in the propensity of a specific customer to churn. For example, let us assume that two percent of the customers churn every month. And let us further assume that we could allocate a segment of customers who are 10 times more probable to churn (10 times! That’s a lot). Still, in this segment, 4 out of 5 customers will not churn, which actually multiplies the costs of the proactive actions by 5 (you have to invest in customers who are not really churns), and often makes retention through churn prediction way less attractive than it seems at first. Not being able to act according to the outcomes may be caused by many reasons: we don’t have such actions in mind, law or regulations do not allow such actions, costs are too high, actions are unethical or immoral, and more. While serving our customers, and learning about a new data driven idea we often ask: “And then what? How will you act given this and that outcome?” We believe that having concrete answers to these questions is a must, and in most cases, the answers must be clear before you start.</p><h4><strong>3. A Voodoo-Based Organizational Culture</strong></h4><p>Organizations that are not driven by data actually operate in an open loop. Employees in these organizations make decisions, which make an impact that they cannot measure. Without measuring the outcomes of different decisions, exploring and concluding, no learning occurs. When organizations operate in an open loop for too long, it is often the case that voodoos replace the real feedback.</p><p>In such organizations, there are often a wide set of beliefs regarding causes and effects. These beliefs are sometimes true, but most of the time they are either inaccurate, describe situations of the past, or are complete nonsense.</p><p>A typical example of a voodoo-based culture is referring to a specific segment of customers as VIPs, due to a small number (or sometimes even a single observation) of highly profitable customers in this segment being observed. Another type of voodoo belief results from inappropriate statistical analysis. For example, in funnel analysis (which characterizes many online marketing operations), you sometimes find companies running A/B testing that merely take averages, with no statistical significance, whatsoever. Clearly the power of these A/B tests is merely slightly better than random. Similarly, you often find organizations that mix correlation and causation (having two correlated variables doesn’t mean that one of them affects the other).</p><p>When the voodoo-based culture is strong, even hard evidence is not always convincing. In an adaptive organization, consistent efforts to act based on hard evidence will eventually help in changing the culture, but in a less adaptive organization, we often see that the voodoos win data.</p><h4><strong>4. Data is not available or is in pure quality</strong></h4><p>Being data driven clearly requires data. Sounds trivial, right? Well, as surprising as it may seem, we have been asked so many times to design a data-driven system when there is no relevant data at all. When we ask about that negligible aspect, the typical reply is: “design the algorithm for us, and we will start running it when the data arrives”. Planning working algorithms requires evaluating and testing them. Simulated data is nice, clean, and easy to handle. When the real data arrives, it always, always, proves to be totally different from the plans. It’s poor in quality, has lots of missing value, different formats, unstable sampling rates and more.</p><p>It is good practice to think and plan out a system before you have the data. In such planning, you might gain a better understanding of the nature of the data that you need for your system. But, no working system which is based on data can be designed and launched without real data.</p><h4><strong>5. Under-trained data team</strong></h4><p>With all the advancements that are being made around data, many data analysis tools have become available to the public. Some of these tools are really easy to work with. You can do magic with some BI tools, and with basically no programming skills you can build a machine learning model within a single afternoon. Consequently, many organizations try to analyze data in-house, without having the proper trained team (it is somewhat like a non-handyman using a hammer — finding the tool and holding it is easy, but fixing a new set of shelves with it is a completely different aspiration). In many organizations, we see under-trained teams working with data. This post is on soft pitfalls. The post on technical pitfalls will be longer in at least two orders of magnitudes. With data analysis, you should be careful, knowledgeable, and experience each and every step that you make.</p><h4><strong>6. Wrong KPIs</strong></h4><p>Measurability is a key aspect of being data driven. Leadership and management have soft areas, and some things can be managed beautifully based on soft skills, without being measured. However, data processes should be optimized to measurable KPIs. A common mistake is mixing the set of business KPIs with the measures that are used for optimizing algorithms.</p><p>Leadership and management have soft areas, and some things can be managed beautifully based on soft skills, without being measured. However, data processes should be optimized to measurable KPIs. A common mistake is mixing the set of business KPIs with the measures that are used for optimizing algorithms.</p><p>The set of business KPIs indicate the wellbeing of the organization. It is very like a blood test that reflects something about the proper functionality of several key organs, it contains indications about the different aspects of the business. However, when you optimize a process or algorithm you typically need to decide on a single measure to optimize it to.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b11ac841dbb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/how-not-to-become-a-data-driven-organization-b11ac841dbb2">How NOT to become a Data-Driven Organization</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Agile Meets Machine Learning]]></title>
            <link>https://medium.com/yellowblog/when-agile-meets-machine-learning-851985db95cf?source=rss----4981ab970712---4</link>
            <guid isPermaLink="false">https://medium.com/p/851985db95cf</guid>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[methodology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[YellowRoad]]></dc:creator>
            <pubDate>Mon, 26 Jun 2017 14:35:09 GMT</pubDate>
            <atom:updated>2017-06-26T14:35:09.450Z</atom:updated>
            <content:encoded><![CDATA[<h4>5 Guiding Principles in Implementing Agile Methodology in Research-Intensive Software Environments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*BjKq3D8URE0lrVZmjoTHMQ.jpeg" /></figure><p>Agile software development is becoming the common practice in more and more software organizations. For those organizations that use to work in the traditional, waterfall way, going agile means letting many myths go. It’s never easy, however, when dealing with pure functional products, it is often the case that within a reasonable period of time, the superiority of the new methodology becomes apparent, and that often wins the argument.</p><p>In addition to the regular and inherent pain that is caused by the change, implementing agile in domains that are research intensive (for example, with products that are based on machine learning) has some additional obstacles. Research contains significantly higher degree of uncertainty. Research means finding new knowledge, that is currently unavailable, and predicting the exact form of that knowledge, and the resources that are needed to find it is hard. Moreover, researchers are trained and educated in academy, where in many cases there are no customers to collaborate with, comprehensive documentation is the ultimate outcome, and the tightest compliance with tools and standards is required, in order to compare with previous works. Not surprisingly, the term “research” is intimidating to many managers and business personnel.</p><blockquote>“Does agile methodology fit in research intensive software environments?”</blockquote><p>The question above seems like a rhetoric question, given the title of this post, but it is being asked by many many software organizations, including organizations that run agile fluently, in their pure functional features. Undoubtedly, the answer is a big yes; however, research intensive software environments has some unique characteristics that need to be considered during almost every aspects of agile implementation.</p><p>In this post we suggest 5 key considerations, that deserve closer attention, when implementing agile methodology in research-intensive software environments.</p><p>In pure functional features, there are two important considerations: correctness and engineering quality. The team will typically not commit to a feature if it cannot be supplied correctly and with sufficient quality. In functional features correctness is usually well defined.</p><p>In research intensive features, the knowledge that is required for developing the new feature is initially absent, and so is often the exact outcome. In many cases, there may be several levels of correctness.</p><p>Consider for example an analytic feature that includes a prediction model. With no model, the prediction is random. With the “optimal” model, the prediction accuracy might be, for example, 95%. However, the concept of an optimal model is merely theoretical, since in reality we cannot asses the accuracy of an optimal model, and getting near this 95% requires a lot of efforts. It is often the case that really quickly a skillful data scientist can come with a model that can provide 85% of accuracy, or even 90% of accuracy. Assuming that this model is implemented with high engineering quality, the question of correctness remains: did we fully optimized the solution already or not?</p><p>Going agile requires many times to prefer quick solutions over optimal ones. Having a solution that is already providing significant business value, it is often the case that we better start implementing the model, instead of continue optimizing it.</p><p>As a matter of fact, prediction tasks are an easy example, in the sense that the solutions have objective measures of correctness (the prediction accuracy in this case). How would you measure or compare algorithms that cluster objects into groups, or algorithms that automatically summarize long documents? Research domains often require baby steps, and solutions that improve iteratively. Research is a product that evolves continuously and not a one-time project. Giving up the aim for optimization is sometimes hard for researcher who were educated in academy, where user engagement is no consideration, and it often requires a cultural change.</p><p>Uncertainty regarding the outcomes is common to any significant research. It is true that any planning includes some uncertainty, but in research, the level of uncertainty is by far higher. This huge uncertainty makes it dangerous to include the research and the implementation of the same feature, in a single iteration or plan. In research, failing to prove feasibility is an acceptable outcome (finding another way that is not working is frustrating, but it happens a lot). Failing to deliver a feature is less acceptable. Counting on the research phase to result in a feasible solution, in order to immediately implement this solution, by definition means that many times you will end with undelivered feature.</p><p>In most cases you even have to maintain a small buffer of research-oriented features that are waiting for implementation. Without this buffer, the implementation team will eventually starve to inputs, and be underutilized. Having a too big buffer is not recommended either, because it disturbs the important connection between the research and the customers’ needs and makes production too slow.</p><p>Diversity and heterogeneity are blessed, in many cases. Unfortunately, many organizations see their researchers as individual contributors. Researchers are viewed as highly smart individuals, yet strange and sometimes hard to work with. Individual-contributor researchers get very little feedback from customers, very little feedback from developers and very little feedback from product owners. No wonder that in some cases individual contributors indeed lose their connections with reality, and may provide outcomes that do not take into account all the product and infrastructure constraints.</p><p>Not less important, in the individual contributor mode, customers, product owners and developers get very little feedback from the researchers, and often tempted to think that research is some sort of magic. When this is the case, the product comes with unrealistic request and expectations, and fail to deliver these requests to the researchers, and developers fail to prepare the infrastructures for implementing the new features.</p><p>Teams that conduct research should not be comprised solely from researchers. The developers who will implement the feature under research and the product representative should be part of the team, as well.</p><p>Moreover, cross-role knowledge sharing should be encouraged. Researchers are often highly curious, open minded individuals, that can contribute in many domains, and sharing business knowledge and development knowledge with them, creates a win-win. Research work is often highly interesting, and sharing it with product and development is also a win-win, since it increases the level of work interest, and helps in aligning expectations.</p><p>Retrospective is one of the strongest aspects of agile. Dedicating time to discuss the way that the work is done, rather than the work itself, and adapting the methodology according to the team’s needs and desires ultimately drive teams to become more engaged and self-empowered, and simply makes the work methods better. With heterogeneous teams (see guiding principle no. 3) agile become even more complicated and comes with higher degrees of freedom. The different terminology, mode of work, outcomes and even mentality, should be discussed and learned by all sides with much respect and patience to each other. The more diverse is the team, the more important it becomes to have deep non-judgmental retrospective. Going agile in research intensive environments takes way more time and improvements iterations, and missing the needed room for retrospective jeopardize the level of team satisfaction from the change, the probability of finding some sweet implementation method and eventually the overall success of the change.</p><p>Differently from basic research, in which the sole objective is generating new knowledge, and where wondering about the value of that new knowledge is sometimes non-legitimate, in business domain research should be applicative, and serve the business objectives. Now, when written, it hard to argue that every applicative research should stem from a business need and serve that business need. Nonetheless, there are many efforts, in so many organizations, which are spent on research from no clear reason. This phenomenon especially characterizes emerging and sexy domains (like big data and machine learning, in the recent years).</p><p>Conducting research with no clear business need is not only an almost certain recipe for failure, it is also a stick in the wheels of agile. With no clear business needs there is no clear way to define iterations outcome, no way to commit on anything and therefore no way to proceed. The most that can come out from such non-applicative research is a sudden enlightenment. I know very few organizations that can afford themselves the expectation of arbitrary enlightenment as a business model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=851985db95cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/yellowblog/when-agile-meets-machine-learning-851985db95cf">When Agile Meets Machine Learning</a> was originally published in <a href="https://medium.com/yellowblog">YellowBlog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>