<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by VASHIST NARAYAN Singh on Medium]]></title>
        <description><![CDATA[Stories by VASHIST NARAYAN Singh on Medium]]></description>
        <link>https://medium.com/@vashistnarayansingh?source=rss-454105000991------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*hkzPD0jyO56JPCIV</url>
            <title>Stories by VASHIST NARAYAN Singh on Medium</title>
            <link>https://medium.com/@vashistnarayansingh?source=rss-454105000991------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:29:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vashistnarayansingh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Neural Machine Translation using Simple Seq2Seq]]></title>
            <link>https://medium.com/analytics-vidhya/neural-machine-translation-using-simple-seq2seq-517b5ce1ae0f?source=rss-454105000991------2</link>
            <guid isPermaLink="false">https://medium.com/p/517b5ce1ae0f</guid>
            <category><![CDATA[encoder-decoder]]></category>
            <category><![CDATA[seq2seq]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nmt]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[VASHIST NARAYAN Singh]]></dc:creator>
            <pubDate>Mon, 04 May 2020 09:20:00 GMT</pubDate>
            <atom:updated>2020-05-05T08:58:48.623Z</atom:updated>
            <content:encoded><![CDATA[<h3>Neural Machine Translation using Simple Seq2Seq Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*prLx5jgXucwWMQoz-Cb7rQ.jpeg" /></figure><p>Language translation plays a very crucial role in this modern era of globalization where countries are connected with each other in some way. Irrespective of English being the universally accepted language to communicate, there are still many countries who prefer their native language to communicate. If we keep the global perspective aside for a second there are still people who understand things better in a language they are native to. Hence considering all these needs language translation is the part of human civilization from a very long time. Before the advancement of technology, there are humans who used to act as an interpreter between peoples. But soon after the technology evolves we created machines to do this translation. This machine translation keeps on evolving as time passes and researchers keep researching to make machine translation more accurate and correct. Soon after the advancement of deep learning and neural nets like RNN’s, LSTM’s and more sequence-based neural nets building a simple language translator is not that difficult nowadays.</p><p>So in this tutorial blog, I will show how you can create a simple neural machine translator to translate English sentences into Marathi sentences. I have chosen a Marathi sentence because it helps me to understand the translation in a better way. Also, the Hindi language data set is very small to work with. So without wasting more time lets get started.</p><h3>Table of Content</h3><ol><li>Introduction</li><li>Data Collection and Cleaning</li><li>Input Creation for encoder and decoder</li><li>Model building</li><li>Inference Model Building</li><li>Prediction</li><li>Model Performance</li><li>Scope of Improvement</li><li>Conclusion</li><li>Reference</li><li>Connect with me</li></ol><h3>1. Introduction</h3><p>In this section, I will give a small introduction of what neural machine translation also called NMT is all about. So, NMT is build using a neural network hence “N” stands for “Neural” and since it uses a machine to do the translation “M” stands for “Machine” and in the end, it’s doing translation hence ‘T’ stands for “Translation”. I hope you have understood till here and for a non-machine learning and deep learning readers you guys are wondering what’s this “Seq2Seq Model” for them I will try to explain it in short, In deep learning, we make use of neural networks to train our deep learning model(that makes predictions) so there are some neural networks like RNN, LSTM, GRU that works on a sequence as inputs like a sequence of English sentence in our case. So Seq2Seq model is a representation of these special neural networks. The Seq2Seq model has two parts one is called an encoder and the other is called a decoder and as the name suggests encoder encodes our input English sentence and decoder decodes these encoded English sentence into Marathi sentence. So this is a simple introduction of a simple seq2seq model and based on further improvements and research we keep on finding new techniques to improve the encoding of the input sequence but the heart of seq2seq model is encoder and decoder. For more information on encoder and decoder, you can refer to <a href="https://machinelearningmastery.com/encoder-decoder-long-short-term-memory-networks/">this</a> link.</p><h3>2. Data Collection and Cleaning</h3><p>So after the quick introduction of NMT and seq2seq model its time for us to make our hands dirty by digging into the coding part of this NMT. In this section, we are going to get our raw data and perform some basic text cleaning. To get the dataset you can refer to <a href="http://www.manythings.org/anki/">this</a> link.</p><p>Before loading the data from the link I have given in the above paragraph we need to import all the necessary libraries we are going to use in this project.</p><h4>Import Libraries</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/092e3dcefdb5d4948c4f7f464f18c01d/href">https://medium.com/media/092e3dcefdb5d4948c4f7f464f18c01d/href</a></iframe><p>Now as we imported all the libraries we can now move ahead to load our raw text data file.</p><pre><strong>with</strong> open(&#39;mar.txt&#39;,&#39;r&#39;) <strong>as</strong> f:<br>  data = f.read()</pre><pre><em># len(data)</em></pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/758d56d495d94311ca940602dbce83d1/href">https://medium.com/media/758d56d495d94311ca940602dbce83d1/href</a></iframe><p>The above gist shows the content of the raw text file and you can see that we need to do the cleaning of this file so let’s get into the data cleaning process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/584106c6aa99c2ea579b210f4aea908d/href">https://medium.com/media/584106c6aa99c2ea579b210f4aea908d/href</a></iframe><p>In the above code, I am spitting each line of the raw data text and storing it into the list named uncleaned_data_list. After that, I am separating all the English and Marathi sentences and storing them into english_word and marathi_word list respectively and in the end, I have printed the total number of English sentences and Marathi sentences.</p><p>After that, I created a pandas DataFrame with two columns named English and Marathi and saved it as a CSV file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2aedf5a3ba1b350d37614e5717c6053/href">https://medium.com/media/a2aedf5a3ba1b350d37614e5717c6053/href</a></iframe><pre>language_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*9UjCV3wuqhfh1XeaPw6yzA.png" /></figure><pre>language_data.tail()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*aESAVJz0LZyZKp6hz04wkQ.png" /></figure><p>You can see the head and tail data of our newly formed pandas DataFrame the reason for representing the data into this format is it makes the handling of data more efficient and also with this we can take the advantage of panda’s tools as well. Now let’s move further into our text cleaning process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/500b0d9fb51cda1e68891c5912db9a2a/href">https://medium.com/media/500b0d9fb51cda1e68891c5912db9a2a/href</a></iframe><p>I think the above code section is self-explanatory but still, I will explain it in short. In the above code section, I am doing text preprocessing which includes some basic things like lowercase the text, removing punctuation, removing digits, and whitespace.</p><pre><em># Putting the start and end words in the marathi sentances</em><br>marathi_text_ = [&quot;start &quot; + x + &quot; end&quot; <strong>for</strong> x <strong>in</strong> marathi_text_]</pre><p>The above code section seems to be simple but it is the most important step in neural machine translation using the seq2seq model. I am going to explain the reason for adding “start” and “end” tags in the Marathi sentences when we are going to build the Seq2Seq model until please be patient and with this, we came to the end of data collection and data cleaning section.</p><h3>Input Creation for encoder and decoder</h3><p>Till this section, we have our clean and processed text dataset and in this section, I will show you how you can prepare the data for your NMT model.</p><p>Before we move into the data preparation step we will split our data into train and test here I am doing 90–10 split.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb56328798f526e1b463c12df2ae45b3/href">https://medium.com/media/fb56328798f526e1b463c12df2ae45b3/href</a></iframe><p>The first thing you have to keep in mind is that the seq2seq model wants all the input sequence to be of same length so one way to achieve this is, we calculate the length of each sentence in both English and Marathi and then we choose the max length i.e the longest sentence in both English and Marathi thus, in the end, we have two max_length one for English sentence and another is for Marathi sentence the same you can see in the code section also. I will tell how you can use these max lengths to make all the sequence of English and Marathi of equal length in later sections. After this, since machines understand numbers only and not the text we need a way to convert the input text sequence into number and one such way is to index the words of the sentences. We can do this indexing using the “Tokenizer” method of Keras we also need to take the vocabulary size of both English and Marathi corpus which is required when we create the input data for the model training. All the things which I have explained now are done in the above code block.</p><p>The code below is long but don’t be scared I am going to explain it completely.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fcefc06f745d24442f3d44489d6aacc/href">https://medium.com/media/2fcefc06f745d24442f3d44489d6aacc/href</a></iframe><p>So as you can see I have created a function called “generator_batch” which accepts three-parameter “X”, “Y”, “batch_size” and you can figure out by yourself what all these parameters mean. Now the first thing in the function is a “While True” i.e a while loop and this loop is set to be always running ok! fine I can understand till here but what is happening inside the loop? The same question came to my mind when I first saw this code on one of my referenced blog that I have used to create this project. Actually, it’s not that complex to understand. We are running a “for loop” from 0 to length of our X(train data) and which is stepped by batch_size that is 128 in this case. Inside the loop, I have three variables “encoder_data_input”, “decoder_data_input”, “decoder_target_input” now let’s see each variable in more detail I have created a metrics of all zeros having No. of rows equal to “batch_size” and columns equal to “max_length_english” and similarly we do for the decoder input but can you see the decoder target data it is of 3-dimensional shape because decoder output is 3-dimensional. But wait I have told you guys that decoder is what that gives you the translated sentence as output then why we have created the decoder input also. This is because it makes the training faster and this method is called the “Teacher Forcing” technique. In teacher forcing, we pass the target data as the input to the decoder for example if the decoder is to predict &#39;the&#39; then we also pass ‘the’ in the input of decoder as well you can relate it with the actualy teacher student learning process. Hence it makes the learning process faster.</p><p>So till now, we have created the three matrics of zeros. In the nested for loop, I am taking the index, X_train, and Y_train data one by one there is another for loop which is taking the word from the outer for loop X_train sentence and also the index and after that, I am filling our encoder_data_input zeros metric with the respective index of the words in the input sentence here the outer loop “i” is going row-wise and the “t” of the nested loop is going columns wise and as you observe the all the input sequences are going to be of the same length because I have set the column equal to the max_lenght_english so if the input sequence is less then max_length_english we are padding zeros for rest of its columns. A similar thing was done in the for loop of the decoder_input_data but not in the decoder_target_input remember we have padded the “start” and “end” tags in the target Marathi sentences so at t = 0 we have “start” as the word and we do not want this word in the target input sequence hence we start the sequencing for the decoder_target_input from t&gt;0.</p><p>Till now, I have created our input and outputs for the seq2seq model and using yield I am sending the data batch-wise and not all at the same time. In the following section, I will tell you the reason for appending “start” and “end” tags in the target sentences also working of encoder and decoder in detail and the model code as well.</p><h3>Model building</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDDhDYhipj5owgPKmuSs2Q.png" /><figcaption>Encoder and Decoder</figcaption></figure><p>The above figure is a representation of simple encoder and decoder the horizontal axis here is the timestamp and ‘A’, ‘B’, ‘C’ is the words of the input sequence. So now suppose we have 10 input sentences and are stored in our encoder_data_input metrics each row in the metric is one sentence hence we have 10 rows and max_length_english columns and say X1, X2, ……., X10 is used to represent each sequence.</p><p>Now,</p><p>For X1 say we have 3 words in the sequence then,</p><p>at t = 0 we pass w1 into the encoder and take its state</p><p>at t = 1 we pass w2 into the encoder along with the state at t = 0</p><p>at t = 2 we pass w3 into the encoder along with the state at t = 1</p><p>now at the end of the sequence, we take the output of the encoder and its state at t=2 and pass them as the input to the decoder, and then the decoder output at t=0 is passed as input to the decoder at t=1 and so on till the end of the string is not reached.</p><p>Now the reason for appending “start” and “end” to the decoder input is, “start” tag is used to initiate the decoder to start decoding, and the “end” tag is used to signal decoder to stop decoding process. If we do not use this then the decoder will never be able to produce the first word as we know that if we pass the word to the decoder as input it will predict the next word of the sequence and this prediction will never stop as it is not able to know when to stop.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/306e26eaa552a3e7d6bf1ba14bf995ff/href">https://medium.com/media/306e26eaa552a3e7d6bf1ba14bf995ff/href</a></iframe><p>The above code is doing exactly what I have explained before and also shown in encoder-decoder image. Try to understand the code in reference to the above encoder-decoder image.</p><pre>plot_model(model, to_file=&#39;train_model.png&#39;, show_shapes=<strong>True</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*DHcQ-BMSjsud3LGaDhs6ow.png" /><figcaption>Training Model</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/448fe09350e856d4299ca8ce69197e5c/href">https://medium.com/media/448fe09350e856d4299ca8ce69197e5c/href</a></iframe><p>In the above code section, I am saving the trained model and loading it please note one thing save the model like this only and not any other way because we have to create the inference encoder and decoder to make a prediction later and that model makes use of trained model layers also.</p><h3>Inference Model Building</h3><p>Below section is a code of inference model this model is used to make a prediction.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2a9d10df1a928f3bbf443651ce89727/href">https://medium.com/media/d2a9d10df1a928f3bbf443651ce89727/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*PICBrd2xSM2sdSaWkwbMvQ.png" /><figcaption>encoder inference model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hzkoXZlaffW25Bs9L4lL7A.png" /><figcaption>decoder inference model</figcaption></figure><p>The encoder inference model is similar to the training model the difference is in the decoder inference model. If you see the encoder-decoder diagram we can see that the states of the previous timestamp are pass to the next timestamp hence we need to find a way to preserve the states of the previous timestamp the below code in the decoder inference is doing the same thing.</p><pre><em>#inference decoder</em><br><em># The following tensor will store the state of the previous timestep in the &quot;starting the encoder final time step&quot;</em><br>decoder_state_h_input = Input(shape=(latent_dim,)) <em>#becase during training we have set the lstm unit to be of 50</em><br>decoder_state_c_input = Input(shape=(latent_dim,))<br>decoder_state_input = [decoder_state_h_input,decoder_state_c_input]</pre><p>Then we create the decoder inference model to create the decoder model we have defined the input layer first this is the input layer of the trained decoder model. After that, we create the same layers as we have in the training decoder model but the only difference is the initial state of the decoder LSTM is set to be the states of the previous timestamp as we can see in the diagram of encoder-decoder above, rest of the inference model is similar to the training decoder model.</p><pre><em># inference decoder input</em><br>decoder_input_inf = model_loaded.input[1] <em>#Trained decoder input layer</em><br><em># decoder_input_inf._name=&#39;decoder_input&#39;</em><br>decoder_emb_inf = model_loaded.layers[3](decoder_input_inf)<br>decoder_lstm_inf = model_loaded.layers[5]<br>decoder_output_inf, decoder_state_h_inf, decoder_state_c_inf = decoder_lstm_inf(decoder_emb_inf, initial_state =decoder_state_input)<br>decoder_state_inf = [decoder_state_h_inf,decoder_state_c_inf]<br><em>#inference dense layer</em><br>dense_inf = model_loaded.layers[6]<br>decoder_output_final = dense_inf(decoder_output_inf)<em># A dense softmax layer to generate prob dist. over the target vocabulary</em><br><br>decoder_model = Model([decoder_input_inf]+decoder_state_input,[decoder_output_final]+decoder_state_inf)</pre><h3>Prediction</h3><p>Now we have trained the seq2seq model and created the inference model using the trained model for making prediction so its time to make a prediction using the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9988b51d0cb13c3d5f5f021eac83a496/href">https://medium.com/media/9988b51d0cb13c3d5f5f021eac83a496/href</a></iframe><p>The above function is returning us the translated sentence it first encodes the input sentence and gets the state value of the last timestamp of the encoder. Then we have created the metric of single value called “target_seq” which holds the index of the next predicted Marathi word but to start the decoding process we need to pass the first word i.e “start” in this case hence, in the beginning, we are going to store the index of the word “start” into the “target_seq” then since we have to carry out the decoding process till we reach the “end” word. Thus we need to call the decoder inference model in a loop. In the loop, we pass the “target_seq” and the encoder last timestamp “states” as the input to the decoder inference model. The inference decoder_model returns the next word index and the decoder states, after this, we convert that index back to the word and add it to the string variable “decoder_sentance”, then we put the newly generated word index into the “target_seq” and update the “state_value” with the decoder model states this process continues till we get the predicted word as “end”. Hence at the end of the while loop, we get the complete translated sentence. The following is some of the output I get.</p><pre><strong>for</strong> i <strong>in</strong> range(30):<br>  sentance = X_test[i]<br>  original_target = y_test[i]<br>  input_seq = tokenizer_input.texts_to_sequences([sentance])<br>  pad_sequence = pad_sequences(input_seq, maxlen= 30,padding=&#39;post&#39;)<br>  <em># print(&#39;input_sequence =&gt;&#39;,input_seq)</em><br>  <em># print(&quot;pad_seq=&gt;&quot;,pad_sequence)</em><br>  predicted_target = decode_seq(pad_sequence)<br>  print(&quot;Test sentance: &quot;,i+1)<br>  print(&quot;sentance: &quot;,sentance)<br>  print(&quot;origianl translate:&quot;,original_target[6:-4])<br>  print(&quot;predicted Translate:&quot;,predicted_target[:-4])<br>  print(&quot;==&quot;*50)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b6fe8061086bdd068ecade4683e67c7/href">https://medium.com/media/9b6fe8061086bdd068ecade4683e67c7/href</a></iframe><h3>Model Performance</h3><p>The performance of the model is good as compared to the amount of data used to train the seq2seq model I have got the accuracy of 54.8% for a model trained on 34825 dataset for 50 epochs. The limitation of the simple seq2seq model is that it is not able to translate a lengthy sentence that efficiently. So this model also has this limitation.</p><h3>Scope of Improvement</h3><p>Since this is just a simple model to show you how to create a neural machine translator and as we know that no solution is perfect and everything keeps improving with time this solution is not the only solution, you can improve it further as per your understandings and skills.</p><p>Following things I can suggest to Improve this model further:</p><ol><li>we can train the model on a large dataset with lots of variation in it.</li><li>For lengthy Sentance limitation of it, we can include the Attention Mechanism.</li><li>We can try replacing GRU with LSTM and check the performance of the model.</li></ol><h3>Conclusion</h3><p>This is the conclusion section of the blog. This blog shows you what neural machine translation is all about, what is the need for translation in this world, and how you can create an NMT by yourself using deep learning, you have learned what seq2seq model in detail. In the next blog, I will show you how can you deploy this NMT and create a REST API using Flask so that you can access your model through any application. The following is the video of the end product we are going to build in the next blog. So follow me on medium so that you can receive the notification of my next blog whenever it gets publish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*H0TMzJgkcYjffviSBK4FYQ.gif" /><figcaption>Next blog end product demo</figcaption></figure><h3>References</h3><ul><li><a href="https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html">A ten-minute introduction to sequence-to-sequence learning in Keras</a></li><li><a href="https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq_restore.py">keras-team/keras</a></li><li><a href="https://machinelearningmastery.com/develop-encoder-decoder-model-sequence-sequence-prediction-keras/">How to Develop an Encoder-Decoder Model for Sequence-to-Sequence Prediction in Keras - Machine Learning Mastery</a></li><li><a href="https://machinelearningmastery.com/encoder-decoder-long-short-term-memory-networks/">Encoder-Decoder Long Short-Term Memory Networks - Machine Learning Mastery</a></li><li><a href="https://towardsdatascience.com/word-level-english-to-marathi-neural-machine-translation-using-seq2seq-encoder-decoder-lstm-model-1a913f2dc4a7">Word Level English to Marathi Neural Machine Translation using Seq2Seq Encoder-Decoder LSTM Model</a></li></ul><h3>Connect with me</h3><p>You can connect with me on <a href="http://linkedin.com/in/vashistnarayan-singh">LinkedIn</a>.</p><p>You can find the complete code of this blog on my <a href="https://github.com/vashist1994/Neural-machine-translation-seq2seq">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=517b5ce1ae0f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/neural-machine-translation-using-simple-seq2seq-517b5ce1ae0f">Neural Machine Translation using Simple Seq2Seq</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackoverflow Based Semantic Search Engine]]></title>
            <link>https://medium.com/analytics-vidhya/stackoverflow-based-semantic-search-engine-139e2b2e8860?source=rss-454105000991------2</link>
            <guid isPermaLink="false">https://medium.com/p/139e2b2e8860</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[stackoverflow]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[search-engines]]></category>
            <dc:creator><![CDATA[VASHIST NARAYAN Singh]]></dc:creator>
            <pubDate>Wed, 04 Mar 2020 08:07:18 GMT</pubDate>
            <atom:updated>2020-04-03T17:57:58.761Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*49shmWGsJdHomHodycLPGA.png" /></figure><p>If you have written a code in your life then you must be familiar with StackOverflow. For those who did not familiar with this, StackOverflow provides one of the largest QA platforms for programmers. Users post questions/doubts and their fellow programmers try to provide solutions in the most helpful manner possible. The better an answer, the higher the votes it gets, which also improve the reputation of the user.</p><p>Kowing to its popularity, it’s safe to say that there are lots of data to them, However, this huge amount of information also makes it difficult to search for the solution on is looking for. But it is not big programmers who are aware of programming because they know how to optimize your search query with proper keywords. But for someone who is new to this world, it is not that simple. Therefore, an optimal search engine is necessary to navigate through this mess.</p><h3>Index</h3><ol><li>The Task</li><li>Business Problem</li><li>ML Problem</li><li>Approach</li><li>Data</li><li>Library Import</li><li>Data Analysis and processing</li><li>Tag predictor model</li><li>Creating the Search pipeline</li><li>Scope of Improvement</li><li>References</li></ol><h3>1. The Task</h3><p>Our objective is for the platform to actually understand the content of what the user is trying to search for, and then return the most similar result based o that. I need to gather Questions and Answers that were posted on Stack Overflow. Thus what I need are the following:</p><ul><li>Title</li><li>Question body</li><li>Answers for that question</li><li>Votes for each answer</li></ul><h3>2. Business problem</h3><p>The StackOverflow has a huge amount of information due to which it makes it difficult to search for the solution one is looking for. It’s not that big of an issue for programming veterans and other experienced professionals, because they are aware of the correct keywords required to get an appropriate answer. However, for a baby programmer, this poses a great concern. For instance, if he needs to learn ‘how to make a server’ using Python, it is quite unlikely that he would use the terms ‘Django’ or ‘Flask’ in the search box. Thus there is a need for an optimal search engine.</p><h3>3. ML Problem</h3><p>We have to build a semantic-based search engine that uses the StackOverflow data to do the search.</p><h4>Performance metrics</h4><p>I am using micro f1_score, as the metric for my Tag predictor model, since it is multilabel classification model micro f1_score is a good metric to choose</p><h4>constraints</h4><p>since its a search engine the latency should low.</p><h3>4. Approach</h3><ul><li>First I will be going to get the data from <a href="https://www.kaggle.com/stackoverflow/stackoverflow"><em>https://www.kaggle.com/stackoverflow/stackoverflow</em></a> and do some preprocessing and feature engineering on that.</li><li>Then I train a deep learning model to predict the tag of the input query so that I know which category the input query belongs to, my metric will be micro f1_score. Here I am also using the w2vec model for the vectorizing since I am using cosine similarity along with the number of votes and the sentiment polarity to the answer to select the most voted and positive response and to achieve this I want to represent my query into a vector space.</li><li>In the end, I will create a small UI to get the input and when the user enters the new query I will first process it then I will predict the tag form it and after that will filter the main StackOverflow data related to the predicted tag. After doing this I will restrict my search and embedding to that corpus of filtered data only and based on the similarity along with most voted and positive sentiment. I return the results to the user.</li></ul><h3>5. Data</h3><p>I will use this <a href="https://www.kaggle.com/stackoverflow/stackoverflow"><em>https://www.kaggle.com/stackoverflow/stackoverflow </em></a>dataset. I include an archive of Stack Overflow content, including<br>posts, votes, tags, and badges. This dataset is updated to mirror the Stack Overflow content on the Internet Archive.</p><ul><li>I will use bq_helper which is a helper class to perform read-only BigQuery Tasks. Reference:<a href="https://www.kaggle.com/sohier/introduction-to-the-bq-helper-package"> <em>https://www.kaggle.com/sohier/introduction-to-the-bq-helper-package</em></a></li><li>There are many tables on the Stackoverflow database, but we only need concern ourselves with posts_questions and posts_answers.</li></ul><h3>6. Library Import</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d063750bf65029e20281d5154be6519/href">https://medium.com/media/2d063750bf65029e20281d5154be6519/href</a></iframe><h4>Getting the data</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a4c31261fa79d031d8dc7a527e05ae0/href">https://medium.com/media/8a4c31261fa79d031d8dc7a527e05ae0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*uNvl3F4Y3vAHoRW0lv282Q.png" /><figcaption>Raw Data from Google BigQuery</figcaption></figure><h4>Observation</h4><p>Basically, this query joins 2 tables(<em>stackoverflow.post_questions</em> and <em>stackoverflow.posts_answers</em>) and collects the required data for 10000000 questions. Thus, each row contains a question and one of the answers it has. (<strong>Note</strong>: <em>There may be rows with identical questions but unique answers</em>).</p><h3>7. Data analysis and processing</h3><h4>Missing value check</h4><pre>original_data.isna().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*fHtNIx27hNxfZ-0VV6A91g.png" /><figcaption>Missing value output</figcaption></figure><h4>Observation</h4><ul><li>It’s good to see that there is no missing value in the dataset.</li><li>Hence no need for the data imputation.</li></ul><h4>Checking for Duplicate in data if any</h4><pre>original_data.duplicated().any()<br>--------------------------------<br>True</pre><p>As mentioned above the dataset may contain rows with identical questions but unique answers. We want to combine all these different rows as one while summing up the votes of each answer and combining all answers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0dbeab8dcdce07e613c2eb88655cf0c7/href">https://medium.com/media/0dbeab8dcdce07e613c2eb88655cf0c7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtU-xVzag23UaqvfR6IWZA.png" /><figcaption>Data After combining rows with identical questions</figcaption></figure><p>Now since we have prepared our raw data we can now move onto cleaning and preprocessing this raw text data. I am doing some basic text processing including the following steps:</p><ul><li>Tokenization of the text</li><li>Converting the tokens to lowercase</li><li>Removing the punctuation from the tokens list</li><li>Removing the stopwords from the tokens list</li><li>Tokenization of the code string</li></ul><p>If you noticed the dataset closely, you’ll observe that the raw text for questions and Answer is given along with the HTML markup with which it was displayed on StackOverflow originally. These refer usually to ‘<em>p’ </em>tags, <em>‘h1-h6’</em> tags and the ‘<em>code’</em> tags. So my approach is as follows:</p><ul><li>I constructed a new feature column called ‘post_corpus’ by combining the title, question body, and all the answers</li><li>I prepended the title to the question body</li><li>I use the ‘code’ sections to extract some features.</li><li>I constructed URLs for each question by appending ‘<a href="https://stackoverflow.com/questions/&#39;">https://stackoverflow.com/questions/&#39;</a> with the question id</li><li>I constructed 2 features for sentiment using the open Source <strong>Textblob library</strong></li></ul><p>Each post has a variable number of different tags. In order to narrow down the vast choices for a more accurate model, I decided to with 100 most common tags. The plan is to filter only the data which contains at least one of most_common_tags.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54e5225b5efca4a2c014635c51bc2743/href">https://medium.com/media/54e5225b5efca4a2c014635c51bc2743/href</a></iframe><p>One last task in the preprocessing step is as follows:</p><ul><li>I have created a separate column for the ‘processed_title’ because I wanted to preserve the original title because I wanted to serve the original titles in the final output.</li><li>I am also normalizing the numeric ‘scores’</li><li>I have also created a separate column for the ‘code’ because I want to use it for the tag prediction</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b218f1a17ecd4dc26706474c8fd4f40/href">https://medium.com/media/0b218f1a17ecd4dc26706474c8fd4f40/href</a></iframe><h3>8. Tag predictor model</h3><h4>Importing library</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ded5108daca61b02d051aee0e292ee06/href">https://medium.com/media/ded5108daca61b02d051aee0e292ee06/href</a></iframe><h4>Filter out only the most common tags</h4><p>Although we did filter out a lot of tags in the data preprocessing step, there still exist a lot of ‘stray ’ tags which may appear only once or twice as compared to thousands of occurrences for other tags. This increases the dimensions of the ground truth data, which is not desirable for our model.</p><p>Thus I have extracted the top 1000 tags based on their occurrences finally, I altered the tag data to only include tags from one of these 1000 tags, for better model accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*EPb8MMEjmAu6cOQQIeXBjA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74b29bc46c99da690be0b288e08023dd/href">https://medium.com/media/74b29bc46c99da690be0b288e08023dd/href</a></iframe><h4>Training Word Embeddings using Word2Vec</h4><p>In order for our model to understand the raw text data, we need to vectorize it, Bag of Words and TF-IDF are very common approaches for vectorizing. However, since I would be using an artificial neural network as my model the sparse nature of BOW and TFIDF would pose a problem. Thus I decided to go for word embeddings, which are dense vector representations and thus perfect for our neural network.</p><p>The way people converse on StackOverflow is very technical and they use a very specific vocabulary of words, so it is not a good idea to use pre-trained word embeddings because they are trained on plain English text and would not able to understand the relation between the words in our vocabulary. Thus I decided to train a Word Embeddings model from scratch using Word2Vec.</p><p>Thus after successful training, we get the following results:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50d77f9e7e10b25a5244b3afce4126f2/href">https://medium.com/media/50d77f9e7e10b25a5244b3afce4126f2/href</a></iframe><pre>Terms most similar to &quot;python&quot;<br>[(&#39;python3&#39;, 0.6342744827270508), (&#39;perl&#39;, 0.5586778521537781), (&#39;bash&#39;, 0.5048258304595947), (&#39;python36&#39;, 0.4876842796802521),<br>(&#39;pyhton&#39;, 0.482374906539917), (&#39;python27&#39;, 0.4808519780635834), (&#39;lua&#39;, 0.4791009724140167), (&#39;ruby&#39;, 0.4676322937011719), (&#39;py<br>thon2&#39;, 0.46161943674087524), (&#39;matlab&#39;, 0.46039116382598877)]<br>--------------------------------------------------------------------<br>Terms most similar to &quot;Node&quot;<br>[(&#39;nodes&#39;, 0.6288038492202759), (&#39;firstnode&#39;, 0.5410162210464478), (&#39;nodetext&#39;, 0.5174486041069031), (&#39;new_node&#39;, 0.51485019922<br>25647), (&#39;node1&#39;, 0.5093669891357422), (&#39;nodedata&#39;, 0.4883999228477478), (&#39;newnode&#39;, 0.481780469417572), (&#39;nodename&#39;, 0.4755<br>1754117012024), (&#39;node2&#39;, 0.4689573049545288), (&#39;node3&#39;, 0.45593389868736267)]<br>--------------------------------------------------------------------<br>Terms most similar to &quot;java&quot;<br>[(&#39;scala&#39;, 0.4836808145046234), (&#39;groovy&#39;, 0.46810388565063477), (&#39;c&#39;, 0.4596169888973236), (&#39;python&#39;, 0.44173622131347656), (&#39;jd<br>k7&#39;, 0.4374290406703949), (&#39;kotlin&#39;, 0.4267594814300537), (&#39;jdk&#39;, 0.42633625864982605), (&#39;eclipse&#39;, 0.4209193289279938), (&#39;netbea<br>n&#39;, 0.41293397545814514), (&#39;jdk6&#39;, 0.40997016429901123)]<br>--------------------------------------------------------------------<br>Terms most similar to &quot;server&quot;<br>[(&#39;sever&#39;, 0.6860402822494507), (&#39;client&#39;, 0.666864812374115), (&#39;webserver&#39;, 0.5926132798194885), (&#39;servers&#39;, 0.5850958824157715<br>), (&#39;machine&#39;, 0.5267062187194824), (&#39;service&#39;, 0.5109481811523438), (&#39;remote&#39;, 0.5007282495498657), (&#39;backend&#39;, 0.49881637096<br>40503), (&#39;remotely&#39;, 0.4766204357147217), (&#39;locally&#39;, 0.47090765833854675)]</pre><h4>Training neural models</h4><p>Preparing the data for training the model includes:</p><ol><li>One-Hot encoding the data</li><li>Splitting into training and test data</li><li>Tokenizing and padding</li><li>Creating an embedding matrix</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cd53edcf9ccacf67b97e1442ce1b28f/href">https://medium.com/media/7cd53edcf9ccacf67b97e1442ce1b28f/href</a></iframe><p>Before splitting I have combined the post_corpus and code columns because I wanted to include the code section of the question also because I also contain information that can be used in predicting the tag.</p><p>I am training here three models which are GRU, CNN, LSTM and based on the f1_micro, recall_micro, precission_micro results of all three I choose the final model</p><h4>Model 1</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44ae0fd2093b70d4bd308609d452f557/href">https://medium.com/media/44ae0fd2093b70d4bd308609d452f557/href</a></iframe><h4>Model 2</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d9fb3ddcc7396f22f5da8ad6b1a577c/href">https://medium.com/media/7d9fb3ddcc7396f22f5da8ad6b1a577c/href</a></iframe><h4>Model 3</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05fcd0bcd7ff88b487d9ff9d79816a45/href">https://medium.com/media/05fcd0bcd7ff88b487d9ff9d79816a45/href</a></iframe><h4>Model Comparison</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b57f337f7b32b4939c8529bff2eddb04/href">https://medium.com/media/b57f337f7b32b4939c8529bff2eddb04/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*Ck24J-ClCSOovDBsIezVxg.png" /><figcaption>Model Comparison</figcaption></figure><h4>Observation</h4><ul><li>I trained three neural network model GRU based, CNN based and one LSTM based.</li><li>The best recall, precision, and f1 value found with the GRU model so I choose this model.</li></ul><h3>9. Creating the Search pipeline</h3><p>Till now, we have trained the model which will predict the tag of the input search query of the user.</p><p>Now in this section, I will show you how we are going to use this predicted tag along with so other simple but useful hack with the cosine similarity to get the most similar result for the input search query.</p><p>To know how similar the two sentences are we can find the distance between them. I order to be able to compute such a distance, the sentences must belong in the same vector space. This is done through a sentence embeddings.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1262a55fafd2f383101a1c527f7ed529/href">https://medium.com/media/1262a55fafd2f383101a1c527f7ed529/href</a></iframe><h4>Observation</h4><ul><li>Firstly I loaded the data, trained model, title embedding, and W2Vec model.</li><li>Then I defined the necessary functions like loss, evaluation metrics, vectorizing, etc.</li><li>After that, I calculated the TFIDF of all the titles in the data.</li><li>After all this, I defined a function named ‘searchresult’ which takes two parameters the input search string and the other is the number of the search result to return in the output.</li><li>Inside the function, I am processing the input search query and after that, I am predicting the tags and once the tag is predicted I am filtering the main data to only the data having the predicted tag also I am calculating the TFIDF of the input query and after that, I am calculating the cosine similarity of the input search query and all the filtered data titles.</li><li>After calculating the cosine similarity I am adding the overall score, sentiment polarity and the TFIDF of the input search multiplied with the appropriate weights.</li><li>And In the end, I am returning the dictionary of the desired thing which I want to display in the search output.</li></ul><h4>GUI for the search engine</h4><p>As a search engine should look like a search bar not like an exposed code so in this section I will going teach you how I build on so the user does not have to dig into code to enter the query.</p><p>You can find the code of the GUI below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/755df2b0a5dc2b8d1894d276d8b0d8d5/href">https://medium.com/media/755df2b0a5dc2b8d1894d276d8b0d8d5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*vfYiO9Sijruq0QUqI0TB4g.png" /><figcaption>Search engine GUI</figcaption></figure><p>For the GUI I have used the ‘ipywidgets’ It is a very simple GUI with nothing fancy in it, it contains title, two input box, and one search button.</p><ul><li>I read the two input string from the text box the search query and the number of results to print.</li><li>After that, I preprocess the input query using a function named preprocess_text which I have written in the utils notebook to keep the project more organized.</li><li>preprocess_text returns me a processed text and then I pass this processed text and number of results to print, as an argument to the ‘searchresult’ method which I have defined in a separate notebook named as search.ipynb and the above-explained operation happened and get the most similar results</li><li>And then I display that out as a search result</li></ul><p>You can see the demo of the search engine in the bellow video block:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*qMEacLpqJkiKFTe-_wOX0A.gif" /><figcaption>Search engine demo video</figcaption></figure><h3>10. Scope of Improvement</h3><p>As every model has there scope of improvement this project also has, you are free to do changes and try some other models with different architecture to improve the search result.</p><h3>11. References</h3><ul><li><a href="https://www.appliedaicourse.com/course/11/applied-machine-learning-online-course">https://www.appliedaicourse.com/course/11/applied-machine-learning-online-course</a></li><li><a href="https://www.kaggle.com/stackoverflow/stackoverflow"><em>https://www.kaggle.com/stackoverflow/stackoverflow</em></a></li><li><a href="https://www.kaggle.com/sohier/introduction-to-the-bq-helper-package"><em>https://www.kaggle.com/sohier/introduction-to-the-bq-helper-package</em></a></li><li><a href="https://machinelearningmastery.com/develop-word-embeddings-python-gensim/">https://machinelearningmastery.com/develop-word-embeddings-python-gensim/</a></li><li><a href="https://www.kaggle.com/pierremegret/gensim-word2vec-tutorial">https://www.kaggle.com/pierremegret/gensim-word2vec-tutorial</a></li></ul><p><strong>You can connect with me on LinkedIn</strong><br><a href="https://www.linkedin.com/in/vashistnarayan-singh/">https://www.linkedin.com/in/vashistnarayan-singh/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=139e2b2e8860" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/stackoverflow-based-semantic-search-engine-139e2b2e8860">Stackoverflow Based Semantic Search Engine</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer Transaction Prediction]]></title>
            <link>https://medium.com/analytics-vidhya/customer-transaction-prediction-31c28beba1ea?source=rss-454105000991------2</link>
            <guid isPermaLink="false">https://medium.com/p/31c28beba1ea</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[VASHIST NARAYAN Singh]]></dc:creator>
            <pubDate>Thu, 12 Dec 2019 15:39:26 GMT</pubDate>
            <atom:updated>2019-12-19T04:51:52.015Z</atom:updated>
            <content:encoded><![CDATA[<p>As the title says, this blog is about a kaggle competition titled Santander customer transaction. For any finance-based company, the most crucial thing is to have the information about whether the customer is going to use their service in the future or not. The same situation is given by Santander an online bank who asked us to solve a challenge to predict whether a customer will do a transaction in the future or not. I will tell you how can you solve this challenge and what feature engineering I have done to get the AUC result of 90% and rank in the top 4%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4vg0lW_YVZluXxzdGMfCIQ.png" /></figure><h3><strong>Problem Statement</strong></h3><p>The problem statement of this kaggle challenge is to predict if a customer will do the future transaction or not irrespective of the amount of money transacted. The constraints in this are we should give the probability-based prediction’s.</p><h3><strong>Machine learning problem statement</strong></h3><p>So till now, we have seen what the problem and what are the constraints of the problem, but all those stuff were related to the business point of view. For solving this challenge using machine learning we have to transform this into a machine learning problem statement and guys this thing is not only limited to this particular problem statement but all other challenges as well. Before solving any real-world problem we first have to transform it into a machine learning problem. For this question, our ML problem statement will be like: “<strong>This one is a classical Binary classification machine learning problem where we have to predict the customer will do a future transaction or not with the evaluation metrics be AUC.</strong>”</p><h3><strong>Dataset</strong></h3><p>The dataset is anonymized so we cannot know which feature is what. There is a total of 200 features in this data set along with ID_code and target columns. The target columns contain 0 and 1 value where 0 means the customer will not do a transaction and 1 means the customer will do a transaction.<br>You can download the dataset from the following link:</p><p><a href="https://www.kaggle.com/c/santander-customer-transaction-prediction/overview/evaluation">https://www.kaggle.com/c/santander-customer-transaction-prediction/</a></p><h3><strong>Importing the libraries</strong></h3><p>Till now you are completely aware of the problem statement and its constraints so now let gets are hand dirty and start with the coding part. First, we start by importing all the necessary libraries.</p><pre>import pandas as pd<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br>import numpy as np<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import roc_curve, auc,roc_auc_score<br>from sklearn.model_selection import StratifiedKFold<br>from sklearn.model_selection import RandomizedSearchCV<br>import lightgbm as lgb<br>import timeit<br>import time</pre><h3><strong>Exploratory Data Analysis</strong></h3><p>So we are done with loading the libraries, let’s start with the exploratory data analysis (EDA). Guys EDA is the most important step in solving any machine learning problem it helps you to know your dataset deeper and helps you in deriving the new feature for better learning of your model. we start our EDA by loading the data first.</p><pre>data = pd.read_csv(‘train.csv’)#train data<br>data_test = pd.read_csv(‘test.csv’)#test data</pre><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NDUObygma8EwtZXMo9xvQ.png" /></figure><p>After loading the train and test data I did some columns rearrangement and put the target column to the end and saved the data into new CSV file name data_train.csv.</p><p><strong>Checking for the balance of dataset</strong></p><pre>data_train[‘target’].value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*IF5Kb4MbVt7hymLIrIPY-Q.png" /></figure><pre>ax = sns.countplot(‘target’,data=data_train)<br>print(“percentage of data belongs to 0 :”, data_train[‘target’].value_counts()[0]*100/200000,”%”)<br>print(“percentage of data belongs to 1 :”, data_train[‘target’].value_counts()[1]*100/200000,”%”)<br>ax.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*sUBgzYwAMh_XfVXxhdJhVw.png" /></figure><p>As we can see that around 90% of data belongs to 0 class and only 10% belongs to class 1 hence we can conclude that the data is an imbalanced data</p><p><strong>Checking for the null values</strong></p><pre>data_train.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*25Xsm0MMlcoc-FSyQJVqOQ.png" /></figure><p>A good thing for us is, as we can see there are no null values in any feature so no need for imputation of missing value. <br>So now let’s check what is the distribution of all the feature towards the target values 0 and 1 for that I have written the following function and since we have 200 feature we plot this distribution in two parts in the first part I have taken 100 features and in the second I have taken the rest 100 features.</p><pre>def feature_distribution(data_1,data_2,target_0,target_1,features_list):<br># Here we are setting the style of the plot and grid in it<br>sns.set_style(‘whitegrid’)<br>plt.figure() # Here we are initializing the plt figure object<br># Here we are creating the subplot and initialzing it size and row col size<br>fig, ax = plt.subplots(10,10,figsize=(18,22))<br>for plot_count, feature in enumerate(features_list):<br>#plotting the plots here for every plot feature<br>plt.subplot(10,10,plot_count+1)<br>#plotting the pdf plot for every feature towards the target value<br>sns.distplot(data_1[feature], hist=False,label=target_0)<br>sns.distplot(data_2[feature], hist=False,label=target_1)<br>plt.xlabel(feature, fontsize=9)# Here we are setting the x axis label<br>locs, labels = plt.xticks()<br># Here we are setting the ticks for x and y axis<br>plt.tick_params(axis=’x’, which=’major’, labelsize=6, pad=-6)<br>plt.tick_params(axis=’y’, which=’major’, labelsize=6)<br>plt.show();<br>## Dstribution for the first 100 features<br>target_0_data = data.loc[data_train[‘target’] == 0]<br>target_1_data = data.loc[data_train[‘target’] == 1]<br>features = data.columns.values[1:101]<br>feature_distribution(target_0_data, target_1_data, ‘0’, ‘1’, features)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*ezIYU0af3T9gsLsOUVof9g.png" /><figcaption>sample distribution of first 100 features towards target class</figcaption></figure><p>Similarly, we have done for the rest of the 100 features and came out to be the following conclusions :<br>1) <em>By looking at the distribution of each feature towards the target values I found that most of the features have different distribution<br>for the target values.<br>2) We can also say that there are some features that are quite close to normal distribution not completely but a little.</em><br>3) <em>Hence I can say that there are some kind of processing is done on the data.</em></p><p><strong>Let&#39;s check the distribution of mean and std of the data<br></strong>Like we have done with the features let’s check for the mean and std distribution of data.</p><pre>plt.figure(figsize=(16,6))<br>sns.set_style(‘whitegrid’)<br>features = data_train.columns.values[1:202]<br>plt.title(“Distribution of mean values per row in the data”)<br>plt.xlabel(‘mean value’)<br>plt.ylabel(‘pdf value’)<br>sns.distplot(data_train[features].mean(axis=1),color=”green”, kde=True,bins=120)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*KSpa4dsD4-7gbhqh1JMLxg.png" /><figcaption>Distribution of mean values per row in data</figcaption></figure><p>Following observation is made after looking at the mean graph for row values of data:<br>1) <em>The above graph shows the distribution of the means of each feature along the row and it seems to follow the kind of gaussian.<br>2) The graph looks kind of Gaussian with a mean value of 6.7342.<br>3) From the above graph, we can say that there are around 80% of a feature whose mean lies between 6.5 and 7.0</em></p><p>Similarly, we plot the distribution of mean values per col in data and got the following graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*Tb8UeS2wiZHbXAiZTosoyw.png" /><figcaption>Distribution of mean values per col in data</figcaption></figure><p>Following observation is made after looking at the mean graph for col values of data<br>1) <em>The above graph is of the mean distribution of each feature column-wise.<br>2) The columns wise mean distribution is not gaussian.</em><br>3) <em>The majority of columns having the mean value between -10 and 20.</em></p><p>Similarly, I plot the distribution of standard deviation along the row and col of data and got the following plot and observations respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*jhqRjtdjHTbqjur1sJ8qFQ.png" /><figcaption>Distribution of std values per row in the data</figcaption></figure><p>The observation was as follows:<br>1) <em>we can see that the standard deviation distribution of each feature along the row also kind of follow the Gaussian distribution not<br>exactly but as per the shape of the curve.<br>2) Around 60% of features having the standard deviation around in the range of 9.3–10.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*Ze9E9nHYgv0y3bbR-BwBFA.png" /><figcaption>Distribution of std values per col in the data</figcaption></figure><p>Observations<br>1) <em>As the graph, we can say the distribution of the standard deviation along with features along the column comes from some other distribution.<br>2) There are a large number of features having a deviation in the range of 0 &amp; 6.<br>3) With a minimum standard deviation of 0 and a maximum of around 21 or something.</em></p><p><strong>Distribution of the mean value dataset, grouped by the value of a target.<br></strong>Here I have done the above EDA one more time but with some twist, I have grouped mean by the value of a target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*otv5qi88hFqFqyViXWYjeQ.png" /><figcaption>Distribution of mean values per row in the dataset grouped by the value of the target</figcaption></figure><p>Observations:<br>1) <em>The above graph shows the mean distribution of every features towards the target.<br>2) The distribution of every feature towards each class looks kind of similar.<br>3) Hence the features will do well on identifying the target class.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*PRgeSYstIhBz8gRBe01aCw.png" /><figcaption>Distribution of mean values per col in the data set grouped by the value of the target</figcaption></figure><p>Observation:<br>1) <em>Looking at the above graph shows both the distribution quite similarly.<br>2) All features will do we in identifying the target class.<br>3) The majority of features means lies in the range of -10 to 20.</em></p><p><strong>Distribution of min and max value in data as row and col both wise</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*vRACR_GzrovBNgHtwJwsdQ.png" /><figcaption>Distribution of min values per row in the data</figcaption></figure><p>Observation is as follows:<br>1) <em>The above graph shows the distribution of the min values of each feature.<br>2) The plot looks skewed on the right side.<br>3) Majority of features having the min values in the range of -40 to -20.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*u01N__ry7xPW-Y58LJCRxw.png" /><figcaption>Distribution of min values per col in the data</figcaption></figure><p>Observations:<br>1) <em>The above graph is the column-wise min value distribution of each feature.<br>2) We have observed the lower value i.e -80 as the long queue is at the lower side.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*E9OmB2H2Q-SN6WcJcpaK-w.png" /><figcaption>Distribution of max value per row in the data</figcaption></figure><p>Observations:<br>1) <em>The above distribution is the row-wise distribution of each feature max value.<br>2) We can observe the max value of 70 on the right as the long tail is on the right of the graph.<br>3) The graph is skewed towards the left as the long tail is on the right side.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*ugwwnrwas7VbMlSLGdGbaQ.png" /><figcaption>Distribution of max values pr col in the data</figcaption></figure><p>Observation:<br>1) <em>The above distribution is the col wise distribution of each feature max value.<br>2) We can observe the max value of 80 as the long tail of the graph is on the right side.</em></p><p><strong>Now let’s check the correlation between the features<br></strong>So far all my EDA is based on the graphical distribution so now since the data contains numerical features correlation between the features should be checked. So let’s do it.</p><pre>features = data_train.columns.values[1:200] # Here we are getting all th features<br># And here we are calculating the correlation of every featre ad sorting it in ascending order<br>cor_data = data_train[features].corr().abs().unstack().sort_values(kind=”quicksort”).reset_index() <br>cor_data = cor_data[cor_data[‘level_0’] != cor_data[‘level_1’]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*ztiZNoJHN_RYD-enkRG2RA.png" /><figcaption>topmost correlated data</figcaption></figure><h3><strong>Training a model without any feature engineering</strong></h3><p>Herewith correlation I have ended my EDA but you guys can do some more to find out more interesting information about the data. One way to know what is the baseline for your model should be you can train any model on the raw data without doing any feature engineering. And in this section of the blog, I am doing exactly the same thing.</p><pre>start = timeit.default_timer()<br># %%time<br>lgb_model = lgb.LGBMClassifier(boosting_type= ‘gbdt’, objective=’binary’,feature_fraction=0.05,<br> class_weight=’balanced’,num_leaves=8,n_estimators=2000,max_depth=5,<br> learning_rate=0.05,metric=’auc’,bagging_fraction=0.4, n_jobs=-1)<br>lgb_model.fit(X_train, y_train)</pre><pre>predict_y_train = batch_predict(lgb_model, X_train)<br>predict_y_cv = batch_predict(lgb_model, X_cv)</pre><pre>train_fpr, train_tpr, tr_thresholds = roc_curve(y_train, predict_y_train)<br>test_fpr, test_tpr, te_thresholds = roc_curve(y_cv, predict_y_cv)<br>plt.plot(train_fpr, train_tpr, label=”train AUC =”+str(auc(train_fpr, train_tpr)))<br>plt.plot(test_fpr, test_tpr, label=”test AUC =”+str(auc(test_fpr, test_tpr)))<br>plt.legend()<br># plt.xlabel(“K: hyperparameter”)<br>plt.ylabel(“AUC”)<br>plt.title(“ERROR PLOTS”)<br>plt.grid()<br>plt.show()<br>stop = timeit.default_timer()<br>print(‘Time in mins: ‘,(stop — start)/60)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*DAzh6yXU03BDB4375zcsbw.png" /><figcaption>AUC plot of tian and test</figcaption></figure><p>Observation<br>The test AUC of 89.52% achieved without any feature engineering I aim to increase it further using some interesting feature engineering.</p><h3><strong>Feature Engineering</strong></h3><p>From the EDA we can see that the mean, std, min, max, sum, skew, kurtosis, the median can be used as a good feature</p><pre>%time<br># <a href="https://www.youtube.com/watch?v=LEWpRlaEJO8">https://www.youtube.com/watch?v=LEWpRlaEJO8</a><br>idx = features = data.columns.values[2:202]<br>for dataFrame in [data_test, data_train]:<br> dataFrame[‘sum’] = dataFrame[idx].sum(axis=1) <br> dataFrame[‘min’] = dataFrame[idx].min(axis=1)<br> dataFrame[‘max’] = dataFrame[idx].max(axis=1)<br> dataFrame[‘mean’] = dataFrame[idx].mean(axis=1)<br> dataFrame[‘std’] = dataFrame[idx].std(axis=1)<br> dataFrame[‘skew’] = dataFrame[idx].skew(axis=1)<br> dataFrame[‘kurt’] = dataFrame[idx].kurtosis(axis=1)<br> dataFrame[‘med’] = dataFrame[idx].median(axis=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*fzRldMy3trCVRv4qwufKFQ.png" /><figcaption>feature engineered feature</figcaption></figure><p>One more interesting feature could be the rounding of the values of each feature.</p><pre>%%time<br># <a href="https://www.geeksforgeeks.org/numpy-round_-python/">https://www.geeksforgeeks.org/numpy-round_-python/</a><br>features_value = [col for col in data_train.columns if col not in [‘ID_code’,’target’]]<br># In this we ar rounding of the value of each columns and creating a new freature of the same <br>for feature in features_value:<br> data_train[‘round_2’+ feature] = np.round(data_train[feature],2)<br> data_test[‘round_2’+ feature] = np.round(data_test[feature],2)<br> data_train[‘round_1’+ feature] = np.round(data_train[feature],1) <br> data_test[‘round_1’+ feature] = np.round(data_test[feature],1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IjVBnUUo1-Nb5n86hJ7htA.png" /><figcaption>some round feature</figcaption></figure><p>After doing feature engineering we manage to increase the features up to 626.</p><h3><strong>Final Model Training</strong></h3><p>Once the new features are created its time to train a model and see what is the value of the AUC and did I manage to increase it more than 89.52%.</p><pre>parameter = {<br> ‘bagging_freq’: 5,<br> ‘bagging_fraction’: 0.4,<br> ‘boost_from_average’:’false’,<br> ‘boost’: ‘gbdt’,<br> ‘feature_fraction’: 0.05,<br> ‘learning_rate’: 0.01,<br> ‘max_depth’: -1, <br> ‘metric’:’auc’,<br> ‘min_data_in_leaf’: 80,<br> ‘min_sum_hessian_in_leaf’: 10.0,<br> ‘num_leaves’: 13,<br> ‘num_threads’: 8,<br> ‘tree_learner’: ‘serial’,<br> ‘objective’: ‘binary’, <br> ‘verbosity’: 1<br>}<br># <a href="https://www.kaggle.com/adrianlievano/light-gbm-with-stratified-kfold">https://www.kaggle.com/adrianlievano/light-gbm-with-stratified-kfold</a><br># Getting all the features name except the ID_code , target <br>features = [col for col in data_train_final_.columns if col not in [‘ID_code’, ‘target’]]<br># Inititalizing the K-Fold object<br>K_folds = StratifiedKFold(n_splits=10, shuffle=False, random_state=44000) <br># this creates the empty numpy array of length of x in which we store the prediction of every validation data<br>val_pred = np.zeros(len(x))<br># In this we keep the predicted output of the test data <br>predictions_test = np.zeros(len(data_test_final_))<br>#In this loop we are doing the training and prediction for each folds and we are getting the train and valid data <br># using the trn_idx and val_idx <br>for n_fold, (trn_idx, val_idx) in enumerate(K_folds.split(x.values, y.values)):<br> print(“Fold {}”.format(n_fold))<br> # Getting the train and validation data from the x data<br> train_data = lgb.Dataset(x.iloc[trn_idx][features], label=y.iloc[trn_idx]) <br> valid_data = lgb.Dataset(x.iloc[val_idx][features], label=y.iloc[val_idx])<br> # Here we are training lightgbm model on train and valid dataset<br> num_round = 1000000<br> classifier = lgb.train(parameter, train_data, num_round, <br> valid_sets = [train_data, valid_data],<br> verbose_eval=1000, early_stopping_rounds = 3000)<br> # Here we are doing the prediction on the valid data <br> val_pred[val_idx] = classifier.predict(x.iloc[val_idx][features], num_iteration=classifier.best_iteration)<br> # And here we are doing the prediction on the test data<br> predictions_test += classifier.predict(data_test_final_[features], <br> num_iteration=classifier.best_iteration) / K_folds.n_splits<br>print(“CV score: {:&lt;8.5f}”.format(roc_auc_score(y, val_pred)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*EMO-kVpJ-B234rOkLgn7Qw.png" /></figure><p>As you can see the AUC score of 90% is achieved hence its time to do the submission on kaggle and check the score there.</p><p><strong>Creating the submission CSV</strong></p><pre>sub_df = pd.DataFrame({“ID_code”:data_test[“ID_code”].values})<br>sub_df[“target”] = predictions<br>sub_df.to_csv(“submission.csv”, index=False)</pre><h3><strong>Result’s</strong></h3><p>After creating the submission file its time for the test on kaggle and see the result. so keep your finger crossed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*XoAgnAHCyTqXViHfTvmrFA.png" /><figcaption>Result of kaggle submission</figcaption></figure><p>As you can see I managed to achieve the AUC of 0.90041 on the kaggle test data. And as we know in machine learning no solution is a perfect solution there is always a scope of improvement in every model my solution can also be improved further.</p><h3><strong>Scope of improvement</strong></h3><p>You can improve this solution further by incorporating some other feature engineering one of them is frequency encoding and by removing the synthetic test data thank’s to this awesome technique shared by YaG320 in his kernel <a href="https://www.kaggle.com/yag320/list-of-fake-samples-and-public-private-lb-split">https://www.kaggle.com/yag320/list-of-fake-samples-and-public-private-lb-split</a>). You can also perform data augmentation and improve your AUC by up to 92%.</p><h3><strong>References</strong></h3><p><a href="https://www.kaggle.com/c/santander-customer-transaction-prediction/overview/evaluation">https://www.kaggle.com/c/santander-customer-transaction-prediction/overview/evaluation</a><br><a href="https://www.youtube.com/watch?v=LEWpRlaEJO8">https://www.youtube.com/watch?v=LEWpRlaEJO8</a><br><a href="https://www.kaggle.com/yag320/list-of-fake-samples-and-public-private-lb-split">https://www.kaggle.com/yag320/list-of-fake-samples-and-public-private-lb-split</a>)</p><p><strong>You can connect with me on LinkedIn</strong><br><a href="https://www.linkedin.com/in/vashistnarayan-singh/">https://www.linkedin.com/in/vashistnarayan-singh/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31c28beba1ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/customer-transaction-prediction-31c28beba1ea">Customer Transaction Prediction</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>