<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shubhanshu Gupta on Medium]]></title>
        <description><![CDATA[Stories by Shubhanshu Gupta on Medium]]></description>
        <link>https://medium.com/@shubhanshugupta?source=rss-269f5616997f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tIZGI7lGaIPj8MAVnJTj1A.png</url>
            <title>Stories by Shubhanshu Gupta on Medium</title>
            <link>https://medium.com/@shubhanshugupta?source=rss-269f5616997f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:34:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shubhanshugupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Collocations in NLP using NLTK library]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/collocations-in-nlp-using-nltk-library-2541002998db?source=rss-269f5616997f------2"><img src="https://cdn-images-1.medium.com/max/2600/1*2iHQUtwo0zzIyWUyDz7t-g.jpeg" width="5184"></a></p><p class="medium-feed-snippet">Collocations are phrases or expressions containing multiple words, that are highly likely to co-occur. For example&#x200A;&#x2014;&#x200A;&#x2018;social media&#x2019;&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/data-science/collocations-in-nlp-using-nltk-library-2541002998db?source=rss-269f5616997f------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/collocations-in-nlp-using-nltk-library-2541002998db?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2541002998db</guid>
            <category><![CDATA[text-analytics]]></category>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Thu, 19 Mar 2020 16:04:33 GMT</pubDate>
            <atom:updated>2020-03-19T16:04:33.444Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A Survey of API Management Platforms]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shubhanshugupta/a-survey-of-api-management-platforms-348d80348a3f?source=rss-269f5616997f------2"><img src="https://cdn-images-1.medium.com/max/2600/1*IYTrPoHKZnmUU7xUq58_ig.jpeg" width="5184"></a></p><p class="medium-feed-snippet">In my previous blog, I discussed how I landed up interning at Dentsu. I also discussed that I worked on scouting and building a POC for a&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@shubhanshugupta/a-survey-of-api-management-platforms-348d80348a3f?source=rss-269f5616997f------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shubhanshugupta/a-survey-of-api-management-platforms-348d80348a3f?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/348d80348a3f</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[api-management]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Sat, 25 May 2019 10:36:04 GMT</pubDate>
            <atom:updated>2019-05-25T10:36:04.956Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Multi-Class Classification in Text using R]]></title>
            <link>https://medium.com/data-science/multi-class-classification-in-text-using-r-e6cf72ef1da3?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6cf72ef1da3</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[tedx]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Thu, 20 Dec 2018 01:22:40 GMT</pubDate>
            <atom:updated>2018-12-20T03:29:46.998Z</atom:updated>
            <content:encoded><![CDATA[<h4>Predicting Ted Talk Ratings</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lt7kQ3CGgQwXEKcY" /><figcaption>Photo by <a href="https://unsplash.com/@hermez777?utm_source=medium&amp;utm_medium=referral">Hermes Rivera</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This blog is in continuation to my NLP blog series. In the previous blogs, I discussed <a href="https://medium.com/@shubhanshugupta/data-preprocessing-in-r-2f0e25487bb">data pre-processing steps in R</a> and <a href="https://medium.com/datadriveninvestor/emotions-in-ted-talks-text-analytics-in-r-1b6016f3316e">recognizing emotions present in ted talks</a>. In this blog, I am going to predict the ratings of the ted talks given by viewers. This would require a multi-class classification and quite a bit of data cleaning and preprocessing. We will discuss each step in detail below.</p><p>So, let’s dive in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*riIiqesfHxXnwTS7" /><figcaption>A way to visualize text classification</figcaption></figure><h4>Data Cleaning and Parsing</h4><p>The ratings column in the <a href="https://www.kaggle.com/rounakbanik/ted-talks">ted talk dataset</a> looks like Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z352_WmgvFGp8sOC" /><figcaption>Figure 1</figcaption></figure><p>The screenshot of the column above, Figure 1signifies how many people have rated a particular talk to be “Inspiring”, “Beautiful”, “Ingenious”, “Persuasive”, etc. The count key in the JSON signifies the value of the rating given to the talk. So for example, the talk has been rated “Inspiring” by 385 people and just 2 people have rated the talk to be “Funny”. Our objective, here, is to get the rating with the highest count for each talk.</p><p>One of the primary concern is to parse the ratings column. I have used the <em>gsub</em> function for replacing the single quote with double quotes. I found the details and explanation about <em>gsub</em> <a href="http://rfunction.com/archives/2354">here</a>.</p><pre>library(jsonlite)</pre><pre>formatted_ted_ratings &lt;- gsub(“‘“,’”’,ted_talks$ratings)</pre><p>The next step was to parse the JSON to get a list of id, rating name and rating count. The <em>jsonlite</em> library in R, provides functions to stream, validate and prettify JSON data. The <a href="https://www.rdocumentation.org/packages/RJSONIO/versions/1.3-0/topics/fromJSON">fromJSON function</a> is used to de-serialize JSON objects into R objects. Finally, <em>purrr::map</em> function, applies a function (fromJSON in our case) to each element in the list. The documentation and implementation can be read <a href="https://jennybc.github.io/purrr-tutorial/ls01_map-name-position-shortcuts.html">here</a>.</p><pre>ted_ratings &lt;- purrr::map(formatted_ted_ratings, jsonlite::fromJSON)</pre><p>The code block above gives us a neat list of parsed rating column. It looks like Figure 2 .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*75tY6IIe0SW18jSc" /><figcaption>Figure 2: Parsed rating column</figcaption></figure><p>In the next step, I am going to make a new column “highest_rating” in which I will store the rating with highest count for every talk. Post that, I will convert the column into factor which will effectively give us 13 unique factors (ratings) to deal with. I hope, by this point you must have got a hint that this highest_rating column with 13 factor variables will be used for multi-class classification. In the sense, a binary classification problem has two classes to classify a data point, e.g. True and False. Whereas, in this problem we have to deal with the classification of a data point into one of the 13 classes and hence, this is a multi-class classification problem.</p><pre>for (i in (1:length(ted_ratings))) {<br> ted_ratings_df &lt;- ted_ratings[[i]]<br> highest_rating_count &lt;- ted_ratings_df[which(ted_ratings_df$count == max(ted_ratings_df$count)), ]<br> ted_talks$highest_rating[i] &lt;- highest_rating_count$name<br>}</pre><pre>ted_talks$highest_rating = as.factor(ted_talks$highest_rating)</pre><p>With the above step, our dataset preparation is now complete.</p><h4>Data Modeling</h4><p>We will now split our dataset into training and test. I have divided my dataset in 60:40 ratio.</p><pre>trainObs &lt;- sample(nrow(ted_talks), .6 * nrow(ted_talks), replace = FALSE)<br>testObs &lt;- sample(nrow(ted_talks), .4 * nrow(ted_talks), replace = FALSE)</pre><pre>train_dat &lt;- ted_talks[trainObs,]<br>test_dat &lt;- ted_talks[testObs,]</pre><p>I will now apply all the pre-processing steps to my training and test data (separately). Somehow, I was in dual mindset: whether to split the DTM into train and test or split the dataset and then prepare their DTM individually. Somehow, I chose the latter option. You can try with the former option and let me know if it works out fine for you.</p><p>I also took care of sparsity, something which I discussed in good detail in my <a href="https://medium.com/@shubhanshugupta/data-preprocessing-in-r-2f0e25487bb">blog</a>. I also renamed my target variable as “y” instead of highest_rating for better intuitiveness.</p><pre>train_corpus &lt;- VCorpus(VectorSource(train_dat$transcript))</pre><pre>##Removing Punctuation<br>train_corpus &lt;- tm_map(train_corpus, content_transformer(removePunctuation))</pre><pre>##Removing numbers<br>train_corpus &lt;- tm_map(train_corpus, removeNumbers)</pre><pre>##Converting to lower case<br>train_corpus &lt;- tm_map(train_corpus, content_transformer(tolower))</pre><pre>##Removing stop words<br>train_corpus &lt;- tm_map(train_corpus, content_transformer(removeWords), stopwords(“english”))</pre><pre>##Stemming<br>train_corpus &lt;- tm_map(train_corpus, stemDocument)</pre><pre>##Whitespace<br>train_corpus &lt;- tm_map(train_corpus, stripWhitespace)</pre><pre># Create Document Term Matrix<br>dtm_train &lt;- DocumentTermMatrix(train_corpus)</pre><pre>train_corpus &lt;- removeSparseTerms(dtm_train, 0.4)</pre><pre>dtm_train_matrix &lt;- as.matrix(train_corpus)<br>dtm_train_matrix &lt;- cbind(dtm_train_matrix, train_dat$highest_rating)</pre><pre>colnames(dtm_train_matrix)[ncol(dtm_train_matrix)] &lt;- “y”</pre><pre>training_set_ted_talk &lt;- as.data.frame(dtm_train_matrix)<br>training_set_ted_talk$y &lt;- as.factor(training_set_ted_talk$y)</pre><p>Now that we have our training dataset ready, we can train our model. I am using <em>caret</em> package and <em>svmLinear3</em> method in caret. <em>svmLinear3</em> provides <em>L2 regularization</em> in SVM with <em>Linear Kernel</em>. Agreed, that’s a lot of technical jargon which I am purposely not explaining here because that’s for another blog altogether. Meanwhile, I am going to leave some links for you to understand <a href="https://developers.google.com/machine-learning/crash-course/regularization-for-simplicity/l2-regularization"><em>L2 regularization</em></a>, and <a href="https://www.svm-tutorial.com/2014/10/svm-linear-kernel-good-text-classification/"><em>SVM with Linear Kernel</em></a>.</p><pre>library(caret)</pre><pre>review_ted_model &lt;- train(y ~., data = training_set_ted_talk, method = ‘svmLinear3’)</pre><pre>Preparing our test data. It’s the same repetitive procedure.</pre><pre>test_corpus &lt;- VCorpus(VectorSource(test_dat$transcript))</pre><pre>##Removing Punctuation<br>test_corpus &lt;- tm_map(test_corpus, content_transformer(removePunctuation))</pre><pre>##Removing numbers<br>test_corpus &lt;- tm_map(test_corpus, removeNumbers)</pre><pre>##Converting to lower case<br>test_corpus &lt;- tm_map(test_corpus, content_transformer(tolower))</pre><pre>##Removing stop words<br>test_corpus &lt;- tm_map(test_corpus, content_transformer(removeWords), stopwords(“english”))</pre><pre>##Stemming<br>test_corpus &lt;- tm_map(test_corpus, stemDocument)</pre><pre>##Whitespace<br>test_corpus &lt;- tm_map(test_corpus, stripWhitespace)</pre><pre># Create Document Term Matrix<br>dtm_test &lt;- DocumentTermMatrix(test_corpus)</pre><pre>test_corpus &lt;- removeSparseTerms(dtm_test, 0.4)</pre><pre>dtm_test_matrix &lt;- as.matrix(test_corpus)</pre><h4>Model Accuracy and other metrics</h4><p>I will now check the accuracy/performance of our model on test data.</p><pre>#Build the prediction <br>model_ted_talk_result &lt;- predict(review_ted_model, newdata = dtm_test_matrix)</pre><pre>check_accuracy &lt;- as.data.frame(cbind(prediction = model_ted_talk_result, rating = test_dat$highest_rating))</pre><pre>library(dplyr)<br>check_accuracy &lt;- check_accuracy %&gt;% mutate(prediction = as.integer(prediction) — 1)</pre><pre>check_accuracy$accuracy &lt;- if_else(check_accuracy$prediction == check_accuracy$rating, 1, 0)<br>round(prop.table(table(check_accuracy$accuracy)), 3)</pre><pre>library(performanceEstimation)<br>classificationMetrics(as.integer(test_dat$highest_rating), model_ted_talk_result)</pre><pre>most_common_misclassified_ratings = check_accuracy %&gt;% filter(check_accuracy$accuracy == 0) %&gt;%<br> group_by(rating) %&gt;%<br> summarise(Count = n()) %&gt;%<br> arrange(desc(Count)) %&gt;%<br> head(3)</pre><pre>##Most commong missclassified rating<br>levels(train_dat$highest_rating)[most_common_misclassified_ratings$rating]</pre><p>The model metrics are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grXwT8Zu1qiZKcrxCv8AXw.png" /><figcaption>Model metrics</figcaption></figure><p>The top 3 most commonly <strong>misclassified</strong> ratings are: “Inspiring”, “Informative”, “Fascinating”. You can read more about <em>micro</em> and<em> macro F1 scores</em> from <a href="https://blog.revolutionanalytics.com/2016/03/com_class_eval_metrics_r.html#macro">here</a> and <a href="https://datascience.stackexchange.com/questions/15989/micro-average-vs-macro-average-performance-in-a-multiclass-classification-settin/16001">here</a>.</p><h4>Final Remarks</h4><p>In this article, we have discussed multi-class classification of text. It was pretty challenging data problem for me to solve, primarily because:</p><ul><li>I had uptil now dealt only with binary classification problems</li><li>The data manipulation steps leading to 13 classes for every individual rating column. Single quotes had ruined what looked like a perfect JSON column. Without any intuition for the same, it took a lot of hair pulling to figure out that single quotes were causing the problem. I eventually managed to parse it using gsub.</li></ul><p>Finally, the model scores above are heavily reliant on the method we use in the train method. I used SVM with Linear Kernel and L2 regularization which in itself was computationally heavy. There are other methods which you can try, but computational resources could be an issue. Please do let me know what other method(s) you used and the scores (accuracy and F1) that you get.</p><p>That’s all for now. I will discuss a new task related to NLP in my next blog. I have written other <a href="https://medium.com/@shubhanshugupta/engineering-challenges-of-streaming-a-million-concurrent-json-data-streams-from-product-to-crm-360506b29aca">posts</a> related to software engineering as well. You might want to check them out <a href="https://shubhanshugupta.com/blog/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6cf72ef1da3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/multi-class-classification-in-text-using-r-e6cf72ef1da3">Multi-Class Classification in Text using R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Emotions in Ted Talks: Text Analytics in R]]></title>
            <link>https://medium.datadriveninvestor.com/emotions-in-ted-talks-text-analytics-in-r-1b6016f3316e?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b6016f3316e</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Thu, 29 Nov 2018 19:12:01 GMT</pubDate>
            <atom:updated>2018-12-15T05:53:54.457Z</atom:updated>
            <content:encoded><![CDATA[<h3>Emotion Classification in Text using R</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/0*vIXGcGHIJ8Wa6eqU" /></figure><p>This post is in continuation with my NLP blog series. You might want to checkout my <a href="https://medium.com/@shubhanshugupta/data-preprocessing-in-r-2f0e25487bb">previous blog</a> in which I discussed data pre-processing in R. In this blog, I will determine the emotions that are associated with the Ted Talks. At the end, I will compute a HeatMap of emotions and talks to aid in our visualization.</p><p>So, without further ado, let’s dive in!</p><p>As I have already discussed detailed data pre-processing steps in my last blog post, I will assume that you have already prepared a Document Term Matrix.</p><p>We will begin by importing <a href="http://www.wjh.harvard.edu/~inquirer/homecat.htm">General Inquirer Category Listings</a> Dataset. The dataset sources word categories from 4 different dictionaries. A particular category houses several words pertaining to the semantic meaning for that category. For example — The “Pleasur” category has words indicating the enjoyment of a feeling, including words indicating confidence, interest and commitment.</p><pre>require(“tm.lexicon.GeneralInquirer”)</pre><pre>words_with_emotions &lt;- terms_in_General_Inquirer_categories(“EMOT”)<br>words_with_pleasur &lt;- terms_in_General_Inquirer_categories(“Pleasur”)<br>words_with_pain &lt;- terms_in_General_Inquirer_categories(“Pain”)<br>words_with_virtue &lt;- terms_in_General_Inquirer_categories(“Virtue”)<br>words_with_vice &lt;- terms_in_General_Inquirer_categories(“Vice”)</pre><p>For determining emotions in Ted Talks, choosing categories is an open-ended task. I picked up categories which I felt would best suit the emotions in this talks. Later, as we’ll see, the heatmap would validate my choices. You can pick up completely different categories for your practice and it would still be fine.</p><p>Next, we are going to compute a score which will signify how many terms in the DTM match with the words present in the categories imported in the step above. We use tm_term_score function for this purpose.</p><pre>emo_score &lt;- tm_term_score(dtm, words_with_emotions)<br>pleasure_score &lt;- tm_term_score(dtm, words_with_pleasur)<br>pain_score &lt;- tm_term_score(dtm, words_with_pain)<br>virtue_score &lt;- tm_term_score(dtm, words_with_virtue)<br>vice_score &lt;- tm_term_score(dtm, words_with_vice)</pre><p>After computing the score, I am going to prepare a dataframe containing the category score for each document.</p><pre>library(dplyr)<br>library(tibble)<br>library(tidyr)<br>library(tidytext)<br>library(gplots)</pre><pre>doc_emo_df &lt;- tbl_df(emo_score) %&gt;% rownames_to_column %&gt;% rename(document = rowname) %&gt;% rename(emotional = value)<br>doc_pleasure_df &lt;- tbl_df(pleasure_score) %&gt;% rownames_to_column %&gt;% rename(document = rowname) %&gt;% rename(pleasure = value)<br>doc_pain_df &lt;- tbl_df(pain_score) %&gt;% rownames_to_column %&gt;% rename(document = rowname) %&gt;% rename(pain = value)<br>doc_virtue_df &lt;- tbl_df(virtue_score) %&gt;% rownames_to_column %&gt;% rename(document = rowname) %&gt;% rename(virtue = value)<br>doc_vice_df &lt;- tbl_df(vice_score) %&gt;% rownames_to_column %&gt;% rename(document = rowname) %&gt;% rename(vice = value)</pre><p>One such dataframe looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*F8vwrLr4EF_Trj6H" /></figure><p>Subsequently, I join all the dataframes together by their document number. This results in a single dataframe containing the scores of each category for each document.</p><pre>library(tidyverse)</pre><pre>all_emotions &lt;- list(doc_emo_df, doc_pleasure_df, doc_pain_df, doc_virtue_df, doc_vice_df) %&gt;% reduce(inner_join, by = “document”)</pre><p>This step results in the dataframe that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FpBcx9jB5uKtdJMu" /></figure><p>The next step is to make a heatmap for visualization. I use heatmap.2 function in which we need to feed a numeric matrix of the values to be plotted. Therefore, that’s the first step I undertake — converted the dataframe computed above in the matrix form. Subsequently, that matrix was fed into the heatmap.2 function, along with several other parameters. I have commented the description and usage of every parameter in the code below. I learnt the usage of heatmap from <a href="https://sebastianraschka.com/Articles/heatmaps_in_r.html">Sebastian Raschka’s blog</a>.</p><pre># transform the columns (except document number) into a matrix<br>mat_data &lt;- data.matrix(all_emotions[,2:ncol(all_emotions)]) <br>rownames(mat_data) &lt;- all_emotions[,1][[1]] # assign row names</pre><pre>heatmap.2(mat_data,<br> cellnote = mat_data, # same data set for cell labels<br> main = “Emotions in Ted Talks”, # heat map title<br> notecol=”black”, # change font color of cell labels to black<br> density.info=”none”, # turns off density plot inside color legend<br> trace=”none”, # turns off trace lines inside the heat map<br> margins =c(12,9), # widens margins around plot<br> col=colorRampPalette(c(“red”, “yellow”, “green”))(n = 299), # use on color palette defined earlier<br> Colv=”NA”, # turn off column clustering<br> dendrogram=’none’)</pre><p>The code results in the heatmap below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xngmww32o-GLYyf_" /></figure><p>As you can see in the heatmap above, red color dominates throughout. A clear cut reasoning for that lies in the fact that the DTM is extremely sparse. With 99% sparsity in DTM, the words in the five categories (emotional, pleasure, pain, virtue and vice) could not match the majority of the terms present in our DTM. Small stretches of yellow lines denote the presence of particular emotions — virtue in this case. This means that speakers in their Ted Talks use a lot of words related to virtue in their speeches. The result could be different if you use different category of emotions.</p><p>That’s all for now. I will discuss a new task related to NLP in my next blog. So stay tuned.</p><p>I have written other <a href="https://medium.com/@shubhanshugupta/engineering-challenges-of-streaming-a-million-concurrent-json-data-streams-from-product-to-crm-360506b29aca">posts</a> related to software engineering as well. You might want to check them out <a href="https://shubhanshugupta.com/blog/">here</a>.</p><p><strong>This post</strong> <strong>was initially published at </strong><a href="https://shubhanshugupta.com/emotions-in-ted-talks/">https://shubhanshugupta.com/emotions-in-ted-talks/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b6016f3316e" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/emotions-in-ted-talks-text-analytics-in-r-1b6016f3316e">Emotions in Ted Talks: Text Analytics in R</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Preprocessing in R]]></title>
            <link>https://medium.com/@shubhanshugupta/data-preprocessing-in-r-2f0e25487bb?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0e25487bb</guid>
            <category><![CDATA[ted]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 04:50:28 GMT</pubDate>
            <atom:updated>2018-11-19T04:50:28.206Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*g1DPkAGcwxdgbF5l" /></figure><p>I have recently got my hands dirty with Natural Language Processing (NLP). I know, it’s a little late to the party but I am at least <strong>in</strong> the party!</p><p>To start with a general overview, I implemented quite a few tasks related to NLP including Text Classification, Document Similarity, Part-of-Speech (POS) Tagging, Emotion Recognition, etc. These tasks were made possible by implementing text pre-processing (noise removal, stemming) and text to features (TF-IDF, N-Grams, Topic Modeling, etc). I implemented these in both R and Python. So, I will try to jot down my experiences in both of these environments. Therefore, I will write this as a blog series, wherein each blog will discuss only one particular thing implemented in one particular environment.</p><blockquote>This post is going to be an account of conducting data pre-processing and visualizing the words that are used most frequently in <a href="https://www.kaggle.com/rounakbanik/ted-talks">TED Talks</a>.</blockquote><h4><strong>About the dataset</strong></h4><p>It contains information about all the audio-video recordings of TED talks uploaded to the official TED.com website until 21st September 2017. There are two major files in the dataset. The first one has information about talks like number of views, comments, speakers, titles etc. The second one on the other hand consists of the transcripts of those talks.</p><p>So, let’s dive in!</p><pre>set.seed(5152)<br>ted_main &lt;- read.csv(‘ted_main.csv’)<br>transcripts &lt;- read.csv(‘transcripts.csv’)</pre><pre>ted_talks &lt;- merge(x = ted_main,y = transcripts, by = ‘url’)</pre><p>It’s important to set seed (but optional) so as to get reproducible results for everyone who runs the code. I am also merging the files by the ‘url’ column present in both the files and because it is uniquely identifying a talk.</p><p>Moving on to pre-processing.</p><pre>library(tm)<br>library(SnowballC)</pre><pre>corpus &lt;- VCorpus(VectorSource(ted_talks$transcript))</pre><pre>##Removing Punctuation<br>corpus &lt;- tm_map(corpus, content_transformer(removePunctuation))</pre><pre>##Removing numbers<br>corpus &lt;- tm_map(corpus, removeNumbers)</pre><pre>##Converting to lowercase<br>corpus &lt;- tm_map(corpus, content_transformer(tolower))</pre><pre>##Removing stop words<br>corpus &lt;- tm_map(corpus, content_transformer(removeWords), stopwords(“english”))</pre><pre>##Stemming<br>corpus &lt;- tm_map(corpus, stemDocument)</pre><pre>##Whitespace<br>corpus &lt;- tm_map(corpus, stripWhitespace)</pre><p>With all the steps above, we complete our data pre-processing. Most of the steps are self-explanatory in the sense that we are removing punctuations, numbers, stop words from our text. We are also converting our text to lowercase and subsequently, stemming it. Stemming is the process of stripping suffixes (“ing”, “ly”, “es”, “s”, etc). The <a href="https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf"><strong>tm</strong> package in R</a> presents methods for data import, corpus handling, data preprocessing, creation of term-document matrices etc. The <a href="https://cran.r-project.org/web/packages/SnowballC/index.html">SnowballC package</a> is used for stemming.</p><p>We will now visualize our pre-processed data.</p><pre># Create Document Term Matrix<br>dtm &lt;- DocumentTermMatrix(corpus)</pre><pre># Removing all terms whose sparsity is greater than 95% <br>corpus &lt;- removeSparseTerms(dtm, 0.95)</pre><p>There are two important things in the code block above. First is the creation of Document Term Matrix (DTM) and second is the removal of sparse terms. The DTM is a matrix that lists all occurrences of words in the corpus, by document. That is, the documents form the rows of the matrix and the terms form the column. Therefore, if a word occurs in a document, the corresponding entry for that row and column will be the number of times that word occurred in the document. To illustrate it better, say I have two documents:</p><p><em>Doc 1</em>: I study in National University of Singapore</p><p>Doc 2: I live in Singapore</p><p>It’s DTM would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5eliNhHF7pEXsGOaUROrg.png" /><figcaption>A Sample DTM</figcaption></figure><p>The next important funda that I want to discuss is <strong>removal of sparse terms</strong>.</p><p>When I check what’s in my DTM, I get this:</p><pre>&lt;&lt;DocumentTermMatrix (documents: 2467, terms: 81626)&gt;&gt;<br>Non-/sparse entries: 1146779/200224563<br>Sparsity : 99%<br>Maximal term length: 51<br>Weighting : term frequency (tf)</pre><p>It says that there are 2467 rows, more than 80000 columns. Of this matrix, there’s 99% sparsity. I found a fantastic explanation of what’s sparse terms and why is there a need to reduce the sparsity through this <a href="https://stackoverflow.com/questions/28763389/how-does-the-removesparseterms-in-r-work">link</a>. To sum it up here, a 99% sparse matrix means that 99% of the values on the matrix are zero and 0% sparsity means, the matrix has no zero value and is densely populated with values. So, in my previous code block, I gave a value of 0.95 (in removeSparseTerms function) in my DTM. The result is self-evident:</p><pre>&lt;&lt;DocumentTermMatrix (documents: 2467, terms: 1783)&gt;&gt;<br>Non-/sparse entries: 797530/3601131<br>Sparsity : 82%<br>Maximal term length: 13<br>Weighting : term frequency (tf)</pre><p>Number of columns got reduced to 1783 and my sparsity went down to 82% from 99%. This means that there are lesser zero values in my matrix.</p><p>I will now proceed to visualize the highest occurring terms in my corpus.</p><pre>library(data.table)<br>library(ggplot2)<br>library(ggthemes)</pre><pre>colS &lt;- colSums(as.matrix(corpus))</pre><pre>doc_features &lt;- data.table(name = attributes(colS)$names, count = colS)</pre><pre>ggplot(doc_features[count&gt;8500],aes(name, count)) + geom_bar(stat = “identity”,fill=’lightblue’,color=’black’)+ theme(axis.text.x = element_text(angle = 45, hjust = 1))+ theme_economist()+ scale_color_economist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XM21Sdioi1oJLMw6" /></figure><p>Clearly, the highest frequency words are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8Xs9qXje4CNT_IhSlY0SA.png" /><figcaption>High frequency words in the corpus</figcaption></figure><p>There are few more pre-processing steps that can be done like removal of extreme rare words, most commonly occurring words. But the steps mentioned above, cover majority of the pre-processing needs. Only thing that I feel could be personalized is the sparsity. But then again, it depends on your use-case. It’s okay If it’s just for understanding what’s sparsity but if it’s for, say Text Classification — which I will cover in my subsequent blogs, then sparsity matters a lot. A dense matrix would carry a lot more information about your corpus then a sparse matrix. Even, my Mac (Macbook Pro i5 2.6 GHz, 8 GB Ram) used to crash while running Text Classification on a highly sparse matrix. Anyhow, I will discuss these issues in much more detail in the upcoming blogs.</p><p>Thanks for reading and stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0e25487bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataKind Singapore]]></title>
            <link>https://medium.com/@shubhanshugupta/datakind-singapore-eedfd9ddb3b6?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/eedfd9ddb3b6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[meetup]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[data-for-good]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Sun, 09 Sep 2018 05:40:58 GMT</pubDate>
            <atom:updated>2018-09-09T05:40:58.529Z</atom:updated>
            <content:encoded><![CDATA[<p>I have recently shifted gears in <a href="https://shubhanshugupta.com/">my life</a>. A shift to <a href="http://msba.nus.edu/">academics</a> after spending quite sometime in the industry has been equally exciting and challenging at the same time. Even with all the diversity in the cohort of my own program (in terms of work experience and country of origin) along with a campus hustling with <a href="http://rc4.nus.edu.sg/rc4life/events-and-activities/">activities</a> all day, I wanted to explore the communities ingrained deep within the culture of Singapore. A natural choice for me, was to look for <a href="http://meetup.com/">meetups</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGN0ZEipvM6xhFet1rcLNQ.png" /></figure><p>After subscribing to quite a few meetups of my interest, I finally got a notification of <a href="https://www.datakind.org/chapters/datakind-sg">DataKind Singapore</a> meetup. This is one of the few meetups I wanted to attend since quite some time. Due to the fact that its chapters are present only in 5 places (Bangalore, Washington, UK, San Francisco, Singapore), I could not attend any of its meetups in the past. But not this time!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*llO6ADVHuPs52uecfUgZuA.png" /></figure><h3>DataKind Singapore Meetup</h3><p>DataKind brings together data scientists to collaborate and solve critical humanitarian issues in the field of education, poverty, health, human rights, and environment through data driven approaches. The meetup took place at Halogen Foundation Singapore and saw participation from about 60 data enthusiasts from various organizations and universities. There are Project Accelerators — NGOs who present their problems and bottlenecks. Then there are DataJam, DataDives and DataCorps events which focus on solving specific problems from accelerators over an evening or a weekend or even an entire month.</p><h3>Session and Project Overview</h3><p>In the meetup that I attended, we had <a href="https://halogen.sg/">Halogen Foundation</a>, <a href="https://www.waterpointdata.org/">Waterpoint Data Exchange</a> (WPDx), <a href="https://www.wrs.com.sg/en/conservation/our-work/conservation-of-singapores-own-raffles-bonded-langur.html">Raffles’ Banded Langur Working Group</a>. Halogen offers various leadership and mentoring programmes to equip youth with mindsets and skills to excel in future. They wanted us to come up with a strategy of analyzing their surveys done across their beneficiaries. The survey analysis would have helped them to establish benchmarks for their programmes. WPDx is a community which establishes coordination among organizations in the Water Access, Sanitation and Hygiene (WASH) community. WPDx is currently the largest public repository of water point data. They incorporate data science to improve water services through evidence based decision making. In the meetup, they had several tasks in hand including data cleaning, exploratory data analysis and modeling for risk of failure for data points. The Raffles’ Banded Langur Working Group is working to protect the extremely endangered species of Raffles’ Banded Langur. They leverage machine learning and openCV to identify individual monkey faces. Since it’s almost entirely impossible for the conservationist to tag the monkeys, they use professional cameras to capture their photos (for tagging purpose). Our task was to identify each monkey and track its age from infant to juvenile to adulthood.</p><h3>My Contribution</h3><p>I was subsequently grouped with the WPDx team. Our work initially was to clean the dataset and subsequently perform EDA (Exploratory Data Analysis) followed by data modeling. The huge dataset consisting of about 500,000 rows and 30 features, had ample data anomalies and inconsistencies. We were first assigned to analyze and clean the data. I analyzed and explored several features in the dataset and found inconsistencies in the data such as spelling mistakes, presence of escape characters, incorrect date-time format etc. After due validation and testing, I wrote the functions in a python script and raised its PR (Pull Request) to be merged in the organization’s code base.</p><p>The comprehensive project documentation, data dictionaries, trello boards, python scripts (including tests) helped in the process. The volunteers, subsequently moved on to data exploration part as well.</p><p>What seemed to be the longest meetup that I attended (&gt;2.5 hours), we were hoping for some more time at the end to push our work to production. The coordinators were very organized and had great documentation to make the process of data volunteering as seamless as possible. Looking forward for the next meetup!</p><p>I <a href="https://medium.com/@shubhanshugupta">write</a> about Software Engineering and Data Science. Please read my other posts <a href="https://shubhanshugupta.com/blog/">here</a>.</p><p><strong>This post</strong> <strong>was initially published at </strong><a href="https://shubhanshugupta.com/datakind-singapore-meetup/">https://shubhanshugupta.com/datakind-singapore-meetup/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eedfd9ddb3b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My learning experience with Google’s Machine Learning Crash Course]]></title>
            <link>https://medium.com/@shubhanshugupta/my-learning-experience-with-googles-machine-learning-crash-course-bd9a033213c3?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd9a033213c3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[google-mlcc]]></category>
            <category><![CDATA[ml-crash-course]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Mon, 25 Jun 2018 07:53:50 GMT</pubDate>
            <atom:updated>2018-06-26T05:58:29.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Ab1aP4kjEWZzvGvW" /><figcaption>Machine Learning Crash Course</figcaption></figure><p>I came to know about Google’s Machine Learning Crash Course (MLCC) from Sundar Pichai’s <a href="https://twitter.com/sundarpichai/status/969003071131938817">tweet</a>.</p><h3>Sundar Pichai on Twitter</h3><p>We&#39;re sharing resources for everyone to learn more about AI, including a &quot;Machine Learning Crash Course&quot; which over 18K Googlers have enrolled in! #GoogleAI https://t.co/vCiw5QmNLe</p><p>I then enquired about it with some close acquaintances working in Google. I was soon pretty convinced of pursuing this course, after their good words about it and my own research on the course content. This post is going to be an account of my learnings from MLCC. I will structure the learnings in such a way that it will look more like a review. I will also include what I really liked about the course and things which I think they can possibly improve, if the creators are planning to update the course content. So, let’s get started!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*Z4fLWYmt8ULjMMvU" /></figure><h3><strong>Structuring and Categorization</strong></h3><p>This is one of the best part of the entire MLCC and I would like to give a big thumbs up to the course creators and designers. The way they have structured the course is really commendable. I didn’t realize this initially, but later on as the course progresses, you would notice the smooth flow of the course contents. Each section of the course is completely modular. The flow of the content makes it easier to understand the sections when completed in order. However, many concepts in the modules are linked to the advanced modules. So if someone is interested, he/she can choose to jump on that topic and come back again at a later stage.</p><p>As I mentioned earlier, that I really liked the way the course has been structured. Starting with loss and gradient descent, then building through classification models and subsequently deep neural nets. The smooth flow of the topics helped me visualize the necessity of using deep neural nets when the datasets are not linear. I felt agonized when I was prodded to come up with a linear model that could solve datasets such as these:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/0*l73kw6isd0wL0PL7" /><figcaption>Non Linear Spiral</figcaption></figure><p>That’s when the course instructors explained the beauty of deep neural nets to solve such type of datasets. That’s why, I am not getting tired of appreciating the course creators who did a great job of making a student crave for the next part of the module.</p><h3><strong>Content</strong></h3><p>The course content is designed in such a way that each module has a combination of video lectures, <a href="http://playground.tensorflow.org/#activation=tanh&amp;batchSize=10&amp;dataset=circle&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0&amp;noise=0&amp;networkShape=4,2&amp;seed=0.75822&amp;showTestData=false&amp;discretize=false&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=false&amp;cosY=false&amp;sinY=false&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false">playground exercises</a>, text lessons and programming exercises. The video lectures are pretty short and, succinct. The text lessons mainly transcribe the video lessons. I personally felt, playground exercises to be one of the best part of the MLCC course content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eCjhyIFHAq1E888s" /><figcaption>Playground exercise</figcaption></figure><p>I have rarely seen such interactive learning tools. These are great plug and play exercises which help in visualizing the concepts in a profound manner. But here’s a part which I didn’t like much, that is, the programming exercises. Just a small disclaimer: I have been a pretty decent Software Development Engineer and have been coding in Python (and Django) since last 3 years. This disclaimer was not even necessary but I just gave it to prove that I am not a newbie to programming and am not scared of programming either. But the programming exercises in the course just didn’t appeal to me. I felt they were disconnected inspite of the fact that the creators tried to preserve the context of various pre-built functions in the programming exercises from different modules. That’s where I felt that the exercises were a little disconnected. I soon realized that I would be much better off learning the concepts through other exercises provided in the course and learn the practical implementation by my own, later on.</p><h3><strong>Learning Outcome</strong></h3><p>The program has collaborated with <a href="https://www.kaggle.com/">Kaggle</a><strong> </strong>for its <a href="https://www.kaggle.com/c/donorschoose-application-screening">DonorsChoose</a> competition. The competition is not exclusive to the people who complete Google’s MLCC. It, however, could be a great way to start the Kaggle journey and also implement the learnings accrued in a real world project.</p><h3><strong>Final verdict</strong></h3><p>It’s undoubtedly a great online course. But what I seriously think is that each person who completes the course, might take-away different learning outcomes. For some, it could be a naive repetition of the same concepts and they might feel wastage of time. So if you are a kind of person who is already into the game and have even a decent amount of experience in data modeling (including neural nets), then this course is definitely not for you. However, if you are just starting and if you have or don’t have programming experience, then you will enjoy this course. It’s pretty short and you have the liberty to decide your own pace.</p><p>Best of luck!</p><p>I have written other <a href="https://medium.com/@shubhanshugupta/engineering-challenges-of-streaming-a-million-concurrent-json-data-streams-from-product-to-crm-360506b29aca">posts</a> related to software engineering as well. You might want to check them out <a href="https://shubhanshugupta.com/blog/">here</a>.</p><p><strong>This post</strong> <strong>was initially published at </strong><a href="https://shubhanshugupta.com/machine-learning-crash-course/">https://shubhanshugupta.com/machine-learning-crash-course/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd9a033213c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering challenges of streaming a million concurrent JSON data streams from product to CRM]]></title>
            <link>https://medium.com/@shubhanshugupta/engineering-challenges-of-streaming-a-million-concurrent-json-data-streams-from-product-to-crm-360506b29aca?source=rss-269f5616997f------2</link>
            <guid isPermaLink="false">https://medium.com/p/360506b29aca</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-dev-engineer]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Shubhanshu Gupta]]></dc:creator>
            <pubDate>Tue, 19 Jun 2018 18:24:31 GMT</pubDate>
            <atom:updated>2018-06-19T18:24:31.842Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="http://truebil.com/">Truebil</a>, I was fortunate enough to be given an opportunity to solve a unique engineering problem. We had already outsourced CRM development to a third party but I had to integrate the data flow to CRM from our product and back. I mentioned earlier that I was given a unique engineering problem because of the challenges it posed. The challenge didn’t lie in the CRM integration alone, but the fact that Truebil has umpteen number of in-house products which spawn data close to about 1 Million data streams per hour. Besides building a bastion of such magnitude, I knew that it would be equally challenging to work with three different verticals and stakeholders. This post is going to be an account of my experiences dealing with two things. First, how Truebil catered to the transfer of a million data stream to and from CRM without hampering its operations and customer support. The second experience deals in the art of working with different verticals. Not that I have mastered the art or something, but I will share my own experiences and learnings in this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*4yflpvqpnrOvbVYBNGgARQ.gif" /></figure><h3><strong>The solution</strong></h3><p>We had two problems in hand. First was to send all the data from Truebil to CRM. Second was to receive the data that gets generated in CRM.</p><p>We’ll first address the workflow wherein we started off with taming all our data producers. Whereby producers I mean, all our in-house products including (but not limited to) our website, internal data entry website, data-collection app, crones, PWA, customer agent data collection spreadsheets, etc. With a million data streams being generated every hour, we had several tasks in hand and I will list a few of them in the order below:</p><ul><li>Enqueue each data stream — in order.</li><li>Process the data streams and convert each stream as per a data mart, generated jointly in collaboration with our CRM and, operations teams.</li><li>Hit the CRM server in the same order in which the data streams were generated.</li></ul><p>Let’s discuss the first task. It was of paramount importance for us to maintain the order of the data streams. For instance, a customer does an action on our website for which a login action is a prerequisite. So we can’t let the former go first and then the latter. Similar case exists with all our producers.</p><p>We use <a href="http://gearman.org/">gearman</a> for our queue computation. In order to tackle the ordering issue, we maintained only one queue for our entire computation, lest we run the risk of losing the ordering of the data streams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GtMQYnk2DXuBvdHb" /><figcaption>Flow of data from producers to consumer</figcaption></figure><p>The second task in hand was to process the data streams in a format suitable for CRM consumption. Below are some screenshots to give you a glimpse of how the pre-processed and processed data looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*odsTKOm_IanGgbWk" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cb0JCKxDpBBDCO-W" /></figure><p>It required a good amount of computation for each incoming data stream. For every foreign key out there in the pre-processed data, we had to get only select amount of content from the related table to make it ready for consumption.</p><p>In the third task, after processing the data we had to hit the CRM servers with the processed data stream. We were also supposed to wait for the server to respond with a 200 status and hold other jobs in the queue until then. In cases wherein, where we did not receive 200 status, we used to hold on to that job instead of letting it go and moving on to the next one. It was figured out that the CRM rejects a job and returns a different code only when there is a server issue. While testing the integration, we also figured out that the CRM servers almost always accept the job after a maybe a couple of tries. This could possibly be because of the load that we impose on the CRM servers. Hence, it was decided that the CRM server will be hit with the same job (for which it didn’t return 200 OK status) a maximum of three times in a span of 6 seconds. With each hit, we expected their server to accept our job else we would take down our job server. Simply put, we would cut off the outlet of the queue. This was done so that we do not lose any data stream from our side without CRM’s acceptance. When the worker gets shut down, we devised a solution wherein we would restart our gearman worker after a delay of 10 seconds. This process of re-staring the gearman worker does not involve anyone’s manual intervention. We, instead created a watchdog — a gearman monitor, to keep a watch on the workers and restart if necessary. We also created gearman bots to alert us on slack for every unusual activity that is happening with any of our data processing or with our gearman workers.</p><p>The second workflow consists of receiving data from CRM to our end. We maintain another queue at our end for every data stream that we receive from CRM. Each request goes through several stringent data validation checks. Only after a request has passed the data validation checks, we allow it to enter our database.</p><h3><strong>Our learning</strong></h3><p>I will break down the learnings into two scopes. First will be about, how Truebil handled building a new product for our operations and customer support team without hampering their daily operations and without loss of any lead. Second, revolves around my personal learnings from this project.</p><p>Here at Truebil, this project required a sophisticated setup of a smooth flow of requirements and a clear vision of deliverables, in between all the stakeholders. Lest, we ran the risk of ruining the timelines and a complete messed up CRM. A rogue CRM carries a wide range of repercussions from low leads to low sales. Low sales, goes without saying, leads to an overall risk of everyone’s position in the company. This anecdote was just to portray the gravity of the project. Hence, we had a very capable team of product managers, operations managers and software development engineers. All the stakeholders from the management team did a fine job of setting up an exceptional example of planning and execution. So did the engineering team, in solving complex engineering problems to timely delivery of the actionables.</p><p>As far as my learning goes, as a SDE (Software Development Engineer), it was tremendous. Besides solving the engineering challenges that came along, I learnt many more things during the tenure of this project. It was of utmost importance for me to take care of timely deliverables and, ensure no gap in or lack of communication with any of the stakeholders during the entire development cycle. Moreover, my interaction with the different stakeholders including managers at Truebil and a remote development team, had impacted me profoundly. It gave me a deeper understanding of how a product is planned, executed and managed at upper and mid-management levels. I started developing an acumen to pursue product development by a holistic approach, which considers every stakeholder as an end-user of a product. This approach allowed me to imbibe practices which incorporates a well-rounded and balanced perspective in any feature development.</p><h3><strong>Future scope</strong></h3><p>There are two major challenges to cater in near future. First one would be to make our gearman servers compatible for massive scaling. The increasing footprint of Truebil across the country, would mean an increase in operational activity and user generated data. This means that our job servers would soon need to handle an estimated 10–20 million data streams an hour. At such scale, there’s a possibility that we might suffer a slowdown in the execution of our jobs. With lack of or no free space available in memcache, we won’t be able to enqueue or dequeue any job(s). Second scope, which in part extends from the first scope, is to lower the load on just one queue. We would want to implement sharding and also maintain order among the jobs.</p><p><strong>This post</strong> <strong>was initially published at </strong><a href="http://shubhanshugupta.com/engineering-challenges-for-streaming-a-million-concurrent-json-data/">http://shubhanshugupta.com/engineering-challenges-for-streaming-a-million-concurrent-json-data/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=360506b29aca" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>