<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Shankar on Medium]]></title>
        <description><![CDATA[Stories by Ravi Shankar on Medium]]></description>
        <link>https://medium.com/@ravi07bec?source=rss-5de3088925ae------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MGR1oM0mexcQIRncxBM5Jg.jpeg</url>
            <title>Stories by Ravi Shankar on Medium</title>
            <link>https://medium.com/@ravi07bec?source=rss-5de3088925ae------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:58:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ravi07bec/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ML Algorithms from scratch in Python]]></title>
            <link>https://pub.towardsai.net/ml-algorithms-from-scratch-in-python-5caac512eabc?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/5caac512eabc</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sat, 12 Mar 2022 13:02:10 GMT</pubDate>
            <atom:updated>2022-03-12T13:02:10.806Z</atom:updated>
            <content:encoded><![CDATA[<h4>Self notes for behind the scenes mathematical understanding</h4><p><strong>Goal:</strong> To build Machine Learning algorithms in python without using any over-the-shelf ML library like SKLearn or Tensorflow. Intend to cover Linear Regression, Logistic Regression, KNN, K-Means Clustering, Decision Trees, Random Forest, SVM, XGBoost, Perceptron, Neural Net with Backpropagation, DNN, RNN, LSTM, TF-IDF, Bag Of Words, LDA, and Word2Vec.</p><h3>Linear Regression</h3><p>Gradient Descent is the process of minimizing a function by following the gradients of the cost function. This involves knowing the form of the cost as well as the derivative so that from a given point you know the gradient and can move in that direction, e.g. downhill towards the minimum value.</p><p>We are following Gradient descent to find B0 and B1 which would minimize the loss function(Mean Squared Error) in this case. <br>Initialisations: <br>B0=B1=0 , Learning rate(lr)=0.01, #Epochs=500</p><blockquote>Pseudocode: <br>1. Input Xs and Ys are fed to the system with init values<br>2. Prediction of Y is made with B0*X+B1 <br>3. Error Values are returned for coeffient correction <br>4. B0=B0-lr*Error and B1=B1-lr*X*Error<br>5. After a point lets say 300 epochs, error function would start converging i.e no change and that would be a point of early stopping.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be939205b359f94e5f4ea0f98e5ab21a/href">https://medium.com/media/be939205b359f94e5f4ea0f98e5ab21a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*qrPHxn1Q0sVAuBoV5BCcCA.png" /><figcaption>Linear Regression Results</figcaption></figure><h3>Logistic Regression(Similar as Linear Regression)</h3><p><a href="https://machinelearningmastery.com/implement-logistic-regression-stochastic-gradient-descent-scratch-python/">How To Implement Logistic Regression From Scratch in Python - Machine Learning Mastery</a></p><h3>KNN</h3><p><a href="https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/">Develop k-Nearest Neighbors in Python From Scratch - Machine Learning Mastery</a></p><p>For the classification problem, we find the closest k neighbors by using pair-wise Euclidean distance and taking the majority voting. <br>Pseudocode: <br><strong>Step 1</strong>: Calculate Euclidean Distance.<br><strong>Step 2</strong>: Get Nearest Neighbors.<br><strong>Step 3</strong>: Make Predictions.<br><strong>Step 4:</strong> Evaluate prediction accuracy by cross-validation and optimize metric by changing k or distance metric</p><p>For the classification problem, we find the closest k neighbors by using pair-wise Eucledian distance and take the majority voting. <br>Pseudocode: <br><strong>Step 1</strong>: Calculate Euclidean Distance.<br><strong>Step 2</strong>: Get Nearest Neighbors.<br><strong>Step 3</strong>: Make Predictions.<br><strong>Step 4:</strong> Evaluate prediction accuracy by cross-validation and optimize metric by changing k or distance metric</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d21fcfda5842f0705b6c6ebfa1553b19/href">https://medium.com/media/d21fcfda5842f0705b6c6ebfa1553b19/href</a></iframe><h3>K-Means:</h3><h4>Optimum K is calculated by elbow Method</h4><p>In the Elbow method, we are actually varying the number of clusters ( K ) from 1–10. For each value of K, we are calculating <strong>WCSS ( Within-Cluster Sum of Square ). WCSS is the sum of squared distance between each point and the centroid in a cluster. </strong>When we plot the WCSS with the K value, the plot looks like an Elbow. As the number of clusters increases, the WCSS value will start to decrease. WCSS value is largest when K = 1. When we analyze the graph we can see that the graph will rapidly change at a point and thus creating an elbow shape. From this point, the graph starts to move almost parallel to the X-axis. The K value corresponding to this point is the optimal K value or an optimal number of clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/0*Hnru5j6DY633jDbc.png" /></figure><p>We can pick the number of clusters by varying it in a range of possible values and compute the silhouette score and the distortion at every iteration. The optimal number is the one that maximizes the first and minimizes the second.<br>The two methods to measure the cluster quality are described below:</p><p><strong>Inertia:</strong> Intuitively, inertia tells how far away the points within a cluster are. Therefore, a small of inertia is aimed for. The range of inertia’s value starts from zero and goes up.</p><p><strong>Silhouette score:</strong> The silhouette score tells how far away the datapoints in one cluster are, from the datapoints in another cluster. The range of silhouette scores is from -1 to 1. The score should be closer to 1 than -1.<br>The <a href="https://en.wikipedia.org/wiki/Elbow_method_(clustering)">elbow method</a> looks at the percentage of variance explained as a function of the number of clusters: One should choose a number of clusters so that adding another cluster doesn’t give much better modeling of the data. More precisely, if one plots the percentage of variance explained by the clusters against the number of clusters, the first clusters will add much information (explain a lot of variances), but at some point the marginal gain will drop, giving an angle in the graph. The number of clusters is chosen at this point, hence the “elbow criterion”.</p><p>Kmeans separates the documents(NLP) into disjoint clusters. the assumption is that each cluster is attributed to a single topic. However, descriptions may in reality be characterized by a “mixture” of topics. For example, let’s take an article that deals with the hearing that Zuckerberg had in front of the congress: you’ll obviously have different topics rising based on the keywords: Privacy, Technology, Facebook app, data, etc. <strong>LDA helps deal with this problem.</strong></p><p>Input is <strong>n</strong> data points with <strong>k</strong> clusters, small tolerance limit such as 0.01 after for early stopping and n_iterations as 500. The output is a set of ‘<em>k</em>’ cluster centroids and labeling of the dataset that maps each of the data points to a unique cluster.</p><p><strong>Step 1.</strong> Randomly pick k data points as our initial Centroids.<br><strong>Step 2.</strong> Find the distance (Euclidean distance for our purpose) between each datapoints in our training set with the k centroids.<br><strong>Step 3.</strong> Now assign each data point to the closest centroid according to the distance found.<br><strong>Step 4.</strong> Update centroid location by taking the average of the points in each cluster group.<br><strong>Step 5.</strong> Repeat Steps 2 to 4 till our centroids don’t change.</p><pre>if np.sum((curr - original_centroid)/original_centroid * 100.0) &gt; self.tolerance:<br>	isOptimal = False<br>#break out of the main loop if the results are optimal, ie. the centroids don&#39;t change their positions much(more than our tolerance)<br>if isOptimal:<br>		break</pre><p><a href="https://en.wikipedia.org/wiki/File:K-means_convergence.gif">K-means convergence.gif</a></p><h3>Perceptron</h3><p>Perceptron receives input signals from examples of training data that we weigh and combined in a linear equation called the activation. It is closely related to linear regression and logistic regression that make predictions in a similar way (e.g. a weighted sum of inputs). The weights of the Perceptron algorithm must be estimated from your training data using stochastic gradient descent.</p><p>The weights and biases are initialized as 0 and learned based on error from predicted vs output value by <em>activation = sum(weight_i * x_i) + bias.<br></em>The weight matrix is equal to input matrix or num_neurons</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5e954b377c2fc06723bc46873885ff9/href">https://medium.com/media/b5e954b377c2fc06723bc46873885ff9/href</a></iframe><h3>Neural Network with Backpropagation</h3><p><a href="https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/">How to Code a Neural Network with Backpropagation In Python (from scratch) - Machine Learning Mastery</a></p><p>Given the input training dataset and random weights and biases, can you build a DNN with hidden layers and backpropagation?</p><blockquote>Pseudocode<br>1. Initialise the network with #layers(Input+Output+Hidden) and matrix of weights and bias for every neuron <br>2. Build activation function which would calculate neuron activation for an input by (Wi*Xi)+Bias. <br>3. Pass the activated neuron via activation function like sigmoid, Relu etc<br>4. Forward propagate the predictions and calculate overall error<br>5. Back-propagate errors and blame individual errors across neurons and adjust weights by derivative function and wi=wj+lr*ei<br>6. Continue training the network for minimising error till n epochs with specified learning rate(if not using momentum)</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d6e5562b94e585b7fe819c2d0dcd4b9/href">https://medium.com/media/8d6e5562b94e585b7fe819c2d0dcd4b9/href</a></iframe><h4>Error Backpropagation</h4><p>The first step is to calculate the error for each output neuron, this will give us our error signal (input) to propagate backward through the network. The error for a given neuron can be calculated as follows</p><blockquote>error = (output — expected) * transfer_derivative(output)</blockquote><p>Where <strong>expected</strong> is the expected output value for the neuron, the <strong>output</strong> is the output value for the neuron and <strong>transfer_derivative()</strong> calculates the slope of the neuron’s output value<br><strong>Vanishing gradient </strong>means that weights in layers close to the input layer are not updated in response to errors calculated on the training dataset. A problem with training networks with many layers (e.g. deep neural networks) is that the gradient diminishes dramatically as it is propagated backward through the network. The error may be so small by the time it reaches layers close to the input of the model that it may have very little effect. As such, this problem is referred to as the “<em>vanishing gradients</em>” problem. Vanishing gradients is a particular problem with recurrent neural networks as the update of the network involves unrolling the network for each input time step, in effect creating a very deep network that requires weight updates.<br>Solutions: <em>pre-training, better random initial scaling, better optimization methods, specific architectures, orthogonal initialization, etc.</em></p><p><strong>TF-IDF (Term Frequency-Inverse Document Frequency)</strong></p><p><strong>Term Frequency:</strong> It is the ratio of the number of times the word appears in a document compared to the total number of words in that document.<br>tf(t,d) = count of t in d / number of words in d<br>For corpus- “Ravi is here. Rohan is there”,<br>tf(Ravi)=1/6<br>tf(is)=2/6</p><p>Inverse Document Frequency: Log of the ratio of the number of documents to the number of documents containing the word<br>corpus=[“Ravi is here”,”Rohan is there”,”Good to read ML”,”Bad to read ML”]<br><em>idf(t) = log(N/(df + 1))<br>idf(Ravi)=log(4/2)<br>idf(ML)=log(4/3)</em></p><p>If the model encounters an unknown word other than the vocab, it will give us a Key error as we did not account for any unknown tokens. To use word2vec embedding for such use-cases.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e259eb5020684bac7db9fd79a610783a/href">https://medium.com/media/e259eb5020684bac7db9fd79a610783a/href</a></iframe><p>Other techniques such as count vectorizer, Bag Of Words(Work to Live and Live to Work are same) and Hashing vectorizers were precursors to TF-IDF and can be used for additional features.<br>Disadvantages of TF-IDF:<br>- TF-IDF is based on the bag-of-words (BoW) model, therefore it does not capture the position in the text, semantics, co-occurrences in different documents, etc.<br>- Cannot capture semantics (e.g. as compared to topic models, word embeddings)</p><p>Word embedding represents the vector representation of words in a constrained space e.g 100 Dimensional space as opposed to Millions of possible worlds (dimensions) in the dictionary. Skip-Gram has output as weights between the input layer and hidden layer while for CBOW it is hidden layer and output layer. <br>In Supervised learning, the weights can be retrained as follows:</p><blockquote>1.Initialise a One-Hot encoded matrix with random scores<br>2.Multiply it with a one-dimensional vector representing the word e.g “Nice” has [1,0,0,0] representation<br>3.Combine individual samples for a uniform weight matrix, e.g 12 weights with 4 dimension each. Use padding for small sized reviews.<br>4.Pass the matrix through a neuron, calculate the error against the labelled data(Fake problem) and backpropagate for updating the input weights. <br>5.Perform the iteration for n epochs with x LR for improving binary-cross entropy<br>6. Store the vector weights of individual keywords as embedding weights.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prgFUzf0TzhLTHHBHUrfRw.png" /><figcaption>Embedding calculation in supervised learning</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08c3d2091da7830d5def88510ea00b7c/href">https://medium.com/media/08c3d2091da7830d5def88510ea00b7c/href</a></iframe><p><strong>Recurrent Neural Network(RNN)-</strong> For sequence problems such as next word prediction, time series, speech recognition, etc where output is dependent on the previous sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*aBI8RYDnclweMXbm.png" /><figcaption>RNN</figcaption></figure><p>Steps for RNN from scratch: <br>1. Data preparation with the sequence as input e.g trailing 50 days of stock prices<br>2. Architecture creation for RNN: Inputs are LR, #Epochs, hidden layer, input layer, and output layer similar to DNN with additional parameters as BPPT(Backpropagation through time), Min_Clip and Max_Clip for vanishing/exploding gradient. The core difference in BPTT versus backprop is that the backpropagation step is done for all the time steps in the RNN layer. So if our sequence length is 50, we will backpropagate for all the timesteps previous to the current timestep.<br>In addition to the input and output weight matrix, we would have an additional matrix for shared weights in the RNN layer.<br>3. Train the model: Forward Pass, Calculate loss, Start training, Backpropagate error and update weights<br>4. Generate Predictions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-xInZQ7_BIyuz3dMSpvVg.png" /><figcaption>RNN advantages and Drawbacks</figcaption></figure><p><strong>Gradient clipping</strong> is a technique used to cope with the exploding gradient problem sometimes encountered when performing backpropagation. By capping the maximum value for the gradient, this phenomenon is controlled in practice.<br>Gradient Update rule, New Weight=weight-LR*Gradient,<br>If Gradient is small, no message is reached to earlier layers. <br>If Gradient is large, the Network wouldn&#39;t converge and show NaN errors.</p><p>LSTM: They have Gates to remove unwanted noise. <br>It has a cell state and Gates: Forget gate, Input Gate and Output Gate.<br>Gates contain sigmoid activation i.e update or forget[1,0] unlike Tanh[-1,1]. <br><strong>Forget Gate:</strong> Information from the previous hidden state and the current state is passed through the sigmoid function. Closer to 0 means forget and closer to 1 means keep. <br><strong>Input Gate:</strong> Previous hidden and current input is passed through the Tanh function. Tanh output and Sigmoid output are multiplied for the next state. <br><strong>Output Gate:</strong> Decides what the next hidden state should be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wg38XZlhji3G_4ZiEkpJyA.png" /><figcaption>Pseudocode for LSTM</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6FH8LBecs1jQptz50h7Dyw.png" /><figcaption>How LSTM sees a review</figcaption></figure><p><strong>Sources</strong></p><ul><li><a href="https://machinelearningmastery.com/implement-linear-regression-stochastic-gradient-descent-scratch-python/">How to Implement Linear Regression From Scratch in Python - Machine Learning Mastery</a></li><li><a href="https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/">How to Code a Neural Network with Backpropagation In Python (from scratch) - Machine Learning Mastery</a></li><li><a href="https://www.askpython.com/python/examples/tf-idf-model-from-scratch">Creating a TF-IDF Model from Scratch in Python - AskPython</a></li><li><a href="https://www.analyticsvidhya.com/blog/2019/01/fundamentals-deep-learning-recurrent-neural-networks-scratch-python/">RNN From Scratch | Building RNN Model In Python</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5caac512eabc" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/ml-algorithms-from-scratch-in-python-5caac512eabc">ML Algorithms from scratch in Python</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Book Summary- Data Science from ISLR, OpenIntro Statistics and AB testing]]></title>
            <link>https://medium.com/analytics-vidhya/book-summary-data-science-19a30ecfe13d?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/19a30ecfe13d</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sun, 28 Nov 2021 20:37:57 GMT</pubDate>
            <atom:updated>2023-11-08T18:21:09.272Z</atom:updated>
            <content:encoded><![CDATA[<p>There are people who grasp more from watching tutorials from Courseera/YouTube and then there are others who learn by reading textbooks. I belong to second category and have made notes from 3 books, this post was private for longest of times and opening it to community now.</p><p>Book: Trustworthy Online Controlled Experiments, Link <a href="https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264">here</a></p><p>Designing the experiment, Questions:<br><strong>1. What is the randomisation unit? </strong>Users(Signed in, Cookies, Device ID), Pages, Session- Consideration- Are they independent across granularity e.g Green buy button if randomised across pages or sessions is bad. Should follow SUTVA- Stable unit treatment value assumption. Randomisation unit and analysis unit should generally be same<br><strong>2. What proportion of randomisation unit to target-</strong> High sample is better but concerns around server load, Inter-dependence, Cache Utilisation, Page load time, Guardrail metrics should be considered.<br><strong>3. How large does our experiment need to be?</strong> Depends on MDE, Power, Statistical significance and most importantly Variance. Variance can be minimised by bucketing(Does user watch &gt;XX hours of content) or #Searchers vs #Searches or User Buying vs Actual revenues.<br><strong>4. How long should we run the experiment?</strong> Are we taking into account variability on Day Of Week, Hour of Day, #Expected users, Novelty and Primacy effects and Seasonality.</p><p><strong>Interpreting the results:</strong> <br>1. Overall Evaluation Criterion(OEC) and secondary metrics<br>2. Guardrail metrics such as email/Notification unsubscribe, Sample Ratio Mismatch(SRM), Latency, Redirects etc.<br>3. Statistical results, A/A test output i.e metrics haven’t changed with control and practical significance levels.</p><p><strong>Sample Ratio Mismatch(SRM) can happen due to: </strong><br>1. Browser Redirects- Performance differences(Load time), Bots, Rediects etc<br>2. Lossy instrumentation: Leakage in click capture<br>3. Bad hash function for randimisation<br>4. Triggering impacted by treatment: E.g if segment were dormant users which changes over course of experiment<br>5. Data pipeline issues such as bot filtering failed</p><p>The samples should have a ratio of 1.0 by design and any value not in [0.99–1.01] is questionable. p-value of sample ratio is calculated and if below 0.05, the experiment is void. Can be corrected by removing outliers e.g A bing search experiment by removing users with old version of chrome browser.<br>Debugging SRMs can be done by <strong>a.</strong>validating no difference upstream of randomisation point(e.g a checkout treatment feature cant be mentioned on Homepage, <strong>b.</strong> Ensure Bot filtering is working , and <strong>c. </strong>Validate that variant assignment is correct</p><p><strong>Potential issues with experiment results: </strong><br>1. Statistical issues: Lack of Power, Misinterpreting p-values, Early stopping and peeking at p-values, Multiple treatments<br>2. Multiple Hypothesis tests: Segment wise results, multiple metrics, Multiple iteration of experiments and result peeking<br>3. Threats to internal Validity: SUTVA violation due to Network effects, Survivorship bias, and Sample Ratio Mismatch<br>4. Threats to external Validity: Generalisation across segments(Country/Language), Primacy effects, Novelty effects and Simpson’s paradox i.e across multi-phase experiments, Treatment may be better than Control individually but worse when combined.</p><p><strong>Variant assignment for scaling experimentation:</strong><br>1. Single Layer Method: Divide traffic into groups(40% C, 30%-T1, 30% T2). This should ensure that roughly same number of users in every bucker and key metrics(Goal, Guardrail, Quality) have roughly same values<br>2. Concurrent Experiments: To ensure orthogonality of experiments across layers. Groups may interact with other experiments such as “Blue Font(E1) on Blue Background(E2)”. To follow nested design i.e system paparmeters are partioned into layers and these those poor combination partition should not be fired on same user.</p><p><strong>Metrics: </strong><br>1. Goal metrics: Overall vision of the company e.g connecting users for FB, Organise world’s information for Google etc. Goal metrics should be simple(explainable to stakeholders) and stable(not changing with feature)<br>2. Driver Metrics: Shorter term, Fast moving and sensitive to our experiment design. Can bucketise by HEART i.e Happiness, Engagement, Adoption, Retention and Task oriented. To ensure that driver metrics overall drives towards goal metrics. <strong>Driver metrics should be aligned with goal, actionable and relevant(teams must feel they can act on levers), sensitive enough to measure impact of initiative and resistant to Gaming.</strong><br>3. Guardrail Metrics: Latency, revenue hit at the cost of #clicks increase, Client crashes, SRM, Telemetry fidelty, cache hit rate, cookie write rate etc(Page 224)</p><p><strong>Converting from driver metrics to experiment appropriate metrics:</strong><br>1. Measurable- Bad example would be post-purchase satisfaction<br>2. Attributable- App crash to the treatment<br>3. Sensitive- Insensitive metric can be stock price change with feature. Extremely sensitive metric can be visible page change. <br>4. Timely-<br>Multiple metrics can be combined to an OEC based on standardisation(0–1) and business judgement</p><p>Complementary techniques to online experimentation can be log based analysis for blast radius(e.g Windows phone vs Android vs iOS or &gt;80 age group vs &lt;40Age group), Survey, Focus group and external data. When controlled experiments are not possible, we can use counterFactural(What would have happened if they were not treated) and observational causal studies.</p><p>Few designs for observational causal studies: <br>1. Interrupted time series: Example is surveillance helicopters and its impact on Burgalaries. Seasonality can be a challenge. <br>2. Interleaved experiment: Showing 50% search results from Model 1 and 50% from M2 with duplicates removed. <br>3. Regression discontinuity design: Look closer to the threshold e.g 5% discount rate in Amazon strike price or 80% marks.<br>4. Difference in Difference: To estimate the increase over delta change, e.g Control group in Colorways also increased due to Holidays.</p><p>Ramping experiment exposure should be planned as 100% rolling out may lead to crashes. Ramping down is needed for bad treatment and cleaning it up to prevent bad CX.</p><p><strong>Experiment ramp Phases after statistical significant results without affecting any guardrail metrics: </strong><br>1. Pre Maximum Power Ramp(50%)-Create rings of testing population and gradually expose the treatment to successive rings to mitigate risk. Example rings can whitelisted individuals(Within team) to company employees to Beta users/Early adopters to data Centers.<br>2. Post MPR: To check for concerns about increasing traffic load prior to 100%. <br>3. Long term Holdout set- To keep ~20% users in Hold out set and validate their behavior for novvelty or primacy effects or to check for Goal metric i.e user retention</p><h3>Analyzing experiments(Data Science):</h3><p>For Means(2 sample t-test) and for proportions(CTR, Conversion) it should be z-test. Confidence interval shouldn’t contain 0. Even if the metric is not normal, Normality assumption holds due to Central Limit theorem i.e Means should be normal(Can be further tested by Kolmogorov-Smirnov test). The skewness in data can be reduced by transformation i.e buckets for netflix, Log Transaforms, Binary purchase decisions etc. Power is lower for small effect size as it can be difficult to capture.</p><p><strong>Bias</strong> can arise in Experiment results when estimate and true value of mean is systematically differene e.g Platform bug, Flawed experiment design, Unrepresentative sample such as company exployee or test account. <br>Multiple testing problem(Control against 10 colors): If we see 100 treatments as an output, there is a chance that 5% of those treatments would be statistically significant(False Discovery rate). Bonoferroni correction can be applied i.e p-value/#treatments. This is conservative and we may use tiered approach like 0.05,0.01 and 0.001 p-values for First order metrics(Directly impacted by experiment), 2nd order metrics(Potentially impacted e.g cannibalisation) or 3rd order metrics(Unlikely to be affected).</p><p><strong>Variance</strong> reduction is essential to get high power, low sample size and greater T-Statistic. Possible reasons for high variance can be:<br>1. Ratio Metrics: Variance formula, Var(Y)=Sum(Yi-YHat)²/n-1 has assumption that the samples are independently identically distributes(IID) or at least uncorrelated. This is satisfied if the analysis unit is same as randomisation unit. For ratio metrics such as CTR on a page and randomised on user, there is a within user correlation leading to increase in correlation. Can be corrected by Bootstrapping or randomising by page. <br>2. Outliers: Bots having extremely high clicks affect both mean and variance however greater impact on variance due to squared factor which would lead to T-Statistic reduction. <br><strong>Best practices to improve sensitivity or reduce variance: </strong><br>1. Create evaluation metric with smaller variance while capturing similar information, e.g Number of searchers vs searches or Purchase(Boolean) vs Purchase amount.<br>2. Transform metric through capping, binarisation or log transform(e.g netflix users &gt;XX content)<br>3. Use stratification- Divide sampling region to strata(Web vs Mobile, Chrome vs Firefox) which has lower variance and combine it together. <br>4. Randomise at more granular size e.g for a page experiment such as Green buy botton at page level.</p><p><strong>A/A Test:<br></strong>Split the users into 2 groups as in a regular A/B test and make B identical to A. If the system is operating correctly:<br>1. About 5% of the time a given metric should be statistically significant with p-value&lt;0.05<br>2. The distribution of p-values in t-test should be close to a uniform distribution.</p><p>Why to do A/A Test(***):<br>1. Ensure that Type-I errors are controlled<br>2. Ensure that no biases exist between treatment and control groups<br>3. Compare data to the system of record e.g Engagement, CTR etc<br>4. Capture user leakage and perform upstream analysis as needed<br>5. Estimate variance for statistical power calculation<br>6. Check for platform issues such as Browser redirects in Bots, Unequal percentages in cache utilisation etc.</p><p>Counterfactual triggering for ML models(Page 212)- To read more</p><p><strong>Leakage and interference between Variants(Assumption is SUTVA):<br></strong>1. Direct Connections: If C and T share same social network or physical space at the same time. If treatment is using more video chat feature or messaging to friends, Control would also be affected, Results are underestimated.<br>2. Indirect connections: If they have common latent variable or shared resources such as Ad campaign budget or LRU caching. AirBNB or uber- More usage by treatment leaves less resources for control. Results are Overestimated.</p><p><strong>Best practices for isolation or minimizing network effects: <br></strong>1.Splitting shared resources e.g predefinedad budget <br>2. Geo Based randomisation: NYC vs SF for uber or Doordash<br>3. Time based randomisation<br>4. Network cluster randomisation: Separate cluster of nodes in FB/Linkedin which are closer together <br>5. Network ego centric randomisation: Ego and Alter ego i.e variant assignment for egos and alters separately</p><h3>Notes from OpenIntro Statistics,Link <a href="https://www.amazon.com/OpenIntro-Statistics-Fourth-David-Diez/dp/1943450072">here</a></h3><p>Distribution of Random Variables:</p><p><strong>Normal distribution:</strong> Mean of 0 and sd=1 is standard normal distribution. For SN, Mean=Median=1<br>Z=(x-μ/σ<strong>)<br></strong>1σ:2σ:3σ for 2 tailed is 68:95:99.7% area.</p><p><strong>Geometric Distribution:</strong> Used to describe how many trials it takes to achieve a success for independent and identically distributed(IID) Bernoulli RV. Follows <strong>Bernoulli RV</strong> i.e any individual trial has 2 outcomes with probability p and 1-p.</p><blockquote>For Bernoulli RV, μ=p and σ²=p(1-p)<br>For Geometric Distribution, μ=1/p and σ²=(1-p)/p2 . Prob of success in nth trial=p*(1-p)^n-1</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*6rkcy0wVV2TDh2sL.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cZWqDRE7kv5aKTJG.jpg" /></figure><p><strong>Binomial Distribution:</strong> The binomial distribution is used when there are exactly two mutually exclusive outcomes of a trial. These outcomes are appropriately labeled “success” and “failure”. The binomial distribution is used to obtain the probability of observing <em>x</em> successes in <em>N</em> trials, with the probability of success on a single trial denoted by <em>p</em>. The binomial distribution assumes that <em>p</em> is fixed for all trials.<br>Conditions for Binomial Distribution:<br>1. Fixed number of <em>n</em> trials.<br>2. Each trial is <a href="https://www.statisticshowto.com/probability-and-statistics/dependent-events-independent/#or">independent</a>.<br>3. Only two outcomes are possible (Success and Failure).<br>4. Probability of success (p) for each trial is constant.<br>5. A random variable Y= the number of successes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*X21Zd97kGzgmns4-.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*d0gbmHRCMSWyoWnI.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*XDvPJvJ_l405Y3YC.gif" /><figcaption>Probability Density function for Binomial</figcaption></figure><p>As we can see in PDF above, binomial distribution approximates to normal distribution when sample size is large or to verify np and n(1-p) &gt;10</p><p><strong>Negative Binomial Distribution:<br></strong>The <strong>negative binomial</strong> is similar to the binomial with two differences (specifically to numbers 1 and 5 in the list above):</p><ul><li>The number of trials, n <strong>is not fixed</strong>.</li><li>A <a href="https://www.statisticshowto.com/random-variable/">random variable</a> Y= the number of trials needed to make r successes.</li></ul><p>Example: Take a standard deck of cards, shuffle them, and choose a card. Replace the card and repeat until you have drawn two aces. Y is the number of draws needed to draw two aces. As the number of trials isn’t fixed (i.e. you stop when you draw the second ace), this makes it a negative binomial distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/0*wk_8pRxuCNtA9Joc.gif" /><figcaption>Properties of Neg Binomial</figcaption></figure><p><strong>Poisson Distribution: </strong>Used for estimating the number of events in a large population over a unit of time. For events such as having a heart attack, stuck by lightening etc, PD helps describe number of such events that will occur in a day for a fixed population if the individuals within the population are independent.<br>The parameter in poisson distribution is rate(λ) or how many events we expect to observe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0Q2a4s_9opq7b93P.jpg" /><figcaption>Probability of X events in a period</figcaption></figure><p><strong>The average number of major storms in your city is 2 per year. What is the probability that exactly 3 storms will hit your city next year?</strong></p><p>Step 1: Figure out the components you need to put into the equation.</p><ul><li>μ = 2 (average number of storms per year, historically)</li><li>x = 3 (the number of storms we think might hit next year)</li><li>e = 2.71828 (e is <a href="https://calculushowto.com/eulers-number/">Euler’s number</a>, a constant)</li></ul><p>Step 2: Plug the values from Step 1 into the Poisson distribution formula:</p><ul><li>P(x; μ) = (e-μ) (μx) / x!</li><li>= (2.71828–2) (23) / 3!</li><li>= (0.13534) (8) / 6</li><li>= 0.180</li></ul><p>The probability of 3 storms happening next year is 0.180, or 18%</p><p>Poisson vs Binomial: <br>If your question has an <strong>average probability</strong> of an event happening per unit (i.e. per unit of time, cycle, event) <strong>and </strong>you want to find probability of a certain number of events happening in a period of time (or number of events), then use the Poisson Distribution.</p><ul><li>If you are given an <strong>exact probability</strong> and you want to find the probability of the event happening a certain number of times out of x (i.e. 10 times out of 100, or 99 times out of 1000), use the <a href="https://www.statisticshowto.com/probability-and-statistics/binomial-theorem/binomial-distribution-formula/">Binomial Distribution formula</a>.</li></ul><p>Chi-Square Distribution: To characterise datasets and statistics that are always positive and typically right skewed. The Chi-square distribution has just one parameter called Degrees of Freedom which influences the shape, center and shape of the distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*Cg5BstKbVnWWv_zr.jpg" /><figcaption>PDF with Degrees of Freedom</figcaption></figure><h3>Inference and Hypothesis testing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70jPhwlmLpriymrxUNGSzg.png" /></figure><p>Central Limit Theorem: When observations are <strong>independent</strong> and sufficiently large(np≥10 and n(1-p)≥10), the sample proportion <strong>p̂ </strong>will tend to follow a normal distribution with following mean and standard error:</p><blockquote>μ=p, Variance =p*(1-p)/n</blockquote><p>95% Confidence Interval for a parameter: The confidence interval (CI) is a range of values that’s likely to include a population value with a certain degree of confidence. It is often expressed as a % whereby a population mean lies between an upper and lower interval. When the distribution of a point estimate qualifies for the CLT and closely follows Normal distribution, <br>95% CI=point estimate+-1.96*SE<br>Margin Of Error: z*SE<br>E.g- We are 95% confident that 88%-91% of American adult(Overall Population) support a change.</p><p><strong>Type I error: </strong>Rejecting the Null hypothesis when it is actually true, False Positives, e.g a Man is pregnant. <br><strong>Type II error:</strong> Failing to reject Null hypothesis when alternative is actually true. Beta. 1-TypeII=Power.</p><p><strong>p-Value: </strong>A p value is used in <a href="https://www.statisticshowto.com/probability-and-statistics/hypothesis-testing/">hypothesis testing</a> to help you <a href="https://www.statisticshowto.com/probability-and-statistics/hypothesis-testing/support-or-reject-null-hypothesis/">support or reject the null hypothesis</a>. The p value is the evidence <strong>against</strong> a <a href="https://www.statisticshowto.com/probability-and-statistics/null-hypothesis/">null hypothesis</a>. The smaller the p-value, the stronger the evidence that you should reject the null hypothesis.</p><p>One-Sided Hypothesis tests: Represented by single tail area. Has a challenge that any findings in the other direction is diregarded, .eg Amazon is cheaper than Walmart, One sided test would disregard the other possibility.<br>For estimating a sample size when p is unknown, we can assumpe p=0.5 for worst case as p(1-p)/n is highest.</p><p>Types of Inference problems: <br><strong>1. Single Proportion(proportion):</strong> Point estimate +-z*SE. E.g Does majority population approve this? μ=p and σ²=p(1-p)/n<br><strong>2. Difference of 2 proportions(proportion): </strong>E.g Difference in Click Through rates for 2 groups in experiment. Conditions:<br>a. Data are independent within and between the groups<br>b. The success failure condition holds for both groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTlqXD0VU0iLoSZykzYfEQ.png" /></figure><p>Even if the sample sizes are not equal, above SE can be applied for hypothesis testing. <br>Pooled Proportion: Hypothesis that p1-p2=0<br>3<strong>. Testing for Goodness of fit using Chi-Square- Categorical Data</strong>(Example of Jury from population across Races)- Determine if a sample(Multiple categorical variable) is representative of the population or if the data is derived from a particular distribution. Can be used in A/A test as a precursor to experiment and verify for any systematic error. Chi-square test for categorical variables determines whether there is a difference in the population proportions between two or more groups. In the medical literature, the Chi-square is used most commonly to compare the incidence (or proportion) of a characteristic in one group to the incidence (or proportion) of a characteristic in other group(s).For example, you might use the Chi-Square test to compare the incidence PONV between patients that received ondansetron, patients that received droperidol, and patients that received a placebo.<br>Pearson’s chi-squared test is used to determine whether there is a <a href="https://en.wikipedia.org/wiki/Statistical_significance">statistically significant</a> difference between the expected <a href="https://en.wikipedia.org/wiki/Frequency_(statistics)">frequencies</a> and the observed frequencies in one or more categories of a <a href="https://en.wikipedia.org/wiki/Contingency_table">contingency table</a>.</p><p>Now, we look into hypothesis testing with numerical data such as mean lift of a metric in Treatment group. For normality, n&gt;30 and samples must be independent</p><p><strong>4. One Sample Mean with T-Distribution: </strong>In absence of Population SD, we use <strong>s/sqrt(n)</strong> as sample S.E. The t-distribution is centered at 0 and has degree of freedom as single parameter, df=n-1. When df&gt;30, t-distribution becomes same as normal distribution. E.g Race times difference in 2006 and 2017 marathon with n=100, Mean=XX and SD for 2017=3Mins</p><p><strong>5. Paired Data: </strong>Two sets of observations are paired if each observation in one set has a special correspondence with exactly one observation in another set. UCLA vs Amazon price example for books or Conversion for an ASIN at different prices. n is same for both sample, Mean and SD is calculated from diff values. SE(diff)= s(diff)/sqrt(n)</p><p><strong>6. Difference of 2 Means, T Statistic</strong>(Most important for causal inference in experiment e.g Engagement difference between Control and treatment group): Similar to 2 proportion test, Conditions:<br>a. Data are independent within and between the groups<br>b. The success failure condition holds for both groups i.e Data is normal across groups.</p><blockquote>S.E=sqrt(Var1/n1+Var2/n2)</blockquote><p>Pooled SD estimate- When SD between 2 groups are similar</p><p><strong>7. Comparing Many means(ANOVA)-F Test </strong>Multiple testing scenario(10 colors in experiment group) and better alternative to pairwise comparison. Null hupothesis is mean across all groups are equal. Conditions: <br>1. Observations are independent within and across groups<br>2. Data within each group are nearly normal<br>3. Variability across groups is nearly equal.</p><p>F-Test is used to answer if Variability in sample means is so large that it seems unlikely from chance alone. ANOVA uses test statistic, F which represents a standardised ratio of variability in sample means relative to variability between groups. <br>F=(Mean Square Between Groups-MSG/Mean Square Error-MSE). <br>Higher F-Statistic is good for lower p-values with df.</p><p>For ANOVA calculation: <a href="https://www.pythonfordatascience.org/anova-python/">https://www.pythonfordatascience.org/anova-python/</a></p><p>If Null hypothesis of equal means is rejected by ANOVA, we use pairwise comparison using 2 sample T-Test. The scenario of testing many pairs of groups is called multiple comparison and <strong>Bonferroni correction</strong> suggests a significance level of <strong>α/k </strong>where k are number of comparisons , k=n*n-1/2</p><p>Probability Notes from CourseEra and Textbooks: <br>Plan: Tues-Dec 28, CourseEra Probability <a href="https://www.coursera.org/learn/probability-theory-statistics/home/week/1">Course</a><br>Wednesday- CourseEra Quizzes and OpenIntro Exercises <br>Thursday- OpenIntro+ Intro Probability questions in Python</p><p>A and B are independent events if :</p><ul><li>P(A/B)=P(A)</li><li>P(B/A)= P(B)</li><li>P(A Intersection B)= P(A)*P(B)</li></ul><p><a href="https://medium.com/@mlengineer/joint-probability-vs-conditional-probability-fa2d47d95c4a">https://medium.com/@mlengineer/joint-probability-vs-conditional-probability-fa2d47d95c4a</a> For difference between Independent and Joint event. <br>Joint probability is the likelihood of <strong>more than one event</strong> occurring at the <strong>same time</strong> P(A and B). The probability of event A and event B occurring together. It is the probability of the intersection of two or more events written as <strong>p(A ∩ B)</strong>.<br>When events are independent, <strong>p(A ∩ B)=p(A)*p(B)</strong></p><p>When events are dependent, <strong>p(A)=p(A|B)*p(B)<br>Suppose you have an event with only two possible outcomes where the probability of success is <em>P </em>and consequently the probability of failure is <em>1 — P. </em>What is the expected number of times you have to perform the event in order to get the first success?<br></strong>You perform the event and get a failure with probability <em>1 — P</em> thereby increasing the expected number of times you’ll have to perform the event to get first success to <em>1 + E</em>.Therefore, we can form the following equation:<br>𝐸=𝑃+(1−𝑃)⋅(𝐸+1)<br>or, E=1/P</p><p>If the sampling is done with replacement, binomial distribution is not a valid choice since probability can change from Trial to Trial. the <strong>hypergeometric distribution</strong> is a <a href="https://en.wikipedia.org/wiki/Probability_distribution#Discrete_probability_distribution">discrete probability distribution</a> that describes the probability of k successes (random draws for which the object drawn has a specified feature) in n draws, <em>without</em> replacement. <br>It is relatively simple, e.g from an urn of 6 red and 3 white balls, probability of 2 white balls and 1 red ball without replacement. <br>3C2*6C1/9C3</p><p>Power Calculations: <a href="https://www.moresteam.com/whitepapers/download/power-stat-test.pdf">https://www.moresteam.com/whitepapers/download/power-stat-test.pdf</a></p><h3>ISLR Notes</h3><p>Parametric Methods: Reduces the problem of estimating f down to one of estimating a set of parameters, e.g linearity assumption of a function.A more restrictive and smoother curve has less degrees of freedom and thus lower variance.</p><p>Non Parametric methods: They do not make explicit assumptions about the functional form of f and need larger number to data points to estimate f.</p><p>U-Shape of test MSE is characterised by Bias²+Variance. <strong>Bias</strong> refers to error introduced by approximating a real-life problem, which may be extremely complicated by a much simpler model. A K-NN classifier with low k(e.g 1) is highly wiggly i.e high variance and low bias.</p><p><strong>Linear Regression: </strong>Null hypothesis states that there is no relationship between X and Y i.e Beta=0. T-Statistic is computed by Beta-0/SE(Beta).Accuracy of the model can be estimated by Residual Standard error(RSE) which implies the average amount that the response will deviate from true regression line, R² statistic or proportion of variance explained.</p><p>Questions prior to building multiple regression model:<br>1. Is at least one of the predictors X1,X2,…XN useful in predicting the response(F-Statistic by ANOVA test). We use ANOVA as hypothesis of B1=B2=B3=0 would have at least 5% of variables satisfying the hypothesis in individual t-test<br>2. Do all the predictors help to explain Y or is only a subset of predictors useful. Forward selection, Backward selection, L1 regularisation, PCA etc can be used for feature subsetting.<br>3. How well the model fit the data. R², Adjusted R², AIC, BIC can be used<br>4. Given a set of predictor values. what response value should we predict and how accurate is our prediction. Prediction interval is wider than confidence interval since they incorporate both the errors in estimate of f(x) and irreducible error.<br>The <strong>hierarchical principle </strong>states that if we include an interaction in a model, we should also include the main effects, even if the p-values associated with their coefficients are not significant.</p><p>Potential Problems(Assumptions) of Linear Regression Model: <br>1. Non-Linearity of the response predictor response<br>2. Correlation of error terms<br>3. Non-Constant variance of error terms(Heterodescacity)<br>4. Outliers<br>5. High Leverage Points<br>6. Collinearity- To compute variance Inflation Factor(VIF) for Multi-collinearity.Variance inflation factors range from 1 upwards. VIF is ratio of variance Bj when fitting the full model divided by Bj when fit on its own.The numerical value for VIF tells you (in decimal form) what <a href="https://www.statisticshowto.com/statistics-basics/calculate-percentages/">percentage</a> the variance (i.e. the <a href="https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/what-is-the-standard-error-of-a-sample/">standard error</a> squared) is inflated for each coefficient. For example, a VIF of 1.9 tells you that the variance of a particular coefficient is 90% bigger than what you would expect if there was no multicollinearity — if there was no correlation with other predictors.<br>VIF=1/(1-R²)<br>A <strong>rule of thumb</strong> for interpreting the variance inflation factor:1 = not correlated,Between 1 and 5 = moderately correlated.Greater than 5 = highly correlated.<br>Code sample for VIF and Multicollinearity <a href="https://www.geeksforgeeks.org/multicollinearity-in-data/">here</a></p><p><strong>Logistic Regression: </strong>Rather than modelling the response Y directly, LR models the probability that Y belongs to a particular category<br><strong>log(P(X)/1-P(X))=B0+B1X.</strong> This is called log-odds or logit.<br>We use Maximum likelihood with LR and seek estimates for B0 and B1 such that the predicted probability-p(x) corresponds as closely as possible to individual’s observed status. The estimates B0 and B1 are chosen to maximise this likelihood function.</p><p><strong>Resampling Methods:</strong> They involve repeatedly drawing samples from a training set and refitting a model of interest on each sample in order to obtain additional observation about the fitted model.<br>Types of Cross Validation: <br>1. Leave One Out CV(LOOCV)- A Single observation is used for validation set and model is trained on remaining data. CV(n)=Sum(MSE)/n. Its advantages over validation set approach are less Bias and consistency of results. However, they are expensive to implement since multiple model training is needed and also has high variance.<br>2. K-Fold CV- Randomly dividing the set of observations into k-folds, train on k-1 and validate on kth. Gives more accurate estimates of test error rate than L00CV based on Bias Variance tradeoff. Unlike LOOCV, in K-Fold, we are averaging the output of k fitted models that are less correlated with each other since the overlap between the training sets between each model is smaller. this leads to reduction in variance.</p><p>Bootstrapping: Instead of repeatedly obtaining independent datasets from the population, we obtain distinct datasets by repeatedlt sampling observations from original dataset.</p><p><strong>Linear Model selection and regularisation: <br></strong>For reducing number of features/curse of dimensionality-<br>1. Subset Selection: Identify set of p predictors(p&lt;n). Though it is simple, it has computational limitations and number of models increases exponentialy i. e 2^p models for p features. Can be done by Forward stepwise selection, Backward stepwise selection and hybrid approaches with evaluation metric as AIC, BIC or R². <br>2. Shrinkage: Shrinking the coefficient estimates significantly reduce variance. Lambda is shrinking parameter here<br>a. Ridge Regression(L2): Lambda*Sum(Beta²), called shrinkage penalty is small when Betas as closer to 0 and has the effect of shrinking the estimates towards 0. No change in intercept. It improves over least squares as Lambda increases, the flexibility of ridge regression fit decreases leading to decreased variance and increased Bias which provides a minima in Bias-Variance tradeoff until best Lambda. Ridge regression works well in high variance situations by trading off a small increase in Bias by large decrease in Variance. Unlike subset selection, Ridge regression will include all p predictors in the model.<br>b. Lasso Regression(L1):Lambda*Sum(Beta) is the Lasso penaltyand shrinks the coefficient estimates towards 0. The l1 penalty has the effect of forcing some of the coefficient estimates to be exactly equal to zero when tuning parameter(Lambda) is sufficiently large. Hence, it also performs variable selection.</p><p>Soft thresholding: In ridge regression, each coefficient estimate is shrunken by the same proportion while the lasso shrinks each least square coeffient towards 0 by a constant amount Lambda/2. The least square coefficients that are less than Lambda/2 are shrunken entirely to 0 which is called soft thresholding.</p><blockquote><strong>L1 Regularization<br><em>L1 penalizes sum of absolute value of weights.</em></strong><em><br></em><strong><em>L1 has a sparse solution</em></strong><em><br>L1 has multiple solutions<br></em><strong><em>L1 has built in feature selection</em></strong><em><br></em><strong><em>L1 is robust to outliers</em></strong><em><br></em><strong><em>L1 generates model that are simple and interpretable but cannot learn complex patterns</em></strong></blockquote><blockquote><strong>L2 Regularization</strong><br><strong><em>L2 regularization penalizes sum of square weights.</em></strong><em><br></em><strong><em>L2 has a non sparse solution</em></strong><em><br>L2 has one solution<br>L2 has no feature selection<br>L2 is not robust to outliers<br></em><strong><em>L2 gives better prediction when output variable is a function of all input features<br>L2 regularization is able to learn complex data patterns</em></strong></blockquote><p>3. Dimension Reduction: Linear techniques such as PCA, LDA,etc.</p><p>PCA: It is a technique for reducing the dimensionality of big datasets which improves model interpretability while minimizing information loss. The new principal components are created which are uncorrelated with each other with the aim to maximize variance i.e maximum information of large feature data into lower dimensional principal components. Based on the incremental variance explained by principal components e.g 90% of Variance, we can decide on the cutoff.The explained variance tells you how much information (variance) can be attributed to each of the principal components. This is important as while you can convert 4 dimensional space to 2 dimensional space, you lose some of the variance (information) when you do this. <br>Principal component loadings are calculated as combination of orthogonal vectors whose squared sum is equal to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVVkTE2dVXtUG-iGFrem9w.png" /><figcaption>Scree Plot for Proportion Of Variance Explained(Elbow Method)</figcaption></figure><p>T-Stochastic Neighbor embedding- It is a non-linear technique for dimensionality reduction and generally used for visualization of high dimensional datasets. Its implementation is based on minimizing the divergence between two distributions: a distribution that measures pairwise similarities of the input objects and a distribution that measures pairwise similarities of the corresponding low-dimensional points in the embedding.</p><p>Principal Component Regression(PCR, Unsupervised based just on data)- Involves constructing the first M principal components, Z1, Z2,… Zm and then using these components as predictors in linear regression model that is fit using least squares. As more principal components are used in regression model, the bias decreases and variance increases and showing equivalent results when M=P. It works well in cases when first few principal components are suffiencient to capture most of the variation in the predictors as well as relationship with response.<br><strong>Partial Least Squares</strong> are supervised alternative to PCR and identifies new principal components in supervised manner i.e makes use of response Y.</p><p><strong>Tree Based Models:<br></strong>1. We divide the predictor space i.e set of possible values for X1, X2, Xp into J distinct and non-overlapping regions. We build this by recursive binary splitting i.e top down greedy approach. The process continues until a stopping criteria is reached, e.g no region contains &gt;5 observations. <br>2. For every observation that falls into region, Rj, we make the same prediction i.e mean for regression and median for classification problem.<br>Tree Pruning: To build a large tree and prune it back for subtree with lowest test error rate. Cost complexity pruning performs this i.e rather than considering every possible subtree, we consider a sequence of trees indexed by a non-negative parameter Alpha which may be chosen by k-fold cross validation.<br><strong>Gini Index:</strong> Gini index says, if we select two items from a population at random then they must be of same class and probability for this is 1 if population is pure.</p><p>1.It works with categorical target variable “Success” or “Failure”.<br>2. It performs only Binary splits<br>3.Higher the value of Gini higher the homogeneity.<br>4. CART (Classification and Regression Tree) uses Gini method to create binary splits.<br>4 methods to decide on split are Gini information gain, Chi-Square test,Entropy(Information gain) and Variance.</p><p>Decision trees have high variance and we use aggregation methods such as bagging, random forest and boosting to improve predictive performance.<br><strong>Bagging: </strong>We build a separate prediction model using each training set and average the training set thus reducing the variance by Sigma²/n. For a given test observation, we record the class predicted for each of B trees and then take majority vote. Works well for feature importance calculation as we record the total amount of RSS decreased due to splits over a given predictor, averaged over B trees. All the bagged trees may look similar to each other if the predictor is strong e.g FAANG for income estimation.<br><strong>Random Forest: </strong>Provides improvement over bagging by decorrelating trees. It forces each split to consider only subset of predictors typically sqrt(n)<br><strong>Boosting: </strong>Weak learners are combined sequentially to build strong learner, does not involve bootstrap sampling and instead each tree is fit on modified version of original dataset. Tuning parameters for boosting: <br>1. Number of trees(B)- CV is used for selecting B. <br>2. Shrinkage parameter(Lambda)- Controls the rate at which boosting learns. <br>3. Number of splits in each tree, d=1 works well i.e tree is a <strong>stump</strong> considering only a single split with low bias.<br>XGBoost is comparatively better than other techniques:</p><ol><li><strong>Regularization:</strong>Standard GBM implementation has no regularisation like XGBoost.Thus XGBoost also helps to reduce overfitting.</li><li><strong>Parallel Processing:</strong>XGBoost implements parallel processing and is faster than GBM .XGBoost also supports implementation on Hadoop.</li><li><strong>High Flexibility:</strong>XGBoost allows users to define custom optimization objectives and evaluation criteria adding a whole new dimension to the model.</li><li><strong>Handling Missing Values:</strong>XGBoost has an in-built routine to handle missing values.</li><li><strong>Tree Pruning:</strong>XGBoost makes splits up to the max_depth specified and then starts pruning the tree backwards and removes splits beyond which there is no positive gain.</li><li><strong>Built-in Cross-Validation:</strong>XGBoost allows a user to run a cross-validation at each iteration of the boosting process and thus it is easy to get the exact optimum number of boosting iterations in a single run</li></ol><p><strong>SVM: TO ADD</strong></p><p>Hierarchical Clustering: One potential drawback of k-means is requirement to pre-specify k(By elbow plot).</p><p>If you enjoyed going through the summary, give it a clap so that medium algorithm picks it up and benefits greater number of learners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19a30ecfe13d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/book-summary-data-science-19a30ecfe13d">Book Summary- Data Science from ISLR, OpenIntro Statistics and AB testing</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Books summary- ML design patterns for building ML powered systems]]></title>
            <link>https://ravi07bec.medium.com/books-summary-ml-design-patterns-d5dc0c5adf42?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5dc0c5adf42</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sun, 28 Nov 2021 08:36:29 GMT</pubDate>
            <atom:updated>2022-02-04T05:40:31.717Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.amazon.com/Machine-Learning-Design-Patterns-Preparation/dp/1098115783">https://www.amazon.com/Machine-Learning-Design-Patterns-Preparation/dp/1098115783</a></p><p>Common Challenges in Building ML Systems:<br>1. Data Quality<br>2. Data Completeness<br>3. Data Consistency</p><p>Methods for scaling data:<br>Linear scaling- Min-Max scaling, Clipping to remove outliers and then scale to [-1,1], Z-Score normalisation and Winsorizing which uses the empirical distribution in the training dataset to clip the data tp [P10-P90] range.<br>Non Linear Scaling- Log Based, Bucketisation</p><p><strong>Design Pattern 1: Hashed Feature- </strong>To address problems associated with categorical features such as incomplete vocabulary(Different OHE in train and test set), Model size due to cardinality(100s of categories) and cold start(Feature not available now but can occur in future)</p><p>Solution: To build hashed categorical feature using n buckets, e.g for 347 airports, an average of 35 airports will get same hash bucket code if we hash into 10 buckets.This would keep the system’s memory and size requirements practical even if there are millions of categorical variables.</p><p>Tradeoff- Lose model accuracy as features are averaged across the hash with bucket collision. Possible solution can be adding an aggregate feature to individual categories such as avg delay from airport.</p><p><strong>Design Pattern 2: Embeddings- </strong>Addresses the problem of representing high cardinality data(Text/Images) densely in a lower dimension by passing input data through an embedding layer that has trainable weights. Keras has helper function pad_sequence to ensure common dimensionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4AWJcHqgSrPclqNYthEE7A.png" /></figure><p>Tradeoffs: Loss of information in going from high cardinality to low dimensional representation. Example is autoencoder network consisting of bottleneck layer which is just an embedding layer where encoder(before the bottleneck layer) compresses and decoder expands it. Similar to PCA achieving linear dimension reduction, bottleneck layer of AE achieves non-linear dimension reduction.</p><p><strong>Design Pattern 3: Feature Cross- </strong>Helps model learn relationships between inputs faster by combining features. It is a synthetic feature formed by concatenating 2 or more features in order to capture the interaction between them, e.g For a Taxi Ride, Day and Time combination would differentiate Monday 5:00 pm from Friday 5:00 pm.Has high cardinality and generally not used with raw numerical features. Cardinality can be corrected by L1 regularisation.</p><p>Design Pattern 4: Multimodal Input- Combining numerical, image and text based feature, e.g Price Gouging capture using item name and image.</p><p><strong>Design Pattern 5: Multilabel-</strong>To assign more than one label , e.g Ranked Genre. For multi-class Softmax is good enough which predicts an image as [0.9,0.08,0.02] as [Cat, Dog, Rabbit] and argmax can be done for final prediction.<br>In this case, sigmoid activation function is used in output layer which each indidual value in sigmoid array between [0,1].The main difference in output between the sigmoid model here and the softmax example in the Problem section is that the softmax array is guaranteed to contain three values that sum to 1, whereas the sigmoid output will contain three values, each between 0 and 1.<br><strong>Hierarchical labels</strong> can also be generated e.g Action -&gt;SuperHero-&gt; Batman which be handled by :<br>1. Use a flat approach and put every label in the same output array regardless of hierarchy, making sure you have enough examples of each “leaf node” label.<br>2. Build one model to identify higher-level labels. Based on the high-level classification, send the example to a separate model for a more specific classification task.</p><p>Another technique for handling Multilabel classification is to train multiple binary classifiers instead of one multilabel model. This approach is called one versus rest. Benefit is simplistic binary model however added complexity of training many different classifiers.</p><p><strong>Design Pattern 6: Ensembles- </strong>Demonstrated in Bagging, Boosting and Stacking.Model averaging methods like <strong>bagging</strong> work because typically the individual models that make up the ensemble model will not all make the same errors on the test set. In an ideal situation, each individual model is off by a random amount, so when their results are averaged, the random errors cancel out, and the prediction is closer to the correct answer. In short, there is wisdom in the crowd.<strong>Boosting</strong> works well because the model is punished more and more according to the residuals at each iteration step. With each iteration, the ensemble model is encouraged to get better and better at predicting those hard-to-predict examples. <strong>Stacking</strong> works because it combines the best of both bagging and boosting. The secondary model can be thought of as a more sophisticated version of model averaging.</p><p><strong>Design Pattern 7: Rebalancing-</strong> For imbalanced data PR is more informative than AUC as it places more emphasis on how many predictions the model got right out of total number it assigned to positive call. It gives more weightage to positive class unlike AUC which gives equal importance to both. <br>Ways to treat imbalanced data for ML Models: <br>1. Downsampling: Reduce the majority class<br>2. Upsampling: Increase the minority class by synthetic Minority Oversampling (SMOTE)<br>3. Weighted Classes: Gives more importance to minority positive class during training. classs_weights parameter in keras and scale_pos_weight in xgboost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IOjxrLqUraZVZc04Eq9_aw.png" /><figcaption>Typical Stochastic Gradient Descent(Batch)</figcaption></figure><p><strong>Design Pattern 8: Checkpoints(Ensures resilience)-</strong>Here, we store the full state of the model periodically so that we have partially trained models available. These partially trained models can serve as the final model (in the case of early stopping) or as the starting points for continued training (in the cases of machine failure and fine-tuning).<br>Problem: The more complex the model is, the larger time it takes.f the training loop is interrupted for any reason, we can go back to the saved model state and restart. However, when doing this, we have to make sure <strong>to save the intermediate model state, </strong>not just the model due to below reasons:<br>1. An exported model does not contain which epoch and batch number the model is currently processing. <br>2. In order to carry out gradient descent effectively, the optimizer might be changing the learning rate on a schedule. This learning rate state is not present in an exported model. <br>3. Additionally, there might be stochastic behavior in the model, such as dropout. This is not captured in the exported model state either. <br>4. Models like recurrent neural networks incorporate history of previous input values. In general, the full model state can be many times the size of the exported model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9_KHDjYRBKncQAkAp1ibNA.png" /></figure><p>For Model playground and seeing how output varies with hidden layers, activation functions, Learning rate, Regularisation etc. <br><a href="https://bit.ly/3D0l4gf">https://bit.ly/3D0l4gf</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aB_3BUtKIEvKNebxXWgIUQ.png" /><figcaption>Model Training stages- Need for Checkpoints</figcaption></figure><p>Training Phases: <br>1. Between stages (a) and ©, the model is learning high-level organization of the data.<br>2. Between stages and © and (e), the model is learning the details.<br>3. Stage (f), the model is overfitting</p><p>Tradeoffs and advantages of Checkpoints: <br>1. Early Stopping- Model may stop training when validation error begins to rise. However it is not uncommon for validation set to rise and then drop again which can be captured as post-processing stage if model checkpoints are in place.<br>To choose between early stopping and checkpoint selection depending on cost(ES is preferred) or model accuracy(Checkpoint selection is preferred)</p><p>FineTuning: In a well-behaved training loop, gradient descent behaves such that you get to the neighborhood of the optimal error quickly on the basis of the majority of your data, then slowly converge toward the lowest error by optimizing on the corner cases.Now, imagine that you need to periodically retrain the model on fresh data. You typically want to emphasize the fresh data, not the corner cases from last month. You are often better off resuming your training, not from the last checkpoint, but the checkpoint marked by the blue line</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ok_P677MpIORqf-OV-tNVw.png" /></figure><p><strong>Design Pattern 9: Transfer Learning-</strong> We take layers of a previosly trained model, freeze the weights and incorporate these non-trainable layers into a new model that solves a similar problem but on a smaller dataset. We do transfer learning as large datasets(Mns of Images and Reviews) are not available for inferencing on small dataset. <br>Typically the penultimate layer of the bigger model is chosen as bottleneck layer which acts as hidden layers for our small scale model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_c_p_QB7evdIiZuVq9IPOQ.png" /></figure><p>The bottleneck layer is conceptually similar to embedding with main difference that weights in an embedding layer can be trained whereas all the layers leading upto and including the bottleneck layer have their weights frozen. <br>In a typical convolutional neural network (CNN), the learning is hierarchical. The first layers learn to recognize edges and shapes present in an image. In the cat example, this might mean that the model can identify areas in an image where the edge of the cat’s body meets the background. The next layers in the model begin to understand groups of edges — perhaps that there are two edges that meet toward the top-left corner of the image. A CNN’s final layers can then piece together these groups of edges, developing an understanding of different features in the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ePE2nJZsERn3UIM1fL13TA.png" /></figure><p><strong>Design Pattern 10: Distribution strategy-</strong> Here, the training loop is carried out at scale over multiple workers, often with caching, hardware acceleration and parallelization.<br>Problem: Millions of model parameters with massive amounts of data which may take weeks to train.<br>Solution: Split across machines by using either <strong>Data Parallelism</strong> or <strong>Model Parallelism</strong>.In data parallelism, computation is split across different machines and different workers train on different subsets of the training data. In model parallelism, the model is split and different workers carry out the computation for different parts of the model.</p><p>Data Paralellism:There is a method for different workers to compute gradients and share that information to make updates to the model parameters. This ensures that all workers are consistent and each gradient step works to train the model.<br><strong>Synchronous Training</strong>:In synchronous training, the workers train on different slices of input data in parallel and the gradient values are aggregated at the end of each training step. This is performed via an <em>all-reduce</em> algorithm. This means that each worker, typically a GPU, has a copy of the model on device and, for a single stochastic gradient descent (SGD) step, a mini-batch of data is split among each of the separate workers. Each device performs a forward pass with their portion of the mini-batch and computes gradients for each parameter of the model. These locally computed gradients are then collected from each device and aggregated (for example, averaged) to produce a single gradient update for each parameter. A central server holds the most current copy of the model parameters and performs the gradient step according to the gradients received from the multiple workers. Once the model parameters are updated according to this aggregated gradient step, the new model is sent back to the workers along with another split of the next mini-batch, and the process repeats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0bY4Z6JMGH5MjRMjp6Us5Q.png" /></figure><p><strong>Asynchronous Training:</strong>In asynchronous training, the workers train on different slices of the input data independently, and the model weights and parameters are updated asynchronously, typically through a <a href="https://oreil.ly/Wkk5B">parameter server architecture</a>. This means that no one worker waits for updates to the model from any of the other workers. In the parameter-server architecture, there is a single parameter server that manages the current values of the model weights</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Ay6iojtRoCsuiCeB3yYZw.png" /></figure><p>The key difference between synchronous and asynchronous training is that the parameter server does not do an <em>all</em>-reduce. Instead, it computes the new model parameters periodically based on whichever gradient updates it received since the last computation. Typically, asynchronous distribution <strong>achieves higher throughput</strong> than synchronous training because a slow worker doesn’t block the progression of training steps.<strong> If a single worker fails, the training continues as planned with the other workers while that worker reboots.</strong> As a result, some splits of the mini-batch may be lost during training, making it too difficult to accurately keep track of how many epochs of data have been processed.<br>Synchronous training is particularly vulnerable to slow devices or poor network connection because training will stall waiting for updates from all workers. This means synchronous distribution is preferable when all devices are on a single host and there are fast devices (for example, TPUs or GPUs) with strong links. On the other hand, asynchronous distribution is preferable if there are many low-power or unreliable workers. If a single worker fails or stalls in returning a gradient update, it won’t stall the training loop. The only limitation is I/O constraints.</p><p><strong>Model Paralellism: </strong>By partitioning parts of a network and their associated computations across multiple cores, the computation and memory workload is distributed across multiple devices. Each device operates over the same mini-batch of data during training, but carries out computations related only to their separate components of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n-ht52sJQtcO6bSfU88zRQ.png" /></figure><p><strong>Design Pattern 11: Hyperparameter Tuning-</strong> Can be performed by Grid search, Random Search, AutoML’s HyperOpt, Bayesian Optimisation etc.<br>Instead of training our model each time we try a new combination of hyperparameters, Bayesian optimization defines a new function that emulates our model but is much cheaper to run. This is referred to as the <em>surrogate function</em> — the inputs to this function are your hyperparameter values and the output is your optimization metric. The surrogate function is called much more frequently than the objective function, with the goal of finding an optimal combination of hyperparameters <em>before</em> completing a training run on your model. With this approach, more compute time is spent choosing the hyperparameters for each trial as compared with grid search.<br>We can also use <strong>Genetic algorithm</strong> which has a fitness function which measures the quality of a particular trial, and can typically be defined by your model’s optimization metric (accuracy, error, and so on). After defining your fitness function, you randomly select a few combinations of hyperparameters from your search space and run a trial for each of those combinations. You then take the hyperparameters from the trials that performed best, and use those values to define your new search space. This search space becomes your new “population,” and you use it to generate new combinations of values to use in your next set of trials. You continue this process, narrowing down the number of trials you run until you’ve arrived at a result that satisfies your requirements.</p><h3>Design Patterns for Resilent Serving</h3><p><strong>Design Pattern 12: Stateless Serving Function(Used for Online Prediction)- </strong>Allows a production ML system to <strong>synchronously</strong> handle thousands to millions of prediction requests per second(Fraud detection, BOT filtering). The production ML system is designed around a stateless function that captures the architecture and weights of a trained model.<br>A stateless function is a function whose outputs are determined purely by its inputs. This function, for example, is stateless:</p><pre>def stateless_fn(x):<br>    return 3*x + 15</pre><p>Another way to think of a stateless function is as an immutable object, where the weights and biases are stored as constants:</p><pre>class Stateless:<br>    def __init__(self):<br>        self.weight = 3<br>        self.bias = 15<br>    def __call__(self, x):<br>        return self.weight*x + self.bias</pre><p>A function that maintains a counter of the number of times it has been invoked and returns a different value depending on whether the counter is odd or even is an example of a function that is stateful, not stateless.<br><strong>Because stateless components don’t have any state, they can be shared by multiple clients.</strong>The life cycle of stateless components needs to be managed by the server. For example, they need to be initialized on the first request and destroyed when the client terminates or times out. Because of these factors, stateless components are highly scalable, whereas stateful components are expensive and difficult to manage.<br>Web applications, for example, are often designed to work based on REST APIs, and these involve transfer of state from the client to the server with each call.In a machine learning model, there is a lot of state captured during training. Things like the epoch number and learning rate are part of a model’s state and have to be remembered because typically, the learning rate is decayed with each successive epoch. By saying that the model has to be exported as a stateless function, we are requiring the model framework creators to keep track of these stateful variables and not include them in the exported file.When stateless functions are used, it simplifies the server code and makes it more scalable but can make client code more complicated. For example, some model functions are inherently stateful. A spelling correction model that takes a word and returns the corrected form will need to be stateful because it has to know the previous few words in order to correct a word like “there” to “their” depending on the context.</p><p>Issues with calling out model.predict() in stateless functions: <br>1. Need to load entire keras model into memory which can be quite large e.g embeddings for full vocabulary of English words. <br>2. Disconnect between languages(Python) vs different languages for Android or iOS<br>3. The preceding architecture imposes limits on the latency that can be achieved because calls to the predict() method have to be sent one by one.</p><p>Proposed Solution:</p><ol><li>Export the model into a format that captures the mathematical core of the model and is programming language agnostic(Tensorflow has ProtoBuf export).</li><li>In the production system, the formula consisting of the “forward” calculations of the model is restored as a stateless function.</li><li>The stateless function is deployed into a framework that provides a REST endpoint.(AWS Lambda, Heroku etc frameworks take care of autoscaling the infra so as to handle large number of predictions, They are fully managed and language neutral). Sagemaker can be used to deploy TF endpoint.</li></ol><p><strong>Design Pattern 13: Batch Serving:</strong> Used for distributed data processing to carry out inference on a large number of instances all at once, e.g personalised daily music playlist for users. Also used when client code needs to maintain state e.g 5th transaction prediction for a customer after 4 orders in a single session.<br>Stateless serving function (Online) is set up for low-latency serving to support 1000s of simultaneous queries and using them for ocassional or periodic processing of Mns. of items can get expensive.If these requests are not latency-sensitive, it is more cost effective to use a distributed data processing architecture to invoke machine learning models on millions of items. The reason is that invoking an ML model on millions of items is an <strong>embarrassingly parallel problem</strong> — it is possible to take the million items, break them down into 1,000 groups of 1,000 items each, send each group of items to a machine, then combine the results. The result of the machine learning model on item number 2,000 is completely independent of the result of the machine learning model on item number 3,000, and so it is possible to divide up the work and conquer it.<br>E.g- In the background, the recommendations are refreshed periodically. For example, we might retrain the recommendation model hourly based on the latest actions on the website.<br>A production ML system that supports both online serving and batch serving is called a <a href="https://oreil.ly/jLZ46"><em>Lambda architecture</em></a> — such a production ML system allows ML practitioners to trade-off between latency (via the Stateless Serving Function pattern) and throughput (via the Batch Serving pattern).</p><p><strong>Design Pattern 14: Continued Model Evaluation</strong>The Continued Model Evaluation design pattern handles the common problem of needing to detect and take action when a deployed model is no longer fit-for-purpose.Two of the main reasons models degrade over time are <strong>concept drift</strong> and <strong>data drift.<br>Concept drift</strong> occurs whenever the relationship between the model inputs and target have changed. This often happens because the underlying assumptions of your model have changed, such as models trained to learn adversarial or competitive behavior like fraud detection, spam filters, stock market trading, online ad bidding, or cybersecurity. In these scenarios, a predictive model aims to identify patterns that are characteristic of desired (or undesired) activity, while the adversary learns to adapt and may modify their behavior as circumstances change. Think for example of a model developed to detect credit card fraud. The way people use credit cards has changed over time and thus the common characteristics of credit card fraud have also changed. For instance, when “Chip and Pin” technology was introduced, fraudulent transactions began to move more online. As fraudulent behavior adapted, the performance of a model that had been developed before this technology would suddenly begin to suffer and model predictions would be less accurate.<br><strong>Data drift</strong> refers to any change that has occurred to the data being fed to your model for prediction as compared to the data that was used for training. Data drift can occur for a number of reasons: the input data schema changes at the source (for example, fields are added or deleted upstream), feature distributions change over time (for example, a hospital might start to see more younger adults because a ski resort opened nearby), or the meaning of the data changes even if the structure/schema hasn’t (for example, whether a patient is considered “overweight” may change over time).</p><p>Capturing Ground Truth: It is also necessary to capture the ground truth for each of the instances sent to the model for prediction. For example, when a user chooses one of the proposed alternate routes in Google Maps, the chosen route serves as an implicit ground truth. More explicitly, when a user rates a recommended movie, this is a clear indication of the ground truth for a model that is built to predict user ratings in order to surface recommendations.<br><strong>Warning:</strong><br>However, if your model suggests a user will abandon their shopping cart and you offer them free shipping or some discount to influence their behavior, then you’ll never know if the original model prediction was correct. In short, you’ve violated the assumptions of the model evaluation design and will need to determine ground truth labels some other way. This task of estimating a particular outcome under a different scenario is referred to as <strong>counterfactual</strong> reasoning and often arises in use cases like fraud detection, medicine, and advertising where a model’s predictions likely lead to some intervention that can obscure learning the actual ground truth for that example.</p><p>Why it works: <br>Continuous model evaluation provides a framework to evaluate a deployed model’s performance <em>exclusively</em> on new data. This allows us to detect model staleness as early as possible. This information helps determine how frequently to retrain a model or when to replace it with a new version entirely.</p><p>Triggers for Retraining:Continuous evaluation allows you to measure precisely how much in a structured way and provides a trigger to retrain the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amMzY1yq3ZUmoWHTLAkYVw.png" /></figure><p>Scheduled Retraining:Where continued evaluation may happen every day, scheduled retraining jobs may occur only every week or every month.Once a new version of the model is trained, its performance is compared against the current model version. The updated model is deployed as a replacement only if it outperforms the previous model with respect to a test set of current data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2LY2Va98GOmgHYCy0RERfg.png" /></figure><p>The most extreme version of this is <a href="https://oreil.ly/Mj-DA">online machine learning</a>. Some machine learning applications, such as ad placement or newsfeed recommendation, require online, real-time decision, and can continuously improve performance by retraining and updating parameter weights with each new training example.</p><p><strong>Design Pattern 15: Workflow Pipeline </strong>In the Workflow Pipeline design pattern, we address the problem of creating an end-to-end reproducible pipeline by containerizing and orchestrating the steps in our machine learning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XsAT9ZFrsSE_RWYmDHWEmw.png" /></figure><p>In recent years, monolithic apps have been replaced in favor of a <em>microservices</em> architecture where individual pieces of business logic are built and deployed as isolated (micro) packages of code. With microservices, a large application is split into smaller, more manageable parts so that developers can build, debug, and deploy pieces of an application independently.</p><p>Solution: <br>To handle the problems that come with scaling machine learning processes, we can make each step in our ML workflow a separate, containerized service. Containers guarantee that we’ll be able to run the same code in different environments, and that we’ll see consistent behavior between runs. These individual containerized steps together are then chained together to make a <em>pipeline</em> that can be run with a REST API call.</p><p>Why it Works: <br>Without running our ML code as a pipeline, it would be difficult for others to reliably reproduce our work. They’d need to take our preprocessing, model development, training, and serving code and try to replicate the same environment where we ran it while taking into account library dependencies, authentication, and more. If there is logic controlling the selection of downstream components based on the output of upstream components, that logic will also have to be reliably replicated. The Workflow Pipeline design pattern lets others run and monitor our entire ML workflow from end to end in both on-premises and cloud environments, while still being able to debug the output of individual steps.</p><p>Integrating CI/CD with pipelines</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6u8FH2vwXpIo8hXWaFUNEw.png" /></figure><p>A production pipeline also allows for <strong><em>orchestration</em></strong> of an ML workflow. By orchestration, we mean adding logic to our pipeline to determine which steps will be executed, and what the outcome of those steps will be. For example, we might decide we only want to deploy models to production that have 95% accuracy or higher. When newly available data triggers a pipeline run and trains an updated model, we can add logic to check the output of our evaluation component to execute the deployment component if the accuracy is above our threshold, or end the pipeline run if not.</p><p><strong>Design Pattern 16: Feature Store-</strong>The <em>Feature Store</em> design pattern simplifies the management and reuse of features across projects by decoupling the feature creation process from the development of models using those features.<br>Need for Feature Store: <br>1. Many times, the data used for the ML model isn’t drawn from a single source. Some data may come from a data warehouse, some data may sit in a storage bucket as unstructured data, and other data may be collected in real time through streaming.<br>2.The structure of the data may also vary between each of these sources, requiring each input to have its own feature engineering steps before it can be fed into a model.</p><p>Problems with ad-hoc feature generation:<br>1. Ad hoc features aren’t easily reused. Features are re-created over and over again, either by individual users or within teams, or never leave the pipelines (or notebooks) in which they are created.<br>2. Ad hoc features used for training and serving are inconsistent — i.e., training–serving skew. Training is typically done using historical data with batch features that are created offline. However, serving is typically carried out online. If the feature pipeline for training differs at all from the pipeline used in production for serving (for example, different libraries, preprocessing code, or languages), then we run the risk of training–serving skew.<br>3.Productionizing features is difficult. When moving to production, there is no standardized framework to serve features for online ML models and to serve batch features for offline model training.</p><p><strong>Solution:</strong>To create a shared feature store, a centralized location to store and document feature datasets that will be used in building machine learning models and can be shared across projects and teams.This way, there is a central repository to house precomputed features, which speeds development time and aids in feature discovery. This also allows the basic software engineering principles of versioning, documentation, and access control to be applied to the features that are created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m7em2kVYBq5ECzOWC-t_Mw.png" /><figcaption>Sample FS</figcaption></figure><p>Characteristics of Feature Store:<br>1.Tooling to process large feature data sets quickly<br>2.Way to store features that supports both low-latency access (for inference) and large batch access (for model training).<br>3.Metadata layer that simplifies documentation and versioning of different feature sets and an API that manages loading and retrieving feature data.</p><p>Rather than creating feature pipelines to support a single ML model, the Feature Store pattern decouples feature engineering from model development.After features are created, they are housed in a data store to be retrieved for training and serving. For serving feature retrieval, speed is optimized. A model in production backing some online application may need to produce real-time predictions within milliseconds, making low latency essential. However, for training, higher latency is not a problem. Instead the emphasis is on high throughput since historical features are pulled in large batches for training. A feature store addresses both these use cases by using different data stores for online and offline feature access. For example, a feature store may use Cassandra or Redis as a data store for online feature retrieval, and Hive or BigQuery for fetching historical, large batch feature sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSop8ELDs5CdYirj-gKKng.png" /></figure><p>Users and production systems retrieve feature data through a Feast serving data access layer. Since Feast supports both offline and online store types, it’s common to have Feast deployments for both</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEi_HSH3TAE1driS5LuIwg.png" /></figure><p><strong>WHY IT WORKS </strong>Feature stores work because they decouple feature engineering from feature usage, allowing feature development and creation to occur independently from the consumption of features during model development. As features are added to the feature store, they become available immediately for both training and serving and are stored in a single location. This ensures consistency between model training and serving.<br>There is no single database that can handle both scaling to potentially terabytes of data <em>and</em> extremely low latency on the order of milliseconds. The feature store achieves this with separate online and offline feature stores and ensures that features are handled in a consistent fashion in both scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pm64-E596HR_i2PFrlpwow.png" /></figure><p><strong>Design Pattern 17: Model Versioning(Can be used in AB test)-</strong>In the Model Versioning design pattern, backward compatibility is achieved by deploying a changed model as a microservice with a different REST endpoint.Training on this updated dataset improves our overall model accuracy, but slightly reduces accuracy on older “classic” books. To handle this, we’ll need a solution that lets users choose an older version of our model if they prefer.</p><p><strong>Design Pattern 18: Heuristic Benchmark- </strong>It<strong> </strong>compares an ML model against a simple, easy-to-understand heuristic in order to explain the model’s performance to business decision makers.uppose a bicycle rental agency wishes to use the expected duration of rentals to build a dynamic pricing solution. After training an ML model to predict the duration of a bicycle’s rental period, they evaluate the model on a test dataset and determine that the mean absolute error (MAE) of the trained ML model is 1,200 seconds. When they present this model to the business decision makers, they will likely be asked: “Is an MAE of 1,200 seconds good or bad.Avoid the temptation to train even a simple machine learning model, such as a linear regression, on a dataset and use that as a benchmark.<br>As we develop our model, it is a good idea to compare the performance of our ML model against this benchmark. In order to do this, we will be evaluating model performance on different stratifications of the evaluation dataset.By doing so, we can easily diagnose whether our model is overemphasizing the busy, popular stations and ignoring infrequent stations in the training data. If that is happening, we can experiment with increasing model complexity or balancing the dataset to overweight less popular stations.</p><p>Utility Value: Is a 1% improvement good? Is it worth the hassle of putting an ML model into production rather than the simple heuristic rule?”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5dc0c5adf42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Modern Web Application- Personal Notes for standing a Complete WebApp on AWS]]></title>
            <link>https://ravi07bec.medium.com/build-a-modern-web-application-personal-notes-for-standing-a-complete-webapp-on-aws-747334c78c96?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/747334c78c96</guid>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sun, 03 Oct 2021 01:55:41 GMT</pubDate>
            <atom:updated>2021-10-07T05:13:41.416Z</atom:updated>
            <content:encoded><![CDATA[<p>In my journey to learn about full stack software development, I created my individual AWS account and started playing with multiple services. <br>Aim was to visualise the system design interviews learnings in action i.e For an Instagram System design, they talk about No-SQL databases, Blob for image storage, Network load balancer to prevent server overload, Version control and CI/CD frameworks for improving efficiency and cache for high latency.</p><p>I stumbled upon an AWS tutorial repository(Final Product- <a href="https://mythicalmysfits.com/">https://mythicalmysfits.com/</a>) which took me along a journey of creating static website to a modern website with a capability to register users, store clickstream actions and populate webpage based on No-SQL database. It took me 2 days to go through this as explained below and I feel way more confident about system designs now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xpMGDeAPczElC8Bu5EFClw.png" /><figcaption>Modern Web-App architecture</figcaption></figure><p>High point from this course was CI/CD pipeline and docker containers which are much needed even in my Model deployment in Prime Video. Also, I feel comfortable with Load Balancers, API, Microservices, DynamoDB, Cloudwatch and UserPool now. Learning by doing makes it so clear, summarised version of workflows below.</p><p>Step 1: Create a static website</p><ul><li>Copy a static html file to s3</li><li>Publish the website content to s3: <a href="http://testbucketraviaws.s3-website-us-west-2.amazonaws.com/">http://testbucketraviaws.s3-website-us-west-2.amazonaws.com/</a></li></ul><p>Step 2: Build dynamic website using Microservices</p><ul><li>Use AWS Fargate which allows to deploy containers without having to manage any clusters or servers. For our website backend, we will use Python and create a Flask app in a Docker container behind a Network Load Balancer.</li><li>We will start with creating Fargate cluster followed by registering an ECS task definition.A task in ECS is a set of container images that should be scheduled together</li><li>Create a Network Load balancer to sit in front of our service tier and prevent directly exposing to internet.This would enable our frontend website code to communicate with a single DNS name while our backend service would be free to elastically scale in-and-out based on demand or if failures occur and new containers need to be provisioned.</li><li>We will integrate our website with new API backend instead of using the hard coded data that we previously uploaded to S3.</li><li><strong>Automate Deployments(</strong>Continuous Integration and Continuous Delivery or CI/CD)- We will create a fully managed CI/CD stack that will automatically deliver all of the code changes that we make to code base to the service</li><li>For CI/CD Pipeline : <strong>a.</strong> Create A S3 Bucket For Pipeline Artifacts <strong>b. </strong>Create a CodeCommit Repository , <strong>c.</strong>Create a CodeBuild Project- With a repository to store our code in, and an S3 bucket that will be used for our CI/CD artifacts, lets add to the CI/CD stack with a way for a service build to occur. This will be accomplished by creating an AWS CodeBuild Project. Any time a build execution is triggered, <a href="https://aws.amazon.com/codebuild/">AWS CodeBuild</a> will automatically provision a build server to our configuration and execute the steps required to build our Docker image and push a new version of it to the ECR repository we created (and then spin the server down when the build is completed), <strong>d.</strong>Create a CodePipeline Pipeline- We continuously integrate our CodeCommit repository with our CodeBuild project so that builds will automatically occur whenever a code change is pushed to the repository. Then, we need a way to continuously deliver those newly built artifacts to our service in ECS , and <strong>e. </strong>Enable Automated Access to ECR Image Repository- With a CI/CD pipeline in place, you won’t be manually pushing container images into ECR anymore. CodeBuild will be pushing new images now.</li></ul><p>Step 3: Storing information to a No-SQL database(DynamoDB)- Rather than having all of the data be stored in a static JSON file, we will store them in a database to make the websites future more extensible and scalable.<br>Substeps:<br>- Create DynamoDB table <br>- Add Items To The DynamoDB Table in a batch write<br>- Commit the change to git</p><p>Step 4: User Registration and custom personalized features<br>To allow users to vote for their favorite mysfit and adopt a mysfit, we need to first have users register on the website.To make sure that only registered users are authorized to like or adopt mysfits on the website, we will deploy an REST API with <a href="https://aws.amazon.com/api-gateway/">Amazon API Gateway</a> to sit in front of our NLB</p><p>Substeps:<br>-Create Cognito User Pool for email address confirmation<br>- Create An API Gateway VPC Link<br>- Create The REST API Using Swagger<br>- Deploy The API</p><p>Step 5: Capture User Clicks</p><p>Following the link: <a href="https://aws.amazon.com/getting-started/hands-on/build-modern-app-fargate-lambda-dynamodb-python/module-two/">https://aws.amazon.com/getting-started/hands-on/build-modern-app-fargate-lambda-dynamodb-python/module-two/</a></p><p>Next Tutorial Series:Build a Basic Web Application</p><h3>Deploy a web application and add interactivity with an API and a database</h3><p>Tutorial Followed: <a href="https://aws.amazon.com/getting-started/hands-on/build-web-app-s3-lambda-api-gateway-dynamodb/">https://aws.amazon.com/getting-started/hands-on/build-web-app-s3-lambda-api-gateway-dynamodb/</a></p><p>What we built in AWS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70-Q8k84862efUz7o2byDw.png" /><figcaption>WebApp with Interactivity and DynamoDB update</figcaption></figure><p>This provided an overview of launching an html app/form via Amplify. We added dynamic functionality to these pages using AWS Lambda and Amazon API Gateway to call remote RESTful APIs which ingested data in No-SQL DB.</p><p><strong>Problem Statement 3: ML OPS</strong></p><p>The following diagram describes the general overview of the MLOps CI/CD pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*LgO64l9jtfwKlIu7.jpg" /><figcaption>ML OPs using Sagemaker</figcaption></figure><p>The workflow includes the following steps:</p><ol><li>The data scientist works on developing custom ML model code using their local notebook or a SageMaker notebook. They commit and push changes to a source code repository.</li><li>A webhook on the code repository triggers a CodePipeline build in the AWS Cloud.</li><li>CodePipeline downloads the source code and starts the build process.</li><li>CodeBuild downloads the necessary source files and starts running commands to build and tag a local Docker container image.</li><li>CodeBuild pushes the container image to Amazon ECR. The container image is tagged with a unique label derived from the repository commit hash.</li><li>CodePipeline invokes Step Functions and passes the container image URI and the unique container image tag as parameters to Step Functions.</li><li>Step Functions starts a workflow by initially calling the SageMaker training job and passing the necessary parameters.</li><li>SageMaker downloads the necessary container image and starts the training job. When the job is complete, Step Functions directs SageMaker to create a model and store the model in the S3 bucket.</li><li>Step Functions starts a SageMaker batch transform job on the test data provided in the S3 bucket.</li><li>When the batch transform job is complete, Step Functions sends an email to the user using <a href="http://aws.amazon.com/sns">Amazon Simple Notification Service</a> (Amazon SNS). This email includes the details of the batch transform job and links to the test data prediction outcome stored in the S3 bucket. After sending the email, Step Function enters a manual wait phase.</li><li>The email sent by Amazon SNS has links to either accept or reject the test results. The recipient can manually look at the test data prediction outcomes in the S3 bucket. If they’re not satisfied with the results, they can reject the changes to cancel the Step Functions workflow.</li><li>If the recipient accepts the changes, an <a href="https://aws.amazon.com/api-gateway">Amazon API Gateway</a> endpoint invokes a Lambda function with an embedded token that references the waiting Step Functions step.</li><li>The Lambda function calls Step Functions to continue the workflow.</li><li>Step Functions resumes the workflow.</li><li>Step Functions creates a SageMaker endpoint config and a SageMaker inference endpoint.</li><li>When the workflow is successful, Step Functions sends an email with a link to the final SageMaker inference endpoint.</li></ol><p>Sources:</p><ul><li><a href="https://www.youtube.com/watch?v=-UYyyeYJAoQ&amp;ab_channel=DataWorksSummit">https://www.youtube.com/watch?v=-UYyyeYJAoQ&amp;ab_channel=DataWorksSummit</a></li><li><a href="https://www.youtube.com/watch?v=0nr6TPKlrN0&amp;ab_channel=Jay">https://www.youtube.com/watch?v=0nr6TPKlrN0&amp;ab_channel=Jay</a></li><li><a href="https://aws.amazon.com/blogs/machine-learning/build-a-ci-cd-pipeline-for-deploying-custom-machine-learning-models-using-aws-services/">https://aws.amazon.com/blogs/machine-learning/build-a-ci-cd-pipeline-for-deploying-custom-machine-learning-models-using-aws-services/</a></li><li><a href="https://operational-machine-learning-pipeline.workshop.aws/module_conclusion_6.html">https://operational-machine-learning-pipeline.workshop.aws/module_conclusion_6.html</a></li><li><a href="https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/scaling_aws/README.md">https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/scaling_aws/README.md</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=747334c78c96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design Fundamentals notes]]></title>
            <link>https://ravi07bec.medium.com/system-design-fundamentals-notes-abe28b0cd8ce?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/abe28b0cd8ce</guid>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Tue, 28 Sep 2021 04:33:46 GMT</pubDate>
            <atom:updated>2021-09-30T20:46:29.651Z</atom:updated>
            <content:encoded><![CDATA[<p>&lt;&lt;Personal Notes, Using Medium as a cloud repository&gt;&gt;</p><p>I am a Data scientist at Amazon and have worked across science, product, business teams and most recently joined tech team in Prime video. Though I have built ML and Deep learning models in the past, I would want to lead deployment efforts and starting off with learning CS fundamentals.</p><p>I have started with system design and following few medium blogs and Youtube courses/Interviews to get a grasp of the subject.</p><p>Wring down my key takeaways so far:</p><p>First question in sysis Functional and Non-Functional requirements. Functional requirement would cover core function of the system i.e for TikTok it would be uploading video, building user feed and ability of users to like, comment and share. Non-Functional requirements cover availability, reliability, scalability, bandwidth and latency requirements.</p><p>There may be monolithic/single machine and distributed systems. Benefits of distributed systems are reliability/fault tolerance, scalability, cost effectiveness and lower latency.</p><p>Glossary:</p><ul><li>Scalability-Ability of system to grow and manage increased traffic</li><li>Reliability-Probability a system will fail during a period of time, Measured by Mean time between Failure</li><li>Availability- Amount of time a system is operational during a period of time. Poorly designed software requires downtime for updates.</li><li>Efficiency-How well the system performs with increase in traffic, metric is latency and throughput</li><li>Manageability-Speed and difficulty involved with maintaining systems. Difficulty in deploying updates.</li></ul><p>For a design interview, engineers should have Traffic(DAU * Reads or writes per second), Memory(To include 20% as cache), Storage and bandwidth calculations as back of head.</p><p>Scaling an application:</p><ul><li>Horizontal scaling: Multiple servers, complexity up front but efficient long term. Redundancy or backup server built-in and need load balancer to manage traffic. Kubernetes, Docker and Hadoop can be used.When it comes to horizontal scaling, one of the more common techniques is to break up your services into partitions, or shards. The partitions can be distributed such that each logical set of functionality is separate; this could be done by geographic boundaries, or by another criteria like non-paying versus paying users. The advantage of these schemes is that they provide a service or data store with added capacity.</li><li>Vertical scaling: Single server, Becomes slow with increase in traffic. Easiest way to scale an application with diminishing returns(Bigger CPUs wont be cheaper) and single point of failure. Consistent data and fast inter process communication vs horizontal.</li></ul><p>Latency: Communication happens via Fiber optic cable at speed of light however there is a lag in ms if 2 players are playing in Aus and US.</p><p>System Design Components:</p><ul><li>Load Balancer- Balances incoming traffic to multiple servers, can perform dynamic scaling to add or subtract servers. Increases reliability as no Single Point Of Failure. Types of load balancers are Round Robin(Cycle through server),Least connection(Chat app), Least Response Time and IP Hash.Located between client and Server. Elastic Load Balancer(ELB) is managed by AWS and guarantees availability and reliability</li><li>Caching- Located between App server and database and prevents continuous hitting of DB. Reading from memory is 50X faster than from database, Can serve traffic with high efficiency. Can pre-compute and serve data as used by Twitter timeline feed generation. Reads have more use case for caching than Writes. Example- Redis Cache. Redis can be deployed as standalone docker container. Cache runs on SSD and is expensive which prohibits all the data to be stored in cache. Cache policy determines cache entry and cache eviction with examples as LRU,LFU, Sliding window policy. Distributed/Global cache such as Redis is closer to database and is resilient and ensures data consistency.</li><li>Caching layers- DNS, Content Distribution Network(Pictures and Videos), Distributed cache and cache eviction i.e cleaning up stale data. TTL(Time to live) decides on time when cache memory would be cleared lets say front page from NYT. Caching can be cleared by Least Recently used or Least Frequently used .</li><li>Database scaling- Most webapps are generally reads i.e 95% read:write ratio. Most used scaling techniques are Indexes(E.g lookup for popular celebrities), Denormalisation, Connection Pooling(Allows multiple application threads to use same DB i.e JB content if accessed by one user in SF can be shared to others), Caching(Most relevant) and Vertical Scaling.</li><li>Synchronous requests-When the server receives more requests than it can handle, then each client is forced to wait for the other clients’ requests to complete before a response can be generated. Degrades performance.<br>Solution is Queue i.e a task comes in, is added to the queue and then workers pick up the next task as they have the capacity to process it.Queues enable clients to work in an asynchronous manner, providing a strategic abstraction of a client’s request and its response. While the client is waiting for an asynchronous request to be completed it is free to perform other work, even making asynchronous requests of other services.</li></ul><p>Tradeoffs for Database scaling:</p><ul><li>Replication( Create replica servers to handle reads)</li><li>Partitioning, Horizontal Partitioning using Sharding- Schema of table remains same but split across multiple DBs. Can result in uneven traffic e.g XYZ names have lower hits than ABC.</li><li>Partitioning, Vertical Partitioning-Divide schema into multiple tables. Insta split like count on photos vs other user attributes for high performance.</li><li>Relational Database vs NoSQL: More structured, If a single user has multiple videos , Relational DB is better. Free form searching is better for Non SQL. Relational DB is space and speed efficient. Advantages of NoSQL databases are insertions and deletions, schema is changeable and built for Scale with problems of non-read optimized, non-guaranteed ACID as compared to SQL DB.</li></ul><p>Design a Youtube clone-</p><ol><li>Functional requirement is uploading video, being able to view video, search functionality, comment and video stats.</li></ol><p>2. Estimates: Capacity(Storage and Bandwidth)<br>Database Design: Video table (VideoID,User, Likes, Dislikes) , User Table(User, emailid) and Comments table(user, VideoID)</p><p>3. System Architecture-</p><ul><li>CDN should store top 20% of video</li><li>Webserver: Distributing traffic across web servers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oRkKxwMZ1QfjMVcDtGmK_w.png" /><figcaption>System Architecture for YT cline</figcaption></figure><p>Design TikTok:</p><p>Functional Requirement is user should be able to upload videos, view feed and perform user actions such as like, share etc. <br>Non-Functional- It should be scalable to 1Mn DAU with 99.9% availability time and a storage of 10Mb per user.</p><p>Database should have Blobs(10Mb per Video) and Relational(User, Content id etc). Caching is an important consideration to pre-load the feed for every user id on app launch, caching should connect to RDB read only data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Te0qrRFP8nxZjlR6YjQtg.png" /></figure><p>Key principles that influence the design of large-scale web systems:</p><ul><li><strong>Availability:</strong> The uptime of a website is absolutely critical to the reputation and functionality of many companies. For some of the larger online retail sites, being unavailable for even minutes can result in thousands or millions of dollars in lost revenue, so designing their systems to be constantly available and resilient to failure is both a fundamental business and a technology requirement. High availability in distributed systems requires the careful consideration of redundancy for key components, rapid recovery in the event of partial system failures, and graceful degradation when problems occur.</li><li><strong>Performance:</strong> Website performance has become an important consideration for most sites. The speed of a website affects usage and user satisfaction, as well as search engine rankings, a factor that directly correlates to revenue and retention. As a result, creating a system that is optimized for fast responses and low latency is key.</li><li><strong>Reliability:</strong> A system needs to be reliable, such that a request for data will consistently return the same data. In the event the data changes or is updated, then that same request should return the new data. Users need to know that if something is written to the system, or stored, it will persist and can be relied on to be in place for future retrieval.</li><li><strong>Scalability:</strong> When it comes to any large distributed system, size is just one aspect of scale that needs to be considered. Just as important is the effort required to increase capacity to handle greater amounts of load, commonly referred to as the scalability of the system. Scalability can refer to many different parameters of the system: how much additional traffic can it handle, how easy is it to add more storage capacity, or even how many more transactions can be processed.</li><li><strong>Manageability:</strong> Designing a system that is easy to operate is another important consideration. The manageability of the system equates to the scalability of operations: maintenance and updates. Things to consider for manageability are the ease of diagnosing and understanding problems when they occur, ease of making updates or modifications, and how simple the system is to operate. (I.e., does it routinely operate without failure or exceptions?)</li><li><strong>Cost:</strong> Cost is an important factor. This obviously can include hardware and software costs, but it is also important to consider other facets needed to deploy and maintain the system. The amount of developer time the system takes to build, the amount of operational effort required to run the system, and even the amount of training required should all be considered. Cost is the total cost of ownership.</li></ul><p>Sources:</p><ul><li><a href="https://www.youtube.com/watch?v=MbjObHmDbZo">https://www.youtube.com/watch?v=MbjObHmDbZo</a> — Primer Course on System Design</li><li><a href="https://www.youtube.com/watch?v=Z-0g_aJL5Fw">https://www.youtube.com/watch?v=Z-0g_aJL5Fw</a>- Design Tiktok</li><li><a href="https://www.youtube.com/watch?v=REB_eGHK_P4">https://www.youtube.com/watch?v=REB_eGHK_P4</a></li><li><a href="https://www.youtube.com/watch?v=Z3SYDTMP3ME&amp;ab_channel=AWSTrainingCenter">https://www.youtube.com/watch?v=Z3SYDTMP3ME&amp;ab_channel=AWSTrainingCenter</a>- AWS services for system design</li><li><a href="https://gist.github.com/vasanthk/485d1c25737e8e72759f">https://gist.github.com/vasanthk/485d1c25737e8e72759f</a>- System Design Cheatsheet</li><li><a href="http://www.aosabook.org/en/distsys.html">http://www.aosabook.org/en/distsys.html</a>- Book on system design</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abe28b0cd8ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Hello world to Quantum computing of dashboard in Python (Part 1 of 3)]]></title>
            <link>https://medium.com/analytics-vidhya/from-hello-world-to-quantum-computing-of-dashboard-in-python-part-1-of-3-f9f5565a64a7?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9f5565a64a7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dash]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Thu, 29 Oct 2020 08:53:36 GMT</pubDate>
            <atom:updated>2021-02-15T23:46:41.958Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tell your Data Story in Hours and not Days using DASH</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y6KISHKJoVaFqM52" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The <em>New York Times</em> on an average Sunday contains more information than a Renaissance-era person had access to in his entire lifetime.We’re getting better and better at collecting data, but we lag in what we can do with it. Lots of data is out there, but it’s not being used to its greatest potential because it’s not being visualized as well as it could be.</p><p>As legends have it, &gt;90% of Data science projects don’t see the light of the day and ML models die their slow deaths within jupyter notebooks. Absence of a sweet intersecting spot between Data Scientists and business stakeholders prevents 2-way communication and <em>“We Will get back to you” </em>meeting never materializes. What if the meeting starts with a good data based storytelling where the stakeholders keep asking follow up questions and getting the answer right there keeping both the parties involved.I have noticed it more than once that a good story makes for a great first impression.A good story needs to stand by itself, should be self serving and should blow the audience off their feet.Now, a good story cant be told for sure in a redshift table or a jupyter notebook as they have shareabaility concerns and have steep learning curve for business audiences. Business folks may come back with N! questions from a minimal dataset with 6 columns on New York Taxi Fare dataset( <a href="https://www.kaggle.com/c/new-york-city-taxi-fare-prediction">https://www.kaggle.com/c/new-york-city-taxi-fare-prediction</a>) — ID, Fare, Time of Trip, Passengers and Location. Their questions may range from:</p><blockquote><em>1. How the fares have changed Year-Over-Year?</em></blockquote><blockquote><em>2. Has the number of trips increased across the years?</em></blockquote><blockquote><em>3. Do people prefer traveling alone or they have company?</em></blockquote><blockquote><em>4. Has the small distance rides increased as people have become lazier?</em></blockquote><blockquote><em>5. What time of the day and day of week do people want to travel?</em></blockquote><blockquote><em>6. Is there emergence of new hotspots in the city recently except the regular Air Port pickup and drop?</em></blockquote><blockquote><em>7. Are people taking more inter-city trips?</em></blockquote><blockquote><em>8. Has the traffic increased leading to more fares/time taken for the same distances?</em></blockquote><blockquote><em>9. Are there cluster of pick-up and Drop points or areas which see high traffic?</em></blockquote><blockquote><em>10. Are there outliers in data i.e 0 distance and fare of $100+ and so on?</em></blockquote><blockquote><em>11. Do the demand change during Holiday season and airport trips increase?</em></blockquote><blockquote><em>12. Is there any correlation of weather i.e rain or snow with the taxi demand?</em></blockquote><p>However, if we provide a good story telling platform a.k.a dashboard hosted on a Webapp, they would have a secondary platform to quench all such curiosities even after in-person meeting.</p><p>On impact of powerful storytellings, there are dashboards that have created ripples right from the word go:</p><ol><li>New York Times portrayal of how Coronavirus travelled- The visualization demonstrates lifecycle of Coronavirus in US starting with 1st infection to Millions with a clear virus trajectory and how unrestrained people movement has led to situation going out of hand. This makes a compelling case for Stay-In-Home orders and subsequently help in flattening the curve.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0OrbGUdeUzs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0OrbGUdeUzs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0OrbGUdeUzs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/85a3aef930e3b7d5d3e35a12b0b16e80/href">https://medium.com/media/85a3aef930e3b7d5d3e35a12b0b16e80/href</a></iframe><p><a href="https://www.nytimes.com/interactive/2020/us/coronavirus-spread.html">How the Virus Won</a></p><p>2. Average day in life of an American: This visualization portrays average day in the life of an American starting from 4 am to Midnight and interspersed between Sleeping, Work, Leisure, chores etc. The animation is right on point to establish the life patterns and can be of so much use to technology companies for reaching out to their target group e.g Push notification for Netflix at 7:00 pm.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLQXk8iwUcfo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLQXk8iwUcfo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLQXk8iwUcfo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6cbcc7c684662f35ce65015299e9042/href">https://medium.com/media/a6cbcc7c684662f35ce65015299e9042/href</a></iframe><p><a href="https://flowingdata.com/2015/12/15/a-day-in-the-life-of-americans/">A Day in the Life of Americans</a></p><p>During the course of this article spread across 3 parts, we will be never moving out of python ecosystem and deploying Plotly based Dash WebApps for showcasing to the world.Series of Animations/Screenshot below represents my handpicked components while working on Dash for last 2 years. During next few weeks we will be building these dashboards together in our open source datasets.</p><ol><li><strong>Integrating Tensorboard projector:</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*lAcTwxRRz4PMvVoRbC9hMQ.gif" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96c83e7ebbb7820658cf3a1f013709c7/href">https://medium.com/media/96c83e7ebbb7820658cf3a1f013709c7/href</a></iframe><p>I have written a blog post with detailed blog post here:</p><p><a href="https://medium.com/towards-artificial-intelligence/build-floating-movie-recommendations-using-deep-learning-diy-in-10-mins-26f585821697">Build Floating Movie Recommendations using Deep Learning — DIY in &lt;10 Mins</a></p><p>2. Creating network diagram using <a href="https://github.com/WestHealth/pyvis">Pyvis</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*UQnvvDK5tBFXbmtS_9RcJw.png" /></figure><p>3. Wordcloud</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*_wPSqnPJwIwk8rn0k4hkMg.gif" /></figure><p>Code for generating wordcloud</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0266bbee112ddb1d6b4c69ee15cdef45/href">https://medium.com/media/0266bbee112ddb1d6b4c69ee15cdef45/href</a></iframe><p><strong>4. Travelling Salesman Problem</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*qUpzTiWdHMZqQ46eq80dEQ.gif" /></figure><p><strong>5. Playing Youtube video live</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*74v6sUW8OZXu4_BkjxYITQ.gif" /></figure><p>Code for attaching YouTube video in Dash</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07b3225a0e289d9533cd6f77eeacc533/href">https://medium.com/media/07b3225a0e289d9533cd6f77eeacc533/href</a></iframe><p>6. <strong>Treeplot to dive deep into sub segments:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*tN7-KlX4D-n2c7sgkx0wHw.gif" /></figure><p>Code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da79a37101a604c5a66ec0ab8e222b8c/href">https://medium.com/media/da79a37101a604c5a66ec0ab8e222b8c/href</a></iframe><p><strong>8. Integrate Google trends Data:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*mGyqo6QPuOgopERn_09bzg.gif" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0a88dc5c1b37c32e1e340c83c02392b/href">https://medium.com/media/f0a88dc5c1b37c32e1e340c83c02392b/href</a></iframe><p><strong>9. Dynamic Drill-down using Dash tables and radio buttons:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*fWhmHQFUv_u-FVKXheR13g.gif" /></figure><p><strong>10. Images display and aggregated information in Dash Table:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*rZMA1GTRGJoo2hoWpoQzLA.gif" /></figure><p>I love storytelling and I have been looking out for that perfect dashboard toolkit since last 6 years and the journey has stopped at Dash where I have found a sweet mix of everything.Chronological order of my tryst with dashboards:</p><p><strong>2014- Excel Dashboards: </strong>Started fidgeting with spreadsheets, macros and pivot charts. Since it was an extension of regular excel data analysis, the learning curve was smooth. However, excel limits the data size to 1Mn rows in addition to slow data joins across tables. Since the reporting is ad-hoc, the dashboards are not shareable with static data.<br><strong>2015-Shiny: </strong>Shiny is an R package that makes it easy to build interactive web apps straight from R.<strong> </strong>It<strong> </strong>combines the computational power of R with the interactivity of the modern web. However, I started facing challenges with Data scaling up in R owing to latest developments being constrained to python, difficulty to launch R studio on linux servers and better community and my company support to Python.<br><strong>2016–17- Tableau: </strong>I was blown by the richness of possible visualizations with mere Drag N Drop. In fact I loved tableau so much that I made my entire CV in tableau. Legends has it that my tableau CV went viral and got called for interview at Amazon(To share in another post). I could connect Redhsift/CSV files in tableau and the charts would update automatically based on updated data, Seemed just like magic. However, there are costs associated with such rich features(Not just talking about $15.5Bn. Tableau Acquisition by Salesforece). The trial is expensive in addition to difficulty in share-ability (Either TWBX file or tableau online). Free version is provided by tableau public but the dashboards cant be saved in local machine. I maintain my tableau public profile <a href="https://public.tableau.com/profile/ravi.shankar1671#!/">here</a>, would love to get feedbacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ojh7q6qUUCiv2A3PrLovDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9t3HySQybBfM_qzU7fEpCw.png" /><figcaption>CV and Recommendation engine in Tableau</figcaption></figure><p><strong>2017–18- </strong><a href="https://superset.apache.org/"><strong>Apache Superset:</strong></a><strong> </strong>Since tableau had difficulty in creating calculated columns and entire data had to be fetched every time, I started exploring open source tools which could integrate seamlessly with cloud based data warehouses. Key advantages of superset is easy deployment on localhost and linux with clear charts on flask ecosystem. Since it runs SQL queries in the backend, calculated columns is a breeze using enterprise based warehouses(Redshift/Big Query/Athena). However, the number of charts are limited and it needs reshuffling between python &amp; sql ecosystem for advanced data visualizations.<br><strong>2019-Ongoing- Dash: </strong><em>“This is how I met the love of my dashboarding life”</em> i.e Dash.Since last 1.5 years , I have spent ~25% of my working time on Dash and it has helped me create scalable solutions not just for myself in jupyter notebooks but also has led to my evolution from mere model building on jupyter notebooks to taking the projects live.</p><p>Summary Table of Comparison :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6AxycaBCnoYPitRsUKtiRA.png" /></figure><p><strong>Flowchart for a dashboard deployment:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AwmT9JjZXGu8RUS9fAOMkQ.jpeg" /></figure><ol><li>Resourcing and stakeholders- Identify the core team driving the dashboard with clear deliverables, timelines and checkpoints.</li><li>Design- Always start with a rough sketch on paper as it gives more room for creativity and scribbling seemingly random dashboard components at start.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxrpsRe_okGd4miiBxdywQ.png" /></figure><p>3. Create the components in a modular fashion and keep getting feedbacks. Also, consider the coding best practices like code comments and scalability with proper classes &amp; modules into consideration as the dashboard must outlive the org tenure.</p><p>4. Create a scalable pipeline to update data (Connection to redshift, code running by DJS, Batch processing) with minimal manual interference. It is an oft-ignored component but starts creating trouble once dependencies are built on the dashboard.</p><p>5. Test on localhost/non prod servers and do a dry run with colleagues to prevent it breaking on prod systems</p><p>6. Deploy using Servers(EC2, Heroku, Elastic Beanstalk)</p><p>7. Optional- Get a professional name, everyone prefers a name like www.buyyourbunny.com vs www.ec2jjjdxxrrr.com:8080</p><p>Summary:</p><ol><li>We Covered grounds on need of a good storytelling</li><li>Potential to improve the success rate of Data Science projects by having a compelling visualization</li><li>Shareability power of dashboards so that the stakeholders become Force Multiplier in seeking additional funding for the project</li><li>My individual tryst with dashboards across last 6 years and my favorite creations in Dash</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9f5565a64a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/from-hello-world-to-quantum-computing-of-dashboard-in-python-part-1-of-3-f9f5565a64a7">From Hello world to Quantum computing of dashboard in Python (Part 1 of 3)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mr. Market has Already Prophesied How the Post-Covid World Will Look Like]]></title>
            <link>https://medium.com/swlh/mr-market-has-already-prophesied-how-post-covid-world-will-look-692b250bc694?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/692b250bc694</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[coronavirus]]></category>
            <category><![CDATA[probability]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sun, 25 Oct 2020 06:07:48 GMT</pubDate>
            <atom:updated>2020-11-05T21:57:35.769Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/data-science">Data Science</a></h4><h4>The market knows it all, and it has prepared itself for the post-Covid world using millions of data points, Have you?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLmm6-JXuRUKiLn_LMfpvw.png" /><figcaption>Source: image by Author</figcaption></figure><p><em>“In life, unlike chess, the game continues after checkmate”</em>-<a href="https://www.goodreads.com/author/show/16667.Isaac_Asimov">Isaac Asimov</a>.</p><p>Coronavirus took everyone by surprise, and it seemed the end of the world and of the human race, which was next to checkmate in Chess. However, life must go on, and after analyzing 30Mn.+ data points across 35 years ranging from Dow jones index to sectors to individual companies for price movements, financial results, news, management guidance, etc., Mr. Market has a clearer idea of how the post-Covid world is going to shape.</p><p>If Mr. Market is to be believed, below is a summary of the new normal based on stock performances, i.e., prices on the 1st week of January vs. last week of September in 2020. To give a perspective on the quantum of price changes, Warren Buffet has a CAGR return of 19.1%, and Dow Jones has a CAGR of 5.1%. Savings accounts in the US and India provide an interest rate of ~2% and ~5%, respectively.</p><ul><li>People will wake up and go for a run in their athletic clothes-Lululemon(+32%) or Nike(+28%) using Apple Watch(+54%) while tuning to Spotify(+16%)</li><li>They would be getting ready for WFH in comfortable clothes, No Levi Strauss(-29%) or Guess(-48%)</li><li>Since they are not traveling for work, less oil consumption from British Petroleum(-52%) and Schlumberger(-39%). Also, there are fewer carpools from Lyft(-38%).</li><li>They would fire up their workstations using more Wifi, Cable One(+25%), and less mobile network, AT&amp;T(-23%). Faster processing is of utmost importance, which will benefit chip manufacturers such as Nvidia(+125%) and AMD(+68%). Since the storage and computing would be on servers, external memory drives such as Seagate(-16%) and Western Digital(-43%) would slim down in importance.</li><li>They will join their Zoom calls(<strong>+670%</strong>) using Logitech Webcam(+62%) and will invest in Home improvements/Furniture using Wayfair(<strong>+212%</strong>) &amp; Lowe’s (+39%). No wonder we are living in Zoom Economy.</li><li>Companies will engage Cloud-based information security companies ,e.g Zscaler(+197%),DocuSign(+183%) &amp; Okta(+82%)</li><li>With increasing data size, Cloud-based data companies like Mongo DB(+72%), AWS, and financial data providers such as S&amp;P Global(+31%) and Moody(+20%) will thrive.</li><li>To prevent the social gathering, the majority of customer demands would be fulfilled from e-commerce companies such as Amazon-Everything Store(+66%), Wayfair-Furniture(+212%), Etsy- Personalised items(+169%), and Chewy- Dog Food(+85%). 12 out of 13 listed e-commerce has grown in a pandemic with a median growth of 105% and the highest growth in Overstock(+932%) and Wayfair(212%).</li><li>People will prefer fashion shopping at home and visit less of departmental stores like Macy’s(-64%), Nordstrom(-69%), and Kohl’s(-61%).</li><li>Since offices are closed and less of urban focussed development, Real estate companies such as EPR Properties(-58.5%), Kimco Realty(-41%), etc. will take a huge hit</li><li>Since small businesses are migrating online, companies serving B2B customers such as Shopify(+151%), Wix(+99%), and Salesforce(+51%) will see a huge demand shift in their favor. Also, since it is an ecosystem change, end-to-end integration would need seamless payment infrastructure, i.e. rise of Paypal(+88%), Square(+155%), etc.</li><li>For Covid symptoms identification, Teladoc(+163%) and Veeva(+96%) would rise to the occasion. Biotech research companies, Novavax<strong>(+2313%)</strong>, Seres Therapeutics (+745%) with a shot to Covid vaccine have seen a meteoric rise in their valuation.</li><li>Netflix(+52%) would become a de-facto entertainment source with a major blow to theatres such as Imax(-41%) and WWE(-36%).</li><li>People will restrict to in-home entertainment i.e Gaming on Sea Limited(+284%),Gravity Co.(+269%), Playstation etc</li><li>For holidays, people will prefer to stay put, leading to a sharp drop in airlines, hotels, cruise, Theme parks, and gambling stocks.</li><li>Downturn and Zero FED interest rates would lead to a sharp drop in banks valuation due to drying up of their major revenue source</li></ul><p>Every coin has 2 sides and with the world order changing, there have been clear winners and losers as identified objectively by Mr. Market. Covid-19 has been an opportunity for some and disaster for others, e.g <strong>9.1%</strong> of 2k+ US-listed companies(Valuation of $1Bn.+) had price change of <strong>&gt;50%</strong> while <strong>5%</strong> of companies saw its stock prices slashed by &gt;60%.</p><p>The choppiness in Dow Jones in 2020 is evident by market performances across the past 35 years. The below chart represents the Dow Jones index movement on 1st vs Last trading day of the year and compares against the average daily volatility. 2020 has been 3rd most fluctuating year in history after 2002 and 2008 i.e 2020 witnessed an average 1.1% up/down every day in Dow Index. However, to an outsider, the overall market is stable i.e Dow started the year at 28,868 and ended Q3 at 27,781 which ignores the storm brewing within, and during the course of this article, we will explore who is sailing against the wind. Random walks by Mr. Market also represent its inability to decide its movement trajectory who is confused with #Covid cases, Fed rates, Vaccine announcements, Stimulus checks, presidential elections, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xutac-fA41Lr9d12WvoBPg.png" /><figcaption>Dow Jones- Risk vs Returns since 1990</figcaption></figure><p>Imagine someone, let’s say Mr. Time Traveller went to space at the end of 2019 and returned to an exciting new decade all charged up and buzzing with energy. On his way back, he met Mr. Market a.k.a Dow Jones, and wanted to catch up on events that have happened in 2020 and understand Mr. Market’s vision about the future.</p><blockquote><strong>Mr. Time Traveller:</strong> Hello, Mr. Market, Howdy!Whats happening on planet earth?<br><strong>Mr. Market: </strong>2020 has been all about a novel Coronavirus which originated somewhere in china. It is a respiratory contagious disease with ongoing research for vaccine. It has changed the entire course of human mankind where tech adoption has been accelerated, oil crisis and pollution is under control and remote collaboration is possible at a scale never witnessed before</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bhvdK7UMOjTyBNnGz4TrHg.jpeg" /><figcaption>2020 by Word Cloud</figcaption></figure><blockquote><strong>Mr. Time Traveller:</strong><em> This is unbelievably huge, May I know the source and depth of your knowledge?<br></em><strong>Mr. Market: </strong><em>I have 35 years data of dow jones and stock market data for every day for 4000 stocks i.e 4000*365*35years, Financial performances and news about every company. Feel free to ask follow up questions</em></blockquote><blockquote><strong>Mr. Time Traveller:</strong> <em>Wow thats some big data at your disposal. Can you give me an executive overview of major changes due to this pandemic?</em><strong><br>Mr. Market:</strong><em>Pre-Covid, Energy and Financial Services used to account for 7.7% of Market share which has dropped to 6.3%(-18%) implying that people are using less oil, traveling less and not engaging much with banks. Technology and Consumer Cyclical has seen massive jump from 17.5% to 24.6%(+41%) implying rapid tech adoption .</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etTcCThqI35ka5n8rOkRgA.png" /><figcaption>Sector Weights change in 2020</figcaption></figure><blockquote><strong>Mr. Time Traveller:</strong> <em>This is an exciting change, Can you provide a deep-dive at sector level?<br></em><strong>Mr. Market:</strong><em>Sure, Below chart represents performance of individual stocks in the industry, e.g if there are 3 stocks in Internet Retail with price change of +10%,+15% and +20% respectively, mean industry change would be 15%.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pDrcJHKsZJqchH3TmXNCPQ.png" /><figcaption>Industry performances during 2020</figcaption></figure><p>Learnings: Companies within the Solar industry have increased by 1.8X implying people’s adoption of free abundant sunlight for power generation. Internet retail(+1.1%) represents a social distancing impact and prefer the convenience of orders at fingertips. Home Improvement retail(+0.9X) represents focus on improving their infrastructure as it is new Home+Office( Zoom video must have also played a role)</p><p>Industries with sharp drops in their underlying stocks:</p><ol><li>Oil &amp; Gas Equipment(-50%): People aren’t traveling and the world can live without burning extra XX gallons oil every day, Good for pollution control!</li><li>Airlines and marine Shipping</li><li>Department stores- Zero-sum game between internet retail and offline retail</li></ol><blockquote><strong><em>Mr. Time Traveller:</em></strong><em> How has the importance of companies changed ? Which are the companies which gained and lost most in their valuation?<br></em><strong><em>Mr. Market:</em></strong><em> Green-shots are visible in tech companies with special shoutout to Tesla(+$340Bn.) and Nvidia(+174 Bn.). Energy companies and banks are trending in red due to less commute and zero FED interest rates.Below bubble plot represents the $Bn. difference in Top 20 companies since the start of pandemic.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2s67FlP8ADOOtKvTpzdYeg.png" /></figure><blockquote><strong><em>Mr. Time Traveller:</em> </strong><em>At an overall market cap level, similar stories of Amazon, Apple and Microsoft is emerging, Can you provide insights into reshuffling of market order across multiple valuation tiers? Which of the companies has jumped/lost most spots?<br></em><strong><em>Mr. Market: </em></strong>Sure, Interesting question. Below charts represent the jump in valuation spot, e.g 29 in Nvidia represents it jumped 29 spots from 42nd position in Jan’2020 to become 13th most valuable company owing to the importance of faster processing on GPU and Artificial Intelligence trend. Red bar represents the carnage i.e General Electric and BP dropped by 60 and 58 spots respectively.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aZOGCXopk-kEsnGAqzpNUQ.png" /><figcaption>Positions change of Large Cap companies</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iaxW-FF9i_w6Qbm7TsGPqQ.png" /><figcaption>Positions change of Mid Cap companies</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5bL5sTPKQjyVK-mD89-8A.png" /><figcaption>Positions change of Small Cap companies</figcaption></figure><blockquote><strong><em>Mr. Time Traveller:</em> </strong>This presents an interesting viewpoint across market caps and order shuffling, Can we have an overall visibility on sector trend?<br><strong>Mr. Market-</strong> Below chart is a violin plot where the small white dot at centre represents median change of companies within the sector. The fatness represents number of companies at every range e.g Technology has seen extremes(lean graph) vs Financial services sector is constrained in a small range(Potbelly shape graph).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0QWe-JIdCt--QBtG7fV3qw.png" /></figure><p>The technology and healthcare sectors have been the biggest gainers. Finance, Real estate, and the Energy sector has taken hits as a result of FED involvement, WFH policies, and behavioral changes introduced by Covid</p><blockquote><strong><em>Mr. Time Traveller:</em></strong> How are individual industries in the respective sectors performing?<br><strong>Mr. Market:</strong> Brace yourself for a lot of data-points across individual sectors.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUQb7XrE9P3wNqVlNqNJuQ.png" /></figure><p><strong>Technology: </strong>Solar industry has increased by 180% i.e on average all solar companies have roughly doubled. Infrastructure, data, Semiconductors, and Saas companies are leading to the tech revolution and thus have been valued generously. However, there are dark spots in Hardware(Memory drives) and distribution companies probably due to cloud migration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UNuZBAhwXui0DMzR3R5hxw.png" /></figure><p><strong>Consumer Cyclical-</strong> Internet Retail and Home improvement have led the pack while gambling, luxury goods(faces hidden behind masks don’t really add much value to the luxury quotient) and textile manufacturing(Why would anyone wear jeans in WFH) has been brutally hit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9EzK0sPUwx7k7GBaVdFhfg.png" /></figure><p><strong>Healthcare: </strong>Coronavirus is a health crisis and who else would benefit apart from this sector. It&#39;s majorly green-shots with R&amp;D and health information companies providing silver light to the early vaccine and fighting misinformation. Retailers and generic drug manufacturers are trending in the red owing to social distancing and lower spread of non-Covid d.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3kwDdIgGKUIJ1YlW6NN_Wg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCOsSMIXJxbO4vMvzL4Kdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQ7BcytcCDwyPmKiAkh-tA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CL9jyIMEvc3Dy3RmL4JZUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnYL7P4Bkx0gFz_Bw2oxew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FNX65eVUd_NTLQabImTQNA.png" /></figure><p><strong>Consumer Defensive:</strong> Balanced sector with discount stores and grocery stores trending upwards while Household products and farm products in green. Reduction in tobacco companies probably due to less stress in WFH and need for healthy lungs to fight Covid( Joke attempt 😃)</p><p><strong>Basic Materials-</strong> Gold and silver have suddenly become important currencies to hedge against the dollar due to huge stimulus checks and possible devaluation of the currency. The economy is on standstill will fewer construction works involving steel, aluminum, and Coking coal.</p><p><strong>Communication Services: </strong>Gaming industries have witnessed a boom probably to engage kids with remote schools. Publishing has slowed down as content is available on the internet/kindle while Advertising agencies slowdown can be attributed to general industry slowdown and reduced purchasing power.</p><p><strong>Industrials-</strong> There has been clear demarcation in this sector with integrated freight and Trucking companies such as FedEx(+64.2%) benefitting from e-commerce proliferation while airlines and Marine shipping taking a hit to prevent transmission of infection.</p><p><strong>Financial Services:</strong> Banks have taken a huge hit due to FED policies and the general economic slowdown. Valuations of JP Morgan and Wells Fargo have dipped by more than $100 bn. However financial data provider companies have seen an uptrend.</p><p><strong>Energy:</strong> Despite all the negative effects of Covid, it has definitely solved pollution and diminishing energy problems. Reduced demand has led to carnage in this sector with the silver lining of movement from conventional coal-based power to nuclear energy</p><blockquote><strong><em>Mr. Time Traveller:</em></strong> Wow thats a lot of information and learning to absorb, If I had to make some quick bucks, can you please provide an overview of companies performance in best and worst performing industries?</blockquote><blockquote><strong>Mr. Market: </strong>Sure, I am listing down the industries and stocks on both ends of spectrum and their performances across the year, e.g Tesla in Auto Manufacturers industry has seen 398% price jump while Norwegian Cruises in Travel services has lost 71% of its market cap in 2020.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTWpcb0xmQwvnlgnU05Fuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9com0SOI7eBQcoc_k_zKEA.png" /><figcaption>Top 20 Gaining and Losing sectors</figcaption></figure><blockquote><strong><em>Mr. Time Traveller:</em></strong> Thats really good and I will learn more about them by googling, However can you provide a split of performances by Marketcap where I can play around and make informed decision based on my risk appetite.</blockquote><blockquote><strong>Mr Market:</strong> Sure, let me integrate a plotly chart below so that you click on Market cap buttons and get returns across individual companies in 2020.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Ep12ravishankar%2F10.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Ep12ravishankar%2F10%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/8237eaa60a826c274f9b60afb071432a/href">https://medium.com/media/8237eaa60a826c274f9b60afb071432a/href</a></iframe><blockquote><strong><em>Mr. Time Traveller:</em></strong><em> This is very helpful. Thanks for detailed peek into the future and insights on how the new normal look like. Let’s be connected.</em></blockquote><blockquote><strong><em>Mr. Market:</em></strong><em> Sure, I enjoyed our conversations, please ping back in comments if you remember something later during deep dives and would want to know more about the datasets, codes and methodologies. Feel free to connect on </em><a href="https://www.linkedin.com/in/ravi-shankar-83863441/"><em>linkedin</em></a><em>, I greatly enjoy data driven conversations. Also, check out my other stories on </em><a href="https://ravishankar-22148.medium.com/"><em>medium</em></a><em> for other topics across </em><a href="https://medium.com/towards-artificial-intelligence/billions-of-rows-milliseconds-of-time-pyspark-starter-guide-c1f984023bf2"><em>Pyspark</em></a><em>, </em><a href="https://medium.com/analytics-vidhya/pandas-on-steroids-dask-end-to-end-data-science-with-python-code-1845d3722c8a"><em>Dask</em></a><em>, </em><a href="https://medium.com/towards-artificial-intelligence/build-floating-movie-recommendations-using-deep-learning-diy-in-10-mins-26f585821697"><em>Movie Recommendations</em></a><em>, </em><a href="https://medium.com/analytics-vidhya/a-picture-is-worth-a-thousand-words-lets-figure-out-the-relevant-ones-15cbb56443ae"><em>Neural Networks</em></a><em> etc.<br>And who doesn’t enjoy a small clap on the story, Keep learning. <br></em><a href="https://www.spanishdict.com/translate/Adi%C3%B3s.%20Hasta%20la%20pr%C3%B3xima.?langFrom=es">Adiós</a>!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=692b250bc694" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/mr-market-has-already-prophesied-how-post-covid-world-will-look-692b250bc694">Mr. Market has Already Prophesied How the Post-Covid World Will Look Like</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Floating Movie Recommendations using Deep Learning — DIY in <10 Mins]]></title>
            <link>https://pub.towardsai.net/build-floating-movie-recommendations-using-deep-learning-diy-in-10-mins-26f585821697?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/26f585821697</guid>
            <category><![CDATA[netflix]]></category>
            <category><![CDATA[movie-review]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorboard]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Wed, 21 Oct 2020 07:02:04 GMT</pubDate>
            <atom:updated>2020-10-30T01:47:34.412Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/machine-learning/deep-learning">Deep Learning</a></h4><h3>Build Floating Movie Recommendations Using Deep Learning — DIY in &lt;10 Mins</h3><h4>Recommendations served on your own Webapp.com, Watch on Netflix.com</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xG7o-gFF2vInVUMg.gif" /><figcaption>A deep-learning-based floating movie recommender</figcaption></figure><p>Have your ever watched inception and wondered about a dream within a dream and the possibility of real-world items floating in space? Well, we can’t float real items in space (at least so far) but can definitely project the python vectors in Multi-dimensional space.</p><p>Follow along and by end of this post, we will have a floating recommender system for movies to your name without any fuss around renting servers, writing flask codes, data scraping, or portal maintenance. Additionally, why just limit ourselves to movies and not extend it to our social network, stocks, sportsperson, TV shows, books, etc. Guess, we are on the same page or at least similar pages when projected in multi-dimensional vector space.</p><p>For those who would want to have a preview of the web-app, <a href="https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/715f5656ae5dc2753f62ce4e9f89e7cb88f4e76e/test2.json">Link here</a></p><p>Major Pre-requisites: Curiosity to learn new concepts and eagerness to deploy their own web-apps.</p><p>Minor Pre-Requisites: Understanding of word2vec embedding, Neural Nets, Tensorflow, Github, and recommender systems.</p><h4><strong>Toolkit/Concepts:</strong></h4><p>1. Embeddings- They are low dimensional space which can project the high dimensional vector. Since an image/word can have millions of parameters as characterized by pixels or tokens, we need a constrained uniform structure which can be an input to our ML models. When high dimensions or a large number of features are modeled to derive a pattern in limited data size, we generally encounter the curse of dimensionality problem i.e the model is unable to extract relevant patterns from input data. We may use techniques like Principle Component Analysis (PCA), TSNE, or L1 regularization to bring down the dimensions. However, for extremely high dimensions and sparse datasets, e.g Millions of tokens in Wikipedia corpus, neural net-based embeddings are more suitable to extract the relevant attributes.</p><p>Embeddings capture the semantic similarity across the tokens or pixels and project them in user-defined vector space. E.g- <em>“Federer”</em> and <em>“Tennis”</em> are mentioned together and thus on a 100-D embedding, Federer would be closer to tennis than Trump.</p><p>Sample diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*miWLQ0PYJ1qyEQ5R" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t3BCs3dsexSrNvAE.png" /><figcaption>Word2Vec vector similarity</figcaption></figure><p>2. Tensorboard Projector- A web application open-sourced by Tensorflow that interactively visualizes embeddings by reading them from our model and rendering them in lower dimensions. Sample projector for Wikipedia and MNIST corpus us available on <a href="https://projector.tensorflow.org/">tensorboard portal</a>. Please spend a good amount of time understanding the intuitive projector, filters, embedding space, and various dimension reduction algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BnZq02prTeQVlX4_.png" /><figcaption>Tensorboard Projector trained on Wikipedia corpus</figcaption></figure><p>3. Dimensionality reduction algorithms- Though we provide a 100 Dimensional embedding to the projector, the human eye can only view items till 3 dimensions and we need to bring down our vector into 2-D or 3-D.</p><p>a. Principal Component Analysis- It is a technique for reducing the dimensionality of big datasets which improves model interpretability while minimizing information loss. The new principal components are created which are uncorrelated with each other with the aim to maximize variance i.e maximum information of large feature data into lower dimensional principal components. Based on the incremental variance explained by principal components e.g 90% of Variance, we can decide on the cutoff.</p><p>b. T-Stochastic Neighbor embedding- It is a non-linear technique for dimensionality reduction and generally used for visualization of high dimensional datasets. Its implementation is based on minimizing the divergence between two distributions: a distribution that measures pairwise similarities of the input objects and a distribution that measures pairwise similarities of the corresponding low-dimensional points in the embedding.</p><p>Major differences between PCA and TSNE are the linearity vs Non-Linearity dimensionality reduction and the objective function i.e PCA trying to preserve the global data structure while TSNE algorithm preserving local structure.</p><p><a href="https://www.geeksforgeeks.org/difference-between-pca-vs-t-sne/">Geeks For Geeks</a> provide an 8 pointer difference between TSNE and PCA and act as a good reference point for our future experimentation.</p><p>4. Movie lens data -Considered as the Gold standard dataset for movie recommender systems, Movie lens data provides [userid-movieid] level data for as big as 26Mn Ratings and 58k movies. Since 1997, Movie lens data has played a prominent role in the evolution of recommender systems ranging from Netflix to Amazon to individual projects. The landslide influence of the Movie lens can be gauged from the fact that <em>“a search for “movielens” produced 2,750 results in Google Books and 7,580 in Google Scholar”</em>.These datasets can be used to build recommender systems using item-based, user-based, SVD matrix, Neural nets, etc with the dependent variable as rating prediction for a new user based on their past preferences.</p><p>Using the concepts above, let us strive to create a floating recommendation web app with a Movies universe in multiples of Netflix e.g 3.7k Netflix movies vs 58k movies in our dataset, Source of data <a href="https://www.foxbusiness.com/technology/how-many-movies-on-netflix">here</a></p><p>Similar to how a word2vec embedding identifies closest words in vector space, we are creating closest movies in 100-dimensional vector space i.e reducing dimensions from 58k to 100.</p><p>The input dataset is sourced from MovieLens which is a transaction data of users giving 0–5 rating to movies. To prevent noise MovieLens has removed ratings from users who have fewer than 20 ratings and inexact demographics. There are multiple variants of data i.e ranging from [1k Users * 1.7k Movies] to [280k Users * 58k Movies], feel free to check out their <a href="https://grouplens.org/datasets/movielens/">data repository</a>. For our user-based Collaborative filtering, we can ignore the quality of ratings and consider the user-movie combination as a binary relationship i.e whether the user has rated or not rated a movie.</p><p>Recollecting the concepts, a Movie recommendation can be of 3 types:</p><p>1. Popularity Based- Identifies the movies watched with the highest watch time in trailing X time period and recommends these movies to all the users.</p><p>2. Item Based- Based on metatags of the movies i.e actor, director, Language, release year, etc, However, the attributes of the items don’t change with time and the recommendations are not refreshed in real time user behavior is not taken into consideration.</p><p>3. User-based- Tags users to a particular cohort/subgroup based on their watching patterns and recommendations are other movies in the same cohort, e.g If I have started with Inception and Dark Knight and those who are on the platform before I have watched Prestige after watching these 2 movies, I will be recommended Prestige during my next visit.</p><p>I have written a detailed blog post a couple of years back on creating recommender systems using item-based collaborative filtering. Please feel free to check it out and provide feedback/comparisons of item-based vs user-based recommendation results.<br>Blog link: <a href="https://towardsdatascience.com/build-your-own-recommendation-engine-netflix-demystified-demo-code-550401d4885e">https://towardsdatascience.com/build-your-own-recommendation-engine-netflix-demystified-demo-code-550401d4885e</a></p><p>For emerging OTT platforms, user-based CF is a quick-start way as they may not have resources to generate meta tags and label movies and instead would prefer the implicit user behavior who interact with their platform. For bigger players like Netflix, Youtube, Prime Video hybrid recommendations are the de-facto approach. For the purpose of this post, we are constraining ourselves to user-based filtering i.e defining a Movie by its user vectors. The data structure would look as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/0*xL64xW28Udr40lrQ.png" /><figcaption>Input Data from MovieLens</figcaption></figure><p>Similar to word2vec embedding, we train this shallow neural net based on User IDs, Once the model is trained we can find the <em>‘Tennis-Federer’ </em>equivalent of user ids. Using the work so far, we can find look-alike users and probably divide users into clusters for targeted communication via Push Notifications/emailers. However more than user targeting, we are interested in movie recommendations which would require an additional step of comparing movies.</p><p>In comes, sentence embedding, where the users would be treated as a sentence, and the semantic similarity of sentences would be captured. There has been a lot of action going in sentence embedding space with the entry of BERT, Elmo, Doc2Vec, and Universal Sentence Encoder. However, speed is of the essence here, and based on my individual benchmarking, I have considered <a href="https://github.com/oborchers/Fast_Sentence_Embeddings">Fast_Sentence_Embeddings</a>(FSE) as a go-to way.FSE creates a composite vector based on individual word2vec vector projections and we are not going to get any value additions by using pre-trained models on Wikipedia or Google News as the user ids are illegible for their SOTA systems. There are no existing semantic similarities to be captured by these models and thus we can safely consider the speed of our algorithm as a predominant deciding factor(FSE claims ~500K sentence processing per second while my personal experience with Universal sentence encoder has seen sub 100/sec speed).</p><p>Once we have sentence vectors, we can project them in multi-dimensional space,e.g “<em>Obama speaks to the media in Illinois”</em> would be similar to <em>“The President greets the press in Chicago”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*bCsq2lt0HXf_MEfb.png" /><figcaption>Context-based sentence similarity</figcaption></figure><p>Code for creating movie embeddings using Movielens data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3b5aa4736018378129aea436ecf9db3/href">https://medium.com/media/b3b5aa4736018378129aea436ecf9db3/href</a></iframe><p>Once we have the movie embeddings, we will be using a combination of tensorboard projector and Github to traverse the last mile of deployment, Special thanks to Google for open sourcing their tensorboard projector.</p><p>Step by step process to creating an amazing portal using embeddings:</p><p>1. Upload the embeddings data to a GitHub gist and update it in the config JSON. For starters, you can freely use my embeddings <a href="https://gist.github.com/ravi07bec/c8b76059933c95e31e8e7f12c858b81b">hosted here</a>.</p><p>2. Upload the metadata on a Github <a href="https://gist.github.com/ravi07bec/b2dfd16c94c1571bb5d70149ace8a532">gist</a> so that the bubbles have a label</p><p>3. Create a JSON file and provide the URL to Tensorflow, Sample<a href="https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/715f5656ae5dc2753f62ce4e9f89e7cb88f4e76e/test2.json"> JSON config</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/0*WhrucwWxzL7n_Nzt.png" /><figcaption>Input Config file location from Github</figcaption></figure><p>4. Deploy the web app and voila!</p><p>Webapps deployed using the above steps:</p><p>a. <a href="https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/715f5656ae5dc2753f62ce4e9f89e7cb88f4e76e/test2.json">Floating recommender without images</a> and b. <a href="https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/915e0dbc1c69b6e9d31100c4967249a7a2548e85/test2.json">Floating recommender with images</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*sfXJL429dKTrIoQz.png" /><figcaption>Webapp and Reccos for Ratatouille</figcaption></figure><p>5. Play around with projector, click on a movie, find closest movies, Project movie vectors using TSNE, or PCA.</p><p><strong>Summary: </strong>We have learned about processing a cross-grid of Movie lens data, concepts behind vector embeddings(User IDs in this case), sentence similarity matrix using FSE, creating a sprite Image, and deploying a DIY projector using Google’s TensorFlow. Snapshot from <a href="https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/ravi07bec/4b033bc6299b655a2abda8b441e82364/raw/915e0dbc1c69b6e9d31100c4967249a7a2548e85/test2.json">final portal</a> below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*DU32DY1Ehyk4QLXn.gif" /><figcaption>Recommender portal with Movie Posters</figcaption></figure><p>I would be happy to see your DIY web apps on movies or related datasets, feel free to hit me up here or Linkedin, Happy to chat if you face any issues/improvements.</p><p>Related datasets which can be used for recommendations:</p><p>1. Books- Kaggle <a href="https://www.kaggle.com/zygmunt/goodbooks-10k">https://www.kaggle.com/zygmunt/goodbooks-10k</a></p><p>2. E-Commerce product purchases-<a href="https://www.kaggle.com/c/instacart-market-basket-analysis">https://www.kaggle.com/c/instacart-market-basket-analysis</a></p><p>3. Restaurants-<a href="https://towardsdatascience.com/how-to-build-a-restaurant-recommendation-system-using-latent-factor-collaborative-filtering-ffe08dd57dca">https://towardsdatascience.com/how-to-build-a-restaurant-recommendation-system-using-latent-factor-collaborative-filtering-ffe08dd57dca</a></p><p>4. Cuisines: <a href="https://github.com/hasan-kamal/Cuisine-Prediction">https://github.com/hasan-kamal/Cuisine-Prediction</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26f585821697" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/build-floating-movie-recommendations-using-deep-learning-diy-in-10-mins-26f585821697">Build Floating Movie Recommendations using Deep Learning — DIY in &lt;10 Mins</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas on Steroids: Dask- End to End Data Science with python code]]></title>
            <link>https://medium.com/analytics-vidhya/pandas-on-steroids-dask-end-to-end-data-science-with-python-code-1845d3722c8a?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/1845d3722c8a</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[parallel-processing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dask]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sun, 18 Oct 2020 23:50:12 GMT</pubDate>
            <atom:updated>2020-10-21T12:27:47.251Z</atom:updated>
            <content:encoded><![CDATA[<h4>End to End Parallelized Data Science from Reading Big Data to Data Manipulation to Visualisation to Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Oro-2xdjFqraa3Mpz28eQg.gif" /><figcaption>Dask- Familiar pandas with superpowers</figcaption></figure><p>As the saying goes, a data scientist spends 90% of their time in cleaning data and 10% in complaining about the data. Their complaints may range from data size, faulty data distributions, Null values, data randomness, systematic errors in data capture, differences between train and test sets and the list just goes on and on.</p><p>One common bottleneck theme is the enormity of data size where either the data doesn’t fit into memory or the processing time is so large(In order of multi-mins) that the inherent pattern analysis goes for a toss. Data scientists by nature are curious human beings who want to identify and interpret patterns normally hidden from cursory Drag-N-Drop glance. They need to wear multiple hats and make the data confess via repeated tortures(read iterations 😂 )</p><p>They wear multiple hats during exploratory data analysis and from a minimal dataset with 6 columns on New York Taxi Fare dataset( <a href="https://www.kaggle.com/c/new-york-city-taxi-fare-prediction">https://www.kaggle.com/c/new-york-city-taxi-fare-prediction</a>) - ID, Fare, Time of Trip, Passengers and Location, their questions may range from:</p><blockquote>1. How the fares have changed Year-Over-Year?</blockquote><blockquote>2. Has the number of trips increased across the years?</blockquote><blockquote>3. Do people prefer traveling alone or they have company?</blockquote><blockquote>4. Has the small distance rides increased as people have become lazier?</blockquote><blockquote>5. What time of the day and day of week do people want to travel?</blockquote><blockquote>6. Is there emergence of new hotspots in the city recently except the regular Air Port pickup and drop?</blockquote><blockquote>7. Are people taking more inter-city trips?</blockquote><blockquote>8. Has the traffic increased leading to more fares/time taken for the same distances?</blockquote><blockquote>9. Are there cluster of pick-up and Drop points or areas which see high traffic?</blockquote><blockquote>10. Are there outliers in data i.e 0 distance and fare of $100+ and so on?</blockquote><blockquote>11. Do the demand change during Holiday season and airport trips increase?</blockquote><blockquote>12. Is there any correlation of weather i.e rain or snow with the taxi demand?</blockquote><p>Even after answering these questions, multiple sub-threads can emerge i.e can we predict how the Covid affected New year is going to be, How the annual NY marathon shifts taxi demand, If a particular route if more prone to have multiple passengers(Party hub) vs Single Passengers( Airport to Suburbs).</p><p>To quench these curiosities, time is of the essence and its criminal to keep the data scientists waiting for 5+ minutes to read a csv file(55 Mn rows) or do a column add followed by aggregation. Also, since the majority of Data Scientists are self-taught and they have been so much used to pandas data frame API that they wouldn’t want to start the learning process all over again with a different API like numba, Spark or datatable. I have tried juggling between DPLYR(R), Pandas(Python) and pyspark(Spark) and it is a bit unfulfilling/unproductive considering the need for a uniform pipeline and code syntax. However, for the curious folks, I have written a pyspark starter guide here: <a href="https://medium.com/@ravishankar_22148/billions-of-rows-milliseconds-of-time-pyspark-starter-guide-c1f984023bf2">https://medium.com/@ravishankar_22148/billions-of-rows-milliseconds-of-time-pyspark-starter-guide-c1f984023bf2</a></p><p>In subsequent sections, I am trying to provide a hands on guide to Dask with minimal architectural change from our beloved Pandas:</p><ol><li>Data Read and Profiling</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9cc38a13e657fcf49b2fb22f49098fff/href">https://medium.com/media/9cc38a13e657fcf49b2fb22f49098fff/href</a></iframe><p>How is Dask able to process data ~90X faster i.e Sub 1 secs to 91 secs in pandas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXtMfGFLYzdcPZuY9z7UcQ.gif" /><figcaption><em>Source: </em><a href="https://dask.org/"><em>https://dask.org/</em></a></figcaption></figure><p>What makes Dask so popular is the fact that it makes analytics scalable in Python and not necessarily need switching back and forth between SQL, Scala and Python.The magical feature is that this tool requires minimum code changes. It breaks down computation into pandas data frames and thus operates in parallel to enable fast calculations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/0*g24z1YXjVkRpjRko.png" /><figcaption><em>source:</em><a href="https://docs.dask.org/en/latest/"><em>https://docs.dask.org/en/latest/</em></a></figcaption></figure><p>2. Data Aggregation:</p><p>With absolutely 0 change from Pandas API, it is able to perform aggregation and sorting in milliseconds, Please note .<strong>compute() </strong>function at the end of lazy computation which brings the results of big data to memory in Pandas Data Frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cdc58075684712c30794896436d591c/href">https://medium.com/media/5cdc58075684712c30794896436d591c/href</a></iframe><p>3. Machine Learning:</p><p>Code snippet below provides a working example of feature engineering and ML model building in Dask using XGBoost</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57cc935c660ae0f4725768d359474088/href">https://medium.com/media/57cc935c660ae0f4725768d359474088/href</a></iframe><p><strong>Conclusion:</strong></p><p>Dask is a powerful tool offering parallel computing, big data handling and creating end to end Data Science pipeline. It has a steep learning curve as the API is almost similar to pandas and it can handle Out Of Memory computations(~10X of RAM) like a breeze.</p><p>Since it is a living blog, I will be writing subsequent parts in Dask series where we will be targeting Kaggle leaderboard using parallel processing. Let me know in comments if you are facing any issues in setting up Dask or unable to perform any Dask Operations or even for a general chit-chat. Happy Learning!!!</p><p>Sources:</p><ol><li><a href="https://ml.dask.org/">https://ml.dask.org/</a></li><li><a href="https://dask.org/">https://dask.org/</a></li><li><a href="https://medium.com/@ravishankar_22148/billions-of-rows-milliseconds-of-time-pyspark-starter-guide-c1f984023bf2">https://medium.com/@ravishankar_22148/billions-of-rows-milliseconds-of-time-pyspark-starter-guide-c1f984023bf2</a></li><li><a href="https://towardsdatascience.com/how-i-learned-to-love-parallelized-applies-with-python-pandas-dask-and-numba-f06b0b367138">https://towardsdatascience.com/how-i-learned-to-love-parallelized-applies-with-python-pandas-dask-and-numba-f06b0b367138</a></li><li><a href="https://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask">https://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask</a></li><li><a href="https://medium.com/better-programming/what-is-dask-and-how-can-it-help-you-as-a-data-scientist-72adec7cec57">https://medium.com/better-programming/what-is-dask-and-how-can-it-help-you-as-a-data-scientist-72adec7cec57</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1845d3722c8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/pandas-on-steroids-dask-end-to-end-data-science-with-python-code-1845d3722c8a">Pandas on Steroids: Dask- End to End Data Science with python code</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A picture is worth a Thousand Words- Lets figure out the relevant ones]]></title>
            <link>https://medium.com/analytics-vidhya/a-picture-is-worth-a-thousand-words-lets-figure-out-the-relevant-ones-15cbb56443ae?source=rss-5de3088925ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/15cbb56443ae</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Ravi Shankar]]></dc:creator>
            <pubDate>Sun, 10 Mar 2019 09:56:24 GMT</pubDate>
            <atom:updated>2020-10-22T13:33:48.652Z</atom:updated>
            <content:encoded><![CDATA[<h4>Teaching Computers to describe pictures</h4><p>Image Captioning refers to the process of generating textual description from an image — based on the objects and actions in the image. This is a 3 part series to implement Image Captioning as presented by Andrej Karapathy<strong> </strong>in his PHD <a href="https://cs.stanford.edu/people/karpathy/main.pdf">thesis paper</a> at Stanford.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L23MWigQNu2bVA_YLN4i-A.png" /><figcaption>Computer Generated Captions using Neural Nets</figcaption></figure><p>In the process, we would learn basics of Neural Network, create a Convolutional Neural Network(CNN) in Keras(wrapper around Tensorflow), explore State of The Art NLP models(Sequence to Sequence, Glove, BERT etc) and stack together CNN and NLP model using LSTM to generate captions of an image.</p><p>We would take it from there and create Recommender systems based on Pre-Trained Image and Captions vector followed by a live WebApp as the testing ground for Caption Generation as well as Recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5IWkE4ZbuS3ryDfUFsDyg.png" /><figcaption>Recommendations based on pattern of first Footwear</figcaption></figure><p>Table of Content(For Part 1):</p><ul><li>Basics Of Neural Networks</li><li>Convolution Neural Network for Image Recognition</li><li>Setting up a Google Colab Notebook</li><li>Creating a Neural Network in Keras for Image Classification</li></ul><p><strong>Neural Network Basics:</strong></p><p>A neural network is a type of machine learning which models itself after the human brain. This creates an artificial neural network via an algorithm which allows the computer to learn by incorporating new data.</p><p>It takes several input, processes it through multiple neurons from multiple hidden layers and returns the result using an output layer. This result estimation process is technically known as “<strong>Forward Propagation</strong>“.</p><p>Next, we compare the result with actual output. The task is to make the output to neural network as close to actual (desired) output. Each of these neurons are contributing some error to final output. How do you reduce the error?</p><p>We try to minimize the value/ weight of neurons those are contributing more to the error and this happens while traveling back to the neurons of the neural network and finding where the error lies. This process is known as “<strong>Backward Propagation</strong>“. Backward Propagation(BP) updates the weights to minimize the error resulting from each neuron.</p><p>In order to reduce these number of iterations to minimize the error, the neural networks use a common algorithm known as <strong>“Gradient Descent”</strong>, which helps to optimize the task quickly and efficiently. More about Gradient descent <a href="https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/">here</a>.</p><p>The aim of multiple <strong>epochs</strong>(Forward and Backward Propagation) is just to optimize the weights and biases of multiple layers so as to minimize the errors.</p><p>Various Categories of Neural Nets:</p><ul><li>Convolutional Neural Network(CNN)</li><li>Recurrent Neural Network(RNN)</li><li>LSTM and GRUs</li></ul><p>Lets delve deep into CNN here and I will cover other categories in subsequent posts.</p><p>Convolutional Neural Network(CNN): They are Used largely for Image processing tasks(Classification, Object Detection, Localization etc) and constitute 4 major operations namely Convolution, Non Linearity, Pooling and Classification as described below:</p><ol><li><strong>Convolution:</strong> The primary purpose of Convolution in case of a ConvNet is to extract features from the input image. Convolution preserves the spatial relationship between pixels by learning image features using small squares of input data As we discussed above, every image can be considered as a matrix of pixel values. Consider a 5 x 5 image whose pixel values are only 0 and 1:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/0*7xHjnE0YCpLOoVqN" /><figcaption>Input image as a matrix</figcaption></figure><p>Also, consider another 3 x 3 matrix as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/74/0*vmNLweEzjTFLVwKL" /><figcaption>Randomly Initialized weight matrix</figcaption></figure><p>Then, the Convolution of the 5 x 5 image and the 3 x 3 matrix can be computed as shown in the animation below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*ZCjPUFrB6eHPRi4eyP6aaA.gif" /><figcaption>Image * Weight Matrix=Convolved Feature</figcaption></figure><p>We slide the orange matrix over our original image (green) by 1 pixel (also called <strong>‘stride’</strong>) and for every position, we compute <a href="https://en.wikipedia.org/wiki/Matrix_multiplication">element wise multiplication</a> (between the two matrices) and add the multiplication outputs to get the final integer which forms a single element of the output matrix (pink). Note that the 3×3 matrix “sees” only a part of the input image in each stride.</p><p>In CNN terminology, the 3×3 matrix is called a ‘<strong>filter</strong>‘ and the matrix formed by sliding the filter over the image and computing the dot product is called the ‘Convolved Feature’ or ‘Activation Map’ or the ‘<strong>Feature Map</strong>‘.</p><p>Convolution operation on an image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Y0tvNnH7sXHw2zP7TrCKgA.gif" /></figure><p>Notice how these two different filters generate different feature maps from the same original image.In practice, a CNN <em>learns</em> the values of these filters on its own during the training process and the more image features get extracted and the better our network becomes at recognizing patterns in unseen images.</p><p><strong>2. Non Linearity(ReLu):</strong></p><p>ReLu stands for Rectified Linear Unit and is a non-linear operation. The purpose of ReLU is to introduce non-linearity in our ConvNet, since most of the real-world data we would want our ConvNet to learn would be non-linear.Its output is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/0*hr5ij1tsgfb4Mz3d" /><figcaption>Relu Operation</figcaption></figure><p>ReLu replaces all negative pixel values in the feature map by zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RYpFlRz3HDz9lcJkbLaBcA.png" /></figure><p><strong>3. Pooling(To Reduce the dimensions of an image):</strong></p><p>Spatial Pooling (also called subsampling or downsampling) reduces the dimensionality of each feature map but retains the most important information. Spatial Pooling can be of different types: Max Pooling, Average Pooling etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/0*9_zro3kc-ccq64LX.JPG" /><figcaption>Max Pooling and Average Pooling</figcaption></figure><h4>4. Fully Connected(Dense) Layer for Classification:</h4><p>The term “Fully Connected” implies that every neuron in the previous layer is connected to every neuron on the next layer.The output from the convolutional and pooling layers represent high-level features of the input image. The purpose of the Fully Connected layer is to use these features for classifying the input image into various classes based on the training dataset.</p><p>We will use Softmax function for final classification<strong>. Softmax function </strong>takes a vector of arbitrary real-valued scores and squashes it to a vector of values between zero and one that sum to one.</p><h4>The overall training process of the Convolution Network may be summarized as below:</h4><p><strong>Step1:</strong> We initialize all filters and parameters / weights with random values</p><p><strong>Step2: </strong>The network takes a training image as input, goes through the forward propagation step (convolution, ReLU and pooling operations along with forward propagation in the Fully Connected layer) and finds the output probabilities for each class.</p><p><strong>Step3:</strong> Calculate the total error at the output layer (summation over all 4 classes)</p><ul><li><strong>Total Error = ∑ ½ (target probability — output probability) ²</strong></li></ul><p><strong>Step4:</strong> Use Backpropagation to calculate the <em>gradients</em> of the error with respect to all weights in the network and use <em>gradient descent</em> to update all filter values / weights and parameter values to minimize the output error.</p><p><strong>Step5:</strong> Repeat steps 2–4 with all images in the training set.</p><p>The above steps <em>train</em> the ConvNet — this essentially means that all the weights and parameters of the ConvNet have now been optimized to correctly classify images from the training set.</p><p>When a new (unseen) image is input into the ConvNet, the network would go through the forward propagation step and output a probability for each class (for a new image, the output probabilities are calculated using the weights which have been optimized to correctly classify all the previous training examples)</p><p><strong>Setting up Google Colab Notebook: </strong>All You need is a Google Drive account</p><p>Colab as <a href="https://colab.research.google.com/notebooks/welcome.ipynb">described</a> by Google research team is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.With Colab you can write and execute code, save and share your analyses, and access powerful computing resources(including GPU), all for free from your browser.</p><p>Lets create a <strong>CNN model</strong> now for<strong> image classification</strong> based on <strong>Keras</strong> in <strong>Google Colab</strong> using <strong>GPU</strong>:</p><ol><li>Go to <a href="https://colab.research.google.com/notebooks/welcome.ipynb">https://colab.research.google.com/notebooks/welcome.ipynb</a></li><li>Click on RunTime-&gt;Change RunTime Type -&gt;Select GPU for GPU activated environment. On my test runs I found GPU at least 10X faster than CPU using similar system configurations.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oxB_YAcSder1sNmcj2png.png" /><figcaption>New Notebook on Google Colab in a GPU powered environment</figcaption></figure><p><strong>Building a CNN model in Keras for image Classification(Good Old </strong><a href="http://yann.lecun.com/exdb/mnist/"><strong>MNIST database</strong></a><strong>):</strong></p><pre>#Keras Model Creation</pre><pre>import keras<br>from keras.models import Sequential<br>from keras.layers import Dense, Conv2D, Flatten<br>#create model<br>model = Sequential()<br>#add model layers</pre><pre>model.add(Conv2D(64, kernel_size=3, activation=&#39;relu&#39;, input_shape=(28,28,1)))<br>model.add(Conv2D(32, kernel_size=3, activation=&#39;relu&#39;))<br><br>model.add(Dense(10, activation=&#39;softmax&#39;))</pre><p>The model type that we will be using is <strong>Sequential</strong>. It allows you to build a model layer by layer.We use the ‘add()’ function to add layers to our model.Our first 2 layers are Conv2D layers. These are convolution layers that will deal with our input images, which are seen as 2-dimensional matrices.‘Dense’ is the layer type we will use in for our output layer. Dense is a standard layer type that is used in many cases for neural networks.We will have 10 nodes in our output layer, one for each possible outcome (0–9 i.e 10 Digits).The activation is ‘softmax’. Softmax makes the output sum up to 1 so the output can be interpreted as probabilities. The model will then make its prediction based on which option has the highest probability.</p><pre># Keras Model Compile</pre><pre>#compile model <em>using accuracy to measure model performance</em><br>model.compile(optimizer=&#39;adam&#39;, loss=&#39;categorical_crossentropy&#39;, metrics=[&#39;accuracy&#39;])</pre><p>Compiling the model takes three parameters: <strong>optimizer, loss and metrics</strong></p><p>The optimizer controls the learning rate. We will be using ‘adam’ as our optimizer since it adjusts the learning rate throughout training.The learning rate determines how fast the optimal weights for the model are calculated. Higher learning rate implies quick convergence and thus under-fitting and low learning rate implies a long time for model fitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*Zzl5hqDpvK0MNDUV.jpeg" /><figcaption>Learning Rate vs Loss</figcaption></figure><p>We will use ‘categorical_crossentropy’ for our <strong>loss function</strong>. This is the most common choice for classification. A lower score indicates that the model is performing better. ‘<strong>Accuracy’</strong> metric is used to see the accuracy score on the validation set when we train the model.</p><pre>#train the model<br>model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=3)</pre><pre>#Make Predictions in the test set<br>model.predict(X_test[:4])<br></pre><p>Based on this, we are able to see a 97% accuracy on our test dataset. My Colab notebook for codes:</p><p><a href="https://colab.research.google.com/drive/1QI_VDLYS_d5fS1QTXqicTxmLs2HkWDs-">Google Colaboratory</a></p><p>In the next Part, we will work on Flickr 30K dataset <a href="http://bryanplummer.com/">http://bryanplummer.com/</a> which has 5 captions associated with each image. We will build an Encoder-Decoder network followed by Supervised Training based on Captions to generate custom captions based on our image inputs.</p><p>Thanks a lot for reading.Happy Learning!!!</p><p>References:</p><ul><li><a href="https://github.com/karpathy/neuraltalk2">https://github.com/karpathy/neuraltalk2</a></li><li><a href="https://towardsdatascience.com/a-deeper-understanding-of-nnets-part-1-cnns-263a6e3ac61">https://towardsdatascience.com/a-deeper-understanding-of-nnets-part-1-cnns-263a6e3ac61</a></li><li><a href="https://www.analyticsvidhya.com/blog/2017/05/neural-network-from-scratch-in-python-and-r/">https://www.analyticsvidhya.com/blog/2017/05/neural-network-from-scratch-in-python-and-r/</a></li><li><a href="https://towardsdatascience.com/image-captioning-in-deep-learning-9cd23fb4d8d2">https://towardsdatascience.com/image-captioning-in-deep-learning-9cd23fb4d8d2</a></li><li><a href="https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/">https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/</a></li><li><a href="https://medium.com/nybles/create-your-first-image-recognition-classifier-using-cnn-keras-and-tensorflow-backend-6eaab98d14dd">https://medium.com/nybles/create-your-first-image-recognition-classifier-using-cnn-keras-and-tensorflow-backend-6eaab98d14dd</a></li><li><a href="https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8">https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8</a></li><li><a href="https://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-keras-329fbbadc5f5">https://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-keras-329fbbadc5f5</a></li><li><a href="https://towardsdatascience.com/creating-a-movie-recommender-using-convolutional-neural-networks-be93e66464a7">https://towardsdatascience.com/creating-a-movie-recommender-using-convolutional-neural-networks-be93e66464a7</a></li><li><a href="https://towardsdatascience.com/getting-started-with-google-colab-f2fff97f594c">https://towardsdatascience.com/getting-started-with-google-colab-f2fff97f594c</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15cbb56443ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-picture-is-worth-a-thousand-words-lets-figure-out-the-relevant-ones-15cbb56443ae">A picture is worth a Thousand Words- Lets figure out the relevant ones</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>