<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sunil Ray on Medium]]></title>
        <description><![CDATA[Stories by Sunil Ray on Medium]]></description>
        <link>https://medium.com/@sunil.udbhav?source=rss-4be29faab438------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mhCM5CHtEP3mxBVLV2gfaQ.png</url>
            <title>Stories by Sunil Ray on Medium</title>
            <link>https://medium.com/@sunil.udbhav?source=rss-4be29faab438------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:55:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sunil.udbhav/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building your First Neural Network on a Structured Dataset (using Keras)]]></title>
            <link>https://medium.com/analytics-vidhya/build-your-first-neural-network-model-on-a-structured-dataset-using-keras-d9e7de5c6724?source=rss-4be29faab438------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9e7de5c6724</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sunil Ray]]></dc:creator>
            <pubDate>Mon, 20 Aug 2018 07:29:38 GMT</pubDate>
            <atom:updated>2019-08-09T08:18:12.905Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>Have you ever applied a <a href="https://courses.analyticsvidhya.com/courses/computer-vision-using-deep-learning-version2?utm_source=medium&amp;utm_medium=FirstNeuralNetworkOnStructuredData">neural network</a> model on a structured dataset? If the answer is no, which of the following reasons are applicable for you?</p><ol><li>It is very complex to apply</li><li>Neural Network is good for unstructured datasets like image, audio, and text and it does not perform well on structured datasets</li><li>It is not as easy as building a model using scikit-learn/caret</li><li>Training time is too high</li><li>Requires high computational power</li></ol><p>In this article, I will focus on the first three reasons and showcase how easily you can apply a neural network model on a structured dataset using a popular high-level library - “keras”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vADxWb9yIYAbfcAQ_oKfiQ.png" /></figure><h3><strong>Understand the problem statement</strong></h3><p>We will work on the <a href="https://datahack.analyticsvidhya.com/contest/black-friday/">Black Friday</a> dataset in this article. It is a regression challenge where we need to predict the purchase amount of a customer against various products. We have been provided with various information about customer demographics (age, gender, marital status, city_type, stay_in_current_city) and product details (product_id and product category). Below is the data dictionary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BU7i3ny6jS44M4MIC7REbQ.png" /></figure><p>The evaluation metric for this challenge is theRoot Mean Squared Error (RMSE).</p><h3><strong>Pre-requisite</strong></h3><p>In this article, I will solve this Black-Friday challenge using a Random Forest (RF) model using scikit-learn and a basic Neural Network (NN) model using keras. The idea of this article is to show how easily we can build an NN model on a structured dataset (it is similar to building a RF model using the scikit-learn library). This article assumes that you have a fair background of building a Machine Learning model, scikit-learn, and the basics of Neural Network. If you are not comfortable with these concepts, I would recommend going through the below articles first:</p><ol><li><a href="https://www.analyticsvidhya.com/blog/2016/03/introduction-deep-learning-fundamentals-neural-networks/">Introduction-deep-learning-fundamentals-neural-networks</a></li><li><a href="https://medium.com/analytics-vidhya/understanding-and-coding-neural-networks-from-scratch-in-python-and-r-b8c760f0ad1c">Understanding-and-coding-neural-networks-from-scratch-in-python</a></li><li><a href="https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/">Common Machine Learning Algorithms</a></li></ol><h3><strong>Approach to solve the problem using Machine (Deep) learning</strong></h3><p>I’m going to separate my approach broadly into four sub-sections:</p><ol><li>Data Preparation</li><li>Model Building (Random Forest and Neural Network)</li><li>Evaluation</li><li>Prediction</li></ol><h4><strong>Data Preparation</strong></h4><p>In this section, I will focus on basic data preparation steps like loading the dataset, imputing missing values, treating categorical variables, normalizing data and creating a validation set. I will follow the same steps for both the Random Forest and our NN model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jXfdWyIAF-nymLFnQEkHNA.png" /></figure><ul><li><strong>Load Data: </strong>Here, I’ll import the necessary libraries to load the dataset, combine train and test to perform preprocessing together, and also create a flag for the same.</li></ul><pre>#Importing Libraries for data preparation<br>import pandas as pd<br>import numpy as np</pre><pre>#Read Necessary files<br>train = pd.read_csv(&quot;train_black_friday.csv&quot;)<br>test = pd.read_csv(&quot;test_black_friday.csv&quot;)</pre><pre>#Combined both Train and Test Data set to do preprocessing together # and set flag for both as well<br>train[&#39;Type&#39;] = &#39;Train&#39; <br>test[&#39;Type&#39;] = &#39;Test&#39;<br>fullData = pd.concat([train,test],axis=0)</pre><ul><li><strong>Impute Missing Values: </strong>Methods to treat missing values for categorical and continuous variables will be different.<br>So, our first step would be identifying the ID column, target variable, categorical and continuous independent variables. <br>Post this, we will create dummy flags for missing value(s). Why do this? Because sometimes missing values themselves can carry a good amount of information. Finally, we will impute the missing values of continuous variables with the mean of that columns, and for the categorical variable, we will create a new level.</li></ul><pre>#Identifying ID, Categorical<br>ID_col = [&#39;User_ID&#39;,&#39;Product_ID&#39;]<br>flag_col= [&#39;Type&#39;]<br>target_col = [&quot;Purchase&quot;]<br>cat_cols= [&#39;Gender&#39;,&#39;Age&#39;,&#39;City_Category&#39;,&#39;Stay_In_Current_City_Years&#39;]<br>num_cols= list(set(list(fullData.columns))-set(cat_cols)-set(ID_col)-set(target_col)-set(flag_col))</pre><pre># Combined numerical and Categorical variables<br>num_cat_cols = num_cols+cat_cols</pre><pre>#Create a new variable for each variable having missing value with VariableName_NA <br># and flag missing value with 1 and other with 0</pre><pre>for var in num_cat_cols:<br>    if fullData[var].isnull().any()==True:<br>        fullData[var+&#39;_NA&#39;]=fullData[var].isnull()*1</pre><pre>#Impute numerical missing values with mean<br>fullData[num_cols] = fullData[num_cols].fillna(fullData[num_cols].mean())</pre><pre>#Impute categorical missing values with -9999<br>fullData[cat_cols] = fullData[cat_cols].fillna(value = -9999)</pre><ul><li><strong>Treat Categorical Values: </strong>We will create a label encoder for categorical variables.</li></ul><pre>#create label encoders for categorical features<br>from sklearn.preprocessing import LabelEncoder</pre><pre>for var in cat_cols:<br>    number = LabelEncoder()<br>    fullData[var] = number.fit_transform(fullData[var].astype(&#39;str&#39;))</pre><ul><li><strong>Normalize Data: </strong>Scale (normalize) the independent variables between 0 and 1. It will help us to converge comparatively faster.</li></ul><pre>features = list(set(list(fullData.columns))-set(ID_col)-set(target_col))<br>fullData[features] = fullData[features]/fullData[features].max()</pre><ul><li><strong>Create a validation Set: </strong>Here, we will segregate the train-test from the full dataset and remove the train-test flag from features list. While building our model, we have target values for the train dataset only, so we will create a validation set out of this train dataset for evaluating the model’s performance. Here, I’m using <em>train_test_split</em> to divide the train dataset in training and validation in a 70:30 ratio.</li></ul><pre>#Creata a validation set<br>from sklearn.model_selection import train_test_split</pre><pre>train=fullData[fullData[&#39;Type&#39;]==1]<br>test=fullData[fullData[&#39;Type&#39;]==0]<br>features=list(set(list(fullData.columns))-set(ID_col)-set(target_col)-set(flag_col))</pre><pre>X = train[features].values<br>y = train[target_col].values</pre><pre>X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.30, random_state=42)</pre><h4><strong>Model Building using Random Forest</strong></h4><p>This part is fairly straightforward and I have written about this multiple times before. If you still want to review the random forest algorithm and its parameters, I would recommend going through this article: <a href="https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/">Tutorial on Tree Based Algorithms</a>.</p><pre>#import necessary libraries to build model</pre><pre>import random<br>from sklearn.ensemble import RandomForestRegressor</pre><pre>random.seed(42)<br>rf = RandomForestRegressor(n_estimators=10)<br>rf.fit(X_train, y_train)</pre><h4>Model Building using Deep Learning Model (Keras)</h4><p>Here, I will focus on the steps to build a basic deep learning model. This will help beginners in creating their own models in the future. The steps to do this are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uaE9vcY1M2-RTCpurKr9lg.png" /></figure><ul><li><strong>Define Model: </strong>For building a deep learning model, we need to define the layers (Input, Hidden, and Output). Here, we will go ahead with a sequential model, which means that we will define layers sequentially. Also, we will be going ahead with a fully connected network.<br><strong>1. </strong>First, we will focus on <strong>defining the input layer</strong>. This can be specified while creating the first layer with the <em>input dim</em> argument and setting it to 11 for the 11 independent variables. <br><strong>2. </strong>Next, <strong>define the number of hidden layer(s)</strong> along with the number of neurons and activation functions. The right number can be achieved by going through multiple iterations. Higher the number, more complex is your model. To start with, I’m simply using two hidden layers. One has 100 neurons and the other has 50 with the same activation function - “relu”.<br><strong>3.</strong> Finally, we need to <strong>define the output layer</strong> with 1 neuron to predict the purchase amount. The problem in hand is a regression challenge so we can go ahead with a linear transformation at the output layer. Therefore, there is no need to mention any activation function (it is linear by default).</li></ul><pre># Define model<br>model = Sequential()<br>model.add(Dense(100, input_dim=11, activation= &quot;relu&quot;))<br>model.add(Dense(50, activation= &quot;relu&quot;))<br>model.add(Dense(1))<br>model.summary() #Print model Summary</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k9-Hmn55Zy3XRVC8dv5SFw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*FSec3nWjTkspxFwoZL0vgg.png" /></figure><ul><li><strong>Compile Model: </strong>At this stage, we will configure the model for training. We will set the optimizer to change the weights and biases, and the loss function and metric to evaluate the model’s performance. Here, we will use “adam” as the optimizer, “mean squared error” as the loss metric. Depending on the type of problem we are solving, we can change our loss and metrics. For binary classification, we use “binary-crossentropy” as a loss function.</li></ul><pre># Compile model<br>model.compile(loss= &quot;mean_squared_error&quot; , optimizer=&quot;adam&quot;, metrics=[&quot;mean_squared_error&quot;])</pre><ul><li><strong>Fit Model: </strong>Now, the final step of model building is fitting the model on the training dataset (which is actually 70% of the full dataset). We need to provide both independent and dependent variables along with the number of training iterations, i.e. epochs. Here, we have taken 10 epochs.</li></ul><pre># Fit Model<br>model.fit(X_train, y_train, epochs=10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWf4OM-auu3yIiQZCVeHfA.png" /></figure><h4><strong>Evaluation</strong></h4><p>Now that we have built the model using Random Forest and Neural Network techniques, the next step is to evaluate the performance on the validation dataset for both the models.</p><ul><li><strong>Evaluation for Random Forest Model: </strong>We will get the predictions on the validation dataset and do the evaluation with actual target values (y_valid). We get the root mean squared error as ~3106.</li></ul><pre>from sklearn.metrics import mean_squared_error<br>pred=rf.predict(X_valid)<br>score = np.sqrt(mean_squared_error(y_valid,pred))<br>print (score)<br>3106.5008973291074</pre><pre># Evaluation while fitting the model<br>model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))</pre><ul><li><strong>Evaluation for Neural Network Model: </strong>Similarly, we will get the predictions on the validation dataset using the neural network model and calculate the root mean squared error. RMSE with a basic NN model comes out to be ~4214. This is a fairly basic model, you can go ahead and tune the hyper-parameters to build a more complex network. <em>You can pass validation data as an argument while fitting the NN model as well to look at the validation score after each epoch.</em></li></ul><pre>pred= model.predict(X_valid)<br>score = np.sqrt(mean_squared_error(y_valid,pred))<br>print (score)<br>4213.954523194906</pre><h4>Prediction</h4><p>After evaluating the model and finalizing the model parameters, we can go ahead with the prediction on the test data. Below is the code to do this using both random forest and NN models.</p><pre>#Select the independent variables for test dataset<br>X_test = test[features].values</pre><pre>#Prediction using Random Forest <br>y_test_rf = rf.predict(X_test)</pre><pre>#Prediction using Neural Network<br>y_test_nn = model.predict(X_test)</pre><h3><strong>What’s Next</strong></h3><p>The idea of this article was to show how easily we can build an NN model on a structured dataset so we have not focused on other aspects of improving the model’s predictions. Below is my list of ideas which you can apply to build on the neural network:</p><ul><li>Impute missing values after looking at the variable to variable relationships</li><li>Feature Engineering (Product Ids may have some information about the purchase amount)</li><li>Select right hyper-parameters</li><li>Build a more complex network by adding more hidden layers</li><li>Use regularization</li><li>Train for more number of epochs</li><li>Take Ensemble of both RF with NN model</li></ul><p><strong>Summary</strong></p><p>In this article, we have discussed the different stages of model building like data preparation, model building, evaluation and finally prediction. We also looked at how we can apply a neural network model on a structured dataset using keras.</p><p>We further learned about different steps of model building in keras like defining, compiling, and fitting the model. Performance of a simple NN model was impressive and we can improve on that after making it into a more complex network. In the next article, we will look at how we can apply a RNN model on a time series challenge.</p><p>If you have any questions or suggestions, please feel free to connect with me in the comments section below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9e7de5c6724" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/build-your-first-neural-network-model-on-a-structured-dataset-using-keras-d9e7de5c6724">Building your First Neural Network on a Structured Dataset (using Keras)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding and coding Neural Networks From Scratch in Python and R]]></title>
            <link>https://medium.com/analytics-vidhya/understanding-and-coding-neural-networks-from-scratch-in-python-and-r-b8c760f0ad1c?source=rss-4be29faab438------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8c760f0ad1c</guid>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Sunil Ray]]></dc:creator>
            <pubDate>Sun, 28 May 2017 20:51:04 GMT</pubDate>
            <atom:updated>2018-08-16T11:48:07.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/713/0*KYQyUaMOWpfhQDzj.jpg" /></figure><h3>Introduction</h3><p>You can learn and practice a concept in two ways:</p><ul><li><strong>Option 1:</strong> You can learn the entire theory on a particular subject and then look for ways to apply those concepts. So, you read up how an entire algorithm works, the maths behind it, its assumptions, limitations and then you apply it. Robust but time taking approach.</li><li><strong>Option 2:</strong> Start with simple basics and develop an intuition on the subject. Next, pick a problem and start solving it. Learn the concepts while you are solving the problem. Keep tweaking and improving your understanding. So, you read up how to apply an algorithm — go out and apply it. Once you know how to apply it, try it around with different parameters, values, limits and develop an understanding of the algorithm.</li></ul><p>I prefer Option 2 and take that approach to learning any new topic. I might not be able to tell you the entire math behind an algorithm, but I can tell you the intuition. I can tell you the best scenarios to apply an algorithm based on my experiments and understanding.</p><p>In my interactions with people, I find that people don’t take time to develop this intuition and hence they struggle to apply things in the right manner.</p><p>In this article, I will discuss the building block of a neural network from scratch and focus more on developing this intuition to apply Neural networks. We will code in both “Python” and “R”. By end of this article, you will understand how Neural networks work, how do we initialize weigths and how do we update them using back-propagation.</p><p>Let’s start.</p><h3>Table of Contents:</h3><ol><li>Simple intuition behind Neural networks</li><li>Multi Layer Perceptron and its basics</li><li>Steps involved in Neural Network methodology</li><li>Visualizing steps for Neural Network working methodology</li><li>Implementing NN using Numpy (Python)</li><li>Implementing NN using R</li><li>[Optional] Mathematical Perspective of Back Propagation Algorithm</li></ol><h3>Simple intuition behind neural networks</h3><p>If you have been a developer or seen one work — you know how it is to search for bugs in a code. You would fire various test cases by varying the inputs or circumstances and look for the output. The change in output provides you a hint on where to look for the bug — which module to check, which lines to read. Once you find it, you make the changes and the exercise continues until you have the right code / application.</p><p>Neural networks work in very similar manner. It takes several input, processes it through multiple neurons from multiple hidden layers and returns the result using an output layer. This result estimation process is technically known as “<strong>Forward Propagation</strong>“.</p><p>Next, we compare the result with actual output. The task is to make the output to neural network as close to actual (desired) output. Each of these neurons are contributing some error to final output. How do you reduce the error?</p><p>We try to minimize the value/ weight of neurons those are contributing more to the error and this happens while traveling back to the neurons of the neural network and finding where the error lies. This process is known as “<strong>Backward Propagation</strong>“.</p><p>In order to reduce these number of iterations to minimize the error, the neural networks use a common algorithm known as “Gradient Descent”, which helps to optimize the task quickly and efficiently.</p><p>That’s it — this is how Neural network works! I know this is a very simple representation, but it would help you understand things in a simple manner.</p><h3>Multi Layer Perceptron and its basics</h3><p>Just like atoms form the basics of any material on earth — the basic forming unit of a neural network is a perceptron. So, what is a perceptron?</p><p>A perceptron can be understood as anything that takes multiple inputs and produces one output. For example, look at the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/0*G1oRGW6Dej3Xkh-n.png" /><figcaption>Perceptron</figcaption></figure><p>The above structure takes three inputs and produces one output. The next logical question is what is the relationship between input and output? Let us start with basic ways and build on to find more complex ways.</p><p>Below, I have discussed three ways of creating input output relationships:</p><ol><li><strong>By directly combining the input and computing the output</strong> based on a threshold value. for eg: Take x1=0, x2=1, x3=1 and setting a threshold =0. So, if x1+x2+x3&gt;0, the output is 1 otherwise 0. You can see that in this case, the perceptron calculates the output as 1.</li><li><strong>Next, let us add weights to the inputs.</strong> Weights give importance to an input. For example, you assign w1=2, w2=3 and w3=4 to x1, x2 and x3 respectively. To compute the output, we will multiply input with respective weights and compare with threshold value as w1*x1 + w2*x2 + w3*x3 &gt; threshold. These weights assign more importance to x3 in comparison to x1 and x2.</li><li><strong>Next, let us add bias: </strong>Each perceptron also has a bias which can be thought of as how much flexible the perceptron is. It is somehow similar to the constant <em>b</em> of a linear function <em>y = ax + b. It allows us to move the line up and down to fit the prediction with the data better. Without b the line will always goes through the origin (0, 0) and you may get a poorer fit.</em> For example, a perceptron may have two inputs, in that case, it requires three weights. One for each input and one for the bias. Now linear representation of input will look like, w1*x1 + w2*x2 + w3*x3 + 1*b.</li></ol><p>But, all of this is still linear which is what perceptrons used to be. But that was not as much fun. So, people thought of evolving a perceptron to what is now called as artificial neuron. A neuron applies non-linear transformations (activation function) to the inputs and biases.</p><h3>What is an activation function?</h3><p>Activation Function takes the sum of weighted input (w1*x1 + w2*x2 + w3*x3 + 1*b) as an argument and return the output of the neuron.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/0*QrkYzzhhsY9mWKB0.png" /></figure><p>In above equation, we have represented 1 as x0 and b as w0.</p><p>The activation function is mostly used to make a non-linear transformation which allows us to fit nonlinear hypotheses or to estimate the complex functions. There are multiple activation functions, like: “Sigmoid”, “Tanh”, ReLu and many other.</p><h3>Forward Propagation, Back Propagation and Epochs</h3><p>Till now, we have computed the output and this process is known as “<strong>Forward Propagation</strong>“. But what if the estimated output is far away from the actual output (high error). In the neural network what we do, we update the biases and weights based on the error. This weight and bias updating process is known as “<strong>Back Propagation</strong>“.</p><p>Back-propagation (BP) algorithms work by determining the loss (or error) at the output and then propagating it back into the network. The weights are updated to minimize the error resulting from each neuron. The first step in minimizing the error is to determine the gradient (Derivatives) of each node w.r.t. the final output. To get a mathematical perspective of the Backward propagation, refer below section.</p><p>This one round of forward and back propagation iteration is known as one training iteration aka “<strong>Epoch</strong>“.</p><h3>Multi-layer perceptron</h3><p>Now, let’s move on to next part of <strong>Multi-Layer</strong> Perceptron. So far, we have seen just a single layer consisting of 3 input nodes i.e x1, x2 and x3 and an output layer consisting of a single neuron. But, for practical purposes, the single-layer network can do only so much. An MLP consists of multiple layers called <strong>Hidden Layers </strong>stacked in between the <strong>Input Layer</strong> and the <strong>Output Layer</strong> as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xnzyeyCmmIwG4NAS.png" /></figure><p>The image above shows just a single hidden layer in green but in practice can contain multiple hidden layers. Another point to remember in case of an MLP is that all the layers are fully connected i.e every node in a layer(except the input and the output layer) is connected to every node in the previous layer and the following layer.</p><p>Let’s move on to the next topic which is training algorithm for a neural network (to minimize the error). Here, we will look at most common training algorithm known as <a href="https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/">Gradient descent</a>.</p><h3>Full Batch Gradient Descent and Stochastic Gradient Descent</h3><p>Both variants of Gradient Descent perform the same work of updating the weights of the MLP by using the same updating algorithm but the difference lies in the number of training samples used to update the weights and biases.</p><p>Full Batch Gradient Descent Algorithm as the name implies uses all the training data points to update each of the weights once whereas Stochastic Gradient uses 1 or more(sample) but never the entire training data to update the weights once.</p><p>Let us understand this with a simple example of a dataset of 10 data points with two weights <strong>w1</strong> and <strong>w2</strong>.</p><p><strong>Full Batch:</strong> You use 10 data points (entire training data) and calculate the change in w1 (Δw1) and change in w2(Δw2) and update w1 and w2.</p><p><strong>SGD:</strong> You use 1st data point and calculate the change in w1 (Δw1) and change in w2(Δw2) and update w1 and w2. Next, when you use 2nd data point, you will work on the updated weights</p><p>For a more in-depth explanation of both the methods, you can have a look at <a href="https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/">this article</a>.</p><h3>Steps involved in Neural Network methodology</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0Ql4RWwCxmEl3P4R.png" /></figure><p>Let’s look at the step by step building methodology of Neural Network (MLP with one hidden layer, similar to above-shown architecture). At the output layer, we have only one neuron as we are solving a binary classification problem (predict 0 or 1). We could also have two neurons for predicting each of both classes.</p><p>First look at the broad steps:</p><p>0.) We take input and output</p><ul><li>X as an input matrix</li><li>y as an output matrix</li></ul><p>1.) We initialize weights and biases with random values (This is one time initiation. In the next iteration, we will use updated weights, and biases). Let us define:</p><ul><li>wh as weight matrix to the hidden layer</li><li>bh as bias matrix to the hidden layer</li><li>wout as weight matrix to the output layer</li><li>bout as bias matrix to the output layer</li></ul><p>2.) We take matrix dot product of input and weights assigned to edges between the input and hidden layer then add biases of the hidden layer neurons to respective inputs, this is known as linear transformation:</p><p>hidden_layer_input= matrix_dot_product(X,wh) + bh</p><p>3) Perform non-linear transformation using an activation function (Sigmoid). Sigmoid will return the output as 1/(1 + exp(-x)).</p><p>hiddenlayer_activations = sigmoid(hidden_layer_input)</p><p>4.) Perform a linear transformation on hidden layer activation (take matrix dot product with weights and add a bias of the output layer neuron) then apply an activation function (again used sigmoid, but you can use any other activation function depending upon your task) to predict the output</p><p>output_layer_input = matrix_dot_product (hiddenlayer_activations * wout ) + bout<br> <em>output = sigmoid(output_layer_input)</em></p><p><strong>All above steps are known as “Forward Propagation“</strong></p><p>5.) Compare prediction with actual output and calculate the gradient of error (Actual-Predicted). Error is the mean square loss = ((Y-t)²)/2</p><p>E = y-output</p><p>6.) Compute the slope/ gradient of hidden and output layer neurons ( To compute the slope, we calculate the derivatives of non-linear activations x at each layer for each neuron). Gradient of sigmoid can be returned as x * (1-x).</p><p>slope_output_layer = derivatives_sigmoid(output)<br> <em>slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)</em></p><p>7.) Compute change factor(delta) at output layer, dependent on the gradient of error multiplied by the slope of output layer activation</p><p><em>d_output = E * slope_output_layer</em></p><p>8.) At this step, the error will propagate back into the network which means error at hidden layer. For this, we will take the dot product of output layer delta with weight parameters of edges between the hidden and output layer (wout.T).</p><p>Error_at_hidden_layer = matrix_dot_product(d_output, wout.Transpose)</p><p>9.) Compute change factor(delta) at hidden layer, multiply the error at hidden layer with slope of hidden layer activation</p><p><em>d_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer</em></p><p>10.) Update weights at the output and hidden layer: The weights in the network can be updated from the errors calculated for training example(s).</p><p>wout = wout + matrix_dot_product(hiddenlayer_activations.Transpose, d_output)*learning_rate<br> wh = wh + matrix_dot_product(X.Transpose,d_hiddenlayer)*learning_rate</p><p>learning_rate: The amount that weights are updated is controlled by a configuration parameter called the learning rate)</p><p>11.) Update biases at the output and hidden layer: The biases in the network can be updated from the aggregated errors at that neuron.</p><ul><li>bias at output_layer =bias at output_layer + sum of delta of output_layer at row-wise * learning_rate</li><li>bias at hidden_layer =bias at hidden_layer + sum of delta of output_layer at row-wise * learning_rate</li></ul><p><em>bh = bh + sum(d_hiddenlayer, axis=0) * learning_rate<br> bout = bout + sum(d_output, axis=0)*learning_rate</em></p><p><strong>Steps from 5 to 11 are known as “Backward Propagation“</strong></p><p>One forward and backward propagation iteration is considered as one training cycle. As I mentioned earlier, When do we train second time then update weights and biases are used for forward propagation.</p><p>Above, we have updated the weight and biases for hidden and output layer and we have used full batch gradient descent algorithm.</p><h3>Visualization of steps for Neural Network methodology</h3><p>We will repeat the above steps and visualize the input, weights, biases, output, error matrix to understand working methodology of Neural Network (MLP).</p><p>Note:</p><ul><li>For good visualization images, I have rounded decimal positions at 2 or3 positions.</li><li>Yellow filled cells represent current active cell</li><li>Orange cell represents the input used to populate values of current cell</li></ul><p><strong>Step 0: </strong>Read input and output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/0*8Ncz80a91PjS4urO.jpg" /></figure><p><strong>Step 1: </strong>Initialize weights and biases with random values (There are methods to initialize weights and biases but for now initialize with random values)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lT1WZp2cXYX_mkNc.png" /></figure><p><strong>Step 2:</strong> Calculate hidden layer input:<br> hidden_layer_input= matrix_dot_product(X,wh) + bh</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8b_lonk18YIokEv2.png" /></figure><p><strong>Step 3: </strong>Perform non-linear transformation on hidden linear input<br> <em>hiddenlayer_activations = sigmoid(hidden_layer_input)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vo01YKO5FtQ_CCiQ.png" /></figure><p><strong>Step 4: </strong>Perform linear and non-linear transformation of hidden layer activation at output layer</p><p>output_layer_input = matrix_dot_product (hiddenlayer_activations * wout ) + bout<br> <em>output = sigmoid(output_layer_input)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n9RWK6DTL8OHjxUT.png" /></figure><p><strong>Step 5: </strong>Calculate gradient of Error(E) at output layer<br> <em>E = y-output</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K_cIwOw8ZKR1kUct.png" /></figure><p><strong>Step 6: </strong>Compute slope at output and hidden layer<br> <em>Slope_output_layer= derivatives_sigmoid(output)</em><br> <em>Slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6cEMKW0VwOpvFi94.png" /></figure><p><strong>Step 7: </strong>Compute delta at output layer</p><p><em>d_output = E * slope_output_layer*lr</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1hg4ukxZaIaMEsZQ.png" /></figure><p><strong>Step 8: </strong>Calculate Error at hidden layer</p><p>Error_at_hidden_layer = matrix_dot_product(d_output, wout.Transpose)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bw7NKPalf6YAA14B.png" /></figure><p><strong>Step 9: </strong>Compute delta at hidden layer</p><p><em>d_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hNvFGYn3Q8FKnafV.png" /></figure><p><strong>Step 10: </strong>Update weight at both output and hidden layer</p><p>wout = wout + matrix_dot_product(hiddenlayer_activations.Transpose, d_output)*learning_rate<br> wh = wh+ matrix_dot_product(X.Transpose,d_hiddenlayer)*learning_rate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ltEpBZLqri4NqfM_.png" /></figure><p><strong>Step 11: </strong>Update biases at both output and hidden layer</p><p><em>bh = bh + sum(d_hiddenlayer, axis=0) * learning_rate<br> bout = bout + sum(d_output, axis=0)*learning_rate</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JTsv3FoTsjPNv6Zp.png" /></figure><p>Above, you can see that there is still a good error not close to actual target value because we have completed only one training iteration. If we will train model multiple times then it will be a very close actual outcome. I have completed thousands iteration and my result is close to actual target values ([[ 0.98032096] [ 0.96845624] [ 0.04532167]]).</p><h3>Implementing NN using Numpy (Python)</h3><pre>import numpy as np</pre><pre>#Input array<br>X=np.array([[1,0,1,0],[1,0,1,1],[0,1,0,1]])</pre><pre>#Output<br>y=np.array([[1],[1],[0]])</pre><pre>#Sigmoid Function<br>def sigmoid (x):<br>    return 1/(1 + np.exp(-x))</pre><pre>#Derivative of Sigmoid Function<br>def derivatives_sigmoid(x):<br>    return x * (1 - x)</pre><pre>#Variable initialization<br>epoch=5000 #Setting training iterations<br>lr=0.1 #Setting learning rate<br>inputlayer_neurons = X.shape[1] #number of features in data set<br>hiddenlayer_neurons = 3 #number of hidden layers neurons<br>output_neurons = 1 #number of neurons at output layer</pre><pre>#weight and bias initialization<br>wh=np.random.uniform(size=(inputlayer_neurons,hiddenlayer_neurons))<br>bh=np.random.uniform(size=(1,hiddenlayer_neurons))<br>wout=np.random.uniform(size=(hiddenlayer_neurons,output_neurons))<br>bout=np.random.uniform(size=(1,output_neurons))</pre><pre>for i in range(epoch):<br>    <br>    #Forward Propogation<br>    hidden_layer_input1=np.dot(X,wh)<br>    hidden_layer_input=hidden_layer_input1 + bh<br>    hiddenlayer_activations = sigmoid(hidden_layer_input)<br>    output_layer_input1=np.dot(hiddenlayer_activations,wout)<br>    output_layer_input= output_layer_input1+ bout<br>    output = sigmoid(output_layer_input)</pre><pre>    #Backpropagation<br>    E = y-output<br>    slope_output_layer = derivatives_sigmoid(output)<br>    slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)<br>    d_output = E * slope_output_layer<br>    Error_at_hidden_layer = d_output.dot(wout.T)<br>    d_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer<br>    wout += hiddenlayer_activations.T.dot(d_output) *lr<br>    bout += np.sum(d_output, axis=0,keepdims=True) *lr<br>    wh += X.T.dot(d_hiddenlayer) *lr<br>    bh += np.sum(d_hiddenlayer, axis=0,keepdims=True) *lr</pre><pre>print output</pre><h3>Implementing NN in R</h3><p># input matrix<br> X=matrix(c(1,0,1,0,1,0,1,1,0,1,0,1),nrow = 3, ncol=4,byrow = TRUE)</p><p># output matrix<br> Y=matrix(c(1,1,0),byrow=FALSE)</p><p>#sigmoid function<br> sigmoid&lt;-function(x){<br> 1/(1+exp(-x))<br> }</p><p># derivative of sigmoid function<br> derivatives_sigmoid&lt;-function(x){<br> x*(1-x)<br> }</p><p># variable initialization<br> epoch=5000<br> lr=0.1<br> inputlayer_neurons=ncol(X)<br> hiddenlayer_neurons=3<br> output_neurons=1</p><p>#weight and bias initialization<br>wh=matrix( rnorm(inputlayer_neurons*hiddenlayer_neurons,mean=0,sd=1), inputlayer_neurons, hiddenlayer_neurons)<br>bias_in=runif(hiddenlayer_neurons)<br>bias_in_temp=rep(bias_in, nrow(X))<br>bh=matrix(bias_in_temp, nrow = nrow(X), byrow = FALSE)<br>wout=matrix( rnorm(hiddenlayer_neurons*output_neurons,mean=0,sd=1), hiddenlayer_neurons, output_neurons)</p><p>bias_out=runif(output_neurons)<br>bias_out_temp=rep(bias_out,nrow(X))<br>bout=matrix(bias_out_temp,nrow = nrow(X),byrow = FALSE)<br># forward propagation<br>for(i in 1:epoch){</p><p>hidden_layer_input1= X%*%wh<br>hidden_layer_input=hidden_layer_input1+bh<br>hidden_layer_activations=sigmoid(hidden_layer_input)<br>output_layer_input1=hidden_layer_activations%*%wout<br>output_layer_input=output_layer_input1+bout<br>output= sigmoid(output_layer_input)</p><p># Back Propagation</p><p>E=Y-output<br>slope_output_layer=derivatives_sigmoid(output)<br>slope_hidden_layer=derivatives_sigmoid(hidden_layer_activations)<br>d_output=E*slope_output_layer<br>Error_at_hidden_layer=d_output%*%t(wout)<br>d_hiddenlayer=Error_at_hidden_layer*slope_hidden_layer<br>wout= wout + (t(hidden_layer_activations)%*%d_output)*lr<br>bout= bout+rowSums(d_output)*lr<br>wh = wh +(t(X)%*%d_hiddenlayer)*lr<br>bh = bh + rowSums(d_hiddenlayer)*lr</p><p>}<br> output</p><h3>[Optional] Mathematical Perspective of Back Propagation Algorithm</h3><p>Let Wi be the weights between the input layer and the hidden layer. Wh be the weights between the hidden layer and the output layer.</p><p>Now, <strong>h=σ (u)= σ (WiX)</strong>, i.e h is a function of u and u is a function of Wi and X. here we represent our function as <strong>σ</strong></p><p><strong>Y= σ (u’)= σ (</strong>W<strong>h</strong>h<strong>)</strong>, i.e Y is a function of u’ and u’ is a function of Wh and h.</p><p>We will be constantly referencing the above equations to calculate partial derivatives.</p><p>We are primarily interested in finding two terms, ∂E/∂Wi and ∂E/∂Wh i.e change in Error on changing the weights between the input and the hidden layer and change in error on changing the weights between the hidden layer and the output layer.</p><p>But to calculate both these partial derivatives, we will need to use the chain rule of partial differentiation since E is a function of Y and Y is a function of u’ and u’ is a function of Wi.</p><p>Let’s put this property to good use and calculate the gradients.</p><p>∂E/∂Wh = (∂E/∂Y).( ∂Y/∂u’).( ∂u’/∂Wh), ……..(1)</p><p>We know E is of the form E=(Y-t)2/2.</p><p>So, (∂E/∂Y)= (Y-t)</p><p>Now, σ is a sigmoid function and has an interesting differentiation of the form σ(1- σ). I urge the readers to work this out on their side for verification.</p><p>So, (∂Y/∂u’)= ∂( σ(u’)/ ∂u’= σ(u’)(1- σ(u’)).</p><p>But, σ(u’)=Y, So,</p><p>(∂Y/∂u’)=Y(1-Y)</p><p>Now, ( ∂u’/∂Wh)= ∂( Whh)/ ∂Wh = h</p><p>Replacing the values in equation (1) we get,</p><p><strong>∂E/∂Wh = (Y-t). Y(1-Y).h</strong></p><p>So, now we have computed the gradient between the hidden layer and the ouput layer. It is time we calculate the gradient between the input layer and the hidden layer.</p><p>∂E/∂Wi =(∂ E/∂ h). (∂h/∂u).( ∂u/∂Wi)</p><p>But, (∂ E/∂ h) = (∂E/∂Y).( ∂Y/∂u’).( ∂u’/∂h). Replacing this value in the above equation we get,</p><p>∂E/∂Wi =[(∂E/∂Y).( ∂Y/∂u’).( ∂u’/∂h)]. (∂h/∂u).( ∂u/∂Wi)……………(2)</p><p>So, What was the benefit of first calculating the gradient between the hidden layer and the output layer?</p><p>As you can see in equation (2) we have already computed ∂E/∂Y and ∂Y/∂u’ saving us space and computation time. We will come to know in a while why is this algorithm called the back propagation algorithm.</p><p>Let us compute the unknown derivatives in equation (2).</p><p>∂u’/∂h = ∂(Whh)/ ∂h = Wh</p><p>∂h/∂u = ∂( σ(u)/ ∂u= σ(u)(1- σ(u))</p><p>But, σ(u)=h, So,</p><p>(∂Y/∂u)=h(1-h)</p><p>Now, ∂u/∂Wi = ∂(WiX)/ ∂Wi = X</p><p>Replacing all these values in equation (2) we get,</p><p><strong>∂E/∂Wi = [(Y-t). Y(1-Y).Wh].h(1-h).X</strong></p><p>So, now since we have calculated both the gradients, the weights can be updated as</p><p><strong>Wh = Wh + η . ∂E/∂Wh</strong></p><p><strong>Wi = Wi + η . ∂E/∂Wi</strong></p><p>Where <strong>η </strong>is the learning rate.</p><p>So coming back to the question: Why is this algorithm called Back Propagation Algorithm?</p><p>The reason is: If you notice the final form of <strong>∂E/∂Wh </strong>and <strong>∂E/∂Wi </strong>, you will see the term (Y-t) i.e the output error, which is what we started with and then propagated this back to the input layer for weight updation.</p><p>So, where does this mathematics fit into the code?</p><p>hiddenlayer_activations=h</p><p>E= Y-t</p><p>Slope_output_layer = Y(1-Y)</p><p>lr = η</p><p>slope_hidden_layer = h(1-h)</p><p>wout = Wh</p><p>Now, you can easily relate the code to the mathematics.</p><h3>End Notes:</h3><p>This article is focused on the building a Neural Network from scratch and understanding its basic concepts. I hope now you understand the working of a neural network like how does forward and backward propagation work, optimization algorithms (Full Batch and Stochastic gradient descent), how to update weights and biases, visualization of each step in Excel and on top of that code in python and R.</p><p>Therefore, in my upcoming article, I’ll explain the applications of using Neural Network in Python and solving real-life challenges related to:</p><ol><li>Computer Vision</li><li>Speech</li><li>Natural Language Processing</li></ol><p>I enjoyed writing this article and would love to learn from your feedback. Did you find this article useful? I would appreciate your suggestions/feedback. Please feel free to ask your questions through comments below.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2017/05/neural-network-from-scratch-in-python-and-r/"><em>www.analyticsvidhya.com</em></a><em> on May 28, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8c760f0ad1c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/understanding-and-coding-neural-networks-from-scratch-in-python-and-r-b8c760f0ad1c">Understanding and coding Neural Networks From Scratch in Python and R</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Guide to Data Exploration]]></title>
            <link>https://medium.com/analytics-vidhya/a-comprehensive-guide-to-data-exploration-d5919167bf6e?source=rss-4be29faab438------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5919167bf6e</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-exploration]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Sunil Ray]]></dc:creator>
            <pubDate>Sun, 10 Jan 2016 19:49:12 GMT</pubDate>
            <atom:updated>2019-08-12T04:01:17.765Z</atom:updated>
            <content:encoded><![CDATA[<ul><li>A complete tutorial on data exploration (EDA)</li><li>We cover several data exploration aspects, including missing value imputation, outlier removal and the art of feature engineering</li></ul><h3>Introduction</h3><p>There are no shortcuts for data exploration. If you are in a state of mind, that machine learning can sail you away from every data storm, trust me, it won’t. After some point of time, you’ll realize that you are struggling at improving model’s accuracy. In such situation, data exploration techniques will come to your rescue.</p><p>I can confidently say this, because I’ve been through such situations, a lot.</p><p>I have been a Business Analytics professional for close to three years now. In my initial days, one of my mentor suggested me to spend significant time on exploration and analyzing data. Following his advice has served me well.</p><p>I’ve created this tutorial to help you understand the underlying techniques of data exploration. As always, I’ve tried my best to explain these concepts in the simplest manner. For better understanding, I’ve taken up few examples to demonstrate the complicated concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*0q2waT5fkoW0cm76.jpg" /></figure><h3>Table of Contents</h3><ol><li><a href="https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/#one"><strong>Steps of Data Exploration and Preparation</strong></a></li><li><a href="https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/#two"><strong>Missing Value Treatment</strong></a></li></ol><ul><li>Why missing value treatment is required ?</li><li>Why data has missing values?</li><li>Which are the methods to treat missing value ?</li></ul><p><a href="https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/#three"><strong>3. Techniques of Outlier Detection and Treatment</strong></a></p><ul><li>What is an outlier?</li><li>What are the types of outliers ?</li><li>What are the causes of outliers ?</li><li>What is the impact of outliers on dataset ?</li><li>How to detect outlier ?</li><li>How to remove outlier ?</li></ul><p><a href="https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/#four"><strong>4. The Art of Feature Engineering</strong></a></p><ul><li>What is Feature Engineering ?</li><li>What is the process of Feature Engineering ?</li><li>What is Variable Transformation ?</li><li>When should we use variable transformation ?</li><li>What are the common methods of variable transformation ?</li><li>What is feature variable creation and its benefits ?</li></ul><h3>Let’s get started</h3><h3>1. Steps of Data Exploration and Preparation</h3><p>Remember the quality of your inputs decide the quality of your output. So, once you have got your business hypothesis ready, it makes sense to spend lot of time and efforts here. With my personal estimate, data exploration, cleaning and preparation can take up to 70% of your total project time.</p><p>Below are the steps involved to understand, clean and prepare your data for building your predictive model:</p><ol><li>Variable Identification</li><li>Univariate Analysis</li><li>Bi-variate Analysis</li><li>Missing values treatment</li><li>Outlier treatment</li><li>Variable transformation</li><li>Variable creation</li></ol><p>Finally, we will need to iterate over steps 4–7 multiple times before we come up with our refined model.</p><p>Let’s now study each stage in detail:-</p><h3>Variable Identification</h3><p>First, identify <strong>Predictor</strong> (Input) and <strong>Target</strong> (output) variables. Next, identify the data type and category of the variables.</p><p>Let’s understand this step more clearly by taking an example.</p><p>Example:- Suppose, we want to predict, whether the students will play cricket or not (refer below data set). Here you need to identify predictor variables, target variable, data type of variables and category of variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*UzkmKZYPNg-GybGf.png" /></figure><p>Below, the variables have been defined in different category:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*2PP5vrmRVo0RIcBC.png" /></figure><h3>Univariate Analysis</h3><p>At this stage, we explore variables one by one. Method to perform uni-variate analysis will depend on whether the variable type is categorical or continuous. Let’s look at these methods and statistical measures for categorical and continuous variables individually:</p><p><strong>Continuous Variables:-</strong> In case of continuous variables, we need to understand the central tendency and spread of the variable. These are measured using various statistical metrics visualization methods as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*CQLib8vwD2Ow9oVj.png" /></figure><p><strong>Note:</strong> Univariate analysis is also used to highlight missing and outlier values. In the upcoming part of this series, we will look at methods to handle missing and outlier values. To know more about these methods, you can refer course <a href="https://www.udacity.com/course/ud827">descriptive statistics from Udacity</a>.</p><p><strong>Categorical Variables:- </strong>For categorical variables, we’ll use frequency table to understand distribution of each category. We can also read as percentage of values under each category. It can be be measured using two metrics, <strong>Count</strong> and <strong>Count%</strong> against each category. Bar chart can be used as visualization.</p><h3>Bi-variate Analysis</h3><p>Bi-variate Analysis finds out the relationship between two variables. Here, we look for association and disassociation between variables at a pre-defined significance level. We can perform bi-variate analysis for any combination of categorical and continuous variables. The combination can be: Categorical &amp; Categorical, Categorical &amp; Continuous and Continuous &amp; Continuous. Different methods are used to tackle these combinations during analysis process.</p><p>Let’s understand the possible combinations in detail:</p><p><strong>Continuous &amp; Continuous: </strong>While doing bi-variate analysis between two continuous variables, we should look at scatter plot. It is a nifty way to find out the relationship between two variables. The pattern of scatter plot indicates the relationship between variables. The relationship can be linear or non-linear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*ei0Fehx5R3YI1bJM.png" /></figure><p>Scatter plot shows the relationship between two variable but does not indicates the strength of relationship amongst them. To find the strength of the relationship, we use Correlation. Correlation varies between -1 and +1.</p><ul><li>-1: perfect negative linear correlation</li><li>+1:perfect positive linear correlation and</li><li>0: No correlation</li></ul><p>Correlation can be derived using following formula:</p><p><strong>Correlation = Covariance(X,Y) / SQRT( Var(X)* Var(Y))</strong></p><p>Various tools have function or functionality to identify correlation between variables. In Excel, function CORREL() is used to return the correlation between two variables and SAS uses procedure PROC CORR to identify the correlation. These function returns Pearson Correlation value to identify the relationship between two variables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/0*12SbYK-32_oTt82g.png" /></figure><p>In above example, we have good positive relationship(0.65) between two variables X and Y.</p><p><strong>Categorical &amp; Categorical: </strong>To find the relationship between two categorical variables, we can use following methods:</p><ul><li><strong>Two-way table:</strong> We can start analyzing the relationship by creating a two-way table of count and count%. The rows represents the category of one variable and the columns represent the categories of the other variable. We show count or count% of observations available in each combination of row and column categories.</li><li><strong>Stacked Column Chart: </strong>This method is more of a visual form of Two-way table.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hX8SRUwRP1F5sK1h.gif" /></figure><p>Probability of 0: It indicates that both categorical variable are dependent</p><p>Probability of 1: It shows that both variables are independent.</p><p>Probability less than 0.05: It indicates that the relationship between the variables is significant at 95% confidence. The chi-square test statistic for a test of independence of two categorical variables is found by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/0*h7zeIEIpBYhRjIbx.png" /></figure><p>where <em>O</em> represents the observed frequency. <em>E</em> is the expected frequency under the null hypothesis and computed by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/0*_b9J54Duoq75vQOG.png" /></figure><p>From previous two-way table, the expected count for product category 1 to be of small size is 0.22. It is derived by taking the row total for Size (9) times the column total for Product category (2) then dividing by the sample size (81). This is procedure is conducted for each cell. Statistical Measures used to analyze the power of relationship are:</p><ul><li>Cramer’s V for Nominal Categorical Variable</li><li>Mantel-Haenszed Chi-Square for ordinal categorical variable.</li></ul><p>Different data science language and tools have specific methods to perform chi-square test. In SAS, we can use <strong>Chisq</strong> as an option with <strong>Proc freq</strong> to perform this test.</p><p><strong>Categorical &amp; Continuous: </strong>While exploring relation between categorical and continuous variables, we can draw box plots for each level of categorical variables. If levels are small in number, it will not show the statistical significance. To look at the statistical significance we can perform Z-test, T-test or ANOVA.</p><ul><li><strong>Z-Test/ T-Test:-</strong> Either test assess whether mean of two groups are statistically different from each other or not.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/0*mqZSjlcqqbzGSW3y.jpg" /></figure><ul><li>If the probability of Z is small then the difference of two averages is more significant. The T-test is very similar to Z-test but it is used when number of observation for both categories is less than 30.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*TM3ekP-38ZJSiuQr.png" /></figure><ul><li><strong>ANOVA:- </strong>It assesses whether the average of more than two groups is statistically different.</li></ul><p><strong>Example:</strong> Suppose, we want to test the effect of five different exercises. For this, we recruit 20 men and assign one type of exercise to 4 men (5 groups). Their weights are recorded after a few weeks. We need to find out whether the effect of these exercises on them is significantly different or not. This can be done by comparing the weights of the 5 groups of 4 men each.</p><p>Till here, we have understood the first three stages of Data Exploration, Variable Identification, Uni-Variate and Bi-Variate analysis. We also looked at various statistical and visual methods to identify the relationship between variables.</p><p>Now, we will look at the methods of Missing values Treatment. More importantly, we will also look at why missing values occur in our data and why treating them is necessary.</p><h3>2. Missing Value Treatment</h3><h3>Why missing values treatment is required?</h3><p>Missing data in the training data set can reduce the power / fit of a model or can lead to a biased model because we have not analysed the behavior and relationship with other variables correctly. It can lead to wrong prediction or classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*17mFOvRzRqDhi0T2.png" /></figure><p>Notice the missing values in the image shown above: In the left scenario, we have not treated missing values. The inference from this data set is that the chances of playing cricket by males is higher than females. On the other hand, if you look at the second table, which shows data after treatment of missing values (based on gender), we can see that females have higher chances of playing cricket compared to males.</p><h3>Why my data has missing values?</h3><p>We looked at the importance of treatment of missing values in a dataset. Now, let’s identify the reasons for occurrence of these missing values. They may occur at two stages:</p><ol><li><strong>Data Extraction</strong>: It is possible that there are problems with extraction process. In such cases, we should double-check for correct data with data guardians. Some hashing procedures can also be used to make sure data extraction is correct. Errors at data extraction stage are typically easy to find and can be corrected easily as well.</li><li><strong>Data collection</strong>: These errors occur at time of data collection and are harder to correct. They can be categorized in four types:</li></ol><ul><li><strong>Missing completely at random:</strong> This is a case when the probability of missing variable is same for all observations. For example: respondents of data collection process decide that they will declare their earning after tossing a fair coin. If an head occurs, respondent declares his / her earnings &amp; vice versa. Here each observation has equal chance of missing value.</li><li><strong>Missing at random: </strong>This is a case when variable is missing at random and missing ratio varies for different values / level of other input variables. For example: We are collecting data for age and female has higher missing value compare to male.</li><li><strong>Missing that depends on unobserved predictors:</strong> This is a case when the missing values are not random and are related to the unobserved input variable. For example: In a medical study, if a particular diagnostic causes discomfort, then there is higher chance of drop out from the study. This missing value is not at random unless we have included “discomfort” as an input variable for all patients.</li><li><strong>Missing that depends on the missing value itself: </strong>This is a case when the probability of missing value is directly correlated with missing value itself. For example: People with higher or lower income are likely to provide non-response to their earning.</li></ul><h3>Which are the methods to treat missing values ?</h3><ol><li><strong>Deletion: </strong>It is of two types: List Wise Deletion and Pair Wise Deletion.</li></ol><ul><li>In list wise deletion, we delete observations where any of the variable is missing. Simplicity is one of the major advantage of this method, but this method reduces the power of model because it reduces the sample size.</li><li>In pair wise deletion, we perform analysis with all cases in which the variables of interest are present. Advantage of this method is, it keeps as many cases available for analysis. One of the disadvantage of this method, it uses different sample size for different variables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*nkp1kgBsl2QXNhss.png" /></figure><ul><li>Deletion methods are used when the nature of missing data is “<strong>Missing completely at random</strong>” else non random missing values can bias the model output.</li></ul><p><strong>2. Mean/ Mode/ Median Imputation: </strong>Imputation is a method to fill in the missing values with estimated ones. The objective is to employ known relationships that can be identified in the valid values of the data set to assist in estimating the missing values. Mean / Mode / Median imputation is one of the most frequently used methods. It consists of replacing the missing data for a given attribute by the mean or median (quantitative attribute) or mode (qualitative attribute) of all known values of that variable. It can be of two types:-</p><ul><li><strong>Generalized Imputation:</strong> In this case, we calculate the mean or median for all non missing values of that variable then replace missing value with mean or median. Like in above table, variable “<strong>Manpower”</strong> is missing so we take average of all non missing values of “<strong>Manpower”</strong> (<strong>28.33</strong>) and then replace missing value with it.</li><li><strong>Similar case Imputation:</strong> In this case, we calculate average for gender “<strong>Male” </strong>(29.75) and “<strong>Female</strong>” (25) individually of non missing values then replace the missing value based on gender. For “<strong>Male</strong>”, we will replace missing values of manpower with 29.75 and for “<strong>Female</strong>” with 25.</li></ul><p><strong>3. Prediction Model</strong>: Prediction model is one of the sophisticated method for handling missing data. Here, we create a predictive model to estimate values that will substitute the missing data. In this case, we divide our data set into two sets: One set with no missing values for the variable and another one with missing values. First data set become training data set of the model while second data set with missing values is test data set and variable with missing values is treated as target variable. Next, we create a model to predict target variable based on other attributes of the training data set and populate missing values of test data set.We can use regression, ANOVA, Logistic regression and various modeling technique to perform this. There are 2 drawbacks for this approach:</p><ol><li>The model estimated values are usually more well-behaved than the true values</li><li>If there are no relationships with attributes in the data set and the attribute with missing values, then the model will not be precise for estimating missing values.</li><li><strong>KNN Imputation:</strong> In this method of imputation, the missing values of an attribute are imputed using the given number of attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function. It is also known to have certain advantage &amp; disadvantages.</li></ol><p><strong>Advantages:</strong></p><ul><li>k-nearest neighbour can predict both qualitative &amp; quantitative attributes</li><li>Creation of predictive model for each attribute with missing data is not required</li><li>Attributes with multiple missing values can be easily treated</li><li>Correlation structure of the data is taken into consideration</li></ul><p><strong>Disadvantage:</strong></p><ul><li>KNN algorithm is very time-consuming in analyzing large database. It searches through all the dataset looking for the most similar instances.</li><li>Choice of k-value is very critical. Higher value of k would include attributes which are significantly different from what we need whereas lower value of k implies missing out of significant attributes.</li></ul><p>After dealing with missing values, the next task is to deal with outliers. Often, we tend to neglect outliers while building models. This is a discouraging practice. Outliers tend to make your data skewed and reduces accuracy. Let’s learn more about outlier treatment.</p><h3>3. Techniques of Outlier Detection and Treatment</h3><h3>What is an Outlier?</h3><p>Outlier is a commonly used terminology by analysts and data scientists as it needs close attention else it can result in wildly wrong estimations. Simply speaking, Outlier is an observation that appears far away and diverges from an overall pattern in a sample.</p><p>Let’s take an example, we do customer profiling and find out that the average annual income of customers is $0.8 million. But, there are two customers having annual income of $4 and $4.2 million. These two customers annual income is much higher than rest of the population. These two observations will be seen as Outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*_KQB7lXdVoXgpnw9.png" /></figure><h3>What are the types of Outliers?</h3><p>Outlier can be of two types: <strong>Univariate</strong> and <strong>Multivariate</strong>. Above, we have discussed the example of univariate outlier. These outliers can be found when we look at distribution of a single variable. Multi-variate outliers are outliers in an n-dimensional space. In order to find them, you have to look at distributions in multi-dimensions.</p><p>Let us understand this with an example. Let us say we are understanding the relationship between height and weight. Below, we have univariate and bivariate distribution for Height, Weight. Take a look at the box plot. We do not have any outlier (above and below 1.5*IQR, most common method). Now look at the scatter plot. Here, we have two values below and one above the average in a specific segment of weight and height.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JjpVmrOJzR5vZCjZ.png" /></figure><h3>What causes Outliers?</h3><p>Whenever we come across outliers, the ideal way to tackle them is to find out the reason of having these outliers. The method to deal with them would then depend on the reason of their occurrence. Causes of outliers can be classified in two broad categories:</p><p>Let’s understand various types of outliers in more detail:</p><ul><li><strong>Data Entry Errors:-</strong> Human errors such as errors caused during data collection, recording, or entry can cause outliers in data. For example: Annual income of a customer is $100,000. Accidentally, the data entry operator puts an additional zero in the figure. Now the income becomes $1,000,000 which is 10 times higher. Evidently, this will be the outlier value when compared with rest of the population.</li><li><strong>Measurement Error: </strong>It is the most common source of outliers. This is caused when the measurement instrument used turns out to be faulty. For example: There are 10 weighing machines. 9 of them are correct, 1 is faulty. Weight measured by people on the faulty machine will be higher / lower than the rest of people in the group. The weights measured on faulty machine can lead to outliers.</li><li><strong>Experimental Error:</strong> Another cause of outliers is experimental error. For example: In a 100m sprint of 7 runners, one runner missed out on concentrating on the ‘Go’ call which caused him to start late. Hence, this caused the runner’s run time to be more than other runners. His total run time can be an outlier.</li><li><strong>Intentional Outlier<em>: </em></strong>This is commonly found in self-reported measures that involves sensitive data. For example: Teens would typically under report the amount of alcohol that they consume. Only a fraction of them would report actual value. Here actual values might look like outliers because rest of the teens are under reporting the consumption.</li><li><strong>Data Processing Error: </strong>Whenever we perform data mining, we extract data from multiple sources. It is possible that some manipulation or extraction errors may lead to outliers in the dataset.</li><li><strong>Sampling error:</strong> For instance, we have to measure the height of athletes. By mistake, we include a few basketball players in the sample. This inclusion is likely to cause outliers in the dataset.</li><li><strong>Natural Outlier: </strong>When an outlier is not artificial (due to error), it is a natural outlier. For instance: In my last assignment with one of the renowned insurance company, I noticed that the performance of top 50 financial advisors was far higher than rest of the population. Surprisingly, it was not due to any error. Hence, whenever we perform any data mining activity with advisors, we used to treat this segment separately.</li></ul><h3>What is the impact of Outliers on a dataset?</h3><p>Outliers can drastically change the results of the data analysis and statistical modeling. There are numerous unfavourable impacts of outliers in the data set:</p><ul><li>It increases the error variance and reduces the power of statistical tests</li><li>If the outliers are non-randomly distributed, they can decrease normality</li><li>They can bias or influence estimates that may be of substantive interest</li><li>They can also impact the basic assumption of Regression, ANOVA and other statistical model assumptions.</li></ul><p>To understand the impact deeply, let’s take an example to check what happens to a data set with and without outliers in the data set.</p><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*DaHoYChABYUQ6zHu.png" /></figure><p>As you can see, data set with outliers has significantly different mean and standard deviation. In the first scenario, we will say that average is 5.45. But with the outlier, average soars to 30. This would change the estimate completely.</p><h3>How to detect Outliers?</h3><p>Most commonly used method to detect outliers is visualization. We use various visualization methods, like <strong>Box-plot</strong>, <strong>Histogram</strong>, <strong>Scatter Plot</strong> (above, we have used box plot and scatter plot for visualization). Some analysts also various thumb rules to detect outliers. Some of them are:</p><ul><li>Any value, which is beyond the range of -1.5 x IQR to 1.5 x IQR</li><li>Use capping methods. Any value which out of range of 5th and 95th percentile can be considered as outlier</li><li>Data points, three or more standard deviation away from mean are considered outlier</li><li>Outlier detection is merely a special case of the examination of data for influential data points and it also depends on the business understanding</li><li>Bivariate and multivariate outliers are typically measured using either an index of influence or leverage, or distance. Popular indices such as Mahalanobis’ distance and Cook’s <em>D</em> are frequently used to detect outliers.</li><li>In SAS, we can use PROC Univariate, PROC SGPLOT. To identify outliers and influential observation, we also look at statistical measure like STUDENT, COOKD, RSTUDENT and others.</li></ul><h3>How to remove Outliers?</h3><p>Most of the ways to deal with outliers are similar to the methods of missing values like deleting observations, transforming them, binning them, treat them as a separate group, imputing values and other statistical methods. Here, we will discuss the common techniques used to deal with outliers:</p><p><strong>Deleting observations: </strong>We delete outlier values if it is due to data entry error, data processing error or outlier observations are very small in numbers. We can also use trimming at both ends to remove outliers.</p><p><strong>Transforming and binning values: </strong>Transforming variables can also eliminate outliers. Natural log of a value reduces the variation caused by extreme values. Binning is also a form of variable transformation. Decision Tree algorithm allows to deal with outliers well due to binning of variable. We can also use the process of assigning weights to different observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*9IBB677W9zADZUR7.png" /></figure><p><strong>Imputing: </strong>Like <a href="https://www.analyticsvidhya.com/blog/2015/02/7-steps-data-exploration-preparation-building-model-part-2/">imputation of missing values</a>, we can also impute outliers. We can use mean, median, mode imputation methods. Before imputing values, we should analyse if it is natural outlier or artificial. If it is artificial, we can go with imputing values. We can also use statistical model to predict values of outlier observation and after that we can impute it with predicted values.</p><p><strong>Treat separately: </strong>If there are significant number of outliers, we should treat them separately in the statistical model. One of the approach is to treat both groups as two different groups and build individual model for both groups and then combine the output.</p><p>Till here, we have learnt about steps of data exploration, missing value treatment and techniques of outlier detection and treatment. These 3 stages will make your raw data better in terms of information availability and accuracy. Let’s now proceed to the final stage of data exploration. It is Feature Engineering.</p><h3>4. The Art of Feature Engineering</h3><h3>What is Feature Engineering?</h3><p>Feature engineering is the science (and art) of extracting more information from existing data. You are not adding any new data here, but you are actually making the data you already have more useful.</p><p>For example, let’s say you are trying to predict foot fall in a shopping mall based on dates. If you try and use the dates directly, you may not be able to extract meaningful insights from the data. This is because the foot fall is less affected by the day of the month than it is by the day of the week. Now this information about day of week is implicit in your data. You need to bring it out to make your model better.</p><p>This exercising of bringing out information from data in known as feature engineering.</p><h3>What is the process of Feature Engineering ?</h3><p>You perform feature engineering once you have completed the first 5 steps in data exploration — <a href="https://www.analyticsvidhya.com/blog/2015/02/data-exploration-preparation-model/">Variable Identification, Univariate, Bivariate Analysis</a>, <a href="https://www.analyticsvidhya.com/blog/2015/02/7-steps-data-exploration-preparation-building-model-part-2/">Missing Values Imputation</a> and <a href="https://www.analyticsvidhya.com/blog/2015/02/outliers-detection-treatment-dataset/">Outliers Treatment</a>. Feature engineering itself can be divided in 2 steps:</p><ul><li>Variable transformation.</li><li>Variable / Feature creation.</li></ul><p>These two techniques are vital in data exploration and have a remarkable impact on the power of prediction. Let’s understand each of this step in more details.</p><h3>What is Variable Transformation?</h3><p>In data modelling, transformation refers to the replacement of a variable by a function. For instance, replacing a variable x by the square / cube root or logarithm x is a transformation. In other words, transformation is a process that changes the distribution or relationship of a variable with others.</p><p>Let’s look at the situations when variable transformation is useful.</p><h3>When should we use Variable Transformation?</h3><p>Below are the situations where variable transformation is a requisite:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*dPHGi5Z8Rm05NXQF.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*ix-hrPbhIMolRoRe.png" /></figure><h3>What are the common methods of Variable Transformation?</h3><p>There are various methods used to transform variables. As discussed, some of them include square root, cube root, logarithmic, binning, reciprocal and many others. Let’s look at these methods in detail by highlighting the pros and cons of these transformation methods.</p><h3>What is Feature / Variable Creation &amp; its Benefits?</h3><p>Feature / Variable creation is a process to generate a new variables / features based on existing variable(s). For example, say, we have date(dd-mm-yy) as an input variable in a data set. We can generate new variables like day, month, year, week, weekday that may have better relationship with target variable. This step is used to highlight the hidden relationship in a variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/0*J_HymY8I9E87GueH.png" /></figure><p>There are various techniques to create new features. Let’s look at the some of the commonly used methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/0*9VY6u9xcEHvX6kLU.png" /></figure><h3>End Notes</h3><p>As mentioned in the beginning, quality and efforts invested in data exploration differentiates a good model from a bad model.</p><p>This ends our guide on data exploration and preparation. In this comprehensive guide, we looked at the seven steps of data exploration in detail. The aim of this series was to provide an in depth and step by step guide to an extremely important process in data science.</p><p>Personally, I enjoyed writing this guide and would love to learn from your feedback. Did you find this guide useful? I would appreciate your suggestions/feedback. Please feel free to ask your questions through comments below.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/"><em>https://www.analyticsvidhya.com</em></a><em> on January 10, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5919167bf6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-comprehensive-guide-to-data-exploration-d5919167bf6e">A Comprehensive Guide to Data Exploration</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Ultimate Tips and Tricks on Data Visualization in QlikView]]></title>
            <link>https://medium.com/analytics-vidhya/10-ultimate-tips-and-tricks-on-data-visualization-in-qlikview-6fc7f186676e?source=rss-4be29faab438------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fc7f186676e</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Sunil Ray]]></dc:creator>
            <pubDate>Wed, 09 Dec 2015 19:45:37 GMT</pubDate>
            <atom:updated>2019-08-12T04:07:11.580Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>QlikView is a popular and simple to learn tool for data visualization. Its simple interface makes it a favorite among newbies in analytics. I loved it too. In fact, my journey in business intelligence began with QlikView. As from a non-programming background, I couldn’t have asked for a better start.</p><p>Over the years, I’ve discovered some ways of becoming productive at QlikView. I found these ways after spending endless hours working on this tool. Finally, I ended up finding these 10 tricks which can help you work faster. Without these tricks, I was almost spending 15 minutes in a task which can now be done in less than 3 mins.</p><p>In this article, I’ve shared these 10 ultimate tips &amp; tricks which can save your time and increase your productivity while working on this tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*slqjGI3Vyr_VIv8u.jpg" /></figure><h3>List of Tips and Tricks Covered</h3><ol><li>Copy Objects</li><li>Copy Expression</li><li>Move chart components</li><li>Select multiple objects</li><li>Limit dimensions of a chart</li><li>Adding Visual elements to table</li><li>Conditionally show or hide objects</li><li>Use comments in scripting</li><li>Use tabs in scripting</li><li>Keyboard Shortcuts</li></ol><h3>Copy Objects</h3><p>We often create objects which are similar to one of existing objects with minor changes in dimension, expression or any other parameter(s). It’s a time taking process.</p><p>But, you can do it much faster now. Simply, copy the existing object and make the necessary changes. You can copy sheet and chart object both. Method is as follows:</p><ul><li><strong>Chart object:</strong> Right click on the chart title and then select command “<strong>Clone</strong>”. This will create a copy of chart objects. Alternative method is to click on chart title by holding <strong>CTRL</strong> key and drag the object to different location and release mouse button.</li><li><strong>Sheet object: </strong>Right click on sheet tab and select command “<strong>Copy Sheet</strong>”. This will create a similar sheet with name as “Copy of &lt;current sheet name&gt;”. This new sheet will be created at the end of all sheets.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*ihkUorRGvilqmoha.png" /></figure><h3>Copy Expression</h3><p>During dashboard development, we write multiple expressions (simple and complex). There are situations when we end up writing similar expressions for different chart objects. To make it efficient, we can re-use our existing expressions. Let’s look at the methods:</p><ul><li>Open expression. Copy and paste it into a new expression</li><li>Right click on expression. Select “<strong>Copy</strong>” command. Then, paste it to expression window. Here, you do not only copy the expression code. The new column that is created has exactly the same properties as the source expression. This includes column width, alignment, number format etc.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*mt5DncGy-gHspvyp.png" /></figure><ul><li>In dashboard development, there are some expressions which are common to different chart objects. In this case, we can store the expression in a variable and access it with variable name using a <strong>dollar expansion</strong>. To do this, first store expression in a variable (Go to Settings -&gt; Variable Overview and store expression).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*SX1nFvfEON8MKsJE.png" /></figure><ul><li>To access it in any chart objects, write expression as $(Variable_Name).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*qsS3hqQcrzWgodpz.png" /></figure><ul><li>This improves the accuracy and similarity across the dashboard as we are using same expression.</li></ul><h3>Move chart components</h3><p>By default, position of chart components are fixed at the right hand side. But, in some cases we are required to change the position for better visualization. For which, there is no direct option available. I too didn’t have any solution of this issue initially. But, with time I found out a way. This can be done by using combination of keyboard shortcuts. Here is the trick:</p><ul><li>Let’s say, we want to move the chart legend to left for below pie chart</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*osdIpiZTAqfJCshf.png" /></figure><ul><li>Now, select the chart and press Ctrl+Shift. You will see a red border around every chart object</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/0*5qQ8A4nYTaQIaH7d.png" /></figure><ul><li>Now, drag and drop the individual component to left position (or required position).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*dY_nxjoRKqeAHhS0.png" /></figure><ul><li>Above, you have seen that chart legends moved to left side. You can also re-size the individual component using double headed arrow.</li></ul><h3>Select Multiple Objects</h3><p>Sometimes we want to perform similar operation such as changing font size, layout, deletion with multiple objects. In such cases, we select all objects and perform the required operation. For selection, initially, we select one object then press <strong>SHIFT</strong> key. Then, click on the title of other objects. Now, you can perform group operation (applicable) on these objects like delete, using properties option (right click on any selected object) change the layout, font and caption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/0*t6Rnbrfpc4vDwmUS.png" /></figure><h3>Adding Visual elements to Table</h3><p>Usually, we show only numbers in tabular representation. However, we can make the table more informative by adding visual elements to table. In Qlikview, we can apply three visualization techniques to table:</p><h3>Limit Dimension of a Chart</h3><p>Look at the below chart, it has too many dimensions making it difficult to make inferences about data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*J2yB_nfs4ApFkm48.png" /></figure><p>We can solve this challenge by:</p><ul><li><strong>Limiting Dimension</strong>: In properties window of chart object, you can find a for “<strong>Dimension Limits</strong>”, where you can restrict dimension based on value such as show only largest 10, first 10, smallest 10, show values greater than or equals to 5% of total, show values those accumulate to 80% of total.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*kEgJt796kywI--wD.png" /></figure><ul><li><strong>Enabling scroll bar for dimension: </strong>This option is available on Presentation tab of chart properties window. You can enable scroll bar on x-axis and set limit for number of items to show (look at below snapshot). This is one of the less known useful feature of Qlikview. Here, you can also show top 10, bottom 10 by sorting the chart based on expression in descending or ascending order.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*kE_JEU_ZYOrwrwrK.png" /></figure><h3>Conditionally Show or Hide Objects</h3><p>We can show or hide Qlikview objects based on the requirement. It helps to optimize the area of sheet object effectively. Under layout tab of properties window, there is an option to show or hide the sheet object (always or conditionally). Here you can write expression or use variable to control it. Generally, developers use “Buttons” to change the value of variable such as one button assigns value 0 to variable and another assigns 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/0*mCx_CYY2w4Rg2uUi.png" /></figure><p>We can also show or hide dimension and expression of chart object and best use of this feature is “ <strong>Adhoc reporting</strong>” or “ <strong>Customized reporting</strong>”. You can refer article “ <a href="https://www.analyticsvidhya.com/blog/2014/01/customize-reporting-qlikview/">Customized reporting in Qlikview</a> “.</p><h3>Use comments in Qlikview scripting:</h3><p>Great programmers always add comments with different segment of code. It makes easier to read and debug the code. In Qlikview scripting also, this practice is highly recommended. There are several ways to create comments in your script.</p><ul><li>// is for single line comments</li><li>/* …*/ is for a block edit</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/0*VheHz2CMtTV5k6pv.png" /></figure><h3>Use script tabs in Qlikview scripting</h3><p>To keep scripts maintainable, we do follow modular structure in coding. In similar line, we should use scripts tab to standardize our scripts. It helps in faster script editing and debugging. Qlikview scripts are run top to bottom on each tab, from the left most tab to the right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/0*JaGYQEb9DyuONFhr.png" /></figure><p>Above, you can that different tabs for XY Example, FilmsDatabase, Rating and Dictionary.</p><h3>Keyboard Shortcuts</h3><p>Before checking these shortcuts, first turn on “ <strong>Show Shortcut keys in ScreenTips</strong> “ option. With this, it will show shortcut keys whenever we hover the mouse pointer over menu icons. Just like Microsoft Excel does. This helps a lot in memorizing keyboard shortcuts.</p><p>To enable it, right click on toolbar area -&gt; select command “Customize” -&gt; Go to Options tab -&gt;Check box on for “Show shortcut keys in ScreenTips”. Below are some useful keyboard shortcuts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/0*W7qRMnp2rFG1G16m.png" /></figure><h3>End Notes</h3><p>Not only these tips will make you work faster. But, you’ll also get more time to explore new things at your end. Visualizing data is equally important as analyzing data. Hence, you should strive to become better day by day in data visualization.</p><p>In this article, we looked at QlikView tips and tricks which will enhance your dashboard development experience. Here, I have mostly covered tips and tricks related associated with front end or designer point of view like copying objects &amp; expression, work with chart components, selecting multiple objects, limit dimension, visualization with table, conditionally show or hide objects, scripting best practices and finally keyboard shortcuts.</p><p>Did you find this article useful ? Do you want to add some more tips and tricks ? Feel free to share your experience with us, I’d love to hear you.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2015/12/10-tips-tricks-data-visualization-qlikview/"><em>https://www.analyticsvidhya.com</em></a><em> on December 9, 2015.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fc7f186676e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/10-ultimate-tips-and-tricks-on-data-visualization-in-qlikview-6fc7f186676e">10 Ultimate Tips and Tricks on Data Visualization in QlikView</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use of variables in QlikView to create powerful data stories]]></title>
            <link>https://medium.com/analytics-vidhya/use-of-variables-in-qlikview-to-create-powerful-data-stories-438842870870?source=rss-4be29faab438------2</link>
            <guid isPermaLink="false">https://medium.com/p/438842870870</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Sunil Ray]]></dc:creator>
            <pubDate>Thu, 18 Jun 2015 21:26:28 GMT</pubDate>
            <atom:updated>2019-08-12T04:08:11.308Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>An application with good Front-end and poor Back-end is like Beauty without brains. You are awed by it initially, but you get irritated by it over time. On the other hand, efficient Back-end with poor Front-end may not excite users to use the application. In today’s world, you need excellence on both the aspects.</p><p>Sadly, a lot of people associate data visualization with only front end engineering. In today’s article, I want to bring out what does Back end engineering look like in case of QlikView and how can it add tremendous power to your data stories.</p><h3>The Natural progression in a programming language</h3><p>If you have learnt coding on any language, you would have experienced this progression yourself.</p><p>The <strong>first stage of learning a language</strong> is to understand its syntax, ways to print output, doing mathematical calculations etc. At this stage, you typically don’t do any thing complex. This usually isn’t very exciting or helpful. After all you don’t learn a programming language to calculate 2 + 3!</p><p>The <strong>second stage of learning a language</strong> is when you start using variables to accomplish some of the more complex tasks. With use of variables, you can create generic logic, which are more complicated and useful. So it you want to compare 2 columns with a million numbers each, you can’t do that unless you use variables (May be you can, not me!). The beauty of the process is that once you learn use of variables, it is impossible to think that there was even a stage 1 involved in the learning process!</p><p>There are some more stages in learning a new programming language, but for today’s article we will focus on what I just mentioned as second stage of any programming language — the use of variables. If you are a complete newbie to QlikView, you should first learn the basics from our learning path — <a href="https://www.analyticsvidhya.com/blog/qlikview-learning-path/">Resource to learn Qlikview from scratch and become an expert</a></p><h3>Real life situation — variables to my rescue!</h3><p>I started using QlikView some time in early 2012 and I fell in love with it. We had a team of passionate developers and over next 6 months (with a lot of dark nights), we were able to replace the traditional clunky excel / access reports on a nice QlikView dashboard for entire Sales process of our organization. We were now the envy of other Intelligence units in our organization and were quoted as a successful transition across the Organization.</p><p>In September 2012, the Sales Director called me. This was the time of sales planning for next year. So, I expected him to ask the plans for the same. And I was well prepared for this. Over years, we had created an Excel based application which would simulate a whole lot of scenarios and then spit out (literally) the targets for all the sales people.</p><p><strong>Then came the bomb! </strong>The Sales Director said that this time he wants the planning to happen on our QlikView application. He said that he wanted a new sheet in our dashboard, where the sales people could enter parameters for their efforts (e.g. Taking 10 sales calls a day) and the application provides an estimate of where they would end taking seasonality and past trends in account.</p><p>I had no clue how to do this in QlikView, so I just listened to him. I didn’t want all the efforts from last 6 months to end and the people to go back to older ways of running business intelligence. Thankfully, I came across variables and their use to run what-if scenarios, which we went on to build in our application.</p><h3>What is a Variable? What are the methods to create it?</h3><p>Variables store data (static or calculated) of any type (numbers, string, boolean, float) and get referenced using expressions or directly by using its name. There are a few ground rules of variable naming convention in QlikView:</p><p>Whenever we create a variable for static value, we prefix variable name with “ <strong>v</strong> “ (lowercase).</p><p>Whenever we store an expression in a variable, we prefix variable name with “ <strong>e</strong> “ (lowercase).</p><p>After the first character, we should keep relevant names so that it is developer friendly. There are various methods to create a variable. The commonly used are:</p><p><strong><em>Method 1:</em></strong> By going into menu -&gt; settings/variable overview</p><p><strong><em>Method 2:</em></strong> By using SET and LET statements.</p><p>Let’s look at them one by one:</p><h3>Method 1:</h3><p>Go to Setting menu -&gt; Click Variable Overview</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*AFomKhSxDYV2FoZR.png" /></figure><p><strong>Note:</strong> You can define a variable with an expression also.</p><h3>Method -2:</h3><p>Whenever you start a new document and move to edit script, you must have noticed that there are some predefined values which QlikView loads based on your system configuration. Right ?</p><p>This is because, these are the variables that QlikView creates using <em>Set Statement</em>. You can also use “ <strong>Set</strong> “ statement for defining variable(s) in script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*6pc06sH5xN68NL2I.png" /></figure><p>Let’s see how do we do it. Look at the below syntax of defining variable:</p><pre><strong>Syntax:</strong> Set Variable_Name=Value;</pre><pre>Example: Set vPrevYr=&#39;2014&#39;;</pre><p>Another method of variable creation is using “ <strong>Let</strong>” statement. <strong>Let</strong> statement evaluates the expression on the right side of ‘=’ and assigns it to the variable on the left side of ‘=’.</p><p><strong>Syntax:</strong> Let Variable_Name=Expression;</p><pre>Example: Let vTest=5+8;</pre><p>If you will use <strong>Set </strong>statement to store expression output in a variable, it will store the expression as a value. I’ll show the detailed effect of defining a formula with <strong>Let</strong> and <strong>Set</strong> statement in the coming up sections.</p><pre>Example: Set vTest1=5+8;</pre><p>In the screenshot below, you can see all the defined variables(vPrevYr, vTest, vTest1) after running script. Similarly, new variables can also be defined / found here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*fxMKm1nQ26ek8jxC.png" /></figure><p>Also Read: <a href="https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/">How to implement incremental load in qlikview?</a></p><h3>How can we access variable?</h3><p>The value of a variable can be accessed using equals (=) sign. If the variable is prefixed with equals ‘ = ‘ sign, QlikView tries to evaluate the value as a formula (QlikView expression) and then displays or returns the result rather than the actual formula text.</p><p>Let’s understand it by accessing above created variable “ <strong>vTest</strong>” (created using Let) and “ <strong>vTest1</strong> “ (Created using Set).</p><p>Let’s create a text box object and put an expression as <strong>“=vTest”</strong> and similarly in another text box, we put <strong>“=vTest1” </strong>(as shown below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*DvvUTfgIWTSKQsRL.png" /></figure><p>Just notice that, the variable (vTest) created using “ <strong>Let</strong>” has got evaluated well. On the contrary, variable (vTest1) created using “ <strong>Set” </strong>hasnot been evaluated. This is the difference between creating a variable using Set or Let statement.</p><p>Now, to evaluate variable <strong>vTest1, </strong>we can use <strong>Dollar Sign Expansion (DSE)</strong>. It is method to evaluate a formula. Let’s look at this method of using dollar sign expansion. Also, this is also a standard way of accessing a variable in QlikView.</p><pre>Syntax: $(variable_Name)</pre><p>Now, I’ll access variable “ <strong>vTest1″ </strong>and check the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/0*c2UfLFiO6TM__vBb.png" /></figure><p>Also Read: <a href="https://www.analyticsvidhya.com/blog/2014/11/synthetic-keys-qlikview-simplified/">The concept of synthetic keys in Qlikview — Simplified !</a></p><h3>What are most Common Uses of variable(s)?</h3><p>Variables are the mainstay of data visualization softwares like Qlikview, D3.js etc. Optimizing the use of variables always reduces application development time. Let’s look at the some common uses of variables:</p><ol><li><strong>Portability:</strong> Variables are used when we want to store values and use them in multiple objects / charts without wanting to type those values again. Variables are also used when we want to perform dynamic calculations. For example: A variable created for representing previous year as “vPrevYr=$(=Max (Year)-1)”, refers to a dynamic calculation as per current selection and can be accessed by all objects of the document.</li><li><strong>Use in Chart Objects: </strong>We can make use of variables in charts at multiple places using:</li><li><strong>What-if analysis: </strong>We can use variable in what if analysis expression also using “<strong>Input Box</strong>” or “<strong>Slider Object”. </strong>In such cases, we take value of variable from these objects (Input, Slider) and used them in expression.</li><li><strong>In expressions (Set Analysis and Aggr): </strong>You can use variable in advance expression like Set Analysis, Aggr() and others. For more detail of these expression you can follow these articles (<a href="https://www.analyticsvidhya.com/blog/2014/01/set-analysis-qlikview/">Set Analysis</a>, <a href="https://www.analyticsvidhya.com/blog/2014/02/aggr/">AGGR()</a>)</li></ol><p>Apart from the uses mentioned above, we can also use variable for incremental data load and in various scripting methodology to improve our data models.</p><h3>End Notes:</h3><p>In this article, we looked at the importance of variables, methods to create it and the common uses of variable in daily development. Basically, we have looked at two methods (Set &amp; Let and Variable Overview) to create them.</p><p>If I summarize the most common use of variable, then it would be expression development using other charts objects like Input box, Slider, Text Box, Buttons and others. My advise to you would be using the variable in your dashboard. It reduces the static nature of dashboard and can change the value across the expression by changing it at one place only.</p><p>I hope this article helped you to get the detail of variables in QlikView. We would love to hear about it from you. Did you find it useful? Feel free to post your thoughts through comments below.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2015/06/variables-qlikview-data-stories/"><em>https://www.analyticsvidhya.com</em></a><em> on June 18, 2015.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=438842870870" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/use-of-variables-in-qlikview-to-create-powerful-data-stories-438842870870">Use of variables in QlikView to create powerful data stories</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reporting In Qlikview | Ad Hoc Reporting]]></title>
            <link>https://medium.com/analytics-vidhya/reporting-in-qlikview-ad-hoc-reporting-5a99b3b64576?source=rss-4be29faab438------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a99b3b64576</guid>
            <category><![CDATA[google-analytics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[analytics-vidhya]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Sunil Ray]]></dc:creator>
            <pubDate>Thu, 09 Jan 2014 19:13:14 GMT</pubDate>
            <atom:updated>2019-08-12T04:04:57.265Z</atom:updated>
            <content:encoded><![CDATA[<p>As a BI professional, I am used to receiving ad-hoc reporting requirements from business users which need a fast turn-around (sometimes under the name of ‘Analysis’). What is interesting is that 95% of these requirements can be delivered by re-arranging, cross-joining, summarizing or expanding some or the other existing reports.</p><p>So, if we merge all ad-hoc and regular reports, it will become a master report with multiple metrics and multiple dimensions, which can be summarized using following expression:</p><p><strong>Multiple Metrics (KPI) X Multiple Dimensions = Multiple Reports</strong></p><p>This is how I deal with the ad-hoc requirements. Excel provides ‘Pivot tables’ to enable creation of this kind of a master repository, where people can select dimension and metric from the available set of fields.</p><p>So, when I started using Qlikview two years back, I was looking for a similar solution i.e. multiple reports where the user has a feature to select or deselect dimension and metrics as per their requirements i.e. customized reporting. Thankfully, Qlikview 11 has a feature called “ <strong>Conditional enabling of Expression and Dimension</strong> “.</p><p>Depending on the user, you can show or hide certain dimensions or metrics. If you would like end users to create their own tables / reports, you may use this feature to dynamically add/remove dimensions or metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*WXVWyK_o4zChEQ_s.png" /></figure><p><strong>Step 1:</strong></p><p>Identify the entire set of dimension and metric that you want or can think of being used by end users. It is very critical that you bring in all dimensions and metrics, otherwise the purpose of this report gets lost.</p><p>Create separate <strong>INLINE</strong> table for dimensions and metric. I would recommend two columns in each table, one for the descriptor and one to uniquely identify it by a single letter (A-Z)/ Number.</p><p><strong>Example:</strong></p><pre>AdhocDimensions: <br><strong>LOAD</strong> * INLINE [dimension_1,dimesnion_flag <br>Region,1 <br>Coutry,2 <br>State,3 <br>City,4 <br>Year,5 <br>Quarter,6 <br>Month,7 <br>Branch Segment,8 <br>Product,9 <br>Product Group,10 <br>Sales Channel,11 <br>Sales Manager,12 <br>Sales Manager Designation,13 <br>Financial Advisor,14 <br>Finanacial Advisor Vintage Group,15 ];</pre><pre>AdhocParameter: <br><strong>LOAD</strong> * INLINE [Parameter_1, Parameter_flag <br>Sales, A <br>Renewal Collection,B <br>Poliices, C <br>Manpower,D <br>Profit,E];</pre><p>This table will not be associated to any other tables in your model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/0*pfUnZtxfVcla0Bf_.png" /></figure><p><strong>Step 2:</strong></p><p>Move to layout TAB, create a list box and select field “dimension_1” that was declared in <strong>INLINE</strong> table. Preferably select list box with LED check Box style so that selection / deselection of multiple dimensions can happen easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e_5nODn5OkwFbyJk.png" /></figure><p>Create similar list box for metric also.</p><p><strong>Step 3:</strong></p><p>Create object (Pivot Table/ Straight Table/ Chart …) that user will customize basis their selected dimension and expression. I am illustrating example of straight table.</p><p>Select all dimensions available in table <strong>“AdhocDimensions”</strong> (dimension inline table) and write a conditional expression for each dimension which results in the reporting. In the dimension properties, check the option for “Enable Conditional” and enter the following code for dimension “ <strong>Month</strong>”: Dimension Month has defined value 7 in inline table under “ <strong>dimension_flag</strong> “ field.</p><pre>=SubStringCount(Concat(dimension_flag, &#39;|&#39;), 7)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/0*AbkHKY2DggSjinDn.png" /></figure><p>Add similar code to each dimension, changing the corresponding numbers you used in your inline table.</p><p><strong>Step 4:</strong></p><p>Write expression for all metrics and enable conditional expressions by checking the option for “Enable” and writing conditional code for each expression</p><pre>=SubStringCount(Concat(Parameter_flag, &#39;|&#39;), &#39;A&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*xwW2acffPq1PitX7.png" /></figure><p><strong>Step 5:</strong></p><p>Add a condition in general tab of the straight table so that the object only displays when at least one dimension and expression is selected:</p><p>=GetSelectedCount(dimension_1) &gt; 0AND GetSelectedCount(Parameter_1)&gt;0</p><p>Finally, we have customized reporting for end-user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*LSNVs4X7OCerQJLk.png" /></figure><p>This is an example where we have brought out methods to create Customize reporting in Qlikview. What do you think about this methodology? Have you used to similar kind of practice? If yes, how did you create them? Do you think this tip was useful?</p><p>Do let me know your thoughts on using this feature in Qlikview.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2014/01/customize-reporting-qlikview/"><em>https://www.analyticsvidhya.com</em></a><em> on January 9, 2014.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a99b3b64576" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/reporting-in-qlikview-ad-hoc-reporting-5a99b3b64576">Reporting In Qlikview | Ad Hoc Reporting</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>