<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aditya Patkar on Medium]]></title>
        <description><![CDATA[Stories by Aditya Patkar on Medium]]></description>
        <link>https://medium.com/@aditya-pat10?source=rss-f98c4937f418------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*FijrelqPTBXLNZbe</url>
            <title>Stories by Aditya Patkar on Medium</title>
            <link>https://medium.com/@aditya-pat10?source=rss-f98c4937f418------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:31:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aditya-pat10/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Empowering Model Management: Weights & Biases for Seamless MLOps Integration]]></title>
            <link>https://blog.nonstopio.com/empowering-model-management-weights-biases-for-seamless-mlops-integration-86d8401f8a09?source=rss-f98c4937f418------2</link>
            <guid isPermaLink="false">https://medium.com/p/86d8401f8a09</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[weights-and-biases]]></category>
            <dc:creator><![CDATA[Aditya Patkar]]></dc:creator>
            <pubDate>Thu, 17 Aug 2023 11:47:52 GMT</pubDate>
            <atom:updated>2023-08-17T13:26:00.197Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZouZTRfJQ3M0nj27Np4ifw.png" /><figcaption>Credits: Author</figcaption></figure><p><strong>Introduction</strong></p><p>Machine Learning workflows have undergone a transformative evolution in recent years. Instead of relying solely on a single Jupyter notebook to encapsulate the entire process, the landscape has shifted towards a more intricate and organized pipeline, encompassing critical components such as meticulous data versioning, efficient data management, data visualization dashboards, training pipelines, and comprehensive model versioning. However, this increased sophistication and functionality haven’t come without a cost — complexity. The challenge has been to navigate this complexity and transform it into long-term simplicity. In this journey, MLOps platforms like Weights &amp;Biases are vital.</p><p>MLOps, a combination of “Machine Learning” and “Operations,” represents a pivotal paradigm shift in the field of machine learning. It addresses the challenges arising from the increased complexity of deploying, managing, and maintaining machine learning models in production environments. MLOps introduces a systematic approach that encompasses the entire machine learning lifecycle — ranging from data preprocessing, model training, and validation to deployment, monitoring, and ongoing iteration. This framework emphasizes automation, and reproducibility, enabling organizations to bring their machine learning projects from the experimental phase to scalable and sustainable production applications.</p><p><strong>About Weights &amp;Biases</strong></p><p>Weights &amp; Biases is a comprehensive MLOps platform with developer-first approach. It provides end-to-end ML workflow management. The features include Experiment Tracking, Data and Model versioning, Hyper-parameter Logging and Optimization, Automated Workflows, Monitoring and much more. Creating a Weights &amp; Biases account is free for individuals and for education purpose.</p><p><strong>How To Use</strong></p><p>To get started, first you need to create an account on the <a href="http://wandb.ai/">wandb.ai </a>. Once you have an account, create a new project and keep your API key safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*CU7v7kooL_dWuQbY28WIig.png" /><figcaption>Creating a project on WandB</figcaption></figure><p>Now it’s time to jump into the code. We will train a simple convolutional neural network on the CIFAR-10 Dataset. The base code I am using is provided in a <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">PyTorch tutorial</a>.</p><p>The dataset is of images from 10 classes: ‘plane’, ‘car’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’. The demo code builds a simple CNN with 2 convolution layers and 3 fully connected layers.</p><p>First, let’s install the wandb (Weights &amp; Biases) package and login. During login, you will be asked to provide an API key.</p><p><strong>Setup</strong></p><pre>!pip install wandb<br><br>import wandb<br><br>#login to wandb<br>wandb.login(anonymous=&quot;allow&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8gQB9V6ArKWilpDvu92bw.png" /></figure><p>Next, we will import the necessary libraries.</p><pre>#import necessary libraries<br><br>#pytorch related<br>import torch<br>import torchvision<br>import torchvision.transforms as transforms<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import torch.optim as optim<br><br>#other libraries<br>import matplotlib.pyplot as plt<br>import numpy as np<br>from types import SimpleNamespace</pre><p>Now we are ready to create a config. This includes all the hyper-parameters and other variables that you want to log to WandB should go here. A rule of thumb is that anything that has impact on model performance quality and anything that tells more about the task at hand should go in the config.</p><pre>#create the configuration<br>config = SimpleNamespace(<br>    batch_size = 4,<br>    dataset = &#39;CIFAR10&#39;,<br>    classes = (&#39;plane&#39;, &#39;car&#39;, &#39;bird&#39;, &#39;cat&#39;,<br>           &#39;deer&#39;, &#39;dog&#39;, &#39;frog&#39;, &#39;horse&#39;, &#39;ship&#39;, &#39;truck&#39;),<br>    learning_rate = 0.002,<br>    momentum = 0.9,<br>    n_epoch = 5<br><br>)</pre><p>We are defining the batch size, dataset, classes, learning rate, momentum and the number of epochs here. Now, let’s get the data ready.</p><pre>#create a data transformation chain<br>transform = transforms.Compose(<br>    [transforms.ToTensor(), #convert to tensor<br>     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) #normalize the data<br><br>#set the batch size<br>batch_size = config.batch_size<br><br>#get the training set, testing set and create respective dataloaders<br>trainset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=True,<br>                                        download=True, transform=transform)<br>trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,<br>                                          shuffle=True, num_workers=2)<br><br>testset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=False,<br>                                       download=True, transform=transform)<br>testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,<br>                                         shuffle=False, num_workers=2)<br><br>#set the classes<br>classes = config.classes<br><br>device = torch.device(&quot;cuda:0&quot; if torch.cuda.is_available() else torch.device(&#39;cpu&#39;))</pre><p>I won’t go too deep into explaining the ML side, but here we are transforming our data and creating training and testing data loaders that will feed data to our model. We are also checking if we have GPU access.</p><p>Let’s see how the data looks.</p><pre>def imshow(img):<br>    &#39;&#39;&#39;<br>    This function displays an image<br>    &#39;&#39;&#39;<br>    <br>    #denormalize the image<br>    img = img / 2 + 0.5 <br>    <br>    #convert the image to numpy array<br>    npimg = img.numpy()<br>    <br>    #transpose the image to get the correct shape<br>    plt.imshow(np.transpose(npimg, (1, 2, 0)))<br>    <br>    #display the image<br>    plt.show()<br><br># get some random training images<br>dataiter = iter(trainloader)<br>images, labels = next(dataiter)<br><br># show images<br>imshow(torchvision.utils.make_grid(images))<br># print labels<br>print(&#39; &#39;.join(f&#39;{classes[labels[j]]:5s}&#39; for j in range(batch_size)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*Igeo8uwZZ7GcGMBKaRk0zA.png" /><figcaption>Samples from the dataset. Labels: cat, deer, dog, deer</figcaption></figure><p>Next up, let’s create a blueprint of our neural network architecture.</p><pre>class Net(nn.Module):<br>    &#39;&#39;&#39;<br>        This class defines the neural network architecture<br>    &#39;&#39;&#39;<br>    <br>    def __init__(self):<br>        &#39;&#39;&#39;<br>            This constructor function defines the layers of the neural network<br>        &#39;&#39;&#39;<br>    <br>        super().__init__()<br>        self.conv1 = nn.Conv2d(3, 6, 5) # a convolutional layer with 3 input channels, 6 output channels and a kernel size of 5<br>        self.pool = nn.MaxPool2d(2, 2) # a max pooling layer with a kernel size of 2 and a stride of 2<br>        self.conv2 = nn.Conv2d(6, 16, 5) # a convolutional layer with 6 input channels, 16 output channels and a kernel size of 5<br>        self.fc1 = nn.Linear(16 * 5 * 5, 120) # a fully connected layer with 16 * 5 * 5 input features and 120 output features<br>        self.fc2 = nn.Linear(120, 84) # a fully connected layer with 120 input features and 84 output features<br>        self.fc3 = nn.Linear(84, 10) # a fully connected layer with 84 input features and 10 output features<br><br>    def forward(self, x):<br>        &#39;&#39;&#39;<br>            This function defines the forward pass of the neural network<br>        &#39;&#39;&#39;<br>        x = self.pool(F.relu(self.conv1(x))) # apply relu activation to the output of the first convolutional layer and then apply max pooling<br>        x = self.pool(F.relu(self.conv2(x))) # apply relu activation to the output of the second convolutional layer and then apply max pooling<br>        x = torch.flatten(x, 1) # flatten all dimensions except batch<br>        x = F.relu(self.fc1(x)) # apply relu activation to the output of the first fully connected layer<br>        x = F.relu(self.fc2(x)) # apply relu activation to the output of the second fully connected layer<br>        x = self.fc3(x) # apply relu activation to the output of the third fully connected layer which is the output layer<br>        return x<br><br># create an instance of the neural network<br>net = Net().to(device)</pre><p>Our network contains 2 convolutional layers and 3 fully connected layers. We have also defined our forward pass. In the end, we move our network to GPU if available.</p><p>Let’s get our<a href="https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e"> loss function</a> and the <a href="https://towardsdatascience.com/stochastic-gradient-descent-clearly-explained-53d239905d31">Stochastic Gradient Descent</a> optimizer.</p><pre>#define the loss function and the optimizer<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.SGD(net.parameters(), lr=config.learning_rate, momentum=config.momentum)</pre><p>We will also write a helper function to calculate accuracy on our test data.</p><pre>def get_accuracy(testloader, device, net):<br>    &quot;&quot;&quot;<br>    This function calculates accuracy<br>    &quot;&quot;&quot;<br>    correct = 0<br>    total = 0<br><br>    # since we&#39;re not training, we don&#39;t need to calculate the gradients for our outputs<br>    with torch.no_grad():<br>        for data in testloader:<br>            images, labels = data<br>            images = images.to(device)<br>            labels = labels.to(device)<br>            # calculate outputs by running images through the network<br>            outputs = net(images)<br>            # the class with the highest energy is what we choose as prediction<br>            _, predicted = torch.max(outputs.data, 1)<br>            total += labels.size(0)<br>            correct += (predicted == labels).sum().item()<br>    accuracy = 100 * correct // total<br>    return accuracy</pre><p><strong>Training</strong></p><p>We are now ready to train our network. Most of our Weights &amp; Biases code will come here.</p><p>First, we will initialize a WandB run. This will be reflected on the WandB dashboard</p><pre>#start a training run<br><br>run = wandb.init(project=&quot;demo&quot;,<br>                 job_type=&quot;train&quot;,<br>                 config=config)<br><br># we pass the config back from W&amp;B<br>config = wandb.config</pre><p>The output will give us a link to the run in the WandB dashboard.</p><pre>Syncing run electric-sea-21 to Weights &amp; Biases (docs)<br>View project at https://wandb.ai/teamaditya/demo<br>View run at https://wandb.ai/teamaditya/demo/runs/z83or8t2</pre><p>Let’s set our network in the training mode and start training it.</p><pre>net.train()<br>for epoch in range(config.n_epoch):  # loop over the dataset multiple times<br><br>    running_loss = 0.0<br><br>    # linearly decay learning rate<br>    optim.lr = config.learning_rate*(1-epoch/config.n_epoch)<br><br>    <br>    for i, data in enumerate(trainloader, 0):<br>        <br>        # get the inputs; data is a list of [inputs, labels]<br>        inputs, labels = data<br>        inputs = inputs.to(device)<br>        labels = labels.to(device)<br><br>        # zero the parameter gradients<br>        optimizer.zero_grad()<br><br>        # forward + backward + optimize<br>        outputs = net(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        optimizer.step()<br><br>        # print statistics<br>        running_loss += loss.item()<br>        if i % 2000 == 1999:    # print every 2000 mini-batches<br>            print(f&#39;[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}&#39;)<br>            running_loss = 0.0<br><br>        wandb.log({&quot;loss&quot;: loss.item(),<br>                   &quot;lr&quot;: optim.lr,<br>                   &quot;epoch&quot;: epoch})<br>        <br>    net.eval()<br>    ckpt_file = &quot;./cifar_net.pth&quot;<br>    torch.save(net.state_dict(), ckpt_file)<br><br>    artifact_name = f&quot;{wandb.run.id}_model&quot;<br>    at = wandb.Artifact(artifact_name, type=&quot;model&quot;)<br>    at.add_file(ckpt_file)<br>    wandb.log_artifact(at, aliases=[f&quot;epoch_{epoch}&quot;])<br><br>    accuracy = get_accuracy(testloader, device, net)<br>    wandb.log({&#39;accuracy&#39;:accuracy})<br><br>print(&#39;Finished Training&#39;)</pre><p>First, we access the number of epochs from the config and iterate over them, we also keep track of the running loss for printing purpose. We are applying a linear learning rate decay for every epoch.</p><p>Next, we are iterating over the input image and the labels. First we pass them on to the GPU if available. We then pass the input to model, get a prediction and calculate the loss.</p><p><em>wandb.log() </em>command is used to log metrics to the dashboard. We are logging the loss, learning rate and the epoch at every step.</p><p>After every epoch, we are saving the model in WandB using Artifacts. First, we create an artifact using <em>wandb.Artifact() </em>command. Then we add a file to the artifact using <em>add_file() </em>method of the artifact. The file is also tagged with epoch number for convenience. Finally we can push it to the platform using <em>wandb.log_artifact().</em></p><p>Last but not the least, we are calculating the accuracy on the test set and logging it to WandB using <em>wandb.log()</em></p><p><strong>How does the data look on the dashboard?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f3OYIl_s2aAjLt9vqiZzJw.png" /><figcaption>WandB Dashboard: Individual run</figcaption></figure><p>Moving over to the dashboard of the project, we can see that our run has been tracked by WandB. Clicking on the run will give us more information about it.</p><p>Moving to the logs panel, you can see the logs that were generated during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*4llgFGrE0QqAGCzc8aOdbQ.png" /><figcaption>WandB Dashboard: Logs</figcaption></figure><p>The overview page shows our configuration for that particular run and also shows things like GPU count, GPU types etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-SHL8jNvT3BrvlRhAc_J5A.png" /><figcaption>Configuration, runoverview panel</figcaption></figure><p>Now, try running the code multiple times using different hyper-parameters. Once done, the project view shows the comparison of all the runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LkU9a5le8x_dhGLF-Rxkog.png" /><figcaption>Project View: Comparing runs</figcaption></figure><p>Finally, switching over to the artifacts tab would show us the models that we have saved. Each saved model is tagged with the epoch. We can now click on Link to registry to make the model usable in future code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*Rzolmnjp5FkybV0Y5H2Hgw.png" /><figcaption>Linking the model to the registry</figcaption></figure><p><strong>Prediction</strong></p><p>One of the most important uses of WandB is for easily importing the model for predictions. Each model has a uniquely identifiable name using which we can import it into our application. This makes it easy to swap the model for a new version.</p><p>Let’s load our model from WandB. We will create a helper function to achieve this task</p><pre>from pathlib import Path<br><br>#let&#39;s create an helper function for loading the model<br>def load_model():<br>    &quot;&quot;&quot;<br>    This function loads the model from the model registry.<br>    &quot;&quot;&quot;<br><br>    DEVICE = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br>    api = wandb.Api(api_key=&#39;&lt;Your_API_Key&gt;&#39;)<br>    artifact = api.artifact(&quot;teamaditya/model-registry/CIFAR-10 CNN:v0&quot;, type=&quot;model&quot;)<br>    model_path = Path(artifact.download())<br><br>    # recover model info from the registry<br>    producer_run = artifact.logged_by()<br><br>    # load the weights dictionary<br>    model_weights = torch.load(model_path / &quot;./cifar_net.pth&quot;, map_location=&quot;cpu&quot;)<br><br>    # create the model<br>    model = Net()<br><br>    # load the weights into the model<br>    model.load_state_dict(model_weights)<br><br>    # set the model to eval mode<br>    model.eval()<br>    return model.to(DEVICE)</pre><p>Here first we check if we have a GPU. Then we will connect to the WandB API using the API key that is found on the dashboard. Next, we find the model using the unique identifier and download the model. The weights are loaded from the <em>.pth</em> file. We then create an instance of our network class and load the weights. Now this model is ready to be used for prediction. We can now again run the testing code to check the accuracy.</p><pre>#calculate the accuracy of the model<br><br>correct = 0<br>total = 0<br># since we&#39;re not training, we don&#39;t need to calculate the gradients for our outputs<br>with torch.no_grad():<br>    for data in testloader:<br>        images, labels = data<br>        images = images.to(device)<br>        labels = labels.to(device)<br>        # calculate outputs by running images through the network<br>        outputs = net(images)<br>        # the class with the highest energy is what we choose as prediction<br>        _, predicted = torch.max(outputs.data, 1)<br>        total += labels.size(0)<br>        correct += (predicted == labels).sum().item()<br><br>print(f&#39;Accuracy of the network on the 10000 test images: {100 * correct // total} %&#39;) #better than random chance which is 10%</pre><p>We can even use the saved model in actual applications that are deployed or expose these model as an API. The possibilities are endless!</p><p><strong>Next Steps</strong></p><p>This is just the start. There are many more things Weights &amp; Biases can do including saving evaluation samples, creating automated pipelines, designing interactive reports and so on. All of this can be explored. One thing I would recommend as an immediate next step is to tweak the hyper-parameters and run the above training code multiple times and see the kind of graphs are visible on the WandB dashboard.</p><p>Thank you for reading! Read my other articles related to this one:</p><ul><li><a href="https://towardsdatascience.com/garbage-segregation-using-pytorch-6f2a8a67f92c">Garbage Segregation Using PyTorch</a></li><li><a href="https://towardsdatascience.com/5-powerful-pytorch-functions-that-every-beginner-should-know-e75cb26ec8bc">5 Powerful PyTorch Functions That Every Beginner Should Know</a></li></ul><blockquote><em>Please </em><a href="https://aditya-pat10.medium.com/"><em>f</em>ollow me</a><em> and </em><a href="https://blog.nonstopio.com/">my team</a><em> in staying updated on the latest and greatest in the world of AI, UX, Web &amp; Mobile technology, as we work together to strengthen our community through knowledge sharing. </em>🙂</blockquote><p><strong>References:</strong></p><ul><li>The code is inspired by the demo code at <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></li><li><a href="http://wandb.ai/">Weights &amp; Biases</a></li><li><a href="https://drive.google.com/file/d/1DBVCEX1TVescmq804Mt-3vdSaGICFx3Y/view?usp=sharing">Executable Notebook</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86d8401f8a09" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/empowering-model-management-weights-biases-for-seamless-mlops-integration-86d8401f8a09">Empowering Model Management: Weights &amp; Biases for Seamless MLOps Integration</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Learn Exploratory Data Analysis Practically]]></title>
            <link>https://medium.com/data-science/lets-learn-exploratory-data-analysis-practically-4a923499b779?source=rss-f98c4937f418------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a923499b779</guid>
            <category><![CDATA[premier-league]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Aditya Patkar]]></dc:creator>
            <pubDate>Wed, 09 Jun 2021 06:43:12 GMT</pubDate>
            <atom:updated>2021-06-11T16:26:37.042Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Using the ‘English Premier League Results’ dataset | A beginner friendly step-by-step approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*-sOaG3Pzp7yLvyI6" /><figcaption>“That is beyond special” — Peter Drury | Image on <a href="https://unsplash.com/">Unsplash</a> by Nathan Rogers</figcaption></figure><p>Season 2020–21 of English Premier League — Arguably the world’s most entertaining football league, has come to an end. Congratulations to Manchester City for winning the league. It has been a different season without fans in the stadium, but it has given us some nail biting finishes, wild and breathtaking matches, memorable goals and captivating moments. Before we get our studs and shin pads ready for next season, why not check some interesting trends in the 29 seasons of the beautiful game played so far and learn step-by-step approach to performing Exploratory Data Analysis along the way? Let’s dig in.</p><p>The dataset we’re using is from <strong>kaggle</strong>, containing information about more than 10,000 Premier League matches played. The dataset can be found <a href="https://www.kaggle.com/irkaal/english-premier-league-results">here.</a></p><p>We will use python libraries NumPy,Pandas,Matplotlib and Seaborn in this project. If you read through, you will be in a position to perform EDA on any dataset of your choice!</p><h3>Downloading the Dataset</h3><p>Let’s start by downloading the dataset from Kaggle. Here we use the opendatasets library made for python to download the same. By passing the URL of the Kaggle page for the dataset to opendatasets.download(), we will download the dataset.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D4%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D4%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F35de53b1facc48bd8367c69f059077fa%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/383b4f7556dea7b11b89f24a4e5edcd2/href">https://medium.com/media/383b4f7556dea7b11b89f24a4e5edcd2/href</a></iframe><p>The dataset has been downloaded and extracted.</p><h3>Data Preparation and Cleaning</h3><p>In any data analysis project, while working with real world raw data, it is very necessary to ready the data for our analysis. There could be wrong entries, missing values that need to be dealt with. Along with that, we might want to add new columns to the dataset, which are useful for our analysis or we might want to merge a few datasets together, this should be done as a preliminary step.</p><p>In our case, let’s start by converting the dataset into a Pandas dataframe. Pandas is a python library which gives us handy functions for data cleaning, merging, operations etc. It creates an object called as <strong>DataFrame</strong>, which is basically data represented in tabular form. We can read different types of files eg. CSV, JSON, XLSX etc. and create a data frame using the same.</p><p>To know more about Pandas, check my <a href="https://www.notion.so/Pandas-79a9e56f662240bf990fdc8fde1ad40d">Notion notebook</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D14-15&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D14-15&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fae10dd63a0c44189a12e37db1d145bd1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f611bfae32a5621438ecfe6bf97e1770/href">https://medium.com/media/f611bfae32a5621438ecfe6bf97e1770/href</a></iframe><p>Let’s take a look at our dataset</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D17&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D17&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fc1328693750842f183115e0729f9596c%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/67a6a7de517ae81ff67cc9d49aa4ec41/href">https://medium.com/media/67a6a7de517ae81ff67cc9d49aa4ec41/href</a></iframe><p>The dataset contains results of all the matches <strong>1993–94</strong> season to <strong>2020–21</strong> season. Labels to note :</p><ul><li>FTHG : Full time home goals.</li><li>FTAG : Full time away goals.</li><li>FTR : Full time Result.</li><li>HTHG : Half time home goals.</li><li>HTAG : Half time home goals.</li><li>HTR : Half time home goals.</li><li>HS : Home team shots.</li><li>AS : Away team shots.</li><li>HST : Home team shots on target.</li><li>AST : Away team shots on target.</li><li>HC : Home team corners.</li><li>AC : Away team corners.</li><li>HF : Home team Fouls.</li><li>AF : Away team Fouls.</li><li>HY : Home team yellow cards.</li><li>AY : Away team yellow cards.</li><li>HR : Home team red cards.</li><li>AR : Away team red cards.</li></ul><p>We can check the shape of our dataset using the .shape method. It returns a tuple in the form (Number of Rows, Number of columns).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D20&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D20&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fcc255269fa6547f6aaacdf9955018c31%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/91c37434af370f4ecf6f42024a5548c9/href">https://medium.com/media/91c37434af370f4ecf6f42024a5548c9/href</a></iframe><p>As we can see, the dataset contains information about more than 10000 Premier League matches.</p><p>.info method provides us a way to describe data, it shows us name of the column, number of non-null values and type of the data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D22&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D22&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F7b3d77be4b784341a4ab2183d1ef0657%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/033fed7ff15603c9e4265d938f79b48d/href">https://medium.com/media/033fed7ff15603c9e4265d938f79b48d/href</a></iframe><p>As we can see, there are a few columns with null values. This must be as record keeping of exhaustive stats started later than inauguration of Premier League. We can see the number of null cells for every column using isna function and delete the rows which contain the null values using .drop method. We have to provide axis argument where 0 = rows and 1 = columns.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D26&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D26&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F785d1204563e46eb9e2de2d62e9c9793%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/618b8b173c339fe70dd283d1fc9037e6/href">https://medium.com/media/618b8b173c339fe70dd283d1fc9037e6/href</a></iframe><p>We have verified that all the null values have been dropped. Let’s see how our dataset looks now.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D28&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D28&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F93a96531c109460fac0e65dde82da57e%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/dda0ccf53673ed102792c2684b26e90a/href">https://medium.com/media/dda0ccf53673ed102792c2684b26e90a/href</a></iframe><p>The exact time is not really that important for us. We can just extract the month and day using .month and .day attributes of DateTime ojects. We will then drop the DateTime column as it is not needed.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D30&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D30&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fbf30966074664779a3f58a16031b9e68%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3093c71b2c4eb2675d72507b984b85ef/href">https://medium.com/media/3093c71b2c4eb2675d72507b984b85ef/href</a></iframe><p>Our data is now ready for visualizations.</p><h3>Exploratory Analysis and Visualization</h3><p>We have prepared our dataset in such a way that now we can use it to find interesting insights. Let’s compute a few things that will tell us more about the data. For that we will be using two data visualization libraries, namely matplotlib and seaborn. These libraries contain useful methods to plot graphs and charts. Matplotlib provides basic functionality while Seaborn builds on top of Matplotlib to give advanced functionality in very few lines of code.</p><p>To know more about matplotlib and seaborn, check out my <a href="https://www.notion.so/Data-Visualization-34175fa561104d0295a6c8d46ea43045">Notion notebook</a>. You can also go through the documentation for both by simply visiting their respective websites.</p><h4>Data insights using .describe()</h4><p>Before plotting graphs, Let’s first get some insights from our dataset using the .describe() method provided by pandas.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D34&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D34&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F4bed865988af4ecf973839f0895e1e4f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1361a2d1364adbfea15054970bbe5b66/href">https://medium.com/media/1361a2d1364adbfea15054970bbe5b66/href</a></iframe><p>This gives us some interesting preliminary insights into our data. As we can see,</p><ol><li>On an average the home team scores more goals and takes more shots.</li><li>The away team commits more fouls.</li><li>Away team gets more yellow and red cards.</li><li>Most goals scored by one team in a match are 9.</li><li>At most 5 goals are scored at half time.</li></ol><p>Let’s import matplotlib.pyplot and seaborn.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D37%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D37%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fc40b68f9b8da40e09998601b6325dedf%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/253b35e2b4fe6baf220e4a2f721d593c/href">https://medium.com/media/253b35e2b4fe6baf220e4a2f721d593c/href</a></iframe><h4>Season-wise analysis</h4><p>Let’s look at the data and compare the trends based on season. Here we will first sort the data according to the season by using .groupby() function.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D39&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D39&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Ffa3cb4a1f12a431da5673720ece5a114%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b325b3f5fa58ce6f75c33827c4875747/href">https://medium.com/media/b325b3f5fa58ce6f75c33827c4875747/href</a></iframe><p><strong>Goals scored per season</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*k_33ltG-mczkQUs3" /><figcaption>“Aguerooooooooooooooooooooooooooooooooo” | Image on <a href="https://unsplash.com/">Unsplash</a> by Nathan Rogers</figcaption></figure><p>Let’s compare number of goals scored per season. We will create a new column which would be the addition of goals scored by the home team and goals scored by the away team.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D41%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D41%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F6e04124076f04a51960b46ff6a3ba16d%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0062c26d5330bc00b466be6a93e33fe8/href">https://medium.com/media/0062c26d5330bc00b466be6a93e33fe8/href</a></iframe><p>Now that we have the data frame, let’s plot the bar graph. Here we’re using the barplot() function from seaborn library. As arguments we&#39;ll provide Goals series for x axis, Season series for y axis and a data argument. Seaborn has good support for pandas. By providing a data frame in the &#39;data&#39; argument, seaborn will automatically find the x and y series&#39; from the given data frame.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D43%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D43%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F88bb6b85e1db4df58e41d7fdbd367c72%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/48881b1fe6382c4a1112a69384f0f220/href">https://medium.com/media/48881b1fe6382c4a1112a69384f0f220/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0M_WQSsczwdEftRgcpwdBA.png" /></figure><p>As we can see above,</p><ol><li>Number of goals scored per season have been fairly similar, around 1000.</li><li>2018–19 season saw the highest number of goals scored.</li><li>2005–06 season saw the lowest number of goals scored.</li></ol><p><strong>Home Goals vs Away Goals</strong></p><p>We will plot a <strong>double barplot</strong> for the same. Before we do that, SNS requires us to melt our data which lets us use the comparison parameter as hue. It will get clear when you compare the data frame below with our original data frame.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D46&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D46&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F88b478b17df24696ad97d3b023dfe8a3%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/553b766ed3c4d94c56412b6eadfec776/href">https://medium.com/media/553b766ed3c4d94c56412b6eadfec776/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D47%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D47%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F162f287a6dee4ff3aacb25ca125eb75b%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bb210df0195a900b113f61ce4656d73/href">https://medium.com/media/7bb210df0195a900b113f61ce4656d73/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M74ZBD-38mRItKLv6Pz_eg.png" /></figure><ol><li>We can note that Goals scored by home team are almost always more than goals scored by away team.</li><li>In 2020–21 Season, the games were played behind close doors. The absence of fans clearly reflects in the stats. Goals scored by away team are more than those scored by home team for this season.</li></ol><p><strong>Plotting a heatmap for distribution of goals per month</strong></p><p>Now let’s prepare our data for a heatmap that shows us the goals scored per month with respect to each season. For that, we use a special function provided by pandas called .pivot() which converts the data frame into a 2D matrix, with first argument (series) being rows, 2nd argument being columns and third argument being the values corresponding to series 1 and series 2. Check it out below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D50&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D50&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fb927abb369f1415b9ee34a9eb8f846dd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f497f0e5b2be159a3be98c5098ff51f7/href">https://medium.com/media/f497f0e5b2be159a3be98c5098ff51f7/href</a></iframe><p>Now we plot a heatmap by passing the dataset to the heatmap() function provided by seaborn. annot argument can be set to True if you want to show the values in the blocks. Try it out!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D52%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D52%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fef0bc44e96f3433d988620047b3046d3%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d22b022118631cb2e31d25144dee8182/href">https://medium.com/media/d22b022118631cb2e31d25144dee8182/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0g8O3_ESgs3jfPpqiQJ77w.png" /></figure><p>The above graphs tells us more about our data.</p><ol><li>No games are played in the months of June and July, thus no goals are scored.</li><li>Almost every year, December has seen the most number of goals. This is because December sees the most number of matches played.</li><li>The 2019–20 season was disrupted due to the pandemic. Games were played in June and July instead of April and May under project restart.</li><li>The season ends mid-May, thus the low number of goals scored in May.</li></ol><p>Let’s quickly plot another graph of Goals scored per game. It is fairly simple.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D54%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D54%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F8283b68fb68b4d9898fbd59e4f5160d4%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf17d58a977f95174184694a467f1495/href">https://medium.com/media/bf17d58a977f95174184694a467f1495/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D55%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D55%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fd9dfa64541754928bf90e85e04e18223%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ca1176c5629d14e72c1f0b8b58f629b/href">https://medium.com/media/2ca1176c5629d14e72c1f0b8b58f629b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxdnJ6KrI29RcjgsnPL30Q.png" /></figure><p>Here we can observe that Goals per game stat remains fairly consistent, which shows that there are no real visible “Settling period”, “Thriving period” or “Burnout period” or at least it doesn’t affect the score line much. Pretty interesting that September witnessed the highest goals per game twice.</p><p><strong>Shot comparison using line chart</strong></p><p>Now, let’s explore how the Shots taken by home team compare to those taken by away team for every Season. We will use the same EPL_by_season data frame that we created before.</p><p>We can plot multiple the lines in same graph by writing all sns.lineplot() calls in the same cell as seen below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D59%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D59%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fd1d3e7cc32fd478eb881a1d08bdc9f75%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6bcc3bc7819baa4112e4bd1d3aa5dc7e/href">https://medium.com/media/6bcc3bc7819baa4112e4bd1d3aa5dc7e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRcoK1ZRcZepL0v1C3H_9Q.png" /></figure><p>This is a very interesting chart. It tells us that,</p><ol><li>Around 60% of the shots taken end up on target.</li><li>See how no line intersects. Shots taken by home team and shots on target by home team are always more than away team. There’s clearly a home advantage.</li><li>The lowest difference between the lines come during 2020–21 season. This season there were no fans allowed. This shows how fans literally play the role of 12th man of the team.</li><li>Even the the shots lines remain fairly flat, there’s a remarkable fall in Shots on target when it comes to 2013–14 season.</li></ol><h4>Team-wise comparison</h4><p>Let’s see how the stats compare based on teams that have played in the top division of English football. We would need to create two group-by objects because we don’t have a teams column. We have HomeTeam and AwayTeam columns. We will use these objects further in our analysis.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D62%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D62%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fd4113cfe419c4faaa700e2ed8000c776%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ab3d21adf07818e82e79b141c0b18d9/href">https://medium.com/media/7ab3d21adf07818e82e79b141c0b18d9/href</a></iframe><p><strong>Total goals scored by each team over all seasons</strong></p><p>We will sum the goals scored by home teams from the EPL_home object and goals scored by away teams from the EPL_away object. We will plot a simple bar graph to visualize the same.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D64%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D64%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F59f291b0c1984302957e14e152ca250e%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1864c84dbd447c21d603289fc258cc7a/href">https://medium.com/media/1864c84dbd447c21d603289fc258cc7a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D65%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D65%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fb3c48446dde74755a4a051f4033f2124%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e324b53ebc02eb50cdb9ab84d2a211ea/href">https://medium.com/media/e324b53ebc02eb50cdb9ab84d2a211ea/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_OQATqsqUbTcMiEKKvBGw.png" /></figure><p>As we can see,</p><ol><li>Manchester United has scored the highest goals in Premier League since 2000–01 season till date.</li><li>The traditional “big-six” teams occupy the top six positions.</li><li>Bradford has scored the least goals in premier league.</li></ol><p><strong>Team-wise shot conversion rate of teams</strong></p><p>Let’s find which team has been the most lethal. Shot conversion rate is basically how many shots taken by a team end up in the net. We will create a new column which will divide goals scored by shots taken by the team.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D68%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D68%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F9bc440d48e00478b927b8562856efa6f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f5222ef7d65e129a19d6c8644702064/href">https://medium.com/media/4f5222ef7d65e129a19d6c8644702064/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttp%3A%2F%2Fjovian.ml%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D69%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=http%3A%2F%2Fjovian.ml%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D69%26hideOutput%3Dtrue&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0060cdea5a132e730a4b085a5fc5e251/href">https://medium.com/media/0060cdea5a132e730a4b085a5fc5e251/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HL3zPdmKz0CDhmo2xKKVvA.png" /></figure><p>We can see that,</p><ol><li>Over the seasons Arsenal has been the most lethal team followed by Manchester United.</li><li>Huddersfield has been the most wasteful with the shots they have gotten.</li><li>Brighton have always been called out by pandits for playing really attacking football but not converting the chances into goals. Above graph proves that.</li></ol><p><strong>Home win percentage</strong></p><p>Teams refer to their home ground as fortress. Let’s see who has defended their fortress the most. We will calculate percentage of home matches that a team has won. We will use the loc method of data frame to find the rows where FTR is H or in simple language, the home team has won.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D72%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D72%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fbc59d1be925a4c3ba59af340f226eb41%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a6a56eb24d336bdc3d7728ae90061c7/href">https://medium.com/media/8a6a56eb24d336bdc3d7728ae90061c7/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D73%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D73%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F9558706db97d48d583526eca5410f7c7%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/17607f920ad2e4014487a2c597f601e0/href">https://medium.com/media/17607f920ad2e4014487a2c597f601e0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MoNeNeeafeiTlfo-7nnf1A.png" /></figure><ol><li>Manchester United has won the most % of the games played at Old Trafford, followed by Arsenal at Emirates and Highbury, and Chelsea at Stamford Bridge.</li><li>Coventry holds the unwanted record of winning the least games of those played at St. Andrew’s stadium and Ricoh Arena.</li></ol><h3>Deep Dive</h3><p>Now we basically know fundamental details about the dataset we have. It’s interesting how just a few lines of code can make sense of a huge dataset containing raw data and lots of values. The visualizations help to a great extent. We started with knowing nothing about the data except the columns it contained. Now just a few lines of codes later, we know a lot more about what goes on in the Premier League and the major trends.</p><p>Now let’s dive deeper and answer a few interesting questions about the dataset which can help us understand the data even better.</p><p><strong>Q1: Which Referee gave the most red cards?</strong></p><p>To answer this, we can group the data based on Referee and sum the values. We will plot the graph for red cards.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D77%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D77%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F4412cfeb077f45d29283fb218d653a03%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/968272f82c01a0a870e933bc266600a9/href">https://medium.com/media/968272f82c01a0a870e933bc266600a9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q4Be2YxdiN9qTotib1jxkw.png" /></figure><p>Quite a lot of referees, but it is Mike Dean who tops the list, being the only referee to award more than 100 red cards.</p><p><strong>Q2: Which season saw the most comebacks?</strong></p><p>Let’s define a comeback as a situation where a team which is trailing at half time ends up winning the game.</p><p>We will find results where,</p><ol><li>HTR is not equal to FTR</li><li>FTR or HTR is not equal to D or draw.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D80%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D80%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fe93a0d4a2eeb41e2a594b467dc1671b9%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a090595489be6016e54ccd49bdbe58a3/href">https://medium.com/media/a090595489be6016e54ccd49bdbe58a3/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D81%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D81%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Fd693d9febd6943a483457ffca9b35df1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f7d6d923debaa54475fef1bba738cd8/href">https://medium.com/media/0f7d6d923debaa54475fef1bba738cd8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SekUBgjZBqiEDtHJRsHiBA.png" /></figure><p>2019–20 Season, won by Liverpool, witnessed the most comebacks followed by the 2012–13 season.</p><p>Q3: What about the standings at the end of the season?</p><p>It is possible to plot a points table for each season. Let’s see how!</p><p>We’ll first need to find the following stats for home teams.</p><ol><li>Number of home games played.</li><li>Number of home games won.</li><li>Number of home games drawn.</li><li>Number of home games lost.</li><li>Goals scored and goals against.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D85%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D85%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2Ff22387b5d4ec4704a9663669f5e9e68f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3faab76faf1d03ef01fa8d7b6c5977e3/href">https://medium.com/media/3faab76faf1d03ef01fa8d7b6c5977e3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*aBenMiusXCDZcjnKBmnwlg.png" /></figure><p>We can find the same for away games.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D87%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D87%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F91d050b84f034b5596cf748d03e1ce55%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b2a1d3ac1fc25ade1b2ac59f802e5ca/href">https://medium.com/media/8b2a1d3ac1fc25ade1b2ac59f802e5ca/href</a></iframe><p>Now let’s merge both the databases, calculate points as:</p><ol><li>3 points for win.</li><li>1 point for a draw.</li></ol><p>Let’s compute the same for amount pledged and add dots for goals using sns.scatterplot() for comparing.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D90%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D90%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F2aef5952407f4c0f9377237582d029e3%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/52ed2091301558e47bc3cb3a97036b4d/href">https://medium.com/media/52ed2091301558e47bc3cb3a97036b4d/href</a></iframe><p>Now let’s rank the teams based on points. Ambiguity occurs when 2 teams end up with same points. In that case, we rank the teams based on Goal difference or GD.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D92&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D92&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F4d1fbc8249d74fb8a7284ca9e986db05%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5e802a43efab9de663e63f19e2aea1d/href">https://medium.com/media/e5e802a43efab9de663e63f19e2aea1d/href</a></iframe><p>Now let’s plot a sample table.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D94&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fenglish-premier-league%2Fv%2F14%26cellId%3D94&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fff4ce095b4924e6d8558ea0c02deb3a6%2Fpreview%2F7b0fbe26099c4cf6aba4b0f5949d4b47%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/32456e1cc1ab5715f28aa245c8dff34e/href">https://medium.com/media/32456e1cc1ab5715f28aa245c8dff34e/href</a></iframe><p>There we have it. We have successfully visualized the points table from the data we had. We can see,</p><ol><li>Manchester city won the 2020–21 Premier League title.</li><li>Man City, Man Utd, Liverpool and Chelsea would play in Champions league 2021–22 (Top 4)</li><li>Leicester and West Ham would play in Europa league and Tottenham would play in Europa Conference League.</li><li>Unfortunately, Fulham, West Brom and Sheffield Utd have gotten relegated from the Premier League.</li></ol><h3>Conclusion</h3><p>As entertaining as Premier League is to watch, the data surrounding it is interesting to dive into. We saw how stats from different seasons compare, how different teams fare in the Premier League. We got a jist of what Premier League is and how it works. We saw some interesting trends and some obvious ones (Mike Dean, I’m talking about you)</p><p>Jokes aside, the main objective of this article was to help you practically learn how to choose any dataset and do a simple yet in depth Exploratory Data Analysis using the same. The powerful libraries for python that we used make our job ever so easy! Now all you need to do is pick any dataset of your choice from platforms like Kaggle, Google Datasets etc and build an EDA project worthy for your resume. Good luck on your endavours into this amazing data science world!</p><h3>References (Special Thanks)</h3><ol><li>My <a href="https://www.notion.so/58cf2b852a67449ebdea6a541db0ab90?v=d1c79b65c16741079e11e99102896b29">Notion Notebook</a> full of notes, functions and examples.</li><li><a href="https://matplotlib.org/api/index.html">Matplotlib documentation</a></li><li><a href="https://seaborn.pydata.org/">Seaborn Documentation</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2Fwww.stackoverflow.com">www.stackoverflow.com</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2Fwww.quora.com">www.quora.com</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2FJovian.ai">Jovian.ai</a></li><li><a href="https://www.kaggle.com/irkaal/english-premier-league-results">Kaggle Dataset</a></li></ol><h3>Future Work and Inspirations</h3><ol><li>Building interactive plots.</li><li>EDA on player specific datasets.</li><li>Attack vs Defense strategy stats.</li><li>Predicting results using ML algorithms and DL models.</li><li>Predicting expected goals from a match.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a923499b779" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/lets-learn-exploratory-data-analysis-practically-4a923499b779">Let’s Learn Exploratory Data Analysis Practically</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Garbage Segregation Using PyTorch]]></title>
            <link>https://medium.com/data-science/garbage-segregation-using-pytorch-6f2a8a67f92c?source=rss-f98c4937f418------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f2a8a67f92c</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[resnet]]></category>
            <dc:creator><![CDATA[Aditya Patkar]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 10:58:41 GMT</pubDate>
            <atom:updated>2021-01-18T10:55:34.208Z</atom:updated>
            <content:encoded><![CDATA[<h4>A RESNET approach to Convolutional Neural Networks in PyTorch</h4><p>Humans produce millions of tons of garbage everyday. This garbage, as we know, needs to be segregated before it is taken out of our houses. As essential as this process is for the functioning of communities and sustaining of the mother earth, it is Tedious!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Du4tsqFLsPihw1hPY71YDA.jpeg" /><figcaption>Segregate your thoughts | Photo by <a href="https://unsplash.com/@pawel_czerwinski">Paweł Czerwiński</a> on <a href="http://unsplash.com/">Unsplash</a></figcaption></figure><p>We all have, at some point imagined of having a robot personal assistant who will do all the chores for us, including garbage segregation. I mean, whom are we kidding? We hate to spend minutes staring at 6 garbage bins with a soda can in our hand and wondering where it goes. The dream is not a far fetched thought, but a reality thanks to progress in deep learning.</p><p>Today we will be building one such image classification model using PyTorch. PyTorch is a Python library with a wide variety of functions and operations, mostly used for deep learning. We will train this model by using a Kaggle dataset of 2527 images belonging to 6 different garbage bins. Those being -</p><ol><li>Cardboard</li><li>Glass</li><li>Metal</li><li>Paper</li><li>Plastic</li><li>Trash</li></ol><p>The dataset is obtained from <a href="https://www.kaggle.com/asdasdasasdas/garbage-classification">Kaggle</a>.</p><p>We will use the opendatasets library to download the dataset.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D1-2&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D1-2&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fd1983179318e487b8be043520f8ec4c7%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a720c8058a37532fd49e1bb670809a5b/href">https://medium.com/media/a720c8058a37532fd49e1bb670809a5b/href</a></iframe><p>While downloading the dataset, you will be asked to provide your Kaggle username and credentials, which you can obtain using the “Create New API Token” button on your account page on Kaggle. Upload the kaggle.json notebook using the files tab or enter the username and key manually when prompted.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D4&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D4&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fcadd2c595f8545759084e632e7fbc88d%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/15535cbb3bf5c0ee2a5c8901fa374aec/href">https://medium.com/media/15535cbb3bf5c0ee2a5c8901fa374aec/href</a></iframe><p>Now we’ll import the necessary libraries.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D6&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D6&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F0ea6950e1ac24b488bfaa9c2da992507%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9575ba3d581cf48d49021ff3295cb11/href">https://medium.com/media/d9575ba3d581cf48d49021ff3295cb11/href</a></iframe><h3>Exploring the Data</h3><p>We have downloaded the data. Now we will take a look at the images and see how the data is stored.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D8&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D8&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F60570daa270542a5baa43fdac25bc596%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/320754437b77039344ac4e724eabb5aa/href">https://medium.com/media/320754437b77039344ac4e724eabb5aa/href</a></iframe><p>As we can see, we have images classified into above 6 different categories, or in our case, garbage bins.</p><h3>Apply Transformations</h3><p>Before we can use these images for our model, we’ll have to apply a few transformations. One of those is converting the images to Tensors. A tensor is a number, vector, matrix, or any n-dimensional array. When we convert the images into tensors, PyTorch is able to apply different functions and transformations to it. Before we convert the images into tensors, we will resize or downscale them into images of 64 * 64 pixels. This is to reduce training time and not let our GPU go into overdrive. Transforms can be chained using transforms.Compose as shown below</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D11&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D11&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fce6a36628ff14d16b202ab870ef16cc0%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/278bf43d5aae28a24e2ff61b892c8869/href">https://medium.com/media/278bf43d5aae28a24e2ff61b892c8869/href</a></iframe><h4>Checking the Images</h4><p>Before we define our model, let’s take a look at the data we are working with. We will define a helper function which will display the image and the label corresponding to it. We will use matplotlib, which is a python library used for data visualization.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D14-15&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D14-15&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F75bca3b94c874a95811e96b72ea1c332%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a66133247fed002197554bc3d1391eaf/href">https://medium.com/media/a66133247fed002197554bc3d1391eaf/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D16&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D16&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F82aee606b4d94f35a90832a37114fa4b%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a66b03bba9dc7b5c5459128a8e695a66/href">https://medium.com/media/a66b03bba9dc7b5c5459128a8e695a66/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D17&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D17&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F75fbd67b34cf4d0fb6623e2dd26d12ce%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43764c43374ebe4e521fe30ac57c68a/href">https://medium.com/media/c43764c43374ebe4e521fe30ac57c68a/href</a></iframe><p>As we can see, we are working with 64 * 64 pixel images stored in a single dataset. Let’s check how many samples the dataset contains</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D26&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D26&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F64741e340ac948259b23f05d985b139f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee0aa9d863c3c6864972b8e62fc4f791/href">https://medium.com/media/ee0aa9d863c3c6864972b8e62fc4f791/href</a></iframe><h4>Splitting the Dataset</h4><p>Next step in our quest to segregate garbage is to split the dataset into :</p><ol><li>training dataset — On which the model would train.</li><li>Test dataset — Randomly selected part of the dataset for prediction purposes.</li></ol><p>Ideally, a validation dataset is also created for checking the accuracy of the model but as we have limited number of images, we will use the test dataset as the validation dataset to increase accuracy of our model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D19&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D19&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fe19de723ec13401fa7c608e0e224321b%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/36c6c59c09cdd5e46428de548b777824/href">https://medium.com/media/36c6c59c09cdd5e46428de548b777824/href</a></iframe><p>We have used the random_split function which splits the dataset randomly into parts.</p><h4>Creating Data Loaders</h4><p>Now we will create data loaders. These data loaders load the data into our model in batches.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D21&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D21&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fa334db99e7bc4b7cba981c88fb6fd3c2%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b6832fccded83067edc486f6cb85e90/href">https://medium.com/media/3b6832fccded83067edc486f6cb85e90/href</a></iframe><p>By doing this, data would be fed to the model in batches of 18 images. To visualize a batch of data, we can create a helper function as follows.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D23&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D23&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F3be8d72877c441f8aaf659f1727896f0%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdc3c7cdd9cf26df72c37e461db76820/href">https://medium.com/media/bdc3c7cdd9cf26df72c37e461db76820/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D24&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D24&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F3be8d72877c441f8aaf659f1727896f0%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c2d87e603e13668aaf0c78b983e4802/href">https://medium.com/media/4c2d87e603e13668aaf0c78b983e4802/href</a></iframe><p>As we can see, a batch contains randomly selected images.</p><h4>Using a GPU</h4><p>A GPU uses CUDA cores to accelerate our training process. Thus it is better to shift our data and model to the GPU. We will create helper functions for the same.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D28&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D28&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F76814acd2ba142d89227b7833dcac05f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2d5cc7c12c240c7712fd129f86830dd/href">https://medium.com/media/e2d5cc7c12c240c7712fd129f86830dd/href</a></iframe><p>Here we have defined a few functions which check if the GPU is available or not. If GPU is available, the data is transferred to the GPU for faster processing. The data loaders and model is also moved to the GPU.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D29&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D29&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fe17e69aff34c4070a9aa521b753a280c%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7c31886aec622a7f38b82255eea3d8a/href">https://medium.com/media/c7c31886aec622a7f38b82255eea3d8a/href</a></iframe><p>We can now wrap our training and validation data loaders using DeviceDataLoader for automatically transferring batches of data to the GPU (if available).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D31&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D31&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F430952c5bca34e59a7b934412046bdcb%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9af86c82139016e81126dd98c355ed15/href">https://medium.com/media/9af86c82139016e81126dd98c355ed15/href</a></iframe><h4>Defining the Model</h4><p>Now it’s the time to define our model. We will extend the <strong>nn.module</strong> class which contains functions for training. We will define functions which can be used for any image classification class in the <strong>ImageClassificationBase </strong>class.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D33&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D33&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F865d242bc20b466e9ba0a2211ed0abfe%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/839555e384f50716224fd1cd9110ad26/href">https://medium.com/media/839555e384f50716224fd1cd9110ad26/href</a></iframe><ul><li>The <strong>accuracy</strong> function will help us calculate the accuracy of our model after each epoch.</li><li><strong>training_step </strong>will train our model on the training dataset in batches and compute the loss.</li><li><strong>validation_step </strong>would compute the validation loss by applying our model on the validation dataset.</li><li><strong>validation_epoch_end</strong> and <strong>epoch_end</strong> are used for displaying the losses and accuracy after each epoch.</li></ul><h4>Creating a RESNET CNN</h4><p>We have now defined the base class which contains the functions necessary for training the model. We still haven’t defined our model architecture.</p><p>The model we will be using is called as Convolutional Neural Network. Instead of applying weights to every element of the tensor, it starts with a kernel, which is simply a small matrix of weights. This kernel “slides” over the 2D input data, performing an elementwise multiplication with the part of the input it is currently on, and then summing up the results into a single output pixel. [<a href="https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1">source</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*FHjn4GWkS1JNkBqX2wVEmg.gif" /><figcaption>source : <a href="https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1">https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1</a></figcaption></figure><p>The advantages of using a Convolutional Neural Network are —</p><ul><li><strong>Fewer parameters</strong>: A small set of parameters (the kernel) is used to calculate outputs of the entire image, so the model has much fewer parameters compared to a fully connected layer.</li><li><strong>Sparsity of connections</strong>: In each layer, each output element only depends on a small number of input elements, which makes the forward and backward passes more efficient.</li><li><strong>Parameter sharing and spatial invariance</strong>: The features learned by a kernel in one part of the image can be used to detect similar pattern in a different part of another image.</li></ul><p>To our Convolutional model, we will also add a <strong>Residual Block</strong>. This block adds the original input back to the output feature map obtained by passing the input through one or more convolutional layers. We cannot change output channels in a residual block, because then the input and output shape would change making it impossible to do the addition of both.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faakashns%2F05b-cifar10-resnet%2Fv%2F16%26cellId%3D26&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faakashns%2F05b-cifar10-resnet%2Fv%2F16%26cellId%3D26&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F91b76af913f34fc0908821c55ddd3e26%2Fpreview%2F70a4ec89b61744a58c68b4e241657566%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/15c1f2c7ac509478c1ce38100d81d080/href">https://medium.com/media/15c1f2c7ac509478c1ce38100d81d080/href</a></iframe><p>Above is an example of a simple residual block. Here, a convolutional layer is followed by <strong>ReLU</strong> which is a non linear activation function that retains positive values and sets negative values to zero. Another convolutional layer follows ReLU. We can add input back to the output of this layer and then apply ReLU or vice versa.</p><p>Now we’re ready to define our model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D34&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D34&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F5da139ef242649a28d2183f55168541b%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/98843df060c225649dbddab9c4f88a67/href">https://medium.com/media/98843df060c225649dbddab9c4f88a67/href</a></iframe><p>Here we have defined a convolutional block which has 3 layers. A Convolutional layer which converts number of input channels to number of output channels, a <a href="https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/#:~:text=Batch%20normalization%20is%20a%20technique,required%20to%20train%20deep%20networks.">Batch Normalization</a> layer and a ReLU layer. If pool is set to <em>True, </em><a href="https://computersciencewiki.org/index.php/Max-pooling_/_Pooling#:~:text=Max%20pooling%20is%20a%20sample,in%20the%20sub%2Dregions%20binned.">Max Pooling</a> would be applied after first 3 layers.</p><p>We have extended the ImageClassificationBase class to define our architecture. As you can see, we have 2 convolutional blocks followed by one residual block. This is repeated 2 more times. As each convolutional block would halve the dimensions of our image and double the number of channels, our 3 * 64 * 64 image would get converted to 1024 * 4 * 4 after our final residual layer. We have then applied a classifier block which would convert the number of channel to 6 values which represent probability of belonging to each garbage bin. We can now move our model to the GPU.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D35&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D35&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F8a633407953144f081f84d13dd596008%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5be621c8650bc3462a574e10bc807e8/href">https://medium.com/media/e5be621c8650bc3462a574e10bc807e8/href</a></iframe><h4>Training the Model</h4><p>Now we can define the functions to train our model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D37&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D37&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F7c4ef6aae1da4e259cf36b22a6e4bf48%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b019afc5f48fbdee6d072953ba091fdb/href">https://medium.com/media/b019afc5f48fbdee6d072953ba091fdb/href</a></iframe><ul><li>The <strong>evaluate</strong> function would invoke validation_step() and return the validation loss.</li><li>We have used a few techniques to better our model including<strong> Learning Rate Scheduling</strong>, <strong>Weight Decay </strong>and <strong>Gradient Clipping</strong>. More information about those can be found <a href="https://www.notion.so/Data-Augmentation-Regularization-and-Res-Nets-b30e613133b345a3846f724c897ca2dc#6aa005432297462190d9c0436f2ffc41">here</a>.</li></ul><p>Let’s check our initial accuracy and loss.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D38&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D38&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F310d41fee0394677ae87762ae429e876%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e912ef516007d1376313a55e98dc800f/href">https://medium.com/media/e912ef516007d1376313a55e98dc800f/href</a></iframe><p>As the model has not trained yet, the loss is pretty high while accuracy is very low. We are now ready to train our model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D39-40&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D39-40&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fd60c8f48d6cd41b189a9614e66e51b06%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e830de3bd8379de5e546026e5ed26c64/href">https://medium.com/media/e830de3bd8379de5e546026e5ed26c64/href</a></iframe><p>We have trained our model for 10 epochs with a maximum learning rate of 0.001. We have achieved an accuracy of <strong>80%.</strong> Let’s train the model for 5 more epochs with lower learning rate.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttp%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D41-42&amp;dntp=1&amp;display_name=Jovian&amp;url=http%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D41-42&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b90c1b19f53f8c53d16112ae197f5a80/href">https://medium.com/media/b90c1b19f53f8c53d16112ae197f5a80/href</a></iframe><p>The accuracy is bouncing around 80%, this indicates unless we change our model architecture, introduce new techniques, increase dimensions of image or fine tune hyperparameters, we won’t be reaching a higher accuracy.</p><h4>Evaluating Model Performance</h4><p>We now know that our model is around 80% accurate. This is not the end, it is important to know more about why it is so. We will now plot a few graphs to better understand our model.</p><ol><li><strong>Accuracy Vs Epochs :</strong></li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D44&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D44&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F348c4543a85047c69836afc89774fb58%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b3740652549ee3394a333c9f8cddf77/href">https://medium.com/media/9b3740652549ee3394a333c9f8cddf77/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D45&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D45&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fbd5387ed3da34b869dcf96cbf3e24ed0%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e91ada7bbd7120a7373161c539e7e64e/href">https://medium.com/media/e91ada7bbd7120a7373161c539e7e64e/href</a></iframe><p>It is observed that the accuracy increased by maximum margin in the first epoch, later, the curve has been smoothened.</p><p>2. <strong>Training and Validation Loss vs Epochs :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D47&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D47&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fabaa7529cdeb42ae9297812266c378a9%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b6bdde42334824ab2f153ae4dbfd182d/href">https://medium.com/media/b6bdde42334824ab2f153ae4dbfd182d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D48&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D48&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Ff0c58f1d1ea149cc9aaf9f0ba9aa2ab4%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfcd18546230bd5d967b338dbf7cf1dd/href">https://medium.com/media/dfcd18546230bd5d967b338dbf7cf1dd/href</a></iframe><p>It’s clear from the trend that our model isn’t overfitting to the training data just yet as both the curves have gone down. Try removing data augmentation and residual layers one by one to study their effect on overfitting.</p><p>3. <strong>Learning Rate vs Epochs :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D51&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D51&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Feeebf6c3d6ee4104bb4fd614d776927b%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/296046a83c764828aac498f109c6451d/href">https://medium.com/media/296046a83c764828aac498f109c6451d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D52&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D52&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F65675ab554d943d792839fcca7fcb1e4%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/95f68ae6a9b447d4a7fa89303a748f6e/href">https://medium.com/media/95f68ae6a9b447d4a7fa89303a748f6e/href</a></iframe><p>Remember that we trained the model twice. In both the instances, learning rate increased for about 30% of the epochs and then started decreasing. This is the effect of Learning Rate Scheduler.</p><p>Let’s try predicting the labels for some sample images.</p><h4>Generating test predictions</h4><p>We can now make predictions onthe test dataset.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D56&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D56&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F573e261a33b54378be2826be0d5bc162%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c75a7ba129f68311306869ca0e1147d/href">https://medium.com/media/4c75a7ba129f68311306869ca0e1147d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D57&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D57&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fcf707f56ec684b5cbbc42fc2c0cae5c7%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/81f1ddf7ee192eed36ac2a5e5edb8518/href">https://medium.com/media/81f1ddf7ee192eed36ac2a5e5edb8518/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D61&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D61&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F6da0ae8a62a6460c9d44df28296087ef%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/270973b5951e8a812a8f779b2cf687b0/href">https://medium.com/media/270973b5951e8a812a8f779b2cf687b0/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D62&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D62&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fd917230bab0d4d729aa6f3625b1f96bb%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/198d76dfed1ec0a445367c6344106504/href">https://medium.com/media/198d76dfed1ec0a445367c6344106504/href</a></iframe><p>In above examples, our model has guessed correctly. But our model is not 100% accurate. Below is the example of an image that our model got wrong.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D65&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D65&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2F07eff37a55a5421fb0cf8c0af2b3aaa5%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d70c33b1af2710d0bd2d121aafa386ef/href">https://medium.com/media/d70c33b1af2710d0bd2d121aafa386ef/href</a></iframe><p>As we can see, the model can sometimes give false results.</p><h4>Save the Work</h4><p>Jovian lets us save the notebook and log the hyperparameters for future references.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D69&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D69&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fcfd1fa3a33ee4c34a6c234a62c8adbeb%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/73c38ab398fcb99530c9214bb1948fcc/href">https://medium.com/media/73c38ab398fcb99530c9214bb1948fcc/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D70&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fgarbage-segregation%2Fv%2F2%26cellId%3D70&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F73abc7619fe04bb6a40b2bc20787585c%2Fpreview%2Fffdfe851019c4f0ea1ed4ece4425ea97%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/befa301510941297a4b34fc16c7f5487/href">https://medium.com/media/befa301510941297a4b34fc16c7f5487/href</a></iframe><h4>Conclusion</h4><p>Computation is evolving and deep learning is the latest and greatest computation can give us. Garbage segregation, as tedious as the process is, we are in a reality where not just garbage segregation, but almost any tedious, complex and inhumane process can be automated thanks to Artificial Intelligence. We sure are long way away from conscious robots assisting the human world, but surely, it’s a step in the right direction.</p><h4><strong>Credits :</strong></h4><ol><li>This blog is a part of the project for <a href="http://zerotogans.com/">Deep Learning with PyTorch : Zero to GANs</a> a extensive, informative and really fun free certification course by <a href="http://jovian.ai/">Jovian</a>. Special thanks to Aakash N.S. and the whole Jovian team.</li><li>Dataset Courtesy of : <a href="https://www.kaggle.com/asdasdasasdas/garbage-classification">https://www.kaggle.com/asdasdasasdas/garbage-classification</a></li><li>PyTorch documentation : <a href="https://pytorch.org/">https://pytorch.org/</a></li><li>An executable Jupyter notebook can be found <a href="https://jovian.ai/aditya-pat10/garbage-segregation">here</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f2a8a67f92c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/garbage-segregation-using-pytorch-6f2a8a67f92c">Garbage Segregation Using PyTorch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Powerful PyTorch Functions That Every Beginner Should Know]]></title>
            <link>https://medium.com/data-science/5-powerful-pytorch-functions-that-every-beginner-should-know-e75cb26ec8bc?source=rss-f98c4937f418------2</link>
            <guid isPermaLink="false">https://medium.com/p/e75cb26ec8bc</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tensor]]></category>
            <dc:creator><![CDATA[Aditya Patkar]]></dc:creator>
            <pubDate>Fri, 27 Nov 2020 15:42:15 GMT</pubDate>
            <atom:updated>2020-11-28T18:15:58.340Z</atom:updated>
            <content:encoded><![CDATA[<h3>5 Powerful PyTorch Functions Every Beginner Should Know</h3><h4>Torch your Data Science world alight with these Tensor related functions</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*kzTaAWEN43Mmjl1djjF8oA.jpeg" /><figcaption>“How many Marks did you get in your data science class?” “10 out of tensor” | Photo by <a href="https://unsplash.com/@alesnesetril">Ales Nesetril</a> on <a href="http://unsplash.com/">Unsplash</a></figcaption></figure><p>PyTorch proficiency is one of the most sought after skill when it comes to recruitment for data scientists. For those who don’t know, PyTorch is a Python library with a wide variety of functions and operations, mostly used for deep learning. One of the most basic yet important parts of PyTorch is the ability to create Tensors. A tensor is a number, vector, matrix, or any n-dimensional array.</p><p>Now the question might be, ‘why not use numpy arrays instead?’</p><p>For Deep Learning, we would need to compute the derivative of elements of the data. PyTorch provides the ability to compute derivatives automatically, which NumPy does not. This is called ‘Auto Grad.’ PyTorch also provides built in support for fast execution using GPU. This is essential when it comes to training models.</p><p>All Deep Learning projects using PyTorch start with creating a tensor. Let’s see a few MUST HAVE functions which are the backbone of any Deep Learning project.</p><ul><li>torch.tensor()</li><li>torch.from_numpy()</li><li>torch.unbind()</li><li>torch.where()</li><li>torch.trapz()</li></ul><p>Before we begin, let’s install and import PyTorch</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D1-2&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D1-2&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a209f07b9b1127f62e1d384b5daae18/href">https://medium.com/media/1a209f07b9b1127f62e1d384b5daae18/href</a></iframe><h3>Function 1 — torch.tensor</h3><p>Creates a new tensor. Arguments taken are :</p><ol><li><em>Data</em>: The actual data to be stored in the tensor.</li><li><em>dtype</em>: Type of data. Note that type of all the elements of a tensor must be the same.</li><li><em>device</em>: To tell if GPU or CPU should be used.</li><li><em>requires_grad</em>: If you want the tensor to be differentiable (to compute gradient), set to True.</li></ol><p>Returns a tensor object.</p><p><strong>Example 1 (Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D4&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D4&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e64d47830980e65b8b281e855ba16b1a/href">https://medium.com/media/e64d47830980e65b8b281e855ba16b1a/href</a></iframe><p>This creates a tensor with shape (10,2)</p><p><strong>Example 2 (Working Example) :</strong></p><p>We can also create an empty tensor using the same function.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D6&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D6&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d17e753225015a0542d34dd9477efef/href">https://medium.com/media/0d17e753225015a0542d34dd9477efef/href</a></iframe><p>As we can see, an empty tensor with size (0) is created.</p><p><strong>Example 3 (Error) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D8&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D8&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/401da84117cc44394b16f81c8de75887/href">https://medium.com/media/401da84117cc44394b16f81c8de75887/href</a></iframe><p>A tensor with string values cannot be created.</p><p><strong>Summary :</strong></p><p>torch.tensor() forms the core of any PyTorch project, quite literally, as it forms tensors.</p><h3>Function 2 — torch.from_numpy</h3><p>For those who work with NumPy Arrays, this is a vital function. Using this, one can convert a NumPy Array into a Tensor. It is to be noted that as the tensor and the NumPy Array share the same memory, any changes made to the tensor would be applicable to the NumPy Array and vice versa. The arguments taken are :</p><ol><li>A NumPy ndarray</li></ol><p>Returns a tensor.</p><p><strong>Example 1 (Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D16-17&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D16-17&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6caf2f40bcf99563106bb5732f726d5d/href">https://medium.com/media/6caf2f40bcf99563106bb5732f726d5d/href</a></iframe><p>A NumPy array has been converted into a PyTorch Tensor. We can verify that by checking the type of both a1 and t1 as shown above.</p><p><strong>Example 2 (Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D19&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D19&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/51ec54cf89e1a3aa231156daad9782b3/href">https://medium.com/media/51ec54cf89e1a3aa231156daad9782b3/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D20&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D20&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac54c72d7c36e0962f26cf5815204526/href">https://medium.com/media/ac54c72d7c36e0962f26cf5815204526/href</a></iframe><p>We first created two NumPy Arrays which contained (height, weight) and (heart rate) of 3 students. We created a tuple of both the variables. Then using a for loop, we converted the elements of the tuple to a tensor one by one. In the end, we checked the type of the elements to verify if the function has worked.</p><p><strong>Example 3 (Error) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D22&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D22&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd904036b96171a397c6879ef0f65043/href">https://medium.com/media/cd904036b96171a397c6879ef0f65043/href</a></iframe><p>A NumPy array containing string elements cannot be converted to a tensor. The only supported types for a tensor are : float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.</p><p><strong>Summary :</strong></p><p>This function is extremely useful for people who are using NumPy Arrays for their projects. It shows the interoperability of different Python libraries for data science which makes the language a go to for the majority of data science enthusiasts.</p><h3>Function 3 — tensor.unbind</h3><p>This function removes a dimension from a tensor and returns a tuple of slices of the tensor without the removed dimension. Arguments it takes are :</p><ol><li>A tensor</li><li>Dimension to be removed (0 by default)</li></ol><p>Returns a tuple of slices.</p><p><strong>Example 1(Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D27&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D27&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6a27fc0296a5872995d6fa811631df1/href">https://medium.com/media/c6a27fc0296a5872995d6fa811631df1/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D28&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D28&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed3c0492551f7cb0c7c830d353fb7fa3/href">https://medium.com/media/ed3c0492551f7cb0c7c830d353fb7fa3/href</a></iframe><p>The 0th dimension from (0,1) is removed thus we get 3 slices of the tensor in a tuple</p><p><strong>Example 2 (Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D30&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D30&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f770d4d7f273c394f7420d8deed60ca6/href">https://medium.com/media/f770d4d7f273c394f7420d8deed60ca6/href</a></iframe><p>As seen above, now the data is sliced into data of 10 students for each day and stored in a tuple of tensors, that is, 7 different tensors are created, each corresponding to a day of the week. This sliced data can be used to further apply logic based on that particular day of the week.</p><p><strong>Example 3(Error) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D32&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D32&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/49f478ff630b654c6e120ec76781c7e2/href">https://medium.com/media/49f478ff630b654c6e120ec76781c7e2/href</a></iframe><p>We had created a tensor with 2 dimensions. We provided a value of 2 to the tensor.unbind() function. As 2 corresponds to the 3rd dimension, which doesn&#39;t exist in our example, the error &#39;<em>Dimension out of range</em>&#39; is produced.</p><p><strong>Summary:</strong></p><p>This is a powerful PyTorch function that could be useful when you want to work on particular slices of the data along a dimension of the tensor.</p><h3>Function 4 — torch.where</h3><p>This is a really useful conditional function which depending on a condition, a returns a tensor of elements selected from either x or y tensor. Arguments required are :</p><ol><li>A condition</li><li><em>x</em>: Elements from this tensor are selected for indices where Condition is True</li><li><em>y</em>: Elements from this tensor are selected for indices where Condition is False</li></ol><p><strong>Example 1(Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D37&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D37&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6d5cc7c1cd82cdeb4888fc3f108e14f/href">https://medium.com/media/a6d5cc7c1cd82cdeb4888fc3f108e14f/href</a></iframe><p>Here we gave a condition where the elements from X which were positive, got selected. The elements which were not positive were replaced with elements from Y</p><p><strong>Example 2 (Working Example):</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D39&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D39&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a9c99c6338c91311da4a0116cc549a4/href">https://medium.com/media/6a9c99c6338c91311da4a0116cc549a4/href</a></iframe><p>Here we compared head to head scores and got a tensor containing scores of the winners.</p><p><strong>Example 3(Working Example):</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D41&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D41&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/45ef2076e6b7d28f94a269e2a72f588a/href">https://medium.com/media/45ef2076e6b7d28f94a269e2a72f588a/href</a></iframe><p>It is vital for dimensions and/or size of both the tensors x and y to match for comparison. Otherwise, we get the above error.</p><p><strong>Summary :</strong></p><p>Conditional testing is very important for any data analysis. We might need a particular type of data from two sets of data based on a certain criteria. torch.where() is a useful function for such scenarios.</p><h3>Function 5 — torch.trapz</h3><p>This function estimates the definite integral of y with respect to x along the given dimension, based on <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTrapezoidal_rule">&#39;Trapezoidal rule&#39;</a>.</p><p>The arguments required are :</p><ol><li>y : A tensor containing values of the function to integrate. (blue line in the illustration below)</li><li>x : The points at which the function y is sampled. (x axis in the illustration below)</li><li>Dimension for integration</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*m0Ai25gLZMFGAvVkh8cugQ.png" /></figure><p>Returns a tensor of the same shape as the input (except the dimension removed) where each elements represents the estimated integral along the given dimension.</p><p><strong>Example 1 (Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D46&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D46&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/41e0b89718b2887b9ee9ef9edf3b593b/href">https://medium.com/media/41e0b89718b2887b9ee9ef9edf3b593b/href</a></iframe><p>Here we have the estimated definite integration of y with respect to x.</p><p><strong>Example 2 (Working Example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D48&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D48&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c7099109269852ce963da959a8731b6/href">https://medium.com/media/2c7099109269852ce963da959a8731b6/href</a></iframe><p>We can also provide a dx argument where sample points are spaced uniformly at a distance of dx.</p><p><strong>Example 3 (Error) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D50&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2F01-tensor-operations%2Fv%2F8%26cellId%3D50&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F82094a16f8c948c29fd916bff0ea494c%2Fpreview%2Fa1f779c6a2894c598e86d415f1e011cd%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbba41d0b5888dd8abde199c2d7816da/href">https://medium.com/media/cbba41d0b5888dd8abde199c2d7816da/href</a></iframe><p>As we can see, the dimensions and size of both y and x should match, otherwise our code will produce the above error.</p><p><strong>Summary:</strong></p><p>Deep Learning requires a lot of calculus to know more about the models that are being used. torch.trapz() makes our job of finding estimated integral easy.</p><h3>Conclusion</h3><p>This concludes our look at 5 important PyTorch functions. From basic tensor creation to advanced and lesser known functions with specific usecases like torch.trapz, PyTorch provides many such functions which make job of a data science enthusiast easier. This was a beginner friendly introduction to PyTorch. There is much more. From here it would be a good idea to explore the documentation and create your own tensors. Play around and have some fun. As we get a tighter grip on the basics, we can move forward to Neural Networks and Deep Learning.</p><p>Check out the ongoing course regarding Deep Learning by jovian.ai, <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fzerotogans.com">Zero To Gans</a>.</p><p>Also check out my beginner friendly data analysis of <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fmedium.com%2Fswlh%2Fkickstarter-projects-do-they-succeed-f4a789010585">Kickstarter Projects</a> using Pandas, Matplotlib and Seaborn.</p><p>Check out similar 5 powerful functions for beginners provided by NumPy <a href="https://aditya-pat10.medium.com/5-powerful-numpy-functions-for-beginners-20f4cdb49de9">here</a>.</p><h3>Reference Links</h3><ul><li>Official documentation for tensor operations: <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fpytorch.org%2Fdocs%2Fstable%2Ftorch.html">https://pytorch.org/docs/stable/torch.html</a></li><li>Deep Learning with PyTorch: Zero to GANs by Jovian.ai: <a href="https://jovian.ai/learn/deep-learning-with-pytorch-zero-to-gans">https://jovian.ai/learn/deep-learning-with-pytorch-zero-to-gans</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e75cb26ec8bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/5-powerful-pytorch-functions-that-every-beginner-should-know-e75cb26ec8bc">5 Powerful PyTorch Functions That Every Beginner Should Know</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Powerful NumPy Functions for Beginners]]></title>
            <link>https://medium.com/swlh/5-powerful-numpy-functions-for-beginners-20f4cdb49de9?source=rss-f98c4937f418------2</link>
            <guid isPermaLink="false">https://medium.com/p/20f4cdb49de9</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[jovian]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Aditya Patkar]]></dc:creator>
            <pubDate>Fri, 27 Nov 2020 11:44:05 GMT</pubDate>
            <atom:updated>2020-11-28T04:40:18.987Z</atom:updated>
            <content:encoded><![CDATA[<h3>5 Powerful NumPy Functions a Beginner Should Know</h3><p>Learn exactly what they do, along with working and breaking examples</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lovP7SzPhOtGCJumFGOUCg.jpeg" /></figure><p>NumPy, as the name suggest, is a powerful and open source python library that helps us compute operations on primarily numbers, faster. It is an important tool for data science. NumPy lets us create multi dimensional arrays and lets us perform simple as well as complex operations like indexing, broadcasting, slicing, matrix multiplication to name a few. Today we’ll see how to organize your NumPy arrays better and compute some interesting operations using following functions. The functions are :</p><ul><li>numpy.sort</li><li>numpy.count_nonzero</li><li>numpy.where</li><li>numpy.compress</li><li>numpy.trace</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*kIdvDiy5uPsEkWcHFNJMFw.png" /><figcaption>A powerful library every Data Science Enthusiast must know</figcaption></figure><h3>Function 1 — np.sort</h3><p>This returns a sorted copy of the array. Please note that the original array is not changed. The arguments the function normally takes are -</p><ol><li>The Array to be sorted.</li><li>Axis along which to sort.</li><li>The method/ algorithm used for sorting as “kind = ”.</li></ol><p><strong>Example 1 (Working example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D8&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D8&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F60de83007bfd48ada836693872c0dd90%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb50af716e6416167ced928592c66bdc/href">https://medium.com/media/bb50af716e6416167ced928592c66bdc/href</a></iframe><p>As you can see, the values inside our array were sorted along the rows. The original array ‘a’ is unchanged and we get a sorted copy of the array in ‘a_sorted’.</p><p><strong>Example 2(Working example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D10&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D10&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F60de83007bfd48ada836693872c0dd90%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e40e0fb62bb2a8c441d5fb781028ffbe/href">https://medium.com/media/e40e0fb62bb2a8c441d5fb781028ffbe/href</a></iframe><p>As we can see, sort() was useful to find median of the data. Here we provide axis as <em>None</em> as we are using a flat array.</p><p><strong>Example 3 (Error) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D12&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D12&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b7d70690961760a8c86763cd2778d72/href">https://medium.com/media/3b7d70690961760a8c86763cd2778d72/href</a></iframe><p>We return an error saying that the specified axis is out of bounds. We had given a 2 dimensional array with axis (0,1). As the third dimension doesn’t exist for the given array, we get the error. To fix this error, make sure the axis you specify is within the bounds of the dimensions of the given array.</p><p><strong>Summary :</strong></p><p>Use sort() function when you have a collection or an array of random unordered data but the use case requires you to sort the data in organized manner.</p><h3>Function 2 — np.count_nonzero</h3><p>This function can be used to count number of nonzero elements in a NumPy array. The arguments commonly taken are -</p><ol><li>Array from which the nonzero values are to be found.</li><li>Axis along which number of nonzero values are to be found.</li></ol><p>Returns the number of nonzero items in an array OR along an axis.</p><p><strong>Example 1 (Working example)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D17&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D17&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c13c7fecf463fd19cebd2ec5db9ce2ad/href">https://medium.com/media/c13c7fecf463fd19cebd2ec5db9ce2ad/href</a></iframe><p>We create an identity matrix which has 9 rows and 3 columns. We then found how many nonzero values are present in the given array.</p><p><strong>Example 2(Working example) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D19&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D19&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4e235c6c16bd1d45c050f5ea99ed542/href">https://medium.com/media/e4e235c6c16bd1d45c050f5ea99ed542/href</a></iframe><p>This can be a practical use case for using <em>np.count_nonzero.</em></p><p><strong>Example 3(Error) :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D21&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D21&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4d04ccb92f88709fd1efd5894ba3639/href">https://medium.com/media/b4d04ccb92f88709fd1efd5894ba3639/href</a></iframe><p>As we can see here, the axis provided is out of bounds. Thus we get an error. We should make sure we provide an axis that is in the range as far as dimensions of the array are concerned.</p><p><strong>Summary :</strong></p><p>As we saw, this is another function to organize the data or to take out important information from data. like days present for students.</p><h3>Function 3 — numpy.where</h3><p>This is a very interesting conditional function which lets us test a condition and do one operation on those that satisfy the condition and another on those that do not satisfy the condition. Arguments taken are -</p><ol><li>Condition to be checked.</li><li>x — do x for the values that satisfy the condition.</li><li>y — do y for the values that satisfy the condition.</li></ol><p>Returns an array with x and y executed.</p><p><strong>Example 1 (Working example)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D26&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D26&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aedaa7fce5f45abc406d9e8b7f2ae55/href">https://medium.com/media/1aedaa7fce5f45abc406d9e8b7f2ae55/href</a></iframe><p>We check for the values that are greater than 10. The values that were greater than 10, were kept as they were. The values that were lesser than 10 were multiplied by 10.</p><p><strong>Example 2 (Working example)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D28&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D28&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/66ed4252fbfd343af39281e5805c0fdd/href">https://medium.com/media/66ed4252fbfd343af39281e5805c0fdd/href</a></iframe><p>Here we check for the marks which are less than 45 and mark them as <em>failed</em>, and mark others as <em>passed.</em></p><p><strong>Example 3 (Error)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D30&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D30&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec8e8f0cde2776c38f009615642faa65/href">https://medium.com/media/ec8e8f0cde2776c38f009615642faa65/href</a></iframe><p>We have provided only the x argument along with condition. This is vague and doesn’t tell what to do with values that do not match the condition. We should provide both x and y arguments or neither arguments. When only condition is provided, it acts like np.asarray(condition).nonzero().</p><p><strong>Summary :</strong></p><p>This is a really good function for conditional problems where there are different operations to be performed for those who satisfy a condition and for those that do not.</p><h3>Function 4 — numpy.compress</h3><p>This function returns a slice of values that satisfy a conditional array. This function takes following arguments -</p><ol><li>A 1D conditional array or Boolean array.</li><li>Array on which the compression is done.</li><li>Axis along which the compression is done.</li></ol><p><strong>Example 1 (Working example)</strong>:</p><p>This returns a compressed slice of the original array according to the condition.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D35&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D35&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2651607d64b4ad0170db5218086c4e12/href">https://medium.com/media/2651607d64b4ad0170db5218086c4e12/href</a></iframe><p>Here the items that correspond to 1 from a are selected for the new slice of the array.</p><p><strong>Example 2 (Working example)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D37&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D37&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/95f65191651d10556ade99dd228ae3b7/href">https://medium.com/media/95f65191651d10556ade99dd228ae3b7/href</a></iframe><p>A Boolean array can also be used here. The values corresponding to True are selected and those corresponding to False are not selected.</p><p><strong>Example 3 (Error)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D39&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D39&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f28e3fabc7b30ed4c373e87c9e7daa4c/href">https://medium.com/media/f28e3fabc7b30ed4c373e87c9e7daa4c/href</a></iframe><p>Here the Boolean array contains more items than the array a, thus we get an error. We should’ve used only 3 values in the Boolean array as the last “True” doesn’t correspond to anything.</p><p><strong>Summary :</strong></p><p>This is a good function for slicing the array which in turn lets us focus on important parts of the data.</p><h3>Function 5 — numpy.trace</h3><p>This is a really cool function that returns sum of values along the diagonal of an array. Arguments taken are -</p><ol><li>An array.</li><li>Offset which can be negative or positive.</li><li>Axis. (axis 1 and axis 2 which forms a tringle with the diagonal)</li></ol><p>Returns the sum in float for 2 dimensional Array or if more than 2 dimensions exist, it returns an array with sum along all the diagonals in the array.</p><p><strong>Example 1 (Working example)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D44&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D44&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b92fa9af66cc320418bdbacdd6a09be/href">https://medium.com/media/9b92fa9af66cc320418bdbacdd6a09be/href</a></iframe><p>We created an identity matrix with 9 rows and columns and then calculated sum along the diagonal.</p><p><strong>Example 2 (Working example)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D46&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D46&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c058defae43dc0903d8723bc6c2d1762/href">https://medium.com/media/c058defae43dc0903d8723bc6c2d1762/href</a></iframe><p>The values along the 9 diagonals are stored in the returned array.</p><p><strong>Example 3(Error)</strong>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D48&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fnumpy-array-operations%2Fv%2F10%26cellId%3D48&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F7abd3873354f49a3aa4eb96b68058a8e%2Fpreview%2F4855f3b357da4c32a1cd5d2aa9c3463f%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f1163d3228288a5a3026c1893246f66/href">https://medium.com/media/5f1163d3228288a5a3026c1893246f66/href</a></iframe><p>The offset needs to take an integer value and not float. The offset can be negative or positive which corresponds to below the diagonal and above the diagonal respectively</p><p><strong>Summary :</strong></p><p>This function shows that NumPy is great tool which has cool functions for almost anything!</p><h3>Conclusion</h3><p>This was a good look under the hood of NumPy documentation which is quite vast and almost impossible to know 100%. We found out that there are very useful functions that can do a particular job quickly, efficiently and with less code. That’s the power of NumPy library. We saw functions for organizing data, calculations and learnt real life as well as theoretical examples of the same. We also saw the things to avoid when using these functions. There is still a lot more practice needed to get comfortable with this amazing and powerful library. That would be next along with learning about Pandas, another library I’m really excited about.</p><p>If you liked this, please do <a href="https://medium.com/swlh/kickstarter-projects-do-they-succeed-f4a789010585">check out</a> my article about Kickstarter projects where I do an in depth analysis of success rate of Kickstarter projects using Python, NumPy, Pandas, Matplotlib, Seaborn and a dataset containing 300,000+ Kickstarter projects.</p><h3>Reference Links</h3><ul><li>NumPy official tutorial : <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fnumpy.org%2Fdoc%2Fstable%2Fuser%2Fquickstart.html">https://numpy.org/doc/stable/user/quickstart.html</a></li><li>NumPy Documentation : <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fnumpy.org%2Fdoc%2Fstable%2Freference%2Findex.html">https://numpy.org/doc/stable/reference/index.html</a></li><li><a href="http://jovian.ai/">Jovian.ai </a>Data Analysis with Python: Zero to Pandas : <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fjovian.ml%2Flearn%2Fdata-analysis-with-python-zero-to-pandas">https://jovian.ml/learn/data-analysis-with-python-zero-to-pandas</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20f4cdb49de9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/5-powerful-numpy-functions-for-beginners-20f4cdb49de9">5 Powerful NumPy Functions for Beginners</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kickstarter Projects — Do They Succeed?]]></title>
            <link>https://medium.com/swlh/kickstarter-projects-do-they-succeed-f4a789010585?source=rss-f98c4937f418------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4a789010585</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kickstarter]]></category>
            <dc:creator><![CDATA[Aditya Patkar]]></dc:creator>
            <pubDate>Sun, 01 Nov 2020 07:31:23 GMT</pubDate>
            <atom:updated>2020-11-19T17:59:52.464Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kickstarter Projects — Do They Succeed?</h3><p>A Data Science Perspective</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LNYoFhpDbv8wDsosUtx2vQ.jpeg" /><figcaption>“Fundraising is the gentle art of teaching the joy of giving.” — Henry Rosso</figcaption></figure><p>Kickstarter. Most of us have seen Kickstarter projects be that in social media ads, blogposts or search results. Hell, you can even be one of those who treat Kickstarter like Instagram and browse it daily.</p><p>For the small portion of readers who don’t know what Kickstarter is, Kickstarter is a platform for launching your own products or services through crowdfunding. You can create a page with information about your creative product which can be anything from a tech product to a art masterpiece. You then set a funding goal. This money is then used for different aspects related to making that product come to life.</p><p>As cool as it sounds, Kickstarter projects are not guaranteed to succeed. Through this article, we’ll analyze the past performances of Kickstarter projects based on year of release, category, funding goal etc. The dataset we’re using is a Kaggle dataset comprising information about more than 300,000 Kickstarter projects up to the year 2018. Special thanks to <a href="https://www.kaggle.com/kemical">Mickaël Mouillé</a> for making this <a href="https://www.kaggle.com/kemical/kickstarter-projects">dataset </a>a reality.</p><p>A quick shoutout to <a href="http://www.jovian.ai/">www.jovian.ai</a> and its founder <a href="http://jovian.ai/aakashns">Aakash NS</a> ,who is also the instructor of ‘<strong>Data Analysis with Python: Zero to Pandas</strong>’ which is a free of cost course to get started with data science. The course is extremely thorough and easy to understand. In fact, this project is part of the same course. Do <a href="https://jovian.ai/learn/data-analysis-with-python-zero-to-pandas">check</a> it out if you want to get started with Data Science.</p><p>I have compiled my class notes in a <a href="https://www.notion.so/58cf2b852a67449ebdea6a541db0ab90?v=d1c79b65c16741079e11e99102896b29">Notion notebook </a>folder which can be a handy guide for you. Do check it for a reference while going through the code.</p><p>If you want to play around with the code, an executable Jupyter Notebook can be found <a href="https://jovian.ai/aditya-pat10/kickstarter-projects-do-they-succeed">here</a>. Without further ado, let’s get started.</p><h3>Downloading the Dataset</h3><p>Let’s start by downloading the dataset from Kaggle. Here we use the opendatasets library made for python to download the same. By passing the URL of the Kaggle page ofthe dataset to opendatasets.download(), we will download the dataset to our Jupyter notebook directly.</p><p>Let&#39;s begin by downloading the data, and listing the files within the dataset.</p><pre>dataset_url = &#39;https://www.kaggle.com/kemical/kickstarter-projects&#39;</pre><pre>import opendatasets as od<br>od.download(dataset_url)</pre><p>The dataset has been downloaded and extracted.</p><pre>data_dir = &#39;./kickstarter-projects&#39;</pre><pre>import os<br>os.listdir(data_dir)</pre><p>Output: We have listed the files that have been downloaded. We will use the updated &#39;ks-projects-201801.csv&#39; file for our use.</p><pre><strong>Out </strong>:[&#39;ks-projects-201612.csv&#39;, &#39;ks-projects-201801.csv&#39;]</pre><h3>Data Preparation and Cleaning</h3><p>While working with real world raw data, it is necessary to prepare the data for our analysis. There could be wrong values, missing values in the raw data which needs to be dealt with. Along with that, we might want to add new columns which are useful for our analysis to the dataset or we might want to merge a few datasets together, this should be done as a preliminary step.</p><p>In our case, let’s start by converting the dataset into a Pandas dataframe. Pandas is a python library which gives us handy functions for data cleaning, merging, operations etc. It creates an object called as Data Frame which is basically the data displayed in tabular form. If you know a little bit of coding, a <strong>Data Frame</strong> can be considered as a <strong>dictionary of lists</strong>.</p><p>We can read different types of files eg. CSV, JSON, XLSX etc. and create a Data Frame using the same. To know more about Pandas, check my <a href="https://www.notion.so/Pandas-79a9e56f662240bf990fdc8fde1ad40d">Notion notebook</a>.</p><pre>#importing the pandas library</pre><pre>import pandas as pd</pre><pre>#reading the CSV file that we downloaded</pre><pre>kickstarter_data = pd.read_csv(&#39;./kickstarter-projects/ks-projects-201801.csv&#39;)</pre><p>Let&#39;s take a look at our dataset</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D19&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D19&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ef06062ad14d1fe07669b03357ae5ef/href">https://medium.com/media/0ef06062ad14d1fe07669b03357ae5ef/href</a></iframe><p>We can check the shape of our dataset using the .shape method. It returns a tuple in the form (Number of Rows, Number of Columns) As we can see, the dataset contains information about more than 300000 Kickstarter projects.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D21&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D21&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/89e2ac84ce1e6b68d92c5f5407cebbbb/href">https://medium.com/media/89e2ac84ce1e6b68d92c5f5407cebbbb/href</a></iframe><p>We don’t really need the ID and name columns as they don&#39;t play any part in our analysis. We can remove columns by passing a list of their names to .drop method of the Data Frame. We have to provide axis argument where <strong>0 = rows</strong> and <strong>1 = columns</strong>.</p><pre>kickstarter_data = kickstarter_data.drop([&#39;ID&#39;,&#39;name&#39;], axis =1)</pre><p>Going through the Kaggle discussion, it seems that the pledged column contains the amount pledged by crowd in the currency that it is listed in. It is important that the amount for all the rows is converted to one currency for accurate analysis. The amount converted to USD is listed in two columns i.e usd pledged and usd_pledged_real. Similar can be said about the goal column. Going through more discussions on Kaggle page, we can assume that usd pledged column contains conversion made using Kickstarter which contains a lot of errors. The usd_pledged_real and usd_goal_real column is converted using more accurate Fixer.io API, which we will use. Thus we can drop pledged, goal and usd pledged columns.</p><pre>kickstarter_data = kickstarter_data.drop([&#39;pledged&#39;, &#39;goal&#39;, &#39;usd pledged&#39;], axis =1)</pre><p>Now let’s see how the Data Frame looks like.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D26&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D26&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4badd69def8c600cb645bac385081bc/href">https://medium.com/media/c4badd69def8c600cb645bac385081bc/href</a></iframe><p>The exact time to the accuracy of seconds is not really important for us in launched and deadline column. We can just extract the year, month and day of launch for the same. First let&#39;s convert the columns to datetime objects. Datetime objects are recognized by python as dates which will help us to do further operations.</p><pre>#Converting to DateTime objects</pre><pre>pd.to_datetime(kickstarter_data.launched,format=&#39;%Y-%m-%d&#39;, errors = &#39;coerce&#39;);<br>pd.to_datetime(kickstarter_data.deadline,format=&#39;%Y-%m-%d&#39;, errors = &#39;coerce&#39; );</pre><pre>#Extracting year, month and day of the week</pre><pre>kickstarter_data[&#39;launched_year&#39;] = pd.DatetimeIndex(kickstarter_data[&#39;launched&#39;]).year<br>kickstarter_data[&#39;launched_month&#39;] = pd.DatetimeIndex(kickstarter_data[&#39;launched&#39;]).month<br>kickstarter_data[&#39;launched_day&#39;] = pd.DatetimeIndex(kickstarter_data[&#39;launched&#39;]).weekday</pre><pre>#Dropping the deadline column as we don&#39;t need it.</pre><pre>kickstarter_data = kickstarter_data.drop([&#39;deadline&#39;], axis = 1)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D31&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D31&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3ffaeca21bc935424846103c0f0aee5/href">https://medium.com/media/f3ffaeca21bc935424846103c0f0aee5/href</a></iframe><p>With this, we have completed preparing our data for the analysis. Now, let’s explore the characteristics of the data.</p><h3>Exploratory Analysis and Visualization</h3><p>We can now use the Data Frame to get interesting insights into the data. Let’s compute a few things that will tell us more about the data. For doing so, we will be using two data visualization libraries, namely matplotlib and seaborn. These libraries contain useful methods to plot graphs and charts. Matplotlib provides basic functionality while Seaborn builds on top of matplotlib to give advanced functionality in very few lines of code.</p><p>Learn more about matplotlib and seaborn <a href="https://www.notion.so/Data-Visualization-34175fa561104d0295a6c8d46ea43045">here</a>. You can also go through the documentation for both by simply visiting their respective websites.</p><h4>Data insights using .describe()</h4><p>Before plotting graphs, Let’s first get some insights about our dataset using the .describe() method provided by pandas.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D36&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D36&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4e9ec999cf8bb80c126498a21fa2f98/href">https://medium.com/media/d4e9ec999cf8bb80c126498a21fa2f98/href</a></iframe><p>This gives us some interesting preliminary insights into our data. As we can see,</p><ol><li>The<strong> average goal </strong>is around <strong>45000 USD</strong>.</li><li>The <strong>average pledged</strong> is around <strong>9000 USD</strong> which is way less that our average goal.</li><li>The data for goal varies more with respect to the mean than the data for amount pledged.</li><li>We have data varying all the way from 1970 to 2018. Though <strong>75% of data is between 2013 and 2019</strong>.</li><li>On an average, around 105 people have backed a Kickstarter project but the variance is very high. In fact, <strong>75% of projects have had less than 56 backers</strong>.</li></ol><p>Let’s importmatplotlib.pyplot and seaborn.</p><pre>import seaborn as sns<br>import matplotlib<br>import matplotlib.pyplot as plt<br>%matplotlib inline<br><br>sns.set_style(&#39;darkgrid&#39;)<br>matplotlib.rcParams[&#39;font.size&#39;] = 14<br>matplotlib.rcParams[&#39;figure.figsize&#39;] = (9, 5)<br>matplotlib.rcParams[&#39;figure.facecolor&#39;] = &#39;#00000000&#39;</pre><h4>Comparing number of projects per year using bar chart</h4><p>Let’s plot a basic bar graph that shows number of projects per year. Here we will first sort the data according to the year and month by using .groupby() function. Then We create a new data frame containing the launched_year series, launched_month series and the count which we found using .size() function. We name this final column as counts.</p><pre>#Sorting the data into groups </pre><pre>kickstarter_obj =  kickstarter_data.groupby([&#39;launched_year&#39;, &#39;launched_month&#39;])</pre><pre>#Getting counts of projects in a group using .size()<br>kickstarter_by_year = kickstarter_obj.size().reset_index(name=&#39;counts&#39;)</pre><p>This is how the newly created Data Frame looks like :</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D42&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D42&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f51c5d80121a2842e35858b613da0dc/href">https://medium.com/media/6f51c5d80121a2842e35858b613da0dc/href</a></iframe><p>Now that we have the data frame, let’s plot the bar graph. Here we’re using the barplot() function from the seaborn library. As arguments, we&#39;ll provide launched_year series for x-axis, counts series for y-axis and a data argument. Seaborn has a good support for pandas. By providing a data frame in the &#39;data&#39; argument, seaborn will automatically find the x and y series&#39; from the given data frame.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D44%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D44%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9977acf74e7d93c6ba8a58cacb9c7796/href">https://medium.com/media/9977acf74e7d93c6ba8a58cacb9c7796/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*x715f-mMThaQeqZVHU8YPA.png" /></figure><p>As we can see above,</p><ol><li>The<strong> number of projects</strong> on Kickstarter seem to have <strong>peaked in 2015</strong>.</li><li>The <strong>data for 2018 is incomplete</strong> so we should not take it as definitive. This is assuming that all Kickstarter projects from 2009 onwards are included in the dataset.</li><li>The <strong>data from 1970 seems to be bad or insignificant</strong> data.</li></ol><h4>Plotting a heatmap for yearly and monthly distribution of projects</h4><p>Now let’s prepare our data for a heatmap that shows us the counts in terms of months and years. This time we will use the same groupby object that we created for the previous graph.</p><pre>#sorting the data into groups by year and month</pre><pre>kickstarter_by_year_month = kickstarter_obj.size().reset_index(name=&#39;counts&#39;)</pre><p>Now we use a special funcion provided by pandas called .pivot() which converts the Data Frame into a 2D matrix, taking first argument (series) as rows, 2nd argument being columns and third argument being the values corresponding to series 1 and series 2. Check it out below.</p><pre>#Creating a 2D Matrix<br>kickstarter_by_year_month = kickstarter_by_year_month.pivot(&#39;launched_year&#39;, &#39;launched_month&#39;, &#39;counts&#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D50&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D50&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/56efd337c249fde139427cfba22610e8/href">https://medium.com/media/56efd337c249fde139427cfba22610e8/href</a></iframe><p>Now we plot a heatmap by passing the dataset to the heatmap() function provided by seaborn. Theannot argument can be set to True if you want to show the values in the blocks.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D52%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D52%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/413ecb2df6c6fbdd6a702a734df65cc5/href">https://medium.com/media/413ecb2df6c6fbdd6a702a734df65cc5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*qC5elgYbSrAdcnI3jWW-7g.png" /></figure><p>The above graphs tells us more about the data.</p><ol><li>The data collection started in <strong>April 2009</strong>.</li><li>It seems like the <strong>data entries without launch date were initialized to January 1970 or “Unix Time”</strong>. We should ignore those for statistics involving dates, for other statistics, these entries are still relevant.</li><li>The <strong>data available is only up to January 2018</strong>.</li><li>As the bar plot suggested <strong>amount of projects peaked between 2014 and 2015</strong>. With maximum projects coming in July 2014.</li><li>There seems to be<strong> increase in number of projects during October and November</strong> as compared to August of every year and a decrease in December. We can find why so with research.</li></ol><h4>Goal amount vs pledged amount using line chart</h4><p>Now, let’s explore how the amount pledged by backers compares to the goal set, for every year. First we will group the data by year and then compute the mean using mean() for every year. We will have to reset the index again as the <em>groupby()</em> function sets the series that we use to group the data as the index.</p><pre>#Creating groups and computing mean for every group<br>kickstarter_data_mean = kickstarter_data.groupby(kickstarter_data.launched_year).mean()</pre><pre>#Resetting the index <br>kickstarter_data_mean = kickstarter_data_mean.reset_index(&#39;launched_year&#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D56&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D56&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dbef49a7f230e3237cace75a777eb08/href">https://medium.com/media/2dbef49a7f230e3237cace75a777eb08/href</a></iframe><p>This is how our newly created data frame looks.</p><p>We’ll use Seaborn to plot the line plots by passing in both the series’ to sns.lineplot() function. We can plot both the lines in same graph by writing both sns.lineplot() functions in the same cell as seen below. we will limit the x axis values from 2009 to 2018 as we found out that rows with launched_year = 1970 were defaulted to that value and that would not give us accurate results.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D58%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D58%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/af07785d6c6c6cf0a5c956dc9e7feee5/href">https://medium.com/media/af07785d6c6c6cf0a5c956dc9e7feee5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*LWKuMNoGb-r8yUov_bo2xA.png" /></figure><p>This is a very interesting chart. It tells us that,</p><ol><li>As the <strong>lines do not intersect</strong> at any point, the mean amount pledged does not exceed the mean goal for any year.</li><li><strong>The mean goal</strong>, just like number of projects, seems to have <strong>peaked in 2015</strong> before going down in 2016, 2017.</li><li>The<strong> line of mean amount pledged is fairly flat</strong> indicating that even though projects on the platform increased in 2013–2015, the amount pledged fairly remained the same for any project. For being more sure about this, let’s quickly plot the cumulative sum data instead of mean.</li></ol><pre>kickstarter_data_sum = kickstarter_data.groupby(kickstarter_data.launched_year).sum()<br>kickstarter_data_sum = kickstarter_data_sum.reset_index(&#39;launched_year&#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D61%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D61%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/747e96b4a11228ceec93a24cfb0bb6ef/href">https://medium.com/media/747e96b4a11228ceec93a24cfb0bb6ef/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*mWHYbMM8_xfFZ0muKHuClQ.png" /></figure><p>As we can see, even the line for cumulative sum of amount pledged remains flat for all the years where the goal amount peaked. This might have made many projects fail in the peak years, making the goal amount go down again.</p><h4>Yearwise status comparison</h4><p>Let’s see how the status of projects (successful, failed etc.) compare over the year using bar graph. First we will group the data based on state and launched_year. Then we will count the numbers falling in every category using size() and then change the index to a column named counts finally we will plot the bar plot using seaborn. we will provide launched_year for X axis, counts for Y axis and we will set the hue to state. Using this, different bars for each state corresponding to every year will be plotted.</p><pre>#Grouping the data and getting counts<br>kickstarter_state = kickstarter_data.groupby([kickstarter_data.state, kickstarter_data.launched_year]).size().reset_index(name=&#39;counts&#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D65%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D65%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/089aa0b52e9ebc6c9dc6d54988bada8a/href">https://medium.com/media/089aa0b52e9ebc6c9dc6d54988bada8a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*HGyeSiPoaa8DqjxVG4VB3g.png" /></figure><p>As we can see,</p><ol><li><strong>Failed projects are more than successful projects</strong> almost every year.</li><li>As previous graphs told us, while the amount of goal increased during 2014–2015, the amount pledged remained pretty much similar to previous years. This in-turn increased the gap in this graph between number of successful projects and number of failed projects as the goal was not met.</li><li>Until 2014, number of successful projects was on par with number of failed projects.</li><li>Though if we account cancelled + suspended + failed = unsuccessful, <strong>number of unsuccessful projects for every single year is more than number of successful projects</strong>.</li></ol><h4>Pie chart to analyze the overall success rate</h4><p>Now that we pretty much know that more projects on Kickstarter are unsuccessful as compared to the successful ones, let’s calculate the success rate of the projects.</p><p>First we will group the dataset based on state and then count the number of projects for each state before plotting the pie chart.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D68&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D68&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c69a316e9bf21be337f490f6961c2e94/href">https://medium.com/media/c69a316e9bf21be337f490f6961c2e94/href</a></iframe><p>We should drop the live projects as we don’t know if they eventually succeeded. We will also create a percentage column to display percentage of every state in the chart. round() function takes the value and rounds it down to nearest number with specified (here 2) number of decimals.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D70&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D70&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e855d7f4b7d494e49496ce147ec84d60/href">https://medium.com/media/e855d7f4b7d494e49496ce147ec84d60/href</a></iframe><p>Now let’s plot the pie chart.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D71%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D71%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/81c251f816aecd78d652d548631e0b92/href">https://medium.com/media/81c251f816aecd78d652d548631e0b92/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*7uGeA1NlRdXc8lKlyMENDQ.png" /></figure><p>We can now say that out of all the projects in the dataset, <strong>35.64%</strong> have succeeded while 52.6% have failed. Other states along with failed can be deemed as “unsuccessful” as percentage of ‘<em>undefined</em>’ is negligible. On Kickstarter, unsuccessful projects are more than successful projects.</p><h3>Deep Dive</h3><p>Now we basically know fundamental details about the dataset we have. It’s interesting how just a few lines of code can make a huge dataset containing raw data and lots of values can be made sense of. The visualizations help to a great extend. We started with knowing nothing about the data except the columns it contained. Now just a few moments later, we pretty much know the quantitative answer of the question we asked in the title. It’s really fascinating.</p><p>Now we can answer a few more interesting questions about the dataset which can help us understand the data even better.</p><p><strong>Q1: Which main categories have more successful projects than failed ones?</strong></p><p>To answer this, we can group the data based on main_category and state and count the items for each row. We’ll call that column projects. Then we can plot a bar chart with projects on the X axis and main_category on the Y axis with state set as hue to compare and find out the answer.</p><pre>kickstarter_category = kickstarter_data.groupby([kickstarter_data.main_category, kickstarter_data.state]).size().reset_index(name = &#39;projects&#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D78&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D78&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d4880e2215d3e9b5e147cb1eae707ce/href">https://medium.com/media/2d4880e2215d3e9b5e147cb1eae707ce/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D79%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D79%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/faf0056d78e52f59ec2e93d4fbee7670/href">https://medium.com/media/faf0056d78e52f59ec2e93d4fbee7670/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ei9iN6b7ZXnlTZMVLzJOnw.png" /></figure><p><strong>Answer</strong> — <strong>Comics, Dance, Music and Theatre </strong>are the main categories where successful projects are more than failed projects. Insights :</p><ol><li>Looks like <strong>performance arts</strong> projects are succeeding more than they are failing.</li><li>Art, Crafts, Design, Fashion, Film and Video, Food, Games, Journalism, Photography, Publishing and Technology have more failed projects than successful ones.</li><li>Film and Video has the largest number of failed projects.</li></ol><p><strong>Q2: Which category had the most projects?</strong></p><p>Let’s group the data based only on main_category and then use the size() function to calculate the sum. Then we&#39;ll plot a basic bar chart.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D82&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D82&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cfdbe3bdbcf410c4e039da6ce1d52c4/href">https://medium.com/media/4cfdbe3bdbcf410c4e039da6ce1d52c4/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D83%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D83%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c99e839e8c1c07c16e00902bea647ae2/href">https://medium.com/media/c99e839e8c1c07c16e00902bea647ae2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yFmHnRTQH3KRz-esTlnuUA.png" /></figure><p><strong>Answer</strong> — Clearly <strong>film and video</strong> projects are maximum followed by music, which is followed by publishing and technology.</p><p><strong>Q3: Which category has the highest goal amount per project?</strong></p><p>For that, we will group the data based on main_category and compute the sum of usd_goal_real for every column before dividing it by number of projects for each category. Finally, we will plot a bar chart to compare the same.</p><pre>kickstarter_category2 = kickstarter_data.groupby(kickstarter_data.main_category).sum().reset_index(&#39;main_category&#39;)<br>kickstarter_category2[&#39;goal_per_project&#39;] = kickstarter_category2[&#39;usd_goal_real&#39;] / kickstarter_category1[&#39;projects&#39;]</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D87&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D87&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/fed66ddaa6c1bd96d4baea1caebba41a/href">https://medium.com/media/fed66ddaa6c1bd96d4baea1caebba41a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D88%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D88%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/24ae75cae7137190c3878058b36fc721/href">https://medium.com/media/24ae75cae7137190c3878058b36fc721/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_qYVyk86gBI6e6hN6TkXQ.png" /></figure><p><strong>Answer</strong> — The most amount of goal per category in USD is for <strong>technology</strong>. This is a really interesting chart.</p><ol><li>Even though number of projects is maximum for Film and Video, goal set by the project creator for technology exceeds that of any other category.</li><li>Journalism comes second with Film and Video following it.</li><li>The least goal is for dance.</li></ol><p>Let’s compute the same for amount pledged and add dots for goals using sns.scatterplot() for comparing.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D91%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D91%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/95806eeed101c1222a3d2588087ac2af/href">https://medium.com/media/95806eeed101c1222a3d2588087ac2af/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zP7hExceydbmNk9v-Toyaw.png" /></figure><ol><li>Interestingly, pledged amount per project for design exceeds that of technology.</li><li>Games which was not prominent in previous statistics suddenly rises to third spot in the graph.</li><li>Goal amount per project for every category is higher than the pledged amount per project.</li><li>It is still not clear where the difference between goal per project and pledged per project is maximum.</li></ol><p><strong>Q4: Which category has the maximum difference in goal amount and pledged amount per project?</strong></p><p>We will simply compute the difference between goal_per_project and pledged_per_project and then plot the graph of difference.</p><pre>kickstarter_category2[&#39;Goal_pledged_difference&#39;] = kickstarter_category2[&#39;goal_per_project&#39;] - kickstarter_category2[&#39;pledged_per_project&#39;]</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D95%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D95%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bff3635b7ce60e207f639cdb28e02223/href">https://medium.com/media/bff3635b7ce60e207f639cdb28e02223/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DazVtuIP-O2qXdBko9NZOw.png" /></figure><p><strong>Answer</strong> — The difference between Goal per Project and Pledged per Project is maximum for <strong>Journalism</strong>.</p><ol><li>Journalism is followed by Technology and Film and Video.</li><li>The difference is lowest for dance followed by photography.</li></ol><p><strong>Q5: Which category has maximum pledged amount per backer?</strong></p><p>We will calculate this by dividing usd_pledged_real by backers for every category. Finally we will plot a bar chart.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D98&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D98&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c231bf9a69eb2b7664e3b379340b211/href">https://medium.com/media/7c231bf9a69eb2b7664e3b379340b211/href</a></iframe><pre>#Calculating amount pledged per backer<br>kickstarter_category3[&#39;pledged_per_backer&#39;] = kickstarter_category3[&#39;usd_pledged_real&#39;] / kickstarter_category3[&#39;backers&#39;]</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D100%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Faditya-pat10%2Fkickstarter-projects-do-they-succeed%2Fv%2F26%26cellId%3D100%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fa82ce55c1de247619c9b6f0572dd0b57%2Fpreview%2F7ccb396e096f4fb0a6ba6f3f8f5f5af1%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bfa0b923cad8b8b06ac0e244895e93c/href">https://medium.com/media/9bfa0b923cad8b8b06ac0e244895e93c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-DhCj4r-lv1CdJPCMDZHw.png" /></figure><p><strong>Answer </strong>— Pledged per Backer is maximum for <strong>Technology</strong>.</p><ol><li>Technology category is a clear standout here with more than USD 120 per backer.</li><li>Comics has the lowest pledged amount per backer with just over USD 50 per backer.</li></ol><h3>Conclusion</h3><p>Kickstarter is an amazing platform for crowdfunding innovative and creative projects. It is the truth that most of the projects fail rather than succeeding but looking at it positively, nearly 36% of projects have succeeded on kickstarter. Businesses do fail in real life thus it is important to compare success rate with other modes of raising capital. This can be done as a future work. Bottom line is, even though the goal set by most projects is not achieved on kickstarter, it is an easy way to raise capital. It might even be that people who launch projects set the goal higher than necessary. We need more research to imply that.</p><p>The bigger picture is, we learned a lot about data today. How raw data can be made sense of, how visualizations play an important role in analyzing data and how great python libraries are! :)</p><h3>References (Special Thanks)</h3><ol><li>My Notion Notebook : <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fwww.notion.so%2F58cf2b852a67449ebdea6a541db0ab90%3Fv%3Dd1c79b65c16741079e11e99102896b29">https://www.notion.so/58cf2b852a67449ebdea6a541db0ab90?v=d1c79b65c16741079e11e99102896b29</a></li><li>Matplotlib documentation : <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fmatplotlib.org%2Fapi%2Findex.html">https://matplotlib.org/api/index.html</a></li><li>Seaborn Documentation : <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fseaborn.pydata.org%2F">https://seaborn.pydata.org/</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2Fwww.stackoverflow.com">www.stackoverflow.com</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2Fwww.quora.com">www.quora.com</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2FJovian.ai">Jovian.ai</a> and Aakash NS : <a href="http://www.jovian.ai/aakashns">www.jovian.ai/aakashns</a></li><li><a href="https://jovian.ai/outlink?url=http%3A%2F%2FJovian.ai">Jovian.ai</a> Zero to Pandas course : <a href="https://jovian.ai/learn/data-analysis-with-python-zero-to-pandas">https://jovian.ai/learn/data-analysis-with-python-zero-to-pandas</a></li><li>Kaggle Dataset : <a href="https://www.kaggle.com/kemical/kickstarter-projects">https://www.kaggle.com/kemical/kickstarter-projects</a></li></ol><h3>Future Work</h3><ol><li>Comparing success rate with other means of raising capital.</li><li>Comparison based on country.</li><li>Kickstarter vs Indiegogo</li><li>Predicting if a Kickstarter project would succeed or not.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4a789010585" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/kickstarter-projects-do-they-succeed-f4a789010585">Kickstarter Projects — Do They Succeed?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>