<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ANKIT PRATAP SINGH on Medium]]></title>
        <description><![CDATA[Stories by ANKIT PRATAP SINGH on Medium]]></description>
        <link>https://medium.com/@ankit-pratap-singh?source=rss-14d49fd473b9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3KUK2euGNCGyKPA-i3jyAQ.jpeg</url>
            <title>Stories by ANKIT PRATAP SINGH on Medium</title>
            <link>https://medium.com/@ankit-pratap-singh?source=rss-14d49fd473b9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:51:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ankit-pratap-singh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MongoDB: A NoSQL Database]]></title>
            <link>https://ankit-pratap-singh.medium.com/mongodb-a-nosql-database-f3f6cc35d034?source=rss-14d49fd473b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3f6cc35d034</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[nosql-database]]></category>
            <category><![CDATA[pymongo]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ANKIT PRATAP SINGH]]></dc:creator>
            <pubDate>Mon, 01 Aug 2022 14:41:46 GMT</pubDate>
            <atom:updated>2022-08-01T14:43:58.884Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Science</h4><h4>And how to use it using PyMongo.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*vszdMtvmEvaD9m07.png" /></figure><pre><strong>Index Of Contents</strong><br>· <a href="#b412">Introduction</a><br>· <a href="#bcfe">How MongoDB stores data</a><br>· <a href="#ef06">Why and When to use MongoDB</a><br>· <a href="#e08e">Let’s talk about the CRUD operations in MongoDB (with python):</a><br>  ∘ <a href="#4239">PyMongo</a><br>  ∘ <a href="#3233">Create operations</a><br>  ∘ <a href="#3b31">Insert Operations</a><br>  ∘ <a href="#62bc">Query Inserted Documents</a><br>  ∘ <a href="#b137">Query with Filter</a><br>  ∘ <a href="#b8f2">Comparison Operator</a><br>  ∘ <a href="#6cdd">String Filter</a><br>  ∘ <a href="#c823">Update Data Collections</a><br>  ∘ <a href="#794d">Delete or Drop</a><br>· <a href="#58e9">Conclusion</a><br>· <a href="#9bb2">References</a></pre><h3>Introduction</h3><p>MySQL/SQL is a Relational Database, were designed to store structured data in tabular format. MySQL was first launched in the year 1965 and is now managed by Oracle. SQL databases were the only viable database storage solution until the 2000s when the internet and web 2.0 booms started to generate a large amount of data. SQL databases failed to store all that unstructured data. Such unstructured data could not be mapped into a table-like schema and thus arose a need for another different class of database engine.</p><p>MongoDB is used to save unstructured data in JSON format. This is when NoSQL databases started arriving on the scene.</p><p>MongoDB is a No SQL database engine. It was first launched in 2009 and has since become one of the leading databases in the NoSQL space.</p><p>In this article, we will learn about how to create, read, update and delete (CRUD operations) data in the MongoDB database engine with python. But first, let’s dig a little bit deeper into how MongoDB stores data and in what cases it is more useful than MySQL databases.</p><h3>How MongoDB stores data</h3><p>In MySQL data is stored in tables, where each table consists of rows and columns, and the columns of the table represent the attribute, and the rows of the table represent a particular record in the table. These tables, in turn, reside inside the databases.</p><p>While in MongoDB, data is stored in collections of documents where the data is present in form of JSON format (i.e. key-value pair). A collection can consist of many documents, and there can be thousands of such collections inside of a MongoDB database.</p><p>SQL databases have a relational property where different tables are related to each other with foreign keys and primary keys. That’s why SQL databases are called Relational Databases, while in MongoDB we can not establish any relationship between the unstructured collections of data. And that’s why such databases are called Non-Relational Databases.</p><h3>Why and When to use MongoDB</h3><p>MongoDB is an attractive option for developers due to its data storage philosophy being simple and easy to understand for anybody having programming experience. MongoDB stores data in collections with no fixed schema, this flexible approach to storing data in such data collections makes it suitable for developers who may not be database experts, yet want to use a database to support the development of their applications.</p><p>MongoDB databases are commonly used in environments where flexibility is required with big and unstructured data with ever-changing schemas. One of the most common use cases of MongoDB database is in companies where tons of data is being generated day by day which is also called big data. Some of the other use cases of MongoDB databases are-Customer management systems, customer analytics, product management system, and real-time data integration that requires a large volume of high-speed data logging and aggregation.</p><h3>Let’s talk about the CRUD operations in MongoDB (with python):</h3><p>Similar to MySQL, MongoDB also comes with its different products wiz MongoDB Community Server, MongoDB Shell, MongoDB Compass, and MongoDB Atlas. Here I am writing a small introduction of each of its products.</p><ul><li><strong>MongoDB Community Server:</strong> MongoDB Community Server offers a flexible document data model along with support for ad-hoc queries, secondary indexing, and real-time aggregations to provide powerful ways to access and analyze your data. It’s the database engine that stores data in different data collections.</li><li><strong>MongoDB Shell:</strong> The new MongoDB Shell lets you connect to MongoDB to work with your data and configure your database. With its enhanced usability features (intelligent autocomplete and syntax highlighting, easy-to-understand error messages, and contextual help), it’s the quickest way to work with MongoDB and Atlas.</li><li><strong>MongoDB Compass: </strong>MongoDB compass is made for easy work with your data, the GUI is built by-and-for MongoDB. Compass provides everything from scheme analysis to index optimization to aggregation pipelines in a single, centralized interface.</li><li><strong>MongoDB Atlas: </strong>MongoDB Atlas is the multi-cloud developer database storage platform. You can deploy your database across Microsoft Azure, AWS, and Google Cloud using MongoDB Atlas, it’s easy and secure to deploy your database.</li></ul><h4><strong>PyMongo</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SOL83Gor_lH3EhW-.jpeg" /></figure><p>PyMongo is a python distribution containing tools for working with MongoDB in python and is the recommended way to work with MongoDB from python. In this article, I will try to explain the CRUD operation of MongoDB from python.</p><blockquote>You can read the detailed documentation of PyMongo by visiting their official documentation <a href="https://pymongo.readthedocs.io/en/stable/">website</a>.</blockquote><p>To start this journey you first have to install PyMongo in your virtual environment. To install PyMongo you can run the command and then you can import it into your python script or notebook.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D3&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D3&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/041ffa6f1e19eab5a1967a9827818b50/href">https://medium.com/media/041ffa6f1e19eab5a1967a9827818b50/href</a></iframe><p>Once the PyMongo package gets imported into your notebook, now you can head towards the getting database and data collection, Before we dive into data pulling from the database, one thing I want to clarify to all readers is that the data tables in MongoDB are called Data-Collections and each data row is called a document in MongoDB, so don’t get confused with the terms as data collections and data document.</p><p>To connect to a database server you need to import MongoClient from pymongo, the function <strong><em>MongoClient() </em></strong>lets you connect with the database present in your system locally,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D7&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D7&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a81d547afcb16a349a8d9dc0e98317dc/href">https://medium.com/media/a81d547afcb16a349a8d9dc0e98317dc/href</a></iframe><p>If you want to connect with a specific database then you have to pass the <strong><em>host </em></strong>and <strong><em>port </em></strong>in the following way:</p><pre>client = MongoClient(&#39;mongodb://localhost:27017/&#39;)</pre><p>Or you can pass the connection URL to connect your script with the MongoDB Atlas Cloud database. The MongoDB Atlas data cluster URL looks as follows:</p><pre>mongodb://&lt;username&gt;:&lt;password&gt;@ac-ahziyw0-shard-00-00.quzykiv.mongodb.net:27017,ac-ahziyw0-shard-00-01.quzykiv.mongodb.net:27017,ac-ahziyw0-shard-00-02.quzykiv.mongodb.net:27017/?ssl=true&amp;replicaSet=atlas-2tdw68-shard-0&amp;authSource=admin&amp;retryWrites=true&amp;w=majority</pre><p>In the above URL, you have to pass your username and password.</p><p>Once you get connected with your data cluster/database, now you can create a new data collection (data tables) or use existing data collection, insert the data rows, update the data rows, or you can delete the data rows, and delete the data collections.</p><h4>Create operations</h4><p>To create a new database, or new data tables let’s practice the below commands:</p><pre># to print a list of available databases<br>client.list_database_names()</pre><pre># to select an existing database, or to create a new database if not exist<br>db = client.db_name</pre><pre># to print a list of available data tables<br>db.list_collection_names()</pre><pre># to select an existing data table, or to create a new data table if not exist<br>table = db.table_name</pre><h4>Insert Operations</h4><p>Once you create a new data table, now you can insert data in the data table, as MongoDB stores data in <strong>JSON </strong>format, so we also have to parse the data in JSON/python dictionary format. Let’s practice the following commands:</p><pre># first create a data dictionary<br>name = {&#39;Name&#39;:&quot;Ankit Pratap Singh&quot;,<br>       &#39;Role&#39;:&quot;Owner&quot;,<br>       &#39;company&#39;:&quot;&quot;}</pre><pre># to insert into table<br>table.insert_one(name)</pre><p>Like SQL, MongoDB also requires a specific “<strong><em>key: value”</em></strong> pair, which identifies a particular data row or say data document. SQL requires a primary key column, if not mentioned then SQL automatically creates an index ID column. MongoDB also generates an ID, “<strong><em>key: value</em></strong><em>” </em>pair, if it wasn’t provided, but if you provide it, MongoDB will save it as an identifier of that individual document. Like in the above cell we have inserted a data row in the table, and now if we fetch the inserted we will get the following output:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D23&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D23&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9f92fcf1681f76d322970cd8a76c44e/href">https://medium.com/media/a9f92fcf1681f76d322970cd8a76c44e/href</a></iframe><p>In the output we can see, that there is an extra “<strong><em>key: value”</em></strong> pair of “<em>_id: ObjectId(‘62d8df307118397f5930be04’)</em>”, this is a unique identifier of the inserted document in the table. Now if you provide it while inserting the data in the table, it won’t save an auto-generated Id itself. There is a way to get an auto-generated id without the “<em>ObjectId” </em>class, have a look at the following code snippet:</p><pre>import bson</pre><pre># create a list of dictionary<br>data_list = [{&#39;Name&#39;:&quot;Ankit Pratap Singh&quot;,<br>            &#39;Role&#39;:&quot;CEO&quot;,<br>           &#39;Comepany&#39;:&quot;XYZ&quot;,<br>           &#39;_id&#39;: str(bson.ObjectId())},<br>        <br>          {&#39;Name&#39;:&quot;Ankit Pratap Singh&quot;,<br>          &#39;Role&#39;:&quot;CTO&quot;,<br>          &#39;Company&#39;:&quot;ABC&quot;,<br>          &#39;_id&#39;: str(bson.ObjectId())}]</pre><pre># To insert multiple data rows<br>result = table.insert_many(name)</pre><p>Now let’s see the output of the above-inserted documents.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D34&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D34&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/54ff2b53754c1615450de2f978b1805f/href">https://medium.com/media/54ff2b53754c1615450de2f978b1805f/href</a></iframe><p>Now here we have auto-generated ObjectIds, but without ObjectId class mentioned in the document.</p><h4>Query Inserted Documents</h4><p>A query in a database is a request for information from a database. A query response usually returns data from different data tables within the database. Queries in MongoDB, are a little bit different from SQL queries, but they are simple and easy to learn, you can read the official documentation by visiting MongoDB queries <a href="https://www.mongodb.com/docs/manual/tutorial/query-documents/">webpage</a>.</p><blockquote>There are ways to query the data from data collections in MongoDB, we will focus here on <strong><em>data query in MongoDB using Python</em></strong>,</blockquote><p>Let’s have a look at the following code snippets:</p><pre>import pprint</pre><pre># to get one row<br>pprint.pprint(table.find_one())</pre><pre># to get all data rows<br>for name in table.find():<br>     pprint.pprint(name)</pre><p>Here “pprint” is a module from PyMongo, which lets you print the queried document as a plain JSON document.</p><h4>Query with Filter</h4><p>Filtering data is an important part of querying data from the database because we don’t want all the data every time. So we need some filter methods in query sets so that we can fetch the required data. To show query filters I have created some tables and inserted some data from excel sheets. You can find the excel sheet that I am using in the notebook <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2Fe%2F2PACX-1vRAJiiMrrG9H7XKqCRjlnFY176SVTuSXWyZstvCRzA6W0uGJSK8wtV7ONPexjr-ErG7D_AP1x_5q4i4%2Fpub%3Foutput%3Dxlsx">here</a>.</p><p>PyMongo provides some methods for data filtering in queries, these filters are called <strong>Projection Operators </strong>in PyMongo, you can read them in detail by visiting the official page of MongoDB for <a href="https://www.mongodb.com/docs/manual/reference/operator/query/">Query and Projection Operators</a>. There are classes of project operators available in MongoDB, some of them are as follows:</p><ul><li>Comparison Operators</li><li>Logical Operators</li><li>Evaluation Operators</li><li>Element Operators</li><li>Geospatial Operators</li></ul><p>I will not go into deep for each class, but I’ll describe 2–3 types of them.</p><h4>Comparison Operator</h4><p>PyMongo provides a number of comparison operators such as <strong>$gte</strong>, <strong>$gt</strong>, <strong>$in</strong>, and <strong>$nin, </strong>they are greater than equal to, greater than, in the given array, $nin matches none of the elements in the given array, etc.</p><p>Here I am writing some code to demonstrate how to apply these operators:</p><pre>#in authors table,how many authors spend more than 10 hrs </pre><pre>for author in authors_table.find({&#39;Hrs Writing per Day&#39;:{&#39;$gte&#39;:10.0}}):<br>    pprint.pprint(author)</pre><h4>String Filter</h4><p>Now let’s try to filter our query set with some string filters. Let’s have a look on the following code snippet</p><pre>#in authors table, how many authors are from United States<br>for author in authors_table.find({&#39;Country of Residence&#39;:&#39;United States&#39;}):<br>    pprint.pprint(author)</pre><pre># you can limit your query set as well<br>for author in authors_table.find({&#39;Country of Residence&#39;:&#39;United States&#39;}).limit(4):<br>    pprint.pprint(author)</pre><h4>Update Data Collections</h4><p>Updating valuesis the third most important pillar of working with any database. Similar to SQL database we have methods to update data collections in MongoDB as well.</p><p>There are two main types of update in database:</p><ul><li>To insert new fields/columns in any existing data table</li><li>To replace values for predefined column in any table</li></ul><p>Sometime we need to add one or more new data columns in predefined data tables, Let’s add a new column in <strong>authors </strong>table. To update any data collection in MongoDB, PyMongo provides two methods as follows:</p><pre>db.collection.update_one()<br>db.collection.update_many()</pre><p>We will update many values with adding one new column.</p><pre>column = authors_table.update_many({}, {&#39;$set&#39;:{&#39;Created&#39;:str(datetime.datetime.now(IST))}},)</pre><p>The above code will add a new data column in the authors table. to verify let’s have a look on the following code embed</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D88&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D88&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/be6be8bcccd995e236eff25ed4fe6a8d/href">https://medium.com/media/be6be8bcccd995e236eff25ed4fe6a8d/href</a></iframe><p>Now let’s try to replace any value from any column in an existing data collection. We will apply the following update in the <strong>edition</strong> table</p><ul><li>Hardcover =&gt;Hardcover Book,</li></ul><pre>edition_table.update_many({&#39;Format&#39;:&quot;Hardcover&quot;},{&#39;$set&#39;:{&#39;Format&#39;:&quot;Hardcover Book&quot;}})</pre><h4>Delete or Drop</h4><p>This is the fourth foundation pillar to learn working with database. Sometimes you need to delete some records from your database,like you need to delete some data rows only, or you want to delete some data columns, or tables, or want to delete entire database.</p><blockquote>Note: Be careful about these operation. Delete operation in some cases are very sensitive, like if you made a wrong step and you can loose all of your data, so always have backup of your database.</blockquote><p>There are operations to perform <strong>Delete</strong> or <strong>Drop</strong></p><ul><li>db.collection.delete_one(): It deletes only one which occurs at very first position in the collection,</li><li>db.collection.delete_many(): It deletes a number of document that matches the condition that you will provide in the argument of function, or if you don&#39;t provide the filter it will delete all the documents present in the data collection,</li><li>db.collection.drop(): It removes entire data collection from your database</li></ul><p>Let’s have a look on following example,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D97%26hideOutput%3Dtrue&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D97%26hideOutput%3Dtrue&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f08264b28f99122339e82bdd12c6536/href">https://medium.com/media/3f08264b28f99122339e82bdd12c6536/href</a></iframe><p>And let’s verify the above if that worked on table,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D100&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fankit-singh%2Fpymongo-practice%2Fv%2F19%26cellId%3D100&amp;image=https%3A%2F%2Fapi.jovian.ai%2Fapi%2Fgist%2F94897f3318684b9ba0fc38eacf34b368%2Fpreview%2Fb57f4c637b0b46d58b2f2594792ee104&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cf156c3c00d5b9c38f18a7256fc0371/href">https://medium.com/media/2cf156c3c00d5b9c38f18a7256fc0371/href</a></iframe><p>As you can see that there is no document containing, Role:CTO. Now let’s have a look at the following code snippet</p><pre>len(list(authors_table.find({&#39;Country of Residence&#39;:&quot;United States&quot;})))<br>Output: 9</pre><pre>authors_table.delete_many({&#39;Country of Residence&#39;:&quot;United States&quot;})</pre><pre>len(list(authors_table.find({&#39;Country of Residence&#39;:&quot;United States&quot;})))<br>Outut: 0</pre><p>that’s done, we have deleted all the rows , where country of residence was the United States from the author&#39;s table.</p><p>You can find more examples on MongoDB&#39;s official documentation <a href="https://www.mongodb.com/docs/manual/tutorial/remove-documents/">webpage</a>.</p><h3>Conclusion</h3><p>That’s all about <strong>CRUD </strong>operations in MongoDB using PyMongo, although most of these operations are same to use in <strong>MongoShell</strong>, or in <strong>MongoDB Compass</strong>,</p><p>In the above entire article we have covered so far:</p><ul><li>Connection to MongoDB Database</li><li>Creating database</li><li>Creating tables/data collections in database</li><li>Insert data rows, or we should say data documnets</li><li>Data query and how to print them in our terminal or output cell of a jupyter notebook.</li><li>Data query with filters</li><li>Update tables, update columns and replace values in a table</li><li>Delete data columns, rows, and delete data tables.</li></ul><h3>References</h3><p>First of all a big thanks to <a href="https://aakashns.medium.com/?source=collection_about-------------------------------------">Aakash N S</a>, and <a href="https://medium.com/@sidujjain?source=collection_about-------------------------------------">Siddhant Ujjain</a>, for helping me in this entire journey of leaning about data science. a big thanks to my Code Hunters team, who always guide me whenever I stuck at some points.</p><p>To learn more about MongoDB, you can follow along the following links</p><ul><li><a href="https://www.youtube.com/watch?v=ABAOvvBh99w">Installing MongoDB Community Server on your system</a></li><li>Read the PyMongo tutorial documentation <a href="https://pymongo.readthedocs.io/en/stable/tutorial.html">here</a></li><li>Read the application development with advanced MongoDB <a href="https://www.mongodb.com/docs/develop-applications/">here</a></li><li>You can go through the entire jupyter notebook, that I have used in the article <a href="https://jovian.ai/ankit-singh/pymongo-practice">here</a></li><li>You can <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2Fe%2F2PACX-1vRAJiiMrrG9H7XKqCRjlnFY176SVTuSXWyZstvCRzA6W0uGJSK8wtV7ONPexjr-ErG7D_AP1x_5q4i4%2Fpub%3Foutput%3Dxlsx">download</a> the excel sheet that I have used to write examples.</li></ul><p>Thanks for reading the article. I hope, I helped you to learn MongoDB, and how to work with PyMongo. If you have any doubt you can reach out to me on my <a href="https://www.linkedin.com/in/ankitpratap-singh/">LinkedIn</a> profile.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3f6cc35d034" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Data Analysis of Covid-19 Vaccinations]]></title>
            <link>https://ankit-pratap-singh.medium.com/exploratory-data-analysis-of-covid-19-vaccinations-d6aa9ce1fdb0?source=rss-14d49fd473b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6aa9ce1fdb0</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ANKIT PRATAP SINGH]]></dc:creator>
            <pubDate>Thu, 02 Sep 2021 14:12:25 GMT</pubDate>
            <atom:updated>2022-03-12T14:00:42.554Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Science</h4><h4>Using Python, Pandas, and Plotly</h4><figure><img alt="Exploratory Data Analysis" src="https://cdn-images-1.medium.com/max/1024/0*XABQIC3aUjTwW59g" /><figcaption>Photo by <a href="https://unsplash.com/@dawson2406?utm_source=medium&amp;utm_medium=referral">Stephen Dawson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Table Of Contents<br></strong>· <a href="#b398">Introduction to Data</a><br>· <a href="#bcbb">Data Visualization</a><br>· <a href="#419f">Question and Answer</a><br> ∘ <a href="#eb37">How is vaccination progress for the top 5 rich countries?</a><br> ∘ <a href="#a101">How’s the vaccination progress for the top 5 rich countries having a population of more than Ten Million?</a><br> ∘ <a href="#308d">Which are the top 5 countries with the highest percentage of complete immunizations?</a><br> ∘ <a href="#a442">How much percentage share of population by dose 1 and 2 for top 10 most populous countries?</a><br>· <a href="#fa56">Summary</a><br> ∘ <a href="#55b7">Our journey from the beginning of the project :</a><br> ∘ <a href="#d8d9">Key Insights from our Work</a><br> ∘ <a href="#e473">Future Work</a><br>· <a href="#fbc8">References:</a></p><p>Exploratory data analysis is an approach to analyzing the datasets to summarize the main characteristics, often using statistical graphics and other data visualizations tools. The first step of anything you do should be a brief introduction with your data, you have to understand it, get familiar with it. Gaining a good understanding of data helps us to clean and summarize it, which then brings out all the insights and trends which were otherwise unclear.</p><blockquote>Note: We will be using EDA for Exploratory Data Analysis in the entire article.</blockquote><p>Exploratory Data Analysis has no hard and fast rules to follow, there is no right or wrong way to perform EDA (exploratory data analysis). It varies from person to person, There are some major commonly followed steps, which are listed below:</p><ul><li><strong>Looking at the Data: </strong>Before you start doing anything in your EDA, first try to look at your dataset.</li><li><strong>Handling missing data: </strong>Sometimes you may encounter <em>NaN</em> values in some of the columns of your dataset, this happens when data may not have been available during the collection of data,</li><li><strong>Handling duplicate data: </strong>Sometimes you may find there are data points that are occurring repeatedly, these data points are called duplicate data. You have to be very careful about such data points.</li><li><strong>Data visualization: </strong>This is the most important part of your EDA. Data visualization is a way in which you can explain everything about your data.</li><li><strong>Insights: </strong>This is the conclusion part of your EDA.</li></ul><p>For this article, I picked a dataset about “<strong><em>Covid-19 Vaccination Progress”.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DPgkSG_ipttf3dme" /><figcaption>Photo by <a href="https://unsplash.com/@spencerbdavis?utm_source=medium&amp;utm_medium=referral">Spencer Davis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s first talk about the dataset, I picked this dataset from <a href="https://www.kaggle.com/gpreda/covid-world-vaccination-progress">Kaggle</a>, the hosts of this dataset are updating the dataset every day. They are adding more data every day since the vaccination process is running in each country. The dataset contains more than 39 thousand rows of data (on 22nd August 2021), and has several columns, containing information like:</p><ul><li><em>daily_vaccinations:</em></li><li><em>people_vaccinated:</em></li><li><em>people_fully_vaccinated:</em></li><li><em>total_vaccination:</em></li></ul><p>There are a total of 15 such columns in the dataset.</p><p>First of all, you need to download the dataset to download a dataset from kaggle, you can use <em>opendatasets</em> python library, this is the simplest way to download any dataset from Kaggle. you can install it in your notebook and then can use it as follows:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D10&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D10&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F5d3797cd1d2a45dd941ef35d0d39f282&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/854aea329602ea7c9e54064fc95e3bf7/href">https://medium.com/media/854aea329602ea7c9e54064fc95e3bf7/href</a></iframe><p>When you run the above cell, it will ask you to input your kaggle username and kaggle API key, which you can get from your kaggle account go to your profile section click on the <em>account settings</em> and you will see your username and scroll a bit down you will find API key, that you can use to download the dataset.</p><p>Once the dataset is downloaded into your jupyter notebook, now you can start the introduction with your dataset.</p><h3>Introduction to Data</h3><p>Before you do anything you should know about your data, you should know about how many rows are there, how many columns are there, how much data is missing from the dataset, what kind of data is available in each column whether it is numeric data or categorical data, whether the entire data is based on certain time period or not, these are the few things that you should know about your dataset that you want to analyze in your EDA.</p><h4>Know about Pandas and Plotly</h4><p>Since we are going to use 2 python libraries for our EDA, so let’s just take a short introduction to both the libraries, We will use the Pandas library. Pandas library is a python library that helps us to do some operations or queries on tabular data, Pandas actually can deal with a bunch of file formats you can read about them on <a href="https://www.cbtnuggets.com/blog/technology/programming/14-file-types-you-can-import-into-pandas">CBTnuggets</a>, and pandas shows the data in tabular format, pandas library comes with a bunch of methods, modules, and functions to manipulate the dataframe. You can read the official documentation page for further introduction with pandas <a href="https://pandas.pydata.org/docs/getting_started/overview.html">here</a>, or you can download a pdf book for the entire documentation using this <a href="https://pandas.pydata.org/docs/pandas.pdf">link</a>.</p><p>Another library we will be using in our EDA is the Plotly Python library. Plotly is actually a visualization python library, which has its origin from JavaScript, Python, R, and Julia. Plotly provides a wide range of interactive graphs. It’s easy to use, and you get a lot of information in just one graph. You can draw line charts, bar graphs, pie charts, treemaps, scatter plots, mixed plots( mix of line and scatter plot and bar graphs), sun-bursts graphs using plotly. You can also draw 3-D graphs, and animated graphs using plotly.</p><p>Let’s come back to our analysis part, To know the basic information of your dataset, you have a bunch of methods available from pandas. You type</p><pre>!ls -lh covid-world-vaccination-progress<br>!wc -l covid-world-vaccination-progress/country_vaccinations.csv<br>df.shape<br>df.info()<br>df.describe()<br>df.columns</pre><p>The commands that I’ve listed above, will give you information about how many data files the link does have that we have downloaded above, how many rows of data is there in the dataset, what is the shape of the dataframe is like (rows, columns).</p><ul><li><em>df.info() </em>will give you information about missing counts for each column and the datatype for each column.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D24&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D24&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F5d3797cd1d2a45dd941ef35d0d39f282&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2809d5cb59c4f20e7122f96e0775434/href">https://medium.com/media/a2809d5cb59c4f20e7122f96e0775434/href</a></iframe><ul><li><em>df.describe()</em> will give you some statistical analysis about your dataset for numerical columns</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F40%26cellId%3D26&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F40%26cellId%3D26&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2Fdaf0a1375be14e8393a2026718edf5e3&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/321cf1dd21ed4557c7d02a96c891dc39/href">https://medium.com/media/321cf1dd21ed4557c7d02a96c891dc39/href</a></iframe><ul><li><em>df.columns </em>will give you a list of names of all columns of the dataset.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F40%26cellId%3D28&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F40%26cellId%3D28&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2Fdaf0a1375be14e8393a2026718edf5e3&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b339a19a8d6f95c9038ac5ae88d4032/href">https://medium.com/media/5b339a19a8d6f95c9038ac5ae88d4032/href</a></iframe><p>An important part of this introduction is, keep an eye on the missing data for each column, If there is a lot of missing data from the columns that you want to analyze then the best way is to remove all those rows which have missing data, otherwise, you will get wrong results at some points, or you can follow an alternative way, just ask the data hosts for why it is missing and if they can fill or is there any way to fill that missing data. This second option is not much useful, can work if you are working in a company and you are doing your EDA on the company’s data. For public data, this can be done. You can just skip for those rows and do your EDA on the remaining rows.</p><p>You can also do a check for duplicated rows of data. You can type <em>df.duplicated().sum(), </em>it will give you information about duplicated number of rows of data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D36&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D36&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F5d3797cd1d2a45dd941ef35d0d39f282&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0b71ca110c8f1de4a8fdd2d65cf4dd7/href">https://medium.com/media/b0b71ca110c8f1de4a8fdd2d65cf4dd7/href</a></iframe><p>Now it’s time to jump into the most interesting part of every EDA project.</p><h3>Data Visualization</h3><p>Data visualization is the coolest thing about EDA. I can’t stop myself loving it. It’s like the transformation of a story into live-action-movie. Data is like a story that you imagine in your mind but can’t share it with others in an explainable way, but you can share it with others with the help of the power of visualization. Data visualization helps you transform your numbers into an engaging story with details and patterns. Data visualization enables us to recognize emerging trends and respond rapidly on the grounds of what we see. Such patterns make more sense when graphically represented; because visuals and diagrams make it easier for us to identify strongly correlated parameters. Humans can process visual images 60,000 times faster than text.</p><p>There are several Python libraries you can use for data visualization. I am listing some of them here:</p><ul><li>matplotlib</li><li>seaborn</li><li>plotly</li><li>ggplot</li><li>folium (for plotting geographical maps)</li></ul><p>More such libraries also available that you can use. <a href="https://www.kaggle.com/getting-started/108792">Here is a list of top 10 python visualization libraries</a>.</p><p>Here in this project I’ve used plotly.</p><p>Since we have data about <em>total_vaccinations </em>for each country, so everyone is curious to compare the progress between different countries, or they just want to know about <em>total_vaccinations </em>at a place instead of looking at different rows of data. The best way for this is to make a bar plot for those countries about which you want to get the information.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D51&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F38%26cellId%3D51&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F5d3797cd1d2a45dd941ef35d0d39f282&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e435d31f65d5cf75bbd59de6ad2dac5/href">https://medium.com/media/6e435d31f65d5cf75bbd59de6ad2dac5/href</a></iframe><p>In the above graph, I have plotted a bar graph for the top 20 countries by <em>total_vaccinations</em>, anyone can easily see all the information about <em>total_vaccinations</em> count and can compare the data with other several countries. The above graph clearly says that China is much far away than the other countries, India is progressing slowly according to its population while USA has achieved as much as it is a rich country. Someone may different aspect to view the above graph, as anyone can say China is going very fast followed by India, United States and Brazil respectively.</p><blockquote>Tthe column <em>‘people_vaccinated’ </em>is about a country’s total vaccinated people who either got partial vaccination or got full/complete dose of vaccination, while the column ‘<em>people_fully_vaccinated’ </em>is about the number of people who has got complete dose of vaccine or can say who has got complete immunization, while the column ‘<em>total_vaccination’ </em>is about the number of total vaccine a country has delivered to it’s people.</blockquote><p>While we have Data about <em>‘people_fully_vaccinated’ </em>and <em>‘people_vaccinated’</em>, so let’s check what is the status for top 5 country’s according to these two columns,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F44%26cellId%3D53&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F44%26cellId%3D53&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2Ffd4d479facb44a0cbb03fae6dab0eb8a&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/08df6d2e12b159dd14447db03008ee2e/href">https://medium.com/media/08df6d2e12b159dd14447db03008ee2e/href</a></iframe><p>There is a big difference in completely immunized people and partially immunized people for India, but it’s been a great job done by India, at least India is covering it’s large population with a single dose. While for United States, a little difference in both bars, they are covering there population with complete immunization along with the partial immunization, We will try to find the reason behind this.</p><p>Let’s check what is day-wise growth rate for the top 5 countries for total vaccination</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D56&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D56&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/683bfd8cf98bd78de5801cfd82c85786/href">https://medium.com/media/683bfd8cf98bd78de5801cfd82c85786/href</a></iframe><p>Each of the country except China has a linear growth rate, while China has exponential kind of growth. Let’s compare these numbers with total population of the country, to do this, let’s download a population dataset into our notebook, and merge with our country-wise vaccination dataframe.</p><p>After merging both the dataframes, I plotted the bar chart for the <em>people_fully_vaccinated </em>and <em>people_vaccinated </em>and <em>population </em>of the country,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D72&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D72&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a78ba9ebd101ac8ec754d257a862770/href">https://medium.com/media/4a78ba9ebd101ac8ec754d257a862770/href</a></iframe><p>India, A long way to cover it’s total population with complete immunization, while USA has done a nice job for the vaccination, Brazil is at almost about 25% with complete immunization and Japan, Germany and United Kingdom has covered about 50% population with complete immunization.</p><p>Let’s draw a geo mix of all countries with all the information about <em>total_vaccination, people_fully_vaccinated, people_vaccinated </em>and the <em>population</em> of the country.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D87&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D87&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2234d47057f787f6725b4f49f18a27c/href">https://medium.com/media/e2234d47057f787f6725b4f49f18a27c/href</a></iframe><p>When you hover over the points on the map, you will get all the information about the country. The size of the bubble tells you about the <em>total_vaccination</em> of the country.</p><h3>Question and Answer</h3><p>After you finish your work on visualization, this should be the last part of your EDA project, because people may not understand your simple visualization or they may not be interested in your coding skills of plotting visualization, but everyone who is reading/looking at you project notebook, he/she might be interested in some general questions. I answered some questions in my project have a look:</p><h4>How is vaccination progress for the top 5 rich countries?</h4><p>Rich country in the sense of having highest GDP per capita. For this, I downloaded another dataset containing information about GDP per capita and merged to our country-wise vaccination data frame. I sorted the data frame according to the GDP per capita and show the table for the top 5 rich countries, have a look at the table:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D102&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D102&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b882fab261be2462cba7259b8d54fc7/href">https://medium.com/media/0b882fab261be2462cba7259b8d54fc7/href</a></iframe><p>It seems like, Luxembourg has covered 74% of its population with complete immunization. One can notice here in the above table that except the USA all the other countries have a very low population less than 1 million and <br>Norway is less than 10 million, and that’s why they have a very high GDP per capita and they have easily covered the complete immunization, but what about more populous countries? Let’s check</p><h4>How’s the vaccination progress for the top 5 rich countries having a population of more than Ten Million?</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D107&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D107&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/12b30a2106882afb95329764bcfe56c0/href">https://medium.com/media/12b30a2106882afb95329764bcfe56c0/href</a></iframe><p>Wow! as expected, these top 5 rich countries having a population of more than 10 million, have done a very nice job for their population, Except Australia, each one of them has covered more than 50% population with complete immunization, Canada and Belgium with more than 75%.</p><h4>Which are the top 5 countries with the highest percentage of complete immunizations?</h4><p>This might be the next question in your mind, and the answer is:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D113&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D113&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/854153f300502ede9ac7173b2279e1a7/href">https://medium.com/media/854153f300502ede9ac7173b2279e1a7/href</a></iframe><p>Singapore, Aruba with more than 95% while Israel, Iceland, and Seychelles with more than 85% are at the top position in view of the complete immunization.</p><h4>How much percentage share of population by dose 1 and 2 for top 10 most populous countries?</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D119&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D119&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d56f433b8f94666e148014822f0ff710/href">https://medium.com/media/d56f433b8f94666e148014822f0ff710/href</a></iframe><p>Wow! India has covered 40% population with single-dose while, 12% population with complete immunization. The USA with 67% and 57% with single-dose and double dose.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D118&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Feda-project-on-world-wide-covid-vaccination%2Fv%2F43%26cellId%3D118&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fd5810719e0f540a3ab0b2cd65c8e4d55%2Fpreview%2F1ce2356a2e35434090e9c4061b8a1bd8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4e614fc012d8a7070e8889655e603cc/href">https://medium.com/media/e4e614fc012d8a7070e8889655e603cc/href</a></iframe><p>From the above graph, we have seen that Nigeria is pretty much slow with the vaccination progress, If you look at the table and focus on the GDP per capita in Nigeria, you will find they have a very low GDP per capita and that’s why they are much far behind in the race of the vaccination process.</p><p>Well, there are few more interesting questions and answers in my notebook, you can view all of them by visiting my notebook click on this link below</p><blockquote><a href="https://jovian.ai/thakubhai-007/eda-project-on-world-wide-covid-vaccination">Covid-19 Vaccination Data Analysis</a> Project Notebook</blockquote><h3>Summary</h3><p>First of all please accept my gratitude for the time that you had put into reading this article.</p><h4>Our journey from the beginning of the project :</h4><ul><li>We first got introduced to our dataset.</li><li>We then checked for missing values, duplicated values, and what are the data type available in the dataset.</li><li>We then come to the visualization part, which is the best part of each of the EDA projects.</li><li>We then answered some of the interesting questions that a person can ask in general.</li></ul><h4>Key Insights from our Work</h4><p>Finally, we brought some interesting conclusions from our work:</p><ul><li>United States: They have done a very great job with the vaccination process, they have covered almost around 57% of the entire population with complete immunization, the achievement they have got may be due to two reasons: First-They is a very strong country in view of GDP per Capita, and the second one is they are using a vaccine developed by the company Johnson&amp;Johnson, which has single dose vaccination scheme.</li><li>India: India is a very big country by its population and also does not have a good GDP per capita score, India has covered almost 40% of its total population with single-dose and about 11% of the population with complete immunization.</li><li>China: Where this pandemic started firstly, and they started the research for vaccines to stop the spread of this covid-19 virus, but they didn’t perform their job perfectly, even they also have a good GDP per capita score than India.</li><li>Countries with Good Vaccination Progress: It is difficult to vaccinate the entire country with a large population, but still, there are some countries that have done their job very seriously and they really deserve the right award for this. some of those countries are United States, Canada, Belgium, Australia and Nethersland.</li></ul><p>Yea, That’s all we almost cover all the important processes that anyone can put in his/her EDA projects.</p><h4>Future Work</h4><blockquote>“ Any work never ends perfectly, you can always make it better, bring some improvements and you always can do better than the previous one.” -Unknown</blockquote><p>There might be some questions regarding the vaccination progress in your mind, please post them in the comments; I’ll really appreciate your efforts to ask me more questions that I should include in my project.</p><p>Although I am further looking for some more analysis on the time estimation for each country to complete the vaccination process for the entire population. And for the entire world population also.</p><h3>References:</h3><p>First of all a big thanks to<a href="https://aakashns.medium.com/?source=collection_about-------------------------------------"> Aakash N S</a>, <a href="https://medium.com/@sidujjain?source=collection_about-------------------------------------">Siddhanth Ujjain</a>, <a href="https://blog.jovian.ai/">jovian.ai </a>team, and the Code-Hunters(<a href="https://jovian.ai/anubratadas">Anubrata Das</a>, <a href="https://jovian.ai/anushree-k">Anushree K</a>, and <a href="https://jovian.ai/shaswathpatil7">Shashwath Patil</a>) team who helped me a lot in learning all of this.</p><ol><li>I watched tutorials from <a href="https://jovian.ai/learn/zero-to-data-analyst-bootcamp">jovian.ai Data Science Bootcamp.</a></li><li>Information about the dataset on <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fwww.kaggle.com%2Fgpreda%2Fcovid-world-vaccination-progress">Kaggle</a>.</li><li>Visualization techniques and examples from <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fplotly.com%2Fpython%2F">Plotly</a>.</li><li>For Pandas documentation, you can download a pdf book: <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fpandas.pydata.org%2Fdocs%2Fpandas.pdf">pandas.pdf</a></li><li>For all the coding references, please visit my project notebook: <a href="https://jovian.ai/thakubhai-007/eda-project-on-world-wide-covid-vaccination">Covid-19 Vaccination Data Analysis</a></li></ol><p>You can read my previous blog on: <a href="https://blog.jovian.ai/web-scraping-using-python-and-beautifulsoup-adf43cbdb816?source=your_stories_page-------------------------------------">Web-Scraping using Python and BeautifulSoup</a></p><p>Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6aa9ce1fdb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web-Scraping using Python and BeautifulSoup]]></title>
            <link>https://ankit-pratap-singh.medium.com/web-scraping-using-python-and-beautifulsoup-adf43cbdb816?source=rss-14d49fd473b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/adf43cbdb816</guid>
            <category><![CDATA[requests-library]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[ANKIT PRATAP SINGH]]></dc:creator>
            <pubDate>Wed, 30 Jun 2021 07:59:13 GMT</pubDate>
            <atom:updated>2022-03-02T15:03:32.386Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to scrape the data from web pages and save it to a CSV file.</h4><blockquote>“Data is the new oil.” — Clive Humby</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4RIXfWCSrDsagQWHil5unw.jpeg" /><figcaption>Web Scraping Image</figcaption></figure><p>We all live in a time when the internet is rich with enormous amounts of data. Data has become a new fuel. Now the question is,<em> “how do you utilize this fuel?”</em> You can use data for analysis, research, machine learning, artificial intelligence, etc.</p><p>Another critical question is if you are a data scientist or Machine Learning Engineer <em>“How to collect and prepare the data for your projects? “. </em>If you are working for a company, the company may already have a ready dataset, or you may be asked to build one from various sources. If you are working on your projects, you have to collect the data from the different sources available on the internet. Usually, not all the data you need is available in one place, formatted for a direct download. Also, there is no guarantee that such readily available data will be suitable for your research.</p><p>Web scraping deals with extracting or scraping the information from the website. Whether you are a data scientist, engineer, or anybody who analyzes vast amounts of datasets, the ability to scrape data from the web is a valuable skill to have. Let’s say you find data from the web, and there is no direct way to download it; web scraping using Python is a skill you can use to extract the data into a proper form that can then be imported and used in various ways. <em>Web scraping is sometimes referred to as </em><strong><em>web harvesting</em></strong><em> or </em><strong><em>web data extraction</em></strong>. Several tools are available for web scraping, and Python comes with some nice and useful libraries to make this job much easier.</p><blockquote>“Where there is data smoke, there is business fire.” — Thomas Redman</blockquote><p>BeautifulSoup is one of the most helpful library in Python that is beginner-friendly and easy to understand with all beginner levels of coding in Python. At the end of this article, I hope you will be confident that “<em>anyone can scrape the data with all beginner level of coding.</em>”</p><figure><img alt="An image showing how does scraping works" src="https://cdn-images-1.medium.com/max/1024/1*M0dOmunDPdJNzZc4e9500A.jpeg" /></figure><h4>Can we scrape data from everywhere?</h4><p>Before you dive deep into the process of Scraping, I think you should be aware of this, Scraping makes the website traffic spike and may cause the breakdown of the website server. Thus, not all websites allow people to scrape. So how do You know which websites are allowed or which are not? You can look at the ‘robots.txt’ file of the website. You simply put <em>“/robots.txt”</em> after the URL you want to scrape and you will see information on whether the website host allows you to scrape the website.</p><blockquote>Let’s take an example of “value.today/robots.txt”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*_xKQwQ-H16buDL5Pa7pqag.png" /></figure><p>You will find a page like this Here, you will read all the terms and conditions about the web page and what the page allowed for and does not allow for.</p><p>Another way of finding the allowance for Scraping is to find user agreement page some time you will find terms and conditions page on the website, there you will find their T&amp;C for web scraping. Sometimes some web pages do not allow scraping in general, but if you contact the admins by emailing them and ask them that you want to scrape the data for research or study purpose, they may allow you to scrape the data. Otherwise, if none of the conditions holds for the web page, and you still try to scrape the data for your purpose, then the admins may block your I.P. address for their server. And you will start seeing captcha pages instead of web pages.</p><h4>How does web scraping works?</h4><p>Well, It’s like a bot, first, you write some codes, and then you run the code, the program send a request to the web page server and bring content in the HTML form from the server, and then you parse that data in CSV or whatever format you want.</p><p>Let’s take an example here:</p><h4>We want to scrape data regarding the world’s top 500 companies by market capitalization.</h4><p>For this, I have selected a web page <a href="https://www.value.today/world/world-top-500-companies">value.</a> Today, this web page contains data regarding companies worldwide, and they are regularly updating their data on their page. They provide their data for study and research purposes. For more details, you can visit <a href="https://www.value.today/about">here.</a></p><p>There are many ways to scrape the data. In Python also you have dozens of libraries for web-scraping, But some important libraries in Python are: “<strong>Requests</strong>”, “<strong>Beautiful Soup</strong>”, “l<strong>xm</strong>l”, “<strong>Selenium</strong>”, “<strong>Scrapy</strong>”. Everyone should learn requests, because this is the library for web scraping that allows you to communicate with the web servers, rest depends on your use case, like:</p><ol><li><strong>Beautiful Soup: </strong>Beautiful soup library is vital to add into your data science toolkit, as this is yet simple and easy to use but powerful library that you can scrape the data in just few hours of practice. It’s simplicity is definitely it’s greatest strength.</li><li><strong>lxml: </strong>lxml is high-performance, production-quality HTML and XML parsing library. You can rely on it to be good for you no matter what web page you are following for the Scraping. lxml is faster then Beautiful soup, lxml is widely adopted between data science industries.</li><li><strong>Selenium: </strong>Some web pages out there use JavaScript to show their content, on those web pages times you need to go through forms to see the content there or sometimes you need to select options from a dropdown menu before seeing the content, For such web pages you will need more powerful tool then requests, you will need Selenium there, Selenium is a tool that automates the browsers, some times it’s called as web-driver.</li><li><strong>Scrapy: </strong>Scrapy is vast than the above options, Scrapy is technically not even a library, it’s a complete web scraping framework. You can use scrapy to manage requests, preserve user sessions, follow redirects and handle output pipelines.</li></ol><p>For more information about these libraries you can visit : <a href="https://elitedatascience.com/python-web-scraping-libraries">5 Tasty Python Web Scraping Libraries.</a></p><p>Well, let’s start our journey for the above example:</p><p>We will use “Requests”, and “Beautiful Soup” libraries for our project, and we will finish our work in three steps:</p><ol><li>Downloading the web-page using Requests library,</li><li>Inspecting and extracting the data from the web-page using Beautiful Soup library,</li><li>After extracting the data we will write a parser function in Python to write all the required data into a csv file.</li></ol><h4><strong>Step 1:</strong></h4><p>We will start our journey from installing and importing all the required libraries into our jupyter notebook. You can install a library into you notebook using the code below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F55%26cellId%3D7&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F55%26cellId%3D7&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F6b5422f40fe144838630be26af1c1a98%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/08058c161c0c8708ac423c5ab99044b1/href">https://medium.com/media/08058c161c0c8708ac423c5ab99044b1/href</a></iframe><p>Then we will download the web page using requests library. Requests library makes a connection between your notebook or say your machine where you are writing the codes and it sends a request to the web-page server and bring the content from the web-page server in HTML format(that’s a general format, generally a web-page written in). We save all the HTML content in a variable so that we can use it anytime whenever we need, we write code as</p><pre>url = &#39;<a href="https://www.value.today/world/world-top-500-companies">https://www.value.today/world/world-top-500-companies</a>&#39;<br>response = requests.get(url)</pre><p>And now we have an another way to know whether the web-page is ordinary for web-scraping or not. We just add a new code cell there and we check the response code for the web-page,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F55%26cellId%3D11&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F55%26cellId%3D11&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F6b5422f40fe144838630be26af1c1a98%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c3d52c5c16666714830e7a812c46705/href">https://medium.com/media/5c3d52c5c16666714830e7a812c46705/href</a></iframe><p>If this status_code is between 200–299 then the web-page you have chosen is ordinary otherwise not. This status_code actually stands for <strong>Hypertext Transfer Protocol(HTTP) </strong>response status_code. <strong>Status_codes</strong> are issued by a server in response to a client’s request made to the server. You can see a complete list of user guide for these status_codes <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status">here.</a></p><p>Now after downloading the web-page to our notebook, now it’s time to go towards the step 2.</p><h4>Step 2:</h4><p>Now we will inspect our web page, Since HTML uses tags for each attributes and our downloaded web page is also in the form of HTML codes so we will try to fetch all the data from the web page using the HTML tags for each attribute available on the web page. For knowing HTML tag for any attribute on the web page you just go to that particular attribute on the page and right click on your mouse you will see a menu having an inspect button there and you click on the inspect button, a new window pops up either in the left side or bottom side or right side of you current window like the following one,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c7KgOWOwkzk0l6SXBIaKWw.jpeg" /></figure><p>Till now half of our work has been finished, We have downloaded the web page and now we now about the html tags for each attributes, now it’s time collect the data from the web-page using these attributes. Have a look on the below code cell:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D30&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D30&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/51c4ad19fbdb0f4a854264e6f6e788d8/href">https://medium.com/media/51c4ad19fbdb0f4a854264e6f6e788d8/href</a></iframe><p>In the above cell, That’s how does data looks like in an html tag. I chose to fetch a company_block_tag here because there are 500 companies on the web-page and the data for each company lies inside a single block, that’s why I fetched block_tag for each company and now I’ll try to fetch all required data from the respective tag for each respective attributes.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D36&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D36&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e1c1e0e12f8d884b2bb07fbfa10c5c8/href">https://medium.com/media/0e1c1e0e12f8d884b2bb07fbfa10c5c8/href</a></iframe><p>In the above code cell, I fetched the data regarding <strong>rank</strong>, <strong>head quarter location</strong>, <strong>CEO name</strong>,<strong> total employee </strong>and <strong>market capitalization </strong>from the company_block_tags for each company.</p><p>That’s great, we have collected the data from the web page and now it’s time to parse this data into a CSV file, using <strong>Pandas </strong>library. First we save all the collected data into a python dictionary in a sequenced format that we want the data should be on the csv file.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D41&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D41&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/014db603b852d6215dde8e27a4693b10/href">https://medium.com/media/014db603b852d6215dde8e27a4693b10/href</a></iframe><p>This is how a dictionary has been created and now we will make a pandas data-frame so that we can parse the data into a csv file using <a href="https://jovian.ml/thakubhai-007/web-scraping-final/v/59&amp;cellId=44">df.to_csv() </a>function. Finally we have collected the data from the web page that we have selected.</p><h4>Step 3:</h4><p>Till now we have only collected the data from a single web-page let’s go deep into the web scraping let’s write some helper function that will actually help to automate our work, instead of writing codes for a single web page let’s write some codes that actually extracted the data from different web pages of almost same kind,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D55&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D55&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/82fb8cca0e9d57716adc087afbe4f3b5/href">https://medium.com/media/82fb8cca0e9d57716adc087afbe4f3b5/href</a></iframe><p>We will now write some helper function like above, when we call the above function by giving an input in the form of a url, it will check the statu_code and if the status_code is valid then it will get the HTML content from the web-page server and returns a Beautiful Soup object. And then we will write some functions to collect the data from each page,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D59&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D59&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e37822ab6ee2f8da6a20c66adacc305/href">https://medium.com/media/2e37822ab6ee2f8da6a20c66adacc305/href</a></iframe><p>Similar to above one, these function then we will use in an another function which we will use to parse all the data into a python dictionary,</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D72&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D72&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e18497c22641a71aed6600f6f1aa3d5/href">https://medium.com/media/3e18497c22641a71aed6600f6f1aa3d5/href</a></iframe><p>This is our function that will collect all the data form the each company’s page and then will automatically write all the data into a python dictionary. That we will use for write all the data first into a pandas data-frame and then using the pandas data-frame we will write all the data into a csv file.</p><p>Here is a snip how does the pandas data-frame looks like:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D82&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fthakubhai-007%2Fweb-scraping-final%2Fv%2F59%26cellId%3D82&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F0b8226b61c8e4ea9870f976aef155822%2Fpreview%2F75706a64f1e9456092c31b1df110de34%3Fsize%3Dl&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/62bc5bdd46c826ac22ee011fe669acc7/href">https://medium.com/media/62bc5bdd46c826ac22ee011fe669acc7/href</a></iframe><p>Well, finally we have successfully scraped the data from 500 different pages, and we have created a csv file with that data, here is a link to download the entire <a href="https://storage.googleapis.com/jvn/gists/thakubhai-007/0b8226b61c8e4ea9870f976aef155822/raw/b8d7abe088194db78e9f8d7bbeff6161/companies_data.csv?X-Goog-Algorithm=GOOG4-RSA-SHA256&amp;X-Goog-Credential=prod-proxy-user%40jovian.iam.gserviceaccount.com%2F20210630%2Fauto%2Fstorage%2Fgoog4_request&amp;X-Goog-Date=20210630T070902Z&amp;X-Goog-Expires=3600&amp;X-Goog-SignedHeaders=host&amp;X-Goog-Signature=a47e40900a37aad6c97a33e0c5db7d47eed22f9385e3920745d43986145b25f52c6d1536d1ebeb52fad41e0c9643db64a4fad08206ff76a6d65004a645fbec31b5e113486b1b7aceef6053e1d4774ac8adb8c8d073490e54cf376a56a1dfbb6a170caee3f687cc6a53ea71df0d10b073b95343386489e2cf67588c58792bc57d6796c2569f12be77bff4f9d9d80e1040d9fe4745c64731cbc4d7f528f05da7250ae6155a324824d7107e0b7636533beae44c0315dfc8f42a25a202816de94cb4c79977fc6a79d73d860fd6616077631d766fec7e1b5f84554230594c3b6259849ba54e109c9a36799c7a7197ea2acce199ab5c0b60299d712444b98f612a9788">dataset</a></p><h3>Summary:</h3><p>Here is a very short description from beginning to end:</p><ol><li>First of all we installed all the required libraries to our jupyter notebook.</li><li>We download the web page to our notebook using requests library,</li><li>We inspect the web-page for HTML tags for all required attributes regarding each data that we want to scrape from the web-page,</li><li>We then collect the data from each HTML tags and write all the data into a python dictionary,</li><li>For collecting the data from different pages we write some helper functions and then we write a parser function to extract all the data from each different page and then parse the collected data into a python dictionary,</li><li>Finally we write all the data into a csv file using pandas library.</li></ol><p>Now the dataset that we have created, can be used for <strong>exploratory data analysis</strong> on top 500 companies by market capitalization, The data set so created by our own is completely reliable because everything in the data set is in our knowledge. we can still add some more variation in our data set for creating more options with the data set like we can add companies data according to country wise, or according to stock exchange and so many ways to make variations.</p><p>That’s all web-scraping in python stands for.</p><h3>References:</h3><ol><li>To see the complete notebook for codes and all the user guidance plz visit to <a href="https://jovian.ai/thakubhai-007/web-scraping-final">my notebook</a></li><li>A big thanks to Aakash N S, who taught me how how to scrape the web-pages, here is a link of his tutorial <a href="https://youtu.be/RKsLLG-bzEY">https://youtu.be/RKsLLG-bzEY</a></li><li>A complete documentation for requests library <a href="https://pypi.org/project/requests/">here</a></li><li>Documentation for Beautiful Soup 4 library <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">here</a></li><li>A big thanks to Anushree K, Anubrata Das and the entire code-hunters team and <a href="https://www.jovian.ai/zero-to-data-science-bootcamp">jovian.ai</a> team as well, who supports me to learn all things in a group.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adf43cbdb816" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>