<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ABHINAV SHARMA on Medium]]></title>
        <description><![CDATA[Stories by ABHINAV SHARMA on Medium]]></description>
        <link>https://medium.com/@abhinavsharma150?source=rss-57d390b69974------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BNoZvYg8-pXVl3AX.jpg</url>
            <title>Stories by ABHINAV SHARMA on Medium</title>
            <link>https://medium.com/@abhinavsharma150?source=rss-57d390b69974------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:16:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhinavsharma150/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mastering Slowly Changing Dimensions: Part 2 — Learn how to Implement SCDs with easy Examples.]]></title>
            <link>https://abhinavsharma150.medium.com/mastering-slowly-changing-dimensions-part-2-learn-how-to-implement-scds-with-easy-examples-e67fa0490849?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/e67fa0490849</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[scd-type-2]]></category>
            <category><![CDATA[slowly-changing-dimension]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 18:03:40 GMT</pubDate>
            <atom:updated>2025-06-30T18:09:53.041Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering Slowly Changing Dimensions: Part 2 — Learn how to Implement SCDs with easy Examples.</h3><p>Learn how to handle changing data with precision by implementing SCD -Types 1, 2, and 3 in your dimensional model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZB_C9Ma3YmPfeJf7p7G1Lg.png" /></figure><p>In <a href="https://abhinavsharma150.medium.com/mastering-slowly-changing-dimensions-part-1-demystifying-data-evolution-01e235f343ed">Part 1</a> of this series, I explained what exactly is SCD with examples.Now, we shift gears to <strong>how</strong> to implement them using practical examples based on the tables you have already seen. Whether you are updating a customer address or tracking department changed for an employee over time, there is a systematic approach to get it right.</p><p><strong>If you have not Read the part 1, You can check it here.</strong><br><a href="https://abhinavsharma150.medium.com/mastering-slowly-changing-dimensions-part-1-demystifying-data-evolution-01e235f343ed">Mastering Slowly Changing Dimensions: Part 1 — Demystifying Data Evolution</a></p><h3>SCD Type 1: Overwrite the Data, No History Maintained.</h3><p>It is used when you only need current data and not history.</p><h4>Table Recap:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*fw85qEpv5eL_kF00-CSaKw.png" /><figcaption>SCD Type 1 Table</figcaption></figure><p>For the update we would also need a <strong>Staging Table</strong>, that would contain the “delta” of the changes we need to update into our customer table.</p><p>Lets Suppose our staging table name is “<strong>Staging</strong>”. We will use staging to:</p><ol><li><em>Modify old records in Customer based on primary key (CustomerID) , where data is changed.</em></li><li><em>Insert new records directly from Staging into Customers.</em></li></ol><pre>-- Step 1: UPDATE existing records that have changed, based on primary key (CustomerID)<br>UPDATE Customer<br>SET Address = staging.Address,<br>    City = staging.City<br>FROM staging<br>WHERE Customer.CustomerID = staging.CustomerID  -- Match Pks<br>  AND (<br>      Customer.Address &lt;&gt; staging.Address       -- Only update if there is a data change<br>   OR Customer.City &lt;&gt; staging.City<br>  );<br><br>-- Step 2: INSERT new records that don&#39;t exist yet, directly from Staging into Customers.<br>INSERT INTO Customer (CustomerID, Name, Address, City)<br>SELECT staging.CustomerID, staging.Name, staging.Address, staging.City<br>FROM staging<br>LEFT JOIN Customer ON staging.CustomerID = Customer.CustomerID<br>WHERE Customer.CustomerID IS NULL;</pre><h4>Alternatives —&gt; Upsert techniques for SCD1</h4><p>We can use Upsert techniques as an alternative, these alternatives use a Single query to perform SCD1.</p><pre><br>-- Merge Statement, SQL Server, Snowflake, Bigquery<br>MERGE INTO Customer AS target<br>USING staging AS source<br>ON target.CustomerID = source.CustomerID<br>WHEN MATCHED AND (<br>    target.Name &lt;&gt; source.Name OR<br>    target.Address &lt;&gt; source.Address OR<br>    target.City &lt;&gt; source.City<br>) THEN<br>    UPDATE SET <br>      target.Name = source.Name,<br>      target.Address = source.Address,<br>      target.City = source.City<br>WHEN NOT MATCHED BY TARGET THEN<br>    INSERT (CustomerID, Name, Address, City)<br>    VALUES (source.CustomerID, source.Name, source.Address, source.City);<br><br><br>-- MySQL<br>INSERT INTO Customer (CustomerID, Name, Address, City)<br>VALUES (?, ?, ?, ?)<br>ON DUPLICATE KEY UPDATE <br>  Name = VALUES(Name),<br>  Address = VALUES(Address),<br>  City = VALUES(City);<br><br><br>-- Postgres SQL<br>INSERT INTO Customer (CustomerID, Name, Address, City)<br>VALUES (:CustomerID, :Name, :Address, :City)<br>ON CONFLICT (CustomerID) DO UPDATE<br>SET <br>  Name = EXCLUDED.Name,<br>  Address = EXCLUDED.Address,<br>  City = EXCLUDED.City<br>WHERE <br>  Customer.Name &lt;&gt; EXCLUDED.Name OR<br>  Customer.Address &lt;&gt; EXCLUDED.Address OR<br>  Customer.City &lt;&gt; EXCLUDED.City;<br><br>-- Spark SQl ( Delta)<br>MERGE INTO target_table AS target<br>USING source_table AS source<br>ON target.key = source.key<br>WHEN MATCHED THEN UPDATE SET ...<br>WHEN NOT MATCHED THEN INSERT ...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*EDRh3GbZuZnI6zH3w7x6Pg.png" /><figcaption>Different Upsert Methods in various SQL Dialects</figcaption></figure><h3>SCD Type 2: Preserve History of changes</h3><p>Goal:</p><ol><li>Keep older version of record by expiring it.</li><li>Enter a new record every time for the change in the record and mark it active via a flag column ( here IsCurrent).</li></ol><pre>-- Step 1: Expire the old record<br>UPDATE customer<br>SET EndDate = CURRENT_DATE,<br>    IsCurrent = FALSE<br>FROM staging_customer AS s<br>WHERE customer.CustomerID = s.CustomerID<br>  AND customer.IsCurrent = TRUE<br>  AND (<br>    customer.Address &lt;&gt; s.Address OR<br>    customer.City &lt;&gt; s.City<br>  );<br><br>-- Step 2: Insert the new record<br>INSERT INTO customer (CustomerID, Name, Address, City, StartDate, EndDate, IsCurrent)<br>SELECT s.CustomerID, s.Name, s.Address, s.City, CURRENT_DATE, NULL, TRUE<br>FROM staging_customer s<br>LEFT JOIN customer c<br>  ON c.CustomerID = s.CustomerID AND c.IsCurrent = TRUE<br>WHERE c.CustomerID IS NULL<br>   OR c.Address &lt;&gt; s.Address OR c.City &lt;&gt; s.City;</pre><h4>Alternative: Upsert like Syntax</h4><pre>MERGE INTO customer AS target<br>USING staging AS source<br>ON target.CustomerID = source.CustomerID AND target.IsCurrent = true<br><br>WHEN MATCHED AND <br>( target.Address &lt;&gt; source.Address OR<br>    target.City &lt;&gt; source.City)<br><br>THEN<br><br>  UPDATE SET EndDate = current_date(), IsCurrent = false<br><br>WHEN NOT MATCHED THEN<br>  -- New record version<br>  INSERT (...);<br><br><br><br><br>-- Spark Sql Delta<br>MERGE INTO customer AS target<br>USING staging_customer AS source<br>ON target.CustomerID = source.CustomerID AND target.IsCurrent = true<br><br>WHEN MATCHED AND (<br>    target.Address &lt;&gt; source.Address OR<br>    target.City &lt;&gt; source.City<br>)<br>THEN UPDATE SET<br>    target.EndDate = current_date(),<br>    target.IsCurrent = false<br><br>WHEN NOT MATCHED THEN<br>INSERT (CustomerID, Name, Address, City, StartDate, EndDate, IsCurrent)<br>VALUES (source.CustomerID, source.Name, source.Address, source.City, current_date(), NULL, true);<br><br><br>-- Snowflake<br><br>MERGE INTO customer AS target<br>USING staging_customer AS source<br>ON target.CustomerID = source.CustomerID AND target.IsCurrent = TRUE<br><br>WHEN MATCHED AND (<br>    target.Address &lt;&gt; source.Address OR<br>    target.City &lt;&gt; source.City<br>)<br>THEN UPDATE SET<br>    target.EndDate = CURRENT_DATE(),<br>    target.IsCurrent = FALSE<br><br>WHEN NOT MATCHED THEN<br>INSERT (CustomerID, Name, Address, City, StartDate, EndDate, IsCurrent)<br>VALUES (source.CustomerID, source.Name, source.Address, source.City, CURRENT_DATE(), NULL, TRUE);<br></pre><h3>SCD Type 3: Keep a limited history of changes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*qibYlNFQ1ANaQ3SIu6aVBQ.png" /><figcaption>Limited History, SCD Type 3</figcaption></figure><p>For SCD 3 , the table keeps extra columns which can contain a previous values.<br>For example here, for department, a previous value column is present.</p><pre>UPDATE Employee AS target<br>SET <br>    PreviousDept = CASE <br>                      WHEN target.CurrentDept &lt;&gt; staging.CurrentDept THEN target.CurrentDept<br>                      ELSE target.PreviousDept<br>                   END,<br>    CurrentDept = staging.CurrentDept,<br>    Name = staging.Name<br>FROM staging_employee AS staging<br>WHERE target.EmployeeID = staging.EmployeeID<br>  AND (<br>        target.CurrentDept &lt;&gt; staging.CurrentDept<br>     OR target.Name &lt;&gt; staging.Name<br>      );</pre><p>In this second part of the series, we understood how to implement Slowly Changing Dimensions in real-world data systems.</p><p>We explored each SCD type with simple, practical examples:</p><blockquote><strong>SCD Type 1: Overwrites data without keeping history → it’s perfect for non-critical changes.</strong></blockquote><blockquote><strong>SCD Type 2: Preserves full history by inserting new records → it is ideal for audit or history tracking.</strong></blockquote><blockquote><strong>SCD Type 3: Tracks limited history within the same row → it’s useful for tracking recent changes like “Previous Department”.</strong></blockquote><p>Each implementation approach, reflects the tradeoffs between complexity, storage, and data needs (like analysis or record keeping).</p><p>As your data grows and your business questions gets more complex, picking the right SCD strategies becomes essential for building robust and insightful analytics.</p><p><strong>What’s Next Now?</strong></p><p>Now that you’ve learned how to implement Slowly Changing Dimensions using real-world examples and SQL logic, We have covered <strong>“What” </strong>and <strong>“How”</strong>.</p><p><strong>In Part 3, we will explore the “When” —</strong></p><blockquote>When should you choose SCD1 vs SCD2 vs SCD3</blockquote><blockquote>What are the trade-offs in performance, storage, and complexity?</blockquote><blockquote>How do real businesses decide which strategy to apply?</blockquote><p>We will walk through practical frameworks and real world scenarios, that will help you apply the right SCD type at the right time with right knowledge.</p><p>Stay tuned — All the best.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e67fa0490849" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Slowly Changing Dimensions: Part 1 — Demystifying Data Evolution]]></title>
            <link>https://medium.com/analytics-vidhya/mastering-slowly-changing-dimensions-part-1-demystifying-data-evolution-01e235f343ed?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/01e235f343ed</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[slowly-changing-dimension]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Sun, 29 Jun 2025 19:44:50 GMT</pubDate>
            <atom:updated>2025-06-30T18:11:09.695Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering Slowly Changing Dimensions: Part 1 — Demystifying Data Evolution</h3><p>Breaking down the basics of slowly changing dimensions for better data insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CwMYLK77FeZVY18ZrjldjA.png" /></figure><h3>Introduction</h3><h4>What is a dimension in data warehousing?</h4><p>Ever heard of terms “Dimensions” and “Facts” in Data? If you have ever worked with a database or warehouse, there is a fair chance you have seen tables with “dim_” or “fact_” prefixes. Dimensions provide a contextual information about the data, while Facts means a quantitative measure. Still hard to understand? Lets take an example to make it easier.</p><p><strong>Example of Facts and Dimensions:</strong></p><p>Lets assume you manage the data of a F.M.C.G. Store chain, where sales are going on. How should its tables look like?</p><ol><li><strong>Customers</strong> : Having Details of all the customers ( Id, Name, Address, Phone)</li><li><strong>Products</strong>: Details of the products ( Id, Name, Category, Price)</li><li><strong>Stores</strong>: (Store-name, City, State etc)</li><li><strong>Sales</strong>: ( Sale-Id, Store-name, Product-Id, Customer-Id, Date, Total Amount)</li></ol><p>Here, you can see that <strong>First 3 Tables (Customer, Product and Store)</strong> gives the <strong>background details</strong> of every aspect that came together to make<strong> 4th Table (Sales) </strong>a possibility. So here, First 3 Tables are called <strong>Dimensions</strong> and the Sales Table where actual interaction happened is a <strong>Fact</strong>.</p><h4>Why Do Dimensions Change?</h4><p>Dimensions change because they are the real world entities. They are not static, they evolve over time. A few examples Where Dimensions can change are:</p><p><strong>Updates to Customer Information :</strong> Address change, Phone change, Gender change, Name change, Any other corrections.</p><p><strong>Product Changes :</strong> Product can be Renamed, Reclassified, Discontinued etc etc.</p><p><strong>Organizational Changes : </strong>Store Might change its location, Store might close, A Merger or Acquisition.</p><h3>Definition of SCD:</h3><p>Slowly Changing Dimensions (SCD) refers to <strong>techniques used to manage and preserve historical changes in dimension data</strong> over time. Dimension changes are called “slow” because they occur infrequently compared to fast-changing transactional (fact) data.</p><h3>Types of Slowly Changing Dimensions</h3><p><em>Type 1: Overwrite the old data</em></p><p><em>Type 2: Keep history with new rows</em></p><p><em>Type 3: Keep limited history with additional columns</em></p><p><strong>In SCD Type 1</strong>, the old information in the dimension table is replaced by new data. So no history is kept and only the latest details are there. Take this example of Customer Table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*fw85qEpv5eL_kF00-CSaKw.png" /><figcaption>Slowly Changing Dimension: Type 1, Data is changed buy history is not preserved</figcaption></figure><p>This type is useful when you are not much worried about tracking changes over time.</p><p><strong>In SCD Type2</strong>, Whenever there is a change, a new record is added to the table instead of deleting/replacing older one. Lets Take the same example of the Customer Table as above with now with a slight twist for SCD2. You can see 3 extra columns are added. We need to start a record, End Older record in case of change, and there is a flag for easier detection of the current value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*gUA6cRZqKFArMRuj6Yi-Xg.png" /><figcaption>Slowly Changing Dimension: Type 2, Where data is changed and history is also preserved using date columns and isCurrent Flag column</figcaption></figure><p>It is useful when you want to keep full track of changes over time for accurate reporting.</p><p><strong>In SCD Type 3</strong>, we keep only limited history by adding new columns to store previous value. So, you can see the current and one previous state, but no more than that. Lets Take a basic example using Employee Table as shown in below screenshot. You can see when Department is changed, We are keeping only previous dept. It would not contain multiple previous values, but only one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*qibYlNFQ1ANaQ3SIu6aVBQ.png" /><figcaption>Slowly Changing Dimension: Type 2, Where data is changed and history is also preserved but only previous value.</figcaption></figure><p>It works well when you want to track just the last change and not full history.</p><h4>Why Managing SCDs Matters:</h4><p>When you manage Slowly Changing Dimensions properly, it has a big impact on your reporting and analytics.</p><p>If changes in dimension data aren’t tracked correctly, your reports can show inaccurate or misleading information — for example, sales numbers tied to the wrong customer details.</p><p>Good SCD management ensures the data accuracy by preserving historical context, so you can analyze the trends over time and make better business decisions.</p><blockquote>Hope you have found this explanation helpful and it made working with Slowly Changing Dimensions a bit easier for you.</blockquote><p><strong><em>Next up: In Part 2, we’ll dive into how to implement SCDs effectively, with practical tips and examples. Stay tuned.</em></strong></p><p>Check Part 2 here: <a href="https://abhinavsharma150.medium.com/mastering-slowly-changing-dimensions-part-2-learn-how-to-implement-scds-with-easy-examples-e67fa0490849">Mastering Slowly Changing Dimensions: Part 2 — Learn how to Implement SCDs with easy Examples.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01e235f343ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/mastering-slowly-changing-dimensions-part-1-demystifying-data-evolution-01e235f343ed">Mastering Slowly Changing Dimensions: Part 1 — Demystifying Data Evolution</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySpark Cheat Sheet: Part 1 — Installation, Setup, and Data Loading]]></title>
            <link>https://medium.com/analytics-vidhya/pyspark-cheat-sheet-part-1-installation-setup-and-data-loading-4d0bd5aa0aa8?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d0bd5aa0aa8</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[setup]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Fri, 02 Jun 2023 14:22:45 GMT</pubDate>
            <atom:updated>2023-06-02T14:22:45.031Z</atom:updated>
            <content:encoded><![CDATA[<h3>PySpark Cheat Sheet: Part 1 — Installation, Setup, and Data Loading</h3><p>Efficient Big Data Processing with Pyspark. Why Pyspark? What is Pysaprk?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXAepzStFSKbysU8WKQxvA.jpeg" /><figcaption>Photo by Neo: <a href="https://www.pexels.com/photo/lines-of-code-2653362/">https://www.pexels.com/photo/lines-of-code-2653362/</a></figcaption></figure><p>Welcome to the first part of our PySpark Cheat Sheet series. In this blog post, we’ll cover the essential steps for installing PySpark, setting up the environment, and loading data into PySpark for efficient big data processing. Let’s dive in!</p><h3>What is Pyspark?</h3><p>PySpark, the Python API for Apache Spark, is a powerful framework for efficient big data processing. Built on Apache Spark, PySpark combines the simplicity of Python with the scalability of Spark, enabling data engineers, data scientists, and analysts to work with large datasets. Let’s explore why PySpark has become a go-to tool for processing big data.</p><p>PySpark simplifies big data processing by leveraging Spark’s distributed computing capabilities. It allows you to write Python code while benefiting from Spark’s scalability, enabling fast and efficient processing of large datasets. PySpark integrates seamlessly with popular Python libraries such as NumPy, Pandas, and Scikit-learn, making it versatile for end-to-end data processing and machine learning workflows. With PySpark, you can tackle structured data, perform complex analytics, and build machine learning models at scale.</p><h3>Installation and Setup</h3><p>PySpark requires Apache Spark to be installed. Follow these steps to set up PySpark on your machine:</p><h4>Step 1: Install Java Development Kit (JDK):</h4><p>PySpark requires Java to run. Install the latest JDK by visiting the Oracle website or using a package manager like Homebrew (for macOS) or apt-get (for Ubuntu).</p><h4>Step 2: Download Apache Spark:</h4><p>Go to the official Apache Spark website (<a href="https://spark.apache.org/downloads.html">https://spark.apache.org/downloads.html</a>) and download the latest version of Apache Spark.</p><h4>Step 3: Extract Apache Spark:</h4><p>Extract the downloaded Spark archive using the following command:</p><blockquote><em>tar -xvf spark-&lt;version&gt;.tgz</em></blockquote><h4>Step 4: Set Environment Variables:</h4><p>Add the following lines to your .bashrc or .bash_profile file to set the necessary environment variables:</p><blockquote>export SPARK_HOME=/path/to/spark-&lt;version&gt;<br>export PATH=$PATH:$SPARK_HOME/bin</blockquote><h4>Step 5: Install PySpark:</h4><blockquote>pip install pyspark</blockquote><h3>Data Loading and Preparation</h3><p>Now that we have PySpark set up, let’s explore how to load and prepare data for processing.</p><h4>Loading Data from CSV</h4><p>To load data from a CSV file into a PySpark DataFrame, use the spark.read.csv() function:</p><blockquote>f<strong>rom pyspark.sql import SparkSession</strong></blockquote><blockquote><strong>spark = SparkSession.builder.getOrCreate()</strong></blockquote><blockquote><strong>df = spark.read.csv(‘path/to/data.csv’, header=True, inferSchema=True)<br>df.show()</strong></blockquote><p>The header=True option specifies that the first row of the CSV file contains column headers. The inferSchema=True option automatically infers the data types of columns.</p><h4>Loading Data from JSON</h4><p>To load data from a JSON file, use the spark.read.json() function:</p><p><strong>df = spark.read.json(‘path/to/data.json’)<br>df.show()</strong></p><h4>Loading Data from Parquet</h4><p>Parquet is a columnar storage format that is highly optimized for big data processing. To load data from a Parquet file, use the spark.read.parquet() function:</p><blockquote>df = spark.read.parquet(‘path/to/data.parquet’)<br>df.show()</blockquote><h3>Conclusion</h3><p>In this first part of our PySpark Cheat Sheet series, we covered the installation and setup process for PySpark. We also explored how to load data from CSV, JSON, and Parquet files into PySpark DataFrames. Stay tuned for the next part, where we’ll delve into data manipulation and analysis with PySpark.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d0bd5aa0aa8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/pyspark-cheat-sheet-part-1-installation-setup-and-data-loading-4d0bd5aa0aa8">PySpark Cheat Sheet: Part 1 — Installation, Setup, and Data Loading</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CHEAT SHEET — Learn Snowflake in 5 Days]]></title>
            <link>https://abhinavsharma150.medium.com/cheat-sheet-learn-snowflake-in-5-days-79960f6d4275?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/79960f6d4275</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cheatsheet]]></category>
            <category><![CDATA[snowpipe]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Tue, 23 Aug 2022 15:58:59 GMT</pubDate>
            <atom:updated>2025-06-30T18:24:58.745Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>CHEAT SHEET</strong> — Learn Snowflake in 5 Days</h3><p>Stepwise Approach and resources required to learn Snowflake</p><h3>What exactly is Snowflake?</h3><p>Snowflake is a Software as a Service (SaaS) offering. Basically a Big Data Warehouse with unlimited compute and Storage.</p><h3>How to Learn Snowflake Big Data Technology in easy Steps?</h3><h4><strong>1. Sign Up for a free account</strong></h4><p><strong>Link for Sign Up:</strong> <a href="https://signup.snowflake.com/">https://signup.snowflake.com/</a></p><p><strong>Video Help for Sign up:</strong> <a href="https://www.youtube.com/watch?v=AYVwv3K0MIw">Click here on Video Link</a> , Start at timestamp — 2:30</p><h4>2. <strong>Learn Basic SQL</strong></h4><p>Snowflake uses Basic SQL commands for everything.(Skip this if you already know SQL).</p><h4>3. Snowflake Architecture</h4><p><strong>Video Link:</strong> <a href="https://www.youtube.com/watch?v=C8_os0-ti48">Click here for Video Link</a> — 12 Mins</p><h4>4. Snowflake UI</h4><p><strong>Video Link for Snowflake UI : </strong><a href="https://www.youtube.com/watch?v=l6lwBrPuixY">Click here for Video Link </a>— 15 Mins</p><p><strong>Text Link for Understanding Snowflake UI:</strong> <a href="https://hevodata.com/learn/snowflake-ui/#:~:text=The%20Snowflake%20UI%20allows%20the,address%20through%20the%20user%20interface.">Click Here </a>— 5 Mins Read</p><h4>5. Working With Snowflake</h4><p><strong>Tables/Databases/Schema/Container Etc →</strong></p><p>a. <a href="https://www.youtube.com/watch?v=Pi3z1NyBd-Y&amp;list=PLba2xJ7yxHB7SWc4Sm-Sp3uGN74ulI4pS&amp;index=7">Click Here For Video Link</a> — 30 Mins</p><p>b. <a href="https://docs.snowflake.com/en/user-guide/getting-started-tutorial-create-objects.html">Click for Hands-on Documentati</a>on — 5 Mins</p><p><strong>Virtual Warehouses:</strong></p><p>Link for Virtual Warehouses : <a href="https://docs.snowflake.com/en/user-guide/warehouses-overview.html">Documentation 20 Mins Read</a></p><p>Video for Virtual Warehouse : <a href="https://www.youtube.com/watch?v=q4F4xIQVEh0\">Video Link 13 mins</a></p><p><strong>Staging Data on Local File →</strong></p><p><a href="https://docs.snowflake.com/en/user-guide/getting-started-tutorial-stage-data-files.html">Hands-on For Data Staging</a> — 5 Mins</p><p><strong>Stage Data from AWS S3 Bucket →</strong></p><p><strong>Option 1 : Click Here : </strong><a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html">Connect to S3 with Storage Integration</a></p><p><strong>Option 2 :</strong> <strong>Click Here:</strong> <a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-aws-iam-user.html">Connect to S3 with IAM User Credentials</a></p><blockquote><strong>Note : </strong>If you do not have an AWS account, You need to Create an AWS account first : <strong>Resources for creating free AWS account — </strong><a href="https://www.youtube.com/watch?v=L3TTvSk11MQ"><strong>click here on Video Link</strong></a><strong>— 7 mins video</strong></blockquote><p><strong>Data Loading into Snowflake</strong></p><p><strong>Copy into commands :</strong></p><p><a href="https://www.youtube.com/watch?v=81831yy_y64">Video Link 1 — Click here</a> – 8 mins short video</p><p><a href="https://www.youtube.com/watch?v=lI5OQPjuj-8&amp;list=PLba2xJ7yxHB7SWc4Sm-Sp3uGN74ulI4pS&amp;index=9">Video Link 2 — Click here </a>— 1 hour detailed video (<strong>Recommended</strong>)</p><p><strong>Continuous Ingestion (SnowPipe)</strong></p><p><strong><em>SnowPipe Resource links Below :</em></strong></p><p><a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro.html">Click Here for Official Documentation</a> — 15 mins Read</p><p><a href="https://www.youtube.com/watch?v=PNK49SJvXjE&amp;list=PLba2xJ7yxHB7SWc4Sm-Sp3uGN74ulI4pS&amp;index=10">Click Here for Video Link 1</a> –- 25 Mins Detailed</p><p><a href="https://www.youtube.com/watch?v=b5iqAiiHvPI">Click Here for Video Link 2</a> – 10 Mins (Less Detailed, but crisp, can be confusing to some)</p><p><strong>Snowflake Tasks and Streams</strong></p><p><strong>Tasks And Streams</strong> —</p><p>Link for <a href="https://community.snowflake.com/s/article/Using-Streams-and-Tasks-inside-Snowflake">Snowflake community Resource </a>— 7 mins read</p><p><a href="https://www.youtube.com/watch?v=me4chjq0JCQ">Short Video for Task and Stream</a> — 50 Mins</p><p>Detailed Videos on Tasks and Streams :</p><p><strong>Tasks </strong>: <a href="https://www.youtube.com/watch?v=165hXZ_o4Vk">Video Link for Snowflake Tasks </a>, <strong>Streams </strong>: <a href="https://www.youtube.com/watch?v=DXI0GDSwE_E">Video Link for Snowflake Streams</a></p><p><strong>Slowly Changing Dimensions:</strong></p><p><strong>Link for SCD:</strong></p><p><a href="https://medium.com/@abhinavsharma150/mastering-slowly-changing-dimensions-part-1-demystifying-data-evolution-01e235f343ed">Mastering Slowly Changing Dimensions: Part 1</a> – 5 mins Read</p><p><a href="https://abhinavsharma150.medium.com/mastering-slowly-changing-dimensions-part-2-learn-how-to-implement-scds-with-easy-examples-e67fa0490849">Mastering Slowly Changing Dimensions: Part 2 </a>– 4 mins Read</p><p><strong>Time Travel In Snowflake:</strong></p><p><a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html">Documentation link</a> — 15 Mins Read</p><p><a href="https://www.youtube.com/watch?v=IK1LwL_Xl_4">Time Travel Video Link 1</a> –13 Mins</p><p><a href="https://www.youtube.com/watch?v=3jkyhA0L3Xw">Time Travel Video Link 2</a>– 10 Mins</p><p><strong>Data Sharing In Snowflake</strong></p><p><a href="https://www.youtube.com/watch?v=2d6d-zZMXtA&amp;list=PLba2xJ7yxHB7SWc4Sm-Sp3uGN74ulI4pS&amp;index=16">Video Link 1</a>–25 Mins</p><p><a href="https://www.youtube.com/watch?v=GcrEK-VFEB8">Video Link 2</a>–15 Mins</p><p><a href="https://docs.snowflake.com/en/user-guide/data-sharing-intro.html">Official Documentation for Data Sharing</a> — 10 Mins Read</p><p><strong>Role Based Access Control</strong></p><p><a href="https://www.youtube.com/watch?v=b-YRXJgjDC8">Video Link</a>–10 Mins.</p><p><strong>Caching In Snowflake</strong> — <a href="https://www.youtube.com/watch?v=hHA2oGLZiVU">Video Link </a>— 6 Mins, <a href="https://community.snowflake.com/s/article/Caching-in-Snowflake-Data-Warehouse">Documentation link </a>— 3 mins read</p><p><strong>Micro Partitioning and Clustering in Snowflake </strong>— <a href="https://www.youtube.com/watch?v=rAFanMOQNhw">Video Link</a> — 10 Mins</p><p><strong>Resource Monitoring </strong>: <a href="https://www.youtube.com/watch?v=aSSOKOttjT0">Video Link</a></p><p>That covers everything…</p><p>It something is missing, Please feel free to add in comments, I will happily add the topic here along with the resources.</p><p>Thanks You …….</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79960f6d4275" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Big Data Technology? What? Why? Where? and How?]]></title>
            <link>https://medium.com/analytics-vidhya/the-big-data-technology-what-why-where-and-how-fdeac3dc1c7b?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdeac3dc1c7b</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Mon, 22 Aug 2022 17:08:33 GMT</pubDate>
            <atom:updated>2022-08-27T15:40:26.177Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*71TTPkOJaZrLAD9z4EEoDw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mbaumi?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Mika Baumeister</a> on <a href="https://unsplash.com/s/photos/spreadsheet?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Also Read, What is <strong>Data Lake</strong>? The difference between <strong>Databases, Data Warehouses, </strong>and <strong>Data Lakes.</strong></p><p>The Digital world means tons of apps, websites, social media pages, and more. What value does it generate? It generates leisure, pleasure, ease, and automation. But, the most elemental attribute it generates is the <strong>Data.</strong></p><p><strong>What is Data? </strong>It is anything in a digital form, be it a fact, a figure, or a stat. It can be a pic of your loving dog, an entire movie, or your identification details. Every one of the things mentioned in one or some other form is data. When the size of the data becomes big, it gets difficult to manage and can be called big data. But, how to define it? And what to do with it.? Let’s dive deep into it with 3W’s and 1 H. What, Why, Where, and How.</p><h3><strong>What is Big Data?</strong></h3><p>When the data exceeds one or more of the 4 V’s, it can be considered Big. What are those 4 V’s?</p><ol><li><strong>Volume:</strong> Simply the size of the data.</li><li><strong>Velocity: </strong>The speed with which data is generated.</li><li><strong>Variety:</strong> The various types, can be images, text files, videos, gifs, or something else.</li><li><strong>Veracity:</strong> How relevant is the data? What is its truthfulness?</li></ol><h3>Why use Big Data?</h3><p>You might have experienced sometimes how we instantly recognize a song, sometimes people get intuition when it is about to rain or something bad is going to happen, or we almost know when it is going to be a goal in football. Why is that? Because of patterns. Our brain can detect, learn and recognize patterns, given we have sufficient data points. The same goes with Data Analysis, and ML/AI technologies, they work on patterns.</p><p>The more data, the more patterns we have, the more stats, the more models, and thus the better and more precise prediction. A basic example is YouTube recommendations, even if you log in with a new account and watch some videos for a day, YouTube will detect patterns for you and put your IP in a category of people who think like you, and then recommend per your taste. Because it already has millions of rows of YouTube history of thousands of people who think like you.</p><p>Data patterns can help predict what should be kept on a Walmart shelf in certain weather or what could be the temperature for the next 20 days, or what Instagram feed would their customer like. The patterns can even tell what disease might a person living in a particular area or from a particular gene pool can get in the future. All this is possible with Big Data.</p><h3>Where is the Big Data Used?</h3><p>It has many use cases. A few of them are listed below.</p><ol><li><strong>Natural Language Processing (NLP)</strong>, is an ML/AI-specific technology used for computers to read and understand human text and language with the help of keywords.</li><li><strong>Image Detection and Classification</strong>, another ML/AI tech, can detect faces, objects, sizes, and borders in images and videos. Used for face detection, robotic vision to enable robots to work, fault detection in industries, etc.</li><li><strong>Weather Analysis</strong>, to read weather patterns to issue meteorological warnings to fishermen, farmers, ships, airplanes, and common people.</li><li><strong>Traffic Optimization</strong>, finding the fastest routes, traffic bottlenecks, and ensuring a smooth driving experience for people.</li><li><strong>Crime prediction and prevention</strong>, real-time intelligence models can be created based on CCTV feeds across the city to predict an alert wherever and whenever required.</li><li><strong>Ad targeting,</strong> make ads reach the correct set of audience, with help of data. The recent use of deep fakes in the Indian Ad Industry by companies like Zomato and Cadbury to show localized versions of Ads is an interesting use case.</li></ol><h3>How Big Data is implemented?</h3><p>There are several tools and technologies already available in the market. I will list down a few.</p><h3>Data Storage:</h3><ol><li><strong>Data Lakes,</strong> like Cloud Blob Storage like Amazon S3, Azure Blob or GCP Storage Bucket, Apache Hadoop HDFS.</li><li><strong>Data Warehouses,</strong> like Snowflake, Google BigQuery, Amazon Redshift, Azure Synapse Analytics, and IBM Db2.</li><li><strong>Databases</strong>, like AWS Dynamo DB, Azure Cosmos Db, Oracle, MongoDB, and MySQL.</li></ol><h3>Data Transformation:</h3><p>Tools Like Apache Spark, Hadoop, Talend, Informatica Power Centre, Dataiku (works on top of Snowflake), etc.</p><h3>Data Streaming:</h3><p>Apache Kafka, Google Cloud Dataflow, Amazon Kinesis, Stream SQL, Spark Streaming, etc.</p><h3>Data Visualization:</h3><p>Tableau, QlikView, Microsoft Power BI, Oracle Visual Analyzer, Databricks Visualization, Pandas with Matplotlib, etc.</p><h3>Bonus Topic: The Data Lake</h3><p><strong>What is a Data Lake?</strong></p><p>Data Lake is a large storage space, that is designed to store and secure great variety and an immense amount of data in its native type. This data can be of any size, any scale, and can be structured, unstructured, or semi-structured.</p><p><strong>How Data Lake is different that Data Warehouse?</strong></p><p>Data Warehouse can hold data at a high scale, but the data needs to be in a structured format, in rows and columns. The data in the data warehouse may be used in the future, so it is kept in format for quick analysis.</p><p>On the other hand, Data Lake acts as a data dump, it would contain all the data but in native format. Consider some Ad-hoc analysis is required, a transformation can be done and data can be made ready as per demand.</p><p><strong>How Database is different than Data Warehouse?</strong></p><p>The database is used for operational purposes, like bank transactions, customer purchases, etc. It captures and maintains the current data. The database is updated per transaction. The database is usually <a href="https://www.ibm.com/cloud/learn/oltp">OLTP-type</a>.</p><p>A Data warehouse is more likely to be used as a historical data dump. Which can create multiple years of history and is used more for analysis. The data here is kept for and according to the purpose of analysis e.g. tables would be created not similar to transactional purpose but would have tables for analysis. It is updated in batch mode in the scheduled process. The database here is usually <a href="https://www.ibm.com/cloud/learn/olap">OLAP-type</a>.</p><p>That was all about Big Data and the Type of Databases. I hope it cleared your concepts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdeac3dc1c7b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/the-big-data-technology-what-why-where-and-how-fdeac3dc1c7b">The Big Data Technology? What? Why? Where? and How?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ten Unwritten Rules for a Coding Beginner]]></title>
            <link>https://medium.com/analytics-vidhya/ten-unwritten-rules-for-a-coding-beginner-5c58d414dea4?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c58d414dea4</guid>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[how-to]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Sun, 27 Sep 2020 18:23:54 GMT</pubDate>
            <atom:updated>2020-10-02T17:02:39.379Z</atom:updated>
            <content:encoded><![CDATA[<h4>An “Easy to Follow” Rule-Book for a Rookie Coder.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGwG5AJ4P88K2bNO_upjOQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lucabravo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Luca Bravo</a> on <a href="https://unsplash.com/s/photos/coding?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Moving to the third decade of the century, Programming is one of the most desired skills. Along with the inflow of more and more digital trends, Coding is gaining more prominence in the skill market. As the result there are many new coders coming to the market every year … We learn from our experiences, but we can also learn from other’s experience.</p><p>So if you are new, try following these rules which were very effective and helpful to me. Hopefully they would help you too.</p><h3><strong>1. No Need to Rush, But No Excuse to Slow Down</strong></h3><p><strong>Fix consistent number of hours per day and commit yourself to it</strong>. Do not worry, You have just started. It will take its course. But don’t skip, <strong>Think twice before a Skipping even a day</strong>. It can easily become a habit.</p><h3>2. Learn To Implement</h3><p>Never read a topic and then move on to the next. Learn it to Implement it. <strong>Read a Topic, Think of a situation where it can be used, Apply It. </strong>If nothing comes in mind, Google for it, Write some Short Code, Debug and Run. But never leave the topic until implemented or you will most probably forget it.</p><h3><strong>3. The Basics are the Foundation</strong></h3><p><em>“A Skyscraper cannot be made on a Weak Base”. </em>You can’t go high if your basics are weak. <strong>Read the Basics, Apply the Basics, Revise the Basics</strong>. You must have a good knowledge of Libraries, OOPS Concepts, Data Types, Memory Allocation and all other basic stuff for the language.</p><h3>4. Always Keep your Code in Classes and Functions</h3><p><strong>This is important.</strong> Doesn’t matter the Code is of <strong>Two Lines</strong> or <strong>Two Hundred, </strong>Write it by making <strong>Functions </strong>at least<strong> if not Classes</strong>. Don’t Write long Paras of Code. <strong>Keep it Structured, Not Freestyle. </strong>“<strong><em>It is a Code, not a Hip-hop” .</em></strong></p><h3>5. Never Copy-Paste the Code</h3><p>Remember, <strong>You are a Noob, Not a Pro. </strong>Write it yourself, Never Copy Paste. Copy-Paste make the job Quick and Easy, but being a new learner you will keep forgetting it and it will slow down your grasping while writing big codes, The Real Codes.</p><h3><strong>6. Errors are Good, Errors are Important</strong></h3><p>Getting Errors in the code? Don’t Worry. No one can code without Errors. <strong>Embrace the Error, Learn from the Error</strong>. Be Patient, Debug the Errors, think, If not getting the answer, Search on Stackoverflow. But try not to repeat the same error many times.</p><h3>7. Don’t Just Learn it, But Live it</h3><p><em>“Human is a social Animal”. </em>No matter how much introvert you are, you survive on Socialisation. Same goes with Programming. <strong>Join Facebook groups</strong>, <strong>Read Coding Blogs, Watch Tutorials, Read Comments</strong> <strong>there</strong>, <strong>Follow Coding Meme Pages on Instagram, Discord, Reddit. </strong>There is a whole community of Coding, Be a part of it.</p><h3>8. Free Projects are Goldmine</h3><p>Thousands of free Projects are on internet, there is a whole community working for you, for people like us. <strong>Read free Projects, See the Flow of Code. Read how the Pros do it. Explore Github, Try to understand it.</strong> Doesn’t matter if you are new, You will get it in some time giving a try.</p><h3>9. Break your Code into Smaller Pieces, Add Comments on Important lines</h3><p><em>Try writing a complex piece of code and read it after a month, You will not Understand your own code</em></p><p>Always write in small pieces, <strong>Small codes are Fast, Manageable, Clean and Easy to Understand. </strong>Add comments to define what you are doing. It makes the <strong>Code Maintenance much Easier</strong>. Remember that you are not the only one who is going to read and work on your code.</p><h3>10. Sit like a Monk, Not Like a Monkey</h3><p>Yes it is funny, but it holds too much value. Ask anyone who sits for long hours. <strong>Posture is important</strong>. You can only work if you are healthy. Coding for long hours <strong>Sitting Improperly</strong> can make serious <strong>Posture Deformities</strong> as well as <strong>Cardio-Vascular diseases</strong>. Most prominent one is <strong>Carpel-Tunnel. </strong>Do<strong> Stretching Exercises at intervals, Stay Hydrated, Eat on time</strong> to stay healthy and fit. It is a sedentary job but it is too much on brain. Relax on regular intervals and exercise daily.</p><h4><strong>And Last but not the Least:</strong></h4><h3>Contribute to the Coding Community:</h3><p>You have taken too much from them, coders can’t be coders if there is no Community Support. It is a selfless work, an honest one. Think it as a payback. Answer queries of various Platforms, Contribute to Github, help people on coding.</p><h4>I really hope this piece of work will help you.</h4><h4>Thanks for reading this!!!!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c58d414dea4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/ten-unwritten-rules-for-a-coding-beginner-5c58d414dea4">Ten Unwritten Rules for a Coding Beginner</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a Telegram News-Bot in less than 10 Minutes]]></title>
            <link>https://levelup.gitconnected.com/create-a-telegram-news-bot-in-less-than-10-minutes-e558e3fd1db6?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/e558e3fd1db6</guid>
            <category><![CDATA[automate]]></category>
            <category><![CDATA[api-news]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[telegram-bot]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Wed, 23 Sep 2020 15:58:07 GMT</pubDate>
            <atom:updated>2020-09-24T17:18:49.791Z</atom:updated>
            <content:encoded><![CDATA[<h4>An easy Setup using Telegram Bot-API, Powered by <a href="https://newsapi.org/">NewsAPI.org</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJMlSskm4YqOhY2WDz5UkA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@christianw?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Christian Wiediger</a> on <a href="https://unsplash.com/s/photos/telegram?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Telegram is a Russian Cloud-Based IM, VOIP, Video-Telephony Software Service.Telegram Client Apps are available on all the major platforms and APIs are available for independent developers. We will be using Bot API here.</p><h3>Create a Bot with BotFather</h3><p>Install and open <strong>Telegram</strong> and search <strong>BotFather. </strong>Type <strong>/newbot</strong> and then give a name to it. You will get a token, save it somewhere safe for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fzr1Y8qIugPBfkSaT0qA3Q.jpeg" /></figure><h4>The Bot API requests</h4><p>You can get messages by easy Get Methods, just make sure to write bot along with token. Example if your token is <strong>234563:37fhh48fhsnbahfe</strong> then it will go as <a href="https://api.telegram.org/bot959342943:AAE20yYuip5BBY4p7HmpdlSJJ1mhUPswo3w/getupdates"><strong>https://api.telegram.org/bot</strong></a><strong>234563:37fhh48fhsnbahfe</strong><a href="https://api.telegram.org/bot959342943:AAE20yYuip5BBY4p7HmpdlSJJ1mhUPswo3w/getupdates"><strong>/getupdates</strong></a></p><p>The Recent messages can be retrieved with the Update ID with Offset <a href="https://api.telegram.org/bot959342943:AAE20yYuip5BBY4p7HmpdlSJJ1mhUPswo3w/getupdates"><strong>https://api.telegram.org/bot</strong></a><strong>234563:37fhh48fhsnbahfe</strong><a href="https://api.telegram.org/bot959342943:AAE20yYuip5BBY4p7HmpdlSJJ1mhUPswo3w/getupdates"><strong>/getupdates</strong></a><strong>?offset=UpdateID</strong></p><h3>Setting up The NewsAPI</h3><p>Go to <a href="https://newsapi.org/"><strong>https://NewsAPI.org</strong></a><strong> </strong>and create an account there. Its a free service for developers.</p><p>A key will be generated, Which can be sent in the URL or inside the headers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWy9bmsOm4FYGALmlRriNA.png" /></figure><h3><strong>Now the Coding Part</strong></h3><p>The whole code is available on GitHub on my repository <a href="https://github.com/abhinav1321/news_bot_with_telegram">https://github.com/abhinav1321/news_bot_with_telegram</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IeTUoGi9IrQCv_529YNuzA.png" /></figure><h4><strong>The Code</strong></h4><p>Sharing GitHub Gist below, Alternatively you can go here <a href="https://github.com/abhinav1321/news_bot_with_telegram">https://github.com/abhinav1321/news_bot_with_telegram</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e10af0a5630fdb7de067f30b9b3d514/href">https://medium.com/media/5e10af0a5630fdb7de067f30b9b3d514/href</a></iframe><h3>Lets Run this and see</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkRbpn_-BUiqZxFjTPTRCw.jpeg" /></figure><blockquote>Thank You for reading this. I hope You found it useful and enjoyed it.</blockquote><h3>Good Bye!!!! Have a Great Day Ahead</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e558e3fd1db6" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/create-a-telegram-news-bot-in-less-than-10-minutes-e558e3fd1db6">Create a Telegram News-Bot in less than 10 Minutes</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Didn’t understand Heap? Lets Implement and know.]]></title>
            <link>https://medium.com/analytics-vidhya/didnt-understand-heap-lets-implement-and-know-eeb80395198?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeb80395198</guid>
            <category><![CDATA[heap-implementation]]></category>
            <category><![CDATA[priority-queue]]></category>
            <category><![CDATA[heap]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 18:36:14 GMT</pubDate>
            <atom:updated>2020-09-25T13:34:45.626Z</atom:updated>
            <content:encoded><![CDATA[<h3>Didn’t understand Heap? Let’s implement and know.</h3><p>A Sample case of Priority Queue, giving preference to Urgent Businesses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IcZBAdoVJNlzLP6mfBo6qg.jpeg" /></figure><h3><strong>The Heap Data Structure</strong></h3><p>Binary Heap is a “<em>Tree Based Data Structure”</em> having a complete “Binary Tree”.</p><blockquote>Heaps Can Be classified into two, <strong>MIN HEAP</strong> and <strong>MAX HEAP</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*geISo4yT4ZSF4S4IsIGqpQ.png" /><figcaption><strong>MIN HEAP</strong>: Both The Left Child and Right Child must be greater than Parents thus min on top whereas in the <strong>MAX HEAP : </strong>The parent is greater than both the Child, So Maximum goes onto the top</figcaption></figure><h3>Use Case of Heap, <strong>The Priority Queue.</strong></h3><p>A Heap is useful when there is constant addition and removal of data. Suppose we have a list of elements and we need to draw the highest element in one go but meanwhile, there is a live addition of elements is going on. Let’s understand it with below example.</p><p><strong>A case of Call Centre</strong></p><p>There is a call centre somewhere, employing half a dozen call attendees, the calls are bursting in, some calls being <strong>Basic Issues</strong> while a few of them <strong>The</strong> <strong>Real Businesses</strong>. How would an Automated System identify and Prioritise this? Here, using a Priority Queue is a feasible option and thus <strong>The Heap Data Structure</strong> becomes a perfect candidate to recruit. See the below Image for a better understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*9YfS4RBgR8gN-WZ4ybjqjg.png" /><figcaption>Calls Will be Prioritised on the based on their Importance</figcaption></figure><p>We can Create Heap ourselves, But Python has an Inbuilt Library named “<strong>heapq” </strong>and to keep the code short, we are using it<strong>. </strong>If you still want to code Heap for yourself ,You can find a suitable code for yourself from a list of articles available on “geeksforgeeks.org” →<a href="https://www.geeksforgeeks.org/medium/heap/2/"><strong>Click Here</strong></a><strong>.</strong></p><p>So lets Come to the Coding Part to get the essence of it.</p><h3><strong>Starting Up</strong></h3><pre>#Importing Libraries<br>import heapq<br>import time<br>import random<br>import schedule</pre><pre># Adding some calls into the list of tuples<br># Priority(1 to 5), ‘Caller name’)<br>call_queue = [<br> (3, ‘c1’),<br> (1, ‘c4’),<br> (4, ‘c3’),<br> (5, ‘c2’),<br> (2, ‘c5’),<br> ]</pre><pre># heapify function makes a heap of the given list<br>heapq.heapify(call_queue)</pre><h3><strong>Adding Functions</strong></h3><p><strong>Random Call Addition : </strong>Since we are creating an example, I am Adding extra Random calls in Queue by scheduling for certain time.</p><pre>def add_call():</pre><pre>    # With some Random Priorities and Random users<br>    call_tuple = (<br>              random.randint(1,5), <br>              random.choice([&#39;c13&#39;,&#39;c15&#39;,&#39;c17&#39;,&#39;c18&#39;,&#39;c11&#39;,&#39;c14&#39;,]))<br>              <br>    print(&#39;New Incoming call in queue, Caller: &#39;,call_tuple[1],<br>          &#39;| Priority :&#39;,call_tuple[0])<br>    <br>    heapq.heappush(call_queue,call_tuple)</pre><p><strong>Popping out “Next Priority” Call</strong></p><pre>def call_manager(): <br>if len(call_queue)==0:<br>     return<br> else:<br>     popped = heapq.heappop(call_queue)<br>     print(‘Call Started for: ‘,popped[1],’ Priority: ‘,popped[0])</pre><h3><strong>Scheduling the Jobs and Running the Schedule</strong></h3><pre># Scheduling Sample Calls<br>schedule.every(3).seconds.do(call_manager)<br>schedule.every(10).seconds.do(add_call)</pre><pre># Running the Schedule<br>while True:<br>    schedule.run_pending() <br>    time.sleep(1)</pre><p><strong>Output : </strong>Lets See what is Happening in the Terminal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*vwqQcNhbrxHEPyb3eAO0Ug.png" /><figcaption>Whenever be a “High Priority Call” is Added, It is getting “First Priority”</figcaption></figure><p>This was an Example of Priority Queue Based on Min Heap. There can be other uses like Finding Min, Max or Median from a list which is constantly appending new elements, Finding a time period with maximum Active Covid cases. Finding the day with minimum or maximum Population.</p><p>In Next Blog for Heap, I will create an example for above Call Centre case most probably as an App.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*nrsXFKYGdPBWKdaA0zyl8Q.jpeg" /></figure><blockquote><strong>Hope you enjoyed this Piece.</strong></blockquote><blockquote><strong>Please like and follow.</strong></blockquote><blockquote>Good Bye !!!!! Have a Nice Day</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeb80395198" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/didnt-understand-heap-lets-implement-and-know-eeb80395198">Didn’t understand Heap? Lets Implement and know.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set Up a Text Message API With Python in 5 Minutes.]]></title>
            <link>https://medium.com/swlh/set-up-a-text-message-api-with-python-in-5-minutes-fdfc46b744c3?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdfc46b744c3</guid>
            <category><![CDATA[sms]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[rest-api]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Tue, 08 Sep 2020 15:42:25 GMT</pubDate>
            <atom:updated>2020-09-21T06:21:47.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B8LMbgRdkHHhLtG3RYvCxA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lastly?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tyler Lastovich</a> on <a href="https://unsplash.com/s/photos/iphone?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Digitally connected world use automated messages in heavy way. Bills, rent, One Time Passwords there are unlimited scopes of an Automated text.</p><p>Suppose you own a mini store and a new item arrives and you want to notify 50 to 100 of your clients. How would you do?, Easy— Text them, Automatically.</p><p>Lets setup a SMS Client for quick learning, I found this website <a href="https://control.textlocal.in/">www.textlocal.in</a> with reasonable prices in India. It is providing Bulk message options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xFcj9ex9iG8t8Yfc9xX-Gg.png" /></figure><p><strong>Create an Account</strong>, They give 10 credits, so lets get trying.</p><p>After Sign-up go to Settings &gt;API &gt; <strong>Generate a new API Key </strong>and save it somewhere safe. You can allow Specific IPs for security, But lets keep the fields empty for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28HFkCgKSYGjeyhpK5_-9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*F_D6eggdc3zycNzi-0Q9tw.png" /></figure><p>Requests will go to link <a href="https://api.textlocal.in/%7Bcommand%7D"><strong>https://api.textlocal.in/{command}</strong></a><strong> </strong>with command-name in command and data in headers<strong>.</strong></p><p><strong>Lets Start Coding Part</strong></p><p>Libraries : For now Just One &gt; “<strong>import requests”</strong></p><pre>import requests<br>url = ‘<a href="https://api.textlocal.in/&#39;">https://api.textlocal.in/&#39;</a></pre><pre>params = {‘username’:’<a href="/service/mailto:abhinavsharma150@gmail.com">yourusername@mailid.com</a>’,<br>          ‘apiKey’:’yourGenerated-APIKey’<br>         }</pre><p>Lets write some Functions():</p><pre>def check_balance(url):<br>    url = url+&#39;balance&#39;<br>    response = requests.get(url,params=params)<br>    return response.json()<br></pre><pre>def send_sms(url,params):<br>    url=url+&#39;send&#39;</pre><pre>    #Phone numbers inside braces {} in commas<br>    numbers={&#39;9911111xxxx&#39;}<br>    message = {&#39;Hi, This is a Sample message&#39;}<br>    params[&#39;numbers&#39;] = numbers<br>    params[&#39;message&#39;] = message</pre><pre>    response = requests.post(url,params=params)<br>    return response.json()</pre><pre><br>def inbox(url):<br>    url = url+&#39;get_inboxes&#39;<br>    response = requests.get(url,params=params)<br>    return response.json()</pre><p>Thats it, Code finished!! In 5 minutes you can setup SMS message for single or bulk messages.</p><p>In Next Part I will show you to make a quick app to send message, and OTP confirmation too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*JzfZRRCXFC7gHb43plaK_g.jpeg" /></figure><h3>Till then, Good Bye!!!!</h3><p>Priority queue with Heap → <a href="https://medium.com/@abhinavsharma150/didnt-understand-heap-lets-implement-and-know-eeb80395198"><strong>Didn’t understand Heap? Let’s implement and know</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdfc46b744c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/set-up-a-text-message-api-with-python-in-5-minutes-fdfc46b744c3">Set Up a Text Message API With Python in 5 Minutes.</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Reporting with Python (Part 3) : Scheduling, Creating and Sending periodical COVID19…]]></title>
            <link>https://medium.com/analytics-vidhya/automated-reporting-with-python-part-3-scheduling-creating-and-sending-periodical-covid19-842f6794421d?source=rss-57d390b69974------2</link>
            <guid isPermaLink="false">https://medium.com/p/842f6794421d</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[reporting]]></category>
            <category><![CDATA[gmail-api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[scheduling]]></category>
            <dc:creator><![CDATA[ABHINAV SHARMA]]></dc:creator>
            <pubDate>Mon, 24 Aug 2020 15:23:09 GMT</pubDate>
            <atom:updated>2020-09-08T08:31:59.312Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automated Reporting with Python (Part 3) : Scheduling, Creating and Sending periodical COVID19 data as Reports.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d2pqL9LlfBH8xBe4cYFvkw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@tobiasrehbein?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tobias Rehbein</a> on <a href="https://unsplash.com/s/photos/covid-data?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>You can find Previous 2 parts here →</p><p>Part 1 → <a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-1-generating-pivot-tables-and-charts-and-saving-to-html-ecd6590cd1b1"><strong>Pivot Tables, Charts and reports to HTML</strong></a></p><p>Part 2 → <a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-2-converting-an-html-file-to-pdf-and-sending-automated-2fb85eec89e9"><strong>HTML to PDF and automated mails by Gmail API</strong></a></p><p>Suppose every day, week or n number of times in a day, you need to fetch or download data from a source like some API link, shared drive, or maybe you are filing it manually. You are creating a report and sending to your boss.Its repetitive, boring as well as time consuming.Why don’t Automate it?</p><p>In this part, I will be explaining a <strong>periodical reporting mechanism for sending automated reports in mail</strong>.</p><p>For illustration purpose I am taking COVID19 data from below link <a href="https://api.covid19india.org/csv/latest/district_wise.csv">https://api.covid19india.org/csv/latest/district_wise.csv</a></p><p>I recommend you to use both Jupyter Notebook and some IDE, If not available use text editor- Sublime Text3, It works with Python well.</p><p>Create two seperate files : <strong>report.py </strong>and <strong>scheduler.py</strong></p><p>Can do it with IDE or use <strong>touch</strong> command on terminal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*FkPlUkgoYpZujtvUFcE_nw.png" /></figure><p>We will work on Jupyter for simplicity first and then will move the code to .py files</p><p><strong>Importing Libraries</strong></p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt</pre><pre>import requests<br>import datetime</pre><p><strong>Fetching the data </strong>:</p><pre>link = &#39;<a href="https://api.covid19india.org/csv/latest/district_wise.csv&#39;">https://api.covid19india.org/csv/latest/district_wise.csv&#39;</a></pre><pre>response = requests.get(link)<br>content = response.content</pre><pre># Name the file as current day<br>file = open(&#39;covid/covid_data_&#39;<br>            +str(datetime.date.today())+<br>            &#39;.csv&#39;,&#39;wb&#39;)<br>file.write(content)</pre><pre>df = pd.read_csv(&#39;covid/covid_data_&#39;+str(datetime.date.today())+&#39;.csv&#39;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*c4281BoCsrClmIfgQNQzKQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*Tazuk6uWMV9FRLbBu35vVg.png" /><figcaption><strong>df.head()</strong> shows top rows, while <strong>df.columns</strong> shows name of the columns</figcaption></figure><p>Wrangling: lets grouping it State wise</p><pre>state_wise_data = df.groupby(‘State’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*u2mw_yetF-e9Sc5xH8kbFg.png" /></figure><p>We will select the data of most affected states by Covid. <strong>Lets sort the dataframe on ‘Confirmed’ column.</strong></p><pre>state_wise_data = df.groupby(&#39;State&#39;)</pre><pre>state_data = []</pre><pre>for state,data in state_wise_data:<br># summing all the districts data[columns-&gt;confirmed:deceased ] to a variable</pre><pre>row_list=[]<br>    row_list.append(state)<br>    for d in data.iloc[:,5:14].sum():<br>        row_list.append(d)<br>    state_data.append(row_list)</pre><pre>new_columns=[&#39;state&#39;]<br>for col in df.columns[5:14]:<br>    new_columns.append(col)</pre><pre>new_data = pd.DataFrame(state_data,<br>                        columns=new_columns<br>                       ).sort_values(<br>                        by=[&#39;Confirmed&#39;],<br>                        ascending=False<br>                                       )<br>new_data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*W9rWtsRVrtSxGtmbxkMB5w.png" /></figure><p>lets get first 5 rows and first five columns from sorted dataframe</p><pre>df_to_plot = new_data.iloc[:5,:5]</pre><p>Plotting it :</p><pre>plt=df_to_plot.plot(kind=’bar’,width=0.5,figsize=(10,5))<br>plt.set_xticklabels(df_to_plot[‘state’],rotation=’30&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*BIhufbIRIHEzDVbro0g3ig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*6jzTZMSbJFSJ2MMEJ5jJ5Q.png" /></figure><p><strong>Saving to HTML, Conversion to PDF, Gmail API </strong>: → It has already been explained in <a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-1-generating-pivot-tables-and-charts-and-saving-to-html-ecd6590cd1b1"><strong>Reporting Part 1</strong></a> and <a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-2-converting-an-html-file-to-pdf-and-sending-automated-2fb85eec89e9"><strong>Reporting Part 2</strong></a><strong> &lt;- </strong>Please click to open<strong>.</strong></p><pre>from datetime import datetime<br>#Converting to HTML</pre><pre># saving plot image to local file<br>image = plt.get_figure().savefig(‘covid_plot.png’)<br>image_tag = ‘&lt;img src=”covid_plot.png”&gt;’</pre><pre>#Table<br>table = data_to_plot.set_index(‘state’)</pre><pre>#writing HTML Content<br>heading = ‘&lt;h1&gt; Scheduled Covid Report&lt;/h1&gt;’</pre><pre># Using .now() from datetime library<br>now = datetime.now()<br>current_time = now.strftime(“%m/%d/%Y %H:%M:%S”)<br>header = ‘&lt;div class=”top”&gt;’ + heading +’&lt;/div&gt;’</pre><pre>footer = ‘&lt;div class=”bottom”&gt; &lt;h3&gt; This Report has been Generated on ‘<br>footer = footer+current_time +’&lt;/h3&gt; &lt;/div&gt;’</pre><pre>content = ‘&lt;div class=”table”&gt; ‘+ table.to_html()+’ &lt;/div&gt; &lt;div class=”chart”&gt; ‘<br>content = content+ image_tag +’&lt;/div&gt;’</pre><pre>html = ‘&lt;body&gt;’ + header + content + footer +’&lt;/body&gt;’</pre><pre># Writing the file<br>with open(‘covid_report.html’,’w+’) as file:<br> file.write(html)</pre><pre># HTML to PDF <br>import pdfkit<br>pdfkit.from_file(‘covid_report.html’,’covid_report.pdf’)</pre><p>Gmail API :- Explained in <a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-2-converting-an-html-file-to-pdf-and-sending-automated-2fb85eec89e9"><strong>Part </strong></a><strong>2 : </strong>Just pasting the code here</p><pre>from googleapiclient.discovery import build<br>from google_auth_oauthlib.flow import InstalledAppFlow<br>from google.auth.transport.requests import Request</pre><pre># Local Libraries<br>import base64<br>from email.mime.text import MIMEText<br>from email.mime.audio import MIMEAudio<br>from email.mime.application import MIMEApplication<br>from email.mime.multipart import MIMEMultipart</pre><pre>SCOPES = [‘<a href="https://mail.google.com/&#39;">https://mail.google.com/&#39;</a>]<br>class Main:</pre><pre>def __init__(self):<br> self.creds = None<br> # The file token.pickle stores the user’s access and refresh tokens, and is<br> # created automatically when the authorization flow completes for the first<br> # time.<br> if os.path.exists(‘credentials/token.pickle’):<br> with open(‘credentials/token.pickle’, ‘rb’) as token:<br> self.creds = pickle.load(token)<br> # If there are no (valid) credentials available, let the user log in.<br> if not self.creds or not self.creds.valid:<br> if self.creds and self.creds.expired and self.creds.refresh_token:<br> self.creds.refresh(Request())<br> else:<br> flow = InstalledAppFlow.from_client_secrets_file(<br> ‘credentials/credentials.json’, SCOPES)<br> self.creds = flow.run_local_server(port=0)<br> # Save the credentials for the next run<br> with open(‘credentials/token.pickle’, ‘wb’) as token:<br> pickle.dump(self.creds, token)</pre><pre>self.service = build(‘gmail’, ‘v1’, credentials=self.creds)</pre><pre>class CreateAndSend:</pre><pre>def __init__(self,service, to, sender, user_id, subject, message_text,attachment):</pre><pre>message = MIMEMultipart()<br> message[‘to’] = to<br> message[‘from’] = sender<br> message[‘bcc’] = ‘’<br> message[‘subject’] = subject<br> body = MIMEText(message_text)<br> message.attach(body)<br> <br> with open(attachment, “rb”) as f:<br> #attach = email.mime.application.MIMEApplication(f.read(),_subtype=”pdf”)<br> attach = MIMEApplication(f.read(),_subtype=”pdf”)<br> <br> attach.add_header(‘Content-Disposition’,<br> ‘attachment’,<br> filename=str(attachment))<br> message.attach(attach)<br> <br> to_send = {<br> ‘raw’: base64.urlsafe_b64encode(message.as_string().encode()).decode()<br> }<br> print(message)</pre><pre>try:<br> to_send = (service.users().messages().send(userId=user_id, body=to_send)<br> .execute())<br> print(‘Message Id: %s’ % to_send[‘id’])</pre><pre>except Exception as e:<br> print(‘An error occurred’, e)</pre><pre>main = Main()<br>service = main.service<br>to = ‘<a href="/service/mailto:abhinavsharma150@gmail.com">sender_id@something.com</a>’<br>sender = ‘’<br>user_id = ‘<a href="/service/mailto:mailstream99@gmail.com">gmailapiid@gmail.com</a>’<br>subject = ‘Automated Report mail’<br>message_text = ‘Hi,\n\n This report has been generated automatically for a demo purpose.\n\n\nSender Signature.’<br>attachment=’covid_report.pdf’</pre><pre>def send_mail():<br> CreateAndSend(service, to, sender, user_id, subject, message_text,attachment)</pre><pre>send_mail()</pre><p><strong>Scheduling the task with nohup (linux)</strong></p><p>nohup is a POSIX command, here ‘hup’ stands for ‘Hang Up’ . hup is sent as intimation to the system to hang up as the user has logged off , nohup intercepts this command and allow process to continue running.</p><p><strong>Syntax :</strong></p><pre><strong>nohup</strong> <em>command</em> [<em>command-argument</em> ...]</pre><pre><strong>nohup</strong> <em>command</em> [<em>command-argument</em> ...] &amp;</pre><p>The &amp; at the end of the command instructs bash to run command at foreground. Lets Try an Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*jf6bVLThUcOP84Q5v6HOjQ.png" /><figcaption>Creating Python File</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*DCYGXiwQD57bEGSdTyXwfA.png" /><figcaption>Editing Python file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*eTGHbYJUHOJ84vweBKTbng.png" /><figcaption>Runnimg as nohup</figcaption></figure><p>When we run nohup it creates an nohup.out file with output</p><p>Closing nohup process: It can be done by bringing process to foreground</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*Y2_3pWfK0fSykUHOZ8nK8A.png" /><figcaption>We can stop running jobs by bringing it to foreground by fg %n ,where n is the number of job</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/1*q4OdVW1q7L11PwwTwolCJw.png" /></figure><p>Now Lets create a scheduler.py and write some code in it</p><p><strong>Python Schedule Library </strong>click here<strong> →</strong><a href="https://pypi.org/project/schedule/#:~:text=Schedule%20lets%20you%20run%20Python,and%20the%20clockwork%20Ruby%20module."><strong>Schedule Library</strong></a><strong> :- </strong>for official documentation. can be downloaded by <strong>pip3 install schedule</strong></p><p>Lets Try Scheduling report every 3 minutes.</p><pre>import schedule<br>import time,datetime<br>import os</pre><pre>def run_report():<br> # os.system runs system commands same as in os<br> os.system(‘python3 report.py’)</pre><pre>schedule.every(2).minutes.do(run_report)</pre><pre>while True:<br> schedule.run_pending()<br> time.sleep(1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*lVFkIvYceyY3ccl67lN7Vg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*p0xvfxc3LjbH8cmrHPSEHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*Cq3zwqZEKImbmu_88zv8zw.png" /></figure><p><strong>Full Code Below:</strong></p><ol><li><strong>Report.py</strong></li></ol><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import requests<br>import datetime<br>import os</pre><pre>link = ‘<a href="https://api.covid19india.org/csv/latest/district_wise.csv&#39;">https://api.covid19india.org/csv/latest/district_wise.csv&#39;</a><br>response = requests.get(link)<br>content = response.content<br># Name the file as current day<br>file = open(‘covid_data_’<br> +str(datetime.date.today())+<br> ‘.csv’,’wb’)</pre><pre>file.write(content)<br>df = pd.read_csv(‘covid_data_’+str(datetime.date.today())+’.csv’)</pre><pre>state_wise_data = df.groupby(‘State’)</pre><pre>state_data = []<br>for state,data in state_wise_data:<br># summing all the districts data[columns-&gt;confirmed:deceased ] to a variable<br> row_list=[]<br> row_list.append(state)<br> for d in data.iloc[:,5:14].sum():<br> row_list.append(d)<br> state_data.append(row_list)<br> new_columns=[‘state’]<br> for col in df.columns[5:14]:<br> new_columns.append(col)</pre><pre>new_data = pd.DataFrame(state_data,<br> columns=new_columns<br> ).sort_values(<br> by=[‘Confirmed’],<br> ascending=False<br> )</pre><pre># Lets plot first 5 rows<br>df_to_plot = new_data.iloc[:5,:5]</pre><pre>plt=df_to_plot.plot(kind=’bar’,width=0.5,figsize=(10,5))<br>plt.set_xticklabels(df_to_plot[‘state’],rotation=’30&#39;)</pre><pre># to HTML<br>from datetime import datetime<br>#Converting to HTML<br># saving plot image to local file<br>image = plt.get_figure().savefig(‘covid_plot.png’)<br>image_tag = ‘&lt;img src=”covid_plot.png”&gt;’<br>#Table<br>table = df_to_plot.set_index(‘state’)<br>#writing HTML Content<br>heading = ‘&lt;h1&gt; Scheduled Covid Report&lt;/h1&gt;’<br># Using .now() from datetime library<br>now = datetime.now()<br>current_time = now.strftime(“%m/%d/%Y %H:%M:%S”)<br>header = ‘&lt;div class=”top”&gt;’ + heading +’&lt;/div&gt;’<br>footer = ‘&lt;div class=”bottom”&gt; &lt;h3&gt; This Report has been Generated on ‘<br>footer = footer+current_time +’&lt;/h3&gt; &lt;/div&gt;’<br>content = ‘&lt;div class=”table”&gt; ‘+ table.to_html()+’ &lt;/div&gt; &lt;div class=”chart”&gt; ‘<br>content = content+ image_tag +’&lt;/div&gt;’<br>html = ‘&lt;body&gt;’ + header + content + footer +’&lt;/body&gt;’<br># Writing the file<br>with open(‘covid_report.html’,’w+’) as file:<br> file.write(html)</pre><pre># HTML to PDF <br>import pdfkit<br>pdfkit.from_file(‘covid_report.html’,’covid_report.pdf’)</pre><pre># Running Gmail File to Send mail<br>os.system(‘python3 gmail.py’)<br></pre><p>3. gmail.py</p><pre># Google API Libraries<br>from __future__ import print_function<br>import pickle<br>import os.path<br>from googleapiclient.discovery import build<br>from google_auth_oauthlib.flow import InstalledAppFlow<br>from google.auth.transport.requests import Request</pre><pre># Local Libraries<br>import base64<br>from email.mime.text import MIMEText<br>from email.mime.audio import MIMEAudio<br>from email.mime.application import MIMEApplication<br>from email.mime.multipart import MIMEMultipart</pre><pre>SCOPES = [‘<a href="https://mail.google.com/&#39;">https://mail.google.com/&#39;</a>]<br>class Main:</pre><pre>def __init__(self):<br> self.creds = None<br> # The file token.pickle stores the user’s access and refresh tokens, and is<br> # created automatically when the authorization flow completes for the first<br> # time.<br> if os.path.exists(‘../credentials/token.pickle’):<br> with open(‘../credentials/token.pickle’, ‘rb’) as token:<br> self.creds = pickle.load(token)<br> <br># If there are no (valid) credentials available, let the user log in.<br> if not self.creds or not self.creds.valid:<br> if self.creds and self.creds.expired and self.creds.refresh_token:<br> self.creds.refresh(Request())<br> else:<br> flow = InstalledAppFlow.from_client_secrets_file(<br> ‘../credentials/credentials.json’, SCOPES)<br> self.creds = flow.run_local_server(port=0)<br> # Save the credentials for the next run<br> with open(‘../credentials/token.pickle’, ‘wb’) as token:<br> pickle.dump(self.creds, token)</pre><pre>self.service = build(‘gmail’, ‘v1’, credentials=self.creds)</pre><pre>class CreateAndSend:</pre><pre>def __init__(self,service, to, sender, user_id, subject, message_text,attachment):</pre><pre>message = MIMEMultipart()<br> message[‘to’] = to<br> message[‘from’] = sender<br> message[‘bcc’] = ‘’<br> message[‘subject’] = subject<br> body = MIMEText(message_text)<br> message.attach(body)<br> <br> with open(attachment, “rb”) as f:<br> #attach = email.mime.application.MIMEApplication(f.read(),_subtype=”pdf”)<br> attach = MIMEApplication(f.read(),_subtype=”pdf”)<br> <br> attach.add_header(‘Content-Disposition’,<br> ‘attachment’,<br> filename=str(attachment))<br> message.attach(attach)<br> <br> to_send = {<br> ‘raw’: base64.urlsafe_b64encode(message.as_string().encode()).decode()<br> }<br> print(message)</pre><pre>try:<br> to_send = (service.users().messages().send(userId=user_id, body=to_send)<br> .execute())<br> print(‘Message Id: %s’ % to_send[‘id’])</pre><pre>except Exception as e:<br> print(‘An error occurred’, e)</pre><pre>main = Main()<br>service = main.service<br>to = ‘to_recipient<a href="/service/mailto:abhinavsharma150@gmail.com">@g</a>mail.com’<br>sender = ‘’<br>user_id = ‘your_api_gmail_id@gmail.com’<br>subject = ‘Automated Report mail’<br>message_text = ‘Hi,\n\n This report has been generated automatically for a demo purpose.\n\n\nSender Signature.’<br>attachment=’covid_report.pdf’</pre><pre>def send_mail():<br> CreateAndSend(service, to, sender, user_id, subject, message_text,attachment)</pre><pre>send_mail()</pre><p>3. scheduler.py</p><pre>import schedule<br>import time<br>import datetime<br>import os</pre><pre>def run_report():<br> # os.system runs command same as system command line <br> os.system(‘python3 report.py’)</pre><pre># Just select or edit the way the scheduling is needed from below <br>schedule.every(2).minutes.do(run_report)</pre><pre>#-------- SOME MORE WAYS IT CAN BE SCHEDULED-------<br># schedule.every(10).minutes.do(job)<br># schedule.every().hour.do(job)<br># schedule.every().day.at(“10:30”).do(job)<br># schedule.every(5).to(10).minutes.do(job)<br># schedule.every().monday.do(job)<br># schedule.every().wednesday.at(“13:15”).do(job)<br># schedule.every().minute.at(“:17”).do(job)</pre><pre>while True:<br> schedule.run_pending()<br> time.sleep(1)</pre><p>If any doubt in Report making or Gmail API , you can read my previous two blogs, links given below</p><blockquote><a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-1-generating-pivot-tables-and-charts-and-saving-to-html-ecd6590cd1b1">Automated Reporting with Python (Part 1): Generating Pivot tables and Charts and saving to HTML</a></blockquote><blockquote><a href="https://medium.com/@abhinavsharma150/automated-reporting-with-python-part-2-converting-an-html-file-to-pdf-and-sending-automated-2fb85eec89e9">Automated Reporting with Python (Part 2) : Converting an HTML file to PDF and sending Automated Mails with Gmail API</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*CbCAKQYoZHQ5s_e9MGZGMw.jpeg" /></figure><p>If any issues with the blog ,any correction or editing you seem fit, feel free to comment.</p><p><strong>Good Bye!!!!!!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=842f6794421d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/automated-reporting-with-python-part-3-scheduling-creating-and-sending-periodical-covid19-842f6794421d">Automated Reporting with Python (Part 3) : Scheduling, Creating and Sending periodical COVID19…</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>