<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shritam Kumar Mund on Medium]]></title>
        <description><![CDATA[Stories by Shritam Kumar Mund on Medium]]></description>
        <link>https://medium.com/@shritam?source=rss-42e0902ac2c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1H-gsa_GyOUdo_4tMGznWg.jpeg</url>
            <title>Stories by Shritam Kumar Mund on Medium</title>
            <link>https://medium.com/@shritam?source=rss-42e0902ac2c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:58:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shritam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Papa Gifted Me the Bhagavad Gita on My Birthday]]></title>
            <link>https://shritam.medium.com/papa-gifted-me-the-bhagavad-gita-on-my-birthday-3ac1a0b9da37?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ac1a0b9da37</guid>
            <category><![CDATA[spirituality]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[lifestyle]]></category>
            <category><![CDATA[war]]></category>
            <category><![CDATA[krishna]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 18:59:34 GMT</pubDate>
            <atom:updated>2025-06-17T18:59:34.739Z</atom:updated>
            <content:encoded><![CDATA[<p>Today was my birthday.<br>And Papa gifted me the Bhagavad Gita.</p><p>I’ve heard so much about this book.<br>Some say it opens your mind in ways you can’t imagine.<br>Some say it breaks you down before it builds you up.</p><p>Over the years, my parents have given me many gifts.<br>But this one felt different.<br>Not expensive. Not fancy.<br>But heavy with meaning.</p><p>It didn’t feel like just a gift.<br>It felt like a message.<br>Like Papa was trying to say something.<br>Not with words.<br>But through a book.</p><p>It felt like a moment he’d been waiting for.<br>Like he knew there would be a time to pass it on.<br>And today was that day.</p><p>I felt a strange mix of things,<br> Curiosity<br> Nervousness<br> Responsibility</p><p>This wasn’t like when he taught me to walk, holding my hand.<br>Now, he’s watching from a distance.<br>Still guiding me.<br>But silently.</p><p>I haven’t read a single page yet.<br>But just holding it…<br>It feels serious.<br>It feels like a beginning.</p><p>What if I’m not ready?<br>What if it changes me?<br>What if it doesn’t?</p><p>So many thoughts.<br>So many questions.<br>And somehow, that feels right.<br>Because the Gita starts with a war.<br>A mind full of doubts.<br>A man at a crossroads.</p><p>Maybe that’s where I am too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IlIPmS7iP56rGtIWxB0Nfg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ac1a0b9da37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Data Apps with Streamlit on Snowflake: A Step-by-Step Guide]]></title>
            <link>https://shritam.medium.com/building-data-apps-with-streamlit-on-snowflake-a-step-by-step-guide-f011522b25f7?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f011522b25f7</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Tue, 03 Oct 2023 03:37:14 GMT</pubDate>
            <atom:updated>2023-10-03T03:37:14.743Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Data to Delight</h4><p>Last month, I had the opportunity to attend the <a href="https://www.snowflake.com/events/data-cloud-world-tour-bangalore/">Snowflake Data Cloud World Tour</a>, which introduced me to the fascinating world of Snowflake. During the event, I discovered numerous exciting insights, including the ability to create Data Apps within the Snowflake dashboard and effortlessly share them with the world. This inspired me to explore building Data Apps within the Snowflake platform. In this article, we will explore the process of creating Data Apps on the Snowflake data dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6IaMjCmwwQYcary3vaSG3g.png" /></figure><h3>Article Overview</h3><ol><li>Introduction to Streamlit</li><li>Why Building with Snowflake ❄️</li><li>Creating Data App</li><li>Considerations and Limitations</li><li>Wrapping Up with a Conclusion</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8X9k8km7fZpZY_KJnKlUAQ.jpeg" /></figure><h3>Introduction to Streamlit:</h3><p>Imagine turning your Python scripts into shareable web apps effortlessly, all without needing any front-end expertise. That’s exactly what <a href="https://streamlit.io/">Streamlit</a> does. In recent years, it has become the go-to tool for creating data apps in Python. It’s so popular that 80% of Fortune 50 companies use it, and it has captured the hearts of hundreds of thousands of Data Engineer worldwide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*q0Z8s_rt599i8n57vPlWvg.gif" /></figure><h3>Why Building with Snowflake ❄️</h3><p>Streamlit has been a trusted name in the data engineering world, with a history of enabling data app development since 2018. It has consistently been the top choice for building Data Apps. Major industry giants such as <a href="https://www.blackrock.com/">BlackRock</a>, <a href="https://www.instacart.com/">Instacart</a>, <a href="https://www.lacework.com/">Lacework</a>, and others were already harnessing its power to create customer-facing apps within Snowflake.</p><p>Recognising the growing industry demand, Snowflake made an exciting move in early 2022 — <a href="https://techcrunch.com/2022/03/02/snowflake-acquires-streamlit-for-800m-to-help-customers-build-data-based-apps/">they acquired Streamlit</a>. Their objective was crystal clear: democratise data apps by merging the best data application framework with a secure, large-scale deployment platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*T9GtUF-hugjnStBokO6y2g.png" /></figure><h3>Creating Data Apps</h3><h4>Step 1: Signup</h4><ul><li><a href="https://signup.snowflake.com/?utm_cta=trial-en-www-homepage-top-right-nav-ss-evg&amp;_ga=2.74406678.547897382.1657561304-1006975775.1656432605&amp;_gac=1.254279162.1656541671.Cj0KCQjw8O-VBhCpARIsACMvVLPE7vSFoPt6gqlowxPDlHT6waZ2_Kd3-4926XLVs0QvlzvTvIKg7pgaAqd2EALw_wcB">Create an Account</a> with AWS in the US Oregon Region</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*E8yM61cVXtlWIO3t6ppYXA.png" /></figure><h4>Step:2 Logging In and Activating Your 30-Day Free Trial</h4><ul><li>Sign in using your credentials to access your snowflake account, which comes with a complimentary 30-day trial.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vw2Lk8po39c_vksVgtvuLw.png" /></figure><h3>Creating Data App</h3><h4>What are we relay gonna make in this article?</h4><p>We will beworking on the development of a hit rate calculator for DAAS projects, utilizing the Snowflake Marketplace and Streamlit. Now, you might be wondering what exactly hit rate is. Allow me to explain.</p><p>Ever wondered how websites maintain their performance while accommodating relentless web crawlers? Hitrate — a strategic gem that helps Data Engineer determine the optimal number of hits a website can handle without overloading the server or risking being blocked. It’s the secret sauce for achieving a seamless web crawling experience.</p><h4>What is the method to determine the maximum hit rate?</h4><p><a href="https://www.similarweb.com/">Similarweb</a> provides valuable analytics and web traffic data for domains, enabling us to determine the daily or hourly request limits. This information assists us in planning the timing for URL crawling based on our specific needs.</p><p>To achieve this I found Similarweb’s Traffic Growth dataset on <a href="https://app.snowflake.com/qtblbua/si95041/#/data/shared/SNOWFLAKE_DATA_MARKETPLACE/listing/GZT1ZA3NJL?originTab=databases&amp;database=GLOBAL_PUBLIC__PRIVATE_COMPANIES_TRAFFIC_GROWTH">Snowflake’s marketplace</a>. I added it to my database by simply clicking on ‘<strong>Get</strong>’.</p><h4>How to calculate Hitrate?</h4><p>In this dataset, while there are numerous columns, we’re primarily interested in just two, for hit rate calculations:</p><ol><li><strong>TOTAL_ESTIMATED_VISITS</strong>: The total number of website visitors.</li><li><strong>TOTAL_PAGES_PER_VISIT</strong>: The average number of pages per visit.</li></ol><p>Now, let’s talk about playing nice with web servers. Just like in real life, online politeness is crucial. Web servers have rules, both unspoken and explicit, that dictate how fast we can crawl their content. To stay on their good side, we should limit our requests to just 8–10% of their capacity per day. It’s all about being a courteous web guest!</p><blockquote>hitrate = (<strong>TOTAL_ESTIMATED_VISITS</strong>*<strong>TOTAL_PAGES_PER_VISIT</strong>)/30*(<strong>Politeness Policy</strong>/100)</blockquote><p>Alright, now that you’ve got the math behind the calculation down, let’s put it into action.</p><h4>Set Up Your Streamlit Environment</h4><p>To build your Streamlit application in Snowsight(Snowflake UI), follow these straightforward steps:</p><ol><li>Click on Streamlit on the left navigation menu.</li><li>Click on + Streamlit App on the top right</li><li>Enter App name</li><li>Select Warehouse (X-Small) and App location (Database and Schema) where you’d like to create the Streamlit applicaton. If you encounter a ‘no database found’ error, simply navigate to the ‘Data’ section in the menu bar, create a new database, and then give it another try.</li><li>Click on Create</li></ol><p>Ta-daaaa!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLoQaDaElWBwJRyMlXu0Ng.png" /></figure><p>This is a sample application that allows you to make instant code changes on the left and test them on the right for your data app.</p><p>Since I already have Similarweb’s Traffic Growth dataset in my database, I wrote a SQL query to extract the required data from it.</p><pre>SELECT CLEAN_SITE, YEAR, MONTH, MAIN_CATEGORY, TOTAL_ESTIMATED_VISITS, TOTAL_PAGES_PER_VISIT<br>        FROM datafeeds.GLOBAL_GROWTH<br>        WHERE <br>        clean_site = &#39;amazon.com&#39;<br>        AND year = 2022<br>        AND month</pre><p>Let’s enable this query using Streamlit and design a stunning user interface for it:</p><pre>import streamlit as st<br>import re<br>import json<br>import requests<br>from math import floor<br>from snowflake.snowpark.context import get_active_session<br>import pandas as pd<br>import plotly.express as px<br><br>session = get_active_session()<br><br># Create a dictionary to map month numbers to names<br>month_mapping = {<br>    1: &#39;January&#39;,<br>    2: &#39;February&#39;,<br>    3: &#39;March&#39;,<br>    4: &#39;April&#39;,<br>    5: &#39;May&#39;,<br>    6: &#39;June&#39;,<br>    7: &#39;July&#39;,<br>    8: &#39;August&#39;,<br>    9: &#39;September&#39;,<br>    10: &#39;October&#39;,<br>    11: &#39;November&#39;,<br>    12: &#39;December&#39;<br>}<br><br>def similarweb_data_query(domain):<br>    # Load CO2 emissions data<br>    snow_df = session.sql(f&quot;&quot;&quot;<br>        SELECT CLEAN_SITE, YEAR, MONTH, MAIN_CATEGORY, TOTAL_ESTIMATED_VISITS, TOTAL_PAGES_PER_VISIT<br>        FROM GLOBAL_PUBLIC__PRIVATE_COMPANIES_TRAFFIC_GROWTH.datafeeds.GLOBAL_GROWTH<br>        WHERE <br>        clean_site = &#39;{domain}&#39;<br>        AND year = 2022<br>        AND month<br>    &quot;&quot;&quot;)<br>    <br>    #view the data table<br>    #st.table(snow_df)<br>    <br>    df = snow_df.to_pandas()<br>    # Use map to replace month numbers with names<br>    df[&#39;MONTH&#39;] = df[&#39;MONTH&#39;].map(month_mapping)<br>    <br>    TOTAL_ESTIMATED_VISITS_avg = df[&#39;TOTAL_ESTIMATED_VISITS&#39;].mean()<br>    #st.write(TOTAL_ESTIMATED_VISITS_avg)<br>    <br>    TOTAL_PAGES_PER_VISIT_avg = df[&#39;TOTAL_PAGES_PER_VISIT&#39;].mean()<br><br>    # Create a bar plot using Plotly Express<br>    fig = px.bar(<br>        df,<br>        x=&quot;MONTH&quot;,<br>        y=&quot;TOTAL_ESTIMATED_VISITS&quot;,<br>        labels={&quot;MONTH&quot;: &quot;Month&quot;, &quot;TOTAL_ESTIMATED_VISITS&quot;: &quot;Total Estimated Visits&quot;},<br>        title=f&quot;Total Estimated Visits vs. Month for {domain}&quot;,<br>    )<br>    <br>    st.plotly_chart(fig)<br><br>    <br>    return TOTAL_ESTIMATED_VISITS_avg, TOTAL_PAGES_PER_VISIT_avg<br><br><br>def human_format(num):<br>    num = float(&#39;{:.3g}&#39;.format(num))<br>    magnitude = 0<br>    while abs(num) &gt;= 1000:<br>        magnitude += 1<br>        num /= 1000.0<br>    return &#39;{}{}&#39;.format(&#39;{:f}&#39;.format(num).rstrip(&#39;0&#39;).rstrip(&#39;.&#39;), [&#39;&#39;, &#39;K&#39;, &#39;M&#39;, &#39;B&#39;, &#39;T&#39;][magnitude])<br><br><br>def result(Hit_rate,domain):<br>    Hit_rate_readable = human_format(Hit_rate)<br><br>    col1,col2 = st.columns(2)<br>    col1.metric(&quot;Hitrate&quot;, str(Hit_rate))<br>    col2.metric(&quot;Compact Hitrate&quot;, str(Hit_rate_readable))<br>    st.info(f&quot;Your DAAS project can make a maximum of {Hit_rate_readable} daily hits for {domain}&quot;)<br><br><br>    <br>magnitudeDict={0:&#39;&#39;, 1:&#39;Thousand&#39;, 2:&#39;Million&#39;, 3:&#39;Billion&#39;, 4:&#39;Trillion&#39;, 5:&#39;Quadrillion&#39;, 6:&#39;Quintillion&#39;, 7:&#39;Sextillion&#39;, 8:&#39;Septillion&#39;, 9:&#39;Octillion&#39;, 10:&#39;Nonillion&#39;, 11:&#39;Decillion&#39;}<br>def simplify(num):<br>    num=floor(num)<br>    magnitude=0<br>    while num&gt;=1000.0:<br>        magnitude+=1<br>        num=num/1000.0<br>    return(f&#39;{floor(num*100.0)/100.0} {magnitudeDict[magnitude]}&#39;)<br><br><br>#hitrate from webpage<br>def hitrate(domain,pp):<br>    hit_data = similarweb_data_query(domain)<br>    visit = int(float(hit_data[-1]))<br>    PagePerVisit = float(hit_data[0])<br>    print(visit, PagePerVisit)<br>    hitrate = (visit*PagePerVisit)/30*(pp/100)<br>    return result(hitrate,domain)<br><br><br>st.title(&quot;❄️ Hit Rate Calculator ❄️&quot;)<br>#st.header(&quot;**Enter the domain name: **&quot;)<br>try:<br>    form_1 = st.form(key=&#39;my-form1&#39;)<br>    url = form_1.text_input(&quot;Enter the domain name:&quot;)<br>    pp = form_1.number_input(&quot;Politeness Policy %: &quot;,10)<br>    submit = form_1.form_submit_button(&#39;Submit&#39;)<br><br>    if submit:<br>        st.header(&quot;**Result**&quot;)<br>        if pp&gt;10:<br>            st.success(f&quot;It&#39;s Recommended To Follow The Politeness Policy: 10%.&quot;)<br>        hit_web_page = hitrate(url,pp)<br>        <br>except IndexError:<br>    #submit = form.form_submit_button(&#39;Submit&#39;)<br>    st.success(f&#39;Input more then a Trillion is out of scope!&#39;)<br>except Exception as e:<br>    #submit = form.form_submit_button(&#39;Submit&#39;)<br>    st.write(e)<br>    st.success(f&#39;Please Enter Valid Input!!&#39;)</pre><p>Ready Steady Go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4z5OICxfv4ynGz3tLJqD5g.gif" /></figure><p>Awesome! We’ve just witnessed how Snowflake can be leveraged to build real-world data applications, and it’s never been this easy and efficient before.</p><h3>Considerations and Limitations</h3><ul><li>Please note that Streamlit is currently in Preview when integrated with Snowflake. Preview features are primarily provided for evaluation and testing purposes and should not be used in production systems or with production data.</li><li>Streamlit apps in Snowflake have a 16MB data retrieval limit per query. Going beyond this limit triggers a ‘MessageSizeError: Data Size exceeds message limit’ error.</li><li>Many Streamlit features, such as file_uploader, cache, and download_button, are currently unsupported in Streamlit within Snowflake.<a href="https://docs.snowflake.com/en/developer-guide/streamlit/limitations#unsupported-streamlit-features">Read more on this…</a></li></ul><h3>Wrapping Up</h3><p>As our community continues to grow, so does the demand for Data Apps. With more companies embracing modern data solutions, the need for quick and insightful decision-making tools has never been greater. In a vast landscape of transactional data spread across the internet, data companies have long sought a way to efficiently leverage their data for optimization.</p><p>Enter Snowflake — the game-changer that makes it all possible. And when paired with Streamlit, it becomes a dynamic duo, enabling users to make decisions faster and access valuable insights effortlessly, all on a single platform. Snowflake and Streamlit are shaping the future of data-driven decision-making and product enhancement. It’s a thrilling journey toward smarter and more efficient choices!</p><p>Join me on this data-driven adventure!</p><p>Connect with me on <a href="https://www.linkedin.com/in/ishritam/">LinkedIn</a> 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f011522b25f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AI War: Mastering the Art of Prompt Engineering in the Era of Large Language Models]]></title>
            <link>https://ai.plainenglish.io/the-ai-war-mastering-the-art-of-prompt-engineering-in-the-era-of-large-language-models-c50ae6af0188?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c50ae6af0188</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[language]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Mon, 13 Feb 2023 04:41:36 GMT</pubDate>
            <atom:updated>2023-02-15T01:31:25.342Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unleashing the Potential of AI with Strategic Prompt Engineering Techniques</h4><blockquote>The hottest new programming language is English.<br> — Andrej Karpathy</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3iiZEXKguGbWUzmI3WIKag.jpeg" /></figure><p>Artificial Intelligence has been the buzzword of the decade and one AI model that has been creating waves is none other than <strong>ChatGPT</strong>. This chatbot has taken the world by storm and has received an unexpected 1 billion visits on the web since its launch in November. From writing articles to composing emails and even passing top university exams, ChatGPT has proven to be the go-to tool for language-related tasks. Its ability to understand human language and generate coherent responses has been nothing short of extraordinary, making it the talk of the town.</p><p>The possibilities are endless. So buckle up, and get ready to explore the future with me. I promise you won’t be disappointed as I share my knowledge and insights on this exciting new article.</p><h3>Article Summary:</h3><ol><li>The inception and evolution of the AI war.</li><li>Large Language Models (LLMs)</li><li>What is Prompt Engineering?</li><li>Type of Prompting</li><li>Prompting Use Cases</li><li>Prompting Techniques</li><li>Limitations of LLMs</li><li>Conclusion</li></ol><h3>The inception and evolution of the AI war.</h3><p>Imagine a world where the inner workings of our own minds were a mystery. A hundred years ago, this was the reality. But then, a rival neuroscientist changed the game. His revolutionary findings brought human biology to life and sparked an explosion of discovery. The scene was set for one of the greatest scientific breakthroughs of all time. <strong>Santiago Ramón y Cajal</strong>, the father of modern neuroscience, fearlessly sliced into a cat’s brain and dipped it in watercolor, revealing the intricate web of neurons that make up our minds. He examined each part under a microscope, discovering patterns and connections that still astound us today. With this bold act, he opened the door to a new understanding of the human brain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*rFSVB6c3xJFbdwiwT3Pwyw.png" /><figcaption><strong>Santiago Ramón y Cajal with cat brain images</strong></figcaption></figure><p>Sounds like a scary tale right? That’s exactly how I felt when I read about him.</p><p>Later, for a few decades, so many different scientists conducted research. Later, a computer scientist named <strong>Jeffrey Hinton</strong> began working on it and proposed some future improvements in the AI direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*97WQwYyCsZEn5NWra2PANw.jpeg" /></figure><p>The origins of the modern AI movement can be traced back to <strong>September 30th, 2012</strong>. On this day, Jeffrey and his colleagues developed the first artificial deep neural network to be used on a widely recognized benchmark. And the rest is history.</p><h3>Large Language Models (LLMs)</h3><p>In simple terms, LLMs are computer programs that have been trained to understand and generate human language. They use vast amounts of text data to learn how words and sentences are put together as a prompt, and can then use that knowledge to perform tasks like answering questions, translating text, or even generating new text that is similar to what a human might write. These programs are called “large” because they have been trained on a huge amount of data, which allows them to produce more accurate and sophisticated results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tteBjZt-t9RvxN1HQ3wm7A.png" /><figcaption><a href="https://docs.cohere.ai/docs/prompt-engineering">https://docs.cohere.ai/docs/prompt-engineering</a></figcaption></figure><h3>What is Prompt Engineering?</h3><p>Prompt engineering is a method of controlling the outputs of a language model. It works by providing a starting point for the model, known as a prompt, and then allowing the model to generate the rest of the text based on what it has learned. By carefully crafting the prompt, we can guide the model to produce specific types of text as to our need.</p><h3>Type of Prompting</h3><p>One thing is clear in Prompting—</p><blockquote>High-quality questions will get High-quality answers.</blockquote><p>In general, there are 4 methods of prompting,</p><ol><li>Prompting by Instructions</li><li>Prompting by Role</li><li>Prompting by Examples</li><li>Chain of Thought Prompting</li></ol><h4>1. Prompting by Instructions</h4><p>One of the simplest promoting methods is just giving instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*2N6px1920gnOzDC0KEjd5g.png" /></figure><p>Even though we did not explicitly tell what changes need to do, the changes were made to remove any personally identifiable information (PII) and replaced it with the appropriate placeholders.</p><h4>2. Prompting by Role</h4><p>Another technique for prompting the AI is to assign it a role. For example, your prompt could begin with, “You are a doctor” or “You are a lawyer,” and then ask the AI to answer a medical or legal question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*klkB6ApFxSMPbiyQNCQdpA.png" /></figure><h4>3. Prompting by Examples</h4><p>Exemplars are specific examples of the task that the prompt is attempting to solve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*Hv1GpV9T4PdrodvC10Je3w.png" /></figure><h4>4. Chain of Thought Prompting</h4><p>Chain of Thought Prompting is a creative problem-solving technique that helps individuals generate ideas and solutions by prompting them to continue a chain of thought.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*7u_mtIO_W9tATEbDGwPWKQ.png" /></figure><h3>Use cases:</h3><p>The possibilities in prompt engineering are endless and exciting! Prompts can help the following use cases:</p><ol><li>Unleashing creativity with<strong> text generation.</strong></li><li>Get the gist with ease through <strong>text summarization.</strong></li><li>Break language barriers with <strong>translation.</strong></li><li>Find the answers you seek with <strong>Q&amp;A.</strong></li><li>Write <strong>Code and Debug</strong></li><li>Unleashing the imagination with stunning <strong>image generation</strong>.</li></ol><p>And the sky’s the limit! The best part?</p><p>Prompts can be tailored to each user, making the experience even more personalized. As the field grows, the potential for even more amazing developments is huge.</p><h3>Prompting Technique</h3><ul><li><strong>Length Control</strong>: As part of the prompt, we can specify a desired word count or character count.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*TzGUaAXuVE0t77T8KA6tWg.png" /></figure><ul><li><strong>Tone Control: </strong>The prompt can use specific words or phrases that indicate the desired tone to control the tone of the generated text.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*dexirapBCKgBQgtf1pxLBA.png" /></figure><ul><li><strong>Style Control: </strong>The prompt can specify the desired writing style to control the style of the generated text.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*WwN2jFp6NgKOQKsoTn_Vgw.png" /></figure><ul><li><strong>Audience Control: </strong>The prompt can be used to control the target audience of the generated text.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*ZpRRk-VwY3r5ZPYDmGY2EA.png" /></figure><ul><li><strong>Keyword-Based Control</strong>: The prompt can include keywords relevant to the desired output to guide the model toward specific outputs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*cf3IcK9gKexwZtfY4KMiEA.png" /></figure><ul><li><strong>Scenario-based Control</strong>: The prompt can describe a specific scenario in order to direct the model to generate text that fits that scenario.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*hhoKGncQwhAEphVEKTqfOQ.png" /></figure><ul><li><strong>Preventing Unwanted Outputs: </strong>The prompt can be used to exclude words from a response.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*SYEVUR9hMxQuDlDlVQMOKA.png" /></figure><h3>Limitations of LLMs</h3><blockquote>Every simple solution has its own set of difficult corners.</blockquote><ul><li><strong>The accuracy of the output</strong></li></ul><p>Google’s lost $100 billion in market value on a factual error in the first demo of its new AI, ChatGPT competitor, <strong>Bard.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*Qb1Eb-0EJAxldgE5GSIPLw.png" /><figcaption>B<a href="https://www.businesstoday.in/markets/global-markets/story/google-loses-over-100-billion-m-cap-after-chatbot-bard-gives-wrong-answer-in-ad-369572-2023-02-08">usiness Today</a></figcaption></figure><ul><li><strong>Training Data Bias</strong></li></ul><p>When it comes to training minority data groups, LLMs have shown bias. As a result, it is critical to improve the data transparency of the models in order to reduce bias in this technology.</p><ul><li><strong>Sustainability</strong></li></ul><p>There is a <a href="https://twitter.com/tomgoldsteincs/status/1600196981955100694?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1600196981955100694%7Ctwgr%5E7ee513747a55cf900311e3c6041a4cdb2368d116%7Ctwcon%5Es1_&amp;ref_url=https%3A%2F%2Fwww.anaconda.com%2Fblog%2Fthe-abilities-and-limitations-of-chatgpt">thread</a> on Twitter about how many GPUs are needed to run ChatGPT. The takeaway here is that running ChatGPT is extremely costly. LLM training will be prohibitively expensive as time passes.</p><p>If you want to learn more about this topic, I recommend reading this <a href="https://www.forbes.com/sites/johnkoetsier/2023/02/10/chatgpt-burns-millions-every-day-can-computer-scientists-make-ai-one-million-times-more-efficient/?sh=40c177e66944">Forbes article</a>.</p><h3>Conclusion</h3><p>The growth of big AI systems has created new possibilities in the field of AI. Thanks to Prompt Engineering, we now have the ability to guide and use these systems to meet our requirements. From creating new text to translating languages, the options are endless. By making prompts specific to each user, the potential for even more exciting advancements is huge. The future of AI is exciting and it’s time for us to take advantage of the opportunities it brings and become experts in Prompt Engineering.</p><h3>References:</h3><ol><li>Learn Prompt Engineering — <a href="https://learnprompting.org/">https://learnprompting.org/</a></li><li>GitHub link of best “Act As” Prompts — <a href="https://github.com/f/awesome-chatgpt-prompts">https://github.com/f/awesome-chatgpt-prompts</a></li><li>cohere.ai Documentation — <a href="https://docs.cohere.ai/docs/prompt-engineering">https://docs.cohere.ai/docs/prompt-engineering</a></li><li><a href="https://github.com/SudalaiRajkumar/Talks_Webinars">https://github.com/SudalaiRajkumar/Talks_Webinars</a></li><li>First Prompt Engineering Job Description — <a href="https://jobs.lever.co/Anthropic/e3cde481-d446-460f-b576-93cab67bd1ed">https://jobs.lever.co/Anthropic/e3cde481-d446-460f-b576-93cab67bd1ed</a></li></ol><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for more such content. 😃</p><p><a href="https://www.linkedin.com/in/ishritam/">Follow me on LinkedIn</a> for all the latest updates.🚀</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Interested in scaling your software startup</em></strong><em>? Check out </em><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c50ae6af0188" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/the-ai-war-mastering-the-art-of-prompt-engineering-in-the-era-of-large-language-models-c50ae6af0188">The AI War: Mastering the Art of Prompt Engineering in the Era of Large Language Models</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Big is BIGDATA?]]></title>
            <link>https://becominghuman.ai/how-big-is-bigdata-16d10b9a892e?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/16d10b9a892e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Fri, 14 Oct 2022 05:07:49 GMT</pubDate>
            <atom:updated>2022-10-18T17:10:35.415Z</atom:updated>
            <content:encoded><![CDATA[<h3>How big is BIGDATA?</h3><h4>No fuss, straight talk.</h4><p>One of the most popular catchphrases emphasizing the importance of data is:</p><blockquote>data is the new oil.</blockquote><p>During the COVID-19 pandemic of the last two years, the importance of quality data has skyrocketed. As a result, the demand for up-to-date, accurate data as well as <strong>methods for organizing data </strong>increased dramatically.</p><h3>Exactly how big is Big Data?</h3><p>Very big.</p><p>And now, time to wrap up… Kidding, kidding.</p><p>Calculating the real amount of data is challenging given the size of the internet. However, if we’re talking about the volume of data that qualifies as big data, we can classify it as such when it doesn’t fit on an Excel or Sheets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xt7fWT9WaGWI1VhZgxBmvg.jpeg" /></figure><h3>Article Summary</h3><ol><li>Small Data Vs Big Data</li><li>Need for Big Data</li><li>Data Storage</li><li>Batch Processing Vs Stream Processing</li><li>Scalability of Data</li><li>Hadoop Ecosystem</li><li>Extract Transform Load (ETL)</li><li>What is Spark?</li></ol><h3>Small Data Vs Big Data</h3><p>Let’s compare small data, sometimes referred to as traditional data, with big data in depth.</p><h4>Small data:</h4><ol><li>Small data is a fixed-size set of data that is often structured in a tabular fashion.</li><li>The small data can be derived up to MB, GB, or max to max TB.</li><li>Small data steadily grows in size. And the gradual process can occasionally be insignificant.</li><li>As a result of centralization, all users must access one server, which houses data on a single node.</li><li>Small data can be stored quickly in Excel files, SQL servers, and database servers.</li></ol><h4>Big data:</h4><ol><li>Large swaths of organized and primarily unstructured data can be used to represent Big data. Any type of format, including video, WARC, XML, etc., can be used for big data.</li><li>Big data can be kept in petabyte, exabyte, or yetabyte storage sizes.</li><li>Big data grows exponentially.</li><li>Globally data can be present and accessible. It is thus decentralized.</li><li>Big data requires extremely sophisticated software and technologies, such as Hadoop, Spark, Big Query, Hive, etc., to store the data.</li></ol><h3>Need of Big Data.</h3><p>Understanding the 4 V’s is necessary to comprehend why we need big data and what problems it answers. Let’s begin defining each one in turn.</p><ol><li><strong>Volume</strong>: People saw that their data volume was rapidly growing and that vast volumes of data were being stored.</li><li><strong>Velocity: </strong>Data was expanding at a lightning-fast rate all the time.</li><li><strong>Variety: </strong>The data formats were growing in all types of forms, including PDF, mp4, and images.</li><li><strong>Veracity: </strong>The data that was available was quite disorganized and required a lot of cleaning.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYuQSysgV60yru9yx5v6UQ.png" /></figure><p>There were challenges to face at each stage in order to overcome all of these. The three main difficulties were <strong>Storage</strong>, <strong>Processing</strong>, and<strong> Scalability</strong>.</p><h3>Data Storage:</h3><p>Data warehouses are designed to store large amounts of data in order to reduce storage complexity. The majority of data warehouses are of two types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*xBFN5to_jDWpXrRHpf4FXw.png" /></figure><ol><li><strong>Monolithic: </strong>Monolithic data warehouses imply that we put up a very large server and added a huge number of resources in order to store the data on a single server.</li><li><strong>Distributed:</strong> A distributed data warehouse is a collection of different servers linked together via a network. The idea is for these individual servers to appear as a single global data warehouse.</li></ol><h3><strong>Data Processing:</strong></h3><p>Data processing specifies two methods for processing data stored in warehouses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*NaMeKdgCKtIguSPyDHvhwA.png" /></figure><ol><li><strong>Transactional processing:</strong></li></ol><p>Stored data is utilized to perform <a href="https://medium.com/analytics-vidhya/fastapi-the-modern-age-api-framework-for-pythonista-4b2cd1e6652">CURD</a> operations on stored data inside warehouses. This processing includes all processes such as retrieving data from the warehouse, updating it to a different value, or deleting data from the warehouse. This is true for the vast majority of online apps and eCommerce websites.</p><p>2. <strong>Analytical processing:</strong></p><p>Analytical processing is required to visualize the data or generate a stat of the data. Analytic processing can be used to obtain insightful data information.</p><p>The majority of Machine Learning and Deep Learning processes rely on analytical processing to obtain useful data for training models.</p><p>Now that we’ve covered two categories of data processing, let’s look at two data processing techniques.</p><ol><li>Batch Processing</li><li>Stream Processing</li></ol><h4>Batch Processing</h4><p>Batch processing is the processing of large amounts of data in batches over a set period of time. It processes a large amount of data at once. When the data size is known and finite, batch processing is performed.</p><p>Batch processing is utilized in food processing systems, billing systems, and other similar applications.</p><h4>Stream Processing</h4><p>Stream processing is the processing of a continuous stream of data as it is produced. It analyses real-time streaming data. When the data size is unknown, unlimited, and continuous, stream processing is used.</p><p>Stream processing is utilized in the stock market, e-commerce transactions, social networking, and other applications.</p><p>Yea, you got it right. Twitter’s trending hashtags are also a type of stream processing. A framework called <strong>Kafka</strong> is used to store, read, and analyze streaming data. It has a streaming method for data redundancy and fault tolerance.</p><h3>Scalability of Data</h3><p>As the amount of data increased, engineers and scientists began investigating the issue, and for the first time in human history, Google took the initiative and released two research papers referring to the GFS technique (Google File System). Later in 2013, it was renamed to <a href="https://ieeexplore.ieee.org/abstract/document/6558077">HDFS</a> (Hadoop Distributed File System) under Apache.</p><h4>Hadoop Components</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBptPk3IacUiXQofV2Gccw.png" /></figure><p>Hadoop is a distributed, open-source system used to manage, store, and process large amounts of data. Let’s examine each of the three elements.</p><ol><li>Storage: <strong>HDFS</strong></li></ol><p>This is nothing more than a storage unit with a structure built to hold various data kinds. Two internal nodes are present, a worker node for data storage and a Manager Node to track metadata about data, such as file structure and directory contents.</p><p>2. Compute Engine: <strong>Map Reduce</strong></p><p>The computational engine, MapReduce, is in charge of making the process happen. The process is divided among various servers or locations using a map. And every piece of material is kept in the worker node. Reduce is used to merge the processes once the map activity is completed.</p><p>3. Resource Manager: <strong>YARN</strong></p><p>YARN is in charge of determining how the worker would get instructions to carry out the process.</p><h4>Extract Transform Load (ETL)</h4><p>As the amount of data, data sources, and data types in organizations grows, so does the importance of using that data in both transactional and analytical processing. ETL (extract, transform, and load) is the process that data engineers use to extract data from various sources, transform the data into a usable resource, and load that data into systems that end-users can access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y23JGg8elRFJLShvEn6uIA.png" /></figure><p>While ETL was critical, with the exponential growth of data sources and types, building and maintaining reliable data pipelines became one of the more difficult aspects of data engineering.</p><h4>Hadoop Ecosystem</h4><p>After a few successful years, engineers encountered challenges writing their ETL pipeline in Hadoop because the majority of the components were built in JAVA. Businesses like Facebook created their own Hadoop-like system called <strong>Hive</strong>.</p><p>But what was the main cause?</p><p>See, Facebook has always desired a system in which they wouldn’t have to hire numerous engineers with diverse backgrounds, such as JAVA, just to complete the process. Once more, they desired something that would serve as a mediator system that would allow them to use SQL queries to access the big data store.</p><p>However, because there were only a few components available, techies always desired a solution that could accommodate all of their needs as well as, most importantly, a variety of programming languages. With these drawbacks, <strong>Spark</strong>, one of the most potent big data tools was born.</p><h3>What hack is Spark?</h3><p>Spark is merely a distributed, general-purpose, in-memory computing engine.</p><p>One of the primary reasons Spark is ranked best in big data is because, while it was created in Scala, our ETL process can be written in Python, JAVA, SCALA, SQL, and R.</p><p>Spark replaces Hadoop’s map-reduce compute engine. As a result, Spark is not reliant on Hadoop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e0sQIfBnBxEtSznTzDpdEQ.png" /></figure><p>However, we continue to use HDFS as storage units and YARN as a resource manager. What if we decide not to utilize them?<br>Yes, we now have a variety of big data storage options to replace HDFS, such as <strong>AWS S3</strong>, and we can also replace YARN with<strong> Kubernetes</strong> or <strong>Apache Mesos</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NwRB1EKwnvSXpAw8LeukdA.png" /></figure><h3>Wrap Up</h3><p>Nowadays, whether we use a mobile device to open an app, run a Google search, or simply move around, data is continuously created.<br>Millions of data points are generated each time one of us opens a device or makes an online transaction.<br>The total amount of data generated worldwide is predicted to exceed 570 zettabytes by 2030.<br>When the question “how big is big data” is brought up, perhaps in 2030, it will be fascinating to observe how we will be handling such a large amount of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ITcUP20hwNAHJPDa37sITQ.jpeg" /></figure><p>The next post will be going to cover a deep dive into Spark and how we can start writing the data pipeline using pySpark (spark-based python framework).</p><p>I hope, you will find these insights useful for your use case, and let me know if you have any other suggestions for me too!</p><p>If you are passionate about this topic, or simply want to stay connected, do follow me on Medium.</p><h3>Thank you for reading!</h3><p><a href="https://www.linkedin.com/in/ishritam/">Follow me on LinkedIn</a> for all the latest updates.🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16d10b9a892e" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/how-big-is-bigdata-16d10b9a892e">How Big is BIGDATA?</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ONDC — E-commerce Revolution or A Hype?]]></title>
            <link>https://shritam.medium.com/ondc-e-commerce-revolution-or-a-hype-c9bca0a0d39?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9bca0a0d39</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[economy]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[indian-economy]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Tue, 24 May 2022 09:16:53 GMT</pubDate>
            <atom:updated>2022-05-24T09:36:07.882Z</atom:updated>
            <content:encoded><![CDATA[<h3>ONDC — E-commerce Revolution or A Hype?</h3><p>ONDC — Open Network for Digital Commerce</p><p>Recently, the term “ONDC” has been floating around all over the internet for the past few days. It is expected to change the whole Indian online commercial economy. In this article, we’ll go through the specifics of “what” and “how” the large Indian tech experiment aimed to democratize digital commerce business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1FZVVYY-qsycFkfuLzzi6A.jpeg" /></figure><h3><strong>What is ONDC?</strong></h3><p>ONDC is a government initiative driven by private enterprises to provide new opportunities, reduce digital monopolies, and assist micro, small, and medium firms, as well as small merchants, in gaining access to online platforms. The goal is to assist thousands of small shops that are unable to access the e-commerce industry.</p><h4>I am a tech geek. Help me understand ONDC in my language.</h4><p>Sure!</p><p>T. Koshy (CEO, ONDC) <a href="https://www.youtube.com/watch?v=nVbClmwm1OY">In one of his interviews</a> has explained it very well. <br>A Gmail user can send an email to another user with a yahoo account. How? Because all emails use the same underlying SMTP protocol. Similarly, ONDC is to digital commerce what SMTP is to email.</p><h3>Why ONDC?</h3><p>Because of its rising internet user base and favorable market conditions, India has significant e-commerce potential. Despite the tremendous increase in e-commerce in India over the last decade, overall buyer participation remains less than 10%. The next generation of vendors and buyers will require more user-friendly and comprehensive platforms.</p><p>Many local and foreign enterprises compete for the most prominent market share. As of April 2017, <a href="https://www.statista.com/statistics/297355/leading-e-retailers-in-india-ranked-by-annual-web-e-commerce-sales/">Amazon India</a> was the country’s top online marketplace, with more than $500 million in sales. The local market was dominated by Flipkart and Myntra. In such circumstances, the Indian government had an opportunity to attempt something to encourage small businesses to step out and take leads. With all of this, it gave rise to ONDC — Open Network for Digital Commerce.</p><h3>How does ONDC benefit the average buyer and seller?</h3><p>Consider a housewife who excels at making several types of pickles. How can she begin selling to thousands of people online without using several platforms (Amazon/Flipkart) or creating a personal website?</p><p>You got it correctly: the answer is ONDC. She only needs to be on the ONDC network to make the pickle catalog readily available to a large number of buyers online via Paytm/Flipkart/etc. At the moment, only buyers and sellers inside the same network (such as Flipkart) can deal. Because ONDC is open and free, merchants no longer need to pay commission to the enabling platform, as opposed to today’s proprietary systems. This should result in higher volumes and cost benefits for the end consumer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*zmBvRiIwjmMGhdTTLP7M8w.png" /><figcaption>source: <a href="https://medium.com/@nilendudas/ondc-open-network-for-digital-commerce-d7b334567a6a">Nilendu Das</a></figcaption></figure><h3>Will it spell the end of the present big players?</h3><p>According to T. Koshy (CEO, ONDC), a seller can be on Amazon to reap its benefits while also being on ONDC to broaden reach and penetration. ONDC does not compete with anyone; rather, it serves as a facilitator of digital commerce. As of now, most of the major players are in discussions, and some have even begun to integrate into the network.</p><h3>How are we getting along with the experiment?</h3><p>On April 29, 2022, the ONDC pilot soft <a href="https://timesofindia.indiatimes.com/business/india-business/ondc-pilots-start-today-want-small-business-on-board/articleshow/91165712.cms">launch</a> took place in five cities across India, including the Delhi National Capital Region, Bhopal, Bengaluru, Shillong, and Coimbatore. Paytm is now collaborating with ONDC and doing active experiments in five different cities. They are carrying out a few transactions and fine-tuning the network with real-world data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*z4MVd__uFGf6FpKYoW97aA.jpeg" /><figcaption>source image: <a href="https://inc42.com/">https://inc42.com/</a></figcaption></figure><p>Some of the businesses in the process of integrating with the network include Flipkart’s logistics arm eKart Logistics, hyperlocal delivery firm Dunzo, and payment service provider PhonePe.</p><h3>Is it really that simple?</h3><p>Yes! The basic concept of ONDC is to make the entire e-commerce grow faster and easier to access. However, every simple solution has its own set of difficult corners. These are some of the unanswered questions,</p><ol><li>How will refunds and exchanges be handled?</li><li>Who will ensure the products’ quality?</li><li>Who is responsible for the delivery timelines?</li><li>Who should we contact for assistance with other problems and concerns?</li></ol><h3>Conclusion:</h3><p>As India has recently registered its 100th unicorn, the ONDC endeavor is about to set India on the road we’ve always desired in the e-commerce space. With that said, it will be very intriguing to watch India on track to become a USD 5 trillion economy by FY26.</p><h3>Reference:</h3><ol><li><a href="https://twitter.com/ONDC_Official?">ONDC Official</a></li><li><a href="https://www.statista.com/">statist</a>a</li><li><a href="https://medium.com/@nilendudas">medium</a></li><li><a href="https://inc42.com/buzz/24-startups-including-dunzo-ekart-phonepe-gearing-up-efforts-to-join-ondc/">inc42</a></li></ol><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for more such content. 😃</p><p><a href="https://www.linkedin.com/in/ishritam/">Follow me on LinkedIn</a> for all the latest updates.🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9bca0a0d39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web 3.0 -  The Beginning Of Third Generation  Internet]]></title>
            <link>https://shritam.medium.com/web-3-0-the-beginning-of-third-generation-internet-94928741b1a3?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/94928741b1a3</guid>
            <category><![CDATA[future]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[elon-musk]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[web3]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Mon, 27 Dec 2021 11:10:43 GMT</pubDate>
            <atom:updated>2021-12-27T12:04:25.250Z</atom:updated>
            <content:encoded><![CDATA[<h3>Web 3.0— The Beginning Of Third Generation Internet</h3><h4>Making sense of the next-gen internet.</h4><p>You probably heard about Web 3.0, but do you know, what does that even mean? If you still don’t know what it means, then don’t worry, you are not alone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkhExzBcb3xCfKMEPVhR6Q.jpeg" /></figure><h3>Web 1.0 vs 2.0 vs 3.0</h3><p>To understand what Web 3.0 is, we first have to understand the previous versions of the Internet.</p><h4>Web 1.0</h4><p>The Internet started evolving around the 1980s. And got a good hype in the 90s, because business started using internet on a high level. As it was just the time to calm before the storm, businesses were using it to promote their products instead of Radios and Magazines.<br>From 1994 to 2004, these ten years belonged to <strong>Web 1.0</strong>. Web 1.0 was about reading and not writing. It was static instead of dynamic. And with that cons, it gave birth to Web 2.0.</p><p><strong>Web 2.0</strong></p><p>Now, this is where the era started in 2004. The next major phase of the Internet was all about interactivity and users. With emerging technologies, Web 2.0 got so many benchmarks, and lots of tech companies got comfortable with it. Users created most of the content on platforms such as Facebook, YouTube, Instagram, Twitter, and yeah, obviously who will forget TikTok.</p><p>Now the problem is, when most of the platforms are free, how are they even making billions on it. That starts the problem right!<br>In this world, NOTHING is free!</p><blockquote>If you are not paying for a product, then you are a product.</blockquote><p>With Web 2.0, users are also providing personal information and data to the companies that controlled these platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*lwhxilJjlDozjpBy7sEi1Q.gif" /><figcaption>tenor.com</figcaption></figure><p>By seeing this, users had to think of ways to react against the excesses and market dominance of the world’s largest tech companies. A new way of essential has been developed for individuals to use the Internet without giving up their privacy and valuable data, Web 3! the next step of the Internet.</p><p><strong>Web 3.0</strong></p><p>Powered by Blockchain technology, <strong>Web 3.0</strong> comes with an idea to make the whole internet decentralized. The key innovation of these networks is the creation of platforms that no single entity controls, yet everyone can still trust. That’s because every user and operator of these networks must follow the same set of hard-coded rules, known as <a href="https://en.wikipedia.org/wiki/Consensus_(computer_science)">consensus protocols</a>.</p><p><strong>Centralized vs Decentralized</strong></p><p>When we say <strong>centralized</strong>, this purely indicates there is a concept of Client and Server hidden in it. That means the data are going to store on a single central server, which will be owned by an individual or a company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*_ocReqg-zmVo4N2vCXhe0Q.png" /></figure><p>On the other side, <strong>decentralized</strong> indicates, Instead of storing the data on a central server it’ll be stored on individual nodes. Thanks to blockchain, the data is immutable and safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mzNAVD0KM0pIUjT4GSjBPA.jpeg" /></figure><p>And the whole decentralized internet is known as <strong>Web 3.0</strong>.</p><h3>How do you use Web 3.0?</h3><p>With Web 3.0, the network is decentralized, so no one entity controls it. With emerging web 3 technology like Crypto, Ethereum and Blockchain there has been a new term popping up these days, <strong>DApps</strong> (decentralized applications). Anyone is able to build and connect with different dapps without permission from a central company.</p><p>There are many Web 3.0 dapps that are available. Currently, I am exploring these two;</p><ol><li><a href="https://www.stacks.co/">https://www.stacks.co/</a></li><li><a href="https://dfinity.org/">https://dfinity.org/</a></li></ol><h3>The Future</h3><p>The web is an unfinished project. The future of the web is about to increase connectivity, privacy, and scalability. For sure, we are moving from Web 2.0 to Web 3.0 in 2022. We can say we are already at the entry gate of the Web 3.0 era because Crypto and NFT are getting famous these days(thanks to X Æ A-12’s dad).</p><p>Yes, It’ll take some time to give a complete shape to it, but we’ll soon use all the decentralized services on a daily basis.</p><p>The change can’t happen without your help:</p><p>do clap, and share this article.</p><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for the latest updates. 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94928741b1a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Sheets: A Database?]]></title>
            <link>https://shritam.medium.com/google-sheets-a-database-c4e3fef6e0bc?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4e3fef6e0bc</guid>
            <category><![CDATA[google-sheets]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Sat, 24 Jul 2021 20:12:22 GMT</pubDate>
            <atom:updated>2021-07-26T04:47:00.636Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WoX8Bvw7Y6iSav2h.png" /><figcaption>Made by <a href="https://www.canva.com/en_in/">Canva</a></figcaption></figure><p>What if I tell you there is a database that is,</p><blockquote><em>1. 100% Free</em></blockquote><blockquote><em>2. Has a world-class admin UI</em></blockquote><blockquote><em>3. And almost everybody in the world knows how to use it.</em></blockquote><p>Believe me or not but Google Sheets can be used as a lite weight database. And in today’s article, we’ll learn how to integrate it using Python3. Without westing further ado, let’s get started.</p><h3>Article Summary</h3><ol><li>Create Google Sheets</li><li>Create Google Service Account</li><li>Get All Records From Google Sheets Using Python</li><li>CRUD Operations</li><li>Conclusion</li></ol><h3>Create Google Sheets</h3><p>We are having the data of an E-commerce website <a href="https://www.zalando.co.uk/">Zalando.uk</a>, stored the data into a spreadsheet and named it “<em>Zalando Data</em>”. You can access the data <a href="https://docs.google.com/spreadsheets/d/1MCm8pxuIBc69N8Yw48j4cvY8arrwAOCMrvhkiGqoLZw/edit?usp=sharing">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dlOHCR3C4lm55zG0.png" /></figure><h3>Create Google Service Account</h3><p>In order to interact with this spreadsheet using python, we will have to create a <a href="https://cloud.google.com/iam/docs/service-accounts">Service Account</a>. It is a special type of account that is used to make authorized API calls to Google Cloud Services. The interesting part here is you can even create it without spending a penny. Let’s follow the following steps and get the credential ready.</p><h3>Step:1</h3><p>Go to <a href="https://console.cloud.google.com/">Google Cloud Platform</a>, and create a new project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HaV9BRtrUuLMp7qD.gif" /></figure><p>Once it is created, you’ll see a notification that gives the confirmation of our new project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/0*esuhvpUpGAmaC7na.png" /></figure><p>Yes, I agree. I did that typo while naming the project. It should be “ZalandoData” instead of “ZlandoData”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*pf3NOjJUHHYobWZOVcQtxg.jpeg" /></figure><h3>Step 2:</h3><p>Now, that our project is created we need to enable the APIs that we require to make CRUD operation to the database. Oops! “<strong><em>to the google sheets</em></strong><em>”</em>.</p><p>we will add two APIs for our project i.e “ZlandoData”.</p><ul><li>Google Drive API</li><li>Google Sheets API</li></ul><p>GCP provides an API library to enable and manage the APIs for the projects.<br>All we need is it enables the Drive API and get the credentials, we will use it later in this tutorial.</p><ol><li><strong>Enable the Drive API</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q604SRNgDdGlYeAi.gif" /></figure><p>2.<strong> Save the service account credentials</strong></p><p>Once the Drive API is enabled, we can create the <em>service account credentials</em> by clicking on the “+<em>Create Credentials</em>” option under the <em>Credential</em> menu.</p><p>Under the service account, details put your name, which will reflect on the credential file. Select Editor or Owner access to add credentials to your project. Once your service account is ready, download the key credentials in JSON format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5d-GG_yQmG-VOwfh.gif" /></figure><p>Now, a JSON file will be downloaded which contains the keys to access the API. For a naming convention, rename it as <strong><em>Cred.json</em></strong>. Great! our google service account is ready to use.</p><p>3.<strong> Enable the Sheets API</strong></p><p>Similarly, enable the Sheets API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LOVyiFLvHROa0pyQ.gif" /></figure><p>We have already created the service account so we don’t need to create credentials again.</p><h3>Step:3</h3><p>To allow access to our Google Sheets from our API we will share the sheets with the <strong><em>client_email</em> </strong>which is present inside our <strong><em>Cred.json</em></strong> file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dBhi_OUYFqHCFGB6.gif" /></figure><h3>Get All Records From Google Sheets using Python</h3><p>Now, we are ready to code and access the sheets using python.</p><p>It’s always a good practice to create a virtual environment. Let’s create a virtual environment and install <a href="https://docs.gspread.org/"><strong>gspread</strong></a> and <a href="https://oauth2client.readthedocs.io/"><strong>oauth2client</strong></a> packages using PIP so that we can use the API.</p><pre><strong>$ pip install gspread oauth2client</strong></pre><p>The API provides all the functionalities to achieve the sheet as a database. Let’s write an app.py that can print all the data present on our spreadsheet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d504b5d7b0eee3aaf260c6a283ad8154/href">https://medium.com/media/d504b5d7b0eee3aaf260c6a283ad8154/href</a></iframe><h3>Details about the code above:</h3><ol><li><strong>scope</strong>: Scopes are the predefined dependencies that provide easy and secure access to our API.</li><li><strong>Credentials</strong>: we can pass the credential JSON which we have downloaded from the GCP project.</li><li><strong>client.open(“Zalando Data”).worksheet(“zalando_data”)</strong>: Here “Zalando Data” is our Google Spreadsheet name where zalando_data is the sheet name under that spreadsheet.</li><li><strong>get_all_records()</strong>: This method gives output as all the records present inside the sheet in the list of dictionary format.</li></ol><p>Let’s check the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rnFNi0CrU8zKgXbd.gif" /></figure><h3>CRUD Operations:</h3><p>CRUD! Create, Read, Update, and Delete are the four basic operations of persistent storage. Let’s perform is CRUD operation on our Google Sheet.</p><h3>Create:</h3><p>To insert a row into our google sheets we can use the <strong>insert_row<em>() </em></strong>functionality. Let’s insert a new row on row number 8.</p><pre># Insert the list as a row at index 8<br><strong>insertRow = [“GA111A28J-A11”,”Gabor”,”Trainers — weiß/ice”,”https://img01.ztat.net/article/spp-media-p1/bc1c433d1649346e9eb02d316961bfc9/dd8d3ad466d64631b956405046eb9ca1.jpg&quot;,&quot;£89.99&quot;,&quot;https://www.zalando.co.uk/gabor-trainers-weissice-ga111a28j-a11.html&quot;]<br>sheet.insert_row(insertRow, 8)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cjnzKxHVZQ6du0d9.gif" /></figure><p>As you can see, we have inserted a new row on the 8th index. But the cool part is it didn’t overwrite. It creates a completely new row on the 8th index and pushes everything down.</p><h3>Read:</h3><p>As we can access the sheet correctly let’s see how we can print the exact row and column by calling its number.</p><ol><li><strong>Call by row:</strong></li></ol><p>Get a specific row, we can use the <strong><em>row_values()</em></strong> and pass the row number.</p><pre># Get a specific row<br><strong>row = sheet.row_values(3)<br>pprint(row)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uEpQ6oVadMcTGy8k.png" /></figure><p>2.<strong> Call by Column:</strong></p><p>Similarly, we can use the <strong><em>col_values()</em></strong> function and get the column value by passing the column number.</p><pre># Get a specific column<br><strong>col = sheet.col_values(3)<br>pprint(col)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9aZjdTza-2914qRl.png" /></figure><h3>Update:</h3><p>We can update a row or column value by simply using the <strong><em>update_cell</em></strong> method. Let’s update the “<em>Brand Name</em>” of the 8th row from “Gabor” to “ADIDAS”.</p><pre># Update one cell<br><strong>sheet.update_cell(8, 2, “ADIDAS”)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XBgffnSI35_61Drb.gif" /></figure><p>The first parameter takes the row number where the second parameter takes the column number.</p><h3>Delete:</h3><p>To delete a row we can use the <strong><em>delete_row</em></strong> method by calling the row number. Let’s delete the 8th row from our sheet.</p><pre># Delete the row<br>sheet.delete_row(8)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o_SZycPDpHsIh47z.gif" /></figure><p>That’s amazing, right!</p><h3>Conclusion:</h3><p>Great! you added a new database skill under your belt. There are a few caveats before you use it as a production database.</p><ol><li>Google Sheets was never built for a database, so this does not support Join operations or complex queries.</li><li>Google Sheets API has a limit of 500 requests per 100 seconds per project, and 100 requests per 100 seconds per user.</li></ol><p>But even with these limitations, Google Sheets can be the best choice for prototyping your project. And most importantly it’s super easy to use.</p><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for the latest updates. 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4e3fef6e0bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FastAPI: The Modern Age API Framework For Pythonista]]></title>
            <link>https://medium.com/analytics-vidhya/fastapi-the-modern-age-api-framework-for-pythonista-4b2cd1e6652?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b2cd1e6652</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Sun, 18 Jul 2021 14:53:18 GMT</pubDate>
            <atom:updated>2021-07-26T05:34:40.072Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build And Host A Production-Ready API In Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Xd1Cx1nKpuAeGwnVVU5ZA.jpeg" /><figcaption>The Flash by <a href="https://www.dccomics.com/">DCcomic</a></figcaption></figure><p>Web APIs have become increasingly important to the operation of modern businesses. As companies become more reliant on data, the importance of data communications continues to grow. While data technologies are emerging frequently, we need a fast, high-performance, and robust framework to build our APIs.</p><h3>Why <strong>FastAPI?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*du7p50wS_fIsaC_lR18qsg.png" /><figcaption><a href="https://fastapi.tiangolo.com/">https://fastapi.tiangolo.com/</a></figcaption></figure><p><strong>FastAPI, </strong>a modern, fast (high-performance), a web framework for building APIs with Python 3.6+. Its performance can be compared with NodeJS and Go and it is rated as o<a href="https://fastapi.tiangolo.com/#performance">ne of the fastest Python frameworks available</a>. Tech giants like <strong>Microsoft, Netflix, Uber </strong>amongst many other corporations are already started building their APIs with the <strong>FastAPI</strong> library. The framework is designed to optimize the experience so that we can write simple code to build production-ready APIs with best practices by default. Without further ado, let’s learn how can we build a robust API solution using FastAPI.</p><h3>Article Summary</h3><ol><li>Problem Statement</li><li>Installation and Setup</li><li>“Hello World” to FastAPI</li><li>Integrate the Database</li><li>Build the real-world API</li><li>FastAPI Swagger UI</li><li>Host it on cloud</li><li>Conclusion</li></ol><h3>Problem Statement</h3><p>Today we are going to build and deploy an API for the <strong>Retail Industry,</strong> having the data of retail products. The API will have the ability to,</p><ol><li>Add data to a database</li><li>Update the existing data (This’ll include marking the data as updated and the time when the data is getting updated)</li><li>Delete data from the database</li><li>View the data by calling the unique id</li></ol><p>The database we&#39;re going to use today is MongoDB. And we’ll explore the MongoDB Atlas to connect the database to our API.</p><h3>Installation and Setup</h3><p>It’s always good to start by creating a virtual environment. Create the virtual environment and install FastAPI using pip3.</p><pre><strong>$ pip3 install fastapi</strong></pre><p>We will also need an ASGI server for production, such as <a href="https://www.uvicorn.org/">Uvicorn</a>.</p><pre><strong>$ pip3 install uvicorn</strong></pre><p>Later, we need a MongoDB driver to connect our API with the database. For this, we’ll go with <a href="https://motor.readthedocs.io/en/stable/index.html"><strong>Motor</strong></a> driver. Motor presents a coroutine-based API for MongoDB which supports <a href="https://docs.python.org/3/library/asyncio.html">asyncio</a>.</p><pre><strong>$ pip install motor</strong></pre><p>We need a DNS toolkit for Python which can support almost all record types.</p><pre><strong>$ pip install dnspython</strong></pre><p>These are all the dependencies that we need to create a full-fledge high-performance API using python3.</p><h3>Hello World to FastAPI</h3><p>To get started, we’ll create a “hello world” app using <em>FastAPI </em>and run it using <em>uvicorn server</em>. This will give you a very quick overview of how everything works.</p><p>Create a file app.py with:</p><pre>from fastapi import FastAPI<br><br>app = FastAPI()<br><br>@app.get(&quot;/&quot;)<br>async def root():<br>    return {&quot;message&quot;: &quot;Hello World&quot;}</pre><p>Run the server with:</p><pre><strong>$ uvicorn app:app -- reload</strong></pre><p>Open your browser at <a href="http://127.0.0.1:8000/">http://127.0.0.1:8000/</a></p><p>You will see the JSON response as: {&quot;message&quot;: &quot;Hello World&quot;}</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*_QiWPqaOeqOy8-vpNxAN2g.png" /></figure><p>The JSON response is the same dictionary that you returned from the function in your FastAPI application.</p><p>You can also view the interactive API documentation at <a href="http://localhost:8000/docs">http://localhost:8000/docs</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*6GnxyYds-VIzhBU4oMaXvg.png" /><figcaption><a href="http://localhost:8000/docs">http://localhost:8000/docs</a></figcaption></figure><h3>Integrate the app to the Database</h3><p>As it has been clear so far, we are using MongoDB as our database, let me tell you we are going to use the most advanced cloud database service on the market, <a href="https://www.mongodb.com/cloud/atlas/efficiency?utm_source=google&amp;utm_campaign=gs_apac_india_search_core_brand_atlas_desktop&amp;utm_term=%2Bmongo%20%2Bdb%20%2Batlas&amp;utm_medium=cpc_paid_search&amp;utm_ad=b&amp;utm_ad_campaign_id=12212624347&amp;gclid=CjwKCAjwos-HBhB3EiwAe4xM9wC3-xjOERrId6ylD-TBAJcnoWc4eNmjd_o3WMs-StnuFS-INn3zOxoCBNgQAvD_BwE"><strong>MongoDB Atlas</strong></a><strong>.</strong></p><ol><li>Let’s create a cluster in MongoDB Atlas with the default 512MB configuration. And name it “<em>FasterAPI</em>”.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*xC2LNBNQeTMGmsKdMwgioA.png" /></figure><p>2. To connect the database to our app, we’ll take the help of MongoDB Compass. Let’s connect it using the configuration that has been set during the creation of the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*gp9kPWsu0XPQBddIUgI6eg.png" /></figure><p>3. Connect with MongoDB compass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*zXnI4S3s5H7A7PIGJ_3TUg.png" /></figure><h3>Build the Realworld API</h3><p>To build a real-world Retail API, we will use some basic HTTP functionalities which are known as <strong>CRUD</strong> operations. <strong>CRUD</strong> stands for Create, Read, Update, and Delete, which are four primitive database operations.</p><p>Using FastAPI we can Create, Read, Update, and Delete data in our database using 4 predefined methods.</p><ul><li>@app.get()</li><li>@app.post()</li><li>@app.put()</li><li>@app.delete()</li></ul><p>This HTTP functionality should execute below the <strong>app</strong> instance, which is an instance of the class FastAPI. This will be the main point of interaction to create our API.</p><pre>app = FastAPI()</pre><p>And this is the same <em>app</em>, which initiates the API while we run using uvicorn app:app --reload</p><p>It’s always a good practice to defined a Pydantic <a href="https://pydantic-docs.helpmanual.io/usage/schema/">Schema</a> that represents how the data should be stored in the Database. In Pydantic, let’s define a schema to store our data to MongoDB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a39922c0abcebb4f2584dfffa8aaf2b/href">https://medium.com/media/0a39922c0abcebb4f2584dfffa8aaf2b/href</a></iframe><p>In the code above, we defined a Pydantic Schema called Items , which defines all the fields with respect to their type. The <em>schema_helper</em> function will help the <strong>CRUD</strong> operation data to be mapped as a dictionary with the Retail parameters i.e SKU, Brand Name, Title, Thumbnail, Available Price, and MRP.</p><p>Yey! The script below is all we need to build our API, where we have defined CRUD functionality along with the schema.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1b22d1eeb725aaa797c4c7fd0d4625b/href">https://medium.com/media/c1b22d1eeb725aaa797c4c7fd0d4625b/href</a></iframe><h3>FastAPI Swagger UI</h3><p>FastAPI provides web user interfaces. As the framework is based on OpenAPI, there are multiple options to be explored yet, but let’s look into the most common and important features of this framework.</p><p>Switch to <a href="http://localhost:8000/docs">http://localhost:8000/docs</a> to interact with FastAPI swagger.</p><h4>POST Method:</h4><p>We’ll create a data to out collection using the POST method,</p><pre>{“sku”: “NI114D0EY-Q11”,<br> “brand_name”: “Nike Sportswear”,<br> “title”: “AIR MAX 2090 UNISEX — Trainers — black/wolf grey/black/anthracite”},<br> “thumbnail”: “<a href="https://img01.ztat.net/article/spp-media-p1/9866ece8595630bfbc16501686540429/9c6d5c0c4ab14091b9714501865dcd66.jpg">https://img01.ztat.net/article/spp-media-p1/9866ece8595630bfbc16501686540429/9c6d5c0c4ab14091b9714501865dcd66.jpg</a>&quot;,<br> “available_price”: “£55.99”,<br> “mrp”: “£59.99”<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjIwEytq2Wzr2WIJ5xbrVA.gif" /></figure><h4>GET Method:</h4><p>Get the updated data by giving id as the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gR3IYKYqhLplo-k6uoYifg.gif" /></figure><h4>PUT Method:</h4><p>Update the data. Here if we update the data using the Post method, 2 fields will be added to the data fields,</p><pre>1. “update”: True</pre><pre>2. “updated_time”: &lt;Time when the data was updated&gt;</pre><h4>DELETE Method:</h4><p>Delete the record from the collection using the Id field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wPQOs53TPC_6FZqzN-D7kQ.gif" /></figure><h3>Host the API on the cloud:</h3><p>Lastly, let’s deploy our API into Heroku.</p><p><a href="https://www.heroku.com/"><strong>Heroku</strong></a> is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud.</p><p>To deploy it on the production environment, collect all your requirements into a requirements.txt file.</p><pre><strong>$ pip freeze &gt; requirements.txt</strong></pre><p>Please make sure <em>uvloop</em> and <em>httptools </em>are part of your requirements file.</p><p>API Url: <a href="https://retail-fastapi.herokuapp.com/">https://retail-fastapi.herokuapp.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*43nrDWy1a8Yylu3R97RepA.gif" /></figure><p>That’s great!</p><p>Feel free to fork and play with the code I used to create this API in <a href="https://github.com/ishritam/FastAPI">this repository</a>.</p><h3>Conclusion:</h3><p>Congratulations, your first production-ready API with FastAPI is live. I hope it was easy and helpful. Feel free to take this FastAPI article as a reference and try it out and build some amazing web APIs.</p><p>I hope this article will create a motivation for you to build and host your APIs into your production.</p><p>I will see you on another cool topic next time. Till then,</p><p>Stay Safe..! Keep Learning..!</p><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for the latest updates. 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b2cd1e6652" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/fastapi-the-modern-age-api-framework-for-pythonista-4b2cd1e6652">FastAPI: The Modern Age API Framework For Pythonista</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Manage Your Workflows With Apache Airflow.]]></title>
            <link>https://medium.com/analytics-vidhya/manage-your-workflows-with-apache-airflow-7d39e12b302b?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d39e12b302b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Sun, 20 Jun 2021 12:39:18 GMT</pubDate>
            <atom:updated>2021-07-05T06:24:42.228Z</atom:updated>
            <content:encoded><![CDATA[<h4>Write Your First Real-Time DAG In Python.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WvTO9j5QvLnQG0DLfnlsiw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@chaitaastic">Chait Goli</a> on <a href="https://www.pexels.com/">Pixel</a></figcaption></figure><blockquote><strong><em>NOTE:</em></strong> This article is part of a series called — “<strong>Manage Your Workflows With Apache Airflow</strong>”. This is the second part of a two-part series. You should read the <a href="https://shritam.medium.com/manage-your-workflows-with-apache-airflow-e7b0e45544a8">first part first</a>.</blockquote><blockquote><em>part1:</em><a href="https://shritam.medium.com/manage-your-workflows-with-apache-airflow-e7b0e45544a8"><em> Introduction, Installation, and Setup Apache Airflow in Python</em></a></blockquote><blockquote><em>part2: Write Your First Real-Time DAG In Python.</em></blockquote><p>In the previous article, we have installed and set up the environment to run our first DAG in Airflow. As an expectation, this article assumes you have the Airflow environment configured and all set to write and manage workflow.</p><p>In this follow-up post, I’d like to explain with the following agenda,</p><ol><li>Real-world problem statement</li><li>Data Source Explanation</li><li>Data Overview</li><li>Write an ETL pipeline</li><li>Write DAG to manage the ETL workflow</li><li>Quality check</li><li>Conclusion</li></ol><h3>Real-World Problem Statement:</h3><p>We are going to operate a simple task of finding a real-world API where we can obtain the data in JSON format, next we’ll transform the data into required fields and at last save the transformed data to CSV format.</p><h3>Data Source Explanation</h3><p>With the rise of modern web app frameworks like React and Vue.js, more and more sites are using REST API to send and receive data, then render the final layout on the client-side.</p><p>For this whole process, we’ll extract the data from <a href="https://www.zalando.co.uk/">Zalando-UK</a> , Men’s Bag category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPgr61chHWvjoNyrUMzucw.gif" /></figure><blockquote><em>But wait! how do we catch the API which contains the product data?</em></blockquote><ol><li>Many websites along with Zalando uses the concept of ‘one page displaying’. Open the webpage and inspect it. Go to the XHR of networks and reload the page. You can see a list of all of the requests which are the type of XHR (data specific). Analyze all network requests and hit the ‘next page’ button, the API request will be made.</li><li>Once you get the API, copy the Curl and convert it to Python request from <a href="https://curl.trillworks.com/">CrulTrillwork</a>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hu4gcfbOyBr64ryuT8aZcQ.gif" /></figure><p>Let’s have a look, how the data looks like once you hit a GET request to the API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XTqkIK9VAE4g9xYqhI9L9g.png" /><figcaption>API JSON data</figcaption></figure><h3>Data Overview</h3><p>We have successfully extracted the API, now it’s time to become <em>Sherlock Holmes.</em> Let’s analyze the data and find the field where all the product details are being stored. For simplicity, I’ll only consider the first page of the API.</p><p>If you look closely at the “<em>API JSON data</em>” image, all the products are stored inside the field called <strong><em>article</em></strong>. And it seems, it’s a list of dictionaries.</p><p>Yes, We got it right.</p><p>Let’s explore some useful fields to make our final CSV file,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*HLDO06m1-O8rnu-rNVaPiw.png" /><figcaption>Fileds to take for ETL process</figcaption></figure><h3>Write an ETL pipeline</h3><p>We need to understand few things before ETL,</p><h4><strong>What is DAG?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/0*51XJchsJBdD7OFBB.png" /></figure><p>A <em>DAG</em> (Directed Acyclic Graph) is the core concept of Airflow, collecting each task together, organized with dependencies and relationships to say how they should run. For example, Here it defines four Tasks — A, B, C, and D — and dictates the order in which they have to run, and which tasks depend on what others.</p><blockquote>Dag files should have <em>.py</em> extention and stored inside a folder named dagswhere you have initiated the db.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*BFdSbGj8Yqm2ZEPQi8WrOw.png" /><figcaption>directory tree</figcaption></figure><ol><li>Let’s create a folder and create a python file and name it as zalando_etl.py</li><li>Create a directory as Data/csv to store your final output.</li></ol><h4><strong>What is X-com?</strong></h4><p>XComs (short for “cross-communications”) is a mechanism that lets <em>Tasks</em> talk to each other. There is a method called “<em>xcom_push</em>” in airflow, that can send some values in the Xcom table in Apache Airflow Backend Database. This can be defined withti(<em>task instance)</em>, which is used to identify the tasks.</p><p>As we are all set to write ETL pipeline, without further ado let’s define a couple of Python functions for Extracting, Transforming, and Loading this data.</p><h4>Extraction:</h4><p>This function will make GET requests to the API URL along with the given headers that we have collected from <a href="https://curl.trillworks.com/">CrulTrillwork</a>. Xcom-push will push the result as the value of the function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b7e42a2d09743c32f4e1dee98137705/href">https://medium.com/media/8b7e42a2d09743c32f4e1dee98137705/href</a></iframe><h4>Transform:</h4><p>We need some preprocessing in this step. If you see the product URL filed and thumbnail field, they are not in proper URL format. We need to transform it into a proper URL.</p><pre><strong>1. url_key</strong>: “under-armour-weekend-bag-black-un244e055-q11” <strong><em>to </em></strong>“<a href="https://www.zalando.co.uk/under-armour-weekend-bag-black-un244e055-q11.html">https://www.zalando.co.uk/under-armour-weekend-bag-black-un244e055-q11.html</a>”</pre><pre><strong>2. path</strong>: “spp-media-p1/506932a1e2c740acb95444b7d9c84994/36e1a4ed9c5c4d9d884dd5297011595f.jpg”<strong> to</strong> &quot;<a href="https://img01.ztat.net/article/spp-media-p1/506932a1e2c740acb95444b7d9c84994/36e1a4ed9c5c4d9d884dd5297011595f.jpg">https://img01.ztat.net/article/spp-media-p1/506932a1e2c740acb95444b7d9c84994/36e1a4ed9c5c4d9d884dd5297011595f.jpg</a>&quot;</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11d08af8dcf869577dc9d736aa9943e0/href">https://medium.com/media/11d08af8dcf869577dc9d736aa9943e0/href</a></iframe><h4>Load:</h4><p>Finally, we need to define a function that will load the transformed data to a CSV file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e8f37f792c96885cd7b764d867e0f74/href">https://medium.com/media/5e8f37f792c96885cd7b764d867e0f74/href</a></iframe><h3>Write DAG to manage the ETL job</h3><p>Yay! Now we can write the DAG in Airflow to run the 3 different jobs one after another. The script below is all we need to create the DAG file to perform the workflow.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1ed236848c85d95c893a3a9e9ce4f36/href">https://medium.com/media/e1ed236848c85d95c893a3a9e9ce4f36/href</a></iframe><h4>Details about the code above:</h4><ol><li>After defining all 3 ETL functions, we have defined the Tasks sequentially.</li><li><em>dag_id </em>should be unique for all the dags you’ll create in the future.</li><li>To execute Python callables always use the PythonOperator method.</li><li>Task are sequentially defined with <em>&gt;&gt;</em> symbols.</li></ol><p>Let’s check whether the DAG has been listed by airflow or not. Activate your virtual environment for airflow and run the following command to check all the dags.</p><pre><strong>$ airflow dags list</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*XDmxUm8EiDiNfpAILldEGA.png" /><figcaption>DAGs list</figcaption></figure><p>You might see some other example dags on your list. But no worries, those are for demo purposes only. It’ll not make any changes to your <em>zalando_workflow</em> DAG.</p><p>Cool! Let’s open two separate shell tabs to run Scheduler and Webservice.</p><h4>Start the Scheduler:</h4><pre><strong>$ airflow scheduler</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nW2f4QVvxuZxlFBm.png" /></figure><h4>Start the WebServer</h4><pre><strong>$ airflow webserver — port 8080     </strong>#default port is 8080</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VHsHk0lAo01nR7QY.png" /></figure><p>Now let’s visit <a href="http://0.0.0.0:8080/"><em>localhost:8080</em></a> in the browser and you’d see the Airflow Webserver UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uGFkeNjCev2tPGJYA5hsHg.png" /><figcaption>Airflow Webserver Homepage</figcaption></figure><p>We are done with the DAG configuration. Let’s start the workflow, unpause the DAG and Trigger the DAG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uY5SdyPI1kmYWg7VQmotCw.gif" /><figcaption>DAG Workflows</figcaption></figure><p>After triggering it, check the <strong><em>Graph View</em></strong> to track the workflow. You can check the status of each task by hovering on it. Once the workflow will get finished, all the task’s borders will be shown in the <em>Green</em> color, with the status as “<em>Success</em>”.</p><h3>Quality check</h3><p>Once the DAG run is successfully completed for today, you can see the CSV file will be generated under Data/csv directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3WLULPqghl6jkDeKfXEAxw.png" /><figcaption>Zalando final csv file</figcaption></figure><p>Let’s read the CSV in Pandas data frame,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AB5DxGgtdn6iDyQOcKeoYw.png" /><figcaption>csv to pandas dataframe</figcaption></figure><p>That’s great!</p><h3>Conclusion:</h3><p>Congratulations, your first real-world DAG with Airflow is live. I hope it was easy and helpful. Feel free to take this Airflow series as a reference and try it out with some other new sources.</p><p>I hope this article will create a motivation for you to write and manage your workflows with Apache Airflow 2.1.0.</p><p>Register for <a href="https://www.crowdcast.io/e/airflowsummit2021/register?utm_campaign=AstronomerMarketing&amp;utm_source=Website&amp;utm_medium=Bar">Airflow Summit 2021</a>.</p><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for the latest updates. 😃</p><h3>Reference</h3><ul><li><a href="https://airflow.apache.org/docs/">Documentation</a></li><li><a href="https://www.astronomer.io/">Astronomer: The Best Place to Run Apache Airflow®</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d39e12b302b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/manage-your-workflows-with-apache-airflow-7d39e12b302b">Manage Your Workflows With Apache Airflow.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Manage Your Workflows With Apache Airflow.]]></title>
            <link>https://medium.com/analytics-vidhya/manage-your-workflows-with-apache-airflow-e7b0e45544a8?source=rss-42e0902ac2c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7b0e45544a8</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Shritam Kumar Mund]]></dc:creator>
            <pubDate>Sat, 19 Jun 2021 12:15:11 GMT</pubDate>
            <atom:updated>2021-06-30T08:56:19.359Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction, Installation, and Setup <strong>Apache Airflow</strong> in <strong>Python</strong>.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KE5l4k92GSvNymPVx4H-cw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@pixabay">Pixabay</a> on <a href="https://www.pexels.com/">Pixel</a></figcaption></figure><blockquote>Management is doing things right;</blockquote><blockquote>leadership is doing the right things.</blockquote><blockquote>— Peter Drucker</blockquote><blockquote>part1: Introduction, Installation, and Setup Apache Airflow in Python</blockquote><blockquote>part2: <a href="https://shritam.medium.com/manage-your-workflows-with-apache-airflow-7d39e12b302b">Write Your First Real-Time DAG In Python.</a></blockquote><p>Data is at the center of every business today. As companies become more reliant on data, the importance of data engineering continues to grow. Data engineering helps make data more useful and accessible for consumers of data. Data Engineers play a major role in the end-to-end process as “data pipelines.” Each pipeline has one or more sources and one or more destinations. Within the pipeline, data may undergo several steps of transformation, validation, enrichment, summarization, or other steps. Data engineers create these pipelines and manage the workflow using some efficient engineering tools. While data technologies are emerging frequently, <strong>Apache Airflow</strong> stands at the top when we think about workflow management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*kpbqN0GZTDTA7t4fA9L1Eg.png" /><figcaption><a href="https://airflow.apache.org/">https://airflow.apache.org/</a></figcaption></figure><h3>What is Airflow?</h3><p>As a Data Engineer there will be a point when you think up to orchestrating your workflows, and here come Apache Airflows to fill the gap. Originally created at <a href="https://airbnb.io/projects/airflow/">Airbnb</a> in 2014, Apache Airflow is an open-source tool for orchestrating complex workflows and data processing pipelines. It is a platform to programmatically Authoring, Scheduling, and Monitoring workflows.</p><p><strong>Authoring: </strong>Workflows in Airflow are written as<strong> </strong>Directed Acyclic Graphs(DAGs) in Python Programming Language. It allows the user to write their custom workflows in python.</p><p><strong>Scheduling: </strong>The user can specify when a workflow should start, end, and after what interval it should run again.</p><p><strong>Monitoring: </strong>The Airflow UI makes it easy to monitor and troubleshoot your data pipelines. It has a bunch of tools to track your workflow in real-time.</p><h3>Installation and Setup</h3><h4>Step: 1</h4><p>Now that we know about Airflow’s different components, let’s start with setting up Airflow on our workstation so that we can locally test and run the pipelines that we build. The first thing we will do is create a new folder and create a virtual environment with Python 3 and activate it in which we will install and run Airflow. Let’s do it quickly,</p><pre><strong>$ mkdir Airflow</strong>                   # make a folder<br><strong>$ cd Airflow                      </strong># directed to the directory<br><strong>$ python3 -m venv airflowenv</strong>      # create a vertual environment<br><strong>$ source airflowenv/bin/activate  </strong># activate the environment</pre><h4>Step: 2</h4><p>The easiest way to install the latest stable version of Airflow is with pip but before that upgrade, the pip version and install airflow:</p><pre><strong>$ pip install --upgrade pip         </strong># pip upgrade<strong><br>$ pip install apache-airflow        </strong># installing airflow</pre><h4>Step: 3</h4><p>After installation check the airflow version you have installed on your system,</p><pre><strong>$ airflow version<br></strong>&gt;&gt; 2.1.0</pre><h4>Step: 4</h4><p>Airflow requires a location on your local system to run known as <em>AIRFLOW_HOME</em>. If we don’t specify this it will default to your route directory. I recommend you to set up the <em>AIRFLOW_HOME</em> under the same directory where you are currently in i.e where you created the virtual environment.</p><pre><strong>$ export AIRFLOW_HOME=.</strong></pre><h4>Step: 5</h4><p>After configuration, you’ll need to initialize the database before you can run tasks:</p><pre><strong>$ airflow db init</strong></pre><p>This will initiate the prerequisite to run your airflow task. If you see closely there will be some folder created under your Airflow directory. Will look into it closely some other day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*pDxifz5xA0AQcO7bRsocUQ.png" /><figcaption>files created after initializing the metadata DB</figcaption></figure><h4>Step: 6</h4><p>As you are getting started with airflow for the first time, it might ask you to log in. To set up your user id and password run the following command. This will set up your username as “<em>admin</em>” and password also as “<em>admin</em>”.</p><pre><strong>$ airflow users  create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin</strong></pre><h4>Step: 7</h4><p>Now we are ready to run Airflow Web Server and Scheduler locally. Use two separate shell tabs to run both processes and don’t forget to activate the virtual environment in the new tab.</p><p>For good practice, I suggest you run Scheduler first and then go for the Web Server.</p><h4>Start the Scheduler:</h4><pre><strong>$ airflow scheduler</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DyPe991Ukfpcz0DMSjwHHw.png" /></figure><h4>Start the WebServer</h4><pre><strong>$ airflow webserver — port 8080     </strong>#default port is 8080</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZHBxq5sBdDdLoLvs3-5fQ.png" /></figure><p>To access the Web Server UI, visit <em>localhost:8080</em> in the browser and use the admin account you just created to log in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v9QLAz6e4jSwkzCn9eXh9g.gif" /></figure><h3>Conclusion:</h3><p>Congratulation, you have successfully set up your Airflow environment. In the next blog post, I’ll show you how to run a real-time workflow using Airflow. Till then keep your setup ready and stay motivated.</p><h3>+1 <strong>Announcement:</strong></h3><p>It’s expected that more than 10K software engineers, data scientists, business intelligence analysts, and DevOps engineers will virtually attend the Airflow Summit 2021 (July 8th-16th). Airflow Summit is a free online conference for the worldwide community of developers and users of Apache Airflow. The event will consist of keynotes, community talks, and in-depth workshops.</p><p>Register for <a href="https://www.crowdcast.io/e/airflowsummit2021/register?utm_campaign=AstronomerMarketing&amp;utm_source=Website&amp;utm_medium=Bar">Airflow Summit 2021</a>.</p><h3>Thank you for reading!</h3><p><a href="https://medium.com/@shritam">Follow me on Medium</a> for the latest updates. 😃</p><h3>Reference:</h3><ul><li><a href="https://airflow.apache.org/docs/">Documentation</a></li><li><a href="https://www.astronomer.io/">Astronomer: The Best Place to Run Apache Airflow®</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7b0e45544a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/manage-your-workflows-with-apache-airflow-e7b0e45544a8">Manage Your Workflows With Apache Airflow.</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>