<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by The Data Girl on Medium]]></title>
        <description><![CDATA[Stories by The Data Girl on Medium]]></description>
        <link>https://medium.com/@the_data_girl?source=rss-11885125cf06------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rCeUJHDUzsQTc1YfyzG4UQ.jpeg</url>
            <title>Stories by The Data Girl on Medium</title>
            <link>https://medium.com/@the_data_girl?source=rss-11885125cf06------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 11:53:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@the_data_girl/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Product Research: How is BluSmart building its cab and EV business?]]></title>
            <link>https://medium.com/@the_data_girl/how-is-blusmart-taking-over-ev-cab-market-04729d238081?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/04729d238081</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[case-study]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[product-case-study]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Mon, 11 Dec 2023 11:45:57 GMT</pubDate>
            <atom:updated>2024-02-18T13:15:55.829Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In this article, I tried analyzing the business model of the biggest cab service providers in India(Ola and Uber) and comparing them with the BluSmart business model.</em></p><p><em>Let’s explore the USP of BluSmart and how this startup is seeking to challenge Uber and Ola for market share in the country with bets on an all-electric taxi fleet.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*8pZrKyw0iCxlLID0c_-JIg.png" /><figcaption>Source: <a href="https://vygrnews.com/business/blusmart-intends-to-quadruple-its-fleet-size-to-10-000--cofounder-anmol-singh-jaggi">https://vygrnews.com/business/blusmart-intends-to-quadruple-its-fleet-size-to-10-000--cofounder-anmol-singh-jaggi</a></figcaption></figure><h4>About BluSmart</h4><p>BluSmart is a 100% electric cab service that began its operation in 2020 by providing airport rides to Delhi and now has extended its services to Bangalore.</p><blockquote>Blusmart since it’s launch saw rapid growth with the average car ride increasing from 1,400 per day to 17,000 trips per day. (Read more <a href="https://www.forbesindia.com/article/take-one-big-story-of-the-day/electric-is-the-trick-why-blusmart-is-on-a-highvoltage-ride/82309/1">here</a>)</blockquote><blockquote>The company is even planning on growing its fleet to 14,000 taxis next year and 100,000 in five years, expanding to four more cities.</blockquote><h4><strong>How is BluSmart different?</strong></h4><p>BluSmart has correctly identified user pain points such as cab users (especially in India) face <strong>high cab cancellation rates, unclean cabs, and high surge prices</strong> when using Ola or Uber.</p><p>Blusmart considered these problems and turned them into their USP. They provide cab services with:</p><ol><li><strong>Zero cancellation rate</strong></li><li><strong>Clean and hygienic cabs</strong> (through direct management of its fleet and drivers)</li><li><strong>No surge prices</strong></li></ol><h4>How is BluSmart offering clean rides with zero cancellations?</h4><p>BluSmart <strong>owns the</strong> <strong>complete infrastructure i.e. EV cabs, and charging setup(use Asset-heavy model), </strong>and only hires drivers.</p><p>By directly managing the cab and providing incentives for clean-cab rides they were able to <strong>provide better quality service such as cleaner cabs and no ride cancellations</strong> (which increased their trust among customers).</p><h4><strong>Unique Incentives by BluSmart</strong></h4><p>Incentives provided by BluSmart:</p><ol><li><strong>Flexible working hours</strong> that allow drivers to log in and out as per their schedule.</li><li><strong>Free accidental insurance up to 2 lakhs</strong></li><li><strong>Cab quality-based incentives</strong> that motivate the drivers to keep the cabs clean.</li></ol><p>These incentives have attracted many Ola and Uber drivers to work for BluSmart. Almost 43% of BluSmart drivers are from Uber.</p><h4><strong>OLA/UBER Working Model and their Drawbacks</strong></h4><p>Ola and Uber work on an <strong>asset-light</strong> model i.e. they don’t own any of the cabs. The drivers own the cab and the <strong>company takes a commission</strong> from the drivers for using their platform.</p><p>This aggregation model comes with its own set of problems such as the driver being unhappy about the ride prices as most of what they make goes into paying off their car EMI and with Ola, Uber&#39;s poor incentives schemes the drivers end up quitting for other jobs that pay more. Due to this, Ola and Uber <strong>faced a huge shortage of cabs and cab drivers</strong>.</p><p>Since Ola, and Uber don’t own the cabs it became difficult for them to control the <strong>cab&#39;s hygiene, its reliability and provide quality rides to the customers</strong>. As drivers started quitting due to lower pay, Ola and Uber had to increase incentives to attract drivers and keep ride prices lower than their competition to acquire new customers. This resulted <strong>in more cash drain than cash inflow</strong>.</p><h4><strong>Comparing incentives provided by Uber, Ola, and BluSmart</strong></h4><p><em>Incentives play a key role in keeping drivers content and motivated to work. The happier drivers are with their jobs, the better the service provided by them.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j_4bpzg5zV9O9cH3XiXLQQ.png" /><figcaption>OLA INCENTIVES</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fyzq1rdA9rBNd8coT86j-w.png" /><figcaption>Uber Incentives</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rWg2VMsitmFDByLDMutUdA.png" /><figcaption>BluSmart Incentives</figcaption></figure><p>Incentives provided by Ola or Uber are in terms of reward points. The more rides the drivers complete, the more points they will get. They can then convert those points either into money or attractive coupons.</p><blockquote>But this type of incentive framework only encourages drivers to accept more rides but doesn’t ensure better ride quality.</blockquote><p>Whereas BluSmart provides <strong>more variety of incentives</strong> that are more lucrative to drivers. Their incentive frameworks motivate drivers to <strong>keep the cab clean and provide better quality cab rides</strong> to the passengers.</p><p>Incentives such as minimum business guarantee ( i.e. drivers are guaranteed INR 2.8k worth of rides per day) build trust between the driver and the company.</p><h4><strong>How has BluSmart increased its asset utilization (cab and EV charging setup)?</strong></h4><ol><li><strong>20% increase in revenue by optimizing the driver-car ratio<br></strong>The company ownership model<strong> allowed them to staff more than one driver for one vehicle</strong>. Using this approach, the vehicle can be operated for much longer hours and higher revenue can be generated even while the lease cost remains fixed, thus increasing profitability. The optimization of the <strong>driver-car ratio resulted in a 20% increase in revenue per vehicle</strong>. (Read more <a href="https://blu-smart.com/assets/ev-report.pdf">here</a>)</li><li><strong>Strategically placing charging hubs to decrease dry-run time<br></strong>When the company learned that the drivers faced an <strong>increase in dry-run time</strong>(Dry run denotes the percent of the time that the vehicle is running idle without a paying passenger).<br>Time begins to increase when <strong>charging infrastructure is placed away from high-demand regions</strong> since this would mean additional time spent by drivers traveling away from pick-up zones to charge the vehicle. <br>To avoid this, the company <strong>strategically placed charging hubs in key demand catchment areas </strong>which helped them to achieve two key operational requirements: <strong>optimally meeting demand and reducing dry-run time. </strong>(Read more <a href="https://blu-smart.com/assets/ev-report.pdf">here</a>)</li><li><strong>Providing EV charging services<br></strong>To increase revenue and utilization of electric charging provided by the company. <strong>BluSmart Charge</strong> was introduced which is one of BluSmart’s business segments to <strong>offer charging services to other EV users and increase asset utilization of chargers</strong>.<br>This allows EV users to locate, book, and pay for charging sessions. Currently, BluSmart cabs charge using both captive BluSmart chargers and public chargers. (Read more <a href="https://blu-smart.com/assets/ev-report.pdf">here</a>)</li></ol><h4>How did BluSmart raise female passengers by 1.8 x?</h4><p>With continuous driver training and ensuring safety standards, BluSmart has created a secure community. To provide safety for women passengers the company has even<strong> hired and trained women drivers</strong>.</p><p>The percentage of female passengers as a proportion of total users on the <strong>BluSmart</strong> <strong>platform is around 1.5 to 1.8 times higher than that on other ride-hailing platforms</strong>. (Read more <a href="https://blu-smart.com/assets/ev-report.pdf">here</a>)</p><h4><strong>Challenges in the BluSmart Business Model</strong></h4><p><strong>1. Cost of setting up charging infrastructure<br></strong>One major issue with BluSmart&#39;s business model is the <strong>cost of doing business</strong>. Since they own the complete infrastructure it becomes a cost-heavy operation as they have to lease EV cabs( which are 25–30% more expensive than petrol cars) and even provide charging infrastructure in their target cities.</p><p>Although the <strong>cost of ownership and operation of EV cabs is cheaper</strong>. The company petrol/diesel car. The total cost of operating <strong>EVs is INR 4.6/km</strong> when compared to <strong>INR 7.1/km for CNG </strong>vehicles or <strong>INR 8.2/km for diesel</strong> vehicles. Setting up charging infrastructure is expensive.</p><p><strong>2. Lack of vehicle options<br></strong>As of now <strong>BluSmart only offers sedan cars</strong>( which have a capacity of 4), unlike other cab-ride providers who offer different types of cars ranging from sedans to SUVs, bikes, and autos.</p><h4><strong>BluSmart collaboration with different EV car manufactures</strong></h4><p>With few automobile companies manufacturing EV cars, Blusmart has limited companies to partner with and purchase cars in bulk.</p><p>They get most of their cars from<strong> Tata Motors </strong>(the only affordable EV maker in India right now). They have already raised Rs 1,200 crore for vehicle leasing. 95 % of their cars are <strong>Tata Tigors</strong> (the electric variant) and the remaining 5 % are premium cars such as <strong>Hyundai Kona Electric, MG ZS EV, and BYD E6</strong>. The company plans to keep the composition the same.</p><h4><strong>BluSmart Cab Services Stats</strong></h4><p>BluSmart has over<strong> 2,600 charging points, has a fleet of over 3,500 cars, claims to add 500 cars every month</strong>, and has over 4,700 drivers. Over <strong>2.4 million users have downloaded</strong> the BluSmart app. BluSmart plans to have 10,000 EVs in FY24.</p><p>The firm has reported operating revenue of Rs 170 crores as of FY23 as against Rs 29 crores in FY22.</p><h4>The potential of EV Market in India</h4><p>EVs are a huge growing market in India as they play a vital role in <strong>transitioning India to a zero-emission transport system</strong>.</p><p>Many e-commerce and delivery companies like Amazon, Flipkart, Zomato, Swiggy, and others,<strong> have committed to 100% fleet electrification</strong> over the next decade. This provided BluSmart with the opportunity to collaborate <strong>with other e-commerce giants and enter into the delivery market</strong>.</p><h4>Conclusion</h4><p>Bluesmart <strong>vision of providing a clean and green space business with a focus on sustainability, and better quality service</strong> is its recipe for success. Within a short period, it has challenged established companies such as Uber and Ola. Blusmart has identified and targeted the appropriate market to address the prevailing issues in cab services.</p><h4>Reference</h4><ol><li><a href="https://economictimes.indiatimes.com/tech/startups/blusmart-chalks-a-new-route-on-winding-road-to-e-mobility/articleshow/98434151.cms">Economics Times</a></li></ol><p>2. <a href="https://www.hindustantimes.com/car-bike/blusmart-vs-ola-uber-indian-startup-bets-on-an-all-electric-taxi-fleet-101681882525265.html">Hindustan Times</a></p><p>3. <a href="https://blu-smart.com/en-IN/">BluSmart</a></p><p>4. <a href="https://www.forbesindia.com/article/take-one-big-story-of-the-day/electric-is-the-trick-why-blusmart-is-on-a-highvoltage-ride/82309/1">Forbes India</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04729d238081" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[E-commerce customer behaviour analysis using Tableau]]></title>
            <link>https://medium.com/@the_data_girl/e-commerce-customer-behavior-analysis-using-tableau-d50d6341d330?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/d50d6341d330</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Fri, 17 Nov 2023 18:18:41 GMT</pubDate>
            <atom:updated>2024-01-03T14:54:24.087Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqR-Jh0MruHpbamwtVJ2MA.png" /><figcaption>Source: <a href="https://unsplash.com/photos/turned-on-black-and-grey-laptop-computer-mcSDtbWXUZU">Unsplash</a></figcaption></figure><p>In this article, we will try to understand customer&#39;s behavior (by analyzing their events history ) and different brands performance. You can get the dataset from <a href="https://www.kaggle.com/datasets/mkechinov/ecommerce-behavior-data-from-multi-category-store">Kaggle</a>.</p><h4>ABOUT THE DATASET:</h4><p>The file contains behavior data from a large multi-category online store.</p><p><strong>Dataset columns:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5VRfn5MaNtdyt9A0QVK1qg.png" /><figcaption>Source: Kaggle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oDoJkdefZSlfPRsjN4XjTA.png" /><figcaption>Data snapshot (Source: Kaggle)</figcaption></figure><h4><strong>CLEANING THE DATASET:</strong></h4><p>Before we begin our analysis, cleaning the data to remove the noisy part that might give incorrect results is important.</p><p>Steps followed to clean the data:</p><ol><li>Checked for rows containing any null values and removed them</li><li>Removed duplicate data</li><li>Ensured each column had a consistent and correct data type</li></ol><p><em>Data cleaning also involves standardization, and normalization (also known as feature engineering )of the numerical data. This is done to bring the dataset to a similar scale. This step is mostly done while training a model to ensure all features contribute equally to the model and to avoid the domination of features with larger values. As this step is not relevant to our use case we will skip it.</em></p><p><strong>Simple Python code to clean the dataset</strong></p><pre>import pandas as pd<br>import os<br><br>#DATASET PATH<br>path=os.path.join(os.path.dirname(os.path.realpath(__file__)),&#39;data/2019-Nov.csv&#39;)<br>df_ecommerce=pd.read_csv(path)<br>print(&#39;DATAFRAME INFORMATION &#39;,df_ecommerce.info())<br>print(&#39;SIZE OF THE DATAFRAME &#39;,df_ecommerce.shape)<br><br>#DROPING ROWS WITH NULL VALUES<br>df_ecommerce.dropna(axis=0,inplace=True)<br><br>#DROPPING DUPLICATE ROWS<br>df_ecommerce.drop_duplicates(subset=[&#39;event_time&#39;,&#39;event_type&#39;,&#39;product_id&#39;,&#39;category_id&#39;,&#39;user_id&#39;,&#39;user_session&#39;],inplace=True)<br>print(&#39;FINAL DATAFRAME SIZE &#39;,df_ecommerce.shape)<br><br>#WRITING THE CLEANED DATA INTO A CS FILE<br>df_ecommerce.to_csv(&#39;cleaned_ecommerce.csv&#39;,sep=&#39;,&#39;, index=False, encoding=&#39;utf-8&#39;)</pre><h4>TABLEAU DASHBOARD</h4><p>I used the clean dataset for building the dashboard.</p><blockquote><strong>E-COMMERCE CONSUMER ANALYSIS</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQNz3cIlspeZqGQ8H3AXhA.png" /><figcaption>Dashboard 1</figcaption></figure><p>This dashboard displays brand performance and consumer purchase behavior.</p><p><strong>Top Brands: </strong>The chart<strong> </strong>displays the top 10 brands where consumers purchase most products. We can see Samsung and Apple leading the race.</p><p><strong>Top Category Purchased</strong>: The chart shows the top product category under purchased under every brand. For Oppo, it is smartphones, and for Apple and Acer it is notebooks. It even provides actionable insights for brands, directing them towards categories with increased customer purchasing activity.</p><p><strong>Event Type Trend</strong>: The chart displays consumer activity trends such as viewing, purchasing, or adding the product to the cart in November(2019).</p><p>Most of the customers who added the products to their cart ended up buying them. We can even observe an increase in sales during the middle of the month. This informs brands about peak user activity times, allowing strategic deployment of methods like ads or discounts to prompt increased product purchases.</p><p>On average <strong>13,10,474</strong> customers view products per day, <strong>68,716 </strong>added the product to their cart, and <strong>22,733</strong> purchased the product(I got these values by adding the average line available in the Analytics pane section for each event type in Tableau).</p><p>Around 1.74% of users who viewed the product purchased it. The percentage of consumers who have viewed the product and purchased it is significantly less.</p><blockquote><strong>E-COMMERCE PRODUCT ANALYSIS</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46LkEQWpCWXUarTNB1-r-w.png" /><figcaption>Dashboard 2</figcaption></figure><p>The dashboard shows the performance of the top 10 brand’s products and their price ranges.</p><p><strong>Product Purchase Rate</strong>: The chart shows the percentage of products under each brand that are purchased by the customers. Samsung has the highest purchase rate( approx 0.9%).</p><p>The % of purchases is very less compared to % of the view and approximately similar to the % of the cart. Encouraging users to add viewed products to their carts, through incentives such as discounts, can effectively boost sales for these brands.</p><p><strong>Highest Purchase Category</strong>: The chart shows the most purchased product category which is smartphones. Smartphones are the highest purchased products followed by notebooks and desktops and shoes being the least purchased product.</p><p>This chart furnishes brands with insights into customer product preferences, enabling the development of similar offerings.</p><p><strong>Price Range of Purchased Products</strong>: The chart shows the different price ranges in which users have purchased the products. Observing that the majority of purchases fall below $500, producing products within this price range could enhance sales opportunities for brands, as higher prices correlate with reduced chances of purchase.</p><p>Through the provided dashboards, we got insights into customer product preferences, assessed the performance of various brands, and identified a correlation between price and purchase behavior, facilitating a comprehensive analysis of consumer trends.</p><h4>REFERENCES:</h4><ol><li><strong>Data cleaning</strong>: <a href="https://www.analyticsvidhya.com/blog/2021/06/data-cleaning-using-pandas/">https://www.analyticsvidhya.com/blog/2021/06/data-cleaning-using-pandas/</a></li></ol><p><em>That’s all I have to say in this article. Thank you readers for your precious time!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d50d6341d330" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s the use of SELF in Python?]]></title>
            <link>https://medium.com/@the_data_girl/whats-the-use-of-self-in-python-c12644d2f0db?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/c12644d2f0db</guid>
            <category><![CDATA[oops-concepts]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[code]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 06:19:23 GMT</pubDate>
            <atom:updated>2023-07-24T06:19:23.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*flCjLnXIFhgFJkB30YjN0g.jpeg" /><figcaption>source: <a href="https://unsplash.com/photos/ieic5Tq8YMk">Unsplash</a></figcaption></figure><p>I have noticed many Python programmers get confused regarding when to use the self and what purpose it serves. In this article, I am planning to give an overview of the self and its correct usage.</p><blockquote>WHAT IS A SELF?</blockquote><p>Before understanding self, let’s discuss a few terms.</p><p><strong>Instance method</strong>: In OOPs programming instance method is associated with class objects. They can access or modify the object state and accept self as the first input parameter. They are mainly used to change the value of instance variables. You can refer <a href="https://pynative.com/python-class-method-vs-static-method-vs-instance-method/">this</a> article to learn more about instance, static and class methods.</p><p>Self keyword refers to the current object in a class (it replicates the behavior of this pointer in Java). It is the first argument passed to an instance method and it represents an instance of the class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*OSmQZlcKSxl4GyuXLMM1yw.png" /><figcaption>Code Snippet</figcaption></figure><p>In the above code snippet you can see Class vehicles has an instance method called description which uses the self keyword as its first parameter. Car and truck are the two objects (that have their own attributes) of the class whose attributes are passed to the instance method.</p><p>Output of the above code snippet will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*_de88w_7dQ_BjkLnb0splA.png" /><figcaption>Output snippet</figcaption></figure><p>From the output, we can infer that the self keyword helps in storing the current state of the object, hence we were able to print the respective attributes of the car and truck objects.</p><p>In case the self-argument was not present then the same class could not hold information about 2 or more objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*OB60w2UjJuqDQP_1PRmy6Q.png" /><figcaption>Code snippet</figcaption></figure><p>The above code without self argument produces an errored output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*Ny6FGgf1WRHlUiJNf9ULGQ.png" /><figcaption>Output snippet</figcaption></figure><p>This is because every instance method and constructor requires self to be the first input. As python religiously follows ‘<em>Explicit better than implicit’ </em>policy, we have to explicitly define self as an argument.</p><blockquote>WHEN TO USE SELF?</blockquote><p>Whenever you are dealing with class and instance methods, it is advised to use self. It should be used while referencing the current instance attribute.</p><p>To sum it up, self is mainly used to refer to the current state of the instance. You can even avoid using self by making the method static (since all static methods act as normal functions).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*g8gltUaX6ZNx_dAhCegSYQ.png" /><figcaption>Code Snippet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*DsSlP8lyOl4q3lZBftoedA.png" /><figcaption>Output snippet</figcaption></figure><p>@staticmethod acts as a function decorator and makes the name method static. Here, we were able to print the output without passing self.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c12644d2f0db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handy Guide For Data Engineers]]></title>
            <link>https://medium.com/@the_data_girl/handy-guide-for-data-engineers-97cfe487041?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/97cfe487041</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Sat, 24 Sep 2022 17:58:56 GMT</pubDate>
            <atom:updated>2022-09-24T17:58:56.168Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6FT-pQqe3YduwMZEvdB2Q.jpeg" /><figcaption>source: <a href="https://unsplash.com/photos/0nOP5iHVaZ8">Unsplash</a></figcaption></figure><p>Data engineering is a vast and rapidly growing field. With so many emerging tools &amp; platforms, it might feel overwhelming for beginners to understand all the tools, and programming languages used in data engineering. In this article, I have tried my best to curate almost all the tech stacks and tools commonly used by data engineers.</p><p>Ps: I have provided only a brief explanation about all the tools. You can use this as a guide to upskill yourself.</p><ol><li>PROGRAMMING LANGUAGES</li></ol><p>The most common languages used by data engineers (DE) are SQL, Python, Scala, Java, and R.</p><p>2. DATABASES</p><p>I.SQL (RDBMS): It stores data in a structured format( i.e. tabular format). The data is stored in rows and columns. A few of the popular relational databases used by DE are MySQL, Oracle SQL, IBM DB2, and Microsoft SQL Server.</p><p>II. NO-SQL: NoSQL is used when data exists in a non-tabular format. For example, document-based, graph-based, columnar-based, and key-value based. A few of the common NoSQL databases used by DE are MongoDB, Apache HBase, and Apache Cassandra.</p><p>3. DATA REPOSITORIES</p><p>I. DATA WAREHOUSE: It acts as a central data repository for an organization that usually stores structured data which are ready for analysis. Common providers of DW are Snowflake(Virtual DW), Amazon Redshift, Google BigQuery, Azure Synapse Analytics, IBM DB2, and Firebolt.</p><p>II. DATA MART: It can be considered as a subset of a data warehouse i.e. it contains data that is relevant to the application/business. For example sales team data, finance team data, and many more. Common vendors of DM are Oracle, IBM, Snowflake, and Sybase.</p><p>III. DATA LAKE: It is used to store data in raw form and can store structured,semi-structured, and unstructured data. Unlike a data warehouse, you can’t perform any transactions or enforce data quality on a data lake. Common vendors of DL are AWS Data lake, Cloudera Data Platform, Azure Data Lake, Databricks, and Google Data Lake.</p><p>IV. DATA LAKEHOUSE: It is the perfect amalgamation of Data Warehouse and Data Lake. It provides cheaper storage of data (in all formats) and even allows ACID transactions, and supports schema enforcement, and data governance. Common vendors of Data lakehouse are Delta Lake (By Databricks), Cloudera, and IBM DB2.</p><p>V. HDFS: Used by Hadoop to store big data in distributed format. It is responsible for breaking the input data and storing them in different clusters.</p><p>4. BIG DATA PROCESSING</p><p>I. Mapreduce: It is software used for distributed processing of big data on a computer cluster.</p><p>II. Spark: It is also used for processing big data, the main difference between MapReduce and spark is the way they process the data and their speed.</p><p>5. RESOURCE MANAGER: They are mainly responsible for allocating and monitoring cluster resources (such as Memory, disk space, CPU, and many more). Commonly used RM are :</p><ol><li>YARN</li><li>MESOS</li><li>Spark Standalone RM</li></ol><p>6. DATA INGESTION</p><p>When you working with real-time streaming data there is a possibility you might need a data ingestion tool to avoid data loss in case your data repository fails to store the data. Most commonly used data ingestion tools are Apache Kafka(general purpose ingestion tool), Apache Flume( mostly designed for Hadoop), Apache Kinesis(AWS managed Kafka), Apache Sqoop(Designed for the relational database), Airbyte, Apache Storm, and many more.</p><p>7. Zookeeper</p><p>The main task of Zoopkeeper is to “Watch the Watchers” i.e. it ensures that all the managing/master nodes are up and running, in case of failure it will assign a substitute node to become the master node. Thus it helps in maintaining synchronization among large clusters in distributed systems.</p><p>8. Oozie</p><p>Oozie is a workflow management tool for Hadoop jobs. It provides a platform to create a pipeline from these jobs and run them at a given schedule.</p><p>9. Workflow management</p><p>Workflow management tools are mainly used to automate the pipeline run. For instance, if you have a pipeline that you need to run once at 10 a.m. every day, instead of running it manually you can schedule the run. In these scenarios, workflow management tools are used that allow you to schedule the run. Apart from this, they are even helpful in debugging your pipeline. You can even send alert notifications/emails in case of an error or failure of the pipeline. Two of the most common workflow tool used are airflow and prefect.</p><p>10. Cloud</p><p>Needless to say, the cloud is the most important concept for a data engineer. Cloud is nothing but a virtual computer, it delivers all its services(like compute power, storage, ram, servers) via the internet. Cloud is very useful once you start working with big data since they have the ability to store petabytes of data, scale the application as per request and increase the overall efficiency and load of the application. Common cloud services used by many companies are AWS(Amazon), GDP(Google), and Azure(Microsoft).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97cfe487041" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Data Lakehouse Architecture]]></title>
            <link>https://medium.com/@the_data_girl/demystifying-data-lakehouse-architecture-91955b138774?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/91955b138774</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Fri, 08 Jul 2022 07:41:41 GMT</pubDate>
            <atom:updated>2022-07-08T07:45:11.661Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Lakehouse which is considered the modern era of central repositories is the perfect amalgamation of Data Lake and Data Warehouse. It was developed with the aim of mitigating the major limitations of the data lake and data warehouse.</p><p>Data Lakehouse acts as a ‘one-stop platform’ providing almost all the features of data lakes and data warehouses. It provides the facility to store data in raw format with directly-accessible storage. It also provides analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization.</p><p>Examples of data lakehouses vendors are Delta Lake( by Databricks), Snowflake, Amazon Athena and Redshift( by AWS), Bigquery( by Google), and many more.</p><p>In short, it is a data management architecture that provides cost-efficient, flexible, and scalable storage along with data that can be used by analysts for complex SQL analysis, and by data scientists for training their models.</p><p>If you want a deeper understanding of Data Lakehouse, check out this <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html">article</a>.</p><p>Assuming you have knowledge about Data Lakehouse and how it is different from Data Warehouse and Data Lake. Let’s dive right into its <strong>architecture.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bcrc2cbp0FmjFCjz.png" /><figcaption>Source: <a href="https://databricks.com/glossary/data-lakehouse">Databricks</a></figcaption></figure><p><strong><em>Data Source</em></strong>: Data lakehouse unlike data warehouse accepts structured( table),semi-structured(emails,JSON,NoSQL), and unstructured data(images,voice clips).The raw data is ingested into the storage layer. This can be done using Snowpipe, Amazon DMS, Apache Spark, and Apache flume for streaming data, etc.</p><p><strong><em>Storage layer</em></strong>: One of the pivotal ideas while implementing lakehouse was to use a low-cost object storage system(such as an Amazon S3 bucket). Then a transactional metadata layer is built on top of the object-store. This allows the system to perform ACID transactions or other management features within the metadata layer. This helps in keeping the original raw data in the object-store and perform data transformations in the same layer.</p><p><strong><em>Metadata layer</em></strong>: This is present in the storage layer. It is this layer that sets the architecture of the data lakehouse apart from the data warehouse and data lake.Features provided by this layer are:</p><ol><li><em>Management</em>: Metadata layer is usually on top of data lake and has the capability to implement ACID transactions,indexing,caching,query optimization and other DBMS management and performance features.</li><li><em>Data quality enforcement</em>: It allows users to implement schema enforcement to ensure that the data is uploaded in the correct format into the table.They can even set constraints on the incoming data (like username can’t have an underscore or pincode should be a 6-digit number) using constrain APIs.</li><li><em>Goveranance</em>: It provides access to control and audit logging. It can restrict users with no read credentials from accessing the tables.</li></ol><p>This layer makes data lakehouse performance more efficient and faster and even allows it to implement data warehouse features.</p><p><strong><em>API layer</em></strong>: The <a href="https://databricks.com/wp-content/uploads/2018/12/sigmod_structured_streaming.pdf">declarative DataFrame APIs</a> allow the data lakehouse to speed up advanced analytics workload. Various ML libraries such as( Tensorflow or Spark MLlib) are implemented by querying the metadata layer to figure out which files are currently in use and just passing those files to the libraries.</p><p><strong><em>Consumption layer</em></strong>: This is the final layer which is consumed by the end users. It contains various analytics and visualization tools such as Power BI and Tableau. Since the client has access to all the stored data( and metadata) they can use it for Business Intelligence, creating interactive dashboards, or training ML models.</p><p>This is one of the many data lakehouse architecture that can be implemented by utilising the power of metadata layer. According to your use-case the architecture might vary .For example the current metadata layer offers high availability and high read bandwidth,but has slow transactions/second rate( due to high latency since, it stores the transaction logs in the same object store). To avoid this you can even use a faster storage system.</p><p><strong>Advantage of Data Lakehouse over Data warehouse and Data Lake</strong></p><ol><li>Supports all form of data i.e. Structured, un-structured, streaming, and open format such as Apache Parquet and ORC (Optimized Row Columnar).</li><li>Decoupled storage and compute cluster improves the scalability and allows parallel processing of data which increases the efficiency of the system.</li><li>Supporting dataframe API allows data optimization and quering metadata layer specific to the requirement( i.e. querying for necessary ML libraries)</li><li>It even provides DBMS management features(like ACID support), flexible schema( data lake feature) ,data governance and BI analytics ,vizualition support all in one platform.</li></ol><p>As the data lakehouse architecture is still in development phase, there are a few drawbacks such as the SQL performance of lakehouse is not as good as the warehouse(which stores data on SSDs,has efficient access methods and more efficient compute engine). Data lakehouse has high latency rate and supports transactions of one table at a time.</p><p>There is still good amount of work to be done on data lakehouse before its implementation in the real world projects.Inspite of the drawbacks data lakehouse shows great potential in replacing data warehouse and become the next-generation of central repository.</p><p><strong>References</strong></p><ol><li><a href="https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf">https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf</a></li><li><a href="https://databricks.com/glossary/data-lakehouse">https://databricks.com/glossary/data-lakehouse</a></li><li><a href="https://www.altexsoft.com/blog/data-lakehouse/">https://www.altexsoft.com/blog/data-lakehouse/</a></li></ol><p><em>That’s all I have to say in this article. Thank you readers for your precious time!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91955b138774" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are Big Data and Data Warehouse the same?]]></title>
            <link>https://medium.com/@the_data_girl/are-big-data-and-data-warehouse-the-same-61a616a0b0f0?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/61a616a0b0f0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Fri, 01 Jul 2022 17:57:27 GMT</pubDate>
            <atom:updated>2022-07-01T17:57:27.348Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*a1xTBd8f9Z2rWJR6" /><figcaption>Source: <a href="https://unsplash.com/photos/KeFyYzxqmH0">Unsplash</a></figcaption></figure><p>I have seen many people get confused between big data and data warehouses. Big data and data warehouses are not interchangeable terms, instead, they have pretty clear differences.</p><blockquote>DATA WAREHOUSE</blockquote><p>A data warehouse is a <strong>central data repository that stores data from various sources</strong>. Since it contains clean and structured data, it is used by analysts to make informed decisions that help in the growth of the business.</p><blockquote>BIG DATA</blockquote><p>Big data is <strong>larger and more complex data sets. </strong>Any data that satisfies the 3v’s (Velocity, Variety, and Volume) can be termed as big data.</p><p>Velocity: The speed at which data arrives in the repository.</p><p>Variety: Talks about data coming from different sources/applications.</p><p>Volume: Talks about the amount/size of data that enters the repository.</p><p>Traditional systems (OLTP or OLAP) can’t process big data. Hence we require a different approach to handle and process big data (which is ideally cloud).</p><blockquote>KEY DIFFERENCES</blockquote><p>1.<strong>Big data is a technology, unlike a data warehouse which is an architecture for storing data</strong>. Big data provides a solution to handle and process such a vast amount of data.</p><p>2. Data warehouse <strong>accepts only structured or semi-structured data(i.e. it does not accept video, image, or gif data), whereas big data accepts data of any format</strong>(structured,semi-structured and unstructured).</p><p>3. Data warehouses <strong>used ELT/ELT to store and process the data, whereas big data does not use any such method</strong>.</p><p>4. Big data <strong>use distributed systems to load data, which is not used by data warehouse</strong>.</p><p>5. Data warehouse<strong> mainly holds historical data for processing, big data also holds historical data for analysis but it also provides an option to perform real-time analysis of data</strong>.</p><p>6. Processing <strong>data in a data warehouse is time-consuming and is slower than big data processing</strong>(which can be done using Hadoop).</p><p>7. Data warehouse <strong>has a fixed schema(Star or Snowflake) format, since big data accepts data in all formats there is no pre-defined schema</strong> for data.</p><p>Many people assume that big data is going to replace data warehouses, which I feel is incorrect since <strong>both have different use-cases and sometimes are even complementary to each other. Big data can even be combined with a data warehouse to improve the efficiency of processing and analyzing huge volumes of data</strong>. (a.ka. <strong>Big Data Warehouse</strong>)</p><blockquote>REFERENCES:</blockquote><ol><li>Big Data vs Data Warehouse:<a href="https://www.educba.com/big-data-vs-data-warehouse/">https://www.educba.com/big-data-vs-data-warehouse/</a></li></ol><p><em>That’s all I have to say in this article. Thank you readers for your precious time!</em></p><p><em>Don’t forget to leave a clap </em>👏, <em>if this article has helped you in any way. This keeps me motivated to write more articles and try my best to contribute to the growth of our data engineering community :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61a616a0b0f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NoSQL for Beginners]]></title>
            <link>https://medium.com/@the_data_girl/nosql-for-beginners-62ad24b1c515?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/62ad24b1c515</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Sat, 05 Mar 2022 17:54:17 GMT</pubDate>
            <atom:updated>2022-03-05T17:54:17.578Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Rjf3sJlTrtWe1PyJ" /><figcaption>Source:<a href="https://unsplash.com/">https://unsplash.com/</a></figcaption></figure><p>NoSQL a.k.a <strong>Not Only SQL </strong>encompasses all the databases that follow the below-mentioned condition<strong>:</strong></p><ol><li>The <strong>data is Non-relational which means the data can’t be expressed in the standard row and column(Tabular) format</strong>.</li></ol><p>Popular NoSQL examples:<em> MongoDB, Apache Cassandra DB, Oracle NoSQL Database, Riak.</em></p><blockquote><strong>Why was NoSQL created?</strong></blockquote><p>One of the obvious reasons is to store and process non-relation type of data(Eg. Document, Key-Value type data, Structured, Unstructured data). Another reason was to build a database that can handle big data processing. Since Big Data came into existence after RDBMS, it is difficult to scale RDBMS horizontally( <a href="https://www.cloudzero.com/blog/horizontal-vs-vertical-scaling#:~:text=Horizontal%20scaling%20(aka%20scaling%20out,server%20may%20be%20your%20solution.">horizontal-scaling</a>) as a result it was inefficient to process big data ( even now it is very expensive to process big-data via RDBMS). Since NoSQL databases can be easily scaled horizontally they proved to be a better option while working with big data. Also to create a database that is distributed in nature which makes it easy to deploy on the cloud.</p><blockquote><strong>Types of NoSQL</strong></blockquote><p>NoSQL largely supports 4 types of data formats:</p><ol><li>Key-Value</li><li>Document</li><li>Graph</li><li>Column</li></ol><blockquote><strong>1.Key-Value Database</strong></blockquote><p>Data is stored in the format of Key-Value as individual records. Both key and value can be of any data type. Here the key acts as a unique identifier. This is the least complex type of data format which is usually represented as a hashmap. It scales very well and is not intended for complex queries. It is ideal for basic CRUD (Create, Read, Update, Delete) operations.</p><p>Data format: {“Key”: “Value”}</p><p>In this type, the value isn’t visible and can’t be queried, thus limiting the scope of operations on the data.</p><p><strong>The most suitable use-case of Key-Value database:</strong></p><ol><li>They are suitable for non-interconnected data that doesn’t involve frequent updates or complex queries. For eg. Storing and retrieving session information(log file) of a web application.</li><li>Can be used in cache management and blockchain implementation.</li><li>Can be used to store in-app user profile data, their preferences. For eg. storing shopping cart data in e-commerce apps (such as Amazon, Myntra, Flipkart, etc.)</li></ol><blockquote><strong>The most unsuitable use-case of Key-Value database:</strong></blockquote><ol><li>When the data is complex and highly interconnected with a many-to-many relationship.</li><li>When an application runs queries based on value instead of a key. (As you don’t have access to value in the Key-Value database, hence you can’t query records on basis of value.)</li></ol><p><em>Eg. of Key-Value databases are Amazon Dynamo DB, Redis, Oracle NoSQL.</em></p><blockquote><strong>2. Document Database</strong></blockquote><p>The database stores information in form of a document.(For Eg: JSON,XML)It is based on the key-value database, but here the value is visible and can be queried.</p><p><strong><em>JSON format example</em>:</strong></p><p>{ “information”:[</p><p>“_id”: 1,</p><p>“first_name”: “Mike”,</p><p>“email”: “mike@example.com”,</p><p>“cell”: “869–585–0000”,</p><p>“likes”: [ “fashion”, “spas”, “shopping” ]</p><p>]</p><p>}</p><p>JSON in simple term is collection of key-value pairs that are grouped to form one record.</p><p>It is comparatively easier to add new information and add additional fields to existing documents in the document database. Searching and querying for information is also faster.</p><p>Document Database offers a variety of features:</p><ol><li>Flexible Schema: It is not compulsory for each document in a collection to have the same column name and type, it is up to the developers to store different fields in each document and create a collection of interconnected documents. But there is also a provision to lock-down schemas if needed.</li><li>Atomic Operation: It guarantees an atomic operation on a single document.</li></ol><p>3. Distributed: The data can be distributed i.e. it can be stored on multiple interconnected nodes/servers. This increases the query speed and makes data more available. It can also be horizontally scaled up.</p><p>4. Resilient: The data is replicated among multiple nodes so that it is always available despite a server or hardware failure.</p><blockquote><strong>Suitable use-cases of document database:</strong></blockquote><ol><li>It can be used in applications similar to online blogs (content management)where there is a requirement to store user data, their activity such as likes, posts, comments, etc.</li><li>It can be used when the data must be available all the time. Eg. Sensor data that is being used by an IoT product/application, payment portals.</li><li>Can be used in mobile apps to store operational datasets and metadata.</li></ol><blockquote><strong>Unsuitable use-cases of document database:</strong></blockquote><ol><li>Not an ideal option when ACID transactions are involved.</li><li>Not ideal for a large amount of sparse data.</li></ol><blockquote><strong>Column Database:</strong></blockquote><p>Unlike RDBMS where data is stored in form of rows, it is stored in groups of columns or in a column.Several such columns are grouped to create column families which has several rows with unique keys that belong to one or more columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*i_7BafQJBp5orjNB.png" /><figcaption>Source: <a href="https://www.kdnuggets.com/2021/02/understanding-nosql-database-types-column-oriented-databases.html">https://www.kdnuggets.com/2021/02/understanding-nosql-database-types-column-oriented-databases.html</a></figcaption></figure><p>Here UserProfile is the column family that has 3 rows with unique keys (“Bob”, “Britney”,”Torl”) which further have columns storing information. This is the basic syntax of how data is stored in column database.</p><p>Each row in column family has a row key and a column.</p><p>The column further has three rows namely:</p><ol><li>The name of the column</li><li>It’s value</li><li>Timestamp</li></ol><p>Column-based database providers more flexible schema compared to RDBMS.</p><blockquote><strong>Suitable use-cases of Column database:</strong></blockquote><ol><li>They are widely used when there is a large amount of sparse data.</li><li>Since column databases can handle being deployed across multiple clusters. This makes quering faster which in turn reduces latency. Therefore they can be used for event logging and blogs.</li><li>Few column database also come with TTL(Time To Live) parameter making them useful for data with an expiration value.</li><li>Column database are usually used in cases when an application is heavy on read and light on writes.</li></ol><blockquote><strong>Unsuitable use-cases of Column database:</strong></blockquote><ol><li>Not suitable for an application where query patterns change frequently as it leads to numerous changes of column designs.This can be costly and can even slow down the production.</li></ol><p><em>Examples of column databases: Cassandra, Hypertable.</em></p><blockquote><strong>Graph Database:</strong></blockquote><p>Graph database is pretty different from other NoSQL databases. It stores information in nodes/entities and in edges/relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/0*1Tq1BfczBlD4pH8J.png" /><figcaption>Source: <a href="https://www.geeksforgeeks.org/graph-data-structure-and-algorithms/">https://www.geeksforgeeks.org/graph-data-structure-and-algorithms/</a></figcaption></figure><p><strong>Node: </strong>It is equivalent to rows in a table.They are the vertex in a graph.</p><p><strong>Relationships/Edge</strong>: They are edges in the graph. They establish relationship between two nodes.</p><p>It is mostly used when the data resembles graph like data-structure.They are the only database which are ACID compliment and can’t be scaled horizontally.</p><blockquote><strong>Suitable use-cases of Graph Database:</strong></blockquote><ol><li>Ideal for highly connect and related data. You can find them widely used in social networking applications such as Instagram, Facebook where each node represents a user and edges establish relationship between two nodes such as friends or friends of friends.</li><li>They are even used in routing,map apps and in recommendation engines.</li></ol><blockquote><strong>Unsuitable use-cases of Graph Database:</strong></blockquote><ol><li>They don’t work well with distributed data as graph don’t shard well. It is because of the fact that is it difficult to traverse a graph with nodes split across multiple clusters.</li></ol><p><em>Example of Graph Database: Neo4j, Amazon Neptune, Apache Giraph, Janus Graph.</em></p><p>REFRENCES:</p><ol><li><a href="https://www.mongodb.com/nosql-explained">https://www.mongodb.com/nosql-explained</a></li><li><a href="https://www.kdnuggets.com/2021/02/understanding-nosql-database-types-column-oriented-databases.html">https://www.kdnuggets.com/2021/02/understanding-nosql-database-types-column-oriented-databases.html</a></li></ol><p><em>That’s all I have to say in this article. Thank you readers for your precious time!</em></p><p><em>Don’t forget to leave a clap </em>👏, <em>if this article has helped you in any way. This keeps me motivated to write more articles and try my best to contribute towards the growth of our data science community :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62ad24b1c515" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PL/SQL for Data Engineers]]></title>
            <link>https://medium.com/@the_data_girl/pl-sql-for-data-engineers-9a571ae0f75?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a571ae0f75</guid>
            <category><![CDATA[plsql]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 16:58:30 GMT</pubDate>
            <atom:updated>2022-02-16T06:08:09.115Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eGHkpxzlnzwfR6yBG1JgWw.jpeg" /><figcaption>Source: <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>I recently came across an article that talks about how PL/SQL is dead and has no scope in the future. I disagree with this statement and believe that PL/SQL will be used by SQL developers and Data Engineers in the future. I am writing this article to explain how programmers can leverage PL/SQL features to process data more efficiently.</p><blockquote><strong>What is PL/SQL? How is it different from SQL?</strong></blockquote><p>PL/SQL stands for Procedural Language extensions to SQL offered by Oracle and is available on Oracle and other databases. It has the combined power of SQL queries and procedural programming.</p><p><strong><em>SQL+PROCEDURAL PROGRAMMING=PL/SQL</em></strong></p><p><strong>SQL(Structured Query Language)</strong>: It is a programming language that is used to manipulate, define, query data, and perform other operations on data. In short, it is used to communicate with databases.</p><p><strong>Procedural Programming</strong>: It is a programming model that is used to create structured/modular scripts. Each module is written to perform one action/task. It works on the concept of calling a procedure. A procedure is a routine/function/sub-routines that are developed to compute a series of instructions step-by-step.</p><p>PL/SQL can not only execute SQL queries but also wrap the queries into a block that can be called any number of times to perform operations on different databases.</p><p>PL/SQL advantages over SQL:</p><p><strong>Procedural Language Support</strong>: It allows developers to use conditional statements(if,if-else), loops, throw exceptions, build packages that help them in adding more functionality to their application, and even in building routines that can be reusable.</p><p><strong>Error Checking and Handling</strong>: Unlike SQL, PL/SQL offers developers to use exceptions to catch the error and give instructions on how to handle them. PL/SQL provides inbuilt as well as user-defined exceptions.</p><p><strong>Data Structures</strong>: Unlike SQL, PL/SQL offers few data structures like varray, records, collections that can store collections of elements. Programmers can use them to make code more readable and less complex.</p><p><strong>Dynamic SQL</strong>: A programming technique that enables you to build SQL statements dynamically at runtime. This can be used when your query execution depends upon the user’s input. For eg. on basis of the user’s location, you execute different queries. If the user’s location is India then you execute queries that use the India records.</p><p><strong>Reduce Network Traffic</strong>: As PL/SQL statements are encapsulated into a block that is executed all at once in the oracle engine it helps in reducing the network traffic.</p><p>Data engineers mainly work on maintaining, manipulating, updating the data in the database. They can use PL/SQL to build structured modular procedures which can be reused across multiple applications and data pipelines.</p><p><em>Another advantage of PL/SQL is that it is very easy to learn. Anyone who has a basic understanding of any programming language (C++, Java, Python) and SQL will be able to learn the basics of PL/SQL within 2–3 weeks.</em></p><blockquote><strong>Resources to learn PL/SQL</strong>:</blockquote><p><a href="https://www.tutorialspoint.com/plsql/index.htm"><strong>TutorialsPoint</strong></a>: They have covered all the basic concepts of PL/SQL and have explained them in a concise and intelligible manner.</p><p><em>That’s all I have to say in this article. Thank you readers for your precious time!</em></p><p><em>Don’t forget to leave a clap </em>👏, <em>if this article has helped you in any way. This keeps me motivated to write more articles and try my best to contribute towards the growth of our data science community :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a571ae0f75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A-Z about Data Engineers]]></title>
            <link>https://medium.com/@the_data_girl/a-to-z-about-data-engineers-1a69e85cf7be?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a69e85cf7be</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databse]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Sun, 30 Jan 2022 16:59:09 GMT</pubDate>
            <atom:updated>2022-01-30T17:26:32.852Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*mf8hJXzYzB3FAI4xKGsQNA.jpeg" /><figcaption>Source <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>Data on the internet has been increasing at an exponential rate. Best estimates suggest that at least 2.5 quintillion bytes of data is produced every day (that’s 2.5 followed by 18 zeros!). <strong><em>Nowadays data is being considered the most invaluable asset to any organization</em></strong>. This has made organizations realize the importance of storing and using data wisely to their benefit. <strong>This increased the demand for data engineers which grew by almost 50% over the past few years. </strong>It is considered to be one of the fastest-growing tech roles in 2021.</p><blockquote>Who are Data Engineers?</blockquote><p>The work of data engineers differs from company to company and also the industry in which they work. Usually, data engineers are people who come after Database Administrators and before Data Analysts and Data Scientists. <strong><em>Their main aim is to create data that is analysis-ready which will be used by data scientists and analysts</em></strong>. A few common tasks that data engineers perform are:</p><ol><li>They usually deal with extracting raw data from different sources.</li><li>They are responsible for transforming the raw data(i.e cleaning, normalizing, standardizing the data).</li><li>They are also responsible for storing and maintaining the data in the database. (Eg. Data warehouse, Data Mart, Data Lake, etc)</li><li>They make use of cloud services to provide efficient and scalable data.</li><li>They even know how to use big data tools (such as Apache Hadoop, Spark, etc.) to process and store big data.</li><li>They also use visualizing tools to display the data to the end-users (eg. to stakeholders, business clients, etc)</li><li>In some cases, they also use machine learning algorithms to extract information that will in turn help in improving the business of an organization.</li></ol><blockquote>Data Engineers v/s Data Analyts</blockquote><p>Data Analysts don’t need to worry about extracting and storing data or building data pipelines to move data from point A to point B. Their main role is to use visualizations tools to plot data and make inferences from it. They don’t need to learn SQL, big data tools, or know about the ETL/ELT process. Whereas data engineers don’t need to know statistics which is a required skill for data analysts.</p><p>Data engineers prepare the data which is used by the data analyst for their analysis.</p><blockquote>Data Engineers v/s Data Scientist</blockquote><p>Data scientists make use of the data built by data engineers. They usually feed the data to machine learning models to either make predictions (eg. predicting the sale of a product for the next 6 months) or for classifications(eg. classifying if an MRI scan has a tumor or not, classifying emails as spam or not spam).</p><p>Data scientists must be proficient in Machine Learning algorithms, deep learning, statistics which are not crucial skills for data engineers.</p><p>In conclusion data engineers solely work with the data in the enterprise. You can also call them the guardians of the data as they are responsible for designing and automating data pipelines that will collect, store, analyze data at scale.</p><blockquote>Skills required to be a Data Engineer</blockquote><p>Data engineers are expected to know at least the basics of the below-mentioned skills:</p><ol><li>SQL Database (Eg. MySQL, Oracle, IBM DB2)</li><li>No-SQL Database (Eg. Mongo DB, Cassandra, Redis)</li><li>Python/Java/Scala</li><li>ETL (Eg. Informatica, IBM Infosphere Information Server, Talend)</li><li>Data Repositories like Data warehouse, Data Mart, and Data Lake.</li><li>Big data tools (Eg. Apache Hadoop, Spark, Hive, Kafka)</li><li>Cloud Platforms (Eg. Azure, AWS, GCP)</li><li>Linux (Basics)</li><li>Web-scraping (Eg. Beautiful Soup, Selenium, Scrapy)</li><li>Machine Learning Algorithms</li><li>Data APIs</li></ol><p><em>That’s all I have to say in this article. Thank you readers for your precious time!</em></p><p><em>Don’t forget to leave a clap </em>👏, <em>if this article has helped you in any way. This keeps me motivated to write more articles and try my best to contribute towards the growth of our data science community :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a69e85cf7be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Math important for Data Science(DS)?]]></title>
            <link>https://medium.com/@the_data_girl/is-math-important-for-data-science-ds-e8c3e951d91d?source=rss-11885125cf06------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8c3e951d91d</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[The Data Girl]]></dc:creator>
            <pubDate>Sun, 23 Jan 2022 18:48:24 GMT</pubDate>
            <atom:updated>2022-01-24T14:21:01.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*m75d1BOEjoi6jxksaiMqAQ.jpeg" /><figcaption>Source: Unsplash</figcaption></figure><p>Data Scientist <em>(“The Sexiest Job of the 21st Century.”) is a profession being chased by many people from different educational backgrounds (they are mainly divided into two categories engineers and non-engineers).</em></p><p><em>All of them have the same doubt is</em><strong><em> MATH REALLY IMPORTANT TO BECOME A DATA SCIENTIST?.</em></strong><em> If so, then what kind of topics are crucial to study and where can I find resources to study them?</em></p><p>This article is <strong>primarily for those of you who are clueless about whether to study math and how much to study to become a data scientist</strong>. I have also included a few resources from where you can learn the required topics.</p><blockquote><strong>WHY IS MATH IMPORTANT FOR DATA SCIENTIST?</strong></blockquote><p>Before we dive into what math topics are important to become a Data Scientist, let’s understand the importance of maths in DS.</p><p>Data Science in layman terms is a field that extracts insights and information from structured or unstructured data. The information derived is later used to either boost business, identify why the product is not doing well in the market, forecast sales for the next month, etc. Use cases of data science are indefinite, to get a valid result it is crucial to accurately extract information from the data. For this data scientists use various algorithms(popularly known as Machine Learning Algorithms) such as Regression, SVM, Random Forest, and many more. Sometimes they even make use of neural networks(Deep Learning) to get precise outcomes. <strong>These algorithms are fundamentally built by 4 math topics: <em>STATISTICS, LINEAR ALGEBRA, CALCULUS, and PROBABILITY</em>.</strong></p><p>To get correct information from the data it is important to understand how different algorithms work(i.e. how a change in data affects the change in the output) and for this, it is crucial to learn the above-mentioned topics.</p><p>If you are not from a math background you don’t have to get scared by these topics. Once you start learning them you will realize they are pretty easy and interesting. <strong>You need to get your fundamentals strong to move ahead in your DS journey</strong>. <strong>Once you understand the math, coding is very easy.</strong> <em>CODING WILL BE EASY IF YOU UNDERSTAND THE MATH BEHIND THE ALGORITHM</em>.</p><p>I have personally seen many students ignoring the math part of DS and jumping right into building models, this usually leads to them developing poor-performing models because they don’t understand how the model actually functions and what type of data will give better results.</p><p>You don’t have to be a pro at math to become a good Data Scientist all you need is a basic understanding of a few core topics.</p><blockquote><strong>WHAT TOPICS TO STUDY?</strong></blockquote><p>Now that we know the role of maths in DS, let’s learn what topics we must study.</p><p>As I had mentioned before there are 4 topics that must be studied. I have explained them in brief for you to get a rough understanding of how they are used in DS.</p><ol><li><strong>STATISTICS</strong></li></ol><p>It deals with the collection, analysis, and presentation of data. Statistics is the cornerstone of DS. Most of the real-world data is unstructured and must be cleaned before feeding it into the model. Stats are useful whenever we deal with unstructured data as it helps us to identify outliers(they are data points noticeably from a different dataset), handle missing values, understand the distribution of the data, identify the correlation between features.</p><p>Data scientists use dashboards, charts, graphs to visualize the data and use statistical formulas and concepts to derive insights from those visuals. In short, statistics is used in visualizing the data, interpreting data distributions, and in cleaning the data.</p><p>2. <strong>LINEAR ALGEBRA</strong></p><p>Linear algebra deals with linear equations, matrices, and vector spaces. It acts as a foundational block in machine learning algorithms. The understanding of linear algebra will make you consider ML algorithms less of a black box. It will allow you to learn how model learns pattern from data and this knowledge will help you choose better hyperparameters and develop robust models.</p><p>Linear algebra is commonly used in :</p><p><strong>Loss functions</strong>- Tells how different your predicted values are from actual values</p><p><strong>Regularizations</strong>- Regularizations techniques(like L1 and L2 norms) prevent you from overfitting the models.</p><p><strong>Covariance Matrix</strong>- They are used to understand the relationship between two attributes/features(continuous in nature).</p><p><strong>Dimensionality Reduction</strong>- While working with massive data(such as images, videos) that have a huge number of matrices, processing them can be a tedious process(even for supercomputers).In such cases, the original data is reduced to a smaller subset.</p><p>PCA(Principal Component Analysis) is a common technique to reduce the dimensions of the dataset. It uses linear algebra concepts such as eigenvectors and values to reduce the data.</p><p><strong>NLP(Natural Language Processing)</strong>-It deals with processing human languages(which means it deals with words, not numeric data). As you are aware algorithms need numeric data as input. Word embeddings is a common method used to convert words into numerical data, it represents words as low dimensional vectors and tries to preserve their context in the document. It uses the concept of matrices and vectors.</p><p><strong>Computer Vision(CV)</strong>-It deals with processing images and videos. To process images, they are represented in form of 2D matrices. Many operations are performed on images such as convolution(it is just element-wise multiplication of two matrices followed by their sum) which is used in edge detection, sharpening the image, scaling of image(grey scaling), and many more.</p><p>3. <strong>PROBABILITY</strong></p><p>Probability can be defined as the likeliness of an event occurring.</p><p>The famous Bayes theorem uses conditional probability(the probability of an event’s occurrence concerning one or multiple other events). This concept is widely used in ML for classification, optimization, and model hypothesis. The Naive Bayes algorithm is constructed using the Bayes theorem.</p><p>Probability is used in training(Maximum Likelihood Estimation, Maximum a Posteriori Estimation), tuning, evaluation(cross-entropy) of models.</p><p>4. <strong>CALCULUS</strong></p><p>Calculus in simple terms means studying the rate of change of quantities. In ML algorithms we use the loss function which tells us how efficient is our model. To optimize our model, we use gradient descent( gradient measures how much the output changes if you change the input data a little bit) which makes use of calculus to change the parameters(weights and biases) of the model to increase its accuracy and make it robust.</p><blockquote><strong>RESOURCES TO STUDY THESE TOPICS</strong></blockquote><p>After knowing what we have to study, I have included a few resources that helped me understand these topics.</p><p><strong>STATISTICS</strong></p><ol><li><a href="https://youtu.be/Vfo5le26IhY">https://youtu.be/Vfo5le26IhY</a> -(Covers basic statistics concepts)</li><li>Think Stats: Probability and Statistics for Programmers by Allen B. Downey (This book even includes how to implement statistics concept through code(Python))</li></ol><p><strong>PROBABILITY</strong></p><ol><li><a href="https://www.analyticsvidhya.com/blog/2021/04/statistics-and-probability-concepts-for-data-science/">https://www.analyticsvidhya.com/blog/2021/04/statistics-and-probability-concepts-for-data-science/</a> (This article will give you a basic overview of the topics that you must study in probability)</li><li><a href="https://youtu.be/sEte4hXEgJ8">https://youtu.be/sEte4hXEgJ8</a></li></ol><p><strong>CALCULUS</strong></p><ol><li><a href="https://youtu.be/WUvTyaaNkzM">https://youtu.be/WUvTyaaNkzM</a> (Must watch the entire calculus playlist by 3Blue1Brown, he has explained calculus in the most simplest and interesting manner!)</li></ol><p><strong>LINEAR ALGEBRA</strong></p><ol><li><a href="https://www.youtube.com/playlist?list=PLUl4u3cNGP63oMNUHXqIUcrkS2PivhN3k">MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018</a></li></ol><p>These resources cover the basics of each topic, with this knowledge you can start building and optimizing models and then gradually learn more complex concepts.</p><p><em>That&#39;s all I have to say in this article. Thank you readers for your precious time!</em></p><p><em>Don’t forget to leave a clap </em>👏, <em>if this article has helped you in any way. This keeps me motivated to write more articles and try my best to contribute towards the growth of our data science community :)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*5tJ4MgkPOWI1kLb2C6IMdg.jpeg" /><figcaption>Source: <a href="https://memegenerator.net/"><strong>Meme Generator</strong></a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8c3e951d91d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>