<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shivudu Jangli on Medium]]></title>
        <description><![CDATA[Stories by Shivudu Jangli on Medium]]></description>
        <link>https://medium.com/@shivajangili125?source=rss-10ba5e33399b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Shivudu Jangli on Medium</title>
            <link>https://medium.com/@shivajangili125?source=rss-10ba5e33399b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:42:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shivajangili125/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Dataflow Gen2 vs PySpark vs SQL in Microsoft Fabric: Which One Should You Use?]]></title>
            <link>https://medium.com/@shivajangili125/dataflow-gen2-vs-pyspark-vs-sql-in-microsoft-fabric-which-one-should-you-use-db5152316c57?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/db5152316c57</guid>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 06:51:28 GMT</pubDate>
            <atom:updated>2026-08-31T06:51:28.815Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-N3Nc4x-Lj_yX-GyDpCmQ.png" /></figure><h3>Introduction</h3><p>While working with Microsoft Fabric, I noticed that there are often multiple ways to solve the same data transformation problem.</p><p>For example, a transformation can be built using Dataflow Gen2, a PySpark notebook, or SQL. All three can handle common tasks such as filtering data, joining tables, and performing aggregations, but the way they approach the problem is different.</p><p>This raises a practical question:</p><blockquote><strong><em>Which one should I use?</em></strong></blockquote><p>The answer isn’t simply based on how much data you’re processing. The complexity of the transformation, performance requirements, cost, maintainability, and the skills available within the team can all influence the decision.</p><p>In this article, I’ll compare Dataflow Gen2, PySpark, and SQL from a practical data-engineering perspective and look at where each approach fits best.</p><p>The goal isn’t to find a single winner. It’s to understand when one approach makes more sense than the others.</p><h3>Understanding the Three Options</h3><p>Before comparing them, let’s quickly understand what each option brings to a Fabric data pipeline.</p><h4>Dataflow Gen2</h4><p>Dataflow Gen2 provides a low-code, visual approach to data transformation. It uses Power Query, so tasks such as filtering, changing data types, removing duplicates, merging data, and creating calculated columns can be done through a visual interface.</p><p>It can be a good fit when the transformation is relatively straightforward and you want to avoid writing code for every step.</p><h4>PySpark</h4><p>PySpark takes a code-first approach to data transformation. It gives you access to Spark’s distributed processing capabilities along with Python.</p><p>This becomes useful when the transformation involves more complex logic, custom processing, large-scale data, or situations where you need more control over how the data is processed.</p><h4>SQL</h4><p>SQL is a natural choice when you’re working with structured, relational data.</p><p>Operations such as joins, filtering, aggregations, CTEs, and window functions can often be expressed clearly and concisely using SQL.</p><p>So, at a high level:</p><blockquote><strong><em>Dataflow Gen2</em></strong><em> → visual and low code<br> </em><strong><em>PySpark</em></strong><em> → flexible and code-driven<br> </em><strong><em>SQL</em></strong><em> → concise and relational</em></blockquote><p>None of these is automatically better than the others. The right choice depends on the type of transformation, workload, performance requirements, and how the solution will be maintained.</p><p>That’s what we’ll focus on in the rest of the article.</p><h3>What Actually Matters When Choosing?</h3><p>Now that we’ve looked at the three options, the next question is: what should we actually consider before choosing one?</p><p>There isn’t one factor that decides everything. A few things usually come into play.</p><h4>Transformation Complexity</h4><p>Start with the transformation itself.</p><p>If you’re doing simple operations such as filtering, removing duplicates, changing data types, or basic joins, a low-code approach like Dataflow Gen2 may be enough.</p><p>As the logic becomes more complex and requires custom processing, PySpark can give you more flexibility. For relational operations, SQL can often be the simplest option.</p><h4>Data Characteristics</h4><p>The type and structure of the data also matter.</p><p>Are you working with structured tables, semi-structured files, or a large volume of data? Are the datasets already stored in a format that works well with your chosen approach?</p><p>Understanding the data before choosing the tool can help avoid unnecessary complexity.</p><h4>Performance</h4><p>Performance becomes important when pipelines start handling larger workloads or need to meet specific processing times.</p><p>It’s difficult to say that one tool is always faster than another. The actual performance depends on the workload, transformations, joins, data layout, and how the solution is implemented.</p><h4>Cost and Capacity</h4><p>Cost is another consideration, especially in Fabric where workloads consume capacity resources.</p><p>Instead of assuming that one tool is always cheaper, it’s better to look at how much and how often the workload runs and what resources it consumes.</p><h4>Development Effort</h4><p>How quickly can you build and modify the solution?</p><p>For straightforward transformations, Dataflow Gen2 can reduce the amount of code you need to write. SQL can also be very concise for relational operations, while PySpark may require more development effort but provides greater flexibility.</p><h4>Maintainability</h4><p>A solution isn’t finished when it works for the first time.</p><p>Someone may need to modify, debug, or extend it later. A simple Dataflow might be easier for a team to understand, while well-structured SQL or PySpark code can be easier to version and reuse.</p><p>The key is to consider who will maintain the solution and how complex it is likely to become.</p><h4>Team Skills</h4><p>Finally, consider the people working on the project.</p><p>A team that is comfortable with SQL may naturally prefer SQL for relational transformations. A data engineering team with strong Python and Spark experience may get more value from PySpark. For teams that prefer a visual development experience, Dataflow Gen2 can be a better fit.</p><p>So, rather than starting with “Which tool is the best?”, I would start with:</p><blockquote><strong><em>What am I trying to do, what does the data look like, and what makes the most sense for the team that will maintain it?</em></strong></blockquote><p>That gives us a much better starting point for comparing the three approaches.</p><h3>Choosing Based on Real Scenarios</h3><p>The choice becomes easier when we look at the type of work we’re trying to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yHwnp83X9JJ_nR7J65Wkw.png" /></figure><p>These are not strict rules. A large dataset doesn’t automatically mean PySpark, and a simple transformation doesn’t always mean Dataflow Gen2. The right choice depends on the data, the transformation, the workload, and the people maintaining it.</p><h3>Can We Use Them Together?</h3><p>Yes. And this is probably how you’ll see them used in an actual project.</p><p>There’s no rule saying a pipeline has to use only Dataflow Gen2, only PySpark, or only SQL.</p><p>For example, you could use Dataflow Gen2 for some initial cleansing, move the data into a Lakehouse, use PySpark when the transformation gets more involved, and then use SQL for the final aggregations or analysis.</p><p>Something like:</p><p><strong>Source → Dataflow Gen2 → Lakehouse → PySpark → SQL</strong></p><p>But I wouldn’t add all three just for the sake of using them.</p><p>If a transformation can be handled cleanly with SQL, there’s no need to bring in PySpark. Similarly, if a simple transformation can be handled through Dataflow Gen2, writing a notebook for it might just add unnecessary complexity.</p><p>So for me, the point isn’t to choose one tool for the entire project.</p><p>It’s about looking at each part of the pipeline and deciding what fits there.</p><h3>Conclusion</h3><p>After looking at Dataflow Gen2, PySpark, and SQL, I don’t think there is one tool that can be called the best.</p><p>For simple transformations, Dataflow Gen2 can save development effort. For complex processing and custom logic, PySpark gives you more flexibility. And when the problem is mainly relational, SQL can often be the simplest option.</p><p>The important thing is not to choose a tool just because it is popular or because it can handle more data. Look at the actual requirement, the workload, the performance you need, and who will maintain the solution.</p><p>And sometimes, the answer isn’t one tool.</p><p>You can use Dataflow Gen2, PySpark, and SQL together when different parts of the pipeline have different requirements.</p><p><strong><em>Don’t choose the most powerful tool. Choose the tool that makes the problem simpler.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db5152316c57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with dbt in Microsoft Fabric]]></title>
            <link>https://medium.com/@shivajangili125/getting-started-with-dbt-in-microsoft-fabric-b52702fbb1b2?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b52702fbb1b2</guid>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 11:46:30 GMT</pubDate>
            <atom:updated>2026-08-06T11:46:30.093Z</atom:updated>
            <content:encoded><![CDATA[<h4>A hands-on guide to building, testing, and managing SQL transformations using dbt in Microsoft Fabric.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ldjfbGJEoZIOxuWJwiOmQ.png" /></figure><p><strong>INTRODUCTION</strong></p><p>Data engineers often spend a significant amount of time writing SQL transformations to prepare data for reporting and analytics. As projects grow, these SQL scripts become difficult to maintain, test, document, and deploy. Managing dependencies between transformations and ensuring consistent data quality also becomes increasingly challenging.</p><p>To address these challenges, Microsoft Fabric now provides native support for dbt (Data Build Tool) through dbt Jobs. This integration enables developers to build modular SQL transformations, validate data quality, generate documentation, and manage analytics projects using software engineering best practices , all within the Microsoft Fabric environment.</p><p>Unlike traditional ETL tools that perform extraction, transformation, and loading,<strong> </strong>dbt focuses only on the transformation layer. It works with data that already exists in a Fabric Warehouse, allowing developers to transform raw data into analytics-ready models using SQL.</p><p>In this article, we’ll explore how to create a dbt project in Microsoft Fabric, configure it with a Warehouse, understand the project structure, and work with models, seeds, tests, snapshots, and other advanced features.</p><p><strong>Creating a dbt Job in Microsoft Fabric</strong></p><p>Before you can start building data models with dbt, you need to create a dbt Job in Microsoft Fabric. A dbt Job provides the execution environment where your dbt project is configured, managed, and executed.</p><p>To create one, navigate to your Fabric workspace and select New item. From the available items, choose dbt Job (Preview). This creates a dedicated workspace for your dbt project, where you can develop SQL models, run tests, load seed data, capture snapshots, and execute other dbt commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*cEnaXNSupul_fchsMTr--Q.png" /></figure><h4>Configuring the dbt Job</h4><p>After creating the dbt Job, the next step is to configure it by selecting a Fabric Warehouse. The Warehouse acts as the execution target where dbt reads source tables and materializes models as tables or views.</p><p>Once the Warehouse is configured, the dbt project is ready for development. You can create models, organize project files, execute dbt commands, and manage your transformations directly within the Microsoft Fabric environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*S3L2t5Y5Iu7yYkDtmAG61A.png" /></figure><p><strong>Seeds</strong></p><p>In many data engineering projects, not all data comes from transactional systems. Some datasets are relatively small and change infrequently, such as country codes, product categories, currency mappings, status codes, or business lookup tables. Instead of manually creating and populating these tables in the Warehouse, dbt provides a feature called Seeds.</p><p>A seed is simply a CSV file stored in the project’s seeds folder. When you execute the dbt seed command, dbt reads each CSV file and automatically creates a corresponding table in the configured Microsoft Fabric Warehouse.</p><p>This approach eliminates manual data loading and ensures that reference data can be version-controlled alongside your SQL models.</p><p><strong>Models</strong></p><p>Models are the core building blocks of any dbt project. A model is simply a SQL file stored in the project’s models folder that defines how data should be transformed. Each model represents a specific business transformation, such as cleaning raw data, creating dimension tables, or building fact tables for reporting.</p><p>When you execute the dbt run command, dbt compiles these SQL files and materializes them as database objects in the configured Microsoft Fabric Warehouse. Depending on the materialization configuration, a model can be created as a table, view, incremental table, or ephemeral model.</p><p>By breaking complex transformations into smaller, modular SQL files, models make data pipelines easier to understand, maintain, and reuse. They also allow developers to build transformations in a structured way by referencing other models using the ref() function, which automatically manages dependencies and execution order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*GvK8-GE49AgNoFxNE1VtdA.png" /></figure><p>As a dbt project grows, organizing models into logical folders improves readability and maintainability. A common approach is to separate models into staging and marts layers.</p><p><strong>Tests</strong></p><p>Maintaining data quality is essential for building reliable analytics solutions. dbt provides a built-in testing framework that helps validate your data before it is consumed by reports or downstream processes. Tests are typically defined in a schema.yml file or as custom SQL tests within the project&#39;s tests folder.</p><p>When you execute the dbt test command, dbt runs all defined tests against the models in your Microsoft Fabric Warehouse and reports any failures. This allows you to identify issues such as duplicate records, missing values, invalid relationships, or unexpected data before they impact business users.</p><p>dbt supports several built-in generic tests, including:</p><ul><li><strong>not_null</strong> – Ensures a column does not contain null values.</li><li><strong>unique</strong> – Verifies that all values in a column are unique.</li><li><strong>relationships</strong> – Validates referential integrity between related tables.</li><li><strong>accepted_values</strong> – Confirms that a column contains only predefined values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*GwA8z22YgsGBRJH9b2Vd9g.png" /></figure><p><strong>Snapshots</strong></p><p>In many business scenarios, it’s important to track how data changes over time rather than simply storing the latest values. For example, a customer may change their address, phone number, or membership status. If the source table is updated directly, the previous information is lost.</p><p>To solve this problem, dbt provides Snapshots, which capture historical versions of records whenever changes occur. Instead of overwriting existing data, dbt preserves previous versions by maintaining validity periods for each record.</p><p>When you execute the dbt snapshot command, dbt compares the current data with the previously captured snapshot and records any detected changes. This enables historical analysis and supports Slowly Changing Dimension (SCD Type 2) implementations.</p><p>The snapshot table includes metadata columns such as:</p><ul><li><strong>dbt_valid_from</strong> – The timestamp when the record became valid.</li><li><strong>dbt_valid_to</strong> – The timestamp when the record was superseded. A NULL value indicates that the record is the current active version.</li><li><strong>dbt_scd_id</strong> – A unique identifier used internally by dbt to track different versions of the same record.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fOVCUMU9u1SKTScNWS2-9w.png" /></figure><p><strong>Macros</strong></p><p>As dbt projects grow, you may find yourself writing the same SQL logic repeatedly. Macros<strong> </strong>help eliminate this duplication by allowing you to define reusable SQL code using Jinja templating. Instead of copying the same logic across multiple models, you can create a macro once and call it wherever needed. This improves code consistency, reduces maintenance effort, and makes your transformations easier to manage.</p><p><strong>Analyses</strong></p><p>The analyses folder is intended for ad hoc or exploratory SQL queries that help you analyze and validate your data. Unlike models, queries stored in this folder are not materialized as tables or views in the Warehouse. They are useful for investigating data, testing business logic, or performing one-time analyses without affecting your production models.</p><p><strong>Assets</strong></p><p>The assets folder is used to store supporting files that complement your dbt project but are not executable SQL models. These may include images, diagrams, sample datasets, or other project resources. While this folder is optional, it helps keep related project assets organized and accessible alongside your dbt code.</p><p><strong>Schema.yml</strong></p><p>The schema.yml file is used to define metadata for dbt models. It allows you to add descriptions to models and columns while also configuring data quality tests. Instead of embedding these details within SQL files, dbt keeps them separate in a YAML file, making projects easier to document and maintain.</p><p><strong>dbt_project.yml</strong></p><p>The dbt_project.yml file is the primary configuration file for a dbt project. It defines project-wide settings and controls how dbt behaves during execution. Whenever a new dbt project is created, this file is automatically generated and serves as the central place for configuring the project.</p><p><strong>Best Practices</strong></p><ul><li>Organize models into staging and marts folders.</li><li>Use ref() instead of hardcoded table names.</li><li>Use snapshots only when historical tracking is required.</li></ul><p><strong>Conclusion</strong></p><p>If you’re working with Microsoft Fabric, learning dbt is a valuable step toward building cleaner, more maintainable data transformation pipelines. Instead of managing large SQL scripts, you can organize your transformations into reusable models, validate data quality with tests, load reference data using seeds, and preserve historical changes through snapshots.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b52702fbb1b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Retail Insights Through Microsoft Fabric]]></title>
            <link>https://medium.com/@shivajangili125/unlocking-retail-insights-through-microsoft-fabric-e4d794bf7551?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4d794bf7551</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 16:56:28 GMT</pubDate>
            <atom:updated>2026-07-14T16:56:28.673Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Retail businesses generate thousands of sales transactions every day, but those records have limited value until they are transformed into meaningful business insights. Decision-makers need more than raw data — they need clear answers to questions such as:</p><ul><li>Which product categories contribute the most to revenue?</li><li>Which stores or regions deliver the highest profitability?</li><li>How do sales and profit trends change over time?</li><li>What factors influence overall business performance?</li></ul><p>To explore these questions, I developed an end-to-end Retail Sales Analytics solution using Microsoft Fabric. The project brings together multiple Fabric capabilities into a unified analytics platform that automates the journey from raw data ingestion and transformation to data modeling and interactive business reporting.</p><p>The goal was not only to build dashboards but also to create a scalable and modern data engineering solution that converts transactional data into actionable insights for business users.</p><h3>Project Objective</h3><p>The goal of this project was to design and implement an automated analytics solution that:</p><ul><li>Ingests retail sales data</li><li>Cleans and transforms the data</li><li>Implements the Medallion Architecture</li><li>Builds a centralized analytical warehouse</li><li>Creates a Star Schema using dbt</li><li>Develops a Semantic Model</li><li>Delivers business insights through Power BI dashboards</li></ul><h3>Business Problem</h3><p>Retail businesses generate large volumes of sales data, but extracting meaningful insights from raw transactional data is often challenging. A centralized analytics solution is needed to monitor revenue, profitability, product performance, regional sales, and sales trends through interactive dashboards instead of manual spreadsheet analysis.</p><h3>Solution Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tHadd6oPhwIPGsyn0G0t7Q.png" /></figure><p>The solution follows the Medallion Architecture, where data flows through multiple layers before reaching business users.</p><p>The complete workflow is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8XvJp_mLVq0_1iFVfvVAQ.png" /></figure><p>Each component has a dedicated responsibility, making the solution modular, scalable, and easy to maintain.</p><p><strong>Step 1 — Metadata-Driven Data Pipeline</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfioqTGfh4awwWTeQYGNMQ.png" /></figure><p>The journey begins with raw CSV files.</p><p>Instead of hardcoding every dataset, I implemented a metadata-driven pipeline using:</p><ul><li>Lookup Activity</li><li>ForEach Activity</li><li>Copy Activity</li></ul><p>This makes the ingestion process reusable for multiple datasets with minimal maintenance.</p><p><strong>Step 2 — Bronze Layer</strong></p><p>The Bronze Layer stores raw data exactly as received.</p><p>No transformations are performed here.</p><p>Its purpose is to preserve source data for auditing and recovery.</p><p><strong>Step 3 — Silver Layer</strong></p><p>The Silver Layer performs data cleansing and transformation.</p><p>I used two different transformation approaches.</p><p><strong><em>Dataflow Gen2</em></strong></p><ul><li>Data type conversion</li><li>Data cleansing</li><li>Derived columns</li></ul><p><strong><em>PySpark Notebook</em></strong></p><ul><li>Duplicate removal</li><li>Data validation</li><li>Additional transformations</li></ul><p>Using both approaches helped me understand when low-code ETL is sufficient and when Spark offers greater flexibility.</p><p><strong>Step 4 — Warehouse</strong></p><p>After transformation, the cleaned data is loaded into the<strong> </strong>Microsoft Fabric Warehouse.</p><p>I automated warehouse loading using:</p><ul><li>CTAS</li><li>Stored Procedures</li></ul><p>This removed manual intervention and simplified data movement.</p><p><strong>Step 5 — Dimensional Modeling with dbt</strong></p><p>To optimize analytical queries, I implemented a Star Schema using dbt.</p><p>The model consists of:</p><p><strong><em>Dimension Tables</em></strong></p><ul><li>dim_customer</li><li>dim_product</li><li>dim_store</li><li>dim_date</li></ul><p><strong><em>Fact Table</em></strong></p><ul><li>fact_sales</li></ul><p>This design improves query performance and simplifies report development.</p><h3><strong>Semantic Model</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*lRfccVGKOsLWU6dt2TvFqQ.png" /></figure><p>After building the warehouse, I created a Semantic Model by defining relationships between the fact and dimension tables.</p><p>I also created business measures including:</p><ul><li>Total Revenue</li><li>Total Profit</li><li>Total Orders</li><li>Total Quantity</li><li>Average Order Value</li><li>Profit Margin %</li></ul><p>This layer acts as the foundation for Power BI reporting.</p><h3>Power BI Dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*M1TWnAG3QgMkfRsVI4H5Wg.png" /></figure><h3>Conclusion</h3><p>This project demonstrates how Microsoft Fabric can be used to build a complete analytics solution — from data ingestion and transformation to dimensional modeling and interactive reporting. By combining Data Factory, Lakehouse, Dataflow Gen2, PySpark, Warehouse, dbt, Semantic Models, and Power BI, I created a unified pipeline that transforms raw retail data into actionable business insights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4d794bf7551" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Fresher’s Guide to Agile and Scrum]]></title>
            <link>https://medium.com/@shivajangili125/a-freshers-guide-to-agile-and-scrum-0656dfb5bdc9?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0656dfb5bdc9</guid>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 14:42:13 GMT</pubDate>
            <atom:updated>2025-04-17T14:42:13.893Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong></p><p>As a fresher giving job interviews in the tech and business world, I kept hearing and reading about terms like “Agile” and “Scrum”. When I first saw them, I had no idea what they meant or why they were important. So I made them part of my skillset, and I’m glad I did.</p><p>In short, agile and scrum are mainstream and they are not fancy words professionals just use, it is the way of working that many companies have adopted to manage projects efficiently. This is based on a simplified version of my learning, so, in this blog, I am going to explain Agile and Scrum in the easiest form. This post is written for all of you who are starting from 0 as well!</p><p><strong>What is Agile?</strong></p><p>Agile is a flexible and collaborative way of working that helps teams deliver work in small steps, get quick feedback, and improve continuously.</p><p><strong>What is Agile Manifesto?</strong></p><p>Basically Agile manifesto is a document that have values and principles expressed in the form of Agile. It was created in early 2001. It just has 4 values, and 12 guiding principles. This manifesto makes it easier for a team to work, bringing more data and measurable structure, which all promote teamwork, iterative methods etc. This is a state of the art framework that is tailored towards making the development methodologies better.</p><p><strong>4 values</strong></p><p>Individuals and interactions over processes and tools</p><p>Working software over comprehensive documentation</p><p>Customer collaboration over contract negotiation</p><p>Responding to change over following a fixed plan</p><p><strong>12 principles</strong></p><ol><li>Customer satisfaction is the top priority</li></ol><p>2. Welcome changes — even late in the process</p><p>3. Deliver working software regularly</p><p>4. Business and developers must work together daily</p><p>5. Build projects around motivated people</p><p>6. Face-to-face communication is best</p><p>7. Working software is the main measure of progress</p><p>8. Maintain a sustainable pace</p><p>9. Continuous attention to quality and good design</p><p>10. Simplicity is key</p><p>11. Teams should self-organize</p><p>12. Reflect and improve regularly</p><p><strong>What is Scrum?</strong></p><p>Scrum is a lightweight framework that helps people, teams, and organizations generate value through adaptive solutions for complex problems</p><p><strong>Key roles in scrum</strong></p><p>Product owner</p><p>Scrum master</p><p>Development team</p><p><strong>Scrum Terms You Need to Know</strong></p><p><strong>Sprint:</strong></p><p>Sprint/iteration is a short, fixed period of time, typically anywhere from 1 to 4 weeks, in which the team works to complete a unique set of tasks and deliver a potentially usable increment of the product.</p><p>This involves planning, daily stand-ups, development, review and retrospective for ensuring continuous improvement and timely delivery of the product.</p><p><strong>Product Backlog</strong></p><p>A Product Backlog is a list of all the features, changes, and fixes that need to be done in a product. It is managed by the Product Owner and is constantly updated based on feedback and priorities.</p><p><strong>Sprint Backlog</strong></p><p>A Sprint Backlog is a list of tasks selected from the Product Backlog that the team plans to complete during the current sprint.</p><p><strong>Sprint Planning</strong></p><p>Sprint Planning is a meeting at the beginning of a sprint where the team selects which items from the Product Backlog they will work on and estimates the effort required — often using techniques like Planning Poker. The team discusses what can be done and how to do it.</p><p><strong>Daily Standup</strong></p><p>The Daily Standup is a short, daily meeting where the team quickly discusses what they worked on yesterday, what they will work on today, and any obstacles they’re facing. It helps ensure everyone is aligned and can address issues early.</p><p><strong>Sprint Demo</strong></p><p>A Sprint Demo is a meeting at the end of the sprint where the team shows the work they completed to stakeholders. It helps gather feedback and see if the product is moving in the right direction.</p><p><strong>Sprint Retrospective</strong></p><p>A Sprint Retrospective is a meeting held at the end of the sprint where the team reflects on what went well, what didn’t, and how they can improve in the next sprint.</p><p><strong>Velocity</strong></p><p>Velocity is the amount of work a team completes in a sprint, usually measured in story points. It helps in planning how much work the team can handle in future sprints.</p><p><strong>My Personal Takeaways</strong></p><p>It’s not just theory — it’s practical. At first, Agile and Scrum felt like buzzwords, but once I understood the roles, events, and purpose behind each step, it started to make real sense.</p><p>Teamwork and communication matter a lot.<strong> </strong>Even though I haven’t worked in a real team yet, I realized how important it is to stay connected, share updates, and support each other daily.</p><p>It’s okay to not get everything at once. Terms like backlog, sprint, and velocity were confusing at first — but once I connected them to real-life examples, they became easier to understand.</p><p><strong>Conclusion</strong></p><p>Learning Agile and Scrum as a fresher was one of the best decisions I made while preparing for interviews. These are not just tools used in IT companies — they’re ways to think, plan, and work better as a team. Understanding the basics has helped me see how projects are handled in the real world.</p><p>If you’re also starting your journey, don’t worry about mastering everything in one go. Take your time, focus on the core ideas, and try to relate them to simple examples. The more you explore, the clearer it gets. Keep learning — and you’ll be surprised how useful this knowledge becomes down the line!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0656dfb5bdc9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering for Freshers: What You Need to Learn.]]></title>
            <link>https://medium.com/@shivajangili125/data-engineering-for-freshers-what-you-need-to-learn-3f05fa427cc0?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f05fa427cc0</guid>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Sat, 15 Feb 2025 17:36:47 GMT</pubDate>
            <atom:updated>2025-02-15T17:36:47.002Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zdX_snISSlDc5a55hEygrA.jpeg" /></figure><p>With the increasing whirlwind of data around the world, a Data Engineer is now one of the most popular jobs around. Data engineers are professionals responsible for designing, building, and maintaining the systems and infrastructure that enable organizations to effectively process, store, and retrieve data. But if you are a fresher looking to get into this area, you must know these skills:</p><p>As a data engineer, you will be working a lot with databases. Understanding query optimizations, performance considerations, and how to work with large datasets is essential. Focus on:</p><p><strong>SQL</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ib0RR1BtOk96yZwXd3_hJQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rubaitulazad?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Rubaitul Azad</a> on <a href="https://unsplash.com/photos/text-CMFglgJg3d4?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Query complexity: JOINs, subqueries, CTEs</p><p>Database Normalization and Indexing</p><p>Interaction with relational database management systems: MySQL, PostgreSQL, SQL Server</p><p>MongoDB, Cassandra About NoSQL databases</p><p><strong>PROGRAMMING:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBK3top74w5boIFOeOOtNw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@hiteshchoudhary?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Hitesh Choudhary</a> on <a href="https://unsplash.com/photos/person-holding-sticky-note-D9Zow2REm8U?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Programming in data engineering is needed to process, change, and also automate information flows. Certain content might have been seen as being too easy due to its extensive accessibility. Learn the</p><p>Data manipulation with pandas</p><p>Writing ETL (Extract, Transform, Load) scripts</p><p>Handling APIs and automation</p><p>Scala or Java basics (good for working with Spark)</p><p><strong>Big Data Technologies</strong></p><p>You need to know big data tools if you work with large-scale data. Here are some of the critical technologies:</p><p>Apache Spark (distributed data processing)</p><p>Hadoop (HDFS, MapReduce, Hive)</p><p>Kafka (real time data streaming)</p><p><strong>ETL &amp; Data Pipelines</strong></p><p>Data engineers create and maintain the ETL(Extract-Transform-Load) that transfers data from one system to another. You should:</p><p>Know ETL workflows design</p><p>Get familiar with tools like Airflow, Apache NiFi, dbt</p><p>ETL services on cloud working (AWS Glue, Google Dataflow)</p><p><strong>CLOUD PLATFORMS (AWS, AZURE, GCP)</strong></p><p>Many enterprises are adopting cloud data solutions. Learning cloud platforms will add an extra edge. Focus on:</p><p>AWS (S3, Lambda, Redshift, Glue)</p><p>Azure (Data factory, Synapse, Blob Storage)</p><p>Google cloud (BigQuery, dataflow, cloud storage)</p><p><strong>Data Warehousing Concepts</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DKgrRJWKb4Pqp37Nf3lmag.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@zmachacek?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Zdeněk Macháček</a> on <a href="https://unsplash.com/photos/silver-and-diamond-studded-cross-pendant-rGzUMs-QsCM?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Data warehouse is used to hold structured data, which can be used for analysis. Understand:</p><p>Star and Snowflake schema</p><p>Fact and dimension tables</p><p>Indexing and partitioning of data</p><p><strong>Version Control &amp; CI/CD</strong></p><p>Now, cross-team work in data engineering also requires version control and automation. Learn:</p><p>Git and GitHub/GitLab for change tracking</p><p>Deployment with CI/CD tools such as Jenkins and GitHub Actions</p><p><strong>DevOps &amp; Containerization — The Basics</strong></p><p>Docker and Kubernetes: Knowledge of these two technologies is essential to manage scalable data pipelines in a cost-effective way. And, how to deploy data pipelines using CI/CD.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f05fa427cc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Tutorials: The importance of hands-On Learning]]></title>
            <link>https://medium.com/@shivajangili125/beyond-tutorials-the-importance-of-hands-on-learning-f12b1eeb94c6?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f12b1eeb94c6</guid>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 12:02:51 GMT</pubDate>
            <atom:updated>2024-08-03T12:02:51.750Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s digital world, we can be lifelong students. There are a huge number of videos, courses and tutorials available to us. I think we deceive ourselves into believing that just by watching or reading, we will become experts in a skill. While these resources help us understand concepts, they don’t provide practical knowledge.</p><p>A video or tutorial online can make us feel like we are mastering a topic. Let us take an example, you are trying to learn coding. You watch a couple of hours’ worth of tutorials on Python and, at the end, you feel confident. You watched the instructor writing code, explaining things and even solving some examples. It all makes sense.</p><p>However, when you go to write your python script from scratch ,you may face some difficulty. The concepts you thought you understood are not so easy. You realize that knowing is different from doing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*HqcgW4sgyGZ8aBQio-_z7A.jpeg" /></figure><p>This is why practice matters. Practice turns theoretical knowledge into practical skills.</p><p>For example, if you are learning to play the guitar. You might watch a ton of tutorials on how to play chords, strumming techniques and maybe even a few lessons on playing your favourite songs. You understand the theory, but when you pick up your guitar, your fingers won’t move as you expect them to.</p><p>The only way to get better is through practice. The more you practice the chords, the more your fingers will remember where to go. This will improve muscle memory and help you master the art of strumming in a more natural way. The same applies to coding or any other skill.</p><p>Let’s go back to our coding example. For instance, you want to learn how to write a simple web application. Watching hours of tutorials on HTML, CSS and JavaScript won’t help unless you start writing code on yourself.</p><p>Start small. Try to create a simple webpage with some text and images. Experiment with different styles in CSS and make it interactive with JavaScript. Make mistakes, debug them yourself, and try to learn from the errors. By doing this continuously, your knowledge and abilities grow.</p><p>Tutorials and videos are great starters. They provide understanding and foundational knowledge, but they do not replace practical implementation. Applying what you know and making errors will actually help your skills grow.</p><p>So, the next time you find yourself watching tutorials, just remember: real learning starts when you start to DO.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f12b1eeb94c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should care about Data Analytics(Even if you’re starting out)]]></title>
            <link>https://medium.com/@shivajangili125/why-you-should-care-about-data-analytics-even-if-youre-starting-out-b5398e8f65ff?source=rss-10ba5e33399b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5398e8f65ff</guid>
            <dc:creator><![CDATA[Shivudu Jangli]]></dc:creator>
            <pubDate>Sat, 27 Jul 2024 11:32:15 GMT</pubDate>
            <atom:updated>2024-07-27T11:32:15.311Z</atom:updated>
            <content:encoded><![CDATA[<p>Data is everywhere in today’s world. Every click and swipe generates data, leads to lots of information. If we don’t understand the data, what’s the use of it ? That’s where data analytics comes in.</p><p>For example, You own a small bakery and run a website for people to order from. You also use social media handles to promote your business. Eventually, you find that some products have more buyers than others but you don’t know why. This is a common problem faced by many businesses.</p><p>You can find patterns and trends from the data of your website and social media. E.g. You might notice that a product is most popular among 25–34 year olds. So, you can make more relevant ads targeting youth. Data analytics not only helps in understanding customers, but also improves efficiency and profits by reducing costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cTSJ2xnvEeUg4PXlMWkaFw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@dengxiangs?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Choong Deng Xiang</a> on <a href="https://unsplash.com/photos/graphical-user-interface--WXQm_NTK0U?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Data analytics plays a major role in many industries. It helps in predicting diseases by monitoring large amounts of data. It also improves operational efficiency of hospitals by maintaining sufficient amount of staff, beds etc. It specifies investment strategies, manage risks and detects fraud in finance. In sports, analysts use data to access player performance, strengths, weakness, strategies etc.</p><p>Data analytics turns unprocessed data into valuable knowledge. It not only allows you to understand your current situation, it also accurately predicts the future outcomes. The best part is, you don’t need to be a data scientist to get started .Ten years ago, data analytics was reserved for specialised experts but now user-friendly tools and resources enable everyone to harness its power.</p><p>Data analytics may be complex at first, but by building your skills, you can solve problems ,make decisions, and achieve goals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5398e8f65ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>