<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venkatesh Muley on Medium]]></title>
        <description><![CDATA[Stories by Venkatesh Muley on Medium]]></description>
        <link>https://medium.com/@venkateshmuley111?source=rss-59d4c0687309------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*m2YnU3mjzufEk5x9sRL8ug.png</url>
            <title>Stories by Venkatesh Muley on Medium</title>
            <link>https://medium.com/@venkateshmuley111?source=rss-59d4c0687309------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:19:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@venkateshmuley111/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking Efficiency: A Deep Dive into Databricks Workflow for Seamless Pipeline Management]]></title>
            <link>https://medium.com/@venkateshmuley111/unlocking-efficiency-a-deep-dive-into-databricks-workflow-for-seamless-pipeline-management-402131ab0119?source=rss-59d4c0687309------2</link>
            <guid isPermaLink="false">https://medium.com/p/402131ab0119</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks-basics]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-pipeline]]></category>
            <category><![CDATA[databricks-workflows]]></category>
            <dc:creator><![CDATA[Venkatesh Muley]]></dc:creator>
            <pubDate>Tue, 27 Feb 2024 11:05:26 GMT</pubDate>
            <atom:updated>2024-02-27T11:07:26.808Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N62xUK2dithDjyhiG7ajbA.jpeg" /><figcaption>Source : Unsplash.com</figcaption></figure><p>The Databricks platform is connected with a fully managed orchestration solution called Databricks workflow. <br>It primarily aids in the creation, management, and monitoring of dependable and maintainable pipelines.</p><p>Databricks Workflow Features:<br>* Databricks facilitates workflow automation, reducing the need for * manual intervention and ensuring efficient execution of tasks.<br>* Enables collaboration among team members through shared notebooks and integrated version control, promoting teamwork in building and refining workflows.<br>* Seamlessly integrates with machine learning frameworks, allowing the inclusion of advanced analytics and predictive modeling within the same environment.<br>* Efficiently manages compute clusters, optimizing resource utilization for various workloads and ensuring scalable performance.<br>* Provides robust monitoring and logging capabilities for tracking job executions, identifying bottlenecks, and troubleshooting issues during workflow development and execution.<br>* Integrates seamlessly with cloud services, facilitating easy storage, retrieval, and processing of data directly from cloud storage solutions.<br>* Supports both Python and Scala languages in notebooks, providing flexibility for data engineers and data scientists with diverse coding preferences.</p><p>Any databricks workflow requires 3 things to be configured. <br>1.Tasks<br>2.Cluster<br>3.Schedule</p><ol><li>Tasks <br>Tasks are basically a flow or the hierarchy of databricks notebook to be executed. <br>Tasks can be further classified in three types. <br> i. Sequence <br> ii. Funnel<br> iii. Fanout /Star pattern</li></ol><p>i. Sequence:</p><p>Data Stages: Involves sequential data stages like Bronze (raw data), Silver (cleaned and transformed data), and Gold (high-quality processed data).<br> Data Quality: Emphasizes ensuring data quality and integrity throughout the transformation process.</p><p>ii. Funnel:</p><p>Data Integration: Focuses on integrating data from diverse sources, enabling a unified view for analysis.<br> Consolidation: Involves consolidating and merging data streams from different origins for a holistic perspective.</p><p>iii. Fanout/Star Pattern:</p><p>Scalability: Ideal for scenarios requiring scalability with a single primary data source distributed to multiple downstream processes.<br> Parallel Processing: Allows parallel processing of data, improving efficiency in data distribution or ingestion.</p><p>2. Cluster:</p><p>A cluster is a computing power source. You can use an already-existing cluster or set up a job to execute on a shared cluster for all purposes. <br>To run your pipeline, you may also define a job cluster. <br>When a job cluster occurs, Databricks will start a new cluster, run the pipeline through it, and then shut it down. This procedure aids in reducing the actual cost.</p><p>3. Schedule:</p><p>Our pipeline can be set up to run on a regular basis. <br>The job is started automatically by Databricks at the scheduled runtime.<br>We have two options for doing this: we may start the pipeline manually or make it automated.</p><p>Finally, I would conclude by saying <br>The goal of databricks workflow is to define a pipeline in its entirety. <br>We must ask ourselves three questions while using workflow.<br>1. What are your goals? (Response: action to be taken)<br>2. How you wish to carry it out. (Response: set up the cluster appropriately)<br>3. At your convenience. (Scheduling pipeline run)</p><p>Stay tuned for more upcoming blogs. Happy exploring!”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=402131ab0119" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Outlier Insights: Unveiling the Hidden Patterns within Data]]></title>
            <link>https://medium.com/@venkateshmuley111/outlier-insights-unveiling-the-hidden-patterns-within-data-01c706d2984e?source=rss-59d4c0687309------2</link>
            <guid isPermaLink="false">https://medium.com/p/01c706d2984e</guid>
            <category><![CDATA[z-score]]></category>
            <category><![CDATA[box-plot]]></category>
            <category><![CDATA[outliers]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[interquartile-range]]></category>
            <dc:creator><![CDATA[Venkatesh Muley]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 07:57:54 GMT</pubDate>
            <atom:updated>2024-01-08T07:57:54.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jt29dUW_YXPsFKXwmuu5tA.jpeg" /></figure><p>To put it simply, an outlier is a data point which lies far away from the average value for a given collection of statistics.<br>Mathematically outlier can be defined as an observation that deviates more than two or three standard deviations or how widely distributed the data is around the average . <br>In certain use case outliers may be crucial, while in others, they may not.</p><p>One of the two reasons why outliers appear in a dataset is either due to data variability or incorrect data capture. <br>Additionally, if outliers are present in the dataset it may have an impact on the statistical analysis of the dataset, which may have an impact on the mean calculation.</p><p>Types of Outliers :<br>i.Global <br>ii.Contexual(conditional)<br>iii.Collective</p><p>i. Global Outlier:<br>If a data point’s value significantly deviates from the values of the entire dataset, it is said to be a global outlier.</p><p>ii. Contextual Outlier:<br>Contextual outliers are in a particular region of a dataset, they behave as outliers at times but, when the entire dataset is considered, this differentiation behavior becomes a pattern.</p><p>iii. Collective Outlier:<br>A collection of numbers in a group that substantially deviates from the dataset is known as a collective outlier.</p><p>There are several approaches to identify outliers.<br>1. Box plot <br>2. Scatter plot <br>3. Using Z score <br>4. Using Inter Quartile Range</p><p>1.Box Plot : <br>A box plot displays five-number summary of a set of data.<br>Boxplots are a visual way of displaying the distribution of data based on a five number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”).<br>This type of plot is used to easily detect outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*0HPz1JcPiwYdqu3ZYf5BeQ.jpeg" /><figcaption>image source: Geeks for Geeks</figcaption></figure><p>2. Scatter plot : <br>Every value in the dataset is represented as a dot in a scatter plot, which is a statistical visualization. <br>The dots’ placement along the vertical and horizontal axes will determine the value of that particular data point.<br>To show the values of the variables in a data set, scatter plots use Cartesian coordinates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*WSeNn9I5Oy18unNBU7efJg.jpeg" /><figcaption>image source : Adobe Stock</figcaption></figure><p>3. Using Z Score : <br>When implementing the Z score technique, we use the formula z=(x — mean)/standard deviation to determine the z score. <br>Additionally, we gave a threshold value. <br>This indicates that we won’t see a value of z as an outlier if it is below the threshold value.<br>We will classify it as an outlier if it deviates significantly from the standard deviation.</p><p>4. Using Inter Quartile Range :<br>Last but not the least how to use the Inter Quartile Range IQR method to obtain your outliers. <br>The data are first rearranged in ascending order.<br>Next, we determine percentile using the quartiles 0 (minimum), 25–50, 75–100, and 100 (maximum). <br>IQR can now be calculated using the formula 75% -25% percentile quartile. <br>Finally, using the distribution pattern as a reference, we multiply the first and third quartiles by 1.5 to determine the lower and upper bond values.<br>(The determined most appropriate value is 1.5).<br>Which essentially suggests that any result that falls outside of the lower or upper bound should be considered as an outlier.</p><p>As we draw to a close our investigation, keep in mind that outliers in the field of data analytics are more than just oddities.</p><p>Stay tuned for more upcoming blogs. Happy exploring!”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01c706d2984e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Exploratory Data Analysis: A Beginner’s Guide]]></title>
            <link>https://medium.com/@venkateshmuley111/mastering-exploratory-data-analysis-a-beginners-guide-28af740414a4?source=rss-59d4c0687309------2</link>
            <guid isPermaLink="false">https://medium.com/p/28af740414a4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[eda]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Venkatesh Muley]]></dc:creator>
            <pubDate>Sun, 10 Dec 2023 11:30:07 GMT</pubDate>
            <atom:updated>2023-12-10T11:32:21.851Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58tFUxpRA7tMUQOqNPfL3Q.jpeg" /><figcaption>You don’t know how it tastes until you taste it..</figcaption></figure><p>The technique of analyzing your data so that it summarizes itself and speaks for itself is known as exploratory data analysis.</p><p>For instance, if someone has to introduce the CEO of a large corporation to the audience at a function, they usually give them his name, occupation, and some other pertinent information.</p><p>In a nutshell, details that will enable anyone to strike up a conversation with him.</p><p>Any data science or data engineering project should begin with exploratory data analysis, or EDA for short. This process is also known as data discovery. methods for comprehending the data. <br>This typically aids in the comprehension of the data and the issues at hand by the developer, analyst, engineer, stakeholder, and end user.</p><p>Items or elements to take into account when carrying out EDA <br>1. When conducting exploratory data analysis (EDA), our first goal should be to understand how our data relates to the issue statement we are working on.</p><p>2. How much data you can access, or which data sources are available to you.</p><p>3. It will also be easier to obtain a suitable sample of data that will cover the majority of the data dependencies if one has some basic knowledge of the business or domain.</p><p>Generally speaking, you can process data using Python, Pandas, Numpy, and SQL. <br>However, you can utilize the seaborn and matplotlib libraries to get some images.</p><p>In general, Eda can assist you with processes including data manipulation and cleaning.<br>Examples include data summarization, handling missing values, handling outliers, standardizing the data, and determining whether filtering is necessary. <br>In our next blogs, we will go into more detail on each of these aspects, but for now, let’s just go over the eda overview.</p><p>In this data exploration procedure, you sample your favorite dish at a new restaurant and note how the food is presented, how it tastes, and how it smells. <br>and when you bite into it while keeping your eyes closed, you can taste the various flavors of the dish.</p><p>To put it simply, based on its explorations, eda primarily assists in the descriptive and diagnostic analysis of your data.</p><p>Just as tasting a new dish reveals its flavors, EDA reveals the richness of your data, making it an essential step in any data journey.</p><p>Stay tuned for detailed insights into data summarization, outlier handling, and more in upcoming blogs. Happy exploring!”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28af740414a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Synergy of Data Science and Data Engineering : Building Scalable Analytics]]></title>
            <link>https://medium.com/@venkateshmuley111/the-synergy-of-data-science-and-data-engineering-building-scalable-analytics-5b088a45f45b?source=rss-59d4c0687309------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b088a45f45b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Venkatesh Muley]]></dc:creator>
            <pubDate>Tue, 05 Dec 2023 08:52:58 GMT</pubDate>
            <atom:updated>2023-12-05T08:52:58.033Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Synergy of Data Science and Data Engineering : Building Scalable Analytics</h3><p>In the dynamic landscape of data, the collaboration between data science and data engineering plays a pivotal role in unlocking the true potential of analytics. In this blog post, we’ll explore how the synergy between these two domains can lead to scalable and efficient data-driven solutions.</p><p>1. Data Science and Data Engineering: A Symbiotic Relationship</p><p>Data science and data engineering are often seen as distinct entities, but in reality, they are deeply intertwined. Data engineers provide the foundation for data scientists to build robust models and derive meaningful insights. Successful data-driven projects require seamless collaboration between these two teams.</p><p>2. Building a Robust Data Pipeline</p><p>A well-architected data pipeline is the backbone of any successful analytics project. Data engineers design and implement pipelines that ensure the smooth flow of data from diverse sources to storage and processing layers. This infrastructure sets the stage for data scientists to explore, analyze, and extract valuable insights.</p><p>3. Ensuring Data Quality and Consistency</p><p>Data quality is paramount in the world of analytics. Data engineers play a crucial role in ensuring that the data ingested into the system is clean, consistent, and reliable. This enables data scientists to work with confidence, knowing that the data they analyze is accurate and representative.</p><p>4. Scalability Challenges and Solutions</p><p>As the volume of data continues to grow, scalability becomes a significant challenge. Data engineers need to design systems that can scale horizontally to handle increasing workloads. This scalability is vital for accommodating the resource-intensive tasks that data scientists often perform, such as training complex machine learning models.</p><p>5. Leveraging Cloud Technologies</p><p>Cloud platforms offer a wealth of tools and services that can streamline both data engineering and data science workflows. From scalable storage solutions to managed machine learning services, the cloud provides an ecosystem for building end-to-end data pipelines. Integrating these technologies allows for more agile and cost-effective analytics solutions.</p><p>6. Continuous Integration and Deployment for Data Science</p><p>Just as in software development, continuous integration and deployment (CI/CD) practices are gaining traction in the realm of data science. Data engineers and data scientists collaborate to implement CI/CD pipelines for deploying models into production seamlessly. This ensures that insights derived from data science are quickly translated into business value.</p><p>And the journey towards scalable analytics, the collaboration between data science and data engineering is not just a necessity but a strategic advantage. By recognizing the symbiotic relationship between these two domains, organizations can build resilient, scalable, and future-ready data ecosystems.</p><p>Stay tuned for more insights in upcoming posts!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b088a45f45b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>