<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhaval Raval on Medium]]></title>
        <description><![CDATA[Stories by Dhaval Raval on Medium]]></description>
        <link>https://medium.com/@dhaval.sony.504?source=rss-7381659939bc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0cNYBREgSOLQ4zgpJXfd1Q.png</url>
            <title>Stories by Dhaval Raval on Medium</title>
            <link>https://medium.com/@dhaval.sony.504?source=rss-7381659939bc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:17:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhaval.sony.504/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring the Features and Components of Databricks]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@dhaval.sony.504/exploring-the-features-and-components-of-databricks-ab2d7647b143?source=rss-7381659939bc------2"><img src="https://cdn-images-1.medium.com/max/1024/1*cWHLTftGjG79mSpX-wAfrA.png" width="1024"></a></p><p class="medium-feed-snippet">In today&#x2019;s data-driven world, organizations are constantly looking for platforms that can handle big data, advanced analytics, and machine&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@dhaval.sony.504/exploring-the-features-and-components-of-databricks-ab2d7647b143?source=rss-7381659939bc------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@dhaval.sony.504/exploring-the-features-and-components-of-databricks-ab2d7647b143?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab2d7647b143</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Sun, 05 Oct 2025 06:33:33 GMT</pubDate>
            <atom:updated>2025-10-05T06:33:33.363Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Databricks: A Platform Where Your Data Meets Possibility]]></title>
            <link>https://medium.com/@dhaval.sony.504/databricks-a-platform-where-your-data-meets-possibility-0a3e401a76fe?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a3e401a76fe</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Sat, 31 May 2025 20:40:18 GMT</pubDate>
            <atom:updated>2025-05-31T20:44:00.166Z</atom:updated>
            <content:encoded><![CDATA[<p>Databricks is a cloud-based platform that take cares of all your data requirements. It is a open-source analytical platform for data manipulation and AI solutions at scale. <br>It is available on top of your existing cloud, whether that’s Amazon Web Services (AWS), Microsoft Azure, Google Cloud, or even a multi-cloud combination of those.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prKltRKTYKczzoh1aSPwHA.png" /><figcaption><em>image source:- </em><a href="https://docs.databricks.com/aws/en/assets/images/data_intelligence_engine-5293849e711ddffe8e6e86418e36ae31.png"><em>docs.databricks.com</em></a></figcaption></figure><h3>Databricks Uses:-</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*DV-CpnH_uYk6-VQ6Ckbr9g.png" /><figcaption>image source:- <a href="https://images.app.goo.gl/fBd5YLSKtzR7HtyX8">google.com</a></figcaption></figure><p>To understand the unique semantic of your data, databricks uses generative AI with the data lakehouse.</p><p><strong><em>With databricks you can:</em></strong></p><ul><li><em>Bring all your data into one place</em></li><li><em>Work with both batch and real-time data</em></li><li><em>Clean and organize your data</em></li><li><em>Run queries</em></li><li><em>Analyze data</em></li><li><em>Build machine learning models</em></li><li><em>Create reports and share insights</em></li></ul><h3><strong>Key Component s of Databricks Platform:-</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INHxty2rXeBYuf4sacWPMw.jpeg" /><figcaption>image source:- <a href="https://images.app.goo.gl/3R3rMegqxyAxFP4D6">google.com</a></figcaption></figure><p>Understanding the main building blocks of Databricks will help you make the most of its capabilities. Here&#39;s a quick overview of its core components:</p><ul><li><strong>Notebooks: </strong><em>Interactive web-based environments where you can write code in languages like Python, SQL, Scala, or R. Great for data exploration, visualizations, and collaborative development.</em></li><li><strong>Clusters: </strong><em>Scalable computing environments that run your code. You can create all-purpose clusters for development or job clusters for specific tasks.</em></li><li><strong>Jobs: </strong><em>Used to schedule and automate tasks such as ETL pipelines, model training, or running notebooks at set intervals.</em></li><li><strong>Delta Lake: </strong><em>A powerful storage layer that brings ACID transactions and schema enforcement to your data lakes. It ensures reliability and consistency for both batch and streaming data.</em></li><li><strong>Unity Catalog: </strong><em>A centralized governance layer that helps you manage data access, metadata, and permissions across your Databricks workspaces. It simplifies data security and compliance.</em></li><li><strong>Databricks SQL: </strong><em>A SQL-native interface for querying data and creating dashboards—perfect for analysts and business users.</em></li></ul><h3>What Makes Databricks Different?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kaVmqXit0BEBxmrw3VezNw.png" /><figcaption>image source:- <a href="https://images.app.goo.gl/ERAkww6gpfTJhT839">google.com</a></figcaption></figure><p>Databricks is not just a database or a data warehouse. It’s a unified data platform designed for modern data needs. Here’s how it stands apart:</p><ul><li><strong>More than just storage</strong>: <em>While databases and data warehouses focus mainly on storing and querying structured data, Databricks goes further by allowing you to process, transform, and analyze both structured and unstructured data.</em></li><li><strong>Handles all data types</strong>: <em>Databricks supports batch and streaming data, unlike traditional warehouses which are typically optimized for batch loads only.</em></li><li><strong>Built for big data and AI:</strong> <em>Databricks is built on Apache Spark, making it ideal for large-scale data processing and machine learning workflows, which go beyond what traditional databases are designed for.</em></li><li><strong>Unified platform</strong>: <em>Instead of needing separate tools for ETL, data science, analytics and reporting, Databricks provides a single workspace where all of this can happen collaboratively.</em></li><li><strong>Lakehouse architecture</strong>: <em>Databricks combines the reliability of data warehouses with the flexibility of data lakes, using its Delta Lake technology. This is known as the Lakehouse approach, which simplifies data architecture and reduces costs.</em></li></ul><h3>Things Can Be Done Using Databricks:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_aEiutOIWs7XKjZ1_OI0GA.png" /><figcaption>image source:- <a href="https://images.app.goo.gl/TxcZHHT3jRLuiTda7">google.com</a></figcaption></figure><ul><li><strong>Data Engineering: </strong><em>Build pipelines to move, clean, and transform data.</em></li><li><strong>Data Science &amp; ML: </strong><em>Train and deploy machine learning models at scale.</em></li><li><strong>Data Analytics: </strong><em>Explore and analyze data using SQL, Python, or other tools.</em></li><li><strong>Business Intelligence: </strong><em>Create dashboards and reports for decision-making.</em></li><li><strong>Real-Time Processing: </strong><em>Work with live streaming data for instant insights.</em></li><li><strong>Data Governance: </strong><em>Manage data access, security, and compliance.</em></li><li><strong>Lakehouse Architecture: </strong><em>Use a single platform for both analytics and AI, combining the best of data lakes and warehouses.</em></li></ul><h3>Conclusion:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>Databricks brings together data engineering, analytics, and AI in one powerful platform. Whether you’re building data pipelines, analyzing trends, or training machine learning models, Databricks provides the tools and scalability to get the job done efficiently. Its unified approach, combined with support for both real-time and batch data, makes it a go-to solution for modern data teams. As data continues to grow in volume and importance, platforms like Databricks will play a key role in turning raw data into real business value.</p><p>In the next blog, we’ll explore some features and components of Databricks in more detail.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a3e401a76fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the term Cloud Computing]]></title>
            <link>https://medium.com/@dhaval.sony.504/understanding-the-term-cloud-computing-0a6dc200c4bb?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a6dc200c4bb</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[understanding]]></category>
            <category><![CDATA[cloud-services]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Mon, 28 Oct 2024 16:49:31 GMT</pubDate>
            <atom:updated>2024-10-28T16:49:31.526Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Conepk49dSMyUHID" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We all have heard about the term Google Drive, and most of us have probably uses it in our day to day life. And because of this, somewhat we are familiar with the term cloud computing. We know the basics that our send our data to a virtual space which is easily accessible via internet rather than storing data on physical systems.</p><p>But let’s take a deeper look at what Cloud computing really is. <br>First, let us breakdown each term:</p><blockquote><strong>Cloud:-</strong> <em>When we say</em> <em>‘Cloud’, its often depicted as a servers, data centers and services </em><strong><em>accessible over the internet</em></strong><em>. It means it is </em><strong><em>stored or processed</em></strong><em> on </em><strong><em>remote servers</em></strong><em>, accessed via the internet rather than on local servers or personal computers. <br>In short, “The Cloud” refers to servers that are </em><strong><em>accessed over the internet</em></strong><em> and the software and databases that run on the servers.</em></blockquote><blockquote><strong>Computing:-</strong> <em>When we say ‘Computing’, it refers to the activities that uses computers to </em><strong><em>manage</em></strong><em>, </em><strong><em>process </em></strong><em>and </em><strong><em>store</em></strong><em> data. Computing can involve hardware and software. It encompasses a wide range of tasks from basic arithmetic calculations to complex data analysis and software development. <br>In short, “Computing” in Cloud Computing refers to the processing and manipulation of the data and the </em><strong><em>execution of the tasks performed</em></strong><em> on remote servers accessed via the internet.</em></blockquote><p>Let us see different types of well-known Cloud Computing definitions:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*fQGkzWTXGy-EEkH3xW8r5g.jpeg" /><figcaption>Image Source:- Google</figcaption></figure><p><strong>Amazon Web Services:- </strong>Cloud Computing is the on-demand delivery of IT resources over the internet with pay-as-you-go pricing. Instead of buying, owning and maintaining physical data centers and servers, you can access technology services such as power, storage and databases, on an as-needed basis from a cloud provider.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9h1J9SQ9LWueHkkGK5ZNg.jpeg" /></figure><p><strong>Microsoft Azure:-</strong> Simply put, cloud computing is the delivery of computing services — including servers, storage, databases, networking, software, analytics and intelligence — over the internet (“The Cloud”). You typically pay only for cloud services you use, helping you lower your operating costs, run your infrastructure more efficiently and scale as your business needs change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*Fj7hpSnRUOZLBztqQ44-KA.jpeg" /></figure><p><strong>IBM Cloud:-</strong> Cloud Computing is the on-demand access of computing resources — physical server or virtual servers, data storage, networking capabilities, application development tools, software, AI-powered analytics tools and more — over the internet with pay-per-use pricing. This model offers customers greater flexibility and scalability compared to traditional on premises infrastructure.</p><p><strong>Let’s understand some few Real-Life Examples for better understanding.</strong></p><ul><li>For storage, Google Drive, Dropbox, etc. store files online, accessible from anywhere.</li><li>For streaming, Netflix, Spotify, etc. stream movies and music directly from the cloud.</li><li>For gaming, services like Google Stadia stream games from cloud servers, so you don’t need high-end hardware.</li><li>For healthcare, Fitbit, Telemedicine, etc. platforms store health data in the cloud for easy access and monitoring.</li><li>For social media, Facebook, Instagram, etc. store and deliver user photos and videos using cloud servers.</li><li>For smart home, Alexa, Google Nest ,etc. use the cloud to process commands and provide responses.</li></ul><p>And so on…<br>These examples show how cloud computing makes our everyday apps and services faster, more accessible, and collaborative.</p><p><strong>Let’s see some Key Characteristics of Cloud Computing.</strong></p><ul><li>You don’t need any IT personnel for accessing and managing your resources, you can easily whenever you need by yourself.</li><li>Cloud services are accessible from anywhere on any internet-connected device, like laptops, tablets, and smartphones.</li><li>Cloud providers serve multiple customers using a shared pool of resources like servers, storage, etc, which helps optimize resource use and lowers costs. Users share the infrastructure but remain isolated from each other.</li><li>Cloud services can automatically scale resources up or down based on demand, so you’re never using more than necessary.</li><li>For instance, if an app gets a surge in users, the cloud can handle the load by increasing resources temporarily.</li><li>You’re billed only for what you use, similar to a utility bill. If you only need a small amount of storage or compute power, you’re only charged for that.</li></ul><p><strong>Let’s see some advantages of Cloud Computing.</strong></p><ul><li>No need for upfront investments in hardware or infrastructure. You pay only for what you use, saving costs on maintenance and upgrades.</li><li>Cloud resources can be scaled up or down easily based on demand, so businesses can handle varying workloads without overprovisioning.</li><li>Access your data and applications from anywhere with an internet connection, which supports remote work and global collaboration.</li><li>The cloud offers built-in backup and recovery options, ensuring business continuity in case of data loss or technical failures.</li><li>Many cloud providers have advanced security measures, such as encryption, firewalls, and compliance with industry standards, which helps protect data and applications.</li><li>Teams can collaborate in real-time, with easy access to shared files and applications, which improves productivity and project management.</li></ul><h3>Conclusion: -</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>Cloud Computing has transformed how businesses and individuals access, manage, and store information by offering a flexible solutions. For Businesses, Cloud Computing has become more essential for modern organization for innovation, agility, cost-savings, scalability, remote access, enhance security, business continuity and disaster recovery and so on. With benefits like on-demand resources, global accessibility, automatic updates, and enhanced security, the cloud has become an essential tool in today’s digital age.</p><p><strong><em>Let’s drive deep by understanding about types of cloud computing models and cloud service models in next blog.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a6dc200c4bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know T — distribution & Z — test]]></title>
            <link>https://medium.com/@dhaval.sony.504/know-t-distribution-z-test-765fb34a2ef3?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/765fb34a2ef3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[z-test]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[t-distribution]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Wed, 31 May 2023 14:06:18 GMT</pubDate>
            <atom:updated>2023-05-31T14:06:54.326Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Know T — Distribution and Z — Test</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*k19QblVo30Cb2Q_rOs0QAA.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>The Z distribution is a special case of the normal distribution with a mean of 0 and standard deviation of 1. The t-distribution is similar to the Z-distribution, but is sensitive to sample size and is used for small or moderate samples when the population standard deviation is unknown.</p><p><em>Let us discuss both one by one simultaneously.</em></p><h3><strong>T — distribution: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*XE6TOo8LbhpcAZ9sAtp4-g.png" /><figcaption>Image source: — Google Image</figcaption></figure><p><strong>The t-distribution, also known as Student’s t-distribution, is a way of describing data that follow a bell curve when plotted on a graph, with the greatest number of observations close to the mean and fewer observations in the tails.</strong></p><p>It is a type of normal distribution used for smaller sample sizes, where the variance in the data is unknown.</p><h4><strong>T-distribution and T-score: -</strong></h4><p>A t-score is the number of standard deviation from the mean in a t-distribution. You can typically look up a t-score in at-table, or by using an online t-score calculator.</p><p>In statistics, t-scores are primarily used to find two things:</p><p><em>· The upper and lower bounds of a confidence interval when the data are approximately normally distributed.</em></p><p><em>· The p-value of the test statistic for t-tests and regression tests.</em></p><h4><strong>What is the difference between T— distribution and standard normal distribution?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IQrhIShccdp9L4b8" /><figcaption>Photo by <a href="https://unsplash.com/@possessedphotography?utm_source=medium&amp;utm_medium=referral">Possessed Photography</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The t-distribution gives more probability to observations in the tails of the distribution than the standard normal distribution (a.k.a. the z-distribution).</p><p>In this way, the t-distribution is more conservative than the standard normal distribution: to reach the same level of confidence or statistical significance, you will need to include a wider range of the data.</p><h4><strong>Properties of the T-distribution: -</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUS2xsib6lnak_xgZg1_mA.jpeg" /><figcaption>Image source: — Google Image</figcaption></figure><p><em>1. The variable in t-distribution ranges from -∞ to +∞ (-∞ &lt; t &lt; +∞).</em></p><p><em>2. t- distribution will be symmetric like the normal distribution if the power of t is even in the probability density function(pdf).</em></p><p><em>3. For large values of ν(i.e., increased sample size n); the t-distribution tends to a standard normal distribution. This implies that for different ν values, the shape of t-distribution also differs.</em></p><p><em>4. The t-distribution is less peaked than the normal distribution at the centre and higher peaked in the tails. From the above diagram, one can observe that the red and green curves are less peaked at the centre but higher peaked at the tails than the blue curve.</em></p><p><em>5. The value of y(peak height) attains highest at μ = 0 as one can observe the same in the above diagram.</em></p><p><em>6. The mean of the distribution is equal to 0 for ν &gt; 1 where ν = degrees of freedom, otherwise undefined.</em></p><p><em>7. The median and mode of the distribution is equal to 0.</em></p><h3><strong>Z — test: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*cGgzAl9sc6Rc4-vliUo8ng.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>A z-test is a statistical test used to determine whether two population means are different when the variances are known and the sample size is large. A z-test is a hypothesis test in which the z-statistic follows a normal distribution.</p><p>A z-statistic, or z-score, is a number representing the result from the z-test. Z-tests are closely related to t-tests, but t-tests are best performed when an experiment has a small sample size.</p><p><em>Z-tests assume the standard deviation is known, while t-tests assume it is unknown.</em></p><blockquote><strong><em>Note: —</em></strong><em> The z-test is best used for greater-than or equal to 30 samples (n ≥ 30)because, under the central limit theorem, as the number of samples gets larger, the samples are considered to be approximately normally distributed.</em></blockquote><h4><strong>When should you use a Z-test?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mHsu9xd2dnpQlFHr" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If the standard deviation of the population is unknown and the sample size is greater than or equal to 30, then the assumption of the sample variance equaling the population variance should be made using the z-test.</p><p>Regardless of the sample size, if the population standard deviation for a variable remains unknown, a t-test should be used instead.</p><h4><strong>What is a Z-score?</strong></h4><p>A z-score, or z-statistic, is a number representing how many standard deviations above or below the mean population the score derived from a z-test is.</p><p>Essentially, it is a numerical measurement that describes a value’s relationship to the mean of a group of values.</p><p>If a z-score is 0, it indicates that the data point’s score is identical to the mean score. A z-score of 1.0 would indicate a value that is one standard deviation from the mean.</p><blockquote><strong><em>Note: —</em></strong> <em>Z-scores may be positive or negative, with a positive value indicating the score is above the mean and a negative score indicating it is below the mean.</em></blockquote><h4><strong>What’s the difference between a T-test and Z-test?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGq53MQgQ1-CxudiTAproA.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>Z-tests are closely related to t-tests, but t-tests are best performed when the data consists of a small sample size, i.e., less than 30.</p><p>Also, t-tests assume the standard deviation is unknown, while z-tests assume it is known.</p><h3><strong>Conclusion: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>In conclusion, the t-distribution and the z-test are important statistical concepts used for hypothesis testing and confidence interval estimation.</p><p>Understanding the t-distribution and z-test enables researchers and analysts to appropriately select and apply the correct statistical methods for hypothesis testing and confidence interval estimation. It allows for accurate inference and decision-making based on the nature of the data and the available sample size and information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=765fb34a2ef3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Basic Explanation Of p-values]]></title>
            <link>https://medium.com/@dhaval.sony.504/basic-explanation-of-p-values-a52c9398a04f?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a52c9398a04f</guid>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[p-value]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Tue, 30 May 2023 18:40:24 GMT</pubDate>
            <atom:updated>2023-05-31T14:09:08.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q7IHNtQ1ZWGRoGCrE5kZ7Q.jpeg" /><figcaption>Image source: — Google Image</figcaption></figure><p><strong>The p-value is a number, calculated from a statistical test, that describes how likely you are to have found a particular set of observations if the null hypothesis were true.</strong></p><p>p-values are used in hypothesis testing to help decide whether to reject the null hypothesis. <strong>The small the p-values, the more likely you are to reject the null hypothesis.</strong></p><blockquote><strong>Note: —</strong><em> </em>The null hypothesis is that there is no relationship between your variables of interest or that there is no difference among groups.</blockquote><h3><strong>Calculate the p-values: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hi0JfTYmQoNdVdht" /><figcaption>Photo by <a href="https://unsplash.com/@dancristianpaduret?utm_source=medium&amp;utm_medium=referral">Dan Cristian Pădureț</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>P values are usually automatically calculated by your statistical program (R, SPSS, etc.).</p><p>You can also find tables for estimating the p value of your test statistic online. These tables show, based on the test statistic and degree of freedom (number of observations minus number of independent variables) of your test, how frequently you would expect to see that test statistic under the null hypothesis.</p><p><strong>The calculation of the p value depends on the statistical test you are using to test your hypothesis:</strong></p><p><em>· Different statistical tests have different assumptions and generate different test statistics. You should choose the statistical test that best fits your data and matches the effect or relationship you want to test.</em></p><p><em>· The number of independent variables you include in your test changes how large or small the test statistic needs to be to generate the same p value.</em></p><h3><strong>Does a p-value tell you whether your alternative hypothesis is true?</strong></h3><p><em>‘No’.</em> The p-value only tells you how likely the data you have observed is to have occurred under the null hypothesis.</p><p>If the p-value is below your threshold of significance (typically p &lt; 0.05), then you can reject the null hypothesis, but this does not necessarily mean that your alternative hypothesis is true.</p><p><em>Below you might understand clearly what it is trying to say:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DXTZPJv0zy1w1w2S3KLJWw.png" /><figcaption>Image source: — Google Image</figcaption></figure><h3><strong>Conclusion: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>The concept of p-values is widely used in statistical hypothesis testing to determine the significance of an observed effect. After conducting a hypothesis test, the p-value represents the probability of obtaining results as extreme as the observed data, assuming the null hypothesis is true. The p-value is then compared to a predetermined significance level (often denoted as α) to make a decision about the null hypothesis.</p><p>In conclusion, p-values are a statistical tool that can help in assessing the strength of evidence against the null hypothesis.</p><p>However, they should be used and interpreted cautiously, considering other factors and in conjunction with effect sizes, confidence intervals, study design, and domain knowledge to draw meaningful conclusions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a52c9398a04f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Short About Hypothesis Testing]]></title>
            <link>https://medium.com/@dhaval.sony.504/short-about-hypothesis-testing-3facbb607cae?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/3facbb607cae</guid>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Tue, 30 May 2023 17:30:19 GMT</pubDate>
            <atom:updated>2023-05-31T14:11:41.954Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*-aFy-Yuq4K_uFn5pnddBHA.jpeg" /><figcaption>Image source: — Google Image</figcaption></figure><p>Hypothesis Testing is a type of statistical analysis in which you put your assumptions about a population parameter to the test. It is used to estimate the relationship between 2 statistical variables.</p><p>e.g., There will be profits or there will be no profits at all.</p><h3><strong>Null Hypothesis &amp; Alternative Hypothesis: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u5iayU005zGUJ0Zc" /><figcaption>Photo by <a href="https://unsplash.com/@bermixstudio?utm_source=medium&amp;utm_medium=referral">Bermix Studio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>· <strong><em>Null Hypothesis:</em></strong> The Null Hypothesis is the assumption that the event will not occur. A null hypothesis has no bearing on the study’s outcome unless it is rejected.</p><blockquote><em>H0 is the symbol for it, and it is pronounced H-naught.</em></blockquote><p>· <strong><em>Alternative Hypothesis: </em></strong>The Alternate Hypothesis is the logical opposite of the null hypothesis. The acceptance of the alternative hypothesis follows the rejection of the null hypothesis.</p><blockquote><em>H1 is the symbol for it.</em></blockquote><h4><strong><em>Let’s take an example;</em></strong></h4><p>A sanitizer manufacturer claims that its product kills 95 percent of germs on average.</p><p><em>Solution: </em>— To put this company’s claim to the test, create a null and alternate hypothesis.</p><p><em>H0 (Null Hypothesis): Average = 95%.</em></p><p><em>Alternative Hypothesis (H1): The average is less than 95%.</em></p><h3><strong>Procedure of Hypothesis Testing: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8UiXYmoXiH74YwjW" /><figcaption>Photo by <a href="https://unsplash.com/pt-br/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>1. The steps for hypothesis tests are as follows:</em></p><p><em>2. Define null and alternative hypothesis inside your data.</em></p><p><em>3. Identify the test statistic to be used for testing the validity of the null hypothesis for example, z — test or t — test.</em></p><p><em>4. Decide the significance value (α). Typical value used for is 0.05 or 5% or 5% of error.</em></p><p><em>5. Calculate the p value (probability value), which is the conditional probability of observing the test statistic value when the null hypothesis is true. We will use the functions provide in scipy.stats module for calculating the p value.</em></p><p><em>6. Take the decision to reject or retain the null hypothesis based on the p value and the significance value.</em></p><p><em>7. Present your result.</em></p><blockquote><strong><em>Note: — Calculate the significance probability p values from test statistics.</em></strong></blockquote><blockquote><strong><em>So,</em></strong></blockquote><blockquote><strong><em>Determine the acceptance of hypothesis using p values</em></strong></blockquote><blockquote><strong><em>· If p is less than α, reject Ho and accept H1.</em></strong></blockquote><blockquote><strong><em>· If p is greater than α, accept Ho and reject H1.</em></strong></blockquote><h3><strong>To Frame Hypothesis: -</strong></h3><p>Hypothesis is a starting position that is open to a test and rejection in light of strong adverse evidence.</p><p><strong>· The initial belief is called the null hypothesis(Ho).</strong></p><p><em>- Action taken: Do nothing</em></p><p><strong>· Its negation is called the alternative hypothesis(H1).</strong></p><p><em>- Often a claim to be tested, or a change to be detected</em></p><p><em>- Action taken: Do something</em></p><p><strong>· Two hypothesis are</strong></p><ul><li><em>Mutually exclusive</em></li><li><em>Collectively exhaustive</em></li></ul><blockquote><strong><em>Note: — If σ is known then go for z — score. (n &gt; 30)</em></strong></blockquote><blockquote><strong><em>If σ is unknown then go for t — score. (n &lt; 30)</em></strong></blockquote><h3><strong>Simple and Composite Hypothesis Testing: -</strong></h3><p>Depending on the population distribution, you can classify the statistical hypothesis into two types.</p><p><em>· </em><strong><em>Simple Hypothesis:</em></strong><em> A simple hypothesis specifies an exact value for the parameter.</em></p><p><em>· </em><strong><em>Composite Hypothesis:</em></strong><em> A composite hypothesis specifies a range of values.</em></p><h4><strong><em>Example: -</em></strong></h4><p>A company is claiming that their average sales for this quarter are 1000 units. This is an example of a simple hypothesis.</p><p>Suppose the company claims that the sales are in the range of 900 to 1000 units. Then this is a case of a composite hypothesis.</p><h3><strong>Type — I and Type — II Errors: -</strong></h3><p><em>· </em><strong><em>Type — I error:</em></strong><em> It is when you wrongly reject a true null hypothesis. (Punished an innocent guy)</em></p><p><em>· </em><strong><em>Type — II error:</em></strong><em> It is when you wrongly accept a false null hypothesis. (Let guilty go free)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*ARZwDOWqmT-G4C1N3nGkTQ.png" /><figcaption>Image source: — Google Image</figcaption></figure><ul><li><em>Probability of committing a Type — I error is the same as p value.</em></li><li><em>α — value can be interpreted as the acceptable probability of making a Type — I error (also called significance level).</em></li></ul><h3><strong>Hypothesis Testing Steps: -</strong></h3><p><strong><em>Null &amp; Alternative</em></strong></p><p><strong><em>|</em></strong></p><p><strong><em>Select the test (z or t)</em></strong></p><p><strong><em>|</em></strong></p><p><strong><em>Test statistics</em></strong></p><p><strong><em>|</em></strong></p><p><strong><em>p-value &lt;-&gt; α (allowable error)</em></strong></p><p><strong><em>|</em></strong></p><p><strong><em>p-values &gt; α values — — — If p high Ho fly -&gt; fail to hypothesis</em></strong></p><p><strong><em>p-values &lt; α values — — — If p low Ho go -&gt; reject Ho</em></strong></p><p><strong><em>and,</em></strong></p><p><strong><em>One tail -&gt; A &lt; B</em></strong></p><p><strong><em>A &gt; B</em></strong></p><p><strong><em>Two Tail -&gt; A = B (P = P1 + p2) or (2xP)</em></strong></p><p><strong><em>A ≠B</em></strong></p><h3><strong>Conclusion: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>In conclusion, hypothesis testing is a fundamental statistical technique used to make decisions or draw conclusions about a population based on sample data.</p><p>Understanding hypothesis testing enables researchers and analysts to evaluate the strength of evidence and make informed decisions based on statistical analysis. It helps in drawing conclusions about populations, comparing groups, and determining the presence of relationships or effects in data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3facbb607cae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Basics Of Central Limit Theorem]]></title>
            <link>https://medium.com/@dhaval.sony.504/basics-of-central-limit-theorem-f7c4d05ac6ac?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7c4d05ac6ac</guid>
            <category><![CDATA[central-limit-theorem]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Mon, 29 May 2023 20:28:41 GMT</pubDate>
            <atom:updated>2023-05-31T14:14:41.058Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZX0dU0znCt2hbaHaRq_EqA.jpeg" /><figcaption>Image source: — Google Image</figcaption></figure><p><strong>The central limit theorem states that if you take sufficiently large samples from a population, the samples’ means will be normally distributed, even if the population isn’t normally distributed.</strong></p><p>The central limit theorem relies on the concept of a sampling distribution, which is the probability distribution of a statistic for a large number of samples taken from a population.</p><p><strong>Imagining an experiment may help you to understand sampling distributions:</strong></p><p><em>· Suppose that you draw a random sample from a population and calculate a statistic for the sample, such as the mean.</em></p><p><em>· Now you draw another random sample of the same size, and again calculate the mean.</em></p><p><em>· You repeat this process many times, and end up with a large number of means, one for each sample.</em></p><p>The distribution of the sample means is an example of a sampling distribution.</p><p>The central limit theorem says that the sampling distribution of the mean will always be normally distributed, as long as the sample size is large enough. Regardless of whether the population has a normal, Poisson, binomial, or any other distribution, the sampling distribution of the mean will be normal.</p><h3><strong>Central Limit Theorem formula: -</strong></h3><p>We can describe the sampling distribution of the mean using this notation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*XJpJeSgQGoRZRKebp0-unA.png" /></figure><h3><strong>Sample Size and the Central Limit Theorem: -</strong></h3><p>The sample size <em>(n)</em> is the number of observations drawn from the population for each sample. The sample size is the same for all samples.</p><p>The sample size affects the sampling distribution of the mean in two ways.</p><p><strong>1. Sample size and normality: -</strong></p><p>The larger the sample size, the more closely the sampling distribution will follow a normal distribution.</p><p>When the sample size is small, the sampling distribution of the mean is sometimes non-normal. That’s because the central limit theorem only holds true when the sample size is “sufficiently large.”</p><blockquote><strong><em>By convention, we consider a sample size of 30 to be “sufficiently large.”</em></strong></blockquote><p><em>· When n &lt; 30, the central limit theorem doesn’t apply. The sampling distribution will follow a similar distribution to the population. Therefore, the sampling distribution will only be normal if the population is normal.</em></p><p><em>· When n ≥ 30, the central limit theorem applies. The sampling distribution will approximately follow a normal distribution.</em></p><p><strong>2. Sample size and standard deviations: -</strong></p><p>The sample size affects the standard deviation of the sampling distribution. Standard deviation is a measure of the variability or spread of the distribution (i.e., how wide or narrow it is).</p><p><em>· When n is low, the standard deviation is high. There’s a lot of spread in the samples’ means because they aren’t precise estimates of the population’s mean.</em></p><p><em>· When n is high, the standard deviation is low. There’s not much spread in the samples’ means because they’re precise estimates of the population’s mean.</em></p><h3><strong>Conditions of the central limit theorem: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l0rl9b5JpDs9jGXb" /><figcaption>Photo by <a href="https://unsplash.com/@medicalertuk?utm_source=medium&amp;utm_medium=referral">MedicAlert UK</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The central limit theorem states that the sampling distribution of the mean will always follow a normal distribution under the following conditions:</p><p><em>1. The sample size is sufficiently large. This condition is usually met if the sample size is n ≥ 30.</em></p><p><em>2. The samples are independent and identically distributed (i.i.d.) random variables. This condition is usually met if the sampling is random.</em></p><p><em>3. The population’s distribution has finite variance. Central limit theorem doesn’t apply to distributions with infinite variance, such as the Cauchy distribution. Most distributions have finite variance.</em></p><h3><strong>Importance of the central limit theorem: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kFks_ZCictKrDVmZ" /><figcaption>Photo by <a href="https://unsplash.com/fr/@freegraphictoday?utm_source=medium&amp;utm_medium=referral">AbsolutVision</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><ul><li><em>The central limit theorem is useful because it allows one to assume that the sampling distribution of the mean will be normally — distributed in most cases. This allows easier statistical analysis and inference.</em></li><li><em>This theorem states that if you take a sufficiently large sample size from a population with a finite level of variance, the mean of all samples from that population will be roughly equal to the population mean.</em></li><li><em>Central limit theorem helps you balance the time and cost of collecting all the data you need to draw conclusions about the population.</em></li></ul><h3><strong>Example of Central Limit Theorem: -</strong></h3><p>Suppose that you’re interested in the age that people retire in the United States. The population is all retired Americans, and the distribution of the population might look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p_iqLcYGXd1-cdoH_vq-iQ.png" /><figcaption>Image source: — Google Image</figcaption></figure><p><em>Solution: </em>— Age at retirement follows a left-skewed distribution. Most people retire within about five years of the mean retirement age of 65 years. However, there’s a “long tail” of people who retire much younger, such as at 50 or even 40 years old. The population has a standard deviation of 6 years.</p><p>Imagine that you take a small sample of the population. You randomly select five retirees and ask them what age they retired.</p><p>Example: sample of n = 5</p><p><em>68, 73, 70, 62, 63</em></p><p>The mean of the sample is an estimate of the population mean. It might not be a very precise estimate, since the sample size is only 5.</p><p>So,</p><p><em>Mean = (68 + 73 + 70 + 62 + 63) / 5</em></p><p><em>Mean = 67.2 years.</em></p><p>Suppose that you repeat this procedure 10 times, taking samples of five retirees, and calculating the mean of each sample. This is a sampling distribution of the mean.</p><p>Means of small samples;</p><p><em>60.8, 57.8, 62.2, 68.6, 67.4, 67.8, 68.3, 65.6, 66.5, 62.1</em></p><p>If you repeat the procedure many more times, a histogram of the sample means will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*nfB8gNg7Qmh11Ip0yFexZA.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>Although this sampling distribution is more normally distributed than the population, it still has a bit of a left skew.</p><p>Notice also that the spread of the sampling distribution is less than the spread of the population.</p><p>This sampling distribution of the mean isn’t normally distributed because its sample size isn’t sufficiently large.</p><p>Now, imagine that you take a large sample of the population. You randomly select 50 retirees and ask them what age they retired.</p><p>So, sample of n = 50</p><p><em>73, 49, 62, …, 64, 68 (till 50 retirees randomly)</em></p><p>The mean of the sample is an estimate of the population mean. It’s a precise estimate, because the sample size is large.</p><p>Mean of a large sample,</p><p><em>Mean = 64.8 years.</em></p><p>Again, you can repeat this procedure many more times, taking samples of fifty retirees, and calculating the mean of each sample:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*hCJa3Bbix6vw3kGizvd0lQ.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>In the histogram, you can see that this sampling distribution is normally distributed, as predicted by the central limit theorem.</p><h3><strong>Conclusion: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>In conclusion, the Central Limit Theorem (CLT) is a fundamental concept in statistics that states that the sampling distribution of the mean of a large number of independent and identically distributed (i.i.d.) random variables approaches a normal distribution, regardless of the shape of the original population distribution.</p><p>Understanding the Central Limit Theorem helps statisticians and data analysts make accurate and reliable inferences about population parameters. It demonstrates the power of sampling and the convergence of sample means to a normal distribution, even when the population distribution may be non-normal.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7c4d05ac6ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know About Confidence Interval]]></title>
            <link>https://medium.com/@dhaval.sony.504/know-about-confidence-interval-1f66d9ef7d0f?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f66d9ef7d0f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[confidence-interval]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Mon, 29 May 2023 19:06:28 GMT</pubDate>
            <atom:updated>2023-05-31T14:15:44.713Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0O8aK-3wSWDGxxT01-VtA.jpeg" /><figcaption>Image source: — Google Image</figcaption></figure><p><strong>A confidence interval is a range of values we are fairly sure our true value lies in.</strong></p><p>A confidence interval is the mean of your estimate plus and minus the variation in that estimate. This is the range of values you expect your estimate to fall between if you redo your test, within a certain level of confidence.</p><p><em>‘Confidence’, in statistics, is another way to describe probability.</em></p><p><em>For example,</em> if you construct a confidence interval with a 95% confidence level, you are confident that 95 out of 100 times the estimate will fall between the upper and lower values specified by the confidence interval.</p><p>Your desired confidence level is usually one minus the alpha(α) value you used in your statistical test:</p><blockquote><strong>Confidence level = 1 — α</strong></blockquote><p>So, if you are an alpha value of p &lt; 0.05 for statistical significance, then your confidence level would be:</p><p><em>1 — 0.05 = 0.95 = 95%.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EhkQt2grNtZCOlaS" /><figcaption>Photo by <a href="https://unsplash.com/@srz?utm_source=medium&amp;utm_medium=referral">sydney Rae</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong><em>Let’s take an example;</em></strong></h4><p>What is the probability of tomorrow’s temperature being 420 exactly?</p><p><em>Ans:</em> Probability is ‘0’.</p><p>Because temperature is in continuous values. So we cannot predict tomorrow’s temperature as exactly 420.</p><p><em>You can say, it can be between 400 to 440.</em></p><p><strong>· Sometimes samples don’t give quite the right result.</strong></p><p>Point estimators(<em>sample mean</em>) are valuable, but they may give errors.</p><p>Because we are not dealing with the entire population, all we are doing is giving a best estimator.</p><p>If the sample we use is unbiased, then the estimate is likely to be close to the true value of the population.</p><p><strong>· Specify a ‘range’ instead of single point of estimate.</strong></p><h3><strong>Interval estimates of parameters: -</strong></h3><p>· Based on sample data(imaginary),</p><p>- The point estimate for mean balance = <em>$1990</em></p><p>- Can we trust this estimate? -&gt; <em>No</em></p><p>· What do you think will happen if we took another random sample of 140 alumni? -&gt; <em>May Differ</em></p><p>· Because of this uncertainty, we prefer to provide the estimate of an interval (range) and associate a level of confidence with it.</p><blockquote><strong><em>Interval estimate = Point Estimate ± Margin of Error</em></strong></blockquote><h3><strong>Confidence Interval for population mean: -</strong></h3><p>· Start by choosing a confidence level (1 — α) % (e.g., 99%, 95%, 90%).</p><p>· Then, the population mean will be within <strong>X̄ ± Z * σ/√n.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*Jp8R6UWM8SmIdbdXhzyYuw.png" /></figure><p>· Margin of errors depends on the underlying uncertainty confidence level and sample size.</p><h4><em>Let’s take an example,</em></h4><p>X̄ = 1990, σ = 2500, n = 140, Z = 1.96</p><p><em>Solution: — </em>σ1 = σ/√n = 2500 / √140 = 211.29</p><p>Therefore, 1990 ± 1.96 * 211.29</p><p>Therefore, Range lies between [1576, 2404]</p><p><strong><em>Note: —</em></strong> <em>In python, we write as;</em></p><p><em>From scipy import stats</em></p><p><em>stats.norm.ppf(.975) { .975 is just a random digit</em></p><h3><strong>Calculating a Confidence Interval: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dOKWKDzw7z6C_gF3" /><figcaption>Photo by <a href="https://unsplash.com/@homajob?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you want to calculate a confidence interval on your own, you need to know:</p><p><em>1. The point estimate you are constructing the confidence interval for</em></p><p><em>2. The critical values for the test statistic</em></p><p><em>3. The standard deviation of the sample</em></p><p><em>4. The sample sizes</em></p><p>Once you know each of these components, you can calculate the confidence interval for your estimate by plugging them into the confidence interval formula that corresponds to your data.</p><h3><strong>When to use Confidence Interval: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dMw_LNhlj1r_lsEP" /><figcaption>Photo by <a href="https://unsplash.com/pt-br/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You can calculate confidence intervals for many kinds of statistical estimates, including:</p><p><em>· Proportions</em></p><p><em>· Population means</em></p><p><em>· Differences between population means or proportions</em></p><p><em>· Estimates of variation among groups</em></p><p>These are all point estimates, and don’t give any information about the variation around the number. Confidence intervals are useful for communicating the variation around a point estimate.</p><h3><strong>Conclusion: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>In conclusion, a confidence interval is a statistical concept that provides a range of values within which a population parameter is likely to fall.</p><p>Understanding confidence intervals helps researchers and analysts communicate the precision and reliability of their estimates. It acknowledges the inherent uncertainty in statistical inference and provides a measure of the range within which the true population parameter is likely to fall.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f66d9ef7d0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything You Should Know About Kurtosis]]></title>
            <link>https://medium.com/@dhaval.sony.504/everything-you-should-know-about-kurtosis-15892f92f113?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/15892f92f113</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kurtosis]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Sat, 27 May 2023 20:40:47 GMT</pubDate>
            <atom:updated>2023-05-31T14:17:01.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*jAEu7SgPtUV4Be2pU_x9IA.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>Kurtosis is a measure of the tailedness of a distribution. Tailedness is how often outliers occur.</p><p>In simple term, a measure of peakedness of the distribution, i.e., kurtosis will measure peakedness inside your data.</p><p>· Distributions with medium kurtosis (medium tails) are mesokurtic.</p><p>· Distributions with low kurtosis (thin tails) are platykurtic.</p><p>· Distributions with high kurtosis (fat tails) are leptokurtic.</p><p>Tails are the tapering ends on either side of a distribution. They represent the probability or frequency of values that are extremely high or low compared to the mean. In other words, tails represent how often outliers occur.</p><h3><strong>Types of Kurtosis: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*ZO83sRtIAwFFcmCGEu-4YQ.png" /><figcaption>Image source: — Google Image</figcaption></figure><p><strong><em>1.Mesokurtic Distribution: — </em></strong>A mesokurtic distribution is medium-tailed, so outliers are neither highly frequent, nor highly infrequent. Kurtosis is measured in comparison to normal distributions. Normal distributions are mesokurtic.</p><p>· Normal distributions have a kurtosis of 3, so any distribution with a kurtosis of approximately 3 is mesokurtic.</p><p>· Normal distributions have an excess kurtosis of 0, so any distribution with an excess kurtosis of approximately 0 is mesokurtic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*sGZ6wOfwJLXqpH-sLAsa4A.png" /><figcaption>Image source: — Google Image</figcaption></figure><p><em>Mesokurtic distributions have outliers that are neither highly frequent, nor highly infrequent.</em></p><p><strong><em>2. Platykurtic Distribution: — </em></strong>A platykurtic distribution is thin-tailed, meaning that outliers are infrequent. Platykurtic distributions have less kurtosis than a normal distribution. In other words, platykurtic distributions have:</p><p>· A kurtosis of less than 3</p><p>· An excess kurtosis of less than 0</p><p>Platykurtosis is sometimes called negative kurtosis, since the excess kurtosis is negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*dQm5-h2kgPZlv1CnXO-4DQ.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>The frequency distribution (shown by the gray bars) doesn’t follow a normal distribution (shown by the dotted green curve). Instead, it approximately follows a uniform distribution (shown by the purple curve). Uniform distributions are platykurtic.</p><p><em>Platykurtic distributions have a low frequency of outliers.</em></p><p><strong><em>3. Leptokurtic Distribution: — </em></strong>A leptokurtic distribution is fat-tailed, meaning that there are a lot of outliers. Leptokurtic distributions are more kurtotic than a normal distribution. They have:</p><p>· A kurtosis of more than 3</p><p>· An excess kurtosis of more than 0</p><p>Leptokurtosis is sometimes called positive kurtosis, since the excess kurtosis is positive.</p><p><em>For example,</em> Imagine that four astronomers are all trying to measure the distance between the Earth and Nu2 Draconis A, a blue star that’s part of the Draco constellation. Each of the four astronomers measures the distance 100 times, and they put their data together in the same dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*DgBwZ91uPS_0sZtcuYkSgg.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>The frequency distribution (shown by the gray bars) doesn’t follow a normal distribution (shown by the dotted green curve). Instead, it approximately follows a Laplace distribution (shown by the blue curve). Laplace distributions are leptokurtic.</p><p>The astronomers calculate that the kurtosis of the sample is 6.54 and its excess kurtosis is 3.54. They conclude that the distribution is leptokurtic.</p><p>The distribution of the astronomers’ measurements has more outliers than you would expect if the distribution were normal, with several extreme observations that are less than 50 or more than 150 light-years.</p><p><em>Leptokurtic distributions have frequent outliers.</em></p><h3><strong>Calculate Kurtosis: -</strong></h3><p>Mathematically speaking, kurtosis is the standardized fourth moment of a distribution. Moments are a set of measurements that tell you about the shape of a distribution.</p><h4><strong><em>Kurtosis of a population: -</em></strong></h4><p>The following formula describes the kurtosis of a population: -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*T0HwUSVROMKmZOco_uTAzA.png" /></figure><h4><strong><em>Kurtosis of a sample: -</em></strong></h4><p>The kurtosis of a sample is an estimate of the kurtosis of the population.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*9vJ8Cq_d6b1S-c7w_DI-pA.png" /></figure><h3><strong>Conclusion: -</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>In conclusion, kurtosis is a statistical measure that describes the shape of a distribution by focusing on the tails and the peakedness or flatness of the curve.</p><p>Understanding kurtosis helps researchers gain a deeper understanding of the shape and behaviour of the data distribution. It provides valuable information about the tails and the peakedness or flatness of the curve, allowing for better decision-making and appropriate statistical analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15892f92f113" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything You Should Know About Skewness]]></title>
            <link>https://medium.com/@dhaval.sony.504/everything-you-should-know-about-skewness-c5edf381cc6d?source=rss-7381659939bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5edf381cc6d</guid>
            <category><![CDATA[skewness]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dhaval Raval]]></dc:creator>
            <pubDate>Sat, 27 May 2023 19:30:29 GMT</pubDate>
            <atom:updated>2023-05-31T14:23:54.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*bHglrUGg4CGLouOfFn9ZJw.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>Skewness is used to measure the level of asymmetry in our graph. It is the measure of asymmetry that occurs when our data deviates from the norm.</p><p>Sometimes, the normal distribution tends to tilt more on one side. This is because the probability of data being more or less than the mean is higher and hence makes the distribution asymmetrical.</p><p>This also means that the data is not equally distributed.</p><p><strong><em>Calculate the skewness of a distribution to: -</em></strong></p><p>· Describe the distribution of a variable alongside other descriptive statistics.</p><p>· Determine if a variable is normally distributed. A normal distribution has zero skew and is an assumption of many statistical procedures.</p><h3><strong>Types of Skewness: -</strong></h3><p>A distribution can have right (or positive), left (or negative), or zero skewness. A right-skewed distribution is longer on the right side of its peak, and a left-skewed distribution is longer on the left side of its peak.</p><p><strong><em>1. Zero Skew: —</em></strong> When a distribution has zero skew, it is symmetrical. Its left and right sides are mirror images. Normal distributions have zero skew, but they’re not the only distributions with zero skew. Any symmetrical distribution, such as a uniform distribution or some bimodal (two-peak) distributions, will also have zero skew.</p><p>The easiest way to check if a variable has a skewed distribution is to plot it in a histogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*oGOY5y0pUiFysa3FxjjTYQ.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>The distribution is approximately symmetrical, with the observations distributed similarly on the left and right sides of its peak. Therefore, the distribution has approximately zero skew.</p><p><em>Note: —</em> In a distribution with zero skew, the mean and median are equal.</p><blockquote><strong><em>Zero Skew : — mean = median</em></strong></blockquote><p><strong>2. <em>Right Skew(Positive Skew): </em>—</strong> A right-skewed distribution is longer on the right side of its peak than on its left. Right skew is also referred to as positive skew.</p><p>You can think of skewness in terms of tails. A tail is a long, tapering end of a distribution. It indicates that there are observations at one of the extreme ends of the distribution, but that they’re relatively infrequent. A right-skewed distribution has a long tail on its right side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*kjOGiyyi_jOF95D79K-rkA.png" /><figcaption>Image source: — Google Image</figcaption></figure><p>The distribution is right-skewed because it’s longer on the right side of its peak. There is a long tail on the right, meaning that every few decades there is a year when the number of x values observed is a lot higher than average.</p><p><em>Note: — </em>The mean of a right-skewed distribution is almost always greater than its median. That’s because extreme values (the values in the tail) affect the mean more than the median.</p><blockquote><strong><em>Right Skew : — mean &gt; median</em></strong></blockquote><p><strong><em>3. Left Skew(Negative Skew): </em>—</strong> A left-skewed distribution is longer on the left side of its peak than on its right. In other words, a left-skewed distribution has a long tail on its left side. Left skew is also referred to as negative skew.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*uYFxprLbq7bqP_1BoD_tOw.png" /><figcaption>Zoology test score (Image source: — Google Image)</figcaption></figure><p>The distribution is left-skewed because it’s longer on the left side of its peak. The long tail on its left represents the small proportion of students who received very low scores.</p><p><em>Note: —</em> The mean of a left skewed distribution is almost always less than its median.</p><blockquote><strong><em>Left Skew : — mean &lt; median</em></strong></blockquote><p><em>For example,</em> the mean zoology test score was 53.7, which is less than the median of 55.</p><h3><strong>Calculate Skewness: -</strong></h3><p>There are several formulas to measure skewness. One of the simplest is Pearson’s median skewness. It takes advantage of the fact that the mean and median are unequal in a skewed distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*xMmt94APxEbl5OmIhxaNsQ.png" /></figure><p>Pearson’s median skewness tells you how many standard deviation separates the mean and median.</p><p>Real observations rarely have a Pearson’s median skewness of exactly 0. If your data has a value close to 0, you can consider it to have zero skew.</p><p><em>For example,</em> Calculating Pearson’s median skewness</p><p>Pearson’s median skewness of the number of sunspots observed per year: Mean = 48.6, Median = 39, Standard Deviation = 39.5.</p><p><em>Solution: -</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*1KEPGsqO_EV78QXmiCPBrQ.png" /></figure><p><em>Note: —</em> If this value is between:</p><p>· -0.5 and 0.5, the distribution of the value is almost symmetrical</p><p>· -1 and -0.5, the data is negatively skewed, and if it is between 0.5 to 1, the data is positively skewed. The skewness is moderate.</p><p>· If the skewness is lower than -1 (negatively skewed) or greater than 1 (positively skewed), the data is highly skewed.</p><h3><strong>What to do if your data is skewed: -</strong></h3><p>You generally have three choices if your statistical procedure requires a normal distribution and your data is skewed:</p><p>1. Do nothing. Many statistical tests, including t tests, ANOVAs, and linear regressions, aren’t very sensitive to skewed data. Especially if the skew is mild or moderate, it may be best to ignore it.</p><p>2. Use a different model. You may want to choose a model that doesn’t assume a normal distribution. Non-Parameter tests or generalized linear models could be more appropriate for your data.</p><p>3. Transform the variable. Another option is to transform a skewed variable so that it’s less skewed. “Transform” means to apply the same function to all the observations of a variable.</p><h4><strong>Transformation based on the type of skew: -</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*fCNUJb_bciqSyE0a2ViFag.png" /><figcaption>Image source: — Google Image</figcaption></figure><h3>Conclusion: -</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*yHTux3GrfU2pmSu4-39ITQ.jpeg" /></figure><p>In conclusion, skewness is a statistical measure that helps describe the asymmetry of a distribution.</p><p>Understanding skewness allows researchers to better characterize and analyse data, helping them make informed decisions and choose appropriate statistical methods. It provides insights into the asymmetry of the data distribution, allowing for a more comprehensive understanding of the dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5edf381cc6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>