<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mayank Aggarwal on Medium]]></title>
        <description><![CDATA[Stories by Mayank Aggarwal on Medium]]></description>
        <link>https://medium.com/@tech.mayankagg?source=rss-528e973b8d36------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vtibWNciKSD90jt178kNdQ.jpeg</url>
            <title>Stories by Mayank Aggarwal on Medium</title>
            <link>https://medium.com/@tech.mayankagg?source=rss-528e973b8d36------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:08:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tech.mayankagg/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[2. Python : A Feature Rich Journey]]></title>
            <link>https://medium.com/@tech.mayankagg/2-python-a-feature-rich-journey-5bbc4ad0e155?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bbc4ad0e155</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Sat, 13 Jul 2024 16:12:30 GMT</pubDate>
            <atom:updated>2024-07-13T16:12:30.287Z</atom:updated>
            <content:encoded><![CDATA[<h3>2. Python : A Feature Rich Journey</h3><h3>Introduction</h3><p>Python’s popularity stems from its simplicity, versatility, and rich feature set.</p><blockquote>I personally have coded in more than 10+ languages but python still seems to be my first choice whenever I have to do something quickly, the ease of getting started as well as straight forwardness of the code make it the first choice to automate anything.<br>Once you learn it, you will surely appreciate its beauty and simplicity and be amazed by its versatility.</blockquote><p>Let’s embark on a journey to uncover the key features that make<strong> Python a favourite among developers worldwide.</strong></p><h3>1. Simple</h3><p>Imagine you’re learning a new language, and the syntax is so straightforward that you grasp it quickly.</p><p>Well, that’s Python for you. Its clean and readable syntax allows you to focus on solving problems rather than deciphering complex code structures. For instance, to print “Hello, World!” in Python, you only need one line of code.</p><blockquote><strong>Also for below code, you don’t need to define any class or function or anything and it is as straight forward as writing english or pseudo code.</strong></blockquote><pre>a = 5pytho<br>b = 6.5 <br><br>print(a+b)<br>print(a*b)</pre><h3>2. Dynamically Typed</h3><p>In Python, you don’t need to specify the data type of a variable. This dynamic typing feature allows you to change the type of a variable on the fly, making your code more flexible and concise:</p><pre>x = 10  # x is an integer<br>x = &quot;Hello&quot;  # now x is a string</pre><blockquote><strong>In other languages like C++ or Java, the same is not possible and you will have to always worry about losing data or getting error.</strong></blockquote><blockquote><strong>Thanks to above you can also catch all the results and catch data without worrying about its type</strong>.</blockquote><h3>3. Robust</h3><p>Python’s robustness comes from its strong error-handling capabilities and extensive standard library. It provides built-in modules and functions for various tasks, ensuring that your code is reliable and error-free.</p><p>The libraries you have in python are</p><h4>4. Multiple Programming Paradigms Supported</h4><p>Python supports multiple programming paradigms, including procedural, object-oriented, and functional programming. This flexibility allows you to choose the paradigm that best suits your project’s needs.</p><h4>5. Compiled and Interpreted</h4><p>Python is both compiled and interpreted. It first compiles your code into bytecode, which is then executed by the Python interpreter. This approach combines the speed of compiled languages with the flexibility of interpreted languages.</p><h4>6. Cross-Platform</h4><p>Python is cross-platform, meaning it can run on various operating systems. This portability allows you to write code once and run it on any platform, making Python highly versatile and accessible.</p><h4>7. Extensible</h4><p>Python’s extensibility allows you to integrate Python code with other languages, such as C or C++. This capability enables you to leverage existing libraries and tools, enhancing Python’s power and flexibility.</p><h4>8. Embedded</h4><p>Python can be embedded within other applications, adding scripting support and extending their functionality. This feature is particularly useful for applications that require customisation or automation.</p><h4>9. Extensive Library Support</h4><p>Python’s extensive standard library provides ready-to-use modules and functions for a wide range of tasks. Additionally, its rich ecosystem of third-party libraries and frameworks further extends its functionality, making Python a go-to choice for developers.</p><p>In conclusion, Python’s simplicity, versatility, and feature-rich nature make it an ideal choice for beginners and experienced developers alike. Whether you’re developing web applications, data analysis tools, or machine learning models, Python offers a robust platform to bring your ideas to life.</p><p>Stay tuned for more Python tutorials, where we will explore these features in depth and showcase their real-world applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bbc4ad0e155" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discovering Python: A Journey into the World of Programming]]></title>
            <link>https://medium.com/@tech.mayankagg/discovering-python-a-journey-into-the-world-of-programming-c9febcd90f35?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9febcd90f35</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Tue, 07 May 2024 15:18:14 GMT</pubDate>
            <atom:updated>2024-05-07T15:18:14.269Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome on starting your journey in Python and in the programming world.</p><p>The series of article I would write would cover a wide array of topics post which you should be good to take up a field from Data Science, ML or Artificial Intelligence to Development in Python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GfLdZpC7V5Owdf6j" /></figure><h4>Introduction</h4><p>Welcome to the world of Python! This versatile programming language has captured the hearts of developers around the globe. In this article, we will embark on a journey to explore Python’s history, applications, and how you can get started with your first Python program.</p><h4>A Brief History Python</h4><blockquote>Before learning any language I believe it is very important to learn about its origin and story as that help in understanding it and makes one eager to learn.</blockquote><p>Python was created by Guido van Rossum in the late 1980s and has since become one of the most popular programming languages in the world. The name “Python” was inspired by the British comedy series “Monty Python’s Flying Circus.” Today, Python is managed by the Python Software Foundation, which oversees its development and promotes its use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*qFO-fN-h6rUfw7Y6.jpg" /><figcaption>Guido Van Rossum: Benevolent Dictator for life</figcaption></figure><blockquote>Can you guess which from Python or Java is older ? <br>Find it out on web and be amazed ;)</blockquote><h4>The Versatility of Python</h4><blockquote>Python’s versatility is one of its key strengths. I have coded in 10+ languages but always find python best as it finds it application in every field from being and undisputed king in Data Science/AI to Development.</blockquote><p>It can be used to develop a wide range of applications, including:</p><ul><li><strong>Desktop Applications</strong>: Python’s simplicity and ease of use make it a great choice for developing desktop applications.</li><li><strong>Web Applications</strong>: Frameworks like Django and Flask make it easy to build powerful web applications with Python.</li><li><strong>Data Analysis</strong>: Python is widely used in data analysis and visualization, thanks to libraries like NumPy, Pandas, and Matplotlib.</li><li><strong>Machine Learning and Artifical Intelligence</strong>: Python has become the go-to language for machine learning and artificial intelligence, with libraries like TensorFlow and Scikit-learn leading the way.</li><li><strong>Raspberry Pi &amp; IoT</strong>: Python is the official programming language of the Raspberry Pi, making it ideal for IoT projects and education.</li><li><strong>Game Development</strong>: Python’s simplicity and readability make it a great choice for game development, with libraries like Pygame making it easy to get started.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*-gwrXNV6j-Kti5X7_C88FQ.png" /><figcaption>Python Application and famous Libraries for each field</figcaption></figure><h4>Getting Started with Python</h4><blockquote>Initially we can run python on Terminal or Google Colab. In next Article I will cover about where all can we write our code.</blockquote><p>Let’s dive right in and write our first Python program. Open your favorite code editor and type the following code:</p><pre># As a custom, we will print Hello World as our first code on Python<br># Ofcourse, It is very easy as compared with other languages<br>print(&quot;Hello, World!&quot;)</pre><p>In this program, print(&quot;Hello, World!&quot;) is used to display the text &quot;Hello, World!&quot; on the screen. <br>Comments in Python start with a # and are ignored by the Python interpreter.</p><h4>Conclusion: What’s Next?</h4><p>Congratulations! You’ve written your first Python program. In the next article, we will explore the fundamentals of Python programming, including variables, data types, and basic operations. Stay tuned for more Python adventures!</p><p><strong>Are you excited to continue your Python journey? <br></strong>Click the clap button if you enjoyed this article and stay tuned for more Python tutorials. <br>Don’t forget to share your thoughts in the comments section below and share the same with your friends. Your feedback is valuable to me and will help me to improve my content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9febcd90f35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Necessity of Programming : Learn before you learn to code.]]></title>
            <link>https://medium.com/@tech.mayankagg/the-necessity-of-programming-learn-before-you-learn-to-code-af17eec074d2?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/af17eec074d2</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[compilers]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Tue, 26 Mar 2024 13:00:19 GMT</pubDate>
            <atom:updated>2024-03-26T13:00:19.147Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Necessity of Programming : Learn before you learn to code.</h3><p><strong><em>If you are someone who is new to the world of programming or are just thinking on which language and/or technology to start with, I feel it very important to first understand quickly on the necessity of programming before diving deep.</em></strong></p><p>The same would help you to make your coding journey a lot easier.</p><blockquote><strong>Programming is the language of computers, enabling us to communicate our thoughts and ideas to these powerful machines. Just as we use language to convey our thoughts to others, programming allows us to instruct computers to perform tasks.</strong></blockquote><p>In this article, we’ll explore why programming is essential in today’s digital age, using simple examples to illustrate its importance so that you are ready to deep dive into programming once everything is done. :)</p><h3>Anecdote 1: The Politician’s Speech</h3><p>Imagine a world leader giving a speech in a language that no one in the audience understands. Everyone is puzzled and unable to grasp the message. Suddenly, a skilled translator steps in. With eloquence and clarity, the translator converts the leader’s words into a language the audience understands. The message is now crystal clear, and everyone can appreciate the speech’s significance.</p><p>In the world of computers, programming languages act as translators, converting our human-readable instructions into a language (binary code) that computers understand. Just as the translator bridges the gap between the leader and the audience, programming bridges the gap between humans and computers, enabling us to communicate effectively with these powerful machines.</p><h3>Anecdote 2: The Travel Guide</h3><p>Imagine you’re in a foreign country, eager to explore its rich culture and history. However, you don’t speak the local language. Fortunately, you have a knowledgeable guide who speaks both your language and the local language. This guide acts as a bridge, translating signs, directions, and conversations, allowing you to fully immerse yourself in the experience.</p><p>Similarly, programming languages serve as guides in the world of computers. They translate our commands and instructions into a language that computers understand, enabling us to navigate and interact with technology seamlessly. Just as the guide enhances your travel experience, programming enhances our ability to harness the power of computers in our daily lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIEzQ8wvR8PCprvfuS8yzg.png" /><figcaption>Types of Translators in Programming World</figcaption></figure><h3><strong>Why Do We Need Programming?</strong></h3><ol><li><strong>Computers Speak Binary</strong>: Computers understand and process information in binary code, which consists of 0s and 1s. While this language is efficient for computers, it’s complex and impractical for humans to use directly. Programming languages act as intermediaries, allowing us to write instructions in a more human-readable format.</li><li><strong>Automation and Efficiency</strong>: Programming enables automation, which is crucial for streamlining tasks and increasing efficiency. For example, a simple program can automatically sort and organize large amounts of data, saving time and effort.</li><li><strong>Problem Solving</strong>: Programming teaches us to think logically and analytically. It helps us break down complex problems into smaller, manageable parts, making it easier to find solutions. This skill is invaluable in various fields, from science and engineering to business and finance.</li></ol><h3>Examples of Programming in Action</h3><ol><li><strong>Creating a Calculator</strong>: Consider a simple calculator program. Without programming, we would need to perform calculations manually. With programming, we can create a calculator application that can add, subtract, multiply, and divide numbers with just a few clicks.</li><li><strong>Developing a Website</strong>: Building a website involves writing code to define its structure, style, and functionality. Programming languages like HTML, CSS, and JavaScript are used to create web pages that can interact with users, display content, and perform various tasks.</li></ol><p>Now I am pretty sure you have a far better idea on what programming actually is and would be better off to start with any language of your choice than to just directly dive into the same.</p><p><strong>Follow me on Medium as well as checkout my other social handles at Mayank Aggarwal (</strong><a href="https://linktr.ee/mayank953">https://linktr.ee/mayank953</a><strong>) for all things tech and make your journey into the world of technology and AI easier. :)</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af17eec074d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Install Anaconda to Run Python for Data Science]]></title>
            <link>https://medium.com/@tech.mayankagg/how-to-install-anaconda-to-run-python-for-data-science-7a6a0b0928d8?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a6a0b0928d8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 12:06:31 GMT</pubDate>
            <atom:updated>2024-02-28T12:06:31.903Z</atom:updated>
            <content:encoded><![CDATA[<p>Python is the most commonly used language in Data Science and Machine Learning. To run python for the use case of Machine Learning, Deep learning &amp; Generative AI, either one can install all the libraries required manually or download everything altogether in a bundle via anaconda.</p><p>Normally the way to install via Anaconda is preferred as it contains all the required dependency and libraries and acts like a complete solution for our use case.</p><p><strong>Note:</strong> Installing libraries manually can also lead to interdependency issues between libraries which can result in code not running properly.</p><p><strong>Anaconda Website: </strong><a href="https://www.anaconda.com/download">Free Download | Anaconda</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*S2kkcLC8IExHKkhvmpw_Lg.png" /></figure><h3>Installing Anaconda</h3><h4>Mac OS:</h4><ol><li>Download Anaconda:</li></ol><p>2. Visit the Anaconda website and download the macOS installer for the latest version of Anaconda.</p><p>3. Run the Installer:</p><ul><li>Open the downloaded .pkg file and follow the installation wizard instructions.</li><li>Agree to the license agreement, choose the installation location, and proceed with the installation.</li></ul><p>4. Verify Installation:</p><ul><li>Open a terminal window and type conda list. If Anaconda is installed correctly, it will display a list of installed packages.</li></ul><h4>Windows:</h4><ol><li>Download Anaconda:</li><li>Navigate to the Anaconda website and download the Windows installer for the latest version of Anaconda.</li><li>Run the Installer:</li></ol><ul><li>Double-click the downloaded .exe file and follow the installation wizard instructions.</li><li>Choose whether to install for “Just Me” or “All Users,” select the installation location, and proceed with the installation.</li></ul><p>4. Verify Installation:</p><ul><li>Open Anaconda Navigator from the Start menu. If it opens without errors, Anaconda is installed correctly.</li></ul><h4>Linux:</h4><ol><li>Download Anaconda.</li><li>Go to the Anaconda website and download the Linux installer for the latest version of Anaconda.</li><li>Run the Installer:</li></ol><ul><li>Open a terminal and navigate to the directory where the downloaded script is located.</li><li>Run the following command to start the installation:</li></ul><pre>bash Anaconda3-&lt;version&gt;-Linux-x86_64.sh</pre><h3>In extreme cases when you are not able to install anaconda, or your PC is old. Google Colab comes to one’s rescue.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7SEAgSVUKT15Not8X_COA.png" /><figcaption>Anaconda Navigator : use it to launch Jupyter Lab or JupyterNotebook</figcaption></figure><h3>Using Google Colab:</h3><p>If you encounter any issues with installing Anaconda locally or prefer to work in a cloud-based environment, you can use Google Colab, which provides a free Jupyter notebook environment with support for Python.</p><ol><li>Access Google Colab:</li></ol><p>2. Go to Google Colab and sign in with your Google account.</p><p>3. Create a New Notebook:</p><ul><li>Click on “New Notebook” to create a new Python notebook.</li></ul><p>4. Write and Execute Code:</p><ul><li>You can write Python code in Colab cells and execute them by pressing Shift+Enter.</li><li>Colab provides access to various Python libraries and supports code execution with GPU and TPU acceleration.</li></ul><p>5. Save and Share:</p><ul><li>Save your work on Google Drive or GitHub and share it with others.</li></ul><p>Using Google Colab allows you to work with Python notebooks without the need for local installation, making it convenient for beginners and experts alike.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6VAHeE77CK9Yh--Lqc8ANA.png" /><figcaption>Google Colab supports running Code directly on web</figcaption></figure><p>If the above guide was helpful, please follow me on Medium. I regularly write on Medium on Data Science, Machine Learning and Artificial Intelligence.</p><p>Additionally, subscribe to my <a href="https://www.youtube.com/channel/UCcYKjf55fmOsJW_on8UIs9w">YouTube Channel</a> to understand things in an intuitive and simple manner.</p><p><a href="https://www.youtube.com/channel/UCcYKjf55fmOsJW_on8UIs9w">Mayank aggarwal — YouTube</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a6a0b0928d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cross-Industry process for data mining]]></title>
            <link>https://medium.com/@tech.mayankagg/cross-industry-process-for-data-mining-286c407132d0?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/286c407132d0</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Sun, 07 Jan 2018 16:48:23 GMT</pubDate>
            <atom:updated>2018-01-07T16:49:09.765Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>CRISP-DM stands for cross industry standard process for data mining. It is a comprehensive data mining methodology and process model that provides anyone from novices to data mining experts with a complete blueprint for conducting a data mining project. CRISP-DM breaks down the life cycle of a data mining project into six phases. These 6 high-level phases of CRISP-DM are still a good description for the analytics process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*pMy1coY9API1o5NNTJGM9A.png" /></figure><p><strong>Phases of CRISP-DM:</strong></p><p><strong>1.</strong> <strong>Business Understanding</strong></p><p>First and foremost step in Analytics project is understanding client’s business to formulate problem statement. Once we define problem statement then we can drive data accordingly.</p><p>Focuses on understanding the project objectives and requirements from a business perspective, then converting this knowledge into a data mining problem definition and a preliminary plan designed to achieve the below objectives</p><p>• What the client really wants to accomplish?</p><p>• Uncover important factors (constraints, competing objectives)</p><p><strong>2.</strong> <strong>Data Understanding</strong></p><p>We start with an initial data collection and proceeds with activities in order to get familiar with the data, to identify data quality problems, to discover initial level of insights into the data or to detect interesting subsets to form hypotheses for hidden information.</p><p>Collect data</p><p>· List the datasets acquired (locations, methods used to acquire, problems encountered and solutions achieved).</p><p>· Describe data</p><p>· Check data volume and examine its gross properties.</p><p>· Accessibility and availability of attributes. Attribute types, range, correlations, identities.</p><p>· Understand the meaning of each attribute and attribute value in business terms.</p><p>· For each attribute, compute basic statistics (e.g., distribution, average, max, min, standard deviation, variance, mode, skewness).</p><p>Explore data</p><p>· Analyze properties of interesting attributes in detail</p><p>· Distribution, relations between pairs or small numbers of attributes, properties of significant sub-populations, simple statistical analyses.</p><p>Verify data quality</p><p>· Identify special values and catalogue their meaning.</p><p>· Does it cover all the cases required? Does it contain errors and how common are they?</p><p>· Identify missing attributes and blank fields. Meaning of missing data.</p><p>· Do the meanings of attributes and contained values fit together?</p><p>· Check spelling of values (e.g., same value but sometime beginning with a lower case letter, sometimes with an upper case letter).</p><p>· Check for plausibility of values, e.g. all fields has the same or nearly the same values.</p><p><strong>3.</strong> <strong>Data Preparation</strong></p><p>It covers all activities to construct the final dataset from the initial raw data. Converting raw data into analytical dataset is very important. Quality of cleaned data will impact on model performance. Data preparation tasks are likely to be performed multiple times and not in any prescribed order. Tasks include table, record and attribute selection as well as transformation and cleaning of data for modeling tools.</p><p>Select data</p><p>1. Reconsider data selection criteria</p><p>2. Decide which dataset will be used</p><p>3. Collect appropriate additional data (internal or external)</p><p>4. Consider use of sampling techniques</p><p>5. Explain why certain data was included or excluded</p><p>Clean raw data using pre-processing techniques like</p><p>1. Missing value Analysis</p><p>2. Outlier Analysis</p><p>3. Feature Engineering</p><p>4. Scaling numerical variables</p><p>5. Sampling</p><p>6. Smooting</p><p><strong>4.</strong> <strong>Modeling</strong></p><p>In this phase, various modeling techniques are selected and applied and their parameters are calibrated to optimal values. Typically, there are several techniques for the same data mining problem type. Some techniques have specific requirements on the form of data. Therefore, stepping back to the data preparation phase is often necessary.</p><p>You can use Machine learning algorithms or statistical methods to build analytical model. Some of the famous algorithms are Decision tree, random forest, KNN, naïve Bayes, K means, linear regression and logistics regression.</p><p>5. <strong>Evaluation</strong></p><p>Thoroughly evaluate the model and review the steps executed to construct the model to be certain it properly achieves the business objectives. A key objective is to determine if there is some important business issue that has not been sufficiently considered. At the end of this phase, a decision on the use of the data mining results should be reached</p><p>Points to consider:</p><p>1. Understand data mining result. Check impact for data mining goal.</p><p>2. Check result against knowledge base to see if it is novel and useful.</p><p>3. Evaluate and assess result with respect to business success criteria</p><p>4. Rank results according to business success criteria. Check result impact on</p><p><strong>6.</strong> <strong>Deployment</strong></p><p>In this phase we need to determine how the results to be utilized. The knowledge gained will need to be organized and presented in a way that the customer can use it. However, depending on the requirements, the deployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise.</p><p>There are multiple methods of model deployment</p><p>1. Traditional method using schedulers.</p><p>2. Online method using AWS, online Tools.</p><p>Pictorial representation of each of the phases and its functionality</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*tpmbkjQZ5zPlZZHyc0yzAg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=286c407132d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HYPOTHESIS TESTING]]></title>
            <link>https://medium.com/@tech.mayankagg/hypothesis-testing-92b7270976de?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/92b7270976de</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Tue, 02 Jan 2018 15:08:28 GMT</pubDate>
            <atom:updated>2024-02-27T10:06:11.912Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Hypothesis Testing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*FoqDznIdc4Mz6g19.jpg" /></figure><p><strong>Introduction:</strong></p><p>A Hypothesis test is a statistical test that is used to determine whether there is enough evidence in a sample of data to infer that a certain condition is true for the entire population.</p><p>When your car breakdown you will make an educated guess that there may be not enough petrol or may be some technical problem. Then you will take car to the nearest workshop to validate your guess/assumption/hypothesis. Depend on Mechanic answer you will reject one hypothesis and accept another hypothesis.</p><p>Here Null hypothesis is “not enough petrol”; Alternate hypothesis is may be some technical problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*IiAXE0Axq0mEBTqymSYUEA.png" /></figure><p>A hypothesis test examines two opposing hypotheses about a population: the null hypothesis and the alternative hypothesis.</p><p>Null hypothesis (H0):</p><p>The null hypothesis states that a population parameter is equal to a value. The null hypothesis is often an initial claim that researchers specify using previous research or knowledge.</p><p>Alternative Hypothesis (H1):</p><p>The alternative hypothesis states that the population parameter is different than the value of the population parameter in the null hypothesis. The alternative hypothesis is what you might believe to be true or hope to prove true.</p><p>Based on the sample data, the test determines whether to reject the null hypothesis. You use a p-value, to make the determination. If the p-value is less than or equal to the level of significance, which is a cut-off point that you define, and then you can reject the null hypothesis.</p><p>A common misconception is that statistical hypothesis tests are designed to select the more likely of two hypotheses. Instead, a test will remain with the null hypothesis until there is enough evidence (data) to support the alternative hypothesis.</p><p>Examples of questions you can answer with a hypothesis test:</p><ul><li>Does the mean height of undergraduate women differ from 66 inches?</li><li>Is the standard deviation of their height equal less than 5 inches?</li><li>Do male and female undergraduates differ in height?</li></ul><p><strong>Simple Example to understand hypothesis testing:</strong></p><p>You can follow six basic steps to correctly set up and perform a hypothesis test. For example, the manager of a pipe manufacturing facility must ensure that the diameters of its pipes equal 5cm. The manager follows the basic steps for doing a hypothesis test.</p><ol><li>Specify the hypotheses.</li></ol><p>First, the manager formulates the hypotheses. The null hypothesis is: The population mean of all the pipes is equal to 5 cm. formally, this is written as: H0: μ = 5</p><p>Then, the manager chooses from the following alternative hypotheses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*kkRNPs4Cdd1TGJCHY05mig.png" /></figure><p>Because they need to ensure that the pipes are not larger or smaller than 5 cm, the manager chooses the two-sided alternative hypothesis, which states that the population mean of all the pipes is not equal to 5 cm. formally, this is written as H1: μ ≠ 5</p><p>2. Determine the power and sample size for the test.</p><p>The manager uses a power and sample size calculation to determine how many pipes they need to measure to have a good chance of detecting a difference of 0.1 cm or more from the target diameter.</p><p>3. Choose a significance level (also called alpha or α).</p><p>The manager selects a significance level 0.05, which is the most commonly used significance level.</p><p>4. Collect the data.</p><p>They collect a sample of pipes and measure their diameters.</p><p>5. Compare the p-value from the test to the significance level.</p><p>After they perform the hypothesis test, the manager obtains a p-value of 0.004. The p-value is less than the significance level of 0.05.</p><p>6. Decide whether to reject or fail to reject the null hypothesis.</p><p>The manager rejects the null hypothesis and concludes that the mean pipe diameter of all pipes is not equal to 5cm.</p><p><strong>Test to validate hypothesis:</strong></p><p>1. ANNOVA</p><p>2. Chi-square</p><p>3. T-Test</p><p>4. F-Test</p><p>5. A/B testing</p><p>Hypothesis tests can be used to evaluate many different parameters of a population. Each test is designed to evaluate a parameter associated with a certain type of data. Knowing the difference between the types of data, and which parameters are associated with each data type, can help you choose the most appropriate test.</p><p><strong>ANNOVA</strong></p><p><strong>Introduction:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*BTItIqUAN9JYmDaCbCos2g.png" /></figure><p>ANOVA is a statistical method that stands for analysis of variance. ANOVA was developed by Ronald Fisher in 1918 and is the extension of the <em>t</em> and the <em>z</em> test. Before the use of ANOVA, the <em>t</em>-test and <em>z-</em>test were commonly used. But the problem with the T-test is that it cannot be applied for more than two groups ANNOVA is used to do the analysis of variance between and within the groups whenever the groups are more than two. ANOVA creates a way to test several null hypotheses at the same time.</p><p>The logic behind this procedure has to do with how much variance there is in the population. It is likely the researcher will not know the actual variance in the population, but they can estimate this by sampling and calculating the variance in the sample. You compare the differences in the samples to see if they are the same or statistically different while still accounting for sampling error.</p><p><strong>Use of ANNOVA in different ways:</strong></p><p>Commonly, we can use ANOVA in three ways: <a href="http://www.statisticssolutions.com/data-analysis-plan-one-way-anova/">one-way ANOVA</a>, two-way ANOVA<a href="http://www.statisticssolutions.com/academic-solutions/resources/directory-of-statistical-analyses/factorial-anova/">,</a> and N-way Multivariate ANOVA.</p><p>One-Way: When we compare more than two groups, based on one factor (independent variable), this is called one way ANOVA. For example, it is used if a manufacturing company wants to compare the productivity of three or more employees based on working hours. This is called one way ANOVA.</p><p>Two-Way: When a company wants to compare the employee, productivity based on two factors (2 independent variables), then it said to be two-way (Factorial) ANOVA. For example, based on the working hours and working conditions, if a company wants to compare employee productivity, it can do that through two-way ANOVA. Two-way ANOVA’s can be used to see the effect of one of the factors after controlling for the other, or it can be used to see the INTERACTION between the two factors. This is a great way to control for extraneous variables as you are able to add them to the design of the study.</p><p>N-Way: When the factor comparison is taken, then it said to be n-way ANOVA. For example, in productivity measurement if a company takes all the factors for productivity measurement, then it is said to be n-way ANOVA.</p><p><strong>General procedure to calculate ANNOVA:</strong></p><p>In an ANOVA, first sets up the null and alternative hypothesis.</p><p>H0: The null hypothesis assumes that there is no significant difference between the groups.</p><p>H1: Alternative hypothesis assumes that there is a significant difference between the groups.</p><p>After cleaning the data, we must test the above assumptions and see if the data meets them. They must then do the necessary calculation and calculate the F-ratio. After this, we must compare the critical value of the F-ratio with the table value or simply look at the p value against the established alpha. If the calculated critical value is greater than the table value, the null hypothesis will be rejected, and the alternative hypothesis is accepted. Rejecting the null hypothesis, we will conclude that the mean of the groups is not equal. If the calculated value is less than the table value, we will accept the null hypothesis and reject the alternative hypothesis. This will tell you that there is a difference in what you were testing but does not tell you WHERE the difference is. This is to say, if we were testing several groups against one another, we would know that there is a difference between the means of the groups but not which individual groups are different.</p><p><strong>Applications:</strong></p><p>ANOVA is used very commonly in business, medicine or in psychology research.</p><p>· In business, ANOVA can be used to compare the sales of different designs based on different factors.</p><p>· A psychology researcher can use ANOVA to compare the different attitude or behavior in people and whether or not they are the same depending on certain factors.</p><p>· In medical research, ANOVA is used to test the effectiveness of a drug.</p><p><strong>Additional information:</strong></p><p>Data level and assumption plays a very important role in ANOVA. In ANOVA, the dependent variable can be continuous or on the interval scale. Factor variables in ANOVA should be categorical. Like the T-test, ANOVA is also a parametric test and has some assumptions, which should be met to get the desired results. ANOVA assumes that the distribution of data should be normally distributed.</p><p>ANOVA also assumes the assumption of homogeneity, which means that the variance between the groups should be equal. ANOVA also assumes that the cases are independent to each other or there should not be any pattern between the cases. As usual, when planning any study, extraneous and confounding variables need to be considered. ANOVA is a way to control these types of undesirable variables.</p><p><strong>Chi-square Analysis</strong></p><p><strong>Introduction:</strong></p><p>There are two types of chi-square tests. Both use the chi-square statistic and distribution for different purposes:</p><p>· A chi-square goodness of fit test determines if a sample data matches a population.</p><p>· A chi-square test for independence compares two variables in a contingency table to see if they are related. In a more general sense, it tests to see whether distributions of <a href="http://www.statisticshowto.com/what-is-a-categorical-variable/">categorical variables</a> differ from each another.</p><p>o A very small chi square test statistic means that your observed data fits your expected data extremely well. In other words, there is a relationship.</p><p>o A very large chi square test statistic means that the data does not fit very well. In other words, there isn’t a relationship.</p><p>Chi-square test can be used for checking variable independencies, Homogeneity, goodness of fit.</p><p>A chi-square statistic is one way to show a relationship between two <a href="http://www.statisticshowto.com/what-is-a-categorical-variable/">categorical variables</a>. In statistics, there are two types of variables: numerical (countable) variables and non-numerical (categorical) variables. The chi-squared statistic is a single number that tells you how much difference exists between your observed counts and the counts you would expect if there were no relationship at all in the population.</p><p>There are a few variations on the chi-square statistic which one you use depends upon how you collected the data and which hypothesis is being tested. However, all of the variations use the same idea, which is that you are comparing your expected values with the values you actually collect. One of the most common forms can be used for <a href="http://www.statisticshowto.com/what-is-a-contingency-table/">contingency tables</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*-xSo8glpTHUEUmfnRy35qg.png" /></figure><p>Where O is the observed value, E is the expected value and “i” is the “ith” position in the contingency table.</p><p>A high value for the chi-square statistic means there is a high correlation between your two sets of data. Deciding whether a chi-square test statistic is “large enough” isn’t as easy it seems. It would be nice if we could say a chi-square test statistic &gt;10 means a correlation but unfortunately that isn’t the case. Being able to decide whether the statistic is large enough requires you to have a good grasp of hypothesis testing.</p><p>In order to decide if your test value is high enough, first state the <a href="http://www.statisticshowto.com/what-is-the-null-hypothesis/">null hypothesis</a> and the <a href="http://www.statisticshowto.com/what-is-an-alternate-hypothesis/">alternate hypothesis</a>. Then generate a chi-square curve for your results along with a p-value.</p><p>The Chi-square Test for Association which is a non-parametric test; therefore, it can be used for nominal data too. It is a test of statistical significance widely used bivariate tabular association analysis. Typically, the hypothesis is whether or not two populations are different in some characteristic or aspect of their behaviour based on two random samples. This test procedure is also known as the Pearson Chi-square test.</p><p><strong>Example:</strong></p><p>First case</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*zVZ0O9JwGp3X5df06ZQEfQ.png" /></figure><p>· Is your manager’s mood associated with the weather?</p><p>· This appears to be no association between your manager’s mood and the weather here because the row percentages are in same in each column.</p><p>Second case</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*SYa87BnZ63gkHW7X7ztT9g.png" /></figure><p>· Is your manager’s mood associated with the weather?</p><p>· There appears to be an association here because the row percentage is different in each column.</p><p><strong>Additional information:</strong></p><p>The Chi-square statistic can only be used on numbers. They can’t be used for percentages, proportions, means or similar statistical value. For example, if you have 10 percent of 200 people, you will need to convert that to a number (20) before you can run a test statistic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92b7270976de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Covariance & Correlation]]></title>
            <link>https://medium.com/@tech.mayankagg/covariance-correlation-def860c4d4ab?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/def860c4d4ab</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Tue, 02 Jan 2018 15:01:00 GMT</pubDate>
            <atom:updated>2018-01-02T15:02:49.346Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*Kd2NLWG_PZGOrzve.jpg" /></figure><h3><strong>Covariance</strong></h3><p><strong>Introduction:</strong></p><p>Covariance is a measure of how changes in one variable are associated with changes in a second variable. Specifically, covariance measures the degree to which two variables are linearly associated. However, it is also often used informally as a general measure of how monotonically related two variables are.</p><p>In <a href="https://en.wikipedia.org/wiki/Probability_theory">probability theory</a> and <a href="https://en.wikipedia.org/wiki/Statistics">statistics</a>, covariance is a measure of how much two <a href="https://en.wikipedia.org/wiki/Random_variable">random variables</a> change together. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the lesser values, i.e., the variables tend to show similar behavior, the covariance is positive. For example, as a balloon is blown up it gets larger in all dimensions. In the opposite case, when the greater values of one variable mainly correspond to the lesser values of the other, i.e., the variables tend to show opposite behavior, the covariance is negative. If a sealed balloon is squashed in one dimension then it will expand in the other two. The sign of the covariance therefore shows the tendency in the <a href="https://en.wikipedia.org/wiki/Linear_relationship">linear relationship</a> between the variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*E1ZxVMAsx0rzupARhGXosA.png" /></figure><p>Variables whose covariance is zero are called <a href="https://en.wikipedia.org/wiki/Uncorrelated">uncorrelated</a> variables.</p><p>Covariance can be calculated as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*E_RQsnbiHS1vIisi2KdUng.png" /></figure><p>Where E(X) = mean of variable X</p><p>E(Y) = mean of variable Y</p><p>Various tools have function or functionality to identify covariance between variables. In Excel, function COVAR() is used to return the covariance between two variables and SAS uses procedure PROC COV to identify the covariance.</p><p>Applications:</p><ol><li>By using covariance, a portfolio manager can identify if the portfolio is adequately diversified</li></ol><h3><strong>Correlation</strong></h3><p><strong>Introduction:</strong></p><p>Correlation refers to the extent to which two variables have a linear relationship with each other. It is a statistical technique that can show whether and how strongly variables are related. It is a scaled version of covariance and values ranges from -1 to +1.</p><p>It can be calculated as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*LLQExBr_pHBILnKlxThRyQ.png" /></figure><p><strong>Difference between Covariance and Correlation:</strong></p><p>In <a href="https://en.wikipedia.org/wiki/Probability_theory">probability theory</a> and <a href="https://en.wikipedia.org/wiki/Statistics">statistics</a>, the mathematical concepts of covariance and correlation are very similar. Both describe the degree to which two <a href="https://en.wikipedia.org/wiki/Random_variable">random variables</a> or <a href="https://en.wikipedia.org/wiki/Set_(mathematics)">sets</a> of random variables tend to deviate from their <a href="https://en.wikipedia.org/wiki/Expected_value">expected values</a> in similar ways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*CS-B07d6ZHZXLImYh-mnLg.png" /></figure><p>If X and Y are two <a href="https://en.wikipedia.org/wiki/Random_variables">random variables</a>, with <a href="https://en.wikipedia.org/wiki/Mean">means</a> μX and μY, and standard deviations σX and σY, respectively, then their covariance and correlation are as follows:</p><p>Where E is the expected value operator. Notably, correlation is <a href="https://en.wikipedia.org/wiki/Dimensionless">dimensionless</a> while covariance is in units obtained by multiplying the units of the two variables. The correlation of a variable with itself is always 1 (except in the <a href="https://en.wikipedia.org/wiki/Degenerate_case">degenerate case</a> where the two variances are zero, in which case the correlation does not exist).</p><p><strong>Ways to detect Correlation between variables:</strong></p><p>1. Graphical Method: While doing bi-variate analysis between two continuous variables, we should look at scatter plot. It is a nifty way to find out the relationship between two variables. The pattern of scatter plot indicates the relationship between variables. The relationship can be linear or non-linear.</p><p>Scatter plot shows the relationship between two variable but does not indicates the strength of relationship amongst them. To find the strength of the relationship, we use statistical technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*09jO8ZNNIX7udiXWD6pOfg.png" /></figure><p>2. Non-graphical method: Build the correlation matrix to understand the strength between variables. Correlation varies between -1 and +1.</p><p>a. -1: Perfect negative linear correlation</p><p>b. +1: Perfect positive linear correlation</p><p>c. 0: No correlation</p><p><strong>Ideal assumptions:</strong></p><p>1. High Correlation between dependent and independent variable.</p><p>2. Less correlation between independent variables.</p><p>Generally, if the correlation between the two independent variables are high (&gt;= 0.8) then we drop one independent variable otherwise it may lead to multi collinearity problem. Various tools have function or functionality to identify correlation between variables. In Excel, function CORREL() is used to return the correlation between two variables and SAS uses procedure PROC CORR to identify the correlation. These function returns Pearson Correlation value to identify the relationship between two variables.</p><h3>Multicollinearity</h3><p><strong>Introduction:</strong></p><p>Multicollinearity (also collinearity) is a phenomenon in which two or more predictor <a href="https://en.wikipedia.org/wiki/Variable_(mathematics)">variables</a> (Independent variables) in a <a href="https://en.wikipedia.org/wiki/Multiple_regression">regression</a> model are highly <a href="https://en.wikipedia.org/wiki/Correlation_and_dependence">correlated</a>, meaning that one can be linearly predicted from the others with a substantial degree of accuracy. In this situation the <a href="https://en.wikipedia.org/wiki/Regression_coefficient">coefficient estimates</a> of the multiple regression may change erratically in response to small changes in the model or the data. Collinearity is a linear association between two <a href="https://en.wikipedia.org/wiki/Explanatory_variable">explanatory variables</a>. Two variables are perfectly collinear if there is an exact linear relationship between them.</p><p><strong>Types of multicollinearity:</strong></p><p>There are two types of multicollinearity:</p><p>1. Structural multicollinearity is a mathematical artifact caused by creating new predictors from other predictors — such as, creating the predictor <em>x</em>2 from the predictor <em>x</em>.</p><p>2. Data-based multicollinearity, on the other hand, is a result of a poorly designed experiment, reliance on purely observational data, or the inability to manipulate the system on which the data are collected.</p><p><strong>Detect Multicollinearity:</strong></p><p>Indicators that multicollinearity may be present in a model include the following:</p><p>1. Large changes in the estimated regression coefficients when a predictor variable is added or deleted</p><p>2. Insignificant regression coefficients for the affected variables in the multiple regression, but a rejection of the joint hypothesis that those coefficients are all zero (using an <a href="https://en.wikipedia.org/wiki/F-test">F-test</a>)</p><p>3. If a multivariable regression finds an insignificant coefficient of a particular explanator, yet a <a href="https://en.wikipedia.org/wiki/Simple_linear_regression">simple linear regression</a> of the explained variable on this explanatory variable shows its coefficient to be significantly different from zero, this situation indicates multicollinearity in the multivariable regression.</p><p>4. VIF (Variance inflation factor) can be used to detect multicollinearity in the regression model{\displaystyle \mathrm {tolerance} =1-R_{j}^{2},\quad \mathrm {VIF} ={\frac {1}{\mathrm {tolerance} }},}</p><p><strong>Why is this problem?</strong></p><p>• Collinearity tends to inflate the variance of at least one estimated regression coefficient.</p><p>• This can cause at least some regression coefficients to have the wrong sign.</p><p><strong>Ways of dealing with collinearity</strong></p><p>· Ignore it. If prediction of y values is the object of your study, then collinearity is not a problem.</p><p>· Get rid of the redundant variables by using variable sélection technique.</p><p>There are multiple techniques to select variables which are less correlated with high importance</p><p>1. Correlation method</p><p>2. PCA (Principal Component Analysis)</p><p>3. SVD (Singular value Decomposition)</p><p>4. Machine learning algorithms (Random Forest, Decision trees)</p><h3>Remedies for multicollinearity</h3><p>1. Drop one of the variables. An explanatory variable may be dropped to produce a model with significant coefficients. However, you lose information (because you’ve dropped a variable). Omission of a relevant variable results in biased coefficient estimates for the remaining explanatory variables that are correlated with the dropped variable.</p><p>2. Obtain more data, if possible. This is the preferred solution. More data can produce more precise parameter estimates (with lower standard errors), as seen from the formula in <a href="https://en.wikipedia.org/wiki/Variance_inflation_factor">variance inflation factor</a> for the variance of the estimate of a regression coefficient in terms of the sample size and the degree of multicollinearity.</p><p>3. Try seeing what happens if you use independent subsets of your data for estimation and apply those estimates to the whole data set. Theoretically you should obtain somewhat higher variance from the smaller datasets used for estimation, but the expectation of the coefficient values should be the same. Naturally, the observed coefficient values will vary, but look at how much they vary.</p><p>4. Standardize your independent variables. This may help reduce a false flagging of a condition index above 30.</p><p>5. It has also been suggested that using the <a href="https://en.wikipedia.org/wiki/Shapley_value">Shapley value</a>, a game theory tool, the model could account for the effects of multicollinearity. The Shapley value assigns a value for each predictor and assesses all possible combinations of importance.</p><p>6. If the correlated explanators are different lagged values of the same underlying explanator, then a <a href="https://en.wikipedia.org/wiki/Distributed_lag">distributed lag</a> technique can be used, imposing a general structure on the relative values of the coefficients to be estimated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=def860c4d4ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistical Quantities]]></title>
            <link>https://medium.com/@tech.mayankagg/statistical-quantities-90467abe7e26?source=rss-528e973b8d36------2</link>
            <guid isPermaLink="false">https://medium.com/p/90467abe7e26</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[median]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[mean]]></category>
            <category><![CDATA[nsit]]></category>
            <dc:creator><![CDATA[Mayank Aggarwal]]></dc:creator>
            <pubDate>Tue, 02 Jan 2018 14:49:31 GMT</pubDate>
            <atom:updated>2018-01-02T14:49:31.751Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*N-mItgNojajKwr7H.jpg" /></figure><p><strong>Introduction:</strong></p><p>There are five basic statistical quantities Mean, Median, Mode, Variance and standard deviation. Mean, Median, Mode are also called central Tendency. Central tendency (or measure of central tendency) is a central or typical value for a <a href="https://en.wikipedia.org/wiki/Probability_distribution">probability distribution</a>. It may also be called a center or location of the distribution. A central tendency can be calculated for either a finite set of values or for a theoretical distribution, such as the <a href="https://en.wikipedia.org/wiki/Normal_distribution">normal distribution</a>.</p><p><strong>1.</strong> <strong>Mean:</strong></p><p>In simple terms, Arithmetic mean (or simply, mean) is the sum of all measurements divided by the number of observations in the data set.</p><p>• The mean is:</p><p>– the arithmetic average of all the scores ( SX)/N</p><p>– the number, m, that makes S(X — m) equal to 0</p><p>– the number, m, that makes S(X — m)2 a minimum</p><p>• The mean of a population is represented by the Greek letter m; the mean of a sample is represented by X</p><p>• You should use the mean when</p><p>– the data are interval or ratio scaled</p><p>• Many people will use the mean with ordinally scaled data too</p><p>• The mean is preferred because it is sensitive to every score</p><p>– If you change one score in the data set, the mean will change</p><p><strong>Care to be taken</strong></p><p>· Mean is very sensitive to outliers (An abnormal value in the data). Do not uses mean if data contains outliers else lead to false assumptions.</p><p>· Mean is correct metric if data is normally distributed.</p><p><strong>2.</strong> <strong>Median</strong></p><p>The middle value that separates the higher half from the lower half of the data set. The median and the mode are the only measures of central tendency that can be used for ordinal data, in which values are ranked relative to each other but are not measured absolutely.</p><p>The median is simply another name for the 50th percentile</p><p>• It is the score in the middle; half of the scores are larger than the median and half of the scores are smaller than the median</p><p>• The median is often used when the distribution of scores is either positively or negatively skewed</p><p>– The few really large scores (positively skewed) or really small scores (negatively skewed) will not overly influence the median</p><p><strong>3.</strong> <strong>Mode</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*eTCesc1ydL_YUM8exxNYiA.png" /></figure><p>The most frequent value in the data set, one that occurs maximum number of times. This is the only central tendency measure that can be used with nominal data, which have purely qualitative category assignments.</p><h3>Relationships between the mean, median and mode</h3><p>For <a href="https://en.wikipedia.org/wiki/Unimodal_distribution">unimodal distributions</a> the following bounds are known and are sharp</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*dMRnlJGxQpOe331qe9RpvA.png" /><figcaption>Relationship between Median Mean &amp; Mode.</figcaption></figure><p>Where μ is the mean, ν is the median, θ is the mode, and σ is the standard deviation.</p><p>For every distribution,</p><p>The mean, mode and median do a nice job in telling where the center of the data set is, but often we are interested in more. For example, a pharmaceutical engineer develops a new drug that regulates iron in the blood. Suppose she finds out that the average sugar content after taking the medication is the optimal level. This does not mean that the drug is effective. There is a possibility that half of the patients have dangerously low sugar content while the other half has dangerously high content. Instead of the drug being an effective regulator, it is a deadly poison. What the pharmacist needs is a measure of how far the data is spread apart. This is what the variance and standard deviation do.</p><p><strong>Variance:</strong></p><p>Variance measures how far a set of numbers are spread out from their mean. Variance is calculated by taking the differences between each number in the set and the mean, squaring the differences (to make them positive) and dividing the sum of the squares by the number of values in the set.</p><p>It is used in descriptive statistics, statistical inference, hypothesis testing, goodness of fit, Monte Carlo sampling, amongst many others.</p><p><strong>Standard Deviation:</strong></p><p>Standard deviation tells you how much data deviates from the actual mean. It is the square root of the Variance</p><p>A low standard deviation indicates that the data points tend to be close to the mean, while a high standard deviation indicates that the data points are spread out over a wider range of values. A useful property of the standard deviation is that, unlike the variance, it is expressed in the same units as the data.</p><p>Standard Deviation can be used to check the data distribution (Ideal assumption is normally distributed data). If a data distribution is approximately normal then about 68 percent of the data values are within one standard deviation of the mean (mathematically, μ ± σ, where μ is the arithmetic mean), about 95 percent are within two standard deviations (μ ± 2σ), and about 99.7 percent lie within three standard deviations (μ ± 3σ). This is known as the <a href="https://en.wikipedia.org/wiki/68-95-99.7_rule">68–95–99.7 rule</a>, or the empirical rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*Vj1MHhq-InHE-vy450jVUA.png" /></figure><p><strong>Additional information:</strong></p><p>Normal distribution is also called standard normal distribution in which μ=0 and σ=1.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90467abe7e26" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>