<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akash Mishra on Medium]]></title>
        <description><![CDATA[Stories by Akash Mishra on Medium]]></description>
        <link>https://medium.com/@akashmishra77?source=rss-5184c919d895------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oY-0qtb8jtAWI6KwFCd8Qw.jpeg</url>
            <title>Stories by Akash Mishra on Medium</title>
            <link>https://medium.com/@akashmishra77?source=rss-5184c919d895------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:33:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akashmishra77/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unmasking Deception: Navigating the Shadows of Modern Influences]]></title>
            <link>https://medium.com/@akashmishra77/unmasking-deception-navigating-the-shadows-of-modern-influences-fff92afcffec?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/fff92afcffec</guid>
            <category><![CDATA[gambling-games]]></category>
            <category><![CDATA[reality-check]]></category>
            <category><![CDATA[self-reflection]]></category>
            <category><![CDATA[self-awareness]]></category>
            <category><![CDATA[detox]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Sun, 12 May 2024 09:00:13 GMT</pubDate>
            <atom:updated>2024-05-12T09:00:13.241Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2MM2r5ZYSXVUpCt8xloB9g.png" /></figure><p>In today’s world, what we see isn’t always what we get. Behind the glitz and glamour of advertisements and endorsements lies a hidden truth, often obscured by the allure of fame and fortune. Let’s embark on a journey to uncover the illusions that pervade our society and discover how we can navigate through them with clarity and conviction.</p><h4><strong>Deceptive Allure: The Dark Side of Celebrity Endorsements</strong></h4><p><strong>Ever noticed your favorite celebrity flashing a smile while promoting products that could harm your health?</strong> <br>- It’s a common sight, with pan masala brands roping in stars to endorse their products, despite the well-known risks of cancer associated with them. <br><strong>-</strong> Similarly, the world of cricket isn’t immune to this trend, with popular players lending their names to gambling apps like <strong>Dream11 and MPL</strong>, blurring the lines between entertainment and addiction.</p><h4>Redefining Success: Beyond Wealth and Fame</h4><p>In a world where success is often measured by material possessions and social status, it’s easy to lose sight of what truly matters. Contrary to popular belief, financial prosperity doesn’t always equate to happiness, and fame doesn’t guarantee fulfillment.</p><p>Many affluent individuals find themselves isolated and discontent, while others who lack fame or fortune lead deeply fulfilling lives. It’s a reminder that true wealth lies in the richness of experiences and relationships, not in the size of our bank accounts.</p><h4>The Mirage of Social Media: Unveiling the Illusions</h4><p>Scrolling through Instagram, it’s hard not to feel a twinge of envy at the seemingly perfect lives showcased on our screens.</p><p>Yet, behind the filtered photos and carefully crafted captions lies a world of insecurity and anxiety. People often go to great lengths to maintain a facade of perfection, even if it means plunging themselves into debt or sacrificing their mental well-being.</p><p>But here’s the truth: those fleeting moments of admiration from strangers hold little weight in the grand scheme of things. True happiness comes from being true to oneself, not from seeking validation from others.</p><h4>Finding Clarity in a World of Illusions</h4><p>Amidst the noise and distractions of modern life, it’s essential to cultivate a sense of clarity and purpose. Instead of blindly following trends or succumbing to peer pressure, take the time to reflect on your own values and aspirations.</p><p>Embrace authenticity over conformity, and prioritize meaningful connections over superficial acclaim. Remember, the path to true fulfillment lies not in chasing illusions, but in embracing the beauty of reality.</p><h4><strong>Conclusion</strong></h4><p>As we navigate through the shadows of deception, let’s remember to stay true to ourselves and our values. By seeing beyond the illusions that surround us, we can carve out a path that leads to genuine happiness and fulfillment. So, the next time you’re tempted to believe everything you see, take a step back, and question the narrative. After all, the truth is often hidden beneath the surface, waiting to be uncovered.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fff92afcffec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why de-normalisation is so important in data and analytics world?]]></title>
            <link>https://medium.com/@akashmishra77/why-de-normalisation-is-so-important-in-data-and-analytics-world-7cf354a45f0b?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cf354a45f0b</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Sat, 13 Apr 2024 11:03:56 GMT</pubDate>
            <atom:updated>2024-04-13T11:03:56.153Z</atom:updated>
            <content:encoded><![CDATA[<p>While many of us understand the advantages of organizing via normalization, let’s dive into another crucial concept: <strong>denormalization</strong>.</p><p><strong>denormalization is </strong>about accelerating data searches in analytics, where large datasets and complex queries are common.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*i-C2t9pqDlNuGqPLcu2axQ.png" /></figure><p>Let’s explore!</p><p><strong>Denormalization</strong> is a game-changer for analytics. By duplicating certain data, it speeds up analytical queries, making them faster and more efficient.</p><p>In contrast, <strong>normalization</strong>, while essential for data integrity, can slow down queries due to the need for complex joins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*a2KbQ-ep2FXJeLko" /></figure><p><strong>Why is denormalization vital for analytics?</strong></p><p>Because it improves the speed of analytical queries, enabling faster data insights and decision-making. Without denormalization, analytical queries may suffer from slow performance, hindering timely reporting and analysis.</p><p><strong>How to denormalise a database?</strong></p><p>1- Identify areas in your database where analytical queries are slow. Normalized databases often require extensive joins to retrieve data.</p><p>2- Duplicate certain table data which reduces the need for complex joins, resulting in faster query execution and improved reporting speed. This ensures that analytical insights are delivered in a timely manner, facilitating better decision-making.</p><p>3- Track improvements in query performance, reporting speed, and overall efficiency to assess the impact of denormalization on analytical operations.</p><p><strong>Conclusion</strong></p><p>Thanks for the read. Hope you have enjoyed it.</p><p>Stay tuned for more insights on data and software engineering. Let’s keep learning and sharing together!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cf354a45f0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seamless Data Ingestion: Harnessing PySpark and Snowflake’s Magic]]></title>
            <link>https://medium.com/@akashmishra77/seamless-data-ingestion-harnessing-pyspark-and-snowflakes-magic-1e9830c62342?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e9830c62342</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Mon, 25 Dec 2023 09:41:23 GMT</pubDate>
            <atom:updated>2023-12-25T09:41:23.785Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p9pCcXRfJDI1-glOCC8DfA.png" /></figure><h3>Introduction</h3><p>In this blog, get ready to dive into the fascinating realm of data ingestion, fueled by the powerful synergy of PySpark and Snowflake. Data ingestion plays a pivotal role in data engineering, where we gather diverse data pieces and prepare them for analysis. Imagine it as solving a puzzle, with PySpark acting as the puzzle master, skillfully processing and arranging data pieces, while Snowflake provides the canvas for the final picture. We’ll cover everything from environment setup to various ingestion methods, data transformation, and loading into Snowflake. Whether you’re a beginner or an experienced data engineer, this blog offers a valuable resource for mastering seamless data ingestion with PySpark and Snowflake, creating a symphony of insights, piece by piece.</p><h3>Understanding Data Ingestion and its Challenges</h3><ul><li>Data ingestion is the crucial first step in the data lifecycle, where data is collected and brought together from different sources for analysis.</li><li>Challenges during data ingestion include dealing with diverse data formats, managing large volumes of data, and ensuring data cleanliness and quality.</li><li>A scalable and efficient data ingestion process is essential to handle growing data volumes and maintain a smooth data pipeline.</li><li>Think of data ingestion as solving a puzzle with scattered pieces of information, where the goal is to create a seamless and reliable process for gathering and preparing data for analysis.</li><li>In this section, we’ll explore the importance of data ingestion and tackle the challenges to enable a successful data engineering journey.</li></ul><h3>Introduction to PySpark and Snowflake</h3><p><strong>1. Overview of PySpark and its Role in Data Processing</strong></p><p>PySpark is a powerful tool that plays a crucial role in data processing, especially when dealing with big data. It combines the versatility of Python with the processing capabilities of Apache Spark, making it a go-to choice in the big data ecosystem. PySpark’s popularity stems from its ability to handle vast amounts of data through distributed computing, enabling parallel processing for lightning-fast and efficient data operations. Imagine having a team of experts working together to process your data simultaneously!</p><p><strong>2. Introducing Snowflake: A Cloud-Based Data Warehousing Solution</strong></p><p>Meet Snowflake, the cloud-based data warehousing solution designed to cater to modern data management needs. Think of Snowflake as a high-tech storage warehouse that not only manages structured data but also seamlessly handles semi-structured data like JSON and Parquet files. One of Snowflake’s remarkable features is its elasticity and scalability, allowing it to dynamically adjust its storage capacity based on your data needs. This ensures cost-effectiveness and flexibility in managing your data.</p><p><strong>3. The Synergy between PySpark and Snowflake for Seamless Data Integration</strong></p><p>When PySpark and Snowflake join forces, the result is a perfect data integration duo! PySpark’s data processing capabilities complement Snowflake’s storage prowess, creating a seamless environment for data ingestion and analysis. Imagine PySpark as your data chauffeur, efficiently processing data from various sources, while Snowflake acts as your data vault, securely storing all the valuable insights. This powerful synergy enables seamless movement of data between PySpark and Snowflake, facilitating a smooth and efficient data engineering workflow.</p><p>In this section, we will explore the magic of PySpark and the wonders of Snowflake, unraveling the unique strengths that make them top choices for data engineers. We’ll dive into their roles and advantages, empowering you to leverage their capabilities and unlock the true potential of your data. Get ready to embark on a journey through the world of PySpark and Snowflake as we delve into their roles in modern data engineering.</p><h3>Preparing the Environment</h3><p><strong>Guide to Setting up the Environment for PySpark and Snowflake Integration</strong></p><p>Before we dive into the exciting world of PySpark and Snowflake integration, let’s ensure our environment is properly set up for a smooth data engineering experience. Follow these steps to create the perfect workspace for seamless data ingestion and analysis.</p><p>Open Python editor or Jupyter Notebook and execute the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*HxkceBrCuhblweI5" /></figure><p><strong>Step-by-Step Instructions for Installing and Configuring PySpark and the Snowflake Connector</strong></p><p>Next, let’s install the Snowflake connector, enabling seamless data transfer between PySpark and Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/0*PF4smbW3h9H80a_Y" /></figure><p>Now, it’s time to configure PySpark to work with Snowflake. Update the <strong>spark-defaults.conf</strong> file to include Snowflake configurations.</p><p><strong>echo “spark.jars.packages net.snowflake:snowflake-jdbc:3.13.4,net.snowflake:spark-snowflake_2.12:2.8.8-spark_3.0” &gt;&gt; $SPARK_HOME/conf/spark-defaults.conf</strong></p><p><strong>echo “spark.sql.extensions net.snowflake.spark.snowflake:SnowflakeSparkExtension” &gt;&gt; $SPARK_HOME/conf/spark-defaults.conf</strong></p><p><strong>Addressing Potential Compatibility Issues and Offering Troubleshooting Tips</strong></p><p>During the installation, you might encounter compatibility issues. Here are some common concerns and troubleshooting tips:</p><ul><li>If you face version conflicts, ensure you use compatible versions of PySpark and the Snowflake connector.</li><li>Check if your system meets the requirements for installing both PySpark and the Snowflake connector.</li></ul><h3>Data Ingestion Methods with PySpark and Snowflake</h3><p>In this section, we’ll explore how to ingest data from local files such as CSV, JSON, or Parquet into PySpark and then load it into Snowflake for further analysis. Let’s start with ingesting CSV files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*VjwjST7SOhJ8wno7" /></figure><p>Run the command <strong>df_csv.show()</strong> to see the loaded data in pyspark</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*Yg08-ah_2S0OYfAq" /></figure><p>Now we will load this data into snowflake:</p><p>Connect to snowflake account using the following snippet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/0*36n0_Zm6RVyQws98" /></figure><p>To load the spark dataframe, you can execute following code snippet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/0*OQhyq9MQxI6BRIyp" /></figure><p>we can also convert it into a pandas dataframe and ingest the data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*Wwh3jl7uZXjgO4F9" /></figure><p>Creating the database, table and insert this dataframe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*ko15M4Wn0PCWWtrx" /></figure><p>we can see the database and table have been created in Snowflake</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0UtyKbVgn5Z5VVMm" /></figure><p>but it has no data so now let’s ingest this data<br> <strong>1- Ingesting it using SQL Inserts</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*JpVZyjVVL3epwm-c" /></figure><p><strong>2-Ingesting it through the library</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*pzX9s_7aEi69CMiP" /></figure><p>Now, let’s check the results in the snowflake table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Aenk8KibmV3Cz8Od" /></figure><p>we can see that our data load from pyspark to snowflake has happened successfully.</p><h3>Transforming and Cleaning the Data</h3><p><strong>Importance of Data Transformation and Cleaning</strong></p><p>Data transformation and cleaning play a crucial role in the data ingestion process. Raw data often comes in various formats and may contain errors, inconsistencies, or missing values. Before loading data into Snowflake, it’s essential to transform and clean the data to ensure its quality, accuracy, and compatibility with the target schema. By performing these vital steps, we can enhance the reliability of our analysis and decision-making processes, ultimately leading to better insights and outcomes.</p><p><strong>Common Data Transformations using PySpark</strong></p><p>In this section, we’ll explore some common data transformations using PySpark to prepare the data for ingestion into Snowflake. We’ll showcase how PySpark’s powerful data processing capabilities allow us to clean, enrich, and structure the data efficiently. Examples of data transformations include:</p><ol><li><strong>Cleaning Missing Values</strong>: Handling missing data is critical to avoid skewed results. We’ll demonstrate how to identify and replace missing values with meaningful alternatives or drop rows with missing data, ensuring data completeness.</li><li><strong>Data Type Conversion</strong>: Data from various sources may have different data types. We’ll show how to convert data types to match the target schema in Snowflake, enabling seamless data integration.</li><li><strong>String Manipulation</strong>: Sometimes, data may contain irrelevant characters or need to be split into separate columns. We’ll illustrate how to perform string manipulation operations like trimming, substring extraction, or concatenation.</li><li><strong>Data Aggregation</strong>: When dealing with large datasets, aggregating data is essential to summarise information effectively. We’ll present techniques to group data and calculate meaningful insights using aggregation functions.</li><li><strong>Handling Duplicates</strong>: Duplicate records can distort analysis results. We’ll guide you through identifying and removing duplicate data, ensuring data accuracy.</li><li><strong>Data Filtering</strong>: We’ll demonstrate how to filter out irrelevant data using conditions, allowing us to focus on the most relevant information for our analysis.</li></ol><h3>Error Handling and Data Quality</h3><p><strong>Strategies for Handling Errors during Data Ingestion</strong></p><p>In this section, we’ll explore how to handle errors that might occur during the data ingestion process. Just like a safety net, error handling ensures that even if something goes wrong, we can gracefully recover and keep the data pipeline running smoothly. We’ll discuss different strategies for error handling, such as logging errors for later review, retrying failed data loads, or notifying the team when issues arise. By employing these strategies, we can maintain data integrity and prevent disruptions in our data flow.</p><p><strong>Techniques to Ensure Data Quality during Ingestion</strong></p><p>Ensuring data quality is like being a diligent inspector; we want to catch any issues with the data before they can cause trouble. In this part, we’ll share techniques to maintain data quality throughout the ingestion process. We’ll talk about performing data validation checks to detect anomalies, outliers, or inconsistencies in the data. Additionally, we’ll explore ways to handle data with errors or discrepancies, such as excluding it from the process or applying data correction methods.</p><h3>Conclusion</h3><p>In this blog, we embarked on a data engineering journey exploring the dynamic duo of PySpark and Snowflake in the realm of data ingestion and warehousing. We uncovered the significance of data ingestion, the challenges it poses, and how PySpark and Snowflake come to the rescue as powerful tools in this process.</p><p>Throughout our adventure, we learned how to ingest data into Snowflake using PySpark, unlocking various methods such as importing data from local files. We witnessed the magic of transforming and cleaning the data, ensuring its quality and compatibility with Snowflake’s schema.</p><p>Creating tables in Snowflake allowed us to organize and structure our data effectively for analysis. Emphasizing the importance of error handling and data quality, we equipped ourselves with strategies to handle errors gracefully and techniques to validate and maintain the integrity of our data during ingestion.</p><p>As we conclude our exploration, we cannot overlook the significance of PySpark and Snowflake in empowering data engineers to achieve seamless data ingestion, data transformation, and warehousing. The synergy of these technologies enhances data management, analysis, and decision-making processes, setting the stage for data-driven success.</p><p>We encourage you, our avid readers, to take these learnings beyond the blog and apply them fearlessly in your data engineering projects. Embrace the power of PySpark and Snowflake to create robust and scalable data solutions, unlocking valuable insights and driving your projects to new heights.</p><p>Thank you for joining us on this enlightening journey, and may your data engineering ventures be filled with triumphs and innovations. Happy data engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e9830c62342" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Box Plots: Detect and remove outliers from distribution]]></title>
            <link>https://medium.com/@akashmishra77/box-plots-detect-and-remove-outliers-from-distribution-a124ee88cf3e?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/a124ee88cf3e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 02:08:08 GMT</pubDate>
            <atom:updated>2023-07-25T02:08:08.841Z</atom:updated>
            <content:encoded><![CDATA[<p>In Machine Learning, certain data points make the model do so well or so bad as compared to other data points. Such data points when examined further, tend to be quite unrealistic in terms of the domain of the data. These data points are called outliers and in this blog, we shall see how we can visualize and then detect and remove the outliers from a dataset.</p><blockquote><em>In statistics, an outlier is a data point that differs significantly from other observations.</em></blockquote><p>These outlier data points can occur due to a range of factors like experimental error, variability in the measurement, documentation error, etc.</p><h3>Five Number Summary and Box Plot</h3><p>One of the most popular ways to detect the outliers in a dataset is by using boxplot visualization and we’ll use this plot to detect and remove we shall use inter-quartile range.</p><p>Let’s see this with the help of an example :</p><p>Suppose, we have a distribution and need to detect and remove the outliers.</p><p>data_distribution = [1, 2, 2, 2, 3, 3, 4, 5, 5, 5, 6, 6, 6, 6, 7, 8, 8, 9,27]</p><p>At first, we need to find the five-number summary of the distribution.</p><p>The 5 number summary which is: minimum, first quartile, median, third quartile, and maximum are as follows :</p><ul><li><strong>Minimum:</strong> It represents the minimum value of the distribution. In this case, it is 1</li><li><strong>first quartile: </strong>It is also known as the 25 percentile. A percentile is a value below which a certain percentage of the observations lie. Here, the first quartile is 3 which means 25% of the observations in the distribution lie below the value 3.</li><li><strong>Median: </strong>A median is a middle number in a sorted list of numbers. In this case, the median is 6.5.</li><li><strong>third quartile: </strong>It is also known as the 75 percentile. Here, the third quartile is 5 which means 75% of the observations in the distribution lie below the value 5.</li><li><strong>Maximum:</strong> It represents the maximum value of the distribution. In this case, it is 27</li></ul><p><strong>Finding the above values using code :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*6z-7OUvX4ezJDNFB.png" /></figure><h3>Box Plot</h3><p>Box Plot is a type of visualization that shows the data from a five-number summary including one of the measures of central tendency. The components of the box plot are shown below :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*XG2sFucPoFMg6NeV.png" /></figure><p>As we can see in the above definition of box plot, if a value is outside the minima and maxima, it is an outlier. So, now let us visualize our distribution and see if we can find any.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*b9DHRrU-K8nib_qf.png" /></figure><p>WOW! We detect the presence of an outlier in our distribution and we need to remove it as this can affect our model if we train it for prediction using machine learning.</p><h3>Removing outliers using Inter-Quartile Range</h3><p>The difference between the upper and lower quartile is known as the inter-quartile range. It is given by :</p><p><strong>Interquartile range = Upper Quartile — Lower Quartile = Q­3 — Q­1</strong></p><p>We then need to calculate lower_boundry and upper_boundry. This would give us the lowest value from our distribution below which the data point is an outlier and the highest value above which our data point is an outlier.</p><p><strong>lower_boundry = q1–1.5 * IQR</strong> <strong>upper_boundry = q3 + 1.5 * IQR</strong></p><p>We then get a range, outside which the data points are outliers :</p><p>range = [lower_boundry, upper_boundry]</p><p>Let’s see this in code :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*7__aB--QiDEFr9YA.png" /></figure><p>Now that we got our range, we can check if any value falls outside this range which is an outlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*uC6lHbhcd2XUnX3y.png" /></figure><p>We can see the value 27 is an outlier in this distribution and the same was also indicated by the box plot visualization. Now all we need is to remove this outlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*vl5RyprIYUH-YGU8.png" /></figure><p>Vola!! We have successfully removed the outlier.</p><h3>Conclusion</h3><p>Outliers are present in almost most of the real-world data sets and they should be dealt with properly. Box plots are one of the most effective and simple ways to visualize the outliers and I hope this blog was able to explain all the logic behind outlier detection and removal.</p><p>Thanks for your read, keep learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a124ee88cf3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Imbalance Datasets in Machine Learning]]></title>
            <link>https://medium.com/@akashmishra77/imbalance-datasets-in-machine-learning-642a1f8be1fe?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/642a1f8be1fe</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 00:33:06 GMT</pubDate>
            <atom:updated>2023-07-24T00:33:06.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*xpCL27p0CyCDA2Yd.png" /></figure><h3>Introduction</h3><p>When we work with real-world datasets for example let us take the case of a fraud detection scenario. Millions of transactions happen all around the world on a daily basis and in the dataset more than 90% of the transactions are correct. A significantly less number of transactions are fraudulent and can create a massive impact on the credit card company. This basically creates an imbalance in the datasets where a majority of the data points belong to one class and we, therefore, need to handle this imbalance.</p><blockquote><em>A classification data set with skewed class proportions is called imbalanced. Classes that make up a large proportion of the data set are called majority classes. Those that make up a smaller proportion are minority classes.</em></blockquote><h3>How to handle it?</h3><p>It is really important to handle the imbalance in a dataset otherwise it’ll give us a sense of being actually correct while being wrong.<br>Suppose, there are 10,000 transactions and out of which 100 are fraud transactions and 9,900 are correct transactions. If we don’t handle the imbalance and just simply write a function that marks every single transaction as a correct transaction, it would still have an accuracy of 99% but would not be handling any fraud transaction at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*8asGw5gmnj-i_2ZM.png" /></figure><p>There are many ways to handle the imbalance but we shall go through the four ways which are being used the most.</p><h3>1- Under Sampling Majority Class</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*J5xoYd4UQSS-A3KU.png" /></figure><ul><li>In this approach, we reduce the number of data points of the majority class by randomly choosing the data and bringing the number down to the same size as the minority class.</li><li>However, this might not be the best approach as we are throwing away lots of data that might be actually useful.</li></ul><h3>2- Oversampling Minority Class</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*grGOfW2Ix9w2E0vR.png" /></figure><ul><li>In this approach, we increase the number of data points of the minority class by duplicating the data and bringing the number up to the same size as the majority class.</li><li>This approach does not lead to the loss of data and the machine learning model building is much more effective.</li><li>However, as we are generating more data this actually increases the learning time and overfitting might be a concern here.</li></ul><h3>3- Oversampling minority class using smote</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*elTVlpQ0o0__Zzt8.png" /></figure><ul><li>SMOTE stands for Synthetic Minority Oversampling Technique. In this approach, we use K-Nearest Neighbours(kNN) algorithm and try to produce some synthetic samples from the data of the minority class.</li><li>The main advantage this has over Oversampling Minority Class is this creates patterns of data using a learning algorithm to get rid of the class imbalance problem.</li></ul><h3>4- Ensemble Techniques</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*36pcFLs_4IrN-Azw.png" /></figure><ul><li>In this approach, we divide out majority class data into batches and the size of each batch is the same as the size of the minority class. We then take the first batch and combine it with the minority class and build a model. We then repeat the same for all the other batches with minority classes and build multiple models. In the end, we take the majority vote from all the models and use that on our dataset.</li></ul><h3>Conclusion</h3><p>The imbalanced dataset problem is very common in the classification problems like fraud transaction detection, cancer prediction, device failure, etc. Every dataset is different and no one strategy is there to handle every scenario so just try those approaches which is the best fit for your data.</p><p>Hope you learned something from this, thanks for reading! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=642a1f8be1fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Find the best model and parameters in Machine Learning]]></title>
            <link>https://medium.com/@akashmishra77/find-the-best-model-and-parameters-in-machine-learning-f0348679db1a?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0348679db1a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Sat, 22 Jul 2023 14:42:01 GMT</pubDate>
            <atom:updated>2023-07-22T14:42:01.386Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*KY_bC8XXhf8kL4BZ" /></figure><p>While learning machine learning in the initial stages, I often used to ask myself questions like :</p><ul><li>I know different types of learning models but is there a way to validate these models which one is better?</li><li>Can I find the best machine learning model and the best parameter in one go without any extra writing code and evaluating results?</li></ul><p>The answer to those questions could have saved a lot of my efforts.</p><p>What I used to do before was using the model validation techniques like :</p><ul><li>train_test_split : It splits the data into one training set and one testing set and model’s accuracy is measured by evaluating it on testing data.</li><li>cross validation : It is more accurate but slower than train_test_split because here the data is divided into multiple folds and training test and testing set are different in different iterations and the average accuracy gives us the accuracy of a model</li></ul><p>But the problem with these techniques was :</p><p>Let’s say I have model_A and model_B and If I had to find which is better, I just trained my model using each of them, found the best accuracy and picked the one with better score. But what is the number of models out of which I have to find the best one is 20? That means I had to train my model 20 times (which consumes lots of time, energy and monitoring) and it becomes a tedious job. 😭</p><p>Then one day, I was really tired with this and decided to search a better and efficient way to find the best model and parameter in one go and I came across the concepts like GridSearchCV and that made my life much easier. 😄 I’m still so happy that I came across this concept and a little sad why I didn’t do this research earlier.</p><h3>Grid Search CV 😎</h3><p>The Grid Search Cross Validation is a technique used for estimating test-set performance of a model. Grid-search is a way to select the best of a family of models, parametrized by a grid of parameters. It basically means, we need to give the names of the models and parameters and it will give us the best model and it’s parameter.</p><p>Let’s see how can we implement it in Python :</p><h3>Step 1</h3><p>Import the GridSearchCV and all the models which we want to validate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*37o6fZzRGKS83T7A" /></figure><h3>Step 2</h3><p>Create a model grid, that is a dictionary which contains all the models and their parameters and out of which we have to find the best one in one go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*SFh1MWEPoFLlg-Pd" /></figure><h3>Step 3</h3><p>Iterate the grid using for loop and append the result of each model to a variable scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*fHByK0-9b3VveCNJ" /></figure><h3>Step 4</h3><p>Create a dataframe out of the variable scores which contains a dictionary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ekoK88JNo99OHPUH" /></figure><h3>Step 5</h3><p>See the results and chose the best set of models and parameters! 😇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*gw6nJcf4YAIdcdEh" /></figure><p>Here, we can see that Support Vector Machine learning model with C=1 and kernel = rbf is the best possible model for this dataset and we shall go with it.</p><h3>Conclusion</h3><p>With these simple steps we can find the best model and parameters in Machine Learning very easily. Special thanks to codebasics channel on youtube where I found this concept for the very first time and it is helping me alot to solve such problems.</p><p>I hope it helps you too! Thanks for reading. 😇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0348679db1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML model evaluation using Cross Validation]]></title>
            <link>https://medium.com/@akashmishra77/ml-model-evaluation-using-cross-validation-950a1e10204e?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/950a1e10204e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Tue, 18 Jul 2023 09:29:59 GMT</pubDate>
            <atom:updated>2023-07-18T09:29:59.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*zc-sgNPXYehOA7OF" /></figure><p>At times, in machine learning we always face the questions like :</p><ul><li>How do we decide which model to use?</li><li>Which model will be more efficient?</li><li>Which parameters are the best for a model?</li></ul><p>Cross Validation is a concept which we can use to validate different models and find the best tuning parameters.</p><h3>How does cross validation works?</h3><ul><li>First, we divide our dataset in k number of partitions called folds</li><li>Then we use one of the folds as testing set and remaining folds as training set</li><li>We calculate the testing accuracy and repeat the steps such that we have used each fold as testing set and others as training set</li><li>After that we take the average of these accuracy and estimate the combined accuracy by finding the mean</li><li>We do the same for multiple models and pick up the best model for our use case</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*YWhgUO2CooRhY_H_" /></figure><p>(<em>Figure showing cross validation with 5 folds</em>)</p><h3>Now, let’s see this with an example where we use iris dataset to predict the flower and see if KNN model is better or logistic regression is a better model to go for.</h3><ul><li>Importing the python modules</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pHV_yPQe6ea9wszf" /></figure><ul><li>Loading the iris dataset<br>Iris dataset is a dataset which contains various features like sepal length/width and petal length/width mapped with a target value of the flower. So basically, here we will be predicting the variety of flower based on it’s characteristics.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*2jrALD0R4E359HXo" /></figure><ul><li>Initializing the features X and target y</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*1OYkKILfn5uoPMIe" /></figure><ul><li>The X is a feature matrix which contains a list of features our model needs to get trained. These are independent variables.<br>The y is the target variable which we want to predict and it is dependent on X. In this example, y can be either of (0,1,2) representing a variety of flower which our model will predict.</li><li>Creating the models and validating the accuracy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*kCSftlouDBSyrRSw" /></figure><ul><li>Here our aim is to create two models Logistic Regression and K-Nearest Neighbors and validate them in order to chose the best among them.<br>We have used 10 fold cross validation(cv=10) to validate both the models.<br>The KNN algorithm (96.66%) is slightly better than logistic regression (94%) after the evaluation so we would be using that to build our machine learning model.</li></ul><h3>Finding the best tuning parameter</h3><p>We can find the best tuning parameters using cross validation. Let’s try to find the most optimal value of k to tune the K-Nearest Neighbours algorithm.</p><ul><li>Firstly, we initialise a range of k values from which we want to chose the most optimal one.</li><li>Then we find the mean accuracy for each run using a different k and chose the best value for k.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*qG3lswJIy3VyW9H8" /></figure><p>In the above example we want to chose the best value of k (1 to 20) and their performance accuracy are stored in k_scores list.</p><ul><li>This plot shows the relationship between k and accuracy score and how k=12 can be the most optimal value of k with 98% accuracy and k=1 can be least optimal value with 95% accuracy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*UF5eHDPvXHXLOwaf" /></figure><h3>Conclusion</h3><p>After validating our dataset with cross validation we would want to use KNN Algorithm to create our model with the best value of the tuning parameter k to be 12.</p><p>This is an introduction to machine leaning model validation and I’ll be posting many more content of machine learning and data science.</p><p>Thanks for reading , I’ll be posting more contents related to this. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=950a1e10204e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Analysis with Python (Part 1)]]></title>
            <link>https://medium.com/@akashmishra77/time-series-analysis-with-python-part-1-f02e071cf374?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/f02e071cf374</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Mon, 17 Jul 2023 12:25:45 GMT</pubDate>
            <atom:updated>2023-07-17T12:25:45.429Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*OaSNJ5zxE_0-xTz8" /></figure><p>Time series analysis basically is a way of studying a sequence of data points collected over a period of time. In this, the data is recorded from time to time consistently and then analyzed further to study the various trends in data with time.</p><h3>Examples of Time Series Analysis</h3><ul><li>Stock Market: Time Series analysis is used in stock markets where the prices of various stocks move up and down from time to time. Analysts record these prices at different instances of time and then do a time series analysis to determine multiple trends in stock prices.</li><li>Sales Report: Companies analyze their sales (annually or semi-annually) just to have an idea of how they are performing over time. The sales data is recorded at various instances of time and helps these organizations to study the various trends and perform some actions after analyzing these time series sales data.</li><li>Weather Forecasting: The Meteorological Department weather by looking at historical trends of whether data were recorded over a period of time.</li></ul><h3>Working with Time Series in Python</h3><p>Pandas library in python has a range of tools for working with dates, times and dime indexed data. There are different types of date and time data like :</p><ul><li><em>Timestamp</em>: It references specific moments in time. On September 17, 2021, at 6:30 A.M</li><li><em>Time period</em>: It references to the length of time between a specific beginning point and end point.<br>Eg: The year 2019</li><li><em>Time delta</em>: It specifies the exact length of time. Ex: 67.88 seconds</li></ul><h3>Coding</h3><h3>Native Python dates and times</h3><p>Python has built-in date and time modules: DateTime and dateutil and we can manually perform some functionalities like :</p><ul><li>Building a date using DateTime type :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*dxlsxz37tfrHIrug" /></figure><ul><li>We can also parse dates from a variety of string formats using dateutil like :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*2iWF7EDIK_WHoRRz" /></figure><ul><li>Once we have DateTime object, we can even print the day of the week :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*selxtFKmQgBC-S2k" /></figure><h3>Numpy’s datetime64</h3><p>Numpy’s datetime64 type gives much more functionalities than native DateTime modules. It encodes dates as 64-bit integers and allows an array of dates to be represented very compactly.</p><ul><li>datetime64 can be created by using NumPy :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_DMeB-92Rxf0kypt" /></figure><ul><li>Once we have the date, we can do various operations on it like, adding the next 10 dates in an array :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*iTWnIWnZajTq1fc3" /></figure><p>np.arrange function basically returns evenly spaced values within a given interval.</p><h3>Dates and Times in Pandas (Intro)</h3><p>Pandas offer even more functionalities than NumPy as it is built on NumPy. It combines DateTime and dateutil with much efficient storage and vectorized interface of NumPy’s datetime64.</p><ul><li>We can parse flexibly formatted string dates and use format codes to output the day of the week as shown above :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*bAXi_qEgQ8HHHDM9" /></figure><ul><li>We can do NumPy style vectorized operations directly on the same object :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*gM1YVNmPTdgBsXlg" /></figure><h3>Pandas indexing with time</h3><p>We can index our data by timestamps. Ex: We can actually create a series that has time-indexed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*uQGKWdtkD9-OKzMI" /></figure><p>Now that this data is in series, we can perform any Series indexing patterns. Let’s select only a range of indexes now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*88lH9RBPoZsZQgNd" /></figure><p>There is also a special indexing operation such as passing the year to obtain a slice of that data as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Nb7b81uP1K1OdBrd" /></figure><h3>pd.date_range()</h3><p>It is a very useful function that returns the DateTime index of a fixed frequency.</p><ul><li>By default, the frequency is 1 day</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*uaAB7tgzmrFl4oqM" /></figure><ul><li>We can find business days that excludes weekends by giving freq = ‘B’</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*WurqS9rnQbm9iBdE" /></figure><p>We can perform much more pandas date time operations just by playing around with datatypes and variables.</p><h3>Conclusion</h3><p>This was part 1 of the Time Series Analysis with Python where we have seen what is time series, where we use it, and an introduction to how time and date variables work in python.</p><p>So, stay tuned, and thanks for your read! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f02e071cf374" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Python and the beautiful road ahead]]></title>
            <link>https://medium.com/@akashmishra77/getting-started-with-python-and-the-beautiful-road-ahead-4bbd8ccf0d3d?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bbd8ccf0d3d</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[python-beginner]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-basic-questions]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Sat, 15 Jul 2023 02:11:32 GMT</pubDate>
            <atom:updated>2023-07-15T02:11:32.538Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*gVmv1ZPDmgbJhByE" /></figure><p>Learning to code has the power to create something that can bring a good change in people’s lives. Anyone who knows how to code can make such creations. A century earlier, it was not possible as resources were rare but we are one lucky generation to have that opportunity.</p><p>With this thought, let me introduce you to one of the programming languages of present and future and that is Python.</p><h3>What really is Python programming in simple words? 😌</h3><p>Python is simply a programming language which basically means through this, we give our computers a set of instructions to execute.</p><p>Python is used in creating a variety of stuffs like websites, machine learning models, automating tasks, game development, web scraping, data science, business applications etc.</p><p>It also has a wide range of libraries that we can import whenever we need some particular feature. It makes python flexible and allows us to use various libraries to create some cool stuffs.</p><p>For now, this is enough.</p><h3>How is Python compared to other languages?</h3><ul><li>Python is easy to learn! The syntax is relatively simple compared to programming languages which makes it easy to get started with even with no prior experience in coding.</li><li>Python is free and open-source which basically means it’s code is freely available to the public on repositories (a place to store code on internet)</li><li>Python might be slow than some programming languages like Java and C but it’s memory management and lesser lines of code makes up for it. :)</li></ul><h3>Some Applications of Python and libraries used :</h3><h3>1- Web Development</h3><p>Web Development basically means building a frontend (User Interface that interacts with the clients or users) and a backend (which contains business logic and interacts with a database).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*1Haj2LqmnXMxW3Mq" /></figure><p>Some popular python web frameworks :</p><p><strong>1- Flask</strong> : Flask is a lightweight microframework, which means that we add extensions and libraries that we need as per our code requirements without being provided by the framework from start. It makes our application highly flexible.<br>Flask is highly used in industries for web development and REST APIs.</p><p><strong>2- Django</strong> : Django is a Python web framework that offers is a high-level framework and involves fast and clean development and design. It is very fast, flexible and secure. It is also used a lot in various companies like Instagram and Sportify.</p><h3>2- Machine Learning</h3><p>Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. It is the technology of present and future as the data available is vast and we can predict the future by analysing the past data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ntAp1RvzfNUyKSSD" /></figure><p>Some Machine Learning Frameworks :</p><p><strong>1- Pandas</strong> : Pandas is a python library used for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.</p><p>In simple words, if you want to do anything data related in Python, Pandas is the backbone of it and it is really easy to learn!</p><p><strong>2- Matplotlib</strong> : It is a data visualization library used to make various plots and charts based on the data. The plots can be 2D or 3D and it gives functionalities of variety of charts like bar graph, pie chart, histograms, funnel chart, line plot etc. in a very few steps.</p><p><strong>3- Scikit-learn:</strong> Scikit-learn is probably the most useful library for machine learning in Python. This library contains a lot of efficient tools for machine learning and statistical modelling including classification, regression, clustering and dimensionality reduction.</p><p><strong>4- Tensorflow :</strong> TensorFlow is a Python library for fast numerical computing. It was created and released by Google. It is a foundation library that can be used to create Deep Learning models directly or by using wrapper libraries that simplify the process built on top of TensorFlow.</p><p><strong>5- Keras :</strong> Keras is a powerful and easy-to-use free open source Python library for developing and evaluating deep learning models. It wraps the efficient numerical computation libraries Theano and TensorFlow and allows you to define and train neural network models in just a few lines of code.<br>Yeah the last two are pretty cool once you get into it.</p><h3>Web Scraping</h3><p>A web scraper is a specialised tool designed to accurately and quickly extract data from a website for data analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/0*S8lFHmdwVc1a5udN" /></figure><p>Some Web Scraping Frameworks :</p><p><strong>1- Scrapy</strong> : Scrapy is an open source web scraping library for python which is used to used in monitoring and mining data, as well as automated and systematic testing. It is actually used when we want to do a large scale web scraping and perform some tests along the way.</p><p><strong>2- Beautiful Soup :</strong> It is used to extract useful information from HTML and pages. It is based on parsing and parsers helps us to get the data from such files. It can also detect the type of encoding a web page has and can get more reliable information.</p><p><strong>3- Requests :</strong> It is an HTTP library which is used for API purposes, which allows the user to sent requests to the HTTP server and GET response back in the form of HTML or JSON response.</p><p><strong>4- Urllib :</strong> Urllib is a Python library that allows the developer to open and parse information from HTTP or FTP protocols. It comes with python so you might not have to install it! :)</p><h3>Game Development 😎</h3><p>The increase and advancements of gaming industry is Python has some frameworks which can be used for game development as well. Some of the very popular video games are coded in python like Civilization-IV, Disney’s Toon town Online etc. have been built using Python. and here is some of the libraries that can get you started with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*5A1b5NZRZqnTfGQT" /></figure><p><strong>1- Pygame </strong>: It is an open source python library which enables users to build fully featured games as well as multimedia programs with Python programming. It is highly portable and runs on nearly every platform and operating system.</p><p><strong>2- Panda3D</strong> : It is an open source python library written in Python and C++ and it is used for Realtime 3D games, visualizations, simulations, experiments and more.</p><p><strong>3- PyKyra</strong> : It is one of the fastest game development frameworks for Python. Along with the standard features of this framework, it also supports MPEG video and sound such as mp3, ogg vorbis, .wav, etc.</p><p>There are many other game development libraries like : Pyglet, Python-Ogre, PyOpenGL, Kivy, Cocos2d etc which you can checkout as well.</p><h3>Python is also used for Data Science, Business Applications, Creating GUIs, Automating tasks etc. The scope is endless!</h3><h3>Opportunities with Learning Python😇</h3><ul><li>Data Science is a booming field currently and as we have lots of data collected now that it has ever been, companies need professionals to study the data, do predictive analysis and share insights that how they can improve the product and Python is currently the best language to perform this skill and it is a job that keeps growing with time.</li><li>Artificial Intelligence, Natural language Processing (NLP) and Big Data are technologies of the future and they bank heavily on their implementation in python.</li><li>Many leading organizations like Instagram, Spotify, Amazon, Disney, YouTube, Facebook etc. have a large part of their codebases in python.</li></ul><h3>Conclusion</h3><p>Python is the language of the present and future. It is going no-where. If you are already learning python that’s really cool and if you’re new to this then you can deep dive learning this programming languages where opportunities are endless and resources are easily available!</p><p>Thanks for reading 😄 Happy Coding! 😌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bbd8ccf0d3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Common SQL operations to perform using Pandas]]></title>
            <link>https://medium.com/@akashmishra77/10-common-sql-operations-to-perform-using-pandas-1b856fbc6fb2?source=rss-5184c919d895------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b856fbc6fb2</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-manipulation]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Akash Mishra]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 17:20:32 GMT</pubDate>
            <atom:updated>2023-07-13T17:20:32.250Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*me7dmUvVPEZUUzc7" /></figure><p>One of the key features of being a data analyst is to query the data from files, databases etc to perform some data manipulation or visualisation and sometimes it’s much better if we can do it directly through code instead of looking into the database tables over and over again. Pandas is a python library which can store query results in variables called “dataframes” and it helps us to perform data manipulations, visualisation and can convert our results back in the databases or files as write operations.</p><p>This blog basically describes how one can use 10 basic SQL operations using the pandas library.</p><h3>Dataset which we’ll be using as an example :</h3><p>We will use two datasets :</p><ul><li>Marks of various students in 3 subjects. (Primary keys : class and roll_no)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/0*Ze2NokEZH-cDNkQ2" /></figure><ul><li>Personal details of students. (Primary keys : class and roll_no)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/0*qMtxtMOT9YRA_5D0" /></figure><h3>1. Selecting the data</h3><ul><li>In sql we can select the data of various columns using this querry :<br>SELECT name,roll_no,phone FROM student_details;</li><li>This is how it’s done in pandas :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*F6t2dm1TSl0Nqxct" /></figure><h3>2. Using aggregate functions</h3><ul><li>Now let’s find the maximum marks a student of class 7th got in maths<br>SELECT max(maths) FROM student_marks WHERE class=7;</li><li>Pandas has several functions like min(),max(),mean() etc that works on a column, which one can use to achieve the above operation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*LzOQjwHtFoT3PBrw" /></figure><h3>3. Order By clause</h3><ul><li>This clause in SQL is used for sorting the values in ascending or descending order Query to sort names in ascending order :<br>select * from student_details order by name;</li><li>Query to sort names in descending order :<br>select * from student_details order by name desc;</li><li>In pandas we can do the same with df.sort_values() function This function takes in ‘column_to_be_sorted’, ascending = True for ascending order and ascending = False for descending order sorting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*6J-BJSP3Zm1YnXB6" /></figure><h3>4. Group By clause</h3><ul><li>Let’s count the number of students in different classes<br>select class,count(*) from student_details group by class;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*fiXbCcHASQ0q9oUU" /></figure><ul><li>In pandas we can do the same with df.groupby() function</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*LaACcYyRJ8gRURqQ" /></figure><h3>5. IN and NOT IN</h3><ul><li>Let’s select only those rows where blood group is A or B<br>select * from student_details where blood_group in (&#39;A&#39;, &#39;B&#39;) ;</li><li>In pandas we can do the same with isin() which works on a particular colum</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*dg7hBWTyNlYsfQ6q" /></figure><ul><li>For NOT IN we can just add a negation symbol “~” in the pandas condition :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*MeTj979QvIOPPtRi" /></figure><h3>6. Joins</h3><ul><li>Join statement to get all the details and marks of students together<br>SELECT * from student_details a JOIN student_marks b ON (a.roll_no=b.roll_no AND a.class=b.class);</li><li>In pandas we can do the same using merge() function and we can specify the type of join like inner,outer,left,right as well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*OYzWezglJrAskM8t" /></figure><h3>7. Creating new column using existing ones</h3><ul><li>Now, let’s add a new column total which adds marks in 3 subjects for each student<br>SELECT class,roll_no,science,maths,english,science+maths+english as total FROM student_marks;</li><li>Just add df[‘new_column’] and fill it’s value using other columns or with their operations</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*QeXMgxz4ioPuWK46" /></figure><h3>8. Selecting data conditionally</h3><ul><li>Let’s try filtering the data using multiple conditions.<br>select * from student_details where class=7 and gender=&#39;M&#39; and blood_group=&#39;O&#39;;</li><li>We can provide multiple conditions in braces () followed by AND (&amp;) or OR (|)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*t9lhF4mrv7R_thQc" /></figure><h3>9. Insert and Update</h3><ul><li>Inserting a new row in student_details table :<br>insert into student_details(name,roll_no,class,gender,blood_group,phone) values (&#39;Seth&#39;,38,7,&#39;M&#39;,&#39;AB&#39;,223344);</li><li>In pandas we can do the same by adding the row at the end of the dataframe</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Saw9ekORVcdimFHG" /></figure><ul><li>Updating a row in student_details table :<br>UPDATE student_details SET name = &#39;Seth Andrews&#39; WHERE class=7 AND roll_no=38;</li><li>In pandas, we can update the value conditionally like this :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Umn0k0qhjFVf-3Tp" /></figure><h3>10. Delete</h3><ul><li>Deleting a row in student_details table :<br>DELETE FROM student_details WHERE name = &#39;Seth Andrews&#39; and class = 7;</li><li>In pandas we can do the same with the help of drop() function :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*cwc47bmnpVPQ6UYR" /></figure><h3>Conclusion</h3><p>This is just introduction to how one can execute basic SQL operation using pandas and there are many more operations of SQL which can be easily done using pandas. The datasets and python file I have used can be found here : <a href="https://github.com/akash19x/Tutorials/blob/main/SQL%20operations%20in%20Pandas.ipynb">Python Notebook</a> I’ll be posting much more content of python and data science in my coming blogs.</p><p>Thanks for your time! :) Checkout my youtube video on the same topic : <a href="https://www.youtube.com/watch?v=sUI00FYS0YA">youtube.com/watch?v=sUI00FYS0YA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b856fbc6fb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>