<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kanchanakanta on Medium]]></title>
        <description><![CDATA[Stories by Kanchanakanta on Medium]]></description>
        <link>https://medium.com/@kanchanakanta?source=rss-ee96734c02fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ucls4cp8luAQV8Fs</url>
            <title>Stories by Kanchanakanta on Medium</title>
            <link>https://medium.com/@kanchanakanta?source=rss-ee96734c02fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:22:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kanchanakanta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[INTRODUCTION TO DEEP LEARNING]]></title>
            <link>https://medium.com/@kanchanakanta/introduction-to-deep-learning-749e09cc2142?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/749e09cc2142</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 15:23:07 GMT</pubDate>
            <atom:updated>2026-04-21T15:23:07.841Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>INTRODUCTION TO DEEP LEARNING</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*6RNzC1oR5fL4nevAnpVNoQ.png" /><figcaption><strong>Fig. 1: Structures of the human brain and the Artificial Neural Network</strong></figcaption></figure><p><strong>Learning from the World Around Us:</strong></p><p>Look at these familiar faces — actors, sports personalities, and global icons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*ZXy8ooprOnocQkElOT70Kw.png" /><figcaption><strong>Fig.2: Known personalities</strong></figcaption></figure><p>Did you recognize who these are? Yes, they are the Indian actors Nani, Surya, Allu Arjun, Dhanush, the recent Oscar-winning actor Leonardo DiCaprio, BTS members Jimin and Suga, Ms. Emma Watson, Virat Kohli, and Sridevi. How could you recognize? From our childhood, we have seen their images in movies, on television, and on social media. Over time, our brain learns to identify their unique features such as face shape, expressions and voice. This ability to learn from experience and recognize patterns is a natural form of intelligence.</p><p>Now, imagine if machines could do the same.</p><p>Just as our brains recognize faces from experience, machines can also be trained to recognize patterns. This ability is part of Artificial Intelligence.</p><p>Artificial Intelligence is a field of Computer Science that focuses on creating machines that can perform tasks that require human intelligence, such as understanding language, recognizing images, making decisions, and solving problems. As humans, we perform these activities without conscious effort. AI aims to give such abilities to machines so that they can assist humans and make our lives easier.</p><p>Today, AI is already a part of our everyday lives. When you unlock your phone using your face, talk to voice assistants like Google Assistant, or use navigation apps, we are interacting with AI systems.</p><p>Now, let us look at some common applications where AI is used in real life.</p><p>· Face recognition for unlocking in smart mobiles,</p><p>· Interacting with Google’s voice assistant, which understands voice commands and can answer questions and send messages,</p><p>· Self-Driving cars, which can drive themselves using AI, sensors and cameras, and</p><p>· Chatbots are AI-based programs that interact with humans through text or voice, simulating real conversations. These are some of the example applications that use AI features.</p><p><strong>How do machines actually learn?</strong></p><p>Instead of programming every rule manually, we allow machines to learn from data. This approach is called Machine learning (ML).</p><p>Machine Learning is a subset of AI that enables systems to learn patterns from data and make decisions without being explicitly programmed for every task.</p><p>For example, machine learning is used in applications like</p><p>· Email Spam Detection, Credit card fraud detection, etc.</p><p>· Basic Predictions like House price prediction, weather prediction, students’ performance predictions, etc.</p><p>· Medical diagnosis</p><p>· Customer segmentation, etc.</p><p>Here, in all applications, the problem is well defined, data is structured, and patterns are not very complex.</p><p>Machine learning works well when data is structured, problems are well defined, and patterns are not very complex.</p><p><strong>Where Machine Learning falls short?</strong></p><p>While Machine Learning is powerful, it has limitations when dealing with real-world complex data.</p><p>Consider the application of face recognition in smartphones, which does not consist of simple data. The human face can vary due to lighting, viewing angles (front, side), facial expressions and accessories like glasses, beard, which makes face recognition a complex image problem.</p><p>In traditional Machine Learning, humans must manually define the features, like edges, shapes, distance between facial parts, for identifying the face. This process is time-consuming and often not accurate in complex situations.</p><p>Consider another example, Google Assistant, where people speak in different accents, different speeds, and background noises, which makes it a very complex problem and using machine learning, humans need to define voice variations, natural conversations, and context, which results in less accurate understanding.</p><p><strong>Limitations of Machine Learning</strong></p><p>· Requires manual feature selection</p><p>· Struggles with complex and unstructured data</p><p>· Depends heavily on high-quality data</p><p>· Limited Accuracy for Complex Problems</p><p>· Cannot automatically learn deep patterns from raw data</p><p>With the limitations of machine learning, it is hard to define all possible features, and it is not accurate for real-world conditions.</p><p>To overcome these limitations, we need a system that can automatically learn features from data, and handle complex data. This leads to the concept of Deep Learning.</p><p><strong>What is deep learning?</strong></p><p>Deep Learning is a subset of Artificial Intelligence that uses multi-layered neural networks to automatically learn representations of data and solve complex problems such as image recognition, speech processing, and natural language understanding. The term “deep” refers to the presence of multiple layers in the network, where each layer processes information step by step, similar to how the human brain works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*3kGJqjO4a-7BL2_Ot1mq9A.png" /><figcaption><strong>Fig.3: Human brain neuron Vs Artificial Neural Network</strong></figcaption></figure><p>Deep Learning models are inspired by the structure and functioning of the human brain. Just like neurons in the brain pass signals to each other, artificial neural networks pass information through multiple layers to learn patterns. The key difference is that Machine Learning requires manual feature extraction, whereas Deep Learning learns features automatically.</p><p><strong>Working of Deep Learning Systems:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*BvcUWIJjz2iqLQW0hKAXFw.png" /><figcaption><strong>Fig.4 Deep Learning working process</strong></figcaption></figure><p>Deep Learning works by learning patterns from data using multiple layers, similar to how the human brain processes information step by step, as shown in Fig. 4.</p><p>Step 1: The first layer in the model receives the input data such as image, text or voice and passes to the next layer.</p><p>Step 2: Data passes through many layers and each layer learns something different.</p><p>Step 3: Each connection has a weight showing the importance and the model adjusts these weights while learning.</p><p>Step 4: The output layer makes the final decision, giving the output.</p><p>Step 5: If the answer is wrong, the model corrects itself by adjusting the weights.</p><p>Step 6: This process repeats many times to improve the accuracy.</p><p><strong>Applications of Deep Learning:</strong></p><p>· Image Recognition</p><p>· Speech Recognition</p><p>· Natural Language Processing applications like Chatbots, Google Translate</p><p>· Self-Driving Cars detect roads, vehicles and pedestrians makes driving decisions automatically.</p><p>· Recommendation Systems suggests content based on user behavior like movie recommendations in Netflix, YouTube etc.</p><p>· Healthcare applications to detect diseases from medical images and help in diagnosis and treatment planning.</p><p>· Robotics to help robots see, learn and make decisions used in industries and automation.</p><p>· Entertainment and Gaming, improving graphics and animations, and creating realistic characters, etc.</p><p><strong>Advantages of Deep Learning:</strong></p><p>· <strong>Automatic Feature Learning: </strong>Learns patterns directly from raw data, which overcomes the limitation of manual definition of features as in machine learning.</p><p>· <strong>High Accuracy:</strong> Deep Learning is useful for complex tasks like image recognition, speech recognition etc. and provides very accurate results.</p><p>· Deep Learning handles complex data.</p><p>· Deep Learning can process huge datasets efficiently and improve performance with more data.</p><p>· It learns from mistakes and becomes more accurate with training.</p><p>· Handles variations like lighting, noise, etc. and works well in dynamic environments.</p><p><strong>Limitations of Deep Learning:</strong></p><p>While Deep Learning is powerful, it also has several important limitations.</p><p>· Needs huge datasets to perform well. With small data, it gives poor accuracy.</p><p>· Requires powerful hardware like GPUs/TPUs, and training becomes expensive and time-consuming.</p><p>· Difficult to understand how the model makes decisions.</p><p>· Training Deep Models can take long training time hours, days, or even weeks.</p><p>· Overfitting problem: May perform very well on training data and poorly on new/unseen data.</p><p>· Needs skilled expertise for model design, tuning parameters and data handling.</p><p>· Uses a lot of energy and is not ideal for low-power devices.</p><p>· Hard to identify the errors because of its complex architecture.</p><p><strong>Relationship between Artificial Intelligence, Machine Learning and Deep Learning</strong></p><p>Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning (DL) are closely related concepts in the field of computer science. They are often used together but represent different levels of capability.</p><p>AI refers to machines that are designed to perform tasks that normally require human intelligence, such as reasoning, learning, and decision-making.</p><p>ML focuses on enabling machines to learn from data and improve their performance without programming for every task.</p><p>DL is a subset of ML. It uses multi-layered neural networks to automatically learn complex patterns from large amounts of data. “In simple terms: Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*Xs8sLj06A0abAw86jXy0NA.png" /><figcaption><strong>Fig. 5 Relationship between AI, ML and DL</strong></figcaption></figure><p>As the technology continues to advance, Deep Learning will play a crucial role in building intelligent systems and solving complex real-world problems across industries.</p><p>Note: All the figures are generated using ChatGPT</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=749e09cc2142" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EVALUATION METRICS FOR CLUSTERING]]></title>
            <link>https://medium.com/@kanchanakanta/evaluation-metrics-for-clustering-884ee5398f9b?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/884ee5398f9b</guid>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Sun, 29 Mar 2026 16:16:26 GMT</pubDate>
            <atom:updated>2026-03-29T16:16:26.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nmlwqJ1uAdw3OxKu.png" /><figcaption>Clustering Evaluation</figcaption></figure><p>Clustering is an unsupervised machine learning algorithm which groups similar data into clusters. Clustering works on unlabeled data or data without ground truth labels. The goal is to ensure that the points within the same cluster are more similar to each other than to points in different clusters.</p><p>The quality of these clusters is measured using c<strong>lustering evaluation parameters</strong> or <strong>clustering validation metrics</strong>. The quality of the clusters is measured based on how well the clusters satisfy the two criteria.</p><ol><li><strong>Intra-cluster similarity</strong> also known as <strong>cohesion or compactness</strong> that measures how similar the data points are within the same cluster. It should be high for good clustering.</li><li><strong>Inter cluster similarity</strong> also known as <strong>separation or scatter</strong> that measures how different or dissimilar the data points in one cluster are from the data points in other, different clusters. It should be low for good clustering.</li></ol><p><strong>Clustering evaluation:</strong> Clustering evaluation is the process of measuring how well a clustering algorithm has grouped data points into meaningful clusters. It is the set of techniques and metrics used to assess the performance of clustering algorithms in machine learning. These metrics are categorized based on whether they rely on external reference information (ground truth) or solely on the data itself.</p><p>TYPES OF CLUSTERING EVALUATION METRICS:</p><p>There are two types:</p><ol><li>Internal Evaluation Metrics</li><li>External Evaluation Metrics</li></ol><p><strong>INTERNAL EVALUATION METRICS</strong></p><p>Internal evaluation metrics are used to assess the quality of clustering without requiring ground truth labels. These metrics evaluate how well the clusters are formed based on the data itself and the resulting clusters — focusing on compactness, separation, and density.</p><p><strong>Important Internal Evaluation Metrics:</strong></p><ol><li>Silhouette Coefficient</li><li>Davies-Bouldin Index (DBI)- A ratio-based metric</li><li>Calinski-Harabasz Index — A Variance based metric</li><li>Dunn Index</li></ol><p><strong>1.Silhouette Coefficient: </strong>It is an internal clustering evaluation metric used to measure how well each data point fits within its assigned cluster compared to other clusters. It reflects both cluster cohesion and separation. It helps to determine ( i ) How well the clusters are formed, and (ii) Whether the number of clusters (k) is appropriate.</p><p>Formula:</p><p>For each data point i silhouette coefficient is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*K3b8PtNqCBl6dwJ5xmrtPw.png" /></figure><p>Where</p><p>a(i)= Average distance of point i to all other points in the same cluster (measures cohesion).</p><p>b(i)=Minimum Average distance of point i to points in other clusters (measures separation)</p><p><strong>Overall Silhouette Score:</strong></p><p>To evaluate the entire clustering result with n number of data points</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*NiUIUfiapgDYHyj8eAPRng.png" /></figure><p>Interpretation of Silhouette Value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*yI39GAH86o802gOt2_xMHA.png" /></figure><p><strong>Higher average S</strong> (closer to 1) → better clustering.</p><p><strong>Lower average S </strong>(close to 0 or negative) → poor or overlapping clusters.</p><p>Example: You have 6 data points clustered into two groups:</p><ul><li>Cluster A: Points at (1, 2), (2, 1), (1, 1)</li><li>Cluster B: Points at (8, 9), (9, 8), (8, 8)</li></ul><p>Let’s calculate the silhouette score for point (1, 2):</p><p>Step 1: Let us use Euclidean distance as the distance metric and the formula is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*rz4ka4HkZwI4W5PSOqRKdw.png" /></figure><p>step2 : Calculate the average distance of a(i) for point (1,2):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*_d3KdfJ7dtQ-vXbGHuP_Ng.png" /></figure><p>step 3: Computing b(i)-average distance to points in other clusters. here cluster B (8, 9), (9, 8), (8, 8)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*-TEBZL7-97GYWSQNzWT0Sg.png" /></figure><p>step 4: Silhouette score</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*e87WLfR09BNdkDcn49TBow.png" /></figure><p>Silhouette score for the point (2,1):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*i7UKtrAAlqrqptJ9z3KqfA.png" /></figure><p>Silhouette score for the point (1,1):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*pLaSlnhD8dJjC_pM6268vQ.png" /></figure><p>Average Silhouette score for cluster A= S(A)= (0.87+0.88+0.90)/3 =0.883</p><p>Silhouette score for cluster B points:</p><p>S(8,9):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*90ShEHCT1_LKnIA65XBplw.png" /></figure><p>S (9,8):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*u3jGLE18wCKB-ESndQG2Jg.png" /></figure><p>S (8,8):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*WGCwWsIXQtQesxsP1ZBvwg.png" /></figure><p>Average Silhouette for cluster B :(0.88+0.88+0.894)/3 =0.885</p><p>Overall Silhouette score = S= S(A)+S(B)/2</p><p>= (0.883+0.885)/2= 0.884</p><p>Since the Silhouette score is close to 1, we can say that clusters are well-separated and compact indicating a very good clustering result.</p><p><strong>When to use or when to not use Silhouette coefficient?</strong></p><p>Silhouette coefficient is used when</p><ul><li>We performed unsupervised clustering</li><li>We compare or validate clusters</li><li>We are deciding optimal number of clusters (k)</li></ul><p>Silhouette coefficient is not used when</p><ul><li>Clusters are not spherical</li><li>When we already know the class labels.</li></ul><p><strong>2.Davies Bouldin Index:</strong></p><p>The Davies Bouldin index is an internal clustering evaluation metric used to measure how well clusters are separated (inter cluster distance) and how compact (intra cluster distance) they are without needing ground truth labels. The Davies Bouldin Index (DBI) evaluates the average similarity between each cluster and its most similar one, where the similarity is defined as the ratio of the sum of intra-cluster distances to inter-cluster distance.</p><p>Formula for calculating DBI is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*4WQdiA1_ExpTH_fyAYuv1Q.png" /></figure><p>Where:</p><ul><li>N is the number of clusters</li><li>sᵢ is the intra-cluster distance or scatter or dispersion of cluster i. This is the average distance between all points in cluster i and its centroid.</li><li>sⱼ is the intra-cluster distance of cluster j.</li><li>dᵢⱼ is inter-cluster distance or separation between the centroids of cluster i and cluster j.</li><li>The term (sᵢ + sⱼ)/dᵢⱼ is the similarity measure Rᵢⱼ between clusters i and j.</li></ul><p>DBI interpretation:</p><p>The goal is to have clusters that tight i.e. low sᵢ and sⱼ and well- separated i.e. high dᵢⱼ.</p><ul><li>A lower DBI value is better clustering i.e. clusters are compact and well-separated.</li><li>Higher DBI value is poorer clustering i.e. clusters overlap or are not well-defined.</li></ul><p>Steps to compute DBI:</p><ol><li>Compute the centroid of each cluster.</li><li>Compute intra-cluster distances sᵢ.</li><li>Compute inter-cluster distance dᵢⱼ.</li><li>For each cluster i, find the maximum ratio (sᵢ + sⱼ)/dᵢⱼ across all other clusters j.</li><li>Take the average of all these maximum values i.e. the DBI.</li></ol><p>Example:</p><p>Let there are 3 clusters 1,2,3</p><p>C1 = {(1,1), (2,1), (1,2)}, C2= {(5,4), (6,5), (5,5)}, C3= {(9,8), (9,9), (8,9)}</p><p><strong>Step1: compute centroid of each cluster</strong></p><p>centroid1= (1+2+1)/3, (1+1+2)/3 = (1.33,1.33)</p><p>centroid2= (5+6+5)/3, (4+5+5)/3 = (5.33,4.67)</p><p>centroid3 = (9+9+8)/3, (8+9+9)/3 = (8.67,8.67)</p><p><strong>step2: Compute intra-cluster distance Si</strong></p><p>for each cluster i</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*9gzVEq8bTTfKAJAyMH3ljg.png" /></figure><p>Where</p><ul><li>nᵢ = number of points in cluster i</li><li>x=a data point in cluster i</li><li>cᵢ= centroid of that cluster</li><li>distance= usually Euclidean distance and</li><li>Sᵢ is the average distance of all points in that cluster from the cluster centroid</li></ul><p><strong>for cluster 1 :</strong></p><p>distance from (1,1) to centroid 1 =√ (1–1.33) ² + (1–1.33) ² = 0.47</p><p>distance from (2,1) to centroid 1 =√ (2–1.33) ² + (1–1.33) ² = 0.75</p><p>distance from (1,2) to centroid 1 =√ (1–1.33) ² + (2–1.33) ² = 0.75</p><p>S1= average of all these distances = (0.47 + 0.75 + 0.75)/3 = 0.66</p><p><strong>for cluster 2:</strong></p><p>distance from (5,4) to centroid 2=√ (5–5.33) ² + (4–4.67) ² = 0.75</p><p>distance from (6,5) to centroid 2=√ (6–5.33) ² + (5–4.67) ² = 0.75</p><p>distance from (5,5) to centroid 2=√ (5–5.33) ² + (5–4.67) ² = 0.47</p><p>S2= average of all these distances = (0.75 + 0.75+0.47)/3 = 0.66</p><p><strong>for cluster 3:</strong></p><p>distance from (9,8) to centroid 3=√ (9–8.67) ² + (8–8.67) ² = 0.75</p><p>distance from (9,9) to centroid 3=√ (9–8.67) ² + (9–8.67) ² = 0.47</p><p>distance from (8,9) to centroid 3=√ (8–8.67) ² + (9–8.67) ² = 0.75</p><p>S3= average of all these distances = (0.75 +0.47+ 0.75)/3 = 0.66</p><p><strong>From this we can interpret that all clusters have equal intra cluster distance i.e. they have equal compactness. If even one cluster had points spread far from its centroid, that cluster’s Si would be larger, overall clustering quality would be reduced.</strong></p><p>Step 3: Computer inter-cluster distances dᵢⱼ</p><p><strong>Inter-cluster distance</strong> is the <strong>distance between two clusters</strong>, usually measured as the distance between their <strong>centroids</strong> (cluster centers). It tells <strong>how far apart two clusters are</strong>.</p><p>Formula for finding the inter — cluster distance is :</p><p>For clusters Ci and Cj ​ with centroids ci ,cj​:</p><p>Mij=distance(ci, cj)</p><p>In Davies–Bouldin Index (DBI), this is specifically the <strong>Euclidean distance</strong> between centroids dᵢⱼ=</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/1*xKN-LmUEKwtPYsUPQmGmaA.png" /></figure><ul><li>If <strong>inter-cluster distance is large</strong>, clusters are <strong>well separated</strong> → good clustering</li><li>If <strong>inter-cluster distance is small</strong>, clusters are <strong>close / overlapping</strong> → poor clustering</li></ul><ol><li>Distance between cluster 1 and cluster 2</li></ol><p>substituting the values of centroids d12=</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*uOJj4JjAR75f7fcX8Ah7lw.png" /></figure><p>d12=5.21</p><p>Distance between cluster 1 and cluster 3 ,d13=</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*ZQ_LRZXFnT8gVbkfS8ixJA.png" /></figure><p>d13=10.38</p><p>distance between cluster 2 and cluster 3 d23=</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*uv835xAhknoVQZu9YZ96yg.png" /></figure><p>d23= 5.21</p><p>cluster1 1 and 3 are very far apart and well separated.</p><p>So, Cluster 1’s nearest neighbor is cluster 2 and cluster 3’s nearest neighbor is cluster 2.</p><p>Step 4: For each cluster i, find the maximum ratio (sᵢ + sⱼ)/dᵢⱼ across all other clusters j.</p><p>1) Compute all pairwise ratios (Si+Sj)/dij</p><p>for cluster 1 ,i=1,j=2,3</p><p>R12=(s1+s2)/d12= (0.66+0.66)/5.21 = 0.2533589</p><p>R13= (s1+s3)/d13 = (0.66+0.66)/10.38 = 0.12716</p><p>for cluster 2, i=2,j=1,3</p><p>R21=(s2+s1)/d21= (0.66+0.66)/5.21 = 0.2533589</p><p>R23= (s2+s3)/d23 = (0.66+0.66)/5.21 = 0.2533589</p><p>in this example cluster 1 and cluster 3 are at the same centroid distance from cluster 2.</p><p>for cluster 3, i=3,j=2,3</p><p>R31=(s3+s1)/d31= (0.66+0.66)/10.38 = 0.12716</p><p>R32= (s3+s2)/d32 = (0.66+0.66)/5.21 = 0.2533589</p><p>2) we need to find maximum ratio (sᵢ + sⱼ)/dᵢⱼ across all other clusters j.</p><p>R1​=max (0.2533589,0.12716) =0.2533589 → <strong>best-match (most similar) is cluster 2</strong> (smallest separation → largest ratio).</p><p>R2=max⁡ (0.2533589, 0.2533589) =0.2533589 → ties (either cluster 1 or 3).</p><p>R3=max ⁡(0.12716, 0.2533589) =0.2533589 → <strong>best-match is cluster 2</strong>.</p><p>So, each cluster’s worst-case ratio is ≈ <strong>0.2533589251</strong>.</p><p><strong>Step 5: Take the average of all these maximum values i.e. the DBI</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*58ewHnNNiY6HtxHycuhx1A.png" /></figure><p>DBI= 0.25336</p><p>Interpretation</p><ul><li>DBI is <strong>low (≈0.25)</strong> → clusters are <strong>compact</strong> (small Si​) and <strong>well-separated</strong> (reasonably large dij</li><li>The reason the max ratio for each cluster picked the pair with dij=5.21 is that that separation is the <strong>smallest</strong> for each cluster’s candidate neighbors — a smaller denominator makes the ratio larger, so that neighbor contributes the worst-case (largest) similarity.</li></ul><p><strong>3. The Calinski — Harabasz Index(CHI) </strong>is also called as Variance Ratio Citerion (VRC). CHI is a metric used to evaluate the quality of clustering results. It assesses how well-separated and compact the clusters are. A higher CHI score indicates better clustering. That means the clusters are well-defined and distinct from each other. The index is calculated based on the ratio of between cluster variance to within-cluster variance.</p><p>For a dataset with N samples, K clusters, and d-dimensional features, the typical formula is:</p><ul><li>Let xi​ be a data point, and let:</li><li>μ be the global mean of the data.</li><li>μk be the mean of cluster k.</li><li>Ck be the set of points in cluster k, and ∣Ck∣ its size.</li></ul><p><strong>Within-cluster dispersion (sum of squares within clusters):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*iRWHKSSZfNPhM4SPCyhvPw.png" /></figure><p><strong>Between-cluster dispersion (sum of squares between cluster means and global mean):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*q9G26z00tLfQFXlb42nPag.png" /></figure><ul><li>The CHI (VRC) is then:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*dWEUB8wsKxFcLeJZdqg0Sg.png" /></figure><p>Example:</p><p>Dataset : We have 5 points in 2 clusters:</p><p>Cluster 1: A(1, 1), B(2, 1), C(1, 2)</p><p>Cluster 2:D(6, 5), E(7, 6)</p><p>So:<br> n=5, k=2</p><p>Step1: Find centroids for all the clusters</p><p>c1= (1+2+1)/3, (1+1+2)/3 = (1.33,1.33)</p><p>c1= (6+7)/2, (5+6)/2 = (6.5,5.5)</p><p>overall centroid C= (1+2+1+6+7)/5, (1+1+2+5+6)/5 = (3.4,3.0)</p><p>step 2: Compute Within-cluster dispersion (sum of squares within clusters):</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*iRWHKSSZfNPhM4SPCyhvPw.png" /></figure><p>cluster 1 there are 3 points A=(1,1), B=(2,1), C=(2,1)</p><p>WA=(1–1.33) ² + (1–1.33) ²=0.22</p><p>WB=(2–1.33) ² + (1–1.33) ²=0.55</p><p>WA=(1–1.33) ² + (2–1.33) ²=0.55</p><p>Cluster 1 total with in cluster dispersion W1=0.22+0.55+0.55 = 1.32</p><p>Cluster 2 has 2 points : D(6,5), E(7,6)</p><p>WD=(6–6.5) ² + (5–5.5) ²=0.5</p><p>WE=(7–6.5) ² + (6–5.5) ²=0.5</p><p>Cluster 2total with in cluster dispersion W2=0.5+0.5 = 1</p><p>Total within cluster distance W=W1+W2 = 1.32+1 = 2.32</p><p>Step 3: Compute between cluster dispersion B</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*q9G26z00tLfQFXlb42nPag.png" /></figure><p>Cluster 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*JEeumY6LtiTZwfsk3RSpuA.png" /></figure><p>Cluster 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*xCXvbDwwrTw1WSP6M-NQTg.png" /></figure><p>Total between cluster distance B= B1+B2=19.41+30.12=49.53.</p><p>Step 4: Compute CHI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*dWEUB8wsKxFcLeJZdqg0Sg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*yySaQPVIn7HY4vsyB8tHsA.png" /></figure><p>CHI=64.07. A high CHI interprets clusters are <strong>well separated</strong> and <strong>internally compact</strong>.</p><p>Interpretation of CHI:</p><ol><li>A higher index value indicates that the clusters are dense (low WCSS) and well-separated (high BCSS) which is the good clustering.</li><li>The CHI is typically calculated for a range of possible k values. The value of k that yields the highest Calinski-Harabasz index is chosen as the optimal number of clusters for the dataset.</li></ol><p>Advantages of CHI:</p><ol><li>Simple and fast to compute because it uses only cluster centroids which takes less time to calculate, especially for larger datasets.</li><li>Internal metric which does not need any ground truth labels for evaluation, relying solely on the data and the clustering results.</li><li>Works well with K-means because the metric is based on sums of squares and centroids, it aligns directly with the objective function of the K-means algorithms.</li></ol><p>Disadvantages of CHI:</p><ol><li>Assumption of Spherical/Compact clusters: Since it uses distances to the centroid (squared Euclidean distance), the CHI tends to favor clusters that are convex, spherical, and roughly equal in size/density. It may perform poorly when clusters are non-globular or have complex shapes. (crescent-shaped clusters).</li><li>Relies on Euclidean distance making it suitable for clustering solutions derived using non-Euclidean distance measures.</li><li>No absolute baseline: The index value itself has no inherent acceptable cutoff. It is only useful for relative comparison between different clustering solutions for the same dataset.</li></ol><p><strong>4. DUNN Index:</strong></p><p>Dunn Index is a measure used to evaluate the quality of the clusters. It aims to quantify how well- separated and compact the clusters are.</p><p>The fundamental principle is that good clustering should have:</p><ol><li>Larger distance between clusters (good separation).</li><li>Small distances within clusters (good compactness).</li></ol><p>The Dunn Index is calculated as the ratio of the minimal inter-cluster distance to the maximal intra-cluster distance across all clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*lh2mjDwWhb1w7cLbdlAXhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*Xypx37gqnC84M_LRaN6GFw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*hc-MN_VC_RG1sN5kDx5tRQ.png" /></figure><p>Example:</p><p>Cluster 1: (1, 2), (2, 3), (3, 2)</p><p>Cluster 2: (8, 8), (9, 7), (8, 9)</p><p>Step1: Compute the distances inside all clusters for every pair of points where d with formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*vuusxwVnO93E7U733hH41A.png" /></figure><p>Distances inside cluster 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*OCV__KQXAobW2hs1HBRqoQ.png" /></figure><p>Distances inside cluster 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*HlI95Vf-3U-NWgFXIlryaA.png" /></figure><p>maximum cluster diameter (denominator of DI):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*EmTsTLBwUrPwyjAXRGKxSA.png" /></figure><p>Step 2: Compute minimum of inter cluster distances from every point in C1 to every point in C2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*MFU4Piuq3G9J8S24Zy7jjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*FEXBYbL3G5yDVGrHAAdPIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*mzTXigf_4QT772FaQjhFQQ.png" /></figure><p>Minimum inter cluster distance among 9 distances is (numerator of DI)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*0zgQy48tpVxU5OXuD4cytg.png" /></figure><p>Step 3: Compute Dunn Index</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*YNoi9_AZqzmskgR_Hj620Q.png" /></figure><p>Interpretation: The Dunn Index (DI) has a value between zero and infinity, and it should be maximized for better clustering.</p><p>Higher the DI value is better clustering and lower DI value is poor clustering. The optimal number of clusters k should be chosen.</p><p>Advantages of Dunn Index:</p><ol><li>Directly measures compactness and separation.</li><li>It only uses the clustered data itself, without needing external ground truth labels.</li></ol><p>Disadvantages of Dunn Index:</p><ol><li>For large datasets, calculating all pairwise distances to find the minimum inter-cluster distance and the maximum cluster diameter can be computationally very expensive.</li><li>Sensitivity to outliers: Since it relies on the minimum separation and maximum diameter a single outlier can significantly inflate the denominator and reduce the DI value.</li></ol><p>Summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*RrRwnz2FgWmlf3Ezz2kL1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*QjBe1m9in-TIRCE9Fp3V8g.png" /></figure><p><strong>External Evaluation metrics:</strong></p><p>External evaluation metrics for clustering assess the quality of clustering result by comparing it against an external criterion or ground truth class labels. These ground truth labels for the data points are known beforehand but were not used to perform the clustering.</p><p>External metrics are used when we have the true labels, providing a quantitative measure of how well the clusters align with the predefined structure.</p><p><strong>There are 3 categories of external evaluation metrics. They are</strong></p><ol><li>Pair-Counting Metrics</li><li>Information — Theoretic Metrics</li><li>Set-Matching Metrics</li></ol><p><strong>Pair-Counting Metrics:</strong></p><p>Pair- Counting metrics work by examining all possible pairs of data points in the dataset and classifying how these pairs are clustered by comparing actual label clusters and predicted clusters (model output). They work by checking paris of points and seeing whether each pair is</p><ul><li>grouped together in both clusterings</li><li>grouped separately in both clusterings</li><li>grouped together in one clustering but not the other</li></ul><p>Given a dataset of <strong>N points</strong>, there possible pairs of points are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*rISRiEs4buEpUSKXMLYUzA.png" /></figure><p>For each pair of points check two conditions</p><p>Let: <strong>C1</strong> = true classes</p><p><strong>C2</strong> = predicted clusters</p><p>For each pair of points (i,j), we check:</p><ol><li>Are they in the <strong>same cluster</strong> in C1?</li><li>Are they in the <strong>same cluster</strong> in C2?</li></ol><p>Based on this, each pair belongs to one of four categories:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*3XAoCY2i_CkEmEYWAhWCiA.png" /></figure><p>These counts a,b,c, and d summarize the agreement/disagreement between two clusterings.</p><p>There are 3 pair counting metrics</p><p><strong>i) Rand Index (RI):</strong> Rand Index is a basic form of pair-counting metric that measures the overall similarity between the two assignments by counting how many pairs of data points are consistently grouped or separated in both partitions. Rand Index quantifies the degree of agreement between a clustering solution (the predicted partition) and a known, ground-truth classification ( the true partition).</p><p>The Rand Index is defined as the fraction of all possible pairs of data points that are placed into the same relationship (either grouped together or separated) in both the predicted clustering (C2) and the true classification (C1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*LjOQTauTEWBkVB2lbWz-tQ.png" /></figure><p>Where:</p><ul><li><strong>a</strong> = number of pairs <strong>in same cluster</strong> in both truth and predicted</li><li><strong>d</strong> = number of pairs <strong>in different clusters</strong> in both truth and predicted</li><li><strong>b</strong> = number of pairs <strong>same in truth but different in predicted</strong></li><li><strong>c</strong> = number of pairs <strong>different in truth but same in predicted</strong></li></ul><p>and</p><ul><li>a + d = number of <strong>agreements</strong></li><li>b + c = number of <strong>disagreements</strong></li></ul><p>Example:</p><p>We have <strong>5 points</strong>: P1, P2, P3, P4, P5.</p><p>Ground truth clustering C1</p><ul><li>Cluster 1: {P1, P2}</li><li>Cluster 2: {P3, P4, P5}</li></ul><p>Predicted clustering C2:</p><ul><li>Cluster A: {P1, P3, P5}</li><li>Cluster B: {P2, P4}</li></ul><p>Step 1 : find the total pairs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/87/1*gXJQKn4VyT4mbuXnJh1Dnw.png" /></figure><p>Pairs:<br> (P1,P2), (P1,P3), (P1,P4), (P1,P5), (P2,P3), (P2,P4), (P2,P5), (P3,P4), (P3,P5), (P4,P5)</p><p>Step 2: for each pair label same/different in both clusterings by making a table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*bTeVkHDQkDGeaUxRv7hjgQ.png" /></figure><p>Step 3: Count a,b,c and d</p><ul><li><strong>a (same-same)</strong> = 1</li><li><strong>b (same-different)</strong> = 3</li><li><strong>c (different-same)</strong> = 2</li><li><strong>d (different-different)</strong> = 4</li></ul><p>Check total:</p><p>a+b+c+d=1+3+2+4=10=all pairs</p><p>Step 4: Compute Rand Index</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*_P8EXSEInAgbSvlx_OeIiw.png" /></figure><p>This means <strong>50% of all point pairs were clustered correctly</strong>.</p><p>Interpretation:</p><ul><li>RI = <strong>1.0</strong> → perfect clustering</li><li>RI = <strong>0.0</strong> → completely incorrect</li><li>Here: <strong>0.5 → moderate match</strong> between predicted clustering and ground truth.</li></ul><p><strong>Rand Index (RI) is used to measure the similarity between two clusterings.</strong><br> It is especially used for:</p><ul><li>External cluster validity</li><li>Comparing clustering methods</li><li>Testing stability of clustering</li><li>Ensemble clustering consistency</li></ul><p><strong>ii)Adjusted Rand Index (ARI):</strong></p><p>RI does not account for chance grouping. Even random cluster assignments can get a moderately high RI. <strong>RI corrects this problem</strong> by adjusting RI for the expected similarity of random labelings.</p><p>Range of ARI:</p><p><strong>+1</strong> → Perfect agreement</p><p><strong>0</strong> → Random clustering</p><p><strong>Negative</strong> → Worse than random</p><p>ARI is a popular external clustering validation metric used to evaluate the agreement between a predicted partition and the true prediction. It is an improvement upon the basic Rand Index because it corrects for the agreement expected purely by random chance, making it is a reliable measure of true clustering quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*-FSjGY4MKMdm7_eyzBBjKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*TwZ3rsi7TRS6nZNGIrx10w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*8uPb0qYQrTEbsm95jLLIzQ.png" /></figure><p>Example:</p><p>Let we have two clusters:</p><p>Ground truth partition:</p><ul><li>Cluster A: {1, 2, 3}</li><li>Cluster B: {4, 5, 6}</li></ul><p>Predicted partition:</p><ul><li>Cluster X: {1, 2, 4}</li><li>Cluster Y: {3, 5, 6}</li></ul><p>Step 1: Make contingency table</p><p>n11​=Number of points that are in cluster <strong>A</strong> of C1 and in cluster <strong>X</strong> of C2</p><p>Cluster A= {1,2,3} and Cluster X= {1,2,4}</p><p>Intersect A and X:</p><p>A∩X={1,2}</p><p>Count the elements: n11=∣A∩X∣=2</p><p>Compute n12= A∩Y={1,2,3}∩{3,5,6}= {3}</p><p>n12= |A∩Y|=1</p><p>n21=|B∩X| = |{4,5,6}∩{1,2,4}|=|{4}|=1</p><p>n22=|B∩Y|=|{4,5,6}∩{3,5,6}|=|{5,6}| =2</p><p>make contingency table as given below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*mvU8_wdm43oB3z0MW_3akQ.png" /></figure><p>Step 2: Compute required combinations</p><p>Term1: Sum of combinations inside each cell</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*DM8Yfto83xXHd403x8HF1w.png" /></figure><p>Term 2: Row combinations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*HPD1_VFDHeU8JJoxeJ5ssg.png" /></figure><p>Term 3: Column combinations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*FQ9mPOKzrdvH4bLs7IS4PQ.png" /></figure><p>Term 4: Total Combinations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*81oPmustEzOJIIN0jZaJXQ.png" /></figure><p>Now to find ARI compute numerator and denominator</p><p>Numerator=</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*FaYvMFk2_9aSuhIsVQi-KQ.png" /></figure><p>Denominator=</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*X6VCJS-KIiZv9QpxR_tR3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*v-TgHKiKXDs08SIUkHmOqQ.png" /></figure><p>Interpretation:</p><p>This means:</p><ul><li>The clustering result is <strong>worse than random</strong>.</li><li>The algorithm placed many pairs incorrectly compared to the true labels.</li></ul><p><strong>RI </strong>Measures clustering similarity but biased by chance and <strong>ARI </strong>Corrects RI by accounting for chance; more reliable</p><p><strong>ARI is the preferred metric for comparing two clusterings because it gives an unbiased similarity score.</strong></p><p>iii) <strong>Fowlkes-Mallows Index (FMI): FMI </strong>is an <strong>external clustering validation metric</strong> used to measure the similarity between a clustering solution and a ground-truth classification. It is a pair-counting metric that is calculated as the <strong>geometric mean of the precision and recall</strong> of the pairwise agreements.</p><p>The Fowlkes–Mallows Index is defined using the pair-counting statistics (a, b,c,d) derived from a contingency table comparing the predicted clusters and the true classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*gxZxYJ7nyYQr9yeWWIfGzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*WfTi-lGWzLqOixQ_hDA5QA.png" /></figure><p>Example :</p><ul><li><strong>Ground truth (C1)</strong>: A = {1,2,3}, B = {4,5,6}</li><li><strong>Predicted (C2)</strong>: X = {1,2,4}, Y = {3,5,6}</li></ul><p>Step 1: compute pair counts a,b,c and d which we have already calculated in the previous example</p><ul><li>a = number of pairs <strong>same in both</strong> =</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*VIZn21sspZTqCSh265PyzQ.png" /></figure><ul><li>Pairs that are <strong>same in truth</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*LLFIlIhiTekclOd2EZsiNA.png" /></figure><p>So b (same in truth but different in pred = (same in truth) − a = 6−2=4</p><ul><li>Pairs that are <strong>same in predicted</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*liS3me-arH5fvLeukFwrLg.png" /></figure><p>So, c (different in truth but same in pred) =(same in pred) − a= 6−2=4</p><ul><li>total pairs are</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*Ruco_1uJG53q65TMD2jMHQ.png" /></figure><p>and d=15−(a+b+c)=15−(2+4+4) = 5 and a=2, b=4, c=4</p><p>Step 2 : compute FMI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*fmaBcwUGIg6hGWRk7xhwoQ.png" /></figure><p><strong>Information — Theoretic metrics:</strong></p><p>These external metrics evaluate clustering quality using information theory concepts such as entropy and mutual information. These compare the cluster assignments with the ground truth class labels.</p><p>Information theoretic metrics are based on the distribution of labels and their statistical dependence. These evaluate how much information is shared between a predicted clustering and ground truth labeling.</p><p>The primary building blocks for these metrics are Entropy(H) and Mutual Information(MI). Entropy gives the uncertainty in clustering and MI give the information shared between two clustering&#39;s.</p><p>Main types of information theoretic metrics:</p><ol><li>Mutual Information (MI): MI measures the amount of information shared between the true labels and the predicted clusters.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*Yob9d3ZcvQnLgtsmn0n3oA.png" /></figure><p>MI is used to select features that have high dependency with target variable.</p><p>2. Normalized Mutual Information (NMI): NMI is used to measure the similarity between cluster assignments and true class labels by normalizing mutual information to a scale between 0 and 1. This is often used method because of normalized MI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*oIcrL9N-GPqbFQzhHRODnA.png" /></figure><p>3. Adjusted Mutual Information (AMI): AMI measures the similarity between cluster assignments and true labels by adjusting mutual information for chance or random agreement, in other words AMI measures how much the clustering agrees with true labels after removing the similarity that could occur by random chance. AMI ranges between -1 and +1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*jDFemElFGhnoWFenYD4_5g.png" /></figure><p>AMI is preferred in Machine Learning research for reliable evaluation.</p><p>4. Variation of Information (VI): VI measures how different two clustering s are by quantifying the information that not shared between them. If VI is 0 the clustering is perfect and if it is higher there is a more dissimilarity. Lower VI better clustering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*aFEQkQS-8TrWUKN89-Eehg.png" /></figure><p>VI is generally used to analyze clustering errors by measuring information lost or information gained through extra noise in cluster. VI measures difference not similarity.</p><p>5. Conditional Entropy: Conditional entropy measures the amount of uncertainty remaining in one variable given that another variable is known.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*1kctZ0fznvdBTJIyOdRPXA.png" /></figure><p>It is used to identify features that reduce uncertainty of target variable, calculate information gain for selecting best splits in decision tree algorithms.</p><p><strong>Set Matching metrics:</strong></p><p>Set matching metrics are a type of external evaluation metrics used to assess the quality of by clustering by computing how well each cluster Ci matches a true class Lj using set operations such as intersection and union.</p><p>overlap=|Ci ∩ Lj|</p><p>Most popular set matching metrics are:</p><ol><li>Purity: Measures how much each cluster contains data points from a single dominant class.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*9aVab8w4ySHoWCVmznibXg.png" /></figure><p>Purity is used when a simple and quick evaluation of clustering quality to check how well each cluster contain data points from a single class.</p><p>2.Precision: It is a set matching external evaluation metric that measures the proportion of data points in a cluster that actually belong to the corresponding true class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*jvunoRwXpebJ3D945xUhiw.png" /></figure><p>Precision is used when we want to measure the correctness of cluster assignments, i.e. how many data points in a cluster actually belong to the intended true class. Commonly used for document classification.</p><p>3. Jaccard Coefficient: Jaccard Coefficient is a set matching external evaluation metric that measures the similarity between a cluster and true class based on the ratio of their intersection to union.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*GN_3x56iM8211zjCQ_8nnQ.png" /></figure><p>This metric is used when clusters and classes are treated as sets of data points. Commonly used for comparing documents, keywords, or topics.</p><p>4.Recall: Recall is a set matching external evaluation metric that measures the proportion of data points of a true class that are correctly captured within a cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*Uo_fWoGtmOMU4JWtdA4Cpg.png" /></figure><p>Recall is used when we want to measure how completely a cluster captures all data points of a true class.</p><p>Used for information retrieval applications like search engines retrieving relevant documents, Detection of diseases where false negatives are costly etc.</p><p>5. F-Measure (F1 Score): F1 Score is a set matching external evaluation metric that measures the overall performance by taking the harmonic mean of precision and recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*Uhhu5TWQdtXhbvxvzbIYKQ.png" /></figure><p>F1 Score is the harmonic mean of precision and recall used to evaluate clustering or classification by balancing correctness and completeness.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=884ee5398f9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Type Conversions in Python]]></title>
            <link>https://medium.com/@kanchanakanta/type-conversions-in-python-e05c41013ded?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e05c41013ded</guid>
            <category><![CDATA[type-conversion-functions]]></category>
            <category><![CDATA[type-conversion]]></category>
            <category><![CDATA[explicit-type-conversion]]></category>
            <category><![CDATA[implicit-type-conversion]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 14:05:24 GMT</pubDate>
            <atom:updated>2025-06-28T14:05:24.036Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gl5jPQb_Qr_csVHk.png" /><figcaption>Type conversions in python</figcaption></figure><p><strong>Definition</strong></p><p>Type conversion is a process of converting a value from one data type to another. Type conversion is also called <strong>type casting</strong>. This process is common in all programming languages, such as C, C++, Java, and Python.</p><p><strong>Use of Type Conversion:</strong></p><p>· To match data types for operations</p><p>· To prevent data type mismatch errors</p><p>· To handle input/output conversions</p><p>· To manage memory of smaller types when needed</p><p><strong>Types of type conversions:</strong></p><p>There are two types of type conversions.</p><p>1. Implicit type conversion</p><p>2. Explicit type conversion</p><p><strong>Implicit type conversion:</strong></p><p>Implicit type conversion is also called Automatic type conversion. In implicit type conversion, the compiler or interpreter automatically converts a lower data type to a higher data type to avoid data loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*V9N0GUJTtCFrFNMH3Io-5w.png" /></figure><p>In the above example, we tried to add two numerical values, one is an integer and the other a float. So, the interpreter converts the integer to float values and then adds the two values to give the float output.</p><p>If we try to perform operations between different types of values, those operations are not supported by Python, i.e, implicit conversion is not possible between different types of values. For example, if we try to add one integer and one string value, we get a type error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*eS8yLgsFxlGNpE4FwJSrvg.png" /></figure><p><strong>Explicit type conversion:</strong></p><p>For type errors in implicit conversion, Python has a solution, which is known as Explicit conversion.</p><p>· In Explicit type conversion, users convert the data type of an object to the required data type using the predefined functions like int (), float (), str (), etc. to perform type conversion.</p><p>· This type of conversion is also called type casting, because the user casts or changes the datatype of the objects.</p><p><strong>Syntax: &lt;required_datatype&gt;(expression)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*LMNfMg1PXnMF7SG9sgvwlQ.png" /></figure><p><strong>Common type conversion functions in Python:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*VWSphm4g1gRyBOZjzBLGbw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*7GpmpilytdOjLaRwGYJpVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*aeDqhsUIGPKfguM5tJFkPA.png" /></figure><p><strong>Possible issues with type conversions:</strong></p><p>· <strong>Data loss:</strong> converting float to int truncates the decimal part.</p><p>· <strong>Precision issues:</strong> converting a double to a float might lose precision</p><p>· <strong>Runtime errors:</strong> improper conversions may cause exceptions, value errors, and type errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*UlkaIe02WNYIn_epqcKvfQ.png" /></figure><p>After the first error remaining instructions will not be executed.</p><p>Flash cards:</p><ol><li>What is Type Casting?</li></ol><p>A<strong>:</strong> It is the process of converting the value of one data type to another.</p><p><strong>2.</strong> What are the two types of type casting in Python?<br> <strong>A: 1.</strong>Implicit Type Casting 2. Explicit Type Casting</p><p><strong>3.</strong> What is implicit type casting?<br> <strong>A:</strong> Python automatically converts one data type to another during operations.</p><p>a = 5 # int <br>b = 2.0 # float <br>c = a + b # c becomes 7.0 (float)</p><p><strong>4.</strong> What is explicit type casting?<br> <strong>A:</strong> Manually converting data types using built-in functions like int(), str(), float(), etc.</p><p><strong>5.</strong> How do you convert a string &#39;123&#39; to integer?<br> <strong>A:</strong> int(&#39;123&#39;) → 123</p><p><strong>6.</strong> How to convert an integer 45 to a string?<br> <strong>A:</strong> str(45) → &#39;45&#39;</p><p><strong>7.</strong> What happens if you try int(&quot;abc&quot;)?<br> <strong>A:</strong> Raises a ValueError because &quot;abc&quot; is not a number.</p><p><strong>8.</strong> How to convert a tuple to a list?<br> <strong>A:</strong> list((1, 2, 3)) → [1, 2, 3]</p><p><strong>9.</strong> What does bool(0) return?<br> <strong>A:</strong> False<br> 10. What does bool(&quot;hello&quot;) return?<br> <strong>A:</strong> True</p><p><strong>11.</strong> Why is type casting important in Python?<br> <strong>A:</strong> To ensure correct data types for operations, especially for user input, file I/O, and APIs</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e05c41013ded" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DATA VISUALIZATION TECHNIQUES]]></title>
            <link>https://medium.com/@kanchanakanta/data-visualization-techniques-dbfc5f484caf?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbfc5f484caf</guid>
            <category><![CDATA[bar-chart]]></category>
            <category><![CDATA[histograms]]></category>
            <category><![CDATA[heatmap]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[scatter-plots]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Sun, 04 May 2025 14:51:08 GMT</pubDate>
            <atom:updated>2025-05-04T14:51:08.068Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*Ri6EGGZuDAlc-Qv2bswTWA.jpeg" /></figure><p><strong>DATA VISUALIZATION:</strong></p><p>Data visualization is a process of representing raw data or information in a graphical representation, like charts, graphs, and maps, that is easier for the human brain to understand.</p><p><strong>Advantages of data visualization:</strong></p><p>1. <strong>Enhanced understanding of data:</strong> Data visualization helps in understanding the patterns, trends, and outliers that are present in the data.</p><p>Example:</p><p>a) The stock price of a company steadily increasing over several years is an upward trend.</p><p>b) Sales for a particular product declining year by year is a downward trend</p><p>c) Monthly yields for an active pharma ingredient remain relatively constant in a sideways trend.</p><p>d) Sales of woolen clothes increase during winter is a seasonal pattern.</p><p>e) In business, fraud or unusual customer behavior is an outlier.</p><p>2. Efficient Data Sharing: In business settings, stakeholders need to understand and act on data insights easily. Visual data presentation facilitates effective communication, making information more understandable.</p><p>3. Identifying event relations: Data visualization helps in discovering correlations between events, which may be crucial for understanding the factors affecting the performance. For example, understanding the increase in online sales during festive seasons can help in better planning the business.</p><p>4. Exploring opportunities: Business leaders can have a deeper understanding of customer behaviors and uncover patterns and opportunities from the large data sets.</p><p>5. Geographical visualization: This can be useful for businesses that operate in multiple regions and need to understand regional performance.</p><p>6. Storytelling: To communicate with non-technical audiences, data visualization helps in telling a data story by presenting data facts in an easy-to-understand form.</p><p><strong>Types of visualization techniques:</strong></p><p>Various visualization techniques are based on D<strong>ifferent types of data and Analysis goals.</strong></p><p>Based on the<strong> types of data,</strong> the visualization can be done using the following plots.</p><ol><li><strong>Categorical data</strong>: Categorical data or qualitative data, which represents categories, names, or labels like gender, country, department, etc., the following plots are used for visualizing categorical data.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*fJpTFxxt_hj1zD9pXmrjpg.png" /></figure><p>2. <strong>Numerical data</strong>: Numerical data or quantitative data that can be measured or counted. The data can be either discrete or continuous. Examples for discrete values are the number of students, colors and for continuous values are height, weight which consists decimal values. The following plots are used for visualizing numerical data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*eZeq_cExsMxHbre_zt7r6g.png" /></figure><p>3. Bivariate or Multivariate Data: Analyzing two or more variables together. The following plots are used for visualizing multivariate data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*m1pbZ9bXHAydJi-sHrlVJg.png" /></figure><p>4. Geospatial data: Data with location components (latitude/longitude, region)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*53Q2b5Mh9UEBkIvG2HRsBg.png" /></figure><p><strong>Visualization techniques based on analysis goals:</strong></p><p>1. <strong>Comparisons:</strong> To compare the differences and similarities between the values in a data set we can use the following charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*di7HfzffkaoFZI2It7IQsA.png" /><figcaption>Fig: Visualization charts/plots for comparisons</figcaption></figure><p>2. <strong>Trend Analysis: </strong>Trend analysis is the process of analyzing patterns or trends in data over some time to identify consistent movement, whether increasing, decreasing, or stable. The main purpose of trend analysis is to detect long-term patterns in data, forecast future values or performance, to understand market behavior or user activity over time. Trend analysis also supports data-driven decision making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*js0F-Z9pcyG3gUzBnEfrow.png" /><figcaption>Fig: Visualization charts/plots for trend analysis</figcaption></figure><p><strong>3.</strong> <strong>Understanding Distribution of data:</strong> Distribution of data refers to how values of a variable are spread or dispersed across a range. It helps in understanding the shape, center, spread, and outliers in the dataset. It is important to understand data distribution to identify patterns like symmetry, skewness, to check the normality or irregularities, to detect outliers and data variability, and to choose the appropriate statistical methods.</p><p><strong>Key characteristics of data distributions:</strong></p><p>a. Central tendency: It is a statistical measure that identifies the center point or typical value of a dataset. It gives an idea of where most data points lie. The main measures of central tendency are mean, median, and mode.</p><p>b. Spread: It is also known as dispersion or variability refers to how much the data values vary from the central tendency. It tells us how scattered or concentrated the data is. Spread is used to understand consistency or variability in data, to compare datasets, to detect the presence of outliers, to support risk assessment in finance, science, and analysis.</p><p>c. Shape: Shape of a distribution refers to the general pattern or outline formed by data when plotted. It tells us how values are distributed around the center. The common shapes of distributions are symmetric, skewed, peaked, or flat. Shape is important to determine the appropriate statistical test like median for skewed data, to indicate data quality and presence of subgroups or outliers and helps in predictive modelling.</p><p>d. Outliers: Outliers are data points that are significantly different from the rest of the dataset. They lie far outside the normal range of values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*iSd0X6RP85gn2TE3r6_RxA.png" /><figcaption>Fig: Visualization charts/plots for understanding distributions</figcaption></figure><p>4. <strong>Part — to — Whole Relationship:</strong> A part — to — Whole relationship shows how individual parts contribute to the total or whole. It is a way to visualize proportions, percentages, or fractions of a complete dataset. It is useful to understand how different components contribute to an overall total, to compare proportions across categories, mostly useful for reporting resource allocation, market share, or survey responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*jjQvJWuwxm7HXUvG6YyJ2A.png" /><figcaption>Fig: Visualization charts/plots for part — to — whole relationship</figcaption></figure><p><strong>5. Correlation / Relationship:</strong> Correlation in data analysis refers to the statistical association between two or more variables. It determines how changes in one variable relate to changes in another. The correlation is of 4 types.</p><p>a) Positive Correlation, where variables change as if one increases, the other also increases, or if one decreases other also decreases.</p><p>b) Negative correlation where if one variable increases other variable decreases and vice versa.</p><p>c)No correlation specifies there is no association between the variables.</p><p>d) Nonlinear relationship means that a non-straight line pattern relationship exists between variables.</p><p>The correlation coefficient ranges from -1 to +1. This correlation helps in identifying predictive relationships, for feature selection in machine learning and to detect multicollinearity in regression models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*oHxXsnJvR92G9Ro_UnDzZA.png" /><figcaption>Fig: Visualization charts/plots for correlation</figcaption></figure><p><strong>6. Deviation / Variance</strong>: Deviation is the difference between a data point and the central value, typically the mean. It helps to understand how far each data point is from the average. It is used as a building block for calculating variance and standard deviation.</p><p>Whereas variance is the average of the squared deviations from the mean. It quantifies how spread out the data points are. This is used to measure the dispersion of data. High variance means data points are more spread out; low variance means data points are closer to the mean. It is used in finance, risk analysis, and statistical modelling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*wdtOSwnv1xTfAhtGU1KeCA.png" /><figcaption>Fig: Visualization charts/plots for Deviation / Variance</figcaption></figure><p><strong>7.</strong> <strong>Outlier Detection: </strong>Outlier detection is the process of identifying data points that differ significantly from other observations in a dataset. These outliers may represent errors, rare events, or important discoveries. Outlier detection is important to clean data before modelling, to avoid distortion of statistical measures like the mean or regression coefficients. Generally used to detect fraud, defects, or anomalies in various domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*wGWrmP_cP6taPkVb4eDH9A.png" /><figcaption>Fig: Visualization charts/plots for outlier detection</figcaption></figure><p>8. <strong>Geospatial Analysis:</strong> Geospatial analysis is the process of collecting, visualizing, and analyzing data that has a geographic or spatial component- i.e., data tied to locations on earth. It helps uncover patterns, relationships and trends based on where things happen.</p><p>This analysis is used for planning traffic flow, land use, public facility placement in urban planning, disease outbreak mapping in health analytics, monitoring deforestation, pollution tracking, disaster zones in environmental monitoring, planning for site selection, finding customer density to establish retail and marketing, to monitor crop health via satellite imagery, irrigation planning in agriculture, route optimization and fleet tracking in logistics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*owdkYId9RucDwd50bc8-aA.png" /><figcaption>Fig: Visualization charts/plots for Geo spatial analysis</figcaption></figure><p><strong>9. Multivariate Analysis:</strong> Multivariate analysis is a statistical technique used to analyze more than two variables simultaneously to understand relationships, patterns, or effects among them.</p><p>It helps to answer questions like how do multiple variables together affect an outcome?</p><p>Are there interactions or hidden correlations among features? Etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*9jJ_wGCrxaBFl-ZKGwdtKQ.png" /><figcaption>Fig: Visualization charts/plots for multivariate analysis</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbfc5f484caf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Types of Data]]></title>
            <link>https://medium.com/@kanchanakanta/types-of-data-e3b254cafe41?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3b254cafe41</guid>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 11:06:56 GMT</pubDate>
            <atom:updated>2025-04-29T11:06:56.432Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*KOIJfn1Um8y8pBCNQjOxsg.png" /><figcaption>Types of Data</figcaption></figure><p><strong>Data and its types:</strong></p><p><strong>Data:</strong> Data is a collection of raw facts, figures or values that represent the information about events, processes, objects, or people. It has no meaning until it is processed or analysed.</p><p><strong>Types of Data:</strong></p><p>Data can be classified into various types based on its nature, format, and use in analysis. There are two categories of data, qualitative and quantitative.</p><p><strong>Qualitative data:</strong></p><p>Qualitative data or categorical data cannot be measured or counted as numbers. Data can be divided into specific categories. These categories label observations and describe qualities or characteristics.</p><p><strong>Types of categorical data:</strong></p><p>There are two types of categorical data.</p><p>1. Nominal data</p><p>2. Ordinal data</p><p><strong>Nominal data:</strong> Nominal data classifies data into distinct groups or categories that are mutually exclusive and do not follow any logical order. Nominal data is used to label variables without any order or quantitative values. We cannot compare the values with one value to others.</p><p>We cannot perform any arithmetic operations like addition or averaging and is not suitable for statistical analysis.</p><p>Example:</p><p>Blood group: A, B, AB</p><p>Gender: Male, Female</p><p>Nationality: Indian, American, French</p><p><strong>Ordinal data:</strong> Ordinal data classifies observations into ordered categories, indicating relative position, but not the exact magnitude of difference between them.</p><p>These types of values have a natural ordering while maintaining their class values. The interval between values is unknown or unequal. Arithmetic operations like addition or mean are not valid.</p><p>Often used in surveys, ratings and rankings.</p><p>Example:</p><p>Movie rating: 1 star &lt;2 star &lt; 3star&lt; 4 star &lt;5 star</p><p>Education: high school &lt; bachelor &lt; masters &lt; PhD</p><p>Clothing sizes: small &lt; medium &lt; Large</p><p><strong>Difference between nominal and ordinal data:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*8_HY1iumSxMC5eTGzWKrlw.png" /></figure><p><strong>Quantitative data:</strong> Quantitative data is numeric in nature. This data can be measured or counted. This data allows mathematical operations and is useful for statistical analysis.</p><p>Types of quantitative data:</p><p>1. Discrete data</p><p>2. Continuous data</p><p><strong>Discrete data:</strong> Discrete data consists of distinct, separate values. It is countable, and values cannot be subdivided meaningfully.</p><p>Example:</p><p>1. Number of cars in parking area: 2,3,5</p><p>2. Number of students: 60,75,40</p><p><strong>Continuous Data:</strong></p><p>Continuous data can take any value within a given range. This data can be obtained through measurement and values can be subdivided into infinitely smaller parts. This data can include fractions and decimals.</p><p>Example:</p><p>Height of a person: 172.5 cm, 180cm, 167.2cm</p><p>Temperature: 36.5 degrees, 42.8 degrees</p><p>Differences between discrete data and continuous data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*WA57XlSuRBbiH8RI4S1l3A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3b254cafe41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CLASSIFICATION METRICS]]></title>
            <link>https://medium.com/@kanchanakanta/classification-metrics-6977da248981?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6977da248981</guid>
            <category><![CDATA[precision-recall]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[accuracy]]></category>
            <category><![CDATA[classification-metrics]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Sun, 01 Dec 2024 16:59:58 GMT</pubDate>
            <atom:updated>2024-12-01T16:59:58.443Z</atom:updated>
            <content:encoded><![CDATA[<h3>CLASSIFICATION EVALUATION METRICS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*4UQ-gY_Oj9bLjwvB.png" /><figcaption>REF: <a href="https://www.markovml.com/blog/key-machine-learning-metrics-for-assessing-model-performance">Key Machine Learning Metrics for Assessing Model Performance Today</a></figcaption></figure><p>Classification is a supervised machine learning process to predict the class label of a given data point based on the input data. There are two types of classification methods.</p><p>1. <strong>Binary classification:</strong> This is a machine learning technique used to predict the class label of a new point into either of the two distinct classes. Binary classification deals with only two outcomes.</p><p>Example: email filtering into SPAM or HAM</p><p>Disease diagnosis into yes or no</p><p>2. <strong>Multi-class classification:</strong> This is a machine learning technique used to predict the class label of a new data point into any of the more than 2 class labels.</p><p>Example: identifying objects in the image</p><p>Text classification: classifying documents into topics</p><p>There are many classification algorithms like Logistic Regression for binary classification, Decision tree classifier, Support Vector Machine, Naive Bayes classifier etc.</p><p>Once the classification model is built it is essential to evaluate the performance of the model. The fundamental performance metric is Accuracy.</p><p><strong>Performance evaluation metrics of a classification model:</strong></p><ol><li><strong>Accuracy:</strong></li></ol><p>Accuracy is the fundamental performance evaluation metric of a classification model. It measures the proportions of correct predictions made by the model out of the total number of predictions.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*T5Lg0P_y-tHkxC6VWT8pwg.png" /></figure><p><strong>High accuracy indicates the model is performing well and low accuracy indicates that the model is not distinguishing the classes well.</strong></p><p>From the above example we can say that accuracy of the model is only 66.66% and there are some incorrect predictions.</p><p>In the above example there are 2 types of incorrect predictions known as errors.</p><p><strong>There are two types of errors.</strong></p><p>Type -I error: The model predicts the negative class as positive class which is known as Type -I error. (False Positive)</p><p>Type- II error: The model predicts the positive class as negative class which is known as Type -II error. (False Negative)</p><p><strong>when to use accuracy?</strong></p><p>Accuracy measure is used when the classes are balanced i.e. similar number of instances are there in each class.</p><p><strong>Drawbacks of Accuracy:</strong></p><p>Accuracy measure does not specify the type of errors. It specifies only how many incorrect predictions are there. Therefore, only accuracy measure is not enough for measuring the classification model’s performance. We need a powerful evaluation tool known as confusion matrix to evaluate the performance of a classification model.</p><p><strong>2. Confusion matrix:</strong></p><p>Confusion matrix is a performance evaluation tool for classification models which provides visual representation of how well the model’s predictions match with the actual outcomes. A table that summarizes the performance of a classification model by showing true positives, false positives, true negatives, and false negatives by choosing some threshold of confidence.</p><p><strong>Threshold is the value confidence of the model represents the percentage of data items classified as positive classes. For example, if the confidence of the model is 0.5, 50 % of data items are classified as positive classes. By changing this threshold, the number of True positives, True negatives, False positives and False negatives changes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*8YRqAEYaCRB4f3bCQ1u0qw.png" /><figcaption>Confusion matrix values changes based on the threshold (ref. googledevelopers.com/machine learning/crash course)</figcaption></figure><p>For example, threshold is 0.90, and the probability of data item belonging to positive class is 0.8999 still that item is classified as negative (may be correct or incorrect).</p><p>The confusion matrix is organized as 2 ˟ 2 matrix for a B<strong>inary classification model </strong>consisting of the following components.</p><p><strong>True Positive (TP): </strong>The model correctly predicts the positive class.</p><p><strong>True Negative (TN): </strong>The model correctly predicts the Negative class.</p><p><strong>False Positive (FP): </strong>The model incorrectly predicts the Positive class. (Type — I error).</p><p><strong>False Negative (FN): </strong>The model incorrectly predicts the Negative class. (Type — II error).</p><p><strong>Structure of a confusion matrix:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*GU_PfDbyJsBUpdz7ArCoRA.png" /><figcaption>Confusion Matrix</figcaption></figure><p>Accuracy can be calculated from confusion matrix using the formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*QKlswhGoC2WDHxl8gMUw1A.png" /></figure><p><strong>Example</strong></p><p>Consider a model predicting whether emails are spam:</p><ul><li><strong>TP</strong>: 60 spam emails correctly identified as spam.</li><li><strong>TN</strong>: 100 non-spam emails correctly identified as non-spam.</li><li><strong>FP</strong>: 20 non-spam emails incorrectly identified as spam.</li><li><strong>FN</strong>: 20 spam emails incorrectly identified as non-spam.</li></ul><p>This would result in the following confusion matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*iFKHSn87MW1BLXqDIeyotQ.png" /></figure><p>Accuracy = (60 + 100) / (60 + 20 +20 +100)</p><p>= (160/200) x 100 % = 80%</p><p>In a <strong>multi-class classification</strong> scenario, the confusion matrix is structured as an <strong>N x N</strong> matrix, where <strong>N</strong> represents the number of classes. Each row of the matrix corresponds to the instances of the predicted class, while each column corresponds to the instances of the actual class.</p><p><strong>Structure of a Multi-Class Confusion Matrix</strong></p><p>For example, if you have three classes (let’s say Class A, Class B, and Class C), the confusion matrix would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*Ns0cWYNVwsQ4v6iw3zprqQ.png" /><figcaption>confusion matrix of a multi class classification model</figcaption></figure><p>In this confusion matrix of <strong>N x N</strong> structure, True predictions can be calculated by summing the elements in the diagonal from left to right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*RoS_Sw1Ll2vzJHmjfG4DZA.png" /><figcaption>True predictions are highlighted</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*3W3kZ4obaMoMY8gZO1dg-w.png" /></figure><p>This calculation can be extended to any number of classes.</p><p><strong>Unlike the accuracy, confusion matrix also visualizes the types of errors that our model is making.</strong></p><p><strong>when to use confusion matrix?</strong></p><p>When working with the imbalanced data sets, accuracy do not work well. Confusion matrix allows us to understand the performance on all classes allowing us to identify which classes are being misclassified.</p><p><strong>Drawbacks of confusion matrix:</strong></p><p>a) In multiclass classification, as the number of classes increases matrix size increases, and it becomes difficult to interpret the performance of the model across all the classes effectively.</p><p>b) Misleading accuracy: When working with imbalanced datasets (i.e one class may have a greater number of instances, and the other class have very less instances) the accuracy may not be the good metric for validating the model through confusion matrix.</p><p>Example consider the data set consisting of cancer disease prediction from the x-ray image. Among 1000 people 1 cancer patient is there. This is an imbalanced dataset consisting of 999 non-cancers and 1 cancer patient. After predicting over the dataset using the classification model the following confusion matrix was constructed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*m_ynrtpkGZPl04n2yxMrEw.png" /><figcaption>confusion matrix of an imbalanced dataset-cancer prediction</figcaption></figure><p>In the above example if we calculate the accuracy of a model as accuracy=999+0/1000 = 99.9 %. Here, even though the accuracy of the model is 99.9 % the model cannot be used because, the 1 person who is suffering with cancer is not detected by the model and the person is considered as non- cancer patient and he/she would not take proper treatment which may cause his death.</p><p>so, even though the model is giving high accuracy this model cannot be used for this type of disease predictions. This is called as misleading accuracy.</p><p>In this example False Negatives are costly and may lead to serious consequences.</p><p>Now consider another example. In airport security, a false positive in detecting a weapon can lead to unnecessary delays, humiliation, or even legal consequences for innocent travelers. This can strain resources and create a negative experience for passengers. In this example innocent passengers are detected as fraud. i.e. False Positives are costly and may lead to serious consequences.</p><p>Considering the following confusion matrix of two models detecting the email is spam or not spam:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*lT_GDZkPYLzo8Jw93zR-fg.png" /><figcaption>confusion matrix of two models with same accuracy</figcaption></figure><p>The above two models are giving same accuracy. Then which model can be selected? It should be decided based on the problem we are dealing with. Whether False Positives are to be less, or False Negatives are to less which are leading to serious consequences as discussed in the cancer detection and fraud detection examples above. While detecting the errors False Negatives should be very less. While detecting the fraudulent False positives should be very less.</p><p>Confusion matrix discuss the overall accuracy of the model, but not the accuracy of the model in predicting original positives or original negatives which may lead to the serious consequences.</p><p>To overcome this, we need some more metrics which evaluates the accuracy of positive predictions made by the model and measure the ability of the model to identify all the relevant instances of the positive class. They are precision and recall.</p><p><strong>3. Precision: </strong>Precision is a classification metric to evaluate the accuracy of predicted positives. i.e. it is a fraction of what portion of predicted positives are truly positive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*v5AmhMbePT35kKKCGCWtjQ.png" /><figcaption>Precision formula</figcaption></figure><p>Model 1 has precision = 100/130 = 76.9% and model 2 has a precision of 100/110 = 90.9%</p><p>Coming to this prediction of spam emails, the false positives may lead to serious consequences by losing schedules and opportunities if the mail goes to junk folder. So, model 2 should be selected for email filtering application.</p><p><strong>High precision means the model has more true positives out of total positive predictions. i.e. there is a less chance of predicting the False positives.</strong></p><p><strong>When to use Precision?</strong></p><p>The applications where False Positives are to be less like spam filtering, fraud detection etc. precision measure is to be used.</p><p><strong>Drawbacks of Precision:</strong></p><p><strong>a) Ignores False Negatives:</strong> Precision does not provide information about how many true positives are predicted as false negatives because it focuses only on positive predictions.</p><p><strong>b) Does not reflect overall performance:</strong> Precision reflects only the accuracy of positive predictions but not the overall performance of the model. To understand the overall performance other metrics like recall and F1 score are to be considered.</p><p><strong>c) Class imbalance sensitivity:</strong> In case of imbalance datasets, high precision might be easier to achieve if the positive class is rare. This can give a false sense of model performance if not evaluated with recall and other metrics.</p><p><strong>d) Trade- off with recall: </strong>Focusing on precision may miss the true positives and reduce the recall. There should be a trade-off between precision and recall.</p><p><strong>e) Threshold dependency: </strong>Precision depends on the threshold we set for the classifying the positive instances. As the precision changes with the threshold value, it is difficult to consider it as a standalone metric.</p><p><strong>4. Recall:</strong> Some applications like disease identification or prediction there is a need to reduce the False Negatives (i.e. missing the disease prediction) which may lead to serious consequences. We need to correctly predict the positive instances out of all actual positives. That means no positive instance should be missed and predicted as False Negative.</p><p><strong>Recall is also known as sensitivity or true positive rate (TPR) </strong>is a metric used to evaluate the effectiveness of the machine learning model in identifying all relevant instances within a dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*TwFPJ9EngRh7_4X-Eqddmw.png" /><figcaption>Recall calculation formula</figcaption></figure><p>For the above example</p><p>model 1 recall = 100/100+170 = 37%</p><p>model2 recall=100/190+100 = 34.48%</p><p>The recall value of model 1 is 37% means 37% of actual positives are correctly predicted and model 2 recall is 34.48% means only 34.48% of actual positives are correctly predicted.</p><p><strong>The high value of recall indicates that the model is effectively identifying the positive instances. Low recall indicates that the model is failing to identify the actual positive instances, in otherwards there are many false negatives. This may be due to high decision threshold or an imbalance in the training data.</strong></p><p><strong>Implications of low recall:</strong></p><p>Missed positive cases can be the critical in applications where identifying all positives is important. ex: medical diagnosis and fraud detection. These missed positive cases may lead to serious consequences.</p><p><strong>When to use Recall?</strong></p><p>The applications where False Negatives are to be less like cancer prediction recall measure is to be used.</p><p><strong>Drawbacks of Recall:</strong></p><p><strong>a) Ignores False Positives:</strong> Recall does not provide information about how many true negatives are predicted as false positives because it focuses only on false negative predictions.</p><p><strong>b) Does not reflect overall performance:</strong> Recall reflects only the accuracy of positive predictions from actual predictions but not the overall performance of the model. To understand the overall performance other metrics like precision and F1 score are to be considered.</p><p><strong>c) Class imbalance sensitivity:</strong> In case of imbalance datasets, high recall might be easier to achieve if the positive class is rare. This can give a false sense of model performance if not evaluated with precision and other metrics.</p><p><strong>d) Trade- off with precision: </strong>Focusing on recall may come at the expense of precision. There should be a trade-off between precision and recall.</p><p><strong>e) Threshold dependency: </strong>Recall depends on the threshold we set for the classifying the positive instances. As the recall changes with the threshold value, it is difficult to consider it as a standalone metric.</p><p>Note: When False Positive accuracy is a major concern precision measure is to be used and when the False Negative is a major concern recall measure is to be used.</p><p>Now, let us consider a classification problem where both false positives as well as false negatives are crucial and has equal significance, we can neither select precision nor recall alone. We need a new metric F1-score which makes use of both these precision and recall.</p><p><strong>5.</strong> <strong>F1-Score: F1-Score is a metric used to evaluate the performance of a classification model. It is useful when we need to balance both precision and recall in the case of imbalanced datasets.</strong></p><p>F1-score is the harmonic mean of precision and recall which gives a single metric that reflects both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*Q6v0fr2lJVzoQkf0YaaGGw.png" /></figure><p>Example: consider the Spam filtering in email classification model with the following confusion matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*YZlvHsGMjf1FAst1EXyXTQ.png" /></figure><p>Precision = (70/70+10) X 100 = 87.5% and Recall = (70/70+20) X 100 = 77.7% In spam filtering model sometimes False positives are concerned, and sometimes false negatives are concerned. so, we can use F1-score which make use of both precision and recall and generates a single measure.</p><p>F1-Score = (2 X 87.5 X 77.7)/ (87.5 +77.7) = 82.3 % which is a single good measure balancing both precision and recall.</p><p><strong>When to use F1-score?</strong></p><p>When we need to balance both precision and recall in the case of imbalanced datasets, we can use F1-score.</p><p><strong>6. Specificity or True Negative Rate (TPR): </strong>is a metric used to evaluate the effectiveness of the machine learning model in identifying all negative instances within a dataset.</p><ul><li>High Specificity: Indicates that the model is good at correctly identifying negative instances, which is important in scenarios where false positives can have serious consequences.</li><li>Low Specificity: Suggests that the model is incorrectly classifying many negative instances as positive, leading to a high number of false positives.</li></ul><p><strong>Specificity is crucial in applications where it is important to correctly identify negative instances, such as in medical testing, fraud detection, and security screening. It complements other metrics like sensitivity (recall) to provide a comprehensive evaluation of a model’s performance.</strong></p><p><strong>Receiver Operating Characteristic — Area Under Curve (ROC-AUC):</strong></p><p><strong>7. False Positive Rate (FPR): </strong>It is the proportion of all actual negatives that are incorrectly classified as false positives.</p><p>FPR = incorrectly classified true negatives /all actual negatives</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/1*tDdENX_enWoocBXf0ppFjw.png" /></figure><p><strong>8. Receiver Operating Characteristic (ROC) — Area Under Curve (AUC):</strong></p><p>Need for ROC-AUC curve:</p><p>Till now the model performance metrics discussed are based on one threshold value of the confidence of probability of the data to be in the positive class. If this threshold changes all the performance metrics like True positives, false positives, true negatives and false negatives also changes. If these values change in turn confusion matrix, accuracy, precision, recall and F1score also changes. If we need to evaluate the model’s performance across all the thresholds at a time, we need different tools they are</p><ol><li>ROC-AUC curve and 2. PR curve</li></ol><p>ROC curve is a graphical representation of performance of a binary classifier or multiclass classifier at varying threshold values. ROC curve is a plot of True Positive Rate (TPR) against the False Positive Rate at each threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*X7TILwrvsGRQUCEM.gif" /><figcaption>gif representing the ROC- AUC curve. Ref: <a href="https://nustat.github.io/STAT303-2-class-notes/Lec7_logistic_regression.html">Data Science II with python (Class notes) — 4 Logistic regression</a></figcaption></figure><p>The perfect model has a TPR of 1 and FPR of 0. This can be represented as (0,1) provided all other thresholds are ignored.</p><p>Area Under ROC curve (AUC) represents the probability that the model if given a randomly chosen positive and negative example will rank the positive higher than the negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AhVG4IrEeYRyh5SD.png" /><figcaption><em>ROC and AUC of a hypothetical perfect model</em></figcaption></figure><p>The higher the AUC value the better the model is.</p><p><strong>Drawbacks of ROC-AUC curve: It is difficult to communicate the model’s performance effectively with non-technical stake holders.</strong></p><p><strong>9. Precision — Recall curve (PR curve): </strong>This curve is a graphical representation used to evaluate the performance of the binary classifier. It plots precision against the recall at various threshold settings. Unlike ROC curves PR curves focuses on performance of the positive class. PR curves can be used to compare the performance of the different models. Higher Area Under Curve (AUC) of the PR curve of a model indicates the better that model is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8fWTsoS4cpx_KY_i.png" /><figcaption>PR curve — AUC: ref. github.io</figcaption></figure><p>By analyzing the PR curve, the threshold value can be optimized to maximize the precision or recall or balance both depending up on the application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6977da248981" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[REGULAR EXPRESSIONS WITH PYTHON]]></title>
            <link>https://medium.com/@kanchanakanta/regular-expressions-with-python-7c5c8800812f?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c5c8800812f</guid>
            <category><![CDATA[regular-expressions]]></category>
            <category><![CDATA[meta-characters-in-regex]]></category>
            <category><![CDATA[regex-validation]]></category>
            <category><![CDATA[regex-functions]]></category>
            <category><![CDATA[pattern-matching]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Thu, 07 Nov 2024 09:23:44 GMT</pubDate>
            <atom:updated>2024-11-07T09:23:44.263Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v0HeV4cm07QQ2X25UYyAWg.jpeg" /></figure><p>Regular expressions are sequences of characters that define a search pattern mainly used for string matching and manipulation. These patterns can be used to</p><p>1. To test whether a string matches a pattern Ex: email id</p><p>2. To find the specific substrings within a string.</p><p>3. To replace the substrings that match a pattern with new text.</p><p><strong>Components of Regular Expressions:</strong></p><p>1. <strong>Literals:</strong> These are the characters that match themselves.</p><p>Example: the regex ‘a’ match with a character ‘a’</p><p>2. <strong>Metacharacters:</strong> These are the characters that have special meaning</p><p>Available meta characters are:</p><p>‘ . ‘: dot matches any character except the new line character.</p><p>‘* ‘: Asterisk matches zero or more occurrences of the preceding element.</p><p>‘+ ‘: Plus, character matches one or more occurrences of the preceding element.</p><p>‘ ? ‘: The question mark matches zero or one occurrence of the preceding element.</p><p>‘| ‘: Pipe symbol acts as a logical OR to match the expression before or after that symbol.</p><p>‘() ‘: Parenthesis groups expressions and captures the matched text.</p><p>‘[ ] ‘ : Square brackets define a character class to match any one of the enclosed characters.</p><p><strong>Operations on Regular Expressions for constructing the complex patterns:</strong></p><p>Regular Expressions support the following operations for constructing complex patterns.</p><p>1. Concatenation: This combines the two expressions to match sequences, one followed by the other. For example, the ‘ab ‘expression matches ‘a ‘followed by ‘b ‘.</p><p>2. Union: It is represented by the ‘| ‘operator to match either of two expressions.</p><p>Example: a|b matches a or b</p><p>3. Kleen Star: This ‘ * ‘ operator used to match zero or more repetitions of an expression.</p><p>Example: a* matches ‘ ‘, ‘ a’ , ‘aa’, ‘ aaa’ etc..</p><p>4. Positive Closure: This ‘ + ‘operator is used to match one or more repetitions of an expression.</p><p>Example : ‘ a’, ‘aa’, ‘aaa’ etc..</p><p><strong>Built — In functions provided by re-module:</strong></p><p>To work with regular expressions, we need to import re module.</p><p><strong>Commonly used Functions in re module are:</strong></p><ol><li><strong>re. search ()</strong>: This function searches for a pattern in a string and returns a match object if found. Otherwise, it returns none.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*_Qhd7ufGFbPXPKvspZqQDw.png" /></figure><p>2. <strong>re. match( )</strong>: This function is used to check for a match only at the beginning of a string. If the pattern matches at the beginning of the string it returns a match object, otherwise it returns none.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*bwuYW5lND3wwmfUMnuwAVQ.png" /></figure><p>3. <strong>re.findall( )</strong>: This function is used to return a list of all non-overlapping matches of the patterns in the string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*SkK86Op9tOVUBI6z16xrzg.png" /></figure><p>4. <strong>re.finditer( )</strong>: This function will return an iterator yielding match objects for all non-overlapping matches of the patterns in the string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*N6wI0c28d-HHJl6HD9R-Wg.png" /></figure><p>5. <strong>re. split ()</strong>: This function is used to split a string by the occurrences of a specified pattern. It returns a list of substrings. This function is mainly useful to break the string into parts based on delimiters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*xpfkHlOC_c_SAiqAoVWkSg.png" /></figure><p>6. <strong>re. sub ()</strong>: This function is used to replace the occurrences of a specified pattern in a string with a replacement string. This function is very useful for text manipulation, such as cleaning up data or formatting strings.</p><p>Syntax: re. sub (pattern, replacement, string, count=0, flags=0)</p><p>Where</p><p>· <strong>pattern</strong>: The regex pattern to search for.</p><p>· <strong>replacement</strong>: The string to replace the matched pattern with.</p><p>· <strong>string</strong>: The input string where the search and replace will occur.</p><p>· <strong>count</strong>: Optional. The maximum number of pattern occurrences to replace. The default is 0, which means replace all occurrences.</p><p>· <strong>flags</strong>: Optional. Modifiers that change how the regex is interpreted (e.g., re.IGNORECASE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*CCW8nEPjSHHbdUIAdWlsmA.png" /></figure><p>7. <strong>re.subn( )</strong>: This function is similar to re. sub( ). It substitutes the string and also returns a tuple consisting of substituted string and count specifying how many times that string got substituted. This is useful when we want to know the count of substitutions that occurred during the operation.</p><p>Syntax: re.subn(pattern, replacement, string, count=0, flags=0)</p><p>Where</p><p>· <strong>pattern</strong>: The regex pattern to search for.</p><p>· <strong>replacement</strong>: The string to replace the matched pattern with.</p><p>· <strong>string</strong>: The input string where the search and replace will occur.</p><p>· <strong>count</strong>: Optional. The maximum number of pattern occurrences to replace. The default is 0, which means replacing all occurrences.</p><p>· <strong>flags</strong>: Optional. Modifiers that change how the regex is interpreted (e.g., re.IGNORECASE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*6EvRFtt61rvxVjhRuFQhvw.png" /></figure><p>8. <strong>re.compile( )</strong>: This function is used to compile a regular expression pattern into a regex object. This will improve performance when the same pattern is used many times because it avoids recompiling the pattern each time it is used.</p><p>Syntax: re.compile(pattern, flags=0) Where</p><p>· <strong>pattern</strong>: The regex pattern you want to compile.</p><p>· <strong>flags</strong>: Optional. Modifiers that change how the regex is interpreted (e.g., re.IGNORECASE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*AzroN-Wkk8LNB9l9_u2bbA.png" /></figure><p>· In this example, re.compile(r’[a-zA-Z0–9._%+-]+@[a-zA-Z0–9.-]+\.[a-zA-Z]{2,}’) compiles a regex pattern that matches typical email addresses.</p><p>· The compiled pattern is stored in the variable email_pattern.</p><p>· We then use email_pattern.findall(text) to find all email addresses in the provided text. The output shows the list of found email addresses.</p><p><strong>SETS IN REGULAR EXPRESSIONS:</strong></p><p>In regular expressions, sets allow to define group of characters that are to be matched. Sets are also called as character classes.</p><p><strong>Types of sets:</strong></p><p>1. Basic Sets: A set can be created by enclosing characters in square brackets [ ].</p><p>Example: [abc] matches any single character that is either a or b or c</p><p>2. Ranges: These are used to specify a range of characters using a hyphen.</p><p>Example: [ a — z ] matches any lowercase letter from ‘a ‘to ‘z ‘</p><p>[ 0–9 ] matches any digit.</p><p>3. Negation: To match any character not in the set, the caret ‘ ^ ‘symbol is used at the beginning of the set.</p><p>Example: [^abc] matches any other characters except ‘a’, ‘b’ or ‘c’.</p><p>Example for sets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*vRYsd2LQ3fnMd1z2TYADLg.png" /></figure><p><strong>SPECIAL SEQUENCES IN REGULAR EXPRESSIONS:</strong></p><p>In regular expressions, special sequences are shorthand notations that represent certain types of characters or groups of characters.</p><p>Some commonly used special sequences are:</p><p>1. ‘\d ‘: It matches any digit and is equal to the set [0–9]</p><p>Example: \d {2} matches any 2 digit number.</p><p>2. ‘ \D ‘ : It matches the non-digit character and is equivalent to [^ 0-9]</p><p>Example: \D+ matches one or more non-digit characters.</p><p>3. ‘ \w ‘: Matches any alphanumeric character (letters and digits) plus underscore and is equivalent to [a-zA-Z0–9_]</p><p>Example: \w+ matches one or more-word characters.</p><p>4. ‘ \W ‘: It matches any non–word character and is equivalent to [^a-zA-Z0–9_]</p><p>Example: \W+ matches one or more non-word characters</p><p>5. ‘ \s ‘ : It matches any white space character. (spaces, tabs, newlines)</p><p>Example: \s+ matches one or more white space characters.</p><p>6. ‘ \S ‘: It matches any non-whitespace character.</p><p>Example: \S+ matches one or more non-whitespace characters.</p><p>7. ‘ \b ‘ : It matches a word boundary (the position between a word character and non-word character)</p><p>Example: \bword\b, this matches the word word only if it appears as a whole word</p><p>8. ‘ \B ‘ : It matches a position that is not a word boundary.</p><p>Example: \Bend matches the end only if it is part of a larger word like a bending</p><p>9. ‘\n\t ‘: Matches specific control characters such as new line (\n) or new tab(“\t)</p><p>Example programs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*bSCYl5HJWTMgfhUTTIc6hg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*KM1YMHwhR5Mx54LebagFSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*TwPsCkcr2w2XAg2tsm6kgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*JF1LBx1AWcd83edLp2Gnzw.png" /></figure><p>#Match all new line and tab characters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*RdyCsN7UYIukyxgx9UWreg.png" /></figure><p><strong>Example Applications of Regular Expressions:</strong></p><p>1. Data Wrangling: regex is used to clean and prepare data by removing unwanted characters, extracting specific fields from messy data or reformatting strings.</p><p>2. Data Validation: In web applications regex helps in validating the user input, ensuring that data like email addresses, and phone numbers meet specific formats.</p><p>3. Web Scrapping: Regex is used to extract specific data from web pages.</p><p>4. Text Processing: Regex is widely used in Natural Language Processing (NLP) for tasks like tokenization.</p><p>5. Pattern matching: Regex is used in various programming languages for pattern matching, enabling developers to implement complex search algorithms.</p><p>6. Security applications: Regex is used security software to detect patterns associated with malware activity.</p><p><strong>References:</strong></p><p>1. <a href="https://en.wikipedia.org/wiki/Regular_expression">https://en.wikipedia.org/wiki/Regular_expression</a></p><p>2. <a href="https://blog.finxter.com/what-are-regular-expressions-used-for-10-applications/">https://blog.finxter.com/what-are-regular-expressions-used-for-10-applications/</a></p><p>3. <a href="https://www.analyticsvidhya.com/">https://www.analyticsvidhya.com/</a></p><p>4. <a href="https://docs.python.org/3/library/re.html">https://docs.python.org/3/library/re.html</a></p><p>5. <a href="https://regex101.com/">https://regex101.com/</a></p><p>6. <a href="https://www.w3schools.com/python/python_regex.asp">https://www.w3schools.com/python/python_regex.asp</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c5c8800812f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GRADIENT DESCENT]]></title>
            <link>https://medium.com/@kanchanakanta/gradient-descent-d7acea2658a7?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7acea2658a7</guid>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[learning-rate]]></category>
            <category><![CDATA[updating-parameters]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Sun, 03 Nov 2024 14:09:38 GMT</pubDate>
            <atom:updated>2024-11-03T14:09:38.675Z</atom:updated>
            <content:encoded><![CDATA[<p>GRADIENT DESCENT</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unDwgnxn8SYHw1iuC78Sqg.jpeg" /></figure><p>Gradient: A gradient is a vector that measures the change in all weights for the change in the error or loss. It indicates the direction and rate at which the model’s parameters must be adjusted to minimize the error.</p><p>Gradients are used in the Gradient Descent optimization algorithm, which trains machine learning models by minimizing the error between predicted and actual results. This algorithm minimizes the differentiable function by taking steps in the opposite direction of the gradient. It is widely used to optimize machine learning algorithms’ cost or loss function.</p><p><strong>Gradient Descent (GD) algorithm:</strong></p><p>Step1: Start with an initial set of model parameters</p><p>Step 2: Calculate the gradient of the cost function to the parameters.</p><p>Step 3: Adjust the parameters in the opposite direction of the gradient.</p><p>Step 4: Repeat this process until the cost function is minimized.</p><p>Applications of Gradient Descent algorithm:</p><p>1. Linear Regression: It helps to find the best-fit line by minimizing the mean squared error between the predicted and actual values.</p><p>2. Logistic Regression: The Gradient Descent algorithm is used to optimize the parameters of the logistic function to classify the data points correctly.</p><p>3. Neural networks: GD adjusts the weights and biases in neural networks to minimize the loss function.</p><p>4. Support Vector Machines: GD helps in finding the optimal hyperplane that separates the classes.</p><p>5. Reinforcement Learning: GD helps in optimizing value functions to improve decision-making in dynamic environments.</p><p>Let us understand this algorithm with the Linear Regression error function MSE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*JGRGa7ZhtFaN_TcHGPWKXA.png" /></figure><p>Where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*0X8DzcQagLKXexJC2w4Bng.png" /></figure><p>The loss function evaluates how well a linear regression model fits the data and guide the optimization process to improve accuracy.</p><p>If the line does not fit properly the difference between the actual values and predicted values would be more and the sum of squares of this difference is more. Then the model’s accuracy would be less.</p><p>To improve the accuracy of the model the value of MSE should be reduced to approximate to zero or minimum.</p><p>There are two ways to reduce this loss function value or MSE.</p><p>1. Ordinary Least Squares method (OLS) and</p><p>2. Gradient Descent method</p><p>Let us consider the example data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*VHGW-qWh7wNlRjYOgbP75g.png" /><figcaption>the points are fitted with linear regression model</figcaption></figure><p>In the above example, the linear regression line is fitted to better predict the values.</p><p>The equation for the prediction line or regression line is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/1*_kNrwV8dTYvLn9lAkdQ2nA.png" /></figure><p>The error function is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/1*WAbp0MpnN8XgteUY0Um9sQ.png" /></figure><p>let us call it SS (sum of squares) and find the value of SS for the above data</p><p>Note: For functional convenience here after the term 1/n is ignored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*v7kdWOCo2HQlfsT2oFCdlQ.png" /></figure><p>SS is a loss function that depends on two factors slope of the line and intercept of the line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*fVNPYq0iL1XW30uzyPHAaw.png" /></figure><p>By changing these parameters, we can minimize this loss function.</p><p>As it is difficult to observe the change in the loss function by changing both slope and intercept at once, let us first assume slope =1 and change the intercept to minimize the error.</p><p>Step1: usually the initial intercept value should be selected randomly. Let us initiate the intercept with zero</p><p>Then error = (15–8)^2+(30–18)^2+(25–24)^2</p><p>SS=49+144+1 = 194</p><p>Step 2 : Now update the intercept value to 5</p><p>Then SS=(15-(8+5))^2+(30-(18+5))^2 +(25-(24+5))^2</p><p>=(15–13)2+(30–23)2+(25–29)2</p><p>=4+49+16=69</p><p>By changing the intercept value from 0 to 5 the error function is minimized and reduced to 69</p><p>Now update the intercept value to 10</p><p>SS==(15-(8+10))2+(30-(18+10))2 +(25-(24+10))2</p><p>=(15–18)2+(30–28)2+(25–34)2</p><p>=9+4+81=94</p><p>Choose intercept== 6</p><p>SS=1+36+25=72</p><p>Choose the intercept to 5.5</p><p>2.25+42.5+20.25=65</p><p>Following the same procedure let us calculate the error with different values of intercept keeping the slope constant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*IT18k57ABag-mxbF1kl3Fw.png" /></figure><p>This is a trial-and-error process where we need to select the best intercept</p><p>value to make the loss function near zero.</p><p>In this example, we have checked only for 5 values, and at some intercept values, we got a minimum of 65 among all the loss function values. Even if we check for 100 values of intercept, the minimum SS value may further be reduced with different intercept value.</p><p>There may be a chance to get a minimum SS value where we did not check that intercept value and we may miss that minimum loss value.</p><p>If we draw the graph of the loss function which can be written as f(x)=x^2 it can be shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*wnJls_08Ecl7FCH91H0HZw.png" /><figcaption>loss function follows a parabolic shape</figcaption></figure><p>From the graph, we can observe that the minimum value of the curve can be found at which the slope of the line becomes zero.</p><p>The slope of the line becomes zero where the derivative of the loss function becomes zero.</p><p>The loss function for the linear regression is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*-PsnAc1PEnnEizSrsN8ScA.png" /></figure><p>By considering the slope constant beta1=1 and substituting the input values the equation becomes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*GPjvXnf22Ws6zAKdu713bg.png" /></figure><p>If we know at some point of the curve slope becomes zero, then that would be the best intercept to get the least error. The derivative of the loss equation becomes zero at slope=0. so, let us equate SS to zero to find the best intercept value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*ZrmGahaoh6zFF38YjRoFOw.png" /></figure><p>If we do not know whether at any point of loss line slope becomes zero again that beta0 would be calculated based on the trial-and-error method.</p><p>Computing and searching for the minimum error at different intercept values is a repetitive process and becomes computationally expensive.</p><p>At the same time there is a chance that at any point in the curve, there may not be slope =0. For example, in the following graph slope is not zero at any point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*t3KTvhJ-F-RzCgf438xIkA.png" /></figure><p>As the slope of the line does not become zero at any point, we cannot equate the derivative of the loss equation to zero and we need to do trial and error for all beta0 values till we get the minimum error and still we need to decide what that minimum error should be approaching towards zero.</p><p>Gradient Descent can work with the curves where there is a scope of having zero slope. Gradient Descent can also find the minimum of error with the curves whose slope is not equal to zero at any point.</p><p>Step 1: Initialize some beta0 and find the slope of the error line by taking the partial derivative of the error function with respect to beta0.</p><p>Step 2: Find the MSE at that beta0. If it is far from zero, then update beta0=beta0-new slope</p><p>Step 3: Repeat this procedure until we reach the minimum error approximating towards zero.</p><p>By using this process, we will not have control over the updated step size.</p><p>Considering the above example: At beta0=0, the derivative of the loss line is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*-6C6vOGENRF7rAwA82OVCg.png" /></figure><p>Again, there is a big shift in error from positive to negative. By following this procedure, the error value is taking bigger steps and may change between positive and negative values without converging to the minimum value as shown in the figure below and the model may not converge.</p><p>This process is to be repeated until we reach the minimum error with optimal values of beta0 and beta1.</p><p>There are two methods to choose to stop this iterative process.</p><p>1. Till the step size becomes &lt;=0.001 or</p><p>2. By choosing the number of iterations known as epochs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*blmhM5cyKElkL3JkP0HfYQ.png" /></figure><p>Considering the number of epochs=1000 and update process beta0=beta0-new slope. Change in MSE with a change in beta0 is visualized in the following gif.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*JjIap2kkf7eQYvYFvedDZA.gif" /><figcaption>Gradient Descent without learning rate(alpha)</figcaption></figure><p><strong>Optimization:</strong></p><p>To optimize this algorithm, Gradient Descent introduces a hyperparameter alpha, which is called the learning rate, and specifies how fast the model converges to minimum error. Alpha is a floating point that determines the step size at each iteration of the optimization algorithm.</p><p>The equation of the Loss function is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*p4lbXubHryM7sDJE78HrUw.png" /></figure><p>Till now we have chosen beta1 as constant and updated beta0 value to find the optimum value of beta0 at which error becomes minimum. But in real</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*h2hIJcgludIoJFwjMdWCMA.png" /></figure><p>i.e. the loss function is a function of two variables beta0 and beta1. We need to find the optimum values of these coefficients at which error becomes minimum.</p><p>To find the optimum values of these parameters, while calculating the beta0 the beta1 is kept constant, and while calculating the beta1, the beta0 is kept constant. Find the partial derivative with respect to beta0 and update</p><p><strong>beta0=beta0-learningrate*new slope</strong></p><p>To find the optimum value of the parameters while calculating the beta1, the beta0 is kept constant, and the partial derivative with respect to beta1 and update</p><p><strong>beta1=beta1-learningrate*new slope</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*t-JgCcJ3FapuuI_NG5lUvg.png" /></figure><p>where alpha is the learning rate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*GyZIcQhDzK_wJp5G3m8B0A.png" /></figure><p>In the first step, the error has shifted from -40 to -37.6 using learning rate and the error has been shifted from -40 to 200 without learning rate. The impact of the learning rate is huge in updating the step size. When the error is far from the minimum, the step size will be larger and when the error is nearer to the minimum, the step size will be smaller.</p><p>The learning rate or alpha should not be too high or too low. If the alpha value is too low the model takes more time to converge. If the alpha value is too high the model may not converge and bounces around the weights and bias.</p><p>The following figure shows the change in the error by updating the values of beta0 and beta1 and it also displays the change in line equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*_rHQFao-UkwEt2sqQBgu_g.gif" /><figcaption>change in MSE with an update in beta0 and beta1</figcaption></figure><p>References:</p><p>1. IBM.com</p><p>2. Machinelearningmastery.com</p><p>3. Builtin.com</p><p>4. Campus.com</p><p>5. Datacamp.com</p><p>6. <a href="https://www.cs.cornell.edu/courses/cs4780/2017sp/lectures/lecturenote07.html">Lecture 7: Gradient Descent (and Beyond)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7acea2658a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TYPES OF FUNCTIONS IN PYTHON]]></title>
            <link>https://medium.com/@kanchanakanta/types-of-functions-in-python-e4865064f288?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4865064f288</guid>
            <category><![CDATA[function-generators]]></category>
            <category><![CDATA[higher-order-function]]></category>
            <category><![CDATA[recursion-function]]></category>
            <category><![CDATA[lambda-function]]></category>
            <category><![CDATA[python-functions]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Mon, 14 Oct 2024 15:45:55 GMT</pubDate>
            <atom:updated>2024-10-16T03:52:31.029Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*0uxOtH-fnhzrfiv353o0RQ.jpeg" /></figure><p>FUNCTION DEFINITION:</p><p>A function is a block of code written to perform a specific task. Functions allows us to break the bigger problems into smaller modules.</p><p><strong>The syntax for defining a function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*F2sfjsHfvIdwWGsDR9A2_Q.png" /></figure><p><strong>For a defining a function five key components are required.</strong></p><p>1. A keyword ‘def’ to represent we are defining a function (mandatory)</p><p>2. A function name (to call later or differentiate the present function with other functions) (mandatory)</p><p>3. Arguments to be passed (Optional)</p><p>4. Doc string (optional but good to practice) to describe what the function does.</p><p>5. Function body consisting of one or more executable statements</p><p>6. A return statement (Optional) to return a value from the function.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*VmtCcDrvVZZ4vRMdTDx3CQ.png" /></figure><p>This function call is passing two arguments to the function definition and executes all executable statements in function definition and returns the result.</p><p><strong>Uses of functions:</strong></p><p><strong>1. To organize the code</strong>: functions organize the code into reusable blocks making the program easier to understand, debug and more efficient.</p><p><strong>2. Code re-usability</strong>: Functions promotes the re use of the code without writing the same code repeatedly.</p><p><strong>3. Improve modularity</strong>: Functions allow programs to be divided into small and manageable modules.</p><p><strong>Types of arguments in functions:</strong></p><p>When working with functions we come across two terms</p><p><strong>1. Actual Parameters and 2. Formal Parameters.</strong></p><p><strong>Actual parameters:</strong> These are the values or expressions that are passed to a function when it is called. They are also known as arguments.</p><p><strong>Formal parameters:</strong> These are the values that are defined by the function that receives values when the function is called.</p><p>Actual and Formal parameters help in allowing the functions to work independently of each other and data can be passed between the modules using parameters.</p><p><strong>Types of function arguments:</strong></p><p>There are 5 types of arguments available in python:</p><p><strong>1. Default arguments</strong></p><p><strong>2. Keyword arguments</strong></p><p><strong>3. Positional arguments</strong></p><p><strong>4. Arbitrary positional arguments</strong></p><p><strong>5. Arbitrary keyword arguments.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*OSpcnTEQXoXuJCCL3JBIXw.png" /><figcaption>types of arguments</figcaption></figure><p><strong>Default argument</strong>s:</p><p>Default arguments are the function arguments that are to be used when no arguments are passed from the function call. Default arguments are represented as <strong>argument_name=value</strong>.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*U6GZpcAOSEPdj5wx7m7Onw.png" /></figure><p>In the above function, b=5 and c=10 are the default values to those parameters. If no actual parameters are sent from function call, then b=5 and c=10 will be considered and a value to be passed for the variable ‘a’. if some values are sent for b and c as-well from the function call they override the default values.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*-nYjJUu90G8iyhpxLYb5UQ.png" /></figure><p>In the above function b and c values are override by the actual parameters sent from function call. There fore we get the output of 60.</p><p>Default arguments are optional.</p><p>Default arguments should follow non-default arguments. Otherwise, the function will not be executed and throws an error as shown in the example below.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*GVzExr0Awq1lNrQF-kQaRQ.png" /></figure><p><strong>Keyword Arguments:</strong></p><p>Keyword arguments are also called as named arguments. when the arguments are passed from the function call to the function definition these are identified by the specific names of the arguments. As these are identified by the names there is no need to maintain the order of passing the actual arguments.</p><p>Keyword arguments are described as<strong> kwarg=value.</strong></p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*3FAvy8kLnfyi0n0BBwCZDg.png" /></figure><p>All the keyword arguments in the function call should match the parameters in function definition. Otherwise it throws an error.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*zxBSs2gmdxU9hXWemGOVMA.png" /></figure><p><strong>Positional arguments:</strong></p><p>These are the actual arguments passed from function call without giving the argument names. These arguments have to follow the same order as the parameter defined in the function definition.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*e0frsej4-yNSCdZ43gP2TA.png" /></figure><p>In this example keyword argument names are not given in function call. The values passed from actual arguments to the parameters in the function definition in the same order. 10 is occupied in a,20 is occupied in b and 30 is occupied in c.</p><p>We can also use these positional arguments in combination with keyword arguments, but Keyword arguments should follow positional arguments. Otherwise, it throws an error.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*L_VrAKkB9aS4gwEKMbliug.png" /></figure><p><strong>Arbitrary arguments:</strong></p><p>If we do not know the number of parameters needed in the function in advance then we can use arbitrary arguments. Arbitrary arguments means variable length arguments. Arbitrary arguments come in two types.</p><p><strong>1. Arbitrary positional arguments and 2. Arbitrary keyword arguments</strong></p><p><strong>Arbitrary positional arguments:</strong></p><p>Arbitrary positional arguments are represented with a special symbol asterisk ( * ) before the parameter in function definition to hold the non- key variable length arguments. Here * indicates we can have 0 or more parameters. These arguments would be sent in the tuple.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*vINTFB294PKwp6foCnNSZQ.png" /></figure><p>In the above example x is the parameter that may accept 0 or more arguments. We can pass any number of arguments. By using this arbitrary positional arguments we cannot pass the keyword arguments.</p><p><strong>Arbitrary keyword arguments:</strong></p><p>To pass the variable length keyword arguments python has a keyword kwargs preceded by **. In the function we use double asterisk (**) before the parameter. The arguments are passed as a dictionary and form as a dictionary inside the function definition. So, we can access them as parameter name.items.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*EjPylPtZqiStR_hjftJZTg.png" /></figure><p><strong>There are six types of functions in python.</strong></p><p><strong>1. Built — in functions</strong></p><p><strong>2. User — defined functions</strong></p><p><strong>3. Recursion functions</strong></p><p><strong>4. Lambda functions</strong></p><p><strong>5. Higher — Order functions and</strong></p><p><strong>6. Generator functions</strong></p><p><strong>BUILT-IN FUNCTIONS:</strong></p><p>These are the functions that come with python. Example: print(), len() etc..</p><p>Example:</p><p>print(“hello , world!”)</p><p># in this example print is the in-built function which prints the string given as an argument or parameter</p><p>Output: hello, world!</p><p>In this example there is no function definition, no return statements. We can execute these functions directly by calling the functions and passing arguments.</p><p><strong>USER DEFINED FUNCTIONS:</strong></p><p>These are the functions defined by the users by using the keyword def.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*pchmaCf1pA60wZLRZYkFYQ.png" /></figure><p><strong>Output: hello, Girish</strong></p><p>in the above example ‘def’ is the keyword , greet is the function name, name is the argument or parameter and print is the executable statement.</p><p><strong>RECURSION FUNCITONS:</strong></p><p>In python a recursion is a programming concept where a function calls itself by dividing the bigger problem into smaller similar subproblems.</p><p>Recursion is useful in scenarios where a problem can be divided into smaller similar problems like tree traversal process, divide and conquer algorithms (example: merge sort and quick sort), back-tracking algorithms (example: sudoku and N-queens problem) and dynamic programming algorithms (Fibonacci series and knap sack problem)</p><p><strong>Syntax for recursion functions:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*JSEI5f71ZIfdzGdRZlgMsw.png" /></figure><p><strong>Features of recursion:</strong></p><p><strong>1. Base case</strong>: to prevent infinite recursion and stack overflow errors a base condition is needed under which recursion ends.</p><p><strong>2. Reductive step</strong>: each recursive call should bring the problem closer to the base case to ensure progress</p><p><strong>3. Memory consumption</strong>: recursive functions are more memory intensive, as each call adds a layer to the call stack.</p><p><strong>4. Readability</strong>: recursive functions can lead to clean and readable code.</p><p><strong>Advantages and Disadvantages of Recursion</strong></p><p><strong>Advantages</strong>: Recursive functions can simplify code for complex problems, making it more readable.</p><p><strong>Disadvantages</strong>: Recursive functions can be less efficient in terms of memory usage due to the call stack. They may be slower than iterative counterparts if not optimized properly. Debugging recursive functions can be more challenging.</p><p><strong>Example :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*TdjdcIpef68ViYVlXoLLPA.png" /></figure><p>In this program the factorial function calls itself with a decremented value of x until it reaches the base case x=1. This recursive process can be visualized as a series of function calls, each one waiting for the next to complete, until the base case is reached and the results are combined to produce the final output.</p><p>Execution process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*l0Q3ltbXJMJum14Vs0CQHg.png" /></figure><p>Used python online compiler for visualization.</p><p>ref: <a href="#mode=display">Python Tutor code visualizer: Visualize code in Python, JavaScript, C, C++, and Java</a></p><p><strong>LAMBDA FUNCTIONS:</strong></p><p>Lambda functions are the small unnamed functions which uses the key word lambda. These functions are also called as anonymous functions because there will not be any name to these functions.</p><p><strong>These functions are used</strong></p><p>1. when the function is to be used for shorter time</p><p>2. Calculations are to be made on the fly</p><p>3. Where we need inline functions which does not require any formal function definition.</p><p>Syntax for using lambda function:</p><p><strong>lambda arguments: expression</strong></p><p>lambda functions can take any number of arguments and can have only one expression. Lambda functions will not have statements.</p><p>Differences between regular functions and lambda functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*a8wEb6cCiVN8_fWs0gva5A.png" /></figure><p>Example1: <strong>With one argument:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*ymPgOUypX-PyJZ19gVb9rA.png" /></figure><p>Example2 : with condition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*JS_IyhHyT1Sotf9sXWkMlw.png" /></figure><p><strong>With multiple arguments</strong></p><p>Example 3: with 2 arguments</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*WWj6Su96McoS9qRTzEnpKQ.png" /></figure><p>As this function is not having the name we can execute it by using the variable that is assigned with lambda function.</p><p>Example 4: with 3 arguments and condition checking:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*qqvDfX8o2NiUsgqI9lFCew.png" /></figure><p><strong>HIGHER ORDER FUNCTIONS:</strong></p><p>Higher order functions are the functions which take other functions as arguments or return them as the result. Higher order functions are the fundamental concept of functional programming. Where the functional programming is a programming paradigm which treats computation as an evaluation of mathematical functions and avoids changing state and mutable data.</p><p>Some common higher order functions in python are</p><p>1. map( ) function</p><p>2. filter ( ) function</p><p>3. reduce( ) function</p><p><strong>Map ( ) function:</strong> map is an in-built function that allow for functional programming and making it easier to work with collections of data.</p><p>Map function applies a given function to all items in an input sequence (iterable) and returns a map object which can be converted into a list or set etc.</p><p>Syntax: map(function, iterable )</p><p>Example :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*8yymIjssaCbf0eAZTXyD3A.png" /></figure><p><strong>Reduce ( ) function: </strong>reduce is an in-built function that allow for functional programming and making it easier to work with collections of data.</p><p>Reduce function applies a rolling computation to sequential pairs of values in a list ( or any iterable) and returns a single value. This function is a part of functools module. To use this function we need to import functools module.</p><p>Syntax : reduce(function, iterable)</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*tX83MjxiiX8l09Um-PbHNA.png" /></figure><p>Execution process of reduce:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*nNuRrn8Wu2iOKiMUECLe5Q.png" /></figure><p><strong>Filter() function:</strong> filter is an in-built function that allow for functional programming and making it easier to work with collections of data.</p><p>Filter function constructs an iterator from elements of an iterable for which a function returns true and returns a map object which can be converted into a list or set etc.</p><p>Syntax : filter(function, iterable)</p><p>Example :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*rfedYMG7n_-tJz2H4M7_UQ.png" /></figure><p><strong>GENERATOR FUNCTIONS:</strong></p><p>Generator functions are a special kind of functions that return an iterator, which we can iterate over one value at a time. Generator functions evaluate items on demand and only one item is stored in memory at one time. This helps in avoiding the memory errors.</p><p>Generators use function and yield statement instead of return statement.</p><p>Syntax for creating a generator function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*Nfg2uG_K3E7KvRt_AeWEOw.png" /></figure><p>Key features of generator functions are</p><p>1. Memory efficient: Generator functions generate values on the fly and do not store the entire sequence in memory.</p><p>2. State Retention: Generators maintain their state between successive calls, i.e. they can resume from the previous state where they left off.</p><p>3. Lazy evaluation: Values are produced only when requested, which is useful for large datasets or infinite sequences.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*CIpXc6rGk59Z9MxDwQ6KOQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4865064f288" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which Machine Learning model to choose?]]></title>
            <link>https://medium.com/@kanchanakanta/which-machine-learning-model-to-choose-a4586129714a?source=rss-ee96734c02fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4586129714a</guid>
            <category><![CDATA[parametric-models]]></category>
            <category><![CDATA[machine-learning-models]]></category>
            <category><![CDATA[non-parametric-models]]></category>
            <dc:creator><![CDATA[Kanchanakanta]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 08:54:55 GMT</pubDate>
            <atom:updated>2024-10-03T08:54:55.194Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*E_FeTjqKH4WkLbueZJcknQ.png" /><figcaption>Types of Machine Learning models</figcaption></figure><p>While developing Machine Learning models the first thing we need to decide is which type of model to be selected. Assumptions can greatly simplify the learning process but can also limit what can be learned.</p><p>There are two types of models.</p><p>1. Parametric models and</p><p>2. Non-Parametric models.</p><p><strong>Parametric models</strong>:</p><p><strong>Definition:</strong> Parametric models are the Statistical Machine learning models characterized by set of specific assumptions about the underlying data distribution or the functional form of relationship between the input and output variables. Parametric models work with continuous data.</p><p>Parametric models summarize the data with these set of parameters irrespective of the data size. Once these parameters are calculated these models can be used for prediction. Parametric models fit the data into a mathematical equation which is a function of a set of parameters that define the underlying distribution of data.</p><p><strong>The algorithm involves two steps.</strong></p><p>1. Select a form for the mathematical function to relate the variables and</p><p>2. Learn the coefficients for the function from the training data</p><p>An easy-to-understand functional form for mapping function is a line and is used in linear regression.</p><p>The equation of the line is B0+ B1X1+B2X2 =0</p><p>Where B0, B1 and B2 are the coefficients of the line that control the intercept of the line and slope of the line. X1 and X2 are the two input variables.</p><p>Assuming the functional form greatly simplifies the learning process. Then we need to estimate the coefficients of the line equation and we have a predictive model for the problem.</p><p>In actual the problem may not be linear function like a line and our assumption may go wrong and model may give poor results.</p><p><strong>Example Parametric Machine Learning Algorithms:</strong></p><p>1. Linear regression</p><p>2. Linear Discriminant Analysis</p><p>3. Perceptron</p><p>4. Naïve Bayes</p><p>5. Simple Neural Networks</p><p><strong>Features of Parametric models:</strong></p><p>1. <strong>Fixed/Finite number of parameters:</strong> The number of parameters is fixed and finite and does not change with data size.</p><p>2. <strong>Assumptions:</strong> These models assume a specific mathematical form or a structure for the relationship between the dependent and independent variables. This assumption relies on distribution of data such as normality or linearity.</p><p>3. <strong>Efficiency:</strong> With fewer parameters to adjust, the parametric models can be very efficient in terms of computation. This makes parametric models more useful when the computational resources are less or limited.</p><p>4. <strong>Easy to Understand/interpretation:</strong> Because of their simplistic mathematical structure, parametric models are usually easier to understand.</p><p>5. <strong>Generalization:</strong> Assuming the chosen mathematical form correctly captures the relationship between the variables. Parametric models aim to generalize well from the training data to unseen data.</p><p>6. <strong>Susceptible to model misspecification:</strong> If the assumed model does not accurately represent the true relationship between variables parametric models can give inaccurate solutions.</p><p>7. <strong>Limited Flexibility:</strong> Due to their fixed structure, parametric models are less flexible in capturing complex nonlinear relationships compared to non-parametric models.</p><p><strong>Advantages of Parametric models:</strong></p><p>1. Simple to understand and interpret the result.</p><p>2. These are very fast in learning.</p><p>3. Can work with less data and less computational resources.</p><p><strong>Disadvantages of Parametric models:</strong></p><p>1. Because of the assumptions these models are constrained to a specific form.</p><p>2. These models are suitable only for solving simple problems.</p><p>3. Poor fit: These models are unlikely to match mapping functions.</p><p>4. These can be affected by outliers.</p><p><strong>Non-Parametric Machine Learning models:</strong></p><p><strong>Definition:</strong> Non-Parametric models are a type of Statistical Machine Learning models that do not make assumptions about a specific form of mapping functions for the underlying distribution of data. These models can handle ordinal and ranked data.</p><p>By not having any assumptions these are free to learn any functional form from the training data. Nonparametric models do not rely on any parameter settings and therefore often produce more accurate results.</p><p><strong>Non -Parametric models work well when</strong></p><p>1. There is a lot of data and no prior knowledge about that data and</p><p>2. When we do not want to worry too much about choosing right features</p><p>Some examples of popular non-parametric machine learning algorithms:</p><p>1. K nearest neighbours’ algorithm</p><p>2. Decision trees like CART and C4 .5 and</p><p>3. Support vector machines.</p><p><strong>Features of non-parametric models:</strong></p><p>1. <strong>Flexibility:</strong> Non-Parametric models do not assume a predefined form for the data distribution. This makes them suitable for wide range of data types and structures.</p><p><strong>Infinite dimensional</strong>: They are infinite dimensional because they can grow in complexity as more data becomes available.</p><p><strong>Advantages of non-parametric models:</strong></p><p>1. No assumptions on data distribution</p><p>2. Can work with larger data</p><p>3. Can fit large number of functional forms</p><p>4. Can result in higher performance for prediction</p><p>5. These models are less sensitive to outliers</p><p><strong>Disadvantages of non-parametric models:</strong></p><p>1. Requires huge data for training to estimate the mapping function.</p><p>2. Requires more time to train as there are many parameters to train.</p><p>3. There is a risk of overfitting the data and it is difficult to explain about specific predictions that are made.</p><p><strong>When to choose which model?</strong></p><p>There are several factors to be considered for deciding which model to select.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*YyttOu_zi1Nz1VpUDSdQNQ.png" /><figcaption>Factors deciding which ML model to select</figcaption></figure><ol><li><a href="https://vitalflux.com/difference-between-parametric-vs-non-parametric-models/">Parametric vs Non-Parametric Models: Differences, Examples (vitalflux.com)</a></li><li><a href="https://plat.ai/blog/parametric-modeling/#:~:text=Here%E2%80%99s%20where%20a%20parametric%20model%20steps%20in,%20focusing">Parametric Models in ML: Theory, Advantages, and Comparisons (plat.ai)</a></li><li><a href="https://machinelearningmastery.com/parametric-and-nonparametric-machine-learning-algorithms/">Parametric and Nonparametric Machine Learning Algorithms — MachineLearningMastery.com</a></li><li><a href="https://en.wikipedia.org/wiki/Nonparametric_statistics">Nonparametric statistics — Wikipedia</a></li><li><a href="https://www.kaggle.com/discussions/general/324532">Parametric and Non Parametric models | Kaggle</a></li><li><a href="https://mylearningsinaiml.wordpress.com/parametric-non-parameteric-algos/">Parametric &amp; Non-Parameteric Algos | Foundations of AI &amp; ML (wordpress.com)</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4586129714a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>