<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sümeyra Güleç Kayan on Medium]]></title>
        <description><![CDATA[Stories by Sümeyra Güleç Kayan on Medium]]></description>
        <link>https://medium.com/@sumeyraagulec?source=rss-2c33400a97a8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W_Bq86mPAHiz1P5TKeLeAQ.png</url>
            <title>Stories by Sümeyra Güleç Kayan on Medium</title>
            <link>https://medium.com/@sumeyraagulec?source=rss-2c33400a97a8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 23:24:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sumeyraagulec/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Discovering Insights from Book Review Data: A BERTopic-Based Topic Modeling Approach]]></title>
            <link>https://medium.com/@sumeyraagulec/discovering-insights-from-book-review-data-a-bertopic-based-topic-modeling-approach-a8cde98e7fe0?source=rss-2c33400a97a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8cde98e7fe0</guid>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[book-review]]></category>
            <category><![CDATA[bertopic]]></category>
            <category><![CDATA[text-analytics]]></category>
            <category><![CDATA[text-mining]]></category>
            <dc:creator><![CDATA[Sümeyra Güleç Kayan]]></dc:creator>
            <pubDate>Mon, 06 Jan 2025 14:26:38 GMT</pubDate>
            <atom:updated>2025-01-06T14:26:38.109Z</atom:updated>
            <content:encoded><![CDATA[<p>Text Analysis, Text Mining, Topic Modeling, BERTopic, Book Review Insights</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZtf-Mjrx-mZzi2GRcz3sA.png" /><figcaption>Generated by ChatGPT</figcaption></figure><p>In today’s digital age, online reviews offer a wealth of information that can be used to gain insights into customer opinions and preferences. Amazon, as one of the largest online marketplaces, hosts an extensive collection of book reviews, making it a valuable resource for data analysis. In this blog post, the Amazon book review dataset is going to be explored to uncover underlying patterns and themes using topic modeling algorithms. After the general structure of the reviews is examined, a closer look will be taken at the reviews to understand their specific categories.</p><p>This analysis will be conducted using a Python workflow in <a href="https://colab.google/">Google Colab</a>. To access and use the documents stored in Drive, it needs to be mounted.</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)</pre><p>After the mounting process, the dataset will be downloaded from the <a href="https://www.kaggle.com/datasets/cynthiarempel/amazon-us-customer-reviews-dataset/data?select=amazon_reviews_us_Books_v1_02.tsv">Kaggle link</a> and imported into Jupyter Notebook.</p><pre>import pandas as pd<br>import numpy as np<br>import nltk<br>from nltk.corpus import stopwords<br>from nltk.tokenize import word_tokenize<br>nltk.download(&#39;stopwords&#39;)<br>nltk.download(&#39;punkt&#39;)<br>from transformers import AutoTokenizer, AutoModel<br>from sentence_transformers import SentenceTransformer<br>from bertopic import BERTopic<br><br><br>df = pd.read_csv(&#39;/content/drive/MyDrive/amazon_reviews_us_Books_v1_02.tsv&#39;,on_bad_lines=&#39;skip&#39;,sep=&#39;\t&#39;)<br>print(df.shape)<br>df.head(3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GptJga40ifs_D8EAKFWa1w.png" /></figure><p>The dataset consists of 3.1 Million rows. Let’s start our analysis by examining the contents of the dataset and its columns.</p><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*UYYU1ep3nGmHw1mVk-Cg-w.png" /></figure><p>First, let’s take a look at the data types and adjust them accordingly. review_date field appears as an object. This field needs to be converted into an appropriate date format.</p><pre>df[&#39;review_date&#39;] = pd.to_datetime(df[&#39;review_date&#39;],format = &#39;%Y-%m-%d&#39;)<br>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*Fvxn43G6NaOK-uKspDf9DA.png" /></figure><p>Let’s take a look at the contents of the first column, which is the marketplace field.</p><pre>df.marketplace.value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*AYfuXKPQWab0uRxvhw_orA.png" /></figure><p>The entire dataset consists of US customers. Since no value is added to our analysis by this information, this column will be dropped.</p><pre>df.drop(&#39;marketplace&#39;,axis=1,inplace=True)</pre><p>The product_category field will be examined. Although it is known that this dataset contains only book data, this field should be analyzed to be sure.</p><pre>df.product_category.value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*Ro3hYA0m9tDTIWYWhsP39Q.png" /></figure><p>There is a data quality issue, as this field has been incorrectly entered in four rows. After these four rows are removed, it is observed that the entire dataset contains only book reviews. With this information confirmed, this column can also be dropped from the dataset.</p><pre>df=df[df.product_category==&#39;Books&#39;]<br>df.drop(&#39;product_category&#39;,axis=1,inplace=True)</pre><p>The review_date field will be looked to determine the time range covered by the dataset.</p><pre>print(min(df.review_date))<br>print(max(df.review_date))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*r6HsMLcWjvnAtTMURYu_jg.png" /></figure><p>To summarize, a dataset of book reviews from US customers, covering a 10-year period between 1995 and 2005, is available.</p><p>Next, the dataset will be investigated to identify which fields contain unique values. The first field that comes to mind is review_id. The count of unique values in this field will be compared with the number of rows in the dataset. If these two values are found to be equal, it can be concluded that the dataset is unique based on this field.</p><pre>print(df.review_id.nunique())<br>print(df.shape)<br>df.review_id.nunique()==df.shape[0]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*KOinEPaB2InGBAIbHKA_qQ.png" /></figure><p>It can be concluded that the dataset is distinct based on the review_id.</p><p>Another aspect that can be examined is the number of unique customers who have reviewed different unique products.</p><pre>print(df.customer_id.nunique())<br>print(df.product_id.nunique())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/64/1*9mQrd1RTX7AnX77uM0BNng.png" /></figure><p>1.5M customers have reviewed 779K books.</p><p>Let’s calculate how many reviews each book has received and the average rating given to each book, then sort and examine them by the number of reviews.</p><pre>pivot_table = pd.pivot_table(<br>    df,<br>    values=&#39;star_rating&#39;,                <br>    index=&#39;product_id&#39;,               <br>    aggfunc=[&#39;count&#39;, &#39;mean&#39;]     <br>)<br><br><br>product_table = pd.DataFrame(pivot_table).reset_index()<br>product_table.columns = [&#39;product_id&#39;,&#39;review_count&#39;,&#39;rating_mean&#39;]<br>distinct_product_name =  df[[&#39;product_id&#39;,&#39;product_title&#39;]].drop_duplicates(keep=&#39;first&#39;)<br>product_table= pd.merge(product_table,distinct_product_name,on=&#39;product_id&#39;,how=&#39;left&#39;)<br>product_table = product_table[[&#39;product_id&#39;,&#39;product_title&#39;,&#39;review_count&#39;,&#39;rating_mean&#39;]].sort_values(by=&#39;review_count&#39;,ascending=False).reset_index(drop=True)<br><br>product_table.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*ndWF1yoqYqR0qFwbHI5nvA.png" /></figure><p>The distribution of the review_count and rating_mean will then be looked.</p><pre>product_review_count_rating_dist =product_table[[&#39;review_count&#39;,&#39;rating_mean&#39;]].describe(percentiles=[0.05,0.10,0.15,0.20,0.25,0.30,0.35,0.40,0.45,0.50,0.55,0.60,0.65,0.70,0.75,0.80,0.85,0.90,0.95])<br>product_review_count_rating_dist</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*dwKiP6hkcicbUpGyHtgHZg.png" /></figure><p>80% of the books have received 4 or fewer reviews. Books with more than 13 reviews make up only 5% of all the books.</p><p>Approximately half of the books have an average rating of 5. It’s possible that these books have only one or two review. If we look at the average ratings of books with more than 30 review, this picture is likely to change.</p><pre>product_table[product_table.review_count&gt;30][[&#39;rating_mean&#39;]].describe(percentiles=[0.05,0.10,0.15,0.20,0.25,0.30,0.35,0.40,0.45,0.50,0.55,0.60,0.65,0.70,0.75,0.80,0.85,0.90,0.95])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/135/1*yb5ufgIH8y08LV92fYmnJg.png" /></figure><p>In statistics, the number 30 is commonly considered a guideline for minimum sample size, as it marks the point where the central limit theorem starts to take effect. You can access more detailed information through this <a href="https://www.linkedin.com/pulse/magic-number-30-why-sample-size-often-considered-sufficient/">link</a>. Now, the analysis will continue by selecting books with 30 or more reviews, and a new dataframe will be created.</p><pre>selected_books = product_table[product_table.review_count&gt;=30].product_id<br>df_filtered = pd.merge(df[[&#39;product_id&#39;,&#39;review_body&#39;]], selected_books, on=&#39;product_id&#39;, how=&#39;inner&#39;)<br>print(df_filtered.shape)<br>df_filtered.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*ue6BHDidZbGS-b0ooLQLVg.png" /></figure><p>In the realm of text data analysis, one of the most effective methods for extracting meaningful patterns from large volumes of unstructured text, such as book reviews, is <em>topic modeling</em>. Topic modeling is an unsupervised machine learning approach used to identify hidden themes or “topics” within a corpus of documents. Through the discovery of these topics, the underlying structure of text data can be understood and interpreted, offering insights into common themes or opinions without the need for manually reviewing each entry.</p><p><strong>What is Topic Modeling?</strong></p><p>Topic modeling automates the discovery of topics within a set of documents by grouping words that frequently appear together, hypothesizing that such groups represent coherent themes. Essentially, each topic is a cluster of words that often occur together, representing a general concept within the data. For example, in a collection of book reviews, topics might naturally emerge around themes like “character development,” “plot,” “writing style,” and “genre.”</p><p><strong>Types of Topic Modeling Techniques</strong></p><p>Several traditional approaches are commonly used in topic modeling like Latent Dirichlet Allocation (LDA),Non-Negative Matrix Factorization (NMF),Latent Semantic Analysis (LSA). These traditional topic modeling methods have limitations in handling nuanced language and contextual relationships, especially with large, diverse corpora.</p><p>Recently, modern approaches like <a href="https://medium.com/towards-data-science/interactive-topic-modeling-with-bertopic-1ea55e7d73d8"><strong><em>BERTopic</em></strong> </a>have gained popularity. BERTopic combines BERT embeddings (contextual word representations) with clustering techniques, allowing for more flexible and contextually accurate topic generation. Similarly, Large Language Models (LLMs) like <strong><em>GPT, Llama and Gemma</em></strong> models and others can now be used for topic modeling by interpreting and generating topics or creating vectors from text data.</p><p>In this article, the focus will be solely on the BERTopic Algorithm Method.</p><p><strong>What is BERTopic?</strong></p><p>BERTopic is a powerful tool for topic modeling that leverages modern transformer embeddings, dimensionality reduction, and clustering to identify topics in a way that’s both contextually accurate and highly customizable. It’s particularly effective for applications in customer feedback analysis, social media monitoring, document categorization, and any other use case where understanding underlying topics in large text datasets is valuable. You can find more detailed information about BERTopic through this <a href="https://towardsdatascience.com/interactive-topic-modeling-with-bertopic-1ea55e7d73d8">link</a>.</p><p>Consequently, BERTopic is highly modular and can uphold the quality of topic generation across different sub-models. Put simply, BERTopic enables you to create a customized topic model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTS68h5S7tNZIqW6YxezOA.png" /><figcaption>This image is sourced from the <a href="https://maartengr.github.io/BERTopic/algorithm/algorithm.html#visual-overview">link</a></figcaption></figure><p><strong>Topic Modelling by Using BERTopic</strong></p><p>First, a sample of 1,000 entries will be taken to ensure efficient code execution. Stopwords will be removed from the text data in this sample, and the standard BERTopic algorithm will be applied using the <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">all-MiniLM-L6-v2 model</a> as the transformer.</p><p>The <a href="https://dataloop.ai/library/model/sentence-transformers_all-minilm-l6-v2/">all-MiniLM-L6-v2</a> model is highly effective for natural language processing applications. As part of the sentence-transformers family, it converts sentences and paragraphs into 384-dimensional dense vectors, making it suitable for tasks such as clustering and semantic search.</p><pre>df_filtered_sample = df_filtered.sample(n=1000, random_state=42)<br># stop words english<br>stop_words = set(stopwords.words(&#39;english&#39;))<br># Stop words filter<br>def remove_stop_words(text):<br>    words = word_tokenize(text)  # divide text into words<br>    filtered_words = [word for word in words if word.lower() not in stop_words]  # Stop words filter<br>    return &#39; &#39;.join(filtered_words)  # join filtered words<br><br># filtered text data column<br>df_filtered_sample[&#39;review_body_clear&#39;] = df_filtered_sample[&#39;review_body&#39;].apply(remove_stop_words)<br><br># Sentence Transformers<br>sentence_model = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;) <br><br># review data<br>amazon_reviews = list(df_filtered_sample.review_body_clear)<br><br># We are converting the reviews into embeddings<br>embeddings = sentence_model.encode(amazon_reviews, show_progress_bar=True)<br><br>#Topic Modelling by using embeddings<br>topic_model = BERTopic()<br>topics, probabilities = topic_model.fit_transform(amazon_reviews, embeddings)<br><br><br>#Topics<br>topic_model.get_topic_info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*3Q9H1_JgLyq9embuzoMeKw.png" /></figure><p>In this output, the topics seem to be categorized mostly by the genres of the books. For example, Topic 13 seems to have separated out science fiction books. By adjusting the hyperparameters of the BERTopic algorithm, the number of topics can be increased or decreased to explore the data in more detail.</p><p>By further analyzing the text data, detailing the preprocessing process, and modifying the hyperparameters of the BERTopic algorithm, this model can be significantly improved and enhanced. In this blog post, the aim is to present a small example analysis where the BERTopic algorithm can provide a solution.</p><p><strong><em>REFERENCES</em></strong></p><ul><li><a href="https://towardsdatascience.com/interactive-topic-modeling-with-bertopic-1ea55e7d73d8">Interactive Topic Modeling with BERTopic</a></li><li><a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">sentence-transformers/all-MiniLM-L6-v2 · Hugging Face</a></li><li><a href="https://www.kaggle.com/datasets/cynthiarempel/amazon-us-customer-reviews-dataset/data?select=amazon_reviews_us_Books_v1_02.tsv">Amazon US Customer Reviews Dataset</a></li><li><a href="https://dergipark.org.tr/tr/pub/izsbd/issue/72911/1167719">BERTopic Konu Modelleme Tekniği Kullanılarak Müşteri Şikayetlerinin Sınıflandırılması</a></li><li><a href="https://maartengr.github.io/BERTopic/algorithm/algorithm.html#visual-overview">The Algorithm - BERTopic</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8cde98e7fe0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The importance of Cutoff and Lift Values for Performance Evaluation in Binary Classification Models]]></title>
            <link>https://medium.com/@sumeyraagulec/the-importance-of-cutoff-and-lift-values-for-performance-evaluation-in-binary-classification-models-3554ef27082f?source=rss-2c33400a97a8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3554ef27082f</guid>
            <category><![CDATA[cut-off]]></category>
            <category><![CDATA[model-performance]]></category>
            <category><![CDATA[binary-classification]]></category>
            <category><![CDATA[performance-metrics]]></category>
            <category><![CDATA[lift]]></category>
            <dc:creator><![CDATA[Sümeyra Güleç Kayan]]></dc:creator>
            <pubDate>Thu, 16 Nov 2023 12:35:02 GMT</pubDate>
            <atom:updated>2023-11-16T12:35:02.974Z</atom:updated>
            <content:encoded><![CDATA[<p>Performance Metrics, Lift, Cutoff Value, and More</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*V7HXLnU70rf1UJRnZzej8Q.jpeg" /></figure><p>A classification model is a supervised learning method that enables us to estimate discrete values, such as whether a customer will churn, purchase a product, be considered risky for a bank, or identify objects in an image (e.g., orange/apple/pear). If the number of classes is more than two, it is considered multiclass classification; if there are only two classes, it is binary classification. This article will focus on analyzing the performance of binary classification models.</p><p>After the modeling process, the test data is predicted, yielding score values between 0 and 1 for each sample. A higher score indicates a greater probability of the target coming true.</p><p>Let’s create a synthetic churn target data in python to make it more concrete and understandable.</p><p>Firstly, we import some libraries.</p><pre>import pandas as pd<br>import numpy as np<br>import random<br>import matplotlib.pyplot as plt<br>from sklearn.metrics import roc_auc_score, roc_curve</pre><p>Then we create the data that we need. We need unique ID’s, target values and score values that model predict.</p><p>CUSTOMERID: unique id for a customer</p><p>CHURN_TARGET: a customer is churn(1) or not (0)</p><p>MODEL_SCORE: churn prediction of the model that we trained for a customer</p><p>Target rate is assumed as 0.1 and score values are assumed as normally distributed.</p><p>Let’s create the dataset.</p><pre>#Needed variables are defined<br>df = pd.DataFrame(columns=[&#39;CUSTOMERID&#39;,&#39;CHURN_TARGET&#39;,&#39;MODEL_SCORE&#39;])<br><br>#Random values are assigned for this variable<br>randomlist_customerid = []<br><br>#1000 Random id numbers are defined for customers between 1000 and 5000<br>for i in range(10000):<br>    n_customerid = random.randint(1000,5000)<br>    if n_customerid not in randomlist_customerid:<br>    randomlist_customerid.append(n_customerid)<br>    if len(randomlist_customerid)==1000:<br>        break<br><br>df[&#39;CUSTOMERID&#39;] = randomlist_customerid<br><br>#1000 Random churn target values are assigned for customers either 1 or 0 <br>#Target rate is assumed as 0.1<br>randomlist_churn_target_1 = []<br>for i in range(100):    <br>    n_churn_target = 1<br>    randomlist_churn_target_1.append(n_churn_target)<br><br>randomlist_churn_target_0 = []<br>for i in range(900):    <br>    n_churn_target = 0<br>    randomlist_churn_target_0.append(n_churn_target)<br><br>randomlist_churn_target = randomlist_churn_target_1 + randomlist_churn_target_0<br>random.shuffle(randomlist_churn_target)<br>df[&#39;CHURN_TARGET&#39;] = randomlist_churn_target<br><br>#1000 random score values are determined with the assumption that scores have normal distribution<br>randomlist_model_score_1 = list(np.random.normal(loc=0.7, scale =0.10, size=100))<br>randomlist_model_score_0 = list(np.random.normal(loc=0.4, scale =0.10, size=900))<br>random.shuffle(randomlist_model_score_1)<br>random.shuffle(randomlist_model_score_0)<br><br>print(&#39;Score range having churn target values as 1 : &#39;, min(randomlist_model_score_1),max(randomlist_model_score_1))<br>print(&#39;Score range having churn target values as 0  : &#39;, min(randomlist_model_score_0),max(randomlist_model_score_0))<br><br><br>#These score values are written to the dataframe<br>a=0 <br>b=0<br><br>for i in range(len(df)):<br>    if df[&#39;CHURN_TARGET&#39;].iloc[i]==1:<br>        df[&#39;MODEL_SCORE&#39;].iloc[i] = randomlist_model_score_1[a]<br>        a += 1<br>    else:<br>        df[&#39;MODEL_SCORE&#39;].iloc[i] = randomlist_model_score_0[b]<br>        b += 1<br><br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*2th-n694e3yJQI6hXlFE2w.png" /><figcaption>Output</figcaption></figure><p>We have churn target and score values for 1000 customers.</p><p>We need to compare CHURN_TARGET and MODEL_SCORE values to interpret model performance, because CHURN_TARGET is real churn target value and MODEL_SCORE is model’s prediction of being churn. CHURN_TARGET is categorical variable and MODEL_SCORE is continuous variable. To compare these two variables, we need to make them similar types. To do so, we must identify a cutoff point for the MODEL_SCORE variable. If the score value is higher than the cutoff point, it is defined as 1; if it is lower or equal to the cutoff point, it is defined as 0. With this prediction flag, 1 indicates churn, and 0 indicates non-churn.</p><p>In Python, there is a function called predict (model.predict(data)), commonly utilized for predicting labels rather than probabilities across various models. However, it’s essential to note that this function typically uses a default cutoff point of 0.5. Yet, determining the optimal cutoff point is project-specific and using this function might not always be the most suitable approach.</p><p>The crucial decision lies in determining the cutoff point. In the context of churn data, this determination hinges on two vital conditions. Firstly, it aligns with the business unit’s decision on the number of customers the unit can feasibly address as churn. Secondly, it involves analyzing performance metric values across various cutoff points. Once the performance metrics are scrutinized, reaching an agreement with the business unit becomes pivotal.</p><p>For now, we assume that our cutoff point is 0.6 and we create a new variable for target prediction which is MODEL_FLG. If MODEL_SCORE value is greater than 0.6, MODEL_FLG value is 1 else it is 0.</p><pre>df[&#39;MODEL_FLG&#39;] = np.where(df[&#39;MODEL_SCORE&#39;] &gt; 0.6, 1, 0)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*dmnlA34T95nvAdzPcFpjEw.png" /><figcaption>Output</figcaption></figure><p><strong>Confusion Matrix</strong></p><p>In calculating performance, it’s essential to understand the concept of a confusion matrix. This matrix provides a clear representation of how we distinguish various cases when comparing actual target values with predicted target values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*MlWux1e4jHC4a7eVVW583A.jpeg" /></figure><p>TP (True Positive) : Predicted target is positive and actual target is positive. For the churn case, MODEL_FLG=1 and CHURN_TARGET=1</p><p>TN (True Negative) : Predicted target is negative and actual target is negative. For the churn case, MODEL_FLG=0 and CHURN_TARGET=0</p><p>FP (False Positive) : Predicted target is positive and actual target is negative. For the churn case, MODEL_FLG=1 and CHURN_TARGET=0</p><p>FN (False Negative) : Predicted target is negative and actual target is positive. For the churn case, MODEL_FLG=0 and CHURN_TARGET=1</p><p>Below is the result for our churn model case:</p><pre>print(&#39;True positives: &#39;, len(df[(df.CHURN_TARGET==1) &amp; (df.MODEL_FLG==1) ]))<br>print(&#39;True Negatives: &#39;, len(df[(df.CHURN_TARGET==0) &amp; (df.MODEL_FLG==0) ]))<br>print(&#39;False Positives: &#39;, len(df[(df.CHURN_TARGET==0) &amp; (df.MODEL_FLG==1) ]))<br>print(&#39;False Negatives: &#39;, len(df[(df.CHURN_TARGET==1) &amp; (df.MODEL_FLG==0) ]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*FyGbM3JtADhz46f855juBg.png" /><figcaption>Output</figcaption></figure><p><strong>Performance Metrics for Classification Models</strong></p><p>There are different performance metrics in classification models, such as Accuracy, ROC, Lift, Precision, Recall, and F Score. All of these metrics are calculated using confusion matrix values.</p><p><strong>Accuracy</strong></p><p>Accuracy shows how accurately we predict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*GiYPqyFKSVRP7Zb2Lv4iyw.jpeg" /></figure><p>For churn case, Accuracy = (82+875) / (82+875+25+18) = 0.96</p><p>Accuracy can be misleading, especially when the target rate is very low. For instance, predicting all samples as 0 would yield a high accuracy value, even though it may not be a meaningful or informative outcome.</p><p><strong>Area Under the Curve (AUC) and ROC-AUC Score</strong></p><p>This curve plots the true positive rate versus the false positive rate at different classification thresholds, and the area under the curve is between 0 and 1. When predictions are all wrong, this value will be 0, and when predictions are all true, this value will be 1.</p><p>AUC measures how well predictions rank and represents the probability that a randomly selected churn sample has a higher score than a randomly selected non-churn sample. It provides a comprehensive measure of performance.</p><pre>def plot_roc_curve(true_y, y_prob):<br>    fpr, tpr, thresholds = roc_curve(true_y, y_prob)<br>    plt.plot(fpr, tpr)<br>    plt.xlabel(&#39;False Positive Rate&#39;)<br>    plt.ylabel(&#39;True Positive Rate&#39;)    <br><br>print(f&#39;AUC score: {roc_auc_score(df.CHURN_TARGET, df.MODEL_SCORE)}&#39;)<br>plot_roc_curve(df.CHURN_TARGET, df.MODEL_SCORE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*X31Ns3ieAp2iWokSMrcklw.png" /><figcaption>Output</figcaption></figure><p>With an AUC value of 0.98, we can interpret it as the probability that a randomly selected churn sample has a higher score than a randomly selected non-churn sample.</p><p><strong>Lift</strong></p><p>Lift is a metric that indicates how many times the model has increased the target rate. It is calculated by dividing the target ratio for samples with a score value higher than the cutoff point by the target ratio of the entire model dataset.</p><p>TP+FP = 107, so model flagged 107 customer as churn and 82 of them are real churn.</p><p>Their real churn rate is TP/(TP+FP) = 0.77</p><p>If we choose random 107 customer in data, we expect their churn rate as 0.1 which means churn rate of the entire data. With this model, the rate of increase for target rate is 0.77/0.1 so 7.7</p><p>Lift = churn rate provided by the model prediction/all data real churn rate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*2HgFx8bwySZz031pZ8EQXA.jpeg" /></figure><p>Lift = 0.77 / 0.1 = 7.7</p><p>Lift is one of the most commonly used metrics for interpreting the performance of binary classification models. It is considered more reliable than other metrics and is easily understandable and interpretable for business units.</p><p><strong>Precision</strong></p><p>Precision is the ratio of how many of our positive predictions are actually positive. It also serves as the numerator in the lift ratio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*0AepWThsoJbqWShl9UtBog.jpeg" /></figure><p>For churn data, Precision is 82/(82+25) = 0.77</p><p><strong>Recall</strong></p><p>Recall is the ratio of correctly predicting actual positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*Perq1u-3Ksf3dkJTMuzbGg.jpeg" /></figure><p>For churn data, Recall is 82/(82+18) = 0.82</p><p><strong>F1 Score</strong></p><p>F1 Score combines Precision and Recall metrics and it is the harmonic mean of precision and recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*Rjty8oHQpBqg2anrUZQbUg.jpeg" /></figure><p>For churn data, F1 Score is 2*((0.77*0.82)/(0.77+0.82)) = 0.79</p><p><strong>Performance Metrics for Different Cutoff Values</strong></p><p>As evident, the values of all these metrics rely on confusion matrix values, and these values are determined based on the cutoff value. In this example, we chose 0.6 as the cutoff value. Generally, determining the optimal cutoff value requires an examination of various cutoff values rather than a fixed prescription.</p><p>Analyzing metrics for different cutoff values is essential. This can be achieved by examining every 1 percent group of the data and calculating these metrics for each group. By assessing these metrics in conjunction with the business unit’s decision about the required data, the optimal cutoff point can be determined.</p><p>We can use the following code for this analysis. This code gives outputs of different performance metrics for different cutoff values.</p><p>The data is initially sorted by MODEL_SCORE in descending order. Subsequently, it is divided into 100 buckets based on percentiles. The minimum MODEL_SCORE value within each bucket is then considered as the cutoff point, and the mentioned metrics are calculated using this logic.</p><pre>sorted_score = df.sort_values(by=[&#39;MODEL_SCORE&#39;],ascending=True)<br>data_size = sorted_score[&#39;CUSTOMERID&#39;].count()<br>bucket_size = int(data_size/100)<br>target_ratio = (df[df.CHURN_TARGET==1].CUSTOMERID.count()/df.CUSTOMERID.count())<br><br>all_buckets = []<br>for i in range(0,data_size+1,bucket_size):<br>    all_buckets.append(i)<br><br>    <br>print(&#39;all buckets: &#39;,all_buckets)<br><br><br>df_perf_results = pd.DataFrame(columns=[&#39;Bucket&#39;,&#39;Cutoff_Value&#39;,&#39;TP&#39;,&#39;TN&#39;,&#39;FP&#39;,&#39;FN&#39;,&#39;Predicted_1_cum&#39;,<br>                                        &#39;Lift&#39;,&#39;Recall&#39;,&#39;Precision&#39;,&#39;F_Score&#39;,&#39;Accuracy&#39;])<br><br><br>for i in range(len(all_buckets)):<br>    if all_buckets[i] !=max(all_buckets):<br>        sorted_score_bucket = sorted_score.loc[sorted_score.CUSTOMERID.isin(list(sorted_score[&#39;CUSTOMERID&#39;][all_buckets[i]:all_buckets[i+1]]))]<br>        cust_upper_cutoff = list(sorted_score[&#39;CUSTOMERID&#39;])[all_buckets[i]:]<br>        cust_below_cutoff= list(sorted_score[&#39;CUSTOMERID&#39;])[:all_buckets[i]] <br>        <br>        <br>        df_perf_results = df_perf_results.append({ &#39;Bucket&#39; : i,  <br>                                        &#39;Cutoff_Value&#39;:sorted_score.loc[sorted_score.CUSTOMERID.isin(cust_upper_cutoff)].MODEL_SCORE.min(),<br>                                        &#39;TP&#39;:len(sorted_score.loc[(sorted_score.CUSTOMERID.isin(cust_upper_cutoff) ) &amp; (sorted_score.CHURN_TARGET==1)]),<br>                                        &#39;TN&#39;:len(sorted_score.loc[(sorted_score.CUSTOMERID.isin(cust_below_cutoff) ) &amp; (sorted_score.CHURN_TARGET==0)]),<br>                                        &#39;FP&#39;:len(sorted_score.loc[(sorted_score.CUSTOMERID.isin(cust_upper_cutoff) ) &amp; (sorted_score.CHURN_TARGET==0)]),<br>                                        &#39;FN&#39;:len(sorted_score.loc[(sorted_score.CUSTOMERID.isin(cust_below_cutoff) ) &amp; (sorted_score.CHURN_TARGET==1)]), <br>                                        &#39;Predicted_1_cum&#39;:sorted_score.loc[sorted_score.CUSTOMERID.isin(cust_upper_cutoff)].CUSTOMERID.count()<br>                                         }, ignore_index=True)<br><br>df_perf_results[&#39;Recall&#39;] = df_perf_results[&#39;TP&#39;] / (df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;FN&#39;])<br>df_perf_results[&#39;Precision&#39;] =df_perf_results[&#39;TP&#39;] / (df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;FP&#39;])<br>df_perf_results[&#39;F_Score&#39;] = (2*df_perf_results[&#39;Recall&#39;]*df_perf_results[&#39;Precision&#39;])/(df_perf_results[&#39;Recall&#39;]+df_perf_results[&#39;Precision&#39;])<br>df_perf_results[&#39;Accuracy&#39;] = (df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;TN&#39;])/(df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;FP&#39;]+df_perf_results[&#39;TN&#39;] + df_perf_results[&#39;FN&#39;])<br>df_perf_results[&#39;Lift&#39;] = (df_perf_results[&#39;TP&#39;]/(df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;FP&#39;]))/((df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;FN&#39;])/(df_perf_results[&#39;TP&#39;]+df_perf_results[&#39;FP&#39;]+df_perf_results[&#39;TN&#39;]+df_perf_results[&#39;FN&#39;]))<br><br>df_perf_results<br><br>df_perf_results.tail(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*oTW1hnidfBgLw9kN8rFUiA.png" /><figcaption>Output</figcaption></figure><p>In interpreting this output, if we set the cutoff value as 0.707 for the 95th bucket, we obtain a lift of 9.8 and an F score value of 0.65. A lift value of 9.8 indicates that the model is 9.8 times more effective than random selection in identifying positive instances.</p><p>We can plot a graph to see how cutoff point and lift values vary relative to each other.</p><pre>plt.plot(df_perf_results[&#39;Cutoff_Value&#39;], df_perf_results[&#39;Lift&#39;])<br>plt.xlabel(&#39;Cutoff_Value&#39;)<br>plt.ylabel(&#39;Lift&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*ctxMRYZL-kkaTJvdTygCWQ.png" /><figcaption>Lift Curve</figcaption></figure><p>By examining the graph and the performance results table, we can interpret the model’s performance. After scrutinizing these metrics, it is crucial to assess how well the results align with the business unit’s needs. For instance, if the business unit has a budget for only 70 customers and desires those with the highest churn score, we can refer to the ‘df_perf_results’ table. In this case, selecting the ‘Predicted_1_cum’ column as 70 corresponds to the 93rd bucket row. They specifically require the top 7 percent of customers with the highest churn scores. By analyzing the performance values in this row, we can easily understand and interpret the effectiveness of the data we will send.</p><p><strong>References</strong></p><p><a href="https://lawtomated.com/accuracy-precision-recall-and-f1-scores-for-lawyers/">https://lawtomated.com/accuracy-precision-recall-and-f1-scores-for-lawyers/</a></p><p><a href="https://towardsdatascience.com/understanding-confusion-matrix-a9ad42dcfd62">https://towardsdatascience.com/understanding-confusion-matrix-a9ad42dcfd62</a></p><p><a href="https://www.w3schools.com/python/python_ml_auc_roc.asp">https://www.w3schools.com/python/python_ml_auc_roc.asp</a></p><p><a href="https://thedatascientist.com/an-in-depth-examination-of-lift-in-machine-learning/#:~:text=Lift%20Calculation,actual%20number%20of%20predicted%20positives">https://thedatascientist.com/an-in-depth-examination-of-lift-in-machine-learning/#:~:text=Lift%20Calculation,actual%20number%20of%20predicted%20positives</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3554ef27082f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>