Getting started Python with Rule Based Classification Problem

Potential Customer Expends on Mobile Games

Source: https://www.moneysavingexpert.com/

Hi everyone, at the end of this article you will discover the awareness of Python data structures and the ease of operations on tables as needed with the project of Calculating Lead-Based Expends with Rule-Based Classification

Data Science, which has become very popular in recent years. It is a matter of curiosity by both companies and individuals. There are many people who want to make a place for themselves in this field where a new job opportunity arises every day. In this learning process, the biggest obstacle to becoming a Data Scientist, which involves multiple disciplines such as mathematics and statistics, may seem to be coding.

Machine learning and deep learning models that come directly to mind when saying Data Science but may not be necessary in many cases. The real world has a great lack of making sense of the data before these. Your first priority is to stay motivated by playing and exploring with data.

I will proceed by thinking that you have touched Python or any programming language at some point in your lives. In this article, which does not need technical detailed information, you will observe how Python is used within a very simple project.

Now, I will share with you the functions and some methods that you will take the first step towards as a data scientist. We will do all this within a project.

Before starting, we will analyze the case of Calculating Lead-Based Returns with Rule-Based Classification.

To summarize our project, one of game company would like to learn how much money its new customers will potentially spend by observing the purchasing movements within the application. The company wants to segment its customers and plan the necessary notifications, additional scores or mail marketing. They assign a Data Scientist for all this and assign this task to him. We will consider this case under basic conditions which should be examined and analyzed in detail.

You have a dataset that includes the Country, Source, Age, Sex information of the customers and their game purchases.

You can get persona.csv data from this repo.

We can receive our data, which we will analyze on Python, in formats such as .csv or .xlsx.

Here is our variables:

1. Price: Payments made by costumers
2. Source: The operating system used, including IOS and Android
3. Sex: Gender of users, Female and Male
4. Country : Information from which country the costumers are from
5. Age: Age of costumers

Before moving on to coding, it is useful to get an idea about the Numpy and Pandas libraries. Since this article will contain a simple project-oriented Python explanation, I do not need to go into too much detail. It is only useful to know that the term library is “ a collection of files (called modules) that contains functions for use by other programs”.

There will be many libraries that we will need as data scientists. Among them, we will only use Pandas for now.

pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool

Getting started, we define our previously downloaded persona.csv dataset as a DataFrame with Pandas’ read_csv function.

# Using import to load a library module into a program’s memory.
import pandas as pd
# Reading persona.csv as pandas DataFrame
df = pd.read_csv(r"/data/persona.csv")

We want to observe and summarize our dataset with a set of pandas functions. It’s important to really take your time here and understand what these numbers are saying.

# Analyze descriptive statistics
df.head() # Returns the first 5 rows for quickly checking data
df.shape  # Returns a dimensionality of the DataFrame
df.describe().T # Generates descriptive statistic
df.isnull().values.any() # Returns any value is missing in DataFrame
df.isnull().sum() #  Returns how many missing values exist in the DataFrame

Returns the first 5 rows for quickly checking data

Generates descriptive statistic

Now we look at the unique value numbers and frequencies for a number of categorical variables. We make such operations and observations in order to get to know our data set better.

# Number of unique <SOURCE>
df["SOURCE"].nunique() # Count number of distinct SOURCE elements
df["SOURCE"].value_counts()# Returns counts of SOURCE rows 
df["COUNTRY"].value_counts() # Returns counts of COUNTRY rows

Next, we will examine the values of multiple variables together and observe their breakdown. In this code block we will use groupby and aggregation functions. This is perfectly normal if you are not familiar with the functions I have mentioned. It will not be possible for me to describe it here except to describe it with a few sentences. Through these documents, you can learn the Pandas function you want with detailed examples.

A groupby operation involves some combination of splitting the object, applying a function, and combining the results. This can be used to group large amounts of data and compute operations on these groups.

Get Nafiz ALTAY’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Aggregate(agg) using one or more operations over the specified axis. Here is the operations : mean, median, prod, sum, std, var

# Country breakdown of income averages

df.groupby("COUNTRY")["PRICE"].agg({"mean"})

# Country and Source breakdown of income averages

df.groupby(["COUNTRY", 'SOURCE'])["PRICE"].mean()

Country breakdown of income averages

Country and Source breakdown of income averages

Data analysis can be performed in many different ways with the method we have given above. Now, by grouping many variables in our data, we take the average gains within the combinations into our table in a DataFrame type named agg_df. We sort from largest to smallest by adding the sort_values method to the end of our code.

# Average income on the basis of variables,

agg_df = df.groupby(["COUNTRY", 'SOURCE', "SEX", "AGE"])["PRICE"].mean().sort_values(ascending=False)
agg_df.head()

After the above processes, all our columns that we received groupby gained index feature in DataFrame. Index is like an address, that’s how any data point across the dataframe can be accessed. Rows and columns both have indexes, rows indices are called as index and for columns its general column names. First, we want to get the index property of these columns and qualify it as a variable.

# Convert the index names to variable names

agg_df = agg_df.reset_index() # If False, return a copy. Otherwise, do operation inplace and return None.
agg_df.head() # Lets look at new index numbers

Converted the index names to variable names

We want to divide the customers into certain groups by converting the numerical variable of age into a categorical variable. This process is critical for segmenting our customers.

We defined the ranges of AGE values as string values in the list with the name my_labels.

# Convert AGE variable to categorical variable and adding it to agg_df

my_labels = ['0_18', '19_23', '24_30', '31_40', '41_70']
agg_df["AGE_CUT"] = pd.cut(x=agg_df["AGE"], bins=[0, 18, 23, 30, 40, 70], labels=my_labels)
agg_df.tail(10) # Just checking data

Convert AGE variable to categorical variable and adding it to agg_df

Use cut when you need to segment and sort data values into bins. This function is also useful for going from a continuous variable to a categorical variable. Cut could convert ages to groups of age ranges.

Next, before we segment our customers, we put them in a persona. Here, I recommend you to research the f-string structure and gain the habit of using it. In the code below, there is a list comprehension structure that saves our lives most of the time. List comprehension offers a shorter syntax when you want to create a new list based on the values of an existing list. If you don’t prefer to use list comprehension, you will have to write a for statement.

# Identify new level-based customers (Personas)
agg_df["customers_level_based"] = [f"{i[0]}_{i[1]}_{i[2]}_{i[-1]}" for i in agg_df.values]

Another concept that we need to understand well is the loc method. You can acccess a group of rows and columns by label(s) or a boolean array by loc. It also has a brother, iloc that advances as integer based. loc proceeds as label based. I think you can understand very well from the source here.

The code below should not be confusing to you. In this method, which shows a very simple usage, we use loc function to select all the rows (:) and the columns we want ([“customers_level_based”, “PRICE”]) from the DataFrame named agg_df. As we have seen use before by grouping it with the groupby and aggregation functions that we separate from the largest to the smallest according to the average returns.

agg_df = agg_df.loc[:, ["customers_level_based", "PRICE"]].groupby("customers_level_based").agg({"PRICE": "mean"}).sort_values(by="PRICE", ascending=False).reset_index()
agg_df["customers_level_based"].head() # Just checking data again

Identify new level-based customers

Now we assign the characters of the customers which we have converted to unique values(persona value). We divide the personas into 4 groups from the least profitable customer group to the most profitable customer group. “D, C, B, A”

This time we separate this separation process into quantiles with qcut, not with cut.

qcut : Quantile-based discretization function

# Segment new customers (Personas)

agg_df["SEGMENT"] = pd.qcut(agg_df["PRICE"], 4, labels=["D", "C", "B", "A"])
agg_df.head()

Segment new customers (Personas)

We would like to take a look at the basic statistical values of the customer personas we have segmented. In the last line, there is an example of selecting and describing only the C segment.

Additional note: These examples and usage patterns may be redundant during ongoing code. It is worth reminding again that the purpose here is to show how to select data from a dataset and to perform basic analysis for this.

# Describe the segments and especially "C"

agg_df.groupby(["SEGMENT"]).agg({"PRICE": ["mean", "max", "sum"]})

agg_df[agg_df["SEGMENT"] == "C"].describe()

Describe the segments

Now our segmentation process is complete.

Next is to know which segments the new customers belong to and to potential purchases on average.

Let’s say a 25-year-old French man downloaded the game and processing on game market with his Android device. In this case, which segment does it belong to and how much does it earn on average?

new_user = "fra_android_male_24_30"
print(agg_df[agg_df["customers_level_based"] == new_user])

This customer is in the C segment and spend potentially 33 units of money to the game.

In this way, special marketing strategies can be planned by segmenting all new customers.

I will be waiting for your comments and likes.

I wish everyone success in this learning path.

Thanks especially going to VBO (Veri Bilimi Okulu)

References:

Libraries

pandas documentation - pandas 1.3.2 documentation