1. Motivation

Natural language processing is among the most attractive and difficult field in machine learning. Different from computer vision and other machine learning tasks, NLP does not convey meaning through any physical manifestation. By the virtue of deep learning, NLP achieved tremendous progress in keyword search, machine translation, semantic analysis and etc. In this project, I would like to make a lyrics generator by using both character level and word level RNN(recurrent neural network).

The dataset is from kaggle with 3.8 million song lyrics from various artist.

But I will only use lyrics from Eminem and Michael Jackson. Because they have around 400 songs, it is easier to extract regular patterns from them.

2. Exploratory data analysis

Unlike other machine learning tasks, there is not much visualization we can do with NLP. And data cleaning will be based on the model that we want to explore. For character-based model, it is necessay to keep punctuations as they are part of the characters. However, for word-based model, punctuations are supposed to be removed. To briefly explore the dataset, we will count most frequently used words for different artists (bag of words or unigram). Consequently, very crude sentiment score can be obtained from it.

Bag-of-words

Remove stop words and punctuations from lyrics

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(stop_words='english')
X = vectorizer.fit_transform([eminem_lyrics])

Sentiment score base on bag-of-words

from nltk.sentiment.vader import SentimentIntensityAnalyzer

def sentiment(eminem_lyrics):
    sentiment = SentimentIntensityAnalyzer()
    score = sentiment.polarity_scores(eminem_lyrics)
    return score

Based on bag of words, one can do naïve Bayes to predict the genre of songs, but we will not cover it here. Let us go deeper to deep learning.

Character-level recurrent neural network(RNN)

Why recurrent? Different from vanilla neural network, RNN (see below, pic from wikipedia) is able to process sequences of inputs (such as words and sentences) by utilizing the internel state (memory state). Hence, it is regarded as a very promising candidate to solve NLP tasks.

Inspired by the minimal character-level Vanilla RNN model from Andrej Karpathy, we decided to build a more complicated RNN model to generate lyrics. Below is the summary of my model: 2 LSTM layers and 1 dense layer.

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_1 (LSTM)                (None, 10, 200)           230400    
_________________________________________________________________
dropout_1 (Dropout)          (None, 10, 200)           0         
_________________________________________________________________
lstm_2 (LSTM)                (None, 200)               320800    
_________________________________________________________________
dropout_2 (Dropout)          (None, 200)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 87)                17487     
=================================================================
Total params: 568,687
Trainable params: 568,687
Non-trainable params: 0
_________________________________________________________________

After 600 epochs, the model achieved 64% accuracy for validation set.

Epoch 600/600 - loss: 0.7529 - acc: 0.7858 - val_loss: 1.6630 - val_acc: 0.6402

1. How to prepare the data and train the model?

• One-hot encode all characters ( a-z, 0-9 and punctuations !"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n )

from keras.utils import to_categorical #[1,2,3] -> [[1,0,0], [0,1,0], [0,0,1]] one-hot_X = to_categorical(X, num_classes=vocab_size)

• Make a sliding window that collects 10 characters as input. Model only generates one character.

input: 'hello, world' -> output: 'n', true output: 'p'

• Calculate cross entropy and backpropagate the neural network to update 568,687 parameters.
• Slide the window to the right by one character. Extract new input and iterate above process until cost function reaches the minimum.

2. How to generate lyrics? Teacher forcing

• Make seed lyrics. Feed it to the neural network to generate one character after the seed lyrics, ‘b’.
• input: 'I want to ' -> output 'b'
• Append new character to the seed lyrics and remove the very first character.
• new input : ' want to b'
• Feed the new seed lyrics into the neural network and iterate the above process as many as you want.
• 'I want to ' -> ' want to b' -> 'want to be' -> .... -> 'ing of pop'
• In the end, you might get something like ‘I want to be king of pop’

This process is known as teacher forcing: training neural network that uses model output from a prior time step as an input.

3. Eminem’s lyrics generator

After one epoch, generated lyrics:

input: 'not afraid'    output: 'the me the me the me the me the me the me the me the me the'

Not smart, it repeats the same words over and over.

After 600 epochs, generated lyrics:

20 characters
input: 'slim shady'    output: '(what you was a mom'

40 characters
input: "i'm rap go"    output: "ing, you should see her, she won't even"

60 characters
input: 'the way i'    output: "lounding off, they can try to take your heart and I don't kn"

80 characters
input: 'lose myself'    output: ' in the music, the moment You own it, you better never let it go You only get o'

100 characters
input: 'not afraid' output: ') To take a stand) Maybe a dontte the back of the way that I know what the fuck I say to the motherf'

It is amazing as the generator can spell the word correctly and it is not hard to tell that they have eminem style.

Word-level RNN

1. Word embedding and word2vec

Instead of letting the model learning how to spell words. One can upgrade the model from character-level to word-level. Correspondingly, this endows model the ability to learning semantics from the corpus. Since the number of unique words is much larger than that of characters, it is necessay to introduce a new representation: word embedding. This is basically the only difference from character-based model. However, there is much more wisdom than just dimension reduction. The notion was first referred by Misolov, et al.,2013. Word embedding rotates word vector from one-hot representation to word2vec representation.

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_1 (Embedding)      (None, 10, 100)           600500
_________________________________________________________________
lstm_1 (LSTM)                (None, 10, 300)           481200
_________________________________________________________________
dropout_1 (Dropout)          (None, 10, 300)           0
_________________________________________________________________
lstm_2 (LSTM)                (None, 300)               721200
_________________________________________________________________
dropout_2 (Dropout)          (None, 300)               0
_________________________________________________________________
dense_1 (Dense)              (None, 300)               90300
_________________________________________________________________
dense_2 (Dense)              (None, 6005)              1807505
=================================================================
Total params: 3,700,705
Trainable params: 3,700,705
Non-trainable params: 0
_________________________________________________________________

Training is much harder than character-base model. Only 32% accuracy is obtained from this model.

Epoch 300/300 - loss: 2.1268 - acc: 0.7121 - val_loss: 8.1230 - val_acc: 0.3291

2. Michael Jackson's lyrics generator based on word-level RNN

After 300 epochs:

• Generate 20-word: here to change the world hee were afraid that away when we change my hand verse ill can never change my heart in pieces lost my heart on the carousel to
• Generate 40-word: a circus girl who left my heart in pieces lost my heart on the carousel to a circus girl who left my heart in pieces lost my heart on the carousel to a circus girl who left my heart in just a little bit baby thats all i need thats all
• Generate 60-word: night need you dont understand you need what about hard let me change you comes your truth out and to guess down together together then be her day every thing out the carpet were gonna see this one understand by much then ever so to see you there are long so game if let me get away verse verse cant gonna tell you right just show your face in broad daylight
• Generate 80-word: im create your crescendo how they dance well theres a reason what its a door when here all never door brother its yes we yeah yeah when you start to say if you truth its whole past and they start that can do tell me girl ive wont you have it tell me no little when the door now im cries chorus bridge cause its door made your game chorus ive never stop up and tell me no true oh her dreams left behind everything for the movie scene nothing more
• Generate 100-word: little start i go a i am away im dreams of this life girl girl tell me fall im together so much i feel you all i say that you say into into i lost my heart i can and you game on me baby ill see i can be far away today i love you from your truth cause youre across the bitch baby does it feel it needs me from a door start that not here there was ghost of moon aint a you better he made the you she win your dreams off the madness i never

It generates something with correct grammar at some parts but it is hard to understand the meaning.

3. How to improve word-level model?

a. Using pretrained word embedding

Word-level RNN is essentially concatenation of two neural networks. If we train two parts separately, we should achieve better accuracy. However, after using a pretrained word embedding, the accuracy of validation set decreases to 20%. Such a counterintuitive result!

Epoch 50/50 - loss: 4.7769 - acc: 0.3353 - val_loss: 6.1902 - val_acc: 0.2160

from keras.layers import Embedding
model = Sequential()
model.add(Embedding(num_of_tokens, latent_dim, input_length=seq_length, weights=[pretrained_embedding], train = False))

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_1 (Embedding)      (None, 10, 100)           600500
_________________________________________________________________
lstm_1 (LSTM)                (None, 10, 250)           351000
_________________________________________________________________
dropout_1 (Dropout)          (None, 10, 250)           0
_________________________________________________________________
lstm_2 (LSTM)                (None, 250)               501000
_________________________________________________________________
dropout_2 (Dropout)          (None, 250)               0
_________________________________________________________________
dense_1 (Dense)              (None, 6005)              1507255
=================================================================
Total params: 2,959,755
Trainable params: 2,359,255
Non-trainable params: 600,500
_________________________________________________________________

There are 600,500 non-trainable parameters which are from pretrained word embedding. Using non-trainable embedding seems not working well. Maybe it would be better to train the word embedding particularlly for the dataset by using CBOW or skip-gram.

b. Word-level Seq2Seq model

Seq2Seq model was widely used in neural tranlation machine. But there is nothing wrong to apply it to lyrics generator. The basic idea is to process input in the encoder end and generate a memory state (a vector) that represents the whole input message. Decoder take the memory state and SOS token to generate one token. Generated token becomes the next input for decoder to predict the next token. The process iterates many cycles until EOS is generated or max length of output is reached. Accordingly, unlike many-to-one model, seq2seq model has advantage of generating more than one token.

However, the performance does not change by much. But from recently research, inplementing attention mechanism could significantly improve the performance. Attention mechanism not only resolves long-term dependency problem of vanilla Seq2Seq languange model but also speeds up training process as it dicarded RNN which disfavors parallel computation.

Concluding remarks

Takeaways for tuning hyperparameters:

1. It is easy to overfit the model. It is necessay to add Dropout layer after each LSTM layer.
2. Sometimes GRU is better than LSTM and computationally cheaper.
3. Initialization and luck are very important. Try to restart kernel if model is stuck at local minimum.
4. Try [importance sampling](https://arxiv.org/pdf/1803.00942.pdf) which randomly takes samples from distrution instead of feeding datapoints in order

• Character-based models perform better than word-base. Even though word embedding is a very innovative method in NLP, word vectors by itself hardly convey semantics efficiently.
• Future work:
• 1.Try negative sampling to boost the training and improve the metrics (better than softmax).
• 2. Implement attention mechanism to seq2seq model.

Reference

1. Distributed Representations of Sentences and Documents
2. Distributed Representations of Words and Phrases and their Compositionality
3. Neural machine translation by jointly learning to align and translate
4. Attention Is All You Need
5. Importance Sampling
6. Sequence to Sequence Learning with Neural Networks

Acknowledgement

• Dataset is from Kaggle.
• Thanks for the tutorial GPU-accelerated Deep Learning on Windows 10.

Final Words

If any errors are found, please email me at venkat.chadalavada@gmail.com

Introduction

Sales is one of the essential components of any business. It is the act of selling goods or services to an interested party, in exchange for money. It is the primary process for a business to generate revenue, which explains why it is so vital for any organisation. Without sales, a business will not have the resources necessary to grow or even survive.

Due to its importance, this project will examine the sales process of an organisation and seek to achieve the following goals:-

Goal 1 — Predict the value of sales opportunity (‘Amount’ column) using regression models

Goal 2 — Predict the outcome of sales opportunity (‘Result’ column) using classification models

Table of Contents

Part 1 Pre-Processing
1.1 — Data Overview
1.2 — Data Cleaning
1.3 — Exploratory Data Analysis
1.4 — Feature Engineering
Part 2 Predict Opportunity Amount
2.1 — Feature Selection
2.2 — Prepare training/testing data
2.3 — Model Generation
2.4 — Model Results Evaluation
2.5 — Model Selection
Part 3 Predict Opportunity Outcome
3.1 — Prepare training/testing data
3.2 — Data Imbalance
3.3 — Model Generation
3.4 — Model Results Evaluation
3.5 — Model Selection
Conclusion

Part 1 Pre-Processing

1.1 — Data Overview

In this project, we will be looking at the sales database for an automotive supplies wholesaler. This is a sample dataset provided by IBM Watson Analytics. The goal of the sales process is to identify and communicate with potential leads, and explore opportunities to convert them into customers. For this project, the database contains records of every sales opportunity with information about the client, product, and process status which focuses on 3 key stages:-

1. Identified/Qualifying
2. Qualified/Validating
3. Validated/Gaining Agreement

This dataset consists of 78,025 rows and 19 columns (9 numerical, 10 categorical). The columns include the following:-

1. Amount — Estimated total revenue of opportunities in USD. (Goal 1)
2. Result — Outcome of opportunity. (Goal 2)
3. Id — A uniquely generated number assigned to the opportunity.
4. Supplies — Category for each supplies group.
5. Supplies_Sub — Sub group of each supplies group.
6. Region — Name of the region.
7. Market — The opportunities’ route to market.
8. Client_Revenue — Client size based on annual revenue.
9. Client_Employee — Client size by number of employees.
10. Client_Past — Revenue identified from this client in the past two years.
11. Competitor — An indicator if a competitor has been identified.
12. Size — Categorical grouping of the opportunity amount.
13. Elapsed_Days — The number of days between the change in sales stages. Each change resets the counter.
14. Stage_Change — The number of times an opportunity changes sales stage. Includes backward and forward changes.
15. Total_Days — Total days spent in Sales Stages from Identified/Validating to Gained Agreement/Closing.
16. Total_Siebel — Total days spent in Siebel Stages from Identified/Validating to Qualified/Gaining Agreement.
17. Ratio_Identify — Ratio of total days spent in the Identified/Validating stage over total days in sales process.
18. Ratio_Validate — Ratio of total days spent in the Validated/Qualifying stage over total days in sales process.
19. Ratio_Qualify — Ratio of total days spent in Qualified/Gaining Agreement stage over total days in sales process.

1.2 — Data Cleaning

Before proceeding, it is necessary to conduct checks to evaluate the integrity of the dataset. Incorrect and incomplete data will affect the results of our models, so these issues will need to be identified and addressed first.

Missing values is a common problem faced in data science. Fortunately, our dataset is complete without any missing values.

Next, a check will be conducted on the 3 ratio columns (Ratio_Identify, Ratio_Validate, Ratio_Qualify) to ensure that the total value does not exceed 1. A new column Total_Ratio will be created that sums up the values of these 3 columns.

The results of the new Total_Ratio column shows that there are 471 records with a total ratio that exceeds the total of 1. Upon closer inspection, the exceeded amount for each of these records is very minor and it is safe to assume that this is possibly due to the rounding in the 3 ratio columns and thus these records will remain as is.

However, there is a single record that has an extreme value of 1.007547. This record will therefore be removed.

The string values for the categorical columns will also be cleaned up to ensure a consistent format.

cars[‘Client_Employee’].replace([‘1K or less’, ‘More than 25K’, ‘5K to 15K’, ‘1K to 5K’, ‘15K to 25K’], [‘Below_1K’, ‘Above_25K’, ‘5K_to_15K’, ‘1K_to_5K’, ‘15K_to_25K’], inplace=True)

1.3 — Exploratory Data Analysis

With the data cleaned, we can now carry out EDA and perform an in-depth analysis of the data.

Lets start by examining the correlation between the numerical columns through the construction of a heatmap.

The heatmap above immediately higlights an almost perfect correlation between the Total_Days and Total_Siebel columns. This will have a negative impact on our models, therefore the Total_Siebal column will be dropped. The other numerical columns have a low to moderate correlation value, which are acceptable and no additional measures will be required.

Next, we will analyse the distribution of our features by plotting various graphs.

We will use histograms to represent data in the numerical columns. Majority of the graphs show a positively skewed distribution, with very few data points located on the right side. The 3 ratio columns represent data points that are located at both extreme ends of the scale and a minimal number in the middle.

With regards to the categorical columns, bar graphs illustrate the distribution across the different classes. We can see a decent spread across all classes for the Supplies_Sub, Region and Size columns. However, the other features have a strong class imbalance distribution, with a single dominant class. Even our target feature Result has an imbalance where the majority class is about three times larger than the minority class. Class imbalance can have a negative impact on machine learning algorithms, and may result in predictive models that are biased and inaccurate.

1.4 — Feature Engineering

The next stage of pre-processing requires some changes to be made to the current features. The first step involves converting the values in our target column Result into binary values.

cars['Result'] = cars['Result'].map(lambda x: 0 if x == 'Loss' else 1)

The classes in the Client_Past columns are also changed. The data is grouped into 2 classes to identify if the opportunity is from a new client or existing client, rather than separating into 5 classes.

cars['Client_Past'] = cars['Client_Past'].map(lambda x: 0 if x == '0 (No business)' else 1)

Comparing the Supplies and Supplies_Sub columns, we can see that Supplies_Sub actually is more detailed and provides a clearer picture as compared to Supplies. Therefore, the Supplies column will be dropped.

With regards to the categorical columns, bar graphs illustrate the distribution across the different classes. We can see a decent spread across all classes for the Supplies_Sub, Region and Size columns. However, the other features have a strong class imbalance distribution, with a single dominant class. Even our target feature Result has an imbalance where the majority class is about three times larger than the minority class. Class imbalance can have a negative impact on machine learning algorithms, and may result in predictive models that are biased and inaccurate.

1.4 — Feature Engineering

The next stage of pre-processing requires some changes to be made to the current features. The first step involves converting the values in our target column Result into binary values.

cars['Result'] = cars['Result'].map(lambda x: 0 if x == 'Loss' else 1)

The classes in the Client_Past columns are also changed. The data is grouped into 2 classes to identify if the opportunity is from a new client or existing client, rather than separating into 5 classes.

cars['Client_Past'] = cars['Client_Past'].map(lambda x: 0 if x == '0 (No business)' else 1)

Comparing the Supplies and Supplies_Sub columns, we can see that Supplies_Sub actually is more detailed and provides a clearer picture as compared to Supplies. Therefore, the Supplies column will be dropped.

The Total_Siebel column was previously shown to be highly correlated to the Total_Days column, so that too will be removed.

Due to the number of categorical features in our dataset, we will create 38 new dummy variables to replace the original categorical columns. This new set of dummy variables is then concatenated with the original dataset to create a new dataset to be used our predictions.

cat_dummy = pd.get_dummies(cars2[cat_columns], drop_first=True)
cars2.drop(cat_columns, axis=1, inplace=True)
cars2 = pd.concat([cars2, cat_dummy], axis=1)

With pre-processing completed, we are now ready to commence building our predictive models.

Part 2 Predict Opportunity Amount

All businesses are interested in knowing how much revenue they can make for each transaction, therefore the ability to predict the value of a sales opportunity would be very insightful. In this section, the objective will be to predict the values in the Amount column and through the use of Regression algorithms. But first, we will need to take additional steps to prepare our data for model generation.

2.1 — Feature Selection

We will begin by performing feature selection by using the .summary() command from statsmodels python package to analyse the p-values of each feature. Any features with a p-value or 0.05 and higher will be deemed insignificant and thus dropped.

These are the features that were dropped.

cars2.drop(['Ratio_Identify', 'Supplies_Sub_Electronics', 'Supplies_Sub_Garage_Car_Care', 'Market_Other',
            'Client_Revenue_500K_to_1M', 'Client_Revenue_250K_to_500K', 'Supplies_Sub_Towing_Hitches',
            'Client_Revenue_Above_1M', 'Client_Revenue_Below_100K', 'Client_Employee_1K_to_5K',
            'Client_Employee_5K_to_15K', 'Client_Employee_Below_1K', 'Client_Employee_Above_25K',
            'Supplies_Sub_Tires_Wheels', 'Region_Midwest', 'Elapsed_Days', 'Supplies_Sub_Replacement',
            'Supplies_Sub_Motorcycle_Parts', 'Ratio_Validate', 'Total_Days'], axis=1, inplace=True)

Using this method, 20 out of 44 features were removed. The selected features are shown below.

2.2 — Prepare training/testing data

To prepare our data, we will use the holdout method to split our dataset. We will use training set that comprises of 70% of the data to train our models, and a testing set of 30% to assess their predictions. 5-fold cross validation will also be applied to the training set for all models.

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=100)

2.3 — Model Generation

Model construction can begin by using the new prepared datasets.

5 different regression algorithms will used to create the models:

1. Linear Regression
2. Lasso Regression
3. Ridge Regression
4. SGD Regression
5. Random Forest Regression

The diagram below is an example of the code used to build the Lasso Regression model.

The results for each of the 5 models will be collected and evaluated, before selecting the model that best predicts the opportunity amount.

2.4 — Model Results Evaluation

With the models completed, we can now compile all the results into a new pandas dataframe, making it easier to compare the results. Graphical representations will also be generated to help visualise the output.

Model performance will be judged based on 2 key criterias:-

1. R² score (training, cross validation, testing)
2. Root Mean Squared Error score (RMSE)

We will also take a look at the feature rankings for the models.

When conducting evaluation, we are looking for a model with a high R² score and low RMSE score. The R² score would indicate the goodness of fit of a set of predictions on the actual values, RMSE indicates the magnitude of error between the predicted and actual value in terms of the output value.

Looking at the compiled results, we see that Linear, Lasso and Ridge Regression achieved the highest R² scores, and the results were identical. Even their RMSE scores were almost exact. On the other hand, the Random Forest Regression model performed the worst with the lowest R² score and highest RMSE score.

In terms of feature ranking, all 5 models share the same features that occupy the top 4 ranks, which are Size_above_60K, Size_50K_to_60K, Size_40K_to_50K and Size_30K_to_40K. After that, feature rankings begin to differentiate between models. These top 4 features also have very similar coefficient values, with the exception of the Random Forest Regression model which uses a different weights scale. The coefficient values show us how large of an impact a particular feature has on the overall opportunity amount.

2.5 — Model Selection

As mentioned earlier, the ideal model is judged according to their R² and RMSE scores. As Linear Regression, Lasso Regression and Ridge Regression had the highest R² testing score and their values are identical, we will then compare the RMSE score between these 3 models. Based on our evaluation of the results above, we can conclude that the Ridge Regression model is the best choice with the lowest RMSE score of 39646.3218.

Part 3 Predict Opportunity Outcome

In part 3 of this project, we will now create classification models to predict the sales opportunity outcome (Result column). The ability to predict the outcome enables a business to identify influencing factors and also better utilize their resources.

Previously in our pre-processing stage, we converted the values in the Result column into binary numbers. The value 1 represents the opportunities that were won, while the value 0 signifies the opportunities that were lost. Hence, our goal will be to accurately predict the highest number of 1s.

3.1 — Prepare training/testing data

For this prediction, we will also be using the holdout method to create the training and testing datasets, and incorporating 5-fold cross-validation into the models.

The next step incorporates the SelectFromModel function from sklearn to perform feature selection, and apply the RandomForestClassifier inside the function as an estimator to identify the feature importance.

from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier
select = SelectFromModel(RandomForestClassifier())
select.fit(Xs_train, y_train)
Xs_train = select.transform(Xs_train)
Xs_test = select.transform(Xs_test)
feature_support = pd.DataFrame({'feature': X_train.columns, 'support': select.get_support()})
feature_support.sort_values('support', inplace=True, ascending=False)
feature_support.head(10)

The results show that out of the 44 features in our dataset, only 9 were selected. This means that these 9 features had the biggest influence in prediction the opportunity outcome.

3.2 — Data Imbalance

As previously shown in our distribution plots under the EDA section, there is an imbalance in our target column whereby the majority class 0 has a total count of more than 3 times when compared to the minority class 1. To solve the problem of class imbalance, we will utilize 2 different techniques from the python package imblearn to resample the data in our training set.

RandomOverSampler — randomly oversample the minority class (1) with replacement.

from imblearn.over_sampling import RandomOverSampler
ros = RandomOverSampler(random_state=100)
Xs_train_ros, y_train_ros = ros.fit_sample(Xs_train, y_train)

RandomUnderSampler — randomly undersample the majority class (0) with replacement.

from imblearn.under_sampling import RandomUnderSampler
rus = RandomUnderSampler(random_state=100)
Xs_train_rus, y_train_rus = rus.fit_sample(Xs_train, y_train)

Oversampling will increase the number of minority class from 12401 to 42215, while undersampling will reduce the number of majority class from 42215 to 12401. The end result after resampling is 3 different training sets (default, oversample, undersample).

3.3 — Model Generation

Now that we have our training and testing datasets, we can begin to generate our models.

3 different classification algorithms will used to create the models:

1. Logistic Regression
2. Random Forest
3. K-Nearest Neighbors

3 versions of each algorithm will also be created, with each version tackling a different dataset (default, oversample, undersample). In the end, we will have 9 models to compare. The results for each of the 9 models will be collected and evaluated, before selecting the model that best predicts the opportunity result.

3.4 — Model Results Evaluation

With the models completed, we can now compile all the results into a new pandas dataframe, making it easier to compare the results. Graphical representations will also be generated to help visualise the output.

An example of the output results for the Logistic Regression (default) model is shown below.

Model performance will be judged based on 3 key criterias:-

1. Confusion matrix results (TN, FP, FN, TP)
2. Accuracy scores (training, cross validation, testing, AUC ROC)
3. Precision and recall scores of the minority class

From the precision and recall graphs above, we can see an inverse correlation between them. Models that had a high precision result like Random Forest (default) and KNN (default) did not perform well in recall, while models with high recall results like KNN (undersample) and Random Forest (undersample) had a lower precision result.

Another interesting point to note is that default models achieved higher precision scores while models with undersampling had higher recall scores.

In terms of accuracy scores, our models will need to achieve an accuracy score that is higher than the baseline of 0.77408. From the results, all 9 models were able to exceed the baseline. The Logistic Regression models in general had the lowest scores, while the Random Forest models had the best.

When comparing the 4 accuracy graphs,the random forest (oversample) model was consistently able to achieve one of the highest scores across all 4.

Looking at the confusion matrix results, the KNN (undersample) and Random Forest (Undersample) models had the highest number of correct predictions for the minority class (TP), while the Random Forest (default) and Logistic Regression (default) models had the most number of majority class predictions (TN).

In terms of wrong predictions, Logistic Regression (oversample and undersample) has the most Type I errors (FP) and Logistic Regression (default) the highest number of Type II errors (FN). This means that Logistic Regression models are unable to accurately classify the minority class and would be unsuitable algorithm to achieve our goal.

When comparing the Random Forest and KNN models, the undersampling models had the most False Positive (FP) errors and the lowest False Negative (FN) rates . On the other hand, while default and oversampling models may have a lower FP rate, their FN rate is higher.

3.5 — Model Selection

Since our goal is to predict the highest number of win opportunities, a model with high recall is desired. Being able to identify these accurately will mean more revenue for the business. Based on this, the K-Nearest Neighbors (undersample) model was able to achieve the highest recall score of 0.83, the highest True Positive (TP) rate of 4349, and lowest False Negative (FN) rate of 877. While its Type II error (FP) is one of the highest out of all the models, it is more acceptable to wrongly predict a loss opportunity than an inaccurate prediction of a won opportunity. Therefore, this model would be most suitable to achieve our goal of predicting the opportunity outcome.

Conclusion

After constructing our predictive models and analysing the results, we can conclude that the goals that were originally set have been achieved. The Ridge Regression model is suitable to predict the value of sales opportunities, and the K-Nearest Neighbors (undersample) model can be used to predict the outcome. Overall, this project demonstrates the usefulness of machine learning in creating business insights.