Done by: Lamalharbi, Monirah abdulaziz, Aisha Y Hakami

In this project Notebook we will go through the whole process of cleaning the data and developing a machine learning model on the Titanic dataset which is one of the famous data sets which every Data Scientist explores at the start of their career and here we are. The dataset provides all the information on the fate of passengers on the Titanic, summarized according to sex, age, economic status (class), and survival.

In addition, we join the Titanic: Machine Learning from Disaster challenge on Kaggle according to our work.

Import Libraries

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

%matplotlib inline
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import cross_val_score, train_test_split, GridSearchCV
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import SGDClassifier,Perceptron
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.svm import SVC,LinearSVC

# To ignore unwanted warnings
import warnings
warnings.filterwarnings('ignore')

Data Wrangling

We used the data after applying the neccessary cleaning in the Random Forest Lab by:

• Dealing with null values.
• Gaining more information on text data.

To look deeper see Cleaning Post

train = pd.read_csv('datasets/cleaned_train.csv')
test = pd.read_csv('datasets/cleaned_test.csv')

Data Dictionary

Main features in the dataset before cleaning steps:

Exploratory Data Analysis

The most important point in working with the data is to understand the it first, then try to find patterns in it by using the visualization because in the real-world, so it’s important to ask why this person particulary survived?, to answer we need to look deeper to the features and the relationship.

# grab the quantitative features
quan_train = list(train.loc[:,train.dtypes != 'object'].columns.values)
quan_train.remove('Survived')

# visualize the distribution of each numerical feature

temp = pd.melt(train.drop('Survived',axis=1), value_vars= quan_train)
grid = sns.FacetGrid(temp, col="variable",  col_wrap=5 , size=3.0, 
                     aspect=1.0,sharex=False, sharey=False)
grid.map(sns.distplot, "value")
plt.show()

Observations

• For the Age, SibSp,Parch, value, Embarked_Q, Fare, Cabin are positively skewed.
• The Pclass, sex_male, and Embarked_S are negatively skewed.

# selecting without ID
corr = train.iloc[:, 1:].corr()
plt.figure(figsize=(8, 8))
sns.heatmap(corr, vmax=1, square=True,annot=True,cmap = 'RdBu')

# List the numerical features decendingly by correlation with Survived
cor_dict = corr['Survived'].to_dict()
del cor_dict['Survived']
print("List the numerical features by their correlation with Survived:\n")
for ele in sorted(cor_dict.items(), key = lambda x: -abs(x[1])):
    print("{0}: \t{1}".format(*ele))

Observations

• The distribution of each variable with itself is shown on the diagonal
• To see patterns the observable pattern is that all the variables correlate with each other. On the bottom of the diagonal : we can see the negative correlations.
• On the top of the diagonal : the positive correlation is displayed.

sns.pairplot(train.drop("Name",axis = 1),hue = "Survived",palette='RdBu')

Observations

• The Pair plot helps us to understand the best set of features to explain a relationship between two variables
• We still need to take a deeper look.

# here we take only 4 features to  perform analysis.
sns.set_style("whitegrid");
sns.pairplot(train[["Survived","Pclass","Fare","Age"]], hue="Survived", size=3);
plt.show()

Observarions

• We can see that passengers who paid higher fare or travelling in upper class has a higher chances to survive.
• According to the Age the young persons have a higher chance to survive than old peoples.
• The picture might not very clear so lets take step towards uni-variate analysis to understand what exactly is happening

Uni-variate Analysis

1. Survived Feature

# Countplot counts the each category of value and plot that.
sns.countplot(train['Survived'],data = train)

2. Pclass Feature

sns.countplot(train["Pclass"],hue = train["Survived"],data = train)

Other features

plt.figure(1)
f, axarr = plt.subplots(3, 2, figsize=(10, 9))
Survived = train.Survived.values
axarr[0, 0].hist(train['Pclass'].values,bins=3)
axarr[0, 0].set_title('Pclass')
axarr[0, 1].hist(train.Cabin.values,bins=2)
axarr[0, 1].set_title('Cabin')
axarr[1, 0].hist(train.Age.values, bins=5)
axarr[1, 0].set_title('Age')
axarr[1, 1].hist(train['Parch'].values,bins=7)
axarr[1, 1].set_title('Parch')
axarr[2, 0].hist(train.Embarked_Q.values, bins=2)
axarr[2, 0].set_title('Embarked_Q')
axarr[2, 1].hist(train.Embarked_S.values, bins=2)
axarr[2, 1].set_title('Embarked_S')
f.text(-0.01, 0.5, 'Survived', va='center', rotation='vertical', fontsize = 18)
plt.tight_layout()
plt.show()

Observations

• The Class 3 have the highest survivors.
• Passenger with age between 20–30 has high chance to survive.

Feature Engeneering

For the Parch and SibSp we can use these columns to explore wether the passenger is traveled alone or with relatives.

# add relative
data = [train, test]
for dataset in data:
    dataset['relatives'] = dataset['SibSp'] + dataset['Parch']
    dataset.loc[dataset['relatives'] > 0, 'travelled_alone'] = 0
    dataset.loc[dataset['relatives'] == 0, 'travelled_alone'] = 1

To remove any noise might occur while training the models we decide to split the Age and Fare to quarter classes.

# add age and fare
train['CatAge'] = pd.qcut(train.Age, q=4, labels=False )
train['CatFare']= pd.qcut(train.Fare, q=4, labels=False)

test['CatAge'] = pd.qcut(test.Age, q=4, labels=False )
test['CatFare']= pd.qcut(test.Fare, q=4, labels=False)

# Train: drop the features after extracting info
train = train.drop(['SibSp','Parch'], axis=1)
train = train.drop(['Age', 'Fare'], axis=1)

# Test: drop the features after extracting info
test = test.drop(['SibSp','Parch'], axis=1)
test = test.drop(['Age', 'Fare'], axis=1)

There’s a large variety of titles from the people on the ship.

We will make new features for Titles from Name column, with grouping the people according to Mr, Mrs .. etc.

# Extract titles from Name feature.
import re 
def get_title(name):
    title_search = re.search(' ([A-Za-z]+)\.', name)
    if title_search:
        return title_search.group(1)
    return ""

train['title']=train['Name'].apply(get_title)
test['title']=test['Name'].apply(get_title)

title_lev1=list(train['title'].value_counts().reset_index()['index'])
title_lev2=list(test['title'].value_counts().reset_index()['index'])

title_lev=list(set().union(title_lev1, title_lev2))
# print(title_lev)

train['title']=pd.Categorical(train['title'], categories=title_lev)
test['title']=pd.Categorical(test['title'], categories=title_lev)

title_dum = pd.get_dummies(train['title'],drop_first=True)
train = pd.concat([train, title_dum],axis=1)
train=train.drop(['title'], axis=1)

# test 
test_dum = pd.get_dummies(test['title'],drop_first=True)
test = pd.concat([test, test_dum],axis=1)
test=test.drop(['title'], axis=1)

train=train.drop(['Name'], axis=1)
test=test.drop(['Name'], axis=1)

Correlation after feature engineering

corrmat = abs(train.iloc[:train.shape[0],:].corr())
plt.figure(figsize=(17, 8))
k = 15 #number of variables for heatmap
cols = corrmat.nlargest(k, 'Survived')['Survived'].index
cm = np.corrcoef(train.iloc[:train.shape[0],:][cols].values.T)
sns.set(font_scale=1.50)
hm = sns.heatmap(cm, cbar=True, annot=True, square=True,
                 fmt='.2f', annot_kws={'size': 10}, yticklabels=cols.values, xticklabels=cols.values,
                 cmap = 'RdBu', linecolor = 'white', linewidth = 1)
plt.title("Correlations between Survived and  features including dummy variables", fontsize =15)
plt.show()

Feature and Target

X_train = train[['Pclass','Cabin','sex_male','Embarked_Q','Embarked_S','relatives','travelled_alone','CatAge','CatFare', 'Miss', 'Mr','Mrs']]
y_train = train['Survived']
X_test = test[['Pclass','Cabin','sex_male','Embarked_Q','Embarked_S','relatives','travelled_alone','CatAge','CatFare', 'Miss', 'Mr','Mrs']]

Scaling

from sklearn.preprocessing import StandardScaler

s = StandardScaler()

X_s = pd.DataFrame(s.fit_transform(X_train) , columns=X_train.columns)
Xtest_s = pd.DataFrame(s.transform(X_test) , columns=X_test.columns)

# plot
fig,ax = plt.subplots(figsize=(16,8))
sns.boxplot(data=X_s, orient='h' , fliersize=2, linewidth=3, notch=True,saturation=0.5, ax=ax)
plt.show()

Detect the Outliers

from scipy.stats import zscore
from scipy.stats import iqr
iqr(X_s, axis=0)
z = zscore(X_s)
outlier = X_s[(z < 3).all(axis=1)]

Split the Data

# split

from sklearn.model_selection import  train_test_split, cross_val_score

X_train, X_test, y_train, y_test = train_test_split(X_s, y_train, test_size=0.3 , random_state = 101,stratify=y_train)

Baseline Model

# CODE HERE PLEASE
basline_acc = y_train.value_counts()[0]/(y_train.value_counts()[0]+y_train.value_counts()[1])
print(f"0: {basline_acc}")
basline_acc = y_train.value_counts()[1]/(y_train.value_counts()[0]+y_train.value_counts()[1])
print(f"1: {basline_acc}")

output:

0: 0.6163723916532905
1: 0.38362760834670945

Models

Decision Tree

from sklearn.tree import DecisionTreeRegressor
regressor = DecisionTreeRegressor(splitter = "random")
regressor.fit(X_train, y_train)
dtree_score=round(regressor.score(X_train,y_train)* 100, 2)
dtree_score

output:

74.35

KNN

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(X_train,y_train)
knn_score=round(knn.score(X_train, y_train) * 100, 2)
knn_score

output:

87.0

KNN with Grid Search

#grid search for KNN
params = {
    'n_neighbors':[1,4,6,8,10,5,20,19],
    'algorithm':[ 'ball_tree', 'kd_tree', 'brute'],
}
gs = GridSearchCV(knn, param_grid=params,  verbose = 1,n_jobs=-1)
gs.fit(X_train, y_train)
knn_score_gs=round(gs.best_score_ * 100, 2)
knn_score_gs

output:

83.3

Adabost

from sklearn.ensemble import RandomForestClassifier,  GradientBoostingClassifier , ExtraTreesClassifier , AdaBoostClassifier
gb= GradientBoostingClassifier()
gb.fit(X_train, y_train)
gb_score=round(gb.score(X_train,y_train) * 100, 2)
gb_score

output:

86.84

Adabost with Grid Search

# grid search for adabost
param_grid = {'learning_rate': [0.01],
 'max_depth': [3,5,6,7 ],
 'max_features': ['auto'],
 'min_samples_leaf': [15 ],
 'min_samples_split': [15],
 'n_estimators': [1500]}

grad = GridSearchCV(GradientBoostingClassifier(),
                           param_grid, cv=5, verbose= 1 , n_jobs=-1)
grad.fit(X_train , y_train)
gb_score_gs=round(grad.best_score_ * 100, 2)
gb_score_gs

output:

83.46

Random Forest with Grid Search

rf_params = {
   'n_estimators': [3,5,4,8,9,10,11,12,13,14],
    'max_depth': [1, 2, 3,4,6],
   'criterion':['gini'],
   'min_samples_split' : [3,5],   
   'max_features':[4,13,2,8],
    'bootstrap':[True,False]
}
rf_g = RandomForestClassifier(oob_score =True,random_state=1 )
gs = GridSearchCV(rf_g, param_grid=rf_params, cv=5, verbose = 1,n_jobs=-1)
gs.fit(X_train, y_train)
rf_score=round(gs.best_score_ * 100, 2)

rf_score

output:

85.07

Extra Tree with Grid Search

## extra tree
par = {'bootstrap': [True],
 'max_depth': [12,10,4,6],
 'min_samples_leaf': [4,2,5,7],
 'min_samples_split': [3,3,4,5],
 'n_estimators': [8]}

ex = GridSearchCV(ExtraTreesClassifier(),
                   par , cv = 5 , verbose= 1  , n_jobs= -1)
ex.fit(X_train , y_train)
ex_score=round(ex.best_score_ * 100, 2)
ex_score

output:

82.34

SVM

from sklearn import svm
from sklearn.svm import SVC

param_grid = {'C': [1000 , 10000],  
              'gamma': [ 0.001,0.01], 
              'kernel': [  'rbf'],}  
  
sv = GridSearchCV(SVC(), param_grid, verbose = 3 , n_jobs=-1 , cv = 5) 
  
# fitting the model for grid search 
sv.fit(X_train , y_train)
svm_score=round(sv.best_score_ * 100, 2)
svm_score

output:

84.59

Stochastic Gradient Descent

sgd = SGDClassifier()
sgd.fit(X_train, y_train)
Y_pred = sgd.predict(X_test)
sgd_score = round(sgd.score(X_train, y_train) * 100, 2)
sgd_score

output:

73.35

Linear svc

linear_svc = LinearSVC()
linear_svc.fit(X_train, y_train)
Y_pred = linear_svc.predict(X_test)
linear_svc_score = round(linear_svc.score(X_train, y_train) * 100, 2)
linear_svc_score

output:

82.83

Perceptron

perceptron = Perceptron()
perceptron.fit(X_train, y_train)
Y_pred = perceptron.predict(X_test)
perceptron_score = round(perceptron.score(X_train, y_train) * 100, 2)
perceptron_score

Scores

list the scores and the models:

models = pd.DataFrame({
    'Model': ['DecisionTreeRegressor', 'KNN','KNN with grid search', 'GradientBoostingClassifier', 'GradientBoostingClassifier with grid search'
             , 'RandomForestClassifier', 'ExtraTreesClassifier', 'SVM', 
              'SGDClassifier', 'LinearSVC', 
              'Perceptron'],
    'Score': [dtree_score, knn_score,knn_score_gs, gb_score,gb_score_gs, 
              rf_score, ex_score, svm_score, 
              sgd_score, linear_svc_score, perceptron_score]})
models.sort_values(by='Score', ascending=False)

output:

Kaggle submission scores

Below is screenshots of the highest score we get in the Kaggle competition:

Conclusion

At the end, Of course there is still more processes for improvement, like doing a more extensive feature engineering, by dive more into the features against each other by comparing and plotting to identifying and removing the outliers and noisy features. Moreover, the kaggle score could be improved by applying extensive hyperparameter tuning on the machine learning models.

Problem statement

Usually when we plan to buy a book specially the scientific books, we normally ask about the goods ones, research the book domains, compare the books with similar or read the reviews so here the recommender system is the master of this problem.

Datasets

This project uses two datasets each one containing more that one thousand rows for books with different features. The data was obtained from Kaggle.

The datasets

1. Children’s books in Amazon
2. Goodreads-Computer-Books

Amazon books

import libraries

# Importing necessary libraries
import pandas as pd
import numpy as np
import pandas as pd
import numpy as np
import seaborn as sns
from nltk.corpus import stopwords
from sklearn.metrics.pairwise import linear_kernel
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.tokenize import RegexpTokenizer
from sklearn.neighbors import NearestNeighbors
import re
import string
import random
from PIL import Image
import requests
from io import BytesIO
import matplotlib.pyplot as plt
%matplotlib inline

Import and read dataset

df= pd.read_csv('amazon_books_data.csv')
df.head()

EDA and Cleaning

regex the Description and the Product_Details features.

df.Product_Details[0]

regex is refer to regular expression which is a special sequence of characters that helps to match or find other helpful strings or sets of strings, using a specialized syntax.

df['Product_Details'] = df.Product_Details.str.replace(r'\n', '', regex=True).str.strip()
df.Product_Details[0]

From the result we can clearly see that thePublisher, ISBN would be good features in the data, so we can extract it also using regex and save it as new features.

# Publisher feature
df['Publisher'] = df['Product_Details'].str.extract('((?<=Publisher:).+?(?=;))' ,expand=False) #'ISBN-13:.+?(?= ;)
df['Publisher'] = df.Publisher.str.replace(r'-', '', regex=True).str.strip() #Program

# ISBN feature
df['ISBN'] = df['Product_Details'].str.extract('ISBN-13:(\d+-?\d*)', expand=False)##ISBN consists of 13 digits
df['ISBN'] = df.ISBN.str.replace(r'-', '', regex=True).str.strip()

Check Nulls

hasNAN = df.isnull().sum()
hasNAN = hasNAN[hasNAN > 0]
hasNAN = hasNAN.sort_values(ascending=False)
print(hasNAN,df.shape)

Create new dataframe that consists of desired features

df1=df[['ISBN','Name','Description','Rating_out_of_5','No_of_Ratings','Publisher']]

df1.head()

Drop Nulls

# Shape before dropping nulls
(1200, 6) 

df1 = df1.dropna(how='any',axis=0)

# Shape after dropping nulls 
(873, 6)

Text Preprocessing

Cleaning the book descriptions:

# Function for removing NonAscii characters
def _removeNonAscii(s):
    return "".join(i for i in s if  ord(i)<128)

# Function for converting into lower case
def make_lower_case(text):
    return text.lower()

# Function for removing stop words
def remove_stop_words(text):
    text = text.split()
    stops = set(stopwords.words("english"))
    text = [w for w in text if not w in stops]
    text = " ".join(text)
    return text

# Function for removing punctuation
def remove_punctuation(text):
    tokenizer = RegexpTokenizer(r'\w+')
    text = tokenizer.tokenize(text)
    text = " ".join(text)
    return text

# Function for removing the html tags
def remove_html(text):
    html_pattern = re.compile('<.*?>')
    return html_pattern.sub(r'', text)

# Applying all the functions in description and storing as a cleaned_desc
df1['cleaned_desc'] = df1['Description'].apply(_removeNonAscii)
df1['cleaned_desc'] = df1.cleaned_desc.apply(func = make_lower_case)
df1['cleaned_desc'] = df1.cleaned_desc.apply(func = remove_stop_words)
df1['cleaned_desc'] = df1.cleaned_desc.apply(func=remove_punctuation)
df1['cleaned_desc'] = df1.cleaned_desc.apply(func=remove_html)
df1.head()

Recommendation engine

Now we will build a recommendation engine using the book titles and descriptions. But first we need to convert each book title and description into vectors using TF-IDF and bigram. Our model will recommends a similar book based on title and description. Furthermore, we will define a function that takes the book title as input and returns the top five similar recommended books based on the title and description.

tfidf = TfidfVectorizer(stop_words='english')

df1['cleaned_desc'] = df1['cleaned_desc'].fillna('')

#Construct the required TF-IDF matrix by applying the fit_transform method on the overview feature

overview_matrix = tfidf.fit_transform(df1['cleaned_desc'])

#Output the shape of tfidf_matrix

overview_matrix.shape

(873, 12175)

similarity_matrix = linear_kernel(overview_matrix,overview_matrix)
similarity_matrix

#book index mapping
mapping = pd.Series(df1.index,index =df1['Name'])
mapping[:3]

def recommend_books_based_on_plot(book_input):
    book_index = mapping[book_input]
    #get similarity values with other books
    #similarity_score is the list of index and similarity matrix
    similarity_score = list(enumerate(similarity_matrix[book_index]))
    #sort in descending order the similarity score of book inputted with all the other books
    similarity_score = sorted(similarity_score, key=lambda x: x[1], reverse=True)
    # Get the scores of the 5 most similar books. Ignore the first book.
    similarity_score = similarity_score[1:6]
    #return book names using the mapping series
    book_indices = [i[0] for i in similarity_score]
    return (df1['Name'].iloc[book_indices])

Test the model

Let's now try to get a recommendation for the book ‘If Animals Kissed Good Night’ from the above recommendation function and see what it outputs.

recommend_books_based_on_plot('If Animals Kissed Good Night').to_frame()

In[31]:

df1.Description[649]

Out[31]:

"A must-have board book for all babies. Good night, Gorilla. Good night, Elephant.  It's bedtime at the zoo, and all the animals are going to sleep. Or are they? Who's that short, furry guy with the key in his hand and the mischievous grin? Good night, Giraffe. Good night, Hyena.  Sneak along behind the zookeeper's back, and see who gets the last laugh in this riotous good-night romp."

In [32]:

df1.Description[458]

Out[32]:

'Shaped, die-cut, picture-changing pages add a subtly interactive element to this peaceful, rhyming bedtime book―say "Night, night" and turn the page to watch the animals transform from being awake to asleep.It’s nighttime down on the farm. The animals are in the barn and it’s time to say a soft and cozy, "Night, night." Say goodnight to the horse, the dog, and all their farm friends, as you turn the shaped pages and watch as the animals go to sleep, one by one. The gentle rhymes and sleepy tone make Night Night Farm perfect for settling your little one into bed and ending with your own, quiet, "Night, night." With irresistibly sweet illustrations and a magical sky of glow-in-the-dark stars, Night Night Farm is the perfect way to end the day.'

We can observe from the descriptions that we can see that the recommender books have similar points in the stories from the 'If Animals Kissed Good Nightbook.

Goodreads

Import and read dataset

df= pd.read_csv('goodreads_books_data.csv')
df.head()

Check Nulls

hasNAN = df.isnull().sum()
hasNAN = hasNAN[hasNAN > 0]hasNAN = hasNAN.sort_values(ascending=False)
print(hasNAN,df.shape)

Series([], dtype: int64) (1234, 10)

df.isnull().sum()

EDA

Since the Book_Id column is as in index we will drop it and generate a new ID

df.drop(['Book_Id'],axis=1,inplace=True)

• Check duplicates and drop if exists.

df.shape

(1234, 9)

df[df['book_Title']=='Anime and the Visual Novel: Narrative Structure Design and Play at the Crossroads of Animation and Computer Games']

df.drop_duplicates(inplace=True)

df.shape

(1142, 9)

df[df['book_Title']=='Anime and the Visual Novel: Narrative Structure Design and Play at the Crossroads of Animation and Computer Games']

# Generate a new id 
df=df.assign(id=(df['book_Title']).astype('category').cat.codes)
df.head()

Visualisation

Most frequent authors

plt.subplots(figsize=(10,7))
df.Author_Name.value_counts()[:10].plot(kind="bar")
plt.show()

Content-Based Recommender Systems

# test poinst for ratings_count=40, Avg_Rating= 4.5, Publish_year= 2002, Pages_no= 523
test_point = [40, 4.5, 2002, 523]
X = df.iloc[:, [2, 3, 4, 6]].values

# build a nearest neighbor object, we are searching for just 3 neighbors so n_neighbors=3
nn = NearestNeighbors(n_neighbors=3).fit(X)
# kneighbors returns the neighbor for the test_point
print(nn.kneighbors([test_point]))

(array([[ 3.9028323 , 7.91060048, 12.2137791 ]]), array([[626, 445, 486]]))

# This is the most similar book from all the available books
df.iloc[626].to_frame()

df.iloc[445].to_frame()

From the results we can observe that the recommender books are near to the test points that we fed manually to the model.

Done by: Lamalharbi, Monirah abdulaziz, Aisha Y Hakami

Introduction

In this article we will go through the whole process of cleaning the data and develping a machine learning model on the Titanic dataset which is one of the famous data sets which every Data Scientist explores at the start of their career and here we are. The dataset provides all the information on the fate of passengers on the Titanic, summarized according to sex, age, economic status (class), and survival.

In addition, we join the Titanic: Machine Learning from Disaster challenge on Kaggle according to our work follow up to see the score.

Data Cleaning

# upload the data
train = pd.read_csv('../datasets/train.csv')
test = pd.read_csv('../datasets/test.csv')

Check the null values:

Observation:

• There is a missing values in train: Age, Cabin and Embarked.
• In the test: Age, Fare and Cabin.

fig, ax = plt.subplots(nrows = 1, ncols = 2, figsize = (18, 6))

# train data 
sns.heatmap(train.isnull(), yticklabels=False, ax = ax[0], cbar=False, cmap='viridis')
ax[0].set_title('Train data')

# test data
sns.heatmap(test.isnull(), yticklabels=False, ax = ax[1], cbar=False, cmap='viridis')
ax[1].set_title('Test data');

1. Embarked feature in Train

First we need to check How many ports are in Embarked column,

train.Embarked.value_counts()

the result was:

S 644
C 168
Q 77

Well, now we can drop the two rows with missing Embarked values in train, however, let's fill the nulls with the port of highest embarkation.

from collections import Counter

train.Embarked= train.Embarked.replace(np.nan,Counter(train.Embarked).most_common(1)[0][0])

2. Fare feature in Test

To fill the missing values in Fare we can use the Pclass column, by computing the average of Fare of the missing Class.

class3_mean = train[train['Pclass']==3]['Fare'].mean()
test['Fare'] = test['Fare'].replace({np.nan:class3_mean})

3. Age feature in Train and Test

#defining a function 'impute_age'
def impute_age(age_pclass): # passing age_pclass as ['Age', 'Pclass']
    
    # Passing age_pclass[0] which is 'Age' to variable 'Age'
    Age = age_pclass[0]
    
    # Passing age_pclass[2] which is 'Pclass' to variable 'Pclass'
    Pclass = age_pclass[1]
    
    #applying condition based on the Age and filling the missing data respectively 
    if pd.isnull(Age):

if Pclass == 1:
            return 38

elif Pclass == 2:
            return 30

else:
            return 25

else:
        return Age

By using the above function we can fill the missing data in Age feature according to the mean age with respect to each Pclass.

# (for train) grab age and apply the impute_age, our custom function
train['Age'] = train[['Age','Pclass']].apply(impute_age,axis=1)

# (for test) grab age and apply the impute_age, our custom function 
test['Age'] = test[['Age','Pclass']].apply(impute_age,axis=1)

4. Cabin feature in Train and Test

Now we reach half of the way, the last step is the Cabin column we need to apply some feature engineering.

• If there was a value for Cabin — replace with 1
• If the value is missing/null — replace with 0

#Train:
train.loc[train['Cabin'].notnull(), 'Cabin'] =1
train['Cabin'] = train['Cabin'].replace({np.nan:0})
train['Cabin'] = train['Cabin'].astype(int)

#Test:
test.loc[test['Cabin'].notnull(), 'Cabin'] =1
test['Cabin'] = test['Cabin'].replace({np.nan:0})
test['Cabin'] = test['Cabin'].astype(int)

Now let’s take a look to the heatmap again:

Look’s Great!! No more missing values :)

Feature Engineering

To include the categorical variables as predictors in statistical and machine learning models we need to Dummy the variables.

Dummy the Sex and Embarked columns

# Train
sex = pd.get_dummies(train['Sex'],drop_first=True)
embark = pd.get_dummies(train['Embarked'],drop_first=True)
train = pd.concat([train, sex,embark],axis=1)
train=train.drop(['Sex','Embarked'], axis=1)
train.rename(columns={"male": "sex_male", "Q": "Embarked_Q","S": "Embarked_S"}, inplace=True)
# Same for the Test

Model Preparation

1. First we need to select our features and it will be the following:

[Pclass, Age, SibSp, Parch, Fare, Cabin, Sex_male, Embarked_Q, Embarked_S]

And our target will be the feature: Survived

2. Then we need to write a list comprehension to grab the selected features.

3. Separate the selected_column in X_train and the Survived in y_train.

features_drop = ['PassengerId','Name', 'Ticket', 'Survived']
selected_features = [x for x in train.columns  if x not in features_drop ]
X_train = train[selected_features]
y_train = train['Survived']

Now the data is ready to train on the models.

Problem Statment

Traffic accidents was the main reason of health hazard in Saudi Arabia [1]. Traffic authorities have been working hard on reducing the number of road accidents in Saudi Arabia by forcing new rules [2]. To investigate the problem and solutions this analytics study is conducted to compare between the regions, and to find which has the highest number and causes against the lowest to improve the society and save lives.

Executive Summary

Traffic accidents in Saudi Arabia fall 30% a year [3]. Makkah and Riyadh has the largest numbers in accidents with a dramatic different comparing with the other regions. The new punishments for traffic violations, traffic safety equipment affect in decreasing these numbers [3].

Datasets

• Driving Licenses.
  This dataset contains Saudi Arabia Driving Licenses Issued By Administrative Area for 1993–2016, and 1993–2019. Data from General Authority for Statistics . Follow datasource.kapsarc.org for timely data to advance energy economics research [4].
• Traffic Accidents and Casualties.
  This dataset contains Saudi Arabia Traffic Accidents and Casualties by Region for 2016. Data from General Authority for Statistics. Follow datasource.kapsarc.org for timely data to advance energy economics research [5].

Preprocessing

After exploring the data, we can observe missing data, and unnecessary features to my analytics. In addition, for the Traffic Accidents and Casualties dataset, the dataset needs to be transform to wide to get the most advantage to analysing the casualities. Furthermore, to enhance the merging process the features names need to match.

Statistics and Investigating the trends

• The highest number of issued driving licenses between 1993–2017 was in Riyadh in 2017 with 495307 licenses.

• The lowest number of issued driving licenses between 1993–2017 was in Tabouk in 2015 with 915 licenses.

• The highest mean of traffic accidents between 2016–2017 was in Makkah in 2017 with 145541 accidents.

• The lowest mean of traffic accidents between 2016–2017 was in Al-Baha in 2017 with 1414 accidents.

Visualization and Key Findings

From the above plots we can see that even with the high increasing in the issued driving licenses, there is a high decrease in the number of indicators in the year 2016–2017.

In addition, the high number of issued driving licenses in Riyadh in 2017 consider very high comparing with the other regions and the decreasing in number of accidents is obviously high.

To investigate more about this, I upload an updated dataset that include 2018, 2019.

The number of accidents continue to decrease with the increasing in the driving licences numbers. After allowing the woman to drive in Saudi Arabia the new rules and regulation that forced by the traffic authority help to decrease the accidents numbers.

After analysing the year relations, the next step is to find the highest and lowest regions.

As observed in investigating the trends section, the regions with the high number of accidents are Makkah, Riyadh, and the lowest is Al-Baha.

Conclusions and Recommendations

As we see there is an increasing in driving licences over the years wherever there is a decreasing in traffic accidents. The main causes for decreasing traffic accidents to fall 30% a year are tough new punishments for traffic violations, including fines and lengthy prison sentences for causing injury or death, and a crackdown on mobile phone usage. Furthermore, Saudi Arabia has also introduced traffic safety equipment including high tensile fencing, protective barriers and rotating shock absorbers to reduce injuries [3].

Key takeaways

• From the above analysis the highest number of accidents for 2016–2017 was fro regions Makkah, Riyadh.
• In addition there is a dramatic difference for the issued driving licenses for Makkah, and Riyadh comparing with other regions.
• Makkah has the highest number in traffic accidents with 145541 accidents where lower region is Al-Baha with 1212 accidents.
• Riyadh issued driving licenses was the highest with 495307 licenses in 2017, on the other hand the lowest was in 2015 in Tabouk region.
• The high population of capital Riyadh and Hajj and Umrah seasons in Makkah might consider as main reasons for high number of accidents in these regions.
• The new rules forced by the traffic authority after allowing woman to drive help to decrease the accident numbers.

Recommendation

• Develop driving training program methods, and manage or motivate drivers through supervisory such as, coaching and leadership. Also, the appropriate driving training space with road challenges would help to think and act before accidents happens.
• Rules and punishments from the government for the negligent drivers who break any type of rules.
• Spread the awareness and statistics of number of accidents using the social media to reach to all the society.

References

[1] PhD MRCGPN.J.Shanks, Road traffic accidents in Saudi Arabia.

[2] Lovin Saudi. 2021. The Main Cause Of Road Accidents In Saudi Arabia Has Been Revealed And It Is Hardly A Shocker. [online] Available at: <https://lovin.co/saudi/en/news/main-cause-of-road-accidents-in-saudi/> [Accessed 2 January 2021].

[3] Anderson, R., 2021. Saudi Road Deaths Fall 33% In A Year. [online] Gulf Business. Available at: <https://gulfbusiness.com/saudi-road-deaths-fall-33-year/> [Accessed 2 January 2021].

[4] Datasource.kapsarc.org. 2021. Driving Licenses Issued By Region. [online] Available at: <https://datasource.kapsarc.org/explore/dataset/saudi-arabia-driving-licenses-issued-in-the-kingdom-2004-2008/information/?disjunctive.administritive_area&sort=time_period&location=5,24.37495,45.08024&basemap=jawg.streets> [Accessed 2 January 2021].

[5] Datasource.kapsarc.org. 2021. Traffic Accidents And Casualties By Region. [online] Available at: <https://datasource.kapsarc.org/explore/dataset/saudi-arabia-traffic-accidents-and-casualties-injured-dead-2008/export/?disjunctive.region&disjunctive.indicator&sort=time_period> [Accessed 2 January 2021].