This story is the capstone project of Udacity Data Visualization Nanodegree. Its main goal is to describe how I’ve improved an existing data visualization from Makeover Monday by combining what I’ve learned throughout the program.

Step 1: Scope the Project and Gather Data

Consequently, the first task was to choose an existing dataset. As I was always very interested in economic fields, so I checked datasets used for MakeoverMonday data then I chosen the Personal Consumption Expenditures on Food and beverages purchased for off-premises consumption.

The original data visualization is here; the dataset with all background information and data dictionary is here.

Step 2: Explore and Assess the Data

The main variable/categories in the original dataset are:

• Alcoholic beverages
• Cereals and bakery products
• Fats and oils
• Coffee, tea, and other beverage materials
• Fish and seafood
• Fresh fruits and vegetables
• Meats and poultry
• Sugar and sweets
• Milk, dairy products, and eggs
• Mineral waters, soft drinks, and vegetable juices
• Processed fruits and vegetables

Under these main categories, there are many sub-categories:

• Beer
• Spirits
• Wine
• Bakery products
• Cereals
• Fats and oils
• Fish and seafood
• Fruit (fresh)
• Vegetables (fresh)
• Beef and veal
• Other meats
• Pork
• Poultry
• Eggs
• Fresh milk
• Processed dairy products
• Processed fruits and vegetables
• Sugar and sweets
• Coffee, tea, and other beverage materials
• Mineral waters, soft drinks, and vegetable juices

So we have 14901 combinations of all these parameters. Each of them is the dataset record that can be interpreted as follows: In January 1959 there are 3457 million dollars consumed on beer which is part of Alcoholic beverages.

The main goal of this step is to identify any limitations and biases in data collection, processing, and insights.

What is well with the initial visualization?

1. Data source is mentioned
2. The title is simple and clear so that everyone can understand

What could be improved with the initial visualization?

1. Compare the cereal with other categories.
2. Expenditures per category over years.
3. Expenditures per category in covid 19 period.

Pre-processing done on household income dataset:

1. Downloaded datset (Excel file) here.
2. Created Tableau file
3. create many sheets as follow:

a) Growth Rate to compare the cereal Vs other categories here

b) Growth Rate to compare the cereal Vs other categories by shown that the difference in millions of dolras. here

c) Pie char illustrate the expenditures per category in covid 19 period. here

d) dashboard the combine the charts. here

Thank you for reading.

Problem Identification

Developing a recommender system for IPTV in Jawwy service provided by STC. Python and JupyterLab are tools with Sklearn libraries for implementing the proposed recommender system. The aim is to create a business value for a local company to improve their service, and STC was selected for this project with their service Jawwy since they shared its data in the open data initiative.

In this blog, we will build a recommendation model by using the Surprise method

Surprise is a Python scikit for building and analyzing recommender systems that deal with explicit rating data.

Surprise was designed with the following purposes in mind:

• Give users perfect control over their experiments
• Provide various ready-to-use prediction algorithms such as baseline algorithms, neighborhood methods
• Make it easy to implement new algorithm ideas.
• Provide tools to evaluate, analyze and compare the algorithms’ performance. Cross-validation procedures can be run very easily using powerful CV iterators (inspired by sci-kit-learn excellent tools.

We’ll be working with the jawwy IPTV dataset to develop recommendation system algorithms, with the Surprise library, which was built by Nicolas Hug. Let’s dive in!

Dataset

Saudi operator STC launched an interactive IPTV service called InVision for its DSL customers to coincide with the start of Ramadan. The service is initially available in the three major cities Riyadh, Jeddah, and Dammam free of charge throughout the month of Ramadan. InVision provides free TV channels and pay-TV packages, time-shifting of programs, program recording, a video-on-demand library of religious and documentary programs, and a catch-up TV service to watch programs broadcast over the last seven days. In addition, users have access to a control panel to select programs and movies.

We admire STC's role in order to enable the data community, when STC launched Open Data initiative which can help in preparing a technical-aware Saudi generation in line with Vision 2030, by supporting researchers interested in Data Science, through enabling them to explore and process data through application and practice.

The data consist of 29,487 user behavior within 8k shows all in 3m row. we extract the juice out of all the behavioral patterns of not only the users but also from the movies themselves.

Modeling

In previous posts, we discuss two of the most popular ways to approach recommender systems are collaborative filtering and content-based recommendations. In this blog, we will focus on the collaborative filtering approach, that is: the user is recommended items that people with similar tastes and preferences liked in the past. In another word, this method predicts unknown ratings by using the similarities between users.

Surprise Model

Surprise provides a collection of estimators for rating prediction, classical algorithms are implemented such as the main similarity-based algorithms, as well as algorithms based on matrix factorization like SVD or NMF. It also supports tools for model evaluation like cross-validation iterators and built-in metrics.

Since surprise does not support implicit ratings, and our dataset hasn’t this important data as shown below:

So we did some feature engineering to obtain the rating of watching for each user/movie, the idea is to consider the watching ratio as a rating.

The problem is the difference between movie and series in this dataset, the total movie duration is as the following:

While the series is like the following:

for movie programs, it is easy to obtain the ratio of watching and deal with it as a user rating, by using regex remove (min) word that suffix the number of minutes in duration feature which is the total duration for each movie from IMDB, then transform the time from minutes to seconds, after that divide the duration by the duration_second which is the duration of the user watching for this particular movie.

Unfortunately, we cannot apply these steps for series/ tv shows because the duration is for the season rather than the total duration of the episode of the series, and when we try to obtain this data from an external dataset, we face other problems which are some series/ tv shows will be lost, also there are many different movies have the same name like little women and little women.

So we split the dataset into 2 dataframe, the first one merged_df_series for TV Show and the second one merged_df_movie for Movie programs

for for TV Show programs to assign the total duration, we use .max() that returns the maximum of the values over the duration_seconds by users for each program/Tv Show as a total duration, since we don't have the total duration the dataset

df_series_duration=merged_df_series.groupby(‘program_name’)[[‘duration_seconds’]].max()

Add average_watching and total_duration, we will use it as a target for the surprise model.

#average_watching
merged_df_series[“average_watching”] = merged_df_series.apply(lambda x: 1 if x[‘duration_seconds’] > df_series_duration.loc[x.program_name,’duration_seconds’] else x[‘duration_seconds’]/df_series_duration.loc[x.program_name,’duration_seconds’], axis=1)

# total_duration 
merged_df_series[“total_duration”]= merged_df_series.apply(lambda x: df_series_duration.loc[x.program_name,’duration_seconds’],axis=1)

The head of the series dataset:

Remember that the average watching represents the ratio of watching for each user for a particular program.

For Movie programs:

# Extract minute by using (regex) and convert to appropriate type 
merged_df_movie[‘total_duration’] = merged_df_movie[‘duration’].str.replace(r’min’, ‘’)
merged_df_movie[‘duration_seconds’] = pd.to_numeric((merged_df_movie[‘duration_seconds’]) , errors=’coerce’).astype(‘Int64’)
merged_df_movie[‘total_duration’] = pd.to_numeric((merged_df_movie[‘total_duration’]) , errors=’coerce’).astype(‘Int64’)

# convert from min to sec
merged_df_movie[‘total_duration’]=(merged_df_movie[‘total_duration’]*60)

IF the value of duration_seconds > total_duration, assign the value of total_duration to duration_seconds:

merged_df_movie[“duration_seconds”] = merged_df_movie.apply(lambda x: x[‘total_duration’] if x[‘duration_seconds’] > x[‘total_duration’] else x[‘duration_seconds’], axis=1)

Add average_watching Feature, we will use it as a target for the surprise model.

merged_df_movie[‘average_watching’]=merged_df_movie[‘duration_seconds’]/merged_df_movie[‘total_duration’]

The head of the movies dataset:

Concatenate Dataframes

• concatenate merged_df_movie and merged_df_series and check outliers.

The dataset looks like this:

Some EDA:

Most of the programs in the data watched less than 3 times, and very few programs have many watching, although the most-watched program has been watched 2312 times

Most of the users in the data had less than 10 watched programs, and not many users had many watchings, although the most-active user has been watched 136 programs.

reader = Reader(rating_scale=(0.03, 1.0))
data = Dataset.load_from_df(df_data[[‘user_id’, ‘show_id’, ‘average_watching’]], reader)

benchmark = []
# Iterate over all algorithms
for algorithm in [SVD(), SVDpp(), SlopeOne(), NMF(), NormalPredictor(), KNNBaseline(), KNNBasic(),
 KNNWithMeans(), KNNWithZScore(), BaselineOnly()]:
 # Perform cross validation
 results = cross_validate(algorithm, data, measures=[‘RMSE’], cv=3, verbose=False)
 
 # Get results & append algorithm name
 tmp = pd.DataFrame.from_dict(results).mean(axis=0)
 tmp = tmp.append(pd.Series([str(algorithm).split(‘ ‘)[0].split(‘.’)[-1]], index=[‘Algorithm’]))
 benchmark.append(tmp)

Train and Predict

BaselineOnly algorithm gave us the best rmse, therefore, we will train and predict with BaselineOnly and use Alternating Least Squares (ALS).

print(‘Using ALS’)
bsl_options = {‘method’: ‘als’,
 “random_state”:250,
 ‘n_epochs’: 5,
 ‘reg_u’: 12,
 ‘reg_i’: 5
 }
algo = BaselineOnly(bsl_options)
cross_validate(algo, data, measures=[‘RMSE’], cv=3, verbose=False)

Using ALS
Estimating biases using als...
Estimating biases using als...
Estimating biases using als...

Out:

{'test_rmse': array([0.32846883, 0.32829109, 0.32849049]),
 'fit_time': (0.05880260467529297, 0.04584360122680664, 0.06278657913208008),
 'test_time': (0.1285717487335205, 0.14949917793273926, 0.12757420539855957)}

We use the train_test_split() to sample a trainset and a test set with given sizes, and use the accuracy metric of rmse. We’ll then use the fit() the method which will train the algorithm on the trainset, and the test() the method which will return the predictions made from the testset.

trainset, testset = train_test_split(data, test_size=0.25)
algo = BaselineOnly(bsl_options)
predictions = algo.fit(trainset).test(testset)
accuracy.rmse(predictions)

Out:

Estimating biases using als...
RMSE: 0.3295

To inspect our predictions in detail, we are going to build a pandas data frame with all the predictions.

User’s 13618 behavior

• Here we will study the watching behavior for a particular user his ID is (13618)

# user_id is the 13618
ratings = newdf.loc[newdf[‘user_id’] == 13618]
# obtain the required data of this user
ratings=ratings[[‘user_id’, ‘show_id’, ‘average_watching’]]
ratings

# get the list of the movie ids
unique_ids = newdf[‘show_id’].unique()

# get the list of the ids that the userid 13618 has watched
iids1001 = newdf.loc[newdf[‘user_id’]==13618, ‘show_id’]

# remove the rated movies for the recommendations
movies_to_predict = np.setdiff1d(unique_ids,iids1001)

Recommendation Programs For User’s 13618

algo = BaselineOnly(bsl_options)
algo.fit(data.build_full_trainset())

my_recs = []
for iid in movies_to_predict:
 my_recs.append((iid, algo.predict(uid=’13618',iid=iid).est))
pd.DataFrame(my_recs, columns=[‘iid’, ‘predictions’]).sort_values(‘predictions’, ascending=False).head(10)

That was it! I hope you enjoyed the recommendation (or rather, a rating prediction) journey with Surprise.

Reference:

• Surprise’ documentation
• https://actsusanli.medium.com/
• https://predictivehacks.com/how-to-run-recommender-systems-in-python/

In our daily life when we are shopping online, or looking for a movie to watch, we usually ask the recommendation from our friends for their personal opinion, and when they recommend something that we don’t like but they enjoyed it, what a waste of time right !!. So if there is a system that can understand you, and recommend for you based on your interests, that would be soo cool isn’t it. Well, that precisely what recommender systems are made for.

Problem Statement

Coming to the streaming shows websites, understanding the user’s behavior always consider a challenge. Irrespective of gender, age, or geographical location everyone enjoying watching movies at home. Bunch of people like genre-specific movies to be romance, action, or comedy while others enjoying the lead actors and directors’ visions. We all are connected via this wonderful medium. However, what most exciting is the fact that how distinctive our choices and combinations are in terms of the show’s preferences. But, when we take all that has been said, it’s remarkably difficult to generalize a movie and say that everyone would like it. So here the recommender systems shine to act as special assistants for the user’s needs.

Dataset

We found out that STC launched an Open Data initiative to support those who are interested in Data Science, the data is associated with Jawwy, which is their IPTV service. And we decided that this will be the beginning of the neon recommendation system journey. The data consist of 29,487 user behavior within 8k shows all in 3m row. we extract the juice out of all the behavioral patterns of not only the users but also from the movies themselves.

Modeling

We decided that we want to use 6 models to meet user preferences as much as we can those models are :

Collaborative filtering

This type of filter is based on users’ watching history, and it will recommend us movies that we haven’t watched yet, but users similar to us have and like. To determine whether two users are similar or not, this filter considers the movies both of them watched. By looking at the movies in common, this type of algorithm will basically recommend the movie for a user who hasn’t watched it yet, based on the similar users’ watching history.

Pros and Cons of collaborative filtering technique

Pros:

1. Collaborative filtering systems work by people in the system, and it is expected that people to be better at evaluating information than a computed function.

Cons:

1. Cold Start: a major challenge of the Collaborative Filtering technique can be how to make recommendations for a new user who has recently entered the system; that is called the cold-start user problem.
2. Sparsity:

• The user/rating matrix is sparse.
• Hard to find users that have rated the same items.

3. First rater:

• Cannot recommend an item that has not been previously rated.
• New items, Esoteric items.

4. Popularity bias:

• Cannot recommend items to someone with a unique taste.
• Tends to recommend popular items.

Content-based filters

Based on what we like, the algorithm will simply pick a movie with similar content(story description )to recommend us. This type of filter does not involve other users.

Pros and Cons of content-based technique

Pros :

1. No cold-start problem, unlike Collaborative Filtering, if the programs have sufficient descriptions, we avoid the “new item problem”.
2. Able to recommend to users with unique tastes

Cons :

1. Content-Based tend to over-specialization: they will recommend items similar to those already consumed, with a tendency of creating a “filter bubble”.
2. Never recommends items outside the user’s content profile, people might have multiple interests.

Hybrid filtering :

• Overcomes previous cons.
• Create a weighted recommender (weights are chosen equally, combining the results of predict_cf and predict_cn).
• Create differently weighted recommender (weights are chosen equally, combining the results of predict_cf, predict_cn, and predict_popularity)
• Create recommender based on popularity with weighted CL and CB (which CL and CB predictions affected by popularity prediction)
• Keep the strengths of CL, CB, and popularity models.
• Overcomes CL, CB, and popularity models cons.

indices = pd.Series(df_program_desc.index)
# defining the function that takes in movie title 
# as input and returns the top 10 recommended movies

def recommendations(title,type_of_recommendation, cosine_sim = cosine_sim, cosine_sim_w = cosine_sim_w, prec_watch_mat = prec_watch_mat):
 ‘’’
 type_of_recommendation values:
 
 0: the similarity scores of program description (Content Based),
 1: the similarity scores of watch history (Collaborative filtering),
 2: the similarity scores between watch history and program description (Hybrid),
 3: the similarity scores between watch history and program description 
 with popularity of program as indepent variables (Hybrid),
 4: the similarity scores between watch history and program description 
 as depent variables on popularity of program (Hybrid).
 ‘’’
 
 
 # initializing the empty list of recommended movies
 recommended_movies = []
 
 # gettin the index of the movie that matches the title
 idx = indices[indices == title].index[0]
 
 if type_of_recommendation == 0:
 
 # creating a Series with the similarity scores of program description in descending order
 score_series = pd.Series(cosine_sim[idx]).sort_values(ascending = False)
 
 elif type_of_recommendation == 1:
 
 # creating a Series with the similarity scores of watch history in descending order
 score_series = pd.Series(cosine_sim_w[idx]).sort_values(ascending = False)
 
 elif type_of_recommendation == 2:
 
 # creating a Series with the similarity scores between watch history and program description in descending order
 score_series = pd.Series(cosine_sim_w[idx]*0.5 + cosine_sim[idx]*0.5).sort_values(ascending = False)
 elif type_of_recommendation == 3:
 
 # creating a Series with the similarity scores between watch history and program description 
 # with popularity of program as indepent variables in descending order
 score_series = pd.Series(cosine_sim_w[idx]*0.33 + cosine_sim[idx]*0.33 + prec_watch_mat*0.34).sort_values(ascending = False)
 elif type_of_recommendation == 4:
 
 # creating a Series with the similarity scores between watch history and program description 
 # as depent variables on popularity of program in descending order
 score_series = pd.Series((cosine_sim_w[idx]*0.5 + cosine_sim[idx]*0.5) * prec_watch_mat).sort_values(ascending = False)
 
 else:
 print(‘You have entered wrong value’)
 return
# getting the indexes of the 10 most similar movies
 top_10_indexes = list(score_series.iloc[1:11].index)
 
 # populating the list with the titles of the best 10 matching movies
 for i in top_10_indexes:
 recommended_movies.append(list(df_program_desc.index)[i])
 
 return recommended_movies

How does Cosine Similarity work?

All the filters above will be measured by Cosine similarity which is a metric used to measure how similar the documents are irrespective of their size. Mathematically, it measures the cosine of the angle between two vectors projected in a multi-dimensional space. The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance (due to the size of the document), chances are they may still be oriented closer together. The smaller the angle, the higher the cosine similarity.

# instantiating and generating the count matrix
count = CountVectorizer()
count_matrix_desc = count.fit_transform(df_program_desc[‘cleaned_desc’])

# generating the cosine similarity matrix on program description
cosine_sim = cosine_similarity(count_matrix_desc, count_matrix_desc)

# generating the cosine similarity matrix on watch history
cosine_sim_w = cosine_similarity(watch_crosstab_transpose.values, watch_crosstab_transpose.values)

Also, we use Surprise library that I am going to write an article in a near future.

Demo

Result

we used the Hit-Rate as an evaluation metric for the cosine matrix algorithms

The Hit-Rate Generate the top n recommendation for a user and compare them to those the user watched. If they match then increase the hit rate by 1, do this for the complete dataset to get the hit rate. Since we want to recommend movies that new to the user so the closest to 0 the better. We sum the number of hits for each movie in our top-N list and divide by the total number of movies.

Since we want to recommend movies that new to the user so the closest to 0 the better. The content-based model represents a good score, while the collaborative not that good. The hybrid model keeps the strength of the previous models and got a good score. the hybrid and popularity models are biased to user preferences and represent a not bad score as well.

Thank you for reading.

This article goes through the process of cleaning a dataset which consists of information about the location of the house, price, and other aspects such as square feet, etc. When we work on these sorts of data, we need to modify and aggregate some features to expand their importance and see which column is important for us and which is not. The main goal of this article is to prepare data to do modeling steps that can give us a good prediction on the price of the house based on the other variables.

Data Dictionary

• There are 81 features in this dataset,
• This is the data dictionary of the dataset before cleaning steps:

Check nulls:

df.isnull().sum()

Handling Missing Values

Divide the data into numerical (“quan”), and categorical (“qual”) features.

quan = list( test.loc[:,test.dtypes != ‘object’].drop(‘Id’,axis=1).columns.values )
qual = list( test.loc[:,test.dtypes == ‘object’].columns.values )

Find out how many missing values there are for the quantitative and categorical features

print('Quantitative\n')
hasNAN = df[quan].isnull().sum()
hasNAN = hasNAN[hasNAN > 0]
hasNAN = hasNAN.sort_values(ascending=False)
print(hasNAN)
print ('-' * 100)
print('Qualitative\n')
hasNAN = df[qual].isnull().sum()
hasNAN = hasNAN[hasNAN > 0]
hasNAN = hasNAN.sort_values(ascending=False)
print(hasNAN)

The output is shown here:

Quantitative

LotFrontage     486
GarageYrBlt     159
MasVnrArea       23
BsmtHalfBath      2
BsmtFullBath      2
GarageArea        1
GarageCars        1
TotalBsmtSF       1
BsmtUnfSF         1
BsmtFinSF2        1
BsmtFinSF1        1
dtype: int64
----------------------------------------------------------------------------------------------------
Qualitative

PoolQC          2909
MiscFeature     2814
Alley           2721
Fence           2348
FireplaceQu     1420
GarageCond       159
GarageQual       159
GarageFinish     159
GarageType       157
BsmtExposure      82
BsmtCond          82
BsmtQual          81
BsmtFinType2      80
BsmtFinType1      79
MasVnrType        24
MSZoning           4
Functional         2
Utilities          2
Exterior1st        1
SaleType           1
Exterior2nd        1
KitchenQual        1
Electrical         1
dtype: int64

Rather than the statistical computing methods to fill the missing values like mean, median, or mode, we prefer to do it manually, the benefit of this procedure is to get deep insight and understanding about the data, adding that make enhancements to some features in the feature engineering stage.

Remember the meaning of these features:

• LotFrontage: Linear feet of street-connected to property
• LotShape: General shape of a property
• Neighborhood: Physical locations within Ames city limits

print(df[[‘LotShape’, ‘Neighborhood’,’LotFrontage’]].loc[df[‘LotFrontage’].isnull()])

The output:

LotShape Neighborhood  LotFrontage
7         IR1       NWAmes          NaN
12        IR2       Sawyer          NaN
14        IR1        NAmes          NaN
16        IR1        NAmes          NaN
24        IR1       Sawyer          NaN
...       ...          ...          ...
2847      Reg      CollgCr          NaN
2850      IR2      CollgCr          NaN
2900      IR1       Timber          NaN
2901      IR1       Timber          NaN
2908      IR1      Mitchel          NaN

[486 rows x 3 columns]

Filling missing values for numerical features.

Most of the NAN should mean that The corresponding facility/structure doesn’t exist, so we use zero for most cases

df.LotFrontage.fillna(df.LotFrontage.median(), inplace=True)
df[“LotFrontage”] = df[“LotFrontage”].fillna(df.groupby([“Neighborhood”, “LotShape”])[“LotFrontage”].transform(“median”))

NAN should mean no garage. we temporarily use yr = 0 here. Will come back to this later.

df.GarageYrBlt.fillna(0, inplace=True)

Use zero to fill following nulls

df.MasVnrArea.fillna(0, inplace=True) 
df.BsmtHalfBath.fillna(0, inplace=True)
df.BsmtFullBath.fillna(0, inplace=True)
df.GarageArea.fillna(0, inplace=True)
df.GarageCars.fillna(0, inplace=True) 
df.TotalBsmtSF.fillna(0, inplace=True) 
df.BsmtUnfSF.fillna(0, inplace=True) 
df.BsmtFinSF2.fillna(0, inplace=True) 
df.BsmtFinSF1.fillna(0, inplace=True)
df.Electrical.fillna(df.Electrical.mode()[0], inplace=True)

Filling missing values for categorical features with not available (NA)

df.PoolQC.fillna('NA', inplace=True)
df.MiscFeature.fillna('NA', inplace=True)    
df.Alley.fillna('NA', inplace=True)          
df.Fence.fillna('NA', inplace=True)         
df.FireplaceQu.fillna('NA', inplace=True)    
df.GarageCond.fillna('NA', inplace=True)    
df.GarageQual.fillna('NA', inplace=True)     
df.GarageFinish.fillna('NA', inplace=True)   
df.GarageType.fillna('NA', inplace=True)     
df.BsmtExposure.fillna('NA', inplace=True)     
df.BsmtCond.fillna('NA', inplace=True)        
df.BsmtQual.fillna('NA', inplace=True)        
df.BsmtFinType2.fillna('NA', inplace=True)     
df.BsmtFinType1.fillna('NA', inplace=True)     
df.MasVnrType.fillna('None', inplace=True)   
df.Exterior2nd.fillna('None', inplace=True)

df.KitchenQual.fillna('NA', inplace=True)        
df.Exterior1st.fillna('NA', inplace=True)     
df.SaleType.fillna('NA', inplace=True)
df.Functional.fillna('NA', inplace=True)
df.Utilities.fillna('NA', inplace=True)

MSZoning should highly correlate with the location, so I use the mode values of individual Neighborhoods

for i in df.Neighborhood.unique():
    if df.MSZoning[df.Neighborhood == i].isnull().sum() > 0:
        df.loc[df.Neighborhood == i,'MSZoning'] = \
        df.loc[df.Neighborhood == i,'MSZoning'].fillna(df.loc[df.Neighborhood == i,'MSZoning'].mode()[0])

Qualitative

fig = plt.subplots(nrows = 1, ncols = 1, figsize = (18, 6))
sns.heatmap(df[qual].isnull(), yticklabels=False, cbar=False, cmap=’viridis’)

Check duplicate

# Eliminating the duplicates.
df.duplicated(subset=None, keep=’first’).sum()

0

Feature Engineering

If we use one-hot encoding we will get a huge number of features, to avoid that, we decide to combine features that have similar data meaning.

df.Heating.value_counts()

GasA     2874
GasW       27
Grav        9
Wall        6
OthW        2
Floor       1
Name: Heating, dtype: int64

So we will combine all values (Grav, Wall, OthW, and Floor 1) together since their values have appeared a few times.

df[“Heating”] = df[“Heating”].apply(lambda x: “Other” if x != “GasA” else x)

• MasVnrType

df.MasVnrType.value_counts()

None       1766
BrkFace     879
Stone       249
BrkCmn       25
Name: MasVnrType, dtype: int64

So we will combine all values (Stone, BrkCmn ) to None since their values have appeared a few times.

df[“MasVnrType”] = df[“MasVnrType”].apply(lambda x: “None” if x in [“None”, “BrkCmn”] else x)

And do the same steps for all categorical features.

Converting categorical variables into labels:

df.Alley = df.Alley.map({‘NA’:0, ‘Grvl’:1, ‘Pave’:2})
df.BsmtCond = df.BsmtCond.map({‘NA’:0, ‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.BsmtExposure = df.BsmtExposure.map({‘NA’:0, ‘No’:1, ‘Mn’:2, ‘Av’:3, ‘Gd’:4})
df[‘BsmtFinType1’] = df[‘BsmtFinType1’].map({‘NA’:0, ‘Unf’:1, ‘LwQ’:2, ‘Rec’:3, ‘BLQ’:4, ‘ALQ’:5, ‘GLQ’:6})
df[‘BsmtFinType2’] = df[‘BsmtFinType2’].map({‘NA’:0, ‘Unf’:1, ‘LwQ’:2, ‘Rec’:3, ‘BLQ’:4, ‘ALQ’:5, ‘GLQ’:6})
df.BsmtQual = df.BsmtQual.map({‘NA’:0, ‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.ExterCond = df.ExterCond.map({‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.ExterQual = df.ExterQual.map({‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.FireplaceQu = df.FireplaceQu.map({‘NA’:0, ‘TA’:1, ‘Gd’:2, ‘other’:3})
df.Functional = df.Functional.map({ ‘Min1’:2, ‘Typ’:1})
df.GarageCond = df.GarageCond.map({‘NA’:0, ‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.GarageQual = df.GarageQual.map({‘NA’:0, ‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.HeatingQC = df.HeatingQC.map({‘Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.KitchenQual = df.KitchenQual.map({‘NA’:0,’Po’:1, ‘Fa’:2, ‘TA’:3, ‘Gd’:4, ‘Ex’:5})
df.LandSlope = df.LandSlope.map({‘Sev’:1, ‘Mod’:2, ‘Gtl’:3}) 
df.PavedDrive = df.PavedDrive.map({‘N’:1, ‘P’:2, ‘Y’:3})
df.PoolQC = df.PoolQC.map({‘NA’:0,’Ex’:1})
df.Street = df.Street.map({‘Grvl’:1, ‘Pave’:2})
df.Utilities = df.Utilities.map({‘NA’:0,’ELO’:1, ‘NoSeWa’:2, ‘NoSewr’:3, ‘AllPub’:4})

Update the lists of numerical and categorical features:

newquan = [‘Alley’,’BsmtCond’,’BsmtExposure’,’BsmtFinType1',’BsmtFinType2',’BsmtQual’,
 ‘ExterCond’,’ExterQual’,’FireplaceQu’,’Functional’,’GarageCond’,
 ‘GarageQual’,’HeatingQC’,’KitchenQual’,’LandSlope’,’PavedDrive’,’PoolQC’,
 ‘Street’,’Utilities’]
quan = quan + newquan 
for i in newquan: qual.remove(i)

This is actually a categorical feature, MSSubClass doesn’t mean the meaning of the value of number 20, and so on with other values

df.MSSubClass = df.MSSubClass.map({20:’class1', 30:’class2', 40:’class3', 45:’class4',
 50:’class5', 60:’class6', 70:’class7', 75:’class8',
 80:’class9', 85:’class10', 90:’class11', 120:’class12',
 150:’class13', 160:’class14', 180:’class15', 190:’class16'})

Keeping “YrSold” is enough, the month MoSold: Month Sold (MM) and YrSold: Year Sold (YYYY) is like a redundant feature in somehow.

df=df.drop(‘MoSold’,axis=1)

Update my lists of numerical and categorical features

quan.remove(‘MoSold’)
quan.remove(‘MSSubClass’)
qual.append(‘MSSubClass’)

df[‘Age’] = df.YrSold — df.YearBuilt
df[‘AgeRemod’] = df.YrSold — df.YearRemodAdd
df[‘AgeGarage’] = df.YrSold — df.GarageYrBlt

For the houses without a Garage, I filled the NANs with zeros, which makes AgeGarage ~ 2000.
Here I replace their AgeGarage with the maximum value among the houses with Garages

max_AgeGarage = np.max(df.AgeGarage[df.AgeGarage < 1000])
df[‘AgeGarage’] = df[‘AgeGarage’].map(lambda x: max_AgeGarage if x > 1000 else x)

Some of the values are negative because the work was done after the house was sold. In these cases, I change them to zero to avoid negative ages.

df.Age = df.Age.map(lambda x: 0 if x < 0 else x)
df.AgeRemod = df.AgeRemod.map(lambda x: 0 if x < 0 else x)
df.AgeGarage = df.AgeGarage.map(lambda x: 0 if x < 0 else x)

drop the original time variables

df=df.drop([‘YrSold’,’YearBuilt’,’YearRemodAdd’,’GarageYrBlt’],axis=1)

Update the list of numerical features

for i in [‘YrSold’,’YearBuilt’,’YearRemodAdd’,’GarageYrBlt’]: quan.remove(i)
quan = quan + [‘Age’,’AgeRemod’,’AgeGarage’]

Detect the Outliers

These are somewhat arbitrary

# These are somewhat arbitrary 
index_drop = df.LotFrontage[df.LotFrontage > 300].index
index_drop = np.append(index_drop, df.LotArea[df.LotArea > 100000].index)
index_drop = np.append(index_drop, df.BsmtFinSF1[df.BsmtFinSF1 > 4000].index)
index_drop = np.append(index_drop, df.TotalBsmtSF[df.TotalBsmtSF > 6000].index)
index_drop = np.append(index_drop, df[‘1stFlrSF’][df[‘1stFlrSF’] > 4000].index)
index_drop = np.append(index_drop, df.GrLivArea[(df.GrLivArea > 4000) & (df.SalePrice < 13)].index)
index_drop = np.unique(index_drop)

Make sure we only remove data from the training set

index_drop = index_drop[index_drop < 1460]
df = df.drop(index_drop).reset_index(drop=True)
print(“{} examples in the training set are dropped.”.format(len(index_drop)))

The output:

7 examples in the training set are dropped.

Transform the numerical features that are skewed.

It is necessary to transform the numerical features that are skewed. This is because lots of regression models building assume that the features are distributed normally and have asymmetrical shapes. For most machine learning algorithms, especially linear models, normally distributed features gives us better results.

from scipy.stats import skew
for i in quan:
 print(i+’: {}’.format(round(skew(df[i]),2)))

The output:

LotFrontage: 0.76
LotArea: 3.6
OverallQual: 0.19
OverallCond: 0.57
MasVnrArea: 2.63
BsmtFinSF1: 0.98
BsmtFinSF2: 4.16
BsmtUnfSF: 0.92
TotalBsmtSF: 0.67
1stFlrSF: 1.26
2ndFlrSF: 0.86
LowQualFinSF: 12.07
GrLivArea: 1.07
BsmtFullBath: 0.62
BsmtHalfBath: 3.94
FullBath: 0.17
HalfBath: 0.69
BedroomAbvGr: 0.33
KitchenAbvGr: 4.3
TotRmsAbvGrd: 0.75
Fireplaces: 0.73
GarageCars: -0.22
GarageArea: 0.22
WoodDeckSF: 1.84
OpenPorchSF: 2.53
EnclosedPorch: 4.0
3SsnPorch: 11.36
ScreenPorch: 3.94
PoolArea: 17.67
MiscVal: 21.94
Alley: 4.13
BsmtCond: -3.6
BsmtExposure: 1.12
BsmtFinType1: -0.15
BsmtFinType2: 3.16
BsmtQual: -1.27
ExterCond: 1.31
ExterQual: 0.78
FireplaceQu: 0.61
Functional: 3.39
GarageCond: -3.38
GarageQual: -3.26
HeatingQC: -0.55
KitchenQual: 0.39
LandSlope: -4.98
PavedDrive: -2.97
PoolQC: 26.93
Street: -16.18
Utilities: -33.93
Age: 0.6
AgeRemod: 0.45
AgeGarage: 0.97

Transform those with skewness > 0.5

skewed_features = np.array(quan)[np.abs(skew(df[quan])) > 0.5]
df[skewed_features] = np.log1p(df[skewed_features])

Dummy variables for the categorical features

Create a list of dummy variables that I will drop, which will be the last column generated from each categorical feature

dummy_drop = []
for i in qual:
 dummy_drop += [ i+’_’+str(df[i].unique()[-1]) ]

Create dummy variables

df = pd.get_dummies(df,columns=qual) 

Drop the last column generated from each categorical feature

df = df.drop(dummy_drop,axis=1)

Check nulls values

fig, ax = plt.subplots( figsize = (18, 6))
sns.heatmap(df.isnull(), yticklabels=False, cbar=False, cmap=’viridis’)

• Half of the SalePrice is null which is the test set before merging datasets.

Split the Data

X_train = df[:-1459].drop([‘SalePrice’,’Id’], axis=1)
y_train = df[:-1459][‘SalePrice’]
X_test = df[-1459:].drop([‘SalePrice’,’Id’], axis=1)

Scaling

Fit the training set only, then transform both the training and test sets.

scaler = RobustScaler()
X_train[quan]= scaler.fit_transform(X_train[quan])
X_test[quan]= scaler.transform(X_test[quan])
X_train.shape, X_test.shape

The output:

((1453, 193), (1459, 193))

Now we have 193 features!

By Jason Brownlee on August 17, 2016, in XGBoost. Last Updated on April 22, 2020. XGBoost is an algorithm that has recently been dominating applied machine learning and Kaggle competitions for structured or tabular data. XGBoost is an implementation of gradient boosted decision trees designed for speed and performance.

XGBoost is a scalable and accurate implementation of gradient boosting machines and it has proven to push the limits of computing power for boosted trees algorithms as it was built and developed for the sole purpose of model performance and computational speed.

Feature importance is defined only for tree boosters

!pip install xgboost
from xgboost import XGBRegressor
xgb = XGBRegressor()
xgb.fit(X_train, y_train)
imp = pd.DataFrame(xgb.feature_importances_ ,columns = ['Importance'],index = X_train.columns)
imp = imp.sort_values(['Importance'], ascending = False)

print(imp)

The output:

Requirement already satisfied: xgboost in c:\users\maemt\anaconda3\lib\site-packages (1.3.1)
Requirement already satisfied: numpy in c:\users\maemt\anaconda3\lib\site-packages (from xgboost) (1.19.2)
Requirement already satisfied: scipy in c:\users\maemt\anaconda3\lib\site-packages (from xgboost) (1.5.2)
                   Importance
GarageCond           0.298127
GarageCars           0.157924
OverallQual          0.079050
CentralAir_Y         0.052237
KitchenQual          0.037871
...                       ...
Fence_GdPrv          0.000000
Condition2_RRAn      0.000000
Condition2_PosN      0.000000
Condition2_PosA      0.000000
Exterior2nd_Stone    0.000000

[193 rows x 1 columns]

The most 10 important features are:

Feature ranking with recursive feature elimination and cross-validated selection of the best number of features.

from sklearn.metrics import make_scorer 
from sklearn.feature_selection import RFECV

# Define a function to calculate RMSE
def rmse(y_true, y_pred):
 return np.sqrt(np.mean((y_true-y_pred)**2))

# Define a function to calculate negative RMSE (as a score)
def nrmse(y_true, y_pred):
 return -1.0*rmse(y_true, y_pred)
neg_rmse = make_scorer(nrmse)
estimator = XGBRegressor()
selector = RFECV(estimator, cv = 3, n_jobs = -1, scoring = neg_rmse)
selector = selector.fit(X_train, y_train)

print(“The number of selected features is: {}”.format(selector.n_features_))

features_kept = X_train.columns.values[selector.support_] 
X_train = X_train[features_kept]
X_test = X_test[features_kept]

The output is:

The number of selected features is: 42

So finally we can deal with cleaned 42 features rather than 81 features.

Thank you for reading.

With the largest online selection of leading brands in categories such as electronics, fashion, health & beauty, fragrances, grocery, baby products, and homeware, noon is the one-stop-shopping destination for everyone. In this article, I choose the noon website, data was collected from the perfume section for female, male, and kids departments.

This article talks about web scraping and regex to extract required data from this website.

What is Web Scraping?

If you wonder what web scraping is, “Web scraping is a technique of extracting information from websites. It focuses on the transformation of unstructured data on the web, into structured data that can be stored and analyzed”.

What are Regular Expressions?

A regular expression is a sequence of characters that define a search pattern.

Usually this pattern is then used by string searching algorithms for “find” or “find and replace” operations on strings

The Most Famous Quote in regex-dom

“Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems.” — Jamie Zawinski (Netscape engineer)

There are many places where regex can be run — from your text editor to the bash shell to Python, and even SQL. It is typically baked into the standard library of programming languages.

In Python, it can be imported like so:

import re

As a Data scientist, when to use Regular Expressions?

Regular Expressions help in manipulating textual data, which is often a pre-requisite for data science projects that involve text analytics.

The dataset

After using selenium and beautiful soup to create our dataset, it looks like that:

there are many data that could be extracted from the above features, lets see:

To extract ml from perfume name:

df['ml'] = df['name'].str.extract('(\d+ml)', expand=False)
df['name'] = df.apply(lambda row : row['name'].replace(str(row['ml']), ''), axis=1)

To extract brand from perfume name:

df['name'] = df.apply(lambda row : row['name'].replace(str(row['brand']), ''), axis=1)

To extract concentration from perfume name:

df['concentration'] = df['name'].str.extract('(.{3,5})$', expand=False)
df['name'] = df.name.str.replace(r'(.{3,5})$', '', regex=True)

To extract notes, scents, dispenser_type, andaromatherapy_type from the details feature:

df['Base Note']=df['details'].str.extract('((?<=Base Note).+?(?= &))' ,expand=False)
df['fragrance_notes']=df['details'].str.extract('((?<=Fragrance Notes).+?(?= &))' ,expand=False)
df['middle_note'] = df['details'].str.extract('((?<=Heart/Middle Note).+?(?= &))' ,expand=False)
df['department'] = df['details'].str.extract('((?<=Department).+?(?= &))' ,expand=False)
df['scents'] = df['details'].str.extract('((?<=Scents/Notes).+?(?= &))' ,expand=False)
df['dispenser_type'] = df['details'].str.extract('((?<=Dispenser Type).+?(?= &))' ,expand=False)
df['top_note'] = df['details'].str.extract('((?<=Top Note).+?(?= &))' ,expand=False)
df['aromatherapy_type'] = df['details'].str.extract('((?<=Aromatherapy Type).+?(?= &))' ,expand=False)

To extract the number of seller ratings, scents, dispenser_type, andaromatherapy_type from the details feature:

df['num_seller_ratings']= df.num_seller_ratings.str.replace(r'[()]', '', regex=True)#.str.strip()
df['seller_rating'] = df['num_seller_ratings'].str.extract('^(.{0,3})' ,expand=False)#  .str.replace
df['num_seller_ratings']= df.num_seller_ratings.str.replace('^(.{0,3})' ,'')

After using regex techniques:

noon_perfume Data Dictionary:

Here Data Dictionary after cleaning and feature engineering.

So from8 features we extracted 18 features, GREAT:)

In this blog, we will see how we can build a simple content-based recommender system using Goodreads.com data, computer books, particularly, this data obtained from Kaggle.

Usually, we plan to buy a book specially the scientific books, we normally ask about the goods ones, research the book domains, compare the books with similar or read the reviews so here the recommender system is the master of this problem.

Recommendation system algorithms:

Content-based recommendation system (CB)

Content-based recommendation systems recommend items to a user by using the similarity of items. This recommender system recommends products or items based on their description or features. It identifies the similarity between the products based on their descriptions. It also considers the user’s previous history in order to recommend a similar product.

Pros and Cons of content-based recommendation techniques

Pros :

1. No cold-start problem, unlike Collaborative Filtering, if the programs have sufficient descriptions, we avoid the “new item problem”.
2. Able to recommend to users with unique tastes
3. Able to recommend new & unpopular items — No first-rater problem.

Cons :

1. Content-Based tend to over-specialization: they will recommend items similar to those already consumed, with a tendency of creating a “filter bubble”.
2. Never recommends items outside the user’s content profile, people might have multiple interests.

Collaborative Filtering (CL)

This filtration strategy is based on the combination of the user’s behavior and comparing and contrasting that with other users’ behavior in the database. The history of all users plays an important role in this algorithm. The main difference between content-based filtering and collaborative filtering that in the latter, the interaction of all users with the items influences the recommendation algorithm while for content-based filtering only the concerned user’s data is taken into account.

Pros and Cons of collaborative filtering techniques

Pros:

1. Collaborative filtering systems work by people in the system, and it is expected that people to be better at evaluating information than a computed function.

Cons:

1. Cold Start: a major challenge of the Collaborative Filtering technique can be how to make recommendations for a new user who has recently entered the system; that is called the cold-start user problem.
2. Sparsity:

• The user/rating matrix is sparse.
• Hard to find users that have rated the same items.

3. First rater:

• Cannot recommend an item that has not been previously rated.
• New items, Esoteric items.

4. Popularity bias:

• Cannot recommend items to someone with a unique taste.
• Tends to recommend popular items.

As I mentioned above, we are using Amazon.com data and don’t have a user reading history. Hence, we have used a simple content-based recommendation system. We are going to build two recommendation systems by using a book title and book description.

We need to find similar books to a given book and then recommend those similar books to the user. How do we find whether the given book is similar or dissimilar? A similarity measure was used to find this.

The Data:

This data contains information about different Computer books:

The head of the table:

Check Nulls

EDA

Since the columnBook_Id is as an index, we will drop it and generate a new ID.

df.drop(['Book_Id'],axis=1,inplace=True)

• Check duplicates and drop if exists.

df.shape

(1234, 9)

df[df['book_Title']=='Anime and the Visual Novel: Narrative Structure Design and Play at the Crossroads of Animation and Computer Games']

drop_duplicates and check the shape

df.drop_duplicates(inplace=True)
df.shape

(1142, 9)

df[df['book_Title']=='Anime and the Visual Novel: Narrative Structure Design and Play at the Crossroads of Animation and Computer Games']

# Generate a new id 
df=df.assign(id=(df['book_Title']).astype('category').cat.codes)
df.head()

Visualization

Most frequent authors

plt.subplots(figsize=(10,7))
df.Author_Name.value_counts()[:10].plot(kind="bar")
plt.show()

Donald Ervin Knuth, Gray B.Shelly, and Sumita Arora are the most 3 frequent authors.

Calculating and plotting the word count for book_Title

# Calculating the word count for book description
df[‘book_Title’] = df[‘book_Title’].apply(lambda x: len(str(x).split()))# Plotting the word count
df[‘book_Title’].plot(
 kind=’hist’,
 bins = 50,
 figsize = (12,8),title=’Word Count Distribution for book_Title’)

• The majority of the book's title is between 5~10 words.
• The distribution of the book’s title is right-skewed.

Content-Based Recommender Systems

# test poinst for ratings_count=40, Avg_Rating= 4.5, Publish_year= 2002, Pages_no= 523
test_point = [40, 4.5, 2002, 523]
X = df.iloc[:, [2, 3, 4, 6]].values# build a nearest neighbor object, we are searching for just 3 neighbors so n_neighbors=3
nn = NearestNeighbors(n_neighbors=3).fit(X)
# kneighbors returns the neighbor for the test_point
print(nn.kneighbors([test_point]))

(array([[ 3.9028323 , 7.91060048, 12.2137791 ]]), array([[626, 445, 486]]))

# This is the most similar book from all the available books
df.iloc[626].to_frame()

df.iloc[445].to_frame()

From the above results, we can observe that the recommender books are similar to the test points that we fed manually to the model.

Done by: Lamalharbi, Aisha Y Hakami, and Monirah abdulaziz.

Introduction

This article goes through the whole process of cleaning the data and developing a machine learning model on the Titanic dataset which is one of the famous data sets which every Data Scientist explores at the start of their career and here we are. The dataset provides all the information on the fate of passengers on the Titanic, summarized according to sex, age, economic status (class), and survival.

In addition, you can join the Titanic: Machine Learning from Disaster challenge on Kaggle.

Data Cleaning

# upload the data
train = pd.read_csv('../datasets/train.csv')
test = pd.read_csv('../datasets/test.csv')

Check the null values:

fig, ax = plt.subplots(nrows = 1, ncols = 2, figsize = (18, 6))# train data 
sns.heatmap(train.isnull(), yticklabels=False, ax = ax[0], cbar=False, cmap='viridis')
ax[0].set_title('Train data')# test data
sns.heatmap(test.isnull(), yticklabels=False, ax = ax[1], cbar=False, cmap='viridis')
ax[1].set_title('Test data');

Observation:

• There are missing values in the train set: Age, Cabin and Embarked.
• There are missing values in the test set: Age, Fare and Cabin.

1. Embarked feature in Train

First, we need to check How many ports are in Embarked column:

train.Embarked.value_counts()

The result was:

S 644
C 168
Q 77

Well, now we can drop the two rows with missing Embarked values in the train set, however, let's fill the nulls with the port of highest embarkation.

from collections import Countertrain.Embarked= train.Embarked.replace(np.nan,Counter(train.Embarked).most_common(1)[0][0])

2. Fare feature in Test

To fill the missing values in Fare we can use the Pclass column, by computing the average of Fare of the missing Class.

class3_mean = train[train['Pclass']==3]['Fare'].mean()
test['Fare'] = test['Fare'].replace({np.nan:class3_mean})

3. Age feature in Train and Test

#defining a function 'impute_age'
def impute_age(age_pclass): # passing age_pclass as ['Age', 'Pclass']
    
    # Passing age_pclass[0] which is 'Age' to variable 'Age'
    Age = age_pclass[0]
    
    # Passing age_pclass[2] which is 'Pclass' to variable 'Pclass'
    Pclass = age_pclass[1]
    
    #applying condition based on the Age and filling the missing data respectively 
    if pd.isnull(Age):if Pclass == 1:
            return 38elif Pclass == 2:
            return 30else:
            return 25else:
        return Age

By using the above function we can fill the missing data in Age the feature according to the mean age with respect to each Pclass.

# (for train) grab age and apply the impute_age, our custom function
train['Age'] = train[['Age','Pclass']].apply(impute_age,axis=1)

# (for test) grab age and apply the impute_age, our custom function 
test['Age'] = test[['Age','Pclass']].apply(impute_age,axis=1)

4. Cabin feature in Train and Test

Now we reach half of the way, the last step is the Cabin , the column we need to apply some feature engineering.

• If there was a value for Cabin — replace it with 1
• If the value is missing/null — replace it with 0

#Train:
train.loc[train['Cabin'].notnull(), 'Cabin'] =1
train['Cabin'] = train['Cabin'].replace({np.nan:0})
train['Cabin'] = train['Cabin'].astype(int)

#Test:
test.loc[test['Cabin'].notnull(), 'Cabin'] =1
test['Cabin'] = test['Cabin'].replace({np.nan:0})
test['Cabin'] = test['Cabin'].astype(int)

Now let’s take a look at the heatmap again:

Look’s Great!! No more missing values :)

Feature Engineering

To include the categorical variables as predictors in statistical and machine learning models we need to Dummy the variables.

Dummy the Sex and Embarked columns

# Train
sex = pd.get_dummies(train['Sex'],drop_first=True)
embark = pd.get_dummies(train['Embarked'],drop_first=True)
train = pd.concat([train, sex,embark],axis=1)
train=train.drop(['Sex','Embarked'], axis=1)
train.rename(columns={"male": "sex_male", "Q": "Embarked_Q","S": "Embarked_S"}, inplace=True)
# Same for the Test

Model Preparation

1. First, we need to select our features and it will be the following:

[Pclass, Age, SibSp, Parch, Fare, Cabin, Sex_male, Embarked_Q, Embarked_S]

And our target will be the feature: Survived

2. Then we need to write a list comprehension to grab the selected features.

3. Separate the selected_column in X_train and the Survived in y_train.

features_drop = ['PassengerId','Name', 'Ticket', 'Survived']
selected_features = [x for x in train.columns  if x not in features_drop ]
X_train = train[selected_features]
y_train = train['Survived']

Now the data is ready to train on the models.

Problem Statement:

Build a Recommendation System for children's books from amazon, a dataset that scrapped by Modhi almannaa, see the link.

This dataset was scraped from Amazon, the largest selling platform, it provides all the information that helps the customers to find the best products, by reviewing all the information the customer needs to know before buying the product. This dataset focused only on children's books.

Recommendation system algorithms:

Content-based recommendation system (CB)

Content-based recommendation systems recommend items to a user by using the similarity of items. This recommender system recommends products or items based on their description or features. It identifies the similarity between the products based on their descriptions. It also considers the user’s previous history in order to recommend a similar product.

Pros and Cons of content-based recommendation techniques

Pros :

1. No cold-start problem, unlike Collaborative Filtering, if the programs have sufficient descriptions, we avoid the “new item problem”.
2. Able to recommend to users with unique tastes
3. Able to recommend new & unpopular items — No first-rater problem.

Cons :

1. Content-Based tend to over-specialization: they will recommend items similar to those already consumed, with a tendency of creating a “filter bubble”.
2. Never recommends items outside the user’s content profile, people might have multiple interests.

Collaborative Filtering (CL)

This filtration strategy is based on the combination of the user’s behavior and comparing and contrasting that with other users’ behavior in the database. The history of all users plays an important role in this algorithm. The main difference between content-based filtering and collaborative filtering that in the latter, the interaction of all users with the items influences the recommendation algorithm while for content-based filtering only the concerned user’s data is taken into account.

Pros and Cons of collaborative filtering techniques

Pros:

1. Collaborative filtering systems work by people in the system, and it is expected that people to be better at evaluating information than a computed function.

Cons:

1. Cold Start: a major challenge of the Collaborative Filtering technique can be how to make recommendations for a new user who has recently entered the system; that is called the cold-start user problem.
2. Sparsity:

• The user/rating matrix is sparse.
• Hard to find users that have rated the same items.

3. First rater:

• Cannot recommend an item that has not been previously rated.
• New items, Esoteric items.

4. Popularity bias:

• Cannot recommend items to someone with a unique taste.
• Tends to recommend popular items.

As I mentioned above, we are using Amazon.com data and don’t have a user reading history. Hence, we have used a simple content-based recommendation system. We are going to build two recommendation systems by using a book title and book description.

We need to find similar books to a given book and then recommend those similar books to the user. How do we find whether the given book is similar or dissimilar? A similarity measure was used to find this.

The Data

This data contains information about different children books:

The head of the table:

EDA and regex

The Description feature for the first book looks like this:

'\n#1\xa0NEW YORK TIMES\xa0BESTSELLER • A modern, sophisticated suspense novel from National Book Award finalist, and Printz Award honoree E. Lockhart.A beautiful and distinguished family.A private island.A brilliant, damaged girl; a passionate, political boy.A group of four friends—the Liars—whose friendship turns destructive.A revolution. An accident. A secret.Lies upon lies.True love.The truth.Read it.And if anyone asks you how it ends, just LIE."Thrilling, beautiful, and blisteringly smart, We Were Liars is utterly unforgettable." —John Green, #1 New York Times bestselling author of The Fault in Our Stars\n\n'

The Product_Details feature for the first book looks like this:

'\n\n\n\nProduct details\n\n\n\n\n\nPublisher\n:\n\nEmber; Illustrated edition (May 29, 2018)\n\n\nLanguage\n:\n\nEnglish\n\n\nPaperback\n:\n\n320 pages\n\n\nISBN-10\n:\n\n0385741278\n\n\nISBN-13\n:\n\n978-0385741279\n\n\nReading age\n:\n\n12 - 17 years\n\n\nLexile measure\n:\n\n600L\n\n\nGrade level\n:\n\n7 - 9\n\n\nItem Weight\n:\n\n12 ounces\n\n\nDimensions\n:\n\n5.49 x 0.84 x 8.24 inches\n\n\n\n\n\n\nBest Sellers Rank:\n\n#13 in Books (See Top 100 in Books)\n\n #1 in Teen & Young Adult Family Fiction\n #1 in Teen & Young Adult Fiction about Emotions & Feelings\n #1 in Teen & Young Adult Fiction about Death & Dying\n\n\n\n\n\n\nCustomer Reviews:\n\n\n\n\n\n\n\n4.5 out of 5 stars\n\n\n\n\n\n\n\n\n12,959 ratings\n\n\n\n\n\n\n\n\n\n\n\n\n'

And after some regex:

'Product detailsPublisher:Ember; Illustrated edition (May 29, 2018)Language:EnglishPaperback:320 pagesISBN-10:0385741278ISBN-13:978-0385741279Reading age:12 - 17 yearsLexile measure:600LGrade level:7 - 9Item Weight:12 ouncesDimensions:5.49 x 0.84 x 8.24 inchesBest Sellers Rank:#13 in Books (See Top 100 in Books) #1 in Teen & Young Adult Family Fiction #1 in Teen & Young Adult Fiction about Emotions & Feelings #1 in Teen & Young Adult Fiction about Death & DyingCustomer Reviews:4.5 out of 5 stars12,959 ratings'

So we can extract a lot of information from this column like Publisher, Illustrated edition, Language, ISBN, and so on.

df['Publisher'] = df['Product_Details'].str.extract('((?<=Publisher:).+?(?=;))' ,expand=False)#'ISBN-13:.+?(?= ;)

df['Publisher'] = df.Publisher.str.replace(r'-', '', regex=True).str.strip()#Program

The new feature Publisher :

The new feature ISBN :

df[‘ISBN’] = df[‘Product_Details’].str.extract(‘ISBN-13:(\d+-?\d*)’, expand=False)##ISBN consists of 13 digits
df[‘ISBN’] = df.ISBN.str.replace(r’-’, ‘’, regex=True).str.strip()

So, the above two examples of how easy that extract any data by using the regex technique.

Create a new dataframe that consists of desired features.

Check Nulls

hasNAN = df.isnull().sum()
hasNAN = hasNAN[hasNAN > 0]
hasNAN = hasNAN.sort_values(ascending=False)
print(hasNAN,df.shape)

The new dataframe shape is (1200, 6), so since we have many books and the null values less than 70%, so we will delete the null values.

Calculating and plotting the word count for book description

# Calculating the word count for book description
df[‘word_count’] = df1[‘Description’].apply(lambda x: len(str(x).split()))# Plotting the word count
df[‘word_count’].plot(
 kind=’hist’,
 bins = 50,
 figsize = (12,8),title=’Word Count Distribution for book descriptions’)

• The majority of books description are between 30~200 words.
• We don’t have many lengthy book descriptions. It is clear that Amazon.com provides short descriptions.
• The distribution of books description is right-skewed.

Converting book titles into vectors using TF-IDF using Bigram

#Converting text descriptions into vectors using TF-IDF using Bigram
tf = TfidfVectorizer(ngram_range=(2, 2), stop_words=’english’, lowercase = False)
tfidf_matrix = tf.fit_transform(df[‘Name’])
total_words = tfidf_matrix.sum(axis=0) 
#Finding the word frequency
freq = [(word, total_words[0, idx]) for word, idx in tf.vocabulary_.items()]
freq =sorted(freq, key = lambda x: x[1], reverse=True)
#converting into dataframe 
bigram = pd.DataFrame(freq)
bigram.rename(columns = {0:’bigram’, 1: ‘count’}, inplace = True) 
#Taking first 20 records
bigram = bigram.head(20)

##Plotting the bigram distribution
bigram.plot(x =’bigram’, y=’count’, kind = ‘bar’, title = “Bigram disribution for the top 20 words in the book titles”, figsize = (15,7), )

• The Boxed Set, Box Set, and Harry Potter are the top 3 words in the book titles.

Converting book descriptions into vectors using TF-IDF using Bigram

#Converting text descriptions into vectors using TF-IDF using Bigram
tf = TfidfVectorizer(ngram_range=(2, 2), stop_words=’english’, lowercase = False)
tfidf_matrix = tf.fit_transform(df1[‘Description’])
total_words = tfidf_matrix.sum(axis=0) 
#Finding the word frequency
freq = [(word, total_words[0, idx]) for word, idx in tf.vocabulary_.items()]
freq =sorted(freq, key = lambda x: x[1], reverse=True)
#converting into dataframe 
bigram = pd.DataFrame(freq)
bigram.rename(columns = {0:’bigram’, 1: ‘count’}, inplace = True) 
#Taking first 20 records
bigram = bigram.head(20)

##Plotting the bigram distribution
bigram.plot(x =’bigram’, y=’count’, kind = ‘bar’, title = “Bigram disribution for the top 20 words in the book description”, figsize = (15,7), )

• New York, York Times, board book are the top 3 words in the book description.

Text Preprocessing

Cleaning the book descriptions.

## Function for removing NonAscii characters
def _removeNonAscii(s):
 return “”.join(i for i in s if ord(i)<128)

## Function for converting into lower case
def make_lower_case(text):
 return text.lower()

# Function for removing stop words
def remove_stop_words(text):
 text = text.split()
 stops = set(stopwords.words(“english”))
 text = [w for w in text if not w in stops]
 text = “ “.join(text)
 return text

# Function for removing punctuation
def remove_punctuation(text):
 tokenizer = RegexpTokenizer(r’\w+’)
 text = tokenizer.tokenize(text)
 text = “ “.join(text)
 return text

# Function for removing the html tags
def remove_html(text):
 html_pattern = re.compile(‘<.*?>’)
 return html_pattern.sub(r’’, text)

The above functions converting into lower case and removing HTML tags, punctuation, stop words, and NonAscii characters for the description feature.

# Applying all the functions in description and storing as a cleaned_desc
df1[‘cleaned_desc’] = df1[‘Description’].apply(_removeNonAscii)
df1[‘cleaned_desc’] = df1.cleaned_desc.apply(func = make_lower_case)
df1[‘cleaned_desc’] = df1.cleaned_desc.apply(func = remove_stop_words)
df1[‘cleaned_desc’] = df1.cleaned_desc.apply(func=remove_punctuation)
df1[‘cleaned_desc’] = df1.cleaned_desc.apply(func=remove_html)

The above function store the modification into another feature that called cleaned_desc.

Comparison between description before and after cleaning:

Recommendation engine

• We build a recommendation engine using book titles and descriptions.
• Convert each book title and description into vectors using TF-IDF and bigram.
• The model recommends a similar book based on title and description.
• The model recommends a similar book based on title and description.
• Define a function that takes the book title as input and returns the top five similar recommended books based on the title and description.

tfidf = TfidfVectorizer(stop_words=’english’)
df1[‘cleaned_desc’] = df1[‘cleaned_desc’].fillna(‘’)
#Construct the required TF-IDF matrix by applying the fit_transform method on the overview feature
overview_matrix = tfidf.fit_transform(df1[‘cleaned_desc’])
#Output the shape of tfidf_matrix
overview_matrix.shape

#similarity_matrix
similarity_matrix = linear_kernel(overview_matrix,overview_matrix)
similarity_matrix

#book index mapping
mapping = pd.Series(df1.index,index =df1['Name'])
mapping[:3]

def recommend_books_based_on_plot(book_input):
    book_index = mapping[book_input]
    #get similarity values with other books
    #similarity_score is the list of index and similarity matrix
    similarity_score = list(enumerate(similarity_matrix[book_index]))
    #sort in descending order the similarity score of book inputted with all the other books
    similarity_score = sorted(similarity_score, key=lambda x: x[1], reverse=True)
    # Get the scores of the 5 most similar books. Ignore the first book.
    similarity_score = similarity_score[1:6]
    #return book names using the mapping series
    book_indices = [i[0] for i in similarity_score]
    return (df1['Name'].iloc[book_indices])

Test the model

Let's now try to get a recommendation for the book ‘If Animals Kissed Good Night’ from the above recommendation function and see what it outputs.

recommend_books_based_on_plot(‘If Animals Kissed Good Night’).to_frame()

Top 5 books recommended that are similar to “If Animals Kissed Good Night”.

Comparison between the first two books.

Observation

• From the description, we can see that the recommender books have similar points in the stories from the 'If Animals Kissed Good Night' book.

References:

• https://www.kdnuggets.com/2020/07/building-content-based-book-recommendation-engine.html

Problem Statment

Kingdom of Saudi Arabia (KSA) is the largest country in the Arab states and a member of the “Group of Twenty” (G-20) major world economies. The motor vehicle rate in this country has increased rapidly since the oil boom in the early 1970s, consequently, the number of roads and the transport infrastructures service increased.
In Saudi Arabia car is the main means of transportation, it provides the flexibility and freedom that people really value and want. Road traffic accidents are one of the most critical public health problems worldwide.
The WHO Global Status Report on Road Safety reports that the annual fatality rate per 100,000 people due to traffic accidents in the KSA has increased from 17.4 to 27.4 since the last decade, which is the worst among the countries in the region and is significantly above death rates for other G-20 nations such as the USA, the United Kingdom, Japan, and Australia. The economic losses due to traffic accidents are estimated to be approximately 4.3% of the KSA’s GDP. A study conducted by Turki et al. suggests that more than 19 individuals have lost their lives daily, and approximately 4 people were injured every hour due to road traffic accidents on KSA roads.

This article takes a look at the number of traffic accidents and driving licenses issued in Saudi Arabia (2016–2017), identifies trends in the data, and combines/analyzes provided datasets with outside research to identify likely factors influencing the outcomes of traffic accidents in the various regions in Saudi Arabia.

Datasets

In this project, two datasets were provided:

1. Driving Licenses:
   This dataset contains Saudi Arabia Driving Licenses Issued By Administrative Areas from 1993–2016. Data from the General Authority for Statistics.

2. Traffic Accidents and Casualties
This dataset contains Saudi Arabia Traffic Accidents and Casualties by Region for 2016-2017. Data from the General Authority for Statistics.

Data Import & Cleaning

Identify, diagnose, and treat a variety of dirty data and forming it into an appropriate form for the statistical analysis.

This what the data looked like after the cleaning process:

Exploratory Data Analysis

To give a quick overview of each numeric feature.

Visualize the data

Visualizing the data allows us to quickly convey the findings (even for a non-technical audience), it will often reveal trends in data that escaped when we were looking only at numbers.

Some plots used in this project:

The above heatmap illustrated the strong correlation between accident cases and death and injury cases, which means accidents usually lead to others, and the injury cases lead to death cases.

Histogram is a representation of the distribution of numerical data, all above histograms are right-skewed histogram that has a peak that is left of center and a more gradual tapering to the right side of the graph. This is a unimodal data set, with the mode closer to the left of the graph and smaller than either the mean or the median. The mean of right-skewed data is to the right of the peak and it is a greater value than either the median or the mode. This shape indicates that there are a number of data points, perhaps outliers.

Car accidents in 2016 and 2017 killed more than 16500 people, causing over 80,000 injuries. The most deadly accidents happened in Makkah (3884 death cases), followed by Riyadh (2829 death cases), then the Eastern region (2076 death cases), while the minimum numbers of deadly accidents were in Northern Border (305 death cases), Al-Baha (319 death cases), and Najran (367 death cases).

The most injured accidents happened in Makkah (23006 injured cases) which is a very high number comparing to other regions, followed by the Eastern region (8966 injured cases), then Riyadh (8747 injured cases) while the minimum numbers of the injured cases were in Northern Border (1005 injured case), followed by Najran (1472 injured cases), then Hail (1705 injured cases).

A scatter plot is a type of plot to display values for typically two variables for a set of data. This scatterplot illustrates the traffic accidents for 2016 vs 2017, the total number of accidents in 2017 is 13.6% less than in 2016, the maximum number of accidents in 2016 was in Riyadh, while the maximum number of accidents in 2017 was in Makkah. Najran and Al-Baha are the lowest in both years.

A boxplot is a standardized way of displaying the distribution of data based on a five-number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”).

The above boxplot illustrates the central tendency and spread in both dead and injured variables, there are some outliers shown, the highest outlier in the dead boxplot represents the number of death cases occurred in Makkah (2243 death cases), while the highest one in the injured boxplot represents the number of injury cases that occurred in Makkah (12383 injured cases) followed by the next one which is also in Makkah (10623 cases)

Descriptive and Inferential Statistics

Descriptive statistics describe data (for example, a chart or graph) and inferential statistics allows to make predictions (“inferences”) from that data. With inferential statistics, we take data from samples and make generalizations about a population.

The above histogram shows that the driving license is positively skewed, A positive-skewed distribution is one whose right tail is longer or fatter than its left, When a distribution is positively skewed, the mean is greater than the median, which is greater than the mode.
Positive skew: mode < median < mean

Although the distribution is not normal in previous variables, The central limit theorem (CLT) is a theorem that gives us a way to turn a non-normal distribution into a normal distribution. It tells us that, even if a population distribution is non-normal, its sampling distribution of the sample mean will be normal for a large number of samples (at least 30).

Investigate trends in the data

This section examines the scientific data, to see the overall trend of driving licenses based on years (1993–2017) and traffic accidents numbers for years (2016–2017)

Riyadh got the first four places in terms of the number of driving licenses issued during (1993–2017) followed by Makkah.

In terms of the number of driving licenses issued during (1993–2017), Tabouk and Al-Baha get the lowest places by 915, 997 respectively.

The above is a part of the table that shows the regions that have more Driving Licenses issued each year than the year average, During 1993–2017, Al-Qaseem -5 years-, Makkah -each year-, Riyadh -21 years-, Eastern -21 years-, Assir and Hail -one year-, these are the regions that have more driving licenses issued in each year than the year average. Makkah, Riyadh, Eastern, and Al-Qaseem, these administrative regions are the most populous. The slightly higher than mean in driving licenses in Hail happened in 2016, and In Assir happened in 2015.

The above table shows the regions that have more driving accidents occurred in each year than the yearly average during 2016–2017. The highest number of accidents was in Makkah: 145,541 accident happened in 2017, followed by Riyadh 141736 accidents happened in 2016.

Outside Research

Doing outside research on provincial and central policies that might influence the trends in the number of driving licenses and traffic accident rates. The data is borrowed from the General Authority for Statistics and the Statistical Centre for the GCC.

1. Vision 2030
In fact, various preventive measures have been effectively adapted to reduce the burden of traffic accidents in Saudi Arabia, the Ministry of the Interior (MoI) has set a strong strategy to reduce traffic accidents as part of the government’s Vision 2030 program.

As shown in the above plot, the number of total accidents dropped from over 460000 accidents in 2017 to 287781 accidents in 2019, The ratio down almost 38% percent.

2. Increase the demand for driving licenses

As shown in the below plot, the number of total driving license is getting higher, it raised from 928165 licenses in 2017 to 1026569 licenses in 2019, The ratio up 10.6% percent.

3.The development of the number of death cases in the GCC countries

As shown in the above plot, the GCC occupied the penultimate position, followed by Malaysia. While Japan, Australia, and UE were at the forefront at the lowest traffic accident mortality rate.

Some policies and engineering measures in these leading countries in an attempt to understand the reasons behind their good performance:

• ROAD SAFETY POLICIES:

The Traffic Safety Policies Law in Japan requires the government to report to the Diet, each year, on the status of traffic crashes, on measures being implemented and on plans for traffic safety measures. This is contained in the ‘White paper on traffic safety in Japan’:
-The aim is a crash-free society.
-Dealing with the issue of human error in public transportation: by improving the organizational structures and systems of companies providing transport services.
-Encourage participatory traffic safety activities by enabling citizens to participate in the planning stages of traffic safety measures run by national and local authorities.

• MEASURES TO IMPROVE THE ROAD TRAFFIC ENVIRONMENT

The White Paper A New Deal for Transport: Better for Everyone in the UK made it clear that merely building more new roads is not the answer. The emphasis is now on making the best use of the existing network, giving priority to treating the places with the worst safety, congestion, and environmental records. Key elements of the approach include recognition that good engineering reduces the risk of accidents.

Key takeaways and recommendations

Rapid development, increased number of vehicles together with population growth, are all contributing to a rise in the number of road accidents, injuries, and fatalities. Although road safety remains a low priority for most governments in the developing world, there is a growing awareness of the social, economic, and public health problems caused by traffic accidents.

• The number of total accidents is getting lower, it dropped from over 460000 in 2017 to 287781 in 2019, The ratio down almost 38% percent, through the National Transformation Program and the Kingdom’s Vision 2030, with the participation of relevant sectors, which contributes to unifying efforts and speed of achievement.
• The number of total driving license is getting higher, it raised from 928165 in 2017 to 1026569 in 2019, The ratio up 10.6% percent. the relationship between the number of licenses and the population is a positive correlation, in addition to allowing women to drive a car. On 26 September 2017, King Salman issued an order to allow women to drive in Saudi Arabia.

Recommendations

• Assess the problem, policies, and institutional settings relating to road traffic injury and the capacity for road traffic injury prevention in each region.
• Specific actions are needed to prevent road traffic crashes and to minimize their consequences. These actions should be based on sound evidence and analysis of road traffic injuries, be culturally appropriate and tested locally, and form part of the national strategy to address the problem of road accidents, especially in cities that had the most accidents and deaths: Makkah, Riyadh, and Eastern province.

Additional Data

• Driver age in each accident.
• Explain the causes of each accident:
   — human factors/personnel error.
   — malfunction or failure of vehicle structures or other systems.
   — deficient maintenance.
   — hazardous environment involving weather, animal, birds, etc.
• Drive a vehicle without a valid driver’s license or not.
• Driving under the influence (DUI) or not.

References

amal, A.; Rahman, M.T.; Al-Ahmadi, H.M.; Mansoor, U. The Dilemma of Road Safety in the Eastern Province of Saudi Arabia: Consequences and Prevention Strategies. Int. J. Environ. Res. Public Health 2020, 17, 157

https://discuss.analyticsvidhya.com/t/difference-between-wide-and-long-data-format/8110/2

https://www.stats.gov.sa/6370

https://taifchamber.org.sa/news_details/621/7-%D8%A3%D8%B3%D8%A8%D8%A7%D8%A8-%D9%84%D9%84%D8%AD%D9%88%D8%A7%D8%AF%D8%AB-%D8%A7%D9%84%D9%85%D8%B1%D9%88%D8%B1%D9%8A%D8%A9-%D8%B9%D9%84%D9%89-%D8%B7%D8%B1%D9%8A%D9%82-%D8%A7%D9%84%D8%AC%D9%86%D9%88%D8%A8-%D8%A8%D9%80-%D8%A7%D9%84%D8%B7%D8%A7%D8%A6%D9%81

https://www.moi.gov.sa/wps/portal/Home/sectors/publicsecurity/traffic/contents/!ut/p/z0/04_Sj9CPykssy0xPLMnMz0vMAfIjo8ziDTxNTDwMTYy83V0CTQ0cA71d_T1djI0MXA30gxOL9L30o_ArApqSmVVYGOWoH5Wcn1eSWlGiH1FSlJiWlpmsagBlKCQWqRrkJmbmqRqUZebngB2gUJAKdERJZmqxfkG2ezgAhzhSyw!!/

https://www.stats.gov.sa/ar/1020

https://www.stats.gov.sa/ar/2

https://www.gccstat.org/images/gccstat/docman/publications/116-gcc-accidents-statistics.pdf

https://www.codatu.org/wp-content/uploads/Road-safety-in-five-leading-countries-Rifaat-SHOUKRALLAH.pdf


https://www.alj.com/en/perspective/women-at-the-wheel-whats-changed-in-saudi-arabia/

- https://www.statisticshowto.com/inferential-statistics/#:~:text=Descriptive%20statistics%20describes%20data%20(for,make%20generalizations%20about%20a%20population.