<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohamed Djedidi on Medium]]></title>
        <description><![CDATA[Stories by Mohamed Djedidi on Medium]]></description>
        <link>https://medium.com/@mohameddjedidi?source=rss-acb14bc923a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UKLmqrc2fU21HKnf.</url>
            <title>Stories by Mohamed Djedidi on Medium</title>
            <link>https://medium.com/@mohameddjedidi?source=rss-acb14bc923a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:59:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohameddjedidi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using Starbucks users data to matching user and offers]]></title>
            <link>https://medium.com/@mohameddjedidi/using-starbucks-users-data-to-matching-user-and-offers-c25a27c74580?source=rss-acb14bc923a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c25a27c74580</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mohamed Djedidi]]></dc:creator>
            <pubDate>Tue, 20 Aug 2019 18:26:48 GMT</pubDate>
            <atom:updated>2019-08-20T18:39:01.111Z</atom:updated>
            <content:encoded><![CDATA[<p>Using Starbucks users data to matching user and offers</p><blockquote>For my Udacity Data Scientist Capstone project, I used simulated data from the Starbucks rewards app to predict the propensity of a user to take up an offer</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*O9Mx5xyYoz8WEzTnugnv9w.jpeg" /></figure><h3>Project Overview</h3><p>In this project, I aim to answer 2 main business questions:</p><ol><li><strong>What are the main drivers of an effective offer on the Starbucks app?</strong></li><li><strong>Could the data provided, namely offer characteristics and user demographics, predict whether a user would take up an offer?</strong></li></ol><p>The business motivation behind these 2 questions are two fold: firstly, from the business perspective, Starbucks can better target their offers towards customers with higher propensity to take up the offers. In doing this, they have the potential to maximize on revenues obtained from offer take up to the right customers, and saving on marketing and promotional costs by sending more relevant offers.<br>Secondly, from the customer point of view, the customer gets a better experience with Starbucks and on the app, as they will be receiving personalized, relevant offers to their tastes.</p><p>This capstone project is using data provided by Udacity as part of the Data Scientist Nanodegree course. It contains simulated data that mimics customer behavior on the Starbucks rewards mobile app.</p><h3>Problem Statement</h3><p>As stated above, the problem statement I am aiming to answer are to (1) discover the main drivers of offer effectiveness, and (2) explore if we can predict whether a user would take up an offer</p><p>The data provided consists of 3 datasets:</p><ul><li>Offer portfolio, which consists of the attributes of each offer</li><li>Demographic data for each customer</li><li>Transactional records of events occurring on the app</li></ul><p>Using the data provided, I answer the above two questions using 3 classification supervised machine learning models, feeding in the data from three different offer types.</p><p>I use the model to uncover the feature importances to identify the drivers of offer effectiveness, while exploring if the model itself could be used to predict if a user would take up an offer.</p><p>Lastly, I also explore the characteristics of users who do or do not take up an offer.</p><p>My project aims to answer the two questions above, but I also ended up adding 2 additional models as points of exploration which will be expounded in Part 4 — the first, assessing whether an all-in-one model could be used in place of 3 different models, with the offer types functioning as a categorical variable. Secondly, I also build a regression model to see if we could predict the amount a user would spend, given that the offer is effectively influencing them.</p><p>If I am able to successfully implement the models above, my expected solution would be to find the drivers of the effective offers, which could be demographic attributes of the users, for example the membership tenure of the user.</p><p>Ideally, a successful model would also be able to predict the propensity of a user to take up an offer with at least 75% accuracy. That would be my benchmark accuracy for the 3 models I plan to implement.</p><p>To assess the performance of my models, the metrics I will be using is accuracy, with F1 score to compare against, as these are classification models.</p><p>F1 score provides a better sense of model performance compared to purely accuracy as takes both false positives and false negatives in the calculation. With an uneven class distribution, F1 may usually be more useful than accuracy.</p><p>It is worth noting in this case that the F1 score is based on the harmonic mean of precision and recall, and focuses on positive cases. For the Starbucks app here, it would be fine as we would prioritise more on whether offers are effective, and less focus on why offers are ineffective.</p><p>Meanwhile, accuracy provides a good overall metric to assess my model performance, and would be sufficient for the purposes of predicting the propensity of a user taking up the offer as we will not be too worried about the negatively classified cases. In fact, for business purposes, we may allow some misclassified negative cases, as we may rather grab the possible sale of product via the incentive of the offer, rather than allow a missed opportunity.</p><p>With that, I will explain my project in the following sections:</p><ol><li><strong>Data Exploration </strong>— I explore the attributes of the provided data and decide on an approach for preprocessing the data for modeling</li><li><strong>Data Preprocessing</strong> — I did the heavy duty data cleaning and feature engineering in this section to prepare the data for modeling</li><li><strong>Model Implementation</strong> — I implemented my 3 models and refined on the model implementation, considering the model performance in each version and deciding on the best model based on the results.</li></ol><p>Although I will delve into my code output if required for understanding on how I implemented this project, I will not delve into the nitty gritty every step of the way. For more details on that, you can check my <a href="https://github.com/syuenloh/UdacityDataScientistCapstone">Github repo</a>which contains the Jupyter Notebook of the project.</p><h3>Part 1. Data Exploration</h3><h3>a. Offer Portfolio data</h3><p>According to the information provided by Udacity, the schema is as follows:</p><p><strong>portfolio.json</strong></p><ul><li>id (string) — offer id</li><li>offer_type (string) — type of offer ie BOGO, discount, informational</li><li>difficulty (int) — minimum required spend to complete an offer</li><li>reward (int) — reward given for completing an offer</li><li>duration (int) -</li><li>channels (list of strings)</li></ul><p>Moreover, some further information given about the offers is that there are 3 different offer types:</p><ul><li>BOGO — buy one get one free</li><li>Discount — discount with purchase</li><li>Informational — provides information about products</li></ul><p>Thus, the schema is pretty straightforward, as it contains the attributes of 3 different offer types. While the duration was not explained I assumed from context that it is in terms of number of days.</p><p>After taking a snapshot view of the data, I noted some key elements to do data cleaning during the preprocessing stage. Among them:</p><ul><li>Expand the channel column later during preprocessing to become categorical variables in my dataset</li><li>Feature scaling — the scale of each are different, for example the difficulty is in terms of dollars while the duration is in terms of days.</li><li>With no null values, we can leave it as is.</li></ul><h3>b. Demographic data</h3><p>Demographic data for customers is provided in the profile dataset. The schema and variables are as follows:</p><p><strong>profile.json</strong></p><ul><li>age (int) — age of the customer</li><li>became_member_on (int) — date when customer created an app account</li><li>gender (str) — gender of the customer (note some entries contain ‘O’ for other rather than M or F)</li><li>id (str) — customer id</li><li>income (float) — customer’s income</li></ul><p>It is also relatively straightforward, as it contains the demographic profile on the customer.</p><p>I did some data visualization of the income column and found some further data preprocessing steps required.</p><ul><li>Null values in gender and income column — would need to be dropped if they do not take up a significant portion of the dataset</li><li>Feature engineering for became_member_on column</li></ul><h3>c. Transactional records</h3><p>The schema for the transactional data is as follows:</p><p><strong>transcript.json</strong></p><ul><li>event (str) — record description (ie transaction, offer received, offer viewed, etc.)</li><li>person (str) — customer id</li><li>time (int) — time in hours. The data begins at time t=0</li><li>value — (dict of strings) — either an offer id or transaction amount depending on the record</li></ul><p>This data looks a bit more tricky, as it is ordered by time and has an event and value. In particular, the value column will have to be preprocessed depending on the event.</p><p>I also found that the number of people (i.e. number of unique IDs) in transcript are the same as the number of people in the Demographics Data, so that is good news. But a lot of preprocessing will need to be done in order to extract meaningful insights out of this dataset.</p><p>Now I do a bit of data cleaning for the transcript dataset. In order to extract insights from the value column, I expanded the values into individual columns depending on the event. With that, I end up with a clean transcript dataset to be analysed further.</p><h3>d. Defining approach for preprocessing data for the model</h3><p>Before I proceeded to preprocess the data for the model, I first revisited my objective. Having done a preliminary exploration of the data, I had to reassess how I would clean and prepare the data for the models I intended to build.</p><p>In order to identify the main drivers of an effective offer, I have to first define what an ‘effective’ offer is within the Starbucks app. Thus, I did some further exploration on the datasets and how all three would interact.</p><p>First, I explored the kinds of events within each offer type.</p><p>We know that there are 4 types of events: offer completed, offer received, offer viewed and transaction. But our data shows that we do not have any offer_id associated with transactions, because they are not recorded in the transcript event data. Thus, the first objective in data preprocessing is to define a methodology to assign offer_ids to specific transactions.</p><p>Moreover, we also know that BOGO and discount offers have an offer completed event when offers are completed. However, informational offers do not have this event associated with it. Thus, we also specify the approach to define an effective offer as follows:</p><p>For a BOGO and discount offer, an effective offer would be defined if the following events were recorded in the right sequence in time:</p><p>offer received -&gt; offer viewed -&gt; transaction -&gt; offer completed</p><p>Meanwhile, for an informational offer, since there offer completed event associated with it, I will have to define transactions as a conversion to effective offer:</p><p>offer received -&gt; offer viewed -&gt; transaction</p><h3>Part 2: Data Preprocessing</h3><h3>a. Assigning offer ids to transactions</h3><p>After defining the approach above, we now have to explore methods to assign offer_ids to specific transactions. Among the considerations is to define the following main groups of customers:</p><ol><li><strong>People who are influenced and successfully convert — effective offers:</strong></li></ol><ul><li>offer received-&gt; offer viewed -&gt; transaction -&gt; offer completed (<em>BOGO/discount offers</em>)</li><li>offer received -&gt; offer viewed -&gt; transaction (<em>informational offers — must be within validity period of offer</em>)</li></ul><p><strong>2. People who received and viewed an offer but did not successfully convert — ineffective offers:</strong></p><ul><li>offer received -&gt; offer viewed</li></ul><p><strong>3. People who purchase/complete offers regardless of awareness of any offers:</strong></p><ul><li>transaction</li><li>offer received -&gt; transaction -&gt; offer completed -&gt; offer viewed</li><li>transaction -&gt; offer received -&gt; offer completed -&gt; offer viewed</li><li>offer received -&gt; transaction -&gt; offer viewed -&gt; offer completed</li><li>offer received -&gt; transaction (<em>informational offers</em>)</li><li>offer received -&gt; transaction -&gt; offer viewed (<em>informational offers</em>)</li></ul><p><strong>4. People who received offers but no action taken:</strong></p><ul><li>offer received</li></ul><p>For people in group 2, I would need to check if there are events where there is an offer received and offer viewed event, but no conversion event, i.e. offer completed or transaction - these are cases of ineffective offers.</p><p>I would have to separate out the people in group 2 from people in group 4, as people in group 2 may have viewed an offer but did not take any action, whereas people in group 4 did not even have an offer viewed event.</p><p>Separating the conversions for effective offers (group 1) and people who purchase/complete offers regardless of awareness of any offers (group 3) is particularly tricky. For people in group 3, a conversion is invalid (i.e., not a successful conversion from an offer) if an offer completed or transactionoccurs before an offer viewed. There also may be scenarios where an offer completed occurs after the offer is viewed, but a transaction was done prior to the offer being viewed. In this instance, the offer may have been completed, but it is also not a valid conversion.</p><p><strong>Defining the target variable </strong><strong>effective offer:</strong></p><p>After defining these conditions, we have to decide what the target variable will be.</p><p>We know that group 1 customers will be our target variable effective_offer=1, but there are many ineffective offer definitions for groups 2-4.</p><p>So what would we define as an ineffective offer? As already stated above, group 2 would be within our definition of an ineffective offer; where a user is aware of an offer, but the offer is ineffective as it does not convert the user into a customer. So group 2 can be defined as our target variable effective_offer=0.</p><p>What about group 3 and group 4? Group 3 consists of users who may have received offers but would have purchased regardless. From the business point of view, we would not want to be sending them any offers.</p><p>Meanwhile, group 4 users would be considered low priority customers, as they do not do any action, regardless of whether they receive offers or not.</p><p>So, we can deprioritise group 3 and group 4 users from our model. It would still be worth doing some exploratory analysis onto group 3 and 4, just to explore on their demographics.</p><p>The conditions above are the basis of which I can assign the offer id that ‘influences’ a transaction by ensuring that the transaction occurs after an offer viewed event.</p><p>Any unique person-offer_id belonging to group 1 can thus be considered in our target variable effective_offer=1 group.</p><p>Meanwhile, group 2 is in our target variable effective_offer=0 group. For customers in groups 3 and 4, I deprioritise them for model implementation, but will be doing some exploratory analysis on them later in Part 4.</p><p>As such, I created a dataset for each offer type consisting of both effective and ineffective offers by appending group 1 and group 2 customer datasets together, and successfully prepared the target variables for our BOGO and discount datasets.</p><p>Meanwhile, for informational offers in particular, before we can tag the effective offers column, there is an additional consideration — the validity of the offer. This is because the conversion event is not an offer completedevent, but a transaction.</p><p>For informational offers, the duration of the offer can be considered to be the duration of the influence. Hence, we can make the assumption that an offer should only be considered effective if it is within the duration of the offer.</p><p>Meanwhile, for BOGO and discount offers, we can assume that if there is a conversion/ offer completed event, it should be within duration as it would not make sense for an offer to be completed if an offer is past its validity period.</p><h3>d. Feature engineering</h3><p>Now we have to look back had to look into the features and see how to be creative in creating new features.</p><p><strong>d.i. </strong><strong>became_member_on column to be engineered</strong></p><p>Recalling my preliminary data exploration steps, the became_member_oncolumn were in date format. Hence in order to extract meaningful insights from that feature, we can convert it as a feature indicating tenure of membership. There could be some influence in how long someone has been a member, with whether he takes up an offer.</p><p>Thus, I extracted the membership tenure of each person in days from the became_member_on column, using 2018 as the current year.</p><p><strong>d.ii. Count of offers received</strong></p><p>As part of some further data exploration, I discovered that there could be multiple offers received per person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*KGBr-GyVR8se8ahmVUQgYw.png" /></figure><p>We can see above that the offer received per person in the transactional data could range from 1 to 6 offers received. I had the hypothesis that the frequency of offers received per person might result in more effective offers, so decided to engineer a feature offer_received_cnt to account for this frequency.</p><p><strong>d.ii. Separating user behaviours by transactions</strong></p><p>I also wondered how many transactions were considered ‘invalid’ by my definition. Ordinarily, these would be the sum of transactions done by people not in group 1. The objective of offers are to drive purchases, so it would already be the case that users with high spend in their transactions would be flagged as effective_offers.</p><p>We’ve already defined that there are people in groups 3 and 4, where they are separate pools of users who are loyal spenders, and already tend to purchase more, isolated from the the effect of offers.</p><p>But for users in group 1 have a high amount of ‘invalid spend’ outside of the effect of offers, there might be some predictive power onto the effectiveness of offers; since a loyal user might have a higher tendency of taking up an offer.</p><p>The logic is to wonder if there is some baseline level of spending for users who are highly influenced by certain offers (in group 1), and group 2, and if there is some predictive power in this baseline level of ‘invalid transactions’ that can predict the propensity of a user to take up an offer.</p><p>Thus, I calculated the ‘invalid’ transactions for each person as an additional feature.</p><p>However, in doing so I found out that the missing values were quite extensive in this engineered feature column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*_3C_XK2FRx1s_fjjb1XSdg.png" /></figure><p>Because of the sparsity, it is debatable whether this column amount_invalidwould be useful to include in the model. Since it is so &#39;sparse&#39;, it might not have much information after all. I plan to assess this feature again later during the model implementation phase. For now, I decided to fill the missing amount_invalid column with 0 as it could represent that only 4% of the overall users tend to purchase without offers; the other 96% would only purchase with awareness of an ongoing offer.</p><p>Meanwhile, we had already conducted the analysis above on the incomeand gender columns, which I had already chosen to drop as they are not useful when they are null, and they only make up 7% of the dataset.</p><p><strong>d. iii. Time elapsed between offers received</strong></p><p>I also wanted to include time as a potential feature into my dataset, but since the transactional data starts from time=0, I suspected it would not have been of much predictive power without some feature engineering. I had the hypothesis that if there were multiple offers received per person within a certain time period, there might be some predictive power in the time elapsed between offers received.</p><h3>Part 3: Implementation</h3><p>Now that the datasets are ready, we can proceed to implement the model. Revisiting our objective, we wanted to analyse the drivers of an effective offer, with the target variable being effective_offer.</p><p>Since we have 3 offer types, there are thus 3 different models to be built. Since we are predicting whether an offer would be effective or not, this is effectively a binary classification supervised learning model.</p><p>I decided to compare the performance of a simple decision tree classifier model as a baseline model, with an ensemble random forest classifier model. Reason why I selected a decision tree as the baseline model is because I wanted to prioritise the interpretability of the model. Going back to the objective, since we intend to analyse the feature importance to determine the drivers of an effective offer, a decision tree would provide good interpretability for us to analyse.</p><p>Meanwhile, I also selected random forest as an alternate model to compare the baseline model is as an improvement over simple ensemble bagging of decision trees, in order to drive towards a high accuracy in training the model.</p><p>Before we can proceed, we have to make sure that the classes we are predicting for are balanced in each dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*N5N3L4bIWrq5APrk16Fqqg.png" /></figure><h3>a. Model Implementation</h3><p>Revisiting our objective, we are creating 3 models to predict the effectiveness of an offer within each type, depending on offer attributes and user demographics.</p><p>I performed the usual required steps for modeling:</p><ul><li>define target and feature variables</li><li>split to train and test data</li><li>apply feature scaler</li></ul><p>I defined model pipeline functions to implement my model as I plan to implement 3 different models; hence it would be easier to implement repeatedly.</p><p>I used a DecisionTree Classifier as my baseline model and a Random Forest classifier with randomly assigned parameters to compare the performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c25a27c74580" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to create magic features in Kaggle : Advanced Feature Engineering]]></title>
            <link>https://medium.com/@mohameddjedidi/how-to-create-magic-features-in-kaggle-advanced-feature-engineering-bcac4a19047f?source=rss-acb14bc923a------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcac4a19047f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[features]]></category>
            <dc:creator><![CDATA[Mohamed Djedidi]]></dc:creator>
            <pubDate>Wed, 24 Jul 2019 13:20:59 GMT</pubDate>
            <atom:updated>2019-07-24T14:02:03.590Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to create magic features in Kaggle : Advanced Feature Engineering</h3><p>Feature engineering is the art of understanding the business problem and to use the domain knowledge to select,create and transform the variables that will be fed into your machine learning model. This step is so essential in applied machine learning since the quality and quantity of the features will have great influence on whether the model is good or not.</p><p>Appropriate well-designed features are often the deciding factor of the performance of your final algorithm. For this reason, data scientists often spend 70%-80% of their time on the pre-modelling phase, which largely consists of feature engineering.</p><figure><img alt="Figure 1 : Features Engineering represents one of the most important steps in every machine learning project" src="https://cdn-images-1.medium.com/max/680/1*JrQY1Q5lIA98eH1vZ3n6gA.jpeg" /></figure><h3><strong>Let’s dig deep and try to work on a real example…</strong></h3><p><strong><em>The data which we will use is from a competition I won.<br></em></strong> We will be trying to build a model that predicts the number of seats that Mobiticket can expect to sell for each ride. There are 14 routes in this dataset. All of the routes end in Nairobi and originate in towns to the North-West of Nairobi towards Lake Victoria.</p><p>Variables description:</p><ul><li>ride_id: unique ID of a vehicle on a specific route on a specific day and time.</li><li>seat_number: seat assigned to ticket</li><li>payment_method: a method used by customer to purchase ticket from Mobiticket (cash or Mpesa)</li><li>payment_receipt: unique id number for ticket purchased from Mobiticket</li><li>travel_date: date of ride departure. (MM/DD/YYYY)</li><li>travel_time: scheduled departure time of ride. Rides generally depart on time. (hh:mm)</li><li>travel_from: town from which ride originated</li><li>travel_to: destination of ride. All rides are to Nairobi.</li><li>car_type: vehicle type (shuttle or bus)</li><li>max_capacity: number of seats on the vehicle</li></ul><p>First of all we will import the libraries that we are going to use</p><pre>import pandas as pd #to use dataframes<br>import numpy as np #for linear algebra operations<br>import matplotlib.pyplot as plt #for visualization<br>import seaborn as sns<br>from sklearn.model_selection import train_test_split,KFold<br>import xgboost as xgb #the model we&#39;re going to use<br>from sklearn.metrics import mean_absolute_error #our metric</pre><p>We now need to load our train and test data</p><pre>train=pd.read_csv(“./Data/train.zip”)<br>test=pd.read_csv(“./Data/test.csv”)</pre><p>The data unfortunately doesn’t contain the number of tickets per trip so we need to build our data and try to group it by ride_id to find the right number of tickets per ride.</p><pre>Target=train.groupby([“ride_id”]).seat_number.count().rename(“number_of_ticket”).reset_index()<br>train=train.drop_duplicates(“ride_id”).drop([‘payment_method’, ‘payment_receipt’, ‘seat_number’],axis=1)<br>train=train.merge(Target,how=”left”,on=”ride_id”)</pre><pre>train.drop([“travel_to”],axis=1,inplace=True)<br>test.drop([“travel_to”],axis=1,inplace=True)</pre><p>Now we should process the data columns and add time features which are very important when dealing with time series problems.</p><pre>def add_20(x):<br>   date=x.split(“-”)<br>   date[-1]=”20&quot;+date[-1]<br>   return “-”.join(date)<br>train[“date”]=(train[“travel_date”].apply(add_20)+” “+train[“travel_time”]).astype(str)<br>test[“date”]=(test[“travel_date”]+” “+test[“travel_time”]).astype(str)</pre><p>We will create a function time_features which will extract all the time details of a given data like the hour, day, month, weekend or not…</p><pre>def time_features(x): <br>    x[“date”]=pd.to_datetime(x[“date”],format=’%d-%m-%Y %H:%M’)<br>    x[“dayofweek”]=x[“date”].dt.dayofweek<br>    x[“dayofyear”]=x[“date”].dt.dayofyear<br>    x[“dayofmonth”]=x[“date”].dt.day        x[“year_woy”]=x[“date”].dt.year.astype(str)+x[“date”].dt.weekofyear.astype(str)<br>    x[“hour”]=x[“date”].dt.hour<br>    x[“minute”]=x[“date”].dt.minute<br>    x[“is_weekend”]=x[“dayofweek”].apply( lambda x : 1 if x in [5,6] else 0 )<br>    x[“year”]=x[“date”].dt.year<br>    x[“quarter”]=x[“date”].dt.quarter<br>    x[“month”]=x[“date”].dt.month<br>    return x <br>train=time_features(train)<br>test=time_features(test)</pre><h3>Visualizing</h3><p>It is now time to visualize our data. For that we will focus on categorical features and we will build a function to plot the count and the mean of the number of tickets grouped by the chosen feature.</p><pre>categorical_feautres=[“travel_from”,”car_type”,”dayofweek”,”dayofmonth”,”hour”,”minute”,”year”]</pre><pre>def plot_categorical_feature(feautre):<br>     fig, axes = plt.subplots(nrows=1, ncols=2)<br>     train.groupby(feautre).number_of_ticket.mean()\<br>     .plot(kind=”bar”,ax=axes[0],figsize=(20,5),title=”mean of    number_of_ticket per {}”.format(feautre))<br>     train.groupby(feautre).number_of_ticket.count().\<br>     plot(kind=”bar”,ax=axes[1],figsize=(20,5),title=”ride count per {}”.format(feautre)) <br>     plt.show()<br>for feautre in categorical_feautres : <br>     plot_categorical_feature(feautre)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*Zl7-DCryl_ksvxan61l8tQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*6lFjfBBbCYKirl-dXlwIlg.png" /></figure><h3>Target Encoding</h3><p>We will define now a function which takes a data frame, the name of the categorical feature, the name of the target column and statistical functions as inputs and returns an aggregate data frame grouped by the categorical feature.</p><pre>def target_encoding(data,feautre,<br>                    target=”number_of_ticket”,<br>                    agg_functions={“mean”,”std”}):</pre><pre>    agg=data.groupby(feautre)[target].agg(agg_functions)<br>    agg.columns=[column+”_per_{}_{}”.format(feautre,target) for column in agg.columns.tolist()]<br>    return agg</pre><p>We will be using 3 target encoding approach in this tutorial.<br>So, we will be defining another helper function to calculate the aggregate data frame for each categorical feature and join them with the train and test sets.</p><p>This function will be used as <strong>our first approach for target encoding</strong>:</p><pre>def add_target_encoding_features(train,test):<br>     for feautre in categorical_feautres : <br>          agg=target_encoding(train,feautre)<br>          train=train.merge(agg,how=”left”,on=feautre)<br>          test=test.merge(agg,how=”left”,on=feautre)<br>     return train,test<br>train_TE,test_TE=add_target_encoding_features(train_TE,test_TE)</pre><p>In many cases when you use the whole train set to calculate the aggregate features you will end up with a good score with the training and validation set and a bad score with the test set (overfitting), due to strong correlation between the new generated features and the target in the train set. To avoid this problem we should split the train data to new train and validation sets from the start and calculate the aggregate features using only the new train. We join it later with the validation set and the test set. Then we compare the score of the training and the validation to know if the aggregate features are overfitting or not.</p><p><strong>The second approach </strong>will be executed with this function. We first need to split our train data into 2 parts : the train data(train of the train lol but we kinda need of a validation set from the big train dataset to evaluate our model) and the validation data.</p><pre>train_VTE,val_VTE=train_test_split(train,random_state=1994,test_size=0.1)<br>test_VTE=test.copy()</pre><pre>def add_target_encoding_features_validation(train,val,test):<br>    for feautre in categorical_feautres : <br>        agg=target_encoding(train,feautre)<br>        train=train.merge(agg,how=&quot;left&quot;,on=feautre)<br>        val=val.merge(agg,how=&quot;left&quot;,on=feautre)<br>        test=test.merge(agg,how=&quot;left&quot;,on=feautre)<br>        <br>    return train,val,test</pre><pre>train_VTE,val_VTE,test_VTE=add_target_encoding_features_validation(train_VTE,val_VTE,test_VTE)</pre><p><strong>This final method</strong> is done by using kfolds instead of a simple split.</p><pre>def add_target_encoding_features_Kfold(train,test,split=10):<br>    kf = KFold(n_splits=split,random_state=2222,shuffle=False)<br>    train_final=[]<br>    for train_index, test_index in kf.split(train):<br>        train_fold, val_fold = train.loc[train_index], train.loc[test_index]<br>        for  feautres in categorical_feautres:<br>            agg=target_encoding(train_fold,feautres)<br>            val_fold=val_fold.merge(agg,how=&quot;left&quot;,on=feautres)<br>        train_final.append(val_fold)<br>    <br>    for  feautres in categorical_feautres : <br>        agg=target_encoding(train,feautres)<br>        test=test.merge(agg,how=&quot;left&quot;,on=feautres)<br>        <br>        <br>    return pd.concat(train_final),test</pre><pre>train_KTE=train.copy()<br>test_KTE=test.copy()<br>train_KTE,test_KTE=add_target_encoding_features_Kfold(train=train_KTE,test=test_KTE)</pre><h3>Modeling</h3><p>Now everything is ready. We just need to define our model and train it. To do that, we are going to define a function to train the data in one fold which takes train, validation and test data and returns the prediction of the train, validation and test.</p><pre>def train_model(X_train,Y_train,X_val,<br>Y_val,X_test,parmaters,features_name): <br>    d_train = xgb.DMatrix(X_train, Y_train,feature_names=features_name)<br>    d_valid = xgb.DMatrix(X_val, Y_val,feature_names=features_name)<br>    d_test = xgb.DMatrix(X_test,feature_names=features_name)<br>    list_track = [(d_train, &#39;train&#39;), (d_valid, &#39;valid&#39;)]<br>    model = xgb.train(parmaters, d_train, 2000,  list_track, maximize=False, verbose_eval=50, early_stopping_rounds=50)<br>    train_pred =model.predict(d_train)              <br>    valid_pred =model.predict(d_valid)   <br>    test_pred = model.predict(d_test)<br>    return train_pred ,valid_pred,test_pred</pre><p>Since we want to be more confident in our algorithm performance. We create five different models using our learning algorithm and test it on five different test sets. This technique is implemented in sklearn in Kfold function from sklearn.model_selection.</p><pre>def train_kfold(X_train,Y_train,X_test,<br>                parmaters,features_name,split=5,):</pre><pre>    final_train_pred=np.zeros_like(Y_train)<br>    final_test_pred=np.zeros(len(X_test))<br>    <br>    kf = KFold(n_splits=split,random_state=2222)<br>    i=1<br>    for train_index, val_index in kf.split(X_train):<br>        print(&quot;fold:&quot;+str(i))<br>        train_fold_features, val_fold_features = X_train.loc[train_index], X_train.loc[val_index]<br>        train_fold_target, val_fold_target = Y_train.loc[train_index], Y_train.loc[val_index] <br>        train_pred ,valid_pred,test_pred=train_model(<br>              X_train=train_fold_features,                                                  <br>              Y_train= train_fold_target,<br>              X_val= val_fold_features,<br>              Y_val= val_fold_target,<br>              X_test= X_test,<br>              parmaters=parmaters,         <br>              features_name=features_name )<br>        <br>        final_train_pred[val_index]=valid_pred<br>        final_test_pred=final_test_pred+test_pred/split<br>        i=i+1<br>    return final_train_pred,final_train_pred</pre><p>We need now to remover useless features.</p><pre>columns_to_remove=[&#39;ride_id&#39;, &#39;travel_date&#39;, &#39;travel_time&#39;, &#39;travel_from&#39;, &#39;car_type&#39;,<br>       &#39;max_capacity&#39;, &#39;number_of_ticket&#39;, &#39;date&#39;, &#39;dayofweek&#39;, &#39;dayofyear&#39;,<br>       &#39;dayofmonth&#39;, &#39;year_woy&#39;, &#39;hour&#39;, &#39;minute&#39;, &#39;year&#39;,<br>       &#39;quarter&#39;, &#39;month&#39;]<br>features_name=train_KTE.drop(columns_to_remove,axis=1).columns</pre><h3>Training</h3><pre>train_TE_train,train_TE_val=train_test_split(train_TE,random_state=1994,test_size=0.1)</pre><pre>X_train=train_TE_train[features_name]<br>Y_train=train_TE_train[&quot;number_of_ticket&quot;]</pre><pre>X_val=train_TE_val[features_name]<br>Y_val=train_TE_val[&quot;number_of_ticket&quot;]</pre><pre>X_test=test_TE[features_name]</pre><pre>params = {&#39;eta&#39;: 0.004, &#39;colsample_bytree&#39;: 0.7, <br>               &#39;max_depth&#39;: 9,&#39;subsample&#39;: 0.9, <br>               &#39;lambda&#39;: 4, &#39;nthread&#39;: 8, <br>               &#39;booster&#39; : &#39;gbtree&#39;, &#39;silent&#39;: 1,<br>               &#39;eval_metric&#39;: &#39;rmse&#39;, &#39;objective&#39;: &#39;reg:linear&#39;,<br>               &quot;gamma&quot;:0.1 ,&quot;alpha&quot;:0.01} <br>train_pred ,valid_pred,test_pred=train_model(X_train,<br>                                             Y_train,X_val,<br>                                             Y_val,X_test,<br>                                             params,features_name)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*Lw0vZI5KNmzIrPRAhJzOnQ.png" /></figure><p>The link to the <a href="https://github.com/JedidiMohamed/Tutoriel/">Github</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcac4a19047f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>