<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matteo Fedeli on Medium]]></title>
        <description><![CDATA[Stories by Matteo Fedeli on Medium]]></description>
        <link>https://medium.com/@matteo.fedeli?source=rss-70784f2ce37e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jfh_V3qEpy7a1cm8-mUNrA.jpeg</url>
            <title>Stories by Matteo Fedeli on Medium</title>
            <link>https://medium.com/@matteo.fedeli?source=rss-70784f2ce37e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:28:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matteo.fedeli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My First Time]]></title>
            <link>https://medium.com/digital-futures-publications/my-first-time-621127011e7c?source=rss-70784f2ce37e------2</link>
            <guid isPermaLink="false">https://medium.com/p/621127011e7c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Matteo Fedeli]]></dc:creator>
            <pubDate>Sat, 27 Nov 2021 14:46:20 GMT</pubDate>
            <atom:updated>2022-01-06T15:36:33.248Z</atom:updated>
            <content:encoded><![CDATA[<p>My <em>first time</em> developing, composing, and completing a Data Science project in Python. My <em>first time</em> participating in a Kaggle competition. My <em>first time</em> publishing a Medium article telling about this achievement.</p><p>In this article I am going to present my Capstone Project for the <a href="https://digitalfutures.com/for-individuals/academy/">Digital Futures Data Science Academy</a>, a 12-week programme I attended between July and September of 2021. This programme has provided me with the skills and knowledge required to kickstart my career in data science and allowed me to make very precious connections in the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XjN9hTj-IEvc7ogO03kjZg.jpeg" /></figure><h4><strong>Introduction</strong></h4><p>I chose to undertake the <a href="https://www.kaggle.com/c/tabular-playground-series-sep-2021">Kaggle “Tabular Playground Series — Sep 2021” Prediction Competition</a> as my capstone project for the programme since it allowed me to put into practice the knowledge I have acquired during the 12 weeks and in my free time, exploring the applicability and performance on a new dataset of the models and techniques I have learned. Beyond that, it has also allowed me to discover and learn about new modelling techniques and compare my skills and abilities with many other participants to the Kaggle competition. This competition in particular allowed me to practice and improve my classification and prediction skills.</p><p><strong>The dataset</strong></p><p>The competition consisted of predicting whether a customer made a claim upon an insurance policy based on a synthetic dataset containing a variety of features. The challenge was further complicated by the features themselves being anonymised and some observations being intentionally missing from the dataset. The dataset provided for the competition consisted of synthetic observations generated using a <a href="https://github.com/sdv-dev/CTGAN#overview">CTGAN</a>, a tool whose developers describe as:</p><blockquote>CTGAN is a collection of Deep Learning based Synthetic Data Generators for single table data, which are able to learn from real data and generate synthetic clones with high fidelity.</blockquote><p>The competition data was separated into a train and a test dataset, of sizes 957,919 rows by 120 columns and 493,474 rows by 119 columns respectively. The first column contained the row number and was therefore chosen to be the index and removed as a feature of the dataset. The two datasets contained 118 anonymised feature columns with observations of varying magnitudes and intentionally missing values, accounting for approximately 1.5% of the data for each column not being present. We can observe a missing value in the last row of the “f117” column for the training dataset below. The train dataset also contained the “claim” column, containing either 0 or 1, which would be the target of the predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*bvd-3jn7udqq52JqnT3K3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*QN8x4ycSQhnEO9Zqdjybjw.png" /><figcaption>Figures 1 and 2: First 5 rows of the final columns of train (left) and test (right).</figcaption></figure><h4><strong>Methodology</strong></h4><p>After importing the datasets and installing the necessary Python packages into Kaggle Notebook, I first separated the target column from the train dataset, saving the targets as a Series and the feature columns as a DataFrame. The test dataset, containing no target column by design, was simply saved as given. The datasets were imported into Pandas DataFrames with the row number as the index.</p><p>I then began with some basic exploratory data analysis. One of the first metrics I looked at was the correlation between my features. The correlation matrix below clearly illustrates the degree of linear correlation between the 118 features and the target in the train dataset. The dark colour of the cells outside of the diagonal implies a lack of correlation between individual features, which in turn indicates that every feature is independent of each other. It, however, also suggests that the features do not display any correlation with the target column they are expected to predict. I must admit this initially came as a bit of a shock since it contrasted with many of the datasets I had encountered previously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a_z9_xhRgB824KzLz5bF9A.png" /><figcaption>Figure 3: Heat map representing the correlation between the features and the target column</figcaption></figure><p>The next step was filling the missing values in the datasets. I chose to do so by imputing the column mean in place of the null values using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html">sklearn’s SimpleImputer</a>. I then checked that both the new train and test datasets no longer had any missing values. The two DataFrames with no more missing values were then saved separately so that I still kept a copy of the original datasets. Having done that, I moved to add additional feature columns I expected would improve my prediction in the later steps.</p><p>I created six new columns, namely: “n_missing”, which counted the number of missing values for each row of the original dataset, “any_missing”, a boolean column which indicated whether the number of missing values in each row was greater than 0, “max”, “min”, “avg” and “std” which measured the maximum value, minimum value, average value, and standard deviation for each row, respectively.</p><p>The final step in my feature engineering was scaling the values in the DataFrames, which ranged from 6.7055e-11 to 3.2499e+17, to a common scale since some of the models I planned to use either relied on or greatly benefitted from, observations being all in the same scale. I used <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html">sklearn’s RobustScaler</a> to standardise my observations while also dealing with possible outliers that could have skewed the distribution of the data.</p><p>Having completed the necessary data cleaning and feature engineering, I chose to delete all the unnecessary objects that had been created through the various cleaning steps. This included all the DataFrames that contained partial steps of the cleaning process but also the Imputer and Scaler used to finalise my dataset. This choice was dictated by the large size of the datasets being used and the relatively limited computational power available to me, especially in terms of RAM. It was therefore essential for me to be able to free up as much memory as possible for later steps. Another step I took to reduce the amount of memory used to store my data was to reduce the precision of floating-point data points from the default float64 to float32 and of integers from int64 to int8.</p><p>These steps turned out to be fundamental for the success of the project since skipping them would result in the Kaggle Notebook running out of memory, forcing me to restart my kernel and run all previous steps again.</p><h4><strong>Models and techniques</strong></h4><p><strong>Logistic regression</strong></p><p>Since the challenge, I was facing consisted of predicting whether each row of features corresponded to one of two discrete outcomes, either a claim is made upon an insurance policy or it is not, the first model I considered was logistic regression. This model and its performance metrics were therefore chosen as the benchmark against which I would compare other models I planned to use. The very first model simply used all available feature columns to predict the target column. The use of the <a href="https://www.statsmodels.org/stable/generated/statsmodels.discrete.discrete_model.Logit.html">statsmodels Logit</a> model allowed me to quickly observe the summary statistics for the logistic regression and, despite the successful execution of the model, notice many statistically insignificant coefficients, indicating that the result I obtained was not valid. I, therefore, decided to reduce the number of predictors, iteratively eliminating those that appeared insignificant in the summary table. Over a number of iterations, this left me with a logistic regression where all coefficients were statistically significant. The final logistic regression output is attached below.</p><figure><img alt="Logistic regression output" src="https://cdn-images-1.medium.com/max/516/1*NZgJjUZtFghGrNOCn8nt7g.png" /><figcaption>Figure 4: Logistic regression output with only statistically significant coefficients</figcaption></figure><p>I used this logit model to predict labels for the train dataset it was trained on, to check that in-sample performance was good. At the same time, I checked that the performance was not <em>too good</em> since that would indicate that the model is overfitting the training dataset. I used functions from the <a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics">sklearn metrics</a> package to build a function that would calculate and print various metrics and scores for my predictions. The aptly named “aprf1auc” function takes the predicted labels and the real labels and returns scores for; accuracy, precision, recall, F1 score, and Area Under the Curve. The scores for my in-sample predictions are attached below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*ydDg8Yi7LToQve6wSgEtEg.png" /><figcaption>Figure 5: Logistic regression prediction metrics and confusion matrix</figcaption></figure><p>Predicted labels were also obtained for the test data and were the first set of predictions that could have been submitted to complete the competition. They were therefore saved as a CSV file according to the sample submission file provided. The test data did not contain the “true” labels so metrics could not be obtained for the predictions made.</p><p>The results obtained from the logistic regression model served as a baseline against which further models would be compared. This first model already demonstrated relatively good metrics, especially in terms of recall, while suffering in terms of precision.</p><p>This result can be interpreted as the classifier being very effective at correctly recognising most observations where claims are actually made while being less effective at only including real claims among those it recognises as valid.</p><p>From a practical point of view, the high recall would allow insurance companies to recognise the vast majority of real insurance claims. This means that most policyholders who file a claim are recognised and rightfully compensated. The lower precision would instead lead to some observations where no claim was made being considered valid and compensation would go to policyholders who do not deserve it.</p><p>This result would make policyholders relatively happy since when a claim is made, almost 90% of the time the claim is recognised and compensation is sent. The result would however be quite expensive for the company since about 30% of the observations recognised as valid would not actually be valid and compensation would go to policyholders who do not deserve it, with the company paying out to fraudulent claims.</p><p>The tradeoff would therefore be between not compensating a policyholder who has suffered damages and deserves it (losing customer satisfaction) against compensating a policyholder who has not suffered any damage and is unrightfully compensated (wasting company money).</p><p>Although ideally, the insurance company would want to minimise (ideally eliminate) both occurrences, from a financial point of view, it would be important to reduce the amount that is paid out on fraudulent claims since that money would be completely wasted while also benefitting fraudulent individuals.</p><p>It would therefore be beneficial to improve precision so fewer fraudulent claims are identified as valid. Nonetheless, if recall were to decrease, more valid claims would not be correctly identified and more policyholders that require compensation would be ignored and not receive any payment to cover the damages incurred.</p><p><strong>Support Vector Machine</strong></p><p>Having set my benchmark, it was then time to move to another model that could hopefully improve the quality of my predictions. The second classification model studied during the course was Support Vector Machine, and more specifically <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC">sklearn’s Support Vector Classification</a> implementation. Unfortunately, despite numerous attempts (with and without GPU acceleration) to train this model, the 16 gigabytes of RAM available did not suffice to successfully train the model, which kept running out of memory and forced the restart of the kernel. This modelling technique was therefore abandoned and no predictions were able to be made.</p><p><strong>Random Forest</strong></p><p>This classification method was also covered in detail during the course and had performed well in previous projects. It was therefore expected to be a good candidate model to predict the labels and complete the challenge. I used <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html">sklearn’s RandomForestClassifier</a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html">ExtraTreesClassifier</a> to implement this technique in my project. To test multiple combinations of hyperparameters, I used <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html">sklearn’s GridSearchCV</a> to fit the combinations and return the best performance. The hyperparameters searched were: “max_depth”, “min_samples_split” and “min_samples_leaf”. The parameter “n_estimators” was fixed at 5 for a good balance between processing time and model effectiveness. The function also carried out cross-fold validation to better evaluate the expected performance of the model on unseen data. Across various trial and error steps of choosing adequate ranges of values for the hyperparameters, the best combination obtained was then used to score the model and make predictions. The best-performing model was then returned by the function and the corresponding parameters can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*HHeddGEaYIt0Q0Ce0urWpQ.png" /><figcaption>Best performing Random Forest parameters and score</figcaption></figure><p>The best combination of parameters was then used to fit a RandomForestClassifier model and make predictions on the training data. The performance metrics are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*YqqtTPBU4ftRbWXmSrw51g.png" /><figcaption>Figure 5: Random Forest prediction metrics and confusion matrix</figcaption></figure><p>The metrics show slightly worse performance compared to the benchmark logistic regression model. This can be observed by the lower metric scores and the distribution of labels in the confusion matrix. The lower AUC score suggests this model might be a worse performer than the logistic regression in making predictions on the test dataset. We, therefore, move to a more complex but hopefully more performing model.</p><p>The lower precision of the model (compared to the baseline) would further worsen the financial downside that the insurance company would face since more of the observations classified as valid would actually be fraudulent and more payments would go to undeserving individuals. The recall is also slightly lower, making this model objectively worse performing than the baseline.</p><p><strong>XGBoost</strong></p><p>This fourth model was the first step outside of what I had covered during the course. The choice of this model was motivated by the substantial popularity across many notebooks and high-scoring submissions I found when looking for extensions to my project. This was therefore an excellent way to learn more about a new classification algorithm while also attempting to improve on the predictive performance of previous models. I initially treated the model as a black box, feeding in the adequate datasets and obtaining results, without really understanding the underlying theory. As with previous models, I trained it on the relevant DataFrame, predicted the targets and measured the performance. Different to the previous models, I did not set any hyperparameters, instead, I took the default values. Nonetheless, I still obtained prediction metrics which can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*cZTuLNhNO15C8BNUNZ0flA.png" /><figcaption>Figure 6: XGBoost Classifier prediction metrics and confusion matrix</figcaption></figure><p>The metrics show very similar performance to the benchmark logistic regression model across the various scores and also in terms of the distribution of observations in the confusion matrix.</p><p>The precision and recall for this model are basically identical to the baseline, but since the model required more resources and more time to train and make predictions, it would be a worse candidate since it performs the same as the benchmark, a much simpler and quicker model.</p><p>In an attempt to find better performing scores, I turned to a much more elaborate set of functions, contained in the PyCaret module.</p><p><strong>PyCaret Classification</strong></p><p>This <a href="https://pycaret.readthedocs.io/en/latest/api/classification.html">module</a> was suggested to me by a colleague and initially looked quite complicated and daunting. The variety of available functions and tools took me some time to understand and then put to use. Nonetheless, after having read through some of the documentation and looked at some examples, I really understood the power of this module. The first step was to set up the “training environment” and “create the transformation pipeline”. This involved calling the “setup” function and giving it the train dataset (including the target column) and a string with the name of the target column, which in my case was “claim”. These were the only necessary parameters but, in addition, the setup function also took additional parameters: “data_split_stratify = True<strong>”</strong> to specify that, during the splitting of the dataset into training and validation, the dataset should be stratified by the target column; “use_gpu = True” to have the pipeline use GPU acceleration for training with algorithms that support it; “n_jobs = -1” to use all processors for functions that supports parallel processing; “silent = True” to execute the confirmation of data types automatically and without prompting the user. As mentioned, the function performs a train-test split internally to create a validation dataset not used for training the model and on which out-of-sample prediction metrics can be obtained. The setup function by default also randomly shuffled rows between train and validation and removed perfect collinearity (features with correlation = 1) from the dataset. The setup function was then run and produced a summary table with details about all the processing steps it had taken. Attached below is the code used to set up the PyCaret pipeline</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17832b6a8827a1524f3694f7965fd0d7/href">https://medium.com/media/17832b6a8827a1524f3694f7965fd0d7/href</a></iframe><p>The next step consisted of understanding what models were available to be trained. This was done by simply using the “models()” function, which returns a table of available models for the given dataset. The output for my dataset is presented below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*RokGyDf1454TV-BSg8MliQ.png" /><figcaption>Figure 7: Table produced by PyCaret containing all models available to train on the given dataset</figcaption></figure><p>The “compare_models()” function was then run to obtain metrics for all models compatible with the dataset provided to it. Some models were nonetheless excluded due to a variety of reasons: some were excluded because they did not benefit from GPU acceleration and took too long to run (ada, gbc) while others were excluded since they were run independently before (dt, et). The function fit the models on the train dataset, made predictions on the validation dataset and returned a table containing metrics scores for all models so that they can quickly be compared. The models are shown ranked by AUC score since the competition also ranked predictions based on AUC score. The function also automatically implemented 10-fold cross-validation to better represent out-of-sample performance. The code to compare the models and table of metrics are attached below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d6a271bda98e59549a9b0a49176630e/href">https://medium.com/media/8d6a271bda98e59549a9b0a49176630e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*wQc5pnLdB_LhmdFSi6Iunw.png" /><figcaption>Figure 8: Model metrics comparison</figcaption></figure><p>Having collected metrics for the various models, I started experimenting with creating individual models using the “create_model” function and, using the “tune_model” function, automatically having PyCaret iteratively optimise the model hyperparameters using a grid search, possibly improving the performance of the model. The CatBoost model was chosen for this process since it displayed the highest AUC score and was the best-scoring model among those examined. Attached below are the metrics for a CatBoost model trained on the given dataset. The 10 folds’ performance is measured and finally averaged to provide the overall model performance metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*0KgGfZFmB4XD8GuHTHm9mQ.png" /><figcaption>Figure 9: Metrics for the 10 folds of a CatBoost model prediction</figcaption></figure><p>Hyperparameter optimisation was then explored. The “tune_model” function was also given parameters “optimize = AUC” such that model tuning optimised AUC score and “choose_better = True” such that “the returned object is always better performing” than the base model. Attached below are the code to tune the model and the metrics for the tuned CatBoost model. The process of tuning was nonetheless was very resource and time-intensive and did not make large improvements to the metrics so ended up rarely being used. Indeed, the results show almost negligible improvements over the base model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c402e1facb6c10c68cf46008c45f5cbd/href">https://medium.com/media/c402e1facb6c10c68cf46008c45f5cbd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*rSuJpTNOykZX3YxfLCcsUA.png" /><figcaption>Figure 10: Metrics for the 10 folds of a tuned CatBoost model prediction</figcaption></figure><p>The PyCaret module also has a “plot_model” function that allows the creation of plots and graphs to visually evaluate the performance of the models. The plots I used to evaluate my models were the “auc”, “feature”, and “confusion_matrix”. These display, respectively, the Area Under the Curve, Feature Importance, and Confusion Matrix for the dataset. The plots for a CatBoost model are presented below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*k3aDrGcRTdbFAl0YFSuR9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*1BjsyYTL7p7nh0bTxkJkAg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*QYt4-nNgY7x8bkRY_JXKSw.png" /><figcaption>Figure 11: ROC Curve plot and AUC scores (Left), Feature Importance plot (Centre), Confusion Matrix plot (Right)</figcaption></figure><p>The lower number of observations in the confusion matrix is explained by the fact that it represents the performance of the CatBoost Classifier on the validation dataset, which is only 20% of the overall training dataset originally imported. Nonetheless, we can observe slightly worse recall and marginally better precision, resulting in an almost identical F1 score compared to the models analysed in previous sections.</p><p>During these steps, I also experimented with model stacking and blending, aiming to exploit the performance improvements that may arise from the combination of various models. These ensembles were then treated as regular models and used to make predictions. The process of blending was found to produce the highest-scoring predictions. In particular, a Voting Classifier composed of the three highest-scoring individual models, that is CatBoost, LightGBM and XGBoost, gave the highest AUC score metric achieved throughout the project. Attached below are the composition and plots for the Voting Classifier</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*be-u7Zes7E8vWtEmzFLP4w.png" /><figcaption>Figure 12: The three models (and relative hyperparameters) blended by the Voting Classifier</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*SeizAzi_tF15rAXwSVuy9w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*BCgCIixqGA-y4OItt2AVoA.png" /><figcaption>Figure 13: ROC curves and AUC scores for Voting Classifier (Left), Confusion matrix for Voting Classifier (Right)</figcaption></figure><p>Having trained and tuned a number of models, it was then time to predict labels for the validation set using the train dataset. This would then allow the first measure of out-of-sample performance since the validation dataset had not been used to train the model. The “predict_model” function takes the model as its only parameter and returns metrics for predictions on the validation set. The function output is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*CnSZz4s_7VYIGH_kJB-tAQ.png" /><figcaption>Figure 15: Tuned Voting Classifier prediction metrics on validation dataset</figcaption></figure><p>To train the model on both the train and validation sets, so that the model takes advantage of the most data possible to then predict on the test dataset, the “finalize_model” function is used to “train a given estimator on the entire dataset including the holdout set.” The “predict_model” is then run again but now takes the finalised model as input. The output of this function can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*WlpxEoP7ZwVJ2IQO_5vTaQ.png" /><figcaption>Figure 16: Finalised Voting Classifier prediction metrics on validation dataset</figcaption></figure><p>The finalised model is then compared to the baseline model established at the beginning of the section. We can observe a worsening of recall and an improvement in precision, resulting nonetheless in an improvement in F1 score.</p><p>In practice, this classifier would trade a decrease in customer satisfaction from lower recall for a decrease in wasted money from higher precision. This model was therefore chosen to make the final predictions on the test dataset.</p><p>The last step was to predict labels for the test dataset, which would then be submitted to complete the competition. This was done using the “predict_model” again but now giving it the name of the test DataFrame. This returned the DataFrame itself with the addition of the “Label” and “Score”(probability of predicted class) columns. However, the competition required the labels not to be binary (either 0 or 1) but rather any number from 0.0 to 1.0. To obtain scores instead of binary labels, the “raw_score = True” parameter was also included. This led to the addition of a new column named “Score_1” which represented the probability of a claim and was the column to be submitted. The final columns of the test DataFrame can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*u6NzDq7f7L4V7IlyNBEGDw.png" /><figcaption>Figure 17: Predicted labels for test dataset. Score_1 being the predicted probability of a claim</figcaption></figure><p>I then created a final DataFrame composed of the row index for each row of the test dataset and the corresponding value of “Score_1”, downloaded it as a CSV and submitted it for scoring on the <a href="https://www.kaggle.com/c/tabular-playground-series-sep-2021/submit">competition submission page</a>. The code below illustrates how the test predictions are obtained and saved as a CSV file to be submitted.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76c8d93c06bab4616224d04c53761061/href">https://medium.com/media/76c8d93c06bab4616224d04c53761061/href</a></iframe><p>My best submission was able to achieve an AUC score of 0.81481 on Kaggle’s test dataset, allowing me to score in the top 37% of participants in the final standings and only <a href="https://www.kaggle.com/c/tabular-playground-series-sep-2021/leaderboard">0.0029 lower than the top-scoring competitor</a>. This positive result also persuaded me to participate in the <a href="https://www.kaggle.com/c/tabular-playground-series-oct-2021">October</a>, <a href="https://www.kaggle.com/c/tabular-playground-series-nov-2021">November</a>, and <a href="https://www.kaggle.com/c/tabular-playground-series-dec-2021">December</a> 2021 competitions, the latter of which is still in progress at the time of writing.</p><h4><strong>Limitations</strong></h4><p>The main limitation I faced during this project was most likely the large size of the datasets. This had a substantial impact on the choice of modelling techniques I was able to use to analyse and make predictions based on the dataset. The large size of the dataset also meant very long runtimes, especially when fitting complex models on the training dataset. In the most extreme of cases, when using GridSearch to find the best combination of model hyperparameters, model fitting took upwards of one hour since many different models had to be fit in order to find the best performing combination.</p><p>The relatively short time available was also a limitation for the project. For example, the use of <a href="https://docs.rapids.ai/api/cuml/stable/api.html#random-forest">cuML RandomForestClassifier</a>, which would use GPU acceleration to substantially speed up model fitting, was considered. The model worked amazingly well when parameters were given to it explicitly but failed repeatedly when using GridSearchCV to find optimal combinations of parameters. This was understood as incompatibility between the cuML Random Forest model and the grid searching algorithm. Due to time constraints, the cuML model had to be abandoned but would be a sensible extension to the project.</p><p>The anonymity of the feature columns also made it more difficult to identify which features to prioritise when making predictions, although that was most likely an intentional challenge as part of the competition. The anonymity forced me to treat every feature with the same importance. The lack of any substantial correlation between features also suggested that all features were useful in predicting the target column and could therefore not be removed. The synthetic nature of the dataset also limits the applicability of any findings since no intuition about the real world can be obtained from the results.</p><h4>Next steps</h4><p>Taking into consideration the limitations presented above, the main improvement that could be made would be the availability of additional computing power, both in terms of processing speed and memory. That would not only speed up current models but also allow the experimentation of additional models and techniques that could not be carried out in this project. The use of more complex and powerful models could have provided better performance scores and improved the AUC score achievable on the test dataset.</p><p>A natural next step for this project would be to examine and compare the performance on real data of the models used. This would likely be a challenging objective but would be a real test of the applicability of the models used to a real-life situation. Indeed, being able to confidently predict whether a customer is likely to make a claim upon an insurance policy can have substantial economic and strategic implications for the insurance industry and those who operate within it.</p><p>Beyond insurance claims, being able to predict whether something will happen, and with what probability, can be extremely beneficial in most fields and environments. Examples of this can be found anywhere, from being able to diagnose and treat diseases before they become harmful, to being able to identify indicators of the performance of a company and profit from investments. Many more examples exist, highlighting the universal importance of prediction and classification.</p><p>As a matter of fact, being able to know or do something before others underpin most aspects of our society and are, therefore, a fundamental and priceless ability to possess.</p><h4><strong>Appendix</strong></h4><p><em>X_train</em>: pandas DataFrame containing the feature columns of the train dataset</p><p><em>y_train</em>: pandas Series containing the target column of the train dataset</p><p><em>X_test</em>: pandas DataFrame containing the feature columns of the test dataset</p><p><em>X_train_fill</em> and <em>X_test_fill</em>: copies of the respective pandas DataFrames with null values replaced by the mean value of the column in which each null value is located</p><p><em>scaled_X_train</em> and <em>scaled_X_test</em>: scaled copies of the X_train and X_test DataFrames such that each feature is normalised to improve the performance of machine learning algorithms</p><h4>Summary</h4><p>The aim of this project was to take part in and complete the <a href="https://www.kaggle.com/c/tabular-playground-series-sep-2021">Kaggle “Tabular Playground Series — Sep 2021” Prediction Competition</a>. The competition consisted of predicting the probability of a customer making a claim on an insurance policy. I chose this as my capstone project since it allowed me to put my knowledge and skills into practice but also explore new topics and learn new modelling techniques.</p><p>The best solution I was able to achieve consisted of a voting classifier model containing the three highest-scoring available models I was able to train on my dataset. This solution was able to achieve good predictive performance both in-sample and out-of-sample, with AUC scores in excess of 83% when using the entire dataset to predict labels for the validation dataset. Other metrics scored highly as well, indicating the good performance of the chosen model.</p><p>The project was concluded with the submission of the test predictions to the relevant <a href="https://www.kaggle.com/c/tabular-playground-series-sep-2021/submit">competition submission page</a> to finally be scored. My best predictions allowed me to score in the top 37% of participants.</p><p>If you appreciated this article (<strong>and especially if you did not</strong>) please feel free to leave a comment with any suggestions, questions, or corrections.</p><p>GitHub repository for the notebook: <a href="https://github.com/ilmatts/TPS-Sep-2021">https://github.com/ilmatts/TPS-Sep-2021</a></p><p><em>This project was part of my data science training with Digital Futures. Digital Futures helps individuals from all backgrounds to kickstart their career in tech. If you like what we are doing, join us at </em><a href="http://digitalfutures.com/"><em>digitalfutures.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=621127011e7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/digital-futures-publications/my-first-time-621127011e7c">My First Time</a> was originally published in <a href="https://medium.com/digital-futures-publications">Digital-Futures-Publications</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>