<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fernandao Lacerda Dantas on Medium]]></title>
        <description><![CDATA[Stories by Fernandao Lacerda Dantas on Medium]]></description>
        <link>https://medium.com/@fernandao.lacerda.dantas?source=rss-3342ed687e10------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nkHU3-iU0RTpe285JJD9ng.jpeg</url>
            <title>Stories by Fernandao Lacerda Dantas on Medium</title>
            <link>https://medium.com/@fernandao.lacerda.dantas?source=rss-3342ed687e10------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:59:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fernandao.lacerda.dantas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spaceship Titanic Kaggle competition top 7% score solution]]></title>
            <link>https://medium.com/@fernandao.lacerda.dantas/space-titanic-kaggle-competition-0-8066-score-solution-7a9c401281c6?source=rss-3342ed687e10------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a9c401281c6</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Fernandao Lacerda Dantas]]></dc:creator>
            <pubDate>Wed, 17 Jan 2024 22:05:49 GMT</pubDate>
            <atom:updated>2025-05-18T03:21:14.039Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*-RpjIW4UcpKu0AI2" /></figure><h3>Introduction</h3><p>Kaggle’s Space Titanic machine learning competition is quite similar to the well-known Titanic competition. Given a dataset, we are required to predict which passengers were transported or not by an “anomaly” using records recovered from the spaceship’s damaged computer system. The “lore” of the competition is not so important, what you need to know is to develop a machine learning algorithm capable of correctly predicting the outcome of the spaceship’s passengers.</p><h3>Import libraries</h3><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>from sklearn.model_selection import train_test_split,GridSearchCV,cross_val_score,StratifiedKFold<br>import xgboost as xgb<br>import category_encoders as ce<br>from sklearn.impute import SimpleImputer<br>from sklearn.pipeline import Pipeline<br>from sklearn.pipeline import make_pipeline<br>from sklearn.compose import ColumnTransformer<br>from sklearn.metrics import accuracy_score, roc_auc_score, precision_score, recall_score,f1_score, confusion_matrix<br>from category_encoders import TargetEncoder<br>from sklearn.impute import SimpleImputer,KNNImputer<br>from lightgbm import LGBMClassifier</pre><h3>Let’s Start</h3><p>As you can see, after I loaded the dataset, I removed both the “Passenger Id” and “Name” columns. They are not going to provide any useful or important information to the prediction. Someone’s name or Id does not change the probability of being Transported.</p><pre>df = pd.read_csv(&quot;train.csv&quot;)<br><br>df.drop(columns=[&quot;PassengerId&quot;,&quot;Name&quot;],inplace=True)</pre><p>Now, we are going to discuss a fundamental step I came across after trying to improve my score a thousand times. This step relies on exploring the “Cabin” column. Notice that the rows on the “Cabin” column follow a specific pattern. Something like this: “A/5/S”, “C/1/S”, “F/7/P”. And I decided to investigate it. So, to make things simple I split the rows of the “Cabin” into three columns based on both slashes (”/”) of the rows. For example, the “A/5/S” row would be transformed into three new columns: The first one is named “cabin_code” referring to<em> </em>the character behind the first slash (A). The second one named “id_cabin” refers to the character behind the second slash (5). The third one named “cabin_sector” refers to the character after the second slash (S). And we end up with three new columns.</p><pre>#Splitting<br><br>df[[&quot;cabin_code&quot;,&quot;id_cabin&quot;,&quot;cabin_sector&quot;]] = df[&quot;Cabin&quot;].str.split(&quot;/&quot;, n=2, expand=True)<br><br>df.head(4)</pre><p>First of all, I noticed that “cabin_code” only has 8 different characters which means that the cabins are, somehow, divided into 8 sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*lHG69QxW367yII1Leb_CBA.png" /></figure><p>Also, I asked myself if passengers from a specific section had a higher chance of being transported or if this statement was not true. With the plot below we can conclude that passengers from the B and C sections have a greater chance of surviving and passengers from the E section have a lower chance of surviving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*9yUfI2EXPKpoqH_5_ndZzA.png" /></figure><p>I did the same thing with the “cabin_sector” column and also noticed that there was a difference between the sectors. Passengers from the P sector have a lower chance of being transported, while in the S sector, the opposite happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*qUnWWe93aWeyWxiT5T8A8A.png" /></figure><p>This means that this exploration of the original “Cabin” column is worth it since new insights are being added to the model.</p><p>Now, we can finally delete the “Cabin” column. It will not provide any useful information for the model anymore. We have already extracted everything useful from it.</p><p>I also removed the “cabin_id” and the column that I had created. As I said before, the Id will not interfere with the model’s predictive ability.</p><p>So used: df.drop(columns=[“Cabin”,”id_cabin”], inplace=True) to drop both columns</p><p>Before splitting our data, the “Transported” column must be in a binary format. As you can see, I switched “True” for 1 and “False” for 0.</p><p>Binary transformation: df[“Transported”] = df[“Transported”].map({True:1, False:0})</p><p>I also removed every row that had missing values in the “cabin_code” column.</p><pre><br>#BINARY TRANSFORMATION<br>df[&quot;Transported&quot;] = df[&quot;Transported&quot;].map({True:1, False:0})<br><br><br>#DROPPING COLUMNS<br>df.drop(columns=[&quot;Cabin&quot;,&quot;id_cabin&quot;], inplace=True)<br><br><br>#DROPPING NULLS<br>df.dropna(subset=[&quot;cabin_code&quot;], inplace=True)</pre><p>Now, we can finally split the data and proceed to develop our model.</p><p>After splitting in train and test, I separated the test data into two categories: numerical and categorical. Why is that? We are going to perform different operations depending on the type of the variable. Categorical data must be encoded since most models are not able to understand categorical values and it must be converted to numerical values. Also, we are going to apply different techniques to fill the null values in our dataset, but I will talk more about it later on.</p><pre>#Define X and y<br><br>X = df.iloc[:,0:12]<br>y = df[&quot;Transported&quot;]</pre><pre>#Splitting Data<br><br>X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=42,test_size=0.25)</pre><pre>#Separate categorical and numerical features<br><br>cat_feat = np.array([coluna for coluna in X_train.columns if X_train[coluna].dtype.name == &#39;object&#39;])<br><br>num_feat = np.array([coluna for coluna in X_train.columns if coluna not in cat_feat])</pre><p>We can now create our pipeline. There are going to be two pipelines: one is going to handle the categorical data and the other one is going to handle numerical data. The missing values of the categorical data will be filled with the most frequent value (mode) and after the Target Encoder will be applied to transform categorical variables into numerical variables. The numerical data missing values will be filled with a strategy called <strong>K-nearest neighbors,</strong> which uses the Euclidean distance between the data points to find the best number to fill the missing values. If don’t know how this Pipeline technique works, I recommend you check <a href="https://medium.com/@fernandao.lacerda.dantas/boost-your-pipelines-with-columntransformer-b2c009db096f">my article about Pipelines.</a></p><pre>#Categorical and numerical pipelines<br><br><br>cat_pipe = Pipeline([(&quot;imputer_cat&quot;, SimpleImputer(strategy=&quot;most_frequent&quot;)),(&quot;encoder&quot;, ce.TargetEncoder()),<br>                    ])<br><br>num_pipe = Pipeline([(&quot;imputer_num&quot;, KNNImputer(n_neighbors=3))])</pre><p>And with column transformer, we can attach both transformations to one variable that I named “transformer”. Observe that we also have to specify the type of data to which the pipeline will be applied to: “cat_pipe” will be applied to “cat_feat” and “num_pipe” will be applied to “num_feat”, meaning the categorical pipeline will take care of the categorical data and the numerical pipeline will take care of the numerical data.</p><pre>#Using ColumnTransformer <br><br><br>transformer = ColumnTransformer([(&quot;num_trans&quot;, num_pipe, num_feat),<br>                            (&quot;cat_trans&quot;, cat_pipe, cat_feat)])</pre><p>After finishing the pipelines, we have to apply the transformations to our data. We use “fit.transform” in the “X_train” data to make the model “learn” the transformations and apply “transform” in the “X_test” data.</p><pre># &quot;.fit_transform&quot; in train data&quot;<br><br># &quot;.transform&quot; in test data&quot;<br><br>X_train_transformed = transformer.fit_transform(X_train, y_train)<br>X_test_transformed = transformer.transform(X_test)</pre><p>In the next step, we are going to perform a Stratified Cross-Validation to select the best tree-based model we will use. We are going to try three models: LGBMClassifier, XGBoost and XGBoost (booster=”gblinear”). And based on the accuracy, the cross-validation will give the mean and the standard deviation of the performance of each model.</p><pre>models = []<br><br>models.append((&quot;xgb&quot;,xgb.XGBClassifier()))<br>models.append((&quot;xgbgblinear&quot;,xgb.XGBClassifier(booster=&quot;gblinear&quot;)))<br>models.append((&quot;LGBM&quot;,LGBMClassifier()))<br><br><br>print(models)<br><br>results = dict()<br><br>for  name, model in models:<br>    skf = StratifiedKFold(n_splits = 5, random_state=None)<br>    cv_results = cross_val_score(model,X_train_transformed,y_train,cv=skf, scoring=&quot;accuracy&quot;)<br>    results[name]= (cv_results.mean(), cv_results.std())<br><br>print(&quot;name     results.mean     results.std&quot;)<br><br>for key,value in results.items():<br>    print(key,value)</pre><p>We can see that LGBMClassifier had the best performance, therefore it will the model used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYH6zHHhymWHQvbOUZPrtw.png" /></figure><p>However, the model itself will not be enough to provide great accuracy. Thus, we now have to perform what is called hyperparameter tuning to make the model more precise.</p><p>We set the parameters we want to test and using sklearn’s GridSearchCV we will obtain the best hyperparameters. GridSearchCV will test the parameters that we want and will show us the combination that has the best performance.</p><pre>lgbmc = LGBMClassifier()<br><br><br>lgbm_params = {&quot;n_estimators&quot;:[100,200,300], <br>               &quot;learning_rate&quot;:[0.01,0.05,0.1,0.3],<br>               &quot;num_leaves&quot;:[20,50,80,100]}<br><br>grid_search = GridSearchCV (estimator = lgbmc,<br>                            param_grid = lgbm_params,<br>                            n_jobs=-1,<br>                            cv = 5,<br>                            scoring=&quot;accuracy&quot;,<br>                           error_score=&#39;raise&#39;)</pre><pre>grid_result = grid_search.fit(X_train_transformed, y_train)<br><br><br>final_model = lgbmc.set_params(**grid_result.best_params_)</pre><p>With the best model optimized, we can finally train our model and obtain our predictions.</p><pre><br>#training the model<br>final_model.fit(X_train_transformed, y_train)<br><br><br>#predictions<br>y_pred = final_model.predict(X_test_transformed)</pre><p>After obtaining our predictions, we have to test our model using metrics such as recall, precision, f1 score and accuracy.</p><pre>precision = precision_score(y_pred, y_test)<br>accuracy = accuracy_score(y_pred, y_test)<br>recall = precision_score(y_pred, y_test)<br>f1 = f1_score(y_pred, y_test)<br><br>score = []<br>score.append((&quot;precision&quot;, precision))<br>score.append((&quot;accuracy&quot;,accuracy))<br>score.append((&quot;recall&quot;,recall))<br>score.append((&quot;f1&quot;,f1))<br><br>score= pd.DataFrame(score)<br>score.rename(columns={0: &quot;Metric&quot;, 1:&quot;Result&quot;}, inplace=True)</pre><p>The data frame below shows us some of those metrics and we can conclude that the model is having a great performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*HeMWr9IeA_cN1E7HguIlxA.png" /></figure><p>The strategies that I mention in this article led me to a 0.80664 score in the competition and I am sure you can improve my model to achieve an even higher score with your knowledge!</p><p>If you enjoyed this article, don’t forget to support me or hit me with a follow!</p><p>See you in the next article!<br>-Fernando Dantas</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a9c401281c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boost your pipelines with ColumnTransformer]]></title>
            <link>https://medium.com/@fernandao.lacerda.dantas/boost-your-pipelines-with-columntransformer-b2c009db096f?source=rss-3342ed687e10------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2c009db096f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Fernandao Lacerda Dantas]]></dc:creator>
            <pubDate>Sat, 16 Dec 2023 04:38:34 GMT</pubDate>
            <atom:updated>2023-12-16T04:38:34.782Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*12qD8cSydlhh5o1G6-p9xw.png" /><figcaption>Machine Learning pipeline (via Datatron)</figcaption></figure><p>You already know Pipelines save you a lot of time while building machine learning models. However, Pipelines face an issue when you want to apply different methods depending on the data you are handling.</p><p>Let’s say you built the pipeline below for your model, but now you want to impute the missing data in your dataset with different approaches. You want to fill the missing data of the numerical columns using the mean (strategy = ”mean”) and the missing values of the categorical columns using the most frequent value (”strategy = ”most_frequent”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T79k8lRRIqyaSE7xqRCoUg.png" /></figure><p>If you are paying attention, you have already noticed that this split is a must in this case, because we can not fill categorical values with their mean. The mean can only be calculated from a numerical set of data.</p><p>This is where the Column Transformer comes in! In the last step, we created two pipelines. One that will treat the numerical values and the other one the categorical values. Now, we use the column transformer to merge both pipelines and apply them to the data we want. We will apply num_pipeline for the num_feat (numerical_features) and cat_pipeline for the cat_feat (categorical_features).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7F5lcTagTRWK2WTC5Ui9RQ.png" /></figure><p>Finally, we can select our machine learning model and apply using make_pipeline. At the end, the we can use .fit in our training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSrGhdgdqoCGWWeTRFnbvA.png" /></figure><p>Once you fit your training data Sklearn will display the pipeline used with our steps. So let’s make a quick review of what we did using the output below. Two different pipelines were created based on which type of data we were handling.</p><p>We applied Simple Imputer (using the mean) and MinMax Scaler for the numerical features and Simple Imputer (using the most frequent values) and OneHotEncoder for the categorical features. Both pipelines were merged and XGBClassifier was used to train our model.<br>Now you know how to use Column Transformer to boost your pipelines and apply different methods depending on the data you have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0s7zF0bIVq9PxmzebyTJlg.png" /></figure><p>Now you know how to use Column Transformer to boost your pipeplines and apply different methods depending on the data you have.</p><p>Thanks for reading! See you in the next article!</p><p>Fernando Dantas</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2c009db096f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use Pearson and Spearman coefficient — A beginners guide]]></title>
            <link>https://medium.com/@fernandao.lacerda.dantas/how-to-use-pearson-and-spearman-coefficient-a-beginners-guide-b66bd3c58a36?source=rss-3342ed687e10------2</link>
            <guid isPermaLink="false">https://medium.com/p/b66bd3c58a36</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Fernandao Lacerda Dantas]]></dc:creator>
            <pubDate>Sat, 19 Aug 2023 20:58:14 GMT</pubDate>
            <atom:updated>2023-08-19T20:58:14.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to use Pearson and Spearman coefficient — A beginners guide</h3><p>Pearson and Spearman coefficients are both extremely important concepts for data analysis and knowing how to apply each will boost your data projects. Frequently asked in interviews, they represent a fundamental topic of data: measuring correlation. If you want to learn the theory behind them and thrive in your job interviews just stick to this article!</p><h3>Usability</h3><p>Pearson and Spearman evolve around the goal of measuring the correlation between variables. Let’s say you got a graph just like the one below while working on your projects. It’s not hard to tell that there is a correlation between the two variables. But, how strong is this correlation? That is the question the coefficients will answer. They will tell you if variables are correlated and if the correlation is weak or significant. This is why using coefficients is important, especially in decision-making. Knowing how variables behave and correlate is fundamental to understanding how one is associated with the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8MvN8KMsS27_adrJfOSkDw.png" /></figure><h3>Pearson</h3><p>The Pearson coefficient is used to measure linear correlation, so when the data in your graph shows a linear pattern the Pearson coefficient will tell you how well are they related. Just like the graph below. You see that you can easily trace a straight line that represents the pattern of the data points. The Pearson metric will describe the association more accurately and this is why we have a high coefficient of 0.86.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KB1Cl9nr0lqSY5jJCrxakQ.png" /></figure><p>The Pearson coefficient ranges from -1 to 1, so let me explain to you how to understand the result you get:</p><ul><li>Coefficient equals 1 or close: Strong and positive association (as one increases the other increases).</li><li>Coefficient equals -1 or close: Strong and negative association (as one increases the other decreases).</li><li>Coefficient equals 0 or close: Very weak or no association between the variables.</li></ul><h3>Spearman</h3><p>The Spearman coefficient is used to measure the strength between to ranked variables. It is used when the data does not follow a linear pattern, also known as a monotonic relationship. A monotonic function is one that either never increases or never decreases as its independent variable increases, that is the main point that differs between Pearson and Spearman coefficients in their usability. Take a look at the graph below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RuKCt9gfy42-uLhRA0uMzQ.png" /></figure><p>It is easy to see why we would use Spearman to calculate the correlation in this case, the data does not form a linear pattern and is a monotonic function. And that is the reason the Spearman coefficient is 0.79, a considerably strong correlation between Variable A and Variable B.</p><ul><li>Coefficient equals 1 or close: Strong and positive association.</li><li>Coefficient equals -1 or close: Strong and negative association.</li><li>Coefficient equals 0 or close: Very weak or no association between the variables.</li></ul><h3>Conclusions</h3><p>You choose your correlation metric based on the distribution and shape of your data. The Pearson coefficient will give you great results when your data follows a linear pattern, while Spearman is used when the data points do not follow a linear pattern, but can be described as a monotonic function.</p><p>Now, you know everything a beginner needs to know about data correlation to boost and improve your data projects or analyses.</p><p>Go ahead and put everything you learned into practice!</p><p>If you enjoyed this post, don’t forget to hit me with a follow. See you in the next article!</p><p>-Fernando Dantas</p><p>(<a href="https://www.linkedin.com/in/fernando-dantas-4a12311a2/">https://www.linkedin.com/in/fernando-dantas-4a12311a2/</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b66bd3c58a36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overfitting and Underfitting  —  simplified]]></title>
            <link>https://medium.com/@fernandao.lacerda.dantas/overfitting-and-underfitting-simplified-553a34fcc43?source=rss-3342ed687e10------2</link>
            <guid isPermaLink="false">https://medium.com/p/553a34fcc43</guid>
            <category><![CDATA[machıne]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Fernandao Lacerda Dantas]]></dc:creator>
            <pubDate>Wed, 14 Jun 2023 00:33:06 GMT</pubDate>
            <atom:updated>2023-08-24T01:37:10.951Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overfitting and Underfitting — simplified</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JIbsYS_jkq4siIUV" /></figure><p>Overfitting and underfitting. I’m sure you have already heard those terms while working with machine learning models, but many data scientists don’t know what they mean and that is the reason their models are not having the desired performance. In this article, I will clarify this lingo for you using uncomplicated examples.</p><h4>Overfitting:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*GPdBHgPQEJPrIWx1.jpg" /></figure><p>Let’s say you are a parent teaching your kid basic math. Like every other child, your daughter has a lot of trouble learning introductory topics of mathematics, such as fractions and multiplication. So, you keep showing her many exercises. To optimize her learning ability, you show her different exercises each time you two study together. As time goes on, she gets better and can solve different exercises in no time. And, indeed, she goes pretty well on the school test, achieving a high grade.</p><p>But, what would happen if you kept showing your child similar exercises instead of different exercises? Do you think her grade would be higher or lower?</p><p>The answer is obvious. It would be lower. Solving similar exercises over and over she wouldn’t even be learning, her brain would just find a way to memorize and quickly solve math problems. So, she would struggle when facing different math problems in the school exam. That’s the main point! Doing similar exercises repeatedly will make your kid memorize the steps of solving a math equation. Consequently, they are not retaining any information or learning new ways to deal with math problems. Your kid is even going to memorize the mistakes made while solving the exercises.</p><p>In this case, your daughter is suffering from overfitting. Just like what happened to her, machine learning models can “memorize” patterns and not learn information like how it is supposed to. This is one of the reasons the train-test split method is used. You can’t test your model with the same data you trained it, because, using the math example above, you are using the exactly same exercises.</p><p>Overfitted models perform very well with extremely specific data and have no success operating with new data.</p><h4>Underfitting:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RPl1AhLNEAQR1y1j" /></figure><p>Now, let’s go even further. In this example, you are a busy parent and don’t have time to study with your daughter. However, somehow, on the night before their school test, you manage to study 5 five minutes with her. With such a short time, you can only give her one math exercise. That’s it. Only one.</p><p>How about now? Do you think she will have a good performance on the test the next day?</p><p>Of course not. Your child didn’t get enough exercises to practice and to get full knowledge of the subject. This way, you would be limiting her learning capacity. The understanding of the child is going to be hardly compromised. Her brain won’t make any connection whatsoever due to the lack of given information. As a result, she is going to fail the test.</p><p>If not enough parameters or information is given, the model is suffering from underfitting. The model will not be able to learn advanced metrics of the system and will have poor performance. Again, this is why you should be careful about the train-test split and why you can’t rely on a small percentage of training data.</p><h3>Conclusion:</h3><p>A model suffering from overfitting or underfitting will drastically interfere with your expected results. Always be careful when splitting data for train and test, that is a crucial moment for your model and many complications can make you far from a great machine learning model.</p><p>Now that you gained some insight into the topic, go ahead and try your best to optimize your models!!</p><p>Hope you enjoyed this post! See you in the next article!</p><p>Fernando Dantas</p><p>(@<strong>_fernando_lacerda)</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=553a34fcc43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Poisson Distribution — An introduction]]></title>
            <link>https://medium.com/@fernandao.lacerda.dantas/poisson-distribution-an-introduction-4430f456d02a?source=rss-3342ed687e10------2</link>
            <guid isPermaLink="false">https://medium.com/p/4430f456d02a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[math]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Fernandao Lacerda Dantas]]></dc:creator>
            <pubDate>Sun, 28 May 2023 23:47:59 GMT</pubDate>
            <atom:updated>2023-05-31T22:35:48.073Z</atom:updated>
            <content:encoded><![CDATA[<h3>Poisson Distribution — An introduction</h3><p>The Poisson distribution is a key distribution in statistics and data science, since it is possible to calculate the number of events within a given interval of time or space and there is nothing more important to statistics than to predict a certain value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*Ci8n_43V1DemxWKQkhpahw.jpeg" /><figcaption>Poisson distribution histogram</figcaption></figure><h3>Conditions</h3><p>Before we get into the main topics, the Poisson distribution has to follow some criteria in order to be correctly used:</p><ol><li>The events must be random and independent, meaning the knowledge of an event does not affect the probability of another event.</li><li>The number of occurrences of an event given a certain time or space must be known.</li></ol><p>If a discrete probability distribution follow the criteria above, the Poisson distribution can be used.</p><p>There is no better way to explain the Poisson method than using a real life example.</p><h3>Main Problem</h3><p>Lets say we work on a marketing agency that reach 25000 clients a week and by previous analysis our data team know that the probability of someone buying our product is 0.008%. And we want to calculate the probability of selling <strong>exactly</strong> 3 products on a given week.</p><p>First of all, it is important to tell if this situation can actually be solved by Poisson. The events are random and independent, knowing someone bought your product will not affect the probability of the next costumer buying or not you product. We also have information about the number of occurrences (0.008% every 25000 costumers) in a time span (per week). Thus, the Poisson method can be used to predict the probability of selling exactly 3 products on a week.</p><p>To solve the problem we are going to use formula below. The formula of Poisson distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*CQKIkky2KmEcaI9LOfIVxg.png" /></figure><p>X = is a random variable</p><p>lambda(λ) = mean number of occurrences at an interval</p><p>x = number of occurrences desired</p><p>e = Euler’s number (2.71828)</p><p>The formula seems to be scary, but don’t worry. Lets see how our marketing problem can be solved with this equation.</p><p>For lambda, we have to multiply 0.008% by 25000 so we can have the mean number off purchases in a week, which is 2 in this case. So lambda = 2.</p><p>As the number of occurrences desired we have x = 3, because we want to know the probability of selling exactly 3 products in a week.</p><p>And “e” is a constant (Euler’s number) that can be rounded to 2.71.</p><p>Putting everything together we should get a result of 0.180</p><p>It means that the probability of our marketing agency selling exactly 3 products in a certain week is 0.18 or 18%.</p><h3>Different Question</h3><p>What if we wanted to know the probability of selling <strong>at least </strong>3 products in a week?</p><p>When asking this question, the problem must be approached in a different way. The desired value can be every number equal or greater than 3. Looking at this perspective, the only amount of products that we are not interested are 0, 1 or 2, right? The easiest way to calculate this new situation is to consider 1 as the total probability and subtract the probability of the events that we are not interested (selling 0, 1 or 2 products) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*fEtoBzowwitYUYbBQ3Z4Sg.png" /></figure><p>That is how we should calculate the probability of our marketing agency selling at least 3 products, which is approximately 0.33. See how we approached the problems in different ways according to our needs?</p><h3>Conclusion</h3><p>The Poisson distribution is really relevant in a corporative environment due to its power to “predict” helpful values for a company such as the marketing example above and this is why Poisson is widely used among data experts.</p><p>I showed you the basics of a crucial probability distribution for those who work with data, from now on you should definitely use the Poisson Distribution when handling probability problems.</p><p>Thanks for reading! See you in the next article!</p><p>Fernando Dantas</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4430f456d02a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>