<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ayomide Oyemaja on Medium]]></title>
        <description><![CDATA[Stories by Ayomide Oyemaja on Medium]]></description>
        <link>https://medium.com/@ayomideoyemaja?source=rss-151a976c9387------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GMBjuRfYjT53pvYh2_6uaw@2x.jpeg</url>
            <title>Stories by Ayomide Oyemaja on Medium</title>
            <link>https://medium.com/@ayomideoyemaja?source=rss-151a976c9387------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:43:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ayomideoyemaja/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Text Summarization with NLP: A Simple Approach to Condensing Large Texts.]]></title>
            <link>https://ayomideoyemaja.medium.com/text-summarization-with-nlp-a-simple-approach-to-condensing-large-texts-4f9d51469f7d?source=rss-151a976c9387------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f9d51469f7d</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[text-summarization]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ayomide Oyemaja]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 17:28:16 GMT</pubDate>
            <atom:updated>2025-01-14T00:05:37.257Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjirIJDTGCmwGltnC6ubgA.png" /><figcaption>Natural Language Processing</figcaption></figure><p>The world we live in today is information-driven, managing and quickly understanding large volumes of text is extremely important. Whether it’s news articles, research papers, or reviews, there is always a need to extract key insights from text. This is where <strong>text summarization</strong> comes in — an essential tool in Natural Language Processing (NLP) that can help condense lengthy documents into their most relevant information.</p><p>Traditional methods struggle to adapt to diverse texts and evolving linguistic patterns. The challenge is not just identifying important words, but understanding the context of sentences and how they relate.</p><h3>A Simple NLP-based Approach</h3><p>In this project, I built a <strong>frequency-based text summarization tool</strong> using <strong>Python</strong> and the <strong>Natural Language Toolkit (NLTK)</strong>. The tool identifies the most important sentences in a text and extracts them to generate a summary. Here’s how it works:</p><ol><li><strong>Text Preprocessing</strong>: The text is cleaned by removing punctuation and stopwords (common words like “and” or “the” that don’t add much meaning).</li><li><strong>Word Frequency Analysis</strong>: Each word in the text is analyzed to determine its frequency, which indicates its importance in the context of the document.</li><li><strong>Sentence Scoring</strong>: Sentences are scored based on the frequency of the words they contain. Sentences with higher scores are considered more relevant.</li><li><strong>Summary Generation</strong>: The highest-scoring sentences are selected and combined to form a concise summary.</li></ol><pre>import nltk<br>import string<br>from heapq import nlargest<br><br># Ensure NLTK dependencies are downloaded<br>nltk.download(&#39;stopwords&#39;)<br>nltk.download(&#39;punkt&#39;)<br><br>text = &quot;&quot;&quot;Network Security is an essential aspect of our daily lives nowadays. <br>Classical protection techniques, such as rule-based firewalls and signature-based detection, <br>are limited in many ways. Those methods may need assistance efficiently responding to the <br>continuously increasing demand for protection and the constantly evolving attacking strategies. <br>IDS, applied to network security, helps us maintain, access, and protect our vulnerable information. <br>This paper suggests a machine learning model to efficiently reduce the informational structure of the network <br>and detect the attacks prominently. The Gain Ratio method reduces the data dimension, and Random Forest is <br>applied for classification purposes. We test and train the model with the NSL-KDD data set, and we <br>provide proof that the implementation of the Gain Ratio can help the IDS to successfully recognize <br>malicious traffic by evaluating the reduced structure of the network and detecting the attack.&quot;&quot;&quot;<br><br># Determine the length of the summary<br>if text.count(&quot;. &quot;) &gt; 20:<br>    length = int(round(text.count(&quot;. &quot;) / 10, 0))<br>else:<br>    length = 1<br><br># Removing punctuation<br>nopunc = &quot;&quot;.join([char for char in text if char not in string.punctuation])<br><br># Tokenizing words and removing stopwords<br>stop_words = set(nltk.corpus.stopwords.words(&#39;english&#39;))<br>processed_text = [word.lower() for word in nopunc.split() if word.lower() not in stop_words]<br><br># Calculating word frequencies<br>word_freq = {}<br>for word in processed_text:<br>    word_freq[word] = word_freq.get(word, 0) + 1<br><br># Normalizing word frequencies<br>max_freq = max(word_freq.values())<br>for word in word_freq:<br>    word_freq[word] /= max_freq<br><br># Tokenizing sentences and calculating sentence scores<br>sent_list = nltk.sent_tokenize(text)<br>sent_score = {}<br>for sent in sent_list:<br>    for word in nltk.word_tokenize(sent.lower()):<br>        if word in word_freq:<br>            sent_score[sent] = sent_score.get(sent, 0) + word_freq[word]<br><br># Extracting the top sentences<br>summary_sents = nlargest(length, sent_score, key=sent_score.get)<br>summary = &quot; &quot;.join(summary_sents)<br><br># Print the summary<br>print(&quot;Summary:&quot;)<br>print(summary)</pre><p><strong><em>Original Text:</em></strong><em> Network Security is an essential aspect of our daily lives nowadays. Classical protection techniques, such as rule-based firewalls and signature-based detection, are limited in many ways. Those methods may need assistance efficiently responding to the continuously increasing demand for protection and the constantly evolving attacking strategies. IDS, applied to network security, helps us maintain, access, and protect our vulnerable information. This paper suggests a machine learning model to efficiently reduce the informational structure of the network and detect the attacks prominently. The Gain Ratio method reduces the data dimension, and Random Forest is applied for classification purposes. We test and train the model with the NSL KDD data set, and we provide proof that the implementation of the Gain Ratio can help the IDS to successfully recognize malicious traffic by evaluating the reduced structure of the network and detecting the attack.</em></p><p><strong><em>Summary:</em></strong><em> We test and train the model with the NSL-KDD data set, and we provide proof that the implementation of the Gain Ratio can help the IDS to successfully recognize malicious traffic by evaluating the reduced structure of the network and detecting the attack.</em></p><h3>Why This Approach?</h3><p>By focusing on word frequencies, this method is simple yet effective. It doesn’t rely on complex models or huge datasets. It’s perfect for smaller, less structured texts where the goal is to extract essential information quickly. Using <strong>NLTK</strong> also makes it easy to scale and customize for different languages or types of documents.</p><p>This tool is useful for:</p><ul><li><strong>News Articles</strong>: Quickly summarizing lengthy articles to get the key points.</li><li><strong>Research Papers</strong>: Extracting the abstract or important findings without reading the entire paper.</li><li><strong>Reviews</strong>: Condensing reviews to understand common themes or issues.</li></ul><h3>Conclusion</h3><p>This text summarization tool is an easy-to-implement solution for anyone looking to condense large amounts of text into digestible summaries. By leveraging NLP and Python, the tool efficiently extracts the most important information, saving time and helping you focus on what matters.</p><h3>Libraries Used:</h3><ol><li><strong>NLTK (Natural Language Toolkit)</strong>: A comprehensive library for text processing and NLP tasks, such as tokenization, stopword removal, and sentence parsing.</li><li><strong>string</strong>: A standard Python library for string manipulation, used here to handle and remove punctuation from the text.</li><li><strong>heapq</strong>: A Python library for efficiently finding the largest elements in a list, which is used to select the most relevant sentences based on word frequency.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f9d51469f7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Health Outcomes of Smokers Using Machine Learning: A Study on Hypertension and…]]></title>
            <link>https://ayomideoyemaja.medium.com/analyzing-smokers-health-in-the-us-a-machine-learning-approach-with-regression-and-classification-42f1f68e5c2f?source=rss-151a976c9387------2</link>
            <guid isPermaLink="false">https://medium.com/p/42f1f68e5c2f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression-modeling]]></category>
            <category><![CDATA[classification-models]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ayomide Oyemaja]]></dc:creator>
            <pubDate>Fri, 17 May 2024 16:02:32 GMT</pubDate>
            <atom:updated>2025-01-20T16:48:50.839Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predicting Health Outcomes of Smokers Using Machine Learning: A Study on Hypertension and Cardiovascular Risk Factors.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vRBBR-5ep2ihxq6a" /><figcaption>Photo by <a href="https://unsplash.com/@pawel_czerwinski?utm_source=medium&amp;utm_medium=referral">Pawel Czerwinski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This study leverages machine learning techniques to analyze the health of smokers in the US, utilizing both regression and classification models. By uncovering patterns and predicting health outcomes, the research provides valuable insights into the impact of smoking on public health.</p><h3><strong>Part 1: Regression</strong></h3><p><strong><em>Section A: Data Description</em></strong></p><p>The dataset used for this analysis was filtered from a Hypertension Risk Dataset and it focuses on the health of smokers. It comprises of various factors such as age, sex, blood pressure, and cholesterol levels, aiming to understand their collective impact on heart rate.</p><p><strong>Observational Units:</strong> Each row in the dataset represents an individual.</p><p><strong>Response Variable:</strong></p><ul><li><strong><em>Heart Rate:</em></strong> This is a numeric variable representing the heart rate of each individual.</li></ul><p><strong>Possible Predictor Variables:</strong></p><ul><li><strong><em>Age:</em></strong> Numeric variable indicating the age of each individual.</li><li><strong><em>Sex:</em></strong> Categorical variable indicating the sex of each individual (male/female).</li><li><strong><em>Current Smoker:</em></strong> Categorical variable indicating whether the individual is a current smoker (yes/no).</li><li><strong><em>Systolic Blood Pressure (systolic_bp):</em></strong> Systolic blood pressure of the individual (numeric).</li><li><strong><em>Diastolic Blood Pressure (diastolic_bp):</em></strong> Diastolic blood pressure of the individual (numeric).</li><li><strong><em>Cigarettes Per Day:</em></strong> Numeric variable indicating the number of cigarettes smoked per day by each individual.</li><li><strong><em>Cholesterol:</em></strong> Numeric variable indicating the cholesterol level of each individual.</li><li><strong><em>Number of Observations:</em></strong> The total number of observations in dataset is 3900 observations.</li></ul><p><strong>Statistical Learning Method:</strong></p><p>The dataset includes both numeric and categorical variables, making it suitable for regression analysis. Considering we have a numeric response variable (Heart Rate) and a mix of numeric and categorical predictor variables, I plan to use multiple linear regression for analysis. Multiple linear regression allows us to model the relationship between the response variable and multiple predictor variables simultaneously. It’s useful for understanding the influence of various factors on the response variable.</p><p><strong><em>Section B: Data Pre-Processing &amp; Variable Selection</em></strong></p><p>For the data pre-processing, I cleaned the smokers’ health data. Some of the things i did are:</p><ul><li>Splitting the blood pressure into systolic and diastolic columns to remove the string.</li><li>I removed the original blood pressure column</li><li>I cleaned the dataset by removing the null data by removing the rows with missing data</li><li>I converted the “current_smoker” variables to binary (0 &amp; 1)</li></ul><p>Source Code:</p><pre># Load the dataset<br>data &lt;- read.csv(&quot;smoking_health_data.csv&quot;)<br><br># Separate the blood pressure into systolic and diastolic columns<br>blood_pressure_split &lt;- strsplit(data$blood_pressure, &quot;/&quot;)<br>data$systolic_bp &lt;- as.numeric(sapply(blood_pressure_split, &quot;[&quot;, 1))<br>data$diastolic_bp &lt;- as.numeric(sapply(blood_pressure_split, &quot;[&quot;, 2))<br><br># Remove the original blood pressure column<br>data &lt;- subset(data, select = -blood_pressure)<br><br># Remove rows with missing data<br>clean_data &lt;- na.omit(data)<br><br># Convert &quot;current_smoker&quot; to 0 and 1<br>clean_data$current_smoker &lt;- ifelse(clean_data$current_smoker == &quot;yes&quot;, 1, 0)<br><br># Fit a full model<br>full_model &lt;- lm(heart_rate ~ ., data = clean_data)<br><br># Perform stepwise variable selection<br>selected_model &lt;- step(full_model)</pre><pre>Start:  AIC=19017.17<br>heart_rate ~ age + sex + current_smoker + cigs_per_day + chol + <br>    systolic_bp + diastolic_bp<br><br>                 Df Sum of Sq    RSS   AIC<br>- current_smoker  1      63.3 520178 19016<br>&lt;none&gt;                        520114 19017<br>- chol            1    2039.6 522154 19030<br>- diastolic_bp    1    2100.3 522215 19031<br>- age             1    2354.0 522468 19033<br>- cigs_per_day    1    2599.4 522714 19034<br>- systolic_bp     1    2669.5 522784 19035<br>- sex             1   10742.1 530856 19094<br><br>Step:  AIC=19015.64<br>heart_rate ~ age + sex + cigs_per_day + chol + systolic_bp + <br>    diastolic_bp<br><br>               Df Sum of Sq    RSS   AIC<br>&lt;none&gt;                      520178 19016<br>- chol          1    2035.3 522213 19029<br>- diastolic_bp  1    2068.5 522246 19029<br>- age           1    2430.0 522608 19032<br>- systolic_bp   1    2667.6 522845 19034<br>- cigs_per_day  1    7428.1 527606 19069<br>- sex           1   10891.3 531069 19094<br><br>Call:<br>lm(formula = heart_rate ~ age + sex + cigs_per_day + chol + systolic_bp + <br>    diastolic_bp, data = clean_data)<br><br>Coefficients:<br> (Intercept)           age       sexmale  cigs_per_day          chol  <br>    60.22044      -0.10575      -3.61199       0.12431       0.01709  <br> systolic_bp  diastolic_bp  <br>     0.06625       0.10167 </pre><p>Based on the stepwise regression results, the best model to select for predicting heart rate includes the variables age, sex, cigarettes per day (cigs_per_day), cholesterol (chol), systolic bp &amp; diastolic bp. This model was found to have the lowest Akaike Information Criterion (AIC) of 19015.64.</p><p><strong><em>Section C: Model Setup, Summary &amp; Discussion.</em></strong></p><pre># Fit the model<br>final_model &lt;- lm(heart_rate ~ age + sex + cigs_per_day + chol +<br>systolic_bp + diastolic_bp, data = clean_data)<br><br># View the summary of the model<br>summary(final_model)</pre><pre>Call:<br>lm(formula = heart_rate ~ age + sex + cigs_per_day + chol + systolic_bp + <br>    diastolic_bp, data = clean_data)<br><br>Residuals:<br>    Min      1Q  Median      3Q     Max <br>-31.979  -7.980  -1.154   6.655  66.049 <br><br>Coefficients:<br>             Estimate Std. Error t value Pr(&gt;|t|)    <br>(Intercept)  60.22044    1.71661  35.081  &lt; 2e-16 ***<br>age          -0.10575    0.02487  -4.253 2.16e-05 ***<br>sexmale      -3.61199    0.40116  -9.004  &lt; 2e-16 ***<br>cigs_per_day  0.12431    0.01672   7.436 1.27e-13 ***<br>chol          0.01709    0.00439   3.892 0.000101 ***<br>systolic_bp   0.06625    0.01487   4.456 8.58e-06 ***<br>diastolic_bp  0.10167    0.02591   3.924 8.86e-05 ***<br>---<br>Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1<br><br>Residual standard error: 11.59 on 3872 degrees of freedom<br>Multiple R-squared:  0.07207, Adjusted R-squared:  0.07063 <br>F-statistic: 50.12 on 6 and 3872 DF,  p-value: &lt; 2.2e-16</pre><p><strong>Model Summary</strong></p><ul><li><strong><em>Intercept:</em></strong><em> </em>The intercept term represents the estimated heart rate when all other predictors are zero. In this case, it’s approximately 60.22 beats per minute.</li><li><strong><em>age:</em></strong><em> </em>For each year increase in age, the heart rate decreases by approximately 0.106 beats per minute (bpm), holding other variables constant.</li><li><strong><em>sexmale:</em></strong><em> </em>On average, males have a heart rate approximately 3.612 bpm lower than females, holding other variables constant.</li><li><strong><em>cigs_per_day:</em></strong><em> </em>For each additional cigarette smoked per day, the heart rate increases by approximately 0.124 bpm, holding other variables constant.</li><li><strong><em>chol:</em></strong><em> </em>For each unit increase in cholesterol level, the heart rate increases by approximately 0.017 bpm, holding other variables constant.</li><li><strong><em>systolic_bp:</em></strong><em> </em>For each mmHg increase in systolic blood pressure, the heart rate increases by approximately 0.066 bpm, holding other variables constant.</li><li><strong><em>diastolic_bp:</em></strong><em> </em>For each mmHg increase in diastolic blood pressure, the heart rate increases by approximately 0.102 bpm, holding other variables constant.</li></ul><p>The model has a significant intercept and predictor variables, as indicated by their low p-values (&lt; 0.05). Age, sex, cigarettes per day, cholesterol level, systolic blood pressure, and diastolic blood pressure are all significant predictors of heart rate. The coefficients provide insight into the direction and magnitude of the relationships between predictor variables and heart rate.</p><p><strong>Conclusion</strong></p><p>Overall, this linear regression model provides valuable insights into the factors influencing heart rate in the dataset.</p><h3>Part 2: Classification</h3><p><strong><em>Section A: Data Description</em></strong></p><p>The dataset used for this analysis is the same as Part 1. It is a dataset on the health of smokers.</p><p><strong>Response Variable:</strong></p><p>The response variable i’m using for this analysis is the current smoker. Smoking status of the individual (0 for non-smoker, 1 for smoker)</p><p><strong>Numeric Predictors:</strong></p><ul><li><strong><em>Age:</em></strong><em> </em>Age of the individual (continuous)</li><li><strong><em>Heart_rate:</em></strong><em> </em>Heart rate of the individual (continuous)</li><li><strong><em>Cigs_per_day:</em></strong><em> </em>Number of cigarettes smoked per day (continuous)</li><li><strong><em>Chol:</em></strong><em> </em>Cholesterol level of the individual (continuous)</li><li><strong><em>Systolic_bp:</em></strong><em> </em>Systolic blood pressure of the individual (continuous)</li><li><strong><em>Diastolic_bp:</em></strong><em> </em>Diastolic blood pressure of the individual (continuous)</li></ul><p><strong>Categorical Predictors:</strong></p><ul><li><strong><em>Sex:</em></strong><em> </em>Gender of the individual (categorical — male or female)</li><li><strong><em>Current_smoker:</em></strong><em> </em>Smoking status of the individual (categorical — 0 for non-smoker, 1 for smoker)</li></ul><p><strong>Number of Observations: </strong>3900 observations.</p><p><strong>Statistical Learning Method:</strong></p><p>For this analysis, we intend to use the logistic regression method for binary classification. Logistic regression is a widely used statistical learning method for binary classification tasks, particularly when the response variable is binary and we want to model the probability of an outcome occurring as a function of predictor variables.</p><p><strong><em>Section B: Variable Selection</em></strong></p><p>Variable selection using the feature importance technique with a random forest model. We’ll train a random forest classifier on the dataset and then examine the importance of each predictor variable.</p><pre># Load required libraries<br>library(randomForest)<br><br># Define predictor variables<br>predictors &lt;- c(&quot;age&quot;, &quot;sex&quot;, &quot;heart_rate&quot;, &quot;cigs_per_day&quot;, &quot;chol&quot;,<br>&quot;systolic_bp&quot;, &quot;diastolic_bp&quot;)<br><br># Define response variable<br>response &lt;- &quot;current_smoker&quot;<br><br># Create training data<br>train_data &lt;- clean_data[, c(predictors, response)]<br><br># Train random forest model<br>rf_model &lt;- randomForest(factor(current_smoker) ~ ., data = train_data, ntree = 500)<br><br># Display variable importance<br>var_importance &lt;- importance(rf_model)<br>print(var_importance)</pre><pre>MeanDecreaseGini<br>age                  51.99073<br>sex                  37.96479<br>heart_rate           17.81732<br>cigs_per_day       1769.70192<br>chol                 18.09475<br>systolic_bp          22.44357<br>diastolic_bp         20.22009</pre><p>Based on the Mean Decrease in Gini impurity scores, we can see the relative importance of each predictor variable in predicting the “current_smoker” status. Here’s a brief interpretation:</p><ul><li><strong><em>cigs_per_day:</em> </strong>This variable has the highest importance score, indicating that it is the most influential predictor in determining whether an individual is a current smoker or not. This aligns with common knowledge, as the number of cigarettes smoked per day is strongly associated with smoking behavior.</li><li><strong><em>age: </em></strong>Age also shows significant importance, suggesting that it plays a crucial role in predicting smoking status. This is consistent with the understanding that smoking behavior tends to vary across different age groups.</li><li><strong><em>sex:</em> </strong>Sex (gender) has a moderate importance score, indicating that it contributes to predicting smoking status but to a lesser extent compared to variables like age and number of cigarettes per day.</li><li><strong><em>systolic_bp, diastolic_bp, heart_rate, and chol:</em> </strong>These variables have lower importance scores compared to age, sex, and number of cigarettes per day, but they still contribute to the predictive power of the model.</li></ul><p>Based on these importance scores, we may choose the top predictors with the highest importance scores (e.g. age, sex, and cigs_per_day).</p><pre># Select top predictors<br>selected_predictors &lt;- c(&quot;age&quot;, &quot;cigs_per_day&quot;, &quot;sex&quot;)<br><br># Create training data with selected predictors<br>train_data_selected &lt;- clean_data[, c(selected_predictors, response)]<br>head(train_data_selected)</pre><pre>## age cigs_per_day sex current_smoker<br>## 15 48 0 male 0<br>## 16 58 0 female 0<br>## 17 46 0 female 0<br>## 18 45 0 female 0<br>## 19 37 0 female 0<br>## 20 46 0 female 0</pre><p><strong><em>Section C: Model Building</em></strong></p><p>Now that we have selected the predictor variables, we can proceed with building the classification model using the chosen variables and the “current_smoker” response variable. Given that we have a binary response variable, we use logistic regression.</p><pre># Load required library<br>library(glmnet)<br><br># Split data into training and testing sets<br>set.seed(123) # for reproducibility<br>train_index &lt;- sample(1:nrow(train_data_selected), 0.7 * nrow(train_data_selected))<br>train_set &lt;- train_data_selected[train_index, ]<br>test_set &lt;- train_data_selected[-train_index, ]<br><br># Build logistic regression model<br>logit_model &lt;- glm(current_smoker ~ ., data = train_set, family = &quot;binomial&quot;)<br>logit_model</pre><pre>##<br>## Call: glm(formula = current_smoker ~ ., family = &quot;binomial&quot;, data = train_set)<br>##<br>## Coefficients:<br>## (Intercept) age cigs_per_day sexmale<br>## -20.63259 -0.01116 39.21405 -0.34369<br>##<br>## Degrees of Freedom: 2714 Total (i.e. Null); 2711 Residual<br>## Null Deviance: 3763<br>## Residual Deviance: 2.633e-06 AIC: 8</pre><p><strong>Model Summary</strong></p><p>The logistic regression model has been fitted to the training data. Here are the coefficients for each predictor variable:</p><ul><li><strong><em>Intercept:</em></strong><em> -</em>20.63259</li><li><strong><em>age:</em></strong><em> </em>-0.01116</li><li><strong><em>cigs_per_day:</em></strong><em> </em>39.21405</li><li><strong><em>sexmale:</em></strong><em> </em>-0.34369</li></ul><p>These coefficients represent the log-odds of being a current smoker associated with each predictor variable, holding other variables constant.</p><ul><li>For every one-unit increase in age, the log-odds of being a current smoker decrease by 0.01116.</li><li>For every one-unit increase in cigs_per_day (number of cigarettes per day), the log-odds of being a current smoker increase by 39.21405.</li><li>Being male (sexmale = 1) decreases the log-odds of being a current smoker by 0.34369 compared to being female (sexmale = 0).</li></ul><p>The residual deviance is very close to zero, indicating a good fit of the model to the data. The Akaike Information Criterion (AIC) is also low, suggesting that the model is relatively parsimonious.</p><p><strong><em>Section D: Presentation of the ROC curve and AUC.</em></strong></p><pre># Make predictions on the test set<br>predictions &lt;- predict(logit_model, newdata = test_set, type = &quot;response&quot;)<br>head(predictions)<br><br># Calculate ROC curve and AUC<br>library(pROC)<br><br>roc_curve &lt;- roc(test_set$current_smoker, predictions)<br><br>auc_value &lt;- auc(roc_curve)<br><br># Plot ROC curve<br>plot(roc_curve, main = &quot;ROC Curve&quot;, col = &quot;blue&quot;, lwd = 2)<br>legend(&quot;bottomright&quot;, legend = paste(&quot;AUC =&quot;, round(auc_value, 2)), col = &quot;blue&quot;, lwd = 2)<br><br># Calculate AUC<br>print(paste(&quot;AUC:&quot;, round(auc_value, 2)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KsxSSIe7SCe2sY8O8Wg28Q.png" /></figure><p>An AUC value of 1 indicates a perfect classifier; that is, the model can perfectly distinguish between smokers and non-smokers in the test set. This is an excellent result and suggests that the logistic regression model is highly accurate in predicting smoking status based on the selected predictor variables.</p><h3><strong>Conclusion</strong></h3><p>In conclusion, the analysis of the “Smoker’s Health” dataset using logistic regression has provided valuable insights into the factors influencing smoking behavior. Through the exploration of predictor variables such as age, gender, and number of cigarettes per day, we have developed a robust model capable of accurately predicting an individual’s likelihood of being a current smoker.</p><p>The logistic regression model demonstrated exceptional performance, achieving a perfect Area Under the ROC Curve (AUC) of 1 on the test set. This indicates that the model can effectively distinguish between smokers and non-smokers based on the selected predictors. The coefficients of the model provided valuable insights into the relationship between predictor variables and smoking status, highlighting the importance of age, number of cigarettes per day, and gender in predicting smoking behavior.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42f1f68e5c2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving Averages in Time Series Analysis: Understanding Trends & Forecasting.]]></title>
            <link>https://ayomideoyemaja.medium.com/moving-averages-in-time-series-analysis-understanding-trends-forecasting-28b4c0a9d146?source=rss-151a976c9387------2</link>
            <guid isPermaLink="false">https://medium.com/p/28b4c0a9d146</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[moving-average]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ayomide Oyemaja]]></dc:creator>
            <pubDate>Thu, 16 May 2024 18:30:12 GMT</pubDate>
            <atom:updated>2024-05-16T18:52:36.525Z</atom:updated>
            <content:encoded><![CDATA[<p>In the field of data analysis, especially in the time series domain, we often face a problem of distinguishing actual trends from short term fluctuations and noise. This is where the concept of moving averages is employed as a tool to smooth out irregular data and unveil underlying patterns.</p><p>In this article, we look at moving averages, it’s significance, types of moving averages and practical applications in analyzing time series data. Understanding the concept of moving averages can significantly enhance one’s ability to grasp insights from datasets whether as a professional data scientist or just an enthusiast.</p><h3><strong>Moving Averages &amp; Concept of Smoothing Data.</strong></h3><p>A moving average process is a technique in time series analysis that is used to predict long-term trends from a data set while “smoothing out” short-term fluctuations. It solves a problem we face when dealing with time series data: it differentiates spikes from an actual trend. Moving averages are essentially nonparametric methods for trend (or signal) estimation and are not for model building. Moving averages are useful for several reasons: Smoothing Data, Identifying Trends, Forecasting.</p><p>The concept of smoothing data involves reducing the noise or flunctuations in a dataset to reveal and highlight the underlying patterns or trends more clearly. An example of where smoothing of data is done is stock price prediction analysis, there are random events in the market that can cause a spike in the chart or graph. What the moving average process does is to smoothen these spikes and give us a smoother curve representation of the data.</p><h3><strong>Types of Moving Averages</strong></h3><p>There are different types of moving averages but the three most common used for analysis are:</p><ol><li><strong>Simple Moving Average (SMA):</strong> This is the simplest form of calculating moving averages most commonly used for smoothing data. It is calculated by simply taking the average of a series of data points over a time period (n). For example, a 5-SMA is the average of the data points in the last 5 days. The SMA gives equal weight to each data points in a dataset. One limitation of SMA is that it is less responsive to recent data changes since every data point in the window has an equal impact on the average.</li><li><strong>Exponential Moving Average (EMA):</strong> The exponential moving average is an alternative to the simple moving average. The EMA assigns greater weight and significance on the most recent data points unlike the SMA that assigns equal weight to each data point in the dataset. This means that more recent data points contribute more to the EMA than older ones and the weights decrease exponentially as we move back in time.</li><li><strong>Weighted Moving Average (WMA):</strong> Unlike the SMA which assigns equal weight to the data points in a dataset, WMA gives more importance to recent data points and assigns different weights to each data point in the period.</li></ol><p><strong>SMA</strong> might be better suited for long-term trends where you need a stable and less noisy signal. <strong>EMA</strong> is ideal when needing a more dynamic measure that adapts quickly to new data, useful in short-term trading or variable. analysis. <strong>WMA</strong> allows for customization and can be tailored to specific analytical needs, making it suitable for complex strategies that rely heavily on the recency of data.</p><h3><strong>Practical Application: Analyzing Airline Passenger Data in R.</strong></h3><p>To demonstrate moving averages in a real-world scenario, we will be using the AirPassengers dataset, which contains monthly totals of international airline passengers from 1949 to 1960. We’ll show how to compute SMAs, EMAs, and WMAs using this dataset, alongside visualizations to uncover underlying trends.</p><p><strong>Simple Moving Average (SMA)</strong></p><p>For the first analysis, we compute the SMA for the AirPassengers dataset. The SMA provides a straightforward method of smoothing out fluctuations by calculating the average of data points over a specified period. In our case, we’ll compute the SMA for a 12-month period to capture yearly trends. I will be demonstrating the manual &amp; automatic methods.</p><p><strong>Source Code:</strong></p><pre># The manual way<br><br># Simple Moving Average<br>calculate_SMA &lt;- function(x, period) {<br>sma &lt;- numeric(length(x))<br>for (i in period:length(x)) {<br>sma[i] &lt;- mean(x[(i - period + 1):i])<br>}<br>return(sma)<br>}<br><br># Define the period for SMA<br>sma_period &lt;- 12<br><br># Calculate SMA<br>sma_values &lt;- calculate_SMA(AirPassengers, sma_period)<br>sma_values<br><br>#The automatic way<br><br>#Load the required library<br>library(zoo)<br>data(&quot;AirPassengers&quot;)<br><br># Calculate SMA for a 12-month period<br>sma &lt;- rollmean(AirPassengers, k = 12, fill = NA)<br>sma<br><br># Plot original data along with SMA<br>plot(AirPassengers, type = &quot;l&quot;, col = &quot;blue&quot;,<br>main = &quot;Simple Moving Average for Airline Passengers&quot;,<br>ylab = &quot;Passengers&quot;)<br>lines(sma, col = &quot;red&quot;)<br>legend(&quot;topleft&quot;, legend = c(&quot;Original&quot;, &quot;SMA&quot;), col = c(&quot;blue&quot;, &quot;red&quot;),<br>lty = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qF-COYa-N8Y0ipJWCicpSA.png" /></figure><p><em>From the image above, we can see the difference between the plot for the original data and the simple moving average data. The SMA Smooths out fluctuations in the data, providing a clearer view of long-term trends but may miss out on short-term fluctuations.</em></p><p><strong>Exponential Moving Average (EMA)</strong></p><p>Next, we explore the Exponential Moving Average (EMA), which assigns exponentially decreasing weights to data points, placing greater emphasis on recent observations. This makes EMAs more responsive to changes in the data compared to SMAs.</p><p><strong>Source Code:</strong></p><pre># The manual way<br><br># Exponential Moving Average (EMA)<br>calculate_EMA &lt;- function(x, period) {<br>  alpha &lt;- 2 / (period + 1)<br>  ema &lt;- numeric(length(x))<br>  ema[period] &lt;- mean(x[1:period])<br>  for (i in (period + 1):length(x)) {<br>    ema[i] &lt;- alpha * x[i] + (1 - alpha) * ema[i - 1]<br>  }<br>  return(ema)<br>}<br><br># Define the period for EMA<br>ema_period &lt;- 12<br><br># Calculate EMA<br>ema_values &lt;- calculate_EMA(AirPassengers, ema_period)<br>ema_values<br><br><br># The automatic way<br><br># Load the required library<br>library(TTR)<br><br># Load the airpassengers dataset<br>data(&quot;AirPassengers&quot;)<br><br># Calculate Exponential Moving Average (EMA)<br>ema &lt;- EMA(AirPassengers, n = 12)<br>ema<br><br># Plot original data along with EMA<br>plot(AirPassengers, type = &quot;l&quot;, col = &quot;blue&quot;, <br>     main = &quot;Exponential Moving Average for Airline Passengers&quot;, <br>     ylab = &quot;Passengers&quot;)<br>lines(ema, col = &quot;black&quot;)<br><br>legend(&quot;topleft&quot;, legend = c(&quot;Original&quot;, &quot;EMA&quot;), col = c(&quot;blue&quot;, &quot;black&quot;), <br>       lty = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XsRuD343hizV5HF0inLKmA.png" /></figure><p><em>The Exponential Moving Average balances between capturing short-term fluctuations and long-term trends due to its exponential weighting scheme.</em></p><p><strong>Weighted Moving Average (WMA)</strong></p><p>Finally, we look at the Weighted Moving Average (WMA), which allows for customized weighting of data points to tailor the smoothing process according to specific requirement or to our taste.</p><p>Source Code:</p><pre># The manual way<br><br># Weighted Moving Average (WMA)<br>calculate_WMA &lt;- function(x, weights) {<br>  wma &lt;- numeric(length(x))<br>  for (i in max(weights):length(x)) {<br>    wma[i] &lt;- sum(x[(i - max(weights) + 1):i] * weights) / sum(weights)<br>  }<br>  return(wma)<br>}<br><br># Define the period and weights for WMA<br>wma_period &lt;- 12<br>weights &lt;- 1:wma_period<br><br># Calculate WMA<br>wma_values &lt;- calculate_WMA(AirPassengers, weights)<br>wma_values<br><br># The automatic way<br><br># Load the required library<br>library(zoo)<br><br># Load the airpassengers dataset<br>data(&quot;AirPassengers&quot;)<br><br># Define custom weights for WMA<br>weights &lt;- c(0.1, 0.15, 0.2, 0.1, 0.05, 0.05, 0.1, 0.1, 0.1, 0.05, 0.1, 0.1)<br><br># Calculate Weighted Moving Average (WMA) using filter function<br>wma &lt;- stats::filter(AirPassengers, weights, sides = 1)<br>wma<br><br># Plot original data along with WMA<br>plot(AirPassengers, type = &quot;l&quot;, col = &quot;blue&quot;, <br>     main = &quot;Weighted Moving Average for Airline Passengers&quot;, <br>     ylab = &quot;Passengers&quot;)<br>lines(wma, col = &quot;orange&quot;)<br><br>legend(&quot;topleft&quot;, legend = c(&quot;Original&quot;, &quot;WMA&quot;), col = c(&quot;blue&quot;, &quot;orange&quot;), <br>       lty = 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTzP6sw32anod6EU2hv7Tg.png" /></figure><p><em>By assigning distinct weights to each data point, WMAs offer a flexible approach to trend analysis, also allowing for customization.</em></p><p>Beyond computing and visualizing moving averages, analysts often go deeper into the insights provided by these smoothed representations of data. Some additional steps in the analysis might include:</p><ol><li><strong>Trend Identification:</strong> Once the moving averages have been calculated, analysts can dig deeper into the trends they reveal. This involves checking whether there are consistent upward or downward trends in the data over time and identifying any significant seasonal patterns or recurring cycles.</li><li><strong>Forecasting:</strong> Moving averages can also be a foundation for forecasting future trends. Analysts leverage historical moving average values to project future data points, considering the consistency of the underlying trends.</li><li><strong>Comparison with Other Indicators:</strong> It’s also common to compare moving averages with other models to validate findings or get other insights. For example, analysts might compare moving averages with trend lines or other regression models fitted to the data.</li></ol><h3>Conclusion</h3><p>In this article, we’ve explored the concept of moving averages and their practical applications in time series analysis. By smoothing out erratic data points and uncovering trends, moving averages provide analysts with valuable insights into datasets across various fields.</p><p>Whether it’s the simplicity of Simple Moving Averages (SMAs), the responsiveness of Exponential Moving Averages (EMAs), or the customization of Weighted Moving Averages (WMAs), each method offers unique advantages that can be used to uncover meaningful patterns in data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28b4c0a9d146" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>