<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chidimma Prisca Ogu on Medium]]></title>
        <description><![CDATA[Stories by Chidimma Prisca Ogu on Medium]]></description>
        <link>https://medium.com/@chidimma-p-ogu?source=rss-ac5b91b8667c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PuZhsfBXXhgnXDT0Dh9BuQ.png</url>
            <title>Stories by Chidimma Prisca Ogu on Medium</title>
            <link>https://medium.com/@chidimma-p-ogu?source=rss-ac5b91b8667c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:22:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chidimma-p-ogu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CLASSIFICATION MODELS]]></title>
            <link>https://chidimma-p-ogu.medium.com/classification-models-e274690aa772?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e274690aa772</guid>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Tue, 28 May 2024 19:53:43 GMT</pubDate>
            <atom:updated>2024-05-28T19:53:43.614Z</atom:updated>
            <content:encoded><![CDATA[<p>These models help identify what group a data point point belong to. For example, if we were to decide whether a person qualifies for a loan or not, we need to look at the person’s loan application and classify whether the person should be in the approved loan group or the not a, how likely a customer is to renew their contract, which product a prospective customer is going to buy.</p><p>Other examples of classification problems include:</p><blockquote>Classifying an email as spam or sending it to inbox</blockquote><blockquote>Does a soy bean have a disease or not.</blockquote><blockquote>What type of transportation will employees take to work</blockquote><h4>Classification problems are of two types: Binary(two) and non-binary problems (more than two)</h4><h4>Binary Classification:</h4><p>In binary classification, the task is to categorize data into one of two classes or categories. These classes are often referred to as positive and negative, or class 1 and class 0. The goal is to build a model that can accurately predict the class label of new instances based on their features. Examples of binary classification tasks include:</p><ul><li>Predicting whether an email is spam (class 1) or not spam (class 0).</li><li>Identifying whether a tumor is malignant (class 1) or benign (class 0) based on medical imaging data.</li><li>Classifying whether a transaction is fraudulent (class 1) or legitimate (class 0) in financial data.</li></ul><p>The models used in binary classification are</p><blockquote>Logistic Regression</blockquote><blockquote>Decision Trees</blockquote><h4>Non-binary Classification:</h4><p>Non-binary classification, also known as multi-class classification, involves categorizing data into more than two classes or categories. Each data point can belong to one and only one class out of multiple possible classes. Examples of non-binary classification tasks include:</p><ul><li>Classifying images of animals into categories such as cats, dogs, and birds.</li><li>Predicting the genre of a book from its text, such as fiction, non-fiction, mystery, etc.</li><li>Identifying different types of activities in human motion data, such as walking, running, or cycling.</li></ul><p>The models used in non binary classification are</p><blockquote>Random forests models</blockquote><blockquote>Boosted models</blockquote><h3>Binary classification</h3><p>In binary classification, checking whether all the predictor variables will affect the target variable is an important step. This involves several steps, including exploratory data analysis (EDA), correlation analysis, and feature selection. Here’s a detailed guide on how to perform this check:</p><h4>1. Exploratory Data Analysis (EDA)</h4><p>Before diving into the analysis, you should understand your data by performing EDA. This includes:</p><ul><li>Summary Statistics: Look at the mean, median, standard deviation, and other summary statistics for each variable.</li><li>Data Visualization: Use histograms, box plots, and scatter plots to visualize the distribution of the variables and their relationships with the target variable.</li></ul><h4>2. Correlation Analysis</h4><p>Calculate the correlation matrix to see the linear relationship between the predictor variables and the target variables.</p><h4>3. Feature Selection</h4><p>Use feature selection techniques to identify which predictor variables have a significant effect on the target variable. Some common methods include:</p><p><strong>A. Statistical Tests</strong></p><ul><li>Chi-Square Test: For categorical variables.</li><li>ANOVA (Analysis of Variance): For continuous variables</li></ul><p><strong>B. Recursive Feature Elimination (RFE)</strong></p><p>RFE is a feature selection method that fits a model and removes the weakest features until the specified number of features is reached.</p><p><strong>C. Tree-based Methods</strong></p><p>Tree-based methods can be used to determine feature importance. Tree-based methods are versatile and powerful for determining feature importance. They help in understanding the impact of different features on the target variable, which is essential for feature selection and model interpretation. Decision Trees are best for binary classification models.</p><h4>Binary classification models</h4><p>I stated above that there are models used for binary classification.</p><h3><strong>Logistic regression</strong></h3><blockquote>Logistic regression is one of the most basic forms of regression modeling. It’s part of a family of “generalized linear models” or GLM for short. This basically means that the formula is very similar to that of a linear regression. However since the target variable is binary, instead of a continuous numeric variable, the target variable has to be modified to fit this GLM formula. Logistic regression is a statistical method for analyzing datasets in which there are one or more independent variables that determine an outcome. The outcome is typically binary (yes/no, success/failure, win/lose), which means it has only two possible values.</blockquote><h4>How Logistic Regression Works</h4><p>Logistic regression estimates the probability that a given input point belongs to a certain class. It uses the logistic function, also known as the sigmoid function, to squeeze the output of a linear equation between 0 and 1. This probability can then be used to assign a binary class label.</p><p>The Logistic Function<br>The logistic function is defined as:​</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*CDeNn5U0OOwBa2jKtlEsKg.png" /></figure><h4><strong>Advantages</strong></h4><p><strong>Simplicity:</strong> Logistic regression is easy to implement and understand.<br><strong>Efficiency:</strong> It requires less computational power compared to more complex models.<br><strong>Interpretability:</strong> The model coefficients can be interpreted as the impact of the corresponding features on the log odds of the outcome.<br><strong>Probability Estimates:</strong> Provides probabilities for each class, which can be useful for decision-making.<br><strong>Feature Importance: </strong>Coefficients indicate the relative importance of features.</p><h4>Disadvantages</h4><p><strong>Linear Boundary:</strong> Assumes a linear relationship between the independent variables and the log odds of the outcome, which may not hold true for all problems.<br><strong>Overfitting:</strong> Can overfit if there are too many features, especially with limited data.<br><strong>Not Suitable for Non-linear Problems: </strong>Performs poorly when the decision boundary is non-linear.<br><strong>Sensitive to Outliers:</strong> Outliers can affect the model’s performance.</p><h4>Use Cases</h4><p>Logistic regression is widely used in various fields for binary classification problems. Some common use cases include:</p><p><strong>Healthcare:</strong> Predicting whether a patient has a particular disease based on diagnostic measurements.<br><strong>Finance:</strong> Credit scoring, predicting whether a loan applicant will default.<br>Marketing: Customer segmentation, predicting whether a customer will respond to a marketing campaign.<br><strong>Human Resources:</strong> Employee attrition, predicting whether an employee will leave the company.<br><strong>E-commerce:</strong> Fraud detection, predicting whether a transaction is fraudulent.</p><p>To check the performance of a logistic regression model,</p><blockquote>Check the AIC and BIC values</blockquote><blockquote>the r-squared value</blockquote><p>Logistic regression is a versatile and fundamental algorithm for binary classification. Understanding the model’s structure, transformations like the log-log transformation, and methods for feature selection like stepwise regression are crucial for building robust predictive models. Evaluating these models using appropriate metrics ensures their reliability and effectiveness in real-world applications. For more in-depth information on logistic regression parameters.</p><p>To read more on Logistic regression. Check out my article on the Parameters in Logistic regression</p><h3>Decision Trees:</h3><blockquote>A decision tree is a powerful machine learning algorithm that functions like a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one of the most popular and understandable methods for both classification and regression tasks.</blockquote><h4>Structure of Decision Trees</h4><ol><li>Root Node: This is the topmost node of the tree, representing the entire dataset. It is split into two or more homogeneous sets.</li><li>Decision Nodes: These are sub-nodes that split into further sub-nodes or leaf nodes based on certain conditions.</li><li>Leaf/Terminal Nodes: These nodes represent the outcome or decision and do not split further. In classification, they represent class labels, while in regression, they represent continuous values.</li><li>Branches: The connections between nodes that represent the decision rules or outcomes from the decision nodes.</li></ol><h4>How Decision Trees Work</h4><ol><li>Splitting: The process of dividing a node into two or more sub-nodes based on a certain criterion.</li><li>Pruning: The process of removing sub-nodes of a decision node, which reduces the complexity of the model and helps in avoiding overfitting.</li><li>Tree Depth: The length of the path from the root to the leaf. A deeper tree can capture more information about the data but might overfit.</li></ol><h4>Advantages of Decision Trees</h4><ul><li>Simplicity and Interpretability: Decision trees are easy to understand and visualize. They can be represented in a flowchart-like structure.</li><li>Handling of Non-linear Relationships: They can model complex, non-linear relationships in the data.</li><li>No Need for Feature Scaling: Decision trees do not require normalization or scaling of data.</li><li>Versatility: Can handle both numerical and categorical data, and can be used for both classification and regression tasks.</li></ul><h4>Disadvantages of Decision Trees</h4><ul><li>Overfitting: Decision trees can easily overfit the training data, especially if they are too deep. Pruning and setting a maximum depth can help mitigate this.</li><li>Bias towards Features with More Levels: Features with more levels can dominate the tree’s structure.</li><li>Instability: Small changes in the data can result in a completely different tree. This can be addressed by techniques like bagging and boosting.</li></ul><h4>Use Cases of Decision Trees</h4><ol><li>Classification Problems:</li></ol><ul><li>Customer Segmentation: Classifying customers into different segments based on their purchasing behavior.</li><li>Credit Scoring: Determining the likelihood of a customer defaulting on a loan.</li><li>Medical Diagnosis: Identifying diseases based on symptoms and medical test results.</li></ul><p>2. Regression Problems:</p><ul><li>Price Prediction: Predicting house prices based on various features like location, size, and amenities.</li><li>Sales Forecasting: Estimating future sales based on historical sales data and other predictors.</li></ul><p>3. Feature Importance: Decision trees can rank the importance of features based on how they contribute to the decision-making process. This is useful for feature selection in building other machine learning models.</p><p>4. Anomaly Detection: Identifying unusual patterns in the data, such as fraud detection in financial transactions.</p><p>Decision trees are a fundamental machine learning technique that is easy to understand and interpret. They are widely used in various domains for both classification and regression tasks. However, care must be taken to avoid overfitting, and techniques such as pruning, ensemble methods, and cross-validation should be used to build robust models.</p><p>When building a decision tree model. To check its performance you</p><blockquote>Plot a leaf summary</blockquote><blockquote>Root node error</blockquote><h3>NON-BINARY CLASSIFICATION</h3><p>Non-binary classification, also known as multi-class classification, refers to a type of predictive modeling where the task is to classify instances into one of three or more classes. This is different from binary classification, where the task is to classify instances into one of two classes.</p><h4>RANDOM FOREST</h4><h4>Structure</h4><p>Random Forest is an ensemble learning method that constructs multiple decision trees during training and outputs the mode of the classes (classification) or mean prediction (regression) of the individual trees. It is built on the principle of “the wisdom of crowds,” leveraging the power of multiple models to improve performance and robustness.</p><h4>How It Works</h4><ol><li>Bootstrap Aggregation (Bagging): Random Forest uses bagging, where each tree is trained on a random subset of the training data sampled with replacement. This process helps in reducing variance and overfitting.</li><li>Random Feature Selection: For each split in a tree, a random subset of features is chosen. This introduces more diversity among the trees, further reducing the likelihood of overfitting.</li><li>Tree Building: Each tree in the forest is grown to the largest extent possible, without pruning. The splitting criterion can be Gini impurity or entropy for classification and mean squared error for regression.</li><li>Aggregation: In classification, the final output is the mode of the classes predicted by individual trees. In regression, the output is the average of the predictions made by each tree.</li></ol><h4>Advantages</h4><ol><li>Reduced Overfitting: By averaging multiple trees, the model reduces the risk of overfitting compared to individual decision trees.</li><li>High Accuracy: Random Forest generally provides high accuracy and robustness compared to many other models.</li><li>Feature Importance: The model provides insights into feature importance, helping in feature selection and understanding the dataset.</li><li>Versatility: It can handle both classification and regression tasks effectively.</li><li>Handling Missing Values: It can handle missing values well by estimating them during the training process.</li><li>Robustness to Noise: The random selection of features and bootstrap sampling makes it robust to noise in the dataset.</li></ol><h4>Disadvantages</h4><ol><li>Complexity: The model can be computationally intensive and requires more memory and processing power due to the multiple trees.</li><li>Interpretability: While it provides feature importance, the overall model is less interpretable compared to a single decision tree.</li><li>Training Time: Training can be slower for large datasets with many trees.</li><li>Prediction Time: Making predictions can be slower because each new input needs to be passed through all the trees.</li></ol><p>When building a Random Forest model. To check its performance you</p><blockquote>Cross validate the model</blockquote><h4>Boosted Model</h4><h4>Structure</h4><p>Boosting is an ensemble technique that sequentially combines multiple weak learners, typically decision trees, to form a strong predictive model. Each subsequent tree in the ensemble corrects the errors of the previous ones, aiming to improve the model’s performance.</p><h4>How It Works</h4><ol><li>Initialization: The process begins with an initial prediction. For regression, this is usually the mean of the target variable. For classification, it’s the initial class probabilities.</li><li>Sequential Learning: Trees are added one at a time, and each new tree focuses on reducing the errors (residuals) made by the preceding trees. In each iteration, the model adjusts the weights of the training instances based on the errors of the previous model, giving more importance to misclassified instances (for classification) or higher residuals (for regression).</li><li>Tree Building: Each tree is trained to predict the residuals or errors of the previous model. These trees are generally shallow, meaning they have limited depth to avoid overfitting.</li><li>Aggregation: The final prediction is the weighted sum (for regression) or majority vote (for classification) of the individual tree predictions, where each tree’s contribution is scaled by a learning rate parameter.</li><li>Learning Rate: A key hyperparameter, the learning rate, scales the contribution of each tree. Lower learning rates require more trees but can lead to better performance and generalization.</li></ol><h4>Advantages</h4><ol><li>High Accuracy: Boosted models often achieve high predictive accuracy by focusing on hard-to-predict instances.</li><li>Reduced Bias: By sequentially correcting errors, boosting reduces bias and improves model performance on the training data.</li><li>Flexibility: Can be used for both classification and regression tasks with various loss functions tailored to specific needs.</li><li>Handles Complex Data: Capable of capturing complex relationships in the data due to the sequential correction of errors.</li><li>Feature Importance: Provides insights into feature importance, helping in understanding the dataset and feature selection.</li></ol><h4>Disadvantages</h4><ol><li>Overfitting: If not properly regularized, boosted models can overfit the training data, especially with too many trees or high learning rates.</li><li>Computationally Intensive: Training can be slow and resource-intensive due to the sequential nature of the process and the need to train multiple trees.</li><li>Sensitivity to Outliers: Boosting can be sensitive to outliers as it tries to correct every instance, which might lead to overfitting.</li><li>Complexity: The model is less interpretable compared to single decision trees due to the ensemble of many trees.</li></ol><p>When building a Boosted model. To check its performance you</p><blockquote>Cross validate the model</blockquote><h3>Model Evaluation</h3><p>To evaluate a classification model, you should compare the model’s performance against a holdout sample using the following metrics:</p><ol><li>Accuracy: The ratio of correctly predicted instances to the total instances. It gives a general idea of how well the model performs overall.</li><li>Precision: The ratio of true positive predictions to the total predicted positives. It measures how many of the predicted positives are actually positive.</li><li>Recall (Sensitivity): The ratio of true positive predictions to the actual positives. It measures how well the model captures all the positives.</li><li>F1 Score: The harmonic mean of precision and recall. It provides a single metric that balances both precision and recall.</li><li>AUC-ROC (Area Under the Receiver Operating Characteristic Curve): Measures the ability of the model to distinguish between classes. The higher the AUC, the better the model.</li><li>Confusion Matrix: A table showing the true positives, true negatives, false positives, and false negatives. It provides a detailed breakdown of model performance.</li></ol><p>Additionally, you should evaluate the following visual metrics:</p><ol><li>ROC Curve: Plot the true positive rate (sensitivity) against the false positive rate (1-specificity). Check which model’s curve reaches the top-left corner the quickest, indicating better performance.</li><li>Gain Chart: Similar to the ROC curve, it helps identify which model gains the highest true positives with the least false positives the fastest.</li><li>Calibration Curve: Assesses how well the predicted probabilities of the positive class align with the actual outcomes. A perfectly calibrated model would have a calibration curve along the diagonal line.</li></ol><p>In summary, when comparing models, look for:</p><ul><li>Higher accuracy, precision, recall, and F1 score.</li><li>Higher AUC-ROC scores.</li><li>A ROC curve that quickly reaches the top-left corner.</li><li>Well-calibrated probability predictions.</li></ul><p>By using these metrics and visual evaluations, you can thoroughly assess the performance of your classification models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e274690aa772" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guide to Classification Models: Selection Criteria and Use Cases]]></title>
            <link>https://chidimma-p-ogu.medium.com/guide-to-classification-models-selection-criteria-and-use-cases-a58ca469d1bb?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a58ca469d1bb</guid>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Tue, 28 May 2024 19:49:42 GMT</pubDate>
            <atom:updated>2024-05-29T12:53:29.462Z</atom:updated>
            <content:encoded><![CDATA[<p>The size of data, specifically the number of rows and columns, play a crucial role in determining the appropriate classification models and their use cases.</p><h3>Small Data Sets</h3><p>Number of Rows: &lt; 1,000<br>Number of Columns: &lt; 20</p><p>Models to Use:</p><ol><li>Logistic Regression : This is used in simple binary classification problems, such as predicting yes/no outcomes.</li><li>k-Nearest Neighbors (k-NN) : This is used for problems where you need a model that’s easy to understand and interpret.</li><li>Decision Trees: This is used in simple classification tasks where interpretability and understanding the decision process is important.</li></ol><h3>Medium Data Sets</h3><p>Number of Rows: 1,000–100,000<br>Number of Columns: 20–100</p><p>Models to Use:</p><ol><li>Random Forest: This is used for complex classification tasks, such as medical diagnoses or credit scoring, where accuracy is important.</li><li>Support Vector Machines (SVM): This is used for classification problems with clear margin of separation, such as image classification or text categorization.</li><li>Gradient Boosting Machines (GBM): This is used for highly competitive predictive tasks, such as Kaggle competitions or marketing response prediction.</li><li>Gaussian Naive Bayes: This is used primarily for continuous data and in generating super fast results. Example of its applications is in Medical Diagnosis and real-time prediction.</li></ol><h3>Large Data Sets</h3><p>Number of Rows: 100,000–1,000,000<br>Number of Columns: 100–1,000</p><p>Models to Use:</p><ol><li>XGBoost: This is used in high-performance classification tasks, such as click-through rate prediction or fraud detection.</li><li>Neural Networks (Deep Learning): This is used for complex and high-dimensional data problems, such as speech recognition or image classification.</li><li>LightGBM: This is used for large-scale and high-performance applications, such as web search ranking or large-scale industrial tasks.</li><li>Multinomial Naive Bayes: This is ideal for text classification, the type of Data used here is discrete data, particularly useful for text classification where word counts are used as features like sentiment analysis.</li></ol><h3>Very Large Data Sets</h3><p>Number of Rows: &gt; 1,000,000<br>Number of Columns: &gt; 1,000</p><p>Models to Use:</p><ol><li>Deep Learning (e.g., Convolutional Neural Networks for images, Recurrent Neural Networks for sequences): This is used for extremely large and complex datasets, such as natural language processing or extensive image databases.</li><li>Distributed Machine Learning Frameworks (e.g., Spark MLlib, TensorFlow on distributed systems): This is used for scalable machine learning tasks across distributed computing environments, such as large-scale recommendation systems or social network analysis.</li></ol><h3>Suitability For Binary And Non-Binary Classification</h3><p>Classification models can be categorized based on their suitability for binary and non-binary (multi-class) classification problems. Here’s a detailed breakdown:</p><h4>Binary Classification Models</h4><p>Binary classification involves distinguishing between two classes. The following models are particularly well-suited for binary classification:</p><ol><li>Logistic Regression: This is used for simple binary outcomes, such as spam detection or churn prediction.</li><li>Support Vector Machines (SVM):This is used for binary problems with clear margin of separation, such as image classification with two categories or binary text classification.</li><li>k-Nearest Neighbors (k-NN): This is used for simple and interpretable binary classification tasks, such as medical diagnosis or binary customer segmentation.</li><li>Decision Trees: This is used in situations requiring easy-to-understand decision rules, such as loan approval or binary medical decisions</li><li>Random Forest: This is used for complex binary classification tasks where high accuracy is needed, such as fraud detection or credit scoring.</li><li>Gradient Boosting Machines (GBM): This is used for competitive binary predictive tasks, such as marketing response prediction or high-stakes financial modeling.</li><li>XGBoost: This is used for high-performance binary classification tasks, like click-through rate prediction or binary classification in large datasets.</li><li>Neural Networks (Deep Learning): This is used for complex binary classification tasks with large amounts of data, such as image recognition or sentiment analysis.</li><li>Naive Bayes: This is used for multi-class problems with high-dimensional data, such as text classification or spam filtering in multiple categories.</li></ol><h4>Non-Binary (Multi-Class) Classification Models</h4><p>Multi-class classification involves distinguishing between more than two classes. Many binary classification models can be extended to handle multi-class problems, Some non-binary classification problems and their use cases are:</p><ol><li>Multinomial Logistic Regression: Problems with more than two outcomes, such as document classification or multi-class sentiment analysis.</li><li>Support Vector Machines (SVM) with One-vs-Rest (OvR) or One-vs-One (OvO) strategies: Multi-class problems with clear separations between classes, such as multi-class image classification or multi-category text classification.</li><li>Decision Trees: Interpretable multi-class classification tasks, such as multi-category product recommendation or multi-class diagnostic tasks.</li><li>Random Forest: Multi-class problems requiring robustness and accuracy, such as multi-class fraud detection or multi-class customer segmentation.</li><li>Gradient Boosting Machines (GBM): Competitive multi-class predictive tasks, such as ranking or recommendation systems with multiple classes.</li><li>XGBoost: High-performance multi-class classification tasks, such as multi-class categorization in large datasets.</li><li>Neural Networks (Deep Learning): Complex multi-class classification tasks, such as image classification with many categories (e.g., CIFAR-100) or language processing tasks with multiple classes.</li><li>k-Nearest Neighbors (k-NN): Simple multi-class classification tasks where the interpretability of the model is important.</li><li>Naive Bayes: Multi-class problems with high-dimensional data, such as text classification or spam filtering in multiple categories.</li></ol><h3>Other Key Considerations</h3><ol><li>Data Sparsity: For sparse data, such as text or high-dimensional categorical data, models like Naive Bayes or specialized deep learning architectures may be more appropriate.</li><li>Computation Resources: Models like deep neural networks and ensembles of trees (e.g., Random Forest, GBM) are resource-intensive and require more computational power.</li><li>Interpretability: Simpler models (e.g., Logistic Regression, Decision Trees) are easier to interpret, which is crucial for domains requiring explainability, such as healthcare or finance.</li><li>Scalability: Models like Random Forest, GBM, XGBoost, and deep learning are suitable for large datasets and can handle both binary and multi-class problems.</li></ol><p>Choosing the right model depends on the specific requirements of the classification task, the nature of the data, and the computational resources available. Failure to choose the right model can significantly affect the performance, errors, and scores of the model. It can influence issues like overfitting and underfitting.</p><blockquote>Overfitting occurs when a model learns the noise and details in the training data to the extent that it negatively impacts the model’s performance on new data. A model that is too complex for the given dataset might fit the training data very well but perform poorly on unseen data. For instance, using a deep neural network on a small dataset might result in overfitting, where the model captures random noise in the training data.</blockquote><blockquote>Underfitting happens when a model is too simple to capture the underlying patterns in the data. The model fails to capture important patterns and relationships, leading to poor performance on both training and test data. For instance, using a linear regression model for a complex, non-linear classification task might result in underfitting.</blockquote><blockquote>Accuracy and Error Rates : Different models may yield varying levels of accuracy and error rates depending on how well they match the data and task requirements. Inappropriate model choice can lead to higher error rates and lower accuracy, reducing the model’s effectiveness. For instance, Naive Bayes models might perform well on text classification tasks with independence assumptions but poorly on datasets with strong feature dependencies.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a58ca469d1bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PRE TRAINED DEEP LEARNING MODELS]]></title>
            <link>https://chidimma-p-ogu.medium.com/pre-trained-deep-learning-models-8e5c4c9e7cb4?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e5c4c9e7cb4</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pretrained-model]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Thu, 23 May 2024 16:07:06 GMT</pubDate>
            <atom:updated>2024-05-23T16:07:06.395Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pretrained models refer to machine learning models that have been trained on large amounts of data and are then made available for use without the need for further training on specific tasks. Here is a list of popular pretrained models across different domains:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*3y9Cvrdw7Sckvdq2mxYGCA.png" /></figure><h3>Natural Language Processing (NLP) Models:</h3><ol><li>BERT (Bidirectional Encoder Representations from Transformers):</li></ol><p>Variants: BERT base, BERT large, e.t.c</p><p>Developed by Google, BERT has achieved state-of-the-art performance on various NLP tasks.</p><p>2. GPT (Generative Pretrained Transformer):</p><p>Variants: GPT, GPT-2, GPT-3, etc.</p><p>Developed by OpenAI, GPT models use transformer-based architectures for language generation and understanding tasks.</p><p>3. RoBERTa (Robustly optimized BERT approach):</p><ul><li>Similar to BERT but with modifications to training objectives and hyperparameters for improved performance.</li></ul><p>4. XLNet:</p><ul><li>An autoregressive language model that combines autoregressive and autoencoding methods.</li></ul><p>5. DistilBERT:</p><ul><li>A distilled version of BERT that retains much of its language understanding capabilities but with fewer parameters and faster inference.</li></ul><p>6. T5 (Text-To-Text Transfer Transformer):</p><ul><li>Developed by Google, T5 is a transformer-based model that converts all NLP tasks into a text-to-text format.</li></ul><h3>Computer Vision Models:</h3><ol><li>ResNet (Residual Neural Network):</li></ol><p>Variants: ResNet-50, ResNet-101, etc.</p><ul><li>Deep convolutional neural network architectures known for their effectiveness in image classification and feature extraction.</li></ul><p>2. VGG (Visual Geometry Group):</p><ul><li>Variants: VGG-16, VGG-19, etc.</li><li>Convolutional neural network architectures with a focus on depth and simplicity.</li></ul><p>3. Inception (GoogLeNet):</p><ul><li>Variants: Inception V1, Inception V2, Inception V3, etc.</li><li>Developed by Google, Inception models use multiple parallel convolutional layers to capture different aspects of the input image.</li></ul><p>4. EfficientNet: A family of convolutional neural networks that achieve state-of-the-art accuracy with fewer parameters and computations.</p><h3>Speech Recognition Models:</h3><ol><li>DeepSpeech: Developed by Mozilla, DeepSpeech is an open-source speech-to-text engine based on deep learning techniques.</li><li>Kaldi: An open-source toolkit for speech recognition that includes pretrained models and tools for building speech recognition systems.</li></ol><h3>Reinforcement Learning Models:</h3><ol><li>OpenAI Gym Models: Pretrained reinforcement learning models available through the OpenAI Gym environment, such as Atari games agents.</li></ol><p>These are just a few examples of pretrained models across different domains. The availability of pretrained models continues to grow rapidly with advancements in deep learning research and open-source contributions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e5c4c9e7cb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment Analysis: How to Handle Labeled and Unlabeled Text Data]]></title>
            <link>https://chidimma-p-ogu.medium.com/sentiment-analysis-5b9e0deb819e?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b9e0deb819e</guid>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[sentiment-analysis-python]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Thu, 23 May 2024 15:52:05 GMT</pubDate>
            <atom:updated>2024-05-24T10:46:46.718Z</atom:updated>
            <content:encoded><![CDATA[<p>Handling labeled and unlabeled text data effectively is crucial for building robust sentiment analysis models. This guide explores strategies and best practices for managing both types of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*3y9Cvrdw7Sckvdq2mxYGCA.png" /></figure><p>Sentiment analysis, also known as opinion mining, is the process of computationally analyzing text data to discern the sentiment expressed within it. The primary goal is to determine the emotional tone conveyed by the text, which can range from positive to negative, or even neutral. This analysis is crucial in understanding public opinion, customer feedback, social media conversations, and other forms of textual communication.</p><p>Before performing sentiment analysis, one must first:</p><h4>Preprocess the text data</h4><p>Preprocessing text data is crucial for cleaning and preparing text for analysis. Common preprocessing steps include:</p><ul><li>Language translation</li><li>Removing non-alphanumeric symbols,</li><li>Convert all text to lowercase to ensure consistency.</li><li>Tokenization: Split the text into individual words or tokens.</li><li>Removing stop words: Remove common words (e.g., “the”, “is”, “and”) that don’t contribute much to the meaning of the text.</li><li>Removing Punctuation and Special Characters: Strip out punctuation marks, websites, empty spaces and other non-alphabetic characters.</li><li>Stemming or Lemmatization: Reduce words to their base forms (e.g., “running” to “run”) to normalize the text.</li></ul><p>Sentiment analysis can be performed on</p><blockquote>Labelled text data</blockquote><blockquote>Unlabelled text data</blockquote><p><strong>Labelled text data: </strong>This consists of text samples that have been assigned specific labels or categories. In the context of sentiment analysis, these labels typically indicate the sentiment of the text, such as “positive,” “negative,” or “neutral.”</p><p>Example: A dataset of movie reviews where each review is labelled as either positive or negative.</p><pre>[<br>    {&quot;review&quot;: &quot;The movie was fantastic!&quot;, &quot;sentiment&quot;: &quot;positive&quot;},<br>    {&quot;review&quot;: &quot;I did not enjoy the film at all.&quot;, &quot;sentiment&quot;: &quot;negative&quot;},<br>    {&quot;review&quot;: &quot;It was an okay movie.&quot;, &quot;sentiment&quot;: &quot;neutral&quot;}<br>]</pre><p>Another example will be customer reviews on a product accompanied with a rating.</p><pre>[<br>    {&quot;review&quot;: &quot;This product exceeded my expectations! The quality is superb.&quot;, &quot;rating&quot;: 5},<br>    {&quot;review&quot;: &quot;Not what I expected. The material feels cheap.&quot;, &quot;rating&quot;: 2},<br>    {&quot;review&quot;: &quot;It&#39;s okay for the price, but I&#39;ve seen better.&quot;, &quot;rating&quot;: 3},<br>    {&quot;review&quot;: &quot;Absolutely love it! Will buy again.&quot;, &quot;rating&quot;: 5},<br>    {&quot;review&quot;: &quot;Terrible experience. The product broke after one use.&quot;, &quot;rating&quot;: 1}<br>]</pre><p><strong>Unlabelled text data: </strong>This consists of text samples that do not have any associated labels. In sentiment analysis, this means the texts have no predefined sentiment categories.</p><p>Example: A collection of social media posts without any sentiment labels</p><pre>[<br>    &quot;I just watched the latest episode of the series.&quot;,<br>    &quot;Feeling great after the morning workout!&quot;,<br>    &quot;Why is the weather so gloomy today?&quot;<br>]</pre><h3>Labelled text data</h3><p>In performing sentiment analysis on labelled text data, there are three approaches that can be considered.</p><blockquote><strong>Lexicon-based analysis</strong></blockquote><blockquote><strong>Supervised Machine learning (ML)</strong></blockquote><blockquote><strong>Pre-trained transformer-based deep learning models</strong></blockquote><h4>Lexicon-based Analysis</h4><p>Lexicon-based analysis relies on predefined dictionaries or lexicons containing words and their corresponding sentiment scores (e.g., positive, negative, neutral). Words in the text are matched against the entries in the lexicon, and the sentiment scores are aggregated to determine the overall sentiment of the text.</p><p>There are a number of lexicon and rule-based sentiment analysis tool included in the Natural Language Toolkit (NLTK) library in Python namely:</p><ol><li>TextBlob: This is a Python library built on top of NLTK and provides a simple API for common natural language processing (NLP) tasks, including sentiment analysis. It uses a lexicon-based approach similar to NLTK Vader but offers additional functionalities like part-of-speech tagging and noun phrase extraction.</li><li>Pattern: This is a Python library that includes modules for web mining, natural language processing, machine learning, and more. It provides a sentiment analysis module capable of analyzing sentiments in text using a lexicon-based approach.</li><li>AFINN: This is a list of English words manually rated for valence with an integer between -5 (negative) and +5 (positive). It’s available as a Python library and can be used for simple lexicon-based sentiment analysis.</li><li>SentiWordNet: This is a lexical resource for opinion mining and sentiment analysis. It assigns sentiment scores to words based on their senses (synsets) in WordNet. The Python NLTK library provides an interface to access SentiWordNet.</li><li>VADER Sentiment: The Valence Aware Dictionary and sentiment Reasoner is a lexicon and rule-based sentiment analysis tool included in the NLTK library, as mentioned earlier. It’s specifically designed for analyzing sentiments expressed in social media texts.</li><li>Liu Hu Lexicon: Liu Hu Lexicon is a lexicon-based sentiment analysis tool that assigns a polarity score to words and phrases based on their semantic orientation. It’s commonly used for sentiment analysis in Chinese text but can also be adapted for English sentiment analysis.</li></ol><p><strong>Use Cases: </strong>Lexicon-based analysis is useful when you have limited labeled data or when you need transparency and interpretability in sentiment analysis. It’s commonly used in social media monitoring, brand sentiment analysis, and customer feedback analysis.</p><h4>Supervised Machine Learning (ML)</h4><p>Machine learning approaches for sentiment analysis involve using supervised machine learning techniques to train models on labeled datasets to learn the relationship between text features and sentiment labels. Common ML algorithms used include logistic regression, support vector machines (SVM), random forests, and gradient boosting.</p><p>After data preprocessing, the features of the data are extracted from text data by creating bag-of-words, TF-IDF (Term Frequency-Inverse Document Frequency), word embeddings, or other advanced representations.</p><p><strong>Use Cases:</strong> ML-based sentiment analysis is versatile and can be applied in various domains such as product reviews, social media sentiment analysis, customer feedback analysis, and sentiment classification of news articles.</p><h4>Pre-trained Transformer-based Deep Learning models</h4><p>Transformer-based NLP models like BERT (Bidirectional Encoder Representations from Transformers) and its variants, GPT, XLNet have revolutionized natural language processing tasks, including sentiment analysis. These models are pre-trained on large corpora of text data and fine-tuned on labeled sentiment analysis datasets to predict sentiment labels.</p><p>They capture contextual information and dependencies between words in a sentence, leading to better performance, especially for complex language patterns. They can be fine-tuned to understand the sentiment of nuanced phrases like “not bad” or “mixed feelings.”</p><p><strong>Use Cases:</strong> Transformer-based models are beneficial for sentiment analysis tasks where context and semantics play a crucial role, such as analyzing customer reviews, social media sentiment, and sentiment detection in user-generated content. They are also suitable for multilingual sentiment analysis tasks.</p><h3>Unlabelled text data</h3><p>In performing sentiment analysis on unlabelled text data, there are several approaches that can be considered.</p><blockquote><strong>Lexicon-based and Rule-based Systems</strong></blockquote><blockquote><strong>Unsupervised Sentiment Analysis Models</strong></blockquote><h4>Lexicon-based and Rule-based Systems</h4><p>Lexicon-based methods such as VADER or SentiWordNet assign sentiment scores to words in unlabeled text data, which are then aggregated at the document level to provide an overall sentiment score. Rule-based systems utilize predefined linguistic rules or patterns to infer sentiment. These approaches are useful in scenarios where labeled data is limited or unavailable.</p><h4>Unsupervised Sentiment Analysis Models</h4><p>Unsupervised learning algorithms like autoencoders or generative adversarial networks (GANs) are trained on unlabeled text data to learn latent representations capturing sentiment information. These models can predict sentiment in new, unlabeled data. For instance, training a GAN on a corpus of customer reviews can generate synthetic data with learned sentiment characteristics, aiding in sentiment analysis tasks where labeled data is scarce or expensive to obtain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b9e0deb819e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[API Data Retrieval: Generating a YouTube API key]]></title>
            <link>https://chidimma-p-ogu.medium.com/api-data-retrieval-generating-a-youtube-api-key-b8a84e0ee0a3?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8a84e0ee0a3</guid>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Tue, 07 May 2024 16:12:02 GMT</pubDate>
            <atom:updated>2024-05-08T07:15:27.250Z</atom:updated>
            <content:encoded><![CDATA[<p>This a step by step procedure on how to generate a YouTube API key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*3y9Cvrdw7Sckvdq2mxYGCA.png" /></figure><p>Go to the Google Cloud Console (<a href="https://console.cloud.google.com/">https://console.cloud.google.com/</a>)</p><p>Navigate to the dropdown list labeled “My First Project” at the top.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*btj6nzo6sIg5AFPtoGLIvA.png" /></figure><p>Click on it to get the interface below. Click on “NEW PROJECT” at the top right corner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*I-4J3OFiFH8ll3yhfjNuTg.png" /></figure><p>Rename the new project as you please and CREATE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*JUJUerwtKf_cDvxd3vQIZw.png" /></figure><p>Next, return to the homepage and click on the dropdown list at the top. Select the specific project name from the list until you reach the desired interface below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yv0oRX1mAOu0l5oMlXIxaA.png" /></figure><p>Click on “APIs &amp; Services” in the menu.</p><p>Select “ENABLE APIS &amp; SERVICES” from the dropdown.</p><p>Search for “YouTube Data API v3” in the library.</p><p>Click on the API to select it.</p><p>Click the “ENABLE” button to enable the YouTube Data API v3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Jb4If2cnh_Bhm0YKz0T3Q.png" /></figure><p>Click on Credentials on the left corner</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bQiLaGLQQlrCNq109qF_IQ.png" /></figure><p>Next, click on “+ CREATE CREDENTIALS” and then select “API Key” to generate a new API key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5_VNT9P5H_OafclVGqBzg.png" /></figure><p>The API key has been successfully generated. Please copy it and proceed with its implementation.</p><p>For more information on extract data using the API key, read the documentation below: <a href="https://developers.google.com/youtube/v3/docs">https://developers.google.com/youtube/v3/docs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8a84e0ee0a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LEVERAGING COMMUNITY BUILDING IN YOUR DATA SCIENCE PURSUIT]]></title>
            <link>https://chidimma-p-ogu.medium.com/leveraging-community-building-in-your-data-science-pursuit-as-an-introvert-c964535371e9?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c964535371e9</guid>
            <category><![CDATA[introvert]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[community]]></category>
            <category><![CDATA[breaking-into-tech]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Tue, 24 Oct 2023 11:43:38 GMT</pubDate>
            <atom:updated>2024-05-07T15:34:22.564Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>This is strictly for those trying to break into Data Science. Your fresh graduates without industry experience most of whom did not study data science, data-related, or even STEM courses. As well as professionals who are trying to make the switch to Data Analysis/ Data Science.</strong></h4><p>First things first, have a LinkedIn account, you must do this to have access to resources and Industry professionals and an idea of what is required of you to break in. Now, when I say build your community I am not referring to following every industry expert you see on LinkedIn and connecting with them hoping they’ll give you the time of the day. Mind you, I have nothing against that, if it works for you that is marvelous. All I am saying is that it is not the only way to go. I am particular about this because while venturing into a new field, I kept getting all the “Send cold emails”, and “You have to jump into it” messages and that threw me off and made me want to quit before I started. I am here to tell you that you do not need all that.</p><p>What you do need is a community of fellow learners and Mentors if you are fortunate to have one. Do not wait for recommendations from your bosses or mentors when you can get them from your peers, endorse and recommend each other, share resources for growth with them, and solve problems together. This is vital because as the saying goes:</p><blockquote>Information is key</blockquote><p>Solve projects together, and get into other communities as well. Some of the connections I made from being part of some internships connected me to other opportunities. A connection I made during an internship directed me to the Udacity course which <strong>“insisted”</strong> I made a post about the <strong>#60daysofUdacity</strong> a learning challenge which in turn made me make my first LinkedIn post and broke that initial barrier of posting on LinkedIn. I have a couple of other experiences but this stood out.</p><p>As the saying goes:</p><blockquote>Start where you are, use what you have, do what you can</blockquote><p>There’s no greater truth in that. As long as you are putting in the work, you will excel. And always remember beautiful things take time so be patient with yourself on this search. Always rooting for you.</p><p>Refer to my article: <a href="https://chidimma-p-ogu.medium.com/breaking-into-tech-how-to-escape-tutorial-hell-86222ff66683"><strong>BREAKING INTO TECH: HOW TO ESCAPE TUTORIAL HELL</strong></a> for some resources and communities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c964535371e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BREAKING INTO TECH: HOW TO ESCAPE TUTORIAL HELL]]></title>
            <link>https://chidimma-p-ogu.medium.com/breaking-into-tech-how-to-escape-tutorial-hell-86222ff66683?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/86222ff66683</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[data-science-courses]]></category>
            <category><![CDATA[data-science-projects]]></category>
            <category><![CDATA[breaking-into-tech]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Wed, 04 Oct 2023 11:24:23 GMT</pubDate>
            <atom:updated>2023-10-04T11:24:23.871Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bnEQBu9fPdsnZs4yg1scCw.jpeg" /><figcaption>Image: Prisca Ogu</figcaption></figure><p>How do you break into Data Science or Tech in general? The general answer is to do your research pick a niche, and buckle down till you get a substantial knowledge of the subject matter to apply for jobs. Now, the question remains. How much knowledge is substantial knowledge? It is at this point that people are at risk of what I term:</p><blockquote><strong>Being stuck in Tutorial Hell</strong></blockquote><p>To break the cycle of continuous learning without a metric to gauge the impact on my retained knowledge, I implemented the following strategies:</p><h4>Participated in Virtual Internships:</h4><p>I took proactive steps to secure virtual internships in the field of Data Science, emphasizing my dedication and commitment to this learning journey. Despite being unpaid, these internships played a crucial role in boosting my confidence as they were project-oriented. What made these internships even more valuable was the opportunity to gain industry-like experience without the pressure of being in an actual workplace.</p><p>One memorable aspect of these internships was the exposure to new tools and technologies. For instance, during one internship, I discovered GitHub, a platform I had no prior knowledge of. Understanding its existence and significance was a revelation. Subsequently, I continued to explore and learn about various other tools and concepts in a similar manner.</p><p>These internships not only provided practical experience but also opened doors to a world of knowledge and skills that I might not have encountered otherwise. They were instrumental in shaping my learning journey in Data Science.</p><p>Some of the virtual internships I participated in are.</p><ul><li><a href="https://www.theforage.com/"><strong>Forage</strong></a><strong>: </strong>This consists of internships from renowned organizations like PWC, ACCENTURE, BANK OF AMERICA, KPMG, etc. It is packed with step-by-step projects that explain each stage in the Data Science roadmap.</li><li><a href="https://www.entrylevel.net/"><strong>Entry Level</strong></a><strong>: </strong>This is a project-based six-week training that focuses on the particular skill you want to learn.</li><li><a href="https://www.hamoye.com/"><strong>Hamoye Data Science Internship</strong></a></li></ul><p><strong>Note:</strong> The aforementioned virtual internships are certified and are great for building your resume, especially for Entry-level candidates who have no experience to show. The projects can be used to build a portfolio to showcase your technical expertise. There are other resources out there. Feel free to add them in the comments for a wider reach to those starting.</p><h4>Participated in challenges involving the skills I wanted to build:</h4><p>Another step you can take in building confidence is participating in online challenges. You can easily find these on LinkedIn, promotional emails, and Google search. Some of the challenges I participated in are.</p><ul><li><a href="https://onyxdata.co.uk/data-dna-dataset-challenge/"><strong>The Onyx Data Visualization Challenge on LinkedIn</strong></a></li><li><a href="https://collaborator.omdena.com/"><strong>Omdena Challenges</strong></a></li></ul><h4><strong>Applied for Tech Scholarships:</strong></h4><p>Technology scholarships significantly contribute to confidence-building, as they go beyond merely working on projects to obtaining certifications from reputable organizations. Several notable tech scholarships include:</p><ul><li><a href="https://www.udacity.com/"><strong>UDACITY NANO DEGREES</strong></a><strong>: </strong>Numerous tech scholarships are available, thanks to Udacity’s collaborations with various organizations. Some prominent ones include the Bertelsmann Scholarship and the Access Bank scholarship, among others. What makes Udacity scholarships especially valuable is their ability to facilitate certifications through specific platforms. For instance, students who completed the <strong>Predictive Analytics for Business Nanodegree</strong> were provided with the opportunity to obtain <strong>Alteryx certification</strong>, while those who completed the <strong>Nanodegree in Data Analysis and Visualization with Microsoft Power BI</strong> could access the <strong>Microsoft Power BI certification</strong>, all at no cost.</li><li><a href="https://grow.google/intl/ie/"><strong>GOOGLE CERTIFICATIONS</strong></a><strong>: </strong>The <strong>Grow with Google</strong> program, in partnership with<strong> Umuzi</strong>, provides a diverse selection of certifications. These courses are hosted on <strong>Coursera</strong>, offering hands-on, project-based learning experiences. The remarkable aspect is that these courses are available at no cost, making them accessible to a wider audience. Upon successful completion of these courses, participants receive certification, highlighting their mastery of the subject matter and adding significant value to their skills.</li><li><a href="http://dataquest.io/"><strong>DATA QUEST</strong></a><strong>: </strong>Dataquest stands out as an excellent platform for delving into data science, particularly due to its unique feature of providing a simulated environment directly on your online profile. This eliminates the need for users to worry about finding the perfect laptop specifications and related concerns. While most courses on Dataquest come with a cost, occasional scholarships are available, making it worthwhile for individuals seeking cost-effective and practical data science education. It is highly recommended to explore Dataquest for its innovative learning environment and the potential opportunities for financial assistance through scholarships.</li><li><strong>AWS: </strong>AWS offers an annual scholarship for Cloud practitioners. It’s highly recommended to explore this opportunity, especially if you are interested in building expertise in cloud technologies. Additionally, it’s advisable to check for other available scholarships within the AWS ecosystem and beyond.</li></ul><p><strong>Note:</strong> Many organizations, educational institutions, and tech companies provide scholarships to support individuals in pursuing careers in Data Science and upskilling in technology at large. Regularly checking for such opportunities can be beneficial for those looking to advance their skills and knowledge in the rapidly evolving tech field.</p><h4>Joined communities aligned with my interests:</h4><p>The last but not the least is joining communities in line with your interests. There are a number of them that are prominent. Do well to do your research for those that suit your needs. Communities I recommend are:</p><ul><li><a href="https://shecodeafrica.org/"><strong>She Code Africa</strong></a></li><li><a href="https://internship.zuri.team/"><strong>Zuri Internship</strong></a></li><li><a href="https://ingressive.org/tag/internship/"><strong>Ingressive for Good</strong></a></li></ul><p>And always remember beautiful things take time, so be patient with yourself whilst on this learning journey. Always rooting for you.</p><p>If you have other resources that can help, kindly add them in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86222ff66683" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL TIPS 1: Getting Started with SQL]]></title>
            <link>https://chidimma-p-ogu.medium.com/sql-tips-1-getting-started-with-sql-4625836e9fdc?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4625836e9fdc</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-analytics-tools]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Sun, 17 Sep 2023 20:05:48 GMT</pubDate>
            <atom:updated>2024-02-27T14:37:29.767Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*1mkud2TKOFZcNXooEoTQHA.jpeg" /></figure><p>Not your conventional SQL learning tips, this is a beginner’s level hands-on SQL tip. It is a simple breakdown of the key things to take note of while working with SQL.</p><p>Note: The environment used in this publication is Postgres SQL.</p><h3>WHAT IS SQL?</h3><h4>STRUCTURED QUERY LANGUAGE</h4><p>SQL is designed for working with databases. It allows you to write queries that the computer executes and provide database insights in return.</p><p>In the world of data analysis, one of the first things we encounter when dealing with databases is the concept of unique identifiers — the primary key and the foreign key. These keys play a pivotal role in maintaining data integrity and relationships between tables within a database.</p><p><strong>Primary Key (PK):</strong> A primary key is a unique identifier for each record in a table. It ensures that each row within a table is distinct, and no two rows share the same primary key value. This uniqueness allows for efficient retrieval of data and establishes a solid foundation for defining relationships with other tables.</p><p><strong>Foreign Key (FK):</strong> On the other hand, a foreign key is a field in one table that refers to the primary key in another table. It creates a link between tables, establishing relationships and enforcing referential integrity. Foreign keys are used to maintain consistency in the data and ensure that data in related tables remains synchronized.</p><p>Now that we have a fundamental understanding of primary and foreign keys, let’s delve into essential SQL statements:</p><p><strong>COMMENTING IN SQL</strong></p><ul><li><strong>“- -”</strong>: Single-line comments</li><li><strong>/* — — — — */</strong>: Multiple line comments</li></ul><h4>FUNDAMENTAL SQL STATEMENTS</h4><p><strong>CREATE DATABASE and CREATE TABLE:</strong></p><ul><li><strong>CREATE DATABASE</strong> is used to create a new database in SQL. It provides a logical container for storing tables, views, and other database objects.</li></ul><pre>CREATE DATABASE database_name;</pre><ul><li><strong>CREATE TABLE</strong> is used to define the structure of a table within a database. It specifies the columns, data types, constraints, and primary/foreign keys.</li></ul><pre>CREATE TABLE table_name (<br>    column1 datatype,<br>    column2 datatype,<br>    column3 datatype,<br>    ...<br>);</pre><p><strong>INSERT INTO:</strong></p><p>The <strong>INSERT INTO</strong> statement is used to add new records (rows) to a table. It requires specifying the target table and values for each column.</p><pre>INSERT INTO table_name (column1, column2, column3, ...)<br>VALUES (value1, value2, value3, ...);</pre><p><strong>COPY (.txt, .csv) for a table in a database:</strong></p><p>The <strong>COPY</strong> command is used to bulk load data from a text or CSV file into a table in a database. It’s an efficient way to import large datasets.</p><pre>COPY table_name FROM &#39;file_path&#39; WITH CSV;<br><br>/*OR*/<br><br>COPY my_table FROM &#39;/path/to/mydata.csv&#39; WITH CSV;</pre><p><strong>SELECT:</strong></p><ul><li><strong>SELECT (*) — wildcard Character: </strong>This returns all the values in a table. It retrieves data from all columns for all rows in that table. It’s a convenient way to grab all available data when needed.</li><li><strong>SELECT DISTINCT in PostgreSQL: </strong>This statement fetches unique values from specific columns in a table. This operation removes duplicate rows from the result set.</li></ul><p><strong>WHERE:</strong></p><p>The<strong> WHERE</strong> clause in SQL is used to filter rows from a table or result set based on a specified condition or set of conditions. It allows you to narrow down the rows returned by a query, only including those that meet the specified criteria. It is similar to using the <strong>IF </strong>statement in Excel. The basic syntax is given by :</p><pre>SELECT column1, column2, ...<br>FROM table_name<br>WHERE condition;</pre><h4><strong>LOGICAL OPERATORS (AND|OR|BETWEEN|NOT)</strong></h4><p>This is used for writing queries that have multiple conditions. They are called conjunctive operators.</p><ul><li><strong>AND: </strong>The <strong>AND</strong> operator is used to combine multiple conditions, and it returns true only if all the conditions it joins are true.</li></ul><pre>SELECT * FROM employees <br>WHERE department = &#39;Sales&#39; AND salary &gt; 50000;</pre><ul><li><strong>OR: </strong>The <strong>OR</strong> operator is used to combine multiple conditions, and it returns true if at least one of the conditions it joins is true.</li></ul><pre>SELECT * FROM orders <br>WHERE order_status = &#39;Shipped&#39; OR order_status = &#39;Delivered&#39;;</pre><ul><li><strong>BETWEEN: </strong>The <strong>BETWEEN</strong> operator is used to specify a range for a column’s value. It’s inclusive, meaning it includes both the start and end values.</li></ul><pre>SELECT * FROM products <br>WHERE price BETWEEN 10.00 AND 50.00;</pre><ul><li><strong>NOT: </strong>The <strong>NOT</strong> operator is used to negate a condition. It reverses the logical result of a condition.</li></ul><pre>SELECT * FROM products <br>WHERE price NOT BETWEEN 10.00 AND 50.00;</pre><p><strong>UPDATE</strong>:</p><p>The <strong>UPDATE</strong> statement in SQL is used to modify existing records in a table. It allows you to change the values of one or more columns in one or more rows based on a specified condition.</p><pre>UPDATE table_name<br>SET column1 = new_value1, column2 = new_value2, ...<br>WHERE condition;</pre><p><strong>DELETE:</strong></p><p>The <strong>DELETE </strong>query is used to remove existing records in a table. This is the same as a <strong>SELECT </strong>query with conditions.</p><pre>DELETE FROM table_name<br>WHERE condition;</pre><p><strong>ALTER:</strong></p><p>This is used to change the structure of a table. You can add/drop columns using the alter query. It can be used to change the column type and column name. It can also be used to add/delete constraints.</p><ul><li><strong>Adding a Column: </strong>To add a new column to an existing table, use the <strong>ALTER TABLE</strong> statement with the <strong>ADD</strong></li></ul><pre>ALTER TABLE table_name<br>ADD column_name datatype;</pre><ul><li><strong>Dropping a Column: </strong>To remove a column from an existing table, use the <strong>ALTER TABLE</strong> statement with the <strong>DROP COLUMN</strong> clause:</li></ul><pre>ALTER TABLE table_name<br>DROP COLUMN column_name;</pre><ul><li><strong>Changing Column Data Type: </strong>To modify the data type of an existing column, use the <strong>ALTER TABLE</strong> statement with the <strong>ALTER COLUMN</strong> clause:</li></ul><pre>ALTER TABLE table_name<br>ALTER COLUMN column_name TYPE new_datatype;</pre><ul><li><strong>Renaming a Table: </strong>To change the name of an existing table, use the <strong>ALTER TABLE</strong> statement with the <strong>RENAME TO</strong> clause:</li></ul><pre>ALTER TABLE old_table_name<br>RENAME TO new_table_name;</pre><ul><li><strong>Adding Constraints: </strong>To add constraints (e.g., primary key, foreign key, not null, check if a column &gt; 0, etc.) to an existing table, use the <strong>ALTER TABLE</strong> statement with the <strong>ADD CONSTRAINT</strong> clause:</li></ul><pre>ALTER TABLE table_name<br>ADD CONSTRAINT constraint_name constraint_type (column_name);</pre><ul><li><strong>Removing Constraints: </strong>To remove constraints from an existing table, use the <strong>ALTER TABLE</strong> statement with the <strong>DROP CONSTRAINT</strong> clause:</li></ul><pre>ALTER TABLE table_name<br>DROP CONSTRAINT constraint_name;</pre><h4>SELECTION COMMANDS</h4><p><strong>IN:</strong></p><ul><li>The<strong> IN </strong>operator<strong> </strong>is used to help reduce the need to use multiple OR conditions in a <strong>SELECT, INSERT, UPDATE,</strong> or <strong>DELETE</strong> statement.</li></ul><pre>SELECT *<br>FROM  table_name<br>WHERE column_name IN (value1, value2, ..., valueN);</pre><p><strong>BETWEEN:</strong></p><ul><li>The <strong>BETWEEN</strong> operator is used to filter rows based on a range of values. It checks whether a value falls within a specified range, inclusive of both the lower and upper bounds. i.e. it is used whenever you can use the mathematical expression <strong>5≤x≥10</strong>.</li></ul><pre><br>SELECT *<br>FROM table_name<br>WHERE column_name BETWEEN low_value AND high_value;</pre><p><strong>LIKE:</strong></p><p>The <strong>LIKE </strong>operator is used for <strong>pattern matching</strong> within text data. It allows you to search for strings that match a specified pattern. P.S. Pattern matching implies finding if a string is starting with a particular character or ending with a particular set of characters. With the <strong>LIKE </strong>you perform pattern matching with wildcards.</p><pre>SELECT *<br>FROM table_name<br>WHERE column_name LIKE &#39;string&#39;;</pre><p><strong>The wildcards are of two types:</strong></p><h4><strong>1. “%”</strong>: This allows you to match strings of any length including 0.</h4><p><strong>SAMPLE USE CASES</strong></p><ul><li><strong>“A%”</strong>: In this case, it means match patterns that start with <strong>‘A’ </strong>like Adam, Apple,</li><li><strong>“%A”:</strong> means end with A.</li><li>‘<strong>A%B’</strong>: means start with A and end with B.</li><li><strong>“%ABC%”</strong>: means start with or end with ABC or within the string ABC appears.</li><li><strong>NOT LIKE “S%”</strong>:</li></ul><h4>2. “_ “: This allows you to match on a single character.</h4><p><strong>SAMPLE USE CASES</strong></p><ul><li>“<strong>AB_C</strong>”: Starts with AB, there’s a character, then C.</li></ul><h4>3. Combination of both</h4><p><strong>SAMPLE USE CASES</strong></p><ul><li><strong>“_ _ _ _%”:</strong> Say you want to find out which column_name (value) contains four characters, after that, there can be many numbers of characters.</li></ul><h4><strong>4. “ ” : This allows you to match a statement.</strong></h4><p><strong>N.B: “ \ ” This is the escape character. It is used to escape the meaning of percent as a wildcard.</strong></p><p><strong>ORDER BY:</strong></p><p>The <strong>ORDER BY</strong> operator is used to sort the records in a result. It can be sorted in ascending or descending order. The default when using <strong>ORDER BY </strong>is ascending except otherwise stated.</p><pre>SELECT column1, column2, ...<br>FROM table_name<br>ORDER BY column1 [ASC | DESC], column2 [ASC | DESC], ...;</pre><p><strong>N.B: ORDER BY </strong>statement can only be used in <strong>SELECT </strong>statements.</p><p><strong>LIMIT:</strong></p><p>The <strong>LIMIT</strong> clause is used to limit the number of records returned based on a limit value. OFFSET command is used to skip several rows before the LIMIT command is implemented.</p><pre>SELECT column1, column2, ...<br>FROM table_name<br>LIMIT number_of_rows;<br><br>SELECT column1, column2, ...<br>FROM table_name<br>LIMIT number_of_rows OFFSET number_of_rows;</pre><p><strong>AS:</strong></p><p>The <strong>AS</strong> keyword is used to assign an alias or a new name to a table, column, or expression in a SQL query. Aliases help make query results more readable and for renaming columns or expressions temporarily.</p><pre>SELECT t1.column_name, t2.column_name<br>FROM table_name AS t1, another_table AS t2<br>WHERE t1.column_name = t2.column_name;</pre><h4>AGGREGATE COMMANDS</h4><p><strong>COUNT:</strong></p><p>The <strong>COUNT</strong> function returns the count of an expression. It can be numeric or non-numeric data.</p><pre>SELECT COUNT(column_name)<br>FROM table_name <br>WHERE condition;</pre><p>To count unique values use the code below:</p><pre>SELECT COUNT(DISTINCT product_id) <br>FROM order_details;</pre><p><strong>SUM|AVERAGE|MIN&amp;MAX:</strong></p><pre>SELECT SUM(column_name) <br>FROM table_name <br>WHERE condition;</pre><pre>SELECT AVG(column_name) <br>FROM table_name<br>WHERE condition;</pre><pre>SELECT MIN(column_name) <br>FROM table_name <br>WHERE condition;</pre><pre>SELECT MAX(column_name) <br>FROM table_name <br>WHERE condition;</pre><p><strong>GROUP BY:</strong></p><p>The <strong>GROUP BY</strong> clause is used in a <strong>SELECT</strong> statement to group the results by one or more columns. The <strong>GROUP BY, LIMIT</strong> and <strong>ORDER BY</strong> clause can be used at once in a <strong>SELECT</strong> statement.</p><pre>SELECT column1, column2, aggregate_function(column3)<br>FROM table_name<br>GROUP BY column1, column2;</pre><p>N.B: The columns used to <strong>GROUP BY</strong> should follow a particular order. The unique columns first, then followed by aggregate columns.</p><p><strong>HAVING:</strong></p><p>The <strong>HAVING</strong> clause is used in combination with the <strong>GROUP BY</strong> clause to restrict the groups of returned rows to only those whose condition is true. So, the <strong>HAVING </strong>clause is a conditional statement like <strong>WHERE</strong> but for an aggregate i.e. a new column formed after aggregating.</p><pre>SELECT column1, column2, aggregate_function(column3)<br>FROM table_name<br>GROUP BY column1, column2<br>HAVING aggregate_function(column3) &gt; 0 ;</pre><p><strong>CAST:</strong></p><p><strong>WITH:</strong></p><p><strong>CASE WHEN:</strong></p><p>This expression is a conditional expression similar to <strong>if/else</strong> statements. It follows the syntax.</p><pre>CASE<br>    WHEN condition1 THEN result1<br>    WHEN condition2 THEN result2<br>    ...<br>    ELSE default_result<br>END`</pre><p>It basically implies creating new columns based on the given conditions.</p><h4>SQL JOINS</h4><p><strong>JOINS</strong> are used to combine rows from two or more tables based on a related column between them. They allow you to retrieve data from multiple tables in a single query, enabling complex queries and data retrieval from a relational database. They are of various types.</p><ul><li><strong>INNER JOIN</strong>: An <strong>INNER JOIN</strong> returns only the rows that have matching values in both tables. It filters out rows from both tables that do not have corresponding matches in the other table. Here’s the basic syntax:</li></ul><pre>SELECT columns<br>FROM table1<br>INNER JOIN table2 ON table1.column_name = table2.column_name;</pre><ul><li><strong>LEFT JOINS: </strong>A <strong>LEFT JOIN</strong> returns all the rows from the left table and the matching rows from the right table. If there are no matches in the right table, <strong>NULL</strong> values are included for the columns from the right table. Here’s the basic syntax:</li></ul><pre>SELECT columns<br>FROM table1<br>LEFT JOIN table2 ON table1.column_name = table2.column_name;</pre><ul><li><strong>RIGHT JOINS: </strong>A <strong>RIGHT JOIN</strong> is similar to a <strong>LEFT JOIN</strong> but returns all the rows from the right table and the matching rows from the left table. If there are no matches in the left table, <strong>NULL</strong> values are included for the columns from the left table. Here’s the basic syntax:</li></ul><pre>SELECT columns<br>FROM table1<br>RIGHT JOIN table2 ON table1.column_name = table2.column_name;</pre><ul><li><strong>FULL (OUTER) JOINS: </strong>A <strong>FULL JOIN</strong> returns all rows from both tables and includes <strong>NULL</strong> values where there are no matches. It combines the results of both the <strong>LEFT JOIN</strong> and the <strong>RIGHT JOIN</strong>. Here’s the basic syntax:</li></ul><pre>SELECT columns<br>FROM table1<br>FULL JOIN table2 ON table1.column_name = table2.column_name;</pre><p>Joins in PostgreSQL are powerful tools for combining data from multiple tables, allowing you to create more complex queries and retrieve relevant information from your relational database. Depending on your specific requirements, you can choose the appropriate join type to obtain the desired results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4625836e9fdc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s talk about Outliers]]></title>
            <link>https://medium.com/analytics-vidhya/lets-talk-about-outliers-d82b846000c0?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d82b846000c0</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Thu, 29 Apr 2021 06:58:04 GMT</pubDate>
            <atom:updated>2021-04-30T15:53:43.844Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*vWqF5xBXR_Va8cAvMzPnIg.jpeg" /></figure><h3>How would you describe outliers?</h3><p>They are observations that lie at an abnormal distance from other values in a random sample from a population. In simpler terms, the number or numbers that are much more higher or lower than the range of other close set of numbers is said to be an <strong>outlier</strong>.</p><h3>How do outliers come about?</h3><p>Outliers can emerge from a number of reasons. Some of which include:</p><ul><li>Recording errors</li><li>Unusual sampling of data</li><li>Natural errors due to the data sample</li><li>Data processing error</li></ul><h3>How are outliers identified?</h3><p>They can be easily noticed through the following graphical techniques.</p><ul><li>Scatterplots</li><li>Boxplots</li></ul><p><strong>Scatterplots</strong></p><p>Using a scatterplot, the points that are farthest from the regression line are outliers. The distance to the line is measured perpendicularly to the line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*6yP8572Is3O7nDgP_alDqw.jpeg" /></figure><p>If one point of a scatter plot is farther from the regression line than some other point, then the scatter plot has at least one outlier.</p><p>If a number of points are the same farthest distance from the regression line, then all these points are outliers. If all points of the scatter plot are the same distance from the regression line, then there is no outlier.</p><p><strong>Boxplots</strong></p><p>In descriptive statistics, a boxplot is used for explanatory data analysis to show the outliers in a dataset.</p><p>A boxplot is constructed by drawing box between the upper and lower quartiles with a solid line drawn across the box to locate the median.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*EojSINViurFWlSeDNIaC1g.jpeg" /></figure><p>Any number that is above the upper fence or below lower fence is said to be an outlier. The upper fence(quartile) and lower fence(quartile) are gotten through the following formulas:</p><p>Note: Use the QUARTILEINC function to get the value for Q1 and Q3 of any given field/column.</p><p>IQR = Q3 - Q1</p><p>Upper Fence = Q3 + 1.5 * IQR</p><p>Lower Fence = Q1 - 1.5 * IQR</p><h3>How to deal with outliers</h3><ul><li>Compare if the outliers are gotten from data entry and correct them.</li><li>Change the values of the outliers using the upper fence and lower fence depending on how skewed the data is due to the outlier.</li><li>If the outlier affects both results and assumptions then there is no need to drop the outlier.</li><li>Remove the entries that contain outliers if the percentage are NOT much.</li></ul><p>All these are dependent on the amount of data that is present.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d82b846000c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/lets-talk-about-outliers-d82b846000c0">Let’s talk about Outliers</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Techniques in cleaning large datasets sing pandas.]]></title>
            <link>https://chidimma-p-ogu.medium.com/techniques-in-cleaning-large-datasets-sing-pandas-1e0fb1cb8544?source=rss-ac5b91b8667c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e0fb1cb8544</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Chidimma Prisca Ogu]]></dc:creator>
            <pubDate>Wed, 17 Mar 2021 07:12:15 GMT</pubDate>
            <atom:updated>2024-06-19T08:01:35.565Z</atom:updated>
            <content:encoded><![CDATA[<h3>Techniques in cleaning large datasets using pandas.</h3><p>The dataset we would use in this publication is the Ames Housing Dataset. It can be found here: <a href="https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data">https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*w0-FBK5aaHTJsF3Fib5gPg.jpeg" /></figure><p>First we read the dataset into pandas using this code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZneeiPku7oS4w-3nHwzuA.jpeg" /><figcaption>importing the dataset into pandas</figcaption></figure><p>To have a complete view of the rows and columns, we’d run the code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5tItn5zHwr-KAeH_A-Hqg.jpeg" /><figcaption>code to ensure the maximum display of the rows and columns</figcaption></figure><p>Then we look through the dataset note the number of rows, columns and their attributes. but, before we do that, we save the raw_data so that we can access it when we need to after changes have been made. this would be labelled our first checkpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-WY8VHf_1eOzsHR4sL2beA.jpeg" /><figcaption>checkpoint 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNupa9bbRVKPD2oYGHGaaw.jpeg" /><figcaption>number of rows and columns in the data set</figcaption></figure><p>Next, we will look at the missing numerical variables in the dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJpFCn2zlyUoy1D9HCKR9Q.jpeg" /><figcaption>numerical variables and their description</figcaption></figure><p>There are only two numerical variables with missing data LotFrontage which contains 1201 rows and MasVnrArea that contains 8 missing rows. Since they are not much i.e the percentage of missing values is less than 20 %, so we’ll replace them with the mean value.</p><p>Note that this is not done all the time and will not work for categorical variables that are numerical</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yRqNSrlonIpsPNUuFCBNCg.jpeg" /><figcaption>replacing missing values for numerical values</figcaption></figure><h3>Dealing with missing categorical variables</h3><p>We use the .info() to look at the missing values</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZ79w-BxyiE_zIWKkTHYKw.jpeg" /><figcaption>info on categorical variables</figcaption></figure><p>The MasVnrType</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2NQ9Bsqyg4JX0mWPaxN6Hg.jpeg" /><figcaption>clean the MasVnrType</figcaption></figure><h4>Basement</h4><p>Looking through the dataset we notice that the four columns BsmtQual, BsmtCond, BsmtExposure, BsmtFinType1 and BsmtFinType2 contain missing values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ANwnc7y_yKs2NJ6xTO97A.jpeg" /></figure><p>and are all dependent on the BsmtQual, so<br>for values that have no BsmtQual, the corresponding 3 columns do not have as well, so there are 37 identical missing values for<br>the four columns. But BsmtExposure and BsmtFinType2 have 38 missing values, 37 missing values identical to BsmtQual and one extra missing value which were in row 948 and 333 respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S7p8OUf8m2wmdenjzq2mNA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9VSaIPINhl967cvJ2XCyA.jpeg" /><figcaption>The five corresponding values are related to BsmtQual</figcaption></figure><p>So we first fill in the two columns BsmtExposure and BsmtFinType2 that have 38 missing values. From the description of the data, they have</p><p>BsmtExposure: Refers to walkout or garden level walls</p><p>Gd Good Exposure<br> Av Average Exposure (split levels or foyers typically score average or above) <br> Mn Mimimum Exposure<br> No No Exposure<br> NA No Basement</p><p>BsmtFinType2: Rating of basement finished area (if multiple types)</p><p>GLQ Good Living Quarters<br> ALQ Average Living Quarters<br> BLQ Below Average Living Quarters <br> Rec Average Rec Room<br> LwQ Low Quality<br> Unf Unfinshed<br> NA No Basement</p><p>as their unique entries. So we would refill the BsmtExpo with ‘No’ since it has no exposure and BsmtFinType2 with ‘unf’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*umYc03YnlxlmeRyaUyUKOQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*caVa10mplsT0_wCSQjoo9w.jpeg" /></figure><p>Looking through the dataset, it is seen that the unique values of the basement variables are given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXer1pYoPS6djH4DkcoS_g.jpeg" /><figcaption>unique values of BsmtQual</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-Apj4SnmQOui2IH6VB-AA.jpeg" /><figcaption>unique values of BsmtFintype2</figcaption></figure><p>while from the data description the unique values of BsmtQual were given by</p><p>Ex Excellent (100+ inches) <br> Gd Good (90–99 inches)<br> TA Typical (80–89 inches)<br> Fa Fair (70–79 inches)<br> Po Poor (&lt;70 inches<br> NA No Basement</p><p>So this implies that houses without basements were stored as missing values for all five identical variables. We’d replace all the 37 identical nan entries in the five related basement cells with NA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ZTAK4eZobVrSWaCGgtHow.jpeg" /></figure><h4>Electrical</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8n9tqC7zzdSP9njkyEirTQ.jpeg" /><figcaption>description of the entries in the Electrical column</figcaption></figure><p>It can be seen that the electrical column has only one missing value, and the highest occurring frequency is SBrkr we would fill in the missing value with this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JN79zjc20T_gn-Kyk6B98g.jpeg" /></figure><h4>Fireplaces</h4><p>The two columns that have null values are Fireplaces and FireplaceQu</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qiPp0dvixDBeW-ejKTeqng.jpeg" /></figure><p>We can see that the two columns are related and that every entry that does not have a fireplace has a null fireplace quality. So we will fill in the missing values with NA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqE2H_2n4wXBm1iCvMT4SQ.jpeg" /></figure><h4>Garages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLmK5n3mrlXHTM3HeBvEUw.jpeg" /></figure><p>It can be seen from the diagram above that columns without a garage type have the four variables that relates to them as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKadu59KnE52UW72z52yvg.jpeg" /></figure><h4>Pool</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yny-eVfFH43opjOUVExyiw.jpeg" /></figure><p>We can see that the number of houses that do not have pool area corresponds to the number of poolqc that are not given.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aMS7_eb7zlnqLGr0tFidwQ.jpeg" /></figure><h4>Checkpoint 2</h4><p>We would create our second checkpoint here with the code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DxhWPmBJvShyped7qxOckQ.jpeg" /></figure><h4>Alley, Fence, MiscFeature</h4><p>There are three columns left with empty data alley, fence and miscellaneous feature. We would drop them as they contain only values that make up less than 20% of the data set.</p><p>I would appreciate your responses and reviews. Are there other techniques that are seamless and faster? Please respond.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e0fb1cb8544" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>