<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nardos Solomon - Data Analyst @ UMASS-BOSTON on Medium]]></title>
        <description><![CDATA[Stories by Nardos Solomon - Data Analyst @ UMASS-BOSTON on Medium]]></description>
        <link>https://medium.com/@nardosanalyst?source=rss-df6a4c9c4a0d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZJnO79yJwV4eMdjFSDHM7Q.jpeg</url>
            <title>Stories by Nardos Solomon - Data Analyst @ UMASS-BOSTON on Medium</title>
            <link>https://medium.com/@nardosanalyst?source=rss-df6a4c9c4a0d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:54:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nardosanalyst/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data-Driven Analysis of Motor Vehicle Collisions]]></title>
            <link>https://nardosanalyst.medium.com/data-driven-analysis-of-motor-vehicle-collisions-a8f0b9a7557f?source=rss-df6a4c9c4a0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8f0b9a7557f</guid>
            <category><![CDATA[data-science-projects]]></category>
            <dc:creator><![CDATA[Nardos Solomon - Data Analyst @ UMASS-BOSTON]]></dc:creator>
            <pubDate>Sat, 09 Mar 2024 07:40:22 GMT</pubDate>
            <atom:updated>2024-03-09T07:40:22.774Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*geFmx5HzxQihQ4Sz8u0zWw.png" /><figcaption><a href="https://nardosanalytics.com/">nardosanalytics.com</a></figcaption></figure><h3><strong>1. Introduction</strong></h3><p>The increasing rate of motor vehicle collisions has prompted the need for a data-driven analysis to understand patterns, contributing factors, and severity levels of these incidents. This research aims to leverage machine learning models to provide insights into the factors influencing collisions and develop predictive models to assess the risk levels associated with different scenarios.</p><p><strong>Research Objectives:</strong></p><p>· Analyze and visualize patterns in motor vehicle collisions.</p><p>· Develop machine learning models to predict collision risk levels.</p><p>· Provide recommendations based on the analysis to improve road safety.</p><h3>Data Source and Variable Explanation</h3><p>The dataset used in this study is sourced from <a href="https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Vehicles/bm4k-52h4/about_data">NYC Open Data</a>, provided by the Police Department (NYPD). It includes comprehensive information on motor vehicle collisions, recorded and reported by the NYPD. Key variables include geographical information (such as ‘borough’), contributing factors (like ‘contributing_factor_vehicle_1’ to ‘contributing_factor_vehicle_5’), vehicle types (i.e., ‘vehicle_type_code_1’ to ‘vehicle_type_code_5’), and numbers about different types of injuries and fatalities.</p><pre>pd.set_option(&#39;display.max_columns&#39;, None)  # Show all columns<br>df.head(15)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B5PSuF6tbPDBcdtiGo35mw.png" /></figure><blockquote><em>Note: </em>Before I start working with data, I make sure to thoroughly understand it. This involves checking how big the dataset is, what type of information it contains, and looking for any missing pieces. I don’t just stop at finding missing values; I also try to figure out why they’re missing and what it means for each part of the data. This way, I can make smart decisions about how to handle those gaps and ensure the data is reliable when I process it.</blockquote><pre>df.shape<br>df.dtypes<br>print(df.isnull().sum())<br># Display descriptive statistics of numerical columns<br>print(&quot;\nDescriptive statistics of numerical columns:&quot;)<br>print(df.describe())</pre><h3>2. Data Processing Explanation</h3><p>The dataset underwent thorough preprocessing to ensure its suitability for analysis and modeling:</p><p><strong>Standardization of Column Names:</strong></p><p>Column names were standardized by removing spaces, special characters, and converting to lowercase.</p><pre># Clean up column names by removing spaces and converting to lowercase<br>df.columns = df.columns.str.strip().str.lower().str.replace(&#39; &#39;, &#39;_&#39;)<br>print(df.columns)</pre><p><strong>Time-Related Features:</strong></p><p>‘crash_date’ and ‘crash_time’ columns were converted to date-time format. Among many other reasons, this is done to explore and visualize the distribution of collisions over different months and hours.</p><pre># Explore time-related features: &#39;crash_date&#39; and &#39;crash_time&#39;<br>df[&#39;crash_date&#39;] = pd.to_datetime(df[&#39;crash_date&#39;])<br>df[&#39;crash_time&#39;] = pd.to_datetime(df[&#39;crash_time&#39;])</pre><p>Then we can extract ‘month’ and ‘hour’ columns to analyze collisions over months and hours.</p><pre># Extract month and hour information<br>df[&#39;month&#39;] = df[&#39;crash_date&#39;].dt.month<br>df[&#39;hour&#39;] = df[&#39;crash_time&#39;].dt.hour</pre><p><strong>Total Injuries and Killed:</strong></p><p>‘total_injuries’ and ‘total_killed’ columns were created by summing the relevant injury and fatality columns.</p><p><strong>Risk Level Categorization:</strong></p><p>A new column, ‘risk_level,’ was introduced by categorizing the combined injuries and killed into risk levels based on predefined thresholds.</p><pre>low_threshold = 0<br>medium_threshold = 2<br>high_threshold = 3<br><br>df[&#39;risk_level&#39;] = pd.cut(df[&#39;combined_injured_killed&#39;], bins=[low_threshold,<br>medium_threshold, high_threshold,float(&#39;inf&#39;)],labels=[&#39;low risk&#39;, &#39;medium risk&#39;, &#39;high risk&#39;], right=False)</pre><p><strong>One-Hot Encoding:</strong></p><p>Categorical variables such as ‘borough’ and contributing factors were one-hot encoded to facilitate machine learning model input.</p><p><strong>Handling of Missing Values:</strong></p><p><strong>Removal of Rows with All Null Values:</strong></p><p>Rows with null values in all vehicle types and contributing factors were removed as they represented instances where there was insufficient information about the vehicles involved or the contributing factors, likely indicating less complex incidents with fewer than three, four, or five documented factors or vehicles.</p><p>Rows with missing values of borough were dropped</p><p><strong>Imputation and Replacement:</strong></p><p>Missing values in categorical columns, including vehicle types and contributing factors, were replaced with the label ‘Unknown’ to retain these instances in the analysis while acknowledging the absence of specific information.</p><p>Missing values in numerical columns like ‘number_of_persons_injured’ and ‘number_of_persons_killed’ were imputed with median values.</p><p><a href="https://nardosanalyst.medium.com/a-step-by-step-guide-to-doing-a-data-analytics-project-136639953fbc">A step-by-step guide to doing a data analysis project</a></p><h3><strong>Exploratory Data Analysis (EDA):</strong></h3><p>The distribution of the top 10 contributing factors and the top 10 vehicle types was explored using count plots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWYgx7_W18HRnWXDzjoOag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*03Ga5JQtOcBIiiC9piOlAw.png" /></figure><h4>Distribution of Injuries and Fatalities (Log Scale)</h4><p>The majority of collisions result in no injuries or fatalities, as indicated by the large counts for 0.0 in both figures. Collisions resulting in injuries are more common than those resulting in fatalities, which is expected. The distribution provides a sense of the severity of collisions, with most being minor (few or no injuries) and a smaller number involving more severe injuries or fatalities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*52vqmjzI8tc4m7uXcl7fvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*VbDL6Ir5odkTpX8tbNhMsQ.png" /></figure><h4>Collisions Over Months and Hours</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WHcfut7vElksHcWmHvJysA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kceiei3IubKgCgtfUQogdA.png" /></figure><p><strong>Number of Accidents Over Time by Borough (2021–2022)</strong></p><p>By grouping the data by year, month, and borough, the line plot provides a dynamic representation of how the frequency of accidents varied across different boroughs throughout the specified time frame.</p><p>As shown in the figure, Brooklyn consistently exhibited the highest number of collisions compared to other boroughs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABKK5i1FzetgMsgzsB-tqA.png" /></figure><h4>Collisions on Public Holidays vs Non-Public Holidays</h4><p>Collisions on public holidays were examined, but the observed effect on collision rates appeared to be minimal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*cZRcpNFIyAdk-0rxacMhfg.png" /></figure><h3>3. Modeling</h3><p>In this section, we explore three distinct models aimed at extracting valuable insights from the dataset: clustering to identify major vehicle types tied to collisions, linear regression to predict the number of accidents in an area over time, and decision Tree Classifier was employed to predict the risk level associated with accidents. Which aids in understanding the factors contributing to the severity of accidents.</p><h4><strong>3.1 Cluster Analysis Model:</strong></h4><p>· K-means clustering is a popular unsupervised machine learning algorithm used for grouping similar data points into clusters.</p><p>· It is chosen here for identifying major vehicle types tied to collisions as it can segregate data into distinct groups based on feature similarity.</p><p>To discern major vehicle types associated with collisions, K-means clustering was employed. <strong>The process involved:</strong></p><p>Feature Selection:</p><p>The primary vehicle type (‘vehicle_type_code_1’) was selected as the key feature.</p><p>Determining Optimal Clusters (k):</p><p>The Elbow Method was utilized to identify the optimal number of clusters (k).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*Tqt10Ht9UdDPJyXkFV9sEw.png" /></figure><p>Applying K-means Clustering and Identifying Major Vehicle Types:</p><p>K-means clustering was executed with the chosen k to group similar vehicle types. Which is “3”. Then the major vehicle types in each cluster were determined by selecting the mode within each cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*1uH2UimelF-Hb5cv6qWo8g.png" /></figure><p>Results</p><p>The clustering process revealed three distinct clusters, each associated with major vehicle types tied to collisions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*oYmk2FEx7Z9MEfFUW1K9gg.png" /></figure><p><strong>Clusters Outcome Analysis:</strong></p><p><strong>Cluster 0 analysis:</strong></p><p>This cluster predominantly consists of collisions involving Station Wagons/Sport Utility Vehicles as the primary vehicle type.</p><p>Sedans are also notable within this cluster, suggesting a combination of these two vehicle types in collisions.</p><p><strong>Cluster 1 Analysis:</strong></p><p>This cluster is characterized by collisions where Sedans are the primary vehicle type involved.</p><p>It indicates a specific pattern where accidents are mainly between two sedans, possibly representing typical urban collisions.</p><p><strong>Cluster 2 Analysis:</strong></p><p>This cluster highlights collisions involving Taxis as the primary vehicle type.</p><p>Sedans also appear frequently in these collisions, indicating potential scenarios where taxis collide with sedans.</p><h4><strong>3.2 Linear Regression for Accidents Prediction</strong></h4><p>· Linear regression is a supervised machine learning algorithm used for predicting a continuous outcome.</p><p>· It is chosen here for predicting the number of accidents in an area over time as it models the relationship between input features (year, month, borough) and the target variable (accident count).</p><p>· It can provide insights into how changes in input features relate to changes in the target variable.</p><p><strong>Grouping and Counting:</strong></p><p>The dataset was grouped by year, month, and borough, and the number of accidents was counted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*BP-alEgeK5iQrgb9YgTA4w.png" /></figure><p>Modeling Data Preparation:</p><p>A modeling dataset was created, featuring ‘Year,’ ‘Month,’ ‘borough,’ and ‘Accident Count.’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*2WLr383e_lr9Cg0EmN312Q.png" /></figure><p>Training and Testing Sets:</p><p>The dataset was split into training and testing sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*kn0lfSUQr4mt7O8gWuNhfw.png" /></figure><p>Linear Regression Model:</p><p>A linear regression model was built using the training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*GnZRdFivTjecXFU2EV5zgw.png" /></figure><p>Model Evaluation and result</p><p>The model’s performance was assessed using metrics such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE).</p><p>The linear regression model demonstrated the following performance on the test set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*Jj2QE5kcNsxlEJvwF1Zm_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*RDAH9cw-pv1OM96gPXmm5A.png" /></figure><h4>Interpretation and Acceptance of Results</h4><p>Both models contribute valuable insights to our understanding of collisions. The clustering model identifies major vehicle types associated with collisions, aiding in targeted interventions and policy decisions. The linear regression model predicts the number of accidents over time, providing a tool for proactive planning and resource allocation.</p><p>The identified major vehicle types from clustering and the predictive capability of the linear regression model collectively contribute to a comprehensive understanding of collision patterns.</p><h4><strong>3.3 Decision Making Model</strong></h4><p>One modeling method chosen is the decision tree modeling. We chose this for classification of the “level of risk” on factors that are related to the vehicles (i.e. incident contribution factors, vehicle types, number of injuries, number of fatalities, and the area that it occurred in).</p><pre>low_threshold = 0<br>medium_threshold = 2<br>high_threshold = 3</pre><p>For the model’s risk classification, it tries to categorize the results into 3 levels of risk. We took a unique list of numbers of total injuries and/or fatalities and used that to classify the risk. We mainly used our own judgement for the grouping. “Low Risk” indicates that nobody was injured but the accident was still reported due to meeting the minimum damage cost required for a police officer to report (damage totaling up to over $1,000). “Medium Risk is classified as having up to or equal to 2 total injuries and/or fatalities. “High Risk” would be any total injuries and/or fatalities that occur above the threshold of 2. This model is aimed to predict the most likely risk category that a reported vehicle incident would be.</p><p>The results presented by the model were promising but there are a few issues that need to be addressed before we can accept it. The results currently show that the low-risk category was performing extremely well in predictions, accuracy, and reliability. While this is promising, the other two categories were not so fortunate. To address this issue, we would want to either limit the size of the data to be an even amount across all categories for training and testing, or up sample the minority variables in this case. This would provide us with a “truer” result set and allow us to further identify the “noise” in the dataset and process it further to maintain better accuracy. Due to time constraints related to this project and troubleshooting taking a long time, along with other obligations, we cannot address this issue prior to the presentation or report. Another potential avenue to address the current results situation would be to automatically categorize fatalities as high risk and maintain the current “buckets” for just total injuries. This would be a lot easier to regroup the large dataset into a slightly fairer set of clusters, but up sampling method would still be needed, as it isn’t that likely to have enough “medium/high-risk” categorized data to train and test with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*V5DGn5LlVgLdZH5ywfeZfQ.png" /></figure><p><strong>Conclusion:</strong></p><p>In conclusion, this data-driven analysis of motor vehicle collisions employs machine learning models to understand patterns and predict risk levels, utilizing a comprehensive dataset from NYC Open Data. Thorough preprocessing, exploratory data analysis, and diverse modeling techniques contribute valuable insights. While the clustering and linear regression models offer valuable findings, recognizing the need for improvement in the decision tree model’s predictions is crucial. Continuous refinement and adaptation of methodologies are essential for a more effective understanding of collision patterns and improved road safety interventions. Overall, this study emphasizes the importance of ongoing efforts to enhance accuracy and insight in addressing motor vehicle collisions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8f0b9a7557f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging Data Analytics to Solve Real-Life Problems]]></title>
            <link>https://nardosanalyst.medium.com/leveraging-data-analytics-to-solve-real-life-problems-92b501b59ee6?source=rss-df6a4c9c4a0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/92b501b59ee6</guid>
            <category><![CDATA[data-analyticsapplication]]></category>
            <category><![CDATA[data-analysis-history]]></category>
            <category><![CDATA[real-life-problem-solving]]></category>
            <dc:creator><![CDATA[Nardos Solomon - Data Analyst @ UMASS-BOSTON]]></dc:creator>
            <pubDate>Fri, 06 Jan 2023 16:41:31 GMT</pubDate>
            <atom:updated>2023-01-06T16:41:31.261Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explains the origin of data and how it can be used to address real-life issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sI23dhZB0QcokA2tocx8Hw.png" /></figure><h3>The History of Data</h3><p>To gain insight into how data is being used today and what the future of data holds, it’s essential to understand its evolution. The development of data has altered the way we process it and interact with it.</p><p><em>Data is a form of information that connects individuals, companies, and the larger community. It helps businesses, scientists, and other organizations make sense of the world and improve their performance.</em></p><p>Even before the invention of computers, the need for tools to analyze and store data was apparent. Businesses were collecting information to make decisions on their operations, inventories, and more. Governments collected data to track trade routes and taxes.</p><p>When the first computer was invented in 1958, it revolutionized the way data was analyzed. In addition, the need for data storage increased as more computers were built.</p><p>New technologies also made it easier to store large amounts of data. By the mid-1900s, the storage crisis was a real concern.</p><p>Paper had been the main source of written data for nearly a thousand years. It was not only expensive, but the relative portability of paper meant that records were a less effective form of storage.</p><p>In the 1800s, the need for a more efficient data analysis process became critical. This led to the development of the Hollerith Tabulating Machine. The Hollerith machine used punch cards to tabulate data.</p><p>This system helped researchers analyze census data more quickly and accurately. It also allowed them to extrapolate insights into future events.</p><p>As data science has evolved, a number of new techniques have been developed to help people solve problems with data. Some of these techniques include predictive analytics, using ratios, percentages, and proportional reasoning, and making data more accessible to the average person. The goal is to ensure that people can easily access and interpret information, and use it in ways that will benefit their organizations and their community.</p><h3>Solving Problems With Data Analytics</h3><p>1. <strong>Data analytics can help to identify the question you want to answer: </strong>Data analytics can be a powerful tool to help gain clarity on any given question or issue. It provides an in-depth overview of the situation, allowing for more precise predictions and decisions to be made.</p><p>For example, data analytics can be used to analyze customer behavior patterns and identify trends in order to make better marketing decisions.</p><p><strong>2. Gather the data necessary to answer the question:</strong></p><p>Identify the necessary information to answer the question, such as relevant historical data, current trends, customer feedback, market conditions etc.</p><p>Example:</p><p>Relevant historical data: Sales figures from the past 5 years, customer retention rate over time.</p><p>Current trends: Market share of competitors, consumer spending patterns.</p><p>Customer feedback: Surveys, reviews on social media platforms.</p><p>Market conditions: Macroeconomic indicators such as GDP growth rate, inflation rate etc.</p><p><strong>3. Assess the data to determine what is accessible and what is not</strong></p><p>Once the existing sources of information have been identified, it is important to assess what data points are already available from these sources such as databases or surveys and which ones require additional research or collection of new data points.</p><p>To ensure the accuracy and reliability of the data, it is important to determine if any additional research or collection of new data points is necessary. This may involve further analysis of existing databases or surveys as well as conducting interviews, focus groups, experiments, or other methods to gather information that was not previously available.</p><p><strong>4. Obtain any missing data</strong>:</p><p>Acquire the information that is not available either by collecting it yourself or purchasing it from a third-party provider if needed in order to complete your analysis effectively.</p><p><strong>5. Utilize the data to solve the problem:</strong></p><p>Solve the problem using analytical techniques such as regression analysis, clustering algorithms, decision trees etc., interpreting results accurately and implementing appropriate solutions based on findings in order to achieve desired outcomes efficiently and cost-effectively.</p><h3><strong>Using Data Analytics to Solve Real-Life Problems</strong></h3><p>Data analytics can be used to solve real-life business problems. It can provide a comprehensive overview of the issue and allow for more accurate predictions and decisions. In order to do this, it is important to identify the information necessary to answer the question such as relevant historical data, current trends, customer feedback, and market conditions.</p><p>Once you have determined what information is available from existing sources or needs to be acquired through additional research or collection of new data points, analytical techniques such as regression analysis, clustering algorithms and decision trees can then be applied in order to interpret results accurately and implement appropriate solutions based on findings in order to achieve desired outcomes efficiently and cost-effectively.</p><blockquote>A business needed to identify the most profitable customer segment for its product.</blockquote><p>Solution: Used data analytics to analyze customer feedback, market conditions, and historical data in order to determine which customer segment was most likely to purchase the product and had the highest potential for profit. Examples of tools used in this solution: Data Mining, regression analysis, and decision trees.</p><blockquote>An online retailer wanted to reduce abandoned carts on their website.</blockquote><p>Solution: Utilized data analytics techniques such as decision trees and regression analysis to identify patterns in user behavior that were leading customers away from purchasing products and implemented changes accordingly in order to reduce abandoned cart rates significantly.</p><blockquote>A company wanted to optimize their supply chain operations by reducing costs while maintaining the quality of service levels.</blockquote><p>Solution: Applied clustering algorithms with existing datasets of suppliers’ performance metrics, pricing structures, delivery times, etc., combined with predictive models based on current trends in order to identify areas where cost savings could be achieved without compromising service standards or reliability of supply chains operations</p><blockquote>A bank needed a way of detecting fraudulent activity quickly and accurately</blockquote><p>Solution: Leveraged data analytics tools such as machine learning algorithms which can detect anomalies within large datasets quickly and accurately, allowing them to flag suspicious activities faster than traditional methods used previously would have been able to do.</p><blockquote>An insurance company wanted a better understanding of risk profiles associated with their customers.</blockquote><p>Solution: Applied data analytics techniques such as predictive modeling, regression analysis, and clustering algorithms to analyze customer profiles, past claims history, and other relevant data points in order to gain a better understanding of risk profiles associated with their customers. This allowed them to create more accurate pricing models for different types of customers based on their individual risk profiles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92b501b59ee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uncovering Hidden Insights Through Text Mining]]></title>
            <link>https://nardosanalyst.medium.com/uncovering-hidden-insights-through-text-mining-ab076a59d861?source=rss-df6a4c9c4a0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab076a59d861</guid>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[text-mining-for-business]]></category>
            <category><![CDATA[text-mining-process]]></category>
            <category><![CDATA[text-mining-application]]></category>
            <dc:creator><![CDATA[Nardos Solomon - Data Analyst @ UMASS-BOSTON]]></dc:creator>
            <pubDate>Wed, 04 Jan 2023 19:04:46 GMT</pubDate>
            <atom:updated>2023-01-04T19:04:46.250Z</atom:updated>
            <content:encoded><![CDATA[<h4>How text mining works and why it is such an important tool for businesses today.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuhdrnClRy-A%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuhdrnClRy-A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuhdrnClRy-A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/056f2ea6e322ba95908857a2bb5ca610/href">https://medium.com/media/056f2ea6e322ba95908857a2bb5ca610/href</a></iframe><p>In this article, we will explore how text mining can be used to uncover hidden insights from large amounts of textual data. We will discuss the process and techniques that are utilized in order to gain meaningful insights from unstructured text sources. By the end of this article, you should have a better understanding of how text mining works and why it is such an important tool for businesses today.</p><blockquote>Text mining, a process of extracting meaningful insights from large amounts of unstructured text data, is becoming increasingly popular among businesses. This sophisticated technology can uncover hidden trends and patterns that would otherwise be difficult to detect by traditional methods.</blockquote><h3><strong>How Businesses Utilize Text Mining</strong></h3><p>Text Mining offers tremendous potential for businesses looking for ways to optimize operations and stay competitive in today’s ever-evolving marketplace. With its ability to uncover hidden insights from massive amounts of unstructured data quickly and accurately; it has become an essential tool for organizations seeking an edge over their competition.</p><p>Text mining enables businesses to gain a deeper understanding of customer sentiment, market dynamics, consumer behavior, and more. By automatically analyzing the content within emails, blog posts, social media messages and other sources of textual information, companies are able to gain valuable insights into their customers’ needs and preferences as well as potential opportunities for growth.</p><p>Moreover, businesses can use text mining to identify topics that are trending with their target audience or analyze customer feedback about products in order to make informed decisions about future product development or marketing strategies. Additionally, businesses can also use text mining to automatically classify incoming emails into categories based on their content, which can help streamline processes and improve efficiency. Finally, besides discovering new opportunities for growth and innovation; text mining also helps companies stay ahead of competition by tracking changes in the industry landscape quickly and accurately.</p><h3><strong>HOW DOES IT WORK?</strong></h3><p><strong>Step 1: Collect large amounts of unstructured text data from webpages, databases, social media messages and other sources.</strong></p><p>The first step in text mining is collecting relevant data. To collect large amounts of unstructured data from emails, blogs, social media messages and other sources, businesses can use a variety of tools such APIs or other web scraping tools like Beautiful Soup library in Python programming language</p><p>Web scraping is used to extract data from websites or online databases.</p><p><strong>Step 2: Pre-process the collected data by removing irrelevant content or formatting issues that may affect the accuracy of results.</strong></p><p>Once the dataset has been gathered together it must then be preprocessed by removing noise such as punctuation marks, stop words (commonly used words), special characters etc., to make sure that only useful information remains before further analysis takes place.</p><p>The tool used for this process is called data normalization. Why do we care? Because we need to make sure the data is in a consistent format before use. The data needs to be tidy and usable, in order to derive meaningful insight. If the data is not normalized, it can skew the results of your analysis.</p><p>A real life example of this is how the democrats used trump’s tweets during the 2016 election. They highlighted certain words and phrases from his tweets, without taking into account any context or meaning behind them, which caused an inaccurate representation of his character and intentions.</p><p><strong>Step 3: Analyze the collected text data to identify topics that are trending with your target audience. We can analyze the collected text data using various methods such as clustering and topic modeling.</strong></p><p>Clustering is a process of grouping similar data, for example group tweets about a specific topic together. Why we need to do this, because most people don’t spell things right or use words in the same way, so we need to group these similar words together. For instance, people writing about a specific topic may use different words such as “machine” or “system” or “model”. We need to group these words together in order to be able to pull the topic out of a huge collection of data.</p><p>Topic modeling is the process of constructing a list of topics. Topic modeling enables businesses to identify topics that are trending with their target audience or analyze customer feedback about products in order to make informed decisions about future product development or marketing strategies. It also helps companies stay ahead of competition by tracking changes in the industry landscape quickly and accurately.</p><p><strong>Step 4: Use natural language processing (NLP) techniques such as sentiment analysis to gain insights into customer opinions about products or services.</strong></p><p>Sentimental analysis helps in determining the emotions behind customer comments, thus enabling business to track changes in customer perception. It also helps in identifying potential customers or market segments that are yet to be tapped in to. For example, if a lot of customers are talking about a new social media feature in your product, you could potentially make that feature more prominent in your marketing strategy.</p><p><strong>Step 5: Extract meaningful information from the analyzed data and use it to make informed decisions about future product development, marketing strategies, etc.</strong></p><p>A real-life example of Step 5 would be a company that uses customer feedback data to inform decisions about product development. For instance, the company may analyze customer feedback from surveys or interviews to identify common issues and trends in their products. Based on this analysis, they could then decide to invest resources into developing new features or improvements that address these issues. Additionally, the company could use the same data to inform marketing strategies by determining which messages resonate with their target audience and how best to reach them.</p><p><strong>Step 6: Track changes in industry landscape quickly and accurately using advanced analytics tools like machine learning algorithms.</strong></p><p>Once the data is collected and organized, it can be fed into a machine learning algorithm that will process it to identify patterns or trends. This involves using advanced analytics tools such as machine learning algorithms to quickly and accurately track changes in the industry landscape. These algorithms are able to process large amounts of data, identify patterns, and uncover new insights from them. By tracking changes in the industry landscape, businesses can stay ahead of competitors and better anticipate customer needs. Machine learning algorithms can also be used to create predictive models that help businesses make more informed decisions about their operations. The use of these powerful analytics tools is becoming increasingly popular among organizations looking for a competitive edge in today’s digital economy.</p><p>In conclusion, text mining is a powerful tool that businesses can use to uncover hidden insights from large amounts of unstructured textual data. By utilizing techniques such as clustering and topic modeling, sentiment analysis, and machine learning algorithms; companies are able to gain valuable insights into their customers’ needs and preferences as well as potential opportunities for growth. Text mining enables organizations to stay ahead of the competition by quickly tracking changes in the industry landscape and making informed decisions about product development or marketing strategies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab076a59d861" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A step-by-step guide to doing a data analysis project]]></title>
            <link>https://nardosanalyst.medium.com/a-step-by-step-guide-to-doing-a-data-analytics-project-136639953fbc?source=rss-df6a4c9c4a0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/136639953fbc</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-analysis-project]]></category>
            <category><![CDATA[data-analysis-process]]></category>
            <category><![CDATA[how-to-do-project]]></category>
            <dc:creator><![CDATA[Nardos Solomon - Data Analyst @ UMASS-BOSTON]]></dc:creator>
            <pubDate>Wed, 04 Jan 2023 03:55:08 GMT</pubDate>
            <atom:updated>2023-01-04T17:10:21.351Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Don’t know where to begin with your data analysis project? Follow this step-by-step guide and start today!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*1OjD2AlZv3oAY8MLEbp9jg.png" /></figure><p>You have grasped basic knowledge of data analysis and tools, and you are having a hard time applying them to real-world problems. A good starting point is to work on projects to showcase your ability and see how much you have learned. Working on projects gives you confidence and inspires you to learn more.</p><h3><strong>Choose and research your topic</strong></h3><p>Choosing the right topic for your data analysis project is a crucial step. You need to select a topic that is not only relevant and interesting for you, but also one that has enough data available for analysis.</p><blockquote>A good place to start is by asking yourself what type of questions you are interested in answering with the data.</blockquote><p>Data analysis projects can be used to answer a variety of questions, such as the effectiveness of covid-19 vaccinations, predicting flight and gas prices, understanding the skillsets required for data analysts by most industries and their associated salaries, or determining which cities or countries have a high demand for certain skills. Additionally, data analysis projects can provide insight into what people are discussing on platforms like Twitter. All these and others I haven’t mentioned can be answered using a data analysis project.</p><p>If you are unable to come up with a question that you would like to answer or an issue that you would like to address, proceed to the next step.</p><h3><strong>Find a dataset</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*my6XtvFXPFqSVngc9rWZQA.png" /></figure><p>Once you have identified the questions, explore datasets related to your field of interest to provide answers. If nothing comes to mind, start by exploring datasets related to your field of interest and formulating questions from it. There are many sources for public datasets, including open-source databases and repositories, government and university websites, and industry associations.</p><blockquote>There are plenty of sources that offer public datasets such as:</blockquote><p>1. Kaggle Datasets: Kaggle is a popular platform for data science projects and competitions, and it has an extensive library of datasets available for analysis.</p><p>2. Government Sources: Governments around the world provide open access to their data sets, making them ideal sources for research projects. The US government provides access to many datasets through Data.gov, while other countries may have similar offerings from their own governments or universities.</p><p>3. Academic Resources: Universities often publish data sets associated with academic research in various fields such as economics, sociology, and medicine. Many of these are free to use and can be found on university websites or online repositories like Harvard Dataverse or the Inter-university Consortium for Political and Social Research (ICPSR).</p><p>4. Open Source Projects: Open source software projects often include sample datasets that can be used by developers or researchers who want to experiment with new algorithms or techniques without having to create their own dataset from scratch. For example, Apache Spark includes several sample datasets that are useful for testing out machine learning algorithms on distributed systems like Hadoop clusters</p><blockquote>You can also scrape your own data or use publicly available APIs.</blockquote><p>Web scraping is the process of extracting data from websites. It involves using a program or script to collect information from webpages, such as HTML code, text, images and other media. Web scraping can be used to gather large amounts of data quickly and easily. Tools that can be used for web scraping include web crawlers, HTML parsers, and data extraction tools.</p><p>API stands for Application Programming Interface. It is a set of programming instructions and standards that allow two applications to communicate with each other. APIs are used to access data from external sources, such as databases or web services, in order to display information on a website or application. For example, you can use an API to retrieve weather forecasts from a third-party service and display them on your own website or app.</p><h3><strong>Write your questions</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTQg_DdTzVFylJjrNF0umg.png" /></figure><p>Before you can begin to analyze your data, you need to have clear questions in mind. Writing down the questions that you would like to answer with the data will help to guide your exploration and analysis.</p><p>For example, if you are researching how television viewing habits have changed over the past decade, you might ask questions such as:</p><p>-What percentage of households watch television each day?</p><p>-Which age groups watch the most television?</p><p>-What genres of programming are most popular?</p><p>-What type of content do viewers prefer: broadcast TV or streaming services?</p><p>-Has the number of hours spent watching television increased or decreased over the past decade?</p><p>Asking questions and defining objectives for your research is an important step when doing data analytics. Make sure that you have well-defined questions before you begin to explore and analyze your data.</p><h3><strong>Explore, transform, and clean your data</strong></h3><p>Exploring data is the process of getting to know your dataset .Once you’ve explored your data, it’s time to start transform and clean it. This may include converting values into a consistent format, combining multiple datasets into one, removing duplicate records, and dealing with missing values.</p><blockquote>These are some of the thing you may want to do to prepare your data for analysis:</blockquote><p>1. Start by exploring the data and understanding its structure, values, and any potential issues or inconsistencies. Examining the number of rows and columns in a dataset, identifying which variables are numeric and which are categorical. Remove any irrelevant columns that are not useful for your analysis because they don’t help answer your questions and you will have a more clean and precise dataset.</p><p>2. Identify areas of missing or invalid data and determine how to handle them (e.g., imputation, removal). Finding records with blank values or incorrect formats (e.g., strings instead of numbers).</p><p><strong>These are some of the things you can do:</strong></p><p>Remove null values or replace them with zero.</p><p>Replace null values with some central value like mean, median or mode.</p><blockquote>In order to do that the data should be missing completely at random (MCAR).</blockquote><ul><li>Mean- When the data is numeric and not skewed.</li><li>Median-When the data is numeric and skewed.</li><li>Mode-When the data is an object or numeric.</li></ul><p>3. Check for outliers in the dataset and decide whether to remove them or keep them as part of the analysis.</p><p>Check for outliers: identify some outliers. Whether you decide to remove the outliers, keep them or transform them it is always a good idea to check for outliers. Since they may indicate bad data or may influence your findings.</p><blockquote>Scatter plot and box plot are the most preferred graphs to observe outliers.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGZOQ2gGZz0HpZ192KZpMQ.png" /></figure><p>4. Clean up text fields such as names, addresses, etc., using regular expressions or other methods depending on your needs. Replacing special characters, removing extra whitespace, converting all letters to lowercase/uppercase as needed using regular expressions or other tools/libraries available for this purpose.</p><p>5. Convert categorical variables into numerical ones where necessary so that they can be used in further analysis steps (e.g., one-hot encoding). Encoding nominal categories such as gender (male/female) into binary values (0/1).</p><p>Make sure the formatting is uniform for the data set. For example, some values in a column may have different currency from one another or some numbers are separated by a comma.</p><p>Moreover, some variables may need to be transformed. For example: if you want to find out the relationship between happiness and income and you have each country’s data, you may want to categorize then in to their specific continent.</p><p>6. Standardize values across columns to ensure consistency (e.g., formatting dates/times). Ensuring that all dates have the same format across different datasets by reformatting them from one standard to another (e.g., YYYY-MM-DD to MM-DD-YYYY).</p><p>7. Verify accuracy of all changes made before saving any results from the cleaned dataset(s). Running tests on the cleaned dataset(s) to ensure that any changes made were correct and did not introduce new errors into the analysis results before saving them permanently</p><h3><strong>Answer your questions: Choose your methodology</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uRBh-59ghJjV9RU5NzPrrw.png" /></figure><p>Once your data has been thoroughly explored, transformed, and cleaned, it’s ready for analysis and visualization. This is an important part of the data analysis process and requires a lot of thought. Before you start, make sure that you understand the questions that you want to answer with your analysis.</p><blockquote><strong>Choose the right type of visualization for your dataset.</strong></blockquote><p>When it comes to visualizing data, it is important to choose the right type of visualization for your dataset. Different types of datasets require different types of graphs, tables, or figures in order to effectively communicate the information. For example, if you are looking at a large amount of numerical data with multiple variables that need to be compared against each other, then a bar graph or line graph may be most appropriate. If you are dealing with categorical data and want to show how many items fall into each category, then a pie chart could be used instead. Additionally, if you have time-based data and would like to track changes over time then an area chart might work best. It’s also important to consider the audience when selecting which type of visualization will convey your message most clearly; visuals can often help people understand complex topics more easily than text alone can do. Ultimately, choosing the right type of visualization for your dataset is essential in order for your message or story about the data come across successfully!</p><blockquote><strong>Analyze the dataset.</strong></blockquote><p>There are different methods of analysis that you can use depending on the type of data that you have and the questions that you want to answer. Basically, when it comes to data analysis, there are five main types of analysis. These are:</p><p><strong>Exploratory analysis</strong> is used to explore and understand a dataset, identify patterns, trends and relationships between variables, or gain insights into the data. Exploratory analysis can be done through visualizing data using graphs or tables, summarizing the data using descriptive statistics such as mean, median and mode; or finding correlations between different variables in the dataset. For example, if you have a dataset containing customer demographics such as age and gender along with purchase history information like product type and quantity purchased then exploratory analysis can help you find out which age group buys more of certain products than others.</p><p><strong>Predictive analysis</strong> is a type of data analysis that uses historical data to predict future trends and behaviors. It can be used in any data analytics project to identify patterns, correlations, and trends within the dataset to generate predictions about future outcomes.</p><p><strong>Descriptive analysis</strong> is a type of data analysis that uses descriptive statistics to summarize and describe the data. It can be used to identify patterns, trends, and relationships between variables in a dataset. For example, if you have customer demographic information such as age and gender along with purchase history information like product type and quantity purchased then descriptive analysis can help you find out which age group buys more of certain products than others.</p><p><strong>Diagnostic analysis</strong> is a type of data analysis that uses diagnostic methods to identify the root cause of problems or issues in a dataset. It can be used to determine why certain trends, patterns, and relationships exist within the data. For example, if you have customer purchase history information such as product type and quantity purchased then diagnostic analysis can help you figure out why customers are buying more of one product than another.</p><p><strong>Prescriptive analysis</strong> is a type of data analysis that uses predictive analytics and diagnostic techniques to suggest recommendations or solutions to problems identified in the dataset. It can be used in any data analytics project to identify patterns, correlations, and trends within the dataset then generate predictions about future outcomes as well as recommend solutions for issues found in the dataset. For example, if you have customer purchase history information such as product type and quantity purchased then prescriptive analysis can help you forecast how much of each product will be sold over the next few months and suggest ways to increase sales for certain products.</p><p>Finally, make sure that you document every step in the process so that you can refer back to it later if needed. Documenting every step will ensure that your project is reproducible, which is essential if you want to be able to use it in other projects.</p><h3><strong>Present your findings</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*2_usHQb7bFqNta_x9CYeCg.png" /></figure><p>Now that you’ve worked your way through the process of doing a data analysis project, it’s time to present your findings. Start by summarizing your findings in a few sentences, and then break it down into more detail. Make sure to include visuals to illustrate key points, as this will make your results easier to understand. You can use charts, graphs and tables to help get your message across. If possible, try to focus on what the audience wants to hear — don’t just dump a ton of numbers on them!</p><p>When presenting your results, explain the meaning behind them. For example, if you’re looking at customer spending, don’t just give out the numbers — explain why it’s important, what it means for the business, and how it can be used to make better decisions in the future.</p><p>Finally, remember to emphasize any insights or actions that you discovered from your analysis. Show how the data can be used to inform decision making and create strategies for the future.</p><p>Following these steps will ensure that your presentation is clear and organized while allowing those viewing it to understand your findings and take action accordingly.</p><blockquote>Overall, completing a project is quite straightforward and there are plenty of resources available to help you. These include websites such as GitHub, Medium, Tableau Public, Kaggle and more. By studying how others have approached projects in the past you can gain valuable insights into best practices.</blockquote><blockquote>Good luck!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=136639953fbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Expert Tips to Become a Great Analyst]]></title>
            <link>https://nardosanalyst.medium.com/expert-tips-to-become-a-great-analyst-cfee186cb964?source=rss-df6a4c9c4a0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfee186cb964</guid>
            <dc:creator><![CDATA[Nardos Solomon - Data Analyst @ UMASS-BOSTON]]></dc:creator>
            <pubDate>Mon, 02 Jan 2023 00:35:05 GMT</pubDate>
            <atom:updated>2023-01-02T03:34:15.699Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8GmZwsQHEvjx6k0u30FZfQ.jpeg" /></figure><h3><strong>Introduction</strong></h3><p>Data analysis is an increasingly important skill in the modern world. It’s a powerful tool for understanding large amounts of data and making informed decisions. To become a great data analyst, you need to have strong analytical skills, be able to interpret complex datasets and know how to communicate your findings effectively. Below are some tips on how to become a successful data analyst.</p><p>By following these tips and gaining experience with data analysis tools like Excel or Tableau, you can develop the necessary skill set to excel in this field.</p><h3><strong>Learn the Basics</strong></h3><p>Before diving into data analysis, it’s important to have a strong foundation in the basics of mathematics and statistics. Brush up on your algebra, calculus, regression, probability theory, and other key concepts that make up the backbone of data analysis.</p><p><strong>I recommend the following books:</strong></p><ol><li>Introduction to Probability and Statistics Using R by G. Jay Kerns</li></ol><p>2. Linear Algebra Done Right by Sheldon Axler</p><p>3. All of Statistics: A Concise Course in Statistical Inference by Larry Wasserman.</p><p>If you come from a math or statistics background, you may have taken some of these courses already, but it’s always good to brush up on your skills.</p><h3><strong>Get Familiar with Data Analysis Tools</strong></h3><p>There are many tools out there for analyzing data such as R programming language, Python, Excel, etc., so it’s important to become familiar with these tools before attempting any complex analyses.</p><p>Start by learning how to use one or two programs well before moving on to others. R programming language and Python are two of the most popular tools for data analysis, so it’s a good idea to start with these.</p><p>You can find many tutorials online to get started with these tools. Like any other skill, it takes time and experience to become proficient in these tools. And the best way to master a skill is to practice it regularly.</p><h3>Develop Problem-Solving Skills</h3><p>It is essential for a competitive data analyst to be able to identify problems and come up with solutions using available resources effectively and efficiently.</p><p>This requires an analytical mind capable of breaking down problems into smaller parts that can be solved individually until the whole problem is resolved. This is a skill that can be developed over time through practice.</p><h3>Learn How to Communicate Findings Effectively</h3><p>It is important to learn how to effectively communicate your findings. This means being able to explain the results of your analysis in an understandable and concise way that can be easily understood by non-experts. It also involves presenting your data visually so that it can be quickly reviewed and interpreted.</p><p>You should have strong written communication skills as well as the ability to present your findings clearly through charts, graphs, tables, etc. Additionally, you should understand how to tell stories with your data in order to draw out meaningful insights and conclusions for decision-makers.</p><p>A good place to practice your communication skill is to present to your peers or friends. You can also practice by writing a blog post about your findings.</p><h3>Practice Your Skills</h3><p>The best way to hone your skills as a data analyst is through practice! Take part in online challenges or competitions related to data analysis or join forums where professionals discuss their experiences working with different datasets or software packages used in analytics projects like Tableau or SAS Enterprise Miner etc.</p><p>If you are a beginner, you can start by taking up small projects and working on them. You can start with simple datasets and work your way up to more complex ones.</p><p>In conclusion, becoming a great data analyst requires dedication and practice. It is important to learn the basics of mathematics and statistics, become familiar with data analysis tools such as R programming language or Python, develop problem-solving skills, and learn how to effectively communicate findings. By following these tips and gaining experience with data analysis tools like Excel or Tableau, you can develop the necessary skill set to excel in this field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfee186cb964" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>