<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ANDREY FATEEV on Medium]]></title>
        <description><![CDATA[Stories by ANDREY FATEEV on Medium]]></description>
        <link>https://medium.com/@ANDREY_FATEEV?source=rss-ba326de0c606------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T8CF2FNIHwQ8xg56CMfAZw.jpeg</url>
            <title>Stories by ANDREY FATEEV on Medium</title>
            <link>https://medium.com/@ANDREY_FATEEV?source=rss-ba326de0c606------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:47:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ANDREY_FATEEV/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ANALYSIS OF DRUGS / ALCOHOL INVOLVEMENT IN FATAL TRAFFIC CRASHES]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/analysis-of-drugs-alcohol-involvement-in-fatal-traffic-crashes-580922c86dc5?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/580922c86dc5</guid>
            <category><![CDATA[traffic-crash]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[analysis-of-drugs]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 14:01:11 GMT</pubDate>
            <atom:updated>2021-10-20T14:01:11.073Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Motor vehicles are very important small transportation in everyone’s life, which used for moving people and cargo from one place to another place. It is relatively easy to operate and does not require a lot of knowledge for the test or documents to drive within the United States. However, it is required to have a certain level of law knowledge along with basic driving rules. For a fact, any DMV in Europe country would test driving abilities and law knowledge several times before letting a person to operate any vehicle, but we will get back to it later.</p><p>American company “AAA” in 2015 made a correlation in their studies and compared the number of fatal crashes in two decades, within the United States. Studies showed that young drivers more often getting in fatal crashes rather than it was 10 years ago. If that is true, what is the cause of growing numbers of fatalities? Luck of the experience from the younger generation, reckless driving, irresponsibility and unlawful behavior (drugs / alcohol), or just an accident? If the responsible authority can identify the problems and put enough afford to solve this issue, it will be possibly to save a huge number of lives and lower the number of accidents in the future.</p><p>The causes and pattern of fatal traffic crashes can be identified by this research and used for feasible solution and decision making. The overall analysis is based on a dataset named US_FARS DATA, FARS stands for “Fatality Analysis Reporting System”. By using that dataset, overall analysis, research and problem-solving process of causes and pattern are done. The analysis result can be helpful for the government and regulation authorities to decrease the number of fatal crashes within the United States.</p><h3>Data Preparation</h3><p>Name of the dataset used here is US_FARS DATA and provided by FARS. There are 27 dimensions and 2,126,828 rows of records within that file. By selecting relevant key variables, we left joined FAR table and dimensions to pick up all the relevant fields needed for the analysis. The focus of that research will be, “Alcohol and Drugs Involvement in Fatal Traffic Crashes” as most of the dimensions related to alcohol and drugs involvement. It is relatively easy to test motorists to determine whether they are under the influence of alcohol or drugs and determine whether it was a factor in a crash. However, we will correlate this information with age group, year, location and law involvement to find the causes and patterns. This file contains valuable information like police assumptions on driver influence by drug substances or alcohol, day of week, year, person type, drugs/alcohol involved, if person (driver) tested or not and which tests did they used.</p><h3>Analysis</h3><p>On preliminary analysis we have an assumption that for that past 10 years drivers killed in crashes were more likely to be on drugs than drunk (Appendix 1), but the total fatality crashes within the United States from 1995 to 2015 lowered by 25%. It raises a serious concern at a time when the United States is facing an epidemic of opioid usage and more states legalizing soft psychoactive substances — drugs for medical and recreational usage. How big is the problem, if some of the young generation thinking that soft drugs, for instance opioids do not impair their ability of driving and some even believe that it makes them safer drivers. We have heard numerous times and stories from people who is driving constantly either on drugs or alcohol, sometimes even both and saying it makes them calmer and more relaxed. However, the effects of specific drugs on driving skills depending on how they act in the brain. For instance, marijuana can slow the reaction time on the roads while driving, which can cause a serious accident, or alcohol is a widespread euphoriant and acts as a centra nervous system depressant, that may cause the driver to fall asleep or on the opposite side driving reckless.</p><p>By analyzing the overall dataset and different variables from a different perspective, we can see that there are some top States in using drugs and alcohol while driving with a significant number of fatal crashes occurred. From different visualizations and analyses, we have drawn those insights.</p><p>After Alcohol, marijuana Is the drug most often found in the blood of drivers that involved in fatal traffic crashes (Appendix 2). Since 2012, nine states started decriminalization and recreational of soft drugs, the first states were Colorado, Washington, California, Alaska and Oregon. In most of the mentioned states soft drugs used to be as a medical substance for a while, which gives it easier accesses to the people, for instance California were people have an easy access to soft drugs, it all matters while looking at the analysis of fatal crashes, on the other hand we have states such as Louisiana, Texas were even nowadays all the substances are illegal, but alcohol prevails. In Appendix 3, we can clearly determine the spikes in fatal crashes, using alcohol versus marijuana dimensions. It is clearly that alcohol dominates in tests, however since 2012 when it was first legalized as recreational, we have a progressive upward trend in marijuana been involved in state such as California and later even in Louisiana. The appropriate population ratio used, for one fatal accident in Louisiana there are five fatal accidents in California.</p><p>The overall drug usage that involved in motor fatal accidents in State of California increased by 11% since 2012–2015 and by 10% in state of Louisiana, in only 3 years (Appendix 4). The population, age group with the greatest number of accidents for drugged driving will be teenagers and relatively young adults with the group age 18–24, which can be correlated with an easy access to drugs in certain states and black market of drugs in states where it is illegal (Appendix 5). We also check on adults and seniors as they usually take prescribed medicine, such as painkillers and usually overtaking it, however there was no sign or spikes in analysis of adults and seniors, but we can see on Appendix 6 that overall usage of drugs relatively in stagnation compared to 24-hour usage of alcohol, which tells us that people may take prescribed medication in the morning, noon and evening, compared to alcohol where obvious spikes in the evening after 5–6pm. Does that mean that unlawful behavior towards drinking and driving is more relevant compared to drugs involvement as most of them prescribed?</p><p>Yes and No, as it was mentioned in Appendix 2, cannabinol is number one drug used while driving and it is growing rapidly. Police usually determine unlawful driver by his/her behavior, besides fatal injuries where blood test is the only way to find out what was the cause, they do not usually test on drugs if the driver has reached an illegal blood alcohol level, which is enough information for DUI.</p><p>If we look at the current situation, we should pay attention to alcohol, as for example since 1995 to 2015 on July 4th (Appendix 7) the biggest percentage of traffic accidents reported was due to alcohol, however the overall drug trend is prevailing every single year. Even though alcohol stays number one awareness for the police and independent authorities, drugs are playing bigger and bigger role in our lives.</p><p>It clearly shows the trend of marijuana-related traffic deaths in Colorado increased by 120 percent from 2012–2015, after the state legalized recreational (Appendix 8).</p><p>If we look in detailed analysis on 24-hour chart with different drugs, overall usage (amount) of cannabinol will be first again, and only then prescribed medical drugs, which means overall marijuana trend is growing and it is more relevant than any other drugs or alcohol (Appendix 9).</p><h3>Conclusion</h3><p>What actions people can take in order to prevent drugging driving? The first thing they should develop social strategy in order to teach young generation to prevent them from getting behind the wheel while on drugs or alcohol. For instance, in Europe they have high school classes where professors teach young generation about the drug-alcohol law and involvement in driving, in order to be a responsible citizen not only for your own life but everyone else life, as well as driving abilities and rules in 3 stages. Minimum Driving Age should be changed from 16 to 18, besides having a lack of experience in driving due to age, clearly see in Appendix 10, young people could be easily influenced to something that illegal with little or no responsibility at all.</p><p>Sometime the cause of fatal traffic accidents can even be legally prescribed medicine by the doctors for mental health illness, or in basic wisdom teeth removal. Being itself a strong antidepressant and mixing it with alcohol and other drugs can cause a serious consequence in our lives.</p><p>Especially looking at the worldwide statistic, where United States leading in mental health illness, it is required to take a serious action in order to minimize traffic accidents.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*EY61QQOeJ03cdADe.png" /><figcaption><strong>Appendix 1</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*TGRjvrjxt8IbJ-Na.png" /><figcaption><strong>Appendix 2</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/0*rNeWweBUGmUFFrRe.png" /><figcaption><strong>Appendix 3</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*8kwFl1H7EiS0HFSD.png" /><figcaption><strong>Appendix 4</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/0*G4zSlv09xPXQOLLc.png" /><figcaption><strong>Appendix 5</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/0*QVmMxXsDAUPgNGT9.png" /><figcaption><strong>Appendix 6</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*LsZmth0gOEs2lWMi.png" /><figcaption><strong>Appendix 7</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*_ZX4r_htPKxSwYIe.png" /><figcaption><strong>Appendix 8</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/0*y4IyJWzlNfkpAL-g.png" /><figcaption><strong>Appendix 9</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/0*YRlzAg__wprxr28n.png" /><figcaption><strong>Appendix 10</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=580922c86dc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What variables must be considered to predict the price of a real estate property in Baton Rouge, US.]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/what-variables-must-be-considered-to-predict-the-price-of-a-real-estate-property-in-baton-rouge-us-75588007c891?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/75588007c891</guid>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[predictions]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:58:20 GMT</pubDate>
            <atom:updated>2021-10-20T13:58:20.311Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>The variables included in any statistical model will depend on its purpose, market segment (industry), and spatial/temporal perspective. The goal is to determine what type of variables are significant in predicting house prices within one state.</p><p>The four main key factors that we have to consider are demographics, simply the compositions of the population by race, gender, age, and income as a prominent example Beverly Hills in CA versus Baton Rouge. In addition, interest rates, the state and federal economy, and government policies with their determination of property demand and prices on the estate are all critical variables.</p><p>If we go online and start searching for a new house, the first questions we usually ask are location, then a type of estate we are looking for, home or condo, size in square feet, and how many rooms and restrooms are within the property.</p><p><strong>3V’s.</strong> Big data incessantly contains important information or valuable insights, critical for business success in our case, its price predicting. The challenge we might face is analyzing and processing all the data to derive the set of significant variables. Big data is just the data generated from internet-connected devices, can be different types and forms, simply highly diverse.</p><p>We might face the size (volume) of the available estates on the website or in data storage. Due to the competing independence of companies with their policies and personal information protection, sometimes is not enough time to collect and analyze the data of different properties. On top of that, companies struggle with analysis due to the variety of data collected, which could be structured, semi-structured or unstructured data, as we all want to see, for example, pictures of the property and not just prices and how many rooms it has or any type of links for the website where this property is located.</p><p>All of the data is important to analyze the property but must be prepared correctly following the method chosen. For example, most agencies use regression models and what’s called the hedonic regression method of estimating the demand for a good.</p><p>In our research, we would use JMP to find what significant variables are in determining the real estate prices within the Baton Rouge area. Our focus will be on properties with a similar characteristic to avoid any biases.</p><h3>Data Understanding and Analysis</h3><p>The “<strong>Redfin_2018_Houses</strong>” JMP file contains 2,150 records of houses across Louisiana. Before analysis can be performed, a significant amount of data preprocessing was necessary, including variable reduction, treatment of missing values, screening for outliers, and recoding. Since “City” is nominal, we want to use Recode. Then, we excluded all properties outside Baton Rouge, and we have 2098 properties available to analyze within the Baton Rouge area. We can also use a query and filters to exclude properties we think might not be needed for the analysis.</p><p>Since there are several property types within the data set, as we stated before, we want to avoid any biases and make a model for properties with a similar characteristic to single-family residents and townhouses and Condo/Co-op. We won’t include multi-family<em> </em>or mobile/manufactured homes as the prices are different and don’t have similar characteristics.</p><p>Screening for outliers and missing values was performed.</p><p>We have a couple of outliers within the price, days on the market, HOA/month, and lit size, and we want to exclude these rows.</p><p><em>Our variables will be: price, beds, baths, square feet and $/square feet, lot size, year built, days on the market, HOA/month.</em></p><p>The number of missing values is: 0, 1, 0, 1 and 1, 1556, 61, 15, 787, respectively (Appendix A). From 1780 properties left in our data set, the biggest number of missing values would be within “lot size,” “year built,” and “HOA/month.”</p><p>The more missing values we have, the worst variable it is for predicting the prices of properties. Therefore, for beds and square feet, we used Automated Imputation to impute those two values. As a result, 3142 missing values were replaced, and dimension two approximation was empirically selected, 1570 values excluded.</p><p>Analyze distribution performed for price, beds, baths, square feet, and year built (Appendix B). From distribution analysis, we conclude that most of the properties within the data set are newer properties with 3–5 bedrooms, 2–4 baths, from 1000 to 3000 square feet, and a price range from 200k to 400k dollars. Most of the properties are relatively big, and the market is oversaturated with a big square foot, many beds, and bathes properties within the data set. Los size has a lot of missing values, wasn’t considered for distribution analysis, and year built, especially for the lower-priced properties.</p><p>From the map (Appendix C), we added zip code and price. We can conclude that zip codes of the properties are very randomized, show us different areas of the zip on the map, red dots on the left side, some of them on the right looks like a red triangle with a lot of blue dots (different zip code areas) in the center. Prices north of Baton Rouge have a smaller price on properties than in the middle of Baton Rouge.</p><h3>Conclusion</h3><p>The average annual income in Baton Rouge is $44,470.00, with the median cost of the houses of $339,950.00. It must be a big challenge to own your property. The average property has four bedrooms and three baths. For the sake of prices and average annual income, companies should have relatively small parcels if we are trying to avoid homelessness. Therefore, beds and bathes play a significant role in predicting the price of real estate. As well as location from the map appendix, zip codes and prices correlated, better community based on zip code meaning higher prices.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*gEtZV82flOCyWgxW.png" /><figcaption><strong>Appendix A — Table info, after excluding all missing and outliers.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*pDxBkViLbeaC6wW2.png" /><figcaption><strong>Appendix B — Distribution.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*s0QI0uDgMniozpcf.png" /><figcaption><strong>Appendix C — Map of Zip and Prices.</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75588007c891" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic Regression, Decision Tree, Bootstrap Forest, Boosted Tree, Neural Network, and K-nearest]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/logistic-regression-decision-tree-bootstrap-forest-boosted-tree-neural-network-and-k-nearest-7eb40e74c33d?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/7eb40e74c33d</guid>
            <category><![CDATA[neural-network-algorithm]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:56:38 GMT</pubDate>
            <atom:updated>2021-10-20T13:56:38.493Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>The research objective is to build the following models (Logistic Regression, Decision Tree, Bootstrap Forest, Boosted Tree, Neural Network, and K-nearest neighbor) and assess the results using metrics learned in this course. The data set contains the list of applicants applying for housing loans/mortgages in Louisiana. Different factors within the data set are involved in decisions whether the applicant will be proved or denied, such as the other purposes of loans, which will be predictor variables. The objective of this assignment is to predict which applicants would be rejected loans for financial institutions as this information will determine the potential risk of people that not be able to pay the loans back. Another objective is to find the best model (averaged) that will predict the pros and cons of using the chosen model.</p><h3>Data Understanding and Analysis</h3><p>The overall analysis will be based on a dataset named “HMDA_data_known” using JMP with Principal Component Analysis (PCA). The data set consists of 166,800 records. Before analysis can be performed, a significant amount of data preprocessing was necessary, including treatment of missing values and screening for outliers. For all the variables, we excluded all the rows to avoid biases. Missing values and screening for outliers were performed. Appllncom_num &amp; loanamount_num have 948 and 467 outliers, respectively. reteSpread_num has 156,218 missing values, it is a lot if we compare it to the entire data set, and it’s not a significant variable to use for predictions. All missing values were recorded, and the design is missing. Also, to avoid the curse of having too many dimensions, we have to reduce the number of values for each variable and the number of continuous variables.</p><p>The training set is used to fit a model into a validation set to check if the model validates in a different data set. We set our training set, validation, and test set as 0.60, 0.30, and 0.10, respectively. Further, we evaluated each variable for its relationship with the binary response variable as denied = 1. By using tabulate in JMP and calculating the percentage of it, we can perform binning. Variables, such as COUNTY was binned into 20 separate bins, FFIECMedianFamilyIncome_num was also grouped into 3 bins, as 46,900–52,400 — LowIncome, 54,500–58,300 -MediumIncome and 60,000–65600- HighIncome. The models mentioned above were evaluated to identify the model with the lowest error rate and then compared to the test set and the best fit model with Sensitivity and Specificity. Then the model averaging platform in Model Comparison in JMP is used to create a comparison model.</p><p><strong>Logistic Regression.</strong> Each variable has LogWorth, a function of p-value and representing the ranking of importance for each variable. Even though all the variables were found to be statistically significant, in (Appendix A), LoanPurposeDescription has the highest LogWorth of 915,748, which is the most statistically significant variable. The overall error rate was 23%, simply obtained by dividing the sum of false predictions by the total number of cases. It is essential to understand that the Confusion Matrix evaluates the effectiveness of the model prediction.</p><p>Constructed data is used to explain the various metrics that are available to evaluate a model. The matrix shows the outcome of the predictions of 37,067 loan applicants, of which 26,016 applicants were correctly classified by the model as approved for the loan, while 2,633 applicants were correctly classified as denied for the loan/mortgage (Appendix B). For the applicants being rejected, the Logistic Regression model correctly classifies 29% of the time, called <em>sensitivity</em>. The model accurately predicts 93% of the time for denied cases, as defined by the <em>Specificity</em>. The percentage of predicted declined applicants who were approved was 42%, which is <em>a false</em> <em>positive rate</em>, and the percentage of expected agreed but were denied at 20%, which is called <em>false-negative rate</em>. <em>ROC curve</em> is used to compare models, showing what happened to the sensitivity when the cutoff values are decreased from 1 to 0 vs. 1 specificity. In the chart with the range from 0 to 1, AUC, which is the area under the curve, is standardized to be 1 for a model that correctly classifies all cases. In our case, the ROC chart for the validation data set and AUC number is close to 0.7895. If the curve is closer to 1, the better model we have. The <em>lift curve</em> is another method of evaluating the gain in the prediction. Our lift curve has a lift of more than 2.5, and ideally, it should decline through 30% of the data, which is a good feature of the model. However, we see that at the beginning, it started to decrease by about 10% and then constantly declining. A lift curve of 2 is about 27% of the ranked data, which is a decent percentage.</p><p><strong>Decision Tree.</strong> For every model, we used the same variables as it was for the Logistic Regression. We added additional 2 predictor variables (ApplIncome_num &amp; loanamount_num), and pressed Go, 42 splits. The final error rate was 20%, with a Sensitivity of 42% and a Specificity of 93% (Appendix C).</p><p><strong>Bootstrap Forest.</strong> Default setting used, one predictor variable had no significance, therefore was removed from the test. The minimum splits per tree were 10, and the minimum size split was 166. The overall error rate was at 21%, the sensitivity of 33%, and Specificity of 96% (Appendix C). The false-positive rate is at 31% and false-negative at 20%.</p><p><strong>Boosted Tree.</strong> The same variables were used as in the previous models, and the default setting changed. The number of layers increased to 200 and split per tree. For this model, we decided to include HOEPADDescritiopn as we increased the set of the model. Withing Boosted Tree, one of the most critical variables was RaceDescription. Results in (Appendix C).</p><p><strong>Neural Network.</strong> 2 types of Neural networks were boosted, and the default one models were evaluated. In (Appendix D) all models that we performed to choose the best one. The default Neural Network was performed first, with 3 TanH, 0 Linear, and 0 Gaussian nodes. The overall error rate was 22%. Then, the model with one hidden layer and 3 TanH, 2 Linear, and 2 Gaussian nodes was performed. Again, the overall error rate was 21%. Finally, boosting was completed to this model with 10, which has an overall error rate of 21% (Appendix C).</p><p><strong>K-nearest neighbor.</strong> This model usually takes much time. To save some time, we used fewer variables. To do so, we used screening under response screening and chose variables with the highest LogWorth. Also, the model does not require grouping. However, it performed faster when we used it. The final model has a K-Nearest of 9 with an overall error rate of 21% (Appendix C).</p><h3>Conclusion</h3><p>The objective of the models was to determine who would be approved for a loan and who would be denied based on the data set with comparison (Appendix E). All the models performed relatively the same, but Boosted Tree has the lowest error rate at 19%, the highest ROC curve of 83%, and the best Lift Curve at 0.37, which is the best performance out of all the models. Also, having the lowest false positive rate at 31%, this model shows that based on the results and relatively high sensitivity &amp; Specificity, it can predict if people will be denied for the mortgage or not. Overall, data had 27% of applicants who were rejected, of the data model indicated 12.66% were rejected and 86.2% approved for the loans.</p><p>The potential biases were related to overfitting the models, the reason we used a validation set and added variables to the data set. The second-best model was the Decision tree but had a high False positive rate, which is not the best in predicting denied applicants.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/0*xktUG8xdbB9_sZT2.png" /><figcaption><strong>Appendix A — Logistic Regression.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*wrBnYtEhVG7lTB_K.png" /><figcaption><strong>Appendix B — Example of Confusion Matrix.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*xZiJLSWFl2IEpYO7.png" /><figcaption><strong>Appendix C — Summary Table.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/0*HeuHPjjDwrWhuUEX.png" /><figcaption><strong>Appendix D –Activation Functions for Neural Network Models.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/0*c95c5L3dpQhAzrH0.png" /><figcaption><strong>Appendix E –Model Comparison with ROC and Lift Curve.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*CreypFLjsZDyb7nM.png" /><figcaption><strong>Appendix E –Model Comparison with ROC and Lift Curve.</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7eb40e74c33d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Tree model, Bootstrap Forest, and Boosted Trees to predict alcohol involvement (BAC>=0.08)]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/decision-tree-model-bootstrap-forest-and-boosted-trees-to-predict-alcohol-involvement-bac-0-08-ac14c1b1a89d?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac14c1b1a89d</guid>
            <category><![CDATA[alcohol-involvement]]></category>
            <category><![CDATA[bootstrap]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[prediction-markets]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:53:15 GMT</pubDate>
            <atom:updated>2021-10-20T13:53:15.332Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>This research aims to build a model and run a decision tree model for each predictor to bin all predictors and bootstrap forest and boosted trees models to predict the driver’s alcohol involved in a crash.</p><h3>Data Understanding and Analysis</h3><p>The overall analysis will be based on a dataset named “Alcohol Prediction Data” using JMP with Principal Component Analysis (PCA). The data set consists of 92,924 records; before analysis can be performed, a significant amount of data preprocessing was necessary, including treatment of missing values and screening for outliers. In addition, for all the variables, we excluded all the rows to avoid biases.</p><p>Also, to avoid the curse of having too many dimensions, we have to reduce the number of values for each variable and the number of continuous variables. Finally, the training set is used to fit a model into a validation set to check if the model validates in a different data set. We put our training set, validation, and test set as 0.65, 0.20, and 0.15, respectively.</p><p>The formula used in creating a new column called “DWI” whereas BAC&gt;=0.08 = 1, else 0.</p><p>Our initial process of binning to achieve lower error rate as well as added additional independent variables, instead of 10, we using 11 variables which are: Injury, Gender, Age group, PrevDwiDescription, AnumOccs, HitRunDescription, Month, Hour, Day_Week, AngleDescription, and RestUse.</p><p>To make the first model that is Partitioning, which builds decision trees to predict the response variable. The objective of this model is to watch for R-square, which is based on the difference between observed cell frequency and expected cell frequency, and split the variables with the highest LogWorth, and until Rsquare doesn’t change, it will be stable. Instead of using p-value, our attention is to select the variables with the highest LogWorth. We chose DWI as a response variable and all necessary or statistically significant variables, which is 10: Hour, Day_Week, Age Group, Month, AnumOccs, PrevDwiDescription, HitRunDescription, DrugResultDescription, Sex Description, and Injury as our variables for the test. The overall error rate is 23%, simply obtained by dividing the sum of false predictions by the total number of cases (Appendix A). The model correctly classifies 61% of the time for the alcohol-impaired cases, called sensitivity. For non- alcohol-impaired issues, the model accurately predicts 84% of the time, as defined by the specificity. The false-positive rate is the percentage of positive (1-Yes) predictions that are wrong — 38% and the false-negative rate, which is the percentage of false negative (0-No) — 17%.</p><p>ROC curve is used to compare models, showing what happened to the sensitivity when the cutoff values are decreased from 1 to 0 vs. 1 specificity. In the chart with the range from 0 to 1, AUC, which is the area under the curve, is standardized to be 1 for a model that correctly classifies all cases. In our case, the ROC chart for the validation data set and AUC number is close to 0.80. Therefore, if the curve is closer to 1, the better model we have.</p><p>The lift curve is another method of evaluating the gain in the prediction. Our lift curve has a lift of more than 2.5, and ideally, it should decline through 30% of the data, which is a good feature of the model. However, we see that it started to fall by about 10%, and then constantly declining. The lift curve of 2 is a decent percentage for about 35–40% of the ranked data. This model shows us that it predicts no-alcohol involvement in crashes rather than indicating alcohol impairment in crashes within drivers. It has a similar outcome with ROC Curve and Lifts curve as it was for Prediction Model.</p><p>The next model we performed is Bootstrap, which builds a collection of decision trees using random sampling averages results to predict a response. Again, we used the default settings given by JMP, and we inserted values into the excel formula. Our error rate was 24%, with non- alcohol-impaired cases predicted 88% of the time, as defined by the specificity. False-positive rate — 34% and false negative rate– 21%.</p><p>For the ROC curve, the AUC number was not close to at least 0.80, which is not a great model based on the model we used before (Appendix B).</p><p>The last model would be Boosted tree, which builds decision trees that sequence of smaller trees to predict a response. We used the default settings as well as in Bootstrap Model. Our error rate is 23%, with non- alcohol-impaired cases, which indicates 88% of the time, as defined by the specificity — false positive rate — 33% and false negative rate– 20%. For the ROC curve, AUC number was similar as in the previous models (Appendix B).</p><h3>Conclusion</h3><p>The Decision Tree model, Bootstrap Forest, and Boosted Trees models were developed for the BAC data. Higher sensitivity was achieved in the Decision Tree model, which was one thing to improve for the overall results. Also, the error rate fluctuated from 23% to 24%, we can improve our model by reducing cutoff probability, but it will be reducing in specificity, which we tried to avoid, and improving Regression Model. Therefore, binning has to be redone entirely for the variables chosen; we believe a margin error of +2–3% can be reduced to 20%-21% of the Error Rate. Overall, the best results we achieved with the Regression model and the Decision Tree model, which ROC and Lift curves showing, we still can improve our model, but it would be based on the decisions above.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/0*ACEV9BvYJ1Xm2BDq.png" /><figcaption><strong>Appendix A — ROC Curve &amp; Lift Curve for Partitioning.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/0*eHaJ8pzUnvoMhtuk.png" /><figcaption><strong>Appendix B — ROC Curve &amp; Lift Curve for Random Forest.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/0*vntuliZxfGWViqC6.png" /><figcaption><strong>Appendix C — ROC Curve &amp; Lift Curve for Boosted Tree.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/0*RlQxnSDuVy8pO4mX.png" /><figcaption><strong>Appendix D –Summary with Confusion Matrix of all 4 models.</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac14c1b1a89d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prediction of alcohol involvement of the driver in the crash]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/prediction-of-alcohol-involvement-of-the-driver-in-the-crash-7689314132cf?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/7689314132cf</guid>
            <category><![CDATA[drugs]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[alcohol-involvement]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[crash]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:50:45 GMT</pubDate>
            <atom:updated>2021-10-20T13:50:45.342Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>This research aims to build a logistic regression model to predict the alcohol involvement of the driver in the crash.</p><h3>Data Understanding and Analysis</h3><p>The overall analysis will be based on a dataset named “Alcohol Prediction Data” by using JMP with Principal Component Analysis (PCA). The data set consists of 92,924 records. Before analysis can be performed, a significant amount of data preprocessing was necessary, including treatment of missing values and screening for outliers.</p><p>MOD_YEAR, AnumOccs, HOUR, FATALS, and BAC have a significant number of outliers that we excluded, 162, 14, 658, 30, and 2, respectively.</p><p>BAC and Age Desc have 6026 and 73 missing values that we excluded as well.</p><p>Also, to avoid the curse of having too many dimensions, we have to reduce the number of values for each variable and the number of continuous variables. Finally, the training set is used to fit a model into a validation set to check if the model validates in a different data set. We put our training set, validation, and test set as 0.65, 0.20, and 0.15, respectively.</p><p>Our data set can identify the binary response variable, which is either 0 or 1, depending on the outcome that we want to predict. The legal blood alcohol content/concentration (BAC) value you can have in your body system within the United States of America is 0.08, meaning that there is 0.08g of alcohol for every 100ml of blood if the number is bigger than 0.08 in our case, BAC number different from the 0 considered to be illegal, or driving under the influence of alcohol. The formula used in creating a new column called “DWI” whereas BAC&gt;=0.08 = 1, else 0.</p><p>For the binning process, we look at the specific variables: Age, Injury, Hour, Day of Week, BodyDescription, RestUseDescription, and AngleDescription of the drivers. Then, we build a chart Y by X; as DWI on the Y-axis and Injury and other variables on the X-axis, we must remember to use Local Data Filter as a training data set to avoid unbiases and to prevent overfitting that might occur when we manipulate the data and include the validation set and test set.</p><p>To make a confusion matrix regression model, we chose DWI as a response variable and all necessary or statistically significant variables, which is 10, where p-value &lt; 0.05, not every variable must be &lt;0.05. The limit in the research was 0.03. Target level to 1, as we want to predict DWI = 1, people involved in accidents with alcohol within their body system. In (Appendix D), each variable has LogWorth, a function of p-value and representing the ranking of importance for each variable. The misclassification rate of the validation set is 0.21. It is essential to understand that the Confusion Matrix evaluates the effectiveness of the model prediction. Constructed data is used to explain the various metrics that are available to evaluate a model. The matrix shows the outcome of the prophecies of 17214 drivers, of which the model correctly classified 10397 drivers as involved in the crash by alcohol influence, while 3047 drivers were correctly classified as not influenced by alcohol with BAC &lt; 0.08. The model classified 1352 drivers as having no alcohol involvement, however still getting into the crash, and 2418 drivers were classified as alcohol involvement but did not get in the impact.</p><p>The overall error rate was 35%, simply obtained by dividing the sum of false predictions by the total number of cases. However, after binning correction and substituting with the right statistically significant variables and no bias or unstable, our error rate becomes 22%, which is good. Thus, the model correctly classifies 56% of the time for the alcohol-impaired cases, called sensitivity. For non- alcohol-impaired cases, the model accurately predicts 88% of the time, as defined by the specificity. The false-positive rate is the percentage of positive (1-Yes) predictions that are wrong — 31% and the false-negative rate is the percentage of false-negative (0-No) — 19%. This matrix/model shows that it predicts no alcohol involvement in crashes than indicating alcohol impairment in crashes within drivers. (Appendix C).</p><p>ROC curve is used to compare models, showing what happened to the sensitivity when the cutoff values are decreased from 1 to 0 vs. 1 specificity. The chart with the range from 0 to 1, and AUC, which is the area under the curve, is standardized to be 1 for a model that correctly classifies all cases. In our case, the ROC chart for the validation data set and AUC number is close to 0.80. Therefore, if the curve is closer to 1, the better model we have. (Appendix A).</p><p>The lift curve in (Appendix B) is another method of evaluating the gain in the prediction. Our lift curve has a lift of more than 2.5, and ideally, it should decline through 30% of the data, which is a good feature of the model. However, we see that at the beginning, it started to fall by about 10%, and then constantly declining. Therefore, a lift curve of 2 is for about 35–40% of the ranked data, a decent percentage.</p><h3>Conclusion</h3><p>The logistic regression model was developed for the BAC data.</p><p>The sensitivity needs to be higher than 56%, and it could be achieved by adding additional variables, and fixing binning, to choose more linear relationships between variables that are statistically significant. Also, we can improve our model by reducing cutoff probability, but it will decrease specificity, which we tried to avoid, as we also want to correlate non- alcohol-impaired cases within the model. However, in the ROC curve, we still have 0.20 to improve.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*cff6pP_ff3qgdcmU.png" /><figcaption><strong>Appendix A — ROC Curve.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*ZllPICyNNlUPiS3a.png" /><figcaption><strong>Appendix B — Lift Curve.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*JELPy4SSXhDhpcNH.png" /><figcaption><strong>Appendix C — Confusion Matrix.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*s_UDLfSUmvWVWYbs.png" /><figcaption><strong>AppendixD — Effect Summary.</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7689314132cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Network and Boosted, Neural Network model, predicts the driver’s alcohol involvement…]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/neural-network-and-boosted-neural-network-model-predicts-the-drivers-alcohol-involvement-38736b59006e?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/38736b59006e</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[drugs]]></category>
            <category><![CDATA[alcohol]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:42:52 GMT</pubDate>
            <atom:updated>2021-10-20T13:42:52.599Z</atom:updated>
            <content:encoded><![CDATA[<h3>Neural Network and Boosted, Neural Network model, predicts the driver’s alcohol involvement (BAC&gt;=0)</h3><h3>Introduction and Business Understanding</h3><p>This research aims to build a model and run a Neural Network and Boosted Neural Network models for each predictor to predict the alcohol involvement of the driver in a crash.</p><h3>Data Understanding and Analysis</h3><p>The overall analysis will be based on the “Alcohol Prediction Data” dataset using JMP with Principal Component Analysis (PCA). The data set consists of 92,924 records. Before analysis can be performed, a significant amount of data preprocessing was necessary, including treatment of missing values and screening for outliers. In addition, for all the variables, we excluded all the rows to avoid biases.</p><p>Also, to avoid the curse of having too many dimensions, we have to reduce the number of values for each variable and the number of continuous variables. Finally, the training set is used to fit a model into a validation set to check if the model validates in a different data set. We put our training set, validation, and test set as 0.65, 0.20, and 0.15, respectively.</p><p>The formula used in creating a new column called “DWI” whereas BAC&gt;=0.08 = 1, else 0.</p><p>To make the first model — Neural Network for this research, we corrected our binning process by combining more variables, such as Race Description and Hour, Week. As a result, we would be using 11 variables instead of 10 as before in the last research: HitRunDescription, Injury, DrugDescription, AgeGroup, DayWeek, SexDescription, Month, AngleDescription, AnumOccs, DrugResults, PreDwiDescription.</p><p>Also, fixed binning to achieve better results as we don’t need one for the Neural Network model.</p><p>Neural Network is simply a machine learning process that looks at data and tries to figure out the function or set of calculations that turns the input variables into the output with the associated weight and of each neuron and modifies the strength of each input.</p><p>Neural networks learn from examples and exhibit some capability for generalization beyond the training data.</p><p>The objective is to build the model with a lower error rate, as no matter what we do, the Specificity error would always be more significant than the Sensitivity error. The first model we built was with only one layer, and we had an overall error rate of 25%, which is a lot even compared to the Decision Tree models. As we mentioned before, so far, the best model results were within a Regression Model with an error rate of 21% after fixing binning. On the other hand, even using a regression model, the ROC curve wasn’t that much closer to 1 for a better result.</p><p>With the option of three activation functions used in the hidden layer: TanH, Linear, and Gaussian, which are the hyperbolic tangent function often used for nominal variables, linear that is most often used in conjunction with the non-linear activation functions and gaussian that is radial basis function behavior.</p><p>We used a model with 2,3- and 6-layers (Appendix A as an example), with the lowest error rates. We flagged Transform Covariates as an absolute. The best ROC curve (Appendix B) was with activation functions of TanH: 6, Linear 0, Gaussian: 0. Even though we know that TanH would be the most suitable function for nominal variables, we tried to use different activation functions. We didn’t achieve the results we wanted to.</p><p>The overall error rate is 22%, simply obtained by dividing the sum of false predictions by the total number of cases (Appendix C). For the alcohol-impaired cases, the model correctly classifies 65% of the time, as called sensitivity. For non- alcohol-impaired cases, the model accurately predicts 92% of the time, as defined by the specificity. The false-positive rate is the percentage of positive (1-Yes) predictions that are wrong — 14% and false-negative rate, which is the percentage of false-negative (0-No) — 28%.</p><p>ROC curve is used to compare models, showing what happened to the sensitivity when the cutoff values are decreased from 1 to 0 vs. 1 specificity. In the chart with the range from 0 to 1, AUC, which is the area under the curve, is standardized to be 1 for a model that correctly classifies all cases. In our case, the ROC chart for the validation data set and AUC number is close to 0.85, the highest we have. Therefore, if the curve is closer to 1, the better model we have.</p><p>The lift curve is another method of evaluating the gain in the prediction. Our lift curve has a lift of more than 2.5. Ideally, it should decline through 30% of the data, which is a good feature of the model, so as we see that at the beginning, it started to fall in about 10% sharply and then constantly declining. The lift curve of 2 is about 40–45% of the ranked data, which is 5–10% better than a Decision tree model.</p><p>Overall, this model performed better than any other in determining non-alcohol impaired cases.</p><p>For the next model that we performed, called boosted model, the second layer is not allowed, the number of models chosen to be three.</p><p>For the first, we used the default settings as JMP gave it, and we after inserted values into the excel formula, our error rate was around 24%. We decided to take the activation function of TanH = 6 and kept flagged Transform Covariates as an absolute.</p><p>The overall error rate becomes 23%, Sensitivity 64%, which is not a big difference compared to the model that is not boosted, and for non- alcohol-impaired cases, the model correctly predicts 90% of the time, as defined by the specificity: false-positive rate 13% and false negative rate 29% (Appendix C). ROC curve stays pretty much the same, around 0.83, and lift curve of 2 is for about 5–7% drop compared to the original model.</p><h3>Conclusion</h3><p>Neural Network and Boosted Neural Network models were developed for the BAC data. Even higher sensitivity was achieved compared Decision Tree model, which is a work of binning. Also, the error rate fluctuated from 23% to 24% in the previous research, and we have achieved better results. Overall, ROC and Lift curves show that the best results we achieved with the Neural Network model. Even if we compare to the regression model, we still can improve our model, but it would be based on binning process and levels of layers.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/0*Oc4ZvGLFt-yhqQ12.png" /><figcaption><strong>Appendix A — Example of Neural Network Diagram.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/0*9yu5_DjVqEHX10X7.png" /><figcaption><strong>Appendix B — ROC Curve &amp; Lift Curve for Neural Network.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/0*XhB0ejDasvX-DbID.png" /><figcaption><strong>Appendix B — ROC Curve &amp; Lift Curve for Neural Network.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/0*FUBrN_I_hYeucdUi.png" /><figcaption><strong>Appendix C — Confusion Matrix for Neural Network and Neural Network Boosted.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/0*KKbpRvyuDvfowrnN.png" /><figcaption><strong>Appendix C — Confusion Matrix for Neural Network and Neural Network Boosted.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/0*TpObUdoMq7m5Le_a.png" /><figcaption><strong>Appendix D — ROC Curve &amp; Lift Curve for Neural Network Boosted.</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38736b59006e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Analysis of Causes of Accidents in Flight School]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/text-analysis-of-causes-of-accidents-in-flight-school-b2240f771773?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2240f771773</guid>
            <category><![CDATA[text-analytics]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[accident]]></category>
            <category><![CDATA[text-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:38:21 GMT</pubDate>
            <atom:updated>2021-10-20T13:38:21.480Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>The data that we will analyze is “Aircraft accident reports from NTSB of 1,906 aircraft accidents”.</p><p>Flight School experiences a high-volume aircraft accident and wants to get recommendations from analysts to determine the causes of accidents and in what phases accidents occurred in Instructional Flights and give some corrective actions to reduce these accidents potentially. Flight School also wants Instructional flights, highlights its percentage of crashes, and relates to Personal flight crashes.</p><h3>Data Understanding</h3><p>The “Aircraft Incidents (1)” JMP file contains 1,906 accident reports or number of cases. The data set included 28 categorical variables, and our focus is on “Narrative Cause,” and we will be focusing on “Purpose of the Flight” and in what Phase the aircraft was during these accidents. In the “Purpose of the flight” category, 1,906 of total flights, “Unknown” is nine flights, and we are missing additional information from reports on 158 flights. Our concern is within Instructional Flights as the company has asked even though it’s ranked second in the highest percentages of accidents, first Personal Flights. Flight School had 1092 or 57.3% accidents when they used aircraft for personal purposes and 272 or 14.3% for Instructional Purposes.</p><p>The biggest concern in Instructional flights is that most accidents happened while the plane was landing — Landing Phase, which is 123 times out of 272 or 45.22% of total casualties. The second biggest Phase with accidents is Taking off the aircraft with 46 times or 16.91% of total accidents.</p><h3>Analysis</h3><p>Here we analyze Instructional Flights in the Landing Phase.</p><p>Using <strong>Tabulate</strong>, we’ve created a table showing that the highest percentage of accidents happened in Personal and Instructional Flights 62.47% and 15.56%, respectively. Landing is the highest “Broad phase of flights” for both Personal and Instructional Flights with 27.29% and 45.22% respectively, since Instructional flights have the most significant percentage of crashes in the landing phase, it is our concern from now. Appendix A.</p><p><strong>Text Frequency and Word Cloud</strong> were created. The Text Explorer platform was used to analyze the text. No stemming was selected, and the standard options for words and characters were used. First, the text explorer produces a term and phrase list shown in Appendix B. Next, a local filter was used to obtain the frequencies for Landing Issues only, also shown in Appendix C.</p><p>*Excluding Landing in Instructional Flights, “landing,” “failure,” and “pilot’s” appeared 166, 162, and 125, respectively.</p><p>*Including Landing phase in Instructional Flights, Local Filter used, “landing,” “pilot’s” and “failure” appeared 118, 73, and 68, respectively. If we use to sort similar words and phrases, we will get: land*, pilot* and student* — 122, 105, and 72 times, respectively. However, terms won’t change. By using stemming for combining, the word “student” appears more than “failure.”</p><p>The <strong>Word Cloud</strong> was created to gain a general understanding of the most frequent words Appendix D. The size of each word was proportionate to its frequency. For example, the most frequent word is Landing.</p><p><strong>For phrase frequency in Instructional Flights including Landing,</strong></p><p>“directional control”, “student pilot’s,” and “failure to maintain” used 38, 38, and 33 times, respectively.</p><p><strong>Singular Value Decomposition (SVD)</strong></p><p>Next, Singular Value Decomposition (SVD) plots for the first five singular values for Terms and Document were created for Terms and Documents, as seen in Appendix E, F, and G. SVD analysis groups similar documents into topics. We can cluster text documents or cluster terms that are in a collection of documents.</p><p>Comparing these two appendixes on failure terms shows that the highest failure with Landing Phase is “student” 41%. However, if we exclude Landing “student,” failure appears only at 21%. Looking closer to the Landing Phase, we can tell that the highest percentage of terms related to the accidents are: student issues, landing, and flight issues.</p><p>This implies that a smaller number of word combinations explains the causes of aircraft accidents better. For instance, there could be issues with students due to a lack of experience and knowledge. A higher percentage on the chart tells us which Phase of the flight in student failure occurred more. Appendix G. SVD plot contains a point for each (document, case, row). In this SVD plot, points that are visibly grouped indicate documents with a similar composition of terms.</p><h3>Conclusion</h3><p>The use of Text Explorer in JMP provides some insight into what causes the accidents to happen within Instructional Flights and how to reduce these accidents potentially.</p><p>Instructional flights are second with the percentage of accidents and reports. If we look into phases, Landing accidents occur the most. In reports in that section, we determined the most mentioned terms: landing 118 times, pilot’s 73 times, and failures 68 times. Phase analysis shows the most mentioned phases used in reports causes of the situation and how to reduce them potentially.</p><p>“directional control” — failure to maintain directional control with poor wind conditions.</p><p>“student pilot’s” — young pilot’s failures due to lack of knowledge and experience.</p><p>“failure to maintain” — failures to maintain the aircraft in certain situations before the accidents.</p><p>The SVD helps to distinguish better a cause in different phases by using singular values and topic analysis.</p><p>The highest percentage of terms related to the accidents are student issues, landing, and flight issues.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OqApJTQ8bMiTiyu9.png" /><figcaption><strong>Appendix A. Cross tab Purpose of Flight versus Broad Phase of Flight.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*eDb13zn6VhiEL2jP.png" /><figcaption><strong>Appendix B. Term &amp; Phrase List for Instructional excluding Landing.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*BOGDAP5KUcu6-eNw.png" /><figcaption><strong>Appendix C. Term &amp; Phrase List for Instructional &amp; Landing.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*hKFq-EPbrilABFrp.png" /><figcaption><strong>Appendix D. Word Cloud with Landing and Instruction as Local Filter.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*PhRx5FGIwoIAPW3-.png" /><figcaption><strong>Appendix E. SVD Plot for Terms only Landing.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/0*4Q7Ps7aKO1e7ainx.png" /><figcaption><strong>Appendix F. SVD Plot for Terms Excluding Landing.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*e_V5EvsSmtlYSFSd.png" /><figcaption><strong>Appendix G. SVD Plot for Documents.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/0*aZ-QrpQz0wrLfsiT.png" /><figcaption><strong>Appendix H. Singular Values with Landing and Instruction as Local Filter</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2240f771773" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A cluster analysis on housing data from Baton Rouge, Louisiana, provided by Redfin.com]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/a-cluster-analysis-on-housing-data-from-baton-rouge-louisiana-provided-by-redfin-com-d541ad195c7f?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/d541ad195c7f</guid>
            <category><![CDATA[cluster-analysis]]></category>
            <category><![CDATA[housing]]></category>
            <category><![CDATA[redfin]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[baton-rouge]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:35:08 GMT</pubDate>
            <atom:updated>2021-10-20T13:35:08.504Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>The data that we will analyze is “Housing data from Louisiana.” In our analysis, we will be using Clustering methods to put data into similar groups and help an analysis focus on the critical differences between data points rather than be overwhelmed by individual differences that have no patterns associated with them.</p><p>The file contains 2,150 houses for sale listed on Redfin.com in 2018. There are several variables included in the file. We want to create a cluster of houses within Baton Rouge, Louisiana, identify data issues, and take appropriate actions. Also, we need to identify variables that should be excluded from the analysis and provide a rationale for doing so.</p><p>Overall, the main goal is to prepare the data set for better analysis by using clustering methods.</p><h3>Data Understanding</h3><p>The “<strong>redfin_2018_Houses</strong> <strong>Raw Data</strong>” JMP file contains 2,150 records of houses across Louisiana. Before Clustering could be performed, a significant amount of data preprocessing was necessary, including variable reduction, treatment of missing values, screening for outliers, and recoding. Our variables will be price, beds, baths, square feet, and $/square feet. The amount of missing values is 0, 290, 288, 303, and 303, respectively.</p><p>Also, since “City” is nominal, we need to use Recode. Then, we excluded all properties outside Baton Rouge. We (Recode) or separated Baton Rouge and group other cities as “Others.” Baton Rouge 2,098 reports “Other” cities combined 52. Total as we mentioned, 2,150 reports within Louisiana. We created a new data table, excluded 354 reports, and are ready to perform Clustering analysis. 1,796 rows are left after removing outliers, missing values, and properties outside Baton Rouge.</p><h3>Analysis</h3><p>Two different clustering techniques were performed, K-means and Hierarchical. The K-Means clustering method provided a manageable number of clusters while also providing insight, described below.</p><p>The first analysis will be <strong>a K-means cluster</strong> with several k=5 clusters using the following variables: Price, Beds, Baths, SF, and $/SF. “Cluster Summary” shows that #1 and #3 are the closest with the number of reports 263 and 233 respectively, Appendix A. All variables are relatively close to each other with their “Cluster Means” or distance.</p><p>By looking at the means, we can see that five groups (clusters) are divided into two main subgroups, as they are close to each other with their means (distances).</p><p>1st group clusters 1,4,5 and 2nd group 2 and 3, would be the closest to each other, as we can also see the Biplot of clusters, Appendix B. Shorter distance between clusters means that they have similar observations of houses within Baton Rouge area and the space that incorporates all dimensions that we chose.</p><p>We are not required to perform validation, but if we would need to validate the K- means method, we would use two techniques: cross-validation, and the second method to validate would be to take the sum squared distances to cluster means for k=5.</p><p>The K-means offered insight into the problems, variables. To increase the granularity of the analysis, a Hierarchical clustering technique was employed, with the same number of clusters = 5.</p><p>Hierarchical Clustering is a step-wise procedure that attempts to identify relatively homogeneous groups of cases based on selected characteristics using an agglomerative or divisive algorithm.</p><p><strong>Hierarchical Clustering means</strong> Appendix C.</p><p>Word’s method used, the sum of squares within the cluster summed over all variables as the similarity measures between sets. It tends to join clusters with a small number of observations, and it is strongly biased on producing clusters with the same shape and numbers of observations.</p><p>If we look into “Clustering History,” Cluster 4 and 5 will be the closest means or distances. Also, #4 has the most groups of variables joined together by 184 and 29 Leaders.</p><p>The constellation plot reveals three major clusters with branches to sub-clusters and several outliers, Appendix D.</p><p><strong>We’ve also created a map of 5 geographic clusters using k-means Clustering, which</strong> included five variables: Price, Beds, Baths, SF, and $/SF. The map shows us that clusters 4 and 5 will be closest to each other with their properties that we chose to the “smooth” line on the latitude vs. longitude graph within the Baton Rouge area. Appendix E.</p><h3>Conclusion</h3><p>Clustering in JMP provides some insight into the understanding of massive data sets, with the ability to put data into similar groups and focus on the critical patterns with grouped data rather than be overwhelmed by individual differences.</p><p>Clusters 1,4, and 5 will be the closest in the K-means method and 3,4 and the Hierarchical approach.</p><p>With property type, we have a condo and single-family residential with the most significant percentage.</p><p><strong>The advantages of the K-means method</strong> that we found out are the ability to work with the large data set, as it clearly shows the pattern and numeric output as distances (means). Also, the results are less susceptible to outliers in the data.</p><p>Disadvantages for the K-means method is not efficient when a large number of potential clusters exist. Each cluster solution is a separate analysis. The data can be only numerical.</p><p><strong>Advantages for Hierarchical,</strong> the data can be either numerical or categorical. It tends to join small clusters with an extensive set of data clusters. Similar to a treelike structure.</p><p>Disadvantages for Hierarchical is not suitable for extensive data set — the impact of outliers. To reduce the effects of outliers, the analyst may wish to cluster the data several times.</p><p>Overall, I prefer K-Means and Map as they clearly show the results that we wanted.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/0*i0ZwA2v_IjgiIUji.png" /><figcaption><strong>Appendix A — Table 1. Cluster distribution for K Means</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/0*GcjZOwpUPE33warb.png" /><figcaption><strong>Appendix B — Biplot, K-means, k=5</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/0*ypKKOu3ASi4dcQb-.png" /><figcaption><strong>Appendix C — Hierarchical Clustering Means</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*ldTyW4wJCB-CtgkT.png" /><figcaption><strong>Appendix C — Hierarchical Clustering Means</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*ShndXpg1u8Yc6VEq.png" /><figcaption><strong>Appendix D — Constellation Plot</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/0*OsZfYBT1nKIVkLXQ.png" /><figcaption><strong>Appendix E — Map of 5 Geographic Clusters using K-means Clustering.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/0*CBEI_Kx9hLIO_a3O.png" /><figcaption><strong>Appendix F — Hierarchical Clustering with Location and Property Type and Constellation Plot.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*ZTxnbONd4AIs7sbl.png" /><figcaption><strong>Appendix F — Hierarchical Clustering with Location and Property Type and Constellation Plot.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*UuFKUZuPJDWvIr50.png" /><figcaption><strong>Appendix G — Crosstab of Cluster vs. Property Type</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d541ad195c7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[(ARIMA models) Which model best fits in the “Air Travel.”]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/arima-models-which-model-best-fits-in-the-air-travel-36e73e835de6?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/36e73e835de6</guid>
            <category><![CDATA[arima-model]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[air-travel]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:27:01 GMT</pubDate>
            <atom:updated>2021-10-20T13:27:01.708Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction and Business Understanding</h3><p>We start by using the file “Sept11Travel”, which contains the data of three different monthly time series <strong>between January 1990 and May 2004</strong>. This file contains 172 rows and 32 rows which are excluded from the data table. In the research, we should focus only on “Air Travel.” Our goal is to select the model that fits best, develop regression and ARIMA models.</p><p><strong>Data Understanding</strong></p><p>Using Graph Builder, we can tell how the times series looks (AIR RPM vs. Month) and Smoother them in and out, making a graph linear or heavy volatility.</p><p>In our first model, we want to look at the Regression model. Before doing it, we must add two columns to the data table to account for the model for seasonality. The first column is just a Counting table. To count the rows called — “Month_NUM,” we will use it as one variable in the regression model. The second column would be season — 12 months with a nominal, character data type. It creates 12 dummy variables for us. Finally, we are ready to make models and perform analyses of them.</p><h3>Analysis</h3><p>Firstly, we have to make a Regression Forecast model by using two variables that we have created. Appendix A. Season [12] is a baseline value, measures all the differences among the 12. That is the exact reason we don’t see it in the Parameter Estimates. Most seasonal coefficients are statistically significant at the 0.05 level, except for months [5] and [10]. R square = 0.97, R square adj. = 0.96, we will be using this as well in the future to figure out which model performs better. After creating a prediction formula from the Regression Forecast model, we plot this data to an original data plot. Appendix B. If we decide to unexcluded the 141 rows, we will get the actual forecast. The blue line is the smooth representation of the actual values, and the red line is a forecast from the regression model. Appendix C. We can see that after September 11, the regression model is quite off because it is assumed that the trend is continuous and because of the variable we have created “Month_Num and it is a straight regression line, so it cannot know that after September 11 is a decline. Know we want to compare this model to the ARIMA model. We are excluding back these 141 Rows so that the ARIMA model won’t use this data.</p><p>We select Air RPM as a Y, Time series, and Month as an X, Time in time series. Next, we see the data plot in Appendix D., which contains Autocorrelation functions on the left side and Partial-Autocorrelation tasks on the right side. Bars outside the blue confidence limits for different lags indicate the values p and q.</p><p>Partial-Autocorrelation function: tells us AR(p) process, while Autocorrelation function gives us an indication of the average moving technique, MA(q).</p><p>Partial-Autocorrelation function, AR(p) process of 1, since the partial autocorrelation function is that close to 1, it is closed to be stationary and in that case, we getting a better estimate if we take the first differences and just model the residuals. When Autocorrelation is very close to 1, we take the first differences to make this data stationary. The same with the seasons, season [12], Autocorrelation function rather large partial-autocorrelation, and significant Partial Autocorrelation for the same month. Therefore, we use the standard method of season differences, and then we try to model the remaining Autocorrelation. When Autocorrelation is close to 1, the estimates become very unstable. That is a reason we are taking differences.</p><p>We selected a Seasonal ARIMA process because the Partial-Autocorrelation function at lag one is relatively large. We take the first difference and also because the Partial-Autocorrelation part is around the season is large at Lag [12]. ARIMA of 1 between successful months like July and August. Residuals of ARIMA process Appendix E.</p><p>The residuals of the season difference and the first auto differences much teamer know can fit a model. There has to be a season component in the ARIMA process with MA [12], also AR [1], or AM [1]. We want to try these models first and simplify the understanding of them. We make a Seasonal ARIMA with values, 1,1,1 and 1,1,1 and compare it to the original 0,1,0 and 0,1,0. R adj. square increased, which means we have very useful valuables in the set. Appendix F. AR1,1 is statistically significant, so as MA1,1, but AR2,12 and MA2,12 are not. Since the AR2,12 is very close to being statistically significant, we exclude it and look closely at MA2,12. The next step is to try models 1,1,1, and 0,1,1 by excluding the Autoregressive model. As we can see, we are getting a better result in Appendix G. Our residuals are looking very good within limits, and all estimates are statistically significant, and R adj. square also improved to 0.979. Right now, we want to try to use a different model and see if we can improve our forecast results.</p><p>We have tried model (1,1,1) and (0,0,0) not good results, so our next try is model (1,1,1) and (1,1,0) Appendix H.;</p><p>by excluding Moving Average Order, we have a greatly improved result. R adj. square has increased by little, so as Residuals got closer to our limitations at Lag [24]. By plotting these models into JMP by using our prediction formula, we can see that up to a month of September 11, both forecast for regression and ARIMA forecast quite similar and comparable, but know when we include back these 141 rows, the red curve which is a regression line is continuous up. However, the green and purple lines quickly follow the changing pattern close to the blue line. The deviation was a lot less than the deviation of the regression model, less Autocorrelation. Appendix I.</p><h3>Conclusion</h3><p>2 ARIMA models and Regression Forecast models were used in this assignment to select the model that fits the best to “Air Travel.” ARIMA deviation is a lot less than the deviation of the regression line. R adj. square performed the best in the ARIMA model (1,1,1) and (1,1,0) but pretty close to (1,1,1) and (0,1,1).</p><p>2 ARIMA MODELS, which statistically significant parameters a high adj. R square and residuals that had low Autocorrelation.</p><p>Overall, up to September 11, both forecast for regression and ARIMA forecast are quite similar and comparable. ARIMA produced more accurate results than ordinary regression estimation because of the account for dependencies between observations. The model replicated well.</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/0*F-0vwmSHg-x586xs.png" /><figcaption><strong>Appendix A. Parameter Estimates</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/0*ortlIoj_bNBRv1qo.png" /><figcaption><strong>Appendix B. Data from Reg. Model combined Original Data Plot.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/0*ReTu2lo8JB-37LGB.png" /><figcaption><strong>Appendix C. Forecast Unexcluded 141 rows</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*z_DKaOnOzSyMIPfc.png" /><figcaption><strong>Appendix D. Time Series of Air RPM, Basic Diagnostic.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*XZEKTQ7ZCvnoc1vf.png" /><figcaption><strong>Appendix E. Residuals of ARIMA process</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*5EJmupKqFVwypNki.png" /><figcaption><strong>Appendix F. Seasonal Component, Parameter Estimates for 1,1,1 and 1,1,1</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/0*3Mk11qi1yfrrWKj3.png" /><figcaption><strong>Appendix G. Model (1,1,1 and 0,1,1)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*1SgnvlMgWjwo2xGB.png" /><figcaption><strong>Appendix H. (1,1,1) and (1,1,0)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*bquB6St8iswqvjq8.png" /><figcaption><strong>Appendix I. 2 ARIMA models and Regression Model</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36e73e835de6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The company optimizes the click rate]]></title>
            <link>https://medium.com/@ANDREY_FATEEV/the-company-optimizes-the-click-rate-2bff8d65619?source=rss-ba326de0c606------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bff8d65619</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[click-rate]]></category>
            <dc:creator><![CDATA[ANDREY FATEEV]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 13:23:10 GMT</pubDate>
            <atom:updated>2021-10-20T13:23:10.913Z</atom:updated>
            <content:encoded><![CDATA[<p>The ABC Company optimizes the click rate.</p><h3>Introduction and Business Understanding</h3><p>The ABC company wants to optimize the response (click rate) to their online ads. 13 factors were identified as potentially affecting the clock rate. We start by using the file &quot;DOE_Click,&quot; which contains the 13 factors and two levels we should consider.</p><p>The goal is to determine which response factors will increase the click rates on ABC Company&#39;sCompany&#39;s online ads.</p><h3>Data Understanding</h3><p>ABC company identified 13 factors that will be included in a 20 run Blackett-Burman design. Each element is on two levels, and the response is of continuous modeling type to count the number of clicks. Appendix A.</p><h3>Analysis</h3><p>Several DOE designs were considered using all 13 factors, with the most common the Screening Designs platform under the Classical Design option to create a design matrix. We should select a design with the number of runs. A complete factorial design is not realistic as it will contain 213 runs, too many, better not to run a lot, as it costs much money to complete that many runs. JMP provides multiple options to reduce the design to a fractional factorial, including minimum runs, default runs, and user selection. The minimum requirement is simply several factors plus the mean. The minimum represented the number of factors plus one, which is 14 experiments. The default design in JMP provided us with 16 runs fractional factorial plan, which meets the minimum run requirement, which is more than 14 experiments and is of resolutions &quot;&quot;3 main effects&quot;&quot;, which provide the estimation of all 13 main effects, and the overall mean free of confounding with each other.</p><p>Both 14 and 16 run options were evaluated in the distributions platform, and It was determined that 20 runs would be a less costly, balanced design. Two &#39;factors&#39; interactions are confounded with the main effects.</p><h3>Identifying Statistically Significant Factors</h3><p>After selecting a design, the data file was used to determine the statistically significant factors at a significance level of p=0.05. We performed two-level screening under the DOE platform to analyze the design. Four elements (Discount, Advertising Chanel, Color Schema, and Message Type) were statistically significant below 0.05, Appendix B. Only Discount and Ads. Channel has a Simultaneous value of less than 0.05. The remaining 16 factors had a p-value greater than 0.14, which makes them all not statistically significant. The model was run with the four selected factors to evaluate their relationship to the response of several clicks. Appendix C. All four main effects were statistically significant at p less than 0.05.</p><h3>Confounding</h3><p>Four top factors are unconfounding with each other, but Discount*Ads. Chanel is confounding with some of the main effects, but we can&#39;t see which one, so we have to look at the design, which action it is confounding with, and the confounding structure. We want to determine if their 2-factor interaction of the two most significant factors is confounded with the other weaker main effects. The Alias Matrix Appendix D. shows that the 2-factor exchange of Discount and Advertising Chanel are strongly cofounded with the Color Scheme indicated by the value of 0.6, and the second one is Message type with 0.2 indications.</p><p>We look at the significant values in the positive or negative directions. The sigh is not essential, but the absolute value is, and if it is a relatively large number, if it&#39;s zero, there is no confounding between the 2-factors, which are the 2 top rows, and the main effects of the first column. There are other large numbers in Discount with Color Scheme being strongly confounded with Advertising Chanel and 2-factor interaction between Ad. Chanel with Color Scheme that confounded with Discount, but they are likely to be small interactions.</p><h3>Reduced Design</h3><p>Based on the findings from the Alias Matrix, a new model was created, including the 2-factor interaction Discount with Ads... Chanel with four main effects, which were determined in the first run. Appendix E. Color Scheme and Discount Ads. Chanel both have p larger than 0.05 making them not statistically significant. Appendix F. Discount*Ads. Is the factor that is responsible for the Color Scheme that contributes to the effect.</p><p>We take out Color Scheme as it has the lowest effect. Appendix G. It is out the final result as all main products are statistically significant at p smaller than 0.05. Dominant factors are Discount and Ads. Chanel, they in action with Discount Ads. Chanel. In Appendix H. it shows that the effect of when we have Ads Chanel A, and we added Discount to add A so that it will increase in the click rate, but if we choose Ad B with Discount, it will increase even more compared to Ad A, it is interaction, the effect of one variable is different depending on what the level of the other variable is on the graph.</p><h3>Conclusion</h3><p>Screening designs were used to determine which response factors that will increase the click rates.</p><p>A fractional factorial design was used to limit the number of runs which would lower cost while still providing enough runs to estimate.</p><p>The first 20-run design identified the four most statistically significant factors, and an inspection of the alias matrix indicated that one of the main effects was confounded with the 2-factor interaction, the two most considerable main effects. Interaction plot shows us that switching Ad. Chanel from A to B has a significant impact when there is a discount – YES. Furthermore, the experiment shows that the most click responses occurred when a deal was offered and Ad Canel B (No).</p><p><strong>Research Provided by Andrey Fateev</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*bcebXKCP238C8YV2.png" /><figcaption><strong>Appendix A. Factors and Levels</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*y0yZLy3-AuFbjX8R.png" /><figcaption><strong>Appendix B. Analysis Result</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*SBxIUr5-Atp3a6oM.png" /><figcaption><strong>Appendix C.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*i24cTYwOgrWwT7Pg.png" /><figcaption><strong>Appendix D. Alias Matrix.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*q-YJIKRAKVSKNJUt.png" /><figcaption><strong>Appendix E.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*YX2B6St_LbGxZMRc.png" /><figcaption><strong>Appendix F. Results of the 2nd run.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*MN-x-rybSzAQJolX.png" /><figcaption><strong>Appendix G. Final Model without Color Scheme.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*gE6bFH03O2d5uG55.png" /><figcaption><strong>Appendix H. Interaction Plot for the Final Model.</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bff8d65619" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>