<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wesley Willis on Medium]]></title>
        <description><![CDATA[Stories by Wesley Willis on Medium]]></description>
        <link>https://medium.com/@wwillis125?source=rss-c9c85b209952------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Kb9HEiV9oZ5lwVCj.jpg</url>
            <title>Stories by Wesley Willis on Medium</title>
            <link>https://medium.com/@wwillis125?source=rss-c9c85b209952------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:51:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wwillis125/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Brief Introduction to Tableau]]></title>
            <link>https://medium.com/@wwillis125/a-brief-introduction-to-tableau-14edc356f321?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/14edc356f321</guid>
            <category><![CDATA[tableau]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Mon, 18 May 2020 16:32:00 GMT</pubDate>
            <atom:updated>2020-05-18T16:32:00.958Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Tableau is a fairly popular Business Intelligence (BI) tool that helps to automate various forms of data in order to make quick and concise visualizations and insights. While Tableau offers a free interface, you may be required to purchase a more advanced version in order to take advantage of its more professionally convenient features. While the free version does not limit any of the data visualizations, it does not allow you to connect your servers to the program, which inhibits the type of data you can upload for privacy purposes. However, as this blog post will cover the basic functions of Tableau, the free version will be sufficient.</p><h3>Getting Started</h3><p>In order to download Tableau to your computer, you want to make sure that your system is up to date with the requirements of the latest Tableau software. These requirements can be found at <a href="https://www.tableau.com/products/trial">https://www.tableau.com/products/trial</a> where you can also download a free 14 day trial of Tableau Desktop. If your system is not compatible, I suggest looking into older versions of Tableau in order to find the version of tableau that will integrate with your system. In order to download Tableau Public, the free version, you can go to <a href="https://public.tableau.com/en-us/s/download">https://public.tableau.com/en-us/s/download</a>. Both these downloads come with sample datasets to help the user get a greater understanding of the features and possibilities of this platform.</p><h3>Feature Selection</h3><p>After downloading Tableau to your computer, you will have a few options regarding what data you can play with. Tableau will initially separate your data into two categories, dimensions, which help to separate by a distinct category and measures, which shows the degree or extent of an attribute. For example, using complimentary data from Tableau, for superstores in the US, dimensions would be the region of the store, the items being sold, etc, while measures would be the sales figures, profit, or quantity. By dragging these dimensions to the rows or columns section, and choosing which measure you wish to view, Tableau works similar to Groupby’s in SQL, allowing you to gain a quick and comprehensive understanding of the data. Once you have chosen which dimensions you wish to highlight, you can drag your measures into the row or column you wish to compare it across. This allows you to make quick and comprehensive visualizations across various dimensions for a specific measure. These measures can also be tweaked using the drop down arrow within the measure feature. This allows you to compute across various different statistical measurements, such as maximum, minimum, average, count, or sum. Some advanced options even include standard deviation, variation, and percentile. These features are extremely valuable during data exploration, as it can help uncover patterns and themes across the entire dataset, that might take multiple individual SQL commands in order to duplicate.</p><h4>Marks</h4><p>In order to make the data a bit more comprehensible, you can add measurements and dimensions within the “Marks Card,” traditionally to the left of your data sheet. These marks help to distinguish your data to a higher degree in your visualization. The automatic feature, allows Tableau to determine what is the best way to visualize your features, but you can manually determine what type of graph style you would prefer, between a scatter plot, line or bar graph, and even on a map if given geographical coordinates. These work in conjunction with the “Show Me” tab, which is really just a shortcut version to these same data visualization techniques. The marks tab also allows you to determine color, shape, and size of your data points as well, by dragging the measure or dimension over the box you wish to distinguish by.</p><h3>Dashboards</h3><p>After completing your data exploration, the next step is to bring everything you’ve found into a comprehensive visualization. Tableau helps to facilitate this process through the use of dashboards. Dashboards allow you to bring all the sheets that you’ve compiled regarding a specific dataset and combine the visualizations under one space. These dashboards can be extremely helpful through the use of the legends. Depending on the various dimensions that you used in your data exploration, the dashboards combine the marks into one legend, allowing you to pinpoint specific data across various sheets or graphs to highlight any discoveries you may have made. For instance, if you made multiple sheets that split the data by region, a legend will allow you to prioritize which region, whether south, east, etc, you wish to highlight. These will highlight across the sheets, regardless of the other dimensions you may have differentiated by. This comes in handy when attempting to present your data to a group, or even when exploring the findings of a colleague.</p><h3>Conclusion</h3><p>Tableau is a fairly easy and intuitive BI tool for data exploration and visualization. I highly recommend it to anyone who is starting out in the data field or companies that need to overhaul their data program, as even those without a strong history can understand and draw connections through this tool. The major drawback that I’ve found for Tableau is for professional use, as the public use does not allow you to upload private servers, making the data uploaded to the program public. Also because you cannot connect servers to the program, all the data must be able to fit on the server that Tableau is running on, making it difficult if you have large amounts of data. These are very important considerations to make when determining if you want your companies data on this program, which can come with a heavy price tag for corporate use. However, for aspiring analysts and engineers, I definitely recommend getting a comprehensive understanding of this program and I hope this blog helped to shed some light on the features available to you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14edc356f321" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Visualization in CartoFrames]]></title>
            <link>https://medium.com/@wwillis125/data-visualization-in-cartoframes-5a5d42d00758?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a5d42d00758</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[carto]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Mon, 27 Apr 2020 18:18:33 GMT</pubDate>
            <atom:updated>2020-04-27T18:18:33.068Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>During the Open Data Science Conference (ODSC), I had the opportunity to attend various discussions and demos centered around new and innovative methods to support data modeling, engineering, and visualization. One of these talks was centered around Geospatial Analysis, given by Alvaro Arredonde, a data scientist with Carto, a company dedicated to providing better tools and techniques for data scientists. This blog is dedicated to furthering my understanding of this package and its potential.</p><h3>Installation</h3><p>To install cartoframes on your computer you can use git or jupyter notebooks and run this code:</p><blockquote>$ pip install cartoframes</blockquote><p>If you want to ensure that you’re running the latest version of carto you can install using this code:</p><blockquote>$ pip install — upgrade cartoframes</blockquote><h3>Geocoding</h3><p>Cartoframes works in collaboration with pandas and numpy, allowing it to be highly adaptable to most machine learning workflows. To begin using cartoframes for geospatial data, it is important to convert location data into geometry that can be read on a map, through a process called geocoding. This can be done a few different ways, dependent on the type of data available. The simplest way I’ve found is with latitude and longitudinal data. Given these data points, it is possible to run the data through geopandas, a subsidiary of pandas used for spatial datam converting the data into a geodataframe, and passing the argument geometry with the method points_from_xy:</p><blockquote><em>cases = gpd.GeoDataFrame( </em>source<em>,geometry</em>=gpd.points_from_xy(source<em>.latitude, </em>source<em>.longitude</em>), <em>crs</em>=4326)</blockquote><p>where source is the dataframe, geometry indicates the location of the data and crs is the coordinate system being referred to (in this case longitudinal data)</p><p>Another way to apply geospatial data to cartoframes is by using the cartoframes API for more complex data points. This can be used in conjunction with specific data, such as addresses or zip codes, or broader location points, like city or country names. With countries or regions, the API allows the geometry to be converted into polygons representing the borders of the country. In order to use the Data Services API through Carto, you must apply for a key and set your credentials. You can learn more about the API here:</p><p><a href="https://carto.com/developers/data-services-api/">Data Services API - Home - Documentation - CARTO</a></p><h3>Visualization</h3><p>After geocoding your location data, it is time to visualize the data using Map and Layer classes. The Map class is the foundation for your data visualization, establishing a map through which your data can be viewed. The Layer class stacks your geocoded data on top of the Map, allowing for various functionalities to improve and support your workflow. Multiple layers can be applied to a single map, in order to visualize various aspects of the data, and allow different arguments to pass in order to better assist the visualization. The style arguments can be applied to both categorical or continuous data, and can manipulate the data visualization by either size, color or other, and can even provide a coinciding legend. The Layer class can also take a widget or pop-up argument. This provides a secondary structure next to the map that gives the viewer more information and a greater understanding of the trends and qualities of the data. Multiple widgets can be applied to a layer, which will coincide with a specific attribute of the data. In order to expand coverage of the attributes, another Layer will have to be applied to the Map pertaining to that specific feature.</p><h3>Going Further</h3><p>While these basic functions of CartoFrames provide an interesting and collaborative framework to assist your data exploration, Carto goes a step further to provide projects with vital tools to improve and expand their research. A particularly interesting feature is the ability to create isochrones. Isochrones are areas represented by polygonal geometry that are calculated by a given metric. For instance, if you wanted to calculate the area from a specific location that could be covered within a 15 minute walk, you would use the Isoline class in the data services package. This package can help identify these areas through the Carto API and can even be further improved by applying the Catalog class, which filters by country and category, allowing information like demographic data to be applied to the visualization.</p><h3>Conclusion</h3><p>CartoFrames data visualization package has the potential to truly improve how spatial data is perceived in the future. It’s wide range of application teamed with its simple and intuitive code makes the learning curve worth it to achieve these high quality visualizations. To understand more regarding its application, I encourage you to visit their website for a more in depth and thorough information and thank you for reading</p><p><a href="https://carto.com/developers/cartoframes/">https://carto.com/developers/cartoframes/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a5d42d00758" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Important Considerations with Linear Regression]]></title>
            <link>https://medium.com/@wwillis125/important-considerations-with-linear-regression-cb1793a2c59f?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb1793a2c59f</guid>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 05:51:56 GMT</pubDate>
            <atom:updated>2020-04-17T05:51:56.064Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Overview</strong></h3><p>Linear Regression is one of the most common machine learning techniques within the data science arsenal. Used with numeric data, this model helps to determine the relationship between an independent variable (or variables in the case of multiple linear regression) and a target variable. The idea being that given a specific value for the independent variable, one could produce a ballpark estimate for the target. This technique is commonly associated with the line of best fit and is determined by minimizing the error between the predicted and actual value. This blog will focus on considerations when trying to optimize your linear model, so that you can improve your results.</p><h3><strong>Assumptions</strong></h3><p>To begin, it is important to address the assumptions when dealing with Linear Regression. While one of the more basic models, a linear regression model will not work with just any numerical dataset. These assumptions are linearity, normality, and homoscedasticity.</p><h4>Linearity</h4><p>The trend of the data is a very important aspect to understand. You can find that the line of best fit may work within a suitable error frame, but if the trend is not necessarily linear, when applying this model to new data points, the predictions could be more harmful than helpful. If you find that your model is not giving you the predictions that you need, it can be helpful to try transforming the target variables to find a more suitable trend. By applying an exponential or logarithmic transformation, the model may have an increased predictive power and work better outside the training data.</p><p><strong>Normality</strong></p><p>Another important trait to look into is the distribution of the residuals. The residuals refer to the difference between the predicted values and the actual data point for a given variable. It is important for these residuals to have a normal distribution so that you are able to trust in the predictive power of the model. If the distribution is non-normal, than the confidence in the predictions will vary according to this distribution. In other words, the first 30% of the model may be more reliable than the last 30%. With a normal distribution, you are ensure that the reliability of the model remains consistent and even calculable. While a histogram will show the distribution, a common metric used is the Q-Q plot. Ideally, this graph will show a straight line, but will vary if the distribution is less than ideal. A strong understanding of these graphs can help you identify the issue in your distribution, show where there is a greater weakness in your model, and what steps to take in order to address this weakness.</p><p><strong>Homoscedasticity</strong></p><p>The final assumption with a linear regression model is homoscedasticity, which refers to the spread of the residuals. With a homoscedastic model, the data points would vary evenly throughout, while a heteroscedastic model would see greater areas of variability than others. This is similar to the issue of normality, as it is important for the spread to be even to ensure a consistent level of predictability throughout the model. There are multiple metrics that can be implemented in order to assess a models homoscedasticity (a scatter plot being the simplest), including various significant tests, such as a White general test, or a Cook-Weisberg test.</p><h3>Conclusion</h3><p>These three assumptions are important when determining the validity of your regression model and its predictive power, but are just a few examples of how you can improve and optimize your model. By keeping these types of metrics in mind, you can improve the predictive power and have a better understanding of your data and its output. I hope this blog helps and I encourage everyone to continue to explore the nuances of their statistical models in order to understand and perfect their analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb1793a2c59f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Important Considerations When Working With Time Series]]></title>
            <link>https://medium.com/@wwillis125/important-considerations-when-working-with-time-series-ccc7a3a91f3b?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccc7a3a91f3b</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Fri, 20 Mar 2020 15:20:22 GMT</pubDate>
            <atom:updated>2020-03-20T15:20:22.936Z</atom:updated>
            <content:encoded><![CDATA[<p>In an effort to improve my understanding of Time Series modeling and its potential, I have decided to take a deeper look into some of the concepts that presented a challenge for me. I offer this blog for anyone suffering through a similar experience and want to gain a firmer grasp.</p><p><strong>Trend and Seasonality</strong></p><p>Trend and seasonality are extremely important concepts when working with Time-Series data. Trend refers to the overall direction of the data, whether it is increasing, decreasing, or staying stagnant, over time. Seasonality refers to the patterns the data might exhibit based on the time. Whether it be monthly, daily, or hourly, it is important to distinguish exactly when the pattern begins in order to increase the validity of your predictions.</p><p><strong>White Noise vs Random Walk</strong></p><p>Two important time-series models include White Noise and Random Walk models. A White Noise model has independent but evenly distributed variables with a mean of zero. White Noise is important when discussing predictability of your model, as well as when diagnosing the validity of your model. If your original time series data is a white noise model, than by definition it is random and cannot produce accurate predictions. However, it can also be an indication of the strength of your model when looking at the residuals of your model. If the residuals are not white noise, there may be room for improvement in your model.</p><p>A Random Walk model is a series of data where the next point is a determinant of the previous value. A random walk is difficult to predict, as despite its relationship with the previous value, the next value is still randomly determined from there.</p><p><strong>Autocorrelation vs Partial Autocorrelation</strong></p><p>Correlation is an imperative consideration when working with Time-Series, referring to the amount of influence the previous data point has to the next. Because the correlation is calculated through the same series of observations, just at different times, it is considered an autocorrelation or serial correlation. Partial Autocorrelation attempts to remove the bias in autocorrelation, only looking at the correlation between two observations and diminishing the residual or indirect correlation inherent in the model. This helps to determine a bit more precisely, the effect of each lag or point in the time series independently with one another.</p><p><strong>Autoregression and Moving Averages</strong></p><p>After getting to understand your time series data a little better, the next step is model predictions and forecasting. An Autoregression (AR) model works to capture the dependent relationship between an observation and its seasonality. By comparing an observation with a specific lag, we are able to forecast its potential movement going forward. A Moving Averages (MA) model focuses more on the dependency of residual error when applied to an observation, dealing more with the noise inherent in time series data. This model helps to diminish the residual error in the future, by applying the error from past observations.</p><p>These models help to illuminate certain patterns within the Correlation functions, making it easier to determine what type of data you are dealing with. With an AR model, we would expect the autocorrelation up to its determined lag to be high, and gradually weaken as the lags continue, while partial correlation should not exist past the determined lag, as it only accounts for the direct relationship and not the residual effect. For a MA model the effect is flipped. The Autocorrelation would be strong up to the determined lag, with a sharp decline afterwards, while the Partial Autocorrelation will have a more gradual decline after the determined lag.</p><p><strong>ARIMA Models</strong></p><p>When put together these functions combine to make up an ARIMA model, which allows for predictions and forecasting on our dataset. AR stands for Autoregression, I stands for integrated, referring to the process to make the time series stationary, and MA stands for Moving Averages. When dealing in Python it is important to pass parameters (p, d, q)to determine the scale each of these models are used on the data. By passing 0, you also allow the model to bypass a certain aspect of the model, allowing for individual AR or MA models, or ARMA models as well. In order to find the best values of the parameters for predictions, commonly the Box-Jenkins methodology is used. This methodology incorporates identification, in order to determine the magnitude of the lag by uncovering the trend and seasonality in the data, estimation of the parameters through a fitting procedure on the regression model, and checks the residual errors to find any hidden or latent structure not captured by the model. Furthermore, these models can be further improved by adjusting for seasonality or exogenous variables by using a SARIMA or SARIMAX model.</p><p><strong>Vector Based Models</strong></p><p>While the previous models are designed to work with univariate features, vector based models help to apply these concepts to multivariate time series data. Multivariate data commonly refers to data that has more than one independent variable. The notation for this type of model is similar, as the model mostly involves generalizing the standard model to allow for application across the multiple time series.</p><p><strong>Exponential Smoothing</strong></p><p>Some other univariate time series models involves exponential smoothing, which models the next time step as an exponentially weighted linear function of observations at previous time steps. These can either account for seasonality in trend, such as in the Holt-Winter’s method, or not, with the simple method</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccc7a3a91f3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Brief Introduction to PyTorch]]></title>
            <link>https://medium.com/@wwillis125/a-brief-introduction-to-pytorch-4e53012e58f4?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e53012e58f4</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Thu, 05 Dec 2019 18:23:45 GMT</pubDate>
            <atom:updated>2019-12-05T18:23:45.382Z</atom:updated>
            <content:encoded><![CDATA[<p>PyTorch is a library for Python programs geared to making building deep learning software more efficient and versatile, and is currently one of the most well adapted tools in the deep learning community. With application across a number of professions, such as data science, artificial intelligence, and software engineering, PyTorch aims to address the difficult transition from computer programming to deep learning.</p><p>The idea behind deep learning is simple: delegating more arduous or annoying tasks to computers in hopes of maximizing efficiency and mental stability. Using extensive datasets, deep learning allows for computers to create complex algorithms in order to train models geared towards approximation, predictability, and various other functional purposes that social media has since made humans much too busy for. The true benefit of deep learning is that the machine analyzes the data itself, finding benchmarks and variables to work with and manipulate in order to maximize the efficiency of the model. By allowing the computer to solve these problems in its own, we are able to minimize the nature of human bias and instead strip the datasets to its core characteristics. Characteristics that we assume are intuitive but instead have been engineered and wired in our behavior and deduction for generations no longer influence and distort our findings, but instead we are able to discover certain features that play a much greater role than we may have previously believed. A popular quote by Edgar Dijkstra summarizes the idea quite well.</p><blockquote>“Alan M. Turing thought about the question of whether Machines can think, a question as relevant as whether submarines can swim.”</blockquote><p>PyTorch is a great beginner deep learning software for users who are familiar with Python, adapting many of the more familiar features and packages for use in various machine modeling situations, while addressing and minimizing the limitations prevalent in Python for more in depth applications. For example, PyTorch uses a Tensor, a multidimensional array, similar to Numpy arrays but geared to maximizing computing efficiency in various mathematical sequences as pertaining to deep learning. It also has adapted a high performing C++ runtime in order to address the transition from research to production, and spare the users from being inhibited by Python’s runtime. Furthermore, by utilizing immediate execution, as opposed to deferred execution, PyTorch circumvents potential issues and bugs that might inhibit a model from executing. When the need for deferred execution arrises, PyTorch switches to C++ execution as opposed to Python in order to maximize storage capabilities and create more efficient run times. On the other hand, the tensors so prevalent in PyTorch have cross functionality with many Python packages such as Numpy, Sci-Kit Learn and Scipy. This allows for a more fluid transition in many applications for creating proper models. The structure of a PyTorch project is illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*mfrN1C4HxOhaNsUc1AgY_A.png" /></figure><h3>The Tensor</h3><p>In simple terms, the tensor is a multidimensional array, representing vectors and matrixes. Through the tensor, PyTorch is able to maximize its functionality for manipulation across a wide spectrum of real world data. The tensor operates much like the Numpy array, in the way that they store the information, as well as mathematical manipulation. As Numpy remains the most popular array throughout computer programming, but especially with Python, the tensor in PyTorch not only adapts many of the same commands and functionality, but also allows for the flexibility and incorporation of Numpy in its own programming. However, PyTorch tensors contain within themselves their own impressive operability that helps to make the deep learning and neural network interface more efficient. These include but are not limited to: distributing operations over multiple devices/machines, keeping track of the source graph, and using graphical processing units (GPU) for faster operations. This efficiency is displayed best when considering the different types of data used in PyTorch. When addressing the less conventional forms of data, the Tensor is quite apt at giving them numerical qualities in order to support manipulation through the array. This applies to image and volumetric data, which can be applied within medical devices such as a CT scan. The most common way to encode colors into numbers is through an RGB scale. This scale applies a degree to Red, Green, Blue, in order to produce the desired output across a grid in the form of scalars, ranging from 8–16 bit integers. In order to pivot from a 2 dimensional image to volumetric data, imagine stacking these 2 dimensional images across another scalar, in order to represent density and depth. The ease in which image data is applied in PyTorch has allowed for a wide range of uses in medical, industrial, and scientific applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e53012e58f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Nuances of Statistical Analysis]]></title>
            <link>https://medium.com/@wwillis125/navigating-the-nuances-of-statistical-analysis-ad10bb533cc7?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad10bb533cc7</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Tue, 19 Nov 2019 17:36:46 GMT</pubDate>
            <atom:updated>2019-11-19T18:03:53.817Z</atom:updated>
            <content:encoded><![CDATA[<p>With the ongoing growth in the data science community, there is a higher demand than ever for the ability to understand, manipulate, and interpret data. Every industry is in need of the foresight that data science can provide in almost every aspect of their business, whether analyzing sales, distribution, or production. With the proper application, these services can be invaluable to the improvement of various business practices, increasing efficiency and decreasing risk. However, when poorly applied, scientists run the risk of missing out on key variables and influences that may skew their analysis. This blog is designed to give an introduction into the different forms of statistical analysis and when to apply them, in order to diminish this risk.</p><p>In order to begin, it is important to establish an understanding of the more basic parameters and fundamental concepts of statistical analysis. This blog will assume that the reader has a basic understanding of variance and standard deviation, normal distribution, and standard error.</p><p><strong>Z - Test:</strong></p><p>The z-test are designed to show the distance of an observed point (or z-statistic) is from the expected value, within a normal distribution. The metric they use to communicate the distance is standard error, and the expected value is derived from the null hypothesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*8aDgJQfLKD032c_R.jpg" /></figure><p>When using a z-test, a value greater than 2 shows that the observed value falls outside of the 95% of the data in a normal distribution. The area away from the median underneath the graph is known as a P-value, which stands for the observed significance level and basically represents the probability of getting a specific t-statistic. It is important that the z-test be used with reasonably large, independent samples, that follow a normal distribution.</p><p><strong>T - Test:</strong></p><p>The t-test was developed by W. S. Gossett, a statistician of the early 20th century, as a modification of the z-test, to be used with data with a smaller sample size.</p><p><strong>Wilcoxon-Mann-Whitney Test:</strong></p><p>Similar to the t-test, this test works for data that is not normally distributed, or is non-parametric. However, it is important that the dependent variables is at least ordinal. This test is conducted in a very similar way to the t-test and shows the statistical significance between two means.</p><p><strong>Chi-Squared Test:</strong></p><p>The chi-squared test is used in lieu of the z-test for categorical data when there are more than two possible results for the data. This test is designed to show whether each result is equally possible. The formula for the chi-squared statistic is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*prn2qtKQNAUMCNdw.jpg" /></figure><p>A large statistic indicates that the observed frequencies are far from the expected, while a small statistic suggests that they are fairly close, allowing statisticians an insight into the validity of their expected data. However, it is still necessary to compute the probability of finding a test-statistic of this magnitude, or in other words, understanding the relative distance of the statistic within the model, by finding the corresponding p-value. With a chi-squared test, it is necessary to construct a chi-squared curve, which is determined by the degrees of freedom of the model. These curves tend to have a longer tail on one side of the model, which flattens and expands as the degrees of freedom increases.</p><p><strong>Anova:</strong></p><p>Anova stands for Analysis of Variance, and is used to determine the influence independent categorical data on a dependent variable. For this test, it is important to take a look at the variance within variables, as an Anova test assumed that there is even variance within each independent variable.</p><p>Some variations of the Anova test are Factorial Anova, which takes multiple categorical independent variables, as opposed to one, Multivariate Anova or Manova, which measures two or more dependent variables against a single independent variable, Analysis of Covariance, which applies continuous predictors on top of the categorical ones.</p><p><strong>Kruskal Wallis Test:</strong></p><p>Similar to the Anova test, this metric also tests the independence of the dependent observations from each independent variable. The difference lies in that this test is non-parametric, meaning that the data is not normally distributed.</p><p><strong>Testing for Normal Distribution:</strong></p><p>While most of these tests help to conclude a hypothesis, it is also important to have a foundation for statistical analysis that helps to portray information about your data. The Shapiro-Wilk test, and the K-squared test both help to determine if your data follows a normal distribution, so that further analysis can be concluded. These tests are pertinent to helping scientist understand what analysis test to use, as many of the practices between parametric and non-parametric tests will produce a different result.</p><p><strong>Conclusion:</strong></p><p>While this is just a brief overview of the various potential statistical tests that can be used to analyze our data, it is important to establish a foundation of understanding with the different statistical tests and what they information they can illuminate. Attached is a link that helps to signify when the best time to use a certain statistical test is:</p><p><a href="https://stats.idre.ucla.edu/other/mult-pkg/whatstat/">Choosing the Correct Statistical Test in SAS, Stata, SPSS and R</a></p><p>By using these guides along with your own discernment and objectives, you are closer to wielding data science as a reliable and efficient tool, while avoiding mistakes and interpretations that could lead to ruin.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad10bb533cc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Whole New World: The Problems with Data]]></title>
            <link>https://medium.com/@wwillis125/a-whole-new-world-the-problems-with-data-1c4353e94e2e?source=rss-c9c85b209952------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c4353e94e2e</guid>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Wesley Willis]]></dc:creator>
            <pubDate>Wed, 30 Oct 2019 17:22:06 GMT</pubDate>
            <atom:updated>2019-10-30T17:22:06.339Z</atom:updated>
            <content:encoded><![CDATA[<p>The game is changing…</p><p>And the old rules no longer apply.</p><p>It is no secret that in the past few years the prevalence of technology at our disposal has drastically altered our personal lifestyles and choices. Every 90’s baby was warned early about the dangers of the internet and sharing personal information with complete strangers. However, in recent years, we have shifted completely away from this cautious outlook and willingly, almost aggressively, filled the internet with every detail of our personal information, putting our trust in these strangers for food, transportation, and even romance.</p><p>Often times, the narrative we are given about the potential dangers of our current lifestyle places the responsibility elsewhere. In 2016, in Kalamazoo, Michigan, an Uber driver, in between fares, began opening fire on random civilians, killing 6 and injuring more. In 2017, in Cleveland, Ohio, a man dubbed “The Facebook Killer” posted videos on Facebook of his heinous acts before a manhunt led to his suicide. Both of these companies faced a decent amount of public backlash, causing them to proclaim a shift to stricter practices in monitoring content and background checks. Other than expressing their condolences to the victims, there was no political involvement that led to any significant changes in how these companies operated.</p><p>While these drastic tragedies immediately grabbed the publics attention, they are hardly the only type of offense that we are subjecting ourselves to in this age of technology. Data manipulation and misuse presents a growing issue, one who’s potential magnitude is still as of yet uncapped.</p><p><strong>Data Security</strong></p><p>Data security is a growing issue who’s potential for damage does not always garner adequate concern. Over the past few years, tech giants such as Google, Facebook, and Apple have all experienced massive data breaches, compromising millions of their users privacy and exposing their information without their consent. These breaches were not only taken advantage of by hackers, but also various apps and businesses who were able to use that information to their corporate advantage. For Google, the breach of their Google+ social media exposed roughly 5 million users to their private information within google and elsewhere to be available to companies without their knowledge. Apple was exposed for a bug in their phones that allowed hackers information to users location services, passwords, and personal conversations for over two years. Facebook was responsible for over 500 million of their users data to be posted in public view by third party distributors, exposing their passwords, user interactions and other personal information. These are some of the more prominent incidences within major technology firms in the past few years, but unfortunately, data security has been a major concern for decades now, with breaches in financial institutions, government, and even healthcare. These growing data risks are becoming more frequent and affecting the lives of millions globally.</p><p><strong>Cambridge Analytica</strong></p><p>Recently, the story of Cambridge Analytica has helped to shed some light on the potential that data has to shape and influence our worldview. Cambridge Analytica was brought to the forefront of public awareness after the 2018 presidential election when it was discovered that they had acquired the data of millions of Facebook users without their consent, and used it to launch a misleading campaign geared at exploiting the worst in our inherent biases. While that is the extent of most of the public’s understanding, the issue here lies in that this was not the first instance of their involvement. CA has been associated with multiple political elections globally, using this same strategy of pitting two political or religious rivals against each other, spreading false information in order to increase the rifts and tension for their benefit. Their involvement has led to riots, cultural oppression, and religious genocide. To make matters worse, their influence has been documented for years in various media publications dating as early as their emergence in 2015 but, due to their prominence in media disruption, their practices never fully breached into the public awareness. Facebook, for their role, has agreed to pay a fine of $500k but has refused to accept liability or responsibility. Their fine equals to about 5 minutes of ad revenue.</p><p>As technology and personal lifestyle become ever more intertwined, it is important that we do not fall into the trap of doing what is easy, and instead do what is right. Major companies like Facebook, Google, and Apple are able to circumvent what would normally be their social responsibility by claiming that they are already civil servants, that their services are only applied to the extent that their users allow it. They use our participation as a way to avoid liability, while cashing in on our involvement and manipulating our interests. Stuck behind our computer screens, we distance ourselves from the realization that we are selling our souls for convenience and very little else in return. For those of us in the data and technology sphere, it is perhaps even more important that we acknowledge our involvement and the potential negative impacts our ideas or practices may result in. By taking the extra effort to consider a different perspective, we have the ability to circumvent or at least delay these negative impacts and improve our future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c4353e94e2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>