<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alberto Abreu on Medium]]></title>
        <description><![CDATA[Stories by Alberto Abreu on Medium]]></description>
        <link>https://medium.com/@abreualberto91?source=rss-f5b04783a31f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*DI-yRByN4b-ptq20RyOM3g.png</url>
            <title>Stories by Alberto Abreu on Medium</title>
            <link>https://medium.com/@abreualberto91?source=rss-f5b04783a31f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:43:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abreualberto91/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can budget allocation be related to poor Government performance and corruption?]]></title>
            <link>https://medium.com/data-science/can-budget-allocation-be-related-to-poor-government-performance-and-corruption-f9a1dc397bf0?source=rss-f5b04783a31f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9a1dc397bf0</guid>
            <category><![CDATA[apc]]></category>
            <category><![CDATA[corruption]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alberto Abreu]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 21:46:26 GMT</pubDate>
            <atom:updated>2019-10-04T09:18:52.089Z</atom:updated>
            <content:encoded><![CDATA[<h3>A machine learning analysis</h3><p><em>Special thanks to Prof. Michael Greencare and the UPF Barcelona School of Management.</em></p><h4>Abstract</h4><p><em>This paper aims to study the corruption perception index (CPI) and its relationship with the primary five categories of budget expenditures and five governance indicators variables using principal component analysis (PCA) and clustering analysis. The purpose is to identify similarities between different European countries and the changes between the years 2010 and 2016 by extracting the most important structuring information from the data set and providing a synthesis displayed as a graph. It takes it further by running a linear regression between CPI and the categorical variables to see what categories are strongly associated. Furthermore, the study runs a Regression Tree Analysis to determine the most significative variables to craft a potential explanatory model.</em></p><p><em>It starts with a brief introduction of the corruption perception index and the Governance key performance indicators. In the second part, the dataset and the methodology used; explain in chapter two and three. Results and conclusions are given in section four and chapter five respectively.</em></p><p><strong><em>Keywords</em></strong><em>: Corruption Perception Index, Budget expenditures, Governance Indicator, Rule of Law (RL), Principal Component Analysis, Clustering, data, variables, R-studio.</em></p><h3>1. Introduction</h3><p>The study of corruption is an extreme and delicate issue, recent studies from the World Economic- Forum indicate that the cost of corruption is approximately USD 2.6 trillion per <a href="https://www.icac.nsw.gov.au/about-corruption/why-expose-corru">year</a>. The impact affects the most vulnerable classes of the society and affects the countries’ development, for example, Venezuela which is one of the richest countries in the World but it’s submerged in a network of unprecedented corruption, leaving it in an economical and humanitarian crisis.</p><p>Additionally, corruption has an adverse effect on international investments, consequently weakens economic growth and undermining the rule of law. Likewise, there are more severe consequences of these practices; wasting public resources and money, undermining public trust in their governments and inefficiencies in their operations, affecting their economy.</p><p>The International Transparency is an organization with the mission of<a href="https://www.transparency.org/"> “stop corruption and promote transparency, accountability, and integrity at all levels and across all sectors of society”</a> and have created an index, to rank 180 countries worldwide by their perceived corruption levels in the public sector according to business people and experts. The scale goes from cero to one- hundred, where cero is highly corrupted and the top score, one-hundred is very clean.</p><p>Furthermore, the <a href="http://info.worldbank.org/governance/wgi/index.aspx#home">Worldwide Governance Indicators Project</a>, reports an aggregate and individual governance performance indicators for over 200 countries by using six different dimensions and giving them scores from -2.5 and 2.5, the higher the score, the strong governance performance. For the purpose of this study, we select the first five dimensions:</p><p>• <strong>Voice and Accountability (VA): </strong>Reflects perceptions of the extent to which a country’s citizens can participate in selecting their government, as well as freedom of expression, freedom of association, and a free media.</p><p>• <strong>Political Stability and Absence of Violence (PS-AV): </strong>Measures perceptions of the likelihood of political instability and/or politically-motivated violence, including terrorism.</p><p>• <strong>Government Effectiveness (GE): </strong>Reflects perceptions of the quality of public services, the quality of the civil service and the degree of its independence from political pressures, the quality of policy formulation and implementation, and the credibility of the government’s commitment to such policies.</p><p>• <strong>Regulatory Quality (RQ): </strong>Reflects perceptions of the ability of the government to formulate and implement sound policies and regulations that permit and promote private sector development.</p><p>• <strong>Rule of Law (RL): </strong>Reflects perceptions of the extent to which agents have confidence in and abide by the rules of society, and the quality of contract enforcement, property rights, the police, and the courts, as well as the likelihood of crime and violence.</p><p>Moreover, the <a href="http://ec.europa.eu/budget/figures/interactive/index_en.cfm">European Commission</a>, responsible for preparing and proposing draft budgets for the European Union, shows the UE budget expenditures for European countries by categories of different ranges of expenses. We only took categories that represent more than 2% of the total expenditure budget like:</p><p>• Smart and inclusive growth (SIG)</p><p>• Sustainable growth (SG-NR)</p><p>• Security and citizenship (SC)</p><p>• Administration (ADM)</p><h3>2. Dataset</h3><p>The dataset corresponded to twenty-eight (28) western European countries from the year 2010 and 2016 to see relevant changes during the period mentioned. Moreover, the dataset contains fifteen (15) variables including the CPI, governance indicators, budget expenditures in percentage of the total and, five (5) demographic variables like unemployment rate (UNPL), inflation rate (INF), growth domestic product per capita (GPD), purchasing power parity (PPP) and population (POP). The following table illustrates the head of the data set, showing only the first six countries in both periods, 2010 and 2016. BE10 indicated that the country is Belgium in the year 2010 and so on following the same logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*puk-tWgchEtV04lNWzd0wg.png" /></figure><h3>3. Methodology</h3><h4>3.1. Knowing your data</h4><p>The first step was getting to know the data by introducing a boxplot of the variables. This is a method for graphically portray numerical data through their quartiles and using whiskers to indicate variability outside the upper and lower quartiles. The purpose of this graphic representation is to detect outliers. In the database. The second step was to know the distribution of the predictor variables before starting to analyze them; this is important because certain analysis requires certain distributions. Ideally, a normal distribution is preferred without being skewed to one side.</p><h4>3.2. Clustering analysis</h4><p>The objective of this analysis is to group and extract value from an unstructured data set and make a partition of the countries into logical groupings before attempting to analyze it. This analysis will allow the researchers to take a glance at the data, and form structures base on what we found before going deeper into the analysis subject of this study. Regarding the analysis itself, clusters are sets of countries and years that share similar attributes and separates them into different groups based on their similarities in a hierarchical way. To obtain the cluster, we introduce the following code to R-Studio program:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*6c1pZO-6TC_PJg-1zt3ouA.png" /></figure><p>Furthermore, we analyze the data for each year separately to have a clear vision of the different groups and compare the two periods to see if there are significant changes. To achieve this, we divide the dataset into 2010 and 2016 data with the following code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*1VaI6YphB8saAlKYoQQX6Q.png" /></figure><p>And coding the two-separate cluster analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*xnf724dRkV8_1OIguR5Smg.png" /></figure><h4>3.3. Data standardization and Principal Component Analysis</h4><p>Before applying the PCA, the data set needs to be standardized because some variables can have large variances and other minimal variance. This is important to do before running the PCA since it is a variance maximizing method. The data standardization it’s done by taking out the mean and divide the data by its standard deviation. With all the data standardized, the methodology used was the Principal Component Analysis (PCA) to describe multivariate data.</p><p>One of the main reasons that we used this methodology is due to simplicity, the fact that we do not have missing values and mainly because it not requires normality. <strong>This test is not and hypothesis test and there are no clear distinctions between response variables and explanatory ones</strong> (Alain F. Zuur, Elena N. Leno &amp; Graham M. Smith, 2007). The idea behind this methodology is to reduce a large set of variables to a small set that still contains most of the information in the large dataset. The first principal component explains most of the variance and each succeeding component accounts for as much of the remaining variability. Furthermore, we extract the row principal coordinates from the PCA results.</p><h4>3.4. Linear Regressions, Regression Tree Analysis.</h4><p>Regarding the linear regression model, we first analyze the relationship between the dependent variable (Corruption Perception Index or CPI) and the predictor variables with a scatter plot. The underlying assumption in the LR is that the relationship between the two variables must be linear and additive.</p><p>In the first linear model, we used CPI as the independent variable and the governance indicators as the predictors. A second linear regression model was used with the same dependent variable but with the budget expenditures and the independent variables. Additionally, a third linear model was done with budget expenditures and governance indicators as well.</p><p>Furthermore, we did a Regression Tree Analysis to represent the recursive partition. Each terminal node represents a cell of the partition and each cell is just a constant estimate of the independent variable. The advantage of this analysis is to make predictions faster and there are easy to understand what variables are more significant in making the prediction. The regression tree was made with the variables that were significant in individual linear models as we can see in the following code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2enErb5wntseH08J5p33MA.png" /></figure><p>And lastly, a linear model using CPI and other significant variables with Spain as the reference level for all countries and 2010 as the base year. The purpose is to see if 2016 is significantly higher or lower than the base year and to identify significance among counties and the reference country, Spain.</p><h3>4. Results</h3><h4>4.1. Fist-stage results</h4><p>The results of the first stage, knowing your data, are very helpful to identify outliers and, the distributions. Regarding the identification of outliers, the boxplots analysis showed that Security and Citizenship (SC) have one outlier which is Malta in the year 2016. Many reports indicate that citizenship in the country of Malta is a very important department as they sell citizenship to billionaires and criminals all around the world as a free entrance to Europe.</p><p>This could be one of the causes that make this country an outlier in this category. Furthermore, Administration (ADM) also showed extreme outliers. These are Belgium and Luxemburg which are countries were many of their citizens work in other countries as they are very small and closer to bigger countries and also because they host EU institutions. Likewise, Purchasing Power Parity (PPP) boxplot indicates outliers which in this case are Hungary in both periods, reaching an all-time high in 2016. The following table contains the boxplots of the variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8iGiG0rbYSvHMlwBoGAig.png" /></figure><p>Moreover, table 3 shows the distributions of the variables by graphically demonstrating the density of one of the variables object in this study.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QqXzsrK5bM69bZ6Xbx-1iA.png" /></figure><p>In this analysis, we can confirm the presence of outliers as mention in the previous step. But most importantly, we can identify the distributions of the variables. The idea is that the distribution should be close to a normal distribution, that means, without being skewed to one side. Many variables are skewed to one side, but overall, the distribution its closed to normality, even though is not necessary to run a PCA analysis.</p><h4>4.2. Second stage results</h4><p>Regarding the results of the cluster analysis including both periods, we see many countries that come together, showing little change, but others get clustered with others. The following dendrogram shows the results for both periods, analyzing each country separately by years, for example, Belgium in 2010 and 2016 as the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HbHO8g70YzU7_p_IZ73zg.png" /></figure><p>Most of the countries uphold their positions in their cluster; meanwhile, others suffer drastically changes like Estonia (EE). It went from being in the same cluster with Latvia and Slovakia in 2010 to become part of countries like Sweden and the Netherlands. We can assume that this is an overall achievement in progress as they look more like well-developed countries. On the other hand, France (FR) also suffer a change between years. In 2010 was more like the UK and Germany but, at the end of 2016, it becomes more similar to Spain (ES). Malta is the country that changes the most. In 2010 was positioned with Estonia and Cypress but in 2016, they are unlikely similar to any country.</p><p>If we go deeper into the dendrogram, we can see two main clusters if we don’t count the blue line. On the far left, we can find countries like Belgium, Luxemburg, UK, Sweden, Netherland, and Denmark among others. It is commonly known that these countries are well developed and are top-ranked.</p><p>On the other hand, the right side is dominated by southern nations like Spain, Italy, Portugal, Greece and other countries located far more to the western region of Europe. It is very interesting how nations can change in six years, some of them improve their positions, others suffer a downgrade, but in general, most of the countries maintain their positions.</p><h4>4.3. Third stage results</h4><p>The results of the Principal Component Analysis with all the variables in the data set are shown in the next figure. For better interpretation purposes, country nodes were joint with a dotted line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KFG_fiSDgQByPxAgm8MjrQ.png" /></figure><p>The first dimension explains 43% of the total variance; meanwhile, the second dimension explains 13.5% which is the second biggest jump. So, we considered more the first one to explain the variance even though Sustainable Growth (SG-NR) seems to be the most significant contributor of variance in the second dimension and it’s highly correlated with population size.</p><p>The governance indicators Voice and Accountability (VA), Government Effectiveness (GE), Rule of Law (RL), Regulatory Quality (RG) and the Corruption Perception Index (CPI) are highly correlated. These correlations make sense as well-developed countries tend to have good governance indicators in all their aspects, making the nation very clean regarding corruption. As we can see, the higher governance indicator a country has, the cleaner they are in terms of perceived corruption.</p><p>Furthermore, there is a profoundly negative correlation between the variables mentioned before (circled in color red) and Smart and Inclusive Growth (SIG). It seems that the bigger the percentage expenditure in SIG, the more likely is the country to be very corrupt and have low governance indicators. From this, we can assume that investing in this category, can be a way for politicians to earn profit from public funding and does not generate any smart growth as the name indicates. This assumption can be useful for EU commission to study in detail the expenditure in this category for countries like Hungary that have more than 50% of their total budget in this category.</p><p>Regarding the unemployment rate, it seems to be negatively correlated with CPI and governance indicators. Corrupt countries have higher unemployment rates than those who are cleaner in this aspect. We can see countries like Spain, Italy, and Greece which are well-developed countries suffering from higher unemployment rates have lower CPI scores. From this, we can assume that corrupt governments affect directly to the society, concerning unemployment rates. Moreover, Inflation rate (INF), Security and Citizenship (SC) and the size of the population (POP) are the variables that explain the least the variance of the study.</p><p>And finally, the main result that we can be interpreted from the PCA analysis is that there are significant changes from the year 2010 and 2016. The tendency for most countries is to move to the right side of the graph. The result showed that states are receding in a matter of corruption and governance and this is spread in all the countries. It seems that the European country is getting worse scores regarding corruption and governance indicators and it could be a red alarm for the European Congress. It is interesting to realize that the lower governance scores, the more likely are for the country to be corrupt. We can assume that poor performance is not due to a lack of professionalism but a lack of values.</p><h4>4.4. Fourth stage results</h4><p>The linear regression model in this stage is related to the independent variable CPI and identifies which of the predictor variables explain the most in order to make predictions. We first identify which variables are more likely to be related to the independent variable. The following table shows the relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*896wHMVh-IlcoWLdsXojmQ.png" /></figure><p>The scatter plot and the smoothing line suggest that the governance indicators are linearly increasing with CPI. Others like SC, INF and, PPP showed no linearity. This was confirmed by running a regression with every single one of the variables.</p><p>The regression model with CPI and the governance indicators variables showed that Rule of Law is significantly different from cero with a low p-value. Moreover, the regression’s slope is also significantly different from zero. The R² indicates that the governance indicator variables explain CPI by 88.6%, corroborated by the adjusted R² of 87.5%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vkwPWTh448Ze-e40wlmi2w.png" /></figure><p>Furthermore, the second regression with budget expenditures indicates that all the variables were significantly different from cero with p-values less than a 5% confident level. The model indicates that the variables explain the variance by 48.4%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*SDuYjfT0EUJ_X0rG1DPtTg.png" /><figcaption>Figure 4. Linear Regression with budget expenditure as the explanatory variables and CPI as the target variable.</figcaption></figure><p>Additionally, we run a Regression Tree Analysis. First with all the data set from both years together using the variables shown in the code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jf1xu8RnWrMRguS3eLxirw.png" /></figure><p>And the result is the following regression tree:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G0-0o9YH-duviZVjuw7wlQ.png" /></figure><p>In the regression tree we can analyze that if GPD is lower than 3.5+04 and the Government Effectiveness is less than 0.93, we expect that the CPI scores will be 45.26, which is very low, and 19 countries fall in this side. On the other hand, if GDP is higher than the result and Voice and Accountability score is higher than 1.445, we expect that the CPI scores will be 88.89, 9 countries fall in this area. Let’s remember that countries can be counted twice because we are taking them as separate samples, 2010 and 2016.</p><h3>5. Conclusions</h3><p>• Governance indicators are strongly correlated with the perception of corruption, well- developed countries with higher governance scores are cleaners in terms of corruption than those countries with poor government performance. As we mention before, the lack of values in politicians affect directly the performance of the government and the general population is the most affected one and it can be reflected by the unemployment rates.</p><p>• The majority of European countries are performing poorly as we go into the future, there are significant changes between 2010 and 2016 and we can see in the PCA analysis that all the countries were moving away from good governance scores in 2016.</p><p>• Rules of law (RL) is the indicator that is most correlated with CPI. It seems that a country with a strong rule of law is cleaner in terms of corruption than countries with lower RL indicators. This conclusion makes sense as a strong law will punish those politicians who are corrupt.</p><p>• Many countries suffer changes between 2010 and 2016. France, for example, was on the well- developed countries cloud in the first year of study and move out to the less developed countries by 2016. Well developed countries like the Nordic ones, stay in their positions, meaning that they have not changed during the studied period.</p><p>• And finally, there is a strong negative correlation between Smart and Growth expenditures and CPI. It seems like countries that spent a significant amount of money in this category like Hungry, tent to have lower CPI scores.</p><h3>6. Data sources</h3><p>• <a href="http://info.worldbank.org/governance/wgi/index.aspx#home">http://info.worldbank.org/governance/wgi/index.aspx#home</a></p><p>• <a href="http://ec.europa.eu/budget/figures/interactive/index_en.cfm">http://ec.europa.eu/budget/figures/interactive/index_en.cfm</a></p><p>• <a href="http://ec.europa.eu/eurostat/statistics-explained/index.php/Unemployment_statistics">http://ec.europa.eu/eurostat/statistics-explained/index.php/Unemployment_statistics</a></p><p>• <a href="http://ec.europa.eu/eurostat/statistics-explained/index.php/File:HICP_all-">http://ec.europa.eu/eurostat/statistics-explained/index.php/File:HICP_all</a>-items,_annual_average_inflation_rates,_2006–2016_(%25)_YB17.png</p><p>• <a href="http://ec.europa.eu/eurostat/tgm/table.do?tab=table&amp;init=1&amp;language=en&amp;pcode=tps00001&amp;">http://ec.europa.eu/eurostat/tgm/table.do?tab=table&amp;init=1&amp;language=en&amp;pcode=tps00001&amp;</a> plugin=1</p><p>• <a href="https://data.oecd.org/conversion/purchasing-power-parities-ppp.htm#indicator-chart">https://data.oecd.org/conversion/purchasing-power-parities-ppp.htm#indicator-chart</a></p><p>• <a href="https://data.oecd.org/conversion/exchange-rates.htm#indicator-chart">https://data.oecd.org/conversion/exchange-rates.htm#indicator-chart</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9a1dc397bf0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/can-budget-allocation-be-related-to-poor-government-performance-and-corruption-f9a1dc397bf0">Can budget allocation be related to poor Government performance and corruption?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should I bring my laptop to my next flight?]]></title>
            <link>https://medium.com/@abreualberto91/flight-delays-in-europe-dad1af88711a?source=rss-f5b04783a31f------2</link>
            <guid isPermaLink="false">https://medium.com/p/dad1af88711a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[flightdelays]]></category>
            <category><![CDATA[ironhack]]></category>
            <dc:creator><![CDATA[Alberto Abreu]]></dc:creator>
            <pubDate>Thu, 01 Aug 2019 20:15:26 GMT</pubDate>
            <atom:updated>2019-08-05T22:09:55.055Z</atom:updated>
            <content:encoded><![CDATA[<h4>Statistical Analysis of flight delays in Europe.</h4><p>This analysis is influenced by my repeated behavior of packing unnecessary things when traveling for leisure. I cannot recall the last time I really used my laptop or read all my books when waiting for the airplane to take-off. For this reason, I want to assess you if you really need to pack more than one book or your laptop to make those delays time more pleasant.</p><p>For those who want to see the code and the procedure, <a href="https://github.com/albertoabreu91/Project-Week-5-Your-Own-Project">here</a> is the link to my GitHub repository.</p><h4>Abstract</h4><p><em>This paper aims to study the flights&#39; delays and its relationship with countries, airports, and daytime using correlation analysis and hypothesis testing from real data obtain through the </em><a href="https://aviation-edge.com/"><em>Aviation Edge API</em></a><em> from July 25, 2019, to July 29, 2019. The purpose is to compare Spain’s delay time, in terms of departure, arrival and total delay time in minutes and see if it is statistically different from the rest of Europe. Moreover, the same comparison is provided with the Barcelona Airport (El Prat de Llobregat) against the rest of the airports in Spain.</em></p><p><em>Keywords: Flight’s delays, Europe, Barcelona, correlation, hypothesis testing.</em></p><h3>Objective</h3><p>As mentioned above, the objective looks to compare and test differences between the mean departure delays on different levels.</p><p>1- The mean departure delay in Spain is greater than in the rest of Europe.</p><p>2- The Barcelona Airport mean departure delay is lower in comparison with the airports in Spain.</p><p>3- Vueling is not the best airline if you are in a hurry, as the mean departure delay is greater than the rest of the carriers.</p><p>4- Daytime in terms of morning, afternoon and evening, night has an effect in classifying the flight as a delayed flight. (<a href="https://search.usa.gov/search?query=15+minutes&amp;op=GO&amp;affiliate=dot-bts">By regulations, if the flight is delayed more than 15 min it should be classified as delayed</a>).</p><h3><strong>Introduction</strong></h3><p>Nowadays, more and more people are traveling through the airspace more frequently, either for leisure or work. On average, approx 2.8 million passengers fly per day above the European sky, <a href="https://ec.europa.eu/eurostat/statistics-explained/index.php/Air_transport_statistics#Progressive_growth_in_air_transport_of_passengers_in_the_course_of_2017">reaching up to 1.043 billion at the end of 2017, representing an increase of 7.3% from 2016</a>. Just in Spain, the total number of passengers reached 206 million, around 20% of the total.</p><p><a href="https://www.eurocontrol.int/">Eurocontrol</a>, an aviation platform that tracks all the flights in Europe, calculates that from January 2019 until now (July 2019) the total delay time in minutes is around 83 thousand minutes, more or less 58 days. If we keep at this rhythm we will end up the year with 120 days of delay time. As you have seen, Air travel is not always a stress-free experience. Flight delays are more common and it is of the importance of the airlines since passengers can claim up to €600 if their flight has been delayed for more than three hours under the European Law called EC 261.</p><p>As mentioned before, Spain contributes 20% of the total passengers in Europe, playing an important role in transportation in Europe. For this reason, this paper wants to test if Spain’s mean delay time is greater than the rest of Europe, among others test with its main airline carriers like Vueling.</p><p>Additionally, it is important to understand the differences between departure, arrival and total delay.</p><ul><li><strong>Departure delay</strong>: Amount of time in minutes from Schedule time, which is the time that a plane should close the gate and Actual time, representing the observed time when the airline closed it. If it is positive, means that there where some delay.</li><li><strong>Arrival delay</strong>: Amount of time in minutes from Schedule time, and the time it actually arrived at the gate. Again, if it is positive, means that the airplane should arrive at the gate at 3:45 pm and it arrives at 3:50 pm as an example.</li><li><strong>Total delay</strong>: It is the departure delay time in minutes plus the arrival time.</li></ul><h3>Dataset</h3><h4>Requesting the data</h4><p>The dataset corresponds to around 70 thousand flights gathered in real-time from July 25 and July 29 of 2019, using the <a href="https://aviation-edge.com/">Aviation Edge API</a>. The response is a JSON file with more than 29 attributes of a flight departing from a specific airport and with the status of landed. Among the many attributes, this paper used: Airline name, status, departure airport, scheduled departure time, departure delay time, arrival airport, scheduled arrival time, arrival delay time and country of departure among others.</p><p>A function was created to obtain all the flight information from all the airports of Spain, Italy, France, and Germany with a single loop, looping through a list of airports with their IATA code formats.</p><h4>Cleaning the data</h4><p>After requesting the data, the next step was to select and clean the data of interest. After transforming the JSON file into a Pandas DataFrame, the column headers were dictionaries so, in order to select the attributes of interest, lambdas were needed as seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8BIX3I7HZvJBVNv0f5uaoQ.png" /></figure><p>The final result is a DataFrame as shown below, where you can see a partial screenshot of the pandas&#39; df.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jbWGi3UC6ctIVMTE--jRYw.png" /></figure><h3>Methodology</h3><h4>Knowing your data</h4><p>The first step was getting to know the data by introducing a frequency distribution and a boxplot of the variables. This is a method for graphically portray numerical data through their quartiles and using whiskers to indicate variability outside the upper and lower quartiles. The purpose of this graphic representation is to detect outliers. The second step was to know the distribution of the predictor variables before starting to analyze them; this is important because certain analysis requires certain distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-2DKcIZanrQSqaemSviQwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*5xyIR7mpuPa7K4APXK0arQ.png" /></figure><p>The second step was to eliminate outliers. For the purpose of this analysis, outliers that were more than 3.5 times the interquartile range plus the third quartile were removed. The selection of the criteria was a mix between feeling and the fact that after 3 hours of delay, or 180 min, airlines have to pay their passengers six hundred euros so I wanted to remove those cases from the sample.</p><p>Furthermore, for further analysis, it was relevant to scope the analysis into the more frequent flights in Europe. After some percentage calculations, the market share of the principal carriers was Lufthansa, Air France, and Iberia respectively.</p><h4>Relations between the random variables</h4><p>The final step of this chapter was to detect relations between some variables. As the collection of data was daily, a comparison between the days of the week and the delay times was needed to detect if some days are more likely to have delays in term of minutes than other to make some recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LLh5j67qdeG-7gBONtVuBA.png" /></figure><p>As seen in the figure, it seems that the weekend tends to have more mean delays time than other days of the week. Unfortunately, I could not include Tuesday and Wednesday to the analysis. Also, as a further step, it would be interesting to see if there are significant differences between the mean delay of the weekend and weekdays.</p><p>Moreover, as the scope of this analysis was to compare Spain vs Europe and Barcelona vs the rest of Spain, similar plots were done but only including the relevant data for each comparison. In the figure below, a scatter plot was done to see the relation between departure delay and total delay including a trend line to see the relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hBX6SJCUWGwzlf0wSDmCfg.png" /></figure><p>There seems to be a strong, linear and positive correlation between departure delay and total delay (corr coef 0.934). From a visual perspective, the regression line predicts that if your flight is delayed 20 min, the entire delay will be almost 40 min. So, if someone is waiting for you at the other side of the flight, be a good friend and tell them to arrive later at the airport.</p><p>Another insight that seems a bit logical is between hours of the day and delay time. The later it is, you can expect more delay time between flights with a positive, linear and not so strong relationship as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VTqFPFUMZMz_EDlJFNBqag.png" /></figure><p>Furthermore, as the last hypothesis wants to test if Vueling’s mean departure delay is greater than the rest, a plot of the means was needed to visually detect if there are notable differences. From this perspective, at least comparing it with the rest of most frequent flights in Barcelona, there seems to be no notable difference. But a one-tail test is needed to corroborate the claim that Vueling is actually doing worse than the rest in term of departure delays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l8v-8814OiNtwZgukkt37Q.png" /></figure><h3>Hypothesis testings</h3><p><strong>To answer the first hypothesis</strong>, claiming that the mean departure delay in Spain is greater than the rest of Europe, a t-student one tail test was conducted as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0QyeLX90pf4zHveK-46Jw.png" /></figure><p>The same test was conducted by taking out Spain from the sample, calculating the new mean departure delay and comparing it with Spain. In both cases, we rejected the null hypothesis. Only in one case where the “true mean” gathered from an aviation website was tested against the sample mean, the null hypothesis was rejected, indicating that with a 95% confidence, there is statistical evidence for the alternative.</p><p><strong>The second hypothesis, </strong>where the claim was that the Barcelona airport’s mean departure time is lower than Spain’s one, three tests were conducted.</p><p>In the first one, the mean of Spain included the observations of Barcelona. The second one without Barcelona and the third one with only the main airports in Spain, including Catalan city.</p><p>In all three cases, the was no statistical evidence to reject the null hypothesis, indicating that the mean departure time in Barcelona is equal or greater than the rest of Spain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CtTXHd5SgmpbZcBDll82ag.png" /></figure><p><strong>To test the third hypothesis, </strong>where the hypothesis claims that Vueling&#39;s departure delay time is greater than the rest of the carriers in this sample, an identical test was conducted. In this case, the null hypothesis was rejected, showing that there is evidence for the alternative.</p><p>And last but not least, <strong>the fourth hypothesis was conducted with Fisher’s exact test</strong>. In this test, the study aims to detect if there is an effect regarding the time of the day and if the flight will have more than 15 min of departure delay time. The first step was to classify with 0 if the flight is before 2 pm and 1 if it was afterward. Same for the delay, if it was more than 15 min delay, excluding the flights with no delay, the flight was classified as 0 and more than 15 min as 1. The next step consisted in creating a contingency table and conducting the test as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*eWMv2qjV28lYPUDUaYht7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6KHxsXOjxuqqQSYTfkmraw.png" /></figure><p>As the test demonstrated, there is statistical evidence to reject the null hypothesis and go for the alternative. And a calculation of the probabilities was conducted as an informative table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*QplCla2Ti7OqLdYnqkIJxg.png" /></figure><p>For the scope of this paper, this was enough but a test should only tell you to dig deeper. The analyst should keep digging on the subject to find more relevant insights.</p><h3>Conclusion</h3><p>This paper was one of my projects in the IronHack Bootcamp school for Data Analytics as part of the statistical module. Where the scope of it was to pick a subject and perform some basic statistical analysis. For this reason, I believe that this analysis can take a further step to come up with more insightful information that can help airlines perform data-driven decisions regarding delay times. Furthermore, taking into account more analysis like ANOVA with categorical variables with numerical variables could be of interest when trying to see differences between the means of delays and if the carrier is low cost or not among other variables that can influence on delay times. This was a nice project to put in practice the lectures seen in class and I encourage other data analyst to perform similar and more complex analysis on this subject since there is a lot of data out there.</p><p>Especial thanks to Prof. Claudia Rivera and the TAs who help me in this journey and to <a href="https://aviation-edge.com/">Aviation Edge</a> for providing the information. And remember, always try to buy early flight tickets, avoid traveling during the weekends or at least bring a book to read before you take off.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dad1af88711a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>