<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alvaro Corrales Cano on Medium]]></title>
        <description><![CDATA[Stories by Alvaro Corrales Cano on Medium]]></description>
        <link>https://medium.com/@acorralescano?source=rss-639157103f84------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fzG7Ua6qqS4HCvf6qYRQdw.jpeg</url>
            <title>Stories by Alvaro Corrales Cano on Medium</title>
            <link>https://medium.com/@acorralescano?source=rss-639157103f84------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:39:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@acorralescano/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Don’t Do Laundry Today, It Will Be Cheaper Tomorrow]]></title>
            <link>https://medium.com/data-science/dont-do-laundry-today-it-will-be-cheaper-tomorrow-84a6baa4e626?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/84a6baa4e626</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 16:54:30 GMT</pubDate>
            <atom:updated>2024-10-21T16:54:30.326Z</atom:updated>
            <content:encoded><![CDATA[<h4>Analysing electricity price changes in London through causal inference</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_y27TLD_zk3Ds-dbdyX3sQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@artlambi?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Arthur Lambillotte</a> on <a href="https://unsplash.com/photos/a-person-holding-a-device-YG19gb9jW0I?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Sometimes, we just have to experiment. Whether it’s a company improving a product or a public official aiming for a policy goal, they all face questions like:</p><ul><li>Will a new pricing plan boost customer retention in our telecom?</li><li>Will a new onboarding process increase app conversions?</li><li>Will a reminders system reduce hospital no-show rates?</li><li>If so, by how much?</li></ul><p>Such issues are tricky. The costs of not addressing them can be serious, but the wrong solution can be even costlier. To reduce risk, many turn to experiments: before rolling out a new reminders system to the whole healthcare network of a region, we can test it out in one hospital; before deploying a new onboarding system for all users on a platform, we may only show it to some; before changing our optic fibre tariffs in a whole country, we can test them in just a city. These are all smaller scale versions of the intended solution, experiments, but how do we know that they work? This is where causal analysis kicks in.</p><p>In this article, I propose a causal framework to analyse an experiment that happened in London in 2013: the <a href="https://innovation.ukpowernetworks.co.uk/projects/low-carbon-london">Low Carbon London Project</a> [1]. Among other objectives, the aim of the experiment was to study how households reacted to announced electricity price changes. The raw data showed a decrease in electricity consumption in households who were warned about price changes. But why? Did the initiative <em>actually</em> lead to lower consumption or a more efficient use of the electricity network, or did other external factors play a role? If the program did play a role, how large was the effect? The cost of the project, £2.3 billion, motivates a proper evaluation of the results.</p><h3>The Low Carbon London Project</h3><p>Between January 2011 and December 2014, UK Power Networks conducted the Low Carbon London Project (LCL). The overarching goal of the program was to explore the role of smart electricity networks in supporting the transition to a low-carbon economy through various initiatives. One of these initiatives was a new Dynamic Time of Use (DToU) tariff, which was tested throughout 2013. The tariff consisted of a three-tier electricity pricing mechanism and it was rolled out to 1100 households in London. Most of the time, these households would be subject to a “normal” tariff of 11.76 pence per kilowatt-hour (p/kWh). However, some days they would get a price change notification either in their smart meters or by SMS: for some 30-minute slots in the following day, their tariff would go up to 67.20p/kWh or down to 3.99p/kWh. We call these households our treatment group. Alongside these, a control group of about 4500 other households were subject to a standard, flat tariff of 14.228pence/kWh.</p><p>Upon concluding the project, UK Power Networks released some of the data it had collected [2], enabling researchers the conduct their evaluations of the initiative. The idea of the dynamic tariff was to use the network more efficiently, through a better control of demand. Did it work? Let’s explore the data.</p><h4>Consumption data</h4><p>The LCL data consists of half-hourly smart-meter readings from 5566 anonymous households over the period from November 2011 to February 2014. This amounts to a total of 167,932,474 observations. While this is an impressive amount of data, the granularity can be an overkill for the purposes of our analysis. It takes up a lot of space in memory, and it is unnecessarily noisy. Therefore, we will aggregate it to total daily consumption per household.</p><p>The chart below shows us average daily consumption per household for the whole sample and period, split by tariff group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6GZ_HgaIxyADXeuGXWWdBg.png" /><figcaption>Averarge daily consumption over time. Image by author.</figcaption></figure><p>Ultimately, our goal is to determine whether or not the red line (treatment group) took a divergent path from the blue one due to the existence of the DToU tariff between January and December 2013. If so, we’d like to know how much it cost or saved in terms of energy and money.</p><p>Something that catches our eye is that the red line does seem to run consistently below the blue one, both before and after 1 January 2013. To further analyse this point, we can take the differences in average daily consumption between the two groups. In the chart below, we plot these differences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*fLAqQTQaJuZdt3NxtWSD8g.png" /><figcaption>Difference in average daily consumption over time. Image by author.</figcaption></figure><p>Two main conclusions arise from this chart. The first one is that there were indeed differences between control and treatment group in 2013. Treated households consumed 0.84 kWh less per day on average. The second conclusion, though, is that there were also differences in 2012 — that is, prior to the introduction of the DToU tariff. In 2012, households in the treatment (but still untreated) group, consumed 0.62 kWh less a day on average. It seems that the two groups were already different before the introduction of the tariff. Therefore, we can’t say that the introduction of the tariff led to a reduction in consumption of 0.84 kWh a day. A naive conclusion would be to that the reduction was of 0.22 kWh (0.84 – 0.62), but that would be too simplistic. Is this difference statistically different from zero? Can we attribute it to the program alone or are there other confounding factors muddling up the estimate? To answer these questions, we need causal analysis.</p><p>Notice the volatility of the differences at the beginning of 2012. The reason is that the sample is not complete until mid-to-late 2012, as we can see in the chart below. If we were to include to analyse the data as it is now, the sensitivity of our estimates would be affected, as we could not compare some households’ consumption in 2012 with that of 2013. To work around this issue we will only use households who joined the program in 2012 or before and stayed till the end of the program. This leaves us with 520 households: 77 in the treatment group, 443 in the control group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*OGKF4nYiyTgwg99p8zqL0w.png" /><figcaption>Count of households in the LCL project by month (2012–2013). Image by author.</figcaption></figure><h4>Additional household data</h4><p>On the project site, we can find the times of the tariffs that applied for each half-hour slot in 2013. Approximately 86% of the time, treated households were subject to the normal tariff; 9.5% of the time they got a low tariff, while the rest of the time they got a high tariff. Interestingly, this means that, over the whole year, their average price whas 13.52 p/kWh. Intuitively, we would think that treated households should have consumed more, cheaper energy than their control counterparts! The fact that they didn’t suggests that households’ demand for energy was not only determined by price. Other factors may have also played a role. For instance, households could have delayed energy-hungry activities to days when they knew electricity would be cheaper (like doing laundry). This could in turn lead to a more efficient use of energy, thereby reducing the overall amount of energy consumed in spite of consumption spikes. We’ll test this out in our causal analysis.</p><p>Another interesting dataset associated to the LCL Project is household demographics, as provided by <a href="https://acorn.caci.co.uk/">Acorn</a>. Acorn is a geodemographic segmentation tool that classifies UK households based on census data. The resulting classification ranks households by level of affluence from A (most affluent) to U (most deprived). It would be safe to assume that energy consumption profiles vary according to demographics, so this dataset could be very useful for our analysis.</p><p>At the time of writing, household Acorn classification was not available on the LCL’s website, but I was able to find it on <a href="https://www.kaggle.com/datasets/jeanmidev/smart-meters-in-london">Kaggle</a> [3]. The chart below shows the distribution of Acorn segments in our sample, split by tariff group:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6XWNxiHfQIUKpcuutJrYzg.png" /><figcaption>Distribution of households by Acorn group. Image by author.</figcaption></figure><p>As we can see, the distributions are fairly similar, but not entirely. Households in the treatment group (in orange) tend to be more affluent, with a higher proportion of them in the upper segments of the Acorn classification. The distribution of control households, on the other hand, is proportionally skewed towards the more deprived segments, especially segment Q. This suggests that group assignment was not necessarily random, and thus households cannot be compared without taking demographics into account.</p><h4>Weather data</h4><p>The final dataset that we are going to use in our analysis is about weather. Households tend to use more energy in cold weather to heat their homes. Therefore, variables such as (apparent) temperature, cloud cover and the likes are a big determinant of energy demand in a given day. For our purposes, we will use daily weather data from London from 2012 and 2013, as provided by the Visual Crossing Weather data service [4].</p><h3>Baseline model</h3><p>Before diving into more advanced modeling, we’ll first define a baseline model to serve as a benchmark for future results. Our baseline will be a differences-in-differences (DiD) linear model, with a household’s daily consumption as dependent variable. This model is ideal when we can’t assume that the treatment and control groups are not statistically identical. It only requires that both groups follow a trend with respect to the dependent variable prior to treatment (the so-called parallel trends assumption). In our case, the first chart above indicates that the assumption holds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HM25ifbcfjmZTfDl9892sA.png" /><figcaption>DiD model. Image by author.</figcaption></figure><p>In the above expression, the effect of a treatment on outcome <em>Y </em>for individual <em>i </em>at time <em>t </em>is quantified by <em>β</em>₃, which captures the interaction of being in the treatment group <em>when</em> the treatment took place. In other words<em>, β</em>₃ quantifies the difference in electricity consumption due to the effect of the DToU tariff.<em> β</em>₁<em> </em>and<em> β</em>₂, on the other hand, account for differences across groups and time. Similarly <em>α </em>is a constant term<em>, X </em>is a vector of covariates<em> </em>and<em> ϵ </em>is a random noise component.</p><p>Python’s library Statsmodels allows us to estimate the DiD model very easily. We won’t go into the details of the code, but it is available in this <a href="https://github.com/AlvaroCorrales/smart-meters-london/blob/main/Causal%20analysis.ipynb">notebook</a>. The table below shows the estimated DiD model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJ9I8rxqtombVIxypudqxw.png" /><figcaption>Differences-in-Differences model. Image by author.</figcaption></figure><p>Our treatment effect,<em> β</em>₃, is the coefficient associated to treat_2013. This variable is the product of year_2013 and treat, which are dummy variables. They take value 1 if the observation corresponds to year 2013 and if the household is in the treatment group, respectively. Therefore, treat_2013 takes value 1 only when the year is 2013 <em>and </em>the household is in the treatment group. The value for the coefficient is -0.2958. This means that, in 2013, households under the dynamic tariff consumed 0.2958 kWh less per day than households under the flat tariff — about 108 kWh over the year. To put it in monetary terms, if the treated households had had the flat tariff instead, they would have spent £15.36 more that year on average.</p><h3>DoWhy + EconML Causal model</h3><p>The DiD model is a good benchmark, but it comes with some limitations. The most evident is that it assumes a straight linear relationship between the treatment and the outcome. A more flexible approach is to use graph models. Graph models allow us to incorporate more complex relationships between variables, informed by our domain knowledge of the problem.</p><p>In Python, we can use the library <a href="https://www.pywhy.org/dowhy/v0.11.1/index.html">DoWhy</a> [5][6] to estimate causal effects with graphs. DoWhy is a powerful, easy-to-use tool, and I can’t recommend it enough. One of the features that makes it great is its integration with <a href="https://econml.azurewebsites.net/spec/overview.html">EconML</a> [7], another library for causal analysis that allows us to estimate individualised treatment effects with a variety of machine learning models.</p><h4>Building a causal graph</h4><p>DoWhy’s causal inference flow consists of 4 steps: model definition, identification, estimation and refutal. However, before we can kick of the flow, we need a directed graph that represents our data. While there are methods to infer graph structures from a dataset, it is far more recommendable to define it ourselves by using our domain knowledge about the problem and the data. This is what we do in the snippet below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0021ba59fb52e661fddde3d97089cda/href">https://medium.com/media/c0021ba59fb52e661fddde3d97089cda/href</a></iframe><p>As we can see, after initialising our graph object with networkx, we define a list of causes that could influence energy consumption: day of the week, temperature, perceived temperature, precipitation, cloud cover, and (encoded) Acorn category. We connect all of these with an edge towards our target variable for energy consumption, kwh_hh. On top of these, we draw a link from being in the treatment group to consumption, via receiving treatment in 2013: treat -&gt; treat_2013 -&gt; kwh_hh. This is the causal path that we want to quantify. We know that being on the treatment group is already an indicator of lower consumption, so we know there is another link other than receiving treatment. Therefore, we add an extra edge linking treatment to consumption without passing through treatment in 2013: treat -&gt; kwh_hh. Finally, we believe that Acorn group may have played a role in assignment to treatment, so we draw this edge as well. The following chart depicts the resulting graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRuYiznq-wC8rgiBkMAI1g.png" /><figcaption>Our causal graph. Image by author.</figcaption></figure><p>Notice that this is not an exhaustive graph. Other factors could influence household energy consumption, and including them in the graph (or even rearraging the existing ones) could influence our estimated causal effects.</p><h4>Model definition</h4><p>Now that we’ve got the graph, we can start DoWhy’s causal inference flow. Step number 1 is the definition of a causal model. This causal model allows us to specify the role of a variable in the outcome for another one. Our causal model looks as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1246c3a035691fd24c3c0342c8e585f/href">https://medium.com/media/e1246c3a035691fd24c3c0342c8e585f/href</a></iframe><p>Treatment variable is treat_2013. There is a common factor of both receiving treatment in 2013 and energy consumption in 2013, which is belonging to the treatment group. Finally, effect modifiers are other causes for energy consumption. Visually, we have the following graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*TSf_ophUTCCga3LBxRMerA.png" /><figcaption>Causal model. Image by author.</figcaption></figure><p>Essentially, it is the same as the causal graph, but this is just because our graph is very simple to begin with. For other problems, we could isolate different causal models from more complex graphs.</p><h4>Identification</h4><p>Step 2 in DoWhy’s flow is to identify the relevant estimand for our model. To put it simply, in this step we determine the expression that captures the effect of our treatment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d83c1276d2faa5384c82681f50065fcc/href">https://medium.com/media/d83c1276d2faa5384c82681f50065fcc/href</a></iframe><p>The above code snippet prints the following message:</p><pre>### Estimand : 1<br>Estimand name: backdoor<br>Estimand expression:<br>     d                       <br>────────────(E[kwh_hh|treat])<br>d[treat₂₀₁₃]                 <br>Estimand assumption 1, Unconfoundedness: If U→{treat_2013} and U→kwh_hh then P(kwh_hh|treat_2013,treat,U) = P(kwh_hh|treat_2013,treat)</pre><p>Assumption 1, unconfoundedness, is crucial here: it states that whatever factors affect both treatment and electricity consumption in 2013 are accounted for in the model. Since we didn’t run the experiment, it is a tricky assumption. We have tried to address it by accounting for Acorn group in assignment to treatment, as well as by allowing for treatment effects prior to 2013.</p><h4>Estimation</h4><p>Once we’ve got the expression for the estimand, we can move on to step 3: estimation; that is, computing the expression from step 2. Here is where we leverage DoWhy’s integration with EconML. EconML allows us to model the relationship between variables that may influence both assignment to treatment and the final outcome, using a technique called Double Machine Learning (DML). In DML, both treatment and final outcome are modelled using any machine learning algorithm as function of the rest of covariates — hence the “Double.” The predictions of these models are combined in a final, unbiased model. To learn more about DML, visit <a href="https://econml.azurewebsites.net/spec/estimation/dml.html#usage-faqs">this section of EconML’s docs</a>.</p><p>The code snippet below shows our estimation approach. We have chosen to model treatment and final outcome using a gradient boosting regressor. We are also generating polynomial features of degree 2 out of explanatory variables. The final model is an elastic net, fitted with cross-validation. We can see this model as an updated, regularised version of our linear DiD model, as elastic net is a linear model itself. Notice that we can also get confidence intervals around our estimte: we simply need to set the boolean in line 9 to True (although the process takes a lot of memory).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c95728a1094f150e6247c58f8f4796e/href">https://medium.com/media/2c95728a1094f150e6247c58f8f4796e/href</a></iframe><p>Using this method, we estimate an the average treatment effect of −0.2565, meaning that households with the dynamic tariff used 0.2565 kWh less per day than households in the control group. The estimate is slightly lower than our DiD benchmark, -0.2958, but in the same order of magnitude. The results translate into £13.31 of savings in the treated households’ energy bill over the year, or 93.6 kWh in reduced consumption. To put it in perspective, the average household in our sample consumed 3893 kWh a year, so the reduction seems modest: 2.4% [8].</p><p>EconML’s implementation is not without caveats: the DML approach comes with a lot of intrinsic randomness. Models in the first and/or second stage may need a random seed for initialisation (for instance, in the case of boosted trees, or for cross-validation). For some datasets, especially smaller ones, this can have a big impact in the estimated causal effect. In our case, different combinations of algorithms and seeds in both stages always yielded consitent results in terms of sign — treated households always consumed less energy, but the effect size varied from less than 0.1 to almost 0.4.</p><h4>Refutal</h4><p>Now that we have our estimate, the question is: can we trust it? This is what step 4 of DoWhy’s flow, refutation, tries to address. The idea of this step is that, for the estimate to be a valid causal estimate, it has to pass <em>at least </em>some robustness checks. Notice the emphasis on “at least:” if the estimate passes the tests, it may still be incorrect for other reasons. We should see these robustness checks as a minimum quality bar. DoWhy’s standard checks are:</p><ol><li><strong>Adding a random common cause</strong>. This shouldn’t affect a robust estimate.</li><li><strong>Replacing treatment with a random placebo variable</strong>. Treatment effect should disappear.</li><li><strong>Removing a subset of the data.</strong> The estimate should be more or less the same.</li></ol><p>The code snippet below shows all three tests for our causal estimate. Luckily, it passes them all! (Again, this process takes a lot of time and memory.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fff0ecced15e7c5a98f2f75d86edfbc5/href">https://medium.com/media/fff0ecced15e7c5a98f2f75d86edfbc5/href</a></iframe><h3>Separate effects</h3><p>In the previous sections we quantified the effect of the LCL project on treated households: they consumed less energy than households in the control group by about 0.25 kWh a day, saving over £13 a year. But is that all that we can say about the experiment? The answer is no.</p><p>Remember that households subject to the Dynamic Time of Use tariff were sent <strong>two </strong>price signals: sometimes the price would increase, sometimes it would decrease. Thinking in terms of domestic economics, it would make sense that these price signals had in fact opposite effects: a lower price signal should induce households to consume more electricity, or at least reallocate consumption towards the day with lower prices (remember signals were sent one day in advance of the price change); the opposite should apply for days with higher prices. Fortunately, we can test this hypothesis.</p><p>The code snippet below redefines our original causal graph. The main alterations are paths 1 and 2, in lines 15 and 23. They represent two new edges through which low and high price signals operate: the household is assigned to treatment &gt; treatment starts in 2013 &gt; the household receives a high (low) price signal &gt; the household consumes electricity accordingly. To build this graph, we have created two new variables: got_high_signal and got_low_signal, which take value 1 if the treated household received the corresponding signal, and 0 otherwise. The rest of the graph remains the same, including the edge from treatment in 2013 to consumption: belonging to the treatment group and knowing it’s running may impact consumption decisions irrespective of price signals.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b590d61c6b835ca6e401175dac740183/href">https://medium.com/media/b590d61c6b835ca6e401175dac740183/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muI6P05ld-uIb3sbEaUMMA.png" /><figcaption>Modified causal graph. Image by author.</figcaption></figure><p>From this graph, we can build two separate causal models to estimate the impact of receiving a low and a high price signal, respectively. Each of them will quantify the impact of receiving one of the signals. In other words, got_low_signal and got_high_signal become the new treatment variables. The code snippet below illustrates steps 1 to 3 of the DoWhy flow for the low price signal. We can build an equivalent model for the high price signal value simply replacing the treatment variable in the code below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7763c67b638262e3e4b733bb024c4804/href">https://medium.com/media/7763c67b638262e3e4b733bb024c4804/href</a></iframe><p>The effect of sending a low price signal, as estimated by the model, is indeed positive: on days with lower prices, treated households increased their consumption by 0.326 kWh compared to households in the control group. This means that, even though households under the DToU tariff consumed less overall, they took advantage of lower prices to perform energy-hungry activities (laundry!). The effect of sending a high signal, on the contrary, is not that strong. Even though it’s negative (as we would expect), it is too close to zero for it to be significant: -0.0026.</p><h3>Discussion and parting thoughts</h3><p>In this article, we analysed the results of the Low Carbon London Project, a program that explored how warning people about electricity prices affected their consumption. To do so, we proposed a causal framework to answer the question if the program led to a lower or more efficient use of electricity by households, and by how much.</p><p>While the observed reduction in electricity consumption may seem modest — an average savings of £13 per household over the year, or 2.4% kWh less— the real significance lies in the methodology we used [9]. It allowed us to establish a clear causal link between the intervention and the outcome of the program. These causal links can help organisations inform more reliable cost-benefit analyses and guide them in decision-making processes. It lets them understand not just <em>what</em> happened, but also <em>why</em> it happened and how much it cost them.</p><p>Want to check my code? You can find it on this <a href="https://github.com/AlvaroCorrales/smart-meters-london">Github repo</a>.</p><h3>Notes and references</h3><p>[1] Low Carbon London Project. URL: <a href="https://innovation.ukpowernetworks.co.uk/projects/low-carbon-london#project-single__accordion">https://innovation.ukpowernetworks.co.uk/projects/low-carbon-london#project-single__accordion</a> (last accessed 17 Oct 2024).</p><p>[2] Available at <a href="https://data.london.gov.uk/dataset/smartmeter-energy-use-data-in-london-households">https://data.london.gov.uk/dataset/smartmeter-energy-use-data-in-london-households</a> (last accessed 17 Oct 2024). Licensed under Creative Commons Attribution License.</p><p>[3] Available at <a href="https://www.kaggle.com/datasets/jeanmidev/smart-meters-in-london">https://www.kaggle.com/datasets/jeanmidev/smart-meters-in-london</a> (last accessed 17 Oct 2024). Licensed under Open Data Commons Open Database License (ODbL).</p><p>[4] You can learn more about the service and access the same data yourself at <a href="https://www.visualcrossing.com/">https://www.visualcrossing.com/</a>. (last accessed 17 Oct 2024).</p><p>[5] Sharma, A., &amp; Kiciman, E. (2020). <em>DoWhy: An end-to-end library for causal inference</em>. arXiv preprint arXiv:2011.04216.</p><p>[6] Blöbaum, P., Götz, P., Budhathoki, K., Mastakouri, A. A., &amp; Janzing, D. (2022). <em>DoWhy-GCM: An extension of DoWhy for causal inference in graphical causal models</em>. arXiv preprint arXiv:2206.06821.</p><p>[7] ALICE Team. <em>EconML: A Python package for estimating heterogeneous treatment effects</em>. Microsoft Research. <a href="https://econml.azurewebsites.net/spec/overview.html">https://econml.azurewebsites.net/spec/overview.html</a></p><p>[8] According to the British government, the average household in the UK consumes 3509 kWh every year. Source: <a href="https://assets.publishing.service.gov.uk/media/63a31383e90e075875eae2e7/subnational_electricity_and_gas_consumption_summary_report_2021.pdf">https://assets.publishing.service.gov.uk/media/63a31383e90e075875eae2e7/subnational_electricity_and_gas_consumption_summary_report_2021.pdf</a></p><p>[9] For UK Power Networks, LCL had wider goals that made the project worthwile in other fronts. See the final report: <a href="https://d1oyzg0jo3ox9g.cloudfront.net/app/uploads/2023/10/Project-Closedown-Report-March-2015.pdf">https://d1oyzg0jo3ox9g.cloudfront.net/app/uploads/2023/10/Project-Closedown-Report-March-2015.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84a6baa4e626" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dont-do-laundry-today-it-will-be-cheaper-tomorrow-84a6baa4e626">Don’t Do Laundry Today, It Will Be Cheaper Tomorrow</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting in the Age of Foundation Models]]></title>
            <link>https://medium.com/data-science/forecasting-in-the-age-of-foundation-models-8cd4eea0079d?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cd4eea0079d</guid>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[foundation-models]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Sat, 20 Jul 2024 16:40:24 GMT</pubDate>
            <atom:updated>2024-07-20T16:40:24.101Z</atom:updated>
            <content:encoded><![CDATA[<h4>Benchmarking Lag-Llama against XGBoost</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kxtzFVqoxj1N24qE3WDTmA.jpeg" /><figcaption>Cliffs near Ribadesella. Photo by <a href="https://unsplash.com/@henrymd?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Enric Domas</a> on <a href="https://unsplash.com/photos/body-of-water-near-mountain-during-daytime-FNJfLCMO3Bk?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>On Hugging Face, there are 20 models tagged “time series” at the time of writing. While certainly not a lot (the “text-generation-inference” tag yields 125,950 results), time series forecasting with foundation models is an interesting enough niche for big companies like Amazon, IBM and Salesforce to have developed their own models: Chronos, TinyTimeMixer and Moirai, respectively. At the time of writing, one of the most popular on Hugging Face by number of likes is <a href="https://huggingface.co/time-series-foundation-models/Lag-Llama">Lag-Llama</a>, a univariate probabilistic model. Developed by Kashif Rasul, Arjun Ashok and co-authors [1], Lag-Llama was open sourced in February 2024. The authors of the model claim “strong zero-shot generalization capabilities” on a variety of datasets across different domains. Once fine-tuned for specific tasks, they also claim it to be the best general-purpose model of its kind. Big words!</p><p>In this blog, I showcase my experience fine-tuning Lag-Llama, and test its capabilities against a more classical machine learning approach. In particular, I benchmark it against an XGBoost model designed to handle univariate time series data. Gradient boosting algorithms such as XGBoost are widely considered the epitome of “classical” machine learning (as opposed to deep-learning), and have been shown to perform extremely well with tabular data [2]. Therefore, it seems fitting to use XGBoost to test if Lag-Llama lives up to its promises. Will the foundation model do better? Spoiler alert: it is not that simple.</p><p>By the way, I will not go into the details of the model architecture, but the <a href="https://time-series-foundation-models.github.io/lag-llama.pdf">paper</a> is worth a read, as is this nice <a href="https://towardsdatascience.com/lag-llama-open-source-foundation-model-for-time-series-forecasting-9afdfaf2bd7c">walk-through</a> by Marco Peixeiro.</p><h3>The data</h3><p>The data that I use for this exercise is a 4-year-long series of hourly wave heights off the coast of Ribadesella, a town in the Spanish region of Asturias. The series is available at the Spanish ports authority <a href="https://www.puertos.es/es-es/oceanografia/Paginas/portus.aspx">data portal</a>. The measurements were taken at a station located in the coordinates (43.5, -5.083), from 18/06/2020 00:00 to 18/06/2024 23:00 [3]. I have decided to aggregate the series to a daily level, taking the max over the 24 observations in each day. The reason is that the concepts that we go through in this post are better illustrated from a slightly less granular point of view. Otherwise, the results become very volatile very quickly. Therefore, our target variable is the maximum height of the waves recorded in a day, measured in meters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*YDil_s_KKh5sz3gzVRidRw.png" /><figcaption>Distribution of target data. Image by author</figcaption></figure><p>There are several reasons why I chose this series: the first one is that the Lag-Llama model was trained on <em>some</em> weather-related data, although not a lot, relatively. I would expect the model to find this type of data slightly challenging, but still manageable. The second one is that, while meteorological forecasts are typically produced using numerical weather models, statistical models can still complement these forecasts, specially for long-range predictions. At the very least, in the era of climate change, I think statistical models can tell us what we would typically expect, and how far off it is from what is actually happening.</p><p>The dataset is pretty standard and does not require much preprocessing other than imputing a few missing values. The plot below shows what it looks like after we split it into train, validation and test sets. The last two sets have a length of 5 months. To know more about how we preprocess the data, have a look at this <a href="https://github.com/AlvaroCorrales/wave-height-prediction/blob/main/Process%20data.ipynb">notebook</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*Brlr47eMppY7oTzCuUoBVQ.png" /><figcaption>Maximum daily wave heights in Ribadesella. Image by author</figcaption></figure><h3>The test</h3><p>We are going to benchmark Lag-Llama against XGBoost on two univariate forecasting tasks: point forecasting and probabilistic forecasting. The two tasks complement each other: point forecasting gives us a specific, single-number prediction, whereas probabilistic forecasting gives us a confidence region around it. One could say that Lag-Llama was only trained for the latter, so we should focus on that one. While that is true, I believe that humans find it easier to understand a single number than a confidence interval, so I think the point forecast is still useful, even if just for illustrative purposes.</p><p>There are many factors that we need to consider when producing a forecast. Some of the most important include the forecast horizon, the last observation(s) that we feed the model, or how often we update the model (if at all). Different combinations of factors yield their own types of forecast with their own interpretations. In our case, we are going to do a recursive multi-step forecast without updating the model, with a step size of 7 days. This means that we are going to use one single model to produce batches of 7 forecasts at a time. After producing one batch, the model sees 7 more data points, corresponding to the dates that it just predicted, and it produces 7 more forecasts. The model, however, is not retrained as new data is available. In terms of our dataset, this means that we will produce a forecast of maximum wave heights for each day of the next week.</p><p>For point forecasting, we are going to use the <a href="https://en.wikipedia.org/wiki/Mean_absolute_error">Mean Absolute Error (MAE)</a> as performance metric. In the case of probabilistic forecasting, we will aim for empirical coverage or <a href="https://en.wikipedia.org/wiki/Coverage_probability">coverage probability</a> of 80%.</p><p>The scene is set. Let’s get our hands dirty with the experiments!</p><h3>Defining the XGBoost model</h3><p>While originally not designed for time series forecasting, gradient boosting algorithms in general, and XGBoost in particular, can be great predictors. We just need to feed the algorithm the data in the right format. For instance, if we want to use three lags of our target series, we can simply create three columns (say, in a pandas dataframe) with the lagged values and voilà! An XGBoost forecaster. However, this process can quickly become onerous, especially if we intend to use many lags. Luckily for us, the library <a href="https://skforecast.org/0.12.1/">Skforecast</a> [4] can do this. In fact, Skforecast is the one-stop shop for developing and testing all sorts of forecasters. I honestly can’t recommend it enough!</p><p>Creating a forecaster with Skforecast is pretty straightforward. We just need to create a ForecasterAutoreg object with an XGBoost regressor, which we can then fine-tune. On top of the XGBoost hyperparamters that we would typically optimise for, we also need to search for the best number of lags to include in our model. To do that, Skforecast provides a Bayesian optimisation method that runs Optuna on the background, bayesian_search_forecaster.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b986053c4d85baf16e89592f91a2e58/href">https://medium.com/media/2b986053c4d85baf16e89592f91a2e58/href</a></iframe><p>The search yields an optimised XGBoost forecaster which, among other hyperparameters, uses 21 lags of the target variable, i.e. 21 days of maximum wave heights to predict the next:</p><pre>Lags: [ 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21] <br>Parameters: {&#39;n_estimators&#39;: 900, <br>             &#39;max_depth&#39;: 12, <br>             &#39;learning_rate&#39;: 0.30394338985367425, <br>             &#39;reg_alpha&#39;: 0.5, <br>             &#39;reg_lambda&#39;: 0.0, <br>             &#39;subsample&#39;: 1.0, <br>             &#39;colsample_bytree&#39;: 0.2}</pre><p>But is the model any good? Let’s find out!</p><h3>XGBoost results</h3><h4>Point forecasting</h4><p>First, let’s look at how well the XGBoost forecaster does at predicting the next 7 days of maximum wave heights. The chart below plots the predictions against the actual values of our test set. We can see that the prediction tends to follow the general trend of the actual data, but it is far from perfect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*7LG0IJoinKj47IPIk6rVQg.png" /><figcaption>Maximum wave heights and XGBoost predictions. Image by author</figcaption></figure><p>To create the predictions depicted above, we have used Skforecast’s backtesting_forecaster function, which allows us to evaluate the model on a test set, as shown in the following code snippet. On top of the predictions, we also get a performance metric, which in our case is the MAE.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e98e7e57568c44a5ab5de2f0f1eefc24/href">https://medium.com/media/e98e7e57568c44a5ab5de2f0f1eefc24/href</a></iframe><p>Our model’s MAE is 0.64. This means that, on average, our predictions are 64cm off the actual measurement. To put this value in context, the standard deviation of the target variable is 0.86. Therefore, our model’s average error is about 0.74 units of the standard deviation. Furthermore, if we were to simply use the previous equivalent observation as a dummy best guess for our forecast, we would get a MAE of 0.84 (see point 1 of this <a href="https://github.com/AlvaroCorrales/wave-height-prediction/blob/main/Model%20XGBoost.ipynb">notebook</a>). All things considered, it seems that, so far, our model is better than a simple logical rule, which is a relief!</p><h4>Probabilistic forecasting</h4><p>Skforecast allows us to calculate distribution intervals where the future outcome is likely to fall. The library provides two methods: using either bootstrapped residuals or quantile regression. The results are not very different, so I am going to focus here on the bootstrapped residuals method. You can see more results in part 3 of this <a href="https://github.com/AlvaroCorrales/wave-height-prediction/blob/main/Model%20XGBoost.ipynb">notebook</a>.</p><p>The idea of constructing prediction intervals using bootstrapped residuals is that we can randomly take a model’s forecast errors (residuals) an add them to the same model’s forecasts. By repeating the process a number of times, we can construct an equal number of alternative forecasts. These predictions follow a distribution that we can get prediction intervals from. In other words, if we assume that the forecast errors are random and identically distributed in time, adding these errors creates a universe of equally possible forecasts. In this universe, we would expect to see at least a percentage of the actual values of the forecasted series. In our case, we will aim for 80% of the values (that is, a coverage of 80%).</p><p>To construct the prediction intervals with Skforecast, we follow a 3-step process: first, we generate forecasts for our validation set; second, we compute the residuals from those forecasts and store them in our forecaster class; third, we get the probabilistic forecasts for our test set. The second and third steps are illustrated in the snippet below (the first one corresponds to the code snippet in the previous section). Lines 14-17 are the parameters that govern our bootstrap calculation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68ad73e74a0b432482559a9710d9c26c/href">https://medium.com/media/68ad73e74a0b432482559a9710d9c26c/href</a></iframe><p>The resulting prediction intervals are depicted in the chart below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*XbBY6HQMyQlIoNQOee7eyQ.png" /><figcaption>Bootstraped prediction intervals with XGBoost forecaster. Image by author</figcaption></figure><p>An 84.67% of values in the test set fall within our prediction intervals, which is just above our target of 80%. While this is not bad, it may also mean that we are overshooting and our intervals are too big. Think of it this way: if we said that tomorrow’s waves would be between 0 and infinity meters high, we would always be right, but the forecast would be useless! To get a idea of how big our intervals are, Skforecast’s docs suggest that we compute the area of our intervals by thaking the sum of the differences between the upper and lower boundaries of the intervals. This is not an absolute measure, but it can help us compare across forecasters. In our case, the area is 348.28.</p><p>These are our XGBoost results. How about Lag-Llama?</p><h3>Inference with Lag-Llama</h3><p>The authors of Lag-Llama provide a <a href="https://colab.research.google.com/drive/1DRAzLUPxsd-0r8b-o4nlyFXrjw_ZajJJ?usp=sharing">demo notebook</a> to start forecasting with the model without fine-tuning it. The code is ready to produce probabilistic forecasts given a set horizon, or prediction length, and a context length, or the amount of previous data points to consider in the forecast. We just need to call the get_llama_predictions function below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5a9bae85d8f4d276b97e73e7854c968/href">https://medium.com/media/d5a9bae85d8f4d276b97e73e7854c968/href</a></iframe><p>The core of the funtion is a LagLlamaEstimatorclass (lines 19–47), which is a <a href="https://ts.gluon.ai/stable/api/gluonts/gluonts.torch.model.estimator.html#gluonts.torch.model.estimator.PyTorchLightningEstimator">Pytorch Lightning Estimator</a> based on the <a href="https://ts.gluon.ai/stable/index.html">GluonTS</a> [5] package for probabilistic forecasting. I suggest you go through the GluonTS docs to get familiar with the package.</p><p>We can leverage the get_llama_predictions function to produce recursive multistep forecasts. We simply need to produce batches of predictions over consecutive batches. This is what we do in the function below, recursive_forecast:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f24af841a03192f8929ea04b0bf44b12/href">https://medium.com/media/f24af841a03192f8929ea04b0bf44b12/href</a></iframe><p>In lines 37 to 39 of the code snippet above, we extract the percentiles 10 and 90 to produce an 80% probabilistic forecast (90–10), as well as the median of the probabilistic prediction to get a point forecast. If you need to learn more about the output of the model, I suggest you have a look at the author’s tutorial mentioned above.</p><h3>Zero-shot results with Lag-Llama</h3><p>The authors of the model advise that different datasets and forecasting tasks may require differen context lenghts. In our case, we try context lenghts of 32, 64 and 128 tokens (lags). The chart below shows the results of the 64-token model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*aOdAxfOfCtkj-Cr7UFKdVw.png" /><figcaption>Zero-shot Lag-Llama predictions with a context length of 128 tokens. Image by author</figcaption></figure><h4>Point forecasting</h4><p>As we said above, Lag-Llama is not meant to calculate point forecasts, but we can get one by taking the median of the probabilistic interval that it returns. Another potential point forecast would be the mean, although it would be subject to outliers in the interval. In any case, for our particular dataset, both options yield similar results.</p><p>The MAE of the 32-token model was 0.75. That of the 64-token model was 0.77, while the MAE of the 128-token model was 0.77 as well. These are all higher than the XGBoost forecaster’s, which went down to 0.64. In fact, they are very close to the baseline, dummy model that used the previous week’s value as today’s forecast (MAE 0.84).</p><h4>Probabilistic forecasting</h4><p>With a predicted interval coverage of 68.67% and an interval area of 280.05, the 32-token forecast does not perform up to our required standard. The 64-token one, reaches an 74.0% coverage, which gets closer to the 80% region that we are looking for. To do so, it takes an interval area of 343.74. The 128-token model overshoots but is closer to the mark, with an 84.67% coverage and an area of 399.25. We can grasp an interesting trend here: more coverage implies a larger interval area. This should not always be the case — a very narrow interval could always be right. However, in practice this trade-off is very much present in all the models I have trained.</p><p>Notice the periodic bulges in the chart (around March 10 or April 7, for instance). Since we are producing a 7-day forecast, the bulges represent the increased uncertainty as we move away from the last observation that the model saw. In other words, a forecast for the next day will be less uncertain than a forecast for the day after next, and so on.</p><p>The 128-token model yields very similar results to the XGBoost forecaster, which had an area 348.28 and a coverage of 84.67%. Based on these results, we can say that, with no training, Lag-Llama’s performance is rather solid and up to par with an optimised traditional forecaster.</p><h3>Fine-tuning Lag-Llama</h3><p>Lag-Llama’s Github repo comes with a “best practices” section with tips to use and fine-tune the model. The authors especially recommend tuning the context length and the learning rate. We are going to explore some of the suggested values for these hyperparameters. The code snippet below, which I have taken and modified from the <a href="https://colab.research.google.com/drive/1uvTmh-pe1zO5TeaaRVDdoEWJ5dFDI-pA?usp=sharing#scrollTo=6c8783db-3a6b-4eb6-b2cd-05e25614cbe2">authors’ fine-tuning tutorial notebook</a>, shows how we can conduct a small grid search:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32e9cb79fdecabfa104d97966b78ae7d/href">https://medium.com/media/32e9cb79fdecabfa104d97966b78ae7d/href</a></iframe><p>In the code above, we loop over context lengths of 32, 64, and 128 tokens, as well as learning rates of 0.001, 0.001, and 0.005. Within the loop, we also calculate some test metrics: Coverage[0.8], Coverage[0.9] and Mean Absolute Error of (MAE) Coverage. Coverage[0.x] measures how many predictions fall within their prediction interval. For instance, a good model should have a Coverage[0.8] of around 80%. MAE Coverage, on the other hand, measures the deviation of the actual coverage probabilities from the nominal coverage levels. Therefore, a good model in our case should be one with a small MAE and coverages of around 80% and 90%, respectively.</p><p>One of the main differences with respect to the original fine-tuning code from the authors is line 46. In that line, the original code does not include a validation set. In my experience, not including it meant that all models that I trained ended up overfitting the training data. On the other hand, with a validation set most models were optimised in Epoch 0 and did not improve the validation loss thereafter. With more data, we may see less extreme outcomes.</p><p>Once trained, most of the models in the loop yield a MAE of 0.5 and coverages of 1 on the test set. This means that the models have very broad prediction intervals, but the prediction is not very precise. The model that strikes a better balance is model 6 (counting from 0 to 8 in the loop), with the following hyperparameters and metrics:</p><pre> {&#39;context_length&#39;: 128,<br>  &#39;lr&#39;: 0.001,<br>  &#39;Coverage[0.8]&#39;: 0.7142857142857143,<br>  &#39;Coverage[0.9]&#39;: 0.8571428571428571,<br>  &#39;MAE_Coverage&#39;: 0.36666666666666664}</pre><p>Since this is the most promising model, we are going to run it through the tests that we have with the other forecasters.</p><h3>Fine-tuning results</h3><p>The chart below shows the predictions from the fine-tuned model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*YUfifkwmqnn2YmneZLW1vQ.png" /><figcaption>Fine-tuned Lag-Llama predictions with a context length of 64 tokens. Image by author</figcaption></figure><p>Something that catches the eye very quickly is that prediction intervals are substantially smaller than those from the zero-shot version. In fact, the interval area is 188.69. With these prediction intervals, the model reaches a coverage of 56.67% over the 7-day recursive forecast. Remember that our best zero-shot predictions, with a 128-token context, had an area of 399.25, reaching a coverage of 84.67%. This means a 55% reduction in the interval area, with only a 33% decrease in coverage. However, the fine-tuned model is too far from the 80% coverage that we are aiming for, whereas the zero-shot model with 128 tokens wasn’t.</p><p>When it comes to point forecasting, the MAE of the model is 0.77, which is not an improvement over the zero-shot forecasts and worse than the XGBoost forecaster.</p><p>Overall, the fine-tuned model leaves doesn’t leave us a good picture: it doesn’t do better than a zero-shot better at either point of probabilistic forecasting. The authors do suggest that the model can improve if fine-tuned with more data, so it may be that our training set was not large enough.</p><h3>Discussion and parting thoughts</h3><p>To recap, let’s ask again the question that we set out at the beginning of this blog: Is Lag-Llama better at forecasting than XGBoost? For our dataset, the short answer is no, they are similar. The long answer is more complicated, though. Zero-shot forecasts with a 128-token context length were at the same level as XGBoost in terms of probabilistic forecasting. Fine-tuning Lag-Llama further reduced the prediction area, making the model’s <em>correct forecasts</em> more precise, albeit at a substantial cost in terms of probabilistc coverage. This raises the question of where the model could get with more training data. But more data we did not have, so we can’t say that Lag-Llama beat XGBoost.</p><p>These results inevitably open a broader debate: since one is not better than the other in terms of performance, which one should we use? In this case, we’d need to consider other variables such as ease of use, deployment and maintenance and inference costs. While I haven’t formally tested the two options in any of those aspects, I suspect the XGBoost would come out better. Less data- and resource-hungry, pretty robust to overfitting and time-tested are hard-to-beat characteristics, and XGBoost has them all.</p><p>But do not believe me! The code that I used is publicly available on this <a href="https://github.com/AlvaroCorrales/wave-height-prediction">Github repo</a>, so go have a look and run it yourself.</p><h3>References</h3><p>[1] Rasul, K., Ashok, A., Williams, A. R., Ghonia, H., Bhagwatkar, R., Khorasani, A., Bayazi, M. J. D., Adamopoulos, G., Riachi, R., Hassen, N., Biloš, M., Garg, S., Schneider, A., Chapados, N., Drouin, A., Zantedeschi, V., Nevmyvaka, Y., &amp; Rish, I. (2024). <em>Lag-Llama: Towards foundation models for probabilistic time series forecasting</em>. arXiv. <a href="https://arxiv.org/abs/2310.08278">https://arxiv.org/abs/2310.08278</a></p><p>[2] Shwartz-Ziv, R., &amp; Armon, A. (2021). <em>Tabular data: Deep learning is not all you need</em>. arXiv. <a href="https://arxiv.org/abs/2106.03253">https://arxiv.org/abs/2106.03253</a></p><p>[3] Puertos del Estado. (2024). <em>3106036 — Punto SIMAR Lon: -5.083 — Lat: 43.5, Oleaje</em>. <a href="https://www.puertos.es/es-es/oceanografia/Paginas/portus.aspx">https://www.puertos.es/es-es/oceanografia/Paginas/portus.aspx</a>. Last accessed: 25/06/2024. [More info (in Spanish) at: <a href="https://bancodatos.puertos.es/BD/informes/INT_8.pdf">https://bancodatos.puertos.es/BD/informes/INT_8.pdf</a>]</p><p>[4] Amat Rodrigo, J., &amp; Escobar Ortiz, J. (2024). <em>skforecast</em> (Version 0.12.1) [Software]. BSD-3-Clause. <a href="https://doi.org/10.5281/zenodo.8382788">https://doi.org/10.5281/zenodo.8382788</a></p><p>[5] Alexandrov, A., Benidis, K., Bohlke-Schneider, M., Flunkert, V., Gasthaus, J., Januschowski, T., Maddix, D. C., Rangapuram, S., Salinas, D., Schulz, J., Stella, L., Türkmen, A. C., &amp; Wang, Y. (2020). GluonTS: Probabilistic and neural time series modeling in Python. <em>Journal of Machine Learning Research, 21</em>(116), 1–6. <a href="http://jmlr.org/papers/v21/19-820.html">http://jmlr.org/papers/v21/19-820.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cd4eea0079d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/forecasting-in-the-age-of-foundation-models-8cd4eea0079d">Forecasting in the Age of Foundation Models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to work with censored data]]></title>
            <link>https://medium.com/data-science/how-to-work-with-censored-data-26b7b6e2c7f?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/26b7b6e2c7f</guid>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gig-economy]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 17:01:51 GMT</pubDate>
            <atom:updated>2023-12-05T05:02:19.226Z</atom:updated>
            <content:encoded><![CDATA[<h4>An application of the Tobit model to the gig economy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h5SvI0pSUIGuD_C5" /><figcaption>Photo by <a href="https://unsplash.com/@kaip?utm_source=medium&amp;utm_medium=referral">Kai Pilger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Linear regression is probably the most basic topic in statistical learning. Almost all courses in machine learning, statistics and econometrics start with the fundamentals of Ordinary Least Squares. And why not? Its defining equation <em>y = β</em>₀​+<em>β₁</em>​<em>x + e </em>seems easy enough. But why would you care about the OLS equation? Granted, in the days of large language models and self-driving cars, talking about linear regression sounds almost arcane. And yet, as we will see in this article, sometimes a flavour of linear regression is exactly what you need.</p><p>One of the advantages of linear regression is explainability. In our equation above, we can say that the impact of <em>x </em>on <em>y </em>is<em>, </em>on average, <em>β₁</em>, allowing for some random noise, <em>e</em>, and a constant term,<em> β</em>₀<em>. </em>However, this simple interpretation is underpinned by strong assumptions on how the data is generated: is the relation between <em>x </em>and <em>y </em>actually linear? Are we measuring <em>x </em>and <em>y </em>properly? Are we even seeing a representative sample of our target population? As you may have guessed, the answer to those questions tends to be <em>No</em> in real life. Which leads me to the use case that I want to present in this article: the Tobit model for censored data.</p><h3>Gig economy meets censored data</h3><pre><strong>Note</strong>: This article uses simulated data. This data is meant to be representative of a real-life scenario, but it does not contain any real transactions from any company mentioned in the article nor does it reflect their actual state of operations.</pre><p>Imagine that you are a data scientist at firm in the gig industry. Let’s say your company runs a riding app such as Uber. (This can apply to any match-making business, from food delivery, to home services or transportation.) Your manager wants to know what is the impact of offered rates on the number of rides a driver completes in a day. You’ve got a dataset that contains how many rides drivers completed after logging into the app in a day, including if those drivers who didn’t complete any. The dataset also contains a variable on earnings, which has been normalised by hour so that rides are comparable. “Easy — you think to yourself — I’ll just run a linear regression and thake the beta”. Are you sure? Look at the chart below. It shows how many rides drivers in a city completed as a function of hourly earnings. Anything strange?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*ts04NwPwbge2lo380SqMAw.png" /><figcaption>Rides completed as a function of hourly earnings. Source: Author</figcaption></figure><p>At first sight, not really. While there is a lot of variation, there seems to be a positive correlation between hourly earnings and number of rides completed. In other words, drivers work more if they are paid more. Economics 101. But let’s unpack the data a bit more. Look at the histogram below. It shows the distribution of rides completed per day per driver. As you can see, there’s a big agglomeration of points for 0 rides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*HOtTes9Rsalbhs6XPUInPw.png" /><figcaption>Distribution of rides completed per day per driver. Source: Author</figcaption></figure><p>Now look at the scatter plot again. Doesn’t it feel like the cloud <em>wants </em>to continue on the negative side of the x-axis? “But this wouldn’t make sense — you think — You can’t do negative rides in a day!” And that’s true, you can’t. But your data is not necessarily faulty either. On the contrary, this is a well-known phenomenon in the field of Economics. What we are seeing here is a case of data censoring: drivers only complete a ride if they are willing to do so at a particular rate. However, some other drivers will have logged into the app, looked at the rates being paid at that time and decided that they are not worth their time. These are the ones who did zero rides.</p><h3>The Tobit model</h3><blockquote>“What do you mean, less than nothing?” replied Wilbur. “I don’t think there is any such thing as less than nothing. Nothing is absolutely the limit of nothingness. It’s the lowest you can go. It’s the end of the line. How can something be less than nothing? If there were something that was less than nothing, then nothing would not be nothing, it would be something even though it’s just a very little bit of something. But if nothing is nothing, then nothing has nothing that is less than it is.” <br> — <em>James Tobin’s 1958 quote of E.B. White’s </em>Charlotte’s Web <em>(1952)</em></blockquote><p>This problem was first identified by James Robin who, in his 1958 paper [1], studied the case of individual expenditure on durable goods. In particular, he noticed that his sample of American households showed that most of them would report zero consumption of automobiles or durable goods, as they couldn’t afford them. In other words, his sample necessarily had a lower limit at zero. He cleverly represented this problem in figure 1 of his paper, shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uHFncpTaLiVHhBMdXt7bIA.png" /><figcaption>Screenshot from Tobin (1958) — Vintage, right?</figcaption></figure><p>Notice the important implications for your OLS estimator. If you were to run a simple linear regression, your <em>β₁ </em>would be inconsistent, because your observed data is not distributed linearly across the chart. If we take Tobin’s image above, you would want an estimated <em>β₁ </em>for the slope of the line that goes from A to B — in other words, the unconditional average relationship between <em>x</em> and <em>y</em>. However, because your data is censored, the observed relation (the conditional average) is not linear, but something like the line between O and B. Basically, your model would be seeing far too many zeros for certain hourly rates, which should in reality be negative values if we allowed for “negative rides” to maintain the linear relationship (i.e. if we let the data cloud continue into the negative values of the y-axis). Therefore, because our sample is censored at zero rides, the data will not allow us to draw the true straight line that represents the linear relationship between <em>x</em> and <em>y.</em></p><p>In his paper, Tobin proposed a method for the estimation of <em>β₁ </em>with censored data, which in time would come to be known as the <em>Tobit</em> model. His idea was to correct the way we estimate <em>β₁ </em>by accounting for the probability that an observation is censored at 0 (or at any other value, from below or from above). Why <em>Tobit </em>model and not <em>Tobin </em>model, you ask? The term was coined by Arthur Goldberger, another economist. It is the combination of the words <em>Tobit </em>and and <em>probit, </em>the classification technique used to calculate the probability that an observation is censored.</p><p>While the derivation of the model is beyond the scope of this article, let’s have a quick look at the problem for completeness. The function below shows the maximisation problem that we have to solve by maximum likelihood estimation to get to our true <em>β₁ </em>(shown in vector notation <em>as β, </em>together with <em>β</em>₀)<em>. </em>Tobin’s solution implies that errors are independent and normally distributed, with standard deviation σ. In the function, <em>dᵢ</em> would take value 0 if <em>y = 0</em>, and 1 otherwise. Thus, the left-hand side of the sum (the one multiplied by <em>dᵢ</em>, on the top line) is equivalent to the OLS likelihood function, whereas the right-hand side (multiplied by <em>(1-dᵢ)</em>, bottom line), accounts for the probability that observation <em>i </em>is censored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wd4PjzhF7_0Qo0zJnmUXxw.png" /><figcaption>Likelihood function for the Tobit model. Source: Author</figcaption></figure><h3>Implementation</h3><p>Cool as it is, to the best of my knowledge there is no Python package to use the Tobit model in Python (at least I haven’t seen any on pip or conda). However, I found <a href="https://github.com/jamesdj">James Jensen’s implementation</a> quite useful. I forked my own version of it, which you can find <a href="https://github.com/AlvaroCorrales/tobit">here</a>, and I highly encourage you to do the same!</p><p>Remember that I said that <em>β₁ </em>would be inconsistent if we estimated it by simple OLS? In mathematical terms, that means that our estimate of <em>β </em>will not converge to its actual value. More simply, the slope of our linear relationship will be off. To illustrate what that would look like, the green line in the chart below shows the OLS fitted values in our drivers example. On the other hand, estimating <em>β </em>using the Tobit model gives us a slightly higher value (0.35 vs 0.33), which is represented by the blue line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*RwyIgyPyTnCXmyVVuUzrOw.png" /><figcaption>Best-fit lines between hourly earnings and rides completed: OLS (green) and Tobit model (blue). Source: Author</figcaption></figure><p>The difference between the OLS and the Tobit estimate can vary depending on your model specification — sometimes it’ll be bigger, sometimes it’ll be smaller. Notice that we haven’t included any control variables in our equation, nor do we have any solid identification strategy underpinning it. This means that our model could be improved, which would likely impact our estimates. Hence, we shouldn’t try to derive any meaning from these particular numbers.</p><h3>Marginal effects</h3><p>Now that you have run your Tobit model and come up with an estimate for <em>β₁, </em>you may be tempted to say that each driver will complete <em>β₁</em> more rides for each additional dollar. But it’s not so simple! Let’s see why.</p><p>Generally speaking, our aim is to estimate the average impact of a change in earnings on rides completed. This is what economists call a “marginal effect”. While we will skip mathematical details, this is equivalent to the derivate of <em>y </em>(rides) with respect to <em>x </em>(earnings) in our equation. In the OLS model, this is simply <em>β₁. </em>However, in the Tobit model, the marginal effect of <em>x </em>on <em>y </em>is slightly more complicated. Because we included the probability that an observation is censored in our derivation of <em>β₁</em>, we need to account for it in our interpretation of it as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*_prnajnia5dVi-eOVClrSA.png" /><figcaption>The marginal effect of x on y in the Tobit model. Source: Author</figcaption></figure><p>The equation above shows how a change in earnings translates into rides completed on average — hence the conditional expectation on the left hand side. On the right-hand side, we’ve got our<em> β₁</em>, multiplied by the cummulative distribution function (CDF, represented by Φ) of the Normal distribution evaluated at our model’s estimated value for <em>y, </em>that is, <em>β</em>₀​+<em>β₁</em>​<em>x</em>​, and normalised by our error’s standard deviation, σ.</p><p>What I just described above may sound complicated, but you’ll see it actually isn’t. What the equation for the marginal effect means is that the average impact of a change in hourly earnings indeed depends on <em>β₁. </em>However, the coefficient is weighted by the probability that an individual is willing to work at the present hourly rate, as represented by the CDF. In other words, the marginal effect of offering an extra dollar will be different depending on whether that ride was originally $10 or $40. This makes sense if you think about it: the probability that someone works for $40 is probably higher than working for $10!</p><p>To see how this works in practice, let’s return to our Python implementation. To calculate marginal effects, I created a function, called margins, that builds on top of James Jensen’s solution. With our estimated <em>β₁ </em>of 0.35 and a starting hourly earnings rate of $10, the estimated marginal effect would be 0.17. Changing the starting rate to $40, however, yields a marginal effect of 0.32. If we take the hourly earnigs further up, say $100, the marginal effect is already 0.35. As you can see, the higher the earnings, the closer our marginal effect gets to <em>β₁, </em>as it is more likely for a driver to be willing to complete a ride for higher hourly earnings.</p><h3>Conclusion</h3><p>In this article we have seen how some apparently simple regression problems may actually get tricky if our observed data is censored. This is the case for companies in the gig economy industry, who may want to know how much to offer workers to increase their engagement in their platforms. To account for this censored data problem we have introduced the Tobit model, an econometric model developed by James Tobin in 1958.</p><p>You can find the code to follow this article in this <a href="https://github.com/AlvaroCorrales/tobit">Github repo</a>.</p><p>Have you ever encountered a data set with a time stamp? Perhaps it was about a retailer that wants to predict demand over holidays. It could also have been from a telecommunications provider who wanted to forecast call volumes during major sporting events. Or, maybe, your data set was about an energy company that needed to predict electricity demand during extreme weather conditions.</p><p>Time series are all around us. If you want to learn how to work with them, consider taking my course An Introduction to Time Series, on Educative.</p><p><a href="https://www.educative.io/courses/an-introduction-to-time-series">An Introduction to Time Series - Learn Interactively</a></p><h3>References</h3><p>[1] Tobin, James (1958). <a href="http://cowles.yale.edu/sites/default/files/files/pub/d00/d0003-r.pdf">“Estimation of Relationships for Limited Dependent Variables”</a> (PDF). <em>Econometrica</em>. <strong>26</strong> (1): 24–36.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26b7b6e2c7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-work-with-censored-data-26b7b6e2c7f">How to work with censored data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Myth Behind Alexander the Great]]></title>
            <link>https://medium.com/@acorralescano/the-myth-behind-alexander-the-great-5d8ec9d920b8?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d8ec9d920b8</guid>
            <category><![CDATA[history]]></category>
            <category><![CDATA[war]]></category>
            <category><![CDATA[greece]]></category>
            <category><![CDATA[asia]]></category>
            <category><![CDATA[mythology]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Wed, 24 Nov 2021 09:02:49 GMT</pubDate>
            <atom:updated>2022-10-14T07:38:47.550Z</atom:updated>
            <content:encoded><![CDATA[<h4>Plutarch and the portents of the conquest of Asia</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zso0LdqkmBSDivnLKryIZQ.jpeg" /><figcaption>Alexander Cuts the Gordian Knot (1797), by Jean-Simon Berthélemy. Source: Wikimedia Commons</figcaption></figure><p>October 331 BCE - Alexander III of Macedon, popularly known as Alexander the Great, defeats the army of Darius III of Persia near the village of Gaugamela, in modern-day Iraq. This was the climax of Alexander’s campaigns. In Plutarch’s [1] words:</p><blockquote>After the battle had ended in this way, the authority of the Persian empire was regarded as having been completely overthrown. Alexander was proclaimed king of Asia […].</blockquote><p>In Gaugamela, 24-year-old Alexander achieved his lifelong purpose of subjugating the Persian empire. This battle was by no means an isolated event, but part of a series of military victories as king of Macedon between 336 and 323 BCE. The rest, as they say, is history: after conquering the territories of the Persians (which in that time included places like Egypt, Anatolia and Mesopotamia) he went on as far as modern Pakistan and the Hindu Kush. There, only his army’s mutiny forced him to turn back. Finally, in Babylon, at the age of 32, he died in strange circumstances.</p><p>The grandiosity of Alexander’s feats has inspired countless authors who have come up with various explanations for his glory. Plutarch, who lived some four centuries after Gaugamela, is just one of the many who wrote about Alexander in Antiquity. Writing so many years after Alexander’s time, it is only normal that Plutarch’s <em>Life of Alexander </em>is plagued with legends and impossibilities. His depiction of the great king is full little gestures, extravagant situations and, more importantly, colourful and bizarre omens. Yet, his account also stands out for its vivacity. Let’s dive into it.</p><h3>Born to be a hero</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8GwZpMH8xDUk-3MIkwGeww.jpeg" /><figcaption>Alexander the Great at Achilles’ Tomb, by Johann Heinrich Schönfeld. Source: Wikimedia Commons</figcaption></figure><p>Heroes are born, not made — at least in Greek mithology. The Illiad, Alexander’s (quite literally) bedside book, is full of great warriors of divine stock. The greatest of them is Achilles, son of the sea goddess Thetis. Achilles is the epitome of the Greek hero, and Alexander’s role model. Other mythological characters with semidivine origin included Heracles, son of Zeus, and Dyonisus, who was both the god of wine and parties, and a great conqueror of Asia.</p><p>We do not know whether Alexander thought that, <em>because </em>he was destined to rule the world, he had to be the son of a god, or the other way round. What we do know is that, however he came to that conclusion, he believed he was the son of Zeus. This was of course a tricky thing to say: his claim to the throne of Macedon came from his <em>human</em> father, Phillip. In any case, this did not prevent stories about his origins to arise. In the incredibly funny first chapters of the <em>Life of Alexander</em>, Plutarch tells us that Phillip, suspecting that his wife Olimpia may be cheating on him with a god, decided to spy on her. One night, he went to her chambers and peered through the half-open door. Zeus, conveniently disguised as a snake for the occasion, didn’t like this, and he made Phillip lose his eye.</p><h3>Destiny unfolds</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhxvcWCy2cnV-yGK_tgyTg.jpeg" /><figcaption>Alexander’s empire. Source: Wikimedia commons</figcaption></figure><p>Like many other kings, Alexander was prone to vanity. Upon his arrival in Egypt, he decided to travel to the temple of Zeus-Ammon in the Siwah oasis. A perilous journey through the desert, Alexander was sure that he could “overcome not only his enemies, but even places and seasons of the year” [ch. 26]. His motivation? He wanted the god’s confirmation that he would defeat the Persians and, since he was there, he’d solve once and for all the issue about his father. According to Plutarch, to greet Alexander in his native tongue, the priests of the temple addressed him as ‘<em>O, paidion</em>’ (Greek for ‘O, my son’). However, in the priests’ foreign accent, it sounded as ‘<em>O, pai Dios</em>’ (‘O, son of Zeus’), which was enough for Alexander to confirm his ascendence.</p><p>The interpretation of the Egyptian priests’ accent may sound a little forced. After all, one may think that you wouldn’t need to twist facts so much if you really were the son of a god and destined to rule the world. But Alexander (or rather, the Alexander from Plutarch’s sources) saw nothing wrong with this. In another very famous episode that confirmed Alexander’s destiny, the Macedonian managed to fullfil the prophecy of the Gordian Knot. The story goes that king Midas had dedicated to Zeus a chariot that belonged to his father. In doing so, he had tied it to a pilar in a temple in Gordium using a very intricate knot. The knot was so complex that, with time, a prophecy formed that whoever was able to untie it would rule over the whole world. After capturing the city of Gordium, Alexander decided to prove his worth against the knot. Unable to find an end to pull from, Alexander inpatiently drew his sword and cut the thread. See? Destiny.</p><h3>Restoring balance</h3><p>Even for the great heroes of old, defeat and death were eventually at the end of the road. Achilles was killed after conquering the city of Troy; Hercules was burnt alive before he acquired immortality. This was in fact only natural for the Greeks, a necessary counterbalance that reminded humans of their place in the universe: right below the gods. Boundless victory was their exclusive privilege. Perhaps even more important, a legend needs a tragic, early end to become timeless. And, as it was clear for everyone who cared to look at the signs, Alexander would have his.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TmB2uHA_ZlM-C-faisWD2Q.jpeg" /><figcaption>A coin celebrating Alexander’s campaign in India; Babylon, 322. Source: Wikimedia Commons</figcaption></figure><p>The invasion India, in summer 327 BC, marked a turning point in Alexander’s life. In chapter 57, Plutarch tells us of the various omens that foreboded the bittersweet outcome for the campaign. As the army was preparing to march, an ewe gave birth to a lamb with a strange substance over its head. Oddly, the substance ressembled the Persian king’s tiara and, even worse, it looked as if two testicles were hanging from either side. Alexander was appalled, as he interpreted that an unworthy successor may inherit his throne. A few days after setting off, his servants found a spring of olive oil while pitching Alexander’s tent. Although the king welcomed the omen, his diviners pointed that it forecast a glorious, but painful campaign: oil was only given to men by the gods after their hard labours.</p><p>Toilsome as it was, the Indian campaign brought some victories for Alexander. However, things started going South as he approached Babylon, the new capital of his empire, on his way back. The first portent happened at the gates of the city, when he saw a large group of ravens fighting one another. Shortly after, news came of another bad sign: Apollodorus, the governor of the city, had conducted a sacrifice in honour of the king, only to find that the victim’s liver had no lobe. Dark omens kept coming, until one day the king found a man silently wearing his regalia and sitting on his throne. Alexander had taken them off to go play a ball game with his friends. The man could not explain how he had got there, but Alexander had him executed nonetheless. This was not enough though, and confidence deserted the king. He became fearful and superstitious, and some fevers took his life shortly after. He was only 32 years old, but by then Alexander had conquered much of the world known to the Greeks.</p><h3>Conclusion</h3><p>Alexander the Great is an enormous figure and one of the greatest warriors in History. His achievements were so beyond imagination that some historians like Plutarch could only rationalise them through a lense of divine portents and omens. In doing so, they enlarged Alexander’s legend and effectively made him what they already thought he was: a semidivine hero.</p><h3>Sources</h3><ul><li>[1] Plutarch and Scott-Kilvert, I. (1973). <em>The age of Alexander</em>. Harmondsworth: Penguin Books.</li></ul><p>‌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d8ec9d920b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Assyrian or Babylonian? Language Identification in Cuneiform Texts]]></title>
            <link>https://medium.com/data-science/assyrian-or-babylonian-language-identification-in-cuneiform-texts-4f15a14a5d70?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f15a14a5d70</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linguistics]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Fri, 09 Jul 2021 04:14:02 GMT</pubDate>
            <atom:updated>2023-12-05T05:10:53.394Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>A language model for ancient Mesopotamian dialects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LfPq8VGwuPMA4e5vxaxvOQ.jpeg" /><figcaption>Clay tablet with cuneiform characters. Source: <a href="https://commons.wikimedia.org/wiki/File:Cuneiform_tablet-MAHG_15856-IMG_9472-black.jpg">Rama</a>, Wikimedia Commons</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Mesopotamia">Mesopotamia</a> is an ancient denomination for the region between the Euphrates and Tigris rivers, in what today is Irak and parts of Turkey, Syria, and Iran. Modern societies owe much of their “modernity” to Mesopotamian inventions in fields as diverse as agriculture, math or metallurgy. Among those inventions, one of the greatest achievements of ancient Mesopotamians was to develop the <a href="https://en.wikipedia.org/wiki/Cuneiform"><em>cuneiform </em>script</a>, and with it one of the first writing systems. It is the 3rd millennium B.C., and History has just started.</p><p>The cuneiform writing system consisted of wedge-shaped signs printed into fresh clay tablets. Clay is a material that can resist the passage of time without significant deterioration. Therefore, archeologists have managed to find large sums of these tablets in a very good state of conservation. Clay is also very cheap and abundant in the region. This made this form of scripture very popular, and different peoples adapted the script to their languages for a long period of time. As a consequence, we have a pretty good record of how (written) language evolved in Mesopotamia for thousands of years. In fact, we have so many records spanning such a long period, that they are almost unmanageable. And here is where Data Science comes to the rescue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zoH-Rz1jwrm1cbGoNsD8w.png" /><figcaption>Map of Ancient Mesopotamia. Source: <a href="https://commons.wikimedia.org/wiki/File:N-Mesopotamia_and_Syria_english.svg">Goran tek-en</a>, Wikimedia Commons</figcaption></figure><p>The purpose of this article is to show how we can write a simple custom language model to identify Mesopotamian languages and dialects in cuneiform texts. To do so, we use the dataset from the Kaggle data challenge <a href="https://www.kaggle.com/wilstrup/cuneiform-language-identification"><em>Cuneiform Language Identification</em></a><em>. </em>This dataset contains ~139,000 snippets of cuneiform texts, for which a language or dialect tag is also provided. There are seven tags, including <a href="https://en.wikipedia.org/wiki/Sumerian_language">Summerian</a> (SUX), an isolated language, and six dialects of <a href="https://en.wikipedia.org/wiki/Akkadian_language">Akkadian</a>: Neo-Assyrian (NEA), Standard Babylonian (STB), Late Babylonian (LTB), Neo-Babylonian (NEB), Middle Babylonian Peripheral (MPB), and Old Babylonian (OLB).</p><p>In case you were wondering how these characters look, the image below shows a sample of our dataset. Beautiful, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rq9ZXuEU0S5UaVQ2Ufo-3w.png" /><figcaption>A subset of our data. Source: Author’s analysis of Kaggle <a href="https://www.kaggle.com/wilstrup/cuneiform-language-identification">dataset</a></figcaption></figure><p>Going forward, if you want to replicate this analysis on your local machine, I suggest that you download <a href="http://oracc.museum.upenn.edu/doc/help/visitingoracc/fonts/">one of these fonts</a>. While this will not affect your ability to replicate the analysis, the texts will fail to render properly otherwise.</p><h3>Language Models</h3><p>Generally speaking, a <a href="https://en.wikipedia.org/wiki/Language_model">language model</a> is a probability distribution over sequences of words. Within this broad definition, we find different types of models. Some of the most common use the concept of <em>n-gram</em>, i.e. sequences of n consecutive characters (words) in which we split the text. N-gram models are fairly common in NLP literature. Their applications include speech and handwriting recognition and machine translation. Consequently, there are some well-known packages in Python, such as <a href="http://www.nltk.org/">NLTK</a>, with which we can easily construct an N-gram model.</p><p>For this task, I use an N-gram model to predict the probability that a sequence of cuneiform n-grams belongs to a particular Ancient Mesopotamian language or dialect. In this implementation, we assume that each character is a unigram. Technically, a cuneiform character could be either a whole word or just a sound, but we will ignore this fact. Now, as you may have already guessed, I wouldn’t be writing this article if I had followed the easy path for this challenge. On the contrary, here I show you how to construct an N-gram model <em>from scratch</em>. In fact, the only Python libraries that I use in my code are numpy and pandas. There are two reasons why I decided to do it this way: first, standard NLP libraries are designed to work with Latin characters, so adapting them to cuneiform text is a bit of a pain. Second, it&#39;s fun! I am a firm believer that the best way of testing your knowledge on a subject is to go to the basics and see if you can code it with as few specialised libraries as possible.</p><p>The N-gram model implementation that I have come up with is inspired in <a href="https://towardsdatascience.com/the-beginners-guide-to-language-models-aa47165b57f9">classical N-gram models</a> that use the <a href="https://en.wikipedia.org/wiki/Chain_rule_(probability)">Chain Rule of Probability</a> and the <a href="https://en.wikipedia.org/wiki/Markov_property">Markov assumption</a>. Simply put, this assumption states that the probability of an n-gram in position <em>t</em> depends only on the previous <em>k </em>n-grams (where <em>k</em> is usually 1). Hence, the probability of a sequence can be computed by taking the product of conditional probabilities of its n-grams up to <em>t-k</em>.</p><p>Well, my code does not <em>exactly</em> do that, but something close and simpler. What it actually does is to compute the unconditional probability of all n-grams in a training set of each of the languages. It then estimates the probability that an observed sequence belong to each of the languages by multiplying the unconditional probabilities of all n-grams that appear in the sequence. The language with the highest probability is chosen as the predicted label.</p><p>Obviously, more complex methods can be developed. However, I have found that this very simple model already performs really well on a test set.</p><h3>Implementation</h3><p>Now that we have a better understanding of language models, let’s see how we can code our custom implementation for cuneiform texts. This section is a bit heavy in terms of code, but bear with me: the effort will be worth it! Alternatively, you can move on to the results section to know the end of the story.</p><p>We start with some helper functions. The first one, which we call preprocess , transforms the strings of cuneiform text into a more convenient format for us. It performs two actions: first, it adds a beginning- and an end-of-sentence token (which we call B and E, respectively), and then it adds a space between characters. B and E are useful tokens: they serve us to calculate the probability that a sentence begins and ends with a given character.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ee4b4e690f2ff5241c5f186f994eace/href">https://medium.com/media/8ee4b4e690f2ff5241c5f186f994eace/href</a></iframe><p>Now, we need to count n-grams! The two functions below are the core of our implementation, since they split the text in n-grams and count how many times each appears in our sample.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16c86acabf935b1f326c803ccfad77a1/href">https://medium.com/media/16c86acabf935b1f326c803ccfad77a1/href</a></iframe><p>The last of our helper functions, logprob, will be used to calculate the log probability of a whole sequence of n-grams. In this type of problems, we usually deal with very small probabilities, so taking logarithms is usually a good idea to keep track of our numbers. Here we have another simplification with respect to classical language models. Normally, an N-gram model would include an “unknown” word token in its vocabulary to estimate the probability of finding really unusual words. In our case, however, we will simply assume that the probability of finding such character is 0.00001 (about ten times lower than some of our rarest characters).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9565cbca0f89dc320f45e34b1a13b78/href">https://medium.com/media/d9565cbca0f89dc320f45e34b1a13b78/href</a></iframe><p>And that’s it! Those are all the helper functions that we are going to need. Now it’s turn to define a class to wrap our model, which you can see in the snippet below. For lack of a better name, let’s call it CuneiPy . Our wrapper class follows the classic Scikit-Learn structure, and contains two methods: .fit() and .predict() . The .fit() method calculates the unconditional probability for each n-gram and language (remember, we have seven). This is simply the number of occurrences of the n-gram over the total number of n-grams. The .predict() method takes a sequence of cuneiform characters and calculates a probability that the sequence belongs to one of our languages. Then, it takes the language with the highest probability and chooses it as the predicted label.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/446548c86240a201c34f500e020824d6/href">https://medium.com/media/446548c86240a201c34f500e020824d6/href</a></iframe><h3>Results</h3><p>If you are still with me at this stage, you may be wondering if this thing works at all. And you may already guess the answer: it does! While there is definitely room for improvement, our simple model performs really well on a test set of 1000 observations, with a weighted F1 score of ~0.8 (depending on the train-test split).</p><p>Testing CuneiPy is as simple as this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9854ae318121bc0eaf0423815890ac5e/href">https://medium.com/media/9854ae318121bc0eaf0423815890ac5e/href</a></iframe><p>As we can see in the .fit() part, we are using a bigram model (hence<em> n = 2</em>). The column “cuneiform” is the input of our language model, and our target column is defined by “lang” in the dataset. The classification report below gives us a granular breakdown of where the model performs best. Perhaps not surprisingly, the model is better at identifying Neo-Assyrian (NEA) and Summerian (SUX), two of the most common languages. The model is also good at identifying Late Babylonian (LTB), although only 11% of our the texts belong to this dialect of Akkadian. Middle Babylonian Peripheral (MPB) is the dialect with which our model struggles the most. This is consistent with how few representations of this language we have in the sample (3.9% of all observations).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcyEQCOEtXceWunkBvtkig.png" /><figcaption>Classification report. Source: Author’s analysis of Kaggle <a href="https://www.kaggle.com/wilstrup/cuneiform-language-identification">dataset</a></figcaption></figure><h3>Main takeaway</h3><p>In this article we have seen how to create a custom N-gram language model to predict the dialect or language of a cuneiform text from Ancient Mesopotamia. The model is a version of classical N-gram models that exploit the Chain Rule of Probability and the Markov assumption. Although simple, the model achieves an F1 score of 0.8.</p><p>I hope you enjoyed the article. If you liked it, you can access the code in this <a href="https://github.com/AlvaroCorrales/Cuneiform-classification">Github repository</a>.</p><p>For any feedback, please contact me on <a href="https://www.linkedin.com/in/alvaro-corrales-cano/">LinkedIn</a> or via email!</p><p>Have you ever encountered a data set with a time stamp? Perhaps it was about a retailer that wants to predict demand over holidays. It could also have been from a telecommunications provider who wanted to forecast call volumes during major sporting events. Or, maybe, your data set was about an energy company that needed to predict electricity demand during extreme weather conditions.</p><p>Time series are all around us. If you want to learn how to work with them, consider taking my course An Introduction to Time Series, on Educative.</p><p><a href="https://www.educative.io/courses/an-introduction-to-time-series">An Introduction to Time Series - Learn Interactively</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f15a14a5d70" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/assyrian-or-babylonian-language-identification-in-cuneiform-texts-4f15a14a5d70">Assyrian or Babylonian? Language Identification in Cuneiform Texts</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AdaBoost from Scratch]]></title>
            <link>https://medium.com/data-science/adaboost-from-scratch-37a936da3d50?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/37a936da3d50</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Tue, 27 Apr 2021 11:57:37 GMT</pubDate>
            <atom:updated>2023-12-05T05:05:02.219Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build your Python implementation of one of the most popular “off-the-shelf” algorithms in Data Science</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKyfMtQvDoBwJXdwCmSMXw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@soymeraki?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Javier Allegue Barros</a> on <a href="https://unsplash.com/s/photos/decision?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>A colleague once told me that you don’t really understand an algorithm until you can write it on NumPy from scratch. The claim may be bold, but there is still something beautiful in opening a text book or a paper on machine learning and trying to translate the math into code. That was my whole motivation when I started doing the work behind this article, where I describe how to build your own implementation of the AdaBoost algorithm. Our version of the algorithm comes from <em>The Elements of Statistical Learning </em>(TESL), chapter 10 [1].</p><p>AdaBoost, short for <strong>Ada</strong>ptive <strong>Boost</strong>ing, has become one of the most popular “off-the-shelf” algorithms in Data Science. It’s been part of winning solutions in Kaggle competitions (sometimes in combination with its younger cousins, XGBoost and Light GBM), and its now even part of face detection problems. It was introduced by Freund and Shapire in 1996 [2].</p><p>A small warning: the purpose of this article is not to derive and explain the math of AdaBoost in detail — there are many good posts like that already. On the contrary, this article is about how an implementation of the algorithm works under the hood. It is intended for people that want to have an idea of what ready-made implementations of AdaBoost do, such as the one on <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html">Scikit-learn</a> for Python or <a href="https://cran.r-project.org/web/packages/adabag/adabag.pdf">Adabag</a> for R. Ultimately, I hope it will also inspire you to build your implementations of other algorithms to learn more about them!</p><p>For the rest of the article, I will be following the notation and formulas in Chapter 10 of TESL. We will be implementing AdaBoost.M1 algorithm, called “Discrete AdaBoost” in Friedman et al. (2000) [3], and Algorithm 10.1 in TESL. All code examples are in Python. You can find the code in <a href="https://github.com/AlvaroCorrales/AdaBoost">this Github repository</a>.</p><h3>An overview of Boosting</h3><p>The AdaBoost algorithm is based on the concept of “boosting”. The idea behind boosting is that a set of “weak” classifiers can make up to a robust classifier using a voting mechanism. A weak classifier is one that will only yield slightly better results than tossing a (fair) coin. In other words, if by randomly guessing a binary label we would be right about 50% of times, a weak classifier would be right, say, 55% (or any other number close to 50%).</p><p>With boosting, we train a sequence of weak classifiers on successive modified versions of our sample. Let’s put an example to see how this works. Say that we have a sample with five observations. Each observation is labelled either -1 or 1, such that our labels are <em>y = [1, 1, -1, -1, -1]</em>. We also have some explanatory variables <em>x.</em> In the first boosting round, we train a weak classifier <em>G₁(x)</em>, that yields the following prediction: <em>G₁(x) = [1, -1, -1, 1, -1]. </em>We can see that the second and fourth observations have been misclassified. In the second round of boosting, we will weight each observation in our sample using some weights <em>wᵢ. </em>We will set <em>wᵢ </em>so that the two misclassified observations have a larger impact in the learning process of a new classifier <em>G</em>₂<em>(x) </em>than the other three. The new classifier may miss some of the previously well-classified observations now, but it will be better at predicting the difficult ones.</p><p>After repeating the process described above M times, we will get a set of M weak classifiers that we can combine in a final, robust meta-classifier <em>G(x). </em>This meta-classifier will allocate a prediction for each observation based on a weighted majority vote, as shown in the formula below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*7H2nn8GFReMGjPZlNXlq_A.png" /><figcaption>Our final classifier. Source: The Elements of Statistical Learning, ch. 10</figcaption></figure><p>See that the <strong>weight</strong> of each weak <strong>classifier</strong> in the majority vote is denoted by <em>α</em>. These are not the weights that we apply to each <strong>observation</strong> in each boosting round, which we have called <em>wᵢ </em>(I know, so many weights!).</p><h3>Weak classifiers</h3><p>Remember that we loosely defined a weak classifier as one that would only yield slightly better results than random guessing. A more formal version of this definition would be that the error rate of the classifier is less than but close to 50%. In this case, the in-sample error rate would simply be the number of misclassified observations (i.e. <em>yᵢ </em>≠ <em>G(xᵢ)</em>) over the total sample size, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*mCMMoeNApKcnYqT_eesocg.png" /><figcaption>In-sample error rate. Source: The Elements of Statistical Learning, ch. 10</figcaption></figure><p>Many algorithms could qualify as weak classifiers but, in the case of AdaBoost, we typically use “stumps”; that is, decision trees consisting of just two terminal nodes. Intuitively, in a binary classification problem a stump will try to divide the sample with just one cut across the one of the multiple explanatory variables of the dataset. As you can imagine, very often this is not enough to separte classes in a neat weay, especially in complex datasets.</p><p>Now, if we trained a decision tree classifier for 1,000 times on the same data, we would always get the same cut (for a given random seed, if you are using an approximation heuristic). When drawing the line, the decision tree is minimizing a metric that’s based on the distribution your target values, usually the Gini coefficient or information gain from the resulting categories. And here is where our observation weights, <em>wᵢ</em>, kick in. In every boosting interation, we tell our stump to weigh each observation differently when estimating the metric that it is minimizing. That way, in each iteration we get a different decision tree.</p><p>We don’t go into details about decision trees in this article (in fact, I use the Scikit-learn implementation in my algorithm), but if you want to learn more about them, I encourage you to read chapters 9, 10 and 15 of TESL.</p><h3>Writing our algorithm</h3><p>Time to recap. To solve a binary classification problem, our AdaBoost algorithm will fit a sequence of weak classifiers (stumps) through a series of boosting iterations. These classifiers will form a meta-classifier that will yield a prediction based on a weighted majority vote mechanism. In each boosting iteration, we will give more weight to those observations that were misclassified in the previous iteration. We can formalise this process as shown in the algorithm below. One can show that the formulas for <em>α </em>and <em>w </em>come from minimizing the exponential loss function. The explanation is a bit lenghty and out of the scope of this article, but you can find it in 10.4 of TESL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TcrnpBGsi5MsFog-fSdRMQ.png" /><figcaption>Source: The Elements of Statistical Learning, ch. 10</figcaption></figure><p>We are going to start our implementation by identifying independent chunks of the process that we can code outside of the loop in step 2. Steps 2(b), 2(c) and 2(d) contain operations that are repeated with every iteration, so we are going to define them as Python functions. Let’s see how.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6d77c334382483800fa21dd4f0e9c1c/href">https://medium.com/media/c6d77c334382483800fa21dd4f0e9c1c/href</a></iframe><p>Note that these functions are independent of the algorithm. One could calculate the error rate of any classifier with the <em>compute_error</em> function. Similarly, we could estimate <em>α </em>and <em>w </em>for other contexts.</p><p>It is time to define our AdaBoost classifier. To do so, we are going to construct a Python class called <em>AdaBoost </em>that will be composed of several methods to fit the model and make predictions. It will also contain other relevant information about our fitted model, such as the sequence of weak classifiers and their weights in the final vote, as well as training and prediction errors. The code snippet below shows the initialisation and fit method of our class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2551c5cd9ed6cd528b70252342195ea1/href">https://medium.com/media/2551c5cd9ed6cd528b70252342195ea1/href</a></iframe><p>The .<em>fit()</em> method has a very similar structure to the typical Scikit-learn class. As necessary arguments, it takes a matrix of independent variables <em>X </em>in the form of an array-like object, and a vector with the target variable, <em>y</em>. The target variable has to be binary, and it has to be encoded so that its values are either -1 or 1 (this makes voting easier). The boosting round paramter is optional, with 100 being the default.</p><p>Steps 1 and 2 of the AdaBoost.M1 algorithm are characterised within the <em>for</em> loop. The <em>if</em> statement at the begining initialises the weight for each observation to 1/N in the first round (step 1), and then it updates them after the stump has been fitted and the <em>α </em>for that iteration has been calculated (step 2(d)). The output of this method is a set of stumps, alphas and training errors that we store for further use in the predict method. Step 3 of the algorithm is defined in a separate method, which we call <em>.predict().</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa2b396c7a8acdd9be711bbe77e892d0/href">https://medium.com/media/aa2b396c7a8acdd9be711bbe77e892d0/href</a></iframe><p>Similar to the .<em>fit()</em> method, <em>.predict() </em>takes as argument an array-like matrix of explanatory variables. It then calculates a prediction for each observation and stump. These are stored in a dataframe which will be used in the majority-vote part of the process to calculate the final prediction for each observation.</p><h3>Testing our algorithm</h3><p>We have now defined our implementation of AdaBoost, but how do we know it works? To find out, we are going to test it on the Spambase Dataset, a collection of 4,601 emails labeled based on whether or not they are spam. The independent variables consist of term and character frequencies in each email. This dataset is available at the University of California, Irvine’s public repository, and you can download it <a href="http://archive.ics.uci.edu/ml/datasets/Spambase">here</a> [4]. It is also used across several chapters of TESL, including the one about AdaBoost, so we can benchmark our implementation against other, validated algorithms.</p><p>Just like in TESL, and to ensure that our training sets are as similar as possible, we will not apply any cleaning or feature engineering to the explanatory variables. TESL doesn’t state its random seed, so we can’t replicate the train/test split. However, we can still use the same set sizes so that at least both algorithms are trained on the same number of observations. The code snippet below shows how we read and split the data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff5a395f12dbe41853f81b317aee2f00/href">https://medium.com/media/ff5a395f12dbe41853f81b317aee2f00/href</a></iframe><p>Our AdaBoost class is very simple to implement from here. We just need to pass the training set to the <em>.fit() </em>method and the test set to the <em>.predict() </em>method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46a3ab2780f044a97c9deec6cc12bfc0/href">https://medium.com/media/46a3ab2780f044a97c9deec6cc12bfc0/href</a></iframe><p>And that’s it! If you try the steps above in your machine, you’ll find that you get an error rate of 5.6%. By comparison, in chapter 10.8 of TESL, they get an error rate of 4.5% using gradient boosting, 5.5% using additive logistic regression, 8.7% with a CART tree and 5.5% using MARS. Note that we are not using the same sample exactly, only the same number of observations. But not bad, right?</p><p>As a second test, you can also compare this custom implementation with Scikit-learn’s. The set of predictions we produced above yield a ROC-AUC score of 93.99%. Using Scikit-learn on the same sets, I got 92.79%. The numbers may vary slightly because of random seeds and approximation heuristics, but it’s really close!</p><h3>Back to stumps</h3><p>Remember that AdaBoost fits a set of weak classifiers. To see what this means in practice, let us visualise the error rates for each of the stumps we trained above. In the chart below, we can see that most of the stumps present error rates very close but below 0.5 on the training set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5d0Btlr8AL0V3gD871ZoEQ.png" /><figcaption>Source: Author’s creation</figcaption></figure><p>A well-known property of decision trees is that they tend to overfit the training data. Our stumps are no exception here, as the chart below shows. In fact, many of them are even worse than random guessing. And yet, their combination only gets it wrong 5.6% of times!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fvu-Y6QTWc9aCzvrOQ5WNA.png" /></figure><h3>Main takeaway</h3><p>The goal of this article was to give an idea of how to construct a custom implementation of the AdaBoost classification algorithm in Python. To do that, we followed chapter 10 of <em>The Elements of Statistical Learning. </em>Along the way, we have also given a high-level overview of boosting, a technique that combines weak learners into a robust, meta-algorithm.</p><p>I hope this was useful and it inspired you to build your own custom implementation. It is not only a fun way of learning, but also quite effective! As Nassim N. Taleb puts it: “What I learned on my own I still remember” [5].</p><p>Liked it? You can access the code in this <a href="https://github.com/AlvaroCorrales/AdaBoost">Github repository</a>.</p><p>Have you ever encountered a data set with a time stamp? Perhaps it was about a retailer that wants to predict demand over holidays. It could also have been from a telecommunications provider who wanted to forecast call volumes during major sporting events. Or, maybe, your data set was about an energy company that needed to predict electricity demand during extreme weather conditions.</p><p>Time series are all around us. If you want to learn how to work with them, consider taking my course An Introduction to Time Series, on Educative.</p><p><a href="https://www.educative.io/courses/an-introduction-to-time-series">An Introduction to Time Series - Learn Interactively</a></p><h3>References</h3><p>[1] Hastie, T. et al. (2009), <em>The Elements of Statistical Learning, Second Edition, </em>DOI 10.1007/b94608_10, Springer Science+Business Media, LLC.</p><p>[2] Freund, Y. and Schapire, R. (1996), <em>Experiments with a New Boosting Algorithm, </em>Machine Learning: Proceedings of the Thirteenth International Conference, available at: <a href="http://rob.schapire.net/papers/FreundSc96.pdf">http://rob.schapire.net/papers/FreundSc96.pdf</a></p><p>[3] Friedman J, Hastie T, Tibshirani R (2000). <em>Additive Logistic Regression: A Statistical Viewof Boosting.</em> The Annals of Statistics,28(2), 337–407.</p><p>[4] Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.</p><p>[5] Nassim Nicholas Taleb (2016), The Bed of Procrustes: Philosophical and Practical Aphorisms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37a936da3d50" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/adaboost-from-scratch-37a936da3d50">AdaBoost from Scratch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying Bayesian Networks to Covid-19 diagnosis]]></title>
            <link>https://medium.com/data-science/applying-bayesian-networks-to-covid-19-diagnosis-73b06c3418d8?source=rss-639157103f84------2</link>
            <guid isPermaLink="false">https://medium.com/p/73b06c3418d8</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alvaro Corrales Cano]]></dc:creator>
            <pubDate>Wed, 24 Feb 2021 13:26:40 GMT</pubDate>
            <atom:updated>2021-03-03T20:52:55.668Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h3>Applying Bayesian Networks to Covid-19 Diagnosis</h3><h4>Probabilistic decision making in highly complex scenarios</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GvBvTm4lm2fPLEpj6-b1tg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@marceloleal80?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Marcelo Leal</a> on <a href="https://unsplash.com/s/photos/doctor?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Motivation</h3><p>It is not news that the Covid-19 pandemic has put under strain parts of our societies that we had for long taken for granted. Healthcare services is perhaps one of the clearest examples. Our healthcare professionals have faced a virus that they knew very little about, having to make decisions with too often scarce information and limited resources. In this context, the Data and AI community has stepped in to contribute to the fight against Covid-19 by leveraging what they are best at: processing data to make informed decissions. This has translated into <a href="https://developer.ibm.com/callforcode/">calls for code</a>, <a href="https://www.kaggle.com/roche-data-science-coalition/uncover/tasks">Kaggle challenges</a> and even not-for-profit corporate <a href="https://emergentalliance.org/">alliances</a>.</p><p>Inspired by this collective effort, in these lines we show how Bayesian Networks can be useful for decision makers to make sense out of complex information using a probabilistic approach. In particular, after giving a brief introduction to the concept of Bayesian Networks, we show how to apply this concept to a real world dataset from the Kaggle challenge <a href="https://www.kaggle.com/einsteindata4u/covid19"><em>Diagnosis of Covid-19 and its clinical spectrum</em></a><em>. </em>This dataset contains anonymised medical data from patients that were tested for Covid-19 at the Hospital Israelita Albert Einstein in São Paulo, Brazil, the sponsor of the challenge.</p><p>In the organiser’s own words, the motivation for the challenge was <em>the fact that in the context of an overwhelmed health system with the possible limitation to perform tests for the detection of SARS-CoV-2, testing every case would be impractical and tests results could be delayed even if only a target subpopulation would be tested.</em> The structured knowledge that Bayesian Networks provide is very well suited for this type of situation. As we show in this blog, with Bayesian Networks complex decision making becomes much easier.</p><h3>An introduction to Bayesian Networks</h3><p>Bayesian Networks (BN) are a well-established technique for handling uncertainty within the AI community, to the point that some consider them a capstone for modern AI. As professor Stuart Russell puts it, <em>“BN are as important to AI and ML as Boolean circuits are to computer science.” </em>While one could argue about the extent of this quote, it is clear that BNs are quite popular in the AI community. But, what are BNs exactly?</p><p>BN are a class of Probabilistic Graphical Models renowned for their capabilities to reason under uncertainty. They can be seen as a probabilistic expert system: the domain (business) knowledge is modelled as a direct acyclic graph (DAG). The DAG links, or arcs, represent the probabilistic dependencies between nodes/variables in the domain. These dependencies don’t need to be causal, but can represent a form of correlation or dependency. To put it more formally, a BN is a Joint Probability Distribution (JPD) over a set of random variables. It is represented by a DAG, where the nodes are identifed with random variables and the arcs express the probabilistic dependence between them.</p><p>BNs have been successfully implemented in different industries. Some of their many applications include risk assessment (cancer, water, nuclear safety …), industrial process simulation, monitoring the health of machines (troubleshouting, defect, failure etc ..) and predictive maintenance.</p><p>More generally, BNs can be used as a tool for diagnosis, prediction (aka prognosis) or probable explanations of an observation. In a diagnosis application, we could perform reasoning from consequences to causes following the graph (usually the opposite direction of the arcs in the BN). For example, in our dataset, observations about blood cell counts can be used to update the expert’s belief about the need of admission to ICU. Once we have the BN, we can also start reasoning under uncertainty or, in other words, making <em>what-if</em> type of questions through probabilistic inference. This consists of computing a posterior such as P(Covid-19 = positive| Platelets = 3).</p><p>Going back to or initial motivation, let’s consider the following thought experiment. Take an Intensive Care Unit (ICU) of a hospital during the current pandemic. Healthcare professionals need to have an idea of whether a person will require intensive care in the immediate future to better allocate their resources in moments of great strain. The admission to ICU can be due to Covid-19 or not. Some bioindicators are currently associated with a higher probability of having Covid-19, and can be relatively easily identified through standard tests (see for example this <a href="https://www.researchgate.net/publication/339627510_Laboratory_abnormalities_in_patients_with_COVID-2019_infection">paper</a> by Lippi and Plebani [2020]). Similarly, other indicators not necessarily linked to Covid-19 can also be related to other commorbidities that increase the probability of being admitted to ICU. Such a situation can be modeled using a BN, whose DAG structure is shown in the image below (the image is purely illustrative):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yuwoyra3l3V2TaDA3CvWpQ.png" /><figcaption>Illustrative DAG — Image by the authors</figcaption></figure><p>The BN associates a conditional probability table (CPT) to each node given its parents in the DAG. In our example, the node in the top layer, <em>Bordetella pertussis,</em> doesn’t have any parents; <em>Adenovirus’</em>s parent node is <em>Bordetella pertussis</em>, while its children are <em>CoronavirusNL63 </em>and <em>Strepto A</em>; and in turn <em>Strepto A </em>would not have any children. In this formalism, we consider variables to be discrete. The values of each variable are actually the possible modalities that this variable can take (state) such as 0 or 1 for <em>covid_19</em>.</p><p>What’s more importnat, the BN factorizes the JPD <strong>𝑃</strong> as the product of these CPTs. For the network depicted above, our JPD would be:</p><p><em>P (Bordetella pertussis, Adenovirus, CoronavirusNL63, Coronavirus HKU1, … , Parainfluenza 1, Chlamydophilia pneumoniae) = P (Bordetella pertussis) </em>x <em>P (Adenovirus | Bordetella pertussis)</em> x <em>P (CoronavirusNL63 | Adenovirus, Bordetella pertussis) </em>x<em> P (Coronavirus HKU1 | CoronavirusNL63, Bordetella pertussis) </em>x … x <em>P (Parainfluenza 1 | Bordetella pertussis, Rhinovirus/Entenovirus) </em>x<em> P (Chlamydophilia pneumoniae | Parainfluenza 1, Bordetella pertussis)</em></p><p>It is possible to obtain these CPTs in two different ways: we can either use a frequentist approach by learning a structure from a historical dataset, or we can elicit the CPTs from our domain knowledge. Oftentimes, the most powerful approach will be a combination of both. This is an especially useful characteristic of BN: in practice, we would first want see what the data has to say, so we would use the frequentist approach to learn a structure from it. This will almost always give us new insights on our data and even challenge our previous assumptions about it. Once we have our network, we can apply our domain knowledge to confirm or reject what we have computed via the frequentist approach if we know that certain links have to be there, or some are just impossible. For instance, in healthcare context, we may want to impose a structure such that admission to hospital is the <em>consequence</em> rather than the <em>cause</em> of an illness.</p><p>Another important characteristic of BN is the memory gains that we get from them when making probabilistic reasoning. This comes quite handy when dealing with wide datasets. In the case of our 111-column wide dataset, we would need to store 6347497291776 entries to store the JPD, which is the product of all modality sizes in the JPD. Using the BN, we can encode this compactly as we saw in the factorisation above, meaning that we now we only need to store 817 entries. This is a memory gain of 99.99999998712879%!</p><p>The graphical structure also encodes very interesting information that can be used to derive insights about the data: every node is conditionally independent of its non-descendants given its parents in the DAG. Based on this conditional independence, one can also show that a node is conditionally independent of all nodes given its Markov blanket, i.e., the union of the parents of X, of the children of X and of these children’s parents. In other words, all we need to know to update our belief about a variable is its Markov blanket.</p><p>For instance, in our example dataset, to update our knowledge on the probability that a patient will test positive for Covid-19, we would only need information on the variables <em>Inf A H1N1 2009, Influenza B, Platelets, Coronavirus HKU1, Respiratory Syncytal Virus, regular_ward, </em>and <em>Rhinovirus/Enterovirus.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*MJrhPTynwFMTdNaUJGansw.png" /><figcaption>Markov blanket of covid_19 variable — Image by the authors</figcaption></figure><p>BN have also some caveats. Their power relies on their modeling of probabilistic interaction in complex systems, where human reasoning is weak. BN are mathematically sound and are grounded in theory to have a normative approach to deal with uncertainty. However, they are not particularly powerful tools for classfication tasks in general, although they can used as classifier. This is due to the loss of information we impose by working with states, i.e. discrete variables. There is ongoing research to find more effective ways of discretising continous variables to use them in BN, but the area is relatively new and is outside the scope of this blog.</p><h3>Application to a real-world dataset</h3><p>Both the structure and the parameters (CPT) of a BN can be learned from a dataset. In our case, the Hospital Israelita’s dataset is composed of 5644 observations, each corresponding to a patient that was tested for Covid-19. Alongside the results of the test, we’ve got information on whether or not the person was admitted to a regular ward of the hospital, semi-intensive or intensive care, as well as the results of various other health tests. Note that in this application, we will think of our dataset as a set of random variables with probabilistic interactions.</p><p>The structure of the dataset reflects the reality of a hospital: no two patients are the same, so doctors often need to prescribe different types of tests for each patient upon admission. Hence, our dataset contains a big amount of missing values — in some columns as much 99% of the total. There are several, well-known techniques in the field of Data Science to deal with missing values while keeping as much information as possible. On top of the classic mean/median/mode imputation techniques, clustering and PCA algorithms can be used to fill these missing values when working with BNs. Since the purpose of our blog is mainly educational and we want to keep it as simple and clear as possible, we simply drop all columns with more than 95% missing values. Out of the remaining variables, discrete variables will be filled with the value -999, while continuous variables will be filled with the median. This is motivated by the categorical nature of BNs: filling discrete variables with -999 effectively creates another state, leaving the others untouched. Given that all our continuous varaibles have zero mean and standard deviation of one, flling them with the median ensures that we are still able to capture the variation from the actual values when we discretise them.</p><p>As we’ve said before, our BN works better with discrete variables, so we need to discretise our continous variables. For the purpose of our analysis, we have split the data into buckets based on the belonging to a percentile bucket. We take four asymmetrically bounded buckets. Given our amount of missings is substantial and that we filled values with the median, we are interested mostly in the variations in the extremes of the distribution. Therefore, our first bucket is bounded by percentiles 0 and 5, whereas our last bucket is bounded by percentiles 95 and 100. Everything in between split by percentile 50, forming our two middle buckets.</p><p>An now… we are ready to estimate our first BN! As an initial attempt, if we try to learn the structure only from the data without any business or domain knowledge we get the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7I0Z-4zXG0tUwBUqaj5aA.png" /><figcaption>Learned DAG with no business domain constraints — Image by the authors</figcaption></figure><p>The learned structure may contain some relation that doesn’t make sense from a domain knowledge point of view. The BN framework is flexible to allow a domain expert to intervene to introduce the human knwoledge in the learning process. Additionally, thinking in terms of prediction, we may want to enforce certain arcs from some predictive variables to our target variable(s).<strong> </strong>After imposing some further constraints, we will get a slightly different BN, as shown in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqzSVLNy0CTQkizRV7tssA.png" /><figcaption>DAG after including business domain constraints — Image by the authors</figcaption></figure><p>Once we have fine-tuned the structure of our BN with our domain knowledge, we can start quering it about the learned CPT. Let’s have a look at the CPTs for our regular ward admission indicator. For this particular structure (not necessarily the optimal), the CPT is only determined by the Covid-19 and Influenza B diagnoses. In terms of interpretation, we could say that a patient that tested negative for both Covid-19 and Influenza B has only a probability of 2.95% of needing admission in the regular ward of the hospital.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*61GW5PUAQ1DqAKQLpd74LQ.png" /><figcaption>CPT of regular_ward variable — Image by the authors</figcaption></figure><p>Remember that our BN allows us to ask about <em>what-if </em>scenarios. For instance, we can ask <em>What’s the probability that a patient will test positive for Covid-19 given that she presents these levels of indicators A and B? </em>The answer to this question can be found in the posterior for the variable <em>covid_19</em>. Let’s evaluate this posterior for a patient that has been detected Influenza B but not Coronavirus HKU1 and she is in bucket number 1 of the Platelets variable. The probability that the patient has Covid-19 is as high as 52.89%:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*_5MpbiSVOEFcli9tvUv0ew.png" /><figcaption>Posterior probability of having Covid-19 for a patient with Influenza B but without Coronavirus HKU1 — Image by the authors</figcaption></figure><p>It’s worth noting that the probabilistic engine used only nodes from the Markov blanket to derive such information. To verify this, let’s play again with the Markov blanket of the variable <em>covid_19. </em>As we showed above, the corresponding Markov blanket was composed of the variables <em>Inf A H1N1 2009, Influenza B, Platelets, Coronavirus HKU1, Respiratory Syncytal Virus, regular_ward, </em>and <em>Rhinovirus/Enterovirus. </em>For any set of values for each of these variables, we can get a posterior probability of the variable <em>covid_19.</em> In this particular example, we have given the Marvok blanket the following random values <em>{‘Platelets’: ‘3’, ‘Inf A H1N1 2009’: ‘not_detected’, ‘Influenza B’: ‘detected’, ‘Respiratory Syncytial Virus’: ‘detected’, ‘Coronavirus HKU1’: ‘detected’, ‘Rhinovirus/Enterovirus’: ‘not_detected’,’regular_ward’: 0}</em>, which yielded the posterior in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d50mtQ86lpqhLanZKhF2ng.png" /><figcaption>Posterior probability for covid_19 variable given its Markov blanket — Image by the authors</figcaption></figure><p>This combination of values in the Markov Blanket implies a probability that the patient tests positive for Covid-19 of 56.21%. Let’s now add a couple of observations of nodes outside of the Markov blanket and see the change. For example, let’s add <em>Metapneumovirus </em>and <em>Influenza A rapid test</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*6AAEowMiG0QVr4dMLYCTBw.png" /><figcaption>Posterior probability of having Covid-19 given parameters outside its Markov blanket — Image by the authors</figcaption></figure><p>…which yields no effect on our posterior probability for Covid-19. Note the power of this property of BNs: in the context of a big and potentially complex network, we only need to know about a smaller subset of nodes to model <em>what-if</em> scenarios about a variable of interest.</p><h3>Main takeaway</h3><p>In this blog we introduced Bayesian Networks and highlighted how relevant they are for decision making in uncterain situtaions, with a specific application in the context of the current pandemic.</p><p>As a graphical representation, these expert modelling tools are very intuitive and simplify much the analysis of complex systems. Their value resides mainly on the network structure and probabilistic linkages that they unveil for complex datasets. This makes them extremely beneficial for decision makers subject to great uncertainty, as it has already been proven in many industries and we have demonstrated with our example dataset.</p><p>The code for this analysis is available in <a href="https://github.com/AlvaroCorrales/BayesianNetworks">our Github repo</a>.</p><p>We used the pyAgrum library to do the analysis. Visit its <a href="https://agrum.gitlab.io/">website</a> to learn more about programming Bayesian Networks in Python.</p><p><strong>Authors:</strong></p><ul><li><strong>Hamza Agli </strong>is an international Data Science and AI leader with a PhD in Aritificial Intelligenge and Decision Making.</li><li><strong>Álvaro Corrales Cano</strong> is a Data Scientist. With a background in Economics, Álvaro specialises in a wide array Econometric techniques including causal inference, discrete choice models, time series and duration analysis.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73b06c3418d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/applying-bayesian-networks-to-covid-19-diagnosis-73b06c3418d8">Applying Bayesian Networks to Covid-19 diagnosis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>