<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pipe Salas Bravo on Medium]]></title>
        <description><![CDATA[Stories by Pipe Salas Bravo on Medium]]></description>
        <link>https://medium.com/@felipe.salas.b?source=rss-d484488f49bd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kr2DYgJx5Sjqat6AOAgmjg.jpeg</url>
            <title>Stories by Pipe Salas Bravo on Medium</title>
            <link>https://medium.com/@felipe.salas.b?source=rss-d484488f49bd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:47:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@felipe.salas.b/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Revealing Planetary Changes: Two decades of GRACE data through clustering analysis]]></title>
            <link>https://medium.com/bain-inside-advanced-analytics/revealing-planetary-changes-two-decades-of-grace-data-through-clustering-analysis-3a4890bd4428?source=rss-d484488f49bd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a4890bd4428</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[grace]]></category>
            <dc:creator><![CDATA[Pipe Salas Bravo]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 13:31:47 GMT</pubDate>
            <atom:updated>2024-07-17T14:35:11.554Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ice melting in the last two decades: can we measure it from the space?</h3><h3>Introduction</h3><p>The Gravity Recovery and Climate Experiment (GRACE) is a joint space mission by NASA and the German Aerospace Center (DLR). Since its launch in 2002, GRACE has revolutionized our understanding of earth system’s processes and the effects of human activities on water resources and anthropogenic effects on Earth. In this first part of our analysis, you’ll learn how the GRACE mission has transformed our understanding of climate change and how its data are being used in climate research.</p><p>This is an iniciative of our R&amp;D team from Bain and Company, in an effort for understanding the climate crisis, being conscious about the world’s health and trying to fight against it.</p><h3>What GRACE data looks like</h3><p>GRACE consists of two twin satellites orbiting Earth in tandem, at a distance of about 200 kilometers apart from each other. Equipped with precise instruments, the satellites continuously measure their separation from each other, and this information allows to estimate small variations in the Earth’s gravity field. The key lies in that when the first satellite passes over a high-density object, such as a mountain, it experiences a slight increase in gravity and accelerates slightly, increasing its separation from the second satellite. This separation then decreases when the second satellite passes over the same object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XHuJKO_SW5hgiy8MmikbEw.png" /><figcaption>Fictional picture of GRACE passing over the antarctic</figcaption></figure><p>The raw data collected by GRACE are processed by three independent groups: NASA’s Jet Propulsion Lab, the German Aerospace Center, and the University of Texas’ Center for Space Research. Each lab has its own methodology for correcting errors and transforming the raw data into information that can be used in scientific research.</p><p>The processed data is categorized into three levels. Level 1 data contain the raw data directly from the satellites, including systematic errors and noise. These data are then corrected and cleaned, forming Level 2 data, which contain corrections for orbital errors, atmospheric and tidal variations among others. Level 3 data are even more refined and are presented in a regular global grid for easy use by researchers.</p><p>GRACE data are available in monthly temporal resolution and a spatial resolution of ~100 kilometers (1 degree). This resolution is enough to identify the large modifications in the distribution of Earth’s mass that are occurring due to global warming. The data span from 2002 to the present, providing almost two decades of information about changes in water, ice and solid land mass across the globe.</p><p>The GRACE mission has provided crucial data for our current understanding of climate change. It’s delivered detailed data on ice loss in Greenland and Antarctica, allowed quantifying droughts in California and other regions of the world, and provided an unprecedented view of how earthquakes displace Earth’s mantle. GRACE has also revealed overexploitation of underground aquifers, a growing concern given the key role they play in water supply in many parts of the world.</p><p>According to its website:</p><blockquote><em>The observed monthly changes in gravity are caused by monthly changes in mass. The mass changes can be thought of as concentrated in a very thin layer of water thickness changes near the Earth’s surface (i.e., in a layer up to several kilometers thick). In reality,</em> most <em>of the monthly gravity changes are indeed caused by changes in water storage in hydrologic reservoirs, by moving ocean, atmospheric and land ice masses, and by mass exchanges between these Earth system compartments. Their vertical extent is measured in </em>centimeters of equivalent water thickness<em>, much smaller than the radius of the Earth or the horizontal scales of the changes, which are measured in kilometers. Some changes in gravity are caused by mass redistribution in the ‘solid’ Earth, such as those following a large earthquake, or those due to </em><a href="https://grace.jpl.nasa.gov/data/get-data/gia-trends/"><em>glacial isostatic adjustment</em></a><em>; in those cases, the interpretation of the gravity changes in terms of ‘equivalent water thickness’ are not correct, even though it is still possible to compute this quantity (i.e., by removing the solid Earth effects to isolate the water-related mass changes).</em></blockquote><p><strong>What the data looks like</strong></p><p>The data consists of a square grid of one degree on a monthly basis, with data from 2002-Apr-05 to 2017-Oct-18 and then from 2018-May-22 to Present. For each point of the grid, we have the value of the level water equivalence thickness and an uncertainty value.</p><p>For example, we can look at the time series for a specific coordinate and its neighbors of their <em>lwe</em>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F87.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F87%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/ede899389d1c300df149482e22d7c041/href">https://medium.com/media/ede899389d1c300df149482e22d7c041/href</a></iframe><p>An easy look at the time serie gives us two immediate thoughts:</p><ul><li>there’s a high seasonality</li><li>close coordinates have the same behavior</li><li>the variance between neighbors has been growing (a change in the dynamic)</li></ul><p>So an immediate question that arises is whether we can find “similar” surfaces on earth that have the same behavior in terms of water content.</p><h3>Which global information can we get?</h3><p>In the next section we are going to aboard the question of how we can find similar spots on the earth in terms of their water level equivalence. We want to verify two things: there’s geographical coherence (close coordinates are similar between them), and understand groups of coordinates that has the same evolution over time but are far away from them.</p><p>To find similar spots, we are going to clusterize the time series using k-means, but first we need to correct some issues in the data:</p><ol><li>There are some missing months of data</li><li>The problem with seasonality: while in the north hemisphere is summer, in the south hemisphere is winter, so the seasonalities of those areas are not going to match and are going to introduce noise in ours analysis.</li></ol><p>For the first problem, we will interpolate the missing data using cubic spline interpolation method, and for the latter, we will obtain the trend of each time serie, and clusterize each coordinate based on its trend.</p><h4>Getting trends</h4><p>Once we have no missing data, we can decompose each time serie into its trend and seasonality. For doing that, we are going to use <em>seasonal_decompose</em> from <a href="https://www.statsmodels.org/stable/generated/statsmodels.tsa.seasonal.seasonal_decompose.html">statsmodels</a>. There is one parameter that we need to decide, whether the seasonal component is modeled in an additive or multiplicative way. As the gravity changes that Grace detects are due to shifts in masswater, we decided to model the seasonality in an additive way.</p><h4>Clustering with k-means</h4><p>Once we have the trend of each time serie for each coordinate, we are going to cluster them using the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html">k-means algortithm</a>. For doing that, we need to decide the number of clusters we desire, but we don’t know that number yet. Instead of choosing a cluster number a priori, we are going to clusterize the database from <em>k=2</em> to <em>k=10</em> and then we are going to analyze some metrics in order to decide the optimal number, as well as looking into the <em>phylogeny</em>, which corresponds to the evolution of the clusters as we increment <em>k</em>.</p><p>Those metrics we are going to look at are:</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html">Inertia</a> (Within-Cluster Sum of Squares): Inertia measures the sum of squared distances between data points and their cluster’s centroid. It quantifies how tightly grouped the data points are within their respective clusters. Lower inertia indicates better clustering, suggesting that data points within clusters are closer to their centroids.</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.davies_bouldin_score.html">Davies-Bouldin score</a>: The Davies-Bouldin score assesses the average similarity between each cluster and its most similar neighboring cluster. A lower Davies-Bouldin Index indicates better separation between clusters, meaning that clusters are distinct from each other.</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html">Silhouette Score</a>: The Silhouette Score measures the quality of clustering by evaluating how similar each data point is to its own cluster compared to other clusters. It ranges from -1 to 1, with higher values indicating better clustering. A higher silhouette score means that data points are well-matched to their clusters and poorly matched to neighboring clusters, indicating clear and distinct clusters.</p><h3>Results</h3><p>The following section has 3 components of results:</p><ol><li>Defining the correct number of clusters</li><li>Interpreting each cluster</li><li>See where, in the world, each cluster is allocated</li></ol><p><strong>Elbow, Silhouette and Davies Bouldin graphs:</strong></p><p>For defining the optimal number of clusters we are going to see the metrics we already discussed:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F8.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F8%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/08d162178ffe3c4488d2d17dd0d9ab14/href">https://medium.com/media/08d162178ffe3c4488d2d17dd0d9ab14/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F10.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F10%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2ed90d87f92ed648d07e688e698c29e/href">https://medium.com/media/f2ed90d87f92ed648d07e688e698c29e/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F12.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=http%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F12%2F&amp;image=http%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/94eed501be18894604454a3db3a524da/href">https://medium.com/media/94eed501be18894604454a3db3a524da/href</a></iframe><p>Based on the graphs above, choosing <em>k = 5</em> looks like a good option for the number of clusters, since it has a small value associated with the Davies Bouldin score, and a high value in the Silhouette score. Also, <em>k = 6</em> sounds like a good option, but we need to see the phylogeny first.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F14.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=http%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F14%2F&amp;image=http%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/912620625677918a714af0d557e66abd/href">https://medium.com/media/912620625677918a714af0d557e66abd/href</a></iframe><p>As we can see in the phylogeny, when dealing with 5 clusters, there are 2 big clusters and 3 small. A good option could be going up to <em>k = 6</em>, in that way, we have the same 3 small clusters but also 3 big clusters, and the corresponding values in Davies Bouldin and Silhouette do not change, but for simplicity we are going to stick to <em>k = 5</em>.</p><p><strong>Cluster representation</strong></p><p>To characterize each cluster, we take a sample of trends that belongs to it, and plot them as timeseries in gray color. Also, we plot the representative of the cluster, defined as the trend that is closer to the median value of all samples.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F72.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F72%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7773a0734b9bdb0368b43ada93fb4a3b/href">https://medium.com/media/7773a0734b9bdb0368b43ada93fb4a3b/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F74.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F74%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ba3e0af8750c7502c9503cb3b7e10f2/href">https://medium.com/media/7ba3e0af8750c7502c9503cb3b7e10f2/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F76.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F76%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b4613e1ebd50081b6227260e88ea768/href">https://medium.com/media/2b4613e1ebd50081b6227260e88ea768/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F78.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=http%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F78%2F&amp;image=http%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/88cff8af87d5e04b888e67b8ca0d1780/href">https://medium.com/media/88cff8af87d5e04b888e67b8ca0d1780/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F80.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F80%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/bbbbd3eab2936b0b3c57a526d9f35ddf/href">https://medium.com/media/bbbbd3eab2936b0b3c57a526d9f35ddf/href</a></iframe><p>As we can see from the figures, each cluster has a different behavior, being cluster 1 a set of trends with no slope, almost constant, over time, and clusters 4, 5, 2 and 3 are clusters where the water level equivalence is going down. Also, every time serie has an artifact around the end of 2018, which correspond to the missing data period between one mission and the next. This artifact was created by us during the interpolation step, and one alternative could be improve the interpolation method.</p><p>We can create a grouped chart to see the 5 clusters at the same time:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F40.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F40%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d9f0f031606842e4f350fb5e9394194/href">https://medium.com/media/2d9f0f031606842e4f350fb5e9394194/href</a></iframe><p>From all the clusters, there are two of them (<em>cluster 1</em> and <em>cluster 4</em>) that have a “constant” representative, while having a lot of variance, specially by the end of the period. Contrarily, others clusters (<em>cluster 5</em>, <em>cluster 2</em> and <em>cluster 3</em>) have a decrecent curve as a representative, decreasing the slope of the line as we move from <em>cluster 5</em> to <em>cluster 3</em>.</p><p>This is highly alarming information as it indicates that certain regions are losing water, whether in the form of ice melting or water reservoirs getting dry.</p><p>Another interesting question is about the surface associated to each cluster. A proxy of this is the number of coordinates that belongs to each cluster, so we are going to see that plot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F42.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F42%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee9e1b35772885a1ff8af6c6cfb11c6b/href">https://medium.com/media/ee9e1b35772885a1ff8af6c6cfb11c6b/href</a></iframe><p>From the last figure, we can se that the clusters that are losing more water are the smallest clusters as well, and represent less than 8% of the surface.</p><p>Now, the final question is where, in the world, are located each one on the clusters. For this, we have an interactive map where we can see the geography associated with each cluster across the world.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplotly.com%2F%7Epipers%2F38.embed%3Fautosize%3Dtrue&amp;display_name=Plotly&amp;url=https%3A%2F%2Fchart-studio.plotly.com%2F%7Epipers%2F38%2F&amp;image=https%3A%2F%2Fchart-studio.plotly.com%2Fstatic%2Fwebapp%2Fimages%2Fplotly-logo.8d56a320dbb8.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=plotly" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b6987cd8e002f25bc887678ca301514/href">https://medium.com/media/9b6987cd8e002f25bc887678ca301514/href</a></iframe><p>From the map, first, we can find out that there exists the geographical coherence, where neighbors belong to the same cluster. Secondly, we can easily recognize some geographies associated to each cluster:</p><p><strong>Cluster 5:</strong></p><ul><li>Caspian sea</li><li>Himalayas</li><li>Alaska</li><li>Greenland shore and outside lands</li><li>Chilean Ice fields</li><li>Antarctic territory: Palmer land, Marie Byrd land and Terre Adelie.</li></ul><p><strong>Cluster 2:</strong></p><ul><li>Greenland inside land</li><li>Antarctic: surroundings of Ellsworth Land</li></ul><p><strong>Cluster 3:</strong></p><ul><li>Ellsworth Land</li></ul><p>All the rest of the world that was not mentioned here, belongs whether to <em>cluster 1</em> or <em>cluster 4</em>. Also, our <em>answer first</em> was confirmed by the results where all the locations, except from the Caspian sea, are locations associated with big ice masses.</p><p>If you want to explore the results in more depth, we have a <a href="https://grace-time-series.streamlit.app/"><em>streamlit</em> application</a> running with the results of this article, and where you can try different numbers of clusters and see the associated results.</p><h3>Conclusions</h3><p>Grace satellite provides us with a great open source data resource, from which there is a lot of analysis that can be done and extract very useful — but very scary — information.</p><p>Clustering methodologies as <em>k-means</em> are a great example of nice-to-have tool for every data scientist, and playing with grace data was a great opportunity for showing its potential. The elbow graph, silhouette and davies bouldin scores could be a good heuristic for deciding the best number of clusters, as well as phylogeny and the characterization of each cluster.</p><p>When looking at the results, it is worrying to see that the gravity changes, <em>possibly due to ice losses</em>, are so big that are easily distinguishable from the space, and it might be affecting all the ice masses from the world, at different speed, but in a constant and irreversible way. The most affected ice mass is the Ellsworth Land, which is a high ice plateau. These results are in concordance with the results on <a href="http://gravis.gfz-potsdam.de/ais">Antarctic Ice-Mass Change website</a>.</p><p>Regarding the Caspian Sea, the situation is not less worrying, since it is the world’s largest inland body of water, and our results suggest that what we are looking from the space is the fall in water level that has been reported since years ago. If you want to dive more into this phenomena where inland water bodies are shrinking, you can check on <a href="https://theconversation.com/the-caspian-sea-is-set-to-fall-by-9-metres-or-more-this-century-an-ecocide-is-imminent-152229">this article</a>, this <a href="https://www.themoscowtimes.com/2020/12/25/scientists-sound-the-alarm-over-fast-shrinking-caspian-sea-a72484">other article</a> or this <a href="https://www.nature.com/articles/s43247-020-00075-6">publication</a>.</p><p><strong>Acknowledges</strong></p><p>Acknowledges to the <a href="https://www.cr2.cl/">Center for Climate and Resilience Research</a> (CR2) and specifically to Camila Alvarez-Garreton for her guidance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a4890bd4428" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bain-inside-advanced-analytics/revealing-planetary-changes-two-decades-of-grace-data-through-clustering-analysis-3a4890bd4428">Revealing Planetary Changes: Two decades of GRACE data through clustering analysis</a> was originally published in <a href="https://medium.com/bain-inside-advanced-analytics">bain-inside-advanced-analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time series parameters finding using Prophet and Optuna bayesian optimization]]></title>
            <link>https://medium.com/spikelab/time-series-parameters-finding-using-prophet-and-optuna-bayesian-optimization-e618614bd8b7?source=rss-d484488f49bd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e618614bd8b7</guid>
            <category><![CDATA[optuna]]></category>
            <category><![CDATA[prophet]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[bayesian-optimization]]></category>
            <dc:creator><![CDATA[Pipe Salas Bravo]]></dc:creator>
            <pubDate>Sat, 20 Feb 2021 01:07:38 GMT</pubDate>
            <atom:updated>2021-02-20T01:22:30.183Z</atom:updated>
            <content:encoded><![CDATA[<h4>A smart way for training time series models using <strong>prophet</strong> and <strong>optuna</strong> libraries for creating additive models and finding their optimal parameters through a bayesian approach.</h4><h3>Time series modeling</h3><p>Time series is a big topic in machine learning space and exists in practically all the industries, and, all hands down, is a problem that every data scientist will face on their career..</p><p>Besides the basic of the task, there are a lot of ways of facing and finding models for describing the underlying dynamic of a time series. For example, there’s a lot of classical models, such as space state models (Arima, Armas, etc…), and the use of trees based algorithms as well. In this article, we are going to discuss the use of one tool which is one of the easiest and yet powerful ways of working with time series, <a href="https://facebook.github.io/prophet/">prophet</a>, a Facebook library which:</p><blockquote>Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.</blockquote><p>For getting the best model that prophet can offer, we need to adjust the parameters of the model. The classic way of finding the best combination of parameters is doing a grid search(random search or a cartesian search), but this can be too time-expensive, specially when we are validating the models using a large number of folds during cross-validation.</p><p>An excellent alternative for saving some time and doing a smarter search in the parameters space is doing a <strong>bayesian search</strong>, which will focus on the areas of parameter’s space that have a better value in our objective function. There are some libraries which can do the bayesian search, but we will focus in <a href="https://optuna.org/">optuna</a>, a really good and easy-to-use library. Among the key features of optuna, these are the main ones:</p><ul><li><strong>Eager search space:</strong> Automated search for optimal hyperparameters using Python conditionals, loops, and syntax.</li><li><strong>State-of-art algorithms:</strong> Efficiently search large spaces and prune unpromising trials for faster results.</li><li><strong>Easy parallelization: </strong>Parallelize hyperparameter searches over multiple threads or processes without modifying code.</li></ul><p>One of the things I liked the most, is that we need only a few lines of code for getting optuna to work:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5484d3fd8d3fba35ae147d413f721a1/href">https://medium.com/media/c5484d3fd8d3fba35ae147d413f721a1/href</a></iframe><p>In the code above we see how easy is to implement optuna for a simple optimization problem, and is needed:</p><ul><li>An objective function to be optimized (default is minimizing it).</li><li>A distribution of the variable we are searching (from which we sample). It could be a continuous or discrete distribution.</li><li>Create a study and invoke the optimize method. As the number of trials increases, the solution will be better.</li></ul><p>It sounds very easy, so we will implement it in a prophet parameters search.</p><h3>Lets model it</h3><p>So first we need a time series to work with, and as an example we will work with the maximum temperature in Valdivia, the city where I live.</p><p>To get the data we use the Nasa API which gives temperature information between two given dates. We are going to use the temperature from 2015 to the present day.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af63baf9eaa295582c552196a7314181/href">https://medium.com/media/af63baf9eaa295582c552196a7314181/href</a></iframe><p>And it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDNVGuW0oqy3mQMkz01PBg.png" /><figcaption>Daily maximum temperature from 2015 to 2021 in Valdivia, downloaded from Nasa API</figcaption></figure><h4>How we validate</h4><p>The validation period has a greater importance when dealing with time series. Model performance and optimal parameters depends exclusively in the election of validation and training periods. So what we want is having a robust way of finding the optimal parameters, diminishing the risk of a bad chosen validation period. In order to do this, the error measure should be calculated using time series cross validation. This, and not random split, is the correct way of training a time series in order to prevent data leakage from future data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q7gYPQAGMjmefN0JjskDpw.png" /><figcaption>Idea behind cross validation in time series using a 3 fold split.</figcaption></figure><p>Fortunately prophet has already an implementation for using cross validation which can be parallelized (see prophet documentation for finding different ways of parallelizing).</p><p>The election of the horizon of each validation period depends on the problem we are solving. This needs to have a correspondency to the problem we are willing to solve through the forecast. Lets suppose that we need a 1-week forecast, and we are going to use a 4-fold cross validation (we left the final 4 weeks for validation).</p><p>Code looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6dc9cc4eb1dcec92ae796a8e217190eb/href">https://medium.com/media/6dc9cc4eb1dcec92ae796a8e217190eb/href</a></iframe><p>In the code above, first we need to adjust a prophet model to our data, and then we need to specify the forecast horizon (horizon) of the cross validation, and then, optionally, the size of the initial training period (initial) and the spacing between cutoff dates (period).</p><p>Once we have trained the models, we can get the performance_metrics, which calculate different metrics from the cross validation output, like mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), mean absolute percent error (MAPE), median absolute percent error (MDAPE) and coverage of the yhat_lower and yhat_upper estimates. These are computed on a rolling window of the predictions in df_cv after sorting by horizon (ds minus cutoff). So we already have a robust way for measuring the error of our model.</p><p>Now we need to see which parameters can be tunned. Accordingly to prophet documentation, if we want to tune parameters, the first ones to be tunned should be:</p><ul><li>changepoint_prior_scale: This is probably the most impactful parameter. It determines the flexibility of the trend and, in particular, how much the trend changes at the trend changepoints.</li><li>seasonality_prior_scale: This parameter controls the flexibility of the seasonality.</li><li>holidays_prior_scale: This controls flexibility to fit holiday effects.</li><li>seasonality_mode: Options are [&#39;additive&#39;, &#39;multiplicative&#39;].</li></ul><p>For this example, as temperature is not affected by holidays, our modeling won’t include them.</p><p>Then, we need to specify the support and the distribution for each parameter, so we are going to consider the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/890591e6232c3abbacb0c11d91bcf439/href">https://medium.com/media/890591e6232c3abbacb0c11d91bcf439/href</a></iframe><p>As seasonality mode can take only two values, we are going to sample from an integer distribution which support will be {0,1}, which represent the election of the additive or multiplicative mode.</p><h4>Wrapping prophet and optuna</h4><p>So we already have everything needed for finding the best parameters of our prophet model. The objective function to be optimized for optuna will be the RMSE over the 4 folds of 7 days each, and in order to be resource-efficient, the cross validation will be parallelized.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a92bbbe302428cef22284e27aea388ae/href">https://medium.com/media/a92bbbe302428cef22284e27aea388ae/href</a></iframe><p>Once we have defined our function to be minimized, we need to create the study and define the number of trials we are going to run. Once the study has finished, we have found the parameters that will minimize the RMSE over the folds, so now we can train a final model with all our data in order to start making future predictions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/826fe946f93bfe39e448dfdea086aa93/href">https://medium.com/media/826fe946f93bfe39e448dfdea086aa93/href</a></iframe><h3>Final thoughts</h3><p>Prophet is an easy and quick library for time series prediction, which combined with optuna will unleash its potential. Although in this article we don’t report the error, if we want to reduce it more maybe we need to do some additional tweaks, like using additional regressors or maybe changing and tunning additional prophet’s parameter.</p><p>Optuna is a very useful framework that allows tuning parameters not only for prophet, but for any other model. This is specially important when the searching of parameters comes along with the use of a special metric (different from any usual metric) for evaluating the model performance.</p><p>Please feel free to make any comments or suggestions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e618614bd8b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spikelab/time-series-parameters-finding-using-prophet-and-optuna-bayesian-optimization-e618614bd8b7">Time series parameters finding using Prophet and Optuna bayesian optimization</a> was originally published in <a href="https://medium.com/spikelab">spikelab</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>