<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Peter Kalverla on Medium]]></title>
        <description><![CDATA[Stories by Peter Kalverla on Medium]]></description>
        <link>https://medium.com/@peter.kalverla?source=rss-34317cc61901------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*K2rdrREUzuttU_7Nvc05XQ.jpeg</url>
            <title>Stories by Peter Kalverla on Medium</title>
            <link>https://medium.com/@peter.kalverla?source=rss-34317cc61901------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:00:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@peter.kalverla/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A special gift, December 5th]]></title>
            <link>https://medium.com/escience-center/a-special-gift-december-5th-69f12753581d?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/69f12753581d</guid>
            <category><![CDATA[sinterklaas]]></category>
            <category><![CDATA[poem]]></category>
            <category><![CDATA[tradition]]></category>
            <category><![CDATA[research-software]]></category>
            <category><![CDATA[best-practices]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Fri, 05 Dec 2025 07:49:23 GMT</pubDate>
            <atom:updated>2025-12-05T07:49:23.346Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>A special gift (December 5th)</h3><h4>Ode to the research software engineer</h4><blockquote>Sinterklaas is a Dutch tradition featuring Saint Nicholas, who is said to arrive from Spain with his helpers and travel across rooftops to deliver small gifts and poems. Today, the Sint is visiting the eScience Center.</blockquote><p>As days are getting colder and the nights becoming long, <br>we make our office cozy with hot chocolate and song. <br>The fireplace is lit albeit on the conference screen;<br>it is an office atmosphere unlike you’ve ever seen.</p><p>We sit together ‘round the fire and wait for what’s to come, <br>when suddenly we hear a heavy beating, like a drum. <br>We chase towards the office entrance seeing who is there, <br>but nothing’s to be seen except a letter near the stairs.</p><p>The letter reads “My dearest friends and fellow RSEs, <br>please go back to your conference room and make yourself at ease. <br>You’ve made me feel so welcome and I brought you in return <br>a bundle of delights for you to live and love and learn.”</p><p>Curious we sat once more around the makeshift fire, <br>stunned to find some gifts that any scientist should desire. <br>A bag full of best practices to make your software shine, <br>sparked with digital literacy and beautiful design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byMlMge4UuMF2n-S0cGfFQ.jpeg" /><figcaption>Chocolate letters are part of the tradition</figcaption></figure><p>To start it off we first unpacked a gift so smart and bright: <br>a coffee mug for every soul who codes late in the night.<br>On it was a text that read “your 9 to 5 is fine, <br>but if you rather work at night, some caffeine’s divine”.</p><p>The next gift comes in handy for a project big or small<br>a touch of git to help you keep your code under control.<br>A branching scheme, a tidy diff, a history clear and clean<br>A present many researchers admit they’ve never seen.</p><p>Then slips beneath the office door a test suite neatly wrapped,<br>with unit tests and fixtures all meticulously mapped.<br>“Oh generous Sint,” we all proclaim, “now bugs no more appear!”<br>(Though secretly he knows they will, but less, and less severe.)</p><p>A handbook tied with twine he leaves, with pretty illustrations<br>on licenses and versioning and good documentation.<br>For knowledge lost is hard to find when months have come and gone,<br>but with this helpful guide the happy coding can go on.</p><p>Sint conjures CI pipelines high above us in the cloud,<br>where automated checks and builds assemble clean and proud<br>A magical parade of green ticks marching in a row,<br>the sweetest sight for anyone who’s watched code fail to grow.</p><p>But most of all, the Sint bestows a gift both rare and dear:<br>the quiet craft, the steady hand, the engineer’s career.<br>For research needs more than ideas just drifting in the air;<br>it needs the ones who shape the tools that make discovery FAIR.</p><p>So here’s an ode to those who bridge two worlds with practiced grace:<br>who think in algorithms yet they keep the human pace.<br>Who build, maintain, refactor, teach, who safeguard truth with care,<br>the research software engineers, the heroes unaware.</p><p>And as the Sint sails home again across December seas,<br>he leaves one final message carried softly by the breeze:<br>“Good science thrives when code is sound, transparent, shared, and clear,<br>so honour those who make it so; support them through the year.”</p><p><em>This poem has been crafted in collaboration with AI-Piet</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69f12753581d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/a-special-gift-december-5th-69f12753581d">A special gift, December 5th</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 tips for writing a successful eScience Center proposal]]></title>
            <link>https://medium.com/escience-center/5-tips-for-writing-a-successful-escience-center-proposal-1247bc44a5fd?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/1247bc44a5fd</guid>
            <category><![CDATA[research-software]]></category>
            <category><![CDATA[escience]]></category>
            <category><![CDATA[proposal-writing]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Thu, 28 Aug 2025 07:07:31 GMT</pubDate>
            <atom:updated>2025-08-28T07:07:31.879Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p><em>Over the years, we’ve worked on a wide range of collaborative projects through our open calls. The most successful ones? They tend to have a few things in common right from the proposal stage. Here are five tips to help your proposal lay the foundation for a smooth, impactful, and genuinely rewarding collaboration.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oQKFPzp_2zS-GzxzOMCccg.png" /><figcaption>Image adapted from <a href="https://unsplash.com/photos/a-red-and-white-rocket-ship-flying-through-the-sky-O1jUvZX9DOA">Unsplash</a></figcaption></figure><h4>1. Describe the challenge, not the solution</h4><p>We love a good challenge. If the technical solution is already fully defined, there’s less room for us to contribute meaningfully. For example, instead of saying <em>“we need a dashboard that does X”</em>, describe the challenge: <em>“we struggle to analyze Y because current tools can’t handle Z.”</em> That kind of framing sparks our curiosity and drives the creative collaboration we love.</p><h4>2. Dedicate real time to the project</h4><p>We’re looking for genuine teamwork — whether that’s you, your postdocs, or PhD students. The best projects have collaborators who test the software as it is built. For example: <em>“we tried the new analysis module this week, here’s where it’s slow, and here’s a feature we hadn’t thought of before.”</em> Aligning your proposal with existing commitments makes this kind of feedback loop much easier.</p><h4>3. Be realistic</h4><p>Ambition gets projects funded. And we love bold ideas — but bold doesn’t mean bloated. Make sure your proposal has a solid core to build on, and leave room for exploration, iteration — even failure. A focused project with space to explore usually delivers more than an overpacked one that runs out of steam.</p><h4>4. Build on what you have</h4><p>We love improving workflows and building better tools — but starting from scratch rarely works in one project. If the data is missing, the workflow doesn’t exist, or the whole thing is still hypothetical, that’s a red flag. Don’t think what might be useful for <em>someone, someday</em>. Instead, think what would help <em>you, today, </em>and make your code and data available before submitting the proposal. Even a messy dataset or clunky script gives us a real starting point, so we can hit the ground running.</p><h4>5. Think beyond the project</h4><p>A successful project doesn’t stop when the funding ends. Our calls require a sustainability plan — and for good <a href="https://doi.org/10.5281/zenodo.15646827">reason</a>. The tools that have the biggest impact are those that become part of daily workflows and get shared, adapted, and improved over time by a community of users. So think about how your software fits into your everyday research, and who else might use or build on it after the project wraps up. Check out the <a href="https://smp.research.software/">SMP tool</a> to get a jump start.</p><p><em>Ready to collaborate? Check our </em><a href="https://www.esciencecenter.nl/calls-for-proposals/"><em>website</em></a><em> for an up-to-date overview of past and open calls, or simply </em><a href="https://www.esciencecenter.nl/contact-us/"><em>get in touch</em></a><em>.</em></p><p><em>This blog post represents original content created with AI assistance.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1247bc44a5fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/5-tips-for-writing-a-successful-escience-center-proposal-1247bc44a5fd">5 tips for writing a successful eScience Center proposal</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amsterdam as seen by a weather model]]></title>
            <link>https://medium.com/escience-center/amsterdam-as-seen-by-a-weather-model-b609a340cde4?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/b609a340cde4</guid>
            <category><![CDATA[urban-planning]]></category>
            <category><![CDATA[weather]]></category>
            <category><![CDATA[3d]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Tue, 10 Dec 2024 12:32:13 GMT</pubDate>
            <atom:updated>2024-12-10T12:32:13.298Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Visualizing local climate zones in 3D with the Cityblocks package</h4><p><em>What sets a weather model apart from a video game? While video games prioritize stunning visuals over strict realism, weather models focus entirely on physics. Visualizations come as an afterthought at best. A missed opportunity, since good visualizations can reveal fascinating insights into how these models interpret the world. In this blogpost we bring Amsterdam to life in 3D through the eyes of a weather model — and invite you to try it for your city as well.</em></p><p>Weather models have a peculiar view of a city. To them, the world is made of boxes. Many roughly rectangular boxes. A typical grid box contains more than one building, so weather models see cities at a block level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*l4MLhZBrZbnuEC8AjIcJuQ.png" /><figcaption>Weather and climate models divide the earth in grid cells. Each grid cell encompasses a wealth of information and processes, which must be represented in a simplified form. Source: <a href="https://simple.wikipedia.org/wiki/Climate_model">wikipedia</a></figcaption></figure><p>To describe cities at block level, one option is to use <a href="https://doi.org/10.1016/j.rse.2023.113573">local climate zones</a> (LCZs): a set of typical building blocks modeled after cities across the globe. Each LCZ is associated with certain properties, such as the typical building height, street width, or green fraction. With that, weather models have all the info they need to simulate the interaction between the city and the atmosphere, in a simplified form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_qyITwlKMyiXrfSdUfluQ.png" /><figcaption>Left: Illustration of the local climate zones (<a href="https://doi.org/10.1016/j.rse.2023.113573">source</a>). Right: map of local climate zones for Amsterdam (<a href="https://doi.org/10.5281/zenodo.7670653">source</a>).</figcaption></figure><p>Typically, maps of local climate zones are displayed in 2D — a flat map. In principle, this is good enough, as we have only one LCZ per grid cell. But it doesn’t really appeal to the imagination. From the map above, I cannot tell how realistic this representation of Amsterdam actually is.</p><p>That’s why I started playing around with 3D visualizations instead. The idea is to reproduce the tiles from the illustration above as building blocks, and then place them on a plane in accordance with the pixels on the 2D map. Below, we detail two approaches, one with CityJSON, and one with QGIS.</p><h3>Building a city in CityJSON</h3><p>CityJSON file format is intended as a lightweight and developer-friendly format for spatial information. Like JSON, it is basically a set of key-value pairs. The <a href="https://www.cityjson.org/specs/overview/">CityJSON format </a>specifies which keys are allowed/required and what values they can have. I was not familiar with the format, so this part of the blogpost also represents my learning experience. A simple CityJSON file could look like this:</p><pre>{<br>  &quot;type&quot;: &quot;CityJSON&quot;,<br>  &quot;version&quot;: &quot;2.0&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;geographicalExtent&quot;: [0,0,0,1,1,1]<br>  },<br>  &quot;transform&quot;: {<br>    &quot;scale&quot;: [0.001,0.001,0.001],<br>    &quot;translate&quot;: [0,0,0]<br>  },<br>  &quot;extensions&quot;: {},<br>  &quot;vertices&quot;: [<br>    [0,0,1000],<br>    [1000,0,1000],<br>    [1000,1000,1000],<br>    [0,1000,1000],<br>    [0,0,0],<br>    [1000,0,0],<br>    [1000,1000,0],<br>    [0,1000,0]<br>  ],<br>  &quot;CityObjects&quot;: {<br>    &quot;id-1&quot;: {<br>      &quot;geometry&quot;: [<br>        {<br>          &quot;boundaries&quot;: [<br>            [<br>              [[0,1,2,3]],<br>              [[4,5,1,0]],<br>              [[5,6,2,1]],<br>              [[6,7,3,2]],<br>              [[7,4,0,3]],<br>              [[7,6,5,4]]<br>            ]<br>          ],<br>          &quot;lod&quot;: &quot;1&quot;,<br>          &quot;type&quot;: &quot;Solid&quot;<br>        }<br>      ],<br>      &quot;attributes&quot;: {<br>        &quot;function&quot;: &quot;something&quot;<br>      },<br>      &quot;type&quot;: &quot;GenericCityObject&quot;<br>    }<br>  }<br>}</pre><p>The vertices represent points in 3D space. This file specifies 8 points. Each CityObject specifies an array of boundaries, which are essentially surfaces obtained by connecting the vertices. Here we have 6 surfaces, and if you draw them all up, you will see that this is simply a model for a cube.</p><p>The <a href="https://www.cityjson.org/">CityJSON home page</a> is very welcoming. It contains lots of useful information and examples. There is an official online CityJSON <a href="https://validator.cityjson.org/">validator</a>, as well as a <a href="https://viewer.cityjson.org/">viewer</a> called Ninja. So once you’ve obtained a CityJSON file, you can easily validate it and display it in the browser.</p><p>I started with the <a href="https://www.cityjson.org/datasets/#simple-geometries">example </a>of 2 cubes, and wrote <a href="https://github.com/Urban-M4/cityjson-generator">some Python code</a> to reproduce them as templates, such that I could place blocks randomly everywhere, and scale them in width and height. I created multiple combinations of blocks to reproduce the LCZ archetypes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*EwYAPHFMN43apRcsZYo7QQ.png" /><figcaption>LCZ archetypes reproduced in CityJSON.</figcaption></figure><p>Next, I played a little bit with some random data to see how big of a city I could render. I found that 100 by 100 tiles still worked okay in the Ninja viewer. Effectively, this corresponds to a city of 10 by 10 km. I downloaded the global LCZ map from <a href="https://doi.org/10.5281/zenodo.7670653">Demuzere et al.</a> and cut out the Amsterdam area. The last step was to loop over all pixels in the image, and add the corresponding tile to the CityJSON file.</p><p>The code ran quite slowly, but it worked. The Ninja viewer went into “performance mode”, but eventually was able to display the data just fine. I was ready to observe the highly anticipating result…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*qw1Ym7xaVw_aT_FnEQHvAw.png" /><figcaption>Amsterdam rendered as 3D LCZ tiles. Blue designates correspond to water, yellow designates all other non-urban land use pixels.</figcaption></figure><p>Admittedly, that looks a bit less spectacular than I had hoped for. The interactive view is a bit better, but still. While you can recognize the river, a canal, some parks and different types of buildings, this doesn’t really appeal to the imagination as I had hoped.</p><p>One way to improve upon this initial view would be to render the different LCZ classes with different colors, and make the tiles prettier with things like trees, just like on the example tiles above. However, it seemed the online CityJSON viewer doesn’t support rendering materials, and I was already pushing its limits.</p><p>In order to proceed with CityJSON, we would need to speed things up, and use a better viewer. There are several ways to speed things up. For example, by using <a href="https://www.cityjson.org/cityjsonseq/">CityJSON Sequence</a>s and maybe converting to <a href="https://github.com/3DGI/tyler">3D Tiles</a>. Concerning the viewer, I tried using the Blender and QGIS plugins, but I struggled to use them properly. In the long run, I would like to publish something like this on an interactive webpage, so I would prefer to look at other solutions. The standard viewer uses <a href="https://threejs.org/">three.js</a> under the hood. It seems quite doable to embed such a viewer in a static website hosted from GitHub pages, for example. As such, anyone could clone the template and host their own city model with ease.</p><h3>Making it beautiful with QGIS</h3><p>At this point I was showing my progress to my colleague Maurice. He is one of our GIS experts and helped me to reproduce the workflow in <a href="https://www.qgis.org/">QGIS</a>. This alleviated some of the issues we had with the CityJSON viewer. With QGIS, we could render tiles in different colors, and use true coordinates to combine the tiles with other map layers. For example, here is a version where the colours of the tiles correspond to those on the 2D map above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LlwYC1PNrl72OFSl81SLpA.png" /><figcaption>3D rendering of local climate zones and other land use types for Amsterdam, using data from the global dataset from <a href="https://doi.org/10.5281/zenodo.7670653">Demuzere et al</a>.</figcaption></figure><p>With QGIS, we took a slightly different approach. Instead of storing 3D cityblocks, we made 2D imprints of the LCZ types, and extruded them to a set height during the 3D render. We used Python with <a href="https://rasterio.readthedocs.io/en/stable/">rasterio</a> and <a href="https://geopandas.org/en/stable/">geopandas</a> to read and crop the original data and generate the 2D tiles. Intermediate data was stored as a <a href="https://en.wikipedia.org/wiki/GeoPackage">GeoPackage</a>, which can easily be loaded in QGIS.</p><h4>Introducing Cityblocks: a simple tool to visualize your own city</h4><p>We decided to spend a bit more time to make this workflow available for anyone. We polished the code and converted it into a small and easy to use Python utility called <a href="https://pypi.org/project/cityblocks/">Cityblocks</a>. You can use it to download the global dataset, extract an area of interest, and convert the data into tiles. Subsequently, the data can be visualized in QGIS, or something else in case you prefer. You can find the code and instructions <a href="https://github.com/Urban-M4/cityblocks">here</a>. At the moment, this is mostly tested for our Amsterdam use case, so we are very curious to get feedback on your experience with it for other cities.</p><h3>Making it beautiful</h3><p>At this point, we though it’d be nice to turn this little side quest into a real map that would stand its ground in an atlas of eScience Center projects.</p><p>A good map tells a story that jumps at the viewer without much context. We considered many different options, and settled on a design that places the tiles on a background map of Amsterdam, with a subtle raster to hint at the gridded nature of weather models. Here is the final result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8-_I_H5hGt1j7SPd-BK0Q.png" /><figcaption>3D Local climate zones overlaid on an aerial photograph of Amsterdam. The aerial photograph comes from <a href="https://www.pdok.nl/introductie/-/article/pdok-luchtfoto-rgb-open-">PDOK</a>.</figcaption></figure><p>Seeing the map with and without LCZs makes it easy to compare the “human” view to the “weather model view”. Notice that most of the riverbank area, including the station, is classified as “heavy industry”, whereas the city center is dominated by “compact to open midrise”, for example. Apart from evaluating the classification itself, this view also reveals the relatively coarse structure the weather model sees, and how rigid the city looks. Perhaps it is a sobering view that reminds us that even sophisticated weather models are only an approximation of reality.</p><p>But it is inspiring at the same time! How can we improve upon this? One of the obvious wins is to use local properties like building height, rather than using fixed values from the LCZ classification. We started doing that several years ago in a project called <a href="https://research-software-directory.org/projects/summer-in-the-city">Summer in the City</a>. In that project, we identified properties like building height and “urban fraction” for the Netherlands. Now, in the <a href="https://research-software-directory.org/projects/urban-m4">Urban-M4</a> project, we are adding information on the radiative properties of buildings, by using open street view imagery. At the same time, our project partners at Wageningen University are collecting information on building age and other properties to add better-localized information on things like insulation status.</p><h3>Next steps</h3><p>I also have some ideas regarding the workflow and visualization. If time permits, here are some things I’d like to improve:</p><ul><li>Generalize tile creation: currently the position and dimensions of the buildings are hardcoded based on a tile size of 100x100m. It would be nice to generate them on the fly instead, so we could accomadate various resolutions.</li><li>Randomize the tiles: related to the above, I’ve been thinking a bit on how to add some natural variation to the positioning of buildings. For example by randomly shifting tiles across periodic boundaries.</li><li>Beyond LCZs: a generic tile creation routine could generate tiles based on some key properties like road width, building width, green fraction, et cetera. That would make it possible to vary these parameters independently, rather than coupled through LCZs.</li><li>Combining the workflows for QGIS and CityJSON. Currently they have diverged a bit, but they share many steps. With a bit of work we could streamline and generalize the workflow and add support for other output formats as well.</li><li>Procedural generation: automatically create artificial worlds and gamify urban weather and climate simulation 🙂</li></ul><h3>Concluding remarks</h3><p>When it comes to visualization in weather and climate modelling, we often focus on output data. Here, we introduce an original visualization of the methodology instead. This provides valuable insights into the power and limitations of the LCZ approach. It appeals to the imagination can make our work more captivating.</p><p>What are your thoughs about using visualizations like this? Do you have other ideas, or want to share your own maps made with the Cityblocks package? We have created a <a href="https://github.com/Urban-M4/cityblocks/issues/9">GitHub issue</a> to collect images of cities across the globe. It would be awesome if you would contribute your city as well!</p><p><em>Thanks to Claire Donnely and Ole Mussmann for providing comments on an earlier version, and to Tim Tensen and Maurice de Kleijn for creating the beautiful QGIS map. A chatbot was used to improve the formulation of a few sentences.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b609a340cde4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/amsterdam-as-seen-by-a-weather-model-b609a340cde4">Amsterdam as seen by a weather model</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[(Un)wieldy workflows with WRF]]></title>
            <link>https://medium.com/escience-center/un-wieldy-workflows-with-wrf-4a4947b88a98?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a4947b88a98</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[reproducibility]]></category>
            <category><![CDATA[climate-science]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[workflow]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 07:31:08 GMT</pubDate>
            <atom:updated>2024-09-05T07:31:08.902Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4><strong>Streamlining experiments with the Weather Research and Forecasting model</strong></h4><p>The <a href="https://www.mmm.ucar.edu/models/wrf">Weather Research and Forecasting model</a>, WRF for short, is a beautiful piece of research software with a large community of active users. That’s something we love and advocate for at the eScience Center.</p><p>Given its large user base, one might think this must be an exemplary model to work with. And perhaps it is, in comparison to other models in the scene. But as I’m working with WRF again after several years, I’m reminded of my struggles in keeping my workflows organized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iHKGrQP8dL7wiOTp" /><figcaption>Photo by <a href="https://unsplash.com/@hdbernd?utm_source=medium&amp;utm_medium=referral">Bernd 📷 Dittrich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To try and tackle this issue once and for all, we experimented with different solutions, including a simple shell script, as well as <a href="https://snakemake.readthedocs.io/en/stable/">Snakemake</a>, a python-based workflow management tool often used in bioinformatics. Both approaches turn out to have their own merits and quirks.</p><p>Overall, we did get a better grip on the antipatterns that prevented us from keeping a nice and orderly workflow. From this, we extracted some best practices for managing complex (WRF) modelling workflows. This blogpost documents our journey and the lessons learned, in the hope that it will be useful for other WRF users and those who struggle with managing their experiments.</p><h3><strong>Running basic WRF</strong></h3><p>Let’s start with an overview of the steps required to run a basic WRF (real data) experiment, as per the instructions in the <a href="https://www2.mmm.ucar.edu/wrf/users/wrf_users_guide/build/html/wps.html">user guide</a>. First you must run the WRF preprocessing system (WPS), which itself consists of three programs: geogrid, ungrib, and metgrid. Then you run WRF itself, which also consists of two programs: real and wrf.</p><pre>cd WPS<br><br># Run geogrid (interpolate spatial datasets to the model grid)<br>./geogrid.exe<br><br># Run ungrib (unpack meteorological input data)<br>./link_grib.csh /your/meteo/data/dir<br>ln -s ungrib/Variable_Tables/Vtable.GFS Vtable<br>./ungrib.exe<br><br># Run metgrid (interpolate the meteo data to model grid)<br>./metgrid.exe<br><br>cd ../WRF/run<br><br># Run real (extract initial and boundary fields from preprocessed data)<br>ln -sf ../../WPS/met_em* .<br>./real.exe<br><br># Run WRF (the actual forecast)<br>./wrf.exe</pre><p>This workflow is not great in several ways. WRF and WPS are run in different working directories; yet they “pretend” to be in the same directory by use of symbolic links (ln). Also, exectuables are in the same directory as the input/output data. Now, imagine, for example, that we want to do multiple experiments with slightly different settings.</p><p>One option would be to move the output to a separate directory between each run. We would need to meticulously bookkeep our files, especially since runs may fail at different stages causing us to re-run (part of) the workflow. Also, we would not be able run the experiments in parallel. Clearly, using WRF in this way is a labour-intensive, error-prone process.</p><p>Alternatively, we could duplicate our WPSand WRF/rundirectories. That’s also not great: imagine we want to re-run our experiments with a re-compiled version of WRF. Then we would need to copy over the new executables to all experiment directories.</p><p>Finally, there’s lots of files in these directories that are not needed for our experiments, which makes it hard to tell the wheat from the chaff.</p><h4><strong>Our workflow wishlist</strong></h4><p>Considering the points above, let’s think about how we would like it to be, and start a workflow wishlist:</p><ul><li>Each/all experiment(s) can be re-run with a single command</li><li>One directory per experiment, containing config + output</li><li>Intermediate/output data well organized in subdirectories</li><li>Config files (namelists) specify paths to all relevant input data; no links</li><li>Executables should be in a system path or another dedicated location</li><li>We should be able to (re-)run only part of a workflow</li><li>Bonus: reuse intermediate files that are the same for all experiments</li></ul><h3><strong>Solution #1: using a shell script</strong></h3><p>Let’s see how far we can get with a simple shell script. In this example, we maintain multiple namelists and geogrid tables for each experiment. These are stored alongside the shell script and copied over to the experiment folder once the script is run. The full code is available <a href="https://github.com/Urban-M4/wrf-runner">here</a>.</p><pre># Some initialization of the environment, module loads etc.<br># ...<br><br># Set path to executables<br>WPS_HOME=$HOME/wrf-model/WPS<br>WRF_HOME=$HOME/wrf-model/WRF<br><br># Define experiment name<br>EXP=USGS<br><br># Make new experiment directory<br>export RUNDIR=wrf_experiments/${EXP}<br>mkdir -p $RUNDIR<br>mkdir $RUNDIR/geogrid<br>mkdir $RUNDIR/metgrid<br><br># Copy experiment-dependent files<br>cp namelist.wps_$EXP $RUNDIR/namelist.wps<br>cp namelist.input_$EXP $RUNDIR/namelist.input<br>cp GEOGRID.TBL.ARW_$EXP $RUNDIR/GEOGRID.TBL<br><br># Copy additional input files from WRF/WPS<br>cd $RUNDIR<br>cp $WPS_HOME/metgrid/METGRID.TBL.ARW METGRID.TBL<br>cp $WPS_HOME/ungrid/Variable_Tables/Vtable.ECMWF VTable<br>cp $WRF_HOME/run/CAMtr_volume_mixing_ratio.RCP8.5 CAMtr_volume_mixing_ratio<br>cp $WRF_HOME/run/ozone* .<br>cp $WRF_HOME/run/RRTMG* .<br>cp $WRF_HOME/run/*.TBL .<br><br># Run experiments<br>$WPS_HOME/link_grib.csh &quot;path/to/our/input/data/*&quot;<br>$WPS_HOME/ungrib.exe<br>$WPS_HOME/geogrid.exe<br>grep &quot;ERROR&quot; geogrid.log &amp;&amp; echo &quot;Aborting: ERROR in geogrid.log.&quot; &amp;&amp; exit 1<br>$WPS_HOME/metgrid.exe<br>grep &quot;ERROR&quot; metgrid.log &amp;&amp; echo &quot;Aborting: ERROR in metgrid.log.&quot; &amp;&amp; exit 1<br><br>$WRF_HOME/run/real.exe<br>$WRF_HOME/run/wrf.exe</pre><p>Evidently, WRF and WPS <em>can</em> be run from another directory, by calling them by their full path. Alternatively, we could have added the WPS and WRF/run directories to our PATH variable. However, WRF and WPS expect certain files (tables etc.) in the working directory, so we need to copy those.</p><p>Geogrid and metgrid tables are expected in subfolders, but we can use the namelist setting opt_geogrid_table_path=&#39;.&#39; (default is &#39;./geogrid&#39;) to overwrite that, and similar for metgrid. Geogrid will then look in that folder for a file called GEOGRID.TBL. (Why not simply point to the geogrid table directly in the namelist?)</p><p>We were able to declutter our working directory somewhat by using the namelist.wps options opt_output_from_geogrid_path and similar for metgrid. Unfortunately, that option is not available for ungrib, the worst clutterer of all… We were able to point real.exe to our metgrid subfolder through the namelist.input parameter <a href="https://github.com/wrf-model/WRF/blob/0a11865f97680fdd6865b278ea29d910e5db3ed7/run/README.namelist#L89">auxinput1_inname</a>, so we could skip that linking step as well.</p><p>We wanted the workflow to fail when one of the steps fails. However, metgrid and geogrid <a href="https://github.com/wrf-model/WPS/issues/252#issue-2315032244">don’t fail with a proper exit code</a>. As a simple patch, we added extra commands to look for errors in the log files and raise an exception when needed.</p><p>Despite some peculiarities, this is a good start. We can already tick some of the items off our wishlist. However, the workflow quickly becomes more complex. For example, as we are adding more experiments, we would like to edit the namelists programmatically. And for some runs, we may want to run an intermediate step (e.g. wudapt-to-wrf). Before we know it, the shell script becomes messy, which defeats our purpose of maintaining a tidy and organized workflow.</p><p>To summarize, the following namelist parameters should probably be much better advertised:</p><ul><li>opt_geogrid_table_path (namelist.wps)</li><li>opt_metgrid_table_path (namelist.wps)</li><li>opt_output_from_geogrid_path (namelist.wps)</li><li>opt_output_from_metgrid_path (namelist.wps)</li><li><a href="https://github.com/wrf-model/WRF/blob/0a11865f97680fdd6865b278ea29d910e5db3ed7/run/README.namelist#L89">auxinput1_inname</a> (namelist.input)</li></ul><p>With a similar option for ungrib, the WRF experience would already be much better.</p><h3><strong>Solution #2: using Snakemake</strong></h3><p>Ideally, we want to be able to run only part of a workflow. For example, when real.exe fails, we don’t want to re-run WPS. Similarly, we may want to run all pre-processing first, and then submit WRF as a batch job.</p><p>This is exactly what workflow management tools are for. There are several tools around; we decided to try Snakemake, as it is relatively easy to learn and has the added benefit of being able to use Python code.</p><h4>Brief intro to Snakemake</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BRLzDA3imbZCJnDA160whg.png" /><figcaption>Example Snakemake rules (source <a href="https://snakemake.readthedocs.io/en/stable/project_info/faq.html#what-is-the-key-idea-of-snakemake-workflows">https://snakemake.readthedocs.io/en/stable/project_info/faq.html#what-is-the-key-idea-of-snakemake-workflows</a>)</figcaption></figure><p>A snakemake workflow, aka ‘Snakefile’, consists of rules. Each rule has certain inputs and outputs, and the commands to produce that output, either in Python code (a run block) or as shell commands (a shell block). When you tell snakemake to produce a certain output file, it will check which rule must executed in order to obtain that file. It does so in a smart way: if the required for a rule input already exists, it will not re-run that rule, unless the input is newer than the output.</p><h4>Our desired workflow</h4><p>Before we delve into details, let’s have a look at the workflow that we want to achieve. For demonstration purposes I will focus on WPS only. Snakemake provides a command (snakemake --dag WPS | dot -Tpng &gt; dag.png) to visualize the task graph for our workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSOBOLsjDEF2Iy3ZaQg3eQ.png" /><figcaption>Task graph for our WPS workflow. Boxes (nodes) are rules, arrows indicate dependencies between rules. Dashed lines indicate that these rules have already been executed. Full-line rules still need to be (re)run.</figcaption></figure><p>As you can see, we want to run four independent experiments. While they share all the same steps, they have a different value for the experiment setting. This is achieved through the use of a <em>wildcard</em>.</p><h4>Providing experiment settings</h4><p>The values for the wildcard are provided to snakemake via a configuration file. This is a relatively simple file which, in addition to our experiment-specific settings, contains some global settings like wps_home. Our config file looks like this (only showing the first experiment):</p><pre>wps_home: ~/Urban-M4/WPS<br>wrf_home: ~/Urban-M4/WRF<br>data_home: /projects/0/prjs0914/wrf-data/default<br>working_directory: ./output<br><br>experiments:<br>  usgs:<br>    geog_data_res: [&#39;usgs_30s&#39;,&#39;usgs_30s&#39;,&#39;usgs_30s&#39;,&#39;usgs_30s&#39;]<br>    geog_data_path: &#39;/projects/0/prjs0914/wrf-data/default/static/summerinthecity&#39;<br>    geogrid_table: GEOGRID.TBL.ARW_USGS<br>    use_wudapt_lcz: false<br>    num_land_cat: 24<br>  modis_lcz:<br>    ...</pre><p>The config file is read at the very top of the Snakefile, like so:</p><pre>configfile: workflow.source_path(&quot;config.yaml&quot;)</pre><p>The first rule in our workflow, UPDATE_NAMELIST_WPS, uses a python package <a href="https://f90nml.readthedocs.io/en/latest/">f90nml</a> to load a reference namelist.wps, modify the settings as specified in the experiment configuration, and store the updated namelist in the output folder for that experiment. It looks like this:</p><pre>rule UPDATE_NAMELIST_WPS:<br>    input:<br>        namelist_wps = workflow.source_path(&quot;namelist.wps&quot;)<br>    output:<br>        &quot;{experiment}/namelist.wps&quot;,<br>    run:<br>        # Get parameters from config<br>        geog_data_res = config[&#39;experiments&#39;][wildcards.experiment][&#39;geog_data_res&#39;]<br>        geog_data_path = config[&#39;experiments&#39;][wildcards.experiment][&#39;geog_data_path&#39;]<br><br>        # Read source namelist.wps with f90nml, make some changes, and save in experiment dir<br>        nml = f90nml.read(input.namelist_wps)<br>        nml[&quot;geogrid&quot;][&quot;geog_data_path&quot;] = geog_data_path<br>        nml[&quot;geogrid&quot;][&quot;geog_data_res&quot;] = geog_data_res<br>        nml.write(f&quot;{wildcards.experiment}/namelist.wps&quot;)</pre><p>Notice that we are reading the reference namelist form the workflow source path. This is because it is stored alongside the snakefile. Normally, all paths in snakemake are relative to the working directory.</p><p>Importantly, this rule works for any value of the experiment wildcard (as long as the settings are provided in the configuration file). To invoke the rule for each of our experiments, we use a target rule, like so:</p><pre>rule PRODUCE_ALL_NAMELISTS:<br>    input: collect(&quot;{experiment}/namelist.wps&quot;, experiment=config[&quot;experiments&quot;].keys())</pre><p>The collect function fills the values of experiment with all keys found in config[&quot;experiments&quot;]. When we invoke this target rule, e.g. by running snakemake PRODUCE_ALL_NAMELISTS, Snakemake will look for a rule that can produc the following files (unless they already exist):</p><ul><li>usgs/namelist.wps</li><li>modis_lcz/namelist.wps</li><li>wur/namelist.wps</li><li>wur_lcz/namelist.wps</li></ul><p>It will find UPDATE_NAMELIST_WPSand invoke this rule, inferring that the first segment corresponds to the value of the wildcard.</p><h4>Specifying the other steps</h4><p>Getting Snakemake to use the different experiment settings was by far the most involved part of our journey. Next, we want to add rules for all the other steps. For example, out METGRID rule looks as follows:</p><pre>rule METGRID:<br>    input:<br>        &quot;{experiment}/namelist.wps&quot;,<br>        &quot;{experiment}/finished.ungrib&quot;,<br>        &quot;{experiment}/finished.geogrid&quot;,<br>    output: &quot;{experiment}/finished.metgrid&quot;<br>    shell:<br>        &quot;&quot;&quot;<br>        cd {wildcards.experiment}<br>        mkdir -p metgrid<br><br>        cp {config[wps_home]}/metgrid/METGRID.TBL.ARW METGRID.TBL<br>        {config[wps_home]}/metgrid.exe<br><br>        # Scan logfile for errors and raise if necessary (metgrid can fail silently with 0 exit status)<br>        grep &quot;ERROR&quot; metgrid.log &amp;&amp; echo &quot;Aborting: ERROR in metgrid.log. &quot; &amp;&amp; exit 1<br><br>        touch finished.metgrid<br>        &quot;&quot;&quot;</pre><p>Snakemake infers the dependecies between rules via their input and output filenames. Since WPS output filenames are not predictable without parsing the namelist, we opted for a workaround: at the end of each rule, we create an empty file called finished.&lt;rule&gt;. As such, we inform Snakemake that metgrid must be run after updating the namelist, and after geogrid and ungrib.</p><p>As opposed to the first rule, which executes Python code (a run block), here we use shell commands. This is nice, because it is very similar to what you would do manually on the command line. As an intermediate option we can use params. We use this in our GEOGRID rule:</p><pre>rule GEOGRID:<br>    input: &quot;{experiment}/namelist.wps&quot;,<br>    output: &quot;{experiment}/finished.geogrid&quot;<br>    params: geogrid_table = lambda wildcards: workflow.source_path(config[&#39;experiments&#39;][wildcards.experiment][&#39;geogrid_table&#39;]),<br>    shell:<br>        &quot;&quot;&quot;<br>        cd {wildcards.experiment}<br>        mkdir -p geogrid<br>        cp {params.geogrid_table} GEOGRID.TBL<br><br>        # Run geogrid<br>        {config[wps_home]}/geogrid.exe<br><br>        # Scan logfile for errors and raise if necessary (geogrid can fail silently with 0 exit status)<br>        grep &quot;ERROR&quot; geogrid.log &amp;&amp; echo &quot;Aborting: ERROR in geogrid.log.&quot; &amp;&amp; exit 1<br><br>        touch finished.geogrid<br>        &quot;&quot;&quot;</pre><p>The use of params enables us to execute some Python code to assign a value to a variable, which can then be accessed in the shell block of the rule. Here, we get the geogrid_table from our experiment config. So we can combine the flexibility of Python with the transparancy of plain shell commands.</p><h4>Final workflow</h4><p>Putting everything together, our full workflow looks something like below. Notice that WPS is the target rule in this case.</p><pre>import os<br>from pathlib import Path<br><br>import f90nml<br><br>configfile: workflow.source_path(&quot;config.yaml&quot;)<br>workdir: config[&quot;working_directory&quot;]<br>envvars: &quot;NETCDF&quot;<br><br>rule UPDATE_NAMELIST_WPS:<br>    input:<br>        namelist_wps = workflow.source_path(&quot;namelist.wps&quot;)<br>    output:<br>        &quot;{experiment}/namelist.wps&quot;,<br>    run:<br>        # Get parameters from config<br>        geog_data_res = config[&#39;experiments&#39;][wildcards.experiment][&#39;geog_data_res&#39;]<br>        geog_data_path = config[&#39;experiments&#39;][wildcards.experiment][&#39;geog_data_path&#39;]<br><br>        # Read source namelist.wps with f90nml, make some changes, and save in experiment dir<br>        nml = f90nml.read(input.namelist_wps)<br>        nml[&quot;geogrid&quot;][&quot;geog_data_path&quot;] = geog_data_path<br>        nml[&quot;geogrid&quot;][&quot;geog_data_res&quot;] = geog_data_res<br>        nml.write(f&quot;{wildcards.experiment}/namelist.wps&quot;)<br><br>rule GEOGRID:<br>    input: &quot;{experiment}/namelist.wps&quot;,<br>    output: &quot;{experiment}/finished.geogrid&quot;<br>    params: geogrid_table = lambda wildcards: workflow.source_path(config[&#39;experiments&#39;][wildcards.experiment][&#39;geogrid_table&#39;]),<br>    shell:<br>        &quot;&quot;&quot;<br>        cd {wildcards.experiment}<br>        mkdir -p geogrid<br>        cp {params.geogrid_table} GEOGRID.TBL<br><br>        # Run geogrid<br>        {config[wps_home]}/geogrid.exe<br><br>        # Scan logfile for errors and raise if necessary (geogrid can fail silently with 0 exit status)<br>        grep &quot;ERROR&quot; geogrid.log &amp;&amp; echo &quot;Aborting: ERROR in geogrid.log.&quot; &amp;&amp; exit 1<br><br>        touch finished.geogrid<br>        &quot;&quot;&quot;<br><br>rule UNGRIB:<br>    input: &quot;{experiment}/namelist.wps&quot;<br>    output: &quot;{experiment}/finished.ungrib&quot;<br>    shell:<br>        &quot;&quot;&quot;<br>        cd {wildcards.experiment}<br><br>        # Remove old output if present (ungrib doesn&#39;t like to overwrite stuff)<br>        rm -f FILE*<br><br>        # Link vtable<br>        cp {config[wps_home]}/ungrib/Variable_Tables/Vtable.ECMWF Vtable<br><br>        # Link gribfiles<br>        {config[wps_home]}/link_grib.csh {config[data_home]}/real-time/july2019/*<br><br>        # Run ungrib<br>        {config[wps_home]}/ungrib.exe<br><br>        # Scan logfile for errors and raise if necessary (ungrib can fail silently with 0 exit status)<br>        grep &quot;ERROR&quot; ungrib.log &amp;&amp; echo &quot;Aborting: ERROR in ungrib.log.&quot; &amp;&amp; exit 1<br><br>        # Report ready<br>        touch finished.ungrib<br>         &quot;&quot;&quot;<br><br>rule METGRID:<br>    input:<br>        &quot;{experiment}/namelist.wps&quot;,<br>        &quot;{experiment}/finished.ungrib&quot;,<br>        &quot;{experiment}/finished.geogrid&quot;,<br>    output: &quot;{experiment}/finished.metgrid&quot;<br>    shell:<br>        &quot;&quot;&quot;<br>        cd {wildcards.experiment}<br>        mkdir -p metgrid<br><br>        cp {config[wps_home]}/metgrid/METGRID.TBL.ARW METGRID.TBL<br>        {config[wps_home]}/metgrid.exe<br><br>        # Scan logfile for errors and raise if necessary (metgrid can fail silently with 0 exit status)<br>        grep &quot;ERROR&quot; metgrid.log &amp;&amp; echo &quot;Aborting: ERROR in metgrid.log. &quot; &amp;&amp; exit 1<br><br>        touch finished.metgrid<br>        &quot;&quot;&quot;<br><br>rule WPS:<br>    input: collect(&quot;{experiment}/finished.metgrid&quot;, experiment=config[&quot;experiments&quot;].keys())</pre><p>We’re steadily accumulating lines of code, but overall, we’re quite happy with how interpretable this still looks. As we develop this further, it would be good to split it up into multiple files, and incorporate other best practices as outlined in the <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/best_practices.html">Snakemake documentation</a>.</p><h4>Struggles</h4><p>While we’re quite happy with our progress so far, here are some things that we struggled with:</p><ul><li>Flexibility of output directories: we want to organize our output as follows: ./results/{experiment}. By default, all paths are relative to the working directory. We set the working directory to ./results, so the first rule, UPDATE_NAMELIST, will actually create the file ./results/{experiment}/namelist.wps. Unfortunately, snakemake cannot use wildcards for the working directory. Therefore, we need to cd to the experiment directory in every rule.</li><li>Complex input/output: the output of WPS programs depends on the settings in namelist.wps. Specifically, filenames like met_em.d03.2019... encode information about the domain and (a lot of) dates. We could use something like output: expand(&quot;met_em.{domain}.{datetime}.nc&quot;, domain=DOMAINS, datetime=DATETIMES). For that to work, we’d need to set DOMAINS and DATETIMES in the configuration file, or infer them from the template namelist.wps. Not ideal, and quickly getting complex.</li><li>Atomic inputs/outputs: a bit of a stretch goal, but when using a workflow manager like this, it would be nice if you could do, for example, metgrid.exe --domain=1 --date=20190101. This way, when only one domain/timestamp needs to be re-processed, we don’t need to re-run the workflow for everything else as well. However, this means some of the logic that’s currently encoded in WPS programs, specifically parsing the domain and time control sections of the namelist and looping over them, is moved entirely to the workflow. Do we really want to go down that path?</li><li>Shared input/output: now, we’re saving the output of each experiment in a dedicated directory. However, there is quite some duplication. Instead, we could store WPS output in a shared directory. But what happens if two namelists with different settings create the same file geo_em.d01.nc? We would need to encode the WPS configuration in the path somehow, e.g. geogrid/<strong>usgs</strong>/geo_em.d01.nc and ungrib/<strong>ifs</strong>/FILE:20190101 combine to create metgrid/<strong>usgs/ifs</strong>/met_em.... This is currently not possible since the ungrib directory is not configurable.</li><li>SLURM: snakemake can submit workflows to a batch scheduler like slurm through a plugin. However, we found it to be very cumbersome. For example, we found the main Snakemake program was hanging often and did not seem to check for status; online help often referred to old versions; you can mark jobs for local execution when slurm is the default, but not vice versa as we would have preferred; the working directory was not respected; we had to carefully reinitialize the environment; et cetera. Eventually it seemed easier to wrap the entire Snakemake workflow in a batch script and submit that to slurm, but this is also suboptimal, especially since we cannot differentiate between resources for eacht task then.</li></ul><h4>Verdict</h4><p>Some of these things can probably be solved by digging further into the inner workings of Snakemake. At the same time, we’re conscious of the complexity of our workflow. If it takes more time to build and maintain it than it would take to do the tedious manual bookkeeping, is it really worth the effort?</p><p>Hopefully yes: reproducibility is important, and perhaps others can benefit as well. For now, however, we need to focus on the results for a bit. So let’s do a quick recap and call it a day.</p><p>If you want to follow our progress, keep an eye on our repository <a href="https://github.com/Urban-M4/snakemake-wrf">here</a>.</p><h3>Conclusions</h3><p>We’ve experimented with different options for managing WRF workflows. With a few tweaks, we were able to tackle many of our initial painpoints. Once we got the hang of it, we quite liked the flexibility of Snakemake and its promise of reproducible experiment sets. However, while we aimed for something simple, we found that our solutions quickly grew complex as we tried to bend the tools to our will.</p><p>There’s a few peculiarities about WRF that were very much in the way. For example, that we couldn’t specify the ungrib directory. In terms of compatibility with Snakemake, it would be nice if WRF was less dependent on the current working directory, and/or if Snakemake’s working directory would be able to adapt to wildcard values.</p><p>Perhaps the main conclusion should be that organizing your (WRF) workflows is not as straightforward as it may seem. This is something that many people must struggle with. And while there’s plenty of resources on the WRF documentation and user forum, “how to manage your workflow” does not seem to get the attention it deserves.</p><p>How do you manage your workflows? Please share your tips and tricks in the comments or in any other way that you see fit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a4947b88a98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/un-wieldy-workflows-with-wrf-4a4947b88a98">(Un)wieldy workflows with WRF</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Henry, the City Tweaker]]></title>
            <link>https://medium.com/escience-center/henry-the-city-tweaker-8977b3cce10f?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/8977b3cce10f</guid>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Tue, 21 May 2024 11:39:45 GMT</pubDate>
            <atom:updated>2024-06-21T14:38:10.542Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Imagining the future of urban planning</h4><blockquote>This blogpost is our (<a href="https://www.esciencecenter.nl/news/destination-earth-awards-peter-kalverla-for-his-visionary-proposal-for-the-future/">winning</a> <a href="https://emojipedia.org/party-popper">🎉</a>) submission to Destination Earth’s <a href="https://destination-earth.eu/event/2nd-destine-service-platform-innovation-prize-ideating-the-impact-of-the-desp/">challenge </a>to envision the future use of their platform. The story is set in 2050, when “urban sims” have come to replace traditional city planning tools. Urban sims are like the early century’s sandbox games, say, Cities Skylines, but for real. Design proposals with a net positive impact are eligible for execution, on the condition that they gather sufficient support. The story follows Henry as he experiences the rise of city tweaking firsthand.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d2UZv5N-npijge9C0DoWAA.png" /><figcaption>Mockup of an “urban design interface”, powered by the digital twin engine, that was submitted with the challenge.</figcaption></figure><p>Henry is a city tweaker. He never really pursued his destiny, but in retrospect, it seems all his arbitrary endeavors have been in anticipation of his true calling, which will be revealed to him shortly by an unremarkable lady wearing a white fur hat and a rabbit tattoo.</p><p>His journey starts in the pouring twenties when urban sims are set to democratize urban planning. He makes his debut with an upgrade of the courtyard in his street. Replacing the paved square with a natural playground benefits the local climate, biodiversity, and perceived happiness. The timing is good too: with many new families moving in, there is ample support among the neighbours.</p><p>After this encouraging start, Henry is hooked. As his proficiency grows, people around him start approaching him with their ideas as well. One of his relatives wants to build a sand battery below the communal parking lot, and an acquaintance asks for his help in developing a manure fermentation plant. While urban sims are open to anyone, in practice, by the early 30s, most of the content is created by dedicated individuals, and Henry is your local guru.</p><p>Once he gains enough reputation to earn insider status, he learns to develop mods. This newfound ability draws the attention of companies and entrepreneurs. Mod creators are in high demand, but Henry isn’t in it for the money. He does a few collaborations with crop breeders, adding species to the agriculture module, and works with festival organizers to include an algorithm for acoustic barriers. But he consistently turns down their job offers.</p><p>Instead, he starts participating in so-called “hackathons for the greater good”, organized specifically to tackle grand challenges. He likes the collaborative spirit and the appeal of contributing something substantial, such as the introduction of local distribution networks and a true price concept for public services.</p><p>These are challenging topics indeed. While it is relatively straightforward to understand how design proposals interact with their immediate surroundings, non-local effects are much harder to grasp. Intrigued by the complexity of the problem, Henry spends a big chunk of the 2040s developing an X-AI module to unravel how genuine progress propagates through value chains.</p><p>And so we arrive in 2050 when Henry is startled by the doorbell. “Good morning, Henry.” The friendly voice pauses to let him regain his calm. “I’m looking for a benevolent city tweaker. May I come in?”</p><p>After the usual pleasantries, she turns serious. “Look, Henry. You know that genuine progress comes at the expense of concentrated wealth. You’ve seen it yourself with the distribution networks. Big players with a vested interest in the status quo are seeking to sabotage our system.” She pauses again. “We cannot play cat and mouse. The cat is too clumsy and the mice are too many. We need something of a more ‘<em>viral</em>’ nature. That why we started IMMUNE.”</p><p>“For your long track record of benevolent contributions and proven skill, you will receive an immunity status. You will be granted full privileges and access to a confidential forum. No strings attached. We simply trust that you will continue to serve society.” And with that, she stands up, leaving Henry as startled as she found him.</p><h3>Backstory</h3><p>While the story is completely fictitious, there are a lot of true ideas and concepts that have gone into it. Let me elaborate on some of them.</p><h4>Progress indicators</h4><p>The premise of the story is that “designs with a net positive impact are eligible for execution”. But what is a net positive impact? There are several proposals in the literature on how to evaluate “progress”. Our mockups draw inspiration from the <a href="https://en.wikipedia.org/wiki/Genuine_progress_indicator">Genuine Progress Indicator</a>, and the “<a href="https://doughnuteconomics.org/about-doughnut-economics">economic doughnut</a>”. Both models take many different aspects of well-being into account, including social and environmental well-being.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*atCKoPYy2LoWa1vCrklgJA.png" /><figcaption>Left: the genuine progress indicator (<a href="https://www.recipesforwellbeing.org/genuine-progress-indicator/">source</a>). Right: the economic doughnut (<a href="https://en.wikipedia.org/wiki/Doughnut_%28economic_model%29">source</a>).</figcaption></figure><p>The idea is that the urban sims will evaluate design proposals based on these indicators. Currently, several “local” versions of the doughnut (or rather, the planetary boundaries framework) and the GPI monitor have been published in the scientific literature. Automating this process and making sure that the local scale interacts realistically with the larger scale is probably one of the biggest challenges, both technically and scientifically, in making Henry’s story come true.</p><h4>Democratization</h4><p>Another central element of the story is the democratization of urban planning. At first, it may seem unrealistic to outsource important and complex municipality work to “mere citizens”. On the other hand, we do this all the time with (open source) software. Why not do the same with tangible real estate? Especially since it’s publicly owned.</p><p>In fact, at least in the Netherlands, we already have a process where citizens can apply for building permits with the municipality. It’s dull and bureaucratic, but it is there. Imagine what that could look like if you can simply upload a sketch of your plan as designed by a city builder. The design file would already have all the “technical” details required by the municipality. It would make the request and review process much easier.</p><p>Our submission also included a mockup of a “design proposal page”, where the design is advertised and discussed by the relevant stakeholders (see below). In the story, Henry needs to gather support from the neighbours. I think there is a potential here to address the lack of participation and, consequently, lack of support that we see for example with renewable energy projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cuDleaG9MZcfUpqZvJCjfg.png" /><figcaption>Mockup of “design proposal page” that could be used to advertise urban design proposals, discuss them, and gather support.</figcaption></figure><p>By the way, this design was inspired by <a href="https://www.figma.com/community">Figma</a>’s community pages.</p><h4>Gamification, or rather “sciencification”</h4><p>The urban sims in the story take the idea of gamification to the extreme. Games are optimized for a great user experience, and gamification offers immense potential in making science more accessible and engaging to society.</p><p>Recently, game developers have actually been pursuing similar ideas. For example, check out <a href="https://play.eco/">https://play.eco/</a>, which aims to be a “forever game, growing in depth and breath and […] go beyond entertainment.” Perhaps we should be talking about “sciencifying” existing games instead. I would love to collaborate more with game designers.</p><p><em>On a side note: IMHO science teams could be so much stronger if academia embraced more diverse function profiles. Why not employ dedicated communication specialists (YouTubers / content creators), research software engineers, or game developers in every team/group/department?</em></p><h3>Conclusion</h3><p>Imagination is a key driver of innovation and progress. Lifting the constraints of immediate feasibility inspires us to think big and let our creativity guide us. It helps to break away from doing more of the same and instead direct our research and energy towards an inspiring future.</p><p>Participating in the Destination Earth challenge was a very enjoyable experience, and it would be great to see more such initiatives. In the meantime, I will be on the lookout for opportunities to help make Henry’s story come true.</p><p>What do you think are the biggest challenges in realizing these urban sims? What would you add, or change, and what opportunities did I miss? Perhaps we can start shaping our future together!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8977b3cce10f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/henry-the-city-tweaker-8977b3cce10f">Henry, the City Tweaker</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cracking the (moisture) tracking code]]></title>
            <link>https://medium.com/escience-center/cracking-the-moisture-tracking-code-5849b0001911?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/5849b0001911</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[precipitation]]></category>
            <category><![CDATA[climate-modelling]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Tue, 21 Nov 2023 08:02:41 GMT</pubDate>
            <atom:updated>2023-11-21T08:02:41.112Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>An exclusive interview with our project partners</h4><p>This post is part of a series on our 2021 Small Scale Initiative Call in Software Performance Optimization. This week’s highlight: “<a href="https://research-software-directory.org/projects/optimizing-tracking-of-moisture-in-the-atmosphere">Optimizing tracking of moisture in the atmosphere</a>”.</p><p>What is moisture tracking? Can Python deliver optimal performance? What’s it like to have others modify your code? How can a team avoid alienation in coding? Will the output of this project be part of CMIP? Find out in this exclusive interview with project team members <a href="https://www.linkedin.com/in/imme-benedict-abab09b8">Imme Benedict</a>, <a href="https://www.linkedin.com/in/ruud-van-der-ent-79142b11">Ruud van der Ent</a>, <a href="https://www.linkedin.com/in/chris-weijenborg-17ab1431/">Chris Weijenborg</a>, and Peter Kalverla!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0y5Y-9ERky8W4YN_Vp74Q.png" /><figcaption>Project team. Imme and Ruud, Peter, Chris and Yang (top-to-bottom, left-to-right).</figcaption></figure><h4><strong>Chris, you’re an expert in atmospheric dynamics. Can you explain what moisture tracking is, and why it is useful?</strong></h4><p><em>Chris</em>: The essential question you want to answer with moisture tracking is where moisture leading to precipitation in a certain area is coming from. This can be related to a particular event, think about for example the floods in West-Europe in 2021, but also related to the climatological precipitation. This is useful, since it gives us other insights then just looking at precipitation and or wind fields directly. For example, if the sea surface temperature is increasing in a warmer climate, does this lead to an increased climatological precipitation at another location? Or are the Floods in 2021 caused by increased sea surface temperatures in the Mediterranean Sea or the increased soil moisture over the continent? You might wonder now how all of this works: it’s complicated, but we have a nice animation to show the basics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*tRgidFHb_dAPvjqAlPDmow.gif" /><figcaption>Animation of a moisture-tracking experiment. The video first runs forward, then reversed. On the left, you see the total water content in the atmosphere, on the right only the part that contributed to the 2021 floods over the Eiffel and the Ardennes.</figcaption></figure><h4><strong>Imme, you were the main applicant for this project. What was your original goal with the proposal?</strong></h4><p><em>Imme</em>: I had been working in my PhD with the <a href="https://research-software-directory.org/software/wam2layers">WAM2layers model</a> and also adapted parts of the code. When I got a position as lecturer and researcher, I was keen on continuing the work on moisture tracking, and one way was to work with MSc thesis projects. However, the tool became a bit outdated as it was written to use <a href="https://climatedataguide.ucar.edu/climate-data/era-interim">ERA-Interim dataset</a> as input, while for most recent events only the newest dataset ERA5 was available, as the ERA-Interim dataset was phased out. Therefore,</p><blockquote>I became hesitant to use moisture tracking in my own and students research, which I thought was a shame.</blockquote><p>Then I wrote this proposal to update and speed-up the code such that researchers new in the field (like students) could easily start to use it.</p><h4><strong>Ruud, as the original developer of WAM2layers you published your code many years ago. What were your expectations back then? And how do you look back on that today?</strong></h4><p><em>Ruud</em>: Initially I just wanted to archive the code, so other researchers would more easily be able to find it. I naively thought of the code as something quite static. Since then, demand for moisture tracking in various fields of application has increased enormously and so has the importance to develop and maintain the code. I hope that with the user community around WAM2layers we can keep the software up to date with the latest developments and relevant for several years to come.</p><h4><strong>Peter, you have a background in meteorology @ Wageningen University. What was it like to go back to your roots in this project?</strong></h4><p><em>Peter</em>: Indeed, this project was right up my alley. I benefited a lot from my meteorology background and I enjoyed delving into some of the equations I’d almost forgotten about. I also liked the collaboration in this project. It felt like a real team effort, where everyone brought their own unique expertise.</p><blockquote>When I left Wageningen to join the eScience Center, I wanted to improve my own research software engineering skills so I could apply them to the benefit of academic research. I think that really worked out well with this project.</blockquote><h4><strong>Now that we’re past the introductions, let’s take a deeper dive into the project. You mentioned the code was too slow to work with high-resolution data like ERA5. Did you solve this problem, and how?</strong></h4><p><em>Peter</em>: Yes, we did. The easiest win was changing the code from Python2 to Python3, and then some relatively straightforward changes, for example better use of numpy’s broadcasting capabilities.</p><p>The main bottleneck, however, was the memory use. The model was using a nested time loop, where the outer loop was over the input files, and the inner loop over the hourly data interpolated to an even finer time step (see below). All the interpolated time steps were loaded in memory at once. The upgraded code runs a single loop over the fine-grained model time steps, where the data is interpolated on the fly and only the two neighboring (hourly) time steps are loaded in memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*lrvYhupwEPIsCPW4zWXOTQ.png" /><figcaption>Illustration of the time steps in the input data (above) and in the model time loop (below).</figcaption></figure><h4><strong>WAM2layers was originally written in Matlab, then translated to Python. Wasn’t it time for another switch, especially considering Python’s reputation regarding performance?</strong></h4><p><em>Peter</em>: We considered it. I’ve been experimenting a bit with Julia in the past and it seems like a suitable alternative. However, we wanted to make sure that the code was still understandable for scientists or students: even if they are not actively changing the code, it’s good if they can understand what’s going on. Python still is by far the most widely known language in the field.</p><p>Moreover, Python is fast enough for now. For most operations we use highly optimized numpy functions. If we wanted to squeeze out even more we could still look into numba, for example. Having said that, some of the changes we made to the code structure <em>do</em> actually make it easier to replace expensive operations with optimized code in another language, should we ever wish to do that.</p><h4><strong>In the end, you did more than make it faster. Can you elaborate on some of the unexpected changes?</strong></h4><p><em>Peter</em>: yes, one example is that we made it easier to (pip) install the model and use it from the command line.</p><p><em>Ruud</em>:</p><blockquote>People can now get the software working in a matter of minutes. One does not need to be an expert anymore, which is a great development.</blockquote><p>However, when people do research with it, I still hope that they involve someone from the core development team as there might be some tricky devils in the details.</p><h4><strong>In retrospect, what are the main things you got out of this project?</strong></h4><p><em>Ruud</em>: Personally, I really enjoyed two things: 1) working in a horizontal team, and 2) blocking time in my agenda to do some coding.</p><p><em>Chris</em>: I really enjoyed learning something new again, moreover in the meanwhile I also was ‘forced’ to learn version control with Git.</p><p><em>Imme</em>: Teamwork, and learning Git! But also the possibility to have students working with the tool again.</p><h4><strong>Can you elaborate on the student projects?</strong></h4><p><em>Ruud &amp; Imme</em>: Sure! Below are the results from 4 student projects: Wenyu looked at <a href="http://resolver.tudelft.nl/uuid:28f594b4-b24f-4aec-b01c-662e777ca971">monsoon moisture source</a> for the Yangtze River Basin, Rens studied <a href="http://resolver.tudelft.nl/uuid:4e8cfdc1-350f-403f-aa7e-e76b55f5a800">trends in moisture sources</a> related to rising lakes in Keny, Emma studied the <a href="http://resolver.tudelft.nl/uuid:7185c2b7-8313-4dc9-8a10-f57a7cd5ca35">drivers of rainfall variability</a> in Surinam, and Vincent used WAM2layers to study the moisture sources over the Amazon region during August 2023 when a big field campaign took place to understand biosphere-atmosphere interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XwO3DLM_gWunSvF2NgdHtw.png" /><figcaption>WAM2layers output as produced and visualized in students’ projects. Shading illustrates the moisture sources for the each of the case studies.</figcaption></figure><h4><strong>Recently a follow-up project was granted. So this story will continue?</strong></h4><p><em>Ruud</em>: Absolutely, with the NWO Open Science project and our own time we want to make WAM2layers even more user-friendly, increase the number of features, applicability to climate models, and get a better grip on the uncertainties. To this end we will also engage with the wider moisture tracking community.</p><h4><strong>Will CMIP7 include an “Atmospheric moisture tracking-MIP” in the future?</strong></h4><p><em>Imme</em>: Ha, that would be amazing of course, but realistically I think there are still some steps in between before such a big goal can be achieved. We are currently initiating a moisture tracking intercomparison project, where multiple moisture tracking models will run the same case to address uncertainty of moisture sources between models. This project will be supported by a workshop at the Lorentz centre in Leiden in May 2024.</p><p><em>This post is part of a series on our 2021 Small Scale Initiative Call in Software Performance Optimization. See also our post on the project </em><a href="https://blog.esciencecenter.nl/minds-for-mobile-agents-speeding-up-social-distancing-simulations-62d3fb412ff6"><em>Minds for Mobile Agents</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5849b0001911" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/cracking-the-moisture-tracking-code-5849b0001911">Cracking the (moisture) tracking code</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysis-ready climate data with ESMValCore]]></title>
            <link>https://medium.com/escience-center/easy-ipcc-powered-by-esmvalcore-19a0b6366ea7?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/19a0b6366ea7</guid>
            <category><![CDATA[ipcc]]></category>
            <category><![CDATA[climate-models]]></category>
            <category><![CDATA[research-software]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[climate-science]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Wed, 27 Sep 2023 12:05:41 GMT</pubDate>
            <atom:updated>2023-10-05T08:34:53.782Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4><strong>The “Easy IPCC” blog done differently</strong></h4><p>Co-authored by <a href="https://www.esciencecenter.nl/team/bouwe-andela-msc/">Bouwe Andela</a></p><p>In a <a href="https://medium.com/pangeo/easy-ipcc-part-1-multi-model-datatree-469b87cf9114">recent blog post</a>, Tom Nicholas presented a nice walkthrough of reproducing a famous IPCC figure with <a href="https://xarray-datatree.readthedocs.io/en/latest/">xarray-datatree</a>. Here, we attempt to achieve the same with ESMValCore — the workflow engine that powers the more widely known <a href="http://docs.esmvaltool.org/">ESMValTool</a>. Both xarray-datatree and ESMValCore facilitate the analysis of large climate datasets, but our approach is quite different. For example, xarray-datatree tries to facilitate any hierarchical file structure (more generic), whereas ESMValCore focuses on facilitating CMIP, or CMIP-like data only (more specific, thus tailored). Seeing them side by side is food for thought.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jd1u3gy3--n3UTW0Cy9FNA.png" /><figcaption><em>“</em><a href="https://github.com/zarr-developers/zarr-illustrations-falk-2022/blob/main/1200dpi/kits-deluge-1200dpi.png"><em>Adam uploads</em></a><em>” by </em><a href="https://twitter.com/DrHenningFalk"><em>Henning Falk</em></a><em>, ©2022 </em><a href="https://numfocus.org/"><em>NumFOCUS</em></a><em>, is used under a </em><a href="https://creativecommons.org/licenses/by/4.0/"><em>CC BY 4.0</em></a><em> license.</em></figcaption></figure><p>In case you are not familiar: <a href="https://research-software-directory.org/software/esmvaltool">ESMValTool</a> is a software project that was designed by and for climate scientists to evaluate CMIP data in a standardized and reproducible manner (check out a recent blog <a href="https://blog.esciencecenter.nl/esmvaltool-recipes-for-solid-climate-science-da5b33814f69?source=friends_link&amp;sk=c6510256a10d2a35e7e9a723ec4ae0fb">here</a>). Several years ago, the project was split in two parts: <a href="https://research-software-directory.org/software/esmvalcore">ESMValCore</a> implements the core functionality, whereas ESMValTool is essentially a big collection of “recipes” to reproduce a wide range of CMIP analytics — including some of the IPCC figures. While ESMValCore has matured as a reliable foundation for the ESMValTool, recent additions also make it attractive as a lightweight alternative to its hefty sibling.</p><p>To give you an impression of ESMValCore’s look and feel, we start with a complete code block that exactly reproduces Tom’s figure. In the remainder of this post, we’ll dive into the details: the dataset interface, automatic ESGF downloads, CMOR checks and fixes, and exporting as recipe, meanwhile touching upon some of the similarities and differences with xarray-datatree. At the end of the blogpost we’ll build an ESMValTool recipe that reproduces the same figure with all available models and ensemble members.</p><pre>import cf_units<br>import matplotlib.pyplot as plt<br>from iris import quickplot<br><br>from esmvalcore.config import CFG<br>from esmvalcore.dataset import Dataset<br>from esmvalcore.preprocessor import annual_statistics, anomalies, area_statistics<br><br><br># Settings for automatic ESGF search<br>CFG[&#39;search_esgf&#39;] = &#39;when_missing&#39;<br><br># Declare common dataset facets<br>template = Dataset(<br>    short_name=&#39;tos&#39;,<br>    mip=&#39;Omon&#39;,<br>    project=&#39;CMIP6&#39;,<br>    exp= &#39;*&#39;, # We&#39;ll fill this below<br>    dataset=&#39;*&#39;,  # We&#39;ll fill this below<br>    ensemble=&#39;r4i1p1f1&#39;,<br>    grid=&#39;gn&#39;,<br>)<br><br># Substitute data sources and experiments<br>datasets = []<br>for dataset_id in [&quot;CESM2&quot;, &quot;MPI-ESM1-2-LR&quot;, &quot;IPSL-CM6A-LR&quot;]:<br>    for experiment_id in [&#39;ssp126&#39;, &#39;ssp585&#39;]:<br>        dataset = template.copy(dataset=dataset_id, exp=[&#39;historical&#39;, experiment_id])<br>        dataset.add_supplementary(short_name=&#39;areacello&#39;, mip=&#39;Ofx&#39;, exp=&#39;historical&#39;)<br>        dataset.augment_facets()<br>        datasets.append(dataset)<br><br># Set the reference period for anomalies <br>reference_period = {<br>    &quot;start_year&quot;: 1950, &quot;start_month&quot;: 1, &quot;start_day&quot;: 1,<br>    &quot;end_year&quot;: 1979, &quot;end_month&quot;: 12, &quot;end_day&quot;: 31,<br>}<br><br># (Down)load, pre-process, and plot the cubes<br>for dataset in datasets: <br>    cube = dataset.load()<br>    cube = area_statistics(cube, operator=&#39;mean&#39;)<br>    cube = anomalies(cube, reference=reference_period, period=&#39;month&#39;)  # notice &#39;month&#39;<br>    cube = annual_statistics(cube, operator=&#39;mean&#39;)<br>    cube.convert_units(&#39;degrees_C&#39;)<br><br>    # Make sure all datasets use the same calendar for plotting<br>    tcoord = cube.coord(&#39;time&#39;)<br>    tcoord.units = cf_units.Unit(tcoord.units.origin, calendar=&#39;gregorian&#39;)<br><br>    # Plot<br>    quickplot.plot(cube, label=f&quot;{dataset[&#39;dataset&#39;]} - {dataset[&#39;exp&#39;]}&quot;)<br><br># Show the plot<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*BG2FHkgdqECTHZ06eXwC5g.png" /><figcaption>The famous IPCC figure from Tom’s blogpost reproduced with ESMValCore.</figcaption></figure><p>Cool! This looks exactly like the figure in Tom’s post 🎉. With roughly the same amount of code. If you haven’t read Tom’s post, this figure shows the historical and projected rise in sea-surface temperature from different climate models and future emission scenario’s.</p><h4><strong>Key takeaway </strong>🔑</h4><p>If there is one thing to note from this little script, it is how “scientifically meaningful” it is. For example, we never got to see actual files. We simply defined our datasets with meaningful <em>facets</em>, as described in the CMIP <a href="https://github.com/WCRP-CMIP/CMIP6_CVs">controlled vocabulary</a>. In the background, ESMValCore located the corresponding files on ESGF, downloaded them to our local environment (if we didn’t have them yet), loaded them into memory, checked adherence to the CMOR conventions, and applied fixes for known issues. But those are technical details, not science.</p><p>ESMValCore, in the spirit of ESMValTool, focuses very much on <em>what</em> to do, instead of <em>how</em>.</p><h3><strong>The inner workings </strong>⚙️</h3><p>It’s worth dissecting <em>how </em>ESMValCore does things, though. Firstly because you’ll need some understanding to wield it effectively. But mostly because we scientists are often trying to solve the same problems, so it’s interesting to see how ESMValCore tackles some of these issues so you don’t have to.</p><h4>Configuration</h4><p>ESMValCore works with a configuration file. This is where you store information about, for example, how your data is organized on disk. The defaults are pretty good, we only had to enable automatic downloads. Let’s have a look at some of the other settings.</p><pre>print({key: CFG[key] <br>       for key in [&#39;download_dir&#39;, &#39;drs&#39;, &#39;output_dir&#39;, &#39;rootpath&#39;]<br>})</pre><pre>{<br>    &#39;download_dir&#39;: PosixPath(&#39;/home/peter/climate_data&#39;),<br>    &#39;drs&#39;: {&#39;CMIP3&#39;: &#39;ESGF&#39;, &#39;CMIP5&#39;: &#39;ESGF&#39;, &#39;CMIP6&#39;: &#39;ESGF&#39;, &#39;CORDEX&#39;: &#39;ESGF&#39;, &#39;obs4MIPs&#39;: &#39;ESGF&#39;},<br>    &#39;output_dir&#39;: PosixPath(&#39;/home/peter/esmvaltool_output&#39;),<br>    &#39;rootpath&#39;: {&#39;default&#39;: [PosixPath(&#39;/home/peter/climate_data&#39;)]}<br>}</pre><p>The default configuration specifies that data will be downloaded to a folder called climate data in my home directory. The rootpath setting specifies all folders where ESMValCore should look for data, and it is set to the same default. Finally, ESMValCore defines a dedicated folder to store output. Every ESMValTool run or session will get its own subdirectory in there.</p><p>The key point here is that you don’t have to worry about data management. You can configure some folders if you want to, but even that is not strictly necessary. If you’re working on dedicated climate data infrastructure such as Jasmin or DKRZ, you can also configure ESMValCore to use the Data Reference Syntax (DRS, the filenames + folder structure) used on these machines.</p><h4><strong>Data discovery </strong>🔍</h4><p>Let’s have a closer look at the dataset definition. What happened, for example, when we called augment_facets?</p><pre>dataset = Dataset(<br>    short_name=&#39;tos&#39;,<br>    mip=&#39;Omon&#39;,<br>    project=&#39;CMIP6&#39;,<br>    exp=&#39;historical&#39;,<br>    dataset=&#39;CESM2&#39;,<br>    ensemble=&#39;r4i1p1f1&#39;,<br>    grid=&#39;gn&#39;,<br>)<br>dataset.augment_facets()<br>print(dataset)</pre><pre>Dataset:<br>{&#39;dataset&#39;: &#39;CESM2&#39;,<br> &#39;project&#39;: &#39;CMIP6&#39;,<br> &#39;mip&#39;: &#39;Omon&#39;,<br> &#39;short_name&#39;: &#39;tos&#39;,<br> &#39;activity&#39;: &#39;CMIP&#39;,<br> &#39;ensemble&#39;: &#39;r4i1p1f1&#39;,<br> &#39;exp&#39;: &#39;historical&#39;,<br> &#39;frequency&#39;: &#39;mon&#39;,<br> &#39;grid&#39;: &#39;gn&#39;,<br> &#39;institute&#39;: [&#39;NCAR&#39;],<br> &#39;long_name&#39;: &#39;Sea Surface Temperature&#39;,<br> &#39;modeling_realm&#39;: [&#39;ocean&#39;],<br> &#39;original_short_name&#39;: &#39;tos&#39;,<br> &#39;standard_name&#39;: &#39;sea_surface_temperature&#39;,<br> &#39;units&#39;: &#39;degC&#39;}<br>session: &#39;session-405b0393-0e28-4dae-836e-0d5961e3d30b_20230609_084132&#39;</pre><p>ESMValCore auto-completed our Dataset based on the information it got from the <a href="https://github.com/PCMDI/cmip6-cmor-tables">CMOR tables</a>. With this specification, ESMValCore can automatically search both your local filesystem and the ESGF for available copies of the dataset. Let’s make things a bit more interesting, and look for all available ensemble members. The from_files() method is used to build a list of available dataset based on the search fields.</p><pre>CFG[&#39;search_esgf&#39;] = &#39;always&#39;<br>dataset_search = Dataset(<br>    short_name=&#39;tos&#39;,<br>    mip=&#39;Omon&#39;,<br>    project=&#39;CMIP6&#39;,<br>    exp=&#39;historical&#39;,<br>    dataset=&#39;CESM2&#39;,<br>    ensemble=&#39;*&#39;,<br>    grid=&#39;gn&#39;,<br>)<br>ensemble_datasets = list(dataset_search.from_files())<br><br>print([ds[&#39;ensemble&#39;] for dataset in ensemble_datasets])</pre><pre>[<br>    &#39;r10i1p1f1&#39;,<br>    &#39;r11i1p1f1&#39;,<br>    &#39;r1i1p1f1&#39;,<br>    &#39;r2i1p1f1&#39;,<br>    &#39;r3i1p1f1&#39;,<br>    &#39;r4i1p1f1&#39;,<br>    &#39;r5i1p1f1&#39;,<br>    &#39;r6i1p1f1&#39;,<br>    &#39;r7i1p1f1&#39;,<br>    &#39;r8i1p1f1&#39;,<br>    &#39;r9i1p1f1&#39;<br>]</pre><p>Apparently there’s 11 ensemble members available for this model.</p><p>The wildcard search functionality ( &#39;*&#39; ) is incredibly useful, as there are often inconsistencies in availability between datasets. Asking for ensemble member 1 through 11 may work for CESM2 today, but not for another model, or it may be different tomorrow.</p><p>What if we wanted to plot <em>all </em>available data?</p><pre>CFG[&#39;search_esgf&#39;] = &#39;always&#39;<br>dataset_search= Dataset(<br>    short_name=&#39;tos&#39;,<br>    mip=&#39;Omon&#39;,<br>    project=&#39;CMIP6&#39;,<br>    exp=[&#39;historical&#39;, &#39;ssp585&#39;],<br>    dataset=&#39;*&#39;,<br>    ensemble=&#39;*&#39;,<br>    institute=&#39;*&#39;,<br>    grid=&#39;gn&#39;,<br>)<br>all_of_cmip = list(dataset_search.from_files())<br>print(len(all_of_cmip))</pre><pre>821</pre><p>That’s a lot of datasets! We’ll save that for the end of the post.</p><h4><strong>Adding supplementary information</strong></h4><p>To arrive at global mean temperature, we needed to calculate an area-weighted average. The data for ocean temperature comes at the native model grid, which is irregular and the grid cell area is not easily obtained. To address exactly this issue, the CF conventions include a specification of <a href="http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch07s02.html">cell measures</a>. ESMValCore supports this functionality and makes it easy to add cell measures to a dataset. In our case, we need the variable called areacello.</p><pre># Discard augmented facets as they will be different for areacello<br>dataset = Dataset(**dataset.minimal_facets)<br><br># Add areacello as supplementary dataset<br>dataset.add_supplementary(short_name=&#39;areacello&#39;, mip=&#39;Ofx&#39;)<br><br># Autocomplete and inspect<br>dataset.augment_facets()<br>print(dataset.summary())</pre><pre>Dataset:<br>{&#39;dataset&#39;: &#39;CESM2&#39;,<br> &#39;project&#39;: &#39;CMIP6&#39;,<br> &#39;mip&#39;: &#39;Omon&#39;,<br> &#39;short_name&#39;: &#39;tos&#39;,<br> &#39;activity&#39;: &#39;CMIP&#39;,<br> &#39;ensemble&#39;: &#39;r4i1p1f1&#39;,<br> &#39;exp&#39;: &#39;historical&#39;,<br> &#39;frequency&#39;: &#39;mon&#39;,<br> &#39;grid&#39;: &#39;gn&#39;,<br> &#39;institute&#39;: [&#39;NCAR&#39;],<br> &#39;long_name&#39;: &#39;Sea Surface Temperature&#39;,<br> &#39;modeling_realm&#39;: [&#39;ocean&#39;],<br> &#39;original_short_name&#39;: &#39;tos&#39;,<br> &#39;standard_name&#39;: &#39;sea_surface_temperature&#39;,<br> &#39;units&#39;: &#39;degC&#39;}<br>supplementaries:<br>  {&#39;dataset&#39;: &#39;CESM2&#39;,<br>   &#39;project&#39;: &#39;CMIP6&#39;,<br>   &#39;mip&#39;: &#39;Ofx&#39;,<br>   &#39;short_name&#39;: &#39;areacello&#39;,<br>   &#39;activity&#39;: &#39;CMIP&#39;,<br>   &#39;ensemble&#39;: &#39;r4i1p1f1&#39;,<br>   &#39;exp&#39;: &#39;historical&#39;,<br>   &#39;frequency&#39;: &#39;fx&#39;,<br>   &#39;grid&#39;: &#39;gn&#39;,<br>   &#39;institute&#39;: [&#39;NCAR&#39;],<br>   &#39;long_name&#39;: &#39;Grid-Cell Area for Ocean Variables&#39;,<br>   &#39;modeling_realm&#39;: [&#39;ocean&#39;],<br>   &#39;original_short_name&#39;: &#39;areacello&#39;,<br>   &#39;standard_name&#39;: &#39;cell_area&#39;,<br>   &#39;units&#39;: &#39;m2&#39;}<br>session: &#39;session-ccace7d0-dc3b-4e03-87ea-2716aa127618_20230609_085322&#39;</pre><p>These ancillary datasets are not always consistently available for each variant of a dataset. ESMValCore includes some convenience functionality that tries to find a matching variable if the exact match is not available. For example, it may take the grid cell area from another ensemble member. In this case, it looks like we have an exact match.</p><h4><strong>Data (down)loading</strong></h4><p>We now arrive at the part of the script that calls dataset.load(). In the background, ESMValCore automatically located the files on ESGF and downloaded them for us. Let’s illustrate this.</p><pre># Before load<br>print(dataset.files)</pre><pre>[<br>    ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r4i1p1f1/Omon/tos/gn/v20190308/tos_Omon_CESM2_historical_r4i1p1f1_gn_<br>185001-201412.nc on hosts [&#39;aims3.llnl.gov&#39;, &#39;esgf-data.ucar.edu&#39;, &#39;esgf-data04.diasjp.net&#39;, &#39;esgf.ceda.ac.uk&#39;, <br>&#39;esgf3.dkrz.de&#39;]<br>]</pre><pre># After load (and re-defining the dataset to trigger a new search)<br>print(dataset.files)</pre><pre>[LocalFile(&#39;/home/peter/climate_data/CMIP6/CMIP/NCAR/CESM2/historical/r4i1p1f1/Omon/tos/gn/v20190308/tos_Omon_CESM2_historical_r4i1p1f1_gn_185001-201412.nc&#39;)]</pre><p>Notice that there were copies of this file available on 5 different ESGF nodes. ESMValCore automatically selects the fastest available server and stored the dataset according to the standard CMOR data reference syntax.</p><p>For our example dataset, all the data is apparently contained within one file. However, it’s not always like this. If we look, for example, at another ensemble member, we can see that it is stored in chunks of 50 years:</p><pre>print(ensemble_datasets[1].files)</pre><pre>[ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_185001-189912.nc on hosts [&#39;aims3.llnl.gov&#39;, &#39;esgf-data.ucar.edu&#39;, &#39;esgf-data04.diasjp.net&#39;, &#39;esgf.ceda.ac.uk&#39;, &#39;esgf3.dkrz.de&#39;],<br> ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_190001-194912.nc on hosts [&#39;aims3.llnl.gov&#39;, &#39;esgf-data.ucar.edu&#39;, &#39;esgf-data04.diasjp.net&#39;, &#39;esgf.ceda.ac.uk&#39;, &#39;esgf3.dkrz.de&#39;],<br> ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_195001-199912.nc on hosts [&#39;aims3.llnl.gov&#39;, &#39;esgf-data.ucar.edu&#39;, &#39;esgf-data04.diasjp.net&#39;, &#39;esgf.ceda.ac.uk&#39;, &#39;esgf3.dkrz.de&#39;],<br> ESGFFile:CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Omon/tos/gn/v20190313/tos_Omon_CESM2_historical_r10i1p1f1_gn_200001-201412.nc on hosts [&#39;aims3.llnl.gov&#39;, &#39;esgf-data.ucar.edu&#39;, &#39;esgf-data04.diasjp.net&#39;, &#39;esgf.ceda.ac.uk&#39;, &#39;esgf3.dkrz.de&#39;]]</pre><p>In this case, ESMValCore automatically concatenates the data for us.</p><p>You may have noticed that we assigned the result of the load method to a new variable called cube:</p><pre>cube = dataset.load()</pre><p>Under the hood, ESMValCore uses <a href="https://scitools-iris.readthedocs.io/">iris</a> to represent data cubes. Compared to xarray, iris is more strict on checking adherence to CF conventions. This can make it harder to work with, but it also acts as a safeguard against unexpected behaviour.</p><p>Where Tom used <a href="https://cmip6-preprocessing.readthedocs.io/">xMIP</a>, ESMValCore comes with builtin checks and if there are any (known) issues with the datasets, it automatically fixes them. Also, it automatically loads all supplementary datasets. Notice how the cell area is present as a cell measure.</p><pre>print(cube)</pre><pre>sea_surface_temperature / (degC)          (time: 1980; cell index along second dimension: 384; cell index along first dimension: 320)<br>    Dimension coordinates:<br>        time                                   x                                        -                          -<br>        cell index along second dimension      -                                        x                          -<br>        cell index along first dimension       -                                        -                          x<br>    Auxiliary coordinates:<br>        latitude                               -                                        x                          x<br>        longitude                              -                                        x                          x<br>    Cell measures:<br>        cell_area                              -                                        x                          x<br>    Cell methods:<br>        mean where sea                    area<br>        mean                              time<br>    Attributes:<br>        Conventions                       &#39;CF-1.7 CMIP-6.2&#39;<br>        activity_id                       &#39;CMIP&#39;<br>        branch_method                     &#39;standard&#39;<br>        branch_time_in_child              0<br>        branch_time_in_parent             -492385<br>        case_id                           &#39;18&#39;<br>        cesm_casename                     &#39;b.e21.BHIST.f09_g17.CMIP6-historical.004&#39;<br>        contact                           &#39;cesm_cmip6@ucar.edu&#39;<br>        data_specs_version                &#39;01.00.29&#39;<br>        description                       &#39;This may differ from &quot;surface temperature&quot; in regions of sea ice or floating ...&#39;<br>        experiment                        &#39;all-forcing simulation of the recent past&#39;<br>        experiment_id                     &#39;historical&#39;<br>        external_variables                &#39;areacello&#39;<br>        forcing_index                     1<br>        frequency                         &#39;mon&#39;<br>        further_info_url                  &#39;https://furtherinfo.es-doc.org/CMIP6.NCAR.CESM2.historical.none.r4i1p1 ...&#39;<br>        grid                              &#39;native gx1v7 displaced pole grid (384x320 latxlon)&#39;<br>        grid_label                        &#39;gn&#39;<br>        id                                &#39;tos&#39;<br>        initialization_index              1<br>        institution                       &#39;National Center for Atmospheric Research, Climate and Global Dynamics Laboratory, ...&#39;<br>        institution_id                    &#39;NCAR&#39;<br>        license                           &#39;CMIP6 model data produced by &lt;The National Center for Atmospheric Research&gt; ...&#39;<br>        mipTable                          &#39;Omon&#39;<br>        mip_era                           &#39;CMIP6&#39;<br>        model_doi_url                     &#39;https://doi.org/10.5065/D67H1H0V&#39;<br>        nominal_resolution                &#39;100 km&#39;<br>        out_name                          &#39;tos&#39;<br>        parent_activity_id                &#39;CMIP&#39;<br>        parent_experiment_id              &#39;piControl&#39;<br>        parent_mip_era                    &#39;CMIP6&#39;<br>        parent_source_id                  &#39;CESM2&#39;<br>        parent_time_units                 &#39;days since 1850-1-1 00:00:00&#39;<br>        parent_variant_label              &#39;r1i1p1f1&#39;<br>        physics_index                     1<br>        product                           &#39;model-output&#39;<br>        prov                              &#39;Omon ((isd.003))&#39;<br>        realization_index                 4<br>        realm                             &#39;ocean&#39;<br>        source                            &#39;CESM2 (2017): atmosphere: CAM6 (0.9x1.25 finite volume grid; 288 x 192 ...&#39;<br>        source_id                         &#39;CESM2&#39;<br>        source_type                       &#39;AOGCM BGC&#39;<br>        sub_experiment                    &#39;none&#39;<br>        sub_experiment_id                 &#39;none&#39;<br>        table_id                          &#39;Omon&#39;<br>        time                              &#39;time&#39;<br>        time_label                        &#39;time-mean&#39;<br>        time_title                        &#39;Temporal mean&#39;<br>        title                             &#39;Sea Surface Temperature&#39;<br>        type                              &#39;real&#39;<br>        variable_id                       &#39;tos&#39;<br>        variant_info                      &#39;CMIP6 20th century experiments (1850-2014) with CAM6, interactive land ...&#39;<br>        variant_label                     &#39;r4i1p1f1&#39;</pre><h4><strong>Pre-processing </strong>🛠️</h4><p>ESMValCore comes with a <a href="https://docs.esmvaltool.org/projects/ESMValCore/en/latest/api/esmvalcore.preprocessor.html">large suite</a> of built-in preprocessing functions. These functions have been designed as scientifically sound default implementations for frequently performed operations. An important idea underpinning ESMValCore (&amp;Tool)’s philosophy is that standardization facilitates comparison between, and reproducibility of, our data analysis workflows. Moreover, by bundling our efforts we can develop high quality preprocessor functions more effectively. In that spirit it would be good to see how xMIP could be integrated with ESMValCore’s CMOR checks and fixes.</p><p>We won’t go into detail on each of the preprocessing functions applied above, as they’re fairly self-explanatory. One thing to highlight though, is that in our calculation of anomalies, we deviated a little bit from Tom’s method. He subtracted the overall average of 1950–1980; we used monthly means over that period. It’s not really necessary here, but it facilitates comparing changes in summer temperature, for example 😎.</p><pre>cube = area_statistics(cube, operator=&#39;mean&#39;)<br>cube = anomalies(cube, reference=reference_period, period=&#39;month&#39;)<br>cube = annual_statistics(cube, operator=&#39;mean&#39;)<br>cube.convert_units(&#39;degrees_C&#39;)</pre><h4><strong>Custom code</strong></h4><p>Up to this point, everything we’ve done was “pure” ESMValCore. However, in every data analysis workflow, there comes a point where you want to do custom things. In the traditional context of ESMValTool, such custom code would be added as a “diagnostic script”. In the context of this notebook, we can simply start working with the cube in whatever way we like.</p><p>This can be a good moment to thank iris for its services, and continue with xarray if you prefer. For example:</p><pre>import xarray as xr<br>da = xr.DataArray.from_iris(cube)<br>da.plot()<br>print(da)</pre><pre>&lt;xarray.DataArray &#39;tos&#39; (time: 165)&gt;<br>dask.array&lt;filled, shape=(165,), dtype=float32, chunksize=(1,), chunktype=numpy.ndarray&gt;<br>Coordinates:<br>  * time     (time) object 1850-07-02 13:00:00.000001 ... 2014-07-02 12:00:00<br>    nlon     int64 ...<br>    nlat     int64 ...<br>    lat      float64 ...<br>    lon      float64 ...<br>    year     (time) int64 ...<br>Attributes: (12/57)<br>    standard_name:          sea_surface_temperature<br>    long_name:              Sea Surface Temperature<br>    units:                  degrees_C<br>    Conventions:            CF-1.7 CMIP-6.2<br>    activity_id:            CMIP<br>    case_id:                18<br>    ...                     ...<br>    time:                   time<br>    time_label:             time-mean<br>    time_title:             Temporal mean<br>    title:                  Sea Surface Temperature<br>    type:                   real<br>    cell_methods:           area: mean where sea time: mean longitude: latitu...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*bIYqrtaDGhrBNpOzVzsRUA.png" /><figcaption>The change in sea-surface temperature as plotted by xarray’s builtin plot method.</figcaption></figure><h3><strong>Reproducibility: convert to recipe</strong></h3><p>Traditionally, ESMValTool was designed to work with recipes: static, human-readable files that declare a worfklow from start to finish. This comes back to what we said earlier about describing <em>what</em> to do instead of <em>how</em>. ESMValCore helps us to start building a recipe. With the following code, we can generate the datasets and diagnostics sections of the recipe:</p><pre>from esmvalcore.dataset import datasets_to_recipe<br>import yaml<br><br>for dataset in ensemble_datasets:<br>    dataset.facets[&#39;diagnostic&#39;] = &#39;easy_ipcc&#39;<br>print(yaml.safe_dump(datasets_to_recipe(datasets)))</pre><pre>datasets:<br>- dataset: CESM2<br>  ensemble: r4i1p1f1<br>  institute: NCAR<br>- dataset: IPSL-CM6A-LR<br>  ensemble: r4i1p1f1<br>  institute: IPSL<br>- dataset: MPI-ESM1-2-LR<br>  ensemble: r4i1p1f1<br>  institute: MPI-M<br>- dataset: TaiESM1<br>  ensemble: r1i1p1f1<br>  institute: AS-RCEC<br>- dataset: AWI-CM-1-1-MR<br>  ensemble: r(1:5)i1p1f1<br>  institute: AWI<br>- dataset: AWI-ESM-1-1-LR<br>  ensemble: r1i1p1f1<br>  institute: AWI<br>- dataset: BCC-CSM2-MR<br>  ensemble: r(1:3)i1p1f1<br>  institute: BCC<br>...<br><br>diagnostics:<br>  easy_ipcc:<br>    variables:<br>      tos:<br>        exp:<br>        - historical<br>        - ssp585<br>        grid: gn<br>        mip: Omon<br>        project: CMIP6</pre><p>You still have to add the preprocessors and some metadata (a short description etc.). We envision a workflow where you use ESMValCore interactively for most of your exploratory work, and then port it to a recipe for sharing and reproducibility.</p><p>To complete this post, we’ve gone ahead and extended the recipe above to include all the available CMIP6 data and make a nice plot. As you might expect, this required more labour due to dataset issues for which no fixes were available, and a bit of extra tinkering to make the figure look more like the IPCC version. You can find the corresponding recipe and diagnostic script <a href="https://github.com/ESMValGroup/ESMValTool/pull/3356">here</a>. Be careful with running it, though! This recipe will try to download almost twenty-thousand files… Running this on the German climate compute service, we get the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAyuDmv_n2_W88G-HYXVUg.png" /><figcaption>This image includes all available models &amp; ensemble members.</figcaption></figure><h3><strong>Summary</strong></h3><p>ESMValCore is great addition to any climate analysis toolbox. Its key strengths are its its approach to streamlining workflows to achieve scientific rigor and reproducibility, and its tight integration with the ESGF and CMOR standards.</p><p>If you regularly work with CMIP data, sometimes struggle to retrieve data from ESGF, frequently encounter incompatible datasets, or want to make your workflows available to others, you should definitely give it a shot.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19a0b6366ea7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/easy-ipcc-powered-by-esmvalcore-19a0b6366ea7">Analysis-ready climate data with ESMValCore</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ESMValTool: Recipes for solid climate science]]></title>
            <link>https://medium.com/escience-center/esmvaltool-recipes-for-solid-climate-science-da5b33814f69?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/da5b33814f69</guid>
            <category><![CDATA[research-software]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[climate-modelling]]></category>
            <category><![CDATA[climate-models]]></category>
            <category><![CDATA[reproducible-research]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 09:23:29 GMT</pubDate>
            <atom:updated>2023-09-11T09:23:29.903Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p><em>This story was coauthored by </em><a href="https://www.esciencecenter.nl/team/dr-peter-kalverla/"><em>Peter Kalverla</em></a><em> and </em><a href="https://www.esciencecenter.nl/team/bouwe-andela-msc/"><em>Bouwe Andela</em></a></p><p>Like an artisan chef can improvise a delicious meal, so too can a scientist create new insights from raw data and exploratory analysis. But to run a successful restaurant with personnel and a consistent menu, our chef resorts to writing down his recipes. Similarly, we scientists must make an effort if we want our results to be consistent and widely reusable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RikpZ0cA45k1JSjB" /><figcaption>Photo by <a href="https://unsplash.com/@novokayn?utm_source=medium&amp;utm_medium=referral">Pylyp Sukhenko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is where <a href="https://esmvaltool.org/">ESMValTool </a>comes into play. Designed to ease and automate the analysis of large ensembles of climate model outputs, ESMValTool defines a clear and plain scientific <a href="https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html">“recipe” format</a>. This not only helps to make workflows scientifically robust, but also provides a way to easily share and build upon each others’ work. Even if you’re not a climate scientist, there are valuable lessons to draw from ESMValTool’s approach to FAIR and open science.</p><p>In this blogpost, we’ll explore four aspects of ESMValTool that may be worth your while. Starting with its take on analysis-ready data and its rich suite of pre-processor functions, we’ll set the stage for a broader discussion on how standardization helps making research easy and FAIR and the crucial role of a community in setting itself up for success.</p><h3><strong>Analysis-ready (CMIP) data</strong></h3><blockquote>Strictly speaking, most of the functionality described here is part of ESMValCore. ESMValCore is the engine, if you will, that executes ESMValTool recipes under the hood. ESMValTool, then, is a big collection of recipes. To the user of ESMValTool, this distinction is irrelevant, but we’ll use the terms interchangeably to be as accurate as possible. For a description of ESMValCore as a standalone utility, see the companion blogpost.</blockquote><p>ESMValTool was originally designed for the analysis of CMIP data: output of many climate models assembled in a series of “Coupled Model Intercomparison Projects”. The CMIP <a href="https://wcrp-cmip.github.io/WGCM_Infrastructure_Panel/CMIP6/data_request.html">data request</a> specifies standards that the data should adhere to. Known as the <a href="https://pcmdi.llnl.gov/software/cmorTable/index.html">CMOR standards</a>, after the “Climate Model Output Rewriter” software, they describe things like variable names, units, coordinates (with their direction), and also file name conventions. These standards make it much easier to compare output from different climate models, and they are the foundation on which ESMValTool &amp; Core were built.</p><p>For example, the file name conventions make it possible to specify a dataset like so:</p><pre>Dataset(<br>    short_name = &#39;tos&#39;,<br>    mip = &#39;Omon&#39;,<br>    project = &#39;CMIP6&#39;,<br>    exp = &#39;historical&#39;,<br>    dataset = &#39;CESM2&#39;,<br>    ensemble = &#39;r4i1p1f1&#39;,<br>    grid = &#39;gn&#39;<br>)</pre><p>and ESMValCore is able to locate it, both on your local computer or on <a href="https://esgf.llnl.gov/">ESGF</a>, the shared infrastructure where all of the CMIP data are collected. ESMValCore can automatically download data from ESGF, provided that the data is present there.</p><p>Pause a moment to let this sink in: ESMValCore eliminates hardcoded paths in your scripts, or rather, recipes. This is key to making them interoperable.</p><p>If you’ve worked with CMIP data yourself, you’ll probably know that, despite the strict protocol, small dataset issues are commonplace. That’s why ESMValCore executes an additional CMOR check upon data loading. Where possible, it applies automatic fixes for known and trivial issues. This makes working with the data much easier.</p><p>Another common issue with CMIP data is that files may not or no longer be available. ESMValCore makes it easy to perform wildcard searches or ignore missing datasets, saving you the trouble of figuring out which datasets are available or have been retracted since the last time you ran your recipe. Thus, ESMValCore is also a great utility for CMIP data discovery.</p><p>While ESMValTool &amp; Core were originally developed for CMIP data, it also supports ingestion of other datasets. For example, it can apply “on the fly CMORization” of ERA5 data, and it ships with download and “CMORizer” scripts for many other (observational) datasets. This makes ESMValTool more broadly applicable and moreover, it promotes standardization beyond its original scope.</p><p>To summarize, ESMValTool makes it easy to get to your data. In the next section, we’ll describe how ESMValTool also helps to standardize how you process it (without compromising on versatility).</p><h3><strong>ESMValCore’s rich preprocessor suite</strong></h3><p>Like with a kitchen recipe, after the ingredients come the preparation steps. In ESMValTool &amp; Core, they are called pre-processors. Common preps for climate data include regridding, area extraction, calculating anomalies and all kinds of statistics.</p><p>ESMValCore has built-in functions for each of these pre-processors, and they can be added to a recipe using a very high-level specification. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u2FludEKJb5cgvsOlfnWjQ.png" /><figcaption>Part of an ESMValTool a recipe listing the pre-processing steps that should be applied</figcaption></figure><p>By reading this snippet, you can probably get a pretty good idea of what this recipe is trying to achieve: the mean (temperature) change over Europe with respect to a reference period. This is a declarative syntax, focusing on <em>what</em> should be done, instead of <em>how</em>.</p><p>Of course, it can be very valuable to have a good discussion on <em>how</em> something should be done. But for most practical purposes, it’s often convenient to skip these “implementation details” and focus on the intention. Moreover, a standard way to do things makes it much easier to compare results and also to optimize the performance.</p><p>Ideally, with ESMValCore, the standard pre-processors provide a scientifically sound consensus implementation. For example, it uses state-of-the-art regridding with ESMF/ESMPy, and strict CF-compliant statistics from the Iris package. These functions have been designed by domain experts and are openly developed, so anyone can suggest improvements if they feel the current implementation is insufficient. ESMValCore executes these steps in a standard order which makes sense for most applications. However, you can also specify a custom order as this might make sense for some specific use cases.</p><p>In the illustration below, you can see the default order in which the pre-processors are executed. In your recipe, you can simply select those steps that are relevant for your workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B7DF5wIvtkG2AeyA.png" /><figcaption>The architecture of ESMValTool: recipe specifies which input data is loaded. This data is then ingested and the preprocessor steps specified in the recipe are executed. Finally, pre-processed data is passed onto a diagnostic script to generate the final output.</figcaption></figure><p>After prepping, your data is (optionally) passed on to a diagnostic script. This is the place where you can do custom things, such as making a figure, or doing a more thorough and case-specific analysis. “Combining the ingredients into a tasty dish”, to stick with the chef’s analogy.</p><p>So far, we’ve seen how ESMValTool tries to streamline those parts of your analysis workflow that can be standardized, and tries to accomodate all specific things you might need or want. In the next section, we’ll see how this helps you make your research more <a href="https://www.nature.com/articles/s41597-022-01710-x">FAIR</a>.</p><h3><strong>FAIR and open science</strong></h3><p>ESMValTool is a great example of how open science can work in practice.</p><p>Recently there’s been an increased focus on accessibility, transpancy, and <a href="https://the-turing-way.netlify.app/reproducible-research/reproducible-research.html">reproducibility </a>in science. We’ve seen the advent of open access publications, the introduction of FAIR <a href="https://www.nature.com/articles/sdata201618">standards </a>for research data management. It’s easy to extrapolate these ideas to other aspects of the research process as well. At the same time, this can be daunting, because it may seem like a lot of extra work that you might not have the time or skills for. So let’s see how ESMValTool helps with these concerns.</p><p>One ideal that’s often associated with open science is reciprocity, which we all know in <a href="https://genius.com/27629118">lyrical form</a> as “never take more than you give”. The idea is that scientific progress benefits from mutual exchange and building upon each other’s work. ESMValTool provides a platform for receiving and giving back. By making available a large suite of recipes that other people have already composed. By establishing a process to contribute your own recipes. By setting a standard format to encode recipes. By integrating existing functionality into a streamlined workflow system. And by facilitating a process of feedback based around users’ contributions, and a forum for discussion. Many of the things that are hard to organize on your own, have already been organized by the ESMValTool community. Thus, you’ll be greeted with a warm welcome instead of a daunting prospect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ShuXUZzeVkHT_-CK.jpg" /><figcaption>Illustration from the Turing way book dashes.</figcaption></figure><p>Going more into the specifics of FAIR, we can see how ESMValTool contributes: ESMValTool has great documentation listing all <a href="https://docs.esmvaltool.org/en/latest/recipes/index.html">available recipes</a> and a gallery with outputs. You can also list the available recipes from within the software, or browse the <a href="https://github.com/ESMValGroup/ESMValTool/tree/main/esmvaltool/recipes">source code</a>. So recipes are quite findable, although you could still take it a step further by creating a searchable registry with unique identifiers for each recipe (Note that the package as a whole does have a <a href="https://doi.org/10.5281/zenodo.3401363">DOI</a> for each version).</p><p>ESMValTool recipes are also very accessible. The YAML format is very readable for humans and also by machines; they can be executed. As we’ve seen above recipes are also portable: hardcoded file paths et cetera have been eliminated. For each release of ESMValTool, all recipes are executed to see if they still work, and output is made available.</p><p>To summarize, ESMValTool itself would qualify as “FAIR software. But moreover, ESMValTool recipes also come a long way as “FAIR scientific workflows”. Just like kitchen recipes empower chefs, ESMValTool recipes empower scientists.</p><h3><strong>The crucial role of community</strong></h3><p>ESMValTool would not have been where it is now if it wasn’t for the community. The ESMValTool community is a nice and welcoming crowd of scientists and software engineers from various institutes in Europe and beyond. In addition to lively interactions on GitHub there are monthly online meetings that are open to all, and there’s an in-person workshop once or twice a year. Furthermore there are spontaneous gatherings of ESMValTool contributors at other ocassions such as conferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*hPp8z6MT795wmEfyoCaSvg.png" /><figcaption>Screenshot of the ESMValTool discussion forum announcing some of the community gatherings.</figcaption></figure><p>Over the years the ‘core team’ has gained a lot of experience and become more professional, for example when it comes to processes for decision making and user engagement. There are clear guidelines for contributing to the codebase, and people are always willing to help each other out. To support new users, the user engagement team maintains a <a href="https://tutorial.esmvaltool.org/">tutorial </a>that is regularly taught.</p><p>By fostering enjoyable and personal interaction, the community plays a vital role in facilitating the development and long-term sustainability of ESMValTool.</p><h3><strong>Conclusion</strong></h3><p>In conclusion, ESMValTool provides a valuable platform for scientists to achieve consistency, reusability, and standardization in their research, owing much of its current success to the dedicated community that surrounds it.</p><p>Are you a climate scientists and do you want to be part of this community? Don’t hesitate to reach out on GitHub or another channel that you see fit. Are you passionate about open science but not necessarily in climate science? Then we hope you could get some nice takeaways from ESMValTools experience and we are always happy to exchange more. In any case, the next time you visit a restaurant, remember to appreciate not only the chef but also the thoughtfully curated menu.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da5b33814f69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/esmvaltool-recipes-for-solid-climate-science-da5b33814f69">ESMValTool: Recipes for solid climate science</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[eWaterCycle: Anecdotes of a FAIR expedition]]></title>
            <link>https://medium.com/escience-center/ewatercycle-anecdotes-of-a-fair-expedition-274d1e8e3bba?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/274d1e8e3bba</guid>
            <category><![CDATA[fair]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[hydrology]]></category>
            <category><![CDATA[standardization]]></category>
            <category><![CDATA[modeling]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 09:01:04 GMT</pubDate>
            <atom:updated>2023-03-09T09:01:04.179Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>The journey of a group of hydrologists and research software engineers after FAIRer research software.</p><h3>A grand challenge</h3><p>This story begins in April, 2019. A group of hydrologists and research software engineers had <a href="https://www.lorentzcenter.nl/fair-hydrological-models-automated-comparisons-using-escience-tools.html">gathered in Leiden </a>to discuss one of the grand challenges in contemporary research: <em>how to make our scientific endeavors reproducible and easier to reuse?</em></p><p>The problem was obvious. Hydrological modeling studies had grown complex. So complex, in fact, that it was virtually impossible to reproduce them from scratch. The only way to keep up was to maintain your own virtual research environment, with your own model, your own copy of input data, and your own computational facilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVhO3G5VtX4HqlFP6LaJvQ.png" /><figcaption>An unFAIR landscape: silo structure prohibits the exchange of models and their input/output data</figcaption></figure><blockquote>Side note: a characteristic figure in hydrological modeling studies is a hydrograph. This chart displays river discharge over time. To produce it, you simulate how the precipitation over a certain region propagates through the landscape: seeping into the soils, collecting into rivers, and flowing towards the sea. Hydrographs provide insight in the hydrology of a certain area and can serve to evaluate (and compare) the quality of your simulations.</blockquote><p>Despite the recent push towards more openly sharing datasets and model source code, this had minimal effect. Even with access to the source code, it remained difficult to compile each others’ models. And because modeling choices had diverged, it was not straightforward to swap out components or input datasets, or simply compare outputs of different models. Something had to change.</p><p>A few years earlier, a set of <a href="https://www.go-fair.org/fair-principles/">“FAIR” principles</a> had been formulated with the ultimate goal of making scientific datasets <em>Findable, Accessible, Interoperable, and Reusable</em>. Intuitively, these concepts could be applied to other aspects of the research process as well. It was the right idea at the right time. Thus <a href="https://www.lorentzcenter.nl/fair-hydrological-models-automated-comparisons-using-escience-tools.html">the workshop in Leiden</a> marked the start of our FAIR expedition.</p><h3>FAIR models</h3><p>The first uncharted territory was that of models, the core substance of every modeling study. Hydrological models come in many flavours. They differ in the physical processes they include or in the way in which they describe these processes mathematically. They use different methods to discretize the equations, and different programming languages to implement them. How could such diversity ever be more FAIR?</p><p>We found there was an interesting initiative from Boulder, Colorado: the <a href="https://csdms.colorado.edu/wiki/BMI"><em>Basic Model Interface</em></a> (BMI). Even if models work differently under the hood, we could standardize the way users interact with them. Such a familiar interface would make it easier to reuse and exchange each others’ models.</p><p>At that time, this standard had not been broadly embraced. But we were ambitious and we wanted to set an example of FAIR scientific conduct. So we decided to complement the participating models with a BMI. To top it off, we packaged them in Docker containers and made software to use them in a Python scripting environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CO5kblmYtR5qGYAcvuIbHg.png" /><figcaption>Different models, but to the user they look the same. We decided to embrace Basic Model Interface (BMI).</figcaption></figure><h3>FAIR forcing data</h3><p>Next, we entered the realm of forcing data: the weather that’s fed into the models. Here we discovered a very good standard with the only downside that it didn’t quite fit. Generations of coordinated climate research programmes had established conventions known as <a href="https://cfconventions.org/">CF </a>and <a href="https://pcmdi.llnl.gov/software/cmorTable/index.html">CMOR</a>. They describe how climate model output should be stored, including variable names, filename patterns, file formats, units, et cetera.</p><p>There was also this software called <a href="https://docs.esmvaltool.org/en/latest/">ESMValTool </a>that made working with these datasets more FAIR. You describe your workflow in relatively plain language, and the software takes care of finding the data and executing the specified tasks, such as cropping out a specific area. What’s great is that you can easily share your “recipes” so others can repeat and build upon your work.</p><p>Another great aspect of ESMValTool is its extensibility. By writing so-called <em>CMORizers</em>, we could “teach” the software to also understand the other datasets we wanted to support. At the other end of the workflow, we could write custom code snippets to incorporate model-specific quirks, such as deviating file formats. This allowed us to adopt the existing standards and software for our FAIR hydrological forcing module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCv3lwGmPzeENax9hIhuww.png" /><figcaption>ESMValTool’s extensibility allowed us to adopt it for our FAIR hydrological forcing data module.</figcaption></figure><h3>FAIR parameter sets?!</h3><p>So far, our journey had been quite a feat. We’d seen adopting a standard even if it wasn’t widespread, and being pragmatic if it didn’t quite fit. But what if there is no suitable standard?</p><p>We would soon discover it as we set foot in the land of parameter sets: home to all kinds of maps, from land use, soil types, elevation, you name it. We had envisioned and half-expected something similar to ESMValTool. But our disappointment grew quickly. Apparently, there was no clear authoritative source of parameter sets. No controlled vocabulary, a wild growth of file formats and software to manipulate them, and no consensus on what to use in hydrological models. We felt disoriented and washed out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKcxGe1AcIoAu1_Wf44XAw.png" /><figcaption>By imagining how we would use a standard if there had been one, we were able to “pretend” it exists.</figcaption></figure><p>So we entered upon a game trail. Out of the control and out of time, we headed straight toward our goal: being able to load parameter sets from within our unified modeling environment. We quickly collected example parameter sets for our set of models and dreamed up a minimal set of metadata to describe them: a name, a directory path, the name of a configuration file, and the name of the model for which it was to be used. That was all we needed to load an existing parameter set that we had magically obtained. We added a DOI field, empty for now, so we could eventually publish the datasets and retrieve them again. FAIR enough.</p><h3>A FAIR platform</h3><p>Anecdotes are nice, but experience is nicer. To help you get started, we collected our models, forcing data, parameter sets, and software stack on a dedicated platform. Even if all these components are more FAIR than they’ve ever been, it’s still a bit of work to collect and configure all that. Clearly, we haven’t yet reached transcendence.</p><p>The eWaterCycle demo platform gives a taste of what FAIR can bring in terms of user experience. Simply visit the URL, log in with your credentials, and enter a JupyterLab environment that is ready to go. On top of it, we build a graphical explorer where you can set up a basic experiment with a few clicks — a corresponding notebook will be generated and you can start modeling right away. Notebooks are great for exploratory research and disseminating experiments. Also, there are nice tools to help with things like auto-formatting and version control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-2EFy2k5Gwj7OXMkZgwQGg.png" /><figcaption>A dedicated platform makes all of its components more FAIR and helps to show the merits of FAIR</figcaption></figure><p>While the platform is not necessarily FAIR in itself, we find that it greatly enhances the FAIRness of its components. Technical aspects are hidden from the users, so they can focus on conducting experiments. Models and parameter sets can easily be discovered and used. Our guidelines for adding new models or parameter sets are like a step-by-step guide to making them (more) FAIR.</p><p>With eWaterCycle we hope to inspire a new paradigm for conducting hydrological modeling studies. The flowchart draws our FAIR journey in retrospect. It contains valuable insights, but it is by no means an all-encompassing wisdom chart. Now it is time for your expedition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6MXyXLQhvkjN7PL6Jp4gcw.png" /><figcaption>Our FAIR journey in retrospect (not an all-encompassing wisdom chart)</figcaption></figure><p>For more information on eWaterCycle, see the <a href="https://doi.org/10.5194/gmd-15-5371-2022">publication in Geoscientific Model Development</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=274d1e8e3bba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/ewatercycle-anecdotes-of-a-fair-expedition-274d1e8e3bba">eWaterCycle: Anecdotes of a FAIR expedition</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lilio — a calendar package for machine learning with time series data]]></title>
            <link>https://medium.com/escience-center/lilio-a-calendar-package-for-machine-learning-with-time-series-data-5c197830f4ce?source=rss-34317cc61901------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c197830f4ce</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[research-software]]></category>
            <category><![CDATA[time]]></category>
            <category><![CDATA[forecast]]></category>
            <category><![CDATA[calendar]]></category>
            <dc:creator><![CDATA[Peter Kalverla]]></dc:creator>
            <pubDate>Thu, 09 Feb 2023 09:02:35 GMT</pubDate>
            <atom:updated>2023-08-21T15:34:41.262Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Lilio — a calendar package for machine learning with time series data</h3><h4>Designed with the S2S community, available for everyone</h4><p>Imagine you’re trying to predict the likelihood of a summer drought. Imagine you want to do it weeks, months, or even years in advance. What are all the time aspects you have to take into account?</p><p>Obviously, there is “summer” in there, but when does that start or end? And, do we want just one number or perhaps one for each summer month? When will the forecast be issued? Is it one month in advance, and what does that even mean? A gap of 30 days between the forecast issue date and the first summer month, such that the forecast for “July” should be issued by the end of May? What data are used to drive our prediction? Temperature, pressure, humidity over the past weeks, months, years? Averaged? Could we use previous years to train or tune our model? And if so, then what if we go to longer timescales where data from the last summer is used to predict the next; how do we prevent <a href="https://en.wikipedia.org/wiki/Leakage_(machine_learning)">information leakage</a> in this case?</p><p>These are just some of the questions that scientists in the field of sub-seasonal to seasonal (S2S) weather and climate prediction have to deal with on a daily basis. Especially now that the use of machine learning is <a href="https://s2s-ai-challenge.github.io/">gaining traction</a>. In this blog post, we introduce <a href="https://lilio.readthedocs.io/en/latest/index.html">Lilio</a>, a new calendar <a href="https://pypi.org/project/lilio/">package </a>that was designed to help tackle these questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zkt-hddFFLyDQS2S.png" /><figcaption><a href="https://github.com/AI4S2S/lilio">https://github.com/AI4S2S/lilio</a></figcaption></figure><h3>The niche</h3><p>Before we dive into the capabilities of Lilio, let me briefly review why existing packages didn’t cut the deal for us. <a href="https://pandas.pydata.org/">Pandas</a> is great and we rely on it, but we wanted more flexibility to construct varying intervals with custom gaps in between. And instead of a continuous index, we wanted to exploit the periodicity in our use cases. Existing time series models (<em>e.g. </em><a href="https://medium.com/unit8-machine-learning-publication/time-series-forecasting-using-past-and-future-external-data-with-darts-1f0539585993">Darts</a>) often apply the same model irrespective of the forecast issue date, and the forecasts are anchored to said issue date. By contrast, our experiments are usually anchored to a clear target period, and each target may have a unique set of predictors. We have different models with different precursors for a windy May, a wet monsoon, a mild winter, …</p><h3>The fill</h3><p>So, how does Lilio help with all of that? First of all, we stack the years to obtain a 2-dimensional calendar. Each row represents a year, and columns are intervals within that year. Typically we have several precursor periods leading up to one or more target periods. This aligns nicely with the common representation of samples and features in ML data. In our case, the target is in the rightmost column(s).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*meSE1L3GAeX7BNxsrWu73Q.png" /><figcaption>Conceptual illustration of the structure of our calendars.</figcaption></figure><p>We define the “anchor date” to be between the target and precursor periods. All other intervals are expressed as offsets to this anchor date. Conveniently, this eliminates any ambiguity related to leap years. Here’s a calendar generated with Lilio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*EzXXk_D_G-6zGSVg9n1Flw.png" /><figcaption>Example calendar with uniform 10-day intervals, here represented as offsets to the anchor date.</figcaption></figure><p>By default, we include as many blocks (of a given frequency) as fit in one year but not more. To control this behaviour, Lilio provides options to allow or prohibit overlap. This makes it straightforward to apply existing train/test splitting strategies without leakage.</p><h3>Anchor + offset = date</h3><p>Initially, Lilio calendars don’t include years. Only after we map the calendar onto a given year range or dataset, actual dates can be calculated. Since the start or end date of the calendar doesn’t always nicely align with the input data, the calendar also comes with a method to map it to the range of available data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*nam1xFGCBs58QU409AJX3w.png" /><figcaption>Same calendar as above but plotted on a datetime axis. The anchor date for this calendar is 30 November. Combined with the anchor years 2020 and 2021, the actual dates can be inferred.</figcaption></figure><h3>Visualizations &amp; wishful drawing</h3><p>The visualizations shown above are generated automatically. This turns out to be very helpful in the initial process of setting up your experiments. We even found ourselves making “wishful drawings” to communicate about alternative calendars that we’d like to support.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AVgHOgwMnyC6a-oc.png" /><figcaption>Illustrated feature request for a “rolling calendar”</figcaption></figure><h3>Resampling</h3><p>Lilio’s resampling functionality can be used to aggregate the input data based on the calendar’s intervals. Here, again, we heavily rely on the presence of bounded intervals. By using these as resampling bins, we ensure that all input data ends up on the exact same time axis.</p><h3>Simple and custom calendars</h3><p>For many applications, a simple weekly, monthly, or (n-)daily calendar may be all you need, and Lilio makes this super easy. With only a bit more effort, you can also construct calendars with gaps and overlapping intervals of varying lengths. Here’s a more exotic calendar for predicting the chances of a white Christmas:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*pCbv11Rn4PMtNR46nnpTgA.png" /><figcaption>This calendar uses more fine-grained information closer to the target</figcaption></figure><h3>What’s next?</h3><p>Lilio has been developed in the context of a larger <a href="https://research-software-directory.org/projects/ai4s2s">project </a>in which we are developing a Python package to set up and streamline S2S — machine learning workflows. It will be a key component in our experimental setup. At the same time, the calendar is also very suitable as a stand-alone component for use in other applications. We are curious to learn about new use cases that you may have for it.</p><h3>Final note</h3><p>Collaborative research software development is a <a href="https://github.com/AI4S2S/lilio/graphs/contributors">team effort</a>. Please give appropriate credit and consider<a href="https://github.com/AI4S2S/lilio"> joining us</a> 😊. And if you read up to this point and still wonder who Lilio was: <a href="https://en.wikipedia.org/wiki/Aloysius_Lilius">here you go</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c197830f4ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/escience-center/lilio-a-calendar-package-for-machine-learning-with-time-series-data-5c197830f4ce">Lilio — a calendar package for machine learning with time series data</a> was originally published in <a href="https://medium.com/escience-center">Netherlands eScience Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>