<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tom Clark on Medium]]></title>
        <description><![CDATA[Stories by Tom Clark on Medium]]></description>
        <link>https://medium.com/@thclark?source=rss-3dabf88fa82b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*z4g-ykQB7dCFfmf9q_1yeQ.jpeg</url>
            <title>Stories by Tom Clark on Medium</title>
            <link>https://medium.com/@thclark?source=rss-3dabf88fa82b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:16:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thclark/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From software to academia… what about the Open Source?]]></title>
            <link>https://thclark.medium.com/from-software-to-academia-what-about-the-open-source-64aec183a39d?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/64aec183a39d</guid>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 17:30:07 GMT</pubDate>
            <atom:updated>2026-10-04T17:30:07.572Z</atom:updated>
            <content:encoded><![CDATA[<p>This isn’t a story or standalone essay, but instead is connected to my recent LinkedIn post announcing a career shift.</p><p>For those who’ve been following my work the last 15+ years, this has come as either a huge surprise, or as no surprise at all!</p><p>You can read about the motivation for this move here in Part 1, but I thought it might be useful to explain a bit about what’s happening, because there are quite a few programmes and projects relying on Open-Source Software I’ve built or maintained.</p><h3>Areas of work and what’s happening</h3><p><strong><em>(a TL;DR for worried users)</em></strong></p><ul><li><strong>Octue: </strong>The company will continue as an organisation supporting Open-Source Software, and as a vehicle for occasional individual consultancy. We’ll no longer be searching for projects to support significant new software developments.</li><li><a href="https://bezier.octue.com/"><strong>Bézier:</strong></a><strong> </strong>This tool for securely creating and sharing wind turbine power curves is really on fire — 1000 turbines and growing! This collaboration between Octue and Wind Pioneers is super rewarding, I’ll be continuing to commit similar amounts of time to build Bézier and the data standards that underpin it.</li><li><strong>IEC 61400–16 standard for wind turbine power curve data:</strong> This meshes well with both Bézier and my new work, so I’ll continue to dedicate time. It’s having so much impact across the industry, it’d be crazy not to. We’ll form a committee for ongoing maintenance soon, stay tuned.</li><li><a href="https://strands.octue.com/"><strong>Strands:</strong></a><strong> </strong>This tool for version-controlling and publishing JSONSchema is ultra powerful. I’ll keep it running for my own work, and because the IEC require it, but may change architecture to reduce maintenance and running costs. If you care what happens to Strands, contact me.</li><li><strong>Octue’s SDK (the twined framework):</strong> This is probably the most under-valued project I’ve built (with the talented Marcus Lugg). It’ll be maintained under an ongoing contract, although we’re not planning any major features unless a third party is willing to contribute significantly.</li></ul><h3>Open Source libraries and their support</h3><p>It’s difficult to get reliable analytics, but as far as I can tell, Octue’s and my personal libraries have been used by at least 20 universities world wide (including several ivy-leagues) and thousands of commercial organisations including Google, SpaceX, and NASA… with a variety of applications from securing workaday-web-server applications to deep scientific analysis.</p><p>Only two of those organisations ever bothered sponsoring the work: Wind Pioneers and OST. So I’ve chosen to continue working with these two 💙.</p><p>I don’t need to tell any OSS maintainer about the total mess in the Open-Source scene, funding wise and culture-wise. It’s really extremely hard to maintain a basic income stream by working on OSS and this has taken a severe toll on me over the years.</p><p>With that said, I’ve always felt a responsibility around the projects I’ve maintained so wouldn’t just drop maintenance. Here’s what’s happening:</p><ul><li><a href="https://github.com/django-guardian/django-guardian/"><strong>Django-Guardian:</strong> </a>After a significant chunk of work to resurrect this security-critical and widely used library for the Django web framework, it’s now in very good state. I’m proud of it, and have passed the role of ‘jockey’ onto the the capable maintenance team that emerged.</li><li><a href="https://github.com/octue/django-gcp"><strong>Django-GCP</strong></a><strong>:</strong> <em>To be decided.</em> I’m getting <strong>pretty sick</strong> of putting volunteer work into something that makes Google Cloud a really credible platform for running Django, and getting ZERO recognition from Google for it. Nobody’s even bought me a beer.</li><li><strong>Assorted minor OSS libs: </strong>On a case-by-case basis. I’ll be maintaining libraries which are relevant to my ongoing work, and sunsetting or handing off others. If you’re interested in taking over any projects on either <a href="http://github.com/thclark">thclark</a> or <a href="http://github.com/octue">octue</a>, reach out in the issues or via DM here.</li></ul><p><em>Note: this article contains em-dashes because they’re </em><a href="https://www.grammarly.com/blog/punctuation-capitalization/what-is-an-em-dash/"><em>grammatically correct</em></a><em>, not because it was written by AI ;).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64aec183a39d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A new (old) goal: understanding turbulence in Wind Energy]]></title>
            <link>https://thclark.medium.com/a-new-old-goal-understanding-turbulence-in-wind-energy-3df7eada4ebb?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3df7eada4ebb</guid>
            <category><![CDATA[wind-energy]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[career-change]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 17:15:32 GMT</pubDate>
            <atom:updated>2026-10-04T17:32:22.561Z</atom:updated>
            <content:encoded><![CDATA[<h3>A new (old) goal — understanding turbulence in Wind Energy</h3><p>This month, I’ve made a career shift, and moving from CEO of an open-source software company to Senior Research Associate at a university is a bit of an unusual change of gear!</p><p>For those who’ve been following my work the last 15+ years, this seems to have come as either a huge surprise, or as no surprise at all… so I decided to write a bit about what I’ll be doing, why the shift, and how I made the decision.</p><blockquote>Readers with projects and programmes that depend on my Open-Source Software work can find out what’s happening to that <a href="https://thclark.medium.com/from-software-to-academia-what-about-the-open-source-64aec183a39d">HERE</a>.</blockquote><h3>“Why”</h3><p>I still remember when I fell in love with Fluid Dynamics. The exact moment, sat on a work bench with the autumn sunlight streaming into the lab. I looked at the streaklines, in still-wet emulsion paint on the back of a triangular black plate, and thought “I could do this forever”.</p><p>This was early December 2004, doing an undergraduate lab exercise about delta wing vortices in the tiny little blue wind tunnels on the third floor of CUED. I walked out of the engineering department, down Trinity Street and into my Director of Studies’ office and didn’t leave until I persuaded him to let me up-end my chosen courses so that I could take the supersonic modules too. No mean feat when you’re already halfway through the third year tripos at Cambridge, but I didn’t care.</p><p>Ever since, work has revolved around aerodynamics and hydrodynamics. A Propulsion Engineer at QinetiQ, back for a PhD in turbulence, moving into turbine design for tidal energy, then wind energy for the last 12+ years.</p><p>Octue (which was originally called “Ocean Array Systems”) got started because I’d learned things about turbulence that I thought would be useful in the tidal and wind industries. I knew it’d be vital for things like accurate resource assessment and whole-farm control of turbine arrays.</p><p>After a lot of struggling through semi-funded research projects, it became clear that software development skills were marketable, and niche knowledge about turbulence really wasn’t. So (around the time of the name change), focus shifted to building Open-Source Software tools for scientists.</p><p>Innumerable organisations including NASA JPL, SpaceX, Google, multiple Ivy League / Russel / Brown / Whatever Group universities (including researchers at Harvard and MIT) have used software that we built at Octue. None of them ever paid a bean for it. My jaded opinions on what “Open Source” has come to mean belong in a different essay, however.</p><p>So, it’s not like we did nothing — quite the opposite — but I’ve been existing for a decade in a field <em>adjacent</em> to what I <em>actually</em> want to do. Despite <em>loving</em> the people I work with, and genuinely finding the work to be important, that adjacency grinds over time.</p><p>The hard part of it is actually realising this.</p><h3>How I figured out this very obvious thing</h3><p>Every decision I’ve made for Octue as CEO has been a junction: build toward something more commercial, or build toward better science and technology. I’ve always chosen the latter. Every. Fucking. Time. I just can’t help myself.</p><p><em>It’s just who I am. Is it a failure?</em></p><p>Some would definitely say so. Some would say that as CEO I should have been refining the proposition, marketing harder, chasing the market, networking more, influencing more, generating more leads, generating better leads, and, and, and…</p><p><em>I’ve been judged aplenty. And have judged myself pretty harshly too.</em></p><p>At some point last year, frustrated with how I couldn’t seem to scale Octue to meet my vision, I started working with an Executive and Career Coach — the excellent Anne Duvaux. The brief was that I wanted to get unblocked; become a better CEO.</p><p>I’d already been toying with the idea of going back to academia but could barely admit it — it’s in my nature to succeed, and to try harder if that doesn’t bear fruit. The strategic perspective of “how about doing something different?” was very difficult to register and I just couldn’t see the <em>route</em> to doing it.</p><p>I got <em>so</em> lucky meeting Anne totally by chance on LinkedIn; something about her profile resonated so I dropped her a message. The 15 or so sessions in which we worked together slowly teased out that there actually were options, and how I might come to exercise them.</p><p><strong>Most importantly, I stopped being so harsh on myself for being a shitty CEO.</strong></p><p>So, subconsciously at first, then very consciously, I stepped back some commitments and started laying ground to get back into research. Conversations with academics in the UK were pretty fruitless: A mid-career industrialist really doesn’t fit anywhere in the hierarchy of British academia. No matter how passionate they are about turbulence research.</p><p>It’s funny how things work out. I was browsing LinkedIn and saw a post from Sarah Barber — an amazing researcher based in Switzerland, who Octue already worked with as a commercial partner to one of her projects. She was advertising for a new team member to come and look at the effects of turbulence on wind turbines. <strong>MY DREAM JOB.</strong></p><p>Instead of trying to achieve a thing, the work of stepping back and putting in place the conditions I needed for clear decision making brought me to a moment where the solution was clear. I applied, got the job and spent three months crazily preparing for an international move to Rapperswil, on Lake Zurich!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hlXVI6IAV9BIdVRPD9NAmw.png" /><figcaption>Yeah, this is where I live now.</figcaption></figure><h3>What I’ll be doing</h3><p>In 13 years there were only two organisations that actually properly sponsored my work — paying regularly for consultancy to deliver their products, and sponsoring key features of open-source code that enabled the work. Those organisations were Wind Pioneers — a visionary technical consultancy building the next generation of wind farm design tools and OST, the University of Eastern Switzerland, where I’ve now joined.</p><p>So with some ridiculous luck, I managed to persuade OST to let me continue working part time with Wind Pioneers. So I’ll be doing that at 20% time, through Octue as before, keeping the WindQuest team on their toes and keeping our joint product <a href="https://bezier.octue.com/">Bezier</a> (a wind turbine power curve library) running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DyTli3Y0YA-H8f64PY9KpA.png" /><figcaption>We built a digital power curve library on top of the -16 standard. No way am I abandoning that! With 1000 turbines and countless import/export options it’s seriously starting to be useful for developers worldwide.</figcaption></figure><h4>…and the research?</h4><p>Recently, I had a massive accomplishment: co-writing the IEC 61400–16 standard for Wind Turbine Power Curve Data. I built the schema (mostly me, there were a few smaller collaborators) and worked with a large group of dedicated volunteers (including the brilliant and persistent Emil Moroz, without whom it just never would have happened) to write the standard.</p><blockquote>From next year, every wind turbine in the world will (or at least should!) come with a digital specification sheet, easily loaded into analysis programs and unambiguous, well-defined meaning.</blockquote><p><strong>Impacting the entire industry like that has been hugely satisfying. And after 13+ years of struggle, I now finally understand *how* to do so.</strong></p><p>Way back in 2013 when I started Octue, it was because I knew how important turbulence was to the Wind Industry. I’ve been passionate about that ever since, and now am finally actualised:</p><p>The next three years (and hopefully beyond) will be dedicated to the problem of turbulence, under the guidance of Sarah Barber in OST’s Institut für Energietechnik. We’ll be looking specifically at one of the myriad effects of turbulence: the way it interacts with tip vortices.</p><p>And just wait until you hear the crazy experimental methods we’re using :).</p><h3>One goal</h3><p>It’ll take longer than those three years, but I have a very clear goal:</p><blockquote>Develop a comprehensive framework and methods, through which the wind industry can understand turbulence and its effects.</blockquote><p>The best place for this is from the context of academia, while maintaining close ties to the reality of industry.</p><p>After that I’ll retire. Or become a counsellor for burned out startup executives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3df7eada4ebb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dīrāsha: A City within a City]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-link"><a href="https://thclark.medium.com/d%C4%ABr%C4%81sha-a-city-within-a-city-2f67858a53fa?source=rss-3dabf88fa82b------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://thclark.medium.com/d%C4%ABr%C4%81sha-a-city-within-a-city-2f67858a53fa?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f67858a53fa</guid>
            <category><![CDATA[invisible-cities]]></category>
            <category><![CDATA[creative-writing]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 20:14:38 GMT</pubDate>
            <atom:updated>2026-10-06T14:53:42.022Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Are tailwind’s financial woes REALLY caused by AI?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/are-tailwinds-financial-woes-really-caused-by-ai-151d7cc7fe5f?source=rss-3dabf88fa82b------2"><img src="https://cdn-images-1.medium.com/max/2600/1*iDQFDgopwU8oGXE9pSSSTA.png" width="2928"></a></p><p class="medium-feed-snippet">Recently, stories are popping up all over Medium (here, and here, and here), about Adam Wathan&#x2019;s announcement to significantly reduce&#x2026;</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/are-tailwinds-financial-woes-really-caused-by-ai-151d7cc7fe5f?source=rss-3dabf88fa82b------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/are-tailwinds-financial-woes-really-caused-by-ai-151d7cc7fe5f?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/151d7cc7fe5f</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[tailwind-css]]></category>
            <category><![CDATA[tailwind-ui]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 15:58:55 GMT</pubDate>
            <atom:updated>2026-01-20T17:45:15.057Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Simple local development to debug SSL, CORS and CSRF issues]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://levelup.gitconnected.com/simple-local-development-to-debug-ssl-cors-and-csrf-issues-278327084a38?source=rss-3dabf88fa82b------2"><img src="https://cdn-images-1.medium.com/max/600/1*5xiU1RH-LV3yDMhL5WdD6g.png" width="600"></a></p><p class="medium-feed-snippet">Here, I show you how to get set up with Caddy, enabling you to debug CORS/CSRF for complex backend / frontend configurations</p><p class="medium-feed-link"><a href="https://levelup.gitconnected.com/simple-local-development-to-debug-ssl-cors-and-csrf-issues-278327084a38?source=rss-3dabf88fa82b------2">Continue reading on Level Up Coding »</a></p></div>]]></description>
            <link>https://levelup.gitconnected.com/simple-local-development-to-debug-ssl-cors-and-csrf-issues-278327084a38?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/278327084a38</guid>
            <category><![CDATA[csrf]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[cors]]></category>
            <category><![CDATA[caddy]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Fri, 09 May 2025 16:17:51 GMT</pubDate>
            <atom:updated>2025-05-12T09:27:57.366Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A toast to the Lassies]]></title>
            <link>https://thclark.medium.com/a-toast-to-the-lassies-b666e0817e08?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b666e0817e08</guid>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 07:47:01 GMT</pubDate>
            <atom:updated>2025-01-26T07:49:29.091Z</atom:updated>
            <content:encoded><![CDATA[<p>Given among a dear and intimate group of friends, 25 January 2025.</p><p>On behalf of my fellow gentlemen!</p><p>’Tis <em>splendid</em> to be here, with all our treasured lassies.<br>‘Cos you know that we love you, just masses an’ masses.</p><p>Aye, our lassies;</p><p>With the <em>softest</em> of bosoms but the <em>toughest</em> of souls<br>As the inky calm sea hiding turbulent shoals.</p><p>Aye, our lassies;</p><p>With the <em>kindest</em> demeanours and <em>sharpest</em> of minds<br>That in <strong>all Scotland’s lush mountains and hills we could find!</strong></p><p>Now;</p><p>The tale of lads an’ lasses is a tale of two cities<br>Us with our neeps; an’ yous with your tatties.</p><p>But as us laddies know, there’s no finer gender;<br>As tonight, here with Rabbie, we have cause to remember…</p><p>Our hearts you have reached with a touch <em>fierce</em> but tender<br>So, a toast: To our lassies; <strong>in all your fine splendour!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b666e0817e08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The IronLAB Manifesto]]></title>
            <link>https://thclark.medium.com/ironlab-manifesto-7c47177bd3ee?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c47177bd3ee</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ironlab]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[matlab]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Sun, 05 Nov 2023 12:10:44 GMT</pubDate>
            <atom:updated>2023-11-05T15:48:49.860Z</atom:updated>
            <content:encoded><![CDATA[<p>Tools for scientific coding have not kept pace with progress in computer science, UX, cloud computing or data management.</p><p>IronLAB is an ambitious proposal to bring that back on track.</p><p>IronLAB is not a product (yet). It’s simply an idea and a wishlist. <em>Clap for this story if you wish it too.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7XsSm1-tCaHoxGX3x7LbQ.png" /><figcaption>Rust is a fast, compiled language which has memory-safety and thread-safety guarantees making it a really strong basis for scientific coding. But it’s hard to use and there’s not a huge ecosystem for scientific analysis. LET’S CHANGE THAT.</figcaption></figure><h3>Motivation</h3><p>If you’re a research scientist or engineer, it should be…</p><ul><li>Easy to write scientific algorithms / packages</li><li>Easy to visualise bottlenecks and optimise algorithms for extreme speed</li><li>Easy to run programs in the cloud as data services for others to use</li><li>Easy to work with modern data infrastructure like data lakes</li><li>Easy to create publication-quality plots</li><li>Easy to get started in a productive environment</li><li>Easy to learn more about science and computing <em>from that environment</em></li></ul><p>But it isn’t. It’s still really quite hard to do most of those things. We want to change that.</p><h3>What is IronLAB?</h3><p>IronLAB will be a cross-platform desktop application, with cloud-based counterpart, intended to replace MATLAB and the VSCode + python + numpy/scipy toolchain, with a focus on User Experience for the modern scientist.</p><p>IronLAB will comprise a high-level scripting language and tightly integrate with the Rust ecosystem <em>(the Nature article </em><a href="https://www.nature.com/articles/d41586-020-03382-2"><em>“Why Scientists are turning to Rust”</em></a><em> is a good read)</em>.</p><h3>Features</h3><p>Some killers, some key to success, some neat, and some that really shouldn’t have to be mentioned in this day and age…</p><ul><li><strong>Curated ecosystem. </strong>Inspired by MATLAB toolboxes, we’ll have a set of contrib packages with strictly quality-controlled and well-written documentation, with examples rooted in real science. That won’t preclude anyone publishing their own packages of course! <em>(Many of these can be direct bindings to or collaborations with existing rust scientific packages, or “crates”, of which there’s an increasingly good number. A reddit discussion about this is </em><a href="https://www.reddit.com/r/rust/comments/17oc8a0/comment/k7xnj4b/?utm_source=share&amp;utm_medium=web2x&amp;context=3"><em>here</em></a><em>).</em></li><li><strong>Science-oriented scripting language. </strong>Use an array-oriented scripting language allowing you to succinctly express array and matrix operations and more. <em>(We can take the best inspiration from MATLAB, python/numpy and Julia. Or should we simply copy one of these directly? </em><a href="https://github.com/RustPython/RustPython/"><em>If we chose python it’s already done for us…!</em></a><em>)</em></li><li><strong>Concurrency Support.</strong> Rust’s ownership and borrowing system can be built on to provide safe and efficient concurrency support for parallel processing in scientific and engineering applications.</li><li><strong>Automated and safe binding to Rust packages and modules. </strong>This will allow a transparent shift to Rust for the really intense bits, allow leveraging the rust ecosystem, and (best of all), proper runtime type checking on the interface will mean no more MEX file segfaults!!!</li><li><strong>Documentation editor.</strong> Good quality documentation is what makes code usable so every barrier to this must be broken down. It’s hard to write docs for python packages, and <em>really</em> hard to write docs for MATLAB packages. Major aspects should include figures from code output, doc testing and automated api generation.</li><li><strong>Profiling with great UX.</strong> Good profiling must go far beyond the awkward hell of hooking up austin to run a benchmark. You should be able to one-click-profile any script. Flamegraphs are hard to interpret, so data should be displayed in ways like line highlighting and a drill-down results panel with carefully designed UX.</li><li><strong>Automated code formatting.</strong> Inspired by the great success of <a href="https://black.readthedocs.io/en/stable/">Black</a>.</li><li><strong>Sensible figure generation.</strong> To this day, there’s no single figure solution which allows generation of scientific figures, interactivity, management of figures in IDE context, export to publication-quality graphics and HTML. <em>(Plotly.js definitely comes closest so we should integrate tightly with that, building our own is a whole thing as big as all the rest).</em></li><li><strong>Package management. </strong>Inspired by cargo, poetry, and pnpm, it should be easy both to template new packages and use existing ones. <em>(TBC can we simply leverage cargo? This would be ideal!).</em></li><li><strong>Debugger.</strong> Pretty much goes without saying, but let’s make a point that the debugger integration should be a first-class citizen not an afterthought.</li><li><strong>Integration with Git/GitHub. </strong>There are many in the scientific software community who are still git laggards, although there is momentum now in the right direction, which should be encouraged.</li></ul><h3>Technology Stack</h3><p>We will develop a cross-platform desktop application using the following technologies:</p><ul><li>Tauri application framework (built on Rust itself)</li><li>Next.js rendering framework with tailwind for the visual representation</li><li>A scripting language (MATLAB/Octave-like yet to be defined)</li><li>A Just-In-Time Compiler (yet to be defined)</li><li>The rust toolchain under the hood</li><li>The plotly.js figure/charting tool based</li></ul><h3>Inspiration</h3><p>IronLAB is inspired by our own frustration over decades of producing scientific software. We thought that a high-level language oriented toward science and engineering with a really strong ecosystem and purpose-built IDE, leveraging the state of the art in computer science and software development tooling, could be transformative for the community.</p><p>IronLab is the brainchild of <a href="http://octue.com/">Octue</a>, which is an Open-Source Software foundation dedicated to helping scientists and engineers work with data. But it’ll take a whole village to both make it work and then make it cool.</p><h3>Business Model</h3><p>IronLAB development won’t pay for itself, and VCs are reluctant to invest in non-profit efforts. So we must find ways of securing sustainable funding for the project (sustainable = NOT grants! Grants are great for developing specific chunks, but revenue must grow organically then be reinvested into the ecosystem for the project to succeed in the long term).</p><p>IronLAB will make money by offering:</p><ul><li><strong>Cloud-native workloads as a paid feature.</strong> Often, a scientist will wnat to run 10,000 variations of a thing as a batch. We’ll make this seamless to achieve by deploying to the cloud as a straightforward feature then mark-up the CPU hours fees that we charge on from the cloud provider.</li><li><strong>Data service hosting. </strong>By providing opinionated wrappers, permissions management and other infrastructural logistics for scientific data services (already in motion in the <a href="http://github.com/octue/octue-sdk-python">Octue SDK</a>), scientists can offer their data services on a subscription or pay-per-use model (or make access free and foot the hosting fees themselves). We’ll take a small proportion of paid service fees as well as mark up the CPU hours fees.</li></ul><h3>Unresolved Questions</h3><ul><li>Am I just describing <a href="https://junolab.org/">Juno, the Julia environment</a>? Find out! If yes, go and give them support instead of messing around starting from scratch. <em>Initial skim suggests that compilation of .jl files will result in reasonably-fast but not memory safe executables. I think it’ll also be very difficult to link against existing ecosystem in the compiled language (eg what if we want to call a rust crate?)</em></li></ul><h3>Signing off</h3><p>This was written by me, Tom Clark, on a Sunday morning. The aim of this article is to put an idea down on paper so I can park it for a while, whilst gathering some community support and focusing on existing OSS commitments.</p><p>I’m working on related-but-not-this things at Octue. When we’re in a position to invest outside our current offerings, we’ll come pick this up — and over the next year or so I’ll be keeping my eye open for some significant grant funding to get the ball rolling.</p><p>If this interests you, and you’re able to contribute resource of some kind please do get in touch at <a href="http://octue.com/contact">octue.com/contact</a>. Examples of resource include:</p><ul><li>Organisational sponsorship (from simple letters of support to financial sponsorship)</li><li>Marketing talent</li><li>Design talent</li><li>JIT Compiler / langauage design / tokenisation/ AST Generation expertise</li><li>Rust expertise</li><li>Frontend (Next.js) expertise</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c47177bd3ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[72 Hours at WindEurope: Using the service]]></title>
            <link>https://medium.octue.com/72-hours-at-windeurope-using-the-service-eb080e2e9963?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb080e2e9963</guid>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Thu, 27 Apr 2023 07:02:14 GMT</pubDate>
            <atom:updated>2023-04-29T06:55:03.686Z</atom:updated>
            <content:encoded><![CDATA[<p>Well, it’s all reappeared now and people at the conference are looking at it, evaluating the tools and figuring out the processes.</p><p>The last of the six steps is to use the product, then start mapping out the chess game of what to do next.</p><p>I’ll come back here once we’ve had a decent length of time to do so, and update you with what went on and what we think the next steps are!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb080e2e9963" width="1" height="1" alt=""><hr><p><a href="https://medium.octue.com/72-hours-at-windeurope-using-the-service-eb080e2e9963">72 Hours at WindEurope: Using the service</a> was originally published in <a href="https://medium.octue.com/">Octue</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[72 Hours at WindEurope: Delivering the service.]]></title>
            <link>https://medium.octue.com/72-hours-at-windeurope-delivering-the-service-8fcd50b61649?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fcd50b61649</guid>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[wind]]></category>
            <category><![CDATA[digitalization]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 09:35:44 GMT</pubDate>
            <atom:updated>2023-04-26T09:35:44.121Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s Wednesday Afternoon in Copenhagen, and it’s time to deliver both a service and a presentation. This article is for the fifth of our <em>“Six Steps” </em>— you can read the overview <a href="https://thclark.medium.com/72-hours-of-digitalisation-at-windeurope-7aa786be729d">here</a>.</p><p><em>This article is aimed at engineers, researchers and execs in the Wind Industry, to help understand the process of digitalisation. If you’re qualified in Systems Architecture or Data/Software Engineering, you’re way ahead of this; just get stuck in already!!</em></p><h3>About the ‘Deliver’ Step</h3><p>‘Delivering’ is all about how easy you make it for the consumers (end users) of the data to get it. Essentially, we’re talking about User Experience.</p><blockquote>User Experience (UX) is King. And not just for webapps.</blockquote><p>User Experience (UX) is the way that you, and people in your team, will work with your tool. UX isn’t just for webapps — its how anyone interacts with your tool… and <strong>your project will live or die on UX</strong>. If tools are difficult to use, people avoid them like the plague no matter how smart.</p><p>And this right here is why you, a humble engineer somewhere in the vast Wind Industry, are the perfect person for this bit. <strong>You know what you need</strong>. So don’t be afraid to build something to make your own life easy!</p><p>Some things to think about:</p><ul><li>Should there be a Command Line Utility? <em>Check out the Clicks python library for making user friendly CLIs.</em></li><li>Will you be centralising/storing data somewhere?<em> (See below.)</em></li><li>Can you wrap complicated stuff in a helper library?</li><li>If so, what language(s) are needed?<em> Try to stick to just one for the first few iterations. Python is pretty popular, or if you know it then Rust is fast and extremely versatile.</em></li><li>Will you need an API? <em>Cloud Functions are a great way of creating ultra-simple API endpoints.</em></li></ul><h3>Where we’re at</h3><p>We decided that we’d do two things to make it ultra easy to interface with the service. Sure, users can make requests directly to the API but we wanted to make it familiar for basic users of python, and non-technical users.</p><ul><li><strong>Python client (on pypi):</strong> <a href="https://github.com/octue/windeurope72hours-elevations-client-python">https://github.com/octue/windeurope72hours-elevations-client-python</a></li><li><strong>Map (React/ Javascript using DeckGL):</strong> <a href="https://je3mob.csb.app/">https://je3mob.csb.app/</a></li></ul><h4>Try it</h4><pre>pip install windeurope72hours<br><br>python<br><br>&gt;&gt;&gt; from windeurope72hours import get_coordinate_elevations<br>    elevations, later, estimated_wait_time = get_coordinate_elevations(<br>        [[54.53097, 5.96836]],<br>        resolution=12,<br>    )<br>    print(elevations)<br><br>&gt;&gt;&gt; {(54.53097, 5.96836): 0.0}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qdbmIuCCrQTvhUm0KczfMQ.png" /><figcaption>As you pan the map, it automatically fetches elevations near the centre of the view :)</figcaption></figure><h3>Wrapping up</h3><p>Now it’s over to YOU! Please use the API or the python client, and let us know how it goes!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fcd50b61649" width="1" height="1" alt=""><hr><p><a href="https://medium.octue.com/72-hours-at-windeurope-delivering-the-service-8fcd50b61649">72 Hours at WindEurope: Delivering the service.</a> was originally published in <a href="https://medium.octue.com/">Octue</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[72 hours at WindEurope: Populating data]]></title>
            <link>https://medium.octue.com/72-hours-at-windeurope-populating-data-5637871596a?source=rss-3dabf88fa82b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5637871596a</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[digitalisation]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[wind]]></category>
            <dc:creator><![CDATA[Tom Clark]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 07:01:46 GMT</pubDate>
            <atom:updated>2023-04-26T07:01:46.422Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s Wednesday Morning in Copenhagen, and we’re looking at how to populate data into our database. This article is for the fourth of our <em>“Six Steps” </em>— you can read the overview <a href="https://thclark.medium.com/72-hours-of-digitalisation-at-windeurope-7aa786be729d">here</a>.</p><p><em>This article is aimed at engineers, researchers and execs in the Wind Industry, to help understand the process of digitalisation. If you’re qualified in Systems Architecture or Data/Software Engineering, you’re way ahead of this; just get stuck in already!!</em></p><h3>About the ‘Populate’ step</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e88XMz1imeE5Sc2-8MH75A.png" /></figure><p>There’s really not a lot we can say about populating your data store, because this is the bit that gets totally different each time. In some cases, this might not even be necessary — for example if you’re tying into an existing store. You can safely skip this step if that’s the case.</p><h3>Just show me the code!</h3><p><a href="https://github.com/octue/windeurope72hours-elevations-populator-private">With pleasure :)</a></p><h3>Populating Elevations Data</h3><h4>Ensuring provenance</h4><p>We should always be delivering data with clear sourcing. I can’t stand it when you get data from Google or somewhere, and it seems legitimate but isn’t sourced. You can’t use that for science!</p><p>We’d forgotten this in our brainstorming, so added an extension to the database graph we developed earlier. This allows us to specify the source of the dataset with a proper reference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rhjJoOxedABLlnb2juUCPQ.png" /><figcaption>Adding a graph node to preserve scientific provenance of the data</figcaption></figure><h4>Data sources</h4><p>The underlying dataset we used to provide the elevations is the Copernicus DEM — Global and European Digital Elevation Model (COP-DEM) GLO-30 dataset:</p><ul><li>DOI: <a href="https://doi.org/10.5270/ESA-c5d3d65">https://doi.org/10.5270/ESA-c5d3d65</a></li><li>Direct link: <a href="https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model">https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model</a></li></ul><p>We accessed it via the AWS S3 mirror, which provides easy access to the dataset’s GeoTIFF files:</p><ul><li>Information: <a href="https://copernicus-dem-30m.s3.amazonaws.com/readme.html">https://copernicus-dem-30m.s3.amazonaws.com/readme.html</a></li><li>URL: <a href="https://copernicus-dem-30m.s3.amazonaws.com/">https://copernicus-dem-30m.s3.amazonaws.com</a></li><li>S3 URI: s3://copernicus-dem-30m/</li></ul><p>While developing the populator, for cross-validation purposes, we developed a <a href="https://github.com/octue/windeurope72hours-elevations-populator-private/blob/main/scripts/plot_elevations.py">short script to plot data over a map on a Plotly chart </a>— if we get time, we’ll refactor that into an Observable for people to use.</p><h4>About Resolution</h4><p>Elevations in GLO30 go down to 30 arcseconds spatial resolution, which varies depending on where you are on the globe but broadly is about 30m.</p><p>Looking up the H3 cell statistics, we see that Level 12 hexagons have an edge length of ~10m, making Level 12 the first level that’s finer than the spatial resolution of the data itself. So there’s no point going finer than this; we don’t populate any cells lower than L12.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7dscq5QXGeVX3OHFDpsutQ.png" /><figcaption>These Level 12 hexagons have a 10m edge length. You can see the underlying resolution of the data in this view over some rough terrain. Note that the slight oversampling of the dataset is apparent in the pattern here, but not perfectly because of the nearest neighbour sampling and the different grids.</figcaption></figure><h4>How we populate higher levels</h4><p>The L12 cells are populated by nearest-neighbour sampling the original TIFF files using the centre of the hexagon cell. That’s ideal, because L12 cells are smaller than the grid size.</p><p>But Level 12 is a lot of data to render if you want to cover a whole country! What if we need something coarser? Rather than attempting to analyse the raw data, we were able to take advantage of the graph structure to aggregate values up to coarser resolutions:</p><ul><li>Populate all L12 hexagons in an area.</li><li>To populate a parent L11 cell, take the seven L2 hexagons inside it and average them.</li><li>Keep going until reaching Level 8 (simply because we didn’t think coarser cells would be very useful).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsOWXRmBX5YxG_kRHegi_w.png" /><figcaption>Aggregate up to coarser levels by averaging seven values per parent hexagon</figcaption></figure><p>There are some very powerful ways of doing aggregation in databases, but we stayed simple and just wrote some python code inside the populator service. Easy wins the day!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tlkcBeQ6tzB_xbF5xaJoRg.png" /><figcaption>Levels 9 through 12 of refinement in the same region used for cross-checking above</figcaption></figure><h4>Engineering the Populator service</h4><p>We used <a href="https://octue-python-sdk.readthedocs.io/en/latest/">our own SDK to create and deploy a service</a> to Google Cloud Run. The point of our SDK is to help wrap scientific code and provide it as a data service, so we’re constantly working on features to give the extra helping hand.</p><p>It works on a “question-answer” model. Some features we’re proud of:</p><ul><li>Under the hood it uses an event stream to initiate a “question”, and a second (question-specific) event stream to manage communication (for logs, monitor metrics, progress updates, and optionally a final “answer”).</li><li>It handles errors by capturing all the inputs of a question, so an error can be reproduced for investigation with single line of code.</li><li>Services using the framework can ask each other questions.</li><li>Extra tools for querying and managing files in cloud object stores.</li></ul><p>The populator service only makes use of the most basic aspects of the framework. It sits there. The API can ask it to add a list of hexagons to the database.</p><h4>Security</h4><p>We added a “Secret” to the service (you can Secret Manager resources in our terraform config) and the purpose of that is to add credentials for accessing the database. Never commit credentials to source code!</p><h4>Cross checking</h4><p>We started without using the database at all — just plotting directly. to make sure we were pulling out regions correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EG6ocOjYPhcRHwRlDNIJQ.png" /><figcaption>We used a variety of different locations and maps, plotting hexagons with non-zero opacity to check that we were correctly correlating values with features in the landscape.</figcaption></figure><h4>Lazy Loading</h4><p>The populator is set up to load just a region around a selected area. This allows us to lazily-load data on demand by sending a question to the populator (from the API service).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5637871596a" width="1" height="1" alt=""><hr><p><a href="https://medium.octue.com/72-hours-at-windeurope-populating-data-5637871596a">72 hours at WindEurope: Populating data</a> was originally published in <a href="https://medium.octue.com/">Octue</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>