<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kel Markert on Medium]]></title>
        <description><![CDATA[Stories by Kel Markert on Medium]]></description>
        <link>https://medium.com/@kel.markert?source=rss-15d16d84f736------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*H-VKvwfS1rw65MDZA9cJGw.jpeg</url>
            <title>Stories by Kel Markert on Medium</title>
            <link>https://medium.com/@kel.markert?source=rss-15d16d84f736------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:25:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kel.markert/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Installing Julia on Vertex AI Workbench instances: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@kel.markert/installing-julia-on-vertex-ai-workbench-instances-a-step-by-step-guide-e774bd687bb2?source=rss-15d16d84f736------2</link>
            <guid isPermaLink="false">https://medium.com/p/e774bd687bb2</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[julialang]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <dc:creator><![CDATA[Kel Markert]]></dc:creator>
            <pubDate>Mon, 19 Feb 2024 21:15:34 GMT</pubDate>
            <atom:updated>2024-02-19T21:15:34.035Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eu6uZoNrXTr1WLC_gNYIHA.png" /><figcaption>Image of the Jupiter system taken from the James Webb Telescope. Image credit: <em>NASA, ESA, CSA, Jupiter ERS Team; image processing by Ricardo Hueso (UPV/EHU) and Judy Schmidt. </em><a href="https://blogs.nasa.gov/webb/2022/08/22/webbs-jupiter-images-showcase-auroras-hazes/">https://blogs.nasa.gov/webb/2022/08/22/webbs-jupiter-images-showcase-auroras-hazes/</a></figcaption></figure><p>For those that don’t know, <a href="https://julialang.org/">Julia</a> is a high-performance, dynamic programming language gaining traction in various domains like data science, machine learning, and scientific computing. <a href="https://cloud.google.com/vertex-ai/docs/workbench/instances/introduction">Vertex AI Workbench intances</a> provide a cloud-based Jupyter notebook environment for data exploration and analysis that integrates with other services and the larger ML/AI ecosystem on Google Cloud Platform (GCP).</p><p>This blog post will guide you through the process of installing Julia and its kernel on your Vertex AI Notebook Instance, enabling you to leverage its capabilities with the rest of GCP services within the familiar Jupyter notebook interface.</p><p>While I don’t use Julia every day for work, I am still a fan of the language a think it is valuable to know how to get started with it on GCP.</p><blockquote>Disclaimer: The opinions stated here are my own, not necessarily those of my employer.</blockquote><h3>Prerequisites:</h3><ul><li>A Google Cloud Platform (GCP) project with Vertex AI APIs enabled.</li><li>A Vertex AI Notebook Instance running.</li><li>Basic familiarity with command-line interface (CLI) and Jupyter notebooks.</li></ul><p>This blog does not go through the process of setting up a Vertex AI Workbench instance because there are already great docs on how to do that. If you need to see how it is done, please look here: <a href="https://cloud.google.com/vertex-ai/docs/workbench/instances/create">https://cloud.google.com/vertex-ai/docs/workbench/instances/create</a></p><h3>Steps:</h3><ol><li><strong>Connect to your Notebook Instance:</strong> Open the Cloud Console and navigate to Vertex AI -&gt; Workbench. Select your notebook instance and click on “Open JupyterLab”</li></ol><p>2. <strong>Open a new Terminal session in the Notebook interface</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAM3dozp_Zp2E8aBcBYtdA.png" /><figcaption>Screenshot of the Notebook Launcher page and where to click to start a new terminal</figcaption></figure><p>3. <strong>Install Julia:</strong></p><p>Julia will be installed manually by getting the official binary from the Downloads page.</p><p>First off, set the correct version you need. Here v 1.10.1 is set as this is the latest stable release, but pick whichever one you need.</p><pre>JULIA_VERSION=1.10.1</pre><p>Next, download the tarball with all of dependicies:</p><pre>wget https://julialang-s3.julialang.org/bin/linux/x64/${JULIA_VERSION%.*}/julia-${JULIA_VERSION}-linux-x86_64.tar.gz</pre><p>Unpack the tarball into a local directory:</p><pre>tar -xvzf julia-${JULIA_VERSION}-linux-x86_64.tar.gz</pre><p>Copy the data to a /opt/ directory where all the software and add-on packages that are not part of the default installation should go. Also make a symlink to the julia binary so it can be used throughout the environment:</p><pre>sudo cp -r julia-${JULIA_VERSION} /opt/<br>sudo ln -s /opt/julia-${JULIA_VERSION}/bin/julia /usr/local/bin/julia</pre><p>[Optional] delete the tarball and upacked data. Everything is copied over to /opt/ directory so it is safe to clean up the working directory:</p><pre>rm -rf julia-${JULIA_VERSION}*</pre><p>4. <strong>Install the IJulia Kernel:</strong></p><p>The last step is to install the IJulia kernel which will allow users to use Julia as the backend for notebooks on their instance.</p><p>In the teminal run the following command:</p><pre>julia  -e &#39;using Pkg; Pkg.add(&quot;IJulia&quot;)&#39;</pre><p>5. <strong>Start using Julia</strong></p><p>With everything now setup, you should see the option to select the Julia kernel for the version you just installed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFVArw2Zbgn0bCj9Mko6Ag.png" /><figcaption>Screenshot of the notebook Launcher page now with a Julia kernel option</figcaption></figure><p>Now you can start using Julia for your data science processing and even building models on GCP!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YUteuZ5eadhxFm_c0g5bjQ.png" /><figcaption>Screenshot of a notebook using Julia kernel backend that was installed</figcaption></figure><p>Remember, this is a base installation of Julia so this comes with a brand new environment. You can install packages and manage different Julia environments and versions using Pkg within Julia itself.</p><h4>Bonus Content</h4><p>Vertex AI Workbench instance allow you to spin up an environment with a GPU attached. This is great because it allows you to install a few extra packages within Julia and get started with model development and training. Here is a quick example of the <a href="https://fluxml.ai/Flux.jl/stable/models/quickstart/#man-quickstart">Quickstart </a>using <a href="https://fluxml.ai/">Flux package</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-Z2ofrDX2CbSS5Y-NjP_A.png" /><figcaption>Screenshot showing the installation of packages and the Flux Quickstart example in a notebook</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XgWQEJtjYBQuNN46KhNodA.png" /><figcaption>Screenshot of the model training results from using Julia and Flux on a GPU enabled notebook</figcaption></figure><h3>Conclusion:</h3><p>By following these steps, you can easily install and use Julia on your Vertex AI Workbench instance, adding another tool in the belt for data science and scientific computing capabilities within the cloud-based Jupyter notebook environment. Remember to consult the official Julia and IJulia documentation for further exploration and advanced use cases.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e774bd687bb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Google Earth Engine with Julia]]></title>
            <link>https://medium.com/@kel.markert/using-google-earth-engine-with-julia-ca65c1b1e850?source=rss-15d16d84f736------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca65c1b1e850</guid>
            <category><![CDATA[julialang]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <dc:creator><![CDATA[Kel Markert]]></dc:creator>
            <pubDate>Sat, 05 Sep 2020 00:33:42 GMT</pubDate>
            <atom:updated>2020-09-05T17:00:47.459Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cloud-based geospatial processing with a modern state-of-the-art programming language</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*RG_UZklTxDkAzSkh" /><figcaption>Image credit: <a href="https://unsplash.com/@alesnesetril">Ales Nesetril</a> on <a href="https://unsplash.com/photos/Im7lZjxeLhg">Unsplash.com</a></figcaption></figure><h3>Introduction</h3><p><a href="https://earthengine.google.com/">Google Earth Engine</a> is a powerful cloud-based geoprocessing platform that allows users access to petabytes of earth observation and model data with collocated compute. The Earth Engine (EE) platform is an invaluable resource for Earth science research and applications and has revolutionized the Earth science/geospatial field due to it ease of use and allowing users to quickly prototype algorithms and produce outputs at planetary scale. To interface with the platform users develop workflows using either a JavaScript and/or Python API. By having only a JavaScript or Python API limits some developers to those languages. However, projects (like <a href="https://github.com/r-spatial/rgee">rgee</a>) have emerged to allow developers to interface with EE in their favorite languages.</p><p><a href="https://julialang.org/">Julia</a> is a modern dynamic, high-level programming language, for scientific and numerical computing. It has the feel of a scripting language with the performance compiled languages (thanks to its JIT compilation). It is a programming language full of features for technical computing and is gaining traction for scientific computing.</p><p>This blog post walks though setting up the Julia environment for interfacing with EE API. Additionally, examples are provided to highlight syntax and capabilities. Lastly, new EE features are shown together with Julia to highlight where the real benefit of using the two together can be.</p><p><em>Disclaimer: </em>I have written this article as part of my process in learning Julia. This is meant to be an example for others and a reference for myself. I do not focus on comparing different languages as all languages have their own inherent merits. I hope others find it useful. All opinions in this article are my own.</p><h3>Setting up the environment</h3><p>To use Earth Engine (EE) with Julia, we will use the existing Python API and call the functions using Julia. This is done through Julia’s <a href="https://github.com/JuliaPy/PyCall.jl">PyCall package</a> but we will need to install the EE API for use within the Julia environment (and a few other google cloud packages for later on):</p><pre>using Pkg;<br>Pkg.add(&quot;PyCall&quot;);<br>Pkg.add(&quot;Conda&quot;);<br>Pkg.add(&quot;Plots&quot;); # used for plotting<br>Pkg.add(&quot;Colors&quot;);<br>Pkg.add(&quot;FileIO&quot;);<br>Pkg.add(&quot;ImageIO&quot;);<br>Pkg.add(&quot;Images&quot;);</pre><pre>using Conda;<br>Conda.add(&quot;earthengine-api&quot;,channel=&quot;conda-forge&quot;);<br>Conda.add(&quot;google-api-python-client&quot;,channel=&quot;conda-forge&quot;);<br>Conda.add(&quot;google-auth-httplib2&quot;,channel=&quot;conda-forge&quot;)</pre><p>After we have the environment set up, we can import the package, authenticate, and see if the we can initialize an EE session.</p><pre>using PyCall;<br>ee = pyimport(&quot;ee&quot;);<br>ee.Authenticate(); # authentication is a one-time process<br>ee.Initialize();</pre><p>If all goes well and there are no errors when importing the EE package and initializing a session 🤞, then EE can be used with Julia!</p><h3>EE+Julia Examples</h3><p>To get started illustrate how to execute EE workflows using Julia, some <a href="https://colab.research.google.com/github/google/earthengine-api/blob/master/python/examples/ipynb/ee-api-colab-setup.ipynb">examples using the Python API</a> are replicated using Julia syntax.</p><h4>Hello EE!</h4><p>The first example is focused on importing the packing and performing a small geospatial process. Here the SRTM elevation data is imported and queried at the geospatial coordinates of Mount Everest to get the elevation value. This uses an ee.Image and ee.Geometry to make sure everything is working correctly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fae7a9ddd706c876a35fe1a0d2da699/href">https://medium.com/media/7fae7a9ddd706c876a35fe1a0d2da699/href</a></iframe><p>As seen, the syntax is <em>almost </em>identical to using the Python API aside from small Julia changes. Specifically, Julia uses <a href="https://docs.julialang.org/en/v1/manual/strings/#String-Basics-1">double quotes to define strings</a> most of the time and trying to use single quotations will throw errors. Additionally, when defined the geometry variable xy there is an ellipsis after the coordinate list; this is <a href="https://docs.julialang.org/en/v1/manual/faq/#...-splits-one-argument-into-many-different-arguments-in-function-calls-1">splatting</a> where one argument is split apart into many different arguments for a function with unknown arguments. Again, the syntax is largely the same because we are interfacing with the Python EE package through Julia. More in-depth examples will highlight more Julia syntax.</p><h4>Sampling data from an image</h4><p>As a more extensive example, we will sample data from a raster dataset. This is a common workflow for geospatial sciences whether looking at relationships between variables or sampling data for ML workflows. Here we load in Landsat image, sample band values, and plot the relationship of the bands.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d025c5152b9b9dbccd802ee416e98ef8/href">https://medium.com/media/d025c5152b9b9dbccd802ee416e98ef8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJI7y09UBPqdP9EPsPiL1w.png" /><figcaption>Scatter plot showing the relationship of red and NIR bands. Plot generated from the script above.</figcaption></figure><p>This example ends up loading the data as Julia arrays, this is really helpful if you want to leverage Julia’s powerful processing capabilities. This also highlights using the Julia <a href="http://docs.juliaplots.org/latest/">Plots package</a> for plotting.</p><h4>Image processing and visualization</h4><p>The true value of EE lies in its capability to process large amounts of raster (Earth Observation) data in space and time. This next example is a common workflow to calculate <a href="https://en.wikipedia.org/wiki/Normalized_difference_vegetation_index">NDVI</a> from a Landsat 5 image. For this example we use Julia to define a function, apply it to the image, and pull the results for visualization.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e7e3d557efd7b03436213f89dd52849/href">https://medium.com/media/8e7e3d557efd7b03436213f89dd52849/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W-ekaWbOWL3-o8pPF-9NmQ.png" /><figcaption>Landsat 5 NDVI image result from the above script.</figcaption></figure><p>As we can see from the example, we can access an image, apply a function on the image and visualize the results. We use Julia to handle a lot of the nitty-gritty details of getting a colormap in hex color code with ease allowing users to focus more on the science. These concepts can be abstracted to apply over EE image collections and get results.Also, there is a whole lot more EE functionality that we can access using Julia via the Python API but this would be a <strong><em>very</em></strong> long article if I were to highlight everything.</p><p>The code for all of these examples can be accessed here: <a href="https://gist.github.com/KMarkert/308d2d81eb460603a7ecfa118907ad23">https://gist.github.com/KMarkert/aa90601ca588b97a507df5a056e73892</a></p><h3>Why use Julia to interface with EE?</h3><p>This may be going on in your head, “why use Julia if you are simply calling the Python API?” This is a fair question since this is just interfacing with the Python EE API using there would be minimal performance gains. One answer is user preference in programming languages. However, most of the advantages of Julia come from working earth observation/remote sensing data outside of EE. The Earth Engine team recently released a <a href="https://developers.google.com/earth-engine/reference">cloud REST API for interacting with EE</a> that allows user to access data either <a href="https://developers.google.com/earth-engine/reference/Quickstart#fetching-pixel-values">directly from the EE data catalog</a> or <a href="https://developers.google.com/earth-engine/Earth_Engine_REST_API_computation">after EE has performed some computations</a>! This is huge as it allows processes to be chained together leveraging EE to handle the big computations on EO data and use Julia for ML, modeling, or whatever your heart desires!!!</p><h4>Julia and the EE REST API</h4><p>🚨<em>Setup required!</em>🚨<em> </em>Before going through the REST API examples, please make sure you setup your cloud project and get a service account secret key to access the REST API. Details on how to do this can be found <a href="https://developers.google.com/earth-engine/reference/Quickstart#set-up-your-colab-notebook">here</a>.</p><p>To start off, we will setup the environment and see if we can request information on the Landsat collection. First, we load in the Python Google authentication packages that we installed earlier (not sure if there are Julia packages available to do this), then we authenticate a session using our credentials, and request information from a URL endpoint that corresponds to the Landsat.</p><pre>using PyCall, JSON;</pre><pre>service_account = pyimport(&quot;google.oauth2.service_account&quot;);<br>gauth = pyimport(&quot;google.auth.transport.requests&quot;);</pre><pre>SECRET_KEY = &quot;&lt;PATH_TO_SECRET_KEY&gt;&quot;</pre><pre>credentials = service_account.Credentials.from_service_account_file(SECRET_KEY);<br>scoped_credentials = credentials.with_scopes(<br>    [&quot;https://www.googleapis.com/auth/cloud-platform&quot;]);</pre><pre>session = gauth.AuthorizedSession(scoped_credentials);</pre><pre>url = &quot;<a href="https://earthengine.googleapis.com/v1alpha/projects/earthengine-public/assets/LANDSAT">https://earthengine.googleapis.com/v1alpha/projects/earthengine-public/assets/LANDSAT</a>&quot;;<br>       <br>response = session.get(url);<br>asset = JSON.parse(response.content)</pre><p>If all goes well from the code above the contents of the variable asset should look like this:</p><pre>Dict{String,Any} with 3 entries:<br>  &quot;name&quot; =&gt; &quot;projects/earthengine-public/assets/LANDSAT&quot;<br>  &quot;id&quot;   =&gt; &quot;LANDSAT&quot;<br>  &quot;type&quot; =&gt; &quot;FOLDER&quot;</pre><p>This means we are able to successfully authenticate to use the EE REST API and get results 🎉! It is nice to be able to access the EE catalog metadata using simple HTTP requests but there is more…now we can chain together more extensive workflows to access raw data and read into Julia as arrays.</p><h4>Requesting data</h4><p>To access data directly from EE we create a session to send HTTP requests to EE’s REST API, define some parameters such as the asset to query and spatial domain to request data, then send the request and read as a Julia array. Once we have the raw data as a Julia array, we can leverage the high performance of Julia for additional processing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2bc96d228d1ef1eba852240bb1299bd/href">https://medium.com/media/e2bc96d228d1ef1eba852240bb1299bd/href</a></iframe><p>Link to script: <a href="https://gist.github.com/KMarkert/055a9334d64ad8e2d9e1d6b146f5e5a0">https://gist.github.com/KMarkert/055a9334d64ad8e2d9e1d6b146f5e5a0</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPsSElpXH_nZz22-BKroBw.png" /><figcaption>Sentinel-2 image over the San Francisco Bay Area acquired 2017–04–30. Image created with Julia using data from the Earth Engine REST API from the example script above.</figcaption></figure><p>This was a straightforward example on accessing data from EE using the REST API and visualizing the image. However, there are a whole lot more capabilities available now using the REST API particularly with <a href="https://developers.google.com/earth-engine/Earth_Engine_REST_API_computation#serialize_the_expression_graph">serializing an expression graph</a> and getting the results. Retrieving large amounts of data from EE can be cumbersome and Julia will be able to handle the data and process with ease. Imagine whatever Earth science based process that you wished EE could do and now do it. This is something I am particularly excited about as it allows for more flexibility in working with EE!</p><h3>Concluding remarks</h3><p>Julia is a modern programming language that focuses on performance for technical computing whereas Earth Engine is a cloud-based geoprocessing platform that allows access to petabytes of Earth science data. Using the two together, we can leverage powerful geospatial processing with expressive syntax. Moreover, we can access data directly or after some computations from Earth Engine and work with the results directly in Julia. While the concepts of interfacing with EE and using the REST API are transferable to other languages, <strong>I foresee the EE+Julia being a formidable combination 💪 in the future of geospatial computing and Earth science applications </strong>(this is my opinion). I say this due to the straightforward and expressive syntax that Julia provides and powerful compute that the EE platform and Julia both bring to the table.</p><p>I plan on using EE with Julia and exploring more on how to access compute results and pass them into external processes like ML workflows or hydrology models using Julia. I hope to see Julia’s use with EE and in the broader Earth science field!</p><p>I would lastly like to thank the <a href="https://medium.com/u/5995184886b6">Google Earth</a> team for everything they do and revolutionizing the Earth science field with the platform.</p><blockquote>Circling back to my earlier point of packages in other languages to interface with EE, I would <strong>love</strong> to see an EE.jl package (or something like that) and if there is interest in planning/implementing something like this, please get in contact with me via <a href="https://twitter.com/KelMarkert">Twitter</a> or on <a href="https://github.com/kmarkert">GitHub</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca65c1b1e850" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing Python and Julia for hydrologic modeling]]></title>
            <link>https://medium.com/@kel.markert/comparing-python-and-julia-for-hydrologic-modeling-7334ffa9534b?source=rss-15d16d84f736------2</link>
            <guid isPermaLink="false">https://medium.com/p/7334ffa9534b</guid>
            <category><![CDATA[hydrology]]></category>
            <category><![CDATA[julialang]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Kel Markert]]></dc:creator>
            <pubDate>Thu, 03 Sep 2020 21:18:06 GMT</pubDate>
            <atom:updated>2020-09-04T12:25:41.722Z</atom:updated>
            <content:encoded><![CDATA[<h4>Benchmarking model run and calibration speeds</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*6r_irN52Lort06aX" /><figcaption>Tree-lined stream. Image credit: <a href="https://unsplash.com/@jonflobrant">Jon Flobrant</a> on <a href="https://unsplash.com/photos/rB7-LCa_diU">Unsplash.com</a></figcaption></figure><p>Hydrologic modeling is an import tool for understanding the distribution of water in both space and time without directly measuring specific variables. The ultimate aim of using such models is to assist in decision-making of water related challenges, such as flood protection and water resources planning. Given that the models are based on physical processes, there is quite a bit of math (computations) involved. Furthermore, hydrologic models are iterative in nature, meaning that the equations need to be solved over and over again for an individual model run. Lastly, there are many parameters that need to be calibrated for different models leading to the need for optimization algorithms that are also computationally expensive. All this to say, hydrology models require a lot of computational resources and efficient implementation is required to produce timely results.</p><p>Python is an extremely popular programming language for scientific computing due to the low barrier to entry for translating ideas into code and generating actual scientific data. Python leverages packages like NumPy, Pandas, and others that provide efficient frameworks for scientific computing. However, there is one major downfall in using Python for scientific computing, speed. I know I personally have spent a while waiting on Python to run. Other programming exist that are comparable to Python for scientific programming, such as R, but one in particular has been getting a lot of attention and poised to take over the scientific computing domain, <a href="https://julialang.org/">Julia</a>. Julia is a fastest modern high performance open source computing language for data, analytics, machine learning and artificial intelligence that <a href="https://towardsdatascience.com/bye-bye-python-hello-julia-9230bff0df62">prides itself on processing speed along with many other great features</a>. Given that hydrologic modeling takes a good amount of computational resources, this article aims to benchmark a model implementation in Python and Julia.</p><p><em>Disclaimer:</em> I have been using Python for scientific computing for 6 years now, I really enjoy Python and this is not meant to degrade Python. I am learning Julia and performed this comparison as part of my learning process. There may be ways to better implement the code and the speed tests may not be 100% accurate. While I do my best to support my comments with facts, there are opinions in the article.</p><h3>Experimental setup</h3><p>To perform a benchmarking an experiment was designed to mimic a real-world hydrology application in which Python or Julia could be chosen to run a model. For Python, <a href="https://pandas.pydata.org/">Pandas</a> was used to handle reading of CSV files and data structure and <a href="https://numpy.org/">NumPy</a> was used to handle the array processing and random sampling. For Julia, the <a href="https://juliadata.github.io/CSV.jl/stable/">CSV package</a> was used to handle reading of CSV files, the <a href="https://juliadata.github.io/DataFrames.jl/stable/">DataFrames package</a> was used to handle the data structure, the <a href="https://juliastats.org/Distributions.jl/stable/">Distributions package</a> was used for random sampling. The model implementations are near identical (as best as I could do) aside from small syntax differences to ensure that model runtime differences are based off of language choice and not due to how the model was implemented. All plots were made using <a href="http://docs.juliaplots.org/latest/">Julia’s Plots package</a>.</p><p>The code for the Python and Julia HYMOD implementations with the input data for this example can be found at the following GitHub repo: <a href="https://github.com/kmarkert/hymod">https://github.com/kmarkert/hymod</a></p><h4>Hydrologic model</h4><p>To test processing speeds for hydrologic modeling using Python and Julia the HYMOD model was implemented in both languages. The HYMOD model is a conceptual, lumped rainfall-runoff model that estimates streamflow/discharge for a basin at a daily time step based off of excess precipitation that gets converted to runoff. Evapotranspiration (ET) for the basin is equal to the Potential ET (PET) if the water is available, where PET is estimated using the Hargreaves formula. Streamflow is calculated by dividing runoff into quick flow and slow flow components, which are routed through three identical quick flow tanks (Q1, Q2, Q3) and a parallel slow flow tank. The routing model used is a linear reservoir routing model. There are a total of five parameters that need to be calibrated which control how much runoff occurs based on soil holding capacity, the fraction of runoff that goes into the quick and slow flow reservoirs, and the residence time of the quick and slow reservoirs. More details regarding the model and parameters can be found in the following paper.</p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//piahs.copernicus.org/articles/368/180/2015/piahs-368-180-2015.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d27247e5851896d592ef2e781eefa16/href">https://medium.com/media/6d27247e5851896d592ef2e781eefa16/href</a></iframe><p>This model was chosen for this comparison due to a couple reasons: 1) it is a relatively straightforward model to implement, 2) the model requires few input data, 3) the model has relatively few parameters to calibrate, and 4) the model is light-weight with a quick runtime. Ultimately, this allows for quick setup with minimal effort and the ability to run the model quickly many times (as seen later).</p><h4>Study area and data</h4><p>To run the HYMOD model daily precipitation and PET data is needed. To calculate PET, minimum and maximum temperature data is needed as well. These variables are averaged over the basin. The basin used for this example is the Upper Mekong basin (shown below) from 1986–01–01 to 2006–12–31. <a href="https://earthengine.google.com/">Google Earth Engine</a> was used to create the forcing data over the basin: <a href="https://code.earthengine.google.com/f3cb5014774ab6f0207692f84cefceaa">https://code.earthengine.google.com/f3cb5014774ab6f0207692f84cefceaa</a>. <a href="https://www.chc.ucsb.edu/data/chirps">CHIRPS precipitation</a> and the <a href="https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5">ERA5 reanalysis</a> datasets were used for the precipitation and temperature variables, respectively.</p><p>Observed gauge data for the basin was gathered from the Mekong River Commission data portal: <a href="https://portal.mrcmekong.org/time-series/chart?ts=4233eff2e779473da32364d2b31d6006&amp;sd=1985-01-01&amp;ed=1995-12-31">https://portal.mrcmekong.org/time-series/chart?ts=4233eff2e779473da32364d2b31d6006&amp;sd=1985-1-01&amp;ed=1995-12-31</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c07cd4edc53b745b5efd8e5d6688bf0c/href">https://medium.com/media/c07cd4edc53b745b5efd8e5d6688bf0c/href</a></iframe><h4>Comparison</h4><p>The model runtime using Python and Julia is compared using the model just once to understand the runtimes for single model run. Then the model is calibrated in Python and Julia using 5, 10, 25, 50, 100, 500, 1000, 5000, 10000, 50000, and 100000 iterations. Multiple runs are used to understand the how the model implementations compare in terms of time complexity and if a language is best for differing number of calibration iterations.</p><p>Calibrating a model requires initially selecting random model parameters and optimizing the parameters by running the model thousands of times. The calibration process implemented for this application is a simple randomized search calibration where parameters values are randomly selected <em>n</em> times from a uniform distribution of the parameter space. The model is run <em>n </em>times and the best performing model parameters are selected. The <a href="https://en.wikipedia.org/wiki/Nash%E2%80%93Sutcliffe_model_efficiency_coefficient">Nash-Sutcliffe model efficiency coefficient</a> (NSE) was used to score the model for calibration, where a perfect model has a score of 1 and models over 0.5 are generally accepted as “good”. There are more efficient model calibration procedures but this is simple to implement and gives a good indication of model run times due to iterating model runs <em>n</em> times.</p><p>Now onto the results!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/0*3KgVDLVHQfb4e5wY.jpg" /></figure><h3>Results</h3><h4>One-time model run</h4><p>As stated above the model was executed once to understand how the two implementations perform for a single model run. Here are the results:</p><pre>Julia model run time: 6.977403 seconds<br>Python model run time: 0.251087 seconds</pre><p>From this result, Python executes the model faster than Julia. But wait a second…based on what is on the internet, isn’t Julia supposed to be faster? Well, there is a small detail that explains why Julia took a little while longer. Julia uses <a href="https://en.wikipedia.org/wiki/Just-in-time_compilation">just-in-time compilation</a> strategy which has a noticeable delay in initial execution of applications, however, after the compilation it is much faster. Here is the runtime after the initial run with compilation:</p><pre>Compiled Julia model run time: 0.024758 seconds</pre><p>That is much faster than the original Julia execution! We see that the compile time was most of the time for running the model using Julia. When running the model once compared to Python, <em>Julia is 914.16% faster!!! </em>That is a significant speedup but may not be noticeable since the runtimes are still less than one second. This increase in speed becomes more apparent when running the model a large number of times or when running multiple models for user-facing applications.</p><h4>Model calibration</h4><p>As mentioned above, the model calibration process was run for multiple iterations. The model was compiled for Julia with with a dry-run before the timing the calibration process. From the figure below, we can see that both implementations seem to have O(n) time complexity where processing time scales linearly with number of iterations. This is expected as the calibration procedure simply ran the model over and over again and selected the best case, so more model runs == more time. However, the Julia implementation is much, much faster for calibrating as seen in the figure below (note the log scaling on the plot axes to help with interpretation). For example, Python took 56.6 seconds to run 10000 iterations of the model whereas Julia only took 1.2 seconds. For 100000 iterations, Python took 90 minutes and Julia took less than 2 minutes! On average the calibration using Julia is 3930.72% faster than running the calibration with Python. This seems like a large difference but the increased speed is, again, largely due to Julia compiling the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ulhobd7H6dZTv4h_Tq6FQ.png" /><figcaption>Model calibration runtimes for different number of iterations. Note the log scale for the x and y-axis to help with interpretation.</figcaption></figure><p>Julia obviously wins in terms of computation speed, especially with a large number of iterations. Regardless if you want to use Julia or Python for model runs or calibrations, the model must produce accurate results. So. why do we need to run the model that many time? To answer that question, most calibration algorithms recommend running the model using 5000 to 10000 iterations to achieve acceptable results. So, when using an optimization algorithm we can have our results an order of magnitude quicker using Julia. The following figure illustrates model accuracy score from this randomized search calibration procedure comparing the NSE metric as a function of number of iterations. As seen, the model starts to reach an acceptable NSE score after 1000 iterations, which can take a few minutes in Python. From my experience using Python, the model runtime adds up quickly when calibrating many small sub watersheds over a domain. While this finding has somewhat to do with random chance, having more model iterations usually improves results up to a point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5DnhWwgN-h8GA_MpA1dnQw.png" /><figcaption>Model accuracy after calibration as a function of number of iterations. Note log scaling on x-axis to help with plot interpretation.</figcaption></figure><p>Lastly, because I am a scientist and like to see how well the model performs to observed data, displayed below is a hydrograph from the calibration. The following model run was created using one million model iterations with a final NSE score of 0.665, which is fairly good especially for this simplistic model. It is seen that the calibrated HYMOD model performs quite well for simulating discharge 🎉, particularly during the wet season (high flow periods) and is able to capture the changes between the wet and dry season. <em>Fun note:</em> The HYMOD model does not take into account snowmelt discharge which is the greatest contributing factor in dry-season streamflow for this basin. This is why we see a lower modeled discharge during the low flow periods. If a snow melt module were to be added, I suspect the results would improve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v0rh1aoJMbdt4EnAJ3JiEw.png" /><figcaption>Calibrated discharge for the Upper Mekong basin after 5 million iterations. NSE score is 0.665</figcaption></figure><h3>Final thoughts</h3><p>It is clear that Julia out performs Python in terms of processing speed and should be used for intense hydrologic computations, like calibration, over Python. Although this example provides some insight on how the two programming languages compare in terms of speed for hydrologic modeling, this was a relatively simple application with a computationally light model. I expect that more computationally heavy models (such as VIC or Noah-MP) will have added benefits from using Julia.</p><p>Although compute time is important, it is not the only reason why one may chose to implement workflows in a particular language. For example, development speed, code readability, as well as code base maintainability are important factors. In my case, this is the first time I coded in Julia and it took me one full day to implement the model and a few extra days to debug/figure things out. I contribute the quick development time to the Julia language design/syntax and not my coding abilities. The development speed observed from my end and Julia’s ease of use is a huge bonus for me in terms of using Julia as new language for development. Also worth noting is the that installing Julia and managing packages is really smooth, the same usually cannot be said about Python…</p><p>After having used Julia for this little project to give it a test run, I am sold! I really enjoy the syntax, the process of installing packages and getting things to work is smooth, and the processing speed is really great (especially after getting used to waiting for Python). I hope others in the Earth science field find this little comparison useful. I know I will work to implement future Earth Science based projects using Julia and hope to see its use more in the field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7334ffa9534b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>