<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tom Augspurger on Medium]]></title>
        <description><![CDATA[Stories by Tom Augspurger on Medium]]></description>
        <link>https://medium.com/@tom.augspurger88?source=rss-b66b09f4f9a5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*HQQOH_YwRGa5teSM3OqUGg.jpeg</url>
            <title>Stories by Tom Augspurger on Medium</title>
            <link>https://medium.com/@tom.augspurger88?source=rss-b66b09f4f9a5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:42:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tom.augspurger88/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Managing Environments]]></title>
            <link>https://medium.com/pangeo/managing-environments-5604d848367a?source=rss-b66b09f4f9a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5604d848367a</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[conda]]></category>
            <dc:creator><![CDATA[Tom Augspurger]]></dc:creator>
            <pubDate>Mon, 01 Jun 2020 15:12:59 GMT</pubDate>
            <atom:updated>2020-06-01T15:12:59.624Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managing Pangeo Environments for Distributed Computing</h3><p><em>By </em><a href="https://medium.com/u/b66b09f4f9a5"><em>Tom Augspurger</em></a><em>, </em><a href="https://medium.com/u/a2ec3bba4164"><em>Scott Henderson</em></a><em>, and </em><a href="https://medium.com/u/df9f03847f67"><em>Joe Hamman</em></a></p><p>Managing computational environments has been a persistent challenge for both administrators and users of Pangeo distributed computing infrastructure. While there are still improvements to be made, we’re reasonably happy with our current setup. This post will discuss</p><ol><li>The challenges we face,</li><li>Our goals for Pangeo’s environment management system,</li><li>The solutions we’ve implemented, and</li><li>Guidelines for users wanting to customize an environment.</li></ol><p>But first: what do we mean by environment management? Roughly speaking, we’re talking about the set of packages (typically Python packages with C or Fortran extension modules) available to users when they start a session on Pangeo. A session is typically, though not exclusively, a JupyterLab session.</p><h3>The Challenges</h3><figure><img alt="1987" src="https://cdn-images-1.medium.com/max/492/0*DpBu0ZRMDYWD4WqA.png" /><figcaption><a href="https://xkcd.com/1987/">https://xkcd.com/1987/</a></figcaption></figure><p>While Python environment management can be humorously complex, we have a few additional complexities. First, we’re using <a href="https://dask.org/">Dask</a> for distributed computing, so we need to ensure that the environments on the client, scheduler, and worker machines in the Dask cluster are compatible.</p><p>Second, the user environment has to work in a somewhat complicated “runtime” that involves UI elements like JupyterLab and distributed computing elements like Dask and <a href="https://medium.com/pangeo/pangeo-with-dask-gateway-4b638825f105">Dask Gateway</a>. In the current design, these “runtime” packages must be in the user environment. We’ve found it difficult to ensure that the environment brought by the user contains the versions of, say, Dask that is expected by the runtime. This is a subtle point worth repeating: <em>both users and Pangeo administrators want to dictate specific package versions in the user environment.</em> That negotiation has been one of the toughest technical problems to solve.</p><p>Finally, we can’t tell our users to “just build a Docker image”. We’re using <a href="https://kubernetes.io/">Kubernetes</a> for the cloud deployments, so at the end of the day we do need a Docker image. But building that Docker image has to be Pangeo’s burden, rather than the user’s, who are primarily scientists. We can’t expect that its users are familiar with the intricacies of Docker, and to the extent possible, we’d like our users to not even notice environment management.</p><h3>The Goals</h3><p>In the end, we have a few requirements for our environment management system. We’d like something that</p><ol><li>Builds reproducible Docker images for the client, scheduler, and worker pods that satisfy the runtime constraints of the Pangeo deployment (e.g. the version of Dask-Gateway in the environment must be compatible with the version of Dask-Gateway running on the Kubernetes cluster).</li><li>Provides extension points to customize the built image at various stages, including the set of packages installed, environment variables set, etc., in a way that’s consistent with the first requirement.</li><li>Doesn’t <em>require</em> knowledge of Docker, but provides custom Dockerfiles as an extension point.</li><li>Is reasonably simple to understand and maintain.</li><li>Builds relatively small Docker images for faster startup times when the cluster is scaled.</li></ol><p>To achieve those goals, we’ve implemented a system that’s working pretty well.</p><h3>Our Setup</h3><p>Our current setup involves three high-level pieces:</p><ol><li>Upstream packages</li><li><a href="https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/packages.html#metapackages">Conda metapackages</a></li><li>Pangeo docker images</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*BPbrVfEoTlM4xXEN" /></figure><p>Combined, these pieces resolve to a specific environment for a user. Let’s go through them in turn.</p><p><strong>Upstream packages:</strong> An upstream package is something like Dask or JupyterLab. These are packages <em>in the user environment</em> that are necessary for the “Pangeo runtime”. The Pangeo administrators control this and would like to keep up to date with new releases. We’ve enlisted <a href="https://github.com/pangeo-bot">pangeo-bot</a> to handle regular package updates. When, say, Dask issues a new release to conda-forge, pangeo-bot automatically makes a pull request to update pangeo’s metapackages. More on the pangeo-bot in a future post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*JHM5gdbWX0TSOLV9" /></figure><p><strong>Pangeo’s metapackages: </strong>Pangeo contributors maintain several conda metapackages, a conda package with no files, only metadata. These meta packages allow us to group dependencies and ensure consistent versions across environments.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7938c43b1fd7facd2a744b5881eb8c81/href">https://medium.com/media/7938c43b1fd7facd2a744b5881eb8c81/href</a></iframe><p>Today, we have two meta packages available on conda-forge:</p><ul><li><a href="https://github.com/conda-forge/pangeo-dask-feedstock">pangeo-dask</a>: Packages related to using Dask on Pangeo. These packages need to be in both user and worker environments.</li><li><a href="https://github.com/conda-forge/pangeo-notebook-feedstock">pangeo-notebook</a>: Packages that are only needed in the user notebook environment. Here we include things like JupyterLab and the Dask-labextension.</li></ul><p><strong>Pangeo’s Docker images:</strong> Pangeo deployments are running <a href="https://zero-to-jupyterhub.readthedocs.io/en/latest/">JupyterHub on Kubernetes</a>, which requires that a user’s Jupyter environment be encapsulated in a Docker image. We have created a new repository <a href="https://github.com/pangeo-data/pangeo-docker-images">pangeo-docker-images</a>, which provides continuous building of reproducible Docker images that contain compatible Conda packages using the machinery described above. In brief, this repository controls:</p><ol><li>The <a href="https://hub.docker.com/r/pangeo/base-image/tags"><em>pangeo/base-image</em></a> image, which can be inherited from and customized with <a href="https://repo2docker.readthedocs.io/en/latest/">repo2docker</a>-like extension points.</li><li>Ready-to-use images like <a href="https://hub.docker.com/r/pangeo/pangeo-notebook/tags"><em>pangeo/pangeo-notebook</em></a> and <a href="https://hub.docker.com/r/pangeo/ml-notebook"><em>pangeo/ml-notebook</em></a>, which provide a consistent environment for testing and common use-cases.</li></ol><p>The repo2docker-like extension points in pangeo/base-image are crucial. These are configuration files <a href="https://repo2docker.readthedocs.io/en/latest/config_files.html">specified by repo2docker</a> like <em>environment.yml</em>, <em>postBuild, </em>and<em> start, </em>that allow for users to control certain stages of the Docker building process without having to understand Docker. We previously just used repo2docker directly, but struggled with issues around the complexity of the setup, the size of built images, and Conda timeouts when updating packages. Using a custom base-image gives us more control over how conda commands are run, and allows us to significantly slim down the size of images — our base-notebook environment is now just 250Mb compressed.</p><h3>Customizing environments</h3><p>For Pangeo Binder users, the easiest and most common customization is adding specific packages to the environment. For example, suppose we want an environment with <em>xarray</em>, <em>hvplot</em>, and <em>sat-search</em>. This can be done by including the following files in the binder repo:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e110865d854f95cc9788095677c767e/href">https://medium.com/media/3e110865d854f95cc9788095677c767e/href</a></iframe><p>The <em>binder/Dockerfile</em> ensures that you use our base image, which supports the same customization points as repo2docker. The <em>binder/environment.yml</em> provides a place to specify the conda packages you want in the environment. Including the pangeo-notebook==2020.05.04 metapackage ensures that the versions of <em>dask</em>, <em>jupyterlab</em>, etc. in the environment match the versions expected by the Pangeo runtime.</p><p>The <a href="https://github.com/pangeo-data/pangeo-binder-template">pangeo-binder-template</a> repository provides a template repository with all the necessary configuration for users to create custom environments.</p><p>For many users, however, this even this level of customization is overkill. There’s a set of common packages used by the community that satisfies the needs of most users. So we’ve developed a <a href="https://github.com/pangeo-gallery/default-binder">“default binder” repository</a> with a common set of packages that can be used by many binders. More on that in a future post.</p><h3>Summary</h3><p>Managing computational environments for Pangeo JupyterHubs and BinderHubs has been challenging. Distributed computing means that we have multiple machines whose environments must share a common environment. Additionally, the “Pangeo runtime” requires specific versions of packages installed in the user’s environment. Finally, it’s hard to build a single environment that works perfectly for everyone, so we want to enable users to customize the environment at various places.</p><p>The conda metapackages managed by Pangeo ensures that the user environment has the versions of Dask, Jupyterlab, etc. expected by Pangeo. These can be used with Pangeo’s Docker images with repo2docker-like extension points for successfully creating an environment that fits your needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5604d848367a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pangeo/managing-environments-5604d848367a">Managing Environments</a> was originally published in <a href="https://medium.com/pangeo">pangeo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pangeo with Dask Gateway]]></title>
            <link>https://medium.com/pangeo/pangeo-with-dask-gateway-4b638825f105?source=rss-b66b09f4f9a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b638825f105</guid>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[jupyterhub]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Tom Augspurger]]></dc:creator>
            <pubDate>Tue, 31 Mar 2020 13:01:00 GMT</pubDate>
            <atom:updated>2020-09-15T23:08:33.994Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Tom Augspurger, Joe Hamman, and Scott Henderson</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*kVRJHAkUNLwc6pGOVXYjEQ.png" /><figcaption>Dask Gateway provides a secure, multi-tenant server for managing <a href="https://dask.org/">Dask</a> clusters. It allows users to launch and use Dask clusters in a shared, centrally managed cluster environment, without requiring users to have direct access to the underlying cluster backend (e.g. Kubernetes, Hadoop/YARN, HPC Job queues, etc…). <a href="https://gateway.dask.org/">https://gateway.dask.org/</a></figcaption></figure><p>Over the past few weeks, we have made some exciting changes to Pangeo’s cloud deployments. These changes will make using Pangeo’s clusters easier for users while making the deployments more secure and maintainable for administrators.</p><p>Going all the way back to the <a href="https://medium.com/pangeo/jupyterhub-dask-and-xarray-on-the-cloud-f745d4fe5c16">initial prototype</a>, Pangeo’s cloud deployments have combined a user interface like Jupyterlab with scalable computing. Until recently, Pangeo used Dask Kubernetes to start Dask clusters on a Kubernetes cluster. This worked well for several years, but there were a few drawbacks.</p><ol><li>Customizing the computing environment often required some Kubernetes knowledge</li><li>Dask Kubernetes required broader permissions to interact with the Kubernetes API, including the ability to create and delete pods and services</li></ol><p>To address these problems, we’ve included <a href="https://gateway.dask.org/">Dask Gateway</a>, a secure, multi-tenant server for managing Dask clusters, in <a href="https://github.com/pangeo-data/helm-chart">Pangeo’s Helm Chart</a>. Pangeo’s <a href="https://github.com/pangeo-data/pangeo-binder/">Binders</a> and <a href="https://github.com/pangeo-data/pangeo-cloud-federation">Jupyterhubs</a> now have Dask Gateway enabled, which benefits both users and cluster administrators (who are often just the users wearing a different hat).</p><h3>Benefits for users</h3><p>Previously, users wishing to customize their Dask cluster would often need to interact directly with the <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/">Kubernetes API</a>, which is overwhelmingly large. For example, suppose we have a workload that’s relatively memory intensive: we’re loading a lot of data but doing some very simple transformations. So we’d like to adjust the ratio of CPU-cores to memory to have more memory. The Pangeo admins provided some default values in a configuration file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64c33c1bdafdd966fa2789582d82ec5e/href">https://medium.com/media/64c33c1bdafdd966fa2789582d82ec5e/href</a></iframe><p>Those values are automatically used when creating a dask_kubernetes.KubeCluster. For our “high-memory” workload, we’d need to adjust a few places to get the desired ratio of CPU-cores to memory.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8b7132da1c7016721b5f4cd8defa7bf/href">https://medium.com/media/e8b7132da1c7016721b5f4cd8defa7bf/href</a></iframe><p>That’s a lot of specialized knowledge needed for this relatively simple request of “give me fewer cores and more memory”.</p><p>Compare that with Dask Gateway’s approach to the same problem. Cluster administrators can <a href="https://gateway.dask.org/cluster-options.html">expose options</a> to users. Users can select from these options using a graphical widget or through code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*JRRTdpaQqAQbYvfq" /></figure><p>In our high-memory example, the user would update the values and create their cluster with the new values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c8029314fa5c1ea522b155b85c7de5d/href">https://medium.com/media/0c8029314fa5c1ea522b155b85c7de5d/href</a></iframe><p>This builds on the same technology used throughout the Jupyter ecosystem, so users of libraries like <a href="https://ipywidgets.readthedocs.io/en/latest/">ipywidgets</a> will find the interface familiar.</p><h3>Benefits for administrators</h3><p>Dask-Kubernetes requires <a href="https://github.com/dask/dask-kubernetes/blob/master/doc/source/index.rst#role-based-access-control-rbac">broader permissions</a> on the Kubernetes pods, which may be problematic for some groups. With Dask Gateway, only the service deployed by the administrator needs to interact directly with the Kubernetes cluster. Users are completely isolated.</p><p>Pangeo administrators now have greater control over how the cluster is used. We can set server-side limits on things like the maximum size of a cluster, and the number of concurrent clusters per user. Previously, users could create huge Dask clusters that swamped our Kubernetes cluster, degrading the experience for anyone else using the cluster and costing Pangeo money.</p><h3>Transitioning to Dask Gateway</h3><p>Now that Dask Gateway is available by default on all Pangeo cloud deployments, we’re looking to help users familiar with Dask-Kubernetes transition to Dask Gateway. Luckily, transitioning between the two is pretty simple.</p><p>A typical usage of Dask-Kubernetes cluster:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/460976216519b15fde6c3bf2e3d7400a/href">https://medium.com/media/460976216519b15fde6c3bf2e3d7400a/href</a></iframe><p>Now, to create a Dask Client using Dask Gateway:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/998011362abc4a8e6596e9b5cf4421c0/href">https://medium.com/media/998011362abc4a8e6596e9b5cf4421c0/href</a></iframe><p>Dask Gateway provides some <a href="https://gateway.dask.org/usage.html">great documentation</a> on how to use its API but we’ll unpack a few of the differences anyways:</p><ul><li>gateway = Gateway(): Connects to the Dask Gateway deployment. This gateway will manage Dask clusters and broker access to the Kubernetes API.</li><li>cluster = gateway.new_cluster(): Rather than instantiating a KubeCluster, we ask the Gateway to create a new cluster for us. This returns an object that behaves quite similar to other Dask Cluster objects.</li><li>cluster.scale(): No change here. Scale your cluster up and down using .scale() or .adapt(). Note that Dask Gateway may enforce per-user or per-cluster limits on things like CPU or memory use.</li><li>client = Client(cluster): Again, no change here. Attaching the client to the cluster ensures that the cluster is used for all future computations.</li></ul><p>So that’s it! The primary change is that we’re interacting with the Gateway, rather than directly with the Kubernetes cluster. Everything else, including cluster persistence and scaling, will be the same as before.</p><p><a href="https://binder.pangeo.io/v2/gh/pangeo-gallery/default-binder/master/?urlpath=git-pull?repo=https://github.com/TomAugspurger/pangeo-dask-gateway%26amp%3Burlpath=lab/tree/pangeo-dask-gateway">This binder</a> has a runnable example of using Dask Gateway on a Pangeo binder.</p><p>If you’re using one of Pangeo’s binder deployments (<a href="https://binder.pangeo.io/">on Google</a> or <a href="https://aws-uswest2-binder.pangeo.io/">on AWS</a>), you’ll now need to include Dask Gateway in your environment, rather than dask-kubernetes. See the <a href="https://pangeo-binder.readthedocs.io/en/prod/#configuring-dask">pangeo-binder documentation</a> for more.</p><p><strong>Wrapping up</strong></p><p>We hope this gentle introduction to Dask Gateway and why Pangeo has adopted it proves useful for you. For more details, follow <a href="https://github.com/pangeo-data/pangeo-cloud-federation/issues/496">this GitHub issue</a> and the linked pull requests. While we expect to continue supporting Dask-Kubernetes on Pangeo’s deployments for a while, we will <a href="https://github.com/pangeo-data/helm-chart/issues/79">eventually</a> be turning this integration off.</p><h4>Acknowledgements</h4><p>This work was supported by NSF award <a href="https://www.nsf.gov/awardsearch/showAward?AWD_ID=1740648&amp;HistoricalAwards=false">1740648</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b638825f105" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pangeo/pangeo-with-dask-gateway-4b638825f105">Pangeo with Dask Gateway</a> was originally published in <a href="https://medium.com/pangeo">pangeo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>