<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Guillaume Moutier on Medium]]></title>
        <description><![CDATA[Stories by Guillaume Moutier on Medium]]></description>
        <link>https://medium.com/@guillaume.moutier?source=rss-c821828f30b7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*c4tAo1cQ3CHRb6iNtDSC3A.jpeg</url>
            <title>Stories by Guillaume Moutier on Medium</title>
            <link>https://medium.com/@guillaume.moutier?source=rss-c821828f30b7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:43:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@guillaume.moutier/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlimited scientific libraries and applications in Kubernetes, instantly!]]></title>
            <link>https://medium.com/data-science/unlimited-scientific-libraries-and-applications-in-kubernetes-instantly-b69b192ec5e5?source=rss-c821828f30b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b69b192ec5e5</guid>
            <category><![CDATA[data-science-platform]]></category>
            <dc:creator><![CDATA[Guillaume Moutier]]></dc:creator>
            <pubDate>Mon, 27 Sep 2021 19:15:45 GMT</pubDate>
            <atom:updated>2021-10-07T18:16:20.810Z</atom:updated>
            <content:encoded><![CDATA[<h4>Set up a shared library of packages and applications that you can dynamically load into your Kubernetes Data Science environments.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z3ZRKZYuPjiYGcWFjhR-nQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@michaeldbeckwith?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Michael D Beckwith</a> on <a href="https://unsplash.com/s/photos/scientific-libraries?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>TL;DR:</strong><br>Providing Jupyter-as-a-Service on Kubernetes can be tricky at scale, when you need to provide many different libraries or applications, with different versions, to satisfy every user’s needs.</p><p>Instead of creating overweight container images, or having to manage (and choose from!) hundreds of them, or having people pip/conda reinstall everything from scratch every time they open a notebook, you can build a shared repository of libraries and applications, and let users load them dynamically and instantly.</p><p>Full code and deployment instructions provided for <a href="https://opendatahub.io/">Open Data Hub</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*SXb6YBWGppOSy6MHVxaPuQ.png" /><figcaption>(Screenshot by the author)</figcaption></figure><p><em>Disclaimer:</em> I am working at Red Hat, helping organizations with their Data Science/Data Engineering platforms and solutions. Therefore the implementation described here is using different Red Hat products like <a href="https://www.redhat.com/fr/technologies/cloud-computing/openshift">Red Hat OpenShift</a> (our Kubernetes Enterprise distribution), or <a href="http://Red Hat OpenShift Data Foundation">Red Hat OpenShift Data Foundation</a> (our OpenShift-native storage solution based on Ceph). However, the overall architecture and recipes could be adapted for other platforms.</p><p>And now, let’s go for the lengthy full story!</p><h3>Chapter 1: How the trouble began</h3><p>Kubernetes is a popular choice for Data Science and Data Engineering platforms. Agility, versatility, resource scaling,… make it a platform of choice for those types of workloads. Especially in big organizations with shared environments: packaging your apps, libraries and dependencies into lightweight container images, making them reproducible, consistent and secure seems like the perfect answer to all your problems!</p><p>Thrilled with all what you’ve read about this recently, you package a Jupyter container image in this way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*EF9zWtJ6Nee5CGqR8xp1PQ.png" /><figcaption>(Graphic by the author)</figcaption></figure><p>Because you’re smart and want to anticipate needs (translate: you don’t want to come back to this again and again), you even create some different “flavours” to cover the different use cases you can think of: basic notebook with Scipy only, one with Tensorflow, one with Pytorch,… Perfect, those container images are ready to use.</p><p>So you set up JupyterHub or Kubeflow in your Kubernetes environment, and create a Jupyter-as-a-Service environment for all your users! Mission accomplished, problem solved, you can go back to other (always) urging matters.</p><p>A few weeks pass…</p><p>Then come Alice, Bob, and the rest of the Data Scientists team. They even brought with them some BI people to show their goodwill through the digital transformation the organization is undergoing, and a few Data Engineers (one never knows, there could be a leak to fix in a data pipeline somewhere). But no statisticians, you all know why…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*Cj9B7osELBjJP9rFgbux1w.jpeg" /><figcaption>(Image by the author from imgflip.com)</figcaption></figure><p>Alice started, “Listen, this Jupyter-as-a-Service thing is fantastic! We’ve been playing with it for the past few weeks and we see how this could replace our personal environments: no more things to install and manage on our laptops, which lack processing power and memory anyway. And everything is accessible from anywhere!”.</p><p>“Yes, that is definitely a great improvement.”, continued Bob. “ We are now able to exchange our notebooks easily, as they’re all based on the same container image and use the central data lake. No more of this library or this application missing!”</p><p>Somewhat puzzled, as this had never happened before, you asked, “Ok, then, perfect. But why are you here, not only to thank me I presume?”</p><p>“Oh, almost nothing really, for you it will be easy-peasy!”, exclaimed Charlie. “You know, the different flavours of environments you have created for us are great. It just lacks this library I’m using a lot, you know myFavLib? At version 3.2.4 or higher of course, because up until 3.2.3 there were some broken things. But not from the new 4.x branch, I don’t really like what they’ve done with it.”</p><p>“Yes, and if you could also add LibA, LibB and LibC, that would be great. Oh, by the way, for LibC I will need the last 3 versions, to be able to compare some stuff”, Debbie added.</p><p>In his usual condenscending way, Bob jumped in, “Since we are on this subject… Of course all the older versions of the images and the libraries will always be available, right? Because it’s part of our audit process to be able to redo any calculation at any time, you know, for liability purposes. So of course it has to be always done with the exact same tools. Plus I don’t want to have to modify my code to adapt it to newer versions anyway.”</p><p>Seeing your face slightly decomposing, Alice cut it off, “Well, I guess you got the idea, so we’ll leave you at it then, thanks again!”</p><p>As you are a seasoned professional, you manage to hide the panic rising and quickly switch to solution mode…</p><h3>Chapter 2: Several options, no real solution….</h3><p>After a few days, you contemplate the results of your hard head-scratching sessions…</p><p><strong>Option A</strong>: the bloated image.</p><p>OK, let’s put everything they asked into the same container image! That’s easy: just add some lines to requirements.txt, maybe a few more things to the Dockerfile, docker build and… Done! Now let’s upload the image to the repo. What, ETA 2:34:12 ?! Oh… this image is 38.2GB… Definitely not manageable or workable: “Starting container. Pulling image, please come back tomorrow…”. And anyway, I cannot fit in the multiple versions of the same library as asked.</p><p><strong>Option B</strong>: the never-ending stream of images.</p><p>Well, on top of the existing images, I can always create custom ones to accommodate the various demands. But let’s see,… With only 15 libraries and 2 possible versions for each one, that’s a theoretical 2¹⁵ combinations, 32 768! Even if they only ask for a hundredth of this, that’s still more than 300 container images to choose from. I’m pretty sure the dropdown selector for which image to launch won’t even allow that much. And anyway, even if the users cope with it because they don’t have the choice, the whole thing will be unmanageable in the long term.</p><p><strong>Option C</strong>: let them deal with it.</p><p>After all, it’s not that difficult to do a pip install! With some clever mechanism, I could even have a process looking in the notebook’s folder if there is a requirements.txt, and install everything listed in the background. Of course, that happens every time the notebook is launched, which will cause delays and complaints… And anyway, it won’t solve the problem of having many images to maintain with different sets/versions of base libraries. Unless I want the users to reinstall the whole stack from scratch every time…</p><p>This approach also does not solve the problem of applications that you cannot install using pip or conda. And being in a secured enterprise environment, containers don’t run as root (which is a best practice), so people can’t install anything anyway.</p><p>So what to do…?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*ZDI_mFcV-TC-nB8NaBueQg.png" /><figcaption>Your not-so-good options… (Graphic by the author)</figcaption></figure><h3>Chapter 3: The depths of despair</h3><blockquote>Note from the author: this chapter has been voluntarily filtered out as 1) There is no need for everyone to live this, even by proxy. 2) It has no link whatsoever with IT or Data Science.</blockquote><h3>Chapter 4: This is the way</h3><p>In the past, I had the chance to get to learn what <a href="https://www.computecanada.ca/">Compute Canada</a> and lots of people from the HPC world, at least in Academics, especially the fantastic <a href="https://easybuild.io/">EasyBuild</a> community, were doing to address this very same problem: how do you bring applications and libraries to your users within a shared environment in an effective and reproducible way, making it easy for the data scientists to just do their jobs and not fumble with installations, packages or compilation…?</p><p>Their answer was to use Environment Modules to dynamically “load” pre-packaged libraries or applications. Simplifying it to the extreme, let’s say you have access to some shared folders mounted somewhere in your filesystem, the module engine will simply modify your $PATH, $PYTHONPATH or other environment variables like that to make those folders “visible” into your environment. No more Python complaining that it cannot find the library when you do an “import pytorch” for example because now it knows where to find it!</p><p>Of course all of this is way more refined because it can keep track of dependencies, what’s loaded or not, what to unload and when in case of cross-dependencies…</p><p>But basically, the solution to our problem consists of two things:</p><ol><li>Provide access to a library of “modules” that can be Python libraries, Linux libraries, or full-blown applications. This library can be mounted at spawn time inside your environment.</li></ol><p>2. Have a way to easily load those modules inside the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*7CCdUEEXcvnRk4A1XadnpA.png" /><figcaption>Implementation concept in Kubernetes (Graphic by the author)</figcaption></figure><p>So this is the solution I have implemented as a proof-of-concept within Open Data Hub (<a href="http://opendatahub.io/">Open Data Hub</a>, ODH, is an open source project based on Kubeflow that provides open source AI tools for running large and distributed AI workloads on OpenShift Container Platform):</p><ul><li>The shared library is installed on an RWX volume provided by OpenShift Data Foundation (with CephFS behind the scene).</li><li>ODH is providing Jypyter-as-a-Service environments, in this case with a slightly customized image that is able to load modules, and a JupyterLab extension to do that loadindg/unloading part easily.</li><li>When Jupyter is spawned, the shared library is mounted read-only inside the pod.</li><li>You then only have to load what you need, when you need it!</li></ul><p>And this is what it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*fI_n_0w5zAxGKAIdPJ9Bnw.gif" /><figcaption>Quick demo of the environment (Screenshot by the author)</figcaption></figure><p>This is what happens in this quick overview:</p><ul><li>I launch a basic notebook example that uses <strong>Torchvision</strong> to infer the breed of a dog from a picture (and not just any dog, mine!).</li><li>As I try to run the notebook, a ModuleNotFound Error happens because I don’t have the <strong>torchvision</strong> library available in my environment.</li><li>So on the left panel, in my <strong>Softwares</strong> extension menu, I look for this package by entering a few letters, then I just click on Load.</li><li>As the notebook was already launched, I have to quickly restart my kernel to take the change into account, and Bam! I can run the whole notebook.</li></ul><p>At this point you may say: “That’s interesting and all, but you could have just done a pip install”.</p><p>Granted, especially with this basic example, and this is what we’re used to do, especially on local environments (opposed to shared ones). But this alternative method brings interesting capabilities, all of them without creating specific container images:</p><ul><li>Provide applications that people would not be able to install otherwise as they may not be available as a Conda or Pip package. And in this kind of shared environment people should not be able to yum/apt-get install anything as they’re not supposed to have root access. I hope it’s the case in your environment…</li><li>It gets really easy to tests different versions of the packages/applications. Just load/unload the relevant module(s), no need to do long or complicated installs/uninstalls or juggle with dependencies.</li><li>You can provide many different versions of the packages or applications, and users can choose what they want according to their needs. Even years later, those modules can still be available, no need to remove them because there is an upgrade of the environment. On the opposite, you can provide bleeding-edge alpha versions of modules without putting the whole environment in jeopardy.</li><li>Users’ environments stay clean and don’t get cluttered with installed libraries. Plus they don’t have to do all those installations every time they want to run a specific notebook. Module loading/unloading is way faster as in fact all the content is already mounted into the pod.</li><li>For the system administrators life gets way easier. There is only one container image to manage. Which is why this project has been dubbed Open Data Hub-Highlander, “There can be only one!”.</li></ul><h3>But wait, there is more!</h3><p>In the first basic example, I loaded a Python package. But as I said before it can also be a full application, even one with its own Web UI like RStudio-Server. For this to happen, I’ll just use jupyter-server-proxy with some special configuration to make it appear in the launcher when the module is loaded.</p><p>Load the module:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*mi1fMbJFh5mm67OjokIcQA.png" /><figcaption>Load the module (Screenshot by the author)</figcaption></figure><p>A new tile appears for RStudio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*TgjZ0m0KrorMAhzgogkHdA.png" /><figcaption>New RStudio tile (Screenshot by the author)</figcaption></figure><p>When you click on the tile a new tab opens in your browser and RStudio launched in a few seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lunDWQXH9QqpVFSQM_kZw.png" /><figcaption>RStudio (Screenshot by the author)</figcaption></figure><p>And this is the same container environment as before, meaning you have access to the same files, same tools,… Which can be useful when you want to mix R and Python scripts and vice-versa!</p><h3>Some more details on how to use this</h3><p>In the following repo you will find full deployment instructions and code, including a pre-compiled library of modules that is easy to deploy in your Open Data Hub environment: <a href="https://github.com/guimou/odh-highlander">https://github.com/guimou/odh-highlander</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*yIYnkr8Xcsn2hT9h98lP-w.png" /></figure><p>And this is what you will be able to do once you have deployed everything…</p><p>In your JupyterLab instance you have access to a new extension, “Softwares”. The list of available modules is splitted in two sections. The “featured” modules, and the full list (this is only the modules I chose to provide in this demo environment):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*-fkaLTpQC2cG9sTO.png" /><figcaption>Module list (Screenshot by the author)</figcaption></figure><p>You can use the filter box to search for a specific module (just enter a few letters). Filtering happens simultaneously on both lists:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/0*Dqt1x71LF77Koll7.png" /><figcaption>Filtered list (Screenshot by the author)</figcaption></figure><p>If you click on a module name, a pop-up will give you more information: description, dependencies,…​</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZSpeDlsxF1NNpSQW.png" /><figcaption>Module info (Screenshot by the author)</figcaption></figure><p>To load a module, hover on it and click on the “Load” button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*QYj-brX43fFJhJKe.png" /><figcaption>Module loading (Screenshot by the author)</figcaption></figure><p>The module and all its dependencies is automatically loaded (torchvision in this example):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*8oqpPgvA4iHM9VHo.png" /><figcaption>Module loaded (Screenshot by the author)</figcaption></figure><p>To unload a module, hover it the “loaded” list, and click “Unload”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*3dUKqe3BcS90767B.png" /><figcaption>Unload Module (Screenshot by the author)</figcaption></figure><p>The module and its dependencies will be automatically unloaded.</p><p><em>Note</em>: <a href="https://lmod.readthedocs.io/en/latest/">Lmod</a>, the module manager engine, keeps track of the loaded dependencies for each module. If two different modules share dependencies, a module unloading won’t affect the other one, its dependencies will still be there. They are only unloaded when no module needs them anymore!</p><h3>Advanced functions</h3><h4>Collections</h4><p>If you want to create a specific environment with a set of different modules that you want to use, no need to recreate this from scracth every time! You can simply load those modules, then create a “Collection”. Next time, just restore this collection in two clicks.</p><p>To create a collection, load the modules you want, click on the icon and give a name to the collection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/0*2v6eQNEM_0Pihf5y.png" /><figcaption>Create Collection (Screenshot by the author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*0yFRDEsJ7MviACA7.png" /><figcaption>Name and Save Collection (Screenshot by the author)</figcaption></figure><p>When you want to bring back this environment, just click on the Restore icon, and select and load your collection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/0*ag33VfKCxUkT3RmQ.png" /><figcaption>Restore Collection (Screenshot by the author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/0*OmTs20o9YYp3vWl2.png" /><figcaption>Restore Collection (Screenshot by the author)</figcaption></figure><h4>Imports</h4><p>You can also directly work from your notebooks and your scripts to load the modules that you need. To know which modules you have to use, you can directly export the relevant Python code!</p><p>Click on the “Generate Python code” icon:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*UHmmodJDp8MUUmxe.png" /><figcaption>Generate Code (Screenshot by the author)</figcaption></figure><p>You can then copy paste the full code in you first notebook cell or in your script:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YFhrdBBqX8thnUKF.png" /><figcaption>Python Code (Screenshot by the author)</figcaption></figure><p><em>Note</em>: Of course, for this to work in your notebook or your script, the container image or environment you are using must be “lmod-enabled”, and the library with the relevant modules must be accessible/mounted into this environment.</p><h3>What’s next?</h3><p>As already said, you will find in this repo all the resources and instructions to deploy this solution. And in a follow-up post, I’ll get in more details on the technical implementation, especially:</p><ul><li>How to create an environment (well, tbh a glorified container image…) to create you own modules.</li><li>Instructions and examples on how to use <a href="https://easybuild.io/">EasyBuild</a> to create modules.</li><li>How to create a <a href="https://lmod.readthedocs.io/en/latest/">Lmod</a>-enhanced JupyerLab container image to use the modules library.</li><li>How to create hardware-specific modules (like cuda-enabled, that need a GPU), and make them available only on the relevant infrastructure.</li></ul><h3>Many Thanks</h3><p>There are a number of people and organizations I have to thank, as I did not invent anything here. I simply mashed up some projects, ported some solutions from one world to another, and voilà!</p><ul><li>The <a href="https://opendatahub.io/">Open Data Hub</a> team, for creating this fantastic Data Science environment that is so easy to deploy on OpenShift.</li><li>The <a href="https://easybuild.io/">EasyBuild</a> Community of course for the great tool they created, but also for their warm welcome (thanks Kenneth!), and the willingness to answer my dum newbie questions (thanks Maxime!).</li><li><a href="https://www.computecanada.ca/">Compute Canada</a>, who made me discover this kind of solution. Their setup is sick, especially the way they are distributing modules across the whole organization, kudos!</li><li><a href="https://github.com/cmd-ntrf">CMD-NTRF</a>, Félix-Antoine Fortin, for its fantastic Jupyter-Lmod extension, that brings the solution to another level with a slick UI in Jupyter.</li></ul><h3>References and links</h3><ul><li>Repo and instructions: <a href="https://github.com/guimou/odh-highlander">https://github.com/guimou/odh-highlander</a></li><li>Lmod-enabled JupyterLab : <a href="https://github.com/guimou/s2i-lmod-notebook">https://github.com/guimou/s2i-lmod-notebook</a></li><li>Various EasyConfigs (if you want to see what modules are already available): <a href="https://github.com/easybuilders/easybuild-easyconfigs">https://github.com/easybuilders/easybuild-easyconfigs</a></li><li>Various custom EasyConfigs: <a href="https://github.com/guimou/odh-easyconfigs">https://github.com/guimou/odh-easyconfigs</a></li><li>Open Data Hub documentation: <a href="http://opendatahub.io/docs.html">http://opendatahub.io/docs.html</a></li><li>EasyBuild documentation: <a href="https://docs.easybuild.io/en/latest/">https://docs.easybuild.io/en/latest/</a></li><li>EasyBuild at Compute Canada presentation: <a href="https://www.youtube.com/watch?v=eQ1_iWSnJaI">https://www.youtube.com/watch?v=eQ1_iWSnJaI</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b69b192ec5e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/unlimited-scientific-libraries-and-applications-in-kubernetes-instantly-b69b192ec5e5">Unlimited scientific libraries and applications in Kubernetes, instantly!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated data pipeline using Ceph notifications and KNative Serving]]></title>
            <link>https://medium.com/analytics-vidhya/automated-data-pipeline-using-ceph-notifications-and-kserving-5e1e9b996661?source=rss-c821828f30b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e1e9b996661</guid>
            <category><![CDATA[ceph]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[knative]]></category>
            <dc:creator><![CDATA[Guillaume Moutier]]></dc:creator>
            <pubDate>Thu, 09 Jan 2020 04:13:57 GMT</pubDate>
            <atom:updated>2020-02-21T14:33:03.177Z</atom:updated>
            <content:encoded><![CDATA[<h4>A complete guide to implement the solution and create a demo!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*Js6C0Y4kP36NG0FL1EY4RQ.png" /></figure><p>I recently shared a video where I demonstrated how you can create a fully-automated data pipeline that leverages the new notification features from Ceph, along with Kafka and KNative Eventing and Serving. Maybe it’s better to check it out quickly if you haven’t done it yet, as it will help you understand the rest of this post!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fzhax8jUDMLU%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dzhax8jUDMLU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fzhax8jUDMLU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f0bba9b20c2ad23a4c148f994ceb549/href">https://medium.com/media/9f0bba9b20c2ad23a4c148f994ceb549/href</a></iframe><p>If you don’t have time, here is a summary of the scenario we will implement:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfbLCUhYDQoURO7JsE7bPw.png" /></figure><p>Now that we’re all on the same page, let’s see how you can reproduce this demo from the comfort of your (home|cubicle|airport lounge|batcave)…</p><p><strong>Note</strong>: All the code used here is available at <a href="https://github.com/guimou/dataprep-code/tree/master/examples/xray">https://github.com/guimou/dataprep-code</a></p><h3>Preparing the data</h3><p>Those steps are not completely necessary, as the code I’m sharing already includes a trained model and some demo images that you can use. But if you want to know how the whole thing works or you want to reproduce everything from scratch, just give it a try!</p><h4>The dataset</h4><p>This demo is based on the following dataset, available from Kaggle: <a href="https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia">https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia</a>. It’s a set of chest X-Ray images from patients presenting signs of pneumonia (~1300 images), and from healthy ones (~3900 images). They are of course already classified, and split in different batches for training, validating and testing. So the first step is of course to download it and extract it somewhere (~1.2GB). This gives the following structure (demo_base comes from later operations):</p><pre>.<br>└── data<br>    └── chest_xray<br>        ├── demo_base<br>        │   ├── NORMAL<br>        │   └── PNEUMONIA<br>        ├── test<br>        │   ├── NORMAL<br>        │   └── PNEUMONIA<br>        ├── train<br>        │   ├── NORMAL<br>        │   └── PNEUMONIA<br>        └── val<br>            ├── NORMAL<br>            └── PNEUMONIA</pre><h4>Training the model</h4><p>The notebook I used to train the model, “x-ray-predict-pneumonia-keras-training.ipynb”, is available under examples/xray/utils in the repo. You can use it from the root of the project if you have the above directory structure.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df375fb84659cc87efe64ede806ea8d4/href">https://medium.com/media/df375fb84659cc87efe64ede806ea8d4/href</a></iframe><p>It’s just a matter of running the notebook, being patient, ~ 20mn for me with the help of a GTX1070Ti (yeah, I’m not only doing demos and posts, though I should definitely get a RTX… 😁). If everything goes well you should obtain the “pneumonia_model.h5” file that is used for predictions.</p><h4>Creating demo images</h4><p>Of course the images that we have from the original dataset are completely anonymized. But for the anonymization demo, we want to upload images with personal informations… Well, let’s just create them! That’s done with the help of the “make_demo_images.ipynb” notebook, at the same location than the previous one. No rocket science here: we just take images from the original dataset (here in the val/PNEUMONIA folder) and we generate some random personal information that we paste on top of the images.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f19cec65dae50148eeca73591e0d58b1/href">https://medium.com/media/f19cec65dae50148eeca73591e0d58b1/href</a></iframe><p>That’s it for the data part, we now have a model, and some images that we can use in the demo!</p><h3>Some appetizers before the main course (sounds better than prerequisites)</h3><h4>Ceph</h4><p>You will of course need a Ceph cluster for this demo, but for a complete walk-through of installation that’s another story… Quick hint though: as to perform the rest of the demo you will also need OpenShift or another type of Kubernetes cluster (not tested though!), you can use <a href="https://rook.io/docs/rook/v1.2/ceph-quickstart.html">Rook-Ceph</a> to deploy your Ceph instance (that’s what I’m using here). As of now, the Kafka notifications I am using are only available on the master branch of Ceph (but HTTP and AMQ notifications are already in the latest release). So if you’re using rook-ceph, make sure your cluster yaml file reads at some point:</p><pre>cephVersion:<br> image: ceph/daemon-base:latest-master-devel<br> allowUnsupported: true</pre><p><em>(I will update the post when the feature is available in a standard release).</em></p><h4>Kafka</h4><p>I won’t give much details here, as deploying Kafka and creating topics is also another story in itself. So I will assume that you have a Kafka cluster ready, all the necessary endpoint informations, and that you have created a topic, called “storage” in our example. Another quick hint though: you can use the <a href="https://strimzi.io/">Strimzi</a> operator to deploy Kafka quickly and easily on OpenShift.</p><h4>KNative / OpenShift Serverless</h4><p>Finally, KNative must be installed on your cluster. Again, it’s not the goal of this article to detail this operation, there is already lots of documentation covering this.</p><p>Beware though! For this demo to work, you have to install the KafkaSource resource. It’s a contributed “add-on” to KNative eventing, that you can install following the instructions here: <a href="https://github.com/knative/eventing-contrib/tree/v0.12.4/kafka/source">https://github.com/knative/eventing-contrib/tree/v0.12.4/kafka/source</a></p><h3>Creating the pipeline</h3><p>If you remember from the video (or not if you’ve not watched it), this is the pipeline that we will be building:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilAc2PQdEHUDo1cGUHr1IQ.png" /></figure><p>You can start by creating the 3 buckets we will use for this demo: xray-data-in, xray-data-in-processed and xray-data-in-anonymized.</p><h4>Setting up Bucket notifications to Kafka</h4><p>Now to bucket notifications! To help you with these steps, you will find a Postman collection of all the requests, “Ceph notifications pipeline.postman_collection.json” in the utils folder. Whether you use it or not, <strong>don’t forget to set your credentials</strong> (user Access Key and Secret Key) to be able to interact with the Gateway. In Postman you have to edit the Collection parameters (authorization tab), or you can set them request-by-request.</p><p>There are two different steps involved:</p><ul><li>Create a Kafka topic: that’s a simple POST request to the gateway, passing the required parameters:</li></ul><pre>POST <a href="http://your.ceph.rgw.url/?Action=CreateTopic&amp;Name=storage&amp;push-endpoint=kafka://my-cluster-kafka-bootstrap.kafka:9092&amp;kafka-ack-level=broker">http://your.ceph.rgw.url/?Action=CreateTopic&amp;Name=storage&amp;push-endpoint=kafka://my-cluster-kafka-bootstrap.kafka:9092&amp;kafka-ack-level=broker</a></pre><p>It will return the topic information in the form of <strong>arn:aws:sns:my-store::storage</strong></p><ul><li>Create the notification: that’s a PUT request to the gateway on the bucket where you want to have notifications enabled. In our example it was xray-data-in:</li></ul><pre>PUT <a href="http://your.ceph.rgw.url/data-bckt?notification">http://your.ceph.rgw.url/</a>xray-data-in<a href="http://your.ceph.rgw.url/data-bckt?notification">?notification</a></pre><p>with the parameters passed as XML in the body of the request (including the topic you just created):</p><pre>&lt;NotificationConfiguration xmlns=”<a href="http://s3.amazonaws.com/doc/2006-03-01/">http://s3.amazonaws.com/doc/2006-03-01/</a>&quot;&gt;<br> &lt;TopicConfiguration&gt;<br> &lt;Id&gt;storage&lt;/Id&gt;<br> &lt;Topic&gt;arn:aws:sns:my-store::storage&lt;/Topic&gt;<br> &lt;/TopicConfiguration&gt;<br>&lt;/NotificationConfiguration&gt;</pre><p>As simple as that!</p><h4>Deploying the KNative Serving container</h4><p>It’s the container that will do the image analysis, and eventually its anonymization. The source code is available in the examples/xray/xray-event-container folder of the repo. Here is the Python code itself:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aea71800751356f5aef4c02a555361e1/href">https://medium.com/media/aea71800751356f5aef4c02a555361e1/href</a></iframe><p>It’s pretty much self-explanatory and commented but if you have any question please leave a comment. You can build it yourself, or use the image at quay.io/guimou/xray:master</p><p>After creating a project/namespace “xray”, we will create a secret with the credentials needed to access the different buckets that the container will interact with (don’t forget to create them! You should have xray-data-in, xray-data-processed and xray-data-anonymized for this demo). Here is the “secret.yaml” file (you will have to enter actual key/secret values):</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br> name: s3-secret<br> namespace: xray<br>stringData:<br> AWS_ACCESS_KEY_ID: replace_me <br> AWS_SECRET_ACCESS_KEY: replace_me</pre><p>That we deploy with:</p><pre>oc create -f secret.yaml</pre><p>Then we define our service with “service-xray.yaml” (you will have to enter the right service_point URL):</p><pre>apiVersion: serving.knative.dev/v1alpha1 <br>kind: Service<br>metadata:<br> name: xray<br> namespace: xray<br>spec:<br> template:<br>   spec:<br>     containers:<br>     — image: quay.io/guimou/xray:master<br>       env:<br>       — name: AWS_ACCESS_KEY_ID<br>         valueFrom:<br>           secretKeyRef:<br>             name: s3-secret<br>             key: AWS_ACCESS_KEY_ID <br>       — name: AWS_SECRET_ACCESS_KEY<br>         valueFrom:<br>           secretKeyRef:<br>             name: s3-secret<br>             key: AWS_SECRET_ACCESS_KEY <br>       — name: service_point<br>         value: http://your.gateway.url</pre><p>That we deploy with:</p><pre>oc create -f service-xray.yaml</pre><h4>Deploying the KNative Eventing source</h4><p>That’s the component that will consume our Kafka topic and pass the message to the Service we just created.</p><p>It is defined by “kafkasource.yaml”:</p><pre>apiVersion: sources.eventing.knative.dev/v1alpha1<br>kind: KafkaSource<br>metadata:<br> name: kafka-source<br> namespace: xray<br>spec:<br> consumerGroup: xray<br> bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092<br> topics: storage<br> sink:<br>   apiVersion: serving.knative.dev/v1alpha1<br>   kind: Service<br>   name: xray</pre><p>And again deployed with:</p><pre>oc create -f kafkasource.yaml</pre><p>Everything is now ready and should work!</p><h3>Reviewing the whole pipeline</h3><p>To better understand what’s going on, or if you need to do some debugging, here is step-by-step what’s happening in this pipeline.</p><ul><li>When an image is pushed to the xray-data-in bucket, Ceph will send a notification to Kafka.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*bAGvOleS-870NB1m-s4_Eg.png" /><figcaption>Original image</figcaption></figure><ul><li>This notification will be a message like this:</li></ul><pre>{“Records”: [{“eventVersion”: “2.2”, “eventSource”: “ceph:s3”, “awsRegion”: “”, “eventTime”: “2019–12–12T02:42:43.692977Z”, “eventName”: “s3:ObjectCreated:Put”, “userIdentity”: {“principalId”: “ceph-user-bp3UfKOG”}, “requestParameters”: {“sourceIPAddress”: “”}, “responseElements”: {“x-amz-request-id”: “645c86a7–8062–43b1–972b-f6c6ccd4bc6f.134312.15301”, “x-amz-id-2”: “20ca8-my-store-my-store”}, “s3”: {“s3SchemaVersion”: “1.0”, “configurationId”: “storage”, “bucket”: {“name”: “xray-data-in”, “ownerIdentity”: {“principalId”: “ceph-user-bp3UfKOG”}, “arn”: “arn:aws:s3:::xray-data-in”, “id”: “645c86a7–8062–43b1–972b-f6c6ccd4bc6f.29026.2”}, “object”: {“key”: “data/chest_xray/demo/PNEUMONIA/demo_Anna Mcgehee_3677_1941–11–26_2015–02–10.jpeg”, “size”: 253130, “etag”: “a7d65d30ba7a639a017aa7d2c6fdc28a”, “versionId”: “”, “sequencer”: “23A9F15D4FBC632A”, “metadata”: [{“key”: “x-amz-content-sha256”, “val”: “95fe6dd24fc010ee526e40457ff34ef057d257c71b22ac9fd312094d4d7b2a37”}, {“key”: “x-amz-date”, “val”: “20191212T024243Z”}]}}, “eventId”: “1576118563.711179.a7d65d30ba7a639a017aa7d2c6fdc28a”}]}</pre><ul><li>Kafka will handle this message through the “storage” topic because that’s where we published it.</li><li>The KNative Eventing component we defined is a consumer of the “storage” topic. It will fetch the message, and pass to its “sink”, our KNative Serving Service.</li><li>The Service will launch an image-processing container (if not already running), or will eventually scale it depending on the workload, and pass it the content of the message.</li><li>The image-processing container will parse the message, retrieve the image, do its magic (assessing pneumonia risk and printing it on the image), then save it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*Spu9EzJzkPo-D8X2CLGOdg.png" /><figcaption>risk-assessed image, PII still there</figcaption></figure><ul><li>Eventually (depending on the risk) our KNative service will anonymize the image and save it to another bucket, as per our scenario.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*N5kTb19ZfC6IyFFB6ffgVQ.png" /><figcaption>Anonymized image</figcaption></figure><ul><li>When the process is finished the image-processing container is simply terminated.</li></ul><h3>Final notes</h3><h4>A problem worth mentioning</h4><p>The tricky part during the creation of this demo was figuring out how to listen to KNative Eventing messages. It’s well documented theoretically speaking, but I did not really find any simple implementation in Python (at least simple enough for me to understand, as I’m not a real developer 😁). Until I found <a href="https://blog.elegantmonkeys.com/python-serverless-event-driven-with-knative-728e96e9b35">this post</a> and this <a href="https://github.com/elegantmonkeys/python-kncloudevents">project</a>. It’s a really nice and simple package that will get you quickly going in case you want to play with KNative and Python.</p><p>But if you’ve read all the code, you will ask “So why did you not use it?”. Well, and that’s why I wanted to mention it here, this package uses the ThreadedHTTPServer to listen to events. But as Keras seems to have some problems when predictions are launched from within a thread (and again I must confess I do not totally understand the issue), the prediction function would just fail. That’s why I reimplemented the whole CloudeventsServer class from the package with ForkedHTTPServer, instead of just calling it.</p><h4>Perspectives</h4><p>This simple demo shows that it’s now easier than ever to build automated data pipelines without requiring some orchestrator or other tooling. We can just use native functions from the storage tier and the container/application tier, and bind them together in almost any way we can think of. Stay tuned for other implementations! And if you have use cases that you think would take advantage of this kind of interactions, please feel free to reach out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e1e9b996661" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/automated-data-pipeline-using-ceph-notifications-and-kserving-5e1e9b996661">Automated data pipeline using Ceph notifications and KNative Serving</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a scalable, shared Data Science platform at Université Laval]]></title>
            <link>https://medium.com/@guillaume.moutier/deploying-a-scalable-shared-data-science-platform-at-universit%C3%A9-laval-b26e7e02cba4?source=rss-c821828f30b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b26e7e02cba4</guid>
            <category><![CDATA[ceph]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[vault]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Guillaume Moutier]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 22:07:49 GMT</pubDate>
            <atom:updated>2019-09-03T22:07:49.673Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*DjHMI33_FIClBexcW1QnmQ.png" /></figure><p>TLDR: just to make sure you won’t leave after the first 42 paragraphs, we’ll talk about the integration of JupyterHub, OpenShift, Ceph, Vault, Keycloak,…</p><h3>What’s this?</h3><p>Let’s start with some context!</p><p><strong>Who we are</strong>: Université Laval is located in Québec City, Canada. The very first French-language university in North America (originating back to 1663), ULaval has 43.000 students, 9.300 employees, and ranks 8th out of the top research universities in Canada.</p><p><strong>The project</strong>: Valeria aims at solving different problems that researchers and students are faced with for their data analysis:</p><ul><li><em>Tooling</em>: a lot of tools are available out there. Many. Too many… If you’re already a data engineer or data scientist, you know everything about them (or you try…). But let’s say you’re a biologist, a geologist, an historian, with a small or non-existent background in IT, but a great idea of something you could do with data. How do you start? How do you find the right program, library, or application to use? How do you install it, with all the right prerequisites, the right versions, the compatibility lists,…?</li><li><em>Ressources</em>: now that you know what to do, the next question is “Where?”. Yes, your laptop’s fine, but if you want to scale a little bit, or try to manipulate some heavy data, acquire them 24/7, or simply store them, you may face some issues. For some science fields a USB key is OK, or even a floppy disk (yes, you’d be amazed to see that some labs look likes museums). But if you work in genomics for example, where datasets come by tens or hundreds of terabytes, that’s another game.</li><li><em>Collaboration</em>: more and more projects would like to cross-reference data, combine them from different sources, even from different fields. Or simply expand the datasets with more data shared by colleagues. Researchers need a way to make this process of discovering and sharing data easier.</li><li><em>Security</em>: last subject, but definitely not a small one, how do we ensure that the data stays “secure”, which means maintaining its confidentiality, availability and integrity? That relies of course on a strong architecture and thorough procedures, which involve control, maintenance, audits,… Once again, definitely not your specialty when you are a geographer.</li></ul><p>To sum it up, as a Laval’s Vice-Rector said:</p><blockquote>“We want our researcher to search, not to tinker with systems and applications.”</blockquote><p>So Valeria was built to offer local services to collect, store, process, analyze and share data in a simple and secure way… Without bothering with technology! In short, Valeria is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*vcvgV_S8GIe3_Tubux8KlQ.png" /></figure><p>Now for the tougher part…</p><h3>Architecture</h3><p><strong>Note</strong>: I know there are a lot of solutions and providers on the market, and you can always say “Why did you not use this or this? My brother-in-law knows a better solution!”, but we had to follow some guidelines:<br>- Everything had to be on-prem: for regulations/security (some data are really sensitive, especially in medical science), efficiency (with Petabytes of data you can’t have them move around too much), and cost-effectiveness (ULaval is lucky enough to have 4 datacenters on the campus, with 2 of them being Tier-3, so it’s still much cheaper than going to the Cloud for most workloads).<br>- Storage had to be separated from Compute. As Valeria is to be used by all researchers and students, it’s clearly impossible to forecast the use of resources. Therefore those two have to scale independently.<br>- Whenever possible, solutions have to be Open Source. It’s easier to integrate, expand upon, patch,… And in this area where things are moving really fast² (that’s not a note link, that’s an exponent!), you want to keep all your options opened.<br>- And the most important thing, everything has to be based on open and recognized standards to make sure it can be easily integrated.</p><p>Now…</p><p>There are many elements in Valeria, some are still under development, some will come later. So I will focus here on the core of the platform:</p><ul><li>The Datalake: we wanted our researchers to be able to easily share data, with security and high availability. We also wanted to make it easy to interact with storage, even at long distances (remote labs, or sensors in the field), from any type of device, platform or language. A central object store, S3-compatible, seemed the right way to go and we chose Ceph (we had been looking at it and experimenting with it for a while).<br>2.8 PB have been deployed for this project as a “starter”, spread over the 4 datacenters (don’t worry, they share the same leaf and spine network @100Gbps between them, no latency), with an 8+4 erasure coding profile.</li><li>The Compute part: it was clear from the start that Jupyter notebooks were ideal to provide easy to use environments where researchers would be able to use different languages, libraries,… JupyterHub was also there to handle the multi-user aspect. But how to make it the most efficient and resource-wise? Kubernetes to the rescue! And more specifically OpenShift, on which we were already working for more “standard” development.<br>The project deployed 25 nodes with 256GB of memory with 2x14-cores CPUs, with some of them being GPU-equipped (V100).</li><li>ID Management/Authentication: with the different tools that we would be using, we had to be able to manage OAuth/OIDC, SAML... Keycloak seemed to be a good choice, that we validated after some testings.</li><li>Secrets Management: S3 is great, but it uses secrets for connections to buckets. And of course we don’t want people to put those secrets directly in their code, or any permanent file that could be accessed or compromised. To solve this problem we use Vault from HashiCorp, and access those secrets at runtime only.</li></ul><p>So here is how it looks like from 3000 feet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HcoDrLM0ITDwIXHkJ0k35Q.png" /></figure><ul><li>Users connect to the platform through a web portal (developed using <a href="https://www.patternfly.org/v4/">PatternFly</a>).</li><li>From there they have access to different applications (and help, documentation, onboarding,…): <a href="https://ckan.org/">CKAN</a> as a data catalog, <a href="https://about.gitlab.com/">GitLab</a> to store and version code or programs, and <a href="https://jupyter.org/hub">JupyterHub</a>.</li><li>Every access is authenticated and secured using <a href="https://www.keycloak.org/">Keycloak</a>, and everything runs on <a href="https://www.openshift.com/">OpenShift</a>.</li><li>From JupyterHub, users can launch different “flavors” of notebooks depending on the kernel and the libraries installed (basic Python, SciPy, Tensorflow, Dask, R). And depending on their access level (researcher, student,…) they have different quotas of resources available (CPU, RAM, GPUs).</li><li>With the help of <a href="https://www.vaultproject.io/">Vault</a> (more on that later), the notebooks are automatically spawned with the user’s uid, and injected with the user’s object storage credentials. That allows the notebook to be automatically connected to: 1. the user /home and /scratch folders that are exported through NFS from a Lustre filesystem; 2. all the buckets that the user owns in the <a href="https://ceph.io/">Ceph</a> datalake.</li><li>Standard use of the platform is that all user’s programs are saved in its /home folder, temporary data can be saved in /scratch (automatically erased after 30 days), while the data itself is stored in the Datalake.</li><li>Users can also send and receive data to/from Compute Canada, the array of supercomputers accessible to all Canadian researchers. This is done via Globus for fast parallel transfers.</li><li>Last part (not implemented yet, WIP): to give more flexibility with the kernels and libraries used in the notebooks, ULaval is working to integrate Valeria with the <a href="https://cernvm.cern.ch/portal/filesystem">CVMFS</a> “repo” used at Compute Canada. That will allow for lightweight and easily customizable kernels. If you are interested in this part reach out to me and I’ll get you in touch with the people working on this.</li></ul><h3>Technical details, implementation</h3><p>I’ll start with some good news, everything is available at ulaval GitHub repos: <a href="https://github.com/ulaval/valeria-jupyterhub">https://github.com/ulaval/valeria-jupyterhub</a> and <a href="https://github.com/ulaval/valeria-jupyter-notebooks-s3">https://github.com/ulaval/valeria-jupyter-notebooks-s3</a>.</p><p>Now for the bad news… This platform is tailor made for Valeria. It answers ULaval needs, and takes advantage of what was already implemented and available. So it’s definitely not designed as a turn-key solution. If you’re looking for that kind of things, make sure to have a look at projects such as <a href="https://www.kubeflow.org/docs/about/kubeflow/">Kubeflow</a> or <a href="https://opendatahub.io/">Open Data Hub</a> (I may be biased on this one, I know, more on that later…).</p><p>However, maybe there are some recipes or solutions here that you may find of interest, so I’ll focus on those for the end of this article.</p><h4>JupyterHub on OpenShift</h4><p>First and foremost, our work was heavily based on the fantastic work done by Graham Dumpleton with <a href="https://github.com/jupyter-on-openshift">Jupyter-on-OpenShift</a>, especially <a href="https://github.com/jupyter-on-openshift/jupyterhub-quickstart">JupyterHub-Quickstart</a>. Please read what’s on this project for more details on how this part works. I won’t detail it here as Graham has already documented it quite thoroughly!</p><h4>Vault, JupyterHub and Keycloak integration</h4><p>This is the real tricky (or interesting) part, so bear with me...<br>When a user is created in Valeria, he gets a uid in the environment where he has his /home and /scratch directories (on the Lustre filesystem). When his account is created in Ceph, he also gets an access_key and a secret_key. We store those 3 elements in a Vault instance, at this path (valeria being a kv secret store):<br> valeria/data/users/{{identity.entity.id}}/</p><p>Noticed the {{identity.entity.id}} path? That’s where a dynamic policy attached to it will allow us to make sure that only a user authenticated with Keycloak with this specific id will be able to retrieve a secret stored in this branch. And not a know-it-all backend that would have access to all the keys, with all the security concerns it brings…</p><p>So after that, when a user launches a notebook, what we do is that we reuse the JWT that the user obtained from Keycloak to access JupyterHub, to authenticate against Vault this time and retrieve the uid and the S3 credentials. That involves a custom OAuthenticator class that may be too long to paste here, but here is the <a href="https://github.com/ulaval/valeria-jupyterhub/blob/3ff8aab5088ca599d12f7b10d414a7fc82d79b85/jupyterhub-valeria/.jupyter/jupyterhub_config.py#L76">direct link</a>. The uid is used to spawn the notebook (see below), and S3 credentials are setup as env vars in the notebook for later user (see a little bit more below…).</p><p>Of course many other things could be stored securely this way in Vault, and used only when needed by JupyterHub or the launched notebook.</p><h4>Connection to S3 and NFS (simultaneously)</h4><p>Here I have first to acknowledge the work from <a href="https://github.com/quantopian/pgcontents">PGContents</a> for the HybridContentsManager and <a href="https://github.com/danielfrg/s3contents">S3Contents</a> for the S3ContentsManager. They are the two parts that allow this magic to happen.</p><p><strong>Connection to NFS</strong> is quite easy. First you have to create a service account and custom scc that will allow the notebook pod to mount the NFS volume (detailed in the README file on the repo, but basically it’s the Restricted policy+NFS volume). And then in jupyterhub_config:</p><pre># Setup persistent storage on NFS<br>c.KubeSpawner.service_account = &#39;notebook&#39;<br>nfs_server = os.environ.get(&#39;NFS_SERVER&#39;)<br>nfs_path = os.environ.get(&#39;NFS_PATH&#39;)<br>c.KubeSpawner.volumes = [<br>    {<br>        &#39;name&#39;: &#39;home&#39;,<br>        &#39;nfs&#39;: {<br>            &#39;server&#39;: nfs_server,<br>            &#39;path&#39;: nfs_path<br>        }<br>    }<br>]<br>c.KubeSpawner.volume_mounts = [<br>    {<br>        &#39;name&#39;: &#39;home&#39;,<br>        &#39;mountPath&#39;: &#39;/users&#39;<br>    }<br>]</pre><pre># Start Notebook in user directory<br>c.KubeSpawner.notebook_dir = &#39;/users&#39;<br>c.KubeSpawner.default_url = &#39;/tree/home/{username}&#39;</pre><p>It works well because we have first made sure that the notebook would run with the right uid with <a href="https://github.com/ulaval/valeria-jupyterhub/blob/3ff8aab5088ca599d12f7b10d414a7fc82d79b85/jupyterhub-valeria/.jupyter/jupyterhub_config.py#L121">these lines</a>:</p><pre>secret_version_response_uid = vault_client.secrets.kv.v2.read_secret_version(<br>                    mount_point=&#39;valeria&#39;,<br>                    path=&#39;users/&#39; + vault_entity_id + &#39;/uid&#39;,<br>                )<br>                spawner.uid = int(secret_version_response_uid[&#39;data&#39;][&#39;data&#39;][&#39;uid&#39;])</pre><p>That way, the user has access to its own home directory, but also to other directories for which access may have been granted. From Jupyter he sees only these ones. From a terminal he can list all directories, but cannot enter them. That’s an interesting approach from a collaboration perspective, best suited than mounting a PVC for each user.</p><p><strong>Connections to S3</strong> are made on the notebook side. Remember that we have injected the S3 credentials in the notebook at spawn time? That’s were it’s used. The HybridContentManager assigns different file/content manager depending on the path you request. For the standard filesystem (the one we mounted through NFS) it’s straightforward. For S3 we first find all the buckets a user has, and we configure a S3ContentManager for each one, under the datalake_bucketname path.<br>So in the jupyter_notebook_config.py file we have:</p><pre>#######################<br># Directories mapping #<br>#######################<br>import boto3<br>from s3contents import S3ContentsManager<br>from pgcontents.hybridmanager import HybridContentsManager<br>from notebook.services.contents.filemanager import FileContentsManager</pre><pre># We use HybridContentsManager (<a href="https://github.com/quantopian/pgcontents">https://github.com/quantopian/pgcontents</a>),<br># FileContentsManager for accessing local volumes<br># and S3ContentsManager (<a href="https://github.com/danielfrg/s3contents">https://github.com/danielfrg/s3contents</a>) to connect to the datalake<br>c.NotebookApp.contents_manager_class = HybridContentsManager</pre><pre># Initialize Hybrid Contents Manager with local filesystem<br>c.HybridContentsManager.manager_classes = {<br>    # Associate the root directory with a FileContentsManager.<br>    # This manager will receive all requests that don&#39;t fall under any of the<br>    # other managers.<br>    &#39;&#39;: FileContentsManager<br>}</pre><pre># Get S3 credentials from environment variables<br>aws_access_key_id = os.environ.get(&quot;AWS_ACCESS_KEY_ID&quot;)<br>aws_secret_access_key = os.environ.get(&quot;AWS_SECRET_ACCESS_KEY&quot;)<br>endpoint_url = os.environ.get(&quot;S3_ENDPOINT_URL&quot;)</pre><pre># Add datalake connection information<br>if (aws_access_key_id and aws_access_key_id!=None): # Make sure we have usable S3 informations are there before configuring<br>    # Initialize S3 connection (us-east-1 seems to be needed even when it is not used, in Ceph for example)<br>    s3 = boto3.resource(&#39;s3&#39;,&#39;us-east-1&#39;,<br>                        endpoint_url=endpoint_url,<br>                        aws_access_key_id = aws_access_key_id,<br>                        aws_secret_access_key = aws_secret_access_key,<br>                        use_ssl = True if &#39;https&#39; in endpoint_url else False ) # Provides for test environment with no https<br>    # Enumerate all accessible buckets and create a folder entry in HybridContentsManager<br>    for bucket in s3.buckets.all():<br>        c.HybridContentsManager.manager_classes.update({&#39;datalake_&#39;+bucket.name: S3ContentsManager})</pre><pre># Initalize arguments for local filesystem<br>c.HybridContentsManager.manager_kwargs = {<br>    # Args for the FileContentsManager mapped to /directory<br>    &#39;&#39;: {<br>        &#39;root_dir&#39;: &#39;/users&#39;<br>    }<br>}</pre><pre># Add datalake connections arguments<br>if (aws_access_key_id and aws_access_key_id!=None):<br>    # We don&#39;t have to reinitialize the connection, thanks for previous &quot;for&quot; not being scoped<br>    # Enumerate all buckets and configure access<br>    for bucket in s3.buckets.all():<br>        c.HybridContentsManager.manager_kwargs.update({&#39;datalake_&#39;+bucket.name: {<br>            &#39;access_key_id&#39;: aws_access_key_id,<br>            &#39;secret_access_key&#39;: aws_secret_access_key,<br>            &#39;endpoint_url&#39;: endpoint_url,<br>            &#39;bucket&#39;: bucket.name<br>        } })</pre><h3>Conclusion</h3><p>I will finish by something that will also use as a disclaimer. Valeria has been developed at Université Laval by a fantastic team that I had the chance and the pleasure to lead as Director of Architecture and CTO. I recently left ULaval to join Red Hat, but I am really proud of the work we’ve done on Valeria for the past two years. I’m sure that this will be a great asset for researchers and students, that will allow them to embrace more easily this amazing world of Data Science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b26e7e02cba4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>