<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mike Wilson on Medium]]></title>
        <description><![CDATA[Stories by Mike Wilson on Medium]]></description>
        <link>https://medium.com/@knobby?source=rss-1aed845ca0a2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9n7IuALry-d3lgEjXBVMNA.png</url>
            <title>Stories by Mike Wilson on Medium</title>
            <link>https://medium.com/@knobby?source=rss-1aed845ca0a2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:24:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@knobby/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubeflow with nVidia GPUs on the Canonical Distribution of Kubernetes]]></title>
            <link>https://medium.com/@knobby/kubeflow-with-nvidia-gpus-on-the-canonical-distribution-of-kubernetes-f4c897b1de7d?source=rss-1aed845ca0a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4c897b1de7d</guid>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Mike Wilson]]></dc:creator>
            <pubDate>Fri, 15 Jun 2018 23:09:32 GMT</pubDate>
            <atom:updated>2018-06-15T23:09:32.514Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post we will work through some amazing pieces of software and hardware to jumpstart your machine learning journey. We will go from nothing to a Kubernetes cluster deployed on AWS running machine learning jobs on Kubeflow with nVidia GPUs. That’s a mouthful of acronyms, but stay with me. For now, just know that with CDK, nVidia, and Kubeflow the hardest part of machine learning is determining what you want to build and not how to get things setup and ready for the work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZGoh9eZD0MQkwSBHcLtpQ.jpeg" /></figure><h3>What is Kubeflow</h3><p><a href="https://github.com/kubeflow/kubeflow">Kubeflow</a> is <a href="https://www.tensorflow.org/">Tensorflow</a> on <a href="https://kubernetes.io/">Kubernetes</a>. That might not help much if you don’t know what Tensorflow is in the first place. So in a nutshell, Tensorflow is software for machine learning. This means that with Kubeflow you can build and run machine learning jobs on a Kubernetes cluster. Kubeflow was created to simplify machine learning so that users can focus on what matters: the machine learning jobs.</p><h3>What is CDK</h3><p>The <a href="https://www.ubuntu.com/kubernetes">Canonical Distribution of Kubernetes</a> is a very easy way to get Kubernetes setup via Juju. You can deploy on almost any cloud architecture, on premise, and even offline deployments are possible. As the name suggests, it is a distribution of Kubernetes provided by Canonical, the people that bring you Ubuntu. It uses the same binaries from upstream and provides easy upgrades and maintenance of your Kubernetes cluster.</p><h3>Installation of CDK</h3><p>The easiest way to get off the ground with CDK is to use the tool conjure-up. Conjure-up turns cloud deployments into a series of menus and magic.</p><p>Just run conjure-up kubernetes and then select Canonical Distribution of Kubernetes with nVidia GPU workers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GB0V9pvdq11I7MgfYwYIqg.png" /></figure><p>Select the Kubeflow addon:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9OUCv-1P6OMSw2PrJVRWfg.jpeg" /></figure><p>Configure the Kubeflow section of the options screen. Note that the API token is required because ksonnet downloads information via the github API and unauthenticated requests are heavily throttled causing the install to fail. This API key needs no permissions. More information about this can be found <a href="https://github.com/ksonnet/ksonnet/blob/master/docs/troubleshooting.md#github-rate-limiting-errors">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x36-a_v2TFvFlUZo8xvgyg.png" /></figure><p>Then follow the menus after that to select your cloud(I am using AWS in this example) and then click Deploy All.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIWEDAOTkljf2yx-IJT5_w.png" /></figure><p>You’ll be greeted with a display of the status of the bring-up and you can go get some coffee while all the heavy lifting is done by Juju and CDK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZqi0Pwp5fgokJsCPmClYA.png" /></figure><p>Soon you will have everything setup and ready for you to use and you’ll be greeted by a screen like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5vAlSVtkkgTd0R1op2KKA.png" /></figure><h3>Running a workload</h3><p>The completion screen for conjure-up tells you where Kubeflow is installed on your local machine. It is in ~/.local/share/kubeflow/&lt;conjure-up generated directory&gt;. Go there and to run ksonnet directly. The following commands will run a simple workload to test the use of Kubeflow. The params command sets the job to use a GPU.</p><pre>ks pkg install kubeflow/examples<br>ks generate tf-cnn kubeflow-test --name=cdk-tf-cnn<br>ks param set kubeflow-test num_gpus 1<br>ks apply cdk -c kubeflow-test</pre><p>This will start up a tensorflow job, which spawns some pods. The interesting one being the cnn worker pod, where your work actually happens.</p><pre>$ kubectl get tfjobs<br>NAME       AGE<br>mycnnjob   1m</pre><pre>$ kubectl get po<br>NAME                                               READY     STATUS    RESTARTS   AGE<br>mycnnjob-ps-ax22-0-51r0y                           1/1       Running   0          2m<br>mycnnjob-worker-ax22-0-7qy6i                       1/1       Running   0          2m</pre><p>You can then check out the log of that pod to see the progress.</p><pre>$ kubectl logs mycnnjob-worker-ax22-0-7qy6i<br>INFO|2018-06-15T22:53:20|/opt/launcher.py|48| Launcher started.<br>INFO|2018-06-15T22:53:20|/opt/launcher.py|73| Command to run: python tf_cnn_benchmarks.py --batch_size=32 --model=resnet50<br>...</pre><p>Now it’s time for the data science part of all of this. That’s where you come in, so get to it and make something amazing. Might I suggest the following Kubecon 2018 talk for inspiration. Jeremy Lewi and Stephan Fabel give a Kubeflow intro and also talk about a great github issue naming machine learning job they made as a demo.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FsC8Ce9vUggo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsC8Ce9vUggo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FsC8Ce9vUggo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/851403ecab31a1ccce44cd634e6d2321/href">https://medium.com/media/851403ecab31a1ccce44cd634e6d2321/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4c897b1de7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NFS default storage in Kubernetes with CDK]]></title>
            <link>https://medium.com/@knobby/nfs-default-storage-in-kubernetes-with-cdk-847336cc4a72?source=rss-1aed845ca0a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/847336cc4a72</guid>
            <category><![CDATA[default-storage-class]]></category>
            <category><![CDATA[cdk]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[conjure-up]]></category>
            <category><![CDATA[fn]]></category>
            <dc:creator><![CDATA[Mike Wilson]]></dc:creator>
            <pubDate>Wed, 13 Jun 2018 22:28:39 GMT</pubDate>
            <atom:updated>2018-06-13T22:30:44.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1yW_D0h17LE-E2ZRhl8JQA.png" /></figure><p>I’ve been running Kubernetes backed by NFS persistent volumes for a long time now and one of the rough edges for me has be the creation of persistent volumes(pv) and their claims(pvc). It involves me sshing to my NFS server, creating a directory for the share, creating yaml for the pv/pvc, and then finally applying these to the cluster. This was arduous, but I passed it off as acceptable since it wasn’t too often I was deploying new workloads. This process needed some help and while Kubernetes has supported default storage classes for a while, I didn’t take the time to learn how to use it or how to setup NFS to use it. I should have and hopefully I can help you as well.</p><h3>Potentially Boring Details</h3><p>To get your storage requests in your workloads to automagically get space off your NFS server requires a few things. First, you have to setup your NFS server. Then you have to run a workload on Kubernetes to support the NFS provisioning. This is due to Kubernetes not having NFS support internally and may or may not change in the future. Finally you have to tell your Kubernetes cluster to use this NFS provisioner for storage requests. The list of Kubernetes default providers is <a href="https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner">here</a>. This is all easier to do than to say, so let’s get to it.</p><h3>The Doing</h3><p>First off, we’re going to get a Kubernetes cluster up. The easiest way is to use <a href="https://conjure-up.io/">conjure-up</a>. Run:</p><pre>sudo snap install conjure-up --classic</pre><p>or on MacOS:</p><pre>brew install conjure-up</pre><p>And follow that up with:</p><pre>conjure-up canonical-kubernetes</pre><p>Select your addons, cloud, and a few other things and you’re off.</p><p>A few minutes later, you’re ready to move on to deploying an NFS server. Luckily this is an easy process with juju and the nfs charm. Note the constraint in order to request a larger disk.</p><pre>juju deploy nfs --constraints root-disk=30G</pre><p>Now we need to relate the nfs charm to the kubernetes worker charm:</p><pre>juju relate nfs kubernetes-worker</pre><p>That is it. You now have setup your Kubernetes cluster to hand out persistent volumes. You can verify that things are working by checking the storage classes on the cluster:</p><pre>kubectl get sc<br>NAME                PROVISIONER      AGE<br>default (default)   fuseim.pri/ifs   1h</pre><p>Note the (default) indicating that this provisioner is the default one. Further, we can add a persistent volume and verify that it gets bound and allocated.</p><pre>$ kubectl apply -f - &lt;&lt;EOF<br>kind: PersistentVolumeClaim<br>apiVersion: v1<br>metadata:<br>  name: test<br>spec:<br>  accessModes:<br>    - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 1Gi<br>EOF</pre><pre>$ kubectl get pv<br>NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM          STORAGECLASS   REASON    AGE<br>pvc-1af0c73b-6f58-11e8-bade-0a8bc3df5930   1Gi        RWX            Delete           Bound     default/test   default                  8s</pre><p>That is it. Now you no longer have to ssh to your NFS server or manually create persistent volumes. Hopefully this guide has helped you see how easy it is to use a default provisioner, especially with conjure-up, juju, and the Canonical Distribution of Kubernetes. Be sure to clap if this was useful to you to encourage me to continue to write these guides!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=847336cc4a72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glowing Bear and Weechat in Kubernetes with Helm]]></title>
            <link>https://medium.com/@knobby/glowing-bear-and-weechat-in-kubernetes-with-helm-b150a0eb28b8?source=rss-1aed845ca0a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b150a0eb28b8</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[irc]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Mike Wilson]]></dc:creator>
            <pubDate>Thu, 19 Oct 2017 18:42:23 GMT</pubDate>
            <atom:updated>2017-10-19T18:42:23.270Z</atom:updated>
            <content:encoded><![CDATA[<p>Having a renewed interest in irc again, I found that the best option to stay in touch is to keep <a href="https://weechat.org/">Weechat</a> running all the time and relaying the chat to <a href="https://www.glowing-bear.org/">Glowing Bear</a>. This allows for notifications on my phone of mentions and being able to see missed chat when away from my computer.</p><p>What better place to deploy such a thing than Kubernetes. I recently discovered both <a href="https://github.com/kubernetes/helm">Helm</a> and <a href="https://conjure-up.io/">conjure-up</a> and I am sold on both. I’ve been hand-rolling Kubernetes and then deploying my workloads by hand-editing yaml files and then using kubectl to deploy it all. This works, but it’s time-consuming to say the least. Cluster creation is turned into a no-op with conjure-up and Helm turns deployment into a trivial task. In this entry, we’ll spin up a Kubernetes cluster on LXD and deploy Weechat and Glowing Bear to it as an example of how easy it is to work with both conjure-up and Helm.</p><h4>Conjure-up with LXD</h4><p>This is how to easily spin up a Kubernetes cluster on LXD with conjure-up on an Ubuntu machine.</p><pre>$ sudo snap install lxd<br>$ /snap/bin/lxd init --auto<br>$ /snap/bin/lxc network create lxdbr0 \<br>ipv4.address=auto ipv4.nat=true \<br>ipv6.address=none ipv6.nat=false<br>$ sudo snap install conjure-up --classic<br>$ conjure-up kubernetes-core</pre><p>You might have to add yourself to the lxd group before you can run the lxd commands. Other than that possible snag though, you’re off to the races. Note, you can also `conjure-up kubernetes` if you want a full Kubernetes. At this point you should just be a couple screens away from the install happening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyZ-_UleAMsj731WcRyPSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHQazYgJ8Vc4Wrt7gmIhYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wK6YISmJvxtpqKV3VbbA7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAj6pSSIH05TCVNdMNOEtQ.png" /></figure><p>Congratulations. You now have a Kubernetes cluster up and running on your computer. It’s just that simple with conjure-up.</p><h4>Helm</h4><p>Helm solves lots of tedium with the templating system and it makes life much easier to deploy all but the simplest of things. My initial assumptions of Glowing Bear and how it worked were incorrect; I assumed the client/server diagram looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*690M3IJQxXlCJM9sfXWipg.png" /><figcaption>Incorrect assumption</figcaption></figure><p>But in reality, Glowing Bear is a client-side javascript script that runs on the client and connects to the irc server from the client. This means the picture looks more like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*S4ippO9TQa5MQpNK7TD6cw.png" /><figcaption>How things actually work</figcaption></figure><p>Therefore, the Glowing Bear portion of this is mostly unnecessary unless you’re just concerned that the hosted version on the Glowing Bear page could be compromised. I was too far along to stop at this point, so I claimed that self-hosting Glowing Bear was a better solution and included it in my helm chart. We need a couple ingress rules then; one for Weechat and one for Glowing Bear. This also means you need to be aware of the security implications of opening up your Weechat to the world. You needed to be aware of it before, of course, but now you’re letting people poke two things directly.</p><p>While I am on the topic of security, let me discuss the tls portion of this. Kubernetes allows us to use ingress and something like kube-lego to automagically get certificates for our domains. I will use that here, but you have to realize what this means. You have secure traffic from the client to your ingress termination point(nginx, for example). Then it is <em>clear text</em> from there to the server actually running your application. This is probably fine if you’re in a physically isolated network, but if you’re on a shared tenant network, it might be something to concern yourself with.</p><p>The next step in creating our solution is to determine the images we want to use. I went to the <a href="https://hub.docker.com/">docker hub</a> and searched for Weechat and Glowing Bear and found someone who had made both images, had them available on github, and looked sane. I’m used those as the default images instead of trying to roll my own, because it seems silly to reinvent the wheel all the time. There are enough duplicate public images on docker hub as it is.</p><p>Now we just need to create the Helm files necessary for this. Instead of just duplicating all the files here, I’ll just link to the github repo that I created from those files. You can find it <a href="https://github.com/hyperbolic2346/ircChart">here</a>. The hardest part is ensuring you have an existing persistent volume claim for Weechat config storage. The default is named Weechat, but that can be changed in the values.yaml file. Simply edit the yaml file if you’re interested in changing things such as not deploying Glowing Bear or the persistent volume claim name:</p><pre>git clone <a href="https://github.com/hyperbolic2346/ircChart">https://github.com/hyperbolic2346/ircChart</a><br>cd ircChart<br>vi values-example.yaml<br>helm install --name myircrelay -f values-example.yaml .</pre><p>Now you have a working irc setup that is ready to store and relay irc for you. To get the relay setup, you need to find the persistent volume storage location and either hand-edit the config for Weechat or crank up a Weechat instance and change settings and then copy that config over to your persistent volume. I will show both methods:</p><h4>Interactive setup</h4><pre>docker run -it --rm --tty -v /path/to/persistent/volume:/weechat jkaberg/weechat</pre><p>Once inside your docker container:</p><pre>/relay add weechat 9000<br>/set relay.network.password &quot;supersecretpassword&quot;</pre><h4>Hand-editing</h4><pre>cat &lt;&lt; EOF &gt;&gt; /path/to/persistent/volume/.weechat/relay.conf<br>[network]<br>password = &quot;supersecretpassword&quot;</pre><pre>[port]<br>weechat = 9000<br>EOF</pre><h4>Wrapup</h4><p>Kube-lego handles the TLS for us. This is another really nice package, but remember that once traffic hits the nginx proxy, the communication is no longer secured. At this point you should be able to hop on your Glowing Bear instance(or the hosted one) and use your weechat hostname and password for your ingress. Glowing Bear will then connect to your relay and you can connect to a server and join channels.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b150a0eb28b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>