<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Russ Ferriday on Medium]]></title>
        <description><![CDATA[Stories by Russ Ferriday on Medium]]></description>
        <link>https://medium.com/@russferriday?source=rss-28567a926d9c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*hkjTZc9hckylSiQB.jpg</url>
            <title>Stories by Russ Ferriday on Medium</title>
            <link>https://medium.com/@russferriday?source=rss-28567a926d9c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:39:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@russferriday/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring Capsule Networks with NVIDIA-Docker]]></title>
            <link>https://medium.com/@russferriday/exploring-capsule-networks-with-nvidia-docker-25b843d3964c?source=rss-28567a926d9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/25b843d3964c</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[capsule-networks]]></category>
            <dc:creator><![CDATA[Russ Ferriday]]></dc:creator>
            <pubDate>Mon, 02 Apr 2018 17:48:29 GMT</pubDate>
            <atom:updated>2018-04-02T21:48:14.068Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*nL1KPBWam1BCPp_eumsHtA.png" /></figure><p>I’m considering the potential for <a href="https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b">Capsule Networks</a> in the breast cancer project I’m running. Capsule Networks show great potential to be the <em>next-big-thing </em>in AI and Deep Learning. They add the concept of <em>pose</em> to features found in images. This may be very useful in identifying and aligning tissue prior to differencing.</p><p>There will be much experimentation on my GPU enabled host, therefore I have configured an NVIDIA enabled docker engine so that I can run the experiments in Docker, and never need to worry about conflicts or cleaning up my host.</p><p>Experimental code that I run will have a Dockerfile added, and be X11 enabled where needed. This gives me complete control of versions and libraries, and makes it easy to share with others.</p><p>In this post, I’m documenting my approach to Gram-ai’s <a href="https://github.com/gram-ai/capsule-networks">implementation</a> on PyTorch. Other implementations can be treated in a similar way.</p><p>There are two parts. The first is to build a local Docker image for the pytorch repo. Then we use that as the base for an extension that includes the Capsule Network code and dependencies.</p><h4>Environment</h4><p>Linux 16.04.01 on what was my son’s gaming box with an NVIDIA GeForce GTX 970 GPU.</p><h4>Prerequisites</h4><p>git, docker, nvidia-docker</p><h3>Build a local pytorch image</h3><pre>git clone <a href="https://github.com/pytorch/pytorch.git">https://github.com/pytorch/pytorch.git</a><br>cd pytorch<br>docker build -t pytorch -f docker/pytorch/Dockerfile .<br>cd ..</pre><h3>Build a Capsule Network image</h3><p>This will be based on the local pytorch image from above</p><p>For this we are going to use my repo fork. I have added a requirements.txt and Dockerfile. The new image will be called <em>capsnet. </em>I’ll upstream the Dockerfile, if possible, so if this clone does not work, look here instead: <a href="https://github.com/gram-ai/capsule-networks">https://github.com/gram-ai/capsule-networks</a></p><pre>git clone <a href="https://github.com/topiaruss/capsule-networks.git">https://github.com/topiaruss/capsule-networks.git</a><br>cd capsule-networks<br>docker build -t capsnet .</pre><h3>Load and run the image</h3><pre>nvidia-docker run --rm -it --ipc=host capsnet:latest</pre><p>at the # prompt:</p><pre>python -m visdom.server &amp; python capsule_network.py</pre><p>Some messages should start printing while a few components are downloaded, and then the network will start being computed.</p><h3>Watch</h3><p>To monitor progress, you will access the visdom server, which makes it easy to graph test results. This is already running. Access it in a browser outside the docker image.</p><p>In a new terminal:</p><pre>docker ps</pre><p>take the container id for the capsnet image from the first column and replace it in the following command:</p><pre>docker inspect -f &#39;{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}&#39; &lt;CONTAINER_ID&gt;</pre><p>Take the printed IP address output and put it in your browser URL bar. &lt;IP&gt;:8097</p><p>You’ll see some controls at the top, and eventually some graphs will appear and periodically update.</p><p>Let me know if this is useful. And please share some applause.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25b843d3964c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[App dev: Restart on Kubernetes(K8s)]]></title>
            <link>https://medium.com/@russferriday/app-dev-restart-on-kubernetes-k8s-ac82eefd2e0?source=rss-28567a926d9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac82eefd2e0</guid>
            <category><![CDATA[system-architecture]]></category>
            <category><![CDATA[ci]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Russ Ferriday]]></dc:creator>
            <pubDate>Mon, 20 Nov 2017 17:03:25 GMT</pubDate>
            <atom:updated>2017-11-29T14:30:31.102Z</atom:updated>
            <content:encoded><![CDATA[<p>In the first part of 2016 I began work on an online service to perform web-scraping, analytics, graph-data, and content monitoring. I chose <a href="http://storm.apache.org/">Apache Storm</a> (previously called Twitter Storm) to bind all of the components together into a processing topology. Storm is a great stream computing system, but proved hard to deploy and sustain with the resources I had available at the time for that project.</p><p>I was wrestling with all that, when a friend asked for help with a project at a UK government agency. I chose to put the analytics project on the back-burner for a while, take the <a href="https://en.wikipedia.org/wiki/King%27s_shilling">king’s shilling</a>, and rethink my architecture as a background task.</p><p>A rather frustrating year later, I now have a better plan, and I’m working quickly through it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*5_90jGNJN1zbNNemu4Aqkg.png" /></figure><p>Those intervening months have given time for <a href="https://kubernetes.io/">Kubernetes</a>(K8s) to mature. At the moment it’s the leading solution for container deployment, and I’m embracing it tightly. For my current development, I’ve deployed K8s to <a href="https://azure.microsoft.com/en-gb">Azure</a> using <a href="https://coreos.com/tectonic/">Tectonic</a>. I won’t go into a sales-pitch, but the additional convenience and power when developing and deploying on K8s compared with deploying on the AWS/Azure infrastructure level is mind-blowing.</p><p>Currently, Tectonic is the most dependable way to deploy and manage K8s clusters. I previously evaluated <a href="https://github.com/kubernetes-incubator/kube-aws">kube-aws</a>, which I introduced into the agency six months ago, and <a href="https://github.com/kubernetes/kops">kops</a>. Kops has a lot to be said for it, and I may use it in the future, once my development phase tails off and my deployment becomes more industrial scale. Tectonic though is more polished, has a more effective web UI, and comes with Prometheus built-in. My strategy is to use the Tectonic free license up to 10 instances, and then either switch to kops when the product grows, or just start paying the Tectonic license. Given my experience so far, I think the license fee may pay for itself.</p><h3>Helm</h3><p>With Tectonic installed and a storageClass defined, I immediately deployed <a href="https://docs.helm.sh/">Helm</a>. This is the K8s package management system that is receiving a lot of support across the industry.</p><p>My app relies on several services, such as kafka, redis, zookeeper, postgres, and others. Helm allows me to install production-grade version of these within 10 minutes. By production-grade I mean multiply-redundant, replicated-data and services, with deployments that keep services running should a node fail, and return the system to full service within minutes, fully automatically, without any downtime.</p><h3>Continuous Integration and Deployment (CI/CD)</h3><p>The app will consist of about 10 component-types, designed to be deployed in a layered geo-aware topology across several continents. The architecture has evolved due to separation-of-concerns and what I learned last year about what to replicate, geographically separate, or layer. Of course the components will manifest themselves as Docker images. Clearly this is not a trivial task, and having a maintainable CI pipeline will be critical. Setting up that pipeline was last week’s task. I’ll share my thoughts to help others moving down the same path.</p><p>Firstly I need to share a little philosophy.</p><p>I regard the path to deployment as a matter of life-or-death for a project. In my recent experience, I’ve seen £millions wasted on deployment. In former lives I’ve seen major projects require months of un-budgeted redevelopment because deployment was considered only after a large part of the functional system was developed. Good system-architects are unlikely to be killed by traffic going the wrong way down a one-way street. It about awareness.</p><p>It’s important to know the ‘shape’ of the deployed product early, and choose the tools and deployment environment wisely. In general, I force a narrow product all the way through to deployment before adding a lot of features. The ongoing app development can then adapt to and embrace the realities of the deployment, and deployment-risk is minimised.</p><p>Bearing in mind this philosophy, I evaluated several CI tools starting with Jenkins, along with its plugins. I deployed to my cluster fairly quickly, and Jenkins seemed like a viable choice. I defined pipeline files, got my repos building on commit, and had a plan for managing image deployment. While it could have worked out for me, it also made me queasy. Though the pipeline files sit in the code repository, your config is still split between Jenkins and the repo, and I could foresee having to create new Jenkins jobs to handle development branches. I’m also unhappy about the maintenance burden and the large number of checkboxes you are threatened with in Jenkins. Jenkins does Docker as an afterthought, so I left it to one side to check out the container-native CI systems, <a href="https://github.com/drone/drone">Drone</a> and <a href="https://concourse.ci/">Concourse</a>, with a quick glance at <a href="https://www.spinnaker.io/">Spinnaker</a> on the way. (Edit: <a href="https://blog.spinnaker.io/lifting-the-sail-how-spinnaker-maps-resources-to-kubernetes-57da9c1657ba">this article</a> points out some of the mental mapping needed to use Spinnaker with K8s.)</p><p>Let’s dismiss Spinnaker first. I had heard of this, but never investigated in depth. It’s container friendly, but not container-native. And it does a LOT, including baking AMI images, and monitoring cross-cloud and on premise deployments. There was complexity there that I wanted to avoid. I knew I could find better.</p><p>I first played with Drone about two years ago, though not seriously, and not on a real project. It looked really attractive to me since it had no central configuration — all configuration exists as compact directives in a drone file in each repo. This had been immediately interesting back then, but I chose to go with <a href="https://codeship.com/">Codeship</a> for CI, partly due to the build infrastructure provided as part of the Codeship offering and its ability to configure multi-image testing clusters.</p><p>This time around I spent a couple of hours looking at discussions on the Drone github project, where it is became clear that the dev team has a lot of ideas about what can be done in the K8s age, but not yet a clear plan. They are looking for a volunteer to push this forward. If I could clone myself, I could be that volunteer, but not in this lifetime.</p><p>Codeship was cool, but not cheap, and this time around I KNOW that I’m deploying on K8s, so I wanted to give Concourse a very close look. In case you missed the point here, if you are deploying on K8s then you already <em>have</em> your build infrastructure. Up to a certain project size, your CI pipeline will run on exactly the same cluster where you deploy for production. Can you see why this is not just a convenience but also a key goal for me? Anyway, Codeship is out of the picture.</p><h3>A Winner</h3><p>Long story short: I’m going with Concourse on a trial basis! It has a convenient cli, is configured in pipeline files in one or more watched repos, and is easy to deploy to K8s via Helm. The community is active, and the concepts fit my brain.</p><p>Currently I have a CI pipeline that defines and builds two images that are pushed to <a href="https://quay.io/">Quay.io</a>, loaded to the cluster using my Helm Chart, and communicate via kafka.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*ggbwJoVDrucPjCrWZjCCtA.png" /></figure><p>Regarding Quay.io: I’m making use of the docker image security scanning that’s free with the starter level of Quay, for public images. Here’s a scan of an image that needs some updates:</p><p>I’ll report in more detail when I’m further in.</p><h3>OpenVPN</h3><p>A final thought for my friends at the government agency I mentioned earlier: OpenVPN. A couple of months ago we were using K8s as a test environment for a microservices-based transformation project. I was unable to experiment with installing OpenVPN as part of the K8s cluster. Instead we limped along with a temporary script I created a script that used kubectl port-forwarding. What a shame. Now, with my own cluster, I deployed OpenVPN in about 5 minutes and was immediately able to securely communicate with each pod, service, and node. It means, for instance that I can place intermediate repositories (nexus, devpi) right in the cluster, and build code interchangeably on my laptop or in the cluster, all with the convenience of accurate and fresh DNS resolution.</p><h3>Next</h3><p>My next little task is to setup <a href="https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html">devpi</a> in the cluster, and then refocus on migrating the application parts from the old (Storm) model to the new. I must say that K8s has been a real enabler. Whereas I could have been developing on single instances of redis, for example, or half-baked zookeeper configurations, with a view to upgrading them for deployment, now I’m actually building and testing against production-ready infrastructure, and I have complete control.</p><p>If you find this useful, please clap loudly, and provide feedback. Happy to take corrections/expand. Also share your thoughts on K8s, CI/CD, and getting apps done faster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac82eefd2e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dataflow computing. Roundabouts for Big Data]]></title>
            <link>https://medium.com/@russferriday/dataflow-computing-roundabouts-for-big-data-f41fae718de4?source=rss-28567a926d9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f41fae718de4</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Russ Ferriday]]></dc:creator>
            <pubDate>Mon, 16 May 2016 13:36:01 GMT</pubDate>
            <atom:updated>2016-05-16T13:36:01.553Z</atom:updated>
            <content:encoded><![CDATA[<p>[I published this on LinkedIn a while back. I’m putting it on Medium today, and will follow up soon with the second part.]</p><p>Have you noticed that traffic lights solve the problems that they conveniently create? A red light forces traffic to build up, creating the appearance that there is a lot of traffic, while making the traffic light seem important and useful. Now that there is a queue, the light changes to give the traffic a time-slot to cross the junction. Again the self-importance of the traffic light grows, while other traffic queues build from other directions.</p><p>The roundabout (traffic circle if you prefer) is more modest. It deals in a low-tech way organically with the traffic as it arrives. Momentary backlogs build, but in many cases disperse quickly. Roundabouts are proven to be safer than traffic lights in general; they can be more efficient, while often being much cheaper to build and run.</p><p>In inner-city grids, where density becomes very high, so-called junction-capacity becomes critical, and complex new needs develop. At that point there’s nothing for it but to install lights at all crossings.</p><h3>A relevant analogy</h3><p>The traffic-light/roundabout decision has aspects in common with the current comparisons in Big Data circles between batch-oriented Hadoop/Map-Reduce approaches and dataflow computing.</p><p>In batch-oriented Map-Reduce jobs we enqueue data elements waiting for processing, process them, store results, then start queueing up data again.</p><p>In contrast, dataflow computing takes the approach of processing individual data elements as they arrive, finding a home for them, aggregating them into running totals, and perhaps injecting them into other dataflows. A lot like a roundabout.</p><h3>Advantages of the dataflow approach</h3><ul><li>less latency between data arrival and the outcome</li><li>spreading the processing needs across time removes the high computational load encountered while running a batch job</li><li>simpler, more understandable (more correct?) operations at each step</li><li>flexibility in the face of changing requirements</li><li>can allow flexible scaling — small systems built this way can be scaled organically</li><li>like roundabouts, can be cheaper to build and run than batch systems</li><li>tends to encourage locality-of-reference — a deep topic, but a big performance booster</li></ul><h3>Disadvantages</h3><ul><li>A rethink is needed, if you are batch oriented. No surprise there</li><li>In some cases the communications overhead in the flow overwhelm the computational load. In such systems a batch approach may be better</li></ul><h3>Guidelines?</h3><p>I’ve been working with graph-based data and parallel computation since the mid 80s in a broad variety of applications from data-networking to image-processing, business applications, and intelligence gathering. Many of the applications were real-time, where performance and latency were critical. I’ve gathered a lot of experience of the pros and cons of these approaches, and the kinds of decisions one is faced with while solving high-data-rate computational problems. Much of that experience goes beyond technical, touching on overall project architecture, project management, and development processes.</p><p>I’ll be posting guidelines on using this technology, over the coming months, as time permits. I’ll post here on Medium, with a reference from LinkedIn. After comparing the experience on each, I’ll work out a policy for handling comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f41fae718de4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CI/CD with python packages on Codeship]]></title>
            <link>https://medium.com/@russferriday/ci-cd-with-python-packages-on-codeship-b83bb4d59d37?source=rss-28567a926d9c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b83bb4d59d37</guid>
            <category><![CDATA[codeship]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devpi]]></category>
            <dc:creator><![CDATA[Russ Ferriday]]></dc:creator>
            <pubDate>Sun, 15 May 2016 17:20:00 GMT</pubDate>
            <atom:updated>2016-05-15T19:04:45.440Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KUriivslJiBu080UhkEQeQ.jpeg" /></figure><p>My current project has been developed using a typical <a href="http://doc.devpi.net/latest/">devpi</a> flow. I’m up to about a dozen python packages covering datamodel, api-service, image processing, scraping, and several more. (In passing, I’d like to say, <a href="https://twitter.com/hpk42">@hpk42</a> , <a href="https://twitter.com/fschulze">@fschulze</a>, and other contributors deserve a lot of credit for devpi and related python tools and infrastructure.)</p><p>While my project is going well, it <em>has</em> become a lot to manage. Getting a good fix/stage/deploy flow was always going to require a better process for CI/CD. I often use the existing CI/CD in my clients’ infrastructure, and tend to just-get-by for my own projects by winging-it. Time to change this.</p><p>Most of my deployments now hinge around Docker, a commitment I made two years ago. So I want CI/CD that’s very Docker-centric, but with a lot of flexibility to integrate with non-docker projects. After doing some research, I signed up for a <a href="http://codeship.com/">Codeship</a> Docker trial. I’m now two weeks in and already convinced it’s the way to go. The real beauty is that I can build dockerized test infrastructure even for mundane components, and wire components repeatably together. I don’t need to worry about setting up dev servers, since even integration tests can be run on Codeship for each commit of any branch to a repo.</p><p>My current flow looks like:</p><ul><li>Branch</li><li>Edit locally</li><li>Test locally (in a docker cluster), simply using:</li></ul><pre>$ jet steps</pre><ul><li>Commit as usual to branch, which triggers a hosted codeship build and test.</li><li>On success, codeship can merge to a master branch, then push images to a docker registry, and/or packages to devpi.</li></ul><p>Lacking for me has been the complete story for where and how to publish my <em>private</em> python packages, in this new environment. In development mode, these all sit on my laptop in a local devpi service. How is that going to work when Codeship wants to upload a new version of a tested datamodel? Or when another test wants to pull down that datamodel package?</p><p>The common case would be to setup a private devpi server behind <em>nginx</em>. This raises questions about security, authentication, and longevity of files in the registry. Plus, I’d have the cost/overhead of running an instance.</p><h4>Trying too hard?</h4><p>I did some experiments using a short-lived devpi instance running in the Codeship CI/CD stack alongside my code under test, allowing me to upload python packages when tests pass. I tried persisting my packages in Amazon S3 storage, which is secure, and cheap. Since the Codeship test environment is secure, I would not have to secure the devpi service, and would pass S3 access credentials to devpi in the test environment. I experimented for a day using the yas3fs S3 filesystem, which handles cache invalidation and consistency via notification.</p><p>After a negative experience retrieving packages from the index, and a discussion with <a href="https://twitter.com/fschulze">@fschulze</a>, I have now ditched this approach, since some part of devpi does not play well with a cached remote filesystem like S3. I’d like to get that to work, but don’t have time to spare, unless someone would like to fund the necessary investigation.</p><h4>Final approach</h4><p>After the yas3fs detour, I created a small docker host out on the internet, running with my Azure credits. It hosts a docker container with my packages, in the normal way. In this less aggressive approach, I have not worried too much about a long-term package persistence plan, since I can manage grudgingly without a long version history.</p><p>It’s a compromise, but that’s life, and it Works for Me, for Now. I’ll revisit this later.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b83bb4d59d37" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>