<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeremy Chambers on Medium]]></title>
        <description><![CDATA[Stories by Jeremy Chambers on Medium]]></description>
        <link>https://medium.com/@thehipbot?source=rss-742d49b41197------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*S_cfnXdKDotKqHby.jpg</url>
            <title>Stories by Jeremy Chambers on Medium</title>
            <link>https://medium.com/@thehipbot?source=rss-742d49b41197------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:23:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thehipbot/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Docker for Continuous Delivery: Advanced Topics and Patterns]]></title>
            <link>https://tech.cars.com/docker-for-continuous-delivery-advanced-topics-and-patterns-7532235998af?source=rss-742d49b41197------2</link>
            <guid isPermaLink="false">https://medium.com/p/7532235998af</guid>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jeremy Chambers]]></dc:creator>
            <pubDate>Tue, 05 Sep 2017 16:31:44 GMT</pubDate>
            <atom:updated>2017-09-05T16:31:44.042Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_NQN6_YnxS29m8vFzWYlEg.png" /><figcaption>Image courtesy of <a href="https://www.docker.com/">Docker.com</a></figcaption></figure><p>This post is meant for readers who are already familiar with Docker and know how it works under the hood. If that isn’t the case then I have created an <a href="https://tech.cars.com/docker-for-continuous-delivery-the-basics-6bce7186e872">introductory post</a> just for you! That post introduces the necessary concepts and explain how they relate to Continuous Delivery pipelines.</p><p>So, we have a solid understanding of Docker and where it fits into continuous delivery. So how do we implement it? And furthermore, how do we do so efficiently? We will explore some common patterns, optimizations, and best practices that will allow your organization to leverage Docker with efficiency and consistency.</p><h3>Core Principles</h3><p>Before detailing any specifics, it will be helpful to briefly discuss some of the key principles in your Docker usage. These principles will come up frequently throughout the rest of the post and they should always be in the back of your mind when creating and running Docker containers.</p><p><strong>Keep your Docker images as small as possible.</strong> The smaller the image, the fewer bytes sent over the wire and less overhead when running a container from the image. Smaller images will also mean that your Docker hosts will fill up much slower.</p><p><strong>Standardize on everything.</strong> As you standardize on things like build tools and commands for your applications, you will see many opportunities to reduce code duplication in your Dockerfiles and pipelines. This will also make it easier for developers to use now and also to make the sweeping changes later.</p><p><strong>Decide on your tagging scheme early and enforce that decision.</strong> The tagging scheme will determine much of your pipeline logic. This is also important to provide consistency in your base images (more on this later in the post).</p><p>With that out of the way, let’s dig into some common patterns and techniques to better utilize Docker.</p><h3>Make Use of Your Docker Image Layers</h3><p>Layers can be your best friend or worst enemy when creating Docker images. When used effectively, they can speed up build times through layer caching, but if not they can greatly increase build times and image sizes.</p><p>Each layer is associated with a single command on the underlying Dockerfile. These layers are stacked one atop the other with the bottom most being the first command in the Dockerfile and the top being the last. None of this information is news, it’s covered in my previous post and in the Docker documentation, but it is important to understand how it can cache and reuse these layers.</p><p>Docker will cache layers based on what has changed in the files added to the image and the commands which were run to build the image. The stacking order of image layers is very important because if a layer is unchanged, Docker will pull it from cache or the registry directly rather than rebuilding. Any change in a layer could have an effect on the image up until that point, and thus will require rebuilding all subsequent layers. What that means to us is that the image layers should be ordered by their likelihood to change.</p><p>In general, commands like CMD and ENV which are not likely to change frequently should be up at the top of the file underneath FROM, whereas commands like COPY or ADD which move your code into the image should be at the bottom since the code will change on most builds. As for installing other dependencies or running commands to configure the environment these should be right in the middle.</p><p>Now we can extrapolate upon these ideas with some common patterns useful in your images.</p><h4><strong>Reducing Layer Sizes</strong></h4><p>Aside from stacking order, how you build a single layer can be very important as well. Let’s look at the example in my previous post:</p><pre>RUN apt-get update -y &amp;&amp; apt-get install -y curl</pre><p>You can see that rather than creating two separate run commands, one to do the update and the other to do the install, the two are concatenated. You will see this often in Dockerfiles and this is because the two commands are logically associated and should make up a single layer of the image. Since layers in a Docker image are zipped, stored, and sent separately to assemble the overall image less layers will mean less overhead with things like network latency and running containers.</p><p>One useful bit of information on layers is that Docker will only commit what is there in the filesystem at the time that your layer’s task is complete. That being the case, your Dockerfile commands should also clean up after themselves in the same RUN command to free space on the filesystem before committing the layer that way any extraneous files are not committed. Let’s refactor the previous command to use this method:</p><pre>RUN apt-get update -y &amp;&amp; \<br>    apt-get install -y curl &amp;&amp; \<br>    rm -rf /var/lib/apt/lists<em>/*</em></pre><p>Notice the last command which we added rm -rf /var/lib/apt/lists/*. What you may not know about aptitude and many other package managers, is that when you run an update command it will pull down a manifest of all available packages and where to find them. This manifest is surprisingly large, roughly 40 MB in this example, and we do not need this data after installing curl. By cleaning that out as part of our installation command we have saved that space from the layer and the overall image. This technique could apply to anything that is no longer needed to run the image. Things such as tar files, binaries, or scripts that are used only as part of the build can then be removed.</p><h4><strong>Clearing the Cache</strong></h4><p>A common pattern you will see to leverage layering in the Dockerfile similar to this:</p><pre>ENV ENVIRONMENT_REFRESH 2017-06-22</pre><p>This is setting an environment variable called ENVIRONMENT_REFRESH and it will be positioned directly underneath the FROM command. This will not be used anywhere in the code but is serves as a way to break the cached layers as it will force Docker to rebuild all subsequent layers. This is necessary if you want to install updated versions of dependencies or refresh configurations.</p><h4><strong>Caching for Reduced Build Time</strong></h4><p>We have seen how we can use our knowledge of image layers to reduce our overall image size, but they can greatly reduce your build times when used intelligently. As I have said, the layers will be cached and used in subsequent image builds when there has been no change to the layer. This can be used with great effect to cache any long running commands used to build the image and is especially useful with external dependencies which can take a great deal of time to download and install. The problem here is that the dependencies can change with the code. Let’s take a look at a naive approach with a front-end project built by Node.js:</p><pre>FROM node:6.7.0-wheezy</pre><pre>ENV REFRESHED_AT 2017-03-21</pre><pre>CMD [&quot;npm&quot;, &quot;run&quot;, &quot;build&quot;]</pre><pre>RUN mkdir /app &amp;&amp; \<br>    mkdir /app/dist</pre><pre>WORKDIR /app</pre><pre>COPY . /app</pre><pre>RUN npm set progress=false &amp;&amp; \<br>    npm install --silent</pre><p>Here we need to copy in our current version of the code before we run the npm install command (including the package.json file which defines the projects dependencies). This means that we must install dependencies every time we run a build in our container, but this is avoidable. If we were to cache a layer including our dependencies, we would need an appropriate file to cache with. Since a change in a dependency would require a change to the package.json file, we can cache the layer based on that file. Let’s see how this would look:</p><pre>FROM node:6.7.0-wheezy</pre><pre>ENV REFRESHED_AT 2017-03-21</pre><pre>CMD [&quot;npm&quot;, &quot;run&quot;, &quot;build&quot;]</pre><pre>RUN mkdir /app &amp;&amp; \<br>    mkdir /app/dist</pre><pre>WORKDIR /app</pre><pre>COPY package.json /app</pre><pre>RUN npm set progress=false &amp;&amp; \<br>    npm install --silent</pre><pre>COPY . /app</pre><p>With this change we first copy in the package.json file separate from the rest of our code, run the install, then copy in the remainder of the code. Since we have the package.json copied before the install, Docker will only rebuild these two layers if the package.json file is changed.</p><p>Using this technique is great to reduce build times and overhead for your images, but be wary if you use any mutable versions or version sugar (e.g. Maven SNAPSHOTS or “^1.0.0” in a package.json) which allow different versions of a dependency to be installed depending on what is available at the time of installation. In this case, a break can appear in the artifact produced at build which may not be apparent on a developer’s machine.</p><h3>Create Base Images</h3><p>Even if the public Docker image you are using for your app is as small as it can be and covers all the dependencies you need, you still may want to consider making your own base images. You can contain common steps or logic in your images as well as common authentication making your developers lives easier. This will likely require that you have a private Docker registry, but <a href="https://docs.docker.com/registry/deploying/">these can be pretty simple to setup</a>. As always, keep the images as small as possible.</p><h4><strong>Common Logic in Base Images</strong></h4><p>Putting common logic into base images will reduce the amount of duplicate code and also will make sweeping changes simpler. A great example of some common logic would be the inclusion of some tools used globally for certain types of applications in your organization. Another common example is including information on a private artifact repository which your app relies on. Here is an example of a Dockerfile for a base image of a Node.js application:</p><pre>FROM node:6.11.1-alpine</pre><pre>RUN npm config set registry <a href="http://repository.cars.com/artifactory/api/npm/npm">http://myprivatenpmrepo.com/</a></pre><p>We are using the Node.js official alpine image but setting the npm registry to a private registry. Any images running from that base will automatically be configured to pull Node modules from the private repository rather than the public. If the URL to the repository ever needs to change we can then update the image and users will receive the update.</p><h4><strong>Use ONBUILD Commands in Images</strong></h4><p>Docker provides a directive ONBUILD for your Dockerfiles. What this does is it specifies commands which are to be run when a new image is being built from the image with the ONBUILD commands. One way to think of it is as a form of inheritance for images and it can be very useful in reducing code duplication up your Dockerfiles. In fact, oftentimes Dockerfiles in your individual projects can be just a single FROM command using your onbuild image. The one caveat to this is that you must have a standard set of build and run commands for all projects of that type.</p><p>Let’s look at an example for a Node.js project:</p><pre>FROM node:6.11.1-alpine</pre><pre>ONBUILD CMD [&quot;node&quot;, &quot;/app/index.js&quot;]</pre><pre>ONBUILD COPY package.json /app</pre><pre>ONBUILD RUN npm set progress=false &amp;&amp; \<br>            npm install --silent</pre><pre>ONBUILD RUN npm test &amp;&amp; \<br>            npm coverage</pre><pre>ONBUILD COPY . /app</pre><p>Now, say we build an image from the above Dockerfile and tag it as thehipbot/node:6.11-onbuild and publish this image. Then we can use this image for any Node.js project so long as it has the npm scripts for test and coverage and can be run from the file index.js in the root of the project. The Dockerfile for our projects will be one line:</p><pre>FROM thehipbot/node:6.11-onbuild</pre><p>And all of the logic will live in the shared image.</p><p>It is very beneficial for you and your organization to create and maintain common base images for all types of applications you plan to support and even create onbuild images off of these bases. This provides a good place to make sweeping changes, reduce the amount of code, and simplify your CI pipelines.</p><h3>Separating Build and Run Dependencies</h3><p>Typically, your projects dependencies and tasks can be divided into two categories, those needed to build the application and those needed for it to be run. If we can separate the two stages, we will at least reduce the size of our images but can also speed up things like the time it takes to start our app.</p><p>Ideally we want both stages running in a containerized environment to ensure reproducibility, but we would want the two phases to take place in separate containers with the result of the build being copied into the run image. In that case, we would need only the production dependencies or an artifact generated by the build in our run image.</p><p>As an example of this we will be looking at a sample build for a front-end application, but the same pattern will apply to compiled languages as well. With our front-end code our build stage will verify the code with unit tests and linting, then run a build tool like webpack to minify and bundle. For our run stage, we would like to create an NGINX image to serve up our bundled JavaScript and CSS.</p><p>Assuming we have an npm script build which will test, lint, then package our front-end code, here is what our build stage Dockerfile would look like:</p><pre>FROM node:8-wheezy</pre><pre>WORKDIR /app</pre><pre>CMD [&quot;cp&quot;, &quot;./dist&quot;, &quot;./build/&quot;]</pre><pre>COPY package.json /app</pre><pre>RUN npm set progress=false &amp;&amp; \<br>    npm install --silent &amp;&amp; \</pre><pre>COPY . /app</pre><pre>RUN npm run build</pre><p>Ignore, for a moment, the 3rd line with our CMD directive. It will be important when we combine our stages.</p><p>And here is our Dockerfile for the runnable NGINX container:</p><pre>FROM nginx</pre><pre>COPY build/ /usr/share/nginx/html</pre><p>We are copying the contents of a folder build (look familiar?) into the default directory out of which the NGINX container will serve files.</p><p>Now in older versions of Docker, the only option to connect these pieces is to do so manually, typically in a shell script. This script should run the build, pull the result out of the build container, and then drop the result into an NGINX container which can be run to serve those files. For this case, we need two Dockerfiles which we will call Dockerfile.build and Dockerfile.run. Here’s what that would look like:</p><pre>#!/usr/bin/env bash</pre><pre># build the build-stage image<br>docker build -t thehipbot/fe-build -f Dockerfile.build .</pre><pre># run the build stage image<br>docker run -v $(pwd)/build:/app/build thehipbot/fe-build</pre><pre># build the runnable container<br>docker build -t thehipbot/fe-run -f Dockerfile.run .</pre><pre># now if we run the new image, we will have NGINX servering<br># up our static files<br>docker run -p 80:80 thehipbot/fe-run</pre><p>So all in all, what we do here is volume map (-v $(pwd)/build:/app/build) a folder called build from our local filesystem to the first container so that it can copy the result of the build out of the container. From there the build of our runnable image picks up those files and copies them into the image.</p><p>This technique works great, but as of Docker CE version 17.05.0-ce, released in May, we have a feature called multi-stage builds which provides a cleaner way to do this.</p><h4><strong>Multi-stage builds</strong></h4><p>Multi-stage builds allow us to include multiple FROM commands in your Dockerfiles which creates a new stage with a method to retrieve and use artifacts from previous stages. Let’s see our example refactored to use this.</p><p>Here we have our single Dockerfile with multiple stages:</p><pre>FROM node:8-wheezy as build</pre><pre>WORKDIR /app</pre><pre>COPY package.json /app</pre><pre>RUN npm set progress=false &amp;&amp; \<br>    npm install --silent &amp;&amp; \</pre><pre>COPY . /app</pre><pre>RUN npm run build</pre><pre>FROM nginx</pre><pre>COPY --from=build /app/build/ /usr/share/nginx/html</pre><p>With just a small change we have just one Dockerfile and do not have to worry about copying any files in or out of containers. Notice in our first FROM command we have appended as build which allows us to reference this first stage in the build from other stages in the same Dockerfile. We then use this reference in our COPY --from=build ... command to pull files from the build and copy them into our final result stage.</p><p>Using multi-stage builds (or scripts if you are stuck on an older version of Docker) to reduce image sizes is a great idea for your images. I also have an example of a multi-stage build in <a href="https://github.com/TheHipbot/example-api-go">Golang</a> if you would like a similar example in a compiled language.</p><p>The techniques which we have covered over the course of this post will benefit your practices with Docker and CI in many ways. To name a few, It will reduce the amount of space used on your Docker host and artifact repository, reduce build and deploy times, and reduce duplicate code. If you take nothing else from this post, keep in mind the principles mentioned at the start since many of the patterns arise directly from them.</p><p><em>Do you have any other patterns you have found useful? Please tell me in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7532235998af" width="1" height="1" alt=""><hr><p><a href="https://tech.cars.com/docker-for-continuous-delivery-advanced-topics-and-patterns-7532235998af">Docker for Continuous Delivery: Advanced Topics and Patterns</a> was originally published in <a href="https://tech.cars.com/">Cars Commerce Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker for Continuous Delivery: The Basics]]></title>
            <link>https://tech.cars.com/docker-for-continuous-delivery-the-basics-6bce7186e872?source=rss-742d49b41197------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bce7186e872</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Jeremy Chambers]]></dc:creator>
            <pubDate>Mon, 05 Jun 2017 16:03:57 GMT</pubDate>
            <atom:updated>2017-09-01T14:29:05.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_NQN6_YnxS29m8vFzWYlEg.png" /><figcaption>Image courtesy of <a href="https://www.docker.com/">Docker.com</a></figcaption></figure><p>Recently at <a href="https://www.cars.com/">Cars.com</a>, we have begun to leverage Docker for building our Node.js middleware applications and front end code. This originally came out of necessity since we had some repositories dependent on version 4 of Node and others on 6, but only 4 installed on the Jenkins agents. Rather than install both versions of Node and toggle them using a node version manager like n or, worse yet, using Jenkins tags to have some agents with one version and some with the other, we decided instead to try using Docker to manage the version of Node. This has been working quite well so far and it has come with many other benefits.</p><p>These benefits along with some Docker basics will be the primary topics of discussion in this post and we will explore some common patterns that arose in our Docker usage in Part II.</p><h3>What is Docker and why use it?</h3><p>Docker is a containerization platform which includes many tools to make the process of building and running containers simple and approachable. Docker makes it much easier to use, but the better question here is “What is a container?”.</p><p>Containers are a lightweight form of virtualization which leverages the kernel of the host and does not have the overhead of running an entire operating system. Typically you would want a container to run a single process and nothing more. You can also bundle the container’s, dependencies, such as binaries and runtimes, inside the container itself. What you end up with is an isolated application which is packaged with it’s dependencies. For many reasons, Docker containers are great for not only running applications, but also for building them.</p><p>A key benefit to using Docker in both the development and the release process is that it begins to bridge the gap between a developer’s workstation and the servers used to verify, build, deploy, or run the application. This is due to the fact that running the various lifecycle tasks of the application in a container means that whether its on a developers machine or on a build server, it will be running against the exact same dependencies in the same operating system. This lends itself well to Continuous delivery since you have a controlled environment in which you can build your applications and the resulting Docker image can serve as the runnable artifact. As a result, you will gain an increased confidence that a change will successfully build and deploy, and it also becomes easier for a developer to troubleshoot why a build has failed on their own machine.</p><p>Another benefit to running builds in containers is that it gives the developer the opportunity to manage the dependencies, runtime/language, and OS with which the application needs to run. Developers will control how the application is built as well, which will reduce the friction between development and operations teams and save everyone a great deal of time. This will allow your organization to adopt new technologies faster without worrying about how to build or deploy them and will make engineers happier for the same reason. No longer will technologies need to be dictated by what operations is comfortable with deploying.</p><blockquote>“No longer will technologies need to be dictated by what operations is comfortable with deploying”</blockquote><p>A feature of containers is that they provide complete isolation from the host system and other containers. This helps with the issue of needing different versions of dependencies in different apps but it also provides a more secure way to run multiple applications on a single host. Even if you have multiple apps which need to bind to the same port, you can leave it to Docker to manage the port mappings. This isolation comes in handy for deployment, but also for things like integration tests which may need to start up a mock server or some associated application at a specified port. In these cases, you do not have to worry about coordinating when the tests are run to avoid port conflicts.</p><p>Hopefully at this point, it is clear why you may want to use Docker in your own pipelines, so now let’s learn a bit more about the tool and how it can be used.</p><h3>Docker basics</h3><p><em>In this section we will run through some examples of using Docker. To install Docker on your machine look </em><a href="https://store.docker.com/search?type=edition&amp;offering=community"><em>here</em></a><em>.</em></p><p>Let us start by running a container and seeing what it has to offer:</p><pre>docker container run -it ubuntu bash</pre><p>In the above command we are running a Docker container in interactive mode (-it) so that we see the outputs from the container and can provide input. The container we are running is going to be based on the ubuntu image (more on this shortly), and the command we want to run to start the container is bash to start a shell. You will find yourself at command prompt in an ubuntu container.</p><p><em>Note: If you are already familiar with docker, you may be accustomed to seeing commands like </em>docker run<em> or </em>docker images<em>. These commands and others still exist, but it is recommended to use the new syntax introduced as of version 1.13. The commands have the new counterparts </em>docker container run<em> and </em>docker image ls<em> respectively.</em></p><p>Now let’s try to run a curl command against an open API:</p><pre>curl wttr.in/Chicago</pre><p>You should see output like this: bash: curl: command not found. That’s strange because I have curl installed on my machine and you most likely do as well. This is because the container is isolated from your host operating system and will only have access to binaries or dependencies within the container. What this means is that we will need to go ahead and install curl in the container:</p><pre>apt-get update -y &amp;&amp; apt-get install -y curl<br>curl wttr.in/Chicago</pre><p>Now when we run the curl command it works! Great, but we shouldn’t have to go into the container and install curl every time. Lucky for us Docker also provides a way to capture container states and allow you to start up new containers from that point. To do this we will use the docker container commit, but first we will need the ID of the container we are running on currently.</p><p>The easiest way to get this is to leave our current session on the container open and get the ID in a new terminal window. Let’s start a new terminal window and run the command docker container ls. You should see an ubuntu container running and the first row in the list will have the ID. Copy the ID and run the following:</p><pre>docker container commit {container id} {your github username}/curl<br>docker image ls</pre><p>After the second command you will see another list that should, at very least, have ubuntu and your new image. Now we can kill our old container running in another terminal session and start a new one from where we left off:</p><pre>docker container run -it {your github username}/curl bash<br>curl wttr.in/Chicago</pre><p>Now would be a good point to distinguish a Docker container from an image. It is important to know the difference and also to use the correct terms when talking about one or the other. An analogy I typically will use to differentiate the two is that an image is to a class as a container is to an instance in your object oriented language of choice. This is a good start, but as you get into more advanced features it is good to know that the two are not mutually exclusive.</p><p>A Docker image was created in a containerized environment, but is made read only upon creation (as we have seen above). Running a container from that image then creates a layer on top of the image which has read and write permissions but still has access to the image filesystem. In fact an image will usually consist of a number of read-only layers stacked on top of each other forming a <a href="https://en.wikipedia.org/wiki/Copy-on-write">copy-on-write</a> filesystem (you can find more details about this in the <a href="https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/">documentation for Docker</a>). We will expand upon this idea shortly, as it will be important in the second part of this post.</p><h3>Manual Configuration? No, Thanks</h3><p>We have seen that we can run containers and even configure them and commit the changes to be used later, but at this point you should be skeptical. Manually configuring a container to commit as an image is not ideal. Doing this leaves room for error and also whenever possible, it is nice to have those configurations committed to version control. As you may have guessed, Docker has our back with this one in the form of Dockerfiles.</p><p>A Dockerfile is a script with its own syntax which declares how to create a Docker image and it will fix all the qualms we have with the process we saw previously. It gives us the reproducibility and configuration as code which we so love in continuous delivery.</p><p>As you may have realized from our previous example, when we run a container we start it from an existing image. We initially started with the base ubuntu image (docker container run -it ubuntu bash) and we will do the same here. Dockerfiles have their own set of commands, or directives, and the first one we will use is FROM which will specify our base image. From there we need to install curl, which we can do using RUN, which will expect to be followed by the command you wish to run as part of the image creation.</p><p>Up until this point we have been running containers by passing in the command we want them to start with, but we can also set a default that will be run in the case where we omit this. I think that our curl command to the weather API would make sense here, so we will add this via the CMD directive. Here is the result:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d916bc59264e67c141b6c4a052ab8ac9/href">https://medium.com/media/d916bc59264e67c141b6c4a052ab8ac9/href</a></iframe><p>Most of this is straightforward, but there are a couple of things I would like to point out before moving on.</p><p>The first is that you may have noticed that I had combined the apt-get update and install commands into a single command. This was intentional because with just the single RUN we are creating just one layer in the resulting image instead of two. Any directive will create a new layer in the container (excluding FROM which references the top layer of a different image).</p><p>The next thing to take a look at would be the CMD section which we had passed an array. This is to avoid any issues in running commands in different types of operating systems and is the best practice with Dockerfiles. Before moving on, be sure to create this Dockerfile on your machine by putting the above code in a file simply called Dockerfile.</p><p>Now we have our Dockerfile which tells Docker how to create our image, so all we have left to do is build the image and run a container from that image. To build you will hop into your terminal, change directories :</p><pre>docker build -t {your github username}/curl .</pre><p>Like before when we committed our container to an image, we pass the name of our image prefixed with our github, or dockerhub, user name by using the -t flag. This is to avoid conflicts with others who may have created another curl image and is best practice when publishing your images to the public image registry.</p><p><em>Note: In the case that you have a private registry, instead of your username, you would replace this with the DNS name of your registry which let’s docker know it should push the image into that private registry as opposed to making it public.</em></p><p>The final argument is called the context of the image. It is the directory from which you want to build the image. In our case we use the current directory, ., and Docker will expect a file called Dockerfile to be in this directory (the location or name of the Dockerfile can be overridden using the -f flag). The context will also serve as the root directory for any references to files from the host, but there will be more to come on this in later posts. Upon running the build command, give your machine a moment to create then let’s run it:</p><pre>docker run -i {your github username}/curl</pre><p>If all goes according to plan, you should see the container start up, run the curl command we set using CMD then return you back to your host’s command prompt.</p><h4>Final Thoughts</h4><p>This post will hopefully serve as a good jumping off point before moving on to my next post which will explore common patterns with Docker that will benefit your CI/CD pipelines. In the following post we will go into great detail about how we use Docker at Cars.com and also review some common design patterns that have arisen from that usage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bce7186e872" width="1" height="1" alt=""><hr><p><a href="https://tech.cars.com/docker-for-continuous-delivery-the-basics-6bce7186e872">Docker for Continuous Delivery: The Basics</a> was originally published in <a href="https://tech.cars.com/">Cars Commerce Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributing Your Home Grown Tools to the Masses with Homebrew]]></title>
            <link>https://tech.cars.com/distributing-your-home-grown-tools-to-the-masses-with-homebrew-acb7a62518a8?source=rss-742d49b41197------2</link>
            <guid isPermaLink="false">https://medium.com/p/acb7a62518a8</guid>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[homebrew]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Jeremy Chambers]]></dc:creator>
            <pubDate>Tue, 27 Dec 2016 17:09:30 GMT</pubDate>
            <atom:updated>2017-04-28T14:37:17.004Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3dL3NZ4jxMwLEXAaQ6sc9g.gif" /></figure><p>Whether it’s a Java executable, a Go binary, or just a collection of shell scripts, you or your company likely have some command line tools which you use to simplify an otherwise complex task like querying Splunk or scaffolding out a new project. So, how then do we distribute these tools with ease to others who may need it? We could document step by step instructions to pull down the code, compile it, and add the result to the user’s path but this is tedious and the installation documents can quickly become outdated as the code for the tool evolves. What we need is a program which can automate the downloading and installation of our tool and <a href="http://brew.sh/">Homebrew</a> does just this.</p><p>Homebrew is a package manager originally created for macOS in lieu of a system level package manager built into the OS, but has since been ported to Linux as well. For macOS, there is no built in package manager and Homebrew is among the most widely used third party package managers. Even for Linux though, the flexibility of the package manager may still be a valuable option for your own tools. Homebrew relies on a git repository storing the “formulas” to install various packages, these formulas are Ruby files that define classes extending Homebrew’s own Formula class. These classes only serve as instructions of how to install that specific package on a user’s machine. Creating your own Homebrew “tap” or repository is as simple as having a git repository containing your own formulas, which makes it remarkably easy to create your own public or private repositories to distribute your tools and a great choice for your distribution platform.</p><p>Over the course of this post we will go over the setup process and benefits to creating your own Homebrew tap. The setup is relatively straight forward, the hard part is having all of the pieces necessary (prereqs are listed below) to get started, but in the end we will have a very simple way to retrieve and install our command line tools. I will discuss both open source and private options to set this up as both are very similar. One thing to note is that this strategy is best for tools which can be self contained with dependencies. There are ways to declare a dependency on another package within Homebrew repositories, but this is beyond the scope of this tutorial. Also, if your tool depends on a specific runtime like NodeJS, it is usually best to use the package manager for that runtime (e.g. npm) as opposed to having a dependency on the runtime.</p><p>What you need to set this up:</p><ul><li>Some sort of artifact repository like artifactory, Amazon S3, or at very least a web server to host the binary files (recommended but optional)</li><li>A git repository</li><li>Some (very basic) knowledge of Ruby</li><li>A mac or linux machine with Homebrew installed</li><li>A command line tool or tools which need to be distributed to many users</li></ul><p>Before we get our hands dirty setting this up, let us first take a look the overall process of getting an executable into the Homebrew tap then we will investigate each piece individually to clarify where it fits into this process. Here are the general steps from coding to pulling down the new package or update:</p><ol><li>Commit code to the tool</li><li>Compile a new artifact and push it up to your artifact repository (optional depending on project)</li><li>Update or create a new formula in your Homebrew tap git repository</li><li>Tap the repository on a client machine and brew install your new package or update your existing installation.</li></ol><p>For the tool itself, you should store it in a git repository. This will provide some additional options not available in other VCSs such as being able to pull the code directly from git, install, and compile it through Homebrew. In certain cases this can be easier than pushing a tar or binary into your artifact repository and also gives you the option to include a dev version of the package through Homebrew which you can use to test the edge version of your code. Even if you aren’t doing that, git allows you to easily track and patch versions of the tool through tags and branches.</p><p>Although you may be able to get away with your project being pulled directly from its git repo, there are many cases where it is ideal for Homebrew to pull down a binary file that can then be installed. This can save time in the installation, since you will not have to compile code and will also reduce the dependencies of the project. For example, if my tool was written in Go, but I didn’t want Go as a dependency of the Homebrew package I could instead compile the tool into binary files for the supported architectures. From there I would push them to my repository and put links to those binaries in my Homebrew formula. Another case may be where your project is a set of scripts that you want to archive together and push up. In any case, all you really need from Homebrew’s perspective is a server that can host these binaries which your users would have access to. Homebrew only needs the URL and can manage the rest.</p><p>Once we have the code for our project pushed up to a git repo and a binary artifact that is hosted, we need to link up Homebrew to make it aware of our new package and how it can be installed. First we will need to create another git repository to keep our Homebrew formulas, then we can create the formula itself. We will see an example of a formula shortly, but in the meantime <a href="https://github.com/Homebrew/brew/blob/master/docs/Formula-Cookbook.md">here</a> is the documentation. Once you create your formula commit the new file in git and push it up to your Homebrew tap’s repository.</p><p>I believe the best way to learn how to leverage a new technology is to use it, so now we will walk through a real working example of setting up a tap and publishing a package to the tap. The tool we will be publishing is a simple script that makes a request using curl to <a href="http://wttr.in/">wttr.in</a> to retrieve the weather for the location provided in the argument to the script. For the following example all of my code will be posted to github where it is publicly available, but I will also speak to how this can be kept in private repositories if the code contains sensitive data or is not appropriate to be distributed outside of your organization.</p><p>First we will need two git repositories, one for our tap and another for the tool we will be publishing. I have created each on my github account: <a href="https://github.com/TheHipbot/homebrew-hb-osx">the tap</a> and <a href="https://github.com/TheHipbot/weather">the weather tool</a>. Notice the git repo name has the prefix ‘homebrew-’, this prefix is required if you wish to use the shorthand of tapping using the pattern ‘github username/repo’ but you can also provide the full git url to the brew tap command.</p><p>The code for the weather tool is simple, we issue a curl command to the endpoint including the location input by the user.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/017754ba71498993e5ffa5107388f9c6/href">https://medium.com/media/017754ba71498993e5ffa5107388f9c6/href</a></iframe><p>To avoid going into a lengthy setup of an artifact repository or host for our binary file, I am going to use github to host the file. To do this, I added an “archive” directory to the github repo and pushed a versioned tarball containing my weather script. We will need the download link in the formula so Homebrew knows where to retrieve our executable.</p><p>We have our code hosted alongside a binary which houses the executable weather script, now we will take a look at the Homebrew formula.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/401a729d99a4bf633dd7e9fdb92744b5/href">https://medium.com/media/401a729d99a4bf633dd7e9fdb92744b5/href</a></iframe><p>Let us review the formula. It is a Ruby class extending the Formula class provided through Homebrew. We provide a description, a homepage for the project, a version, and the link to download the tarball. The file in our repo, “weather.rb”, is going to also be the name of the package (minus the rb file extension) when we eventually run our ‘brew install’ command and the Ruby class should have the same name although the name of the class will not affect the ability to install. To verify that the binary is truly the expected file when it is downloaded, you must add a SHA256 hash which can be copied to your clipboard by running the following command on macOS against your tar or binary:</p><pre>shasum -a 256 name-of-your-file | awk &#39;{printf $1}&#39; | pbcopy</pre><p>Then put the hash into the formula with the ‘sha256’ attribute.</p><p>Since our script uses the curl command I have added this as a dependency using the ‘depends_on’ attribute. This will cause curl to be installed through Homebrew if it is not already. Bottling is a feature in Homebrew used to store compiled artifacts, since our code does not need to be compiled we add the line ‘bottle :unneeded’. In the last bit an install method is defined. In it we add the line ‘bin.install “weather”’, bin refers to the binary folder Homebrew has symlinked to the PATH on your system and weather refers to our script which Homebrew then knows to pull out of the tarball.</p><p>With all of the pieces in place, the only thing left to do is to tap your Homebrew repo and brew install your package. This is done by first calling the ‘brew tap’ command. This can take either the pattern I had mentioned earlier, ‘username/repo’ which will assume that is the github username and repo or you can optionally provide a third argument which is a git url. In the latter case the ‘username/repo’ serves as a name for the tap and could be anything, the URL doesn’t have to be an HTTP URL but any which your local machine would have available via git. This would be the method to use if you are using a git server other than github.</p><p>With a working Homebrew tap of your own, creating new packages for friends or colleague to install with a single command becomes quite simple. By greatly simplifying the installation process it will make troubleshooting an installation much simpler through automating the process and can even increase your user base. This also can be a great way to clean up some of your dotfiles by pulling scripts out into Formulas or to publish your open source project. From there you may even want to publish your package to Homebrew’s core package library by submitting a pull request with your Formula to the <a href="https://github.com/Homebrew/homebrew-core">Homebrew core tap</a>.</p><p>Thanks for taking the time to read this guide and please feel free to leave comments for improvements or share it if you found it useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acb7a62518a8" width="1" height="1" alt=""><hr><p><a href="https://tech.cars.com/distributing-your-home-grown-tools-to-the-masses-with-homebrew-acb7a62518a8">Distributing Your Home Grown Tools to the Masses with Homebrew</a> was originally published in <a href="https://tech.cars.com/">Cars Commerce Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>