<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ju5t on Medium]]></title>
        <description><![CDATA[Stories by ju5t on Medium]]></description>
        <link>https://medium.com/@ju5t_?source=rss-cb0055be2cb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oE9-94slTVRO-iHmvsOYrA.jpeg</url>
            <title>Stories by ju5t on Medium</title>
            <link>https://medium.com/@ju5t_?source=rss-cb0055be2cb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:09:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ju5t_/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes, GitLab and DigitalOcean]]></title>
            <link>https://medium.com/@ju5t_/kubernetes-gitlab-and-digitalocean-d73a3dc14d11?source=rss-cb0055be2cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d73a3dc14d11</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[digitalocean]]></category>
            <dc:creator><![CDATA[ju5t]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 13:05:57 GMT</pubDate>
            <atom:updated>2019-11-08T07:02:48.233Z</atom:updated>
            <content:encoded><![CDATA[<h3>How we are using Kubernetes, GitLab and DigitalOcean to deploy our production applications</h3><p>Using Kubernetes, GitLab and DigitalOcean can be a challenge. Even when you’re familiar with all three. This article explains how to integrate DigitalOcean’s Kubernetes with GitLab and how to deploy to it.</p><p>Let’s get started.</p><blockquote>Versions tend to change over time. Although they are up to date at the time of writing, make sure to check for updates when you’re reading this later on.</blockquote><h3>Prerequisites</h3><p>Before you can link your cluster to GitLab, you need to have a DigitalOcean account and access to GitLab. It doesn’t matter what version of GitLab you <br>are using. It can be your own installation or <a href="https://gitlab.com/">gitlab.com</a>.</p><h4>Kubernetes</h4><p>Getting Kubernetes on DigitalOcean is fairly straight forward. You can add a new cluster in the <a href="https://cloud.digitalocean.com/kubernetes/clusters">cloud interface</a>. Click on Create and head for Clusters. The wizard will guide you through the process. If you plan to use <a href="https://docs.gitlab.com/ee/user/project/clusters/serverless/">Knative</a>, make sure your cluster has at least <a href="https://docs.gitlab.com/ee/user/project/clusters/serverless/#installing-knative-via-gitlabs-kubernetes-integration">3 nodes, 6 vCPUs and 22.5GB memory</a>.</p><h4>Install doctl</h4><p>doctl is the command line interface to interact with DigitalOcean, similar to cli-tools from other cloud providers. <a href="https://github.com/digitalocean/doctl#installing-doctl">The installation instructions are available on GitHub</a>. You need this to get the kubectl configuration for Kubernetes.</p><p>When doctl is installed you can authenticate with DigitalOcean.</p><ol><li><a href="https://cloud.digitalocean.com/account/api/tokens">Create an API-token</a>.</li><li>Run doctl auth init. It will ask you for the token you’ve created.</li></ol><p>You now have access to your project at DigitalOcean.</p><h4>Install kubectl</h4><p>kubectl is used to interact with Kubernetes. You can find the installation instructions for all operating systems on the <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">kubernetes.io website</a>.</p><p>When you’ve got kubectl installed, you can use doctl to get and save the Kubernetes configuration from DigitalOcean.</p><pre>doctl kubernetes cluster kubeconfig save &lt;cluster&gt;</pre><p>DigitalOcean rotates all cluster certificates every 7 days. And although this is a good thing, it means you will have to update your configuration occasionally too.</p><h3>Add Kubernetes to GitLab</h3><p>Credits to <a href="https://medium.com/u/52ff72d48c18">Stepan Kuzmin</a> for his <a href="https://gist.github.com/stepankuzmin/796ecff801227773e920f1468e7a1ae6">gist</a> and GitLab for the <a href="https://www.youtube.com/watch?v=7ycuOLa5mtM">video</a>, most of which you can find below.</p><h4>Add an existing Kubernetes cluster</h4><p>When you’re adding a Kubernetes cluster to your project or installation, add an existing one. The steps below will help you get an API URL, CA Certificate and Service Token.</p><p>Create a service account:</p><pre>kubectl create -f - &lt;&lt;EOF<br>  apiVersion: v1<br>  kind: ServiceAccount<br>  metadata:<br>    name: gitlab<br>    namespace: default<br>EOF</pre><p>Create a cluster role giving the gitlab-account cluster-admin privileges.</p><pre>kubectl create -f - &lt;&lt;EOF<br>kind: ClusterRoleBinding<br>apiVersion: rbac.authorization.k8s.io/v1<br>metadata:<br>  name: gitlab-cluster-admin<br>subjects:<br>- kind: ServiceAccount<br>  name: gitlab<br>  namespace: default<br>roleRef:<br>  kind: ClusterRole<br>  name: cluster-admin<br>  apiGroup: rbac.authorization.k8s.io<br>EOF</pre><p>Get the API URL.</p><pre>kubectl cluster-info | grep &#39;Kubernetes master&#39; | awk &#39;/http/ {print $NF}&#39;</pre><p>Get the CA Certificate.</p><pre>kubectl get secret $(kubectl get secrets | awk &#39;/gitlab/ {print $1}&#39;) -o jsonpath=&quot;{[&#39;data&#39;][&#39;ca\.crt&#39;]}&quot; | base64 -D</pre><p>Get the Service Token.</p><pre>kubectl get secret $(kubectl get secrets | awk &#39;/gitlab/ {print $1}&#39;) -o jsonpath=&quot;{[&#39;data&#39;][&#39;token&#39;]}&quot; | base64 -D</pre><p>Make sure you keep RBAC-enabled cluster and GitLab-managed enabled.</p><blockquote>Note: if you keep GitLab-managed enabled GitLab will expose your Kubernetes configuration to deployments with the environment set. Your jobs will depend on it, so leave this enabled.</blockquote><h4>Install helper applications</h4><p>As soon as the cluster is added you will be able to add a number of helper applications that GitLab provides.</p><ol><li>Install Helm Tiller</li><li>Install Ingress*</li><li>Install Cert-Manager</li><li>Install Prometheus</li></ol><p>If you want to run your CI jobs on Kubernetes you can install the runner. I won’t cover this, as there are a few caveats when you do, Docker in Docker being one of them, but it’s still pretty cool.</p><blockquote>* When you add the Ingress application a <a href="https://www.digitalocean.com/pricing/#Tools">load balancer</a> is added to your DigitalOcean project, you will be billed for it.</blockquote><p>Once the helper applications are installed you should see the cluster health in your dashboard. This means Prometheus is running and things should be in good health.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_USMkZzusXlXDsZx.png" /><figcaption>Cluster Health (source: gitlab.com)</figcaption></figure><h3>Setting up your app in GitLab</h3><p>It’s time to deploy our application. Let’s start with a basic .gitlab-ci.yml.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a48e35ee9e314f2e29e4593b310c447/href">https://medium.com/media/0a48e35ee9e314f2e29e4593b310c447/href</a></iframe><p>In this deployment we’re using a few predefined variables.</p><p>$KUBE_NAMESPACE — is the namespace we get from GitLab.<br>$GITLAB_USER_EMAIL — is your e-mail address.<br>$CI_DEPLOY_USER — is the user name of the read only deploy token.<br>$CI_DEPLOY_PASSWORD — is the key of the deploy token.</p><p>These last two variables need some further explanation.</p><h4>Deploy tokens</h4><p>In most environments your app sits in a private registry. GitLab creates tokens that can be used for temporary access. Although Kubernetes could use these, it would break whenever you start scaling your nodes. It wouldn’t be able to use the expired token and Kubernetes will not be able to launch new pods.</p><p>This is why we’re using <a href="https://gitlab.com/help/user/project/deploy_tokens/index#deploy-tokens">Deploy Tokens</a>. A deploy token is persistent. To use them, you need to create a secret in Kubernetes.</p><ol><li>Create a read only <a href="https://gitlab.com/help/user/project/deploy_tokens/index#deploy-tokens">Deploy Token</a>, use gitlab-deploy-token as its name. It only needs access to read the registry. The username is irrelevant.</li><li>If you have named the token gitlab-deploy-token, your CI job will find the username and token <a href="https://docs.gitlab.com/ee/user/project/deploy_tokens/index.html#gitlab-deploy-token">automatically</a>.</li></ol><p>Our before_script creates a secret for you with the name regcred. In most cases there is no need to change this name as we’re deploying each app in its own namespace, but feel free to do. You will need it in your deployment. It should be added to the spec of your deployment template like so.</p><pre>...      <br>imagePullSecrets:<br>- name: regcred<br>...</pre><h4>Preparing for deployment</h4><p>Once the deploy token is configured and the actual job is kicked off, there are a few things happening that need an explanation. You may want to keep a copy of the <a href="https://gist.github.com/ju5t/e8a8bf5dd241ba824fbcfcde5dc6dc82">gist</a> open while reading the below.</p><p>The kubectl commands are different from what you would expect if you have worked with Kubernetes before. We’re using --dry-run=true to get the YAML output and apply this instead.</p><pre>... --dry-run=true -o yaml | kubectl apply -f -</pre><p>You may wonder why. Running kubectl create would have created the same resource, right? Although strictly true, Kubernetes will throw errors when you try to create that same resource. This is why you need apply instead of create.</p><h4>Deploying your app</h4><p>The script to deploy your application is pretty straight forward:</p><pre>envsubst &lt; k8s.yml | kubectl apply -f -</pre><p>We have prefixed it in the gist with a VERSION variable as we use this to label our objects. It helps to quickly track what version a deployment is running.</p><p>More on envsubst below.</p><h3>Your deployment</h3><p>I won’t go into the details of setting up a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment">deployment</a>, <a href="https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service">service</a> or <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource">ingress</a>. It’s up to you how you configure your deployments. There are some things that may help you get up to speed faster though. I will cover these below.</p><h4>Variables</h4><p>envsubst replaces environment variables in standard input. This means we can create YAML-files like this:</p><pre>...<br>  labels:<br>    app: your-app<br>    commit: ${CI_COMMIT_SHORT_SHA}<br>    version: ${VERSION}<br>...</pre><p>And pass them to envsubst with envsubst &lt; k8s.yml. This will be copied to standard output and can then be used to create your resources.</p><p>We’re using a single file for all environments. We’re keeping it <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>.</p><h4>Ingress load balancing</h4><p>Remember the applications you have deployed?</p><p>It includes an ingress controller that is linked to a load balancer. To use it, you have to annotate the ingress object you’re creating.</p><pre>...<br>metadata:<br>  name: ingress-your-app<br>  annotations:<br>    kubernetes.io/ingress.class: &quot;nginx&quot;<br>...</pre><p>This will route traffic from the ingress controller to your ingress object, to your service and your pods.</p><p>Other useful annotations can be found <a href="https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md">here</a>.</p><h4>SSL and Let’s Encrypt</h4><p>Remember the Cert-Manager too?</p><p>It can automatically assign and renew certificates from Let’s Encrypt. To use it, you have to annotate the ingress object and add tls to your spec.</p><pre>...<br>metadata:<br>  name: ingress-your-app<br>  annotations:<br>    ...<br>    certmanager.k8s.io/cluster-issuer: letsencrypt-prod<br>    ...<br>...<br>spec:<br>  tls:<br>  - hosts:<br>    - test.yourapp.com<br>    secretName: test-yourapp-com</pre><p>It can take a while for your certificate to be available. Be patient and make sure your DNS has been set — use a wildcard if you’re using review apps in GitLab to make things easier.</p><p>Other useful annotations can be found <a href="https://github.com/jetstack/cert-manager/blob/master/docs/tasks/issuing-certificates/ingress-shim.rst#supported-annotations">here</a>.</p><h4>Monitoring</h4><p>Last but not least, Prometheus.</p><p>Make sure your deployment, service and ingress are prefixed with the environment slug like so:</p><pre>...<br>metadata:<br>  name: ${CI_ENVIRONMENT_SLUG}-your-app<br>...</pre><p>This makes sure GitLab knows where to pick up data from Prometheus.</p><p>If you’re on the silver or equivalent plans and up and you would like to use the deploy boards feature, make sure your annotations are set up on the deployment itself AND the spec template:</p><pre>...<br>metadata:<br>...<br>  annotations:<br>    app.gitlab.com/env: $CI_ENVIRONMENT_SLUG<br>    app.gitlab.com/app: $CI_PROJECT_PATH_SLUG<br>...</pre><h3>The ‘magic’ namespace</h3><p>If you’re a little familiar with Kubernetes you might have noticed .gitlab-ci.yml has not been configured to create a namespace. Its creation is hidden from your CI job.</p><p>On the first deployment of your app, GitLab will create the $KUBE_NAMESPACE, service accounts and access tokens. Subsequent jobs will use these credentials to deploy your application.</p><p>There’s a downside to this.</p><p>If you decide to delete this namespace, <a href="https://gitlab.com/gitlab-org/gitlab-ce/issues/62032">your jobs will fail</a>. GitLab stores the namespace in a database and once it’s there, it is assumed that it is never deleted. There is no reality check — and that’s a mistake.</p><p>In case you do decide to delete the namespace and want to start over, you need to remove the entry from GitLab’s database.</p><pre># gitlab-rails db<br>gitlabhq_production=&gt; select * from clusters_kubernetes_namespaces;<br>gitlabhq_production=&gt; delete from clusters_kubernetes_namespaces where namespace = &#39;&lt;namespace&gt;&#39;;</pre><h3>Wrapping up: add the Kubernetes dashboard</h3><p>The Kubernetes dashboard is a web-based user interface. It makes it easier to troubleshoot your application(s). You can install the dashboard in the kube-system namespace with the following command:</p><pre>kubectl apply -f <a href="https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml">https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml</a></pre><p>Read more on how to connect to it on the <a href="https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#accessing-the-dashboard-ui">Web UI documentation pages</a>.</p><blockquote>You can reuse the token you’ve created for GitLab to login or create a new token, this is up to you.</blockquote><h3>Closing words</h3><p>Thanks for reading this. I hope this article was useful and helps you get the most out of your <a href="https://about.gitlab.com/">GitLab</a> installation together with <a href="https://m.do.co/c/7249ed50a4c2">DigitalOcean</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d73a3dc14d11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replacing roles for regular expressions in Puppet]]></title>
            <link>https://medium.com/@ju5t_/replacing-roles-for-regular-expressions-in-puppet-560fc71e6811?source=rss-cb0055be2cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/560fc71e6811</guid>
            <category><![CDATA[puppet]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[ju5t]]></dc:creator>
            <pubDate>Sun, 10 Feb 2019 11:11:39 GMT</pubDate>
            <atom:updated>2019-02-10T11:14:16.777Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ve been using <a href="https://puppet.com/">Puppet</a> for the better half of a decade now. Puppet plays a vital role in managing our infrastructure. Every server we run is managed with Puppet. Beginning with Puppet wasn’t easy. Puppet has a pretty intense learning curve, especially when you’re not used to <a href="https://en.wikipedia.org/wiki/Declarative_programming">declarative development</a>.</p><p>When you start learning a new language, whether it’s declarative like Puppet or imperative like Python, there is a lot of trial and error involved. Code is added organically and can grow fast— which often means you will be introducing a lot of technical debt.</p><p>And we were no exception.</p><p>When we started we didn’t really think about structure. We wrote unique code for most servers and repeated ourselves constantly. This lasted until 2014, when we started adding structure through roles and profiles.</p><p><a href="https://puppet.com/docs/pe/2018.1/designing_system_configs_roles_and_profiles.html">Roles and profiles</a> were first introduced in <a href="https://www.craigdunn.org/2012/05/239/">an article by Craig Dunn</a>. Puppet needed a higher level of abstraction. At the time, Puppet didn’t have a lot of best practices on this particular subject and roles and profiles were the most popular solution. When we began implementing them we followed the letter of the law. We introduced new roles in our environment, added profiles to them and assigned these roles to our nodes. These were what Puppet now calls, granular roles.</p><p>But it never felt right. It felt like we were adding roles for the sake of having them, they didn’t have much purpose.</p><h3>Implementing granular roles</h3><p><a href="https://puppet.com/docs/pe/2019.0/designing_convenient_roles.html#first-approach-granular-roles">Granular roles</a> means that each type of node gets its own role. Our roles were looking like the one below:</p><pre><strong>class role::jenkins::master</strong> {<br>  include profile::base<br>  include profile::server<br>  include profile::jenkins::master<br>}</pre><p>Roles are meant to be simple and clean. Typically, you would only assign a <a href="https://github.com/puppetlabs/best-practices/blob/master/puppet-code-abstraction-roles.md">single role to a server</a>.</p><pre><strong>node &#39;master1.jenkins.example.com&#39;</strong> {<br>  include ::role::jenkins::master<br>}</pre><p>Let’s illustrate what happens in an environment with roles and profiles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*SoarNBEJ3favQEFcux5Edw.png" /></figure><p>As you can see it adds another layer of abstraction. You can re-use the role for other servers and you never have to repeat yourself. And for some there will be benefits to that. In our case, not so much.</p><p>Our environment has lots of different servers. They don’t necessarily have things in common. Sometimes they do, but in most cases we can never re-use an existing role. We suffered from role-bloat, a lot.</p><p>We needed something else. Luckily, our server names are boring (read: predictable), which means we can take another approach.</p><h3>Roles and regular expressions in node definitions</h3><p>If we would use Jenkins (we don’t, we use GitLab) our masters would be given names such as master1 and master2. So we don’t need specific node definitions as we can use regular expressions.</p><p>We can match our servers with the following snippet:</p><pre><strong>node /^master\d+\.jenkins\.example\.com$/</strong> {<br>  include ::role::jenkins::master<br>}</pre><p>This is why we decided to cut out the explicit role definition and specify the profiles directly in the node itself.</p><pre><strong>node /^master\d+\.jenkins\.example\.com$/</strong> {<br>  include profile::base<br>  include profile::server<br>  include profile::jenkins::master<br>}</pre><p>Which results in the following flowchart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*yVHGFirKMKd9ACkfMf-OVg.png" /></figure><p>This approach is also explained in <a href="https://puppet.com/docs/pe/2019.0/designing_convenient_roles.html#sixth-approach-building-roles-in-the-node-classifier">the sixth approach of designing convenient roles</a> article in the Puppet documentation. We’ve only added regular expressions into the mix.</p><p>We didn’t do a lot, really. We removed the explicit role definition and moved it into the node classifier. We think it has made our Puppet code cleaner and easier to read, but it works exactly as it did before.</p><p>With this solution we can easily find which roles have been added to a server, and make changes, whenever needed. And although it may not be a solution for everyone, it made managing our infrastructure a lot faster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=560fc71e6811" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we’re using GitLab to test and deploy our application(s) to Docker Swarm]]></title>
            <link>https://medium.com/@ju5t_/how-were-using-gitlab-to-test-and-deploy-our-application-s-to-docker-swarm-e3f556dbf8fe?source=rss-cb0055be2cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3f556dbf8fe</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[ju5t]]></dc:creator>
            <pubDate>Sun, 03 Feb 2019 09:11:29 GMT</pubDate>
            <atom:updated>2019-02-10T11:14:54.541Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article we would like to show you how we often implement and use GitLab CI/CD to test and deploy our application(s) automatically.</p><p>We are not building spaceships and our development process is pretty straight forward. We use git so we often start with a new branch. We write and run tests in this branch and once it is ready it gets merged into develop. When we feel we’re ready for a release, develop is merged into master and tagged. You get the idea.</p><p>This isn’t rocket science.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*Xv5pmPjblrdicjZeEqmAkQ.png" /></figure><p><em>GitLab uses YAML to define a CI/CD pipeline. We have included examples of actual configurations. If some of the keywords used in these examples are unfamiliar to you, have a look the </em><a href="https://docs.gitlab.com/ee/ci/yaml/"><em>GitLab Documentation</em></a><em>.</em></p><h3>Where do we start</h3><p>Most applications will have some sort of unit test already. It depends on the programming language you use, but the first step of any pipeline often is to run this test. For a Python application you could do something like:</p><pre>.unit: &amp;unit<br>  script:<br>    - tox<br>  before_script:<br>    - python -V<br>    - pip install tox<br>  coverage: &#39;/TOTAL.+ ([0-9]{1,3}%)/&#39;</pre><pre>test-python27:<br>  &lt;&lt;: *unit<br>  image: python:2.7<br>  stage: test</pre><pre>test-python36:<br>  &lt;&lt;: *unit<br>  image: python:3.6<br>  stage: test</pre><p>The actual test is created in .unit. &lt;&lt;: merges what is defined in .unit: &amp;unit into test-python27. This is called an <a href="https://learnxinyminutes.com/docs/yaml/">anchor</a>. It allows you to re-use the same block and it keeps your YAML <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>.</p><p>Not a lot happens here. We display the Python version and install <a href="https://tox.readthedocs.io/en/latest/">tox</a>. When this is done we run the tox — command to test our code. That’s it. If this test would fail for whatever reason, the pipeline fails too.</p><p>But we can do more.</p><h3>Hello, Docker</h3><p>We’re using <a href="https://docs.docker.com/get-started/">Docker</a> to run our applications. We build an image for all our applications and we use this image to start containers. The image includes all dependencies, no matter where it runs — it will always work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*gggb_4Nf0HWdm4iVgtypDQ.png" /></figure><p>Building Docker images is easy once you’ve got a Dockerfile. We assume you know how to make them. If not, <a href="https://docs.docker.com/get-started/part2/">Docker tutorials are helpful</a>.</p><p>You have probably used these commands more than once if you’re working with Docker to build an image:</p><pre>docker login registry.gitlab.com<br>docker build -t registry.gitlab.com/user/hello-world:latest .<br>docker push registry.gitlab.com/user/hello-world:latest</pre><p>This isn’t difficult. But what if you’re working in a new feature branch? You probably want to tag it with feature-x instead of latest. And develop should have its own image, too.</p><p>You can see where this is going. It’s boring and tedious and it’s easy to make a mistake. <a href="https://docs.gitlab.com/ee/ci/">You need some automation</a>. The following snippet is what we use to build new Docker images with GitLab. It will tag and push them based on what you’re working on.</p><pre># Build a branch<br>build-branch:<br>  image: docker:latest<br>  stage: build<br>  except:<br>    - develop<br>    - tags<br>    - master<br>  script:<br>    - docker build -t $CONTAINER .<br>    - docker tag $CONTAINER $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG<br>    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG</pre><pre># Build an image for our development branch<br>build-develop:<br>  image: docker:latest<br>  stage: build<br>  only:<br>    - develop<br>  script:<br>    - docker build -t $CONTAINER .<br>    - docker push $CONTAINER</pre><pre># Build a release<br>build-release:<br>  image: docker:latest<br>  stage: build<br>  only:<br>    - tags<br>  script:<br>    - docker build -t $CONTAINER .<br>    - docker tag $CONTAINER $CI_REGISTRY_IMAGE:latest<br>    - docker tag $CONTAINER $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG<br>    - docker push $CI_REGISTRY_IMAGE:latest<br>    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG</pre><p>A full list of CI variables: <a href="https://docs.gitlab.com/ee/ci/variables/">https://docs.gitlab.com/ee/ci/variables/</a>.</p><p>$CONTAINER isn’t available by default. You need to add it yourself. We use variables a lot. We can often re-use blocks and this allows us to keep our code reasonably clean.</p><pre>variables:<br>  CONTAINER: $CI_REGISTRY_IMAGE:develop<br>  DOCKER_DRIVER: overlay<br>  GIT_SUBMODULE_STRATEGY: recursive</pre><p>This is useful but we’re not there yet, as this only publishes our image to our private Docker registry.</p><p><em>You may have noticed that we’re not building an image when changes are committed to the master branch. In fact, we generally don’t run any jobs at all on this branch. Testing the master branch is redundant when you’re tagging releases — and code often has been tested in your develop branch already.</em></p><h3>Knock, knock, who’s there?</h3><p>Once we have a Docker image we can test if it runs.</p><pre>docker run --rm -d --name hello-world $CONTAINER<br>docker logs hello-world</pre><p>In our CI pipeline we use <a href="https://docs.docker.com/compose/">Docker Compose</a> to test the entire stack. Does it run and more importantly, does it do what we expect? For this we often need to start other services too and Docker Compose makes this easy.</p><p><em>In addition, the </em><em>docker-compose.yml file can be re-used when developing new features. We use it to manage services such as Redis or RabbitMQ on a development machine so we can test locally, too.</em></p><p>This is an example of how you could do run these tests:</p><pre>.integration: &amp;integration |<br>  docker build -t $CONTAINER .<br>  docker-compose up -d<br>  echo &quot;Waiting on the environment to come online&quot;<br>  sleep 60<br>  docker logs hello-world || true<br>  CURL=&quot;docker run --rm --net host appropriate/curl -s&quot;<br>  $CURL localhost:8081 | grep -i &#39;text&#39;<br>  docker-compose down</pre><p>Note that this is just a re-usable block. It won’t run anything by itself.</p><p>You need to adjust the script in your build stage. We remove docker build and replace it with our *integration block. We don’t have to build Docker images twice as $CONTAINER has been built in *integration already.</p><pre># Build an image for our development branch<br>build-develop:<br>  image: docker:latest<br>  stage: build<br>  only:<br>    - develop<br>  script:<br>    - *integration<br>    - docker push $CONTAINER</pre><p>We run these tests before we push a Docker image to the registry.</p><p>Now that we have an image we can go live, right? Well, almost. Before we deploy our containers we test one more thing.</p><h3>What are we hiding our containers?</h3><p>One of the challenges with Docker is updates. If you specify an old version of Ubuntu or Alpine in FROM, you may be running a vulnerable container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*ixexGC3GRKRjRizCv5CPHQ.png" /></figure><p><a href="https://github.com/coreos/clair">Clair</a> is an open source tool that you can use to scan your containers for vulnerabilities.</p><p>This is almost an exact copy of what you can find in the <a href="https://docs.gitlab.com/ee/ci/examples/container_scanning.html">GitLab examples</a>.</p><pre>.scan: &amp;scan<br>  image: docker:stable<br>  stage: scan<br>  variables:<br>    DOCKER_DRIVER: overlay2<br>  allow_failure: true<br>  before_script:<br>    - echo &#39;Skipping integration and build process&#39;<br>  services:<br>    - docker:stable-dind<br>  script:<br>    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY<br>    - docker run -d --name db arminc/clair-db:latest<br>    - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6<br>    - apk add -U wget ca-certificates<br>    - docker pull $RELEASE<br>    - wget <a href="https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64">https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64</a><br>    - mv clair-scanner_linux_amd64 clair-scanner<br>    - chmod +x clair-scanner<br>    - touch clair-whitelist.yml<br>    - while( ! wget -q -O /dev/null <a href="http://docker:6060/v1/namespaces">http://docker:6060/v1/namespaces</a> ) ; do sleep 1 ; done<br>    - retries=0<br>    - echo &quot;Waiting for clair daemon to start&quot;<br>    - while( ! wget -T 10 -q -O /dev/null <a href="http://docker:6060/v1/namespaces">http://docker:6060/v1/namespaces</a> ) ; do sleep 1 ; echo -n &quot;.&quot; ; if [ $retries -eq 10 ] ; then echo &quot; Timeout, aborting.&quot; ; exit 1 ; fi ; retries=$(($retries+1)) ; done<br>    - ./clair-scanner -c <a href="http://docker:6060/">http://docker:6060</a> --ip $(hostname -i) -r gl-container-scanning-report.json -t &quot;High&quot; -l clair.log -w clair-whitelist.yml $RELEASE || true<br>  artifacts:<br>    reports:<br>      container_scanning: gl-container-scanning-report.json</pre><p>We’re using $RELEASE to define the image that we want to scan. We use it for clarity but we use this variable elsewhere too.</p><p><em>If you want to fail your pipeline when unapproved vulnerabilities are found, remove </em><em>|| true from the end of the line that starts with </em><em>./clair-scanner.</em></p><p>To use this you can do something like:</p><pre>scan-image:<br>  &lt;&lt;: *scan<br>  stage: scan<br>  variables:<br>    RELEASE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG<br>  except:<br>    - master</pre><p>This will run Clair on all branches except for master.</p><p><em>We run this in a separate stage. But you don’t have to. </em><em>*scan is a block that you could run in other stages too. For example, you could run this before it is pushed to your registry by adding it to the </em><em>build stage.</em></p><p>Unless you’ve got the right GitLab license this won’t show any results in your merge requests. You may need to read the reports in the CI output or upgrade to GitLab Ultimate or Gold (which to be fair, doesn’t come cheap). Either way it can be a useful addition to your test suite.</p><p>So, now that we ran all of our tests we can deploy our application.</p><h3>Getting your application to Docker Swarm</h3><p>Like building containers you don’t want to waste time deploying applications either. And there is no reason to either with GitLab CI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*lVG08ed2UZu1iR2Oljxkaw.png" /></figure><p>Almost all of our applications include a block similar to the one below. This goes in .gitlab-ci.yml.</p><pre>.deployment: &amp;deployment |<br>  docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY<br>  export DOCKER_TLS_VERIFY=&quot;1&quot;<br>  export DOCKER_HOST=&quot;$DOCKER_SWARM_MASTER&quot;<br>  export DOCKER_CERT_PATH=&quot;certs&quot;<br>  mkdir $DOCKER_CERT_PATH<br>  echo &quot;$DOCKER_MACHINE_CA&quot; &gt; $DOCKER_CERT_PATH/ca.pem<br>  echo &quot;$DOCKER_MACHINE_CLIENT_CERT&quot; &gt; $DOCKER_CERT_PATH/cert.pem<br>  echo &quot;$DOCKER_MACHINE_CLIENT_KEY&quot; &gt; $DOCKER_CERT_PATH/key.pem<br>  docker stack deploy --prune -c ci/deploy.yml -c ci/$APP_ENVIRONMENT.yml --with-registry-auth $DEPLOYMENT<br>  rm -rf $DOCKER_CERT_PATH</pre><p>We’re using a lot of variables. You can use *deployment to deploy production, acceptance or to deploy a <a href="https://docs.gitlab.com/ee/ci/review_apps/">review app</a>. It even allows you to deploy to different clusters with $DOCKER_SWARM_MASTER.</p><p>For deployments to work you need to set $DOCKER_SWARM_MASTER. This has to be the full address, e.g. tcp://10.19.1.100:2376. The IP addresses of your CI workers should be allowed access in your firewall (or run in the same network, obviously).</p><p>Connections to Docker Swarm are secured with certificates. We don’t store these in our deployment script. We store them in <a href="https://docs.gitlab.com/ee/ci/variables/#variables">environment variables</a> in each project in our CI/CD settings.</p><p>If you’re using <a href="https://docs.docker.com/machine/">Docker Machine</a> to configure your Docker nodes you can get the certificates from your deployment server:</p><ul><li>DOCKER_MACHINE_CA — ~/.docker/machine/certs/ca.pem</li><li>DOCKER_MACHINE_CLIENT_CERT — ~/.docker/machine/certs/cert.pem</li><li>DOCKER_MACHINE_CLIENT_KEY — ~/.docker/machine/certs/key.pem</li></ul><p>Finally, we’re using stacks to deploy our application. <a href="https://docs.docker.com/v17.12/get-started/part5/">Docker Stacks</a> have a similar configuration as Docker Compose. Below is an example of a stack that you could deploy. We’re using <a href="https://docs.docker.com/compose/environment-variables/#substitute-environment-variables-in-compose-files">variables</a> so we can re-use this for different deployments.</p><p>ci/deploy.yml:</p><pre>version: &#39;3.6&#39;</pre><pre>services:<br>  hello-world:<br>    image: ${RELEASE}<br>    deploy:<br>      update_config:<br>        delay: 60s<br>        monitor: 30s<br>        order: start-first<br>        failure_action: rollback</pre><pre>networks:<br>  default:<br>    external: true<br>    name: ${DOCKER_NETWORK:-test}</pre><p>To make sure that we can override some settings in a review, acceptance or production deployment we use a second configuration file in our deployment step by adding a second-c flag: -c ci/$APP_ENVIRONMENT.yml.</p><p><em>You need to make sure the </em><em>$APP_ENVIRONMENT.yml file exists or it will break your deployment. It cannot be empty — it always needs </em><em>version: &#39;3.6&#39; at the top of the file.</em></p><p><em>We assume that the networks in </em><em>deploy.yml exist. This is often the case — if not you have to create them manually or change </em><em>deploy.yml to create them for you.</em></p><p>And finally you want to deploy the application to production.</p><pre>deploy-production:<br>  image: docker:latest<br>  stage: deploy<br>  variables:<br>    URL: myawesomeapp.com<br>    RELEASE: $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG<br>    DEPLOYMENT: ${CI_PROJECT_NAME}_prod<br>    APP_ENVIRONMENT: production<br>    DOCKER_NETWORK: prod<br>  environment:<br>    name: production<br>    url: <a href="https://$url/">https://$URL/</a><br>  only:<br>    - tags<br>  script:<br>    - *deployment</pre><p>Where:</p><ul><li>$URL is the final location on which the environment will be accessible.</li><li>$RELEASE is the location of the image. Use CI variables if you can.</li><li>$DEPLOYMENT is the name of the project. This is assumed to be unique.</li><li>$APP_ENVIRONMENT is used to find the deployment YAML file.</li><li>$DOCKER_NETWORK is the network we’re binding on.</li></ul><p><em>If you’re using review apps or environments with dynamic urls you need to deploy a load balancer like </em><a href="https://traefik.io/"><em>Traefik</em></a><em> and label your service in </em><em>deploy.yml. Otherwise your URLs are probably not accessible.</em></p><p>When the deployment is finished you will have a new stack running in your Docker Swarm cluster. The name will be ${CI_PROJECT_NAME}_prod and it should be accessible via $URL.</p><p><em>If you don’t feel comfortable deploying your production environment without any intervention you can use </em><a href="https://docs.gitlab.com/ee/ci/environments.html#manually-deploying-to-environments"><em>when: manual</em></a><em> in your deployment stage. This will expose a play button in Gitlab’s UI that triggers the deployment once you click it.</em></p><h3>Build, deploy, and repeat</h3><p>See, this wasn’t rocket science. But if you’re starting from scratch, pipelines in GitLab can be overwhelming. And sometimes it does feel like you’re building a spaceship. In the end GitLab makes it easy to test and deploy applications, you just need some rocket fuel every now and then.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3f556dbf8fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we build our Magento 2 environment using Docker and continuous integration]]></title>
            <link>https://medium.com/@ju5t_/how-we-build-our-magento-2-environment-using-docker-and-continuous-integration-8a6177b8aa5b?source=rss-cb0055be2cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a6177b8aa5b</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[magento]]></category>
            <dc:creator><![CDATA[ju5t]]></dc:creator>
            <pubDate>Mon, 22 Jan 2018 20:03:28 GMT</pubDate>
            <atom:updated>2018-01-22T22:03:06.119Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2017 we began working on a self-service platform based on Magento 2.</p><p>We’re not developers, but we do happen to know a thing or two about hosting applications. We had seen lots of discrepancies between deployments in previous platforms and that fueled the decision to use Docker for development, acceptance and production.</p><p>This is a short story about our journey to run Magento 2 in Docker.</p><p><em>Ps. we had no idea what we got ourselves into.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*6eAGRrD_rKvTPZqax5UgWw.png" /></figure><h3>Choosing a suitable Docker image</h3><p>One thing was clear from the start. Our Docker image should only include the bare minimum to host Magento. We didn’t want to include other processes in the same container as our application. Not only is it bad practice, it also makes scaling more difficult. If you only have one application in your image that’s all you need to worry about. And that’s plenty already.</p><p>This requirement meant we had to find a solution for our database, storage, email and cron, because, well — you need them.</p><p>We already had a few database servers so that was an easy pick. Our static files are stored on S3-compatible storage from <a href="https://fuga.cloud/">Fuga</a>, our sessions are kept in Redis and e-mail is sent out with SMTP. That’s all pretty straight forward.</p><p>But cronjobs aren’t. They can often only run from a single location; duplicate crons are likely to cause havoc. We needed a solution for this but we thought it made sense to cross that bridge when we got there. So we will.</p><p>Unfortunately, none of the available Docker images met our requirements, as they would either deploy several services in a single container or had no intent to be used in production or at scale. Docker seemed to be used a lot for development and that didn’t tick all boxes for us.</p><p>We decided to make our own image.</p><h4>How to dress in layers</h4><p>Docker loves layers. The foundation of our application is based on an existing image being php:7.0-apache. We added all requirements to run Magento 2 to it and published a new image that in turn can be extended again. This doesn’t sound too difficult but proved to be far more challenging than we thought.</p><p>We wanted to use Docker for development, acceptance and production. This meant we needed fast, but above all working, initial deployments. Starting MySQL and Magento together should result in a reusable development or test environment. Once it’s running in production though, it shouldn’t run unintented, not be confused with unattended, database upgrades.</p><p>All of this is done through our ENTRYPOINT. We won’t go into the specifics of Docker here but our <a href="https://github.com/sensson/docker-magento2/blob/master/entrypoint.sh">entrypoint.sh</a> bash script runs on each start. All of the required logic is pushed into ~100 lines of code.</p><p>It took a few dozen runs before all pieces fell in place.</p><p>Let’s go through afew of its features. entrypoint.sh is smart enough to wait for the database to become available. It checks the output of bin/magento setup:db:status for uninstalled plugins and installs them only if UNATTENDED is set. It knows what to do in production ánd development environments and last but not least, it includes cron.</p><p>Wait, what?</p><p>Yes. It includes cron.</p><h4>Cronjobs in Docker</h4><p>We didn’t plan on including cron in the same image when we started. For obvious reasons we didn’t want to have multiple services in the same container — it’s bad when you want to scale, remember?</p><p>Cron is no exception.</p><p>But Magento’s scheduled tasks need access to the preferably same source code as what runs your front facing shop. It made sense to add cron to the same base image to keep things DRY.</p><blockquote>So we broke our own rule.</blockquote><p>We added cronjobs with one exception: the image can only run one service at the same time. This made sure we could scale our web containers without having to worry about crons running simultaneously. This made having two services in the same image acceptable. For us at least.</p><h3>The source of it all: Magento</h3><p>At this point we had a working base image without Magento’s source code.</p><p>We never intended to add the source code of Magento to our base image as extending a base image is much more powerful and flexible. A basic version of a Dockerfile running our application now looks like this.</p><pre>FROM sensson/magento2<br>COPY src/cron /etc/cron.d/magento2<br>COPY src/ /var/www/html/</pre><p>So how do we get Magento’s source in src?</p><p>We needed to be sure that we are always running the same version.</p><p>At first we forked the magento/magento2 git repository and copied its contents to /var/www/html. With our knowledge at that time it seemed to make sense compared to managing a zip-download that we often read about. Little did we know.</p><p>The Magento-repository contained close to 60,000 commits and 1,500,000 objects. It had no impact on our deployment but mixing our own commits with Magento was wrong. Even though we never touched the core it didn’t really fit our Docker ambition to build on top of existing solutions. Our changes were small but the repository was huge. We sometimes struggled looking for our own files. Even though git is great, we needed something else.</p><p>We had to go back to the drawing board.</p><p>Our initial Magento-git-based setup used Composer to keep track of the plugins it required. Somewhere we had missed a turn when Composer was explained as a way to manage Magento too. So that’s what we changed. We rewrote our environment to use a single, smaller composer.json and gave full control to Composer. composer install is all we need now to manage src.</p><h4>Source control</h4><p>The structure below is a snippet of what we keep in source control.</p><pre>.<br>|-- Dockerfile<br>`-- src<br>    |-- app<br>    |   `-- etc<br>    |       |-- config.php<br>    |       `-- env.php<br>    |-- composer.json<br>    `-- cron</pre><p>The two files in app/etc/ turned out to be special. Again this is something we didn’t knew when we began and in fact our first deployments had these files only configured when the container started. This doesn’t sound too bad, but we found out that this had a few side effects.</p><p>The biggest of which was the fact that it ran the installer each time the container started. This didn’t reinstall an existing installation but it did change the default admin password back to the one specified in your deployment. That’s definitely not good. On top of it the start of new containers would be considerably slower.</p><p>Adding env.php and config.php to your image solves this.</p><p>So, what do these files do?</p><h4>Deployment configuration</h4><p>env.php and config.php are referred to as a <a href="http://devdocs.magento.com/guides/v2.0/config-guide/config/config-php.html">deployment configuration</a>.</p><p>env.php holds credentials to your database amongst specific settings about your environment such as caching. entrypoint.sh, what starts our containers, is used to set our database credentials which are configured through environment variables that are passed on to containers in production, test and development environments.</p><p>Don’t store credentials in git.</p><p>Our env.php only includes theinstall and cache_types-keys.</p><pre>&lt;?php<br>return array(<br>  &#39;install&#39; =&gt; array(&#39;date&#39; =&gt; &#39;Tue, 21 Nov 2017 15:06:38 +0000&#39;,),<br>  &#39;cache_types&#39; =&gt; array(<br>    &#39;config&#39; =&gt; 1,<br>    &#39;layout&#39; =&gt; 1,<br>...<br>  ),<br>);</pre><p>We found this to be the minimum requirements. We haven’t had any problems since, but there could be other useful settings that you want to keep in source control.</p><p>config.php is called a shared configuration. It contains the list of installed modules, themes, and language packages; and shared configuration settings.</p><h3>Continuous integration and how it fitted in our development process from test to production</h3><p>Docker was not new to us when we began working on our Magento implementation. Our API and a few other services were all Docker-based already. We knew how to build images from scratch and how to deploy them automatically with continuous integration.</p><p><em>We use Gitlab CI for all of our builds.</em></p><p>We only hit a few minor bumps down the road.</p><p>When we initially build our Docker image we ran composer install at the start of each container. Not the brightest idea. Your containers will not boot as fast you like and that is not what we hoped for. But that is not even the most important reason to move it out of entrypoint.sh. Running composer install on each start means your image won’t be consistent each and every time. That’s bad. This had to be done in the preparation stage, before we build the container.</p><p>That seemed straight forward. We added PHP and Composer to our build container — yes, we build our images in a Docker container — to make sure we could install all plugins to src before building the image. That also meant our build image needed access to our custom plugins and Magento’s repository.</p><p>To access our custom plugins we needed to setup new deploy keys. This process is explained in <a href="https://docs.gitlab.com/ee/ci/ssh_keys/README.html">the documentation of Gitlab</a>. Deploy keys are used to allow build processes read only access to repositories other then their own. As we had them this was a straight forward process for us.</p><h4>Composer</h4><p>Composer was still relatively new to us. The first thing we ran into was authentication. We needed to access repo.magento.com and that meant a username and password had to be injected in our image and stored for Composer to use it. Gitlab offers <a href="https://docs.gitlab.com/ce/ci/variables/README.html#secret-variables">secret variables</a> for this purpose.</p><p>We just had to add them to our build.</p><pre>- composer config -a -g http-basic.repo.magento.com $COMPOSER_REPO_MAGENTO_COM_USERNAME $COMPOSER_REPO_MAGENTO_COM_PASSWORD</pre><blockquote>But it didn’t work. Our builds had PHP syntax errors. We used Docker. Why did Docker fail us?</blockquote><p>As usual with computers, Docker did exactly what it was told. It built the container based on the instructions it was given. The problem was not Docker, it was us.</p><p>Our build container is Alpine-based. As I mentioned before we added PHP and Composer before we prepared our src directory. This installation is done with the following one-liner.</p><pre>- apk add php7 php7-curl php7-openssl php7-json php7-phar php7-dom php7-iconv php7-mbstring php7-zlib php7-ctype php7-gd php7-simplexml php7-mcrypt php7-intl php7-xsl php7-zip php7-pdo_mysql php7-soap php7-xmlwriter php7-tokenizer php7-xml &amp;&amp; curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer</pre><p>This did exactly what we wanted. It installed all of the dependencies we needed. But it installed PHP 7.1. Yes, 7.1. And we were running 7.0 in the final image that we copied src to. We were using the wrong PHP version.</p><p>At first we tried to switch to our base image to prepare our build. But we needed Docker and that meant we had to install it in our base image. No, we had to find another solution. As it turned out you can actually tell Composer which PHP version it should advertise to the repositories it uses. We added a the snippet below to our composer.json and that solved the issue.</p><pre>&quot;config&quot;: {<br>  &quot;platform&quot;: {<br>    &quot;php&quot;: &quot;7.0.23&quot;<br>  }<br>},</pre><p>Our CI now builds perfect images that we can reuse in development, test and production, as many times as we like.</p><h4>Development</h4><p>Having a Docker image is great but above all it has to work. Google recently launched <a href="https://opensource.googleblog.com/2018/01/container-structure-tests-unit-tests.html">Container Structure Tests</a>. This is a useful addition but we needed to be sure that our container runs — and returns a ‘200 OK’ when you try to access it.</p><p>Until now having a Docker container didn’t add much. We have done a lot of work to build a container where you could build a server for Magento in a few hours too. It’s time for it to finally add value to our development process.</p><p>After the image is build it should be ready to rock and roll. That’s what the image is build for. So that’s what we do. With Docker Compose we can build and setup new environments running just a few commands. Our docker-compose.yml includes a database, frontend and cron container. If you want to test a feature all you need to run is docker-compose build &amp;&amp; docker-compose up to get started.</p><h4>Integration tests</h4><p>Testing if it works shouldn’t require you to type — at all.</p><p>We have automated a similar process as our development process during our build. Instead of opening a browser when the environment starts we run a set of curl-commands that mimic the tests we would otherwise run by hand.</p><p>Again — this was easier said than done.</p><p>What made it difficult was the fact that we are running Docker in Docker for our tests. Even though the container running CI has curl installed it turned it had no access to the actual deployment. The solution to this was easy in hindsight. We now run our curl-command within a container that has been given access to the host network with --net host. This would be a near exact simulation of how a real visitor would reach our website.</p><p>Everything combined our simplified integration test looks like this:</p><pre>docker-compose build<br>docker-compose up -d<br>echo &quot;Waiting on the environment to come online&quot;<br>sleep 240<br>docker logs magento-frontend || true<br>docker run --rm --net host appropriate/curl -I -s localhost:8080 | \<br>  grep &#39;200 OK&#39;<br>docker-compose down</pre><p>This is extendable of course. We can add as many curl commands as we need.</p><p>If the test passes we take our integration environment offline, tag the image with the branch it was build for and push it to our Docker registry.</p><p>And finally, we have our image.</p><h3>Closing words</h3><p>Although we are still developing our portal this has helped us a lot. We can now build, test and deploy new environments automatically knowing that they will be consistent throughout the entire process.</p><p>It’s not perfect yet. We’re not testing how traffic will move through our load balancer, we should automate rollbacks and it would be great if we could scale Redis on our cluster automatically. But I think we’re close.</p><p>The next step? Scale up. But now we are using Docker — how hard can it be?*</p><p><em>*We know. It won’t be.</em></p><h4>Open source</h4><p>Our <a href="https://github.com/sensson/docker-magento2">base Magento image is open source</a>. It is available on Docker Hub and you can use it with sensson/magento2.</p><p>We published <a href="https://github.com/sensson/docker-magento2/tree/master/example">a working example</a> of what we have done in GitHub for those of you who prefer reading code instead.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a6177b8aa5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>