<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Lundberg on Medium]]></title>
        <description><![CDATA[Stories by Alex Lundberg on Medium]]></description>
        <link>https://medium.com/@lundbird?source=rss-a3862376a8d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qWT-JrbTA5QZYahD.</url>
            <title>Stories by Alex Lundberg on Medium</title>
            <link>https://medium.com/@lundbird?source=rss-a3862376a8d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:02:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lundbird/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing Gitlab CI templates, Part 2/3: template structure]]></title>
            <link>https://lundbird.medium.com/best-practices-for-writing-gitlab-ci-templates-part-2-3-template-structure-4918c87059d0?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4918c87059d0</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[gitlab-runner]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Mon, 14 Jun 2021 13:00:53 GMT</pubDate>
            <atom:updated>2021-06-16T19:23:23.888Z</atom:updated>
            <content:encoded><![CDATA[<p>Writing CI templates to run build, test, and deploy your project is challenging to do in a way that prioritizes pipeline speed, safety, and easy maintenance. In the first part of this series, I discussed the best practices for developing changes to CI templates. Here I discuss the best ways to setup your templates to reduce code duplication and allow for reduced maintenance.</p><h3><strong>Use a separate repo to host your project templates.</strong></h3><p>Within Gitlab, you can use the includes keyword to reference templates held within another git repo. This allows you to move all of your CI templates to a monorepo which allows you to easy track changes within templates and provide easy tracking of inheritance. When you use a monorepo, you should be careful with file organization to allow easy changes later.</p><p>I find what works best is to have one master pipeline configuration file which contains file references to each stage and the individual jobs within the pipelines organized like following.</p><pre>globals/<br>  setup/<br>    setup.yaml<br>  build/<br>    docker-build.yaml<br>  deploy/<br>    docker-deploy.yaml<br>  promote/<br>    promote.yaml<br>java/<br>node/<br>golang/<br>  full.yaml<br>  setup/<br>    setup.yaml<br>  build/<br>    docker-build.yaml<br>  test/<br>    code-quality.yaml<br>    go-test.yaml<br>    container-scanning.yaml<br>  deploy/<br>    docker-deploy.yaml<br>  promote/<br>    promote.yaml</pre><p>Within the full.yaml you would see include references to each of the files responsible for running each stage of the pipeline. Within each folder you hold specific jobs which run during each stage. If jobs share common code across languages, such as pipeline setup, building docker images, promoting with git tags, and deploying a docker images, you can have these files refer to a template within a global folder.</p><p>This way you have a full working pipeline that developers can add into the projects .gitlab-ci.yml.</p><pre>include:   <br>  - project: &#39;my-group/ci-templates&#39;<br>    ref: main<br>    file: &#39;/golang/full.yaml&#39;<br>variables:<br>  GO_VERSION: 1.17</pre><p>Developers only need to add those five lines of code to implement a full working pipeline in the language of your choice for the version of their choice (Note to leave GO_VERSION empty in the CI templates since each project should define this individually to match the version in their Dockerfile and in their go.mod.)</p><p>Moreover, this configuration allows you to decouple pipeline changes from your projects. So long as you keep the pipeline yaml syntactically correct at all times (can be performed with a CI lint job that runs in your ci-templates repo), you can change or add jobs and have it take effect in projects that reference the master branch. You can also version your CI changes with tags or separate branches for stable and latest.</p><h3>Keep your templates DRY.</h3><p>We want to reduce the amount of code duplication within our templates, and we can do this in many ways. If you use gitlab, you can use the extends keyword which allows you to inherit a jobs template spec from a parent. If you are outside of gitlab, you can use yaml anchors to reference repeated points within. Additionally, within gitlab you can use the reference keyword and the includes keyword to reference configuration held within separate files, or within separate files held within another git repository as shown above.</p><p>Make leverage of theextends keyword with variables and <a href="https://docs.gitlab.com/ee/ci/yaml/README.html#hide-jobs">hidden jobs</a>(jobs starting with a period within gitlabci). Ideally you should be able to move all of your logic into a hidden job, with each non-hidden job describing the implementation via variables. An example is shown below where the logic in handled in the .go-migrate job but the implementation is handled in the go-migrate-dev-up job.</p><pre>variables:<br>  MIGRATE_SEARCH_PATH: public<br>  MIGRATE_SOURCE: db/migrations<br>  MIGRATE_COUNT: &#39;&#39;<br><br>.go-migrate:<br>  image: migrate/migrate:v4.14.1<br>  stage: build<br>  script:<br>  - migrate -path $MIGRATE_SOURCE -database &quot;postgres://$DB_USER:$DB_PASSWORD@$DB_HOST:$DB_PORT/$DB_NAME?sslmode=disable&amp;search_path=$MIGRATE_SEARCH_PATH&quot; $DIRECTION $MIGRATE_COUNT<br><br>.go-migrate-dev:<br>  extends: .go-migrate<br>  except: [master, tags]<br>  environment:<br>    name: dev<br>  variables:<br>    DB_HOST: mydatabase<br>    DB_NAME: dbname<br>    DB_PORT: 5432<br>    DB_USER: postgres<br>    DB_PASSWORD: $DEV_DB_PASSWORD<br><br>    <br>go-migrate-dev-up:<br>  extends: .go-migrate-dev<br>  variables:<br>    DIRECTION: up</pre><pre>go-migrate-dev-down:<br>  extends: .go-migrate-dev<br>  when: manual<br>  variables:<br>    DIRECTION: down</pre><p>In the above we see some global variables, MIGRATE_SOURCE_PATH, MIGRATE_SOURCE, MIGRATE_COUNT, exposed that the end use may want to override in their projects .gitlab-ci.yml. It is best to expose variables that the end-user may want to change under globalvariables rather than variableswithin a non-hidden or hidden job spec.</p><p>We also see two layers of inheritance done to reduce code duplication within the end jobs which simply have 1 use of the extends keyword, and one variable specified. This way if we ever want to adjust the logic of the migration, we only need to implement it at one place.</p><p>Note that the DEV_DB_PASSWORD is not defined in the pipeline configuration — This is up to each project to have this key managed securely.</p><p>Another way to reuse code is with the <a href="https://docs.gitlab.com/ee/ci/yaml/#reference-tags">reference</a> keyword which allows you to insert yaml template from files outside of the current one. An example of this would be to have hidden utility function scripts such as below which can be held in a global utils.yml file</p><pre>.scripts<br>  git-promote:<br>  - VERSION=`git describe --abbrev=0 --tags`<br>  - VERSION=${VERSION/v/}<br>  - VERSION_BITS=(${VERSION//./ })<br>  - VNUM1=${VERSION_BITS[0]}<br>  - VNUM2=${VERSION_BITS[1]}<br>  - VNUM3=${VERSION_BITS[2]}</pre><p>Then reference this within your promote.yml file.</p><pre>hotfix-release:<br>  extends: .promote-template<br>  script:<br>  - !reference [.scripts, git-promote]<br>  - VNUM3=$((VNUM3+1))<br>  - NEW_TAG=&quot;v$VNUM1.$VNUM2.$VNUM3&quot;<br>  - git tag -a &quot;${NEW_TAG}&quot; -m&quot;Hotfix ${NEW_TAG}&quot;<br>  - git push --tags</pre><p>Another example could be to hold your caching configuration with your setup.yml file and reference these cache configurations within later jobs for your pipelines.</p><pre>.cache:<br>  pull-only:<br>    key: $CI_COMMIT_REF_SLUG<br>    paths:<br>        - node_modules/<br>    policy: pull<br>  push:<br>    key: $CI_COMMIT_REF_SLUG<br>    paths:<br>        - node_modules/<br>    policy: push</pre><p>Then reference the above caching configuration within each jobs spec:</p><pre>eslint:<br>  stage: test<br>  cache: !reference [.cache, pull-only]<br>  ...</pre><pre>type-check:<br>  stage: test<br>  cache: !reference [.cache, pull-only]<br>  ...</pre><h3>Conclusion</h3><ul><li>Use a separate Monorepo to store your template configuration.</li><li>Use one file to reference the templates for a fully working end-to-end pipeline, and reference this pipeline file in projects using include</li><li>Use extends reference and hidden jobs to reduce code duplication</li></ul><p>Continue on to part 3 of this story <a href="https://lundbird.medium.com/best-practices-for-writing-gitlab-ci-templates-part-3-3-pipeline-configuration-9a182ef04526">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4918c87059d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Gitlab CI templates, Part 3/3: pipeline configuration]]></title>
            <link>https://lundbird.medium.com/best-practices-for-writing-gitlab-ci-templates-part-3-3-pipeline-configuration-9a182ef04526?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a182ef04526</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[gitlab-runner]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Mon, 14 Jun 2021 13:00:27 GMT</pubDate>
            <atom:updated>2021-06-16T19:22:58.832Z</atom:updated>
            <content:encoded><![CDATA[<p>Writing CI templates to run build, test, and deploy your project is challenging to do in a way that prioritizes pipeline speed, safety, and easy maintenance. In the first part of this series, I discussed the best practices for developing changes to CI templates. In the second part I go over how to setup your templates to reduce code duplication and make changes easy to maintain. In this last part I discuss some common pitfalls, and some general tips to improve your CI. The changes below use GitlabCI as example but can easily be extended to other engines.</p><h3>Understand build, test, and dependency management for your language.</h3><p>Languages and frameworks can differ a lot in how they handle these. And though you could partially eliminate the necessity to worry about the build step by pushing this step down to each projects Dockerfile configuration (if building docker-images), you still need to be concerned about dependency management and testing. Best practices here will vary by your language, so it is important to have some familiarity.</p><p>You can always check what templates Gitlab publishes for your language as there will often a good starting point.</p><h3>Leverage Gitlab artifacts</h3><p>Some job file outputs can be leveraged by gitlab artifacts to show information within the merge request. A common use case of this is exporting code quality metrics through gitlab. Artifacts can be used to push binaries or folders to Gitlab and allow them to be pulled down in later stages. However caching is better suited to this and more flexible. Keep artifacts only for pushing binaries that are used by gitlab to generate reports.</p><h3>Leverage caching</h3><p>Leverage gitlab caching to reduce the amount of time spent downloading files. This is usually done to cache dependencies such as node_modules. Ensure that the policy for jobs pulling/pushing from the cache is correctly setup and extra work is not being performed such as re-uploading the cache.</p><p>Be warned the setting up caching correctly so that it is fast, only runs when needed, and does not perform any unnecessary steps is very challenging. For one, consider when you want the cache upload step to run. A common pattern is to have a separate setup stage and job that runs code to 1) Pull down existing dependencies , 2) update dependencies, and 3) Push them back up. It is a good idea to only run this stage when a change occurs to the projects dependency tracking system such as package.json. All future jobs should have their pull-policy set to pull-only to pull from this cache that the setup job created.</p><pre>setup:<br>  stage: setup<br>  cache:<br>    key: $CI_COMMIT_REF_SLUG<br>    paths:<br>      - node_modules<br>    policy: push<br>  script:<br>    - npm install  <br>  only:<br>    changes:<br>      - package-lock.json</pre><p>A word of warning is to be aware of the tradeoff’s that come when setting your caching infrastructure. You can configure Gitlab runner to store the cache within S3, but be careful that this doesn’t unnecessarily slow down your pipelines, as it may not be any faster than pulling your dependencies normally from the web, or from your own registry mirror. You could configure the gitlab-executors to cache on their host node, but then you have to ensure that future executors run on the same node. Conversely, you could mount an EFS volume to each node that would service your gitlab-executors and use node-selectors or taints/tolerations (if on Kubernetes) to ensure your executors run on those nodes with the cache. You would also want to ensure your CI steps are robust enough to still function during a cache miss. You could do this by writing a command in your before_script section to first check the existing of dependencies, and to download if missing.</p><h3>Compress stages and run jobs in parallel</h3><p>Many times stages can easily run in parallel such as build and test. Other times you want to block further stages, such as deploy, if prior phases fail. Or you need artifacts from previous stages to build your docker image. Consider if you really need the ordering of your stages and if you can rather run those in parallel. You can additionally use the needs keyword to run jobs out of order. For example, start the image build after the artifact is created from a prior stage, but before the testing has fully completed.</p><p>Within GitlabCI, you can use the needs and dependencies keywords to increase your pipeline speed. Needswill run a job if the job under needs has completed rather than waiting for all jobs in the prior stages to complete. This allows you to trigger a job before it would otherwise be triggered if its needs jobs complete. Dependencies will select which artifacts from previous jobs you need to pull for your current job. An example would be an image build job that only pulls the binary artifact instead of the test result artifacts.</p><h3>Use a sensible base images for running your jobs.</h3><p>If you are running apk or yum or apt-get commands in your CI script, Consider instead to search for or even make a new docker image that comes bundled with the dependencies you need.</p><h3>Use a sensible Dockerfile.</h3><p>If you notice your docker build takes long you may be able to rearrange your Dockerfile layers such that operations that are more likely to change are placed lower in your Dockerfile. An example of this would be to move the lines that copy dependencies to the beginning of your Dockerfile as this is less likely to change than an update to the application code. If you notice that your container is large or takes long to startup. Consider using a scratch container or a lightweight base image to reduce the resources usage and boot time.</p><p>Remember that caching does not only need to apply to managing you’re dependencies. If you are building and deploying docker-images, you should also make use of Gitlab’s <a href="https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#make-docker-in-docker-builds-faster-with-docker-layer-caching">docker-image caching</a> using the docker build --cache-from command. This can greatly reduce the build time if you have properly setup the project’s Dockerfile with <a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache">layering</a>.</p><h3>Skip old jobs if new commits are added, and have jobs automatically retry.</h3><p>The interruptible keyword stops jobs running in old pipeline when new pipeline run has begun. This reduces the burden on your executors.</p><p>The retry will automatically retry a job on failure. You can even specify on what conditions you want to start an automatic retry. This should reduce the manual toil when a flaky test fails and needs to be restarted.</p><h3>Check for new Gitlab features.</h3><p>Gitlab constantly upgrades its CI platform to add new features or remediate existing issues. These often can be used to simplify CI pipelines, reduce pipeline time, or add useful metrics for developers. Check to see what other gitlab users are currently doing for your use-case and what gitlab recommends.</p><h3>Conclusion</h3><ul><li>Understand the software lifecycle stages for your language and framework</li><li>Leverage caching</li><li>Make use of Gitlab artifacts for reporting</li><li>Compress stages and parallelize jobs when possible</li><li>Use a sensible base image for running jobs</li><li>Use a Dockerfile with sensible cache layers</li><li>Skip old jobs and have failed jobs automatically retry.</li><li>Check for new Gitlab features.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a182ef04526" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Gitlab CI templates, Part 1/3: local development]]></title>
            <link>https://lundbird.medium.com/best-practices-for-writing-ci-part-1-2-local-development-2332eb6ed3f6?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2332eb6ed3f6</guid>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Sun, 13 Jun 2021 23:37:46 GMT</pubDate>
            <atom:updated>2021-06-16T19:22:20.749Z</atom:updated>
            <content:encoded><![CDATA[<p>Developing continuous integration pipelines is often both time consuming and error-prone. Below are a collection of best practices and helpful tips for developing better pipelines that will improve how quickly and safely you can move code from a developers laptop into production.</p><p>I am most experienced with Gitlab so the recommendations are based on their engine, though most will apply to other CI tools also.</p><h3>Developing CI.</h3><p>How do we write and test and develop CI templates quickly? If your process looks like the following you are doing it wrong:</p><ol><li>Committing a CI change on a branch up to the remote repository</li><li>Waiting for your pipeline runner to start</li><li>Waiting for the jobs to run which come before your changed stage.</li><li>Viewing the behavior of the changes and the logs.</li><li>Based on the results go back to step 1 and repeat.</li></ol><p>This is incredibly slow as the feedback between your changes will be on the order of minutes or more. Expect to be frustrated when after waiting for your changes to be ran, you find it was a simple mistake, only to go back and make the fix to find out you make another small mistake.</p><p>This brings us to our first heuristic for developing CI:</p><p><em>Develop CI locally as much as possible.</em></p><p>This may seem counterintuitive since CI by definition will run on your CI server. But often you can do a majority of your testing locally. If your CI is just a series of shell commands, you should be easily able to run these locally against a test project.</p><h3>Local Development</h3><p>When you develop CI locally you will likely run into the below issues.</p><ol><li>Runner configuration (environment variables, services, secrets)</li><li>Runner environment (OS, program binaries, networking)</li><li>Pipeline logic (branches, approvals, dependent stages, promotion)</li></ol><p>These issues often just make it so people may simply develop CI by pushing new commits to the CI server and going through the five steps above. In some cases, this is inevitable, but we should do everything possible to develop locally. Lets discuss how to fix each of these in turn:</p><p><strong>Runner configuration (environment variables, services, secrets)</strong></p><p>In many cases you can export the project level variables into your local environment. At least for gitlab, you can find these variables in the .gitlab-ci.ymlfile or in the project settings or in the group settings. Additionally, gitlab will add in a whole <a href="https://docs.gitlab.com/ee/ci/variables/predefined_variables.html">list of predefined environment variables</a>. Depending on the needs of your CI, you will have to export these additionally, or stub them out.</p><p>How you handle dealing with secrets depends on your secret-management solution and how much access you have. In some cases, these may just be additional environment variables you need. Consider if you need these or can stub them for testing.</p><p>Services can provide additional resources on the network such as sidecar containers or redis, postgres, kakfa. These have an analog of just running docker-compose locally to spin up these resources.</p><p><strong>Runner environment (OS, program binaries, networking)</strong></p><p>Docker is your best friend here. In general, your CI steps should be using a docker image that packages necessary dependencies anyway, so if you can just use the same docker image that your CI step uses, you should have most of these issues taken care of. Better yet, this means your code developed locally will closely match the environment that it runs within your CI.</p><p>In some cases, you cannot simply run a docker image to replicate your CI environment. If you run within gitlab, you can simply <a href="https://docs.gitlab.com/runner/">run the runner locally</a> and register it to your project. This can further ensure your local environment matches your CI. But this solution is native to gitlab, and has its own <a href="https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797">issues</a>. Another solution specific to gitlab is to use their <a href="https://docs.gitlab.com/ee/ci/pipeline_editor/">pipeline editor</a> which provides a mini-IDE to make CI changes and includes a linter. There may even be local CI development frameworks for your engine, either from the original provider, open source such as <a href="https://github.com/firecow/gitlab-ci-local">gitlab-ci-local</a>, or integration with your IDE.</p><p><strong>Pipeline logic (branches, approvals, dependent stages, promotion)</strong></p><p>When testing logic with the CI itself, you will likely have to push your changes up to test anyway. We can make this easy on ourself in two main ways. These hold for any CI development where you cannot develop locally.</p><ol><li>Collapse your pipeline configuration to only test the desired changes. Skip or stub jobs. Move your changes to the earliest stage as possible. Remove any dependencies among jobs that are not required to test your change</li><li>Create dedicated projects to test CI. Make these as simple as possible so pipelines run quickly, yet thorough enough that they meet CI requirements for real projects.</li></ol><p>Both of these together will ensure that your CI changes are not unreasonably slow. You should never have to wait for three stages to run if they do not affect the job you are testing. Testing your CI changes on a non-dedicated CI project adds extraneous commits and pipelines runs that may confuse the developers of the project, but more importantly it will be slower and less flexible than a project dedicated to testing CI changes.</p><p><strong>Use a custom, dedicated CI runner image.</strong></p><p>One final tip I will leave you with is to consider using a dedicated custom build image. If your CI job has 20 lines of shell scripts, would it be better to package this into a binary, containerize it, then use that container to run your job which would then call your script binary? I see this pattern often in Gitlab’s Auto-Devops CI templates. They use a custom build image and each CI job will have only a few lines of shell scripts which each call a custom command that is contained within the custom build image.</p><p>This not only helps to keep your CI DRY, it allows you to fully leverage bash scripting without being concerned that each of your commands fits into a neat line in your CI file. This becomes especially important when you deal with loops and conditionals as often these might work locally, but will have unexpected behavior when you place them directly into .gitlab-ci.yml.</p><h3>Conclusion</h3><ul><li>Do as much local development as possible to test your CI changes instead of committing these changes to the CI server.</li><li>Export CI environment variables locally, use the same docker containers as your jobs, and use docker-compose to replicate services.</li><li>Investigate your CI engine’s tools for doing local development.</li><li>Use a slim, dedicated CI project, and stub out unneeded jobs when testing.</li><li>Consider using a custom dedicated CI runner image.</li></ul><p>Continue on to part 2 of this series <a href="https://lundbird.medium.com/best-practices-for-writing-gitlab-ci-templates-part-2-3-template-structure-4918c87059d0">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2332eb6ed3f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Linkerd service-mesh in production]]></title>
            <link>https://medium.com/commonbond-engineering/using-linkerd-service-mesh-in-production-1d4a3d8162f6?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d4a3d8162f6</guid>
            <category><![CDATA[linkerd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Sat, 21 Mar 2020 18:29:11 GMT</pubDate>
            <atom:updated>2020-03-23T23:44:30.949Z</atom:updated>
            <content:encoded><![CDATA[<h3>Linkerd service mesh in production</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*ljNAKeAp6isAXAGBifirug.png" /></figure><h3><strong>The need for improved monitoring</strong></h3><p>Recently our company made a push for better metrics on our microservices. Namely, we wanted to quickly know:</p><ul><li>How long services took to respond, what endpoints on a service took longest to respond, and what distribution of response times we receive.</li><li>When a service failed, how it failed(status code, logs), and what route on the service failed.</li><li>What dependencies a service has, and what service is at fault when an outage occurs.</li></ul><p>We had monitoring that offered some coverage. Namely response code and latency metrics from our ingress controller, and more detailed metrics from individual applications instrumented with Prometheus. Often these would not give enough information to quickly diagnose issues and pinpoint the cause of failures. So to improve our monitoring, we looked to add a service mesh solution to our Kubernetes clusters.</p><h4>How a service mesh helps</h4><p>A service mesh works by injecting your application pods with a sidecar that proxies all of the application’s network traffic. The sidecar will gather metrics, such a latency, request rate, success rate, status code, host and destination service for each request. The sidecars will also create encrypted tls connections with each other to give end-to-end encryption within your cluster and prevent network sniffing. Finally, the sidecars give additional functionality into intelligent routing between your services. This can include things like canary deployments, blue-green deployments, retries, timeouts, load-balancing, and circuit breakers.</p><p>Our need for a service mesh was primarily driven for the need to have improved monitoring. The intelligent routing and mutual tls are nice to have as well, but not our focus. We also desired a service mesh that was simple and had low operational overhead.</p><p><strong>Which service mesh do we use?</strong></p><p>We investigated three servicemesh solutions:</p><ul><li>Istio</li><li>Maesh</li><li>Linkerd</li></ul><p>Istio is the largest and the most used service mesh solution on the market currently. It is incredibly comprehensive; however, it is currently very challenging to set up and maintain. We had installed Istio twice but removed it. We found the documentation was a moving target, and various connectivity issues had eroded trust with developers.</p><p>We took a cursory look into other new service mesh solutions such as Maesh. Maesh integrates cleanly with Traefik and runs a Daemonset instead of sidecar proxies. The Daemonset greatly reduces the number of sidecar proxies in your cluster which means less resource consumption. However, if a Daemonset pod fails, network connectivity is stopped for all pods on the node which makes this architecture less resilient to failure.</p><p>We found Linkerd to most closely match our desire from a service mesh. Linkerd describes itself as an “ultralight service mesh for Kubernetes” and has relatively low overhead for installation and maintenance, while still offering comprehensive observability into our microservices. It differentiates itself from Istio, choosing to be less feature-rich, only functional on Kubernetes, and un-opinionated about ingress controller, instead offering a solution that is simpler to manage. Furthermore, Linkerd is well supported as a CNCF project and has many industry players using it in production.</p><h3><strong>Starting off with Linkerd.</strong></h3><p>Linkerd offers a command-line tool to setup and install linkerd. Soon I had installed linkerd into a test cluster, installed the demo “booksapp”, instrumented route monitoring, connected the metrics to our Prometheus instance, and gathered visibility into application performance with their dashboard application. The documentation on the Linkerd website was very easy to follow and I had almost no issues in setting up a minimum working version.</p><p>Once we successfully installed a demo version of Linkerd, we installed it on our development Kubernetes cluster, and gradually injected proxies into each namespace.</p><h3><strong>Development cluster installation with Linkerd.</strong></h3><p>Moving from a demo installation of Linkerd to a semi-production installation requires some changes from the default installation.</p><ul><li>The Linkerd installation and configuration should be committed to code.</li><li>Linkerd should be installed in high availability mode.</li><li>The secrets should not be committed to git and instead managed by a secret-management service.</li><li>Linkerd should function well with our gitops solution, ArgoCD.</li><li>The Linkerd dashboard should be available to users without needing to install and setup the Linkerd cli.</li><li>It should be easy for developers to get enhanced monitoring from linkerd into their application.</li></ul><p>Linkerd offers a helm installation and a high availability option values.yaml file which takes care of the first two bullet points. Linkerd, as of release 2.7.0, allows you to create your root authority and issuer certificate separately and manage cert-rotation with cert-manager which takes care of bullet point three.</p><p>Managing the helm chart with ArgoCD, or any gitops solution, is easily done; however, the app will often appear as out-of-sync from its active manifests in Kubernetes. The certificate rotation of the issuer certificate done through cert-manager, as well as the built-in certificate rotation which rotates each control-plane component tls certificate daily, will cause your original helm installation to never match what is currently active. Worse, syncing the app will cause those certificates to revert and may require that some individual control plan components be restarted to allow for sidecar injection. This is something to be mindful of when we need to update Linkerd in the future.</p><p>Linkerd provides a dashboard that can be accessed through port-forwarding the linkerd-web deployment to your local machine. The dashboard provides a dependency graph of your services, taps into route calls, and offers connectivity metrics such as success rate, latency, and request rate. We wanted all developers and product managers to be able to access the dashboard without needing to install the linkerd cli and port-forward the linkerd-web deployment.</p><p>We discovered that the dashboard creates a WebSocket connection to the end-users machine, and the aws-elb’s provisioned from our Traefik LoadBalancer services will drop the headers needed to provide this connection. We instead set up a new load-balancer and added the missing headers with an Nginx server.</p><h3>Injecting an application with Linkerd</h3><p>It was critical that developers can easily instrument their applications with Linkerd and get per-route monitoring. Linkerd monitors routes based on the serviceprofile CRD which describes the routes to monitor and their timeouts and retries. Linkerd can create serviceprofiles using a swagger spec. We then added the serviceprofile to the Kubernetes manifests and distribute it to the different environments using Kustomize. The only tricky issue was that the serviceprofile name needs to exactly match the FQDN of the service, so distributing the serviceprofile to separate environments requires overwriting the serviceprofile name. We accomplished this by adding an inline Kustomize JsonPatch to replace the name for each environment:</p><pre>patchesJson6902:<br>- target:<br>    group: linkerd.io<br>    version: v1alpha2<br>    kind: ServiceProfile<br>    name: NAME<br>  patch: |-<br>    - op: replace<br>      path: &quot;/metadata/name&quot;<br>      value: &quot;$servicename.$namespace.svc.cluster.local&quot;</pre><p>Linkerd needs to know the Kubernetes destination service for requests coming in from external to the cluster. We added a JsonPatch to each ingress to append the annotation needed:</p><pre>- op: add<br>  path: &quot;/metadata/annotations/ingress.kubernetes.io~1custom-request-headers&quot;   <br>  value: &quot;l5d-dst-override:$servicename.$namespace.svc.cluster.local:serviceport&quot;</pre><p>We controlled the injection of the sidecar at the namespace level, so all developers needed to do to get enhanced monitoring was to generate a serviceprofile from a swagger-spec, then add a name patch for each environment, then add the Linkerd header to their ingress if available.</p><h3><strong>The Road to Production</strong></h3><p>All that needed to be done now was to gradually inject new development namespaces with Linkerd, fix any issues, then install Linkerd in production.</p><h4><strong>Issues we noticed</strong></h4><p>We discovered that one application was failing its readiness and liveness probes, and the logs told us that one of the external dependencies was taking much longer to resolve. This was because the external dependency was a domain name with four periods: x.x.x.x and the Linkerd-proxy assumed that this was a Kubernetes service. After failing to find the Kubernetes service after some timeout, the proxy then looked external to the cluster to correctly find the service. We then created an ExternalName service that pointed to the external service and directed the application to use this service instead. The Linkerd-proxy then resolved the service quickly and our application was operational.</p><p>Some of our applications logged failures during their first second, but resolved themselves later. This was because the application started up faster than the Linkerd-sidecar proxy. The application then tried to route traffic through a sidecar that didn’t exist. For some applications, this resolved itself quickly and no intervention was needed; for others, such as some reverse-proxies that failed to resolve DNS, then failed to provide proxying until their DNS check retriggered, were instead removed from sidecar injection. Future versions of Kubernetes are set to offer sidecar support to prevent race conditions like these from happening on fast-booting pods, but for now, we are stuck with handling these issues as they arise.</p><p>After adding serviceprofiles, some routes would fail to get a response. This was because the default timeout was configured to 10 seconds before the Linkerd proxy would return a 504 error to the requestor. We fixed this by adding a timeout to each route in our serviceprofiles.</p><p>gsed -i &#39;/pathRegex*/a \ \ \ \ timeout: 900s&#39; serviceprofile.yaml</p><p>We had an issue where the Linkerd-proxy was failing to proxy any network traffic. This was an intermittent bug with the stable-2.7.0 proxy and has caused many users to rollback to the stable-2.6.1. We rolled back to the stable-2.6.1 version but noticed that many sidecar proxies were restarting from hitting their memory limit. This was due to a memory leak in the log-management system that was fixed in the latest stable release. We disabled logging except for critical errors which caused our sidecar memory usage to stabilize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*neQmjKBbxMBmzZEj0vWilw.png" /><figcaption>Memory Usage of the sidecar before and after adjusting log level</figcaption></figure><h4><strong>Building an Improved Dashboard</strong></h4><p>We needed additional monitoring that the built-in Linkerd Grafana panels and the Linkerd dashboard didn’t provide as as these don’t provide metrics based on status-code and route. These metrics are present as dimensions in the proxy-metrics, so we created our own Grafana dashboard to monitor latency, success rate, and request rate. This dashboard is available on Grafanlabs <a href="https://grafana.com/grafana/dashboards/11868">https://grafana.com/grafana/dashboards/11868</a>, and a fork of this may be packaged with Linkerd in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gvwuVsePr-PphupngZ3tjw.png" /><figcaption>A custom dashboard that lets us analyze traffic by sender, receiver, status code, and route.</figcaption></figure><p><strong>CpuMageddon</strong></p><p>As we injected more sidecars, we noticed more scheduling failures for new pods. The reason for this was that each proxy sidecar requests a default of 100m CPU when running linkerd in high-availability mode. Multiply this by hundreds or thousands of sidecars in your cluster, and you can very easily hit your resourcequotas or simply consume all the cpus available on your nodes and prevent scheduling of new pods.</p><p>Worse, since these cpus may not be being used, you may fail to trigger a node scale-up depending on your autoscaling solution. To fix this, we simply gathered the historical averages of proxy cpu and memory consumption and adding that to our helm installation as the proxy requests.</p><h3><strong>Installing Linkerd in Production</strong></h3><p>Over three weeks, we monitored development and fixed the issues that arose. We then moved Linkerd into production, namespace by namespace. There were no surprises here. The installation was smooth and had no downtime for all injected applications. I have to give thanks to the Linkerd team for making this process painless and straightforward.</p><h3><strong>Afterthoughts</strong></h3><p>The biggest takeaway here is that our installation with Linkerd in production was seamless because we took the needed time to discover and fix issues in development. Each of the issues we discovered in our development environment could have had disastrous consequences in production if not mitigated. For those wishing to bring Linkerd or any service mesh into production it is important to migrate gradually and to give time to discover and fix issues related to your setup.<br>Again I want to give my thanks to the Linkerd team for their comprehensive documentation and for their promptness in answering my questions on slack.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d4a3d8162f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/commonbond-engineering/using-linkerd-service-mesh-in-production-1d4a3d8162f6">Using Linkerd service-mesh in production</a> was originally published in <a href="https://medium.com/commonbond-engineering">CommonBond Product &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing kubernetes environments with kustomize]]></title>
            <link>https://medium.com/commonbond-engineering/managing-kubernetes-environments-with-kustomize-ab3ab819d079?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab3ab819d079</guid>
            <category><![CDATA[vim]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kustomize]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Thu, 28 Nov 2019 22:49:13 GMT</pubDate>
            <atom:updated>2019-11-28T22:51:15.759Z</atom:updated>
            <content:encoded><![CDATA[<p>Kustomize is a tool that allows you to customize base kubernetes yaml files and manage yamls between environments.</p><p><strong>Comparison to Helm</strong></p><p>Kustomize is similar to helm in that it templates yaml between multiple environments. However, Kustomize base yamls are usable as is. Kustomize doesn’t use golang templating for managing configuration so is far easier to build and maintain. Kustomize has no package management system unlike helm. They are really two separate tools. Helm work best for building external charts and Kustomize shines at managing in-house yamls across separate environments.</p><p><strong>An example use case</strong></p><p>The example here can be found on my github here for you to follow along:</p><p><a href="https://github.com/lundbird/KustomizeTutorial">https://github.com/lundbird/KustomizeTutorial</a></p><p>Our example project consists of a deployment across three environments: dev, staging, and production. Our deployment of an nginx webserver will require a deployment.yaml, a service.yaml, and an ingress.yaml to define access to our service from outside our cluster.</p><p>Right now our three environments live together under a kube/ folder as kube/dev kube/staging and kube/prod and our job is to deduplicate the code across these environments.</p><pre>kube/<br>  -dev/<br>    -deployment.yaml<br>    -service.yaml<br>    -ingress.yaml</pre><pre>-staging/<br>    -deployment.yaml<br>    -service.yaml<br>    -ingress.yaml</pre><pre>-prod/<br>    -deployment.yaml<br>    -service.yaml<br>    -ingress.yaml</pre><p><strong>Deduping the environments:</strong></p><p>We setup our new folder structure:</p><pre>kube/<br>  -base/<br>  -overlays/<br>    -dev/<br>    -staging/<br>    -prod/</pre><p>where the directories under overlays/ are the original folders under kube/. We will have each of the environment overlays refer to the yamls that are stored in the base folder, so they can overlay their configuration overlay on top of the base yamls. Each folder will need a kustomization.yaml to define how to perform the merging of the base and environment yamls.</p><p>Next we need to extract all <em>shared</em> configuration between each environment and place them into the base folder. To do this easily I recommend copying the dev environment into base, then opening each yaml in vi with the -d option to show you the differences between each environment.</p><p>vi -d ../base/deployment.yaml dev/deployment.yaml staging/deployment.yaml prod/deployment.yaml.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XpoTG5_hYdS-3184yCALnQ.png" /></figure><p>Using vim, we can see that each environment lives in a separate namespace, and has a unique ENV variable. Prod also uses a hashed image version and an additional replica. Remove all the environment specific information from the base overlay. The labels differ for each environment. Just remove these for now.</p><p>Next we look at the service using the same command as above but replacing service instead of deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lMFy2OyL_uKHb0PHD4FQMQ.png" /></figure><p>We see that each environment is the same except for the selectors. Remove the labels. We will add these as a commonLabels to our kustomization.yaml</p><p>When you do the same with the ingress, you will find that it has a different hostname. Just remove the ingress file from each environment. We will use a json patch to patch in the different hostname to the base yaml</p><p>Now create a kustomization.yaml file in each folder. For our base folder, we want to tell kustomize to include each yaml in this folder.</p><pre>resources:<br>- deployment.yaml<br>- service.yaml<br>- ingress.yaml<br>- configmap.yaml</pre><p>Next we want to modify our dev kustomization.yaml as follows:</p><pre>resources:<br>- ../../base<br>patchesJson6902:<br>- target:<br>  group: extensions<br>  version: v1beta1<br>  kind: Ingress<br>  name: nginx<br>  path: ingress_patch.yaml<br>commonLabels:<br>  app: dev-nginx<br>namespace: dev<br>configMapGenerator:<br>- name: nginx<br>  behaviour: merge<br>  literals:<br>  - ENV=dev</pre><p>There is a lot going on here. In order:</p><p>“resources” refers to the base directory which we will be overlaying on top of.</p><p>“patchesJson6902” defines the patch we we be adding to our base, in this case a patch to overwrite the original hostname. We will need to add our patch under ingress_patch.yaml:</p><pre>- op: replace<br>  path: /spec/rules/0/host<br>  value: dev-nginx.mydomain.com</pre><p>“commonLabels” will create labels to all of our deployments, pods, services, and deployment and service selectors in this app.</p><p>“namespace” defines which kubernetes namespace to deploy the app.</p><p>“configMapGenerator” is a way for us to manage all of our configuration variables outside of our deployments. We will create a configmap with all of the shared environment variables in base, then merge in our environment specific values in our configMapGenerator. You will need to update the following to your base deployment to import the configs:</p><pre>containers:<br>- name: nginx<br>  envFrom:<br>    configMapRef:<br>      name: nginx</pre><p>as well as create a configmap.yaml in base:</p><pre>apiVersion: v1<br>data:<br>  DB_HOST: postgres<br>  DB_HOST: &quot;8081&quot;<br>kind: ConfigMap<br>metadata:<br>  name: nginx</pre><p>To see the result of the overlay run kustomize build.</p><pre>&gt;&gt;&gt;kustomize build overlays/dev</pre><pre>apiVersion: v1<br>data:<br>  DB_HOST: &quot;8081&quot;<br>  ENV: dev<br>kind: ConfigMap<br>metadata:<br>  annotations: {}<br>  labels:<br>    app: dev-nginx<br>  name: nginx<br>  namespace: dev<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  labels:<br>    app: dev-nginx<br>  name: nginx<br>  namespace: dev<br>spec:<br>  ports:<br>  - name: http-app<br>    port: 80<br>  selector:<br>    app: dev-nginx<br>---<br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: dev-nginx<br>  name: nginx<br>  namespace: dev<br>spec:<br>  metadata:<br>    labels:<br>      app: nginx<br>  selector:<br>    matchLabels:<br>      app: dev-nginx<br>  template:<br>    metadata:<br>      labels:<br>        app: dev-nginx<br>    spec:<br>      containers:<br>      - image: nginx:latest<br>        imagePullPolicy: Always<br>        livenessProbe:<br>          httpGet:<br>            path: /<br>            port: 80<br>          initialDelaySeconds: 10<br>        name: nginx<br>        ports:<br>        - containerPort: 80<br>        readinessProbe:<br>          httpGet:<br>            path: /<br>            port: 80<br>          initialDelaySeconds: 10<br>        resources:<br>          limits:<br>            memory: 500Mi<br>          requests:<br>            cpu: 100m<br>            memory: 100Mi<br>      imagePullSecrets:<br>      - name: gitlab<br>---<br>apiVersion: extensions/v1beta1<br>kind: Ingress<br>metadata:<br>  annotations:<br>    ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0<br>    ingress.kubernetes.io/whitelist-x-forwarded-for: &quot;true&quot;<br>  labels:<br>    app: dev-nginx<br>  name: nginx<br>  namespace: dev<br>spec:<br>  rules:<br>  - host: dev-nginx.mydomain.com<br>    http:<br>      paths:<br>      - backend:<br>          serviceName: nginx<br>          servicePort: http<br>        path: /</pre><p>Next we will do the same for prod. Our kustomization.yaml looks very similar except for two additional fields:</p><pre>replicas:<br>- name: nginx<br>  count: 3<br>images:<br>- name: nginx<br>  newTag: 6b654da03c0e<br>namePrefix: prod-</pre><p>These will update our image to be tagged to a hash of nginx, increase our replica count, and prepend our resource names with prod-</p><p>Again run kustomize build to determine if the results are as you expect.</p><p><strong>Final Thoughts</strong></p><p>Kustomize has a lot of features, only a small fraction of which I have gone through today. Kustomize has some good tutorials on its gettings started page here: <a href="https://github.com/kubernetes-sigs/kustomize/tree/master/examples">https://github.com/kubernetes-sigs/kustomize/tree/master/examples</a>. Additionally for more detailed kustomize documentation use the kubectl docs here: <a href="https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html">https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html</a></p><p>The two tools I use the most in kustomize is the JsonPatch and StrategicMergePatch because they can work on any environment. However, oftentimes you don’t need them. You can often place your differences within the kustomization.yaml itself as shown above for images, commonLabels, and replicas.</p><p>Kustomize can really shine in creating and referring to a a single source of truth. In the above example, we used a whitelist of 0.0.0.0/0, and referred to it in each overlay, but this example can be extended where you keep JsonPatches of whitelists in a separate repository. Then you can refer to these JsonPatches when you build your own ingresses. The same can also be done for whatever centralized patches that you can plug and play.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab3ab819d079" width="1" height="1" alt=""><hr><p><a href="https://medium.com/commonbond-engineering/managing-kubernetes-environments-with-kustomize-ab3ab819d079">Managing kubernetes environments with kustomize</a> was originally published in <a href="https://medium.com/commonbond-engineering">CommonBond Product &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating Kubernetes Clusters with Velero]]></title>
            <link>https://lundbird.medium.com/migrating-kubernetes-clusters-with-velero-62bad36353ca?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/62bad36353ca</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[velero]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Wed, 27 Nov 2019 23:24:32 GMT</pubDate>
            <atom:updated>2020-03-19T14:12:21.208Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*n3IzyEdMq7aiMIEGaKziaQ.png" /></figure><p>Velero can be used to migrate and restore clusters. Migrating to a new cluster is slightly more complicated as you have to bootstrap the minimum resources necessary and install velero correctly to perform a restore. Velero has instructions for migration here: <a href="https://velero.io/docs/master/migration-case/">https://velero.io/docs/master/migration-case/</a>.</p><p><strong>Installing Velero to an existing cluster</strong></p><p>First install the velero client with brew:</p><pre>brew install velero</pre><p>There are a few options you have for installing velero to an existing cluster. <a href="https://velero.io/docs/master/install-overview/">https://velero.io/docs/master/install-overview/</a>. You can also install the velero helm chart. <a href="https://github.com/helm/charts/tree/master/stable/velero">https://github.com/helm/charts/tree/master/stable/velero</a>. You will need to setup an aws bucket and iam role and pass those into velero. You will then need to add a backup schedule for velero which will send backups to the configured bucket. Confirm that you have buckets being created and synchronized to your aws bucket: velero backup get</p><p><strong>Bootstrapping a new cluster with kops</strong></p><p>First use kops to create a new cluster identical to the cluster you are migrating from: <a href="https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_cluster.md">https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_cluster.md</a></p><pre>kops create cluster \</pre><pre>--cloud aws \</pre><pre>--image &quot;KOPS_IMAGE&quot; \</pre><pre>--target=terraform \</pre><pre>--kubernetes-version=KUBE_VERSION\</pre><pre>--out=./terraform \</pre><pre>--name=kube-test.domainname</pre><p>To boot up your cluster apply the changes in the terraform directory.</p><p>Once this is done, you will need to have kops export the kubeconfig specs for you to log into the new cluster:</p><pre>kops export kubecfg kube-test.domainname</pre><p>You should now have a new cluster you can access by switching your kube context.</p><p><strong>Installing Velero to the new cluster</strong></p><p>When you setup velero in your original cluster, velero created a secret called cloud-credentials in the velero namespace. You will need to extract the AWS creds to be able to access the velero backup bucket in your new cluster. Set your kubecontext to the cluster you are migrating from and extract this secret.</p><pre>kubectl get secret cloud-credentials -n velero -o yaml — export &gt; secret.txt</pre><p>Switch your context back to the the new cluster, and run the following:</p><pre>velero install \</pre><pre>--secret-file=secret.txt \</pre><pre>--provider aws \</pre><pre>--backup-location-config region=AWS_REGION \</pre><pre>--snapshot-location-config region=AWS_REGION \</pre><pre>--plugins=velero/velero-plugin-for-aws:v1.0.0 \</pre><pre>--bucket VELERO_BUCKET</pre><p>At the time of this writing, it is required to include a plugin when doing an install. You will need to set the VELERO_BUCKET to the bucket where your backups are being stored to from your original cluster. The snapshot-location-config region is needed to ensure that persistent volumes can be created by velero, else you will get an error where velero restores the persistent volume claims but does not create persistent volumes for them.</p><p>Finally remove the secret file rm secret.txt</p><p>Check that velero is up and running:</p><p>kubectl get pod -n velero</p><p>Check that you can see the existing backups for your existing cluster</p><p>velero backup get</p><p><strong>Restoring the backup:</strong></p><p>In your new cluster, pick out the snapshot you need with velero backup get or just pick the latest snapshot and create a restore from it:</p><p>velero restore create — from-backup `velero backup get | awk ‘FNR==2 {print $1}’` — exclude-namespaces “external-dns,velero”.</p><p>Check that the backup went according to plan by describing the backup. Additionally make sure velero has generated persistent volumes for you when it restored your PVC’s.</p><p><strong>Removing the Cluster</strong></p><p>To remove your cluster run kops delete cluster CLUSTERNAME — yes. Allow enough time for kops to remove resources.</p><p><strong>Notes</strong></p><p>In the backup, you exclude the external-dns and velero namespaces.<a href="https://github.com/kubernetes-sigs/external-dns"> External-dns</a> is a popular kubernetes add-on that will update route53 entries with your ingress routes. Including external-dns will cause all route53 entries for ingresses that external-dns manages to be overwritten to point to the new test cluster instead of your old cluster. Including velero will clash with your existing velero installation.</p><p>Your new velero installation uses the same backup location as your old cluster, you may wish to create new settings with velero backup-location create and velero snapshot-location-create and or create schedules for backups with the new cluster. Note if you want to override the default setting you will need to delete the CRD and recreate it: kubectl delete BackupStorageLocation default -n velero and kubectl deleteVolumeSnapshotLocation default -n velero</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62bad36353ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to automate windows applications with AutoGui]]></title>
            <link>https://lundbird.medium.com/how-to-automate-windows-applications-with-autogui-626c7b452eed?source=rss-a3862376a8d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/626c7b452eed</guid>
            <category><![CDATA[gui]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[windows-10]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[automated-testing]]></category>
            <dc:creator><![CDATA[Alex Lundberg]]></dc:creator>
            <pubDate>Fri, 07 Sep 2018 20:17:12 GMT</pubDate>
            <atom:updated>2018-09-10T01:37:49.273Z</atom:updated>
            <content:encoded><![CDATA[<p>A quick peek at stack overflow shows that there is a lack of simple tools for windows automation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wVZu7dYVXZqwxMZJTR47bQ.png" /></figure><p>Some common tools for windows automation are Sikuli, CodedUI, PywinAuto, and PyAutoGUI. Each of these have their own issues, whether that be complicated syntax for CodedUI, speed for PywinAuto, image recognition for Sikuli, or flexibility for PyAutoGUI.</p><p>Introduce AutoGui. An open-source Python and C# package for windows automation that automates using window properties.</p><h3><strong>Installation</strong></h3><p>You can install AutoGui via pip.</p><pre>pip install autogui</pre><h3><strong>Demo</strong></h3><p>For this example I am going to use the tool RecorderSpy available <a href="https://github.com/lundbird/Recorder-Spy">here</a>. To use the tool, clone the repo and launch the executable.</p><p>Getting started with AutoGui is easy. Lets automate the windows calculator.</p><p>Launch RecorderSpy and start the recording with Ctrl-R. Record a click on a control with Ctrl-W. When you are finished, end the recording with Ctrl-E.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*stG0ZOTWM9jCwOFakQuHxw.gif" /><figcaption>Generating and running a simple script using RecorderSpy and AutoGui</figcaption></figure><p>Note that RecorderSpy is just a generic windows spy tool. You can use any other tools you like. Another good option is inspect.exe that comes bundled with visual studio.</p><p>Lets take a look at the generated Python script.</p><pre>from autogui import *<br>setWindow(&quot;Calculator&quot;)<br>click(&quot;id:=num1Button&quot;)<br>click(&quot;id:=num2Button&quot;)<br>click(&quot;id:=num3Button&quot;)</pre><p>Here we set the active window using the title of the application. Then we select what control to click.</p><p>The syntax for the commands is simple, and you can easily write and edit scripts without the recorder.</p><p>Lets do another example to show how to write to a control using notepad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cdRe0hXjqDB_K1nqvQavcA.gif" /><figcaption>Writing to a control using AutoGui</figcaption></figure><p>The script used to generate the automation:</p><pre>from autogui import *<br>open(&quot;notepad&quot;)<br>click(&quot;File&quot;)<br>click(&quot;Text Editor&quot;)<br>write(&quot;This is AutoGui &quot;,&quot;Text Editor&quot;)<br>append(&quot; writing, appending, and sending keys is easy!&quot;,&quot;Text Editor&quot;)<br>sendkey(&quot;{ENTER}&quot;)<br>read(&quot;Text Editor&quot;)<br>close()</pre><p>We opened notepad with autogui’s open command which automatically sets the active window to notepad. Write() sets the value of the control to the entered text. Append() manually sends each key to the control. A list of keys for sendkey is available <a href="https://pywinauto.readthedocs.io/en/latest/code/pywinauto.keyboard.html">here</a>. Read() returns the text in a control and can be useful for assertions in GUI testing.</p><p>This script was very fast. As is common with GUI automation, you may have to add sleeps to get the desired functionality.</p><p>Lets check out the control properties using RecorderSpy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*-twSWquDsgidRLsf6G_TjA.png" /><figcaption>RecorderSpy shows all the properties that can be used to select an element and can record scripts of actions</figcaption></figure><p>Using the control properties, lets make a python script to automate both the calculator and notepad.</p><pre>from autogui import * <br>open(&quot;calc&quot;)<br>open(&quot;notepad&quot;) <br>write(&quot;this is autogui&quot;,&quot;Text Editor&quot;)<br>setWindow(&quot;calculator&quot;)<br>click(&quot;One&quot;)<br>close(&quot;Untitled - Notepad&quot;)<br>close()</pre><p>Lets run the script</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*-2lN34IxR91oc-eIJJMliQ.gif" /><figcaption>Running multiple applications with AutoGui. To switch to a new window use setWindow().</figcaption></figure><p>Running multiple applications can be done easily so long as you set which window you want to operate on using setWindow().</p><h3><strong>FAQ</strong></h3><p><strong>I can’t find a locator to select the control.</strong></p><p>This will happen sometimes. You will have 20 dialog boxes all with the same characteristics. In this case you can match on all 20 dialog boxes and select which one using the child parameter which is set to 0 by default.</p><pre>autogui.click(“generic dialog box”,5)</pre><p>The above call selects the fifth dialog box with the title “generic dialog box”</p><p><strong>How do I select based on multiple properties?</strong></p><p>You can select based on multiple parameters by passing in parameters in the format: param1:=value1,param2:=value2,param3:=value3.</p><pre>autogui.click(“Editor,class:=EditorClass,id:=EditorID”)</pre><p>In the above, we match for a control with title of Editor, class of EditorClass, and id of EditorID. We do not need to specify the title property. This is assumed when you do not list the property type.</p><p>All windows automation properties are available for selection, although the most useful and common are title/name, class, id, value, and controltype.</p><p><strong>I get an exception after opening or closing a program.</strong></p><p>This can be the case if you have more than one instance of the program running. You can either remove the second instance, or call open() with the setactive parameter equal to false, then set the active window manually.</p><p><strong>The command times out before finding the control</strong></p><p>Each method has an option to adjust the timeout of each call which is set to three seconds by default.</p><p><strong>How does it work under the hood?</strong></p><p>AutoGui is a python port of a c# wrapper for CodedUI which is available as a nuget package as well. AutoGui uses windows automation properties such as title, id, class, value, ect to select controls to act upon. AutoGui searches the active window set using setWindow() or open() for a match on the entered properties.</p><p><strong>Using AutoGui for GUI Testing</strong></p><p>AutoGui is fully able to be used to test your GUI’s. Each time RecorderSpy runs, it also generates a robot-framework script which is a fantastic test acceptance framework tool.</p><p><strong>Source and Further Reading</strong></p><p>The project source with additional documentation is available <a href="https://github.com/lundbird/AutoGui">here</a> on my github.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=626c7b452eed" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>