At ThoughtWorks, we are passionate advocates for continuous delivery — we even wrote the book on it. Jez Humble, the co-author of Continuous Delivery (CD), and the initial product manager for GoCD described CD as

“Continuous Delivery is the ability to get changes of all types — including new features, configuration changes, bug fixes and experiments — into production, or into the hands of users, safely and quickly in a sustainable way.”

One of the primary reasons we understand the challenges behind continuous delivery, and how one may be lulled into thinking they are practicing it, is because we’ve been through this ourselves. This is a story of how we discovered that we could deliver more continuously when we made the decision to go open source in early 2014.

What is GoCD

GoCD is a free and open source continuous delivery server built by ThoughtWorks. It provides complex workflow modeling, advanced traceability and end-to-end visualization of your deployment pipelines. Built specifically to do continuous delivery, GoCD supports all recommended CD patterns out of the box.

The proprietary bubble

GoCD was originally created as a proprietary solution by ThoughtWorks around 2010. Between 2010 and 2014 we had a release cycle of three to four months. The product was being consumed by several enterprise customers and given their resistance to upgrade often, there was no impetus for the team to deliver change any quicker. We had automated test as well as an exploratory and regression test cycle and released high quality releases. Although we had never tried we thought that if we really needed to, we could deliver whenever we wanted.

Using CD to deliver an open source CD product

Going open source came with new expectations from a new audience. We were seeing rapid adoption from a new group of people who expected changes faster than our previous enterprise customers. And who were keen to see their contributions in production. At this point, we thought we were ready to deliver continuously, but we weren’t truly ready When we tried to reduce the release cycle to deliver once every month or month and a half we were challenged. In order to reach this goal we took these steps.

1. Gradually start automating manual parts of the deployment process first

Our first goal was to shorten our release cycle from 3–4 months to 1.5 months. When we looked closely at this process we could see some obvious areas to improve to begin with. For example, although we had a lot of automated tests to help with our regression testing we also had gaps. Our installer testing phase, which tested that each installer for each operating system was functional as well as our performance tests were not automated. To complete these phases manually took us weeks. Likewise publishing our installers at the end of the process was a manual process. Rather than try to change this all at once which was daunting we decided to break down the automation our entire deployment into tiny chunks. Each new release, we automated a new piece of the process, so that we gradually reduced our cycle time and released more often. This incremental approach allowed us to improve without taking too much on at the same time.

2. Find newer things to automate

As we progressed, we started finding newer things to automate such as our Release Notes. Since we were delivering an open source tool to the public, we needed to have reliable and sensible release notes for each release. First, we got everyone who contributed code (the team and external contributors) to write good commit messages and then automated release notes out of these. Each time we added a new automated step like this we made the progress quicker and more reliable.

GoCD deployments today

Our ultimate goal was to get to a place where we could automatically publish releases to production and we are still on that journey but much much closer than we were before. Today, we have automated our installer tests, performance tests, release notes etc., and are closer to this goal. While we make major version releases on a monthly basis, we deploy experimental releases on every commit.

You can take a look at our pipelines at https://build.gocd.org .

One of the real benefits of practicing CD has been that if there’s a problem with the release, we have the ability to release a new version or deploy a fix much more easily. Our pipeline is still not a 100% automated (e.g we manually curate some of our release notes to make them easier to read) and we keep looking for newer things to automate. However, the goal of continuous delivery isn’t complete automation but to ensure that changes get to users safely and reliably. To that end, our journey to continuous delivery is on-going.

Our advice for anyone else moving to CD

The two most important learnings for us in the quest to delivery continuously have been that:

Frequency reduces difficulty

The idea is simple — if you find something difficult, do it more often. If you can bring that pain forward, you will realize that there are a lot of costs and complexity that you can reduce. Every time we do something manual we ask ourselves the question “can we automate this?”

Automation costs pay for themselves

Many times teams will be too daunted to take on the task of automation or they will think that they don’t have time to automate. While there We used to have long cycles and at the end we still spent manual effort checking if all the tests had been carried out and correcting when anything had been missed. We don’t have to worry about that now. There is an initial cost at the beginning automation, but in the end, it definitely pays off. Our approach to incrementally pay-down our manual testing and manual release process allowed us to reduce the time to production as well as continue to deliver. Now we are able to release quickly and reliably whenever we want.

We discovered that our biggest challenge to going open source was our assumption that we were practicing CD when we really weren’t. Just like with CI theatre we recommend that teams challenge themselves to deliver faster, more often and more reliably in order to understand their opportunities to improve.

To get started with GoCD, you can start by downloading it for free. We recommend using our Getting Started Guide for guidance on CD concepts and how to build your deployment pipeline. If you need help with anything related to CD or GoCD setup, do reach out to our community.

GoCD has sponsored Rootconf 2018. The team is seeking feedback and adoption of their open source tool.

How we used Continuous Delivery to deliver a Continuous Delivery tool was originally published in Rootconf Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

During the 10 plus years ThoughtWorks has been in the Continuous Delivery (CD) ecosystem, we’ve regularly come across people wanting to try our tools (GoCD and Snap CI) as they start their journey toward CD. Very often, in attempting to support teams new to CD, we suggest that they pause any tool evaluation and consider whether their organization is actually ready to embark on this journey. If you do not frankly assess your team’s readiness, the result can be a massive failure. The path to CD should not start with the immediate adoption of a CD tool.

In part one of this series, we explored some core development practices that are prerequisites for CD. In this part, we’ll look at a variety of feedback loops — both manual and automated — your organization should have in place before rolling out CD.

Feedback loops and Continuous Delivery

The aim of Continuous Delivery is to release software faster, more reliably, and more frequently. Given that, diagrams of CD typically depict a linear flow. On the surface, this is quite different from Continuous Integration, which is usually shown as a loop.

But CD as a linear flow is an incomplete picture. A good deployment pipeline has numerous feedback loops along the way. At each stage of the pipeline, verifications are run. If they pass, the pipeline continues. If they fail, the pipeline halts and the team responds appropriately to the feedback. The feedback along the way prevents CD from being chaos. Poor quality will almost never reach production in a well-designed pipeline.

Most of the feedback loops you find in a deployment pipeline are good practices in and of themselves. You might already be doing some or most of them. We think you should have many of these in place before moving forward with CD.

Test automation

The most common feedback loop in any deployment pipeline is the execution of automated tests. You must have a solid test automation strategy before attempting CD. Some people like the approach of the test pyramid. We’re actually fine with any sensible approach, as long as it’s fast and reliable. There are myriad types of automated tests, and which ones you use will depend upon your circumstances. Here, we will take a look at three of the most important types: unit, regression, and performance.

Unit tests

Unit tests verify your application at the most granular level, typically methods or classes. They are fast, easy to maintain, and support rapid change of your application. Unit tests should be the foundation of your automation strategy. If your teams don’t value a thorough and fast unit test suite, they won’t be able to move fast or with confidence.

Some points to consider when assessing your team:

• The suite must be fast. What’s fast? A few minutes on a large code base is OK. But faster is better. Slow unit tests result in a slow, horribly frustrating development flow.
• On a mature team, the testers will be comfortable with pushing as much of your test automation as possible into your unit test layer.
• Code coverage is important, but tracking metrics is generally only beneficial for a team learning the basics.
• Some frameworks and platforms are known to be slow when it comes to unit tests. Do not fight or subvert a framework to make tests fast. Instead, consider switching your framework or platform.

Regression test suite

A regression test suite verifies that your entire application actually works. This suite adds a ton of value to a deployment pipeline. For many, the regression stage of a pipeline gives the confidence needed to deploy. We want to make a couple of points about this.

Firstly, regression tests should be 100 percent automated. They are change-detectors and do not require brain power to execute. A manual regression stage in your deployment process will prove painful. Work to get rid of it. Your testers can add more value elsewhere.

Secondly, we reject the notion that a regression suite must mean slow, flaky Selenium tests. Our take is, yes, it’s a fair reputation, but it was earned by many teams doing it wrong. How to author and maintain an automated regression suite is a book-worthy topic, but quickly:

• Don’t couple them to small stories or tasks. Only consider them in the context of the entire application.
• Relentlessly prune the suite. Keep it tight. Err on the side of leaving something uncovered rather than accepting duplication.
• Treat them as production code. Keep things very clean.
• Have programmers write them. Train your testers to code if they are interested in automation. Avoid drag-and-drop programming.
• Do not accept flaky tests. Fix them or get rid of them.
• Even the best suites we’ve seen tend to be slow. Embrace using some combination of hardware, virtualization, and cloud to parallelize execution.

One caveat to note: the best testers will want to do a manual regression every so often, just to help structure how they think about the application. That’s a good thing so long as it’s about their being thoughtful and not how you actually integrate regression checks into your process.

Performance testing

Performance testing — verifying that your application meets specific performance criteria — is a massive topic. There’s no one way to do it: your approach will vary according to request volume and data size. There are many varieties: load, stress, soak, and spike, to name a few. It’s too big a topic for this post. That said, we do have some thinking that can help you assess your maturity:

• Do not leave this phase for last. We cannot stress enough just how difficult this practice is and how much time we’ve seen sunk into failed efforts. Everything about it is difficult: modeling, standing up an environment, building the harness, assessing results, building it into your deployment pipeline.
• It’s critical to test against specific criteria. Don’t worry about getting your criteria wrong at first. You can change the criteria once you have real production data.
• Don’t assume you’ll reach web-scale in month two. You’ll waste huge cycles prematurely optimizing both your application and your tests. (Don’t read this as us saying, “Don’t consider what your actual scale might be.” Just a suggestion that you be realistic and pragmatic.)
• Utilize production monitoring to the greatest extent possible. A canary release can go a long way toward verifying the performance of a new version of your application.

Production monitoring

Do you have a production monitoring strategy? Feedback loops aren’t only for pre-production phases. As much as we try to achieve dev/prod parity, production is truly a unique environment for most organizations and things can — and do — go wrong.

Here are some questions to help you assess your readiness:

• How quickly does your team know something is broken? Do they learn about it via monitoring? And then how fast can they respond?
• Does your team ignore alerts?
• Do your teams tend to invent an approach to monitoring as they go along? Believe it or not, we actually see this a lot and it’s not a good thing. Be as thoughtful about monitoring and alerts as you are with other parts or your application.
• Do you keep a database of events so that you can later query for patterns?

User testing

Sitting users down in front of your application and try it out is a critical feedback loop. In an enterprise setting, we like to see two type of user testing: usability testing and user acceptance testing. Usability testing verifies that users find the application easy to use. User acceptance testing verifies that users can complete transactions with the application in a real-world setting. There can be fair bit of overlap between the two types.

If you do not do user testing, you will struggle getting users to accept frequent releases of new versions. Users will only like rapid changes if the experience remains usable, consistent, and effective.

We also want to call out that these feedback loops are manual processes that often require weeks or months of elapsed time. They are typically not modeled into the deployment pipeline, and that’s fine. But do not leave them both batched up until the end. That’s a long wait period and likely an unknown amount of rework before deploying to prod. If you do this, your process will feel more like waterfall than CD. Run these user tests early and often, while you are writing the code.

Exploratory testing

All this talk of automation does not mean your testers should retire their analytical thinking and learn to program. In fact, test automation should free up your testers to do what they’re best at: use their brains. Exploratory testing is when a tester is simultaneously learning about the system, designing tests, and executing tests. It is when a tester gets into a deep flow, not even knowing what the next test is. This is where a good tester can really shine, doing some of their most valuable work.

For most types of applications, a test strategy should include skilled testers performing exploratory testing. This testing will find problems, teach you about your system, and inform your automated regression suite. As with user testing, this testing should be done throughout the development process and not as a gate at the end.

Summary

This list of feedback loops organizations should have in place before doing CD is not exhaustive (e.g., we didn’t discuss A/B testing). We have presented the more common feedback loops we see where CD has been successful. Obviously every situation presents different problems and has different needs.

You don’t need to have high marks on everything we have presented to begin your journey to CD. But if you are feeling only so-so against a majority of them, we’d suggest working on the individual pieces before approaching CD. Once you get enough of them in place, you will find that you’ve actually completed a large swath of your journey to Continuous Delivery.

In future parts of this series, we plan to explore culture, the last mile, and more.

Are you ready for Continuous Delivery? (Part 2: Feedback Loops) was originally published in Continuous Delivery on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the first in a series of articles that will cover the types of pipelines you should implement to ensure your software is truly ready for production at any time. The culture changes required in most organizations are incredibly important, but I’m going to focus on some technical practices in this series.

Years ago, when I was in management, I had a favorite rule. If asked “is something done?” the answer could not include the word “except” or any of its synonyms.

“It’s done except for…” = “no”.

I hear people say all the time that they’re practicing continuous delivery. This declaration is often followed by something like, “I can let the security team know anytime”, or “I just have to run the performance tests”. If you can’t push your software to production right now, you’re not done with your continuous delivery journey.

Some of the things you might not be running but should …

In this article I’m going to give an overview of some of the types of pipelines that you should be running if you want your software to be ready to ship at all times. Of course this is not an exhaustive list, there are most likely things that are specific to what you’re doing that you should have, just as there are probably things that I will list that don’t make sense for you. The point is that everything possible should be automated as part of your deployment pipeline.

Security testing

All too often this is the primary category of tasks that don’t get run until everything else is “done”. This often results in issues that are very hard to track down since the time between tests has been very long. By writing these tests all the time you’ll have a much easier time tracking down issues before they become too hard to fix.

Many people feel it’s not the greatest idea to have the same person writing the security tests who is writing the code. There’s also the question of skillset; great security people are not common. It’s important that you use a Continuous Delivery server that is capable of using more than one build material for a single pipeline. That way these tests will run whenever the code or the tests are updated.

Performance testing

This one is probably the hardest to run all the time if for no other reason than hardware costs. To properly performance test many applications takes a serious dedication of resources. Luckily public and private cloud infrastructures have made this somewhat easier. Consider having a pipeline where the first stage spins up the machines you need either as virtual machines or containers, runs the tests, and then shuts down those machines.

In this day of “search for a term, hit the link, wait no more than 2 seconds for the page to load” performance is critical. To make matters worse, performance issues are often very hard to track down. You want to know as soon as possible if you’ve introduced a problem.

Management of the environments

It’s been said many times that it’s much easier to break an application by messing up the environment that it is by doing something wrong in the source code. If something like the security advisory comes out and you need to update systems as soon as possible, you should be able to commit the change to a configuration management tool, have that change picked up by your continuous delivery system and run it through exactly the same process as a code change.

Testing of the deployment itself

This isn’t really a type of pipeline all by itself. This is the concept that you should be deploying the software exactly the same way in every environment that you plan to deploy in production. Unfortunately it’s still not uncommon for people to copy over files to a QA server run test and only then run the actual deployment tool that pushes the same software to a production server.

No matter how you’re doing your actual production deployment, whether that is shell scripts, dedicated tools, configuration management tools, or others, you should be deploying in exactly the same way everywhere else. Consider using tools that can read environment specific details from environment variables or other inputs.

Why wouldn’t you do this?

One of the biggest objections I hear to running all of these types of pipelines on every change is that the pipeline will take too long to run. This is why having a continuous delivery server that’s capable of running multiple pipelines in parallel while ensuring that software doesn’t go any further if any of those pipelines fail is so important.

The other objection I hear the most is that people simply lack the automation around these areas. This is certainly valid, and I don’t want to pretend that any of this is easy to do. Don’t be afraid to start with what you can, and then add other things your pipeline as your capabilities grow. A continuous delivery pipeline is a bit of a living system it should be evolving along with your processes.

It’s not Continuous Delivery if you can’t deploy right now was originally published in Continuous Delivery on Medium, where people are continuing the conversation by highlighting and responding to this story.

Testing open source projects has its own set of problems. Unit and integration tests are added by the contributors, however end to end tests are added later or neglected. There are several reasons for this like not being aware of the end to end flow, complexity of automating tests for end to end systems. However to be confident about a release we need end to end tests. To maintain the healthy balance defined by a test pyramid, every release must have its system test cases reviewed and maintained and this needs collaboration.

At GoCD we are have open sourced our functional tests. We believe open sourcing system tests will — improve quality through collaboration, make regression tests efficient, give the community a detailed picture of the system behavior and act as a live document for users.

How GoCD is continuously tested

Let’s take a brief tour of how GoCD does continuous testing. GoCD is built, tested and deployed using GoCD itself. Here’s a snapshot of the GoCD build pipeline.

GoCD is tested using “Server in-the-box” model. In this model we define the system under test — its scope and its boundaries. External dependencies for the system under test (SUT) say git or yum repo are stubbed out or built into the SUT itself. The pipeline “qa packages” does the necessary jobs to bundle the stub code with production code, and then pass on the packages to “smoke” and “acceptance” pipelines to run the tests. Upon successful runs of the upstream pipelines, “distributions-all” pipeline packages only the production code.

Test Classifications

Below are the set of functional test types modeled into the GoCD pipelines. Each of them are run on single or multiple agents as per the size of the tests. Every agent will be cleaned of any previous versions of the build when a new build is started.

• Smoke: As per the classic definition “smoke tests” validates the basic functionality of the system under test. It runs on 2 GoCD server agents — one on windows and one on linux
• Acceptance: These tests validates the necessary set of tests for each functionality or use case flow. It does not execute every scenario in the flow, but validates all possible flows. These tests are run on a total of 17 agents. Smoke and Acceptance are triggered by the upstream pipeline ‘qa packages”.
• Regression: A detailed set of tests which is exhaustive and comprehensive. This job is executed parallel in 17 agents. Regression tests are triggered when Smoke and Acceptance pipelines pass.
• Performance: This pipeline is under development, will discuss this on a separate blog

How this model helps continuous delivery

Continuous delivery depends on a faster feedback loop. However Functional and Performance tests end up being bottlenecks. This can be reduced to a great extent by distributed tests.

Though not the only solution, it’s a first step to achieving the faster feedback loop. Further enhancements and optimizations are on the way.

Twist to Gauge Migration

GoCD functional tests were initially developed using ThoughtWorks Twist automation tool. Since Twist has hit end-of-life as a product and is being replaced by Gauge, we’ve migrated our tests to Gauge. Moving to Gauge now helps us open source these tests (Twist was a commercial product).

Twist to Gauge migration is handled by the migration plugin bundled in Twist.

Gauge Specifications

Gauge specifications are in markdown language making it more readable. Gauge specs intend to be live documents of the system behavior

Gauge Reports

Gauge produces reports in an easily readable format and provides specification execution details like steps executed, time for each spec execution, overall test execution time, steps and specs passed, what failed and the screen shot of the failure page.

Conclusion

A distributed test environment and parallel execution are ways to support CD, by making long running tests short and getting faster feedback. Here we have addressed functional tests, we will soon be working on the performance tests.

If you would like to contribute to GoCD tests, please visit GoCD Functional Tests for contribution.

Why we open sourced our Functional Tests was originally published in Continuous Delivery on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a section called Principles and Practices of Build and Deployment Scripting, the CD book suggests to Use Your Operating System’s Packaging Tools.

“Deploying a bunch of files that need to be distributed across the filesystem is very inefficient and makes maintenance — in the form of upgrades, rollbacks, and uninstalls — extremely painful. This is why packaging systems were invented.”

We concur. This is a good example of cross-pollination of good ideas between development and operations. Software development teams are increasingly packaging their applications using application-specific (maven, gem, wheel, npm,cpan…) or system (rpm, deb, nuget, chocolatey…) packaging tools. Doing so has several advantages:

1. Built in versioning (e.g. rpm, deb, nuget).
2. Dependency management.
3. Distribution via standardized repository layouts, metadata and protocol.
4. Transactional install/upgrade/uninstall via package managers.
5. Managing configuration files: OS packages know how to manage files that are expected to change after installation so you don’t lose your changes when you uninstall or override when you upgrade. e.g. rpm’s %config macro.
6. File verification: Packages have built-in checksums, which can be used for verification before a package is extracted and installed.
7. Signature verification (e.g. rpm).
8. Idempotence — Issuing the same package install/upgrade/uninstall multiple times leaves the system in the same state as issuing it once.
9. Auditability: OS packaging tools allows you to track what package installed what file in your system (versus files appearing or being changed at will)
10. Traceability — what repository did a package come from? (e.g. deb)
11. Infrastructure automation tools like Chef, Puppet and Ansible have good support for popular package management systems. Traditionally used for provisioning servers, and environments, these tools are increasingly being used for application provisioning — more on this point in a follow-up post.

There you go — eleven good reasons to start creating packages for your components and applications and deploying from packages rather than from a tag, branch or binary.

Thanks to Danilo, Ram for their insights on this topic.

Deploy a package, not just a tag, branch or binary was originally published in Continuous Delivery on Medium, where people are continuing the conversation by highlighting and responding to this story.

[Update: It might be better to use the GoCD Docker images available on Docker Hub as they are newer and better maintained and used by many in the GoCD community]

I’m convinced that the ability to automatically create and destroy build agents is key to having a flexible continuous delivery setup. I’m equally convinced that this particular use case is better satisfied using very fast container technologies instead of virtual machines. This is blog entry talks about using Docker for this challenge.

IMPORTANT: It should be noted that this blog entry and code is the result of a spike. This definitely isn’t production ready, but could be useful if you’re thinking about doing the same sort of thing. This will only work on Linux machines, because Docker only works on Linux machines.

If anyone with more current programmer skills wants to fork and make it more production like, I’d be very happy to assist in any way I can.

Setting it up

• Install Docker
• Install GoCD server
• Install at least one GoCD agent if you want to use the pipeline — Yup, even though the whole idea here is to create and destroy agents automatically, we have to install at least one to run the process. The server needs this to actually run the scripts.
• Once the GoCD agent is installed, go to the Agents tab in your GoCD user interface and assign this agent a resource of “manager” if you intend to use the GoCD pipeline below.
• In order for GoCD (and anyone else) to run this without having to become root, you’ll have to follow the instructions under “Giving non-root access” at http://docs.docker.io/installation/ubuntulinux/. I make absolutely no warranties for the security of following those instructions! Again, the point here (for me) was to spike out creating agents on the fly, this is not production code.

The pieces

All of the scripts I’m using can be found here . Please feel free to clone / fork / send pull requests.

• Dockerfile

This is the file that defines how the docker image will be created. Normally you would run the docker build program against this file, but I used a shell script so that I could call it from GoCD easier later.

• go-agent-14.1.0–18882.deb (no longer included)

Update on 22 June — The Dockerfile has been updated to use wget to download the GoCD 14.1 agent debian installer. If you need the RPM version visit www.go.cd/download and change the Dockerfile to match your system.

• go-agent

You’ll need to modify this file to set the IP address of your server. Just to make this more interesting, it needs to be the IP address that Docker is aware of, so probaly not the one you’re used to using.

You can find this IP address using ifconfig…

ifconfig | grep -A 1 docker

On my system, that returns…

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
		  inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0

Because that’s my address, that’s what you’ll see in the go-agent file. As much as I might appreciate your help in supplying agents for me, they won’t be able to connect if you don’t make this match your GoCD server.

• autoregister.properties

This file takes advantage of the autoregister feature in GoCD so that I don’t have to manually approve the new agents. You’ll need to update the key based on the instructions here.

• build_docker_image.sh

Once you’ve modified the above, you’re ready to build your containers. Execute this script at a command prompt.

This step could take quite a while depending on the speed of your internet connection. It has to download the base box, add Java and a bunch of other things and then create the image you’ll be using.

• start_agent_containers.sh

This shell script starts docker containers from the image. You can control how many containers get started by editing the script itself.

• stop_agent_containers.sh

This shell script stops and removes all docker containers — WARNING: I do mean all. If you run this on your box every single Docker container will be stopped and deleted. Did I mention this will remove every single container on your machine?

If someone that’s better with sed / awk / ? would like to modify to only remove the containers tagged with gocd/go-agent please feel free!

• remove_agents.rb

This Ruby script removes disables and removes all agents on the GoCD server tagged with a “docker_created” resource.

• go_config_xml_excerpt.txt

Everything above can be run from the command line just fine. Of course I wanted to have a GoCD pipeline manage all of this. This text file shows the pipeline definition for mine. You’ll need to change the github URL to point to your own. You’re welcome to use mine from above, but I removed it here just so your pipeline doesn’t get executed with my GitHub changes if you don’t want it to.

Summary

It’s my sincere hope that my first attempt to actually use Docker (and Ruby for that matter) will help you get something real done.

1. Unit tests
2. Static and dynamic metric thresholds (e.g. toxicity, coverage)
3. Packaging
4. Integration tests
5. UI tests
6. Performance tests
7. Regression tests
8. Deployment tests (install, uninstall, upgrade)
9. Manual exploratory tests
10. Regulatory, compliance checks
11. Clearance from UAT, staging

Each of these hurdles may be viewed as a promotion level for the build (although, in practice, formal promotion levels are more coarse-grained). These promotion levels may be activated automatically (by the success of the previous level) or manually (by approved promoters). Note that these levels are very different from what is encountered in a source promotion model.

Avoid source or stream promotion models

In an earlier post, we said: “Deploy packages, not just a tag, branch or binary”. With respect to promotions, we could say:

“Promote packages, not just a tag, branch/stream or binary”

Promoting at the level of source (rather than package) violates the build-once rule — one of the key principles outlined in the CD book:

“…the binaries you release should be the same binaries that have been through the rest of your deployment pipeline, so you can be sure that what you release is what you tested. Apart from the fact that nobody has tested the binaries that came from the release stream, there is also a chance that differences could be introduced in the build process, perhaps by the operations team using a different minor revision of the compiler or a different version of some dependency. Such differences can lead to bugs in production that take days to track down.”

Because of this, a source/stream promotion model is fundamentally unfit for the world of continuous delivery:

“One of the objectives of the deployment pipeline is to allow frequent check-ins to mainline on large teams which may result in temporary instabilities, while still allowing you to get rock-solid releases out of the door. In this sense, the deployment pipeline is antithetical to the source promotion model. The main advantage of the deployment pipeline lies in the rapid feedback you get on the effect of every change on the fully integrated application — something that is impossible in the source promotion model.”

Use your CI/CD server to manage promotions

Source based promotion models probably made more sense in a pre-CD era when the version control system was the only source of truth for the level of readiness of a codebase. Today, the CD tool is a more appropriate system of record for builds (and deployments) and their promotions. Again, the CD book backs this up:

“The other essential facility that must be provided by the tool you use to manage your deployment pipeline is the ability, for each stage, to see which builds have passed all the previous stages in the pipeline and are hence ready for the next stage. It should then be possible to choose one of these builds and press a button to have it deployed. This process is known as promotion.

Promoting builds at the press of a button is what turns the deployment pipeline into a pull system, giving everybody involved in the delivery process the ability to manage their own work”

Model your promotion process into the CD tool

In general, a build/deployment pipeline is also a promotion pipeline. GoCD’s support for pipelines and stages with automatic or manual approvals makes it easy to model and visualize your promotion process into the CD value stream. A stage runs only when the previous stage has passed — this is like an automatic promotion gate. Manual stageapprovals may also be used where needed.

Modeling promotion levels (P1…P4) itno the CD value stream

GoCD 13.3 (and up)

GoCD adds more support by letting you store your packages in a first-class external package repository and yet activating different promotion levels within the tool. We now have three ways of activating a promotion level manually or automatically:

1. Stage n activating stage n+1
2. Pipeline X activating Pipeline Y (pipeline dependency)
3. Pipeline Y getting activated by the updation of a package in a package repository (package material dependency)

There are situations when we could use either #2 or #3 above to model a promotion. The package material docs has a section on this.

Support for package material has been implemented as an extension point. Plugins written to this extension point can provide support for different types of repositories. A yum plugin is bundled along with GoCD Server 13.3 (and up). Other non-bundled package material plugins are listed here.

Happy packaging and promoting!

Posted by Sriram Narayan. This article was published on GoCD’s blog.

A common workflow to fail fast

Here’s a generic but common Build-Test-Release workflow. Each code commit will trigger a new release candidate and run through the gauntlet, left to right. Every workflow step is tasked to find an issue with the version of the software it is building, testing or deploying.

To shorten the feedback loop:

• The activities within each workflow step can run in parallel.
• The workflow steps are sequenced to run the most important things first.
• Four build pipelines can run in parallel.
• Four acceptance test pipelines can run in parallel.
• Six different test environments can run in parallel.

Know what’s going on in a workflow

It’s important a team has a holistic context for every release candidate. Every tool talks about “visibility” and “traceability”. A tool that can easily answer these questions provides visibility and traceability:

• Which dependency or commit broke the build?
• Who broke the build?
• What’s the difference between these 2 builds?
• Which features and bug fixes are in this release candidate?
• Which version of our software is running in this environment?
• Who deployed in this environment and when?

How will modeling a workflow help a team fail fast?

Let’s say a release candidate needs to pass 11 steps (gauntlet) before it can make it to an enduser. Sequence is important. The goal is to fail fast. A flat workflow will return feedback but it’s slow since only one step will run at a time.

If it turns out that a few steps can be run in parallel, you will speed up your workflow.

Speed things up even further by increasing the amount of parallelization.

Over time you may find other areas in your Build-Test-Release workflow where you can speed up your feedback loop. This workflow will find an issue much faster than the flat one we started with.

Sequencing and parallelization in GoCD

A GoCD pipeline is a building block (*). Teams using GoCD will come together and model an end-to-end Build-Test-Release workflow by chaining pipelines together. Part of this collaboration is figuring out the sequence of workflow steps and where parallelization makes sense.

Sequencing workflow steps will define priority:

i) Execute the most important activities first:

• Automated acceptance tests before performance tests.

ii) Execute the shortest activities first:

• Unit tests before smoke tests.
• Smoke tests before automated acceptance tests

Parallelization will speed things up:

i) Parallelize activities within a workflow step:

• Speed up an acceptance test suite.

ii) Parallelize different workflow steps:

• Run acceptance tests, performance tests and exploratory testing at the same time.

We treat a workflow as a single entity

It’s extremely important to us that the core behavior of a single pipeline is maintained in a workflow (chain of pipelines), no matter the model. GoCD allows teams to model as they wish and handles stuff like artifact management and if a release candidate should advance to the next step. A lot of the intelligence comes down to how GoCD manages dependencies. We’re proud of this intelligence, especially the “fan-in” support. This is where GoCD really shines.

(*) Jez Humble and David Farley talk about a “deployment pipeline” in their book, Continuous Delivery. A core domain concept in GoCD is called a “pipeline”. They are not the same thing.

• The book talks about the “deployment pipeline” in a holistic sense, a pattern that can be implemented to model your path to production. They summarize a deployment pipeline as “in essence, an automated implementation of your application’s build, deploy, test, and release process”.
• GoCD is a Continuous Delivery tool that allows teams to implement that pattern via GoCD pipelines. A GoCD pipeline is a building block. GoCD pipelines are a way to implement a deployment pipeline. The product overloads the term. On behalf of Jez, sorry!

Posted by Mark Chang. This article was published on GoCD’s blog.

With a traditional release structure all features and bug fixes are released at once, so it is an easy task to determine which features and bug fixes would be released. When we moved to a schedule that pushed new features and bug fixes into our cloud environment as soon as they were ready, we found ourselves with a new crop of questions on our hands.

Tracing features and bugs through a non-trivial build setup was not as easy as it should be

What features and bugs will be in tonight’s deployment to production?

When can we start to dogfood this new feature in our staging environment?

That bug fix we put in, where is that in our pipelines?

Are we going to start testing it today?

I find myself answering or asking some variant of the questions above multiple times a day. Either because I’m eager to start testing a new fix, or because I want to help set our product managers expectations properly, or because we want to know when a customer will be happy because we’ve responded to their feedback, or… you get the idea.

We have multiple pipelines in our path to production. For legacy reasons we have pipelines that run tests using different browsers, databases and operating systems. We also run upgrade tests and performance tests at different frequencies based on need. Since we have a number of pipelines chained together it isn’t always easy to know where a given change is in our deployment process. Consequently, it is difficult to answer questions about what features are in a given build or when a bug fix will be available for testing.

GoCD’s value stream mapping feature

To answer each of the above questions, we now refer a lot to the new value stream mapping feature in GoCD. From any build we can now zoom in to find upstream and downstream pipelines and basically trace a single revision all the way through to production. Below is an example of the overview for a recent build that we are waiting to see promoted to our staging environment (called pasty).

The map above shows the key pipelines involved in the flow from our tests using Firefox and Oracle 11g all the way through to testing our hosting environment. Once the ‘mingle-saas’ pipeline is green I know that the targeted bug fix will be ready for deployment to our dogfooding environment (called ‘pasty’).

Seeing the big picture improves understanding

To answer our key questions easily we needed a big picture view of our pipelines. We needed to know more about our pipeline dependencies and where are artifacts are flowing. These days I find myself using the value stream map multiple times a day and I use it in conjunction with our card wall to drive conversations with our product manager around which new features and bug fixes will be included in the next deployment and when that deployment is likely to be done.

One added benefit we’ve gained from the value stream map is that I now have a much better understanding of the different pieces of our deployment process. This has definitely helped in conversations with the developers:

Developer: “We just have to wait for the GreenInstallers build to pass…”

Me: “GreenInstallers… ah yes I know exactly where that fits in the overall process so I have a good idea about when that build will be ready. Thanks.”

I think it’s safe to say that the new value stream map is my new favorite feature in GoCD.

Posted by Scott Turnquest. This article was published on GoCD’s blog.

This is the first part of a series of posts about Continuous Delivery infrastructure, culture, and process. In this first post, we’ll present questions you need to answer honestly about your own people, teams, and organization to determine your readiness for Continuous Delivery.

Part 1: Practices

Do you put everything into version control?

A foundation of CD is the ability to put a specific version of your application into a given environment at any point in time. CD is actually quite fussy about this. It’s a must. And this can only be done by:

• Putting everything needed to make your application into a version control system
• Any time you change anything, push the changes to version control.
• Write an automated script that, given a version, checks out everything from version control and assembles your application.

CD is impossible when software teams (or the people on a single team) work in isolation from each other. When development work happens in isolation, large periods of integration and testing are required at the end of the development phase. This results in long periods during which your application cannot be released. Avoiding these lengthy integration phases requires that your work be as visible to, and usable by, others as soon as possible. Our preferred mechanism for doing this is called trunk-based development:

• Everyone regularly pulls others’ changes from version control.
• Everyone regularly pushes their changes to version control.
• Everyone works in the same place in version control, typically called “trunk” or “master”.

If you are new to trunk-based development, it might sound ridiculous. But the practices of CI, unit testing, and trunk-based development all playing together, not only makes it reasonable, but a truly pleasant way of working.We don’t have space here to go into the details of making trunk-based development work, but know that it cannot work without version control.

Even if you aren’t planning to do CD, start putting everything into version control now. And we mean everything. Not just your source code. Everything can include images, database scripts, tests, configuration, libraries, documentation, and more. Source code won’t be enough if you need to get back to a specific version of your application, infrastructure, etc. Also, this encourages your entire team — not just the developers — to collaborate.

Do your developers practice Continuous Integration (CI)?

For CD to be successful, the entire organization must trust that your software is high quality and always in a working state. In terms of development team practices, CI is the fundamental building block to achieve this level of trust.

So what is CI? Well, much has been written about CI, but here’s the TL;DR version:

• Developers check code into trunk/master multiple times each day.
• Developers maintain a suite of unit tests that verify the code works before merge, locally, and post merge, on an integration machine or CI server.

The end result is a development team that has high trust that the code in trunk/master actually works. This will leave the development team more willing to push code to testers, or even production, more regularly. With this in place, trust between the groups will quickly grow.

Our experience has been that development teams can only move quickly by combining unit testing, refactoring, and CI. That’s a discussion too broad for discussion here, but know that your teams will never deliver at a fast pace without CI.

If your developers are not practicing CI we would recommend putting your move to CD on hold and shift your focus entirely to supporting the adoption of CI.

Do you automate relentlessly?

CD is very dependent on automation. Automation everywhere is crucial to achieving trusted, one-click, low drama deployments. Manual processes are error-prone and do not lend themselves to repeatability. To practice CD, the entire team needs to get into the mindset of relentless automation of nearly everything.

This mindset means asking “Why can’t this be automated?” every time anyone on the team does anything manually more than once. Some components and aspects of your process that need automation are:

• Tests at different levels, such as unit, integration, UI, regression, security and performance
• Database schema creation, data migration and rollback
• Installer creation and signing (if you have them)
• Generation of documentation for every release
• Last-mile deployment of your application to any environment
• Provisioning of infrastructure all the way from test environments to production
• Provisioning of developer workspaces

We mentioned error reduction and repeatability. There are dozens of other compelling reasons to relentlessly automate. We have a few favorites:

• Scripts codify team knowledge. This ups your bus factor. That’s a good thing.
• It encourages consistency across environments. Once you have a script working in one environment, you’ll want to use it everywhere.
• The output of these scripts provides a detailed audit trail that is hard to match manually.

Relentless automation might seem daunting, particularly if you’re focused on something that truly cannot be automated, such as exploratory testing. In our experience, there will be many more parts of your process that can be automated than those that cannot. The best approach is to figure out the manual processes you are already using, and then make a plan to gradually automate them. As you begin to achieve small successes, you will want to automate more and more.

If your team is hesitant to automate and cannot be convinced about the need, you might have to consider if there’s enough maturity in your team to move toward CD. An automation mindset is a firm prerequisite for CD.

Conclusion

There are numerous small steps you can take early in your CD journey that will have immediate, positive impacts. Don’t waste time flailing around with CD tools if you’ve got a bunch of low-hanging fruit that can provide high value quickly. Also, this approach will set you up for success once you do feel the need to adopt an end-to-end CD tool.

In coming posts we will present similar sets of questions for you to consider in the areas of infrastructure, application design, process, and culture.

Posted by David Rice & Aravind SV. This article was originally published on GoCD’s blog.

Are you ready for Continuous Delivery? (Part 1) was originally published in Continuous Delivery on Medium, where people are continuing the conversation by highlighting and responding to this story.