<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Howard L on Medium]]></title>
        <description><![CDATA[Stories by Howard L on Medium]]></description>
        <link>https://medium.com/@warhod?source=rss-6715884919ab------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*doiyOGFbTKgSNTGwdALMCA@2x.jpeg</url>
            <title>Stories by Howard L on Medium</title>
            <link>https://medium.com/@warhod?source=rss-6715884919ab------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:26:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@warhod/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Continuous Improvement: Delivering Jenkins Through Jenkins Pt 2]]></title>
            <link>https://medium.com/earnest-engineering/continuous-delivery-of-jenkins-by-jenkins-26b649a94fac?source=rss-6715884919ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/26b649a94fac</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <dc:creator><![CDATA[Howard L]]></dc:creator>
            <pubDate>Wed, 16 Jan 2019 02:18:44 GMT</pubDate>
            <atom:updated>2019-01-17T23:30:23.267Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*TEGaUoauOh9g088Y3YHOFg.png" /></figure><p>This is part two of our exploration into Jenkins and how we use it at Earnest. <a href="https://medium.com/earnest-engineering/continuous-improvement-delivering-jenkins-through-jenkins-pt-1-4dd3e5af8cb0">Check out part one if you haven’t already.</a></p><p>Any experienced administrator could tell you Jenkins upgrades are often prone to failures. As Jenkins plugins can modify their behaviors with little to no notice, unsuspecting admins who skipped the fine print may often find themselves with the daunting task of tracking down all the changes on the road to recovery. Some of these behavioral changes may surprise our end users, but the worst of them may cause <a href="https://issues.jenkins-ci.org/browse/JENKINS-50671">Jenkins to crash on launch</a> or even result in the <a href="https://stackoverflow.com/questions/20758905/jenkins-job-disappeared-after-restart">data loss</a>.</p><p>Earnest has a dedicated team that builds the underlying platform and offers it as a service to other engineering teams, where they can build services that contain business logic on top. As such, we have accrued years of experience with the tools and processes for rolling out platform changes. Ideas such as contract testing, zero-downtime deployment, and automated rollbacks are leveraged heavily to ensure our platform is stable despite constant updates. We can borrow those same ideas and treat our Jenkins upgrades just like how we roll out new features in our platform-as-a-service offerings.</p><p>In principle, we need to guarantee upgrades will not result in any major regressions while minimizing downtime, without the intimate knowledge of the various custom CI/CD pipelines that runs on our platform.</p><p>To visualize this, we’ve divided our Jenkins upgrade into four stages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qMFjJqm4NMeheLY3" /></figure><h3>Review and Decide</h3><p>We only ever run the LTS version of Jenkins due to its relative high level of stability. In an ideal world, we could trigger an upgrade immediately upon a new LTS version release. In practice, an LTS release can bundle with it changes that may be undesirable. So to ensure that our end users experience a Jenkins upgrade that is both necessary and of high quality, our team will review the release notes to fully understand what it brings. This process starts with a subscription to the Jenkins release RSS feed and the feed alert will prompt us to review the <a href="https://jenkins.io/changelog-stable/">Jenkins changelog</a>. When reviewing the changelog, we will try to identify key security fixes, notable feature improvements, and any other bug fixes that would make it worth the small level of time commitment and risk involved with upgrading Jenkins. Once the team has reviewed the changes and decides that the benefits outweigh the costs of an upgrade, we move on to the next stage.</p><h3>Building/Deploying Jenkins</h3><p>Jenkins provides a <a href="https://hub.docker.com/r/jenkins/jenkins/">pre-built docker image on DockerHub</a>, so that serves as a good starting point for us to build our own customized image on top.</p><p>Here is a snippet of our Dockerfile for Jenkins:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15f4b54437b1d3cdb3d3203b97b85436/href">https://medium.com/media/15f4b54437b1d3cdb3d3203b97b85436/href</a></iframe><p>Next, we want to also lock in the of plugins that go into our Jenkins master. Fortunately, the official Jenkins image ships with a <a href="https://github.com/jenkinsci/docker/blob/master/install-plugins.sh">install script</a> that we can use to make plugin installation a breeze. Our team selected a set of Jenkins plugins that are applicable to our CI/CD workflow based on our customers’ needs, and they are defined in a <a href="https://github.com/fabric8io/jenkins-docker/blob/master/plugins.txt">plugins.txt</a> file. This file is a manifest of all the plugins with their corresponding versions. This file is in version control and is ultimately copied into the Jenkins image and used by the install script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cb2588f55fd75a0cb2dd16040813a1b/href">https://medium.com/media/5cb2588f55fd75a0cb2dd16040813a1b/href</a></iframe><p>Once the RUN statement is complete our image will have the jenkins plugins in a predictable and controllable version.</p><h3>Running the Jenkins container</h3><p>When running a container, there are multiple parameters that need to be configured per environment. For this reason and to make life easier for ourselves, we generate a `docker-compose.yml` file from an Ansible jinja template per runtime environment. This means we can simply launch Jenkins on any hosts with a simple `docker-compose up -d` command without remembering all the parameters we need to set. This has the added benefit of making it easy for our system administrators to re-configure and restart Jenkins instance on-the-fly, should the need arise.</p><h3>Testing</h3><p>After launching the Jenkins container successfully on our staging environment, we can then proceed to the testing phase. The way we test this new Jenkins instance is by creating an end-to-end test pipeline that combines testing of our both our core <a href="https://jenkins.io/doc/book/pipeline/shared-libraries/">Jenkins library functions</a> and our build pipelines. To create this “pipeline for all pipelines”, we can simply make a matrix capturing all the different types of pipeline and programming language that we support. So that we can have full test coverage no matter what tools and language the project is written in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gi-RXK4N5hQRZqDXvuQM6w.png" /></figure><p>Here is a snippet of the test pipeline stage. We can simply invoke tests from any of our existing git repositories. With this technique, we can re-use test code from our existing projects. This is also extensible and we can add more test stages as the variety of the test pipelines increases. Also, because the tests come from different repositories, we can run them in parallel as there is no inter-dependency. No additional time will be added to our test pipeline. The only thing we’d have to be mindful of is to make sure we have enough Jenkins build executors to support these parallel tests.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95ea088d11b19a239309328f33b4bb18/href">https://medium.com/media/95ea088d11b19a239309328f33b4bb18/href</a></iframe><h3>Promoting Jenkins to production</h3><p>With our Jenkins instance on staging passing all the tests, we’re finally ready to promote it! We use a technique known as <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">blue-green deployment</a> in this stage. This essentially means that we will take the same Jenkins image that we deployed to staging, re-tag it so we know it’s tested and production-ready, and then deploy it to a new stack identical to the existing one, and finally re-routing traffic to the new instance. Since Jenkins uses the <a href="https://jenkins.io/doc/developer/persistence/">underlying file system</a> to store its build data and configuration, we’ll also have to use a file sync tool such as rclone to copy over the data, or use a network file system such as Amazon’s <a href="https://aws.amazon.com/efs/">EFS</a>, before we launch the new Jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jl6o4KYE-1CRpLyE" /></figure><p>Once the new instance is up, the team member responsible for the promotion will ensure that new Jenkins is accessible and \ are retained. Our monitoring tool will also help us confirm that our Jenkins instance is indeed healthy on all the endpoints. With our shiny new instance of “green” Jenkins up and running, we can finally clean up our previous (blue) Jenkins instances.</p><p>Here is what our Jenkins clean-up script looks like, and of course we run this as a Jenkins job (wrapped around an ansible playbook)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d95c7f94a2460568be5fed8086a93515/href">https://medium.com/media/d95c7f94a2460568be5fed8086a93515/href</a></iframe><p>And here is our trusty slackbot reporting on a successful clean up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cEJTCTG1LkmZlnbh" /></figure><h3>Results</h3><p>Although the initial investment in the project required considerable effort, the benefits have compounded in terms of increasing our release (upgrade) velocity and overall confidence when rolling out changes.</p><p>Some of the key benefits of this project include:</p><ul><li>The ability to apply monthly security patches without disrupting users</li><li>Happier customers that are able to take advantage of new pipeline features that make their way into the latest Jenkins versions as soon as they’re available</li><li>The ability to perform upgrades or maintenance during office-hours with minimal downtime</li><li>The ability to maintain high service uptime during upgrade</li></ul><p>This project has also served as a forcing function that paves the road of running Jenkins as a container. So that in the future, we could even opt to migrate it over to a container management service. For now, both our customers and admins are very happy with the results!</p><p>If you found this series interesting and you want to learn more about what we work on, <a href="https://www.earnest.com/careers/#/overview">Earnest is hiring</a> and we’d love to talk!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26b649a94fac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/earnest-engineering/continuous-delivery-of-jenkins-by-jenkins-26b649a94fac">Continuous Improvement: Delivering Jenkins Through Jenkins Pt 2</a> was originally published in <a href="https://medium.com/earnest-engineering">Engineering at Earnest</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Continuous Improvement: Delivering Jenkins Through Jenkins Pt 1]]></title>
            <link>https://medium.com/earnest-engineering/continuous-improvement-delivering-jenkins-through-jenkins-pt-1-4dd3e5af8cb0?source=rss-6715884919ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dd3e5af8cb0</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Howard L]]></dc:creator>
            <pubDate>Wed, 19 Dec 2018 17:01:00 GMT</pubDate>
            <atom:updated>2018-12-19T19:11:53.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*s8mwAOWdb2choX4pj9ozvw.png" /></figure><p><em>By Howard Lo &amp; Amrik Ajimal</em></p><p>Although a plethora of build tools have come into vogue and faded into obscurity with the times, <a href="https://jenkins.io/">Jenkins</a> continues to capture our collective mindshare. When it comes to software integration, automation, and deployment, its flexibility in adapting to all manner of workflows makes it one of the better options on the market.</p><p>However, as most who have had the pleasure of working with Jenkins at their organizations can attest, this flexibility and extensibility often comes with a price often paid in tears, sweat, and caffeine.</p><p>In hindsight, the cause of this dichotomy is actually very simple. After setting up our tools and infrastructure, rarely do we apply the <a href="https://www.atlassian.com/continuous-delivery/ci-vs-ci-vs-cd">Continuous Integration/Delivery (CI/CD)</a> principles to them..</p><p>We propose a more generalized, meta-application of CI/CD principles to the foundational tools and infrastructures responsible for providing CI/CD to wider product and engineering teams. Additionally, we will demonstrate how such an approach not only facilitates holistic improvement to the overall software delivery and support ecosystems we depend on. It also allows us to reap profits ranging from high confidence in infrastructure upgrades to increased peace of mind for those managing such tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*Ngn9_AiUnU_YFLi-" /></figure><h3>Motivation</h3><p>Many factors contributed to our decision to apply a CI/CD approach to our Jenkins setup. For example, Jenkins increasingly became a focal point in our tooling with more tasks coming under its umbrella of responsibilities. While this was helpful in unifying many jobs from disparate systems into a more cohesive experience, it also meant that more teams were dependent on this critical piece of infrastructure. We quickly realized that performing upgrades, even exclusively to <a href="https://en.wikipedia.org/wiki/Long-term_support">Long-Term-Support</a> (LTS) versions of Jenkins, did not end up as stable as one would expect.</p><p>Oftentimes, we would discover that after an upgrade, some pipelines would start to exhibit unexpected behaviors. And such behaviors could block various teams from getting their work done as Jenkins is a central piece to our software delivery process. Once, a minor upgrade to a single plugin even prevented Jenkins from starting at all! Considerable time and effort had to be spent in finding the culprit and eventually we ended up rolling back the change and restored its data to a previous snapshot just to get it working again.</p><p>With that horrid experience, it became clear that deploying Jenkins was similar to the services that Jenkins itself was responsible for building and deploying. For this reason, it made sense to treat Jenkins the same way we treat all <a href="https://en.wikipedia.org/wiki/Microservices">Microservices</a> at Earnest — by utilizing CI/CD principles and only deploy a new version after things have been verified separately to work in a safe environment.</p><h3>Requirements</h3><p>With our frustrations motivations incentivizing us to address some of the shortcomings we experienced, we decided to outline a few key requirements as we embarked on this project.</p><h3>Self-service upgrades</h3><p>One of the critical objectives we wanted to address with our new setup was the “<a href="https://en.wikipedia.org/wiki/Bus_factor">what-if-I-get-hit-by-a-bus</a>” problem. To subvert this inevitable pitfall, we made it a priority to democratize the infrastructure setup for this critical system. Now anyone (with the correct permissions/privileges, of course) could perform an upgrade as easy as pushing a button. This will also allow us to open up the process so our team can contribute improvements with confidence!</p><h3>Blue/Green deployment</h3><p>We wanted to enforce a <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">blue/green</a> deployment model for all proposed changes into our build system. Doing so will allow the rollout with the flip of a network switch. This also gives us the substantial benefit of rolling back an upgrade quickly and seamlessly in case catastrophic issues were found after the rollout.</p><h3>Dogfooding</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/0*d3TJAEQ94l4V5ali" /></figure><p>When addressing the core set of problems above, it is important that everything happening under the hood is not only reliable, but also easy-to-use so it can be performed with minimal guidance.</p><p>To ensure that we were going down the right path as we evolved our setup, we delivered each process of the upgrade in separate pipelines, with minimal documentation. An entire upgrade would be all of these discrete pipelines executed one after another.</p><p>In the end, there is only one way to really validate any of this if you want respect from the <a href="https://en.wikipedia.org/wiki/The_Most_Interesting_Man_in_the_World">Most Interesting Man In The World</a>. We would have a teammate with no prior experience with Jenkins administration to perform each stage of an upgrade while we sat back with a beer and observed. If there were usability problems found during this process, we would address them and have our teammate repeat it all over again. This practice is known as <a href="https://en.wikipedia.org/wiki/Eating_your_own_dog_food">dogfooding</a>.</p><h3>Melting Snowflakes</h3><p>We wanted to ensure any novice Jenkins administrator could perform a Jenkins upgrade and rollback, if a domain expert is unavailable. To achieve this, we required <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">infrastructure-as-code</a> so that Jenkins and its plugins configurations are always available in source control. Doing so would also help us avoid the <a href="https://martinfowler.com/bliki/SnowflakeServer.html">snowflake</a> problem. Even though the initial implementation would take considerable time and effort, we believe this would be beneficial to the team in the long run.</p><h3>Avoid Unanticipated Downtime</h3><p>The goal of all of the above requirements was to fulfill a single need: avoid downtime during any Jenkins upgrade. By implementing blue-green deployment, infrastructure-as-code, and prioritizing usability together, we could ensure that releases to production would be easy and would not interrupt our engineering team’s day-to-day activities.</p><h3>Conclusion</h3><p>Given these requirements, our expectations for this project were pretty high. If we did everything correctly, we could perform an upgrade as soon as a new Jenkins LTS version is released, or anytime a new security patch dotfix dropped. We can ensure our engineering team is always able to use the most cutting-edge Jenkins feature as soon as it is available, without worrying about stability. In our next blog post, we will take a deep dive into how we accomplish such feat, so don’t forget to check back for an article on the details of building that pipeline!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dd3e5af8cb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/earnest-engineering/continuous-improvement-delivering-jenkins-through-jenkins-pt-1-4dd3e5af8cb0">Continuous Improvement: Delivering Jenkins Through Jenkins Pt 1</a> was originally published in <a href="https://medium.com/earnest-engineering">Engineering at Earnest</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>