<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jonathan West on Medium]]></title>
        <description><![CDATA[Stories by Jonathan West on Medium]]></description>
        <link>https://medium.com/@jgwest?source=rss-75e41f0900a0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ST-Sbh66Z45qeEY6-NeBzA.png</url>
            <title>Stories by Jonathan West on Medium</title>
            <link>https://medium.com/@jgwest?source=rss-75e41f0900a0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:31:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jgwest/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ApplicationSet Controller v0.3.0 for Argo CD now available]]></title>
            <link>https://blog.argoproj.io/applicationset-controller-v0-3-0-for-argo-cd-now-available-3f56bd643312?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f56bd643312</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Thu, 16 Dec 2021 18:33:57 GMT</pubDate>
            <atom:updated>2021-12-16T18:33:57.818Z</atom:updated>
            <content:encoded><![CDATA[<p>I am pleased to announce the latest release of the Argo CD ApplicationSet controller, v0.3.0. Many new features were contributed as part of this release, including two new generators, improved error reporting and handling, support for webhook-based refresh triggers, plus doc updates, usability improvements, stability fixes, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rChrRvGcHFFInJX4lPISrQ.png" /><figcaption>The Merge generator is one of two new generators contributed as part of this release.</figcaption></figure><p>You can learn more about this release from the <a href="https://argocd-applicationset.readthedocs.io/">ApplicationSet documentation</a>, or check out the project repository <a href="https://github.com/argoproj-labs/applicationset/">and learn how you can contribute</a>.</p><h3><strong>New in this release</strong></h3><h4><strong>New generator: Pull Request generator</strong></h4><p>With ApplicationSet v0.3.0, a new Pull Request generator has been contributed which uses the API of a SCMaaS provider (e.g. GitHub) to automatically discover open pull requests within a repository. This fits well with users that wish to construct a test environment based on an open pull request.</p><p>In this example, the <em>ApplicationSet</em> will create an Argo CD <em>Application</em> resource for each open pull request:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3181a5053dddba4e69319ec12001850b/href">https://medium.com/media/3181a5053dddba4e69319ec12001850b/href</a></iframe><p>To learn more, check out the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Pull-Request/">Pull Request generator documentation</a> for details. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/366/">@shmurata</a>.</p><h4><strong>New generator: Merge generator</strong></h4><p>Also new in this release is the Merge generator, which is useful when you want to selectively override the parameters generated by one generator, with those generated by another.</p><p>In this example, we first gather the list of clusters from Argo CD using the Cluster generator, then we ‘patch’ only clusters with the label <em>use-kakfa: false</em>, and finally, we enable Redis on a specific cluster:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7a6e3f7d333c1c27e946792bcac52be/href">https://medium.com/media/a7a6e3f7d333c1c27e946792bcac52be/href</a></iframe><p>See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Merge/">Merge generator documentation</a> for a full explanation of the example, and for details on generator behaviour. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/404">@crenshaw-dev</a>.</p><h4><strong>Report error conditions/status for ApplicationSet CR</strong></h4><p>When the user-provided generator/template produces invalid Argo CD Applications, the <em>ApplicationSet</em> resource’s status field will now report errors (or the lack thereof). Here is an example of the new status conditions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c53396ed444aa4106673fcc1e1d191e1/href">https://medium.com/media/c53396ed444aa4106673fcc1e1d191e1/href</a></iframe><p>On parameter generation failure or templating failure, those errors will be reported under the appropriate conditions. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/370">@ishitasequeira</a>.</p><h4><strong>Git Generator: Refresh ApplicationSet resource with Git generator using webhook</strong></h4><p>This feature adds support for refreshing ApplicationSets via a GitHub webhook trigger. It exposes a service which listens for incoming webhook payloads, and once received triggers the ApplicationSet controller to regenerate resources. In contrast, in the previous release the ApplicationSet controller only supported polling the Git repository used by the Git generator every <em>3 minutes</em> (but this <em>is</em> at least customizable).</p><p>See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Git/#webhook-configuration">webhook documentation</a> for details. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/341">@chetan-rns</a>.</p><p>This contribution also adds general support for webhooks by other generators, which is used by the Pull Request generator webhook code, below.</p><h4><strong>Gracefully handle application validation errors</strong></h4><p>This feature changes how the ApplicationSet controller handles ApplicationSets that generate invalid <em>Application</em>s. Previously, if at least one Application in the ApplicationSet was invalid, the controller would refuse to proceed further and would skip <em>all</em> Application processing (i.e. it would ‘fail fast’). Now, the controller will process <em>valid</em> Applications, and only skip <em>invalid</em> Applications (logging information about them to the console).</p><p>Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/372">@alexmt</a>.</p><h4><strong>Pull Request generator: Support for webhooks</strong></h4><p>When using a Pull Request generator, the ApplicationSet controller polls every <em>requeueAfterSeconds</em> interval (defaulting to every 30 minutes) to detect changes. To eliminate this delay from polling, the ApplicationSet webhook server can be configured to receive webhook events, which will refresh the parameters generated by the Pull Request generator, and thus the corresponding <em>Application</em> resources.</p><p>More information on configuring webhooks with the Pull Request generator is available from the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Pull-Request/#webhook-configuration">Pull Request generator documentation</a>. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/417">@shmurata</a>.</p><h4><strong>Support `-logformat=json` as parameter to applicationset-controller</strong></h4><p>This feature adds a new <em>— logformat=json</em> parameter to the applicationset-controller, which switches the logging output of the ApplicationSet controller to JSON. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/373">@shivjm</a>.</p><h4>Improve Git files generator performance (#355)</h4><p>The Git files generator was consuming too much time (and driving up Git requests) due to inadvertently executing ‘git fetch/git checkout’ for each discovered file within the repository. With ApplicationSet v0.3.0, that has improved such that we will now issue a Git checkout/fetch repo once per refresh. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/355">@alexmt</a>.</p><h4><strong>SCM Generator: Provide SHA for latest commit on a branch in variables (#307)</strong></h4><p>This feature adds SHA to the list of parameters exposed by the SCM Generator, with the SHA parameter representing the latest commit. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/307">@TMaYaD</a>.</p><h4><strong>Fixes, test fixes, infrastructure improvements, and documentation updates</strong></h4><p>In addition to the above new features, we fixed plenty of bugs, improved project test coverage, made improvements to the overall CI process, filled in some left over documentation gaps from the 0.3.0 release, and more.</p><p>See the <a href="https://github.com/argoproj-labs/applicationset/releases/tag/v0.3.0">release notes</a> for a detailed list of all changes made in this release.</p><h3><strong>Upgrade Notes</strong></h3><p>When moving from ApplicationSet v0.1.0/v0.2.0, to v0.3.0, there are two behaviour changes to be aware of.</p><h4><strong>Cluster generator: {{name}} parameter value will no longer be normalized, but existing normalization behaviour is preserved in a new {{nameNormalized}} parameter</strong></h4><p>The Cluster generator <em>{{name}}</em> parameter has now reverted to its original behaviour: the cluster name within Argo CD will no longer be <a href="https://github.com/argoproj-labs/applicationset/blob/11f1fe893b019c9a530865fa83ee78b16af2c090/pkg/generators/cluster.go#L168">normalized</a>. The <em>{{name}}</em> parameter generated by the Cluster generator within the ApplicationSet will now be passed unmodified to the ApplicationSet template.</p><p>A new parameter,<em> {{nameNormalized}}</em> has been introduced which preserves the 0.2.0 behaviour. This allows you to choose which behaviour you wish to use in your ApplicationSet, based on the context in which it is used: either using the parameter as defined, or in a normalized form (which allows it to be used in the <em>name</em> field of an <em>Application</em> resource.)</p><p>If your Argo CD cluster names are already valid, no change is required. Otherwise, to preserve the v0.2.0 behaviour of your ApplicationSet, replace <em>{{name}}</em> with <em>{{nameNormalized}}</em> within your ApplicationSet template.</p><p>More information about this change is <a href="https://github.com/argoproj-labs/applicationset/pull/390">available from the issue</a>.</p><h4><strong>If an ApplicationSet contains an invalid generated Application, the valid generated Applications will still be processed</strong></h4><p>The responsibility of the ApplicationSet controller is to convert an <em>ApplicationSet</em> resource into one or more <em>Application</em> resources. However, with the previous releases, if <em>at least one</em> of the generated Application resources was invalid (e.g. it failed the internal validation logic), <em>none</em> of the generated Applications would be processed (the behaviour was to fail fast, so they would neither be created nor modified).</p><p>With the latest ApplicationSet release, if a generator generates invalid Applications, those invalid generated Applications will still be skipped, <em>but</em> the valid generated Applications will now be processed (created/modified).</p><p>Thus no <em>ApplicationSet</em> resource changes are required by this new behaviour, but it is worth keeping in mind that your ApplicationSets which were previously blocked by a failing Application may no longer be blocked. This change might cause valid Applications to now be created/modified, whereas previously they were prevented from being processed.</p><p>More information about this change is <a href="https://github.com/argoproj-labs/applicationset/pull/372">available from the issue</a>.</p><h3><strong>Contributors</strong></h3><p>Thanks to all the folks who have contributed to the ApplicationSet controller since our last release.</p><ul><li>Shunya Murata (@shmurata)</li><li>Michael Crenshaw (@crenshaw-dev)</li><li>Jonathan West (@jgwest)</li><li>Ishita Sequeira (@ishitasequeira)</li><li>Chetan Banavikalmutt (@chetan-rns)</li><li>Alexander Matyushentsev (@alexmt)</li><li>Shiv Jha-Mathur (@shivjm)</li><li>Subhash Chandra (@TMaYaD)</li><li>William Tam (@wtam2018)</li><li>Benoit Gaillard (@benoitg31)</li><li>Michal Barecki (@mbarecki)</li><li>Guillaume Dupin (@yogeek)</li><li>Krzysztof Dąbrowski (@krzysdabro)</li><li>Olve S. Hansen (@olvesh)</li><li>Dewan Ishtiaque Ahmed (@dewan-ahmed)</li><li>Diego Pomares (@DiegoPomares)</li></ul><p>Want to join us for our next release? Check out the <a href="https://github.com/argoproj-labs/applicationset">project repository</a> or visit us on <a href="https://argoproj.github.io/community/join-slack/">#argo-cd-appset</a> on Slack.</p><h3><strong>Installation</strong></h3><p>The ApplicationSet controller must be installed into the same namespace as the Argo CD it is targeting:</p><pre>kubectl apply -n argocd -f <a href="https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.3.0/manifests/install.yaml">https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.3.0/manifests/install.yaml</a></pre><p>Once installed, the ApplicationSet controller requires no additional setup. You can learn more about ApplicationSet controller installation from the <a href="https://argocd-applicationset.readthedocs.io/en/v0.3.0/Getting-Started/">Getting Started</a> page.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f56bd643312" width="1" height="1" alt=""><hr><p><a href="https://blog.argoproj.io/applicationset-controller-v0-3-0-for-argo-cd-now-available-3f56bd643312">ApplicationSet Controller v0.3.0 for Argo CD now available</a> was originally published in <a href="https://blog.argoproj.io/">Argo Project</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ApplicationSet Controller v0.2.0 for Argo CD out now]]></title>
            <link>https://blog.argoproj.io/applicationset-controller-v0-2-0-for-argo-cd-out-now-b2922e5ac883?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2922e5ac883</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Sat, 21 Aug 2021 01:40:25 GMT</pubDate>
            <atom:updated>2021-08-21T01:40:25.839Z</atom:updated>
            <content:encoded><![CDATA[<p>I am happy to announce the second release of the Argo CD ApplicationSet controller, v0.2.0, releasing alongside Argo CD v2.1.</p><p>Many new features were contributed as part of this release, including support for combining generator parameters, support for building Argo CD Applications based on GitHub/GitLab organizations, and support for using custom resources to select clusters, plus oft requested improvements to existing generators, and of course doc updates, usability improvements, stability fixes, and more. Read on for the exciting details!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e9dNLnhSgqXgL58pF-bHxA.png" /><figcaption>The Matrix generator is one of 3 new ApplicationSet generators in this release.</figcaption></figure><p>We encourage you to try out our latest and greatest release, and feel free to let us know what big challenges you are solving with ApplicationSets, and/or what pain points you hit. Got feature requests, bug reports, or want to contribute code? Let us know on our <a href="https://github.com/argoproj-labs/applicationset">project repository</a>, or join us on <a href="https://argoproj.github.io/community/join-slack">#argo-cd-appset</a> on Slack.</p><h3>New in this Release</h3><h4>Matrix generator</h4><p>The Matrix generator is a new generator that combines the parameters generated by two child generators, iterating through every combination of each generator’s parameter set:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/733d72ae58f615693090c51a0a318993/href">https://medium.com/media/733d72ae58f615693090c51a0a318993/href</a></iframe><p>In this example, the parameters generated by a Git generator (looking for applications within a Git Repository), and by a Cluster generator (looking for Argo CD-managed clusters), are combined.</p><p>This produces Argo CD Applications that will deploy every application contained within the Git repository, to every cluster managed by Argo CD.</p><p>See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Matrix/">Matrix generator documentation</a> for an in-depth example of how generator parameters are combined. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/205">@OmerKahani</a>.</p><h4>SCM Provider generator</h4><p>The SCM Provider generator is a new generator that utilizes the API of GitHub/GitLab to automatically discover repositories within an organization, allowing the repository values to be used for targeting generated Argo CD Applications.</p><p>This fits well with GitOps layout patterns that split microservices across many repositories, rather than those patterns that stick to a single repository (which can be handled by other ApplicationSet generators).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68f64684d9280b61f735e9447f9e6be7/href">https://medium.com/media/68f64684d9280b61f735e9447f9e6be7/href</a></iframe><p>The ApplicationSet controller will then scan the provided GitHub/GitLab organization and produce template parameters for each discovered repository/branch, which may be used to generate Argo CD Applications for each of these repositories.</p><p>See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-SCM-Provider/">SCM Provider generator</a> documentation for more information. Initial implementation and GitHub support contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/209">@coderanger</a>, and GitLab support contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/283">@empath-nirvana</a>.</p><h4>Cluster Decision Resource generator</h4><p>The Cluster Decision Resource generator is a new generator that generates a list of Argo CD clusters based on the contents of an external custom resource (CR), with that custom resource managed by an external controller.</p><p>With the Cluster Decision Resource generator you may ‘outsource’ the logic of which clusters to target, to third party controllers/CRs, such as the <a href="https://open-cluster-management.io/concepts/placement/">Open Cluster Management’s Placements</a>.</p><p>This is handled seamlessly using <a href="https://knative.dev/docs/developer/concepts/duck-typing/">duck-typing</a>, which does not require knowledge of the full shape of the referenced Kubernetes resource. The following is an example of a cluster-decision-resource-based ApplicationSet generator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c044007cfa4fae754115829a48c8ec1/href">https://medium.com/media/0c044007cfa4fae754115829a48c8ec1/href</a></iframe><p>This generator references an external resource named quak, managed by an external controller, containing a list of clusters to deploy to, in its ‘decisions’ field:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e0435c7d0296e7e0867e960ffe99414/href">https://medium.com/media/2e0435c7d0296e7e0867e960ffe99414/href</a></iframe><p>See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Cluster-Decision-Resource/">Cluster Decision Resource</a> generator documentation for more information. <a href="https://github.com/argoproj-labs/applicationset/pull/231">Cluster Decision Resource generator</a> and <a href="https://github.com/argoproj-labs/applicationset/pull/272">labelSelector support</a> contributed by <a href="https://github.com/jnpacker">@jnpacker</a>.</p><h4>Preserve Application child resources on deletion of parent ApplicationSet</h4><p>By default, Applications created and managed by the ApplicationSet controller include the <a href="https://argoproj.github.io/argo-cd/user-guide/app_deletion/#about-the-deletion-finalizer">Argo CD resource deletion finalizer</a>. This means that when an ApplicationSet is deleted, its child Applications will also be deleted, as well as the cluster resources of those child Applications. This is the same behaviour as a cascade delete within Argo CD.</p><p>However, this behaviour is not always desirable: one may want to preserve the Application child resources on deletion of the parent Application. To enable this, use the .spec.syncPolicy.preserveResourcesOnDeletion value in the parent ApplicationSet:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b8f55c07a83399eb3e0d84ac54b3299/href">https://medium.com/media/5b8f55c07a83399eb3e0d84ac54b3299/href</a></iframe><p>See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Application-Deletion/">Application Deletion Behaviour</a> documentation for more information. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/223">@mmatur</a>.</p><h4>Allow any key/value pair in List generator</h4><p>The List generator previously only supported a fixed list of cluster name/URL values, with an optional values field for user-defined values. You can now specify any string key/value pair:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68c389e9677cce8e94e54b91b9345a66/href">https://medium.com/media/68c389e9677cce8e94e54b91b9345a66/href</a></iframe><p>This new form is fully backwards compatible with existing v0.1.0-originated ApplicationSets, and no migration steps are needed. See the <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-List/">List generator</a> documentation for more information. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/290">@ishitasequeira</a>.</p><h4>Add YAML configuration file support to Git File generator</h4><p>In the previous ApplicationSet release, only JSON-formatted configuration files were supported by the Git File generator. Now, both <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Git/#git-generator-files">JSON and YAML files are supported</a>, and may be used interchangeably. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/211">@teejaded</a>.</p><h4>Added additional path params to Git File generator</h4><p>The Git File generator now produces {{ path }} and {{ path.basename }} parameters, containing the path of the configuration file, similar to the same parameters already produced by the Git Directory generator. Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/260">@ishitasequeira</a>.</p><h4>Add exclude path support to Git directories</h4><p>The Git Directory generator scans directories within a Git repository, looking for directories that match specific criteria. However, with the previous ApplicationSet release, there was no way to exclude folders from the scan.</p><p>The ability to <a href="https://argocd-applicationset.readthedocs.io/en/master/Generators-Git/#exclude-directories">exclude individual paths from being scanned</a> is now supported:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/986c0df28b289d5dfbe6206158f6c588/href">https://medium.com/media/986c0df28b289d5dfbe6206158f6c588/href</a></iframe><p>Contributed by <a href="https://github.com/argoproj-labs/applicationset/pull/195">@LiorLieberman</a>.</p><h4>Quality of Life Improvements</h4><ul><li>Git Directory generator should ignore all folders starting with ‘.’ (<a href="https://github.com/argoproj-labs/applicationset/pull/201">#201</a>, contributed by <a href="https://github.com/LiorLieberman">@LiorLieberman</a>)</li><li>Add support for custom ‘finalizers’ in Application templates (<a href="https://github.com/argoproj-labs/applicationset/pull/289">#289</a>, contributed by <a href="https://github.com/tenevdev">@tenevdev</a>)</li><li>Enable the ability to set log level on ApplicationSet controller startup (<a href="https://github.com/argoproj-labs/applicationset/pull/249">#249</a>, contributed by <a href="https://github.com/rumstead">@rumstead</a>)</li><li>Add ‘appset/appsets’ shortnames to the ApplicationSet CRD (<a href="https://github.com/argoproj-labs/applicationset/pull/184">#184</a>, contributed by <a href="https://github.com/christianh814">@christianh814</a>)</li><li>Log appset version during controller startup log (<a href="https://github.com/argoproj-labs/applicationset/pull/241">#241</a>, contributed by <a href="https://github.com/chetan-rns">@chetan-rns</a>)</li></ul><h4>Bug fixes, doc updates, and CI infrastructure improvement</h4><p>In addition to the above new features, we fixed plenty of bugs, improved project test coverage, made improvements to the overall CI process, filled in some left over documentation gaps from the 0.1.0 release, and more.</p><p>See the <a href="https://github.com/argoproj-labs/applicationset/releases/tag/v0.2.0">release notes</a> for a detailed list of all changes made in this release.</p><h3>Contributors</h3><p>Many, many thanks to all the folks who have contributed to this ApplicationSet controller release over the past few months. These many contributions, both big and small, general and specific, help to bring a more featureful and polished experience to all Argo CD ApplicationSet users. We could not do this without all of you!</p><ul><li>Omer Kahani (<a href="https://github.com/OmerKahani">@OmerKahani</a>)</li><li>Joshua Packer (<a href="https://github.com/jnpacker">@jnpacker</a>)</li><li>Jonathan West (<a href="https://github.com/jgwest">@jgwest</a>)</li><li>Noah Kantrowitz (<a href="https://github.com/coderanger">@coderanger</a>)</li><li>Lior Lieberman (<a href="https://github.com/LiorLieberman">@LiorLieberman</a>)</li><li>Michael Matur (<a href="https://github.com/mmatur">@mmatur</a>)</li><li>TJ Miller (<a href="https://github.com/teejaded">@teejaded</a>)</li><li>John Thompson (<a href="https://github.com/empath-nirvana">@empath-nirvana</a>)</li><li>Ishita Sequeira (<a href="https://github.com/ishitasequeira">@ishitasequeira</a>)</li><li>Chetan Banavikalmutt (<a href="https://github.com/chetan-rns">@chetan-rns</a>)</li><li>William Tam (<a href="https://github.com/wtam2018">@wtam2018</a>)</li><li>John Watson (<a href="https://github.com/dctrwatson">@dctrwatson</a>)</li><li>Tencho Tenev (<a href="https://github.com/tenevdev">@tenevdev</a>)</li><li>Ryan Umstead (<a href="https://github.com/rumstead">@rumstead</a>)</li><li>Stephan Auerhahn (<a href="https://github.com/palp">@palp</a>)</li><li>Christian Hernandez (<a href="https://github.com/christianh814">@christianh814</a>)</li><li>Mohit Kumar Sharma (<a href="https://github.com/mksha">@mksha</a>)</li><li>Vivien Fricadel (<a href="https://github.com/vivienfricadelamadeus">@vivienfricadelamadeus</a>)</li><li>Gareth Western (@<a href="https://github.com/gdubya">gdubya</a>)</li><li>William Jeffries (<a href="https://github.com/williamcodes">@williamcodes</a>)</li><li>Mehran Poursadeghi (<a href="https://github.com/mehran-prs">@mehran-prs</a>)</li></ul><p>Want to join us for our next release? Check out the <a href="https://github.com/argoproj-labs/applicationset">project repository</a> or visit us on <a href="https://argoproj.github.io/community/join-slack/">#argo-cd-appset</a> on Slack.</p><h3>Installation</h3><p>The ApplicationSet controller must be installed into the same namespace as the Argo CD it is targeting:</p><pre>kubectl apply -n argocd -f <a href="https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.2.0/manifests/install.yaml">https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.2.0/manifests/install.yaml</a></pre><p>Once installed, the ApplicationSet controller requires no additional setup. You can learn more about ApplicationSet controller installation from the <a href="https://argocd-applicationset.readthedocs.io/en/v0.2.0/Getting-Started/">Getting Started</a> page.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2922e5ac883" width="1" height="1" alt=""><hr><p><a href="https://blog.argoproj.io/applicationset-controller-v0-2-0-for-argo-cd-out-now-b2922e5ac883">ApplicationSet Controller v0.2.0 for Argo CD out now</a> was originally published in <a href="https://blog.argoproj.io/">Argo Project</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the ApplicationSet Controller for Argo CD]]></title>
            <link>https://blog.argoproj.io/introducing-the-applicationset-controller-for-argo-cd-982e28b62dc5?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/982e28b62dc5</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[argoproj]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Thu, 08 Apr 2021 04:12:02 GMT</pubDate>
            <atom:updated>2021-04-08T04:12:02.802Z</atom:updated>
            <content:encoded><![CDATA[<p>I am excited to announce the first release of the Argo CD ApplicationSet controller, v0.1.0, releasing now alongside Argo CD v2.0!</p><p>With the ApplicationSet controller:</p><ul><li>You can deploy Argo CD Applications to multiple Kubernetes clusters at once</li><li>You can deploy multiple Argo CD Applications from a single monorepo</li><li>Allows unprivileged cluster users (those without access to the Argo CD namespace) to deploy Argo CD Applications, without the need to involve cluster administrators in manually enabling the clusters/namespaces</li><li>Best of all, all these features are controlled by only a single instance of<em> </em>an <em>ApplicationSet </em>custom resource, which means no more juggling of multiple Argo CD Application resources to target those multiple clusters/repositories!</li></ul><p>Unlike with an Argo CD Application resource, which deploys resources from a <em>single</em> Git repository to a <em>single</em> destination cluster/namespace, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications simultaneously, targeting <em>multiple </em>destination clusters/namespaces.</p><p>If you are loving Argo CD and want to use ApplicationSet’s automation and templating to take your usage to the next level, read on!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/0*RkJIYbECQSTArfRo" /><figcaption>The ApplicationSet controller automates the deployment of Argo CD Application to a large set of clusters, managed by a single ApplicationSet resource.</figcaption></figure><p>The ApplicationSet controller is installed alongside Argo CD (within the same namespace), and it automatically generates Argo CD Applications based on the contents of a new <em>ApplicationSet</em> Custom Resource (CR).</p><p>Here is how an ApplicationSet looks in action — in this example, an ApplicationSet is being used to deploy a guestbook application to multiple clusters, using Argo CD:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fa985ee99571c118ee28cd6a743a8b6/href">https://medium.com/media/7fa985ee99571c118ee28cd6a743a8b6/href</a></iframe><p>The <em>list</em> field under <em>generators</em> is an instance of the List generator. Generators are responsible for producing a set of key-value pairs, which are passed into the template as <em>{{param}}</em>-style parameters.</p><p>The template is a parameterized Argo CD Application stanza, with the same contents as a standalone Argo CD Application resource, including project, source Git repository, and a destination cluster. But, unlike Argo CD, these fields may accept <em>{{param}}</em>-style parameters from the generator, as seen above. Generators are responsible for generating these parameters, and the template is responsible for consuming them, with the rendered template applied to the cluster as Argo CD Applications.</p><p>In this example, the List generator passes the <em>‘url</em>’ and <em>‘cluster</em>’ fields into the Application template as parameters, which are then rendered into the template as three corresponding Argo CD Applications (one for each defined cluster). Targeting new clusters is simply a matter of adding new elements to the <em>ApplicationSet</em> resource, and the corresponding Argo CD Applications will be automatically created. Likewise, changes made to the ApplicationSet template fields (such as branch or namespace) will automatically be applied to every generated Application.</p><p>Managing a set of multiple Argo CD Applications (an ‘application set’) is thus as easy as managing a single ApplicationSet resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*giIO7oUtPcsdmhpQzlMknQ.png" /><figcaption>Applications, generated from an ApplicationSet, as seen in the Argo CD Web UI</figcaption></figure><p>Within the ApplicationSet controller there exist other more powerful generators in addition to the List generator, including the Cluster generator (which automatically uses Argo CD-defined clusters to template Applications), and the Git generator (which uses the files/directories of a Git repository to template applications).</p><h3>Features</h3><h4>Cluster Generator</h4><p>The cluster generator is useful if you are using Argo CD to manage cluster add-ons, such as Custom Resource Definitions (CRDs) or Ingress Controllers, across a fleet of clusters. Instead of manually creating an application for each managed cluster, you can delegate it to the cluster generator.</p><p>In Argo CD, managed clusters are stored within Secrets in the Argo CD namespace. The ApplicationSet controller uses those same Secrets to generate template parameters for which to target available clusters.</p><p>The <a href="https://argocd-applicationset.readthedocs.io/en/stable/Generators/#cluster-generator">Cluster generator</a> will automatically identify clusters defined with Argo CD, and extract the cluster data as parameters, allowing Applications to be created that target using those values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2535ccc9b4d8d939b70eb0a4da21c3f2/href">https://medium.com/media/2535ccc9b4d8d939b70eb0a4da21c3f2/href</a></iframe><h4>Git Directory Generator</h4><p>It is a good practice to define a convention between the Argo CD application name and the location of the deployment manifests directory with the Git repository. For example, you might choose to store all manifests of production applications under “<em>applications/prod/&lt;app-name&gt;</em>” and all staging applications under “<em>applications/staging/&lt;app-name&gt;</em>”.</p><p>The <a href="https://argocd-applicationset.readthedocs.io/en/stable/Generators/#git-generator-directories">Git Directory generator</a> allows you to go one step further and <em>codify</em> that convention, by generating template parameters using the <em>directory structure</em> of a Git repository.</p><p>Whenever a new subfolder is added to the Git repository, the ApplicationSet controller will detect this change and automatically use the subfolder path to template an Argo CD application comprising the manifests contained within that folder (whether they be plain YAML, Kustomize, Helm, etc).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6739347acdf0e8c375c84bf696068a6/href">https://medium.com/media/d6739347acdf0e8c375c84bf696068a6/href</a></iframe><h4>Git File Generator</h4><p>Similar to the Directory generator, the <a href="https://argocd-applicationset.readthedocs.io/en/stable/Generators/#git-generator-files">Git File generator</a> allows you to generate Applications based on Git repository content, but it provides a bit more flexibility. The Git File generator generates template parameters using the contents of JSON files found within a specified repository.</p><p>Git commits containing changes to JSON files are automatically discovered by the Git generator, and the contents of those files are parsed and converted into template parameters.</p><p>This allows the creation of custom Argo CD Applications based on the contents of JSON files within the repository. As new files are added/changed, corresponding Argo CD Applications are created or modified.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05ef6011ff6a67c00e0bc2fa07838c55/href">https://medium.com/media/05ef6011ff6a67c00e0bc2fa07838c55/href</a></iframe><h4>List Generator</h4><p>As seen in the introduction, the <a href="https://argocd-applicationset.readthedocs.io/en/stable/Generators/#list-generator">List generator</a> generates parameters based on a fixed list of cluster name/URL values, with those values passed as parameters into the template. This allows manual control over Application destinations via editing of a literal cluster list.</p><h4>Brand new documentation and examples</h4><p>The ApplicationSet controller has a brand new set of <a href="https://argocd-applicationset.readthedocs.io/">documentation</a> and <a href="https://github.com/argoproj-labs/applicationset/tree/master/examples">examples</a>. Topics include an introduction to ApplicationSet resources, a getting started guide, use cases, how ApplicationSets interact with Argo CD, generators, template fields, Application lifecycle, and more.</p><h4>Template Override</h4><p>In addition to specifying a template within the <em>.spec.template</em> of the ApplicationSet resource, <a href="https://argocd-applicationset.readthedocs.io/en/stable/Template/#generator-templates">templates may also be specified within generators</a>. This is useful for overriding the values of the spec-level template with generator-specific values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f148bd7efd1fac555ec473e77defb592/href">https://medium.com/media/f148bd7efd1fac555ec473e77defb592/href</a></iframe><h4>Support for arbitrary key/value pairs in Cluster generator and List generator</h4><p>Arbitrary key/value pairs may be included within the Cluster and List generators, which will be converted into template parameters during template rendering. This is useful for providing custom parameters for a specific generator instance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecc981e295268189b6eeaebaa932b1af/href">https://medium.com/media/ecc981e295268189b6eeaebaa932b1af/href</a></iframe><h3>Bugs, tests, and infrastructure improvements</h3><p>In addition to the above new features, we delivered lots of bug fixes, new unit tests, a new end-to-end test framework, new end-to-end tests, a new release process, and build/test infrastructure improvements.</p><p>See the<a href="https://github.com/argoproj-labs/applicationset/releases/tag/v0.1.0"> release notes for a detailed list of changes</a> made in this release.</p><h3>Installation</h3><p>The ApplicationSet controller must be installed into the same namespace as the Argo CD it is targeting. You may install it using kubectl like this:</p><p><em>kubectl apply -n argocd -f </em><a href="https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.1.0/manifests/install.yaml"><em>https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.1.0/manifests/install.yaml</em></a></p><p>Once installed, the ApplicationSet controller requires no additional setup. You can learn more about <a href="https://argocd-applicationset.readthedocs.io/en/stable/Geting-Started/">ApplicationSet installation from the Getting Started page</a>.</p><h3>Feedback is very welcome</h3><p>Since this is our first release, we would ❤️ if you would give ApplicationSets a shot, and let us know what cool problems you are solving with it, or what pain points you hit.</p><p>Got feature requests, bug reports, or want to contribute code? Let us know on our <a href="https://github.com/argoproj-labs/applicationset">project repository</a>, or join us on <a href="https://argoproj.github.io/community/join-slack">#argo-cd-appset on Slack</a>.</p><h3>Contributors</h3><p>A great deal of work has gone into bringing this project to life, from many different contributors, all the way from its inception in early 2020 until today. It is terrific to be able to bring all that work together as ApplicationSet’s first release, and make it available to a wider audience… we welcome you to try it out, and let us know what you think!</p><p>A big thanks to all ApplicationSet controller contributors for their hard work over the last year, whether it be contributing code, writing design documentation, performing code reviews, writing user documentation, and opening issues and PRs:</p><ul><li>Omer Kahani (<a href="https://github.com/OmerKahani">@OmerKahani</a>)</li><li>Devan Goodwin (<a href="https://github.com/dgoodwin">@dgoodwin</a>)</li><li>Michael Goodness (<a href="https://github.com/mgoodness">@mgoodness</a>)</li><li>Alexander Matyushentsev (<a href="https://github.com/alexmt">@alexmt</a>)</li><li>Jonathan West (<a href="https://github.com/jgwest">@jgwest</a>)</li><li>Matteo Ruina (<a href="https://github.com/maruina">@maruina</a>)</li><li>Jesse Suen (<a href="https://github.com/jessesuen">@jessesuen</a>)</li><li>Xianlu Bird (<a href="https://github.com/xianlubird">@xianlubird</a>)</li><li>William Tam (<a href="https://github.com/wtam2018">@wtam2018</a>)</li><li>Ratnadeep Debnath (<a href="https://github.com/rtnpro">@rtnpro</a>)</li><li>John Pitman (<a href="https://github.com/jopit">@jopit</a>)</li><li>Shoubhik Bose (<a href="https://github.com/sbose78">@sbose78</a>)</li><li>Alex Sharov (<a href="https://github.com/kvendingoldo">@kvendingoldo</a>)</li><li>Omer Levi Hevroni (<a href="https://github.com/omerlh">@omerlh</a>)</li></ul><p>The ApplicationSet controller would not exist without the contributions of these talented individuals! 🎉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=982e28b62dc5" width="1" height="1" alt=""><hr><p><a href="https://blog.argoproj.io/introducing-the-applicationset-controller-for-argo-cd-982e28b62dc5">Introducing the ApplicationSet Controller for Argo CD</a> was originally published in <a href="https://blog.argoproj.io/">Argo Project</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How the Lagom framework enables scalable, reactive Microservices in Java and Scala]]></title>
            <link>https://medium.com/@jgwest/how-the-lagom-framework-enables-scalable-reactive-microservices-in-java-and-scala-cd8b15c0a3ad?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd8b15c0a3ad</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Wed, 06 Mar 2019 01:09:57 GMT</pubDate>
            <atom:updated>2019-03-06T01:09:57.453Z</atom:updated>
            <content:encoded><![CDATA[<p>Lagom is open source microservice framework for building <em>reactive</em> microservice applications in Java or Scala. The pitch for the <a href="https://www.lagomframework.com/">Lagom framework</a> is that its programming model and architecture allow developers to write microservices that effectively scale across large deployments, that provide desirable application quality-of-life characteristics such as robust error tolerance and application responsiveness, and that take full advantage of the today’s massively-multicore computer hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*-ycSEmPSWZHwzDa-38qoOQ.png" /><figcaption>A Lagom “Hello World” service in Microclimate</figcaption></figure><p>Before we dive in, let’s note for the cynics out there that these advantages aren’t just checkbox-checking technical buzzwords that a marketing department has deemed desirable when associated with microservices. Rather, with Lagom these are technical innovations that are fully baked into the architecture of the Lagom framework and are necessarily a consequence of its design. But before we get to that, let’s start at the beginning!</p><h3>Reactive services and the Reactive Manifesto</h3><p>Lagom is pronounced Luh-gom, after the Swedish word meaning ‘just the right amount’. The reactive services concepts that underpin Lagom (and its parent project Akka) are described in a document called the <a href="https://www.reactivemanifesto.org/">Reactive Manifesto</a>. As per the manifesto, reactive services are responsive (respond quickly to requests), resilient (tolerate errors well), elastic (effortless scaling in arbitrary conditions), and message-driven (asynchronous messaging between services).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*GwFfptN5gYoucvAQ_f8w0g.png" /></figure><p>Reactive is not itself a specific concrete technology: there is no single ‘reactive’ project with source code on a GitHub page, or a single source of Maven dependencies/JAR files that you can add to your project. Rather, reactive is a descriptive term for <em>any </em>technologies that enable the creation of systems that exhibit the capabilities of the Reactive Manifesto.</p><p>Reactive technologies have gained prominence in recent years, with the defining characteristics of those technologies implemented in a number of modern Java libraries, such as Akka Streams or RxJava. The Core APIs used by those modern libraries have been integrated into the Java API as of Java 9, with the introduction of the Reactive Streams API, a “standard for asynchronous stream processing with non-blocking back pressure”, under the Java <em>java.util.concurrent.Flow.*</em> package.</p><h3>IBM+Lightbend = IBM Reactive Platform, incorporating Lagom, Play, and Akka, also included in IBM Microclimate</h3><p>Lagom is based on the Akka framework and supported by the Play framework. All three technologies are incorporated into the <a href="https://developer.ibm.com/technologies/reactive-systems/">IBM Reactive Platform</a> product, which is a ‘collaborative development initiative’ between IBM and Lightbend. If you’re interested in commercial support for these frameworks, plus advanced enterprise features, you should definitely check it out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*OBHoSMFoDK-ITtIGCr0Tug.jpeg" /><figcaption>Layered platform architecture diagram</figcaption></figure><p>For a quick introduction to these technologies, you can use our <a href="https://microclimate-dev2ops.github.io/">Microclimate</a> product to begin to explore the Lagom microservice framework. While it doesn’t include those additional Reactive Platform features, it does allow use of those three foundational technologies: Lagom, Play, and Akka. <a href="https://medium.com/@jgwest/microclimate-a-new-container-based-multi-language-cloud-friendly-development-tool-98a2d03326a9">Microclimate is a container-based development environment</a>, which is available free-of-charge, and which features a built-in browser-based IDE (plus Eclipse and VSCode plugins) for developing Java, Spring, Swift, Node, and Docker-based applications.</p><h3>Why Lagom?</h3><p>You may ask, what makes Lagom the best microservices framework versus the many (many) other technologies that exist?</p><p>According to proponents of Lagom, common modern programming practices do not properly address the needs of demanding modern systems. And by demanding modern systems, they are referring to the capability for modern systems to make available to applications a frankly ludicrous number of CPU cores. Today you can buy CPUs that feature 28 cores (Intel) or 32 cores (AMD) on a single chip die. Even modern smartphones, desktops, and laptops include between three and eight cores as a standard offering.</p><p>As application developers, once we’ve fully saturated the single-threaded performance that we can squeeze out of a single CPU core, the next step for us is to determine how to scale our application across all the remaining cores of the CPU. But critically we need to do that in a way that is both safe and efficient. Now efficient is always good, but safety is crucial: Sharing data between multiple threads is a <a href="https://medium.com/@jgwest/concurrency-pattern-keep-your-java-objects-safe-from-multithreaded-meddlers-1065bbd2a27d">notoriously difficult thing to get right</a>, and Lagom seeks to provide a programming model that makes <em>getting it right</em> easier for developers.</p><p>Lagom also seeks to ensure maximum application scalability in highly demanding conditions. Now, if the goal of your application is to serve only 10 requests per second, or maybe 100 requests per second, you can (arguably) use any modern web technology to write an application that implements this requirement. For example, frameworks that are are based on slower interpreted languages like Ruby and Python are doing this ever day. But what if you want your application to scale to serving thousands or <a href="https://en.wikipedia.org/wiki/C10k_problem">tens-of-thousands of requests on a single machine</a>? With the right technology this is definitely technically feasible, but at this scale, you start to hit fundamental limits of the CPU itself:<br>- <em>Thread Context Switching</em>: How long your CPU takes to switch between thread contexts. <br>- <em>Contention Overhead</em>: How long your CPU threads spend waiting to acquire a resource lock which is owned by another thread<br>- <em>Blocking on I/O</em>: How long your CPU threads spend blocked waiting for I/O requests, such as file/network/database access</p><p>Lagom seems to minimize these bottlenecks by eliminating cross-thread data sharing and blocking synchronous requests, through the user of asynchronous message passing in line with the Reactive manifesto.</p><h3>The Lagom “Secret Sauce”: Actor model of concurrency</h3><p>So, the scalability and performance benefits described above are what makes Lagom special, but how does Lagom realize the vision of Reactive services? What is the technological ‘secret sauce’ that unlocks that promised massive scalability?</p><p>A core feature of Lagom is its use of the <a href="https://doc.akka.io/docs/akka/current/guide/actors-intro.html?language=scala">actor model of concurrency </a>(through Akka), plus smart design decisions and supporting APIs that seek to reduce the friction between how we as developers are traditionally taught to write multithreaded code, versus the asynchronous message passing model prescribed here. The actor model of concurrency is nothing new, as it was first published in 1973, thus we again prove that nothing is new in Computer Science that has not been formally defined and analyzed years before you were born 😄 and/or before the modern personal computer era.</p><p>The actor model of concurrency is best expressed in contrast to the traditional multithreaded Java concurrency model.</p><h3>The traditional multithreaded Java concurrency model</h3><p>In the diagram, you see two threads (red on the left and green on the right) that are both attempting to access a Java object at roughly the same point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*nU9n_1R92QpwgM3Q-7FD7A.png" /></figure><p>Generally speaking, two threads concurrently accessing shared data is fine as long as both threads are only <em>reading</em> that data. But what if one or both of the threads is also <em>writing</em> the data during this time? Now we have a problem: we have entered the oft-nightmarish world of cross-thread concurrency bugs.</p><p>For example, what happens if the green thread reads the data while the red thread is in the middle of updating it? Well, the green thread will get an inconsistent value from this method. What happens if both the green and red threads write data to the object, and both return from the object’s method? Each will incorrectly assume that the data they have written is still present.</p><p>Java gives us a number of concurrency primitives to combat this: the <em>synchronized</em> keyword, synchronized blocks, various lock types, atomic variables, futures, and the thread <em>ExecutorService</em>. But ensuring that all those concurrency primitives are correctly applied across your application is a mammoth task, and ensuring objects in your application are fully and correctly synchronized is a challenge for which limited supporting tools exists. When these types of bugs do arise, often the only solution is a highly caffeinated programmer staring long and hard at the code until the problem becomes clear.</p><p>We know that synchronizing data is hard, but lets presume that somehow you are able to ensure that all shared-thread data is correctly synchronized. Let’s delve deep into our collective imaginations and envision a world where you have synchronized, locked, and futured your way into a complex multithreaded application system which somehow against all odds does not have inconsistent data sharing. 😃</p><p>Even in this magical world of perfect mutexes and locks, you are not yet out of the woods when it comes to concurrency bugs: your application may still experience deadlocks, when two-or-more threads create a dependency cycle by both owning a shared lock and needing a shared lock that another thread owns. In the deadlock scenario, both threads are fully dead in the water without manual intervention.</p><p>Another contention related issue you will experience even with perfect synchronization is race conditions. Back to our thread diagram: imagine a scenario where 95% of the time the red thread updates the data before the green thread, but 5% of the time instead the green thread updates the data first. Timing on thread ordering can affect overall program behaviour, even though the actual code that is executed in both the 95% and 5% scenarios is exactly the same.</p><h3>In contrast to the traditional concurrency model: The actor model</h3><p>In direct contrast to the traditional model of sharing data between threads is the <em>actor model</em>. In this model, an ‘actor’ is a service that contains both some local data, and some local code that is executed on that local data. However, critically, that local code is never allowed to access the data of any other actor. Nor is the thread that actually runs that actor code allowed to touch any other data that is used by any other thread. Actor code is always limited only to that actor’s own data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*XLNDHoe9KadoSZ7TNVEk3A.png" /></figure><p>In this diagram, the blue circles represent these ‘actors’. You can imagine that the top left actor represents a mortgage service which receives customer requests for new mortgages. In order to process a mortgage, a thread (“thread A”) executes the actor code and checks the actor’s local data to see if the customer already has a mortgage. Once this check is complete, the mortgage actor needs to request the customer’s credit rating to determine the mortgage interest rate.</p><p>However, the customer’s credit rating is stored in a separate actor: the blue circle to the immediate right of the mortgage actor. Since the mortgage actor needs to request a credit rating check, it sends an immutable message to the credit rating actor. This immutable credit request message is placed in the incoming mailbox of the credit rating actor.</p><p>At this point, the mortgage actor — having other business to complete — continues to executes any remaining code that does not depend on a response from the credit rating application. At some point the mortgage actor will receive a response to its credit rating request, but it is not blocked on the request and can process other data during this time.</p><p>Independently, on a new thread (“thread B”) the actor code that handles credit ratings is executed. The credit rating code checks its incoming mailbox, and discovers an incoming message from the mortgage actor. The credit actor then checks its internal database of credit ratings, locates the customer’s credit rating, and sends a message containing that data back to the mortgage actor. This new response message is placed in the mortgage actor’s incoming mailbox, which will be processed the next time the mortgage actor code/thread runs. The credit rating thread then continues its other business.</p><p>As you can see, in this scenario these two threads share data between them by passing messages to each other, rather that calling methods on shared Java objects. Notice that at no point did thread A ever have access to the local data of thread B, and vice versa. In both cases, data was shared by passing lightweight message between the two actors.</p><p>This lightweight messaging is the foundation of the actor concurrency model: rather than sharing data by multiple threads calling methods of shared objects, data is passed through lightweight messages that are shared between threads via a messaging system.</p><h3>The actor model drives Reactive programming</h3><p>So that’s the actor concurrency model, and that’s the model that Akka, Lagom, and Play are based on.</p><p>By following the tenants of reactive programming via the actor model, you will get Lagom-based microservices that are:</p><ul><li>Lightweight</li><li>Message-driven</li><li>Use an asynchronous, non-blocking thread model</li><li>Support Apache Kafka for message passing (and general support for non-Kafka message-broker scenarios)</li><li>Provide a holistic solution to developing distributed systems (build tools, test tools, Apache Kafka infrastructure configuration)</li></ul><p>When building within the constraints (‘opinions’) imposed by Lagom, a Lagom-based application will therefore necessarily have the desirable reactive qualities: responsiveness, resilience, scalability, and elasticity. This, combined with the features described above, make it a compelling choice for moving your application development from a monolithic architecture to a scalable microservices-based architecture.</p><h3>Trade-offs</h3><p>Of course, like everything in software development, there are trade-offs to any technical choice and Lagom is no exception. Lagom is a strongly opinionated framework, and like any opinionated framework, the farther you diverge from those opinions the greater the pain. One instance of this is that it can be more difficult to integrate with other (multithreaded) Java libraries that don’t play well Akka’s actor model, for example third-party libraries that manage their own threads with thread pools. These may interfere with Akka’s ability to efficiently pass messages between actors and threads.</p><p>Lagom requires you to split your applications into a set of independent services, which will necessarily be more complex than a traditional monolithic application built on a more traditional framework (but of course with a monolithic application you lose the scaling/performance benefits of Lagom). Lagom requires a greater understanding of the vagaries of distributed computing and concurrent data sharing, in order to avoid the pitfalls/“footguns” inherent to both of these topics.</p><p>Finally, Lagom is arguably a less mature technology than an industry bedrock like Java EE or Spring: Lagom dates back to 2016 (as per the GitHub commits), while Akka has been around since 2009, and the Play framework has been around since 2007.</p><h3>Jump In</h3><p>If you’re looking to dip your toes into the Lagom framework, try creating a Hello World application using IBM Microclimate. <a href="https://microclimate-dev2ops.github.io/">Microclimate</a> is a free-of-charge, container-based, multi-language, cloud-friendly development IDE, that you can download and try right now! Microclimate does not include the ‘advanced’ features of the IBM Reactive Platform, but still incorporates Lagom, Akka, and Play, the fundamental open source technologies of Reactive Platform.</p><p>If you’re in the market for a microservice platform that is developer-focused, enterprise-friendly, and commercially-supported, check out IBM’s Reactive Platform. It’s a collaborative development initiative between IBM and Lightbend to provide reactive technology builds on Lagom, Akka, and Play, while providing advanced features such application management, intelligent monitoring, and enterprise integrations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd8b15c0a3ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Concurrency Pattern: Keep your Java objects safe from multithreaded meddlers]]></title>
            <link>https://medium.com/@jgwest/concurrency-pattern-keep-your-java-objects-safe-from-multithreaded-meddlers-1065bbd2a27d?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1065bbd2a27d</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Mon, 28 Jan 2019 03:15:15 GMT</pubDate>
            <atom:updated>2019-01-28T03:15:15.354Z</atom:updated>
            <content:encoded><![CDATA[<p>As a developer on a large, multithreaded, battle-tested application, I have seen many examples of concurrency — both good and bad. Whether it be cross-thread immutable data sharing (good), synchronization (good), but also thread deadlocks, race conditions, and UI hangs (all bad). Ultimately many of these issues were due to the absent/incorrect usage of the concurrency primitives that the Java language makes available for cross-thread object sharing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*LK-4f6DrUYzF-wN3Gn7PwA.png" /></figure><p>However, even IF you are making use of the built-in concurrency primitives that Java offers (synchronized statements, mutexes, locks, thread executor, etc) there is no easy way to verify that your program IS correctly and consistently handling cross-thread data without requiring extensive testing or analysis. Often you will only see these kinds of issues when your product is under load, and frustratingly enough, only intermittently.</p><p>In many cases what we really want is way to verify <strong>at compile time</strong> that shared data is only ever accessed by a single thread at a time. But we also want a solution that does not impose a significant performance overhead on data access.</p><p>Here is a code pattern that does exactly that.</p><p>First, lets take a look at the types of issues that can arise with multithreaded code.</p><p><strong>Here is a simple, example Customer object:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e177d76d81901986480c931225c90ba3/href">https://medium.com/media/e177d76d81901986480c931225c90ba3/href</a></iframe><p>Our Customer object is a dead-simple object with a name and a list of previous orders. We’re going to use this object to maintain a list of customers (a simple database) to demonstrate bad code.</p><p>There are many problems that can arise from incorrectly using this object across multiple threads, without proper synchronization.</p><p><strong>Problem 1: Developer forgets that synchronized statement only applies to the synchronized method</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94fb4d523f0dcabb3df20d69a82e0774/href">https://medium.com/media/94fb4d523f0dcabb3df20d69a82e0774/href</a></iframe><p>If multiple threads simultaneously call the <em>CustomerUpdate.run</em>() thread, you will intermittently receive a <em>ConcurrentModificationException</em> from inside the <em>run()</em> method when <em>customers.add(…)</em> is called. This is because the synchronized statement on <em>CustomerDb.getCustomers()</em> only applies in the scope of the <em>getCustomers()</em> call, and not outside that scope.</p><p><strong>Problem 2: Developer forgets to synchronize on an instance that should be synchronized</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/192cc2e053876c10136306089f1a8369/href">https://medium.com/media/192cc2e053876c10136306089f1a8369/href</a></iframe><p>In this scenario, all code that accesses the <em>customers</em> <em>List</em> object should be synchronized. However, in <em>removeCustomer(…)</em>, we forgot a synchronized statement, which may cause a <em>ConcurrentModificationException</em> on either the <em>customers.add(…)</em> or <em>customers.remove()</em> call.</p><p><strong>Problem 3: Developer synchronizes on the whole, but not on the part</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1eb096832e5b0a92cd509d58e9cce194/href">https://medium.com/media/1eb096832e5b0a92cd509d58e9cce194/href</a></iframe><p>A developer may correctly synchronize on a parent object and retrieve a child object from that parent object, but not maintain the synchronization context when accessing the child object. Above, we retrieve a list of orders from the Customer object, but then access those orders outside a synchronize on the parent. This may cause <em>ConcurrentModificationException</em>s<em>,</em> but worse, it may not: depending on how a child object is used, the writes to that object may be overwritten by other threads based on data-access ordering and thread scheduling.</p><h4>Solutions to these problems</h4><p>There are many more problems just like these, and while it may be easy to spot why these simple examples are not thread-safe, these are not necessarily representative of real world applications. When developing a large application these types of issues are much better hidden and therefore difficult to spot: the larger the haystack, the smaller the needle.</p><p>The problem of finding thread concurrency issues in applications is further magnified in cases where:</p><ul><li>You didn’t write the code.</li><li>You are fixing existing code, but trying to avoid the need to do a costly review of the code’s thread context.</li><li>You are a human being and thus inherently imperfect at the careful checking of nitty-gritty details 😆 (unlike a computer doing a compile-time static type check, which is our goal with the pattern described by this blog post!)</li></ul><p>Worse still, intermittent bugs are a nightmare to reproduce and then debug. Especially when you aren’t able to reproduce, but your customers can!</p><p>Until we can solve for human infallibility, we need a code pattern to ensure that an object’s data can only ever be accessed by one thread at a time. Likewise, this pattern should use static typing, such that attempts to access the data outside this lock should correctly fail to compile. Finally, to subvert creative workarounds, we should add simple run-time checks (which may, for example, be only enabled in debug/development builds.)</p><p><strong>Here’s how it looks</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f4379d11ef1e12d07d512f6056e3502/href">https://medium.com/media/1f4379d11ef1e12d07d512f6056e3502/href</a></iframe><p>Here, the <em>YourProtectedData</em> class contains some data that you wish to protect from multiple concurrent accesses. In this simple case, the <em>YourProtectedData</em> class contains a single <em>Integer</em> object. While there is a fair bit of (unfortunately necessary) boilerplate, it only needs to be written once, and the actual use of the API is easy, as you can see below.</p><p>With the above pattern in place, we create the shared thread context like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/621226a6921f03fbd953a8f92e2bb000/href">https://medium.com/media/621226a6921f03fbd953a8f92e2bb000/href</a></iframe><p>Why does this work to prevent multiple accesses?</p><p>First, we control how you can get a reference to this object: We only allow access to the object through a custom <em>ISharedThreadRunnable</em> (patterned off <em>java.lang.Runnable)</em>, which provides protected access to the data, and which the user must extend through a lambda expression or anonymous inner class.</p><p>Second, in the implementation of the methods of the contained object (the <em>YourProtectedData</em> class), we also call <em>assertHeldByThread(…)</em> as the first method call in the method. This ensures that we catch cases where the user has (sneakily) moved the data object out of the original thread context (the custom runnable).</p><p>The only way to gain access to the <em>YourProtectedData</em> class is through <em>SharedThreadContext</em>, as:</p><ul><li><em>YourProtectedData</em> is constructed inside <em>SharedThreadContext</em> (rather than outside where the user could then misuse it)</li><li><em>YourProtectedData</em> is a private value (inaccessible to other classes)</li><li><em>YourProtectedData</em> is only shared with classes that call the <em>SharedThreadContext.own(…)</em> method.</li><li>Runtime checks prevent access to the <em>YourProtectedData</em> object, or its instance methods, if the user does somehow gain access to a reference of it outside the intended thread. These runtime checks may be disabled in production to eliminate performance impact, and re-enabled in debug or customer support scenarios.</li></ul><p>Unsafe attempts by the user to acquire this data will fail as follows.</p><p><strong>Attempt 1) — Attempt to access through the SharedThreadContext getter:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69a15fdd7781cec31ecd6c56d12348f0/href">https://medium.com/media/69a15fdd7781cec31ecd6c56d12348f0/href</a></iframe><p><strong>Attempt 2) — A more advanced attempt to subvert thread safety, by moving the protected data between threads:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b8e0d7806df3e237d3f980ecebdd1a8/href">https://medium.com/media/0b8e0d7806df3e237d3f980ecebdd1a8/href</a></iframe><p>In order to ensure that the properties enshrined in this pattern continue to hold:</p><ol><li>The construction of your protected classes must be limited to the <em>SharedThreadContext</em> constructor.</li><li>The private internal reference to your protected classes must not leak out of <em>SharedThreadContext</em>.</li><li><em>YourProtectedData</em> class should hold a reference to it’s parent <em>SharedThreadContext</em></li><li>Each method of <em>YourProtectedData</em> should call the <em>SharedThreadContext.assertHeldByThread()</em> method, as the first call of the method.</li></ol><p>This pattern provides a guarantee that data cannot be incorrectly accessed by multiple threads simultaneously, and provides an ergonomic mechanism to access and manipulate the data (lambda expressions). While there is more boilerplate code then is desirable, this can be further optimized to only what is required by your own code, and ultimately the goal of the pattern is achieved here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1065bbd2a27d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microclimate: A new container-based, multi-language, cloud-friendly development tool]]></title>
            <link>https://medium.com/@jgwest/microclimate-a-new-container-based-multi-language-cloud-friendly-development-tool-98a2d03326a9?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/98a2d03326a9</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Mon, 01 Oct 2018 17:31:58 GMT</pubDate>
            <atom:updated>2018-10-01T17:31:58.302Z</atom:updated>
            <content:encoded><![CDATA[<p>Microclimate is a new free-of-charge development tool from IBM that makes it easy to create Java/Node/Spring/Swift cloud-native microservices. But, unlike traditional IDEs, with Microclimate your full development environment (including the IDE itself) is entirely contained within lightweight Docker containers that are running on your local machine (or optionally, in Kubernetes). Now, you may ask, why have we at IBM created yet another development tool? Excellent question… read on to find out!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78NV4nuvWcsLkqnMP_2jew.png" /><figcaption>Node.js application in Microclimate code editor</figcaption></figure><p>Microclimate is our team’s vision for the future of microservice/cloud-native application development tools, targeting a full range of cloud environments. This include Docker-based containers on IaaS VMs, hybrid cloud environments through IBM Cloud Private, public cloud Kubernetes hosted on the IBM Cloud Kubernetes service, as well as general support for similar environments in the clouds of other major providers.</p><p>As of this writing, the Microclimate project is still new and growing, with it’s first public release dating back to only February 2018 (and it’s first internal release not far behind that).You can learn more by visiting our product page at <a href="https://microclimate-dev2ops.github.io/">https://microclimate-dev2ops.github.io/</a>.</p><p>There are many features that set Microclimate apart from other development environments. Here are just some of the highlights:</p><h3>Polyglot development environment with code intelligence in real time</h3><p>Microclimate is built from the ground up to support the polyglot development model, where small independent applications (ie microservices) written in different languages/runtimes combine to form a single large application. This allows you to choose the best language/runtime combination for the job at hand: for example, a Node.js-based application may prove most adept at implementing a Web frontend that interacts with the browser, while Java may be better suited to backend business logic, or <a href="https://www.ibm.com/cloud/swift">Swift</a> better suited to a mobile applications frontend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yhg8SJuukFSF3_cipTWwrQ.png" /><figcaption>Create a new project in the language and runtime of your choice</figcaption></figure><p>This multi-language multi-runtime world requires a development tool that treats all environments as equal first-class citizens: these different environments should run side-by-side in the UI, while still providing the features that developers expect from an IDE for that language (for example, content assist/intellisense). Microclimate lives in this brave new polyglot world, allowing you to create Java/Node/Spring/Swift applications that work together and exist side-by-side within a single consistent UI. Within this environment we include full real time code intelligence powered by the <a href="http://langserver.org/">Language Server Protocol</a> (LSP) technology. Microclimate is one tool to rule them all!</p><h3>Fully self-contained inside containers, whether on your local desktop or on the cloud</h3><p>When you <a href="https://microclimate-dev2ops.github.io/installlocally">download the Microclimate installer</a>, you will see it is a tiny file: only about 750KB. This is because when you extract and run it, the full contents of Microclimate will download as a self-contained set of Docker images. Microclimate exists entirely within these pre-built Docker images, and the footprint on your machine that is outside these containers is limited to the source code of your Java/Spring/Swift/Node projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11pga94NTTpMbGRYynNaFA.png" /><figcaption>Microclimate IDE and user applications running in self-contained Docker containers</figcaption></figure><p>Microclimate’s container-based architecture enables the polyglot development model described above by <em>automatically</em> downloading the required language/runtime development tools when you first begin developing in a new language.</p><p>For example, when you create or import a Java application, Microclimate will automatically download the Java JDK (the compiler and Java APIs), Maven (build system), and WebSphere Liberty Docker images (web container). Unlike traditional IDEs that require you to download/install these language-specific packages on your own, Microclimate grabs them on your behalf both automatically and seamlessly. And it’s not just Java: the Node, Spring, and Swift development/ecosystem tools are fully supported as well.</p><h3>Develop using the Microclimate browser-based UI, or using your local IDE of choice</h3><p>Microclimate gives you the choice of which editor or IDE you wish to use to develop. Included as part of the Microclimate browser-based UI is a code editor based on <a href="https://www.theia-ide.org/">Eclipse Theia</a>, which is a open source project designed from the ground up to provide a rich development environment delivered through web technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W8owTIHC-oKfZqMMGuQ1yg.png" /><figcaption>The Theia-based code editor, editing a Java class</figcaption></figure><p>In addition to developing using Microclimate’s integrated Theia editor, you can also develop using the IDE of your choice, whether it be Eclipse, VSCode, IntelliJ Idea, Sublime/Atom, etc. Code that is generated or imported into Microclimate is available outside the container as well, using a Docker bind mount, which means that code can easily be imported into your own IDE as easily as importing code from another local directory.</p><p>All your of Microclimate-based project files are accessible as a folder on your local machine under the ‘<em>microclimate-workspace’ </em>directory. Any changes you make here will be visible both inside and outside the Microclimate UI, and from here you may access the other Microclimate functionality. More <a href="https://microclimate-dev2ops.github.io/settingownide#doc">information about using an IDE is here</a>.</p><p>Even if you are not using the Microclimate code editor you can still take advantage of Microclimate’s many other features, including self-contained application containers, internal build system, build and application logs, integrated performance metrics, and Jenkins-based delivery pipeline.</p><h3>Fast development inner loop</h3><p>As developers, our general day-to-day development workflow looks like this: we write some code, we wait for that code to build, we deploy that build, and then we run our application on that deployment. Hopefully our new code made our application better rather than worse, but using feedback from our changes we then do some more coding. Next we wait for that code to build, deploy it, and run it. That loop, what we refer to as the <em>development inner loop </em>— code, build, deploy, run — must be as fast possible, in order to keep your developers both happy and productive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*fyxB0jispmJ-9iVuztsiAQ.png" /><figcaption>The Java application has built, and is now being packaged into a container</figcaption></figure><p>Personally as a developer I like to be as happy and productive as possible 😄, and as such in Microclimate this fast turnaround time has been a core focus. We want to avoid the scenario you may see in some other container-based development tools where the build and deploy steps take <em>much</em> longer than they should, leaving the programmer to get <a href="https://xkcd.com/303/">distracted by nearby shiny objects</a>.</p><p>In Microclimate, using various clever Docker/Kubernetes tactics, we have achieved that fast turnaround time, reducing our waiting time to as slim a slice as possible.</p><h3>From day one, applications are running inside Docker</h3><p>Microservice projects that are generated by Microclimate are container native. That is, they are hosted in, built by, and deployed to, Docker containers, with the container images themselves being as close as possible to the Java/Node/Spring/Swift production containers you’ll find on Docker Hub. All of this combines to ensure that not only is it easy to build and run your project, but also that it will be seamless to transition that container into production.</p><p>Behind the scenes, when you create a new project or import an existing one, we automatically create a Dockerfile for that project as well as configuring an automated build (for example, a Maven build for Java). Microclimate then manages the lifecyle of that application, building and hosting it as a containerized application.</p><p>At any time you can see your application happily running along if you run the <em>docker ps</em> command (to list all running containers), or from within the deployment resource list in the Web UI of IBM Cloud Private (where containers are run at ‘enterprise scale’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*Ut50KiDpdkzzvxHlvoLkig.png" /><figcaption>Applications developed inside Microclimate appear as Kubernetes deployments, and as IBM Cloud Private Helm Releases</figcaption></figure><p>And to double-down on our ease of deployment, with just a few clicks Microclimate will create a Jenkins-based delivery pipeline to automatically deploy your containers to <a href="https://www.ibm.com/cloud/private">IBM Cloud Private</a> or the <a href="https://www.ibm.com/cloud/container-service">IBM Cloud Kubernetes Service</a>.</p><h3>Integrated Jenkins-based DevOps pipeline for pushing to IBM Cloud Private / IBM Kubernetes Service on IBM Cloud Public</h3><p>So you’ve developed an application in Microclimate, and now you need somewhere to host it: Microclimate makes it easy to create a deployment pipeline between your microservice project and the target IBM Cloud Kubernetes server.</p><p>With just a few clicks inside the Microclimate UI, a Jenkins-based delivery/deployment pipeline is configured in your target cloud environment. From that point on, any commits made to your projects will trigger a build that will then be packaged into a Docker image and uploaded to the target Docker image registry. The applications deployed by the Jenkins delivery pipeline are managed as Helm charts (linked to the Jenkins-built image in your Docker registry), and thus are visible with the other Helm releases from within the IBM Cloud Private or IBM Kubernetes Service user interfaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fA8iQ8DZq7KOK2f3V3XfTw.png" /><figcaption>Delivery pipeline based on Jenkins, automatically created and configured by Microclimate</figcaption></figure><h3>A sleek, modern, clean UI with a focus on low barrier to entry</h3><p>In less than a minute and with only a small handful of steps, you can go from starting the Microclimate environment for the first time, to creating and running your first Java/Node/Spring/Swift microservice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rat-FL6-pZGWjJqE2UE_WA.png" /><figcaption>Welcome to Microclimate!</figcaption></figure><p>When designing our user interface we exercised careful, considered focus to ensure that it is intuitive to new users, thus minimizing the barrier-of-entry to getting a real application up and running in the language and runtime of your choice.</p><p>You can throw out the excessively long step-by-step guides required by other IDEs in order to create and deploy a new project, because with Microclimate each next step is obvious and intuitive. Skeptical, or, curious to know more? Browse through the screenshots in this article and then try it for yourself!</p><h3>Integrated performance metrics</h3><p>If you’ve ever hit a Java <em>OutOfMemoryException</em>, or been baffled by why your supposedly lightweight code is eagerly consuming a full CPU, then you have been “<em>lucky”</em> enough to work with code that strains the limits of its virtual environment. Finding and fixing performance issues such as these can be tricky, and having the right performance tool to point you in the correct direction is key.</p><p>That’s why every application running in MC is automatically configured to run with performance monitoring by default, which includes <em>CPU usage</em>, h<em>eap usage</em>, <em>% of time spent garbage collecting</em>, <em>HTTP Incoming Requests</em>, <em>HTTP Throughput</em>, and <em>maximum heap used after GC</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3aitfFxbUKm8rSS6wdjdLg.png" /></figure><p>Also included is response time monitoring for individual HTTP endpoints, which includes <em>total endpoint hits</em>, <em>average response time</em> (ms), and <em>longest response time </em>(ms).</p><p>In addition we include the ability to further drive application load to test performance using <a href="https://jmeter.apache.org/">JMeter</a>, which you can kick off by clicking <em>Run load</em> in the <em>App monitor</em> tab.</p><h3>Front-end based on Web technologies</h3><p>In the past, the user interface of an IDE would traditionally have been coded using a native widget toolkit specific to a particular operating system, in tools like XCode (Cocoa), Visual Studio (Windows Forms/WPF/MFC/etc), or Eclipse (SWT, which interfaces with GTK/Cocoa/Windows).</p><p>In recent years, however, Web technologies like HTML, CSS, JavaScript, the HTML Canvas, and WebGL have been used to create richer user interfaces, as seen in tools like Visual Studio Code, Slack, and Atom. Applications based on Web technologies mostly maintain the overall feel of a native application, though not necessarily the specific look of one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oS-yUO6j0rHupjtFF-ymLg.png" /><figcaption>The Microclimate Pipeline tab, which is one of many components using the IBM Carbon Design System UI</figcaption></figure><p>Microclimate is based on those same web technologies, both through our browser-based code editor Eclipse Theia, but also through the suite of tools that are integrated into Microclimate at the browser level. This ensures that Microclimate provides the same rich UI of other competing tools, while enabling fluid animation, vibrant colours, sophisticated layouts, and custom widgets.</p><p>Our UI is based on IBM’s <a href="http://www.carbondesignsystem.com/">Carbon Design System</a>, which is built to provide beautiful and intuitive designs that are logically oriented around consistent guidelines. Not only is the Microclimate UI attractive and modern, but it also shares a unified design experience with our sister products in IBM Cloud and IBM Cloud Private.</p><h3>Fully at home on IBM Cloud Private</h3><p>Microclimate can be hosted either locally or on IBM Cloud Private (based on Kubernetes). This allow you to host multiple self-contained developer environments in a private cloud environment. Microclimate containers hosted on IBM Cloud Private can be installed through Helm, either through the catalog UI or the Helm CLI tool. Ultimately hosting on IBM Cloud Private means that the resources of the cluster are distributed across developers, and are fully managed inside the cloud, with developers requiring only a thin client (the web browser) to connect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1_81LF8NZKyJajbFoPw6A.png" /><figcaption>Microclimate can be installed directly from the IBM Cloud Private catalogue.</figcaption></figure><p>Hosting your development environment in the cloud has the advantage of persisting your workflow across individual machines and across physical contexts: if you start developing your code, but turn off your machine and head out of the office, when you later return to your cloud-based environment it will be exactly as you left it. Cloud-based development environments are likewise not tied to a specific machine, but are accessible from anywhere.</p><h3>Best of all, it’s available right now!</h3><p>The best part of Microclimate is that you don’t have to take my word on all of its great features… you can try it out for yourself right now! Microclimate is available for download from <a href="https://microclimate-dev2ops.github.io/">our website</a>.</p><p>The installer is less than a single megabyte, and running the installer will automatically download Microclimate into self-contained Docker images hosted on your local machine.</p><p>So try it out, and let us know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98a2d03326a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to keep your users from running away: triaging bugs and features on large projects]]></title>
            <link>https://medium.com/@jgwest/how-to-keep-your-users-from-running-away-triaging-bugs-and-features-on-large-projects-3a63039d48a7?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a63039d48a7</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[bugs]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Tue, 28 Aug 2018 16:40:02 GMT</pubDate>
            <atom:updated>2018-09-10T15:40:40.940Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine you’re a developer on a large project with a sizable customer base, and as a developer you are tasked with prioritizing features and defects on an ever-growing defect backlog. For many, this is not difficult to imagine 😄.</p><p>The question you must first ask is, what bugs are worth fixing and which aren’t? How should we distinguish between the two? How should we balance that work against feature work?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BvKl3K72srQ8RUyaXqWxew.jpeg" /><figcaption>Good old-fashioned triage: fortunately, as developers our triage is not usually life and death</figcaption></figure><p>When senior leaders are pushing hard for new features and developers want to work on those new features (<em>“the old stuff isn’t that interesting”</em>), it can be easy to fall into the habit of letting lower severity defect work slide in favour of more exciting tasks. But this is a slippery slope that must be carefully navigated lest you alienate your existing users.</p><p>As developers, we must consider that customers are using our product to get real work done, and all the bugs they encounter ultimately frustrate that effort no matter how minor those bugs may seem to us. Even if users never encounter a <em>major</em> or <em>show-stopping</em> bug, you still risk a <em>death by a thousand cuts</em>: if folks hit enough of your lower severity defects, they will lose faith in your ongoing support of your product, and go looking for alternatives (<em>“this product is becoming more dysfunctional every day”</em>). Likewise, if folks that are new to your product hit bugs during the crucial introductory phase, they will quickly move on (<em>“I can’t invest my developer’s time on a product with these software quality issues”</em>).</p><p>To avoid falling into that trap we need to understand the scope of the work, set guidelines to fix it, and mitigate/document what can’t be fixed:</p><ol><li>First understand the shape/velocity of your defect backlog.<br> — Are we fixing everything that comes in equally quickly, or are our most important bugs languishing?</li><li>Restrict work on new features in a way that is consistent with your defect goals <br> — Example: “<em>any time our average defect age is above 12 weeks, we will focus on defects until that average is below 9 weeks</em>”</li><li>If you can’t fix a bug, mitigate it. If you absolutely can’t mitigate a bug, then document it. <br> — Documenting a bug as a known issue is fine, but is very much a last resort versus fixing/mitigation</li><li>Finally, recognize that your users usually won’t tell you when they hit your bugs, they may simply give up and switch to using a competitor’s product. <br> — While this is slightly less true in the age of Stack Overflow, you still can’t assume that you have anything close to full visibility over how users are using your product.</li></ol><h4>First, understand the shape/velocity of your defect backlog</h4><p>That the <em>size of a bug backlog</em> is important, is intuitive: we all know how many bugs we have and what their severity are. But there are often other useful metrics we can gather that may better inform planning.</p><p>One useful metric is the <em>distribution of defect age</em> which we can call the <em>shape</em> of the data, as it quite literally corresponds to the geometric curve of a defect age graph. This shape can give us critical insights: What is the average age of your open bugs and what is the median? Are the mean and median significantly different? If so, this would imply that your distribution is a lot of young bugs, and some much older bugs. Is it a short tail (everything is fixed quickly) or a long tail (a small number of defects hang around for years)? Is the pace of new defect creation increasing or decreasing over time? This defect velocity may bode poorly if development work is introducing, rather than fixing, more bugs over time.</p><p>Another useful metric is the number of <em>defects per component </em>or <em>defects per code owner</em>: If we are planning our next release based on which folks are available to work on which pieces, it is crucial to recognize that only certain individuals are able to efficiently fix bugs in their area of specialization. For example, if I am responsible for fixing a large number of data parser bugs, and no one else can work on those (or at a minimum I am much more productive in this area due to code familiarity), my pile of technical debt is much larger than others. In this scenario this may be a signal that greater priority should be placed on fixing those bugs, or that another developer should begin specializing in this area as well.</p><p>Ask yourself: How long is it from when a defect is discovered, to when a fix is delivered? Or, said another way, how long do users have to wait for a fix from when it is discovered? Similar metrics to these are used in technical support (<em>average time to ticket closure</em>) and they are equally useful in development as well, as they capture the full lifecycle from problem to solution.</p><h3>Restrict work on new features in a way that is consistent with your defect goals</h3><p>Once you have an understanding of what your defect backlog looks like, and what you want it to look like, you can move on to planning how to fix it. Ultimately it is up to your team to decide how best to shape your product plans, while taking into account both developer engagement dynamics and product requirements. However, setting a firm policy — when backed by legitimate quality concerns — can ensure that everyone is on the same page and working towards ensuring a long and healthy future for the product (and the developers involved 😄).</p><p>In a trade-off between feature/defect work, here are some example scenarios and guidelines one could use to shape planning:</p><ul><li><em>Our median or average defect age is consistently rising!</em> This likely means defects are not being fixed quickly enough, or insufficient priority is being placed on them. <em>As soon as our average defect is older than 12 weeks, we should solely focus on bug fixing</em>.</li><li><em>Our number of open defects is increasing much faster than it used to!</em> Even if those bugs are ultimately being fixed at a reasonable pace, we should consider <em>reducing product scope or paying down technical debt</em>.</li><li><em>The code that developer A owns has too many bugs!</em> They should fix those before moving on to new work, or we should bring developer B on to work alongside developer A on that code.</li></ul><h4>Fix. Mitigate. Document. — Fix the bug if possible. If not, then mitigate it. If not, then, only as a last resort versus mitigating/fixing it, document it</h4><p>Just because you don’t plan to fix a bug, it doesn’t mean your work is done. Recognize that out of sight is not out of mind for your customers: while the effort required to fix your bug may not exceed your cost-vs-benefit-to-fix threshold, this does not magically prevent customers from hitting it. While you may have long forgotten your unfixed bugs, your customers haven’t, and they will indirectly remind you when you lose their marketshare/mindshare to a competing, less bug-addled product!</p><p>If you don’t plan to fix a bug, consider instead mitigating it in code, to ensure that customers know what to do when they encounter it. For example, it is often easier to detect the failure condition and present a workaround to the user, then to fully fix a lower severity yet problematic bug. Finally, if you absolutely cannot fix or mitigate as above, then should you document any unfixed and unmitigated issues as known issues.</p><ol><li><strong>Fix what is most important</strong><br>- Prioritize bugs that would be seen by a lot of people (<em>incidence rate</em>), are more serious if they are encountered (<em>severity)</em>, or are easier to fix (<em>developer time to fix</em>)<br>- The metric to maximize here is bugs with the highest <em>(severity)x(incidence rate as population %)/(hours to fix)</em> value, and then placing a priority on items with higher values</li><li><strong>Mitigate what you can’t fix</strong><br>- If a bug is too time consuming to fix correctly, you should instead inform the user how to resolve it when it does occur, and attempt to reduce the overall bug incidence.<br>- Mitigation is a code-based solution to help users workaround the problem: ideally the solution should be presented to the user from within the product UI, as users (including we programmers) don’t read documentation except as a last resort.</li><li><strong>Document what you can’t mitigate: Known issues + workarounds.</strong><br>- Document known issues (and workarounds) for any bugs you won’t mitigate or fix.<br>- BUT, as above, often folks don’t read documentation unless it is absolutely necessary to proceed in their task😄.<br>- For this reason, bugs should be fixed or mitigated if possible, and documenting and then dropping known issues should be the least preferred versus other strategies.</li></ol><h4>Don’t assume that your customers aren’t hitting your bugs: they just aren’t telling you about them</h4><p>It is convenient to think that users will let you know whenever they encounter a bug, no matter how insignificant, so that this feedback can inform your future defect fixing. Unfortunately this really isn’t the reality of how software products are used. When most users encounter a <em>non-show-stopping</em> bug they will just shrug and move on. A small percent will open an issue (on open source projects) or a support ticket (on proprietary products), however, the majority won’t. An even smaller percentage will provide ongoing feedback beyond the initial submission (ie test your fix, then verify in a build), to allow you to follow that defect to its logical conclusion.</p><p>Historically, users are not always engaged with bug fixing unless it is directly impeding their work, due to the real (or imagined) hurdle that can come with creating a useful defect report, due to other time commitments, or just due to inertia. Likewise we should not and cannot expect them to be engaged: a user’s time is a valuable commodity that we can’t necessarily ask them to spend on our behalf. This is especially true with proprietary products, as finding answers can also involve navigating the labyrinthine documentation repositories and software support processes of large companies (whereas users of volunteer-based open source projects have different expectations).</p><p>Ultimately you will hear from users if the bugs are pernicious, or if the users themselves are highly engaged. Otherwise, you will hear nothing.</p><p><em>(</em><a href="https://en.wikipedia.org/wiki/Triage#/media/File:Triagemexico.jpg"><em>Medical triage image by CC BY-SA w/ source</em></a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a63039d48a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Many Flavours of DevOps - 2018 Edition]]></title>
            <link>https://medium.com/@jgwest/the-many-flavours-of-devops-2018-edition-78b1159d496f?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/78b1159d496f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ibm]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Tue, 19 Jun 2018 18:04:13 GMT</pubDate>
            <atom:updated>2018-06-19T18:04:13.728Z</atom:updated>
            <content:encoded><![CDATA[<p>While DevOps encompasses a wide variety of topics, it is principally focused around the automation of various aspects of software development and deployment. This includes everything from the ability to run automated unit tests on a developer’s Git deliveries (<em>continuous integration</em>), to provisioning cloud infrastructure (“<em>infrastructure as code</em>”), to configuring applications as they are deployed to production machines (<em>software configuration management</em>).</p><p>Each of these topics can be broken down into these <a href="https://en.wikipedia.org/wiki/DevOps">prescribed toolchain categories</a>: <strong>Code</strong>, <strong>Build</strong>, <strong>Test</strong>, <strong>Package</strong>, <strong>Release</strong>, <strong>Configure</strong>, <strong>Monitor</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*iWQIdDTK9YlxRwZCZd813A.png" /></figure><p><strong>1) Code — Source code management, review</strong><br>- GitLab (open source), Github/Github Enterpise, Bitbucket, Gogs/Gitea (open source)<br>- Review: Gerrit/Phabricator (open source)</p><p>Git has emerged as the dominant version control system (VCS) over the last several years, beating not only centralized VCS products (CVS, SVN, Rational ClearCase/Team Concert, MS Team Foundation Server, etc) but also more modern decentralized VCS projects (Darcs, DVCS, Bazaar, Mercurial). While a major reason for that popularity is likely the success of GitHub, other Git-based products exist including GitLab (open source) and Atlassian BitBucket.</p><p>One may also wish to establish a code review requirement for all commits, using a project like Gerrit or Phabricator (both open source). While we coders may naturally balk at the idea of having to justify our code to our peers, it is hard to argue with the results: versus other QA practices, it is one of the most cost-effective methods for finding and fixing bugs.</p><p>On <a href="https://console.bluemix.net/">IBM Cloud</a>, we provide a managed GitLab-based SCM service through the <a href="https://console.bluemix.net/docs/services/ContinuousDelivery/index.html">Continuous Delivery service</a>, which also includes an integrated delivery pipeline, an in-browser code editor, issue tracking, and more. Likewise we have full integration with public GitHub and may other third-party services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKqvQgXAZrtl1rXbE4oTMQ.png" /><figcaption>GitLab on Continuous Delivery service on IBM Cloud</figcaption></figure><p><strong>2 and 3) Build/Test — Continuous integration / continuous delivery / continuous testing</strong><br>- Package management systems (Language runtime specific tools): JS: NPM; Java: Maven/Gradle; Rust: Cargo; etc<br>- CI/CD: Jenkins (open source), Travis CI (open source<em>ish</em>)</p><p>In most cases your build tool options are largely determined by your language runtime: For Java the most popular options are Maven and Gradle, for JS you have Grunt/Gulp/NPM/etc, for Rust you have Cargo, and so on. In each case the choice is mostly down to personal taste: for instance, for Java builds do you prefer Maven’s opinionated XML-based declarative build files, or do you prefer Gradle’s less-opinionated Groovy-based build files?</p><p>Once you have automated your software build processes, you can next incorporate these builds into an automated continuous integration/continuous delivery service like Jenkins (open source) or Travis. Based on your Jenkins/Travis configuration, your build may run either per-commit or on a scheduled basis (eg a “nightly build”). Likewise automated tests can be configured to run after the build has completed, to verify the correctness of the build artifacts (once again either per commit or scheduled).</p><p>On public IBM Cloud, the delivery pipeline of the <a href="https://console.bluemix.net/docs/services/ContinuousDelivery/index.html">Continuous Delivery service</a> handles much of the heavy lifting here. Each commit to Git will trigger an event in the delivery pipeline service, causing it to build, run automated tests, then deploy the resulting artifacts to IBM Cloud. The integrated delivery pipeline supports all the IBM Cloud deployment options, include Cloud Foundry, Kubernetes (<a href="https://www.ibm.com/cloud/container-service">IBM Container Service</a>), and IaaS VMs (SoftLayer).</p><p>On IBM Cloud Private (based on Kubernetes), you can use our new Microclimate tool to create and build microservices, then use Microclimate’s Jenkins-based delivery pipeline to automatically deploy your microservices to IBM Cloud Private (using Helm). More information on Microclimate is available here on our <a href="https://microclimate-dev2ops.github.io/">Microclimate GitHub page</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3QDI6XqHDLpQaW5Q9zJow.png" /><figcaption>Delivery pipeline of Continuous Delivery service</figcaption></figure><p><strong>4) Package — Build artifact repository manager</strong><br>- JFrog Artifactory, Sonatype Nexus, Apache Archiva (open source)</p><p>You’ve successfully built and blessed your builds, but now where should you put them? Artifact repositories like JFrog’s Artifactory, Sonatype Nexus, or Apache Archiva all exist to provide a centralized location to store, version, publish, and manage artifacts and their dependencies. A large application, such as a microservice-based application, may contain tens or even hundreds of interdependent modules, and private artifact repositories allow them to be managed and consumed by build tools like Maven/Gradle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*ZQdN5vhoNC9m_Tv4Ls7fJg.png" /><figcaption>JFrog Artifactory containing a list of available Maven artifacts</figcaption></figure><p><strong>5) Release — Change management, release automation</strong><br>- IBM UrbanCode Deploy/Release</p><p>The <em>release</em> category of DevOps is focused on release automation, which is the process of packaging the build artifacts of an application in an automated, reliable, auditable fashion. This category is traditionally the domain of large enterprise software products like IBM’s UrbanCode Deploy/Release. There is some overlap here between release automation and software configuration/management (the next step), but principally this category is about blessing a set of build artifacts as a ‘release’, potentially requiring release approvals at each stage, providing those approved artifacts to the configure step below, and then providing the after-the-fact ability to verify/audit which build artifacts were included in which previous historical releases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7ToFDTucmld0Q9HkeVGgw.png" /><figcaption>UrbanCode Deploy deploying JPetStore</figcaption></figure><p><strong>6) Configure — Infrastructure configuration, Software configuration management, “<em>Infrastructure as Code</em>”</strong><br>- Chef, Ansible, Puppet, Salt (open source)<br>- Terraform (open source)<br>- IBM UrbanCode Deploy, IBM Cloud Automation Manager</p><p>Once you’ve built and blessed your built artifacts, and combined them into a release, you need to actually get those artifacts onto production machines 😄.</p><p>The concept of ‘Infrastructure as code’ has emerged to describe the capability of modern tooling to utilize user-written code (or JSON/YAML/Terraform’s HCL) to automatically provision machines, configure the operating system, deploy/configure any required the middleware, and then deploy/configure your application.</p><p>There are many tools available in this category, and which tool you use is once again based on individual taste (push vs pull, declarative vs procedural, agent vs agentless), configuration language (Ruby/Python/custom declarative/HCL), popularity, integration with your existing infrastructure, etc.</p><p>A great place to start is the community plugin repositories for each of the products. If you are looking to deploy/configure a specific product, you may search the various tool’s plugin repositories to see if there are existing plugins available for that product. For example, if I wanted to install and configure IBM WebSphere Liberty, I can visit the Chef Supermarket and find the <a href="https://supermarket.chef.io/cookbooks/was_liberty">was_liberty cookbook</a>, which is an existing plugin that may be used to install/configure Liberty using Chef.</p><p>Community plugin repositories exist for each tool: <a href="https://supermarket.chef.io/">Chef Supermarket</a>, <a href="https://galaxy.ansible.com/">Ansible Galaxy</a>, <a href="https://forge.puppet.com/">Puppet Forge</a>, and <a href="https://developer.ibm.com/urbancode/plugins/">UrbanCode Deploy Plugins</a>.</p><p><a href="https://developer.ibm.com/cloudautomation/">IBM Cloud Automation Manager</a> provides the capability of managing public infrastructure (IBM Cloud, Amazon, Azure, GCP), private infrastructure (OpenStack, VMWare), and deployments to IBM Cloud Private (Kubernetes). VMs/containers/ are provisioned across environment using Terraform, and software/middleware is deployed and configured using Chef and/or UrbanCode Deploy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cxIoYO1BwAFczDJXGrOU8A.png" /><figcaption>IBM Cloud Automation Manager Template Designer</figcaption></figure><p><strong>7) Monitor — Application monitoring</strong><br>- Prometheus, ELK, Grafana (all open source)</p><p>Last but not least, once your application is deployed into production, you need to ensure that it is performing as expected. There are many solutions, but various combinations of software exists including Prometheus, Elasticsearch/Logstash/Kibana (ELK ) and Grafana.</p><p>IBM Cloud provides Kibana through the <a href="https://console.bluemix.net/catalog/services/log-analysis">Log Analysis service</a>, and Grafana through the <a href="https://console.bluemix.net/catalog/services/monitoring">Cloud Monitoring service</a>. IBM Cloud Private incorporates Grafana and Prometheus, both of which are built into IBM Cloud Private by default and are available for use by containers. An ELK stack (Elasticsearch, Logstash, Kibana) is available as an optional install via Helm chart from the IBM Cloud Private catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vLKG53W9BgJw6A8eyq1lPQ.png" /><figcaption>Kibana on IBM Cloud</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78b1159d496f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debugging Eclipse’ SubProgressMonitor and SubMonitor: 0% work completed or work not reported]]></title>
            <link>https://medium.com/@jgwest/debugging-eclipse-subprogressmonitor-and-submonitor-0-work-completed-or-work-not-reported-f482c71cc85c?source=rss-75e41f0900a0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f482c71cc85c</guid>
            <category><![CDATA[eclipse]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Jonathan West]]></dc:creator>
            <pubDate>Wed, 05 Apr 2017 17:26:32 GMT</pubDate>
            <atom:updated>2019-02-06T00:19:51.841Z</atom:updated>
            <content:encoded><![CDATA[<p><em>SubProgressMonitor</em> and <em>SubMonitor </em>are Eclipse classes used to indicate progress on a Job or task running in the Eclipse workbench. <em>SubProgressMonitor</em> was deprecated in a semi-recent Eclipse release, and the preferred (and seemingly better) progress monitor class is <em>SubMonitor</em>. If you are creating new classes you should definitely use <em>SubMonitor</em>: more information is available from this Eclipse article (<a href="https://eclipse.org/articles/Article-Progress-Monitors/article.html">https://eclipse.org/articles/Article-Progress-Monitors/article.html</a>).</p><p>However, if you are working on a mature code base with a long and storied history, it is likely that your code will contains references to both <em>SubProgressMonitor </em>and <em>SubMonitor</em>. While migrating from <em>SubProgressMonitor </em>to <em>SubMonitor </em>is preferable, this is not always possible.</p><p><strong>SubProgressMonitor’s basic pattern is try / beginTask / new / worked / done</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7449b8144ac170f45e9a5de40adf57ee/href">https://medium.com/media/7449b8144ac170f45e9a5de40adf57ee/href</a></iframe><p><strong>SubMonitor’s basic pattern is convert / split / (worked)</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/189323b7cfbc347b4d001bbc213b5eab/href">https://medium.com/media/189323b7cfbc347b4d001bbc213b5eab/href</a></iframe><p>While working with code that used both <em>SubProgressMonitor </em>and <em>SubMonitor </em>interchangeably, I discovered that work that being reported (split/worked) from child <em>SubMonitors </em>was not being reported back to parent <em>SubProgressMonitors</em>. Searching the web for obvious solutions revealed no suggestions, so it was time to pull the code to both classes and see what was happening.</p><p>Both <em>SubMonitor</em> and <em>SubProgressMonitor </em>are implemented in the <em>rt.equinox.bundles repository</em>, in the <em>org.eclipse.equinox.common plugin.</em></p><p><strong>Main repo</strong>:<br><a href="https://git.eclipse.org/c/equinox/rt.equinox.bundles.git/">https://git.eclipse.org/c/equinox/rt.equinox.bundles.git/</a></p><p><strong>Github mirror</strong>:<br><a href="https://github.com/eclipse/rt.equinox.bundles">https://github.com/eclipse/rt.equinox.bundles</a></p><p><strong>To pull the repo, use</strong>:<br>git clone git://git.eclipse.org/gitroot/equinox/rt.equinox.bundles.git</p><p><a href="http://help.eclipse.org/neon/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fcore%2Fruntime%2FSubMonitor.html"><strong>SubMonitor</strong></a><strong> is implemented in</strong>:<br>rt.equinox.bundles\bundles\org.eclipse.equinox.common\src\org\eclipse\core\runtime\SubMonitor.java</p><p><a href="http://help.eclipse.org/neon/topic/org.eclipse.platform.doc.isv/reference/api/org/eclipse/core/runtime/SubProgressMonitor.html"><strong>SubProgressMonitor</strong></a><strong> is implemented in</strong>:<br>rt.equinox.bundles\bundles\org.eclipse.equinox.common\src\org\eclipse\core\runtime\SubProgressMonitor.java</p><p>Import the <strong>org.eclipse.equinox.common</strong> plug-in into your workspace (<strong>Import &gt; Git &gt; Projects from Git</strong>).</p><p>( If you are targeting pre-Neon Eclipse release, use <em>newChild(…)</em> instead of <em>split(..)</em>. As per the above Eclipse article, “the two do pretty much the same thing except that <em>split(…)</em> also performs cancellation checks.” )</p><p>The specific issue I encountered was that progress (work ticks) were not being reported back to the user in the Eclipse workbench UI, even though methods were passed an <em>IProgressMonitor</em>, and were correctly calling <em>SubMonitor</em>.(convert/newChild).</p><p>There can be a few reason for this:<br>1) A <em>NullProgressMonitor </em>was substituted for a Job-backed <em>IProgressMonitor</em><br>2) The tick scale is too large (eg you have indicated that your work will include 1000 ticks, but you have only indicated work of 1–10 ticks, which may round down to 0%)<br>3) One of the parent monitors is discarding the work due to a previous breach of the <em>SubProgressMonitor</em> ‘contract’.</p><p>Each <em>SubMonitor</em>/<em>SubProgressMonitor </em>has a single parent. These monitors form a single-branched hierarchical tree of monitors, beginning with a progress monitor provided by an Eclipse Job. <em>SubProgressMonitor</em> and <em>SubMonitor </em>may be used together, so a monitor tree may look like this:</p><pre>IProgressMonitor provided by job (actually an inner class of Job implementing the interface)<br>\_ SubProgressMonitor<br> \_ SubProgressMonitor<br>  \_ SubMonitor<br>   \_ etc</pre><p>Each <em>SubMonitor</em>/<em>SubProgressMonitor </em>has at most a single parent. When debugging, the parent of a monitor can be accessed from an instance variable of the monitor class:</p><ul><li><em>SubMonitor.root.root</em> of type <em>IProgressMonitor</em></li><li><em>SubProgressMonitor.progressMonitor </em>of type <em>IProgressMonitor</em></li></ul><p>With this information, you may examine the monitor tree of a problematic monitor, by beginning with the child monitor and “climbing the tree” until an issue is identified.</p><h4>How to debug NullProgressMonitor parent in the monitor tree</h4><p>Locate a place in your code where work is being worked but not reported. This would be an <em>IProgressMonitor.worked(…)/SubMonitor.newChild(…)</em> or <em>SubMonitor.split(…)</em> call. Add a breakpoint on this method, and debug your application to cause the statement to be triggered.</p><p>Step inside the call worked/split call. From here you can examine move up the tree of progress monitors to determine if any of the parents is a <em>NullProgressMonitor</em>. If the type is a <em>SubMonitor</em>, check the root.root variable; if the type is a <em>SubProgressMonitor</em>, check the <em>progressMonitor</em> variable.</p><p>If the issue a <em>NullProgressMonitor</em> as one of the parents, you will likely find an instance of this class somewhere up the tree; removing this will likely solve the issue.</p><h4>How to debug a parent monitor discarding a child’s work</h4><p>This is more difficult than merely scanning the tree for <em>NullProgressMonitor</em> as above. The <em>SubProgressMonitor</em> class, as written, will silently discard work reported if the user has broken the “contract” of the class.</p><p>The contract of the class, as it relates to this problem, is:</p><ul><li>You must call <em>beginTask(…)</em> at once on a monitor; if you call it zero times (eg do not ever call <em>beginTask(…)</em> ), work will be ignored.</li><li>You should not call <em>beginTask(…)</em> twice on the same monitor; if you do, all subsequent work will be ignored.</li></ul><p>This can be seen in the <em>SubProgressMonitor.internalWorked(…)</em> method, where if ‘<em>nestedBeginTasks</em>’ is not exactly one, the call is suppressed without warning.</p><p>While we can’t do anything about the implementation of <em>SubProgressMonitor</em> (you should be using <em>SubMonitor</em>, anyways), we can do something about the ‘without warning’ part.</p><p>Lets instrument the class by adding our own warnings when the contract is violated:</p><p>1) Add the following fields to <strong>SubProgressMonitor</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/471fb942186a77049c8c46c2ceebad06/href">https://medium.com/media/471fb942186a77049c8c46c2ceebad06/href</a></iframe><p>2) Replace the 3-arg constructor with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8295a2a7a88a4678e0b4c5fab7cc5dd3/href">https://medium.com/media/8295a2a7a88a4678e0b4c5fab7cc5dd3/href</a></iframe><p>3) Replace <em>beginTask(…)</em> as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73a514dc0effd4c3e2c503ecc7bd0a83/href">https://medium.com/media/73a514dc0effd4c3e2c503ecc7bd0a83/href</a></iframe><p>4) Replace <em>internalWorked(…)</em> as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3c1f1584abffb610affe1e1525a22f8/href">https://medium.com/media/c3c1f1584abffb610affe1e1525a22f8/href</a></iframe><p>Note the <strong>TODO</strong> item inside the if. Since this block of code will get triggered often by other code in the workbench, it is necessary to filter out all threads EXCEPT for your own code.</p><p>With the above changes, you should now receive warnings (along with stack traces) for the following two scenarios:</p><ol><li>Multiple calls to <em>beginTask(…)</em>; the call after the first will have the stack trace reported, along with the valid original first call.</li><li>Zero calls to <em>beginTask(…)</em>; the call to <em>internalWorked(…)</em> will have the stack trace reported, along with the stack trace of the object’s construction.</li></ol><p>Instrumenting <em>SubMonitor</em> is mostly not necessary. It is smarter in how it is written, and includes warnings when it’s contract is violated. See the <em>logProblem(…)</em> method in SubMonitor for details.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f482c71cc85c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>