<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[jorgeacetozi - Medium]]></title>
        <description><![CDATA[Articles about DevOps, tools, software development concepts, best practices, architecture, NoSQL and many other cool stuff! Visit the website: https://jorgeacetozi.com - Medium]]></description>
        <link>https://medium.com/jorgeacetozi?source=rss----af8907a3ade0---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>jorgeacetozi - Medium</title>
            <link>https://medium.com/jorgeacetozi?source=rss----af8907a3ade0---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:22:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/jorgeacetozi" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes Node Components: Service Proxy, Kubelet, and cAdvisor]]></title>
            <link>https://medium.com/jorgeacetozi/kubernetes-node-components-service-proxy-kubelet-and-cadvisor-dcc6928ef58c?source=rss----af8907a3ade0---4</link>
            <guid isPermaLink="false">https://medium.com/p/dcc6928ef58c</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jorge Acetozi]]></dc:creator>
            <pubDate>Mon, 18 Dec 2017 14:26:32 GMT</pubDate>
            <atom:updated>2017-12-18T14:26:31.713Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/jorgeacetozi/kubernetes-master-components-etcd-api-server-controller-manager-and-scheduler-3a0179fc8186">In my last post, I discussed the roles of Kubernetes Master Components: Etcd, API Server, Controller Manager, and Scheduler</a>. Now, let’s dive into Kubernetes Node Components.</p><h3>Kubernetes Node Components</h3><p>Basically, the Kubernetes node runs the Kubelet and Service Proxy components as well as a container engine such as <a href="https://www.docker.com/">Docker</a> or <a href="https://github.com/rkt/rkt">Rocket</a>, which in turns run your containerized applications and makes you and your customers very happy (Lol).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RHsNYkGi10fU5XUv.png" /><figcaption>Figure 1: Kubernetes Components</figcaption></figure><h3>Service Proxy</h3><p>The Service Proxy runs on each node and is responsible for watching the API Server for changes on services and pods definitions to maintain the entire network configuration up to date, ensuring that one pod can talk to another pod, one node can talk to another node, one container can talk to another container, and so on. Besides, it exposes Kubernetes services and manipulates iptables rules to trap access to services IPs and redirect them to the correct backends (that’s why you can access a <a href="https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport">NodePort</a> service using any node IP; even if the node you hit is not the one you are looking for, this node will already be set up with the appropriate iptables rules to redirect your request to the correct backend). This provides a highly-available load-balancing solution with low performance overhead.</p><h3>Kubelet</h3><p>The Kubelet is one of the most important components in Kubernetes. Basically, it’s an agent that runs on each node and is responsible for watching the API Server for pods that are bound to its node and making sure those pods are running (it talks to the Docker daemon using the API over the Docker socket to manipulate containers lifecycle). It then reports back to the API Server the status of changes regarding those pods.</p><p>The main Kubelet responsibilities include:</p><ul><li>Run the pods containers.</li><li>Report the status of the node and each pod to the API Server.</li><li>Run container probes.</li><li>Retrieve container metrics from cAdvisor, aggregate and expose them through the Kubelet <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/stats/v1alpha1/types.go">Summary API</a> for components (such as Heapster) to consume.</li></ul><blockquote>The last responsibility listed above will change in the future as cAdvisor will be turned off for container stats collection and replaced by the <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cri-container-stats.md">Container Runtime Interface</a>.</blockquote><p>The Kubelet also starts an <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go">internal HTTP server</a> on port 10255 and exposes some endpoints (mostly for debugging, stats, and for one-off container operations such as <em>kubectl logs</em> or <em>kubectl exec</em>), such as /metrics, /metrics/cadvisor, /pods, /spec, and so on.</p><h3>cAdvisor</h3><p>The Kubelet ships with built-in support for <a href="https://github.com/google/cadvisor">cAdvisor</a>, which collects, aggregates, processes and exports metrics (such as CPU, memory, file and network usage) about running containers on a given node. cAdvisor includes a built-in web interface available on port 4194 (just open your browser and navigate to http://&lt;node-ip&gt;:4194/).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p2QwedthRRMgSzKT.png" /><figcaption>Figure 2: cAdvisor Usage Metrics</figcaption></figure><p>You can view the cAdvisor /metrics endpoint by issuing a GET request to http://&lt;node-ip&gt;:4194/metrics.</p><p>If this article was helpful to you, please like and share it with your friends. Plus, follow me on <a href="https://www.linkedin.com/in/jorgeacetozi/">LinkedIn</a>, <a href="https://medium.com/jorgeacetozi">Medium</a> and subscribe to <a href="https://www.jorgeacetozi.com/">jorgeacetozi.com</a> to get notified when a new article is published!</p><p>This article was based on contents of my book <a href="https://leanpub.com/continuous-delivery-for-java-apps">Continuous Delivery for Java Apps: Build a CD Pipeline Step by Step Using Kubernetes, Docker, Vagrant, Jenkins, Spring, Maven, and Artifactory</a>, which I invite you to download and read the free sample (110 pages) just clicking on <strong>Read Free Sample</strong> and choosing your preferred format (PDF, EPUB, MOBI, or WEB). Basically, the free sample will guide you through these subjects:</p><ul><li>Unit Tests</li><li>Integration Tests</li><li>Acceptance Tests (Distributed Acceptance Tests with <a href="http://www.seleniumhq.org/projects/grid/">Selenium-Grid</a>)</li><li>Performance Tests using <a href="https://gatling.io/">Gatling</a></li><li>Continuous Integration</li><li>Continuous Delivery</li><li>Continuous Deployment</li><li>Canary Release</li><li>Feature Branch</li><li>A/B Tests</li><li>Feature Flags</li><li>Docker</li><li>ChatOps</li></ul><p>Please check out my other articles:</p><ul><li><a href="https://medium.com/jorgeacetozi/why-to-use-kubernetes-22f28cac68fe">Why to Use Kubernetes?</a></li><li><a href="https://medium.com/jorgeacetozi/kubernetes-master-components-etcd-api-server-controller-manager-and-scheduler-3a0179fc8186">Kubernetes Master Components: Etcd, API Server, Controller Manager, and Scheduler</a></li><li><a href="https://medium.com/jorgeacetozi/continuous-integration-vs-continuous-delivery-vs-continuous-deployment-d5839a85a959">Continuous Integration vs Continuous Delivery vs Continuous Deployment</a></li><li><a href="https://medium.com/jorgeacetozi/cassandra-architecture-and-write-path-anatomy-51e339bcfe0c">Cassandra Architecture and Write Path Anatomy</a></li></ul><p>Thank you very much for your time!</p><p>Jorge Acetozi is a software engineer who spends almost his whole day having fun with things such as AWS, Kubernetes, Docker, Terraform, Ansible, Cassandra, Redis, Elasticsearch, Graylog, New Relic, Sensu, Elastic Stack, Fluentd, RabbitMQ, Kafka, Java, Spring, and much more! He loves deploying applications in production while thousands of users are online, monitoring the infrastructure, and acting quickly when monitoring tools decide to challenge his heart’s health! Check out his books:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iIqxq6CZ6_yQ-WrH.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcc6928ef58c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jorgeacetozi/kubernetes-node-components-service-proxy-kubelet-and-cadvisor-dcc6928ef58c">Kubernetes Node Components: Service Proxy, Kubelet, and cAdvisor</a> was originally published in <a href="https://medium.com/jorgeacetozi">jorgeacetozi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Master Components: Etcd, API Server, Controller Manager, and Scheduler]]></title>
            <link>https://medium.com/jorgeacetozi/kubernetes-master-components-etcd-api-server-controller-manager-and-scheduler-3a0179fc8186?source=rss----af8907a3ade0---4</link>
            <guid isPermaLink="false">https://medium.com/p/3a0179fc8186</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Jorge Acetozi]]></dc:creator>
            <pubDate>Mon, 11 Dec 2017 11:50:06 GMT</pubDate>
            <atom:updated>2017-12-11T11:51:52.229Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s take a look at the following Kubernetes architecture diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r3DyRvMZJ_LUC9w4.png" /><figcaption>Kubernetes Components</figcaption></figure><p>Well, this might look complicated at first glance, but actually, it’s not so much. I’m pretty sure that by the end of this article you will have a clear understanding of how these components interact with each other.</p><h3>Kubernetes Master Components</h3><p>The Kubernetes master runs the Scheduler, Controller Manager, API Server and etcd components and is responsible for managing the Kubernetes cluster. Essentially, it’s the brain of the cluster! Now, let’s dive into each master component.</p><blockquote>In production, you should set up Kubernetes with multiple masters for high availability. See how to <a href="https://kubernetes.io/docs/admin/high-availability/">build high-Availability clusters</a> official guides for further information.</blockquote><h3>Etcd</h3><p><a href="https://github.com/coreos/etcd">Etcd</a> is a distributed, consistent key-value store used for configuration management, service discovery, and coordinating distributed work.</p><p>When it comes to Kubernetes, etcd reliably stores the configuration data of the Kubernetes cluster, representing the state of the cluster (what nodes exist in the cluster, what pods should be running, which nodes they are running on, and a whole lot more) at any given point of time.</p><blockquote>As all cluster data is stored in etcd, you should always have a backup plan for it. You can easily back up your etcd data using the <em>etcdctl snapshot save</em> command. In case you are running Kubernetes on AWS, you can also back up etcd by taking a snapshot of the EBS volume.</blockquote><p>Etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log. Raft is a consensus algorithm designed as an alternative to <a href="https://en.wikipedia.org/wiki/Paxos_(computer_science)">Paxos</a>. The Consensus problem involves multiple servers agreeing on values; a common problem that arises in the context of replicated state machines. Raft defines three different roles (Leader, Follower, and Candidate) and achieves consensus via an elected leader. For further information, please read the <a href="https://raft.github.io/raft.pdf">Raft paper</a>.</p><p><a href="https://github.com/coreos/etcd/tree/master/etcdctl">Etcdctl</a> is the command-line interface tool written in Go that allows manipulating an etcd cluster. It can be used to perform a variety of actions, such as:</p><ul><li>Set, update and remove keys.</li><li>Verify the cluster health.</li><li>Add or remove etcd nodes.</li><li>Generating database snapshots.</li></ul><p>You can play online with a 5-node etcd cluster at <a href="http://play.etcd.io/">http://play.etcd.io</a>.</p><p>Etcd also implements a <strong>watch</strong> feature, which provides an event-based interface for asynchronously monitoring changes to keys. Once a key is changed, its “watchers” get notified. This is a crucial feature in the context of Kubernetes, as the API Server component heavily relies on this to get notified and call the appropriate business logic components to move the current state towards the desired state.</p><h3>API Server</h3><p>When you interact with your Kubernetes cluster using the kubectl command-line interface, you are actually communicating with the master API Server component.</p><p>The API Server is the main management point of the entire cluster. In short, it processes REST operations, validates them, and updates the corresponding objects in etcd. The API Server serves up the <a href="https://kubernetes.io/docs/concepts/overview/kubernetes-api/">Kubernetes API</a> and is intended to be a relatively simple server, with most business logic implemented in separate components or in plugins.</p><blockquote>The API Server is the only Kubernetes component that connects to etcd; all the other components must go through the API Server to work with the cluster state.</blockquote><p>The API Server is also responsible for the authentication and authorization mechanism. All API clients should be authenticated in order to interact with the API Server.</p><p>The API Server also implements a watch mechanism (similar to etcd) for clients to watch for changes. This allows components such as the Scheduler and Controller Manager to interact with the API Server in a loosely coupled manner.</p><p>This pattern is extensively used in Kubernetes. For example, when you create a pod using kubectl, this what happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c2N7STjiWZjCy8we.png" /><figcaption><strong><em>Create Pod Flow. Source: heptio.com</em></strong></figcaption></figure><ol><li>kubectl writes to the API Server.</li><li>API Server validates the request and persists it to etcd.</li><li>etcd notifies back the API Server.</li><li>API Server invokes the Scheduler.</li><li>Scheduler decides where to run the pod on and return that to the API Server.</li><li>API Server persists it to etcd.</li><li>etcd notifies back the API Server.</li><li>API Server invokes the Kubelet in the corresponding node.</li><li>Kubelet talks to the Docker daemon using the API over the Docker socket to create the container.</li><li>Kubelet updates the pod status to the API Server.</li><li>API Server persists the new state in etcd.</li></ol><h3>Controller Manager</h3><p>The Kubernetes Controller Manager is a daemon that embeds the <a href="https://github.com/kubernetes/kubernetes/blob/7c75723867e9e431da323b8cc410bab928cada17/cmd/kube-controller-manager/app/controllermanager.go#L330">core control loops</a> (also known as “controllers”) shipped with Kubernetes. Basically, a controller watches the state of the cluster through the API Server watch feature and, when it gets notified, it makes the necessary changes attempting to move the current state towards the desired state. Some examples of controllers that ship with Kubernetes include the Replication Controller, Endpoints Controller, and Namespace Controller.</p><p>Besides, the Controller Manager performs lifecycle functions such as namespace creation and lifecycle, event garbage collection, terminated-pod garbage collection, cascading-deletion garbage collection, node garbage collection, etc.</p><h3>Scheduler</h3><p>The Scheduler watches for unscheduled pods and binds them to nodes via the <em>/binding </em>pod subresource API, according to the availability of the requested resources, quality of service requirements, affinity and anti-affinity specifications, and other constraints. Once the pod has a node assigned, the regular behavior of the Kubelet is triggered and the pod and its containers are created (see the pod creation flow on API Server section: steps 4 to 11).</p><p>If this article was helpful to you, please like and share it with your friends. Plus, follow me on <a href="https://www.linkedin.com/in/jorgeacetozi/">LinkedIn</a> and subscribe to <a href="https://www.jorgeacetozi.com/">jorgeacetozi.com</a> to get notified when part 2 of this article (<strong>Kubernetes Architecture: Node Components</strong>) is published!</p><p>This article was based on contents of my book <a href="https://leanpub.com/continuous-delivery-for-java-apps">Continuous Delivery for Java Apps: Build a CD Pipeline Step by Step Using Kubernetes, Docker, Vagrant, Jenkins, Spring, Maven, and Artifactory</a>, which <strong>I invite you to download and read the free sample (110 pages) just clicking on Read Free Sample and choosing your preferred format (PDF, EPUB, MOBI, or WEB)</strong>. Basically, the free sample will guide you through these subjects:</p><ul><li>Unit Tests</li><li>Integration Tests</li><li>Acceptance Tests (Distributed Acceptance Tests with <a href="http://www.seleniumhq.org/projects/grid/">Selenium-Grid</a>)</li><li>Performance Tests using <a href="https://gatling.io/">Gatling</a></li><li>Continuous Integration</li><li>Continuous Delivery</li><li>Continuous Deployment</li><li>Canary Release</li><li>Feature Branch</li><li>A/B Tests</li><li>Feature Flags</li><li>Docker</li><li>ChatOps</li></ul><p>Please check out my other articles:</p><ul><li><a href="https://medium.com/jorgeacetozi/why-to-use-kubernetes-22f28cac68fe">Why to Use Kubernetes?</a></li><li><a href="https://www.linkedin.com/pulse/continuous-integration-vs-delivery-deployment-jorge-acetozi/">Continuous Integration vs Continuous Delivery vs Continuous Deployment</a></li><li><a href="https://www.linkedin.com/pulse/cassandra-architecture-write-path-anatomy-jorge-acetozi/">Cassandra Architecture and Write Path Anatomy</a></li></ul><p>Thank you very much for your time!</p><p>Jorge Acetozi is a software engineer who spends almost his whole day having fun with things such as AWS, Kubernetes, Docker, Terraform, Ansible, Cassandra, Redis, Elasticsearch, Graylog, New Relic, Sensu, Elastic Stack, Fluentd, RabbitMQ, Kafka, Java, Spring, and much more! He loves deploying applications in production while thousands of users are online, monitoring the infrastructure, and acting quickly when monitoring tools decide to challenge his heart’s health! Check out his books:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GGtANix00jg4xWz4.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a0179fc8186" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jorgeacetozi/kubernetes-master-components-etcd-api-server-controller-manager-and-scheduler-3a0179fc8186">Kubernetes Master Components: Etcd, API Server, Controller Manager, and Scheduler</a> was originally published in <a href="https://medium.com/jorgeacetozi">jorgeacetozi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agile and Scrum Overview]]></title>
            <link>https://medium.com/jorgeacetozi/agile-and-scrum-overview-fb68fc04aff8?source=rss----af8907a3ade0---4</link>
            <guid isPermaLink="false">https://medium.com/p/fb68fc04aff8</guid>
            <category><![CDATA[extreme-programming]]></category>
            <category><![CDATA[scrum]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Jorge Acetozi]]></dc:creator>
            <pubDate>Mon, 27 Nov 2017 10:52:42 GMT</pubDate>
            <atom:updated>2017-12-11T11:52:12.652Z</atom:updated>
            <content:encoded><![CDATA[<h3>Agile</h3><p>Tell me, how many consecutive years did you fail to accomplish your New Year’s promises and long-term plans for the next year? Long-term plans are just a way for human beings to feel less uncomfortable with the unknown. We kind of get the feeling we have control over stuff when we have a long-term plan.</p><p>The point is that all of the sudden, on January 15th, when you think you are in control of your life, you receive an email with a job offer paying twice more than your current job or you simply get fired.</p><p>What we don’t immediately realize is that a job offer paying twice more sometimes is not good (the job and people there are boring, they don’t care for development best practices, the environment does not encourage communication, and so on) and sometimes getting fired is not bad (your next job can be much better). This happens because human beings tend to react without thinking first. Very often we caught ourselves being thankful for something that initially we thought it would be bad.</p><p>In the software development context, the same happens when the client asks for a change in something that was already delivered or even is being developed. Sometimes, our first reaction is to become upset and think “why didn’t they think about it before asking? These guys don’t know what they want!”. However, after a while we did realize that the change is actually for the best and it will increase a lot of value to the whole product; the client will be more satisfied and the end users will benefit. The second Agile principle states: <strong>Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.</strong></p><p>Agile is not about being faster. It is about being able to adapt well to new situations. Imagine that you are in a building on fire. You can either be really fast and escape from it by throwing yourself out through the window or you can find the fire escape and escape from it in safety. Option one is faster, option two is agile.</p><p>Agile Methodologies for Software Development such as Scrum and Extreme Programming are based on the <a href="http://agilemanifesto.org/">Agile Manifesto</a>, which states the following:</p><ol><li>Individuals and interactions over processes and tools</li><li>Working software over comprehensive documentation</li><li>Customer collaboration over contract negotiation</li><li>Responding to change over following a plan</li></ol><p>That is, while there is value in the items on the right, we value the items on the left more.</p><p>Also, let’s check the <a href="http://agilemanifesto.org/principles.html">Twelve Principles of Agile Software</a>:</p><ol><li>Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.</li><li>Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.</li><li>Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.</li><li>Business people and developers must work together daily throughout the project.</li><li>Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.</li><li>The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.</li><li>Working software is the primary measure of progress.</li><li>Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.</li><li>Continuous attention to technical excellence and good design enhances agility.</li><li>Simplicity — the art of maximizing the amount of work not done — is essential.</li><li>The best architectures, requirements, and designs emerge from self-organizing teams.</li><li>At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.</li></ol><h3>Scrum</h3><p>Developing software in a non-iterative fashion is probably the reason why a lot of good projects with highly-skilled people have failed over the past years.</p><p>As I mentioned, the whole problem with this approach is that when you set up a long-term plan and strictly follow it, like the <a href="https://en.wikipedia.org/wiki/Waterfall_model">Waterfall model</a>, you lose the ability of making changes and adapting the product to what the client really needs, which most of the times happens to be figured out only during the project development phase rather than the in the very first talks to the client.</p><p>In order to avoid this issue, the current recommended way for approaching most of the software projects is by iterations, that is, shorter periods of time (typically weeks) where the whole process happens (planning, development, tests, release, client feedback). This not only allows the client to start using the software earlier (adding value to his business) but also gives the opportunity for him to provide faster feedback, ask for changes earlier, change feature priorities, and so on.</p><p>A very well-known framework for software management that implements this idea is called <strong>Scrum</strong>. It has been successfully used along many years in companies all around the world.</p><p>Of course that there are many other factors that influence the success of a software project such as the code quality, the release process, the application architecture, and so on. Scrum does not define software engineering best practices, it is all about <strong>managing</strong>the software project. There is another Agile Methodology called <a href="http://www.extremeprogramming.org/"><strong>Extreme Programming</strong></a> (XP) that approaches <a href="http://www.extremeprogramming.org/rules/pair.html">Pair Programming</a>, <a href="http://www.extremeprogramming.org/rules/testfirst.html">Test Driven Development</a>, <a href="http://www.extremeprogramming.org/rules/refactor.html">Refactoring</a>, <a href="http://www.extremeprogramming.org/rules/simple.html">Simple Design</a>, <a href="http://www.extremeprogramming.org/rules/integrateoften.html">Continuous Integration</a>, and others software engineering practices. Software development teams often use Scrum and XP together to take advantage of both worlds.</p><p>Please download my free book sample (<a href="https://leanpub.com/continuous-delivery-for-java-apps">Continuous Delivery for Java Apps: Build a CD Pipeline Step by Step Using Kubernetes, Docker, Vagrant, Jenkins, Spring, Maven, and Artifactory</a>) to learn more about Automated Tests (unit, integration, acceptance and performance tests), Continuous Integration, Continuous Delivery, Continuous Deployment, Canary Release, Feature Branch, A/B Tests, Feature Flag and many other relevant concepts.</p><p>The Scrum artifacts are:</p><ul><li><strong>Product Backlog</strong>: a list of items that represent the client “wish list”. These items can be [user stories], bug fixes, non-functional requirements, and so on.</li><li><strong>Sprint Backlog</strong>: a list of items chosen in the Sprint Planning to be developed in a particular sprint.</li><li><strong>Product Increment</strong>: the sum of all the product backlog items completed during a sprint, integrated with the work of all previous sprints.</li></ul><p>A <strong>Scrum Team</strong> consists of:</p><ul><li><strong>Product Owner</strong>: defines and prioritize the items that make up the Product Backlog, answer to business questions that the Development Team might have, etc.</li><li><strong>Development Team</strong>: make up the people that are actually technical, such as developers, system administrators, database administrators and so on.</li><li><strong>Scrum Master</strong>: remove possible impediments that might show up for the development team during the development phase, facilitates Scrum Events, etc.</li></ul><p>In short, Scrum calls each iteration as a <strong>Sprint</strong>. A sprint should always have the same duration (it can vary from 1 to 4 weeks) and it’s essentially composed of these events (in the presented order):</p><ul><li><strong>Daily Scrum</strong>: a quick meeting that takes place every day so that team members can inform what they did yesterday, what they are doing today and if they have any impediments.</li><li><strong>Spring Planning</strong>: a meeting that takes place at the beginning of each sprint where the Scrum Team define which items from the Spring Backlog should be developed in the development phase. Basically, the Product Owner picks the most priority items and answer to business doubts that arise. Then, the developers estimate the effort to develop these items, optionally using the <a href="https://en.wikipedia.org/wiki/Planning_poker">Planning Poker</a>, and based on the average points the team is used to deliver in every Sprint, the Product Owner decides whether he puts some items back to his Product Backlog or he includes more items to the Sprint Backlog.</li><li><strong>Sprint Review</strong>: a meeting that involves the Scrum Team and any stakeholder to review the work that was completed and the planned work that was not completed along with a presentation on the completed work (for example, a demo). It is also a time where everyone collaborates on what to do next, so it provides a valuable input to the subsequent Sprint Planning.</li><li><strong>Sprint Retrospective</strong>: a meeting where the Scrum team discusses what went well in the Sprint, what could be improved and what they will commit to improve in the next Sprint.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uwSK2aX5XtWNUqCb.jpg" /></figure><p>This post should give you an overview about Scrum, but I would really encourage you to read the <a href="http://www.scrumguides.org/">Scrum Guide</a>, which by the way is available in several languages for free.</p><p>If you liked this article, don&#39;t forget to follow me :)</p><p>Check out my other articles:</p><ul><li><a href="https://medium.com/jorgeacetozi/continuous-integration-vs-continuous-delivery-vs-continuous-deployment-d5839a85a959">Continuous Integration vs Continuous Delivery vs Continuous Deployment</a></li><li><a href="https://medium.com/jorgeacetozi/cassandra-architecture-and-write-path-anatomy-51e339bcfe0c">Cassandra Architecture and Write Path Anatomy</a></li></ul><p>Jorge Acetozi is a software engineer who spends almost his whole day having fun with things such as AWS, Kubernetes, Docker, Terraform, Ansible, Cassandra, Redis, Elasticsearch, Graylog, New Relic, Sensu, Elastic Stack, Fluentd, RabbitMQ, Kafka, Java, Spring, and much more! He loves deploying applications in production while thousands of users are online, monitoring the infrastructure, and acting quickly when monitoring tools decide to challenge his heart’s health! Check out his books on <a href="https://www.jorgeacetozi.com/books">https://www.jorgeacetozi.com/books</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IiNd8O6Tjp6-b5CEUvNBxg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb68fc04aff8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jorgeacetozi/agile-and-scrum-overview-fb68fc04aff8">Agile and Scrum Overview</a> was originally published in <a href="https://medium.com/jorgeacetozi">jorgeacetozi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cassandra Architecture and Write Path Anatomy]]></title>
            <link>https://medium.com/jorgeacetozi/cassandra-architecture-and-write-path-anatomy-51e339bcfe0c?source=rss----af8907a3ade0---4</link>
            <guid isPermaLink="false">https://medium.com/p/51e339bcfe0c</guid>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Jorge Acetozi]]></dc:creator>
            <pubDate>Thu, 16 Nov 2017 13:45:41 GMT</pubDate>
            <atom:updated>2017-11-16T13:45:41.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3TADoSRGwNJU__qgBBs_w.png" /></figure><p>Cassandra is a NoSQL database that belongs to the Column Family NoSQL database category. It’s an <a href="http://cassandra.apache.org/">Apache project</a> and it has an Enterprise version maintained by <a href="https://www.datastax.com/">DataStax</a>. Cassandra is written in Java and it’s mainly used for time-series data such as metrics, IoT (Internet of Things), logs, chat messages, and so on. It is able to handle a massive amount of writes and reads and scale to thousands of nodes. Let’s list out here some important Cassandra characteristics. Basically, Cassandra…</p><ul><li>mix ideas from Google’s Big Table and Amazon’s Dynamo.</li><li>is based on <a href="https://en.wikipedia.org/wiki/Peer-to-peer">peer-to-peer</a> architecture. Every node is equal (can perform both reads and writes), therefore there is no master or slave node, that is, there is no master single point of failure.</li><li>does automatic partitioning and replication.</li><li>has tunable write and read consistency for both read and write operations.</li><li>is able to horizontally scale keeping <a href="https://medium.com/netflix-techblog/benchmarking-cassandra-scalability-on-aws-over-a-million-writes-per-second-39f45f066c9e">linear scalability</a> for both reads and writes.</li><li>handles inter-node communication through the <a href="https://en.wikipedia.org/wiki/Gossip_protocol">Gossip protocol</a>.</li><li>handles client communication through the CQL (Cassandra Query Language), which is very similar to SQL.</li></ul><h3>Coordinator</h3><p>When a request is sent to any Cassandra node, this node acts as a proxy for the application (actually, the Cassandra driver) and the nodes involved in the request flow. This proxy node is called as the coordinator. The coordinator is responsible for managing the entire request path and to respond back to the client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UJwKzBurhLTzJ3sL.png" /><figcaption>Figure 1 — Coordinator</figcaption></figure><p>Besides, sometimes when the coordinator forwards a write request to the replica nodes, they may happen to be unavailable at this very moment. In this case, the coordinator plays an important role implementing a mechanism called Hinted Handoff, which will be described in details later.</p><h3>Partitioner</h3><p>Basically, for each node in the Cassandra cluster (Cassandra ring) is assigned a range of tokens as shown in Figure 2 for a 6-node cluster (with imaginary tokens, of course).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wCWBHLmZHgZm68-R.png" /><figcaption>Figure — Token Ranges</figcaption></figure><p>Cassandra distributes data across the cluster using a <a href="https://en.wikipedia.org/wiki/Consistent_hashing">Consistent Hashing</a> algorithm and, starting from version 1.2, it also implements the concept of virtual nodes (vnodes), where each node owns a large number of small token ranges in order to improve token reorganization and avoid hotspots in the cluster, that is, some nodes storing much more data than the others. Virtual nodes also allow to add and remove nodes in the cluster more easily and manages the token assignment automatically for you so that you can enjoy a nice coffee when adding or removing a node instead of calculating and assigning new token ranges for each node (which is a very error-prone operation, by the way).</p><p>Well, that said, the partitioner is the component responsible for determining how to distribute the data across the nodes in the cluster given the <a href="https://dzone.com/articles/cassandra-data-modeling-primary-clustering-partiti">partition key</a> of a row. Basically, it is a hash function for computing a token given the partition key.</p><p>Once the partitioner applies the hash function to the partition key and gets the token, it knows exactly which node is going to handle the request.</p><p>Let’s consider a simple example: suppose a request is issued to node6 (that is, node6 is the coordinator for this request) with a row containing the partition key “jorge_acetozi”. Suppose the partitioner applies the hash function to the partition key “jorge_acetozi” and gets the token -17. As figure 3 shows, node2 token ranges include -17, so this node will be the one handling the request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iSDYBJ2Gvvcio06n.png" /><figcaption>Figure 3 — Partitioner</figcaption></figure><p>Cassandra offers three types of partitioners: <a href="https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/Murmur3Partitioner.java">Murmur3Partitioner</a> (which is the default), <a href="https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/RandomPartitioner.java">RandomPartitioner</a>, and <a href="https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/ByteOrderedPartitioner.java">ByteOrderedPartitioner</a>.</p><h3>Replication</h3><p>Life would be much easier if…</p><ul><li>Nodes never fail</li><li>Networks had no latency</li><li>People did not stumble on cables</li><li>Amazon did not restart your instances</li><li>Full GC meant “Full Guitar Concert”</li></ul><p>And so on. Unfortunately, these things happen all the time and you already chose a software engineer career (your mother used to advise you to study hard and to become a doctor, but you chose to keep playing Counter-Strike instead. Now you are a software engineer, know what AK-47 means and have to care about stuff like that).</p><p>Fortunately, Cassandra offers automatic data replication and keeps your data redundant throughout different nodes in the cluster. This means that (in certain levels) you can even resist to node failure scenarios and your data would still be safe and available. But everything comes at a price, and the price of replication is consistency.</p><h3>Replication Strategy</h3><p>Basically, the coordinator uses the Replication Strategy to find out which nodes will be the replica nodes for a given request.</p><p>There are two replication strategies available:</p><p><a href="https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/locator/SimpleStrategy.java">SimpleStrategy</a>: used for a single data center deployment (not recommended for production environment). It doesn’t consider the network topology. Basically, it just takes the partitioner’s decision (that is, the node that will handle the request first based on the token range) and places the remaining replicas clockwise in relation to this node. For example, in Figure 3, if the table replication factor was 3, which nodes would have been chosen by the SimpleStrategy to act as replicas (besides node2, which was already chosen by the partitioner)? That’s correct, node3 and node4! What if the replication factor was 4? Well, then node5 would also be included.</p><p><a href="https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java">NetworkTopologyStrategy</a>: used for multiple data centers deployment (recommended for production environment). It also takes the partitioner’s decision and places the remaining replicas clockwise, but it also takes into consideration the rack and data centers configuration.</p><h3>Replication Factor</h3><p>When you create a table (Column Family) in Cassandra, you specify the replication factor. The replication factor is the number of replicas that Cassandra will hold for this table in different nodes. If you specify REPLICATION_FACTOR=3, then your data will be replicated to 3 different nodes throughout the cluster. That provides fault tolerance and resilience because even if some nodes fail your data would still be safe and available.</p><h3>Write Consistency Level</h3><p>Do you still remember that when the client sends a request to a Cassandra node, this node is called a coordinator and acts as a proxy between the client and the replica nodes?</p><p>Well, when you write to a table in Cassandra (inserting data, for example), you can specify the write consistency level. The write consistency level is the number of replica nodes that have to acknowledge the coordinator that its local insert was successful (success here means that the data was appended to the commit log and written to the memtable). As soon as the coordinator gets WRITE_CONSISTENCY_LEVEL success acknowledgments from the replica nodes, it returns success back to the client and doesn’t wait for the remaining replicas to acknowledge success.</p><p>For example, if an application issue an insert request with WRITE_CONSISTENCY_LEVEL=TWO to a table that is configured with REPLICATION_FACTOR=3, the coordinator will only return success to the application when two of the three replicas acknowledge success. Of course, this doesn’t mean that the third replica will not write the data too; it will, but at this point, the coordinator would already have sent success back to the client.</p><p>There are many different types of write consistency levels you can specify in your write request. From the less consistent to full consistency: ANY, ONE, TWO, THREE, QUORUM, LOCAL_QUORUM, EACH_QUORUM, ALL.</p><h3>Write Flow Example</h3><p>For simplicity, suppose a write request is issued to a 6-node Cassandra cluster with the following characteristics:</p><ul><li>WRITE_CONSISTENCY_LEVEL=TWO</li><li>TABLE_REPLICATION_FACTOR=3</li><li>REPLICATION_STRATEGY=SimpleStrategy</li></ul><p>First, the client sends the write request to the Cassandra cluster using the driver. We haven’t discussed the role of the driver in this post (maybe in another post), but it plays a very important role as well. The driver is responsible for a lot of features such as asynchronous IO, parallel execution, request pipelining, connection pooling, auto node discovery, automatic reconnection, token awareness, and so on. For example, by using a driver that implements a token-aware policy, the driver reduces network hops by sending requests directly to the node that owns the data instead of sending it to a “random” coordinator.</p><p>As soon as the coordinator gets the write request, it applies the partitioner hash function to the partition key and uses the configured Replication Strategy in order to determine the TABLE_REPLICATION_FACTOR replica nodes that will actually write the data (in this sentence, replace TABLE_REPLICATION_FACTOR with the number 3). Figure 4 shows the replica nodes (in green) that will handle the write request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kKIpPm7xhPb2ZwmA.png" /><figcaption>Figure 4 — Replica Nodes</figcaption></figure><p>Now, before the coordinator forwards the write request to all the 3 replica nodes, it will ask to the <a href="https://github.com/jorgeacetozi/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/FailureDetector.java">Failure Detector component</a> how many of these replica nodes are actually available and compare it to the WRITE_CONSISTENCY_LEVEL provided in the request. If the number of replica nodes available is less than the WRITE_CONSISTENCY_LEVEL provided, the Failure Detector will immediately throw an Exception.</p><p>For our example, suppose the 3 replica nodes are available (that is, the Failure Detector will allow the request to continue) such as shown in Figure 5. Now, the coordinator will asynchronously forward the write request to all the replica nodes (in these case, the 3 replica nodes that were figured in the first step). As soon as WRITE_CONSISTENCY_LEVEL replica nodes acknowledge success (node2 and node4), the coordinator returns success back to the driver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WQ2Rwq2zltznUgIv.png" /><figcaption>Figure 5 — Write Success</figcaption></figure><p>If the WRITE_CONSISTENCY_LEVEL for this request was THREE (or ALL), the coordinator would have to wait until node3 acknowledges success too, and of course that this write request would be slower.</p><p>So, basically…</p><ul><li>Do you need fault tolerance and high availability? Use replication.</li><li>Just bear in mind that using replication means you will pay with consistency (for most of the cases, this is not a problem. Availability is often more important than consistency).</li><li>If consistency is not an issue for your domain, perfect. If it is, just increase the consistency level, but then you will pay with higher latency.</li><li>If you want fault tolerance and high availability, strong consistency and low latency, then you should be the client, not the software engineer (Lol).</li></ul><h3>Hinted Handoff</h3><p>Suppose in the last example that only 2 of 3 replica nodes were available. In this case, the Failure Detector would still allow the request to continue as the number of available replica nodes is not less than the WRITE_CONSISTENCY_LEVEL provided. In this case, the coordinator would behave exactly as described before but there would be one additional step. The coordinator would write locally the hint (the write request blob along with some metadata) in the disk (hints directory) and would keep the hint there for 3 hours (by default) waiting for the replica node to become available again. If the replica node recovers within this period, the coordinator will send the hint to the replica node so that it can update itself and become consistent with the other replicas. If the replica node is offline for more than 3 hours, then a read repair is needed. This process is referred as Hinted Handoff.</p><h3>Write Internals</h3><p>In short, when a write request reaches a node, mainly two things happen:</p><ol><li>The write request is appended to the commit log in the disk. This ensures data durability (the write request data would permanently survive even in a node failure scenario)</li><li>The write request is sent to the memtable (a structure stored in the memory). When the memtable is full, the data is flushed to a SSTable on disk using sequential I/O and the data in the commit log is purged.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wdhGxT-5a5tL7w-1.png" /><figcaption>Figure 6 — Cassandra Node Internals</figcaption></figure><p>I really hope this article has been useful to you. If you enjoyed reading it, let me know if you would like to read another article diving into the Read Request Path. The Read Request Path is a little bit more complicated as it involves Snitches, Bloom Filter, Indexes, and so on, but it’s pretty interesting as well.</p><p>If you are a software developer interested in how to use Cassandra in a realistic scenario coding a real-time chat application from the scratch, please take a look at my book: <a href="https://www.jorgeacetozi.com/books">Pro Java Clustering and Scalability: Building Real-Time Apps with Spring, Cassandra, Redis, WebSocket and RabbitMQ</a></p><p>Thank you very much!</p><h3>Author Books</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7MTfDxtjY63j-NaE.png" /></figure><p><a href="https://leanpub.com/continuous-delivery-for-java-apps">This book</a> will guide you through the implementation of the real-world Continuous Delivery using top-notch technologies that are in high demand by the best companies around the world. Instead of finishing this book thinking “I know what Continuous Delivery is, but I have no idea how to implement it”, you will end up with your machine set up with a Kubernetes cluster running Jenkins Pipelines in a distributed and scalable fashion (each Pipeline run on a new Jenkins slave dynamically allocated as a Kubernetes pod) to test (unit, integration, acceptance, performance and smoke tests), build (with Maven), release (to Artifactory), distribute (to Docker Hub) and deploy (on Kubernetes) a Spring Boot application to testing, staging and production environments implementing the <a href="https://martinfowler.com/bliki/CanaryRelease.html">Canary Release</a> deployment pattern to mitigate risks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/0*7D2CDOQ2pl1oDo81.jpg" /></figure><p><a href="http://www.apress.com/us/book/9781484229842">This book</a> is a step-by-step guide on how to build a real-time chat application using Spring Boot, WebSocket, Cassandra, Redis and RabbitMQ. The application code is available on GitHub and you can start cloning the repository and running it on Docker right now: <a href="https://github.com/jorgeacetozi/ebook-chat-app-spring-websocket-cassandra-redis-rabbitmq">https://github.com/jorgeacetozi/ebook-chat-app-spring-websocket-cassandra-redis-rabbitmq</a></p><p>Although this is a programming book, it also brings many interesting infrastructure discussions and tips about Continuous Delivery, Docker, NoSQL (Cassandra and Redis) and so on. This book also discusses on how to horizontally scale the WebSocket Chat Application using a full STOMP broker like RabbitMQ. My objective writing this book is to bring to readers a new experience mixing a lot of development code with infrastructure interesting and didactic discussions. I’m sure that you’ll really enjoy it! :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FGAWrH92HbHhdvQHYkYeZg.png" /></figure><p><a href="https://www.graylogbook.com/">In this hands-on book</a>, you will learn how to centralize and manage logs using the awesome open source Graylog2 and create a scalable, high-throughput and high-available log processing production infrastructure deploying Elasticsearch and MongoDB clusters as well as Nginx on top of Docker containers. Besides, we will use Fluentd to send logs to many outputs, such as GELF and Amazon S3.</p><p>To make everything even more realistic, we will create our whole environment using different virtual machines managed by Vagrant (deploying everything to “localhost” is not that exciting, you know?). Just grab a beer, open a terminal, type “vagrant up” and have fun!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51e339bcfe0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jorgeacetozi/cassandra-architecture-and-write-path-anatomy-51e339bcfe0c">Cassandra Architecture and Write Path Anatomy</a> was originally published in <a href="https://medium.com/jorgeacetozi">jorgeacetozi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Continuous Integration vs Continuous Delivery vs Continuous Deployment]]></title>
            <link>https://medium.com/jorgeacetozi/continuous-integration-vs-continuous-delivery-vs-continuous-deployment-d5839a85a959?source=rss----af8907a3ade0---4</link>
            <guid isPermaLink="false">https://medium.com/p/d5839a85a959</guid>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <dc:creator><![CDATA[Jorge Acetozi]]></dc:creator>
            <pubDate>Wed, 15 Nov 2017 13:00:51 GMT</pubDate>
            <atom:updated>2017-11-15T13:02:25.490Z</atom:updated>
            <content:encoded><![CDATA[<p>Continuous Integration, Continuous Delivery or Continuous Deployment? There is always a great misunderstanding of these terms out there! Let’s clarify them in only 5 minutes.</p><h3>Continuous Integration</h3><p>Continuous Integration (CI) is a development practice from <a href="http://www.extremeprogramming.org/">Extreme Programming</a> that requires developers to integrate code into a mainline as often as possible, at least once a day, and each check-in is then verified by an automated build that compiles the code and runs the suite of automated tests against it, allowing teams to detect problems early.</p><p>In the Git context, the mainline often turns out to be the master branch and each check-in is a new commit that emerges into the remote repository.</p><p>Implementing continuous integration provides several benefits:</p><p>• Merge hell, that is, endless merge conflicts while merging code into the mainline, is avoided because the code should be integrated to the mainline as often as possible (at least once a day).</p><p>• Allows detecting errors quickly as automated tests run on every single commit that checks-in the remote repository. Hence, fewer bugs get shipped to production.</p><p>• When an error is detected, it’s much easier to fix because the amount of code involved in each commit is much smaller.</p><p>• Promotes human communication earlier when merge conflicts occur. People can sit together and understand how their codes are actually influencing the other’s code.</p><p>• Building the release is easy as all integration issues have been solved early.</p><p>However, everything comes at a price. To implement continuous integration, the development team must understand that:</p><p>• Tasks should be split as small as possible in the planning stage because the code should be developed (including automated tests) and integrated into the mainline as often as possible, at least once a day.</p><p>• Automated tests should be present for each new feature, improvement or bug fix. They will ensure that a change (even a small one) is not breaking any other part of the system.</p><p>The second part of the definition states that “each check-in is then verified by an automated build that compiles the code and runs the suite of automated tests”. In order to run automated tests on every single commit that emerges into the remote repository mainline, the team will need to set up a Continuous Integration server. The essential purpose of the CI server is to run automated tests on every new commit that emerges into the remote repository mainline, but it is actually capable of doing much more than that, such as issuing custom notifications when a test or build fail, triggering releases generation, triggering deployments to a specific environments, and so on. A well-known CI server is <a href="https://jenkins.io/">Jenkins</a>, a leading open source CI server written in Java that provides hundreds of plugins to support building, deploying and automating any project.</p><p>A smooth Continuous Integration flow will ensure that the mainline is always in a deployable state. Continuous Integration and others Extreme Programming practices such as <a href="http://www.extremeprogramming.org/rules/unittests.html">Automated Tests</a>, <a href="http://www.extremeprogramming.org/rules/pair.html">Pair Programming</a>, <a href="http://www.extremeprogramming.org/rules/testfirst.html">Test Driven Development</a>, <a href="http://www.extremeprogramming.org/rules/refactor.html">Refactoring</a>, and <a href="http://www.extremeprogramming.org/rules/simple.html">Simple Design</a> are all great contributors to the quality of an application code. If your team doesn’t care for these (and others) techniques, it’s very likely that you cannot ensure that the code actually works as intended. So, how could you have an automated deployment flow if you have no idea whether the code being delivered works or not? That’s the reason why Continuous Integration is a requirement for implementing both Continuous Delivery and Continuous Deployment.</p><h3>Continuous Delivery</h3><p>I really like Jezz Humble’s definition of continuous delivery:</p><blockquote>Continuous Delivery is the ability to get changes of all types; including new features, configuration changes, bug fixes and experiments into production, or into the hands of users, safely and quickly in a sustainable way.</blockquote><p>In other words, when your team really implements continuous delivery what it actually means that the mainline is always in a deployable state, and one could decide to deploy it to production at any time at the touch of a button. This in turns is only possible because when this button is touched, an automated pipeline (a set of automated steps that your code changes goes through until it is finally deployed to production) is triggered. The key element for achieving continuous delivery is automation!</p><p>Well, if you can deploy to production at any time, the question is: when is it the right time? Of course, the answer may vary depending on your business requirements, but the truth is that if you want to get the benefits of continuous delivery, you should deploy to production as early as possible to make sure that you release small batches that are easy to troubleshoot in case of a problem.</p><h3>Continuous Deployment</h3><p>Continuous deployment is very similar to continuous delivery, but it takes even one step further regarding automation. In continuous delivery, every change pushed to the main repository is ready to be deployed to production, but starting this process still requires human interaction. In continuous deployment, the deployment to production is automatically triggered for every change that passes the test suite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hNG6rNJsnaOOXPqx.png" /><figcaption>Continuous Integration vs Continuous Delivery vs Continuous Deployment</figcaption></figure><p>As in continuous deployment every change that passes the test suite is deployed to production, it heavily relies on <a href="https://martinfowler.com/articles/feature-toggles.html">Feature Flags</a> to release features to end users.</p><p>This article was based on the contents of my book <a href="https://leanpub.com/continuous-delivery-for-java-apps">Continuous Delivery for Java Apps: Build a CD Pipeline Step by Step Using Kubernetes, Docker, Vagrant, Jenkins, Spring, Maven, and Artifactory</a>. <strong>Click on the link and download the free book sample (110 free pages!)</strong> to learn more about Agile, Continuous Delivery, Scrum, Automated Tests (unit, integration, acceptance and performance tests), Feature Branch, Feature Flags, A/B Tests, Canary Releases, Apache Maven, Artifactory, Docker, Kubernetes and more!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8zzj8F392ssPlGWi.png" /></figure><p><a href="https://leanpub.com/continuous-delivery-for-java-apps">This book</a> will guide you through the implementation of the real-world Continuous Delivery using top-notch technologies that are in high demand by the best companies around the world. Instead of finishing this book thinking “I know what Continuous Delivery is, but I have no idea how to implement it”, you will end up with your machine set up with a Kubernetes cluster running Jenkins Pipelines in a distributed and scalable fashion (each Pipeline run on a new Jenkins slave dynamically allocated as a Kubernetes pod) to test (unit, integration, acceptance, performance and smoke tests), build (with Maven), release (to Artifactory), distribute (to Docker Hub) and deploy (on Kubernetes) a Spring Boot application to testing, staging and production environments implementing the <a href="https://martinfowler.com/bliki/CanaryRelease.html">Canary Release</a> deployment pattern to mitigate risks.</p><p>This is the book I wish I had found when I was learning about Continuous Delivery; that’s why I wrote it. It will bring to you years of experience implementing Continuous Delivery in many different projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5839a85a959" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jorgeacetozi/continuous-integration-vs-continuous-delivery-vs-continuous-deployment-d5839a85a959">Continuous Integration vs Continuous Delivery vs Continuous Deployment</a> was originally published in <a href="https://medium.com/jorgeacetozi">jorgeacetozi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why to use Kubernetes?]]></title>
            <link>https://medium.com/jorgeacetozi/why-to-use-kubernetes-22f28cac68fe?source=rss----af8907a3ade0---4</link>
            <guid isPermaLink="false">https://medium.com/p/22f28cac68fe</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jorge Acetozi]]></dc:creator>
            <pubDate>Sat, 19 Aug 2017 22:45:04 GMT</pubDate>
            <atom:updated>2017-08-20T19:54:12.389Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*d9MXw4wPVrLXhgG6eaP-pg.png" /><figcaption>Kubernetes Logo</figcaption></figure><p>In this tutorial, I’m going to introduce you to <a href="https://kubernetes.io/">Kubernetes</a>, an open-source system for automating deployment, scaling, and management of containerized applications.</p><p>Let’s start with a question. Would you deploy an Elasticsearch cluster for your company’s production environment using Docker on a single machine with several containers in it, just like you do on your local machine when you are using or learning Docker? Of course not! Why not? Mainly because you want to keep your job, don’t you? So, what’s wrong with this approach? Let’s list out some arguments below:</p><ul><li>The containers (Elasticsearch instances) would be running on a single machine. What would happen if this machine crashes?</li><li>If some of those containers are down, how would you know about that? Besides, you would have to bring them up again by yourself (issuing the “docker run” command), and that is not funny when it’s 3:00 AM.</li></ul><p>Actually, even if you run these containers on separate machines, you would still have to manage them one by one by yourself.</p><p>Of course that you could address these issues above by running a systemd unit for each container on each machine that would ensure the containers would always be running. However, again, what if the machines crash? Who would have to start new containers in other machines in order to bring the Elasticsearch cluster up again? That’s correct, you! However, if Kubernetes were managing these containers for you, it would be in charge of relaunching them again on the most appropriate machines available in the Kubernetes cluster (for example, the ones that are less overloaded).</p><p>Well, that might be arguments enough, but there are still much more. Forget about the Elasticsearch thing and imagine now that you are running a stateless web application using Docker containers. How would you deploy a new application release? Manually removing containers and launching new ones based on the new Docker image? Really, as a DevOps Engineer (or whatever you wanna call it), does it sound good to read the word “manually”? I bet you know the key for the DevOps culture is automation.</p><p>I think you are already convinced that, for production purposes, running and managing (or… <strong>orchestrating</strong>) container life cycles by yourself may not be a good idea. So, who is going to do this for you? That’s right, Kubernetes! That’s why we call tools like Kubernetes, <a href="https://github.com/docker/swarm">Docker Swarm</a>, <a href="https://aws.amazon.com/ecs/">Amazon ECS</a> and others as <strong>Container Orchestration Tools</strong>.</p><p>Kubernetes has a set of great features that are very useful specially when it comes to production environments:</p><ul><li>Container replication among different nodes to ensure high availability.</li><li>Automatic container recovery when it gets down for whatever reason.</li><li>Container auto-scaling based on Kubernetes cluster metrics such as CPU consumption.</li><li>Rolling deployments and deployment rollbacks.</li><li>Service Discovery. That’s neat for deploying Microservices!</li><li>Load Balancing and Volume management.</li><li>Container health checks.</li><li>Logical resources isolation using Namespaces.</li><li>Controlling resources and quotas by Namespaces.</li><li>Managing cron jobs.</li></ul><p>Actually, there are even more than that! In next tutorials I will dive into some Kubernetes concepts such as Pods, Replica Sets, Services, Deployments, and so on. We will learn how to run Kubernetes locally, then we will deploy some cool stuff in it and get familiar with <a href="https://kubernetes.io/docs/user-guide/kubectl-overview/">kubectl</a>, which is the command line interface for running commands against Kubernetes clusters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22f28cac68fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jorgeacetozi/why-to-use-kubernetes-22f28cac68fe">Why to use Kubernetes?</a> was originally published in <a href="https://medium.com/jorgeacetozi">jorgeacetozi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>