<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[kiratech.io - Medium]]></title>
        <description><![CDATA[DevOps Technical Publication - Medium]]></description>
        <link>https://medium.com/kiratech?source=rss----6e4f7ba0a355---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>kiratech.io - Medium</title>
            <link>https://medium.com/kiratech?source=rss----6e4f7ba0a355---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:04:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/kiratech" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fun with Consul]]></title>
            <link>https://medium.com/kiratech/fun-with-consul-af2b8b04c219?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/af2b8b04c219</guid>
            <category><![CDATA[hashicorp-consul]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[consul]]></category>
            <dc:creator><![CDATA[Ivandal]]></dc:creator>
            <pubDate>Wed, 13 Oct 2021 16:10:06 GMT</pubDate>
            <atom:updated>2021-10-13T18:04:24.735Z</atom:updated>
            <content:encoded><![CDATA[<p>In the era of microservices on Kubernetes, the challenge of Security, Observability and Traffic management can be met by what&#39;s called a <strong>Service Mesh</strong>, an infrastructure layer that helps while tackling these topics.</p><p>In this context arena a contender is <a href="https://www.consul.io/">Consul by Hashicorp</a>.</p><p>In this article, we will learn how to spin up a Kubernetes cluster, install Consul Enterprise Version, and give it a try.</p><h3>Prerequisites</h3><p>All the scripts and the test below were done using Linux. In case you are using Windows or macOS, nothing will be guaranteed apart from an opportunity to try a Linux distribution :-)</p><p>The requirements are both hardware and software:</p><p><strong>Hardware</strong>: 2 vCPUs, 4GB of RAM for running Kubernetes;</p><p><strong>Software</strong>:<br> * <a href="https://minikube.sigs.k8s.io/docs/start/">Minikube</a>;<br> * <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/">kubectl</a>;<br> * <a href="https://helm.sh/">Helm</a>;<br> * <a href="https://releases.hashicorp.com/consul/">Consul</a> installed locally.</p><h3>Running Consul</h3><p>First of all, we will need a valid Consul License, which can be obtained <a href="https://www.hashicorp.com/products/consul/trial">here</a>. Let&#39;s create a new folder and put the license file in here. In this folder, create the following script:</p><pre>#!/bin/bash<br>LICENSE_PATH=&quot;./license.hclic&quot;<br>CONSUL_HELM_VALUES_PATH=&quot;./values.yaml&quot;<br>CONSUL_CHART_VERSION=&quot;v0.32.1&quot;</pre><pre>minikube start --cpus=2 --memory=4gb<br>sleep 120<br>kubectl create secret generic consul-gossip-encryption-key --from-literal=key=$(consul_ent keygen)</pre><pre>secret=$(cat $LICENSE_PATH)<br>kubectl create secret generic consul-ent-license --from-literal=&quot;key=${secret}&quot;<br>helm repo add hashicorp <a href="https://helm.releases.hashicorp.com/">https://helm.releases.hashicorp.com</a><br>helm install -n default consul -f $CONSUL_HELM_VALUES_PATH hashicorp/consul --wait --version &quot;$CONSUL_CHART_VERSION&quot;</pre><p>As you can see, the values.yaml file is missing in the repository. Here is one, not the simplest one, but it is useful for testing some enterprise features:</p><pre>global:<br>  name: consul<br>  datacenter: dc1<br>  image: &quot;hashicorp/consul-enterprise:1.10.3-ent&quot;</pre><pre>gossipEncryption:<br>    secretName: &quot;consul-gossip-encryption-key&quot;<br>    secretKey: &quot;key&quot;</pre><pre>tls:<br>    enabled: true<br>    enableAutoEncrypt: true<br>    verify: true</pre><pre>enableConsulNamespaces: true</pre><pre>acls:<br>    manageSystemACLs: true</pre><pre>server:<br>  replicas: 3<br>  bootstrapExpect: 3<br>  affinity: null<br>  enterpriseLicense:<br>    secretName: &#39;consul-ent-license&#39;<br>    secretKey: &#39;key&#39;</pre><pre>client:<br>  enabled: true<br>  grpc: true</pre><pre>ui:<br>  enabled: true<br>  service:<br>    type: &#39;NodePort&#39;</pre><pre>connectInject:<br>  enabled: true<br>  consulNamespaces:<br>    mirroringK8S: true</pre><pre>controller:<br>  enabled: true</pre><pre>ingressGateways:<br>  enabled: true<br>  defaults:<br>    replicas: 1<br>    service:<br>      type: ClusterIP<br>    consulNamespace: &quot;default&quot;<br>    affinity: null<br>  gateways:<br>    - name: ingress-test<br>      service:<br>        type: NodePort<br>        ports:<br>        - port: 8080<br>          nodePort: 30080<br>        - port: 8443<br>          nodePort: 30443</pre><p>Now, execute the script without fear.</p><p>After few minutes the shell will exit with something like this:</p><pre>|-----------|-----------|-------------|----------------------------|<br>| NAMESPACE |   NAME    | TARGET PORT |             URL             |<br>|-----------|-----------|-------------|----------------------------|<br>| default   | consul-ui | https/443   | <a href="http://192.168.99.150:30580/">http://192.168.99.150:30580</a> |<br>|-----------|-----------|-------------|----------------------------|</pre><p>and a new browser will be opened at the URL showing a beautiful Consul UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBpTCMNKvVoPgxppYyv5NQ.png" /></figure><blockquote>Not so beautiful, you are right.</blockquote><p>As you can notice, we are not logged in, and for doing that we need a token. Why a token? Because we are using the Enterprise version with ACL enabled, therefore a token is required.</p><p>Since we just installed Consul, the “master of puppets” token, or bootstrap-acl-token is in a K8s secrets, just decode and use it to log in to the UI:</p><pre>kubectl get secrets/consul-bootstrap-acl-token --template={{.data.token}} | base64 -d</pre><p>Now the interface will show you some more tabs like Key/Value, Token, Policy, and many others. However, I know there are a lot of people out there screaming that they want the CLI… which is pretty simple to obtain: just setup the environment for the Consul CLI as follows:</p><pre>export CONSUL_HTTP_TOKEN=$(kubectl get secrets/consul-bootstrap-acl-token --template={{.data.token}} | base64 -d)<br>export CONSUL_HTTP_ADDR=$(minikube service --https=true --url=true consul-ui) <br>export CONSUL_HTTP_SSL_VERIFY=false</pre><blockquote>et-voilà, the Consul CLI is configured.</blockquote><p>For instance, in order to obtain the members of the mesh, we can run the following command:</p><pre>consul members</pre><p>The output should be similar the following:</p><pre>Node Address Status Type Build Protocol DC Segment <br>consul-server-0 172.17.0.7:8301 alive server 1.10.1+ent 2 dc1 &lt;all&gt; consul-server-1 172.17.0.9:8301 alive server 1.10.1+ent 2 dc1 &lt;all&gt; consul-server-2 172.17.0.8:8301 alive server 1.10.1+ent 2 dc1 &lt;all&gt; minikube 172.17.0.3:8301 alive client 1.10.1+ent 2 dc1 &lt;default&gt;</pre><p>We now have our service mesh up&amp;running, but no services on it!</p><p>In the next article we will deploy some services and run some nice tests.</p><p>Have fun in the meantime 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af2b8b04c219" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/fun-with-consul-af2b8b04c219">Fun with Consul</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing the HashiCorp Vault CSI on Openshift 4]]></title>
            <link>https://medium.com/kiratech/installing-the-hashicorp-vault-csi-on-openshift-4-6bc0b2c0b841?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/6bc0b2c0b841</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[openshift-4]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[francesco antonio migliorin]]></dc:creator>
            <pubDate>Wed, 29 Sep 2021 12:59:16 GMT</pubDate>
            <atom:updated>2021-09-29T12:59:16.693Z</atom:updated>
            <content:encoded><![CDATA[<p>The HashiCorp Vault Container Storage Interface (CSI) allows you to mount Vault Secrets stored in HashiCorp Vault as ephemeral Kubernetes Volumes.</p><p>However, if you tried to follow the <a href="https://learn.hashicorp.com/tutorials/vault/kubernetes-secret-store-driver?in=vault/kubernetes">official tutorial</a> on an Openshift 4.x cluster, you have most likely had some troubles, mostly due to OpenShift’s strict permission policies.</p><p>In this article, we will learn how to install the HashiCorp Vault CSI on Openshift 4.x.</p><h3>Prerequisites</h3><ul><li><a href="https://helm.sh/">Helm CLI</a>;</li><li><a href="https://docs.openshift.com/container-platform/4.8/cli_reference/openshift_cli/getting-started-cli.html">OpenShift CLI</a>;</li><li>An OpenShift 4.x cluster.</li></ul><h3>Install Vault CSI</h3><ol><li>Let’s start creating a new project for Vault and Kubernetes CSI:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c3b45fb56b7ef778a1a9c31f3a348f2/href">https://medium.com/media/6c3b45fb56b7ef778a1a9c31f3a348f2/href</a></iframe><p>2. Vault CSI and Kubernetes CSI need to have a <em>privileged</em> <a href="https://docs.openshift.com/container-platform/4.8/authentication/managing-security-context-constraints.html">SecurityContextConstraint (SCC)</a>. For this reason we are going to give it to their default service account:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28f27456fc0075e5aa95ea2b2f7c0266/href">https://medium.com/media/28f27456fc0075e5aa95ea2b2f7c0266/href</a></iframe><p>3. Add the Secrets Store CSI driver and the HashiCorp Helm repositories:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/891267d4ef2a0e9d25faf8aa8e72edb5/href">https://medium.com/media/891267d4ef2a0e9d25faf8aa8e72edb5/href</a></iframe><p>4. Install the Kubernetes-Secrets-Store-CSI-Driver:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ce79cfd9b48a06d2fa3fe5e26cc64c1/href">https://medium.com/media/3ce79cfd9b48a06d2fa3fe5e26cc64c1/href</a></iframe><blockquote><em>This command will take a while because we are also waiting until all the pods are up&amp;running.</em></blockquote><p>5. Install the Vault Helm chart. In this article, we will run it in development mode, with the injector service disabled and the CSI enabled:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45ff8dbcb0c58d038a88b4c505baa86c/href">https://medium.com/media/45ff8dbcb0c58d038a88b4c505baa86c/href</a></iframe><p>6. If we check the pods now, we will see all the Vault CSI pods are in a <em>CrashLoopBackOff </em>state:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0b6373996bc6f1981b73f7bc869b982/href">https://medium.com/media/b0b6373996bc6f1981b73f7bc869b982/href</a></iframe><p>This is because the CSI DaemonSet containers need to be run as privileged too. The only way to solve this issue is to patch it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3fa824906de69bd5439a2dda51e11d6/href">https://medium.com/media/a3fa824906de69bd5439a2dda51e11d6/href</a></iframe><p>After some seconds all the pods will be up&amp;running.</p><h3>Configure Vault</h3><ol><li>Start an interactive shell session on the <em>vault-0</em> pod:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28c6802e943e8bcb0fc4ac71414af080/href">https://medium.com/media/28c6802e943e8bcb0fc4ac71414af080/href</a></iframe><p>2. Create a test secret and its policy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94243dd126d961ec5db712af3a3fe84c/href">https://medium.com/media/94243dd126d961ec5db712af3a3fe84c/href</a></iframe><p>3. Configure the Kubernetes authentication method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/424ae5a005ad4b133976e875f075630c/href">https://medium.com/media/424ae5a005ad4b133976e875f075630c/href</a></iframe><p>4. Create a role for the test application:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65a763915822d5ca8cd848b02e5e9298/href">https://medium.com/media/65a763915822d5ca8cd848b02e5e9298/href</a></iframe><p>5. Exit the <em>vault-0</em> pod with the <em>exit </em>command.</p><h3>Run a demo application</h3><ol><li>1. Let’s create a new demo project:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9d282157755f413333b1a4a91da74c1/href">https://medium.com/media/b9d282157755f413333b1a4a91da74c1/href</a></iframe><p>2. The Kubernetes-CSI-Driver provides a CRD named <em>SecretProviderClass</em>. This resource describes the parameters that are given to the Vault CSI provider. So let’s create it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ee16f42b350fe97f287a0965df21775/href">https://medium.com/media/2ee16f42b350fe97f287a0965df21775/href</a></iframe><p>3. Before launching our demo pod, we need to overcome one last obstacle. The pod needs to have the ability to grab CSI volumes. To do so, we need to create a new SCC or patch an existing one and add it to our pod’s service account. In this article, I will patch the <em>restricted</em> SCC and, will add it to the <em>default</em> service account in the <em>demo</em> namespace:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc3cc47842938a11bea77a9c2695a56c/href">https://medium.com/media/cc3cc47842938a11bea77a9c2695a56c/href</a></iframe><p>4. Now we can deploy the demo pod:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bdd3eead15debaa4fb9e3426b40d09f/href">https://medium.com/media/4bdd3eead15debaa4fb9e3426b40d09f/href</a></iframe><p>5. We can finally display the secret password written to the file system:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a8b26b689df3b60685b095779d71194/href">https://medium.com/media/3a8b26b689df3b60685b095779d71194/href</a></iframe><h3>Clean up</h3><p>To remove all the things created in this article, we need to run the following commands:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54a101bd0a82488451d58c7832dac6c6/href">https://medium.com/media/54a101bd0a82488451d58c7832dac6c6/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bc0b2c0b841" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/installing-the-hashicorp-vault-csi-on-openshift-4-6bc0b2c0b841">Installing the HashiCorp Vault CSI on Openshift 4</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster parallel execution of Ansible playbooks]]></title>
            <link>https://medium.com/kiratech/faster-parallel-execution-of-ansible-playbooks-f8a01186ea2?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/f8a01186ea2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[parallel-processing]]></category>
            <category><![CDATA[provisioning]]></category>
            <dc:creator><![CDATA[Lorenzo Pistone]]></dc:creator>
            <pubDate>Mon, 02 Aug 2021 15:58:46 GMT</pubDate>
            <atom:updated>2021-08-30T08:48:41.039Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post I will show you a technique that can be used to parallelize any task loop in Ansible.</p><p>Ansible is an automatization platform, (relatively) simple, and very versatile. Unfortunately, it is quite lacking when it comes to parallelization. There are fundamentally two ways in which you can parallelize operations in Ansible:</p><ul><li>work on multiple hosts at the same time</li><li>launch (and poll) multiple tasks <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html#run-tasks-concurrently-poll-0">concurrently</a> with the poll: 0 keyword.</li></ul><p>Unfortunately these standard solutions have drawbacks. In order to parallelize work across several hosts, you need to have different hosts in the first place: this means that if you aim to parallelize a loop of task within a single host, you are out of luck. The second option requires significant modifications of the playbook, because concurrent task must be polled explicitly for completion with the<a href="https://docs.ansible.com/ansible/latest/collections/ansible/builtin/async_status_module.html">async_status</a> task, which is very unpractical. Most importantly, async_status is not supported for all tasks, and in particular it is not supported in tasks that include roles or other task lists. This means that there is no way to run concurrently third-party roles, which largely limits the usability of this feature.</p><p>The most reliable and effective way to parallelize computations in Ansible is indeed by using the implicit host loop, possibly paired with a large value of the parameter and the <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#selecting-a-strategy">free strategy</a>. How can we (ab)use the host loop to parallelize tasks? It turns out that is possible and quite easy, with minimal modifications to the playbook file.</p><h3>Virtual host slices</h3><p>The idea is to be able to create multiple <em>virtual</em> host copies, from your real inventory hosts, and to distribute work among them. Several virtual hosts will own a <em>slice</em> of your original workload, and just run that, concurrently with the other virtual hosts, according to the number of forks and the play strategy.</p><p>Fortunately, it is possible to add new hosts during the execution of a playbook! You just use the <a href="https://docs.ansible.com/ansible/latest/collections/ansible/builtin/add_host_module.html">ansible.builtin.add_host task</a>. The new hosts can then be selected in a specific play of the playbook.</p><p>Consider this example: your Ansible playbook is configuring some hosts. The configuration consists in running some jobs or workloads with a fictious role run-job, and the job items are are listed in some variable job_ids. Here is a playbook that implements this logic:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95e1551efbc65ea61017f61789ba0e64/href">https://medium.com/media/95e1551efbc65ea61017f61789ba0e64/href</a></iframe><p>We want to parallelize the task Launch all job items, without editing the task files of role run-job. This requires creating some virtual copies to your hosts, each holding a part of the job_ids array. Then, you define a <em>second</em> play in your playbook, which selects on the new host copies, and runs the Launch all job items task unmodified.</p><p>To showcase the result, I have written a ready-to-use Ansible role that you can pull from the Galaxy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36b2993a456dafa5f765128b501da5d3/href">https://medium.com/media/36b2993a456dafa5f765128b501da5d3/href</a></iframe><p>This role takes these two mandatory input:</p><ul><li>payload_var: the variable holding the array that should be sliced</li><li>batch: the size of each slice.</li></ul><p>The number of slice hosts is implicitly determined by this parameter batch. The new hosts will be part (by default) of the new group virtual, which can be used as a host pattern to select on the new virtual hosts.</p><p>Here is how you would modify the playbook:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65c2e6ad615bf09515780ede77419b43/href">https://medium.com/media/65c2e6ad615bf09515780ede77419b43/href</a></iframe><p>The modifications are pretty simple. Now we can try to benchmark the playbooks.</p><h3>Benchmark</h3><p>The code in this post is available <a href="https://github.com/pisto/virtual_slice_hosts_examples">here</a>. The generate-jobs role simulates work IDs by creating a list of strings of length job_count. The run-job role simulates running a job by sleeping for a random delay, between job_duration_ms_min and job_duration_ms_max (milliseconds, default values respectively 10 and 200).</p><p>Let’s write an inventory for three hosts, and let’s generate 100 jobs for each, of a duration between 0 and 200 milliseconds:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1144f91ce5d7c4d1a177059b8a284362/href">https://medium.com/media/1144f91ce5d7c4d1a177059b8a284362/href</a></iframe><p>We can now time the execution of the first playbook, which naively includes the run-job role in a loop:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebbd18916f5fa179018e021a013b1342/href">https://medium.com/media/ebbd18916f5fa179018e021a013b1342/href</a></iframe><p>That is 113 seconds! And we also notice something interesting: Ansible got 100 iterations of the loop for each task, and it is running all of them sequentially, disregarding host-level parallelization. This is because of a <a href="https://github.com/ansible/ansible/issues/30816">known</a> <a href="https://github.com/ansible/ansible/issues/36978">bug</a> (which is not going to be addressed anytime soon I think…) that causes Ansible break host parallelization when the loop items are not the same. Since the generate-jobs mock role creates all different job items in the format of a string{{ inventory_hostname }}-{{ job_id }}, you get this completely serialized execution.</p><p>Now, let’s try to run the modified playbook with virtual hosts, with a job_batch_size of 10, and --forks 20:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5675acbab00e2c18115e6601cfa83ac3/href">https://medium.com/media/5675acbab00e2c18115e6601cfa83ac3/href</a></iframe><p>That is 21 seconds, for a 80% reduction of wall-clock execution time! And we can observe that all the tasks are in parallel across all (virtual) hosts, because we can see in the console interlaced task output from different hosts.</p><h3>Implementation</h3><p>But what is under the hood of the pisto.virtual_slice_hosts slices? It is rather simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efa2a1c7830e21bd8f136593a4b64339/href">https://medium.com/media/efa2a1c7830e21bd8f136593a4b64339/href</a></iframe><p>The role essentially loops over all hosts, and all slices for each host of the payload variable payload_var, and creates a new host in the specified group_name with its own slice of the content of payload_var.</p><p>The important implementation detail here is that in order to copy a host into a virtual slice host, one needs to know which variables should be applied to the new host. These obviously include several connection and privilege escalation variables such as ansible_host or ansible_become. The role copies by default a set of known connection variables (specified in the role default variable implicit_copy_vars), and all variables explicitly declared in role variable copy_vars. For example, you may want to run a benchmark as the one above, but setting different amounts of work for each host. In order to do that, you can specify host-specific values for job_duration_ms_min and job_duration_ms_max in the inventory, then you need to modify the Create virtual slice hosts task into:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/468938b593e2b4e300480d2054293c12/href">https://medium.com/media/468938b593e2b4e300480d2054293c12/href</a></iframe><p>Of course the list of variables depends on your original playbook. You may want to override copy_vars once and for all at the playbook level.</p><h3>Conclusion</h3><p>In this post I showed how you can parallelize arbitrary sections and loops of your playbooks, with minimal modifications to the code of the playbook itself. In particular, loops over items that change on a per-host basis are particularly slow to execute with a naive implementation in Ansible. The trick is to generate new hosts during the playbook execution, using the add_host task, and then execute operations in parallel across these virtual copies of the original host.</p><p>The implementation is logic is trivial, and a basic but versatile implementation is available from the <a href="https://galaxy.ansible.com/pisto/virtual_slice_hosts">Ansible Galaxy</a> as the role pisto.virtual_slice_hosts. The speedup to serial Ansible code is significant: in the synthetic benchmarks shown here there is an 80% reduction of the wall-clock execution time.</p><p>The only significant requirements for employing this technique is that the user must know which variables should be copied over to the virtual hosts, and that in most cases the ansible.builtin.free strategy must be used to unlock the performance gains; otherwise, Ansible will most likely serialize your loops due to long standing limitations of Ansible itself (<a href="https://github.com/ansible/ansible/issues/30816">#30816</a> <a href="https://github.com/ansible/ansible/issues/36978">#36978</a>).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8a01186ea2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/faster-parallel-execution-of-ansible-playbooks-f8a01186ea2">Faster parallel execution of Ansible playbooks</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic vs Static linking, or how the software development changed in the last 10 years]]></title>
            <link>https://medium.com/kiratech/dynamic-vs-static-linking-or-how-the-software-development-changed-in-the-last-10-years-f74cbcb832ee?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/f74cbcb832ee</guid>
            <category><![CDATA[static-library]]></category>
            <category><![CDATA[dynamic-library]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[compilers]]></category>
            <category><![CDATA[operating-systems]]></category>
            <dc:creator><![CDATA[Luca De Pandis]]></dc:creator>
            <pubDate>Fri, 30 Jul 2021 10:06:55 GMT</pubDate>
            <atom:updated>2021-07-30T10:06:55.247Z</atom:updated>
            <content:encoded><![CDATA[<p>When we talk about how executables are made we cannot avoid talking about the <strong>linking </strong>part. Executables are just the product of a lot of steps: the pre-processing of the source code, the generation of the object code and at the last (but not least) the linking.</p><p>There are two kinds of linking: <strong>static </strong>and <strong>dynamic</strong>. Let’s see what’s the difference between them and why we chose the latter for almost 30 years, and how this influenced operating system design choices. We’re going to use Linux as the reference OS for this post, but such concepts are valid for every OS out there.</p><h3>Dynamic Linking</h3><p>The most common linking practice is the dynamic linking. When the linker (ld(1)) is involved during the compilation process, it is going to create an <strong>executable </strong>by merging the object codes generated by the compiler, but it&#39;s not going to include any external dependent code, be it the C runtime or a third-party library used by your code. It is just going to put a reference to a specific symbol and its own version exposed by the library. The symbol version, in POSIX environments, is usually exposed through the library versioning in the filename (e.g. libfoo.so.1).</p><p>Such information is read by the binary loader (ld-linux.so(8)) who&#39;ll try to identify where such libraries are located by searching for them in the default paths (/lib and /usr/lib). Such paths are saved in the /etc/ld.so.cache file, a small and fast database for locating and identifying dynamic loadable libraries.</p><figure><img alt="Dynamic linking" src="https://cdn-images-1.medium.com/max/451/1*lp-sKP9MCYcx-d4iblOdhg.png" /><figcaption>Dynamic linking</figcaption></figure><p>This database can be overridden by two environments variables: $LD_LIBRARY_PATH and $LD_PRELOAD.</p><p>The first is going to define a colon-separated list of paths (just like the$PATH or $MANPATH environment variables) where the loader is forced to search <strong><em>before</em></strong> looking into the paths listed in the cache file. So, if you have a third-party library in a non-standard path (e.g. /opt/foo/lib/libfoo.so.1), you can run your binary by specify that path in the environment variable and everything will work fine. Please note that another way to tell the binary where to find libraries when they are in a non-standard path is to pass the -R &lt;path&gt; or -rpath=&lt;path&gt; flags to the linker and specify the runtime path where the libraries are located. Most proprietary software is linked this way.</p><p>The second environment variable defines a specific library that has to be loaded before the starting of the application. Such libraries don&#39;t have to be dependences. It can also be a third-party library which inject code by overriding routines that the application will run. In my opinion, these two environment variables represent a massive<strong> security hole</strong> in an operating system. Allowing a user to replace dependencies or to inject code at runtime can be a way for exploits to take control and damage an operating system. And this has already <a href="https://www.trendmicro.com/en_us/research/16/i/pokemon-themed-umbreon-linux-rootkit-hits-x86-arm-systems.html">happened</a>. And do you know how to save your systems from that malware? Well, with a statically linked userland! :P</p><h3>Static Linking</h3><p>Static linking is the linking procedure where the linker is going to assemble an executable by including the object code from the third-party libraries that contains the routines your program is running. Static linking has been an ostracized method for linking executables for years, despite it was also the first way to do so in the first years of computing. There was a massive campaign against it, especially about how security was going to be a nightmare with static linking, because people would have to recompile the software whenever a library was exploited and a patch for fixing the problem was released. A major player in the anti-static campaign was the GNU project -especially the GLIBC developer- <a href="https://akkadia.org/drepper/no_static_linking.html">Ulrich Drepper</a>. While, in dynamic linking, libraries are standalone binary objects, in static linking libraries are just AR archives that contains object codes. Therefore the dynamic library libfoo.so.1 would become libfoo.a with static linking. Please note that the most common misconception about static linking is that the executable is going to include the <em>entire</em> libfoo.a.</p><p><strong>THAT’S NOT TRUE!</strong></p><p>As I said before, in static linking <strong>only</strong> the object code that contains the routines needed from the executable is linked. So, if libfoo.a contains foo.o, bar.o and baz.o object codes, and the hello_world executable just makes use of the routines included in baz.o, only that object code is going to be included. That also means that there is no need to <em>always</em> recompile your software if a vulnerability is discovered in a dependency, because the object code included could not even contain the vulnerability.</p><figure><img alt="Static linking" src="https://cdn-images-1.medium.com/max/451/1*jgdCB3t6JfVBwbrZ00SRzQ.png" /><figcaption>Static linking</figcaption></figure><p>In addition, from the architectural point of view, statically linked softwares are more simple, and that’s explain why in this paragraph there are way less technical references about the OS components involved in the runtime than in the previous paragraph. In fact, only the interpreter ld-linux.so(8) is involved in order to run the self-contained executable.</p><p>Another point of the anti-static-linking campaign was that the resulting binaries are too large and they waste space than dynamic linking counterparts.</p><p>Actually, this is partially true, because making them so large is due to the libc they are using. GLIBC was designed to avoid static linking, by bloating the binaries as much as it can. Compiling the same binary with a more neutral and static-friendly libc like Musl will result in a way more compact binary. And that&#39;s why Musl is the libc chosen by Alpine Linux, that provides the smallest Docker images out there.</p><p>Also, a well designed code can even produce static binaries that are <a href="http://harmful.cat-v.org/software/dynamic-linking/">smaller</a> than dynamic ones! Also, one of the main benefits of the static linking binaries is that 32-bit binaries can be run in a 64-bit OS without installing the 32-bit runtime. Not only the third-party dependencies, but even the C runtime.</p><p>You can maintain your OS clean without the mess of multi-architecture libraries and run all the applications you need!</p><h3>Why did we choose dynamic linking?</h3><p>As I said before GLIBC was designed to avoid static linking, and Linux, being just a kernel, needed it for compiling itself and the userland we call GNU/Linux, that is the base for almost all the distributions out there. This influenced a lot the software written on that platform.</p><p>Also, dynamic linking was the main choice on almost every mainstream operating system (see DLLs on Windows or DYLIBs on macOS), so most multi-platform softwares were compiled that way.</p><h3>It was an utterful failure</h3><p>Dynamic linking was a disaster in managing operating systems. By tying up the software with third party libraries and the OS/language runtime, it forced developers to provide copies of the dependent libraries with their applications, making duplications something almost natural. The same libfoo could be installed multiple times in different path for serving different applications. This is how Windows managed things, and how proprietary software do things in Linux as well (think about the -rpath part I talked before).</p><p>Also, especially in Windows, but now even in technology like Flatpak, people started to requires specific version of C runtime. Think about how many times in Windows you installed a Visual C++ runtime, every time in a different version. Also, dynamic linking is fragile even where it should shine. Dynamic linking supporters tell that if you have to patch a dynamically linked library, you can recompile it without worrying about the software that makes use of it.</p><p>This is just plain false, and that is the reason behind WinSxS, the cache that Windows use since Windows Vista. Everytime you&#39;re going to change your source code, you&#39;re going to potentially break the references that your software is make use of. So even when you&#39;re going to dynamically linked your library, you have to check everytime if something broke, and if so <strong>recompile</strong> it.</p><p>In Linux dynamic linking also produce another disaster: it&#39;s tying the applications with the Operating Systems. This is the reason behind getting a specific version of a distribution prevents you from getting the latest and greatest version of your favourite softwares. As software keep evolving it could require specific functions that are not yet included in the library located in the distribution repository, so maintainers have to keep the software to the older and still compatibile version, and wait for another release of a distribution to bump the version of the applications.</p><p>Yes, someone could say that there are third-party repositories that circumvent such limitations (think about PPAs). But they are third-party locations that do not provide <strong>ANY</strong> assurance about don&#39;t going to replace system libraries and breaking something else. That&#39;s why most distribution maintainers consider making use of third-party repoitories a bad practice. And this is the reason behind the development of technologies like <strong>FLATPAK</strong>, <strong>SNAP</strong> and the most known <strong>DOCKER</strong>.</p><h3>OK, so Docker saved the day, huh?</h3><p>Not at all. Docker avoided the dynamic linking stressful situations by putting an entire userland in the same space of the applications, and make use of cgroups and namespaces for running applications in a different context than the host OS. But this is like <strong>killing a fly with a bazooka</strong>. And this bloated software delivery even more.</p><p>Ironically, nowadays the same people who argued about the &quot;benefits&quot; of dynamic linking are telling that bloat is not a problem since storage is cheap. But then, why keep wasting time with dynamic linking at all? We don&#39;t need a specific userlands with specific runtimes for running applications. We just need an executable!</p><p>Also, even the last (false) benefit of the dynamic linking, the recompiling, is faded away. With the advent of the CI/CD pipelines, developers recompile a lot of times their software, so keeping it the same as before and recompiling just the dependencies is something that it doesn&#39;t happen anymore.</p><h3>So we don’t have any chance to avoid this mess, right?</h3><p>Yes, we do! :)</p><p>Golang just came to the rescue. It is designed by the same people who designed <a href="https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs">Plan 9</a> years ago, and it is designed to <strong>statically</strong> link binaries, unless you purposely keep walking the wrong path and dynamically link them. That’s why binaries like helm, oc or kubectl and many other cloud-native softwares are distributed without worrying of the distribution you&#39;re using or about the version of it. They are statically linked binaries that can run everywhere!</p><h3>What about the supporters of dynamic linking?</h3><p>Well, many of them now are just too busy by being excited by the cloud-native era (despite it is being populated by statically linked binaries :P) to think about these things anymore.</p><p>Just think about Ulrich Drepper: now it is researching about Unikernels. And Unikernels are just the evolution of static linking, because they are just going to merge your application with a small, self-contained operating system, that can directly runs on a hypervisor without making use of a host OS with a container runtime (like Docker) and a container orchestrator (like Kubernetes).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f74cbcb832ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/dynamic-vs-static-linking-or-how-the-software-development-changed-in-the-last-10-years-f74cbcb832ee">Dynamic vs Static linking, or how the software development changed in the last 10 years</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing minikube on Windows 10]]></title>
            <link>https://medium.com/kiratech/installing-minikube-on-windows-10-c4905c99b1a?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/c4905c99b1a</guid>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[powershell]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[minikube]]></category>
            <dc:creator><![CDATA[Francesco Furnari]]></dc:creator>
            <pubDate>Tue, 27 Jul 2021 15:35:41 GMT</pubDate>
            <atom:updated>2021-07-27T15:35:41.382Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Installing Minikube on Windows 10</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fMjeHmaGDI5UIzzvyDuUoQ.png" /><figcaption>Minikube’s logo</figcaption></figure><p>The aim of this guide is to teach the reader how to <strong>configure your Windows 10 setup and install Minikube on it</strong>, which in our experience proved to be a tricky process. Following this post will result in you having a fully working Minikube setup on your Windows device!</p><h3>Requirements</h3><ul><li>A <strong>local administrator account </strong>on your computer., which is needed for Hyper-V</li><li><strong>Hyper-V</strong> must be enabled on your system. You can check whether it is (and enable it, if needed) by following <a href="https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v">this guide</a>. We suggest running this <a href="https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v#enable-the-hyper-v-role-through-settings">step</a>.</li></ul><p>This is <strong>mandatory</strong>: otherwise, Minikube won’t work since Hyper-V requires an Administrator account to work.</p><h3>Setup</h3><p>Installing Minikube itself is pretty simple: just download and execute Minikube’s <a href="https://minikube.sigs.k8s.io/docs/start/#stand-alone-windows-installer">Stand-alone Windows Installer</a>.</p><p>Beware of the requirements! As of now, the following are needed:</p><ul><li>2 CPUs or more</li><li>2GB of free memory</li><li>20GB of free disk space</li></ul><h3>Starting Minikube</h3><p>Run <strong>Powershell as Administrator </strong>using your local account. You can perform this step by pressing the <strong>Windows+X</strong> keys and choosing the <em>Windows Powershell (Admin) </em>option.</p><p>In the Powershell terminal type the following command:</p><pre>minikube start --kubernetes-version=v1.18.6 --vm-driver=hyperv</pre><p>This will download all the required files and provision the Minikube environment with Kubernetes v1.18.6.</p><p>Once everything is done you can launch this command to make sure that you have also kubectl running:</p><pre>kubectl get pods -A</pre><p>In case you’re missing the <strong>kubectl </strong>executable, Minikube can download the appropriate version of it, launching this command:</p><pre>minikube kubectl -- get pods -A</pre><p>You can also check the status of Minikube by using the following command.</p><pre>minikube status</pre><p>After this first use you can simply launch <em>kubectl </em>with the command:</p><pre>kubectl</pre><p>And use all of its functionalities (<em>get</em>, <em>create</em>, <em>apply</em>, <em>delete</em>, …)</p><h3>Useful Minikube commands</h3><p>You can <strong>start a cluster</strong> by running:</p><pre>minikube start</pre><p>You can also <strong>access the Kubernetes dashboard</strong> running the following within the minikube cluster:</p><pre>minikube dashboard</pre><p>You can<strong> upgrade your cluster </strong>by typing:</p><pre>minikube start --kubernetes-version=latest</pre><p>And you can <strong>stop your local cluster</strong>:</p><pre>minikube stop</pre><p>In case you want to <strong>pause the kubernetes instance</strong>, you can use:</p><pre>minikube pause</pre><p>To<strong> delete your local cluster</strong>, type:</p><pre>minikube delete</pre><p>Lastly, to<strong> delete all local clusters and profiles </strong>write:</p><pre>minikube delete --all</pre><p>Hopefully, this tutorial has saved you some stressful time. If you have any suggestion, improvement or question, feel free to leave a comment below. We will be glad to get back to you!</p><h4>References</h4><ul><li>Minikube’s official website: <a href="https://minikube.sigs.k8s.io/docs/start/">https://minikube.sigs.k8s.io/docs/start/</a></li><li>Tutorial for installation and setup: <a href="https://redhat-scholars.github.io/kubernetes-tutorial/kubernetes-tutorial/installation.html">https://redhat-scholars.github.io/kubernetes-tutorial/kubernetes-tutorial/installation.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4905c99b1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/installing-minikube-on-windows-10-c4905c99b1a">Installing minikube on Windows 10</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a local three-broker Apache Kafka cluster]]></title>
            <link>https://medium.com/kiratech/setting-up-a-local-three-broker-apache-kafka-cluster-b459b28a6a4f?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/b459b28a6a4f</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[zookeeper]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[francesco antonio migliorin]]></dc:creator>
            <pubDate>Tue, 20 Jul 2021 13:02:43 GMT</pubDate>
            <atom:updated>2021-07-20T13:02:43.555Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a three-broker Apache Kafka cluster on a single node</h3><p>In this post we will learn how to to set up an <a href="https://kafka.apache.org/">Apache Kafka</a> cluster with 3 brokers on a single machine for testing purposes. The procedure is customizable and can be used to run as many brokers as needed on a single node, which can be useful while designing software and infrastructure with fault-tolerance in mind on development environments with limited capacity.</p><p>Apache Kafka is an open-source distributed event streaming platform in real-time. Kafka stores key-value messages that come from sources called <em>producers</em>. Messages are indexed and stored with a timestamp and can be consumed by processes called <em>consumers</em>. Kafka stores messages into different <em>partitions</em>, each belonging to a specific <em>topic</em>. Partitions can be replicated for fault tolerance. Specifically, consumers read from specific partitions within a topic.</p><h3>Prerequisites</h3><ul><li>JDK 8 and above</li></ul><h3>Setup</h3><h4>Step 1: Download Kafka</h4><p>You can download a stable Kafka release from <a href="https://downloads.apache.org/kafka/">the official website</a>. In this tutorial we are going to use kafka_2.12-2.6.2.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c38d4894c74e543168747c2f23c8be68/href">https://medium.com/media/c38d4894c74e543168747c2f23c8be68/href</a></iframe><h4>Step 2: Creating three identical folders</h4><p>Let’s rename the extracted folder and copy it twice, in order to get three identical Kafka folders, one per broker.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8eba6ad1ca4d062d700ad88ddf015db/href">https://medium.com/media/a8eba6ad1ca4d062d700ad88ddf015db/href</a></iframe><h4>Step 3: Starting a Zookeeper server</h4><p>Start a Zookeeper server. Only one Zookeeper is needed, since it can coordinate many Kafka brokers and consumers. In this example, we will run the one located in the kafka_1 folder by means of kafka_1/bin/zookeeper-server-start.sh kafka_1/config/zookeeper.properties</p><p>If zookeeper is running correctly you will see an output similar to the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce233f389b6ae51f85c0fc6127463e43/href">https://medium.com/media/ce233f389b6ae51f85c0fc6127463e43/href</a></iframe><p>This output shows that Zookeeper is up and running, listening on port 2181. Beware: <strong>do not kill this process!</strong></p><h4>Step 4: configuring the first Kafka broker</h4><p>Leave your Zookeeper terminal untouched and open a new terminal window in the same Virtual Machine. We will now conifgure the first Kafka broker by editing the kafka_1/server.properties file with your text editor of choice. In our case, using vi: vi kafka_1/server.properties:</p><ul><li>Set broker.id to 1. This attribute is the broker&#39;s unique identifier. It can be any number, but it must be unique in your cluster.</li><li>Uncomment the listeners=PLAINTEXT://:9092 row and change the port to 9091. This means the broker will be listening on port 9091 for connection requests.</li><li>Edit the log.dirs configuration. As a matter of fact, the log path can&#39;t be the same among different brokers. We suggest using the following configuration: log.dirs=/tmp/kafka_1-logs.</li><li>Save and exit</li></ul><h4>Step 5: running the first Kafka broker</h4><p>We can now start the kafka_1 broker with kafka_1/bin/kafka-server-start.sh kafka_1/config/server.properties.</p><p>If it is running correctly, you will see an output like the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/070912819a05c60129e167babbd6a403/href">https://medium.com/media/070912819a05c60129e167babbd6a403/href</a></iframe><p>Leave this shell open too, and open two other terminal windows, one for kafka_2 and one for kafka_3</p><h4>Step 6: configuring the other two brokers</h4><p>It is now sufficient to repeat points 4 and 5 with kafka_2/server.properties and kafka_3/server.properties in the two terminals you just opened, applying the following changes:</p><ul><li>change broker.id to 2 and 3, respectively;</li><li>use listeners=PLAINTEXT://:9092 in kafka_2 and listeners=PLAINTEXT://:9093 in kafka_3;</li><li>change the log.dirs to /tmp/kafka_2-logs and /tmp/kafka_3-logs, respectively.</li></ul><p>Finally, run kafka_2/bin/kafka-server-start.sh kafka_2/config/server.properties in one terminal and kafka_3/bin/kafka-server-start.sh kafka_3/config/server.properties in the other.</p><h4>Step 7: enjoy!</h4><p>You now have a fully configured Kafka cluster with three brokers running on a single VM, which is wonderful for development purposes!</p><p>Hopefully, this tutorial has saved you quite some time. If you have any idea or question, feel free to leave a comment below — we will be delighted to try and help.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b459b28a6a4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/setting-up-a-local-three-broker-apache-kafka-cluster-b459b28a6a4f">Setting up a local three-broker Apache Kafka cluster</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ansible Tower — Global Workflow Report]]></title>
            <link>https://medium.com/kiratech/ansible-tower-global-workflow-report-5087c884f4ac?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/5087c884f4ac</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Luca Santopadre]]></dc:creator>
            <pubDate>Tue, 01 Jun 2021 15:43:12 GMT</pubDate>
            <atom:updated>2021-06-01T15:43:12.647Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ansible Tower — Global Workflow Report</h3><p><strong>Automation</strong> -both mechanical and computational- has become a hot topic over the last few years. Software systems and infrastructures are getting more complex than ever, making tools such as Infrastructure-as-Code (IaC) ones almost compulsory when working with non-trivial environments.</p><p>In this article, we will take a look at an interesting use-case that involves Red Hat’s <a href="https://www.ansible.com/products/automation-platform">Ansible Tower’s Automation Platform</a>, defined as “an enterprise automation platform for the entire IT organization, no matter where you are in your automation journey”.</p><blockquote><em>This is not an introduction article on </em><strong><em>Ansible Tower</em></strong><em>; we will herebyI assume that you have experience with the following topics:</em></blockquote><p>- Designing <a href="https://docs.ansible.com/ansible-tower/latest/html/userguide/workflows.html">Ansible Tower Workflows</a></p><p>- Using <a href="https://docs.ansible.com/ansible/latest/user_guide/collections_using.html">Ansible Collections</a></p><p>- Working with <a href="https://docs.ansible.com/ansible-tower/latest/html/towerapi/api_ref.html">Ansible Tower’s APIs</a></p><h3>The scenario</h3><p>The following use case arises from a tricky problem faced while using <strong>Ansible Tower Workflow</strong> in a challenging environment, which comprises:</p><ul><li>Up to <strong>15.000 target hosts</strong> in production with different operating systems and middlewares</li><li><strong>Security healthchecks</strong> run in Ansible on every host</li><li><strong>Multiple teams</strong> involved in security analisys of reports generated by ansible playbooks</li><li>A complex <strong>Ansible Tower Workflow</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ljoE_FwFZZSUqbHj8ZH2xA.png" /><figcaption><em>An example of our customer’s workflow</em></figcaption></figure><p>One of the main goals for our customer, in this scenario, is to <em>keep track of the status of each single host during all Workflow executions</em>. Unfortunately, Ansible Tower platform doesn’t offer an out-of-the-box solution to manage this problem.</p><p>So far, our customer had never had a global view for the Ansible Tower Workflow execution status, which made troubleshooting tough and time-consuming. In particular, to check a single host’s status, we needed to <strong>click on each Workflow Node</strong>, scroll down to the end of the log, and view the PLAY RECAP written in it! 😞</p><p>As you can imagine, working with <strong>thousands of host</strong> and Workflows composed by <strong>dozens of Worflow Nodes</strong>, the probability of something failing is not negligible, which makes an easy troubleshooting a key requirement for quick fixes and improvements.</p><p>We therefore asked ourselves: can we create some reports showing the status of the Workflow for each host at a glance?</p><h3>The solution: Ansible Tower Workflow Report</h3><p>The solution we suggest is based on an Ansible role which achieves the following:</p><ul><li>get the job execution status in a Workflow for every node composing the Workflow</li><li>manipulate the data, in order to get information grouped per host</li><li>generate a .csv report with this information</li><li>optionally, send this report by email</li></ul><blockquote><em>Note: all the code shown in this post can be found </em><a href="https://github.com/LucaSantopadre/ansible_tower_workflow_report"><em>on this GitHub repository</em></a><em>.</em></blockquote><h3>Prerequisites</h3><p>We will need:</p><p>The tasks for our ansible_tower_workflow_report role are divided into 3 task groups:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39f7156724a1a73f53bf4a53c3d00542/href">https://medium.com/media/39f7156724a1a73f53bf4a53c3d00542/href</a></iframe><h4>Task #1: Get workflow jobs</h4><p>This file includes all the tasks necessary to get the information needed to generate a report</p><p>1. retrieve a list of <strong>Workflow Nodes Jobs IDs</strong>, calling the <em>Ansible Tower API</em> using <em>*lookup*</em> module with <a href="https://docs.ansible.com/ansible/latest/collections/awx/awx/tower_api_lookup.html">ansible.tower.tower_api</a>().</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9e4dd20466352fc5c824f0ec663d22b/href">https://medium.com/media/d9e4dd20466352fc5c824f0ec663d22b/href</a></iframe><p>2. Iterate over this list to get the <strong>Jobs IDs</strong> related to the Workflow Nodes Jobs ID.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05ef862e35bcbcfeb9b0ecd6fb9d09f4/href">https://medium.com/media/05ef862e35bcbcfeb9b0ecd6fb9d09f4/href</a></iframe><p>3. Query <strong>Job Host Summaries</strong> and create a list of dictionaries with info for each host during Job execution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20536b8fccef3e8f51f2664d71a66f02/href">https://medium.com/media/20536b8fccef3e8f51f2664d71a66f02/href</a></iframe><p>4. In conclusion, <em>the</em> 🎩 <em>trick</em>, <strong>group by host_name</strong>, to get all the information grouped for each host.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f69a27d18f41fa3c42d22c1776441416/href">https://medium.com/media/f69a27d18f41fa3c42d22c1776441416/href</a></iframe><p>Take a moment to understand host_summaries, a data structure composed by a list of lists where:</p><ul><li>list[*].list[0] contains the host_name</li><li>list[*].list[1] is a list of dictionaries, <a href="https://docs.ansible.com/ansible-tower/latest/html/towerapi/api_ref.html#/Jobs/Jobs_jobs_job_host_summaries_list">job_host_summaries</a>, which contains all the information about all the Job Template executed in the Workflow for that host.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e711785e9284baba80519bbadb3b2c23/href">https://medium.com/media/e711785e9284baba80519bbadb3b2c23/href</a></iframe><h4>Task #2: Generating a report</h4><p>To generate .csv report we use a <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">Jinja2 template</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9749ce8ac54c1d3a958899f0120bc594/href">https://medium.com/media/9749ce8ac54c1d3a958899f0120bc594/href</a></iframe><h3>Gluing the blocks</h3><p>To use this role you need an <strong>Ansible Playbook</strong> such as the following one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81cc773af7adb69b857e99e1dfbb5e79/href">https://medium.com/media/81cc773af7adb69b857e99e1dfbb5e79/href</a></iframe><blockquote><em>Make sure you run this playbook on localhost (i.e. the Ansible Tower Machine)</em></blockquote><p>Then set up <em>Ansible Tower Platform</em> :</p><ol><li>Create a <a href="https://docs.ansible.com/ansible-tower/latest/html/userguide/projects.html#add-a-new-project">Project</a> that point on your repository</li><li>Create a <a href="https://docs.ansible.com/ansible-tower/latest/html/userguide/job_templates.html#create-a-job-template">Job Template</a> referring to the workflow_report.yaml playbook</li><li>Add a <a href="https://docs.ansible.com/ansible-tower/latest/html/userguide/job_templates.html#surveys">Survey</a> for this Job Template named workflow_id of type Number</li><li>Find one existing <em>Workflow Job ID</em> for which you want you get the report</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*QwNgLBZVoMSGlZTKQbea-w.png" /></figure><p>5. Finally <em>launch</em> the Job Template passing in Survey the <em>Workflow Job id</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*-HgzfGHg-1sfg8uRJdm3Kg.png" /></figure><h3>Final results</h3><p>Here is an example of the .csv report generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WU7OP2On2up5eFZPMb_Gw.png" /></figure><p>As you can see, we have obtained a complete status of the Workflow per host at a glance.</p><h3>Conclusions</h3><p>The complex scenario described here paves the road to many other topics regarding <em>Ansible Tower Workflow</em> management, which we will describe in future posts. For instance, it is possible to implement a log analysis infrastructure using <a href="https://github.com/dmsimard/ara">ARA</a>.</p><p>What do you think of this approach? Have you ever tried achieving a similar goal? Your feedback is very welcome!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5087c884f4ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/ansible-tower-global-workflow-report-5087c884f4ac">Ansible Tower — Global Workflow Report</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five Terraform tips for common problems]]></title>
            <link>https://medium.com/kiratech/five-terraform-tips-for-common-problems-7c888d09922f?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/7c888d09922f</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[francesco antonio migliorin]]></dc:creator>
            <pubDate>Wed, 19 May 2021 15:31:00 GMT</pubDate>
            <atom:updated>2021-05-19T15:31:00.536Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Terraform </strong>is a <strong>IaC tool </strong>which provides a common syntax to interact with multiple <strong>cloud providers</strong>. Most of the time, though, people face presumably <strong>easy problems </strong>that Terraform transforms into <strong>real nightmares</strong>. In this post, we will present <strong>workarounds </strong>or <strong>solutions </strong>to <strong>five common problems </strong>where novices usually ask for help.</p><blockquote><em>Note that Terraform’s current version is </em><em>0.15 -- hopefully, with future major releases, useful functionalities to face these problems will be introduced.</em></blockquote><h3>Prerequisites</h3><p>Before reading this article you need a basic understanding of Terraform and its syntax. If you never heard of Terraform before we suggest that you read: <a href="https://www.terraform.io/intro/index.html">Introduction — Terraform by HashiCorp</a></p><h3>Don’t abuse count!</h3><p>The instructions count and for_each are very similar but for_each has some advantages because it allows for accessing data structures by key rather than index. Maybe programmers like count because it is easier to write -- we are still trying to figure this out 😄</p><p>But why is count such a bad guy? Let&#39;s take a look at the following example.</p><p>Let’s create a very simple code that prints the favorite operating system of a person:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e706dca2f3d715df5bb70a142c595b7/href">https://medium.com/media/7e706dca2f3d715df5bb70a142c595b7/href</a></iframe><p>This results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bc9e7f41bceb1dd38a484eb19eb0d7c/href">https://medium.com/media/4bc9e7f41bceb1dd38a484eb19eb0d7c/href</a></iframe><p>But what happens if we remove Bob? Now, the local.people variable looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b9673695b41488e7ed5767fd1469b37/href">https://medium.com/media/8b9673695b41488e7ed5767fd1469b37/href</a></iframe><p>If we run terraform apply again, the result may surprise the oblivious Terraform user:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbfea3794b8ee7b9781fe72f0cf13923/href">https://medium.com/media/fbfea3794b8ee7b9781fe72f0cf13923/href</a></iframe><p>Terraform destroyed null_resource.hello[2]... which was Carlo! What about Bob? Why was Carlo recreated?</p><p>Is this a Terraform bug or is it our mistake? What happened? Let’s take a look at the terraform plan:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34394cd6f37eac4b7041c92fa4dea21b/href">https://medium.com/media/34394cd6f37eac4b7041c92fa4dea21b/href</a></iframe><p>Instead of just removing Bob and leaving the current resources untouched, Terraform destroyed Carlo and replaced Bob with Carlo. This is because the unique identifier of Bob and Carlo was their index in the local.people list rather than the name.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62c76350810b2220e1c88c98d2f92862/href">https://medium.com/media/62c76350810b2220e1c88c98d2f92862/href</a></iframe><p>The consequences of this behavior could have been dreadful if we had been working with VMs! Now, let us show you why you should have used for_each instead:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97d693d640132c9ea5bd88dd96564051/href">https://medium.com/media/97d693d640132c9ea5bd88dd96564051/href</a></iframe><blockquote><em>Each element in the iteration needs to have a unique key. This is the design principle behind the </em><em>for_each loop, hence we need to transform our list to a map. For doing this we use </em><em>{ for p in local.people : p.name =&gt; p }. Thanks to this syntax, we create a map where the keys are the names and the values are the objects of the list (i.e. name and favorite OS). Note that we could also map names to favorite OS&#39;s directly</em></blockquote><p>This results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca4682c474f3423cade3519ecbd6cf8e/href">https://medium.com/media/ca4682c474f3423cade3519ecbd6cf8e/href</a></iframe><p>Here you can see the first difference: the name, which is the unique key in your for_each loop, is printed in the log. For instance, null_resource.hello[2] -which was Carlo- is now printed as null_resource.hello[&quot;Carlo&quot;]. We can now remove Bob and see what happens. Now, local.people look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5809e3d5cbc24fe0fa79b958c05e42e8/href">https://medium.com/media/5809e3d5cbc24fe0fa79b958c05e42e8/href</a></iframe><p>This results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55e207951424f69f960dfaa07f1ff9f8/href">https://medium.com/media/55e207951424f69f960dfaa07f1ff9f8/href</a></iframe><p>This is great! Only Bob was deleted and Carlo was left untouched.</p><p>Let’s recap what we learned here. While looping, Terraform uses an unique identifier for each iteration. For count, the identifier is a number; for_each, instead, uses the key of the map.</p><h3>Recreate a resource each time an apply is run</h3><p>Sometimes we want to define a resource which runs every time we execute an apply. Since Terraform saves the result of every execution in its state, we need to bind this &quot;ephemeral&quot; resource to something that always changes. The simplest way is to use a trigger with the timestamp function, as you can see in the following code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c94b1f672d8fdec6a73a1b287bc7dcf/href">https://medium.com/media/1c94b1f672d8fdec6a73a1b287bc7dcf/href</a></iframe><p>Every time we run a terraform apply, the timestamp function will return a different value. Thanks to this behavior, Terraform will be forced to destroy the previous resource and it will recreate a new one.</p><h3>Save variables to be executed during a destroy</h3><p>Provisioners like local-exec and remote-exec give developers the possibility to perform some actions when destroying resources. Usually, these actions are meant for clean-up but developers often encounter an unsuspected error. Take a look at this code and try to find the error:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/533f61b846c7e63042753328b0ecaf8d/href">https://medium.com/media/533f61b846c7e63042753328b0ecaf8d/href</a></iframe><p>Let’s try to perform a terraform apply to figure the error out:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9c64a3adbeffb557f3e17f5bd10ad93/href">https://medium.com/media/e9c64a3adbeffb557f3e17f5bd10ad93/href</a></iframe><p>This error may scare you but, luckily, the solution is very simple. Keep in mind this golden rule: <em>“Every time you need a variable at destroy-time, you need to create a trigger!”</em>. Very simple, right? Let’s see how to fix the previous code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb978f9d6212c5b5da4cfab74de6080e/href">https://medium.com/media/cb978f9d6212c5b5da4cfab74de6080e/href</a></iframe><p>This results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f91a6e8aa74ac963725a20e1ff7c59c8/href">https://medium.com/media/f91a6e8aa74ac963725a20e1ff7c59c8/href</a></iframe><h3>Nested Loops</h3><p>If you are lucky while using nested loops, the first one is on a resource, and the second one is on a block, which means a dynamic block will do the trick. Unfortunately, sometimes nested loops over resources, modules or variables are needed. Let&#39;s imagine this situation: given a list of environments, a resource must be called in order to install, for each environment, the appropriate software.</p><p>A very simplified variable to simulate this situation may look as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/610177d20917bddb65ec0adbd144c358/href">https://medium.com/media/610177d20917bddb65ec0adbd144c358/href</a></iframe><blockquote><em>In this example, we have only two keys for each object, but try to imagine having many different variables related to many different tasks…</em></blockquote><p>With only one for_each the maximum result that we can obtain is something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3f4c5290a4efd290e18799ad9e9eaa3/href">https://medium.com/media/a3f4c5290a4efd290e18799ad9e9eaa3/href</a></iframe><p>In this example, we are not able to loop over the programs variable and, therefore the best we can do is getting one element from (or maybe joining) the list.</p><p>To achieve our goal, we need to loop over a supporting module. Let&#39;s create the folder ./modules/setup; your .tf files inside it will look like this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b24c0e5de1005917365f09ab163b1164/href">https://medium.com/media/b24c0e5de1005917365f09ab163b1164/href</a></iframe><blockquote><em>For the article, we wanted to use as less code as possible for brevity; in your works, follow the official guidelines: </em><a href="https://www.terraform.io/docs/language/modules/develop/index.html"><em>Creating Modules — Terraform by HashiCorp</em></a></blockquote><p>As you can see, in our module we accept one environment and a list of programs as inputs. <em>Inside</em> the module we perform one loop over the programs variable. Now, we can loop over our module to achieve our goal:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d4d488ec49d571c1ebdb2a521b19d84/href">https://medium.com/media/9d4d488ec49d571c1ebdb2a521b19d84/href</a></iframe><p>This results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5c41fa8f44bb4a8681ffab6ea11801d/href">https://medium.com/media/c5c41fa8f44bb4a8681ffab6ea11801d/href</a></iframe><p>In other words, you can nest modules in order to achieve what in traditional programming would be nested looping. This can be repeated for as many layers as desired.</p><h3>Be careful with external!</h3><p>Although Terraform’s developers discourage the usage of the external data-source, it has become very popular because it allows the user to catch the output of an external program, conceptually run as a subprocess.</p><p>The major problem with external is that some developers use it to perform an action <em>once</em>. Since external is a data-source rather than a resource, it will run on <em>every</em> Terraform plan or apply. Therefore, when you design an external script to be used with external, it needs to able to detect when no action needs to be taken, otherwise the defined action will be executed each time over and over again.</p><p>Let’s see a very simple example of this problem. Consider the following Terraform script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/237f90a8066e56bc6149bf124cd693af/href">https://medium.com/media/237f90a8066e56bc6149bf124cd693af/href</a></iframe><p>The script.sh will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2c00a3528f79b30ad94cd8e69fcb506/href">https://medium.com/media/f2c00a3528f79b30ad94cd8e69fcb506/href</a></iframe><p>If we try to run terraform plan you will probably notice that the demo.txt was created. You can now run terraform apply --auto-approve as many times as you want and you will see that a new id will appear every time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c888d09922f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/five-terraform-tips-for-common-problems-7c888d09922f">Five Terraform tips for common problems</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybrid Kubernetes cluster with EKS and Fargate]]></title>
            <link>https://medium.com/kiratech/hybrid-kubernetes-cluster-with-eks-and-fargate-d57ab0475b99?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/d57ab0475b99</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Andrea Gallicchio]]></dc:creator>
            <pubDate>Tue, 11 May 2021 08:22:00 GMT</pubDate>
            <atom:updated>2021-05-11T08:22:00.579Z</atom:updated>
            <content:encoded><![CDATA[<p>According to the results of the <a href="https://www.cncf.io/blog/2020/11/17/cloud-native-survey-2020-containers-in-production-jump-300-from-our-first-survey/">CNCF Cloud Native Survey 2020</a>:</p><ul><li>The use of containers in production has quadrupled in just 4 years, from 23% in 2016 to 92% in 2020</li><li>Kubernetes use in production has increased to 83%</li><li>The use of Serverless technologies in production has increased to 30% of the respondents</li></ul><p>For this reason, in this blog post we are going to learn how to deploy an Hybrid Kubernetes cluster on AWS.</p><h3>Overview</h3><p>First of all, let’s recall the key concepts needed for this post:</p><ul><li>Amazon Elastic Kubernetes Service (<strong>Amazon EKS</strong>) is a managed service that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own Kubernetes control plane.</li><li><strong>AWS Fargate</strong> is a serverless compute engine that provides on-demand, right-sized compute capacity for containers and through you no longer have to think about scaling, sizing and cluster optimization.</li></ul><p>On AWS you can create an EKS cluster that has some worker nodes that are normal EC2 instances and some other worker nodes that are Fargate services.</p><p>In this way <strong>it is possible to manage</strong> both “ <strong>traditional</strong>” containerized applications and <strong>serverless</strong> containerized applications <strong>through the same EKS cluster</strong>.</p><p>This separation of serverless and non-serverless worker nodes is achieved by using different namespaces:</p><ul><li>Your <strong>serverless</strong> workloads will use the Kubernetes namespace defined in a <strong>Fargate</strong> profile, which by default specifies that all resources created in a given namespace will be deployed to the Fargate scheduler.</li><li>If your pod specification defines a <strong>different namespace</strong> than the one specified in a Fargate profile, EKS API will schedule the pod to be run on a non-serverless worker (i.e. <strong>EC2 instance</strong>).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9gFeizR66V8ZS4XimA19lw.png" /></figure><h3>Time for a Demo!</h3><p>First of all it is necessary to get your system ready:</p><ol><li>Make sure that you have an AWS account</li><li>Download and install the <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html">AWS CLI</a></li><li>Configure it with your AWS credentials: AWS Access Key &amp; AWS Secret Access Key</li><li>Install eksctl - a command-line utility that helps us to do any action to interact with the EKS service and simplifies a lot the work to do - as described here: <a href="https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html">AWS docs</a></li><li>Install kubectl - the command line tool that allow us to manage your Kubernetes Cluster - as described here: <a href="https://kubernetes.io/docs/tasks/tools/">Kubernetes docs</a></li></ol><h3>Create the Hybrid Kubernetes Cluster with eksctl</h3><p>To create this hybrid environment, we will start with the following YAML, which will create an EKS cluster with an EC2 instance and a Fargate Profile by means of eksctl. This will ensure that all the pods that will be created in the serverless namespace will run in Serverless mode through Fargate, while all the pods created outside this namespace will run in traditional mode, i.e. on the EC2 instance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d6f9d72d3bd36cb932912d2631cdb2a/href">https://medium.com/media/0d6f9d72d3bd36cb932912d2631cdb2a/href</a></iframe><p>The process of creating a cluster and a Fargate profile may take several minutes. When finished, you should see a similar output confirming that our cluster is ready.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ef13f0ecc198a631d440f366267444f/href">https://medium.com/media/4ef13f0ecc198a631d440f366267444f/href</a></iframe><blockquote><em>One important thing we can notice by looking at the logs is that </em><em>eksctl is using CloudFormation under the hood.</em></blockquote><p>At this point, running kubectl get nodes you can see that there is only one compute node (i.e. the m5.large EC2 instance created via the YAML) and running kubectl get pod --all-namespaces you can see that there are some EKS default pods that are running on this node:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b53e1cbeb039c060cc4fa43a6c5fd5d4/href">https://medium.com/media/b53e1cbeb039c060cc4fa43a6c5fd5d4/href</a></iframe><p>We can check the same cluster using the AWS EKS page to see that configuration more in detail. If we access the EKS page for this cluster we can see in the Compute tab the following information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G3Y60DRpwk-ogLkXpOsBxw.png" /></figure><p>Under Node Groups we can see information related to the EC2 instances that are managed as part of this cluster; as you can see, the Desired Capacity is 1, which is why we have 1 EC2 instance in our cluster. With respect to the Fargate profile, we can see the namespaces set to serverless, which means that all the deployments to those namespaces are going to be deployed using Fargate Tasks.</p><h3>Run the first serverless application</h3><p>Create the serverless namespace, which is the one related to the Fargate Profile and create a pod inside it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de16a1d45a3c7e96767b2b914662338f/href">https://medium.com/media/de16a1d45a3c7e96767b2b914662338f/href</a></iframe><p>The amazing thing is that having created a pod in the namespace related to the Fargate Profile — serverless - Fargate has created a new node on demand, inside which the nginx pod is running -- in serverless mode!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1377223c79a850dc44a2fba09e14d963/href">https://medium.com/media/1377223c79a850dc44a2fba09e14d963/href</a></iframe><blockquote><em>Note that for the provisioning of the new node, Fargate takes about 1 minute.</em></blockquote><h3>Pod CPU and memory</h3><p>Kubernetes allows you to define requests, a minimum amount of vCPUs, and memory resources allocated to each container in a pod. Pods are scheduled by Kubernetes to ensure that at least the resources required for each pod are available on the compute resource.</p><p>Through Fargate, it is also possible to specify the amount of CPU and Memory required for the creation of the pod, but it is necessary to choose one of the possible combinations listed in this table: <a href="https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html">AWS Doc</a>.</p><p>If no vCPU and memory combination is specified, then the smallest available combination is used (.25 vCPU and 0.5 GB memory).</p><p>You can verify the nginx pod&#39;s capacity with the following command kubectl describe pod nginx -n serverless:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa12450fc3286985348f633ff2b4decc/href">https://medium.com/media/aa12450fc3286985348f633ff2b4decc/href</a></iframe><p>The CapacityProvisioned annotation represents the enforced pod capacity and it determines the cost of your pod running on Fargate.</p><p>Note that there is no correlation between the size of the pod running on Fargate and the node size reported by Kubernetes with kubectl get nodes. <strong>The reported node size is often larger than the pod capacity</strong>.</p><p>You can check this by running:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b664cd853eccba69732fd9a2fc7bd2d/href">https://medium.com/media/4b664cd853eccba69732fd9a2fc7bd2d/href</a></iframe><h3>Why you should use this Hybrid Kubernetes Environment?</h3><ul><li>You want to leverage to the Kubernetes ecosystem</li><li>It is possible to run containerized applications both on EC2 instances, if the application itself is better suited to this type of approach (e.g. it is necessary to run it 24 hours a day), and on a serverless infrastructure if it is a on-demand task or a schedule (e.g. cron job)</li><li>Operations no longer need to be over-provisioned as Fargate users only pay for the amount of resources the pod needs to use at any given time and for the time it runs</li><li>You want better isolation, as each pod scheduled on Fargate node does not share the underlying kernel, CPU resources, memory resources, or Elastic Network Interface with another pod</li></ul><p>In conclusion, Amazon EKS and AWS Fargate provide the serverless benefits of Fargate, the best practices of Amazon EKS, and the extensibility of Kubernetes out of the box.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d57ab0475b99" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/hybrid-kubernetes-cluster-with-eks-and-fargate-d57ab0475b99">Hybrid Kubernetes cluster with EKS and Fargate</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python for the XXI century — an overview]]></title>
            <link>https://medium.com/kiratech/python-for-the-xxi-century-an-overview-a59329d0af00?source=rss----6e4f7ba0a355---4</link>
            <guid isPermaLink="false">https://medium.com/p/a59329d0af00</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Lorenzo Soligo]]></dc:creator>
            <pubDate>Mon, 12 Apr 2021 08:55:10 GMT</pubDate>
            <atom:updated>2021-04-14T09:19:13.146Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python for the XXI century — an overview</h3><p>During the last few years, Python has seen its user base <a href="https://www.zibtek.com/blog/the-incredible-growth-of-python/">grow significantly</a>. <a href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved">Developers like Python</a> and even non-techies enjoy its ease of use and versatility. As a matter of fact, <strong>Python works great for a variety of tasks</strong>, including <a href="https://automatetheboringstuff.com/">automating stuff</a>, <a href="https://pytorch.org/">deep learning</a>, and generic “bash-with-decent-syntax” scripting. Unsurprisingly, though, Python has some flaws that people criticize daily. In particular, the <a href="https://en.wikipedia.org/wiki/Duck_typing">duck typing</a> philosophy may come at a high cost in technical debt, especially in fast-growing codebases where code review is not performed systematically. Furthermore, being it an interpreted language, some errors may only arise at the worst possible moment at runtime rather than during a compilation step. In this blog post, we will learn how to use some<strong> tools and functionalities</strong> in order to <strong>enforce higher quality standards in Python codebases</strong>.</p><h3><strong>Typing and MyPy</strong></h3><p>Introduced in Python 3.5, the <a href="https://docs.python.org/3/library/typing.html">typing</a> library introduces support for <strong>type hints</strong>. Although these checks are not enforced in any way at runtime, consistency in <strong>type usage can be verified statically</strong> using <a href="http://mypy-lang.org/">MyPy</a>. For instance, the execution of MyPy can be set in a CI script so that a CI/CD pipeline will not pass unless types are correctly defined.</p><p>For instance, let’s consider the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/220cb391cd9e4eeda92cf467e78e44b9/href">https://medium.com/media/220cb391cd9e4eeda92cf467e78e44b9/href</a></iframe><p>A random execution of the script above would be fine 99% of the time, yet it would fail in case x == 42. Hard to spot for the oblivious developer, the bug will relentlessly be found by MyPy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e62d325f70278d8f231448e50b5dda5b/href">https://medium.com/media/e62d325f70278d8f231448e50b5dda5b/href</a></iframe><h3>Dataclasses</h3><p>Introduced in Python 3.7, <a href="https://docs.python.org/3/library/dataclasses.html">dataclasses</a> provide the user with the capability to <strong>automatically generate special methods in classes</strong>. <a href="https://www.python.org/dev/peps/pep-0557/">PEP 557</a> intuitively describes dataclasses as <em>mutable namedtuples with defaults</em>.</p><p>Think about the highly-renowned class Point, describing a point in the X-Y Cartesian plane. In &quot;classic&quot; Python, we would have to write <em>a lot</em> of boilerplate code: an __init__ constructor, a __repr__, possibly an __eq__, etcetera. Dataclasses free the developer from the burden of defining such methods.</p><p>As a dataclass, Point would look as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5715f15688a426c7446b30a3eb04fee5/href">https://medium.com/media/5715f15688a426c7446b30a3eb04fee5/href</a></iframe><p>In case Point objects were immutable, we could also define the dataclass as frozen to also get a free __hash__. Futhermore, dataclasses can be used in conjunction with typing and MyPy in order to enrich as much code as possible with meaningful data types:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18448cd5ea62f6d2eadd875024944f0f/href">https://medium.com/media/18448cd5ea62f6d2eadd875024944f0f/href</a></iframe><p>We strongly suggest using dataclasses as they are easy and quick to define and can lead to a much more comprehensible and maintainable codebase. Even for small scripts, representing objects or namedtuples using dataclasses allows for more standardization and better static checks before execution. Generally speaking, every non-trivial set of fields should be represented with a <strong>meaningful abstraction</strong>, and dataclasses are great for achieving the goal.</p><h3>Flake8</h3><p><a href="https://flake8.pycqa.org/en/latest/manpage.html">Flake8</a> is a toolkit that wraps many other tools under the hood. It is great for checking whether the code is PEP8-compliant, and perform a <strong>basic static analysis of the codebase against common mistakes or bad practices</strong>. Again, a flake8 step in a CI script is a great idea to enforce code style and a basic static analysis on the codebase.</p><p>For instance, star imports are immediately signaled to the user:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ae1ec60847b75101e1acb1c24713152/href">https://medium.com/media/9ae1ec60847b75101e1acb1c24713152/href</a></iframe><p>And the same holds for undefined variables and wrong numbers of newlines:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/296408fb64ff462407f4d87634df06d8/href">https://medium.com/media/296408fb64ff462407f4d87634df06d8/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40b82e0b346269ea9904eb843ef00ac4/href">https://medium.com/media/40b82e0b346269ea9904eb843ef00ac4/href</a></iframe><h3>Bandit</h3><p>According to its creators, “<a href="https://bandit.readthedocs.io/en/latest/">Bandit</a> is an extremely useful<strong> tool designed to find common security issues in Python code</strong>” via an analysis performed on an AST it creates starting from the codebase. Bandit looks for many different issues by means of <a href="https://bandit.readthedocs.io/en/latest/plugins/#complete-test-plugin-listing">numerous plugins</a>. It looks for the usage of “dangerous” functions such as assert or exec, hardcoded passwords, unsafe loads from files (e.g. YAML) and much much more.</p><p>In the following example, an unsafe YAML load and the usage of exec both raise an alert:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bee4b1f21fddebf9f8111bf7619ea5a4/href">https://medium.com/media/bee4b1f21fddebf9f8111bf7619ea5a4/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1b1bafc09c5751d549fb1297d825e62/href">https://medium.com/media/e1b1bafc09c5751d549fb1297d825e62/href</a></iframe><p>Although Bandit’s tests are far from perfect, they provide a great starting point for <strong>checking for common problems which are easy to fix but may still lead to serious consequences if ignored</strong>. For this reason, we strongly suggest implementing it as a step in your CI pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a59329d0af00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kiratech/python-for-the-xxi-century-an-overview-a59329d0af00">Python for the XXI century — an overview</a> was originally published in <a href="https://medium.com/kiratech">kiratech.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>