<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daisuke Kikuchi on Medium]]></title>
        <description><![CDATA[Stories by Daisuke Kikuchi on Medium]]></description>
        <link>https://medium.com/@kikuchidaisuke.zr?source=rss-12ae18480ae6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*FFavBSaeckfOgOJ-</url>
            <title>Stories by Daisuke Kikuchi on Medium</title>
            <link>https://medium.com/@kikuchidaisuke.zr?source=rss-12ae18480ae6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:21:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kikuchidaisuke.zr/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Secure application.properties with HashiCorp Vault]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/secure-application-properties-with-hashicorp-vault-a07fb39b25f2?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a07fb39b25f2</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sat, 05 Jul 2025 15:42:59 GMT</pubDate>
            <atom:updated>2025-07-05T15:42:59.880Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RagwSKlX0Jm0S8gsIoo8CA.png" /></figure><p>It is one of the common architectural sets to deploy Java application on Kubernetes, and often times you want to inject some credentials to these pods such as database connection, API Key and so on.</p><p>In this article, I am going to show you one of the best practices to secure these credentials using HashiCorp Vault.</p><h3>What is wrong with Kubernetes Secret?</h3><p>Apart from HashiCorp Vault, Kubernetes itself owns secret management object called “Secret”.</p><p>You can create a new secret just like this.</p><pre>kubectl create secret generic my-secret \<br>  --from-literal=username=my-user</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vDfu9XHzdhxFnqXzDTU6DQ.png" /></figure><p>It looks like encrypted but actually it is just base64 obfuscation, and anyone who has an access to this secret objects can decode actual content.</p><p>You can try the following command to decode your secret.</p><pre>k get secret my-secret -o jsonpath=&quot;{.data.username}&quot; | base64 -d</pre><h3>HashiCorp Vault</h3><p>HashiCorp Vault is an opensource tool to give you centralized management with encryption. It can be easily integrated with major platforms such as AWS, Kubernetes and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8y7yBYEn2kRpb5rkie3kBQ.png" /></figure><h3>Install HachiCorp Vault in Kubernetes</h3><p>You can easily install Vault on your cluster with Helm Chart.</p><p>Here is a set of commands to do so.</p><pre># Install Helm<br>curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3<br>chmod 700 get_helm.sh<br>./get_helm.sh<br><br># Install Vault<br>helm repo add hashicorp https://helm.releases.hashicorp.com<br>helm repo update<br>helm install vault hashicorp/vault  -n vault --create-namespace</pre><h3>Initialize your Vault</h3><p>When you install Vault with Helm, it is not yet ready to use. You need to do some initialization tasks, which will be some executions of the commands in Vault Pod.</p><p>First of all, let’s get inside of Vault Pod. Here is the command.</p><pre>kubectl exec -it vault-0 -n vault -- /bin/sh</pre><p>Then, run the following command in the pod.</p><pre>vault operator init</pre><p>It gives you something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Ex8nqU65USugrhA.png" /><figcaption><a href="https://scriptcrunch.com/hashicorp-vault-setup/">https://scriptcrunch.com/hashicorp-vault-setup/</a></figcaption></figure><h4>Unseal</h4><p>Unseal is the process of “giving root key to Vault” and allow it to function as Data Encryption Management tools. Here are some terms you should know to quickly understand what is going on.</p><ul><li>Encryption Key: the key to encrypt your data</li><li>Root Key: they key to encrypt your Encryption Key</li></ul><p>By collecting 3/5 (default) Unseal keys displayed here will be used to recover your root key when Vault is restarted for a few occasion.</p><p>So long story short, you pass 3 unseal keys to Vault, and Vault is able to encrypt/decrypt your data. <br>Usually it is useful when you don’t want to give a single person a full control. You can balance our your responsibility to protect your data,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CcBkSkdZIgDaH5Zhg5qxKw.png" /></figure><p>You can pass an unseal key to Vault by the following command. You do this 3 times with 3 different keys.</p><pre>vault operator unseal</pre><p>Once you’ve done that, you can check the status and confirm it is “unsealed”.</p><pre>vault status</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*jZXjSxoBMWqOOXvNFTqmNQ.png" /></figure><h4>Add Kubernetes Plugin</h4><p>Now, to manage secrets with Kubernetes Integration, you need to enable a plugin. This allows you to authentication/authorize Service Accout in Kubernetes easily with Vault.</p><pre># Enable Kubernetes Auth<br>vault auth enable kubernetes</pre><h4>Add KV Plugin</h4><p>This is the simplest plugin to save your secret in Key-Value format. This time we are going to store any value you want to write in application.properties here.</p><pre># Enable KV Store<br>vault secrets enable -path=kv2 kv-v2</pre><h3>Create a service account</h3><p>Let’s go back to your node terminal from Vault Pod terminal. I want you to create 1 service account to be authenticated/authorized by Vault.</p><pre># Write a yaml file for Service Account<br>cat &lt;&lt;EOF &gt; ./hoge-sa.yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: vault-serviceaccount<br>  namespace: vault<br>EOF<br><br># Create a service account<br>kubectl apply -f hoge-sa.yaml</pre><p>Please note that as HashiCorp Vault, it doesn’t require any Role or ClusterRole to be attached. What this SA can do in HashiCorp will be defined in HashiCorp itself.</p><h3>Define a secret</h3><p>Now let’s go back to The pod of Vault. This time, I will define one simple secret called “password”, but it is up to you add more or change the values.</p><pre>vault kv put kv2/hoge password=SUPER_SECRET </pre><h3>Define a policy</h3><p>In Vault, we have a concept called Policy, which is really similar to Cluster Role or Role in Kubernetes where you define what SA can do.</p><p>And then, it is going to be assembled and attached to SA using Role (a Vault concept). So relation looks like this.</p><ul><li><strong>Policy in Vault</strong>: kind of Role, ClusterRole in Kubernetes</li><li><strong>Role in Vault</strong>: kind of RoleBinding, ClusterRoleBinding in Kubernetes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9pZQw_tTmD_vjmeodTEx2A.png" /></figure><p>※Config is a credential for Vault to access Kube API Server. I’ll explain it in a minute.</p><p>And here is how you create a Policy. We are giving read permission to this policy.</p><pre># Write a HCL for policy<br>cat &lt;&lt;EOF &gt; /home/vault/hoge-policy.hcl<br>path &quot;kv2/data/hoge&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>EOF<br><br># Generate a policy<br>vault policy write hoge-policy /home/vault/hoge-policy.hcl</pre><p>Please note that I am specifying the path “kv2/data/hoge” instead of “kv2/hoge”. This is because KV plugin stores the data in /data/ path from version 2. I will not dive into it in this article but you can check it from here.</p><p><a href="https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2">Key/Value v2 | Vault | HashiCorp Developer</a></p><h3>Define a role</h3><p>As I said, role is a connection between Service Account and Policies.</p><p>This time, we name it “vault-role” and attach “hoge-policy” to it.</p><pre># Create a role<br>vault write auth/kubernetes/role/vault-role \<br>   bound_service_account_names=vault-serviceaccount \<br>   bound_service_account_namespaces=vault \<br>   policies=hoge-policy \<br>   ttl=1h</pre><h3>Define a config</h3><p>This is an easy one. To check a service account and namespaces etc.., Vault needs an access to Kubernetes API Server. The following command should do the job.</p><pre>vault write auth/kubernetes/config \<br>   token_reviewer_jwt=&quot;$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; \<br>   kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \<br>   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt</pre><h3>Create a Pod</h3><p>I created a simple docker image to display the value of password, reading from /vaults/secrets/applications.properties.</p><p>So we are going to use this to test if pod is able to read secret from Vault.<br>If you are curious, you can check the source code in this link.</p><p><a href="https://github.com/Daisuke-lab/vault_tutorial/tree/main/demo">vault_tutorial/demo at main · Daisuke-lab/vault_tutorial</a></p><p>Here is my pod definition.</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: java-pod<br>  namespace: vault<br>  annotations:<br>    vault.hashicorp.com/agent-inject: &quot;true&quot;<br>    vault.hashicorp.com/agent-inject-status: &quot;update&quot;<br>    vault.hashicorp.com/agent-inject-secret-login: &quot;kv2/data/hoge&quot;<br>    vault.hashicorp.com/agent-inject-template-login: |<br>          {{- with secret &quot;kv2/data/hoge&quot; -}}<br>          password={{ .Data.data.password }}<br>          {{- end }}<br>    vault.hashicorp.com/role: &quot;hoge-role&quot;<br>    vault.hashicorp.com/agent-inject-file-login: &quot;application.properties&quot;<br>spec:<br>  serviceAccountName: vault-serviceaccount<br>  containers:<br>  - name: java-app<br>    image: daisukezr/vault-demo-java:latest<br>    ports:<br>      - containerPort: 8080</pre><h4>vault.hashicorp.com/agent-inject-secret-login</h4><p>This is where you specify your secret path. Please note that suffix “-login” can be anything as long as consistent throughout the configuration.</p><h4>vault.hashicorp.com/agent-inject-template-login</h4><p>This is where you customize the output. By default the secret will be injected like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hg7eHr_w1R1jFjJvy9avfw.png" /></figure><p>You have a lot of metadata and unnecessary data and it is quite hard to retrieve what you want.</p><p>After templating, it is going to look like this. That’s why this is useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wk3wI2E9zv5EALXrR9ufiA.png" /></figure><h4>vault.hashicorp.com/role</h4><p>This is self-explanatory. it’s the role you want to use in Vault.</p><h4>vault.hashicorp.com/agent-inject-file-login</h4><p>By default, filename that is going to be mounted to store secret will be the suffix (in this case, “login”). But this parameter allows you to change the filename to anything. This time, we are naming “application.properties”.</p><h3>Run a Pod</h3><p>Name your pod yaml file as “java_pod.yaml” and run the following command.</p><pre>kubectl apply -f java_pod.yaml</pre><p>Once you run the pod, you can see something interesting by inspecting your yaml file again.</p><pre>kubectl get pod java-pod -o yaml</pre><p>You should be able to see Vault automatically adding 1 init container and 1 sidecar.</p><ul><li><strong>vault-agent-init</strong>: This is in charge of ensuring your Service Account and role is correct and retrieving secrets.</li><li><strong>vault-agent</strong>: This side cars monitor the change in Vault continusouly, hot-reload a new secret if needed.</li></ul><p>You should also note that Vault doesn’t your data in “Kubernetes Secret”. It is storing data in PersistentVolume with encryption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ro4wzzvr8iGszTVNKJNy5Q.png" /></figure><p>Now, let’s expose the pod and see the page from the browser.</p><pre>kubectl expose pod java-pod -n vault --port=8080 --type=NodePort</pre><p>Check your open port.</p><pre>k get svc java-pod -n vault</pre><p>Access from the browser. You can see the secret you register in Vault.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*WKjT0UHoqRvP_qaaEMcjww.png" /></figure><h3>Conclusion</h3><p>Once you set up Vault in your cluster, adding Role and ServiceAccount, it is quite easy to add a new secret.</p><p>This little effort makes your system super-secure, production-ready. I hope you learned some from this article. I will see you soon👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a07fb39b25f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Canary Release with Istio]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/canary-release-with-istio-2ee2cbfad803?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ee2cbfad803</guid>
            <category><![CDATA[canary-deployments]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 19:25:37 GMT</pubDate>
            <atom:updated>2025-07-03T21:33:52.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*oWsbqvixmEIrOgwMHxuP7A.png" /></figure><p>Lately, it is more common to have canary release rather than Rolling Update or Blue/Green Deployment to see user’s reaction.</p><p>However, it can be challenging without the right tool.</p><h3>Install Istio</h3><p>I wrote more detailed instruction in this article about how to install Istio in your cluster, so if you are curious, visit this article. In this section, I will just give you commands.</p><p><a href="https://medium.com/@kikuchidaisuke.zr/secure-your-kubernetes-with-service-mesh-b333736af1a5">Secure your Kubernetes with Service Mesh</a></p><pre># Install Helm<br>curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3<br>chmod 700 get_helm.sh<br>./get_helm.sh<br><br># Install Istio<br>helm repo add istio https://istio-release.storage.googleapis.com/charts<br>helm repo update<br>helm install istio-base istio/base -n istio-system --set defaultRevision=default --create-namespace<br>helm install istiod istio/istiod -n istio-system --wait<br><br># Enable injection<br>kubectl label namespace default istio-injection=enabled</pre><h3>Launch v1 and v2 pods</h3><p>To experiment the canary release, let’s run some random pods, one for version 1 and the other one for version 2.</p><h4>Nginx Pod (v1)</h4><p>To make it obvious, I am going to create nginx that shows a title with background color: red</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: nginx-red<br>  labels:<br>    name: red<br>    app: nginx<br>spec:<br>  volumes:<br>    - name: html-volume<br>      emptyDir: {}<br>  containers:<br>    - name: nginx-red<br>      image: nginx<br>      volumeMounts:<br>        - name: html-volume<br>          mountPath: /usr/share/nginx/html<br>      ports:<br>        - containerPort: 80<br>  initContainers:<br>    - name: init-html<br>      image: busybox<br>      command:<br>      - &quot;/bin/sh&quot;<br>      args:<br>        - -c<br>        - |<br>          echo &#39;&lt;h1 style=&quot;background-color: red;&quot;&gt;Hello from custom HTML&lt;/h1&gt;&#39; &gt; /temp/index.html<br>      volumeMounts:<br>        - name: html-volume<br>          mountPath: /temp</pre><h4>Nginx Pod (v2)</h4><p>For version 2, let’s use blue background color with the same configuration.</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: nginx-blue<br>  labels:<br>    name: blue<br>    app: nginx<br>spec:<br>  volumes:<br>    - name: html-volume<br>      emptyDir: {}<br>  containers:<br>    - name: nginx-blue<br>      image: nginx<br>      volumeMounts:<br>        - name: html-volume<br>          mountPath: /usr/share/nginx/html<br>      ports:<br>        - containerPort: 80<br>  initContainers:<br>    - name: init-html<br>      image: busybox<br>      command:<br>      - &quot;/bin/sh&quot;<br>      args:<br>        - -c<br>        - |<br>          echo &#39;&lt;h1 style=&quot;background-color: blue;&quot;&gt;Hello from custom HTML&lt;/h1&gt;&#39; &gt; /temp/index.html<br>      volumeMounts:<br>        - name: html-volume<br>          mountPath: /temp</pre><h4>Service for Nginx</h4><p>As a service level, you create a service that accesses “both pods”. I added app:nginx label to both pods, so let’s use this as a selector!</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: nginx<br>spec:<br>  type: NodePort<br>  selector:<br>    app: nginx<br>  ports:<br>  - port: 80<br>    targetPort: 80</pre><p>To apply these pods, please run the following commands.</p><pre>kubectl apply -f nginx_blue.yaml<br>kubectl apply -f nginx_red.yaml<br>kubectl apply -f nginx_service.yaml</pre><h3>Define VirtualService</h3><p>VirtualService is a CRD from Istio to filter, reroute, and transform your requests.</p><p>In this example, I am going to utilize two route techniques</p><ul><li>Reroute with http header</li><li>Reroute with the weight (Canary Release)</li></ul><pre>apiVersion: networking.istio.io/v1<br>kind: VirtualService<br>metadata:<br>  name: nginx-vs<br>spec:<br>  hosts:<br>  - nginx<br>  http:<br>    - route:<br>      - destination:<br>          host: nginx<br>          subset: v2<br>      match:<br>        - headers:<br>            x-user-type:<br>              exact: beta<br>      <br>    - route:<br>      - destination:<br>          host: nginx<br>          subset: v1<br>        weight: 75<br>      - destination:<br>          host: nginx<br>          subset: v2<br>        weight: 25</pre><p>Let me explain couple fields.</p><ul><li><strong><em>host</em></strong>: This is a service that you want to access. This time we use “nginx”</li><li><strong><em>match</em></strong>: This is where you can define which requests can be rerouted. This time, I defined x-user-type to just have fun later :) (Nothing to do with Canary Relase)</li><li><strong><em>route</em></strong>: This is where you can define how you route it. Most of the fields are intuitive, but let me talk about “subset” in the next section.</li></ul><p>To apply this virtual service, please run this command (name your file “virtual_service.yaml”).</p><pre>kubectl apply -f virtual_service.yaml</pre><h3>Define DestinationRule</h3><p>With DestinationRule, you can group your pods in a different way from “Service.”</p><p>If you remember, We set one extra label for pods, “color:red” and “color:blue”.</p><p>This can be used as a differentiator for pods, and relate it to version number as follows.</p><pre>apiVersion: networking.istio.io/v1<br>kind: DestinationRule<br>metadata:<br>  name: nginx-dr<br>spec:<br>  host: nginx<br>  trafficPolicy:<br>    loadBalancer:<br>      simple: LEAST_REQUEST<br>  subsets:<br>  - name: v1<br>    labels:<br>      color: red<br>  - name: v2<br>    labels:<br>      color: blue</pre><p>To apply this virtual service, please run this command (name your file “destination_rule.yaml”).</p><pre>kubectl apply -f destination_rule.yaml</pre><h3>(Optional) Install Chrome extension</h3><p>To customize your http header on the browser. It’s useful to use this tool.<br>In this specific case, we will see Istio reroute your requests based on headers.</p><p><a href="https://chromewebstore.google.com/detail/modheader-modify-http-hea/idgpnmonknjnojddfkpgkljpfnnfcklj">ModHeader - Modify HTTP headers - Chrome Web Store</a></p><h3>Test Canary Release</h3><p>First of all, let’s get the open port for this nginx pods.</p><pre>kubectl get svc nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0MD9zVF7oA9GZ1IryrCig.png" /></figure><p>In my case, it’s 32576.</p><p>Before accessing it from the browser, I want you to set connection:close as a custom header using the chrome extension so that it stops using pipelining in HTTP/1.1 and go to different pods every time you access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*6VnC61kKTemWXglJYCwKsg.png" /></figure><p>Now, let’s see it from the browser!<br>You should see either of the following 2 pages (red and blue), but more likely to see red pages because it has 75% weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zp9ODHwcCsI7sjG561l4Sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*OAyaKBBVVnAKu2KxtvXopA.png" /></figure><h3>(Optional) Reroute by header</h3><p>Let’s do more fun stuff with Istio. We also set one more routing based on header “x-user-type:beta”. With the same chrome extension, you can add this as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dlC4xRUQ8IaC2moqfA7SUA.png" /></figure><p>When you reload the page now, you should only see blue screen. This is extremely useful when you do some urgent testing in the production set up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*OAyaKBBVVnAKu2KxtvXopA.png" /></figure><h3>Conclusion</h3><p>Today, I mainly focused on VirtualService and DestinationRule CRD from Istio to control your traffic as you wish.</p><p>The source code can be founded below. So if you want to follow along, please check it out.</p><p><a href="https://github.com/Daisuke-lab/istio_tutorial">GitHub - Daisuke-lab/istio_tutorial</a></p><p>See you⭐</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ee2cbfad803" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collect Kubernetes logs with Fluent bit, Elasticsearch and Kibana]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/collect-kubernetes-logs-with-fluent-bit-elasticsearch-and-kibana-e7e97cc77928?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7e97cc77928</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[fluentbit]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 18:03:33 GMT</pubDate>
            <atom:updated>2025-06-20T18:03:33.316Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*VV9Lo5FxEDHZxUPIbxpxCQ.png" /></figure><p>Log management in Kubernetes is one of the biggest challenges, especially in Microservice architecture. <br>You might have different log formats for each service. You might also want to collect logs on the node level apart from pod logs.</p><p>To unify the whole process, there are several options you can take.</p><ul><li>Fluent Bit (Fluentd)</li><li>Loki</li><li>Logstash</li><li>Data Dog</li><li>New Relic</li><li>and more</li></ul><p>In this article, I am going to use</p><ul><li>Fluent bit to collect, filter and parse logs</li><li>Elasticsearch to store logs</li><li>Kibana to visualize logs</li></ul><p>It is often referred as EFK stack just like ELK, but F (Fluent bit) instead of L (Logstash).</p><h3>Install Helm</h3><p>There are several ways to install these tools, but I am going for the easiest way, Helm. Here is the command to install helm on your node.</p><pre>curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3<br>chmod 700 get_helm.sh<br>./get_helm.sh</pre><h3>Install Fluent bit</h3><p>Now let’s install fluent bit into your cluster. You can install it by just following the official website.</p><p><a href="https://docs.fluentbit.io/manual/installation/kubernetes#installing-with-helm-chart">Kubernetes | Fluent Bit: Official Manual</a></p><p>Here is a quick command to do so.</p><pre>helm repo add fluent https://fluent.github.io/helm-charts<br>helm upgrade --install fluent-bit fluent/fluent-bit</pre><p>In Kubernetes, fluent bit is a daemon set, so it is going to run on each node and collect logs from all the pods on the node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MyLgHMuD9Y0fDD-h1rLz2w.png" /></figure><p>You can also check the plugin configuration on fluent bit, exists as a configmap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*qklP5YQWb5eoGAdh2odgxg.png" /></figure><h3>Install Elasticsearch and Kibana</h3><h4>Install ECK</h4><p>ECK stands for Elastic Cloud on Kubernetes, which is CRD and some RBAC/Operator package to make the ease of installation for ELK stack.</p><blockquote>This is not the installation of Elasticsearch or Kibana yet.</blockquote><p>To install ECK, please run the following command.</p><pre>kubectl create -f https://download.elastic.co/downloads/eck/3.0.0/crds.yaml<br>kubectl apply -f https://download.elastic.co/downloads/eck/3.0.0/operator.yaml</pre><p><a href="https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/install-using-yaml-manifest-quickstart">Install ECK using the YAML manifests | Elastic Docs</a></p><p>You can see it creates some interesting CRD named Elasticsearch or Kibana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gIYte2udKKdy-oLiUAR6Ug.png" /></figure><h4>Install Elasticsearch and Kibana</h4><p>Install Elasticsearch</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: elasticsearch.k8s.elastic.co/v1<br>kind: Elasticsearch<br>metadata:<br>  name: quickstart<br>spec:<br>  version: 8.16.1<br>  nodeSets:<br>  - name: default<br>    count: 1<br>    config:<br>      node.store.allow_mmap: false<br>EOF</pre><p><a href="https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/elasticsearch-deployment-quickstart">Deploy an Elasticsearch cluster | Elastic Docs</a></p><p>Install Kibana</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: kibana.k8s.elastic.co/v1<br>kind: Kibana<br>metadata:<br>  name: quickstart<br>spec:<br>  version: 8.16.1<br>  count: 1<br>  elasticsearchRef:<br>    name: quickstart<br>EOF</pre><p><a href="https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/kibana-instance-quickstart">Deploy a Kibana instance | Elastic Docs</a></p><p>Make sure the status is green</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*coyrVYRWaUB_W3I0hQg6xA.png" /></figure><h3>Expose Kibana</h3><pre>kubectl expose svc quickstart-kb-http  --type=NodePort --target-port=5601 --name=kibana-ext</pre><p>get the exposed port by the following command.</p><pre>kubectl get svc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OENZqq8Z3FtOrmC2c2DOgg.png" /></figure><p>Now, let’s access <a href="https://localhost:32179/">https://localhost:&lt;YOUR_PORT&gt;/</a>.</p><p>You should be able to see the login page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*opv2HunjuZRftzFhLNxyPw.png" /></figure><p>So what is username and password?</p><ul><li>username: elastic</li><li>password: run the following command.</li></ul><pre>kubectl get secret quickstart-es-elastic-user -o=jsonpath=&#39;{.data.elastic}&#39; | base64 --decode; echo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*02q7ey_GVgKB0hYMONk8zg.png" /></figure><h3>Run random pods to send logs</h3><p>For the sake of ease, we are going to make 3 simple pods, nginx, apache and busybox (to run curl command inside).</p><pre>kubectl run nginx --image=nginx<br>kubectl run apache --image=httpd<br>kubectl run busybox --image=curlimages/curl -- sleep 10000<br></pre><p>and to create services, run the following commands.</p><pre>kubectl expose pod nginx --port=80<br>kubectl expose pod apache --port=80</pre><h3>Connect Fluent bit with Elasticsearch</h3><p>In Kubernetes, Fluent bit plugin configuration is defined as ConfigMap. Here is the entire configuration that we are going to assign.</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  annotations:<br>    meta.helm.sh/release-name: fluent-bit<br>    meta.helm.sh/release-namespace: default<br>  labels:<br>    app.kubernetes.io/instance: fluent-bit<br>    app.kubernetes.io/managed-by: Helm<br>    app.kubernetes.io/name: fluent-bit<br>    app.kubernetes.io/version: 4.0.3<br>    helm.sh/chart: fluent-bit-0.49.1<br>  name: fluent-bit<br>  namespace: default<br>data:<br>  fluent-bit.conf: |<br>    [SERVICE]<br>        Daemon Off<br>        Flush 1<br>        Log_Level info<br>        Parsers_File /fluent-bit/etc/conf/parsers.conf<br>        HTTP_Server On<br>        HTTP_Listen 0.0.0.0<br>        HTTP_Port 2020<br>        Hot_Reload On<br>        Health_Check On<br><br>    [INPUT]<br>        Name tail<br>        Path /var/log/containers/*.log<br>        multiline.parser docker, cri<br>        Tag kube.*<br>        Mem_Buf_Limit 5MB<br>        Skip_Long_Lines On<br><br>    [FILTER]<br>        Name kubernetes<br>        Match kube.*<br>        Merge_Log On<br>        Keep_Log Off<br>        K8S-Logging.Parser On<br>        K8S-Logging.Exclude On<br><br>    [FILTER]<br>        Name  rewrite_tag<br>        Match kube.*<br>        Rule  $kubernetes[&#39;container_name&#39;] ^nginx$ kube.parsed.nginx true<br>        Rule  $kubernetes[&#39;container_name&#39;] ^apache$ kube.parsed.apache true<br><br>    [FILTER]<br>        Name   grep<br>        Match  kube.*<br>        Regex  $kubernetes[&#39;container_name&#39;] (nginx|apache)<br><br>    [FILTER]<br>        Name parser<br>        Match kube.parsed.nginx<br>        Key_Name log<br>        Preserve_Key On<br>        Reserve_Data On<br>        Parser nginx<br><br>    [FILTER]<br>        Name parser<br>        Match kube.parsed.apache<br>        Key_Name log<br>        Preserve_Key On<br>        Reserve_Data On<br>        Parser apache2<br><br>    [OUTPUT]<br>        Name es<br>        Match kube.parsed.*<br>        Host quickstart-es-internal-http.default.svc.cluster.local<br>        TLS             On<br>        Suppress_Type_Name On<br>        TLS.Verify      Off<br>        HTTP_User       elastic<br>        HTTP_Passwd     XXXXXX<br>        Trace_Error       On<br>        Logstash_Format On<br>        Retry_Limit False<br>    [OUTPUT]<br>        Name    stdout<br>        Match   *<br>        Format  json_lines<br>  parsers.conf: |<br>    [PARSER]<br>        Name   apache2<br>        Format regex<br>        Regex  ^(?&lt;host&gt;[^ ]*) [^ ]* (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+)(?: +(?&lt;path&gt;[^ ]*) +\S*)?&quot; (?&lt;code&gt;[^ ]*) (?&lt;size&gt;[^ ]*)(?: &quot;(?&lt;referer&gt;[^\&quot;]*)&quot; &quot;(?&lt;agent&gt;[^\&quot;]*)&quot;)?$<br>        Time_Key time<br>        Time_Format %d/%b/%Y:%H:%M:%S %z<br><br>    [PARSER]<br>        Name   nginx<br>        Format regex<br>        Regex ^(?&lt;remote&gt;[^ ]*) - (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+) (?&lt;url&gt;\S+) \S+&quot; (?&lt;code&gt;\d+) (?&lt;size&gt;\d+)<br>        Time_Key time<br>        Time_Format %d/%b/%Y:%H:%M:%S %z</pre><p>It’s too long and hard to see what is going on in the first sight, so allow me to break down step by step.</p><h4>Service</h4><p>This is a configuration for Fluent bit itself. You can define</p><ul><li>log level of fluent bit</li><li>parser file path</li><li>and more!</li></ul><pre>[SERVICE]<br>    Daemon Off<br>    Flush 1<br>    Log_Level info<br>    Parsers_File /fluent-bit/etc/conf/parsers.conf<br>    HTTP_Server On<br>    HTTP_Listen 0.0.0.0<br>    HTTP_Port 2020<br>    Hot_Reload On<br>    Health_Check On</pre><h4>Input</h4><p>This is where you collect logs. The easiest input is called “tail” which has really similar function as tail command in Linux, collecting the latest log periodically. In this example, Fluent bit will check /var/log/containers/*.log where kubernetes logs are located by default.</p><p>Please note that you tag all the Kubernetes logs with “kube.*”, which is useful when you have multiple inputs and want to filter or parse specific logs.</p><pre>[INPUT]<br>    Name tail<br>    Path /var/log/containers/*.log<br>    multiline.parser docker, cri<br>    Tag kube.*<br>    Mem_Buf_Limit 5MB<br>    Skip_Long_Lines On</pre><h4>Filter — Kubernetes</h4><p>This is a Kubernetes plugin for filter. All the Kubernetes log (matched with kube.*). You can merge/filter some Kubernetes logs conveniently using parameters. This is the same parameters as default from fluent bit.</p><pre>[FILTER]<br>    Name kubernetes<br>    Match kube.*<br>    Merge_Log On<br>    Keep_Log Off<br>    K8S-Logging.Parser On<br>    K8S-Logging.Exclude On</pre><h4>Filter — Nginx/Apache</h4><p>In this part, you are filtering the logs based on container name, only picking up the related logs.</p><p>Then, we are sending to the parser which we are going to talk about in a moment.</p><pre>[FILTER]<br>    Name  rewrite_tag<br>    Match kube.*<br>    Rule  $kubernetes[&#39;container_name&#39;] ^nginx$ kube.parsed.nginx true<br>    Rule  $kubernetes[&#39;container_name&#39;] ^apache$ kube.parsed.apache true<br><br>[FILTER]<br>    Name   grep<br>    Match  kube.*<br>    Regex  $kubernetes[&#39;container_name&#39;] (nginx|apache)<br><br>[FILTER]<br>    Name parser<br>    Match kube.parsed.nginx<br>    Key_Name log<br>    Preserve_Key On<br>    Reserve_Data On<br>    Parser nginx<br><br>[FILTER]<br>    Name parser<br>    Match kube.parsed.apache<br>    Key_Name log<br>    Preserve_Key On<br>    Reserve_Data On<br>    Parser apache2</pre><h4>Parser</h4><p>In parser, you are going to convert string into hash map (or dictionary in Python or object in Javascript). In the first example, “host” from &lt;host&gt; is going to be a key and parsed value is going to be a value.</p><pre>[PARSER]<br>    Name   apache2<br>    Format regex<br>    Regex  ^(?&lt;host&gt;[^ ]*) [^ ]* (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+)(?: +(?&lt;path&gt;[^ ]*) +\S*)?&quot; (?&lt;code&gt;[^ ]*) (?&lt;size&gt;[^ ]*)(?: &quot;(?&lt;referer&gt;[^\&quot;]*)&quot; &quot;(?&lt;agent&gt;[^\&quot;]*)&quot;)?$<br>    Time_Key time<br>    Time_Format %d/%b/%Y:%H:%M:%S %z<br><br>[PARSER]<br>    Name   nginx<br>    Format regex<br>    Regex ^(?&lt;remote&gt;[^ ]*) - (?&lt;user&gt;[^ ]*) \[(?&lt;time&gt;[^\]]*)\] &quot;(?&lt;method&gt;\S+) (?&lt;url&gt;\S+) \S+&quot; (?&lt;code&gt;\d+) (?&lt;size&gt;\d+)<br>    Time_Key time<br>    Time_Format %d/%b/%Y:%H:%M:%S %z</pre><h4>Output — Elasticsearch</h4><p>Finally, this is the configuration you connect Fluent bit with Elasticsearch. Please note that “kube.parsed.*” is specified in Match so that it extracts only logs after parser. Otherwise, you will receive both unparsed original logs and parsed logs.</p><p>For password, please use the same password you used to login kibana.</p><pre>[OUTPUT]<br>      Name es<br>      Match kube.parsed.*<br>      Host quickstart-es-internal-http.default.svc.cluster.local<br>      TLS             On<br>      Suppress_Type_Name On<br>      TLS.Verify      Off<br>      HTTP_User       elastic<br>      HTTP_Passwd     XXXXXX<br>      Trace_Error       On<br>      Logstash_Format On<br>      Retry_Limit False</pre><h4>Output — stdout</h4><p>This is a optional configuration. It shows parsed logs in stdout of fluent bit.<br>This is extremely useful for debugging.</p><pre>[OUTPUT]<br>    Name    stdout<br>    Match   *<br>    Format  json_lines</pre><h4>Apply ConfigMap</h4><p>Name your file “fluent_cm.yaml” and execute the following command.</p><pre>kubectl apply -f fluentbit_cm.yaml --force</pre><h4>Restart fluent bit(Cold reload)</h4><p>By default, Hot reload is disabled, so you can manually restart pod of fluent bit to affect a new configuration.</p><pre>kubectl get pod -o custom-columns=&quot;NAME:.metadata.name&quot; | grep fluent | xargs -I {} kubectl delete pod {}</pre><p><em>Once you enable the hot reload, you can use following command.</em></p><pre>kubectl exec busybox -- curl -X POST -d &#39;{}&#39; fluent-bit:2020/api/v2/reload</pre><h3>make logs</h3><p>To produce couple of logs on nginx and apache pods, let’s curl them a few times.</p><pre>kubectl exec busybox -- curl nginx<br>kubectl exec busybox -- curl apache</pre><h3>Confirm logs on stdout</h3><p>Before discovering the logs on Kibana, let’s make sure it arrives at fluent bit and got filtered and parsed successfully. To check the log, please run the following command.</p><pre>k get pod -o custom-columns=&quot;NAME:.metadata.name&quot; | grep fluent | xargs -I {} kubectl logs -f {}</pre><p>You should be able to see something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N2o9Zv93nV4QLasC-BgoSQ.png" /></figure><h3>Confirm logs on Kibana</h3><p>Go to discover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQQSXO2zNctFwp9qgWwMvw.png" /></figure><p>Run the following query.</p><pre>FROM logs* | LIMIT 10 | WHERE `kubernetes.container_name` in (&quot;nginx&quot;, &quot;apache&quot;)</pre><p>You can see the whole logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYCuyKZ-ZZoyTODaYpFIKQ.png" /></figure><h3>Conclusion</h3><p>This time, we sent logs to Elasticsearch, but it is possible to have multiple outputs such as AWS S3, Kinesis Firehose or simply database.</p><p>I hope you learned some from this article. If you have any comment, I would love to hear!</p><p>See you⭐</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7e97cc77928" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure your Kubernetes with Service Mesh]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/secure-your-kubernetes-with-service-mesh-b333736af1a5?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b333736af1a5</guid>
            <category><![CDATA[mtls-authentication]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sun, 15 Jun 2025 15:01:04 GMT</pubDate>
            <atom:updated>2025-06-20T00:54:51.883Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*2tfmtu_THJiN0WKGG8sRHg.png" /></figure><p>Istio is one of the prominent service mesh implementation and it can be used in many use-cases including</p><ul><li>Canary Release</li><li>A/B Testing</li><li>Network Monitoring</li><li>mutual TLS</li><li>and more!</li></ul><p>In this article, I am going to explore mutual TLS feature in Istio to secure your Kubernetes cluster.</p><h3>Why mTLS matters?</h3><p>You might think you have firewall, https between user and load balancer, so you don’t need mTLS within the cluster. Why do we need to use TLS even between pods. It just make things too complicated.</p><p>Well, In Kubernetes, you wouldn’t notice easily that it is actually multiple nodes that support 1 cluster. Kubernetes was intentionally made to opinionate the concept of Pod and Deployment, and nodes are just a giant pool of resources.</p><p>But if you think about it, it is possible that you have 1 node in AZ us-east-1 and another node in AZ us-east-2. The connection between them can go through the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sz8vaQMK-DmJLRqZ3Jyg7g.png" /></figure><p>Now, you understand why mTLS is integral to our system, and that is where Istio comes in.</p><h3>Install helm</h3><p>We have many ways to install Istio, but this time, I am going to install it as Deployment in kubernetes and to do so, let’s intall helm first.</p><pre>curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3<br>chmod 700 get_helm.sh<br>./get_helm.sh</pre><h3>Install Istio</h3><pre>helm repo add istio https://istio-release.storage.googleapis.com/charts<br>helm repo update<br>helm install istio-base istio/base -n istio-system --set defaultRevision=default --create-namespace<br>helm install istiod istio/istiod -n istio-system --wait</pre><h3>Enable Istio Envoy Injection</h3><p>Thanks to Istio, you don’t have to edit your any yaml file to integrate Service Mesh. Istio takes care of installing Envoy proxy in each pod as a sidecar, distributing configurations to each proxy, and collecting metrics and more!</p><p>To inject Envoy proxy to default namespace, you run the following command.</p><pre>kubectl label namespace default istio-injection=enabled</pre><h3>Launch 2 random pods for test</h3><p>To establish mTLS, let’s run 2 pods to test it.</p><pre>kubectl run nginx --image=nginx<br>kubectl run busybox --image=curlimages/curl -- sleep 10000</pre><p>Now, you can see each pod actually has 2 containers now. This means it has Envoy sidecar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*uyZ-qhAoaxgtzNRVN5LvKw.png" /></figure><p>Let’s expose nginx port with service.</p><pre>kubectl expose pod nginx --port=80 --target-port=80</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBWo-_H3ipWgdz6-aApXxg.png" /></figure><p>Please note that you can still access the service with http not https. This is because Istio runs as PERMISSIVE mode by default. In this mode, it tries to use mTLS as much as possible, but if it fails, it would use http instead, which is useful when there are some legacy pods or namespace where you haven’t integrated Istio yet, and you can not set up mutual TLS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SVC9jbW6aLSRVIvQpnr9bw.png" /></figure><h3>Test mTLS</h3><p>Now, let’s use RESTRICT mode instead. This is the opposite of PERMISSIVE mode and you can not use http connection anymore.</p><pre>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: security.istio.io/v1<br>kind: PeerAuthentication<br>metadata:<br>  name: &quot;default&quot;<br>  namespace: &quot;istio-system&quot;<br>spec:<br>  mtls:<br>    mode: STRICT<br>EOF</pre><p>Now you can not use http.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*1DfMlLM7JZMxn4n4Mft3iQ.png" /></figure><p>So let’s access from another pod we created. It should return 200 successfully.</p><pre>kubectl exec busybox -- curl nginx</pre><h3>Inspect mTLS by tcpdump</h3><p>So the question now is “is it really mTLS?”. There is no log that shows mTLS. You still use http in curl because Envoy proxy would take care of converting it to https, so there is no obvious way to see it’s mTLS.</p><p>To dimistify this, let’s use tcpdump.</p><p>To run tcpdump, Envoy need more permission to use “sudo” command in envoy proxy, and you can do so by reassigning one value of helm configuration as follows.</p><pre>helm upgrade istiod istio/istiod -n istio-system --set global.proxy.privileged=true</pre><blockquote>ref: <a href="https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/">https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/</a></blockquote><p>Now it’s all set. Let’s set tcpdump in “nginx” pod with the following command.</p><pre> kubectl exec nginx -c istio-proxy -- sudo tcpdump dst port 80  -A</pre><p>Then, let’s access this nginx from “busybox” command.</p><pre>k exec busybox -- curl nginx</pre><p>In nginx pod, you should see some encrypted information that looks like this. This tells you it’s using mTLS correctly via Envoy Proxy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uHPoMGGVmBFMrqphrAJgew.png" /></figure><h3>Conclusion</h3><p>I initially thought Istio is just more advanced proxy with granular and fine-grained control to set routing and filtering, but it’s more than that.</p><p>This security empowerment can be enough reason why you want to integrate it to your cluster.</p><p>I hope you learned some from this article. If you have any comment, I would love to hear!</p><p>See you⭐</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b333736af1a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Your stress testing with Terraform/Ansible Part 3]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/automate-your-stress-testing-with-terraform-ansible-part-3-a37e9fbe8074?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a37e9fbe8074</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jmeter]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sun, 08 Jun 2025 15:17:15 GMT</pubDate>
            <atom:updated>2025-06-08T15:17:15.631Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-23FefhfQC8yAYC8dDu-Q.png" /></figure><p>This is a Part 3 of the series for automating stress testing. If you haven’t check the part 1 and 2 yet, please check it from the link below.</p><ul><li><a href="https://medium.com/@kikuchidaisuke.zr/automate-your-stress-testing-with-terraform-ansible-part-1-db3241249b0d">Automate Your stress testing with Terraform/Ansible Part 1</a></li><li><a href="https://medium.com/@kikuchidaisuke.zr/automate-your-stress-testing-with-terraform-ansible-part-2-0ce7dd2c1c46">Automate Your stress testing with Terraform/Ansible Part 2</a></li></ul><ol><li>Automate test servers with terraform</li><li>Set up Prometheus and Grafana to monitor nodes and containers</li><li><strong>Set Up Ansible</strong></li><li>Initiate Ansible</li><li>Check the result on Prometheus and Grafana</li></ol><h3>Install Ansible</h3><p>In this article, I am going to install it on WSL of Windows since Ansible cannot be installed on Windows as itself.</p><p>If you are using Mac or Other Linux distributions, Please check the official website of Ansible to check how to install it.</p><p><a href="https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html">Installing Ansible - Ansible Community Documentation</a></p><pre>sudo apt update<br>sudo apt install software-properties-common<br>sudo add-apt-repository --yes --update ppa:ansible/ansible<br>sudo apt install ansible</pre><h3>Install Ansible collections</h3><p>Ansible collections are a package including</p><ul><li>roles: public play book</li><li>module: predefined code to convert you Ansible to command.</li><li>plugins: Anything that customizes Ansible (※Module is a part of plugin, task plugin)</li></ul><p>Long story short, it allows you to manipulate aws or any other resources very easily.</p><p>Here is the commands to install the necessary tools.</p><pre>ansible-galaxy collection install amazon.aws<br>ansible-galaxy collection install community.aws<br>ansible-galaxy collection install chocolatey.chocolatey<br>ansible-galaxy collection install ansible.windows<br>ansible-galaxy collection install community.windows<br>sudo apt install python3-pip<br>pip install boto3</pre><h4>Trivia</h4><p><em>The default Ansible includes some famous collections, so you might have experience to create AWS resources without installing collection. The ansible that doesn’t include any collection is called Ansible core.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gIjdXZOOXRMvod9wH2UwA.png" /></figure><h3>Define inventory</h3><p>The next thing to do is to define inventory.</p><p>So what is inventory? They are the collection of hosts that you want to execute. You can define them manually in ini format or yaml format.</p><p>But in this time, we are using AWS EC2 and luckily we have something called Dynamic Inventory. You can select sets of EC2 based on region, AZ, tag and more. If you remember, we created ec2 with tags like “usage:test”, “ os:windows/linux”.</p><p>Create the file called inventory_aws_ec2.yaml with the following content.</p><pre>plugin: amazon.aws.aws_ec2<br>regions:<br>- us-east-2<br>keyed_groups:<br>  - key: &quot;tags.os&quot;<br>    separator: &quot;&quot;<br>filters:<br>  tag:usage: test<br>  tag:os:<br>    - windows<br>    - linux</pre><h3>Set up Ansible config</h3><p>We’ve defined hosts already, so what’s next?</p><p>Next, we need to define how you access these hosts, namely credentials. In linux, you can have shared ssh key but in Windows, you have different password every time you create a new EC2.</p><p><em>*If you have ActiveDirectory, you can always use the same user and password.</em></p><p>To address this issue, you can still use Ansible to retrieve password and generate config file automatically. you can name this file “init_playbook.yaml”.</p><pre>- name: Set up Ansible config<br>  hosts: localhost<br>  gather_facts: false<br>  tasks:<br>    - name: Lookup EC2 instance(s) by tag<br>      amazon.aws.ec2_instance_info:<br>        filters:<br>          &quot;tag:os&quot;: &quot;windows&quot;<br>          &quot;tag:usage&quot;: &quot;test&quot;<br>      register: ec2_info<br>  <br>    - name: print instance_id<br>      debug:<br>        msg: &quot;{{ ec2_info.instances[0].instance_id }}&quot;<br><br>    - name: get the Administrator password<br>      community.aws.ec2_win_password:<br>        instance_id: &quot;{{ ec2_info.instances[0].instance_id }}&quot;<br>        key_file: &quot;~/.ssh/demo.pem&quot;<br>        wait: true<br>      register: password<br><br>    - name: print password<br>      debug:<br>        msg: &quot;{{ password }}&quot;<br><br>    - name: Ensure group_vars directory exists<br>      file:<br>        path: group_vars<br>        state: directory<br><br>    - name: Create group_vars/linux.yml<br>      copy:<br>        dest: group_vars/linux.yml<br>        content: |<br>          ansible_user: ec2-user<br>          ansible_ssh_private_key_file: ~/.ssh/demo.pem<br>          ansible_become: true <br><br>    - name: Create group_vars/windows.yml<br>      copy:<br>        dest: group_vars/windows.yml<br>        content: |<br>          ansible_user: Administrator<br>          ansible_password: {{password.win_password}}<br>          ansible_connection: winrm<br>          ansible_winrm_transport: ntlm<br>          ansible_winrm_server_cert_validation: ignore</pre><p>Run the following command and make sure it creates group_vars/linux.yaml, and windows.yaml.</p><pre>ansible-playbook init_playbook.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*HbX6ab1dEGqIrjVHkvQKoQ.png" /></figure><h3>Playbook for Linux</h3><p>Now, it’s an easy part. We have 3 tasks in total.</p><ul><li>Install Java</li><li>Install Jmeter</li><li>Unpack Jmeter</li></ul><p>Let’s name this file “linux_playbook.yaml”.</p><pre>- name: Ansible Test Playbook<br>  gather_facts: false<br>  hosts: linux<br>  tasks:<br>    - name: Install Java.<br>      ansible.builtin.dnf:<br>        name: java-17-amazon-corretto<br>        state: present<br><br>    - name: Download jmeter<br>      ansible.builtin.get_url:<br>        url: https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.3.tgz<br>        dest: /home/ec2-user/jmeter.tgz<br><br>    - name: Extract jmeter<br>      ansible.builtin.unarchive:<br>        remote_src: true<br>        src: /home/ec2-user/jmeter.tgz<br>        dest: /home/ec2-user</pre><p>Execute it with the following command.</p><pre>ansible-playbook -i inventory_aws_ec2.yaml linux_playbook.yaml</pre><h3>Playbook for Windows</h3><p>Same for Windows. File name is “windows_playbook.yaml”</p><pre>- name: Ansible Test Playbook<br>  gather_facts: false<br>  hosts: windows<br>  vars:<br>    jdk_package: jdk17<br>    java_major_version: 17.0.2<br>  tasks:<br>    - name: Install Java<br>      win_chocolatey:<br>        name: openjdk<br>        state: present<br>        version: &quot;17.0.2&quot;<br><br>    - name: Create directory structure<br>      ansible.windows.win_file:<br>        path: C:\temp<br>        state: directory<br><br>    - name: Download jmeter<br>      ansible.windows.win_get_url:<br>        url: https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.3.zip<br>        dest: C:\temp\apache-jmeter-5.6.3.zip<br><br>    - name: Unzip gz log<br>      community.windows.win_unzip:<br>        src: C:\temp\apache-jmeter-5.6.3.zip<br>        dest: C:\temp</pre><p>Execute it with the following command.</p><pre>ansible-playbook -i inventory_aws_ec2.yaml windows_playbook.yaml</pre><h3>Conclusion</h3><p>Today, we went through the process of install Jmeter on EC2 only using Ansible.</p><p>Next time, we are finally going to execute Jmeter to conduct stress testing.<br>Stay tuned! ⭐</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a37e9fbe8074" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collect logs on EC2/ASG with SSM before shutting down]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/collect-logs-on-ec2-asg-with-ssm-before-shutting-down-060440f2386d?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/060440f2386d</guid>
            <category><![CDATA[aws-system-manager]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[auto-scaling-groups]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sun, 01 Jun 2025 01:51:12 GMT</pubDate>
            <atom:updated>2025-06-01T01:51:12.434Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rkZxikoSNFM1Lem6z6mP5A.png" /></figure><p>When you want to collect logs from EC2, the CloudWatch, DataDog would be better option in most of the cases,</p><p>But in other cases, you might not want to use them to reduce the cost as much as possible or for security reason.<br>In this article, I am going to walk you through how you set up AWS System Manager to access EC2, collect logs and send it to S3 bucket.</p><h3>Create a lifecycle hook in ASG</h3><p>Visit the detail page of your ASG, and click “instance management”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hJu22dTjFDxVJn5ZyYmcA.png" /></figure><p>Then click “Create lifecycle hook”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YrqCMOh2EsPCsHMD7NAPqA.png" /></figure><p>Fillout the inputs as follows. <br><em>You can set the lifecycle transition to “instance launch” depending on your requirement.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-nhV8qLc7EfTe41atGDNg.png" /></figure><h3>Create SSM document</h3><p>In System Manager, find “Document” section and create a new one.<br>Make sure you choose “Command or Session”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5rLEfmNsxp_6TS2n5tWCeA.png" /></figure><p>Put a random name, and as a content, you can isnert the following json as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-CV6EV_7GOaW9JkgtZyprQ.png" /></figure><pre>{<br>  &quot;schemaVersion&quot;: &quot;2.2&quot;,<br>  &quot;description&quot;: &quot;Upload log to S3&quot;,<br>  &quot;mainSteps&quot;: [<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;UploadAcmeJson&quot;,<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;#!/bin/bash&quot;,<br>          &quot;aws s3 cp /LOG/PATH/XXX.log s3://YYYYY/XXX.log&quot;<br>        ]<br>      }<br>    }<br>  ]<br>}</pre><p>This is just one example. You can copy it to S3 or you can run scp command to copy it to whereever you want.</p><h3>Set up EventBridge</h3><p>First of all, visit “default” event bus in EventBridge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jfe2LnEOhFi8-iD530Bzxw.png" /></figure><p>Create a new rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmYYkgKxFvX2LNGrK6GyuA.png" /></figure><p>Fill out a name, and leave other fields as default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zzWUX88c_PcxCQRssIKrTg.png" /></figure><p>Choose “others” as Event Source and select “Custom pattern”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlhHANKJXvxXGqZRWH-ZSg.png" /></figure><p>Then copy and paste the following json as Event Pattern. Don’t forget to change ASG name accordingly.</p><pre>{<br>  &quot;source&quot;: [&quot;aws.autoscaling&quot;],<br>  &quot;detail-type&quot;: [&quot;EC2 Instance-terminate Lifecycle Action&quot;],<br>  &quot;detail&quot;: {<br>    &quot;AutoScalingGroupName&quot;: [&quot;er-diagram-asg&quot;]<br>  }<br>}</pre><p>In the next section, you are going to specify the target who receives the event. Choose “Systems Manager Run Command”, and find your document!</p><p>Also, you can specify The tags used to find EC2 where we run command with SSM document.<br>For ASG, “tag:aws:autoscaling:groupName” is key, and value is your ASG name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJFtdDVyDzBgfu02D0OO0g.png" /></figure><p>After that, you can just click “Next” and you are good.</p><h3>Configure IAM Role for EC2</h3><p>Last but not least, you need to add desired permissions to Instance Profile (IAM Role) to use SSM properly.</p><p>“AmazonSSMManagedInstanceCore” is the only one required. (If you just add permission, it wouldn’t work without rebooting. After the change, please make sure you reboot isntances where you want to catch the event)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nf8iV_uZn3G_ATV9aSKcHg.png" /></figure><blockquote>Depending on OS, you might need to additionally install SSM agent. Please check the following link to see the list of OS where SSM agent is already installed.</blockquote><p><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html">Find AMIs with the SSM Agent preinstalled</a></p><blockquote>You also allow outbound 443 to access SSM.</blockquote><p><a href="https://repost.aws/questions/QUARF56BqXQaKrU27VTPSNYQ/security-group-for-session-manager">security group for session manager</a></p><h3>Test behavior</h3><p>The easiest way to test this, you can change desired capacity of ASG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ursjsmFJxR53afYHKYq-4Q.png" /></figure><p>Once you change it, You can check if it runs propery by looking at “Run Command” history section in System Manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jqoMMteKgNOgi1BkSy5iWg.png" /></figure><h3>Conclusion</h3><p>This time, we explored Lifecycle hooks with SSM but it can be Lambda, SQS, SNS or more!</p><p>This is a nice way to integrate your custom logic to scaling, potentially optimizing your scaling and reducing some costs by being creative.</p><p>See you soon!👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=060440f2386d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CRI & containerd Explained]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/cri-containerd-explained-29ad5c598f8b?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/29ad5c598f8b</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[runc]]></category>
            <category><![CDATA[containerd]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sat, 24 May 2025 16:22:05 GMT</pubDate>
            <atom:updated>2025-05-24T16:22:37.821Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*U_9HWpP-IaKipUk8s5u7wQ.png" /></figure><p>Have you ever wondered what container technologies are being used for Kubernetes?</p><p>Is it Docker? Well, in this article, I’ walk you through how kubernetes and docker try to standardize container technologies and grow together. More specifically, I will answer the question, what is CRI or containerd and how they communicate with each other.</p><h3>Proliferation of container technologies</h3><p>10 years ago, Docker had dominated a container field, and there was no option that you could choose apart from Docker. If you use container, it was Docker. the most obvious choice.</p><p>But few years later, other container runtimes started appearing such as cri-O and rkt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8bAlQH-ELREggRRbG9YDGQ.png" /></figure><h3>Standardization of the interface (CRI)</h3><p>As an container industory, they wanted to standardize the interface so that Kubernetes or other container orchastration tools (e.g. Apache Mesos) doesn’t have to worry about how each container technologies create a container or pull an image. As a Kubernetes, they just want to call the same request and get the same response regardless of the underlying technologies.</p><p>That is where CRI (Container Runtime Interface) comes in. CRI itself is just a Protocol Buffer in gRPC. You can find the actual “.proto” files <a href="https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6jYdqafvC98faDm29qtYMA.png" /></figure><p>If you are not familar with gRPC, you can think of it as a model in any Backend frameworks (e.g. Entity in Spring, Model in Django), but it can be universally used across multiple languages.</p><p>So all the container technologies tried to implement this to accept the request from Kubernetes, except for one.</p><p>It was Docker. Docker has its own defined interface and didn’t adopt the interface for some reasons.</p><h3>The first solution: docker-shim</h3><p>To make docker compatible with Kubernetes, Kubernetes maintained something called “docker-shim”. It was a kind of translator between CRI and Docker, simply converting the request from Kubernetes to docker-specified request and vice versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*tyOR3cK88jPdpTNXAZByjQ.png" /></figure><p>But it was against standardization and Kubernetes needed to maintain 2 interfaces which was the opposite from the initial purpose.</p><p>*<a href="https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/"><em>Kubernetes support for Docker via </em></a><a href="https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/"><em>dockershim is now removed.</em></a></p><h3>The second solution: containerd</h3><p>To overcome this challenge, Docker took an action. They thought why don’t we create the CRI-compatible underlying software and create Docker on the top of it.</p><p>One of the reasons why it was difficult for Docker to adopt CRI was because they have many features as a container service beyond a container runtime. So they created containerd to follow CRI, not Docker itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*-imQhJhKFqgA4Spu.png" /><figcaption><a href="https://blog.purestorage.com/purely-educational/containerd-vs-docker-whats-the-difference/">https://blog.purestorage.com/purely-educational/containerd-vs-docker-whats-the-difference/</a></figcaption></figure><p>Now, Kubernetes and Docker shares common container runtime and containerd would be the most prominent runtime nowadays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*0He_jjNYChNiQRJKEFfp8Q.png" /></figure><p>You can find the implementation of CRI in containerd in the following url.</p><ul><li><a href="https://github.com/containerd/containerd/blob/release/1.6/pkg/cri/server/sandbox_run.go#L61">pkg/cri/server/sandbox_run.go:61</a></li><li><a href="https://github.com/containerd/containerd/blob/release/1.6/pkg/cri/server/sandbox_run.go#L422">pkg/cri/server/sandbox_run.go:422</a></li></ul><blockquote>ref: <a href="https://addozhang.medium.com/source-code-analysis-understanding-cnis-usage-from-kubelet-container-runtime-24d72f29466b">https://addozhang.medium.com/source-code-analysis-understanding-cnis-usage-from-kubelet-container-runtime-24d72f29466b</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zdT6B0FNuvw7nUUUsZ8NNg.png" /></figure><h3>Deep Dive: containerd</h3><p>We understand what contained is and the history behind it. Now let’s unerstand containerd itself. There are a few components you should know.</p><ul><li>containerd daemon</li><li>containerd-shim</li><li>runc</li></ul><h4>containerd daemon</h4><p>This is the main part of containerd. (It is usually just called containerd). This handles all gRPC requests/response from Kubernetes and delegate tasks to containerd-shim per container/pod.</p><blockquote><a href="https://github.com/containerd/containerd/blob/main/docs/historical/design/lifecycle.md">https://github.com/containerd/containerd/blob/main/docs/historical/design/lifecycle.md</a></blockquote><h4>containerd-shim</h4><p>This is a process to handle one pod by default. It takes care of creating/stopping “container” using runc or mounting the filesystem and more. This also communicates with containerd daemon via gRPC (ttrpc).</p><h4>runc</h4><p>This is another big topic of container, but it is essentially the core of container technologies. This will make system calls to kernel, manipulating Linux namespace, cgroup, Seccomp and more to make a process segregated from other resources (aka container). What we discussed earlier is management of containers including containerd daemon, but runc is an actual juicy part of container magic.</p><p>We can summarize these components as the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cb4y6Z4yqZ6uvTRY.png" /><figcaption><a href="https://github.com/containerd/containerd/blob/main/docs/cri/architecture.md">https://github.com/containerd/containerd/blob/main/docs/cri/architecture.md</a></figcaption></figure><p>More specifically, the flow looks like this from kubelet to an actual container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vf7zet89S-seGGMNOoI94A.png" /></figure><h3>Conclusion</h3><p>This time, I focused on container runtime, I hope it helps you undersntand docker or kubernetes better.</p><p>But This is one aspect of container, and there are also Container Network Interface (CNI), and Container Storage Interface (CSI).</p><p>So stay tuned for them. Good bye 👋</p><h3>Reference</h3><blockquote><a href="https://iximiuz.com/en/posts/container-learning-path/">https://iximiuz.com/en/posts/container-learning-path/</a></blockquote><blockquote><a href="https://addozhang.medium.com/source-code-analysis-understanding-cnis-usage-from-kubelet-container-runtime-24d72f29466b">which file being executed for containerd gRPC</a></blockquote><blockquote><a href="https://www.nccgroup.com/us/research-blog/technical-advisory-containerd-containerd-shim-api-exposed-to-host-network-containers-cve-2020-15257/">containerd-shim</a></blockquote><blockquote><a href="https://github.com/containerd/containerd/blob/main/docs/cri/architecture.md">github — Architecture of CRI</a></blockquote><blockquote><a href="https://mkdev.me/posts/the-tool-that-really-runs-your-containers-deep-dive-into-runc-and-oci-specifications">The tool that really runs your containers: <br>deep dive into runc and OCI specifications</a></blockquote><p><a href="https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto">cri-api/pkg/apis/runtime/v1/api.proto at c75ef5b473bbe2d0a4fc92f82235efd665ea8e9f · kubernetes/cri-api</a></p><blockquote><a href="https://addozhang.medium.com/source-code-analysis-understanding-cnis-usage-from-kubelet-container-runtime-24d72f29466b">https://addozhang.medium.com/source-code-analysis-understanding-cnis-usage-from-kubelet-container-runtime-24d72f29466b</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29ad5c598f8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Your stress testing with Terraform/Ansible Part 2]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/automate-your-stress-testing-with-terraform-ansible-part-2-0ce7dd2c1c46?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ce7dd2c1c46</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Sat, 17 May 2025 19:46:52 GMT</pubDate>
            <atom:updated>2025-05-17T19:46:52.506Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*c3Dhuo5RjVGYIGAgo1kOkQ.png" /></figure><p>This is a Part 2 of the series for automating stress testing. If you haven’t check the part 1 yet, please check it from the link below.</p><p><a href="https://medium.com/@kikuchidaisuke.zr/automate-your-stress-testing-with-terraform-ansible-part-1-db3241249b0d">Automate Your stress testing with Terraform/Ansible Part 1</a></p><ol><li>Automate test servers with terraform</li><li><strong>Set up Prometheus and Grafana to monitor nodes and containers</strong></li><li>Initiate the test with Ansible</li><li>Check the result on Prometheus and Grafana</li></ol><p>In this article, we are focusing on the set up of Prometheus and Grafana on Kubernetes cluster. I assume that readers already have the cluster ready but I will simulate the Kubernetes on EC2 using K3s which is the light weight version of kubernetes.</p><h3>Set Up Kubernetes</h3><p>You can spin up 1 Amazon Linux 2023 instance and run command to install kubernetes.</p><p><em>*This should be a different ec2 instance that we created with terraform as testing clients!</em></p><pre>EC2_USER_HOME=&quot;/home/ec2-user&quot;<br>cd $EC2_USER_HOME<br>export INSTALL_K3S_VERSION=&quot;v1.32.3+k3s1&quot;<br>curl -sfL https://get.k3s.io | sh -<br># root<br>echo &#39;alias kubectl=&quot;sudo k3s kubectl&quot;&#39; &gt;&gt; $HOME/.bashrc<br>echo &#39;alias k=&quot;sudo k3s kubectl&quot;&#39; &gt;&gt; $HOME/.bashrc<br># ec2-user<br>echo &#39;alias kubectl=&quot;sudo k3s kubectl&quot;&#39; &gt;&gt; $EC2_USER_HOME/.bashrc<br>echo &#39;alias k=&quot;sudo k3s kubectl&quot;&#39; &gt;&gt; $EC2_USER_HOME/.bashrc<br>source $HOME/.bashrc</pre><h3>Install helm</h3><p>There are so many ways to install prometheus and grafana, but this time, I am going to use helm which is a package manager for kubernetes, allowing you to install common sets of softwares with just one command!</p><p>To install helm, please run the following command.</p><pre>curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3<br>chmod 700 get_helm.sh<br>./get_helm.sh</pre><p>Make sure it is installed correctly</p><pre>helm version</pre><h3>Install Prometheus and Grafana</h3><p>To install them with helm, you need to add a repository as you do in yum or apt. For more details, you can check the prometheus helm github in the following link.</p><p><a href="https://github.com/prometheus-community/helm-charts">https://github.com/prometheus-community/helm-charts</a></p><pre>sudo helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>sudo helm repo update</pre><p>To install Prometheus and Grafana, run the following commands.<br><em>*you might not need “ — kubeconfig=XXX” if you don’t use k3s</em></p><pre>kubectl create namespace prometheus<br>sudo helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n prometheus --kubeconfig=/etc/rancher/k3s/k3s.yaml</pre><p>This command allows you to install everything you need. By default, Kubernetes doesn’t have a way to monitor the resource usage even in command line. To do so, You need to set up the metric server. However this command did it for you. and you can now check the resource usage by command as well.</p><pre>kubectl top pod -n kube-system</pre><h3>Expose Prometheus</h3><p>Currently, prometheus is only available inside of the cluster and you can not access it from your browser. To do so, I am going to expose the existing service with “NodePort” service with an arbitrary port (e.g. 30303)<br><em>(Ideally, you use “LoadBalancer” type service with some cloud providers and map it to port 80/443)</em></p><pre>kubectl expose service kube-prometheus-stack-prometheus --type=NodePort --target-port=9090 --name=prometheus-server-ext -n prometheus</pre><p>Check the port assigned to this service by the following command.</p><pre>kubectl get svc prometheus-server-ext -n prometheus</pre><p>Finally, you open the prometheus homepage at publicIP:PORT</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqWuxkaOs89ENnHNr2pnzg.png" /></figure><p>Let’s run some promQL (query) to make sure Prometheus is successfully collecting information from the existing resources.</p><p>By running the following query, you can check how many pods are running on each namespace</p><pre>count(kube_pod_info) by (namespace)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2J54nSgd9ZDod4LE53GWWw.png" /></figure><h3>Expose Grafana</h3><p>Now let’s move onto Grafana! you run a very similar command that exposes prometheus.</p><pre>kubectl expose service kube-prometheus-stack-grafana --type=NodePort --target-port=3000 --name=grafana-ext -n prometheus</pre><p>And then, check your port.</p><pre>kubectl get svc grafana-ext -n prometheus</pre><p>You should be able to access it from browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ALV5_XICLoIhzN1yxA7rlA.png" /></figure><p>But we don’t know user and password yet!</p><ul><li>user: admin</li><li>password: run the following command</li></ul><pre>kubectl get secret kube-prometheus-stack-grafana -o jsonpath=&quot;{.data.admin-password}&quot; -n prometheus | base64 --decode ; echo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEUEZzIknflJGCo9CcXeDQ.png" /></figure><p>Thanksfully, all the dashboard related to Prometheus is already ready! So let’s go visit one of the dashboard available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23n44d9ruEARRkW0HIzdrQ.png" /></figure><p>*Make sure you change namespace to where you already have some pods</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MoyHwbAQiY1MjDG5HvC8KA.png" /></figure><p>You see prometheus start collecting data for each pod.</p><h3>Conclusion</h3><p>Nowadays, we have rich enterprise monitoring solution such as Data Dog or NewRelic, but Grafana and Prometheus are so powerful and allow you to triage a root cause with cosutomizable dashboard.</p><p>We are going to use this to monitor CPU, Memory, Disk for stress testing.</p><p>See you next time ⭐</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ce7dd2c1c46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Your stress testing with Terraform/Ansible Part 1]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/automate-your-stress-testing-with-terraform-ansible-part-1-db3241249b0d?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/db3241249b0d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[stress-testing]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Mon, 12 May 2025 00:51:50 GMT</pubDate>
            <atom:updated>2025-05-12T00:51:50.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uc1PC7oJRfF6Rq_2BaF-1g.png" /></figure><p>Stress testing is one of the most difficult test to conduct.</p><ul><li>set up the test environment</li><li>configure monitoring tools</li><li>stress your server with consistent requests in a certain time range</li><li>collect data</li><li>visualize data</li><li>analyze it</li></ul><p>it requires a variety of expertises from infrastructure to application.</p><p>Another problem I faced in my previous company was reproducibility.<br>A lot of configurations and the order of these settings tend to cause a human error, and it would end up taking too long than it should be.</p><p>In this article, I will try automating everything as follows</p><ol><li><strong>Automate test servers with terraform</strong></li><li>Set up Prometheus and Grafana to monitor nodes and containers</li><li>Initiate the test with Ansible</li><li>Check the result on Prometheus and Grafana</li></ol><p>This is the part 1. So I will go through provisioning process of infrastructure.</p><h3>Network Provisioning</h3><p>This is the my “network.tf” to set up VPC and import key data.</p><pre>module &quot;vpc&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version = &quot;5.21.0&quot;<br>  name    = &quot;jmeter-demo&quot;<br>  cidr    = &quot;10.0.0.0/16&quot;<br><br>  azs            = [&quot;us-east-2a&quot;, &quot;us-east-2b&quot;, &quot;us-east-2c&quot;]<br>  public_subnets = [&quot;10.0.101.0/24&quot;, &quot;10.0.102.0/24&quot;, &quot;10.0.103.0/24&quot;]<br>}<br><br>data &quot;aws_key_pair&quot; &quot;common&quot; {<br>  filter {<br>    name   = &quot;key-name&quot;<br>    values = [&quot;demo&quot;]<br>  }<br>}</pre><p>This “module” section is really similar to helm chart in kubernetes. Somebody on the internet already wrote a bunch of yaml files for you so that you can just specify some attributes and it’s good to go!<br>In this module, it creates vpc and subnets including internet gateway and route tables in a specified AZs.</p><p>The following “data” section is going to be used to fill an attribute in ec2 that we are going to create soon.</p><h3>EC2 Provisioning</h3><p>This time, I will create 2 instances, 1 Windows instance and 1 Linux instance. Depending on your situation, you can change the following configuration to create 2 or more Linux instances and the same for Windows.</p><h4>Linux EC2</h4><pre>resource &quot;aws_instance&quot; &quot;linux_test&quot; {<br>  instance_type               = &quot;t3.micro&quot;<br>  ami                         = data.aws_ami.amzn_linux_2023_latest.id<br>  subnet_id                   = module.vpc.public_subnets[0]<br>  associate_public_ip_address = true<br>  security_groups             = [aws_security_group.linux_test_sg.id]<br>  key_name                    = data.aws_key_pair.common.key_name<br>  tags = {<br>    usage : &quot;test&quot;<br>  }<br>}<br><br>resource &quot;aws_security_group&quot; &quot;linux_test_sg&quot; {<br>  name        = &quot;linux_test_sg&quot;<br>  description = &quot;Security Group for Linux test server&quot;<br>  vpc_id      = module.vpc.vpc_id<br><br>  ingress {<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>}<br><br>data &quot;aws_ami&quot; &quot;amzn_linux_2023_latest&quot; {<br>  most_recent = true<br>  owners      = [&quot;amazon&quot;]<br><br>  filter {<br>    name   = &quot;name&quot;<br>    values = [&quot;al2023-ami-2023*&quot;]<br>  }<br><br>  filter {<br>    name   = &quot;architecture&quot;<br>    values = [&quot;x86_64&quot;]<br>  }<br><br>  filter {<br>    name   = &quot;root-device-type&quot;<br>    values = [&quot;ebs&quot;]<br>  }<br><br>  filter {<br>    name   = &quot;virtualization-type&quot;<br>    values = [&quot;hvm&quot;]<br>  }<br>}</pre><p>This is a quite long file, but let’s walk through it step by step.</p><p>Let’s look at the last section first.</p><pre>data &quot;aws_ami&quot; &quot;amzn_linux_2023_latest&quot; {<br>  most_recent = true<br>  owners      = [&quot;amazon&quot;]<br><br>  filter {<br>    name   = &quot;name&quot;<br>    values = [&quot;al2023-ami-2023*&quot;]<br>  }<br><br>  filter {<br>    name   = &quot;architecture&quot;<br>    values = [&quot;x86_64&quot;]<br>  }<br><br>  filter {<br>    name   = &quot;root-device-type&quot;<br>    values = [&quot;ebs&quot;]<br>  }<br><br>  filter {<br>    name   = &quot;virtualization-type&quot;<br>    values = [&quot;hvm&quot;]<br>  }<br>}</pre><p>This data secation again is used for importing the existing resources. Specifically in this block, you are importing AMI that is officially available on AWS. Frankly said, it’s just Amazon Linux 2023. You need to filter this out by name and architecture and more details, but that’s the only purpose for this long block.</p><p>And now is the middle block.</p><pre>resource &quot;aws_security_group&quot; &quot;linux_test_sg&quot; {<br>  name        = &quot;linux_test_sg&quot;<br>  description = &quot;Security Group for Linux test server&quot;<br>  vpc_id      = module.vpc.vpc_id<br><br>  ingress {<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>}</pre><p>This more understandable. It is creating a security group for Linux server. Notice that we are referring to the vpc id from the module I mentioned before. It allows you to create the security group in the vpc.</p><p>Finally, the ec2 section.</p><pre>resource &quot;aws_instance&quot; &quot;linux_test&quot; {<br>  instance_type               = &quot;t3.micro&quot;<br>  ami                         = data.aws_ami.amzn_linux_2023_latest.id<br>  subnet_id                   = module.vpc.public_subnets[0]<br>  associate_public_ip_address = true<br>  security_groups             = [aws_security_group.linux_test_sg.id]<br>  key_name                    = data.aws_key_pair.common.key_name<br>  tags = {<br>    usage : &quot;test&quot;<br>  }<br>}</pre><p>It is referring to many attributes from other blocks including AMI, subnet, security group, and key_name. It asssembles all the resources you created together and locate your ec2 in the desired subnet.</p><h4>Windows EC2</h4><p>You can use the following format to create the Windows instance. I will omit the details because it follows the pretty much same format as Linux EC2.</p><pre>resource &quot;aws_instance&quot; &quot;windows_test&quot; {<br>  instance_type               = &quot;t3.micro&quot;<br>  ami                         = data.aws_ami.windows.id<br>  subnet_id                   = module.vpc.public_subnets[0]<br>  associate_public_ip_address = true<br>  security_groups             = [aws_security_group.windows_test_sg.id]<br>  key_name                    = data.aws_key_pair.common.key_name<br>  tags = {<br>    usage : &quot;test&quot;<br>  }<br>}<br><br>resource &quot;aws_security_group&quot; &quot;windows_test_sg&quot; {<br>  name        = &quot;windows_test_sg&quot;<br>  description = &quot;Security Group for Windows test server&quot;<br>  vpc_id      = module.vpc.vpc_id<br><br>  ingress {<br>    from_port   = 3389<br>    to_port     = 3389<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>}<br><br>data &quot;aws_ami&quot; &quot;windows&quot; {<br>  most_recent = true<br>  filter {<br>    name   = &quot;name&quot;<br>    values = [&quot;Windows_Server-2019-English-Full-Base-*&quot;]<br>  }<br>  filter {<br>    name   = &quot;virtualization-type&quot;<br>    values = [&quot;hvm&quot;]<br>  }<br>  owners = [&quot;801119661308&quot;] # Canonical<br>}</pre><h3>Provider Configuration</h3><p>Finally, you can specify the region as follows.</p><pre>provider &quot;aws&quot; {<br>  region = &quot;us-east-2&quot;<br><br>}</pre><h3>Let’s create it now!</h3><p>As long as you put all the configurations in one folder, you can write it in anyway with 1 file or multiple files.</p><p>I organized my files like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7u6JqznZN0zhpsnOtimPVw.png" /></figure><p>To create all the resources, you run the following command.</p><pre>terraform init<br>terraform apply</pre><h3>Conclusion</h3><p>We successfully created the test server, and the next time, I am going to configure Prometheus and Grafana in Kubernetes so stay tuned!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/0*OTj5TTAbpSW2Oj5e.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db3241249b0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linux free command Explained]]></title>
            <link>https://medium.com/@kikuchidaisuke.zr/linux-free-command-explained-a6f2e2d311e7?source=rss-12ae18480ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6f2e2d311e7</guid>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[free]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Daisuke Kikuchi]]></dc:creator>
            <pubDate>Mon, 05 May 2025 18:16:08 GMT</pubDate>
            <atom:updated>2025-05-05T18:16:08.687Z</atom:updated>
            <content:encoded><![CDATA[<p>Linux command “free” is super useful to check your memory status. It is one of the most common commands to be run for debugging as well.</p><p>However, do you fully understand the output of this command?</p><ul><li>What is swap?</li><li>What is buffer?</li><li>What is cache?</li><li>What is shared?</li><li>What is the difference between “free” and “available”?</li></ul><p>In this article, I am going to explore each section to the details and unveil the mystery so that you are comfortable with using command and identify the root cause of issues as quickly as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iaNxrZshdemAUP9HelU9sg.png" /></figure><h3>shared</h3><p>Let’s explore “shared” first. It refers to the memory used for tmpfs, POSIX shared memory and etc.</p><h4>tmpfs</h4><p>So, what is tmpfs. It is a temporary file system in memory not in a disk device (e.g. SSD). The biggest benefit is the speed. It is in memory, so way quicker than retriving/writing data from disk. But of couse, once you shut down your machine, it will be gone just as normal memory you think of.</p><p>One of the most common use-case for this tmpfs is volumes in Kubernetes or Docker.<br>Usually, you would use Persistent Volume to store logs from containers, but it is also common to use “emptyDir” as a volume to pass some data from one container to another.<br>In Kubernetes, you can configure your volume to use tmpfs like this.<br>(ref: <a href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir">https://kubernetes.io/docs/concepts/storage/volumes/#emptydir</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*1Y8s7ZNpDT6Azv_5" /></figure><p>So if your “shared” memory is huge, it is possible that you store too much data here.</p><h4>POSIX shared memory</h4><p>This is a linux kernel feature that allows you to share the memory in 2+ processes. The idea is really similar to a parent process and child threads where the heap memory is shared, but it is completely different implementation that is being used for this POSIX shared memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2CIFjsQCZ5nT9D3m" /><figcaption><a href="https://kuafu1994.github.io/MoreOnMemory/sharedMemory.html">https://kuafu1994.github.io/MoreOnMemory/sharedMemory.html</a></figcaption></figure><p>Nginx is well known for Non-Blocking IO architecture, but it uses this POSIX shared memory to communicate within worker processes.<br>(ref: <a href="https://blog.nginx.org/blog/inside-nginx-how-we-designed-for-performance-scale">https://blog.nginx.org/blog/inside-nginx-how-we-designed-for-performance-scale</a>)</p><p>This will also contribute to the high usage of shared memory.</p><h3>Buffer</h3><p>In free command, buffer and cache section are merged together, I will explain it separately since commands like vmstat displays them separately and they are completely different concepts.</p><p>So buffer in one sentence is “cache for file system metadata in block level”.<br>I will not go dive into file system details here but they are basically 2 types of data in a file system. Content of a file, and metadata (e.g. permission, timestamp).</p><p>This metadata can be about the file system itself or each file or even directory.<br>Obviously, they are being stored in a disk as well as a file content, so OS wants to cache it to speed up tasks.</p><p>Another important thing to think about is where to cache it. I said “it is a block level cache”, which means that OS will cache the data in in-complete format. it is not structure data that is going to be cached, collected from many blocks. OS caches it as a block and it still needs to be assembled together to be something meaningful.</p><h3>Cache</h3><p>Cache, on the other hand, is the cache for an actual file content not metadata, and it will be cached as an entire file content, not like spread block data. <br>More specifically this cache is called “OS Page cache”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/0*3I-tC65JB5ZpGES7" /><figcaption><a href="https://www.usenix.org/legacy/publications/library/proceedings/usenix01/full_papers/kroeger/kroeger_html/node8.html">https://www.usenix.org/legacy/publications/library/proceedings/usenix01/full_papers/kroeger/kroeger_html/node8.html</a></figcaption></figure><p>I want you to notice here something common between buffer and cache. They are both handled by OS not you. It is not you who try to increase or decrease buffer/cache. It’s OS.</p><p>So they are often said to be negligent. Once you have high usage of memory in a user space, those memory for buffer/cache can be reclaimed for you. So even if you see high number here, it is not big deal unless you are a kernel developer😄.</p><h3>available</h3><p>available is free — (buffer,cache). Simple right? Does it mean “buffer/cache” is included in “free”? <br>YES! and you already know why! Because it is reclaimable.</p><h3>Swap</h3><p>To understand “swap”, you need to understand virtual memory. Virtual memory is a technique to extend your memory using a disk (e.g. SSD). the hardware called MMU takes care of mapping from virtual memory address to physical address, and OS utilizes this and take a snapshot of your partial memory and save it to somewhere in a disk. When it is needed, it is going to be reloaded to the actual memory again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*99t0Tgldiip8Ul7b.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Virtual_memory">https://en.wikipedia.org/wiki/Virtual_memory</a></figcaption></figure><p>But it doesn’t help you without the downside. It is speed again. It is much slower to retrieve data from a disk than RAM. If your server is really slow, and swap is really high. You can think of the following reasons.</p><ol><li>Simply the entire memory was already eaten up, so OS had no choice to use virtual memory, and make it slow. (In this case, you need to look for another clue)</li><li>Your swapness is too high. You can specify how aggressively you want to use virtual memory. Obviously you want to use RAM as much as possible for quicker access. So if you have free space in RAM but swap is really high, reconfigure the swapness.<br>(ref: <a href="https://askubuntu.com/questions/157793/why-is-swap-being-used-even-though-i-have-plenty-of-free-ram">https://askubuntu.com/questions/157793/why-is-swap-being-used-even-though-i-have-plenty-of-free-ram</a>)</li><li>Disk I/O is slow. Because virtual memory uses a disk, it is no longer entirely memory issue, you can check if your disk is fine (For example, your disk is almost full, swap can be really slow.)</li></ol><h3>Conclusion</h3><p>In this article, I explored the details of each secion in free command with some examples and use-cases. Although you still need to identify which process and which business logic would cause the high memory usage, free command is a good place to start, and I hope it will help you understand deeper about this command and memory in general.</p><p>I will see you in the next article. Bye👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6f2e2d311e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>