<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akhil Ghatiki on Medium]]></title>
        <description><![CDATA[Stories by Akhil Ghatiki on Medium]]></description>
        <link>https://medium.com/@akhil-ghatiki?source=rss-f26fe154db02------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ekPxL0fG-G53bnwEuc5d9w.jpeg</url>
            <title>Stories by Akhil Ghatiki on Medium</title>
            <link>https://medium.com/@akhil-ghatiki?source=rss-f26fe154db02------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:25:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akhil-ghatiki/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A case study: Duplicate Entries in Event Processing with SQS + Lambda]]></title>
            <link>https://akhil-ghatiki.medium.com/a-case-study-duplicate-entries-in-event-processing-with-sqs-lambda-4f9c6c584a77?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f9c6c584a77</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Wed, 27 Mar 2024 14:05:42 GMT</pubDate>
            <atom:updated>2024-03-27T14:05:42.279Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*czHFddd01IQOci3Nk7et_Q.png" /></figure><h3>Issue Summary</h3><p>Duplicate entries were discovered in a Redshift database, spanning over several months. The duplicates were primarily caused by two factors:</p><ol><li>Multiple consumption of messages by the consumer lambda from the entries FIFO queue.</li><li>Duplicate requests made by the source teams.</li></ol><h4>Root Cause</h4><p>The root cause of the duplicate entries can be attributed to the behavior of AWS Lambda when processing messages from an SQS queue, especially when using the AWS SDK. Here’s how the process typically works:</p><ul><li>AWS Lambda receives a batch of messages from the SQS queue.</li><li>The Lambda function processes each message in the batch.</li><li>If the Lambda function completes successfully (no errors or exceptions), the AWS SDK automatically deletes the messages from the SQS queue.</li></ul><p>However, if the Lambda function encounters an error or exception during processing, the messages will not be automatically deleted. Instead, they will be retried by SQS according to the queue’s configuration. In this case, the duplicates occurred because of partially failed batches. When a message failed to process (e.g., due to a Redshift exception), the Lambda function did not delete the successfully processed messages before the failed one. As a result, these messages were reprocessed by subsequent invocations of the Lambda function, leading to duplicate entries.</p><p>For example, if the 6th message processing failed in a batch of 10, the current Lambda invocation exited without deleting the 5 messages processed successfully before the failed 6th one. These 5 messages were then visible in the queue for another Lambda invocation after the visibility timeout (10 minutes in this case).</p><h4>Conclusion</h4><p>To prevent duplicate entries in the future, it’s important to handle message processing errors more robustly. One approach could be to use a dead-letter queue (DLQ) in SQS to capture messages that cannot be processed after a certain number of retries. Additionally, ensuring that messages are deleted only after successful processing, even in the case of partial failures, can help mitigate the risk of duplicate entries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f9c6c584a77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running a new task using Terraform without a service in AWS ECS]]></title>
            <link>https://akhil-ghatiki.medium.com/running-a-new-task-using-terraform-without-a-service-in-aws-ecs-f815434d28c7?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/f815434d28c7</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Wed, 27 Mar 2024 13:45:34 GMT</pubDate>
            <atom:updated>2024-03-27T13:45:34.007Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ba16ze0EOIzo424FJEkmVg.png" /></figure><p>In Amazon Elastic Container Service (ECS), tasks are the fundamental unit of work that can be executed on a cluster. While services in ECS are used for long-running tasks, there are scenarios where you might need to run a task without associating it with a service. In this blog post, we’ll explore how to use Terraform to define and run a standalone ECS task without a service.</p><h3>First thing first</h3><p>Before proceeding, ensure you have the following:</p><ul><li>An AWS account with permissions to create ECS resources.</li><li>Terraform installed on your local machine.</li><li>Basic knowledge of ECS task definitions and Terraform.</li></ul><h3>Defining the ECS Task</h3><p>To define an ECS task without a service, we’ll first create a task definition. This task definition will specify the container image, resource requirements, and any other configuration needed for the task.</p><pre>resource &quot;aws_ecs_task_definition&quot; &quot;task_definition&quot; {<br>  family                   = &quot;my-task&quot;<br>  network_mode             = &quot;awsvpc&quot;<br>  cpu                      = 256<br>  memory                   = 512<br>  requires_compatibilities = [&quot;FARGATE&quot;]<br><br>  container_definitions = jsonencode([<br>    {<br>      name      = &quot;my-container&quot;<br>      image     = &quot;nginx:latest&quot;<br>      cpu       = 256<br>      memory    = 512<br>      essential = true<br>    }<br>  ])<br>}</pre><p>In this example, we define a task named “my-task” using the nginx container image running on AWS Fargate.</p><h3>Running the ECS Task</h3><p>To run the ECS task, we’ll use a null resource with a local exec provisioner in Terraform. This provisioner will execute the aws ecs run-task command to start the task.</p><pre>resource &quot;null_resource&quot; &quot;migration_task_run&quot; {<br>  depends_on = [<br>    aws_ecs_task_definition.task_definition    // to make sure that you run task only after creating the task definition<br>  ]<br><br>  provisioner &quot;local-exec&quot; {<br>    command = &lt;&lt;EOF<br>    aws ecs run-task \<br>      --cluster &lt;&lt;cluster_name&gt;&gt; \<br>      --task-definition &lt;&lt;task_definition_name&gt;&gt; \<br>      --count 1 --launch-type FARGATE \<br>      --network-configuration &#39;{     // This is required if you have chosen awsvpc in network config for your task definition. Else, this can be ignored<br>      &quot;awsvpcConfiguration&quot;: {<br>      &quot;assignPublicIp&quot;:&quot;DISABLED&quot;,<br>      &quot;securityGroups&quot;: [&quot;&lt;&lt;security_group&gt;&gt;&quot;],<br>      &quot;subnets&quot;: [&quot;&lt;&lt;your subnets&gt;&gt;&quot;]<br>      }<br>      }&#39;<br>EOF<br>  }<br>}</pre><h3>Use cases:</h3><p>Running a standalone ECS task without a service using Terraform provides a flexible way to execute short-lived tasks, such as batch processing or one-time operations. By defining the task with Terraform and using the aws ecs run-task command, you can easily manage and execute ECS tasks as part of your infrastructure automation workflows.</p><ol><li><strong>Automated Deployment Pipeline</strong>: Incorporate the execution of ECS tasks into your CI/CD pipeline. For example, after building and pushing a Docker image, use Terraform to update the ECS task definition and then automatically run the updated task.</li><li><strong>Scheduled Tasks</strong>: Schedule tasks to run at specific times or intervals using a cron job or another scheduling mechanism. Terraform can be used to define and manage these scheduled tasks, ensuring they are executed as intended.</li><li><strong>One-off Tasks</strong>: Run ad-hoc tasks that are not part of your regular ECS services. These tasks might be used for maintenance, data processing, or other temporary operations.</li><li><strong>Data base migration: </strong>To run DB migration jobs as part of your CI/CD pipeline.</li><li><strong>Environment Initialization</strong>: Use Terraform to initialize your ECS environment by running tasks that perform setup operations, such as database migrations, configuration loading, or resource provisioning.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f815434d28c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ECS Task deployment failed alert using Amazon EventBridge — Part 2 (Terraform code)]]></title>
            <link>https://akhil-ghatiki.medium.com/aws-ecs-task-deployment-failed-alert-using-amazon-eventbridge-part-2-terraform-code-965e0d9b1849?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/965e0d9b1849</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 16:31:21 GMT</pubDate>
            <atom:updated>2022-10-25T16:31:21.174Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS ECS Task deployment failed alert using Amazon EventBridge — Part 2 (Terraform code)</h3><p>Please read <a href="https://akhil-ghatiki.medium.com/aws-ecs-task-deployment-failed-alert-using-amazon-eventbridge-e6002d69563b">Part 1</a> for more context before you continue with this blog post</p><p>The <a href="https://akhil-ghatiki.medium.com/aws-ecs-task-deployment-failed-alert-using-amazon-eventbridge-e6002d69563b">part -1</a> talks about event pattern rule to filter events of a task failure. In this blog post, lets take a look at the terraform code for this implementation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*3bxxGwOVr2f0JPAXtiGyMA.png" /></figure><p>This blog post assumes you are quite familiar with the Terraform and you have some hands on experience with it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8115a3288c7e645627349a448b6f3a8/href">https://medium.com/media/f8115a3288c7e645627349a448b6f3a8/href</a></iframe><p>The above snippet creates the event rule as per the pattern we discussed in <a href="https://akhil-ghatiki.medium.com/aws-ecs-task-deployment-failed-alert-using-amazon-eventbridge-e6002d69563b">Part -1</a>.</p><p>And you can create the target to this event rule as below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd99e8ce707f07630c50814e4296e8d2/href">https://medium.com/media/fd99e8ce707f07630c50814e4296e8d2/href</a></iframe><p>Now, you can have your lambda triggered for the events coming from the SNS and the lambda can alert the teams (any communication medium that you use)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=965e0d9b1849" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ECS Task deployment failed alert using Amazon EventBridge]]></title>
            <link>https://akhil-ghatiki.medium.com/aws-ecs-task-deployment-failed-alert-using-amazon-eventbridge-e6002d69563b?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6002d69563b</guid>
            <category><![CDATA[aws-ecs-fargate]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Mon, 17 Oct 2022 12:47:23 GMT</pubDate>
            <atom:updated>2022-10-17T12:47:23.070Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances.</p><p>We have recently decided to move our deployments to ECS. As, the uncle Ben said — With great power, comes great responsibility — setting up the monitoring and alerts for this new implementation was quite interesting and this blog post deep dives into one specific alarm that we configured i.e., <strong>How do you get alerted when an ECS task fails during the deployment ?</strong></p><p>We had microsoft teams webhook in place, to which the alerts have to be sent when a failure / error occurs. This alert dump can be any other application in your case.</p><p>We have leveraged <strong>Amazon EventBridge</strong> rules for creating <strong>alerts</strong> when ever an ECS task fails. This gives us the opportunity to check what went wrong. Below is a high level block diagram of the implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*3bxxGwOVr2f0JPAXtiGyMA.png" /><figcaption>High level block diagram of alerting an AWS ECS task failure</figcaption></figure><p>And yes, I know what your eyes are looking for — The event rule pattern. Here is the pattern we used.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f59b33daef7aaaa787eb2eb7372c8df/href">https://medium.com/media/4f59b33daef7aaaa787eb2eb7372c8df/href</a></iframe><p>You can use any pattern for your rule. In the above example, we are filtering it based on the group key whose value is the service name. So, any event with the service name will be sent to the SNS. Below is a sample event that can help you with other attributes you might need for your rule pattern.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77b5635d6697593ffaebc4adfe8c800e/href">https://medium.com/media/77b5635d6697593ffaebc4adfe8c800e/href</a></iframe><p>God Speed !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6002d69563b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Probes —  Never let your production environment go down during deployment]]></title>
            <link>https://medium.com/swlh/kubernetes-probes-never-let-your-production-environment-go-down-during-deployment-cf3b85184eab?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf3b85184eab</guid>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Mon, 09 Aug 2021 04:45:06 GMT</pubDate>
            <atom:updated>2021-08-09T05:32:19.117Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bh7W4jSFJLPYLeNdaXaYMA.png" /><figcaption><a href="https://www.google.com/search?q=kubernetes+probes&amp;client=firefox-b-d&amp;sxsrf=ALeKk013Hdl_MPnEbbFFhgFyC4YdDr5jqA:1628439687035&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjas9XI6qHyAhVW9nMBHdDhDhMQ_AUoAnoECAEQBA&amp;biw=1680&amp;bih=849#imgrc=KeqXdywX-s5pRM&amp;imgdii=6wuFmXC48xjsMM">image source</a></figcaption></figure><h3>Kubernetes Probes — Never let your production environment go down during deployment</h3><p>Software development teams which work in an Agile mode often focus on speeding up the deployment process and these deployments are quite frequent.</p><h4><strong>First things first:</strong></h4><p>The reader is expected to have a very basic understanding of <a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/">Kubernetes(k8s)</a>, <a href="https://docs.docker.com/get-started/overview/">Docker</a> and <a href="https://www.docker.com/resources/what-container">containers</a>. This blog post assumes the deployment of the application is happening in a k8s cluster. The deployment strategy can be assumed to be in rolling update fashion.(It’s beyond the scope of this post).</p><p>Trying to keep it as simple as possible even if the reader is new to these terms.</p><h4>The need:</h4><p>With great power, comes great responsibility! When the builds and deployments are backed up by CI/CD pipelines, one has to be vigilant about the production environment not going down while deploying the changes. But we often end up in scenarios where the new instances of your application gets deployed and the production environment is puff !! … just gone !!</p><p>There could be many reasons why this happened — new instances get crashed, problem with re-balancing the data if application is state full etc.</p><blockquote>An ounce of prevention is worth a pound of cure<br> — Benjamin Franklin</blockquote><p>Ain’t it better and easier to stop a problem from happening than stopping it or correcting it after it has started ?</p><p>And that is where k8s probes come to rescue us from the chaos. There are three types of probes one can implement —<strong> </strong><strong>Liveness</strong>, <strong>Readiness</strong>, <strong>Startup</strong></p><h4>Common ground:</h4><p>This section tries to set a common ground for all the readers with a basic context to make the post more inclusive. (You can skip this section if you know below terms. Below definitions are taken from k8s documentation. )</p><p><strong><em>Kubelet</em></strong>: The kubelet is the primary “node agent” that runs on each node. It can register the node with the apiserver. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.</p><p><strong><em>Liveness Probe:</em> </strong>The <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/">kubelet</a> uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.</p><p><strong><em>Readiness Probe:</em> </strong>The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.</p><h4>Implementation:</h4><p>That was enough of a theory now ! Let’s take a look implementing these probes are testing them on a developer’s machine(typically referred as local environment).</p><p>Below is the summary of what we are going to do as part of implementation:</p><ol><li>Create an application.</li><li>Figure out the health end points of the application or create custom health end points.</li><li>Setting up k8s and minikube.</li><li>Build a docker image of this application.</li><li>Create a k8s deployment for above docker image.</li><li>Implement the probes in the deployment configuration.</li><li>Update the deployment.</li><li>Test the probes.</li></ol><p><strong>Create an application:</strong></p><p>Head over to <a href="https://start.spring.io/">Spring Initializr</a> and create a Spring boot application with build automation tool of your choice. Add only two dependencies — <strong>Spring Web<em> </em></strong>and<strong><em> </em></strong><strong>Spring Boot Actuator<em>. </em></strong>Actuator<strong><em> </em></strong>expose<em>s </em>default health end points through in the path — <strong>actuator/health</strong></p><p><strong>Note</strong>: Here a spring boot application is deliberately chosen to demonstrate as it has a slower startup time and this helps us in taking a look at what probes are doing from the logs.</p><p>Run the application in your local machine, fire up the browser and check the health status of your application using the endpoint below.</p><pre><a href="http://localhost:8060/actuator/health">http://localhost:8060/actuator/health</a></pre><p>This point should return HTTP.200 OK status with — <strong>{“status”:”UP”}</strong>. If not, check the port, it could be running on a different port. (My server.port is configured to 8060)</p><p><strong>Setting up k8s and minikube:</strong></p><p>Install docker desktop, k8s and minikube based on your OS.</p><p><em>Docker: </em><a href="https://www.docker.com/products/docker-desktop">https://www.docker.com/products/docker-desktop</a><em><br>k8s:</em> <a href="https://kubernetes.io/releases/download/">https://kubernetes.io/releases/download/</a><br><em>minikube: </em><a href="https://minikube.sigs.k8s.io/docs/start/">https://minikube.sigs.k8s.io/docs/start/</a></p><p>minikube is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes.</p><p><strong>Build a docker image of this application:</strong></p><p>Start docker desktop.<br>Start your minikube (Requires docker to be running in your desktop.)</p><pre>minikube start</pre><p><strong>Create a k8s deployment for above docker image.</strong></p><p>You can create deployments in minikube with out pushing your docker images to any public repository. Setting up an environment variable is the key for this.</p><pre>eval $(minikube docker-env)</pre><p>Go to the root folder of your application code and create a docker image. This creates a docker image with name spring-k8s-docker</p><pre>docker build -t spring-k8s-docker .</pre><p>You should now be able to see the image in the listing by running docker image ls</p><p>In the same root folder, now create a deployment file for the image we created using previous command with name deployment.yaml using below command. It creates a basic deployment configuration for you.</p><pre>kubectl create deployment spring-k8s --image=spring-k8s-docker --dry-run -o=yaml &gt; deployment.yaml</pre><p>You should be able to see a deployment.yaml file with below configuration in your code base.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  creationTimestamp: null<br>  labels:<br>    app: spring-k8s<br>  name: spring-k8s<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: spring-k8s<br>  strategy: {}<br>  template:<br>    metadata:<br>      creationTimestamp: null<br>      labels:<br>        app: spring-k8s<br>    spec:<br>      containers:<br>      - image: spring-k8s-docker<br>        name: spring-k8s-docker<br>        resources: {}<br>status: {}</pre><p>The imagePullPolicy pull policy has to be updated to never for this deployment to run in minikube. Update the above config file by adding imagePullPolicy under spec → containers → ..<br>Also, Update the <strong>replicas</strong> in above config file to<strong>3</strong>.</p><pre># rest of the config file remains same. Only add imagePullPolicy. <br>spec:<br>  containers:<br>  - image: spring-k8s-docker<br>    name: spring-k8s-docker<br>    imagePullPolicy: Never<br>    resources: {}</pre><p>Now apply this deployment with below command. This creates a deployment in spring-k8s name space of the minikube.</p><pre>kubectl apply -f deployment.yaml</pre><p>This creates the deployments and runs the pods immediately. To get the pods information, use the command below.</p><pre>kubectl get pods</pre><p>Once, you run this you should be able to see the out come similar to below in your terminal.</p><pre>➜spring-k8s kubectl get pods                   <br>NAME                          READY   STATUS    RESTARTS   AGE<br>spring-k8s-7fb746575f-499p2   1/1     Running   0          3s<br>spring-k8s-7fb746575f-7cvc7   1/1     Running   0          3s<br>spring-k8s-7fb746575f-q4qz2   1/1     Running   0          35s<br>➜spring-k8s</pre><p>Now, this is the problem us. What if the new pods are not ready to take the traffic, this is just running the pods right away, we don’t want that.This can bring down the production environment.</p><p>To look into the logs of any pod running above, run the below command.</p><pre>kubectl logs -f spring-k8s-7fb746575f-499p2</pre><p>Logs should look normal as they got deployed and running successfully.</p><p><strong>Implement the probes in the deployment configuration:</strong></p><p>Add probes to the deployment.yaml file. The updated deployment.yaml is below.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  creationTimestamp: null<br>  labels:<br>    app: spring-k8s<br>  name: spring-k8s<br>spec:<br>  replicas: 4<br>  selector:<br>    matchLabels:<br>      app: spring-k8s<br>  strategy: {}<br>  template:<br>    metadata:<br>      creationTimestamp: null<br>      labels:<br>        app: spring-k8s<br>    spec:<br>      containers:<br>      - image: spring-k8s-docker<br>        name: spring-k8s-docker<br>        imagePullPolicy: Never<br>        resources: {}<br>        readinessProbe:<br>          httpGet:<br>            scheme: HTTP<br>            path: actuator/health<br>            port: 8060<br>          failureThreshold: 3<br>          periodSeconds: 10<br>        livenessProbe:<br>          httpGet:<br>            path: actuator/health<br>            port: 8060<br>          failureThreshold: 3<br>          periodSeconds: 10<br>        startupProbe:<br>          httpGet:<br>            scheme: HTTP<br>            path: actuator/health<br>            port: 8060<br>          failureThreshold: 12<br>          periodSeconds: 10<br><br>status: {}</pre><p><strong>Note: </strong>The paths and ports used in each probe can you different according to what end point your applications is exposing for health checks. They could be custom end points as well.</p><p><strong>Update the deployment:</strong></p><p>Apply the above deployment configuration.</p><pre>kubectl apply -f deployment.yaml</pre><p><strong>Test the pods:</strong></p><p>Now, k8s pods will not run all the pods immeidately. It will wait for the start probe, checks the liveness using liveness probe configuration and similarly readiness.</p><p>Try getting the running pods and you can see the delayed start up of each pod.</p><h4><strong>Conclusion:</strong></h4><p>And yes !! your probes are configured. You should not see your production environment going down anymore during your deployments as the probes will not replace the existing healthy probes unless they find the new instances are ready and able to take the traffic.</p><p><strong>Few exercises for the reader:</strong></p><ol><li>Try misconfiguring the health check end points in the readiness probe (any other probe too). Check if the new pods are replacing the existing pods or not. Bring up the logs to see what is happening inside the the pods when the health check end points are misconfigured.</li><li>Try deleting a healthy pod kubectl delete pod &lt;pod_name&gt; . Checkout what liveness probe does for you:)</li><li>Play around with failureThreshold, periodSeconds and figure out what is the most optimal configuration based on your application’s startup time.</li></ol><p><strong>Clean up:</strong></p><p>Delete the name space. This deletes all the resources under that name space.</p><pre>kubectl delete -n default deployment spring-k8s</pre><p>Stop the minikube</p><pre>minikube stop</pre><p>This post is originally pushed on my blog. Checkout more at: <a href="https://akhil-ghatiki.github.io/#/">https://akhil-ghatiki.github.io</a></p><p><strong>Credits</strong>:<br><a href="https://www.linkedin.com/in/prabhu43/">Prabhu Jayakumar</a> for his valuable inputs.</p><p><strong>References</strong> : <br><a href="https://spring.io/guides/gs/spring-boot-kubernetes/">Getting Started | Spring Boot Kubernetes</a><br><a href="https://spring.io/guides/gs/spring-boot-docker/">Getting Started | Spring Boot with Docker</a><br><a href="https://stackoverflow.com/questions/42564058/how-to-use-local-docker-images-with-minikube">linux — How to use local docker images with Minikube? — Stack Overflow</a><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">Configure Liveness, Readiness and Startup Probes | Kubernetes</a><br><a href="https://developers.redhat.com/blog/2020/11/10/you-probably-need-liveness-and-readiness-probes#example_4__putting_it_all_together">You (probably) need liveness and readiness probes | Red Hat Developer</a><br><a href="https://docs.docker.com/get-started/kube-deploy/">Deploy to Kubernetes | Docker Documentation</a><br><a href="https://blog.container-solutions.com/kubernetes-deployment-strategies">Kubernetes deployment strategies</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf3b85184eab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/kubernetes-probes-never-let-your-production-environment-go-down-during-deployment-cf3b85184eab">Kubernetes Probes —  Never let your production environment go down during deployment</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Code smell Katas — Graceful exception handling]]></title>
            <link>https://akhil-ghatiki.medium.com/code-smell-katas-graceful-exception-handling-ceba4b3f50cc?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceba4b3f50cc</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[test-driven-development]]></category>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[code-smells]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Sun, 28 Feb 2021 09:35:49 GMT</pubDate>
            <atom:updated>2021-02-28T13:12:42.937Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHf7b20ffLSlwYLGUxcGuQ.jpeg" /><figcaption><a href="https://www.google.com/search?q=code+smell&amp;client=firefox-b-d&amp;sxsrf=ALeKk01nYBpHfZNW8qtTQAUdRTyaFt-vRQ:1614504534279&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwi8zYOKoozvAhVEyjgGHaLjBG4Q_AUoAXoECCAQAw&amp;biw=1680&amp;bih=866#imgrc=Xqd8SaMkuh461M">source</a></figcaption></figure><h3>Code smell Katas — Graceful exception handling</h3><p>In computer programming, a <strong>code smell</strong> is any characteristic in the source code of a program that possibly indicates a deeper problem. Determining what is and is not a code smell is subjective, and varies by language, developer, and development methodology. — wikipedia</p><p>Code smells are often symptoms of poor design or implementation choices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*a09RL66NCbzOhSh_RXRJeA.jpeg" /><figcaption><a href="https://www.google.com/search?q=code+smell&amp;client=firefox-b-d&amp;sxsrf=ALeKk01nYBpHfZNW8qtTQAUdRTyaFt-vRQ:1614504534279&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwi8zYOKoozvAhVEyjgGHaLjBG4Q_AUoAXoECCAQAw&amp;biw=1680&amp;bih=866#imgrc=Xqd8SaMkuh461M&amp;imgdii=UfXnfDbwpz8mRM">source</a></figcaption></figure><p>Let’s take up a Code Smell <a href="https://en.wikipedia.org/wiki/Kata">kata</a> in this blog post.</p><p>Consider a business scenario which has to make a HTTP call to a service to update a document if it exists or create new document if it doesn’t exist.</p><p>Below class has only one public method <strong><em>createOrupdateDocument</em></strong> and all other methods are private and have no visibility to other classes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25e89ddeb80f47ff17ddc2807ff1f614/href">https://medium.com/media/25e89ddeb80f47ff17ddc2807ff1f614/href</a></iframe><p>Let’s deviate a bit from code smells for now and take a look at above class. Why should this class take care about creating HttpClient ? How can you test this class for unit testing ? What can you test for unit testing ?</p><p>Consider moving the creation of HttpClient into a different class and passing it to this class as constructor parameter like below or as an autowired bean if you are using Spring framework.</p><pre>private HttpClient httpClient;<br><br>public CodeSmellHttpClient(HttpClient httpClient, String bearerToken) {<br>  this.httpClient = httpClient;<br>  this.bearerToken = bearerToken;<br>}</pre><p>This will help you in mocking the creation of HttpClient and verifying the calls of <strong><em>httpClient.execute(httpGetRequest), httpClient.execute(httpPostRequest),httpClient.execute(httpPatchRequest) </em></strong>respectively when get, post and patch requests are made through the private methods.</p><blockquote><strong>ProTip</strong>: If one is following test driven development, one will not end up into most of the code smells like in above implementation. Test driven development forces you to write cleaner code in many ways. All hail TDD !</blockquote><p>Now, just look into <strong><em>isDocumentExistingWithId(String documentId)</em></strong> method in above code, take a minute and try listing out the code smells in it.</p><p>What happens if there is an exception ?</p><p>Everything works smooth in a happy scenario here. But in case of exception, the method catches the exception and logs it, and then the if statement after the catch is checking the status code in response. One is not very far away form seeing a huge red log lines of NullPointerException in the console when an exception occurs.</p><p>Alright, now lets make some changes to tackle the exception scenario where there is an exception in this method. How about changing the method to below ? — moving the if condition into the try block.</p><pre>private boolean isDocumentExistingWithId(String documentId) {</pre><pre>  final String getTemplateUrl = baseUrl.concat(documentId);<br><br>  HttpGet httpGetRequest = new HttpGet(getTemplateUrl);<br>  httpGetRequest.setHeader(getAuthorizationHeader());<br>  HttpResponse response = null;<br><br>  try {<br>    response = httpClient.execute(httpGetRequest);<br>    LOG.info(response.toString());</pre><pre>    if(response.getStatusLine().getStatusCode()==HttpStatus.SC_OK){<br>      LOG.info(documentId + &quot;already exists&quot;);<br>      return true;<br>    }<br>  } catch (IOException e) {<br>    LOG.error(e.getMessage());<br>    e.printStackTrace();<br>  }<br>  LOG.info(documentId + &quot;does not exists&quot;);<br>  return false;<br>}</pre><p>This seems to mitigate the issue we were discussing a while ago. But wait, did we look into the caller method of this method ? Have a look into <strong><em>createOrUpdateDocument(String requestBody, String documentId) </em></strong>in the code example. (Sorry for making you scroll the page so many times :p)</p><p>The <strong><em>createDocument(newRequest) </em></strong>gets executed because we are returning <em>false</em> when the exception is caught in the <strong><em>isDocumentExistingWithId(String documentId) </em></strong>and this is invoking the createDocument method. We don’t want that to happen.</p><p>Take a minute now and think how can this be handled ! Don’t scroll down to answer before giving it a thought.</p><p>.</p><p>.</p><p>.</p><p>.</p><p>.</p><p>.</p><p>.</p><p>.</p><p>If throwing the exception is resonating in the corridors of your brain, yes, you are right !</p><p>Throw the exception in <strong><em>isDocumentExistingWithId(String documentId) </em></strong>instead of handling it<strong><em>.</em></strong>Let the caller method decide what to do with that exception. This is what I call handling the exception gracefully. Treat the exception like first class citizens in your implementations. Don’t ignore them. <em>Functional</em> <em>programming</em> does amazing job in treating exceptions as first class citizens. Will not discuss more about functional programming as that is way beyond the scope of this blog post.</p><blockquote><strong><em>ProTip</em></strong>: Treat exceptions like first class citizens in your implementations.</blockquote><p>Below is the implementation of throwing the exception as discussed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81c048fa41a5e68609e53be221a7f440/href">https://medium.com/media/81c048fa41a5e68609e53be221a7f440/href</a></iframe><p>Treat the exceptions good and they will treat you better.</p><p>I have intentionally left few code smells unaddressed. Drop them in the comments if you figure out any.</p><p>This post is originally pushed on my blog. Checkout more at: <a href="https://akhil-ghatiki.github.io/#/">https://akhil-ghatiki.github.io</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceba4b3f50cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One point stop for Installing SonarQube server in an AWS EC2 instance]]></title>
            <link>https://medium.com/swlh/one-point-stop-for-installing-sonarqube-server-in-an-aws-ec2-instance-2b287da39e42?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b287da39e42</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sonarqube]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Sun, 14 Jun 2020 14:14:53 GMT</pubDate>
            <atom:updated>2020-06-18T04:33:33.323Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvWYZwZGRnXJPtt-X7lOoA.jpeg" /></figure><p>Writing clean code is an art. It is a pathway towards software craftsman /craftswoman ship. But sometimes not every one is well equipped with this art. As the saying goes it takes 10,000 hours to master a skill. So, we need something that will enforce the developers of the team to write the code in a cleaner and secure way.</p><p>SonarQube is an amazing tool that helps in this regard. It helps us detect the code smells, potential bugs and security vulnerabilities in your code. It can integrate with your existing workflow to enable continuous code inspection across your project branches and pull requests.</p><p>SonarQube can be run in you local machine or as a docker container or can be hosted in a server. When the requirement is for having it for the whole team and planning to integrate in the CI pipeline, you have the only option of hosting it in the server. In this write out we will be walking through hosting it in an AWS EC2 instance.</p><h4><strong>First things first:</strong></h4><p>The reader is expected to have a very basic understanding of cloud, servers, CI/CD, CLI , database.Familiarity with AWS ecosystem is good to have. Trying to keep it as simple as possible even if the reader is new to these terms.</p><h4>Spawning up an AWS EC2 instance:</h4><p>Create a AWS EC2 instance. Just putting out the screen shot of configuration as the intention of this write out is not inclined more towards creating EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aibaO9eKDDu84QvcxGb36A.png" /></figure><p><strong>Important note : </strong>Be careful while choosing the Amazon Machine Image(AMI). Choose <strong>Amazon Linux AMI 2018.03.0 </strong>(ami-005956c5f0f757d37). This image includes PostgreSql which we are going to use in further steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIzcImaBAeW0_fjo-HEG5w.png" /></figure><p>Download the <strong>“</strong>.<strong>pem file” </strong>while<strong> </strong>creating the instance and save it in your machine. This is required to ssh into the instance in the further steps.</p><h4>Installing SonarQube server in the above created instance:</h4><p><strong>SSHing into the instance :</strong></p><p>Connect to the EC2 instance using a secure shell. Thanks to the jumpbox server that lets us do this.</p><pre>ssh -i &lt;&lt;path to your .pem file&gt;&gt; ec2-user@&lt;&lt;ip address of your EC2&gt;&gt;</pre><p><strong>Setup PostgreSql :</strong></p><p>We have to setup a database for sonarqube to save the report analysis. This helps in maintaining the report versions as well. The supported databases include Oracle, Microsoft SQL server and PostgreSQL. We will be going ahead with PostgreSQL here.</p><pre>sudo yum install postgre96 postgresql96-server </pre><p><strong>Note</strong> : Install the version higher than 9.3. The default version that gets installed if you do not mention the version in command will be 9.3 and this is not supported by SonarQube.</p><p>If you are going ahead with different AMI other than the above mentioned, checkout the supporting postgresql versions in <a href="https://www.postgresql.org/download/">this link</a>.</p><p>The below commands are self explanatory. Run them in this order:</p><pre>sudo service postgresql96 initdb //creating a new postgre cluster</pre><pre>sudo service postgresql96 start  // starting the postgre sevice</pre><pre>sudo passwd postgres  // changing the password of the default server</pre><pre>su — postgres // login with the new password</pre><pre>psql // start the shell</pre><p><strong>Create a user and database for sonar:</strong></p><pre>CREATE USER sonar WITH ENCRYPTED PASSWORD ‘sonar_password’;</pre><pre>CREATE DATABASE sonarqube;</pre><pre>GRANT ALL PRIVILEGES ON DATABASE sonarqube to sonar</pre><p><strong>Now we need to edit the authentication modes of postgresql.</strong></p><p>Open the file in below path in vim :</p><pre>sudo vi /var/lib/pgsql96/data/pg_hba.conf</pre><p>Change the modes as shown below: (Highlighted in block letters)</p><pre>TYPE DATABASE USER ADDRESS METHOD<br># “local” is for Unix domain socket connections only</pre><pre>local all sonar <strong>md5</strong></pre><pre>local all all peer</pre><pre># IPv4 local connections:</pre><pre>host all all <a href="http://127.0.0.1/32">127.0.0.1/32</a> <strong>md5</strong></pre><pre># IPv6 local connections:</pre><pre>host all all ::1/128 <strong>md5</strong></pre><p>Restart the postgreSQL server to have the new authentication modes applied.</p><pre>sudo service postgresql96 restart</pre><h4>Installing OpenJDK11</h4><p>We need JDK 11 or higher to run SonarQube 7.9 or more. The AWS EC2 instance comes with Amazon’s corretto distribution of java. But better stick to openJDK.</p><p>Download the zip in the instance:</p><pre>curl -O <a href="https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz">https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz</a></pre><p>Extract the zip:</p><pre>tar zxvf openjdk-11.0.1_linux-x64_bin.tar.gz</pre><p>Move the folder to your required location (This optional)</p><pre>sudo mv jdk-11.0.1 /usr/local/</pre><p>Change the access of the JDK folder:</p><pre>sudo chmod -R 755 /usr/local/jdk-11.0.1</pre><p>Add java home path:</p><pre>export JAVA_HOME=/usr/local/jdk-11.0.1</pre><pre>export PATH=$JAVA_HOME/bin:$PATH</pre><p>check the java version:</p><pre>java -version<br>This should show up the installed java version in the instance</pre><h4>Installing the SonarQube</h4><p>We are going to download the binaries and use them to install.</p><p><strong>Note</strong>: Please be careful about the edition that you are going to install. Sonarqube comes in community, developer and enterprise editions.</p><p>We are going to use community edition in this write out. More details can be found here in <a href="https://www.sonarqube.org/downloads/">this link</a>. Right click on the respective download button and choose copy link location if you aim to install other edition.</p><p>Download the binaries:</p><pre>wget <a href="https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.3.1.34397.zip">https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.3.1.34397.zip</a></pre><p>Unzip the binaries:</p><pre>unzip <a href="https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.3.1.34397.zip">sonarqube-8.3.1.34397.zip</a></pre><p>Move the file path to desired location:</p><pre>sudo mv -v <a href="https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.3.1.34397.zip">sonarqube-8.3.1.34397</a>/* /opt/sonarqube</pre><p>Create a group for sonar users:</p><pre>sudo groupadd sonar</pre><p>Create a user:</p><pre>sudo useradd -c “Sonar System User” -d /opt/sonarqube -g sonar -s /bin/bash sonar</pre><p>Activating the user with password:</p><pre>sudo passwd sonar</pre><p>Add sonar user to EC2 user:</p><pre>sudo usermod -a -G sonar ec2-user</pre><blockquote><strong>Exit from ec2-user bash and reconnect to the server to load new group for ec2-user</strong></blockquote><p>Change the ownership of all the sonar files to the sonar user:</p><pre>sudo chown -R sonar:sonar /opt/sonarqube</pre><p>Change file access privileges:</p><pre>sudo chmod -R 775 /opt/sonarqube</pre><h4>Configuring the SonarQube server:</h4><p>Set run as user:</p><pre>sudo vi /opt/sonarqube/bin/linux-x86–64/sonar.sh</pre><blockquote>Find the line RUN_AS_USER, uncomment it by removing the pound sign and enter sonar user as the value below</blockquote><pre>RUN_AS_USER=sonar</pre><p>Modify the sonar.properties to add the details of postgreSQL:</p><pre>sudo vi /opt/sonarqube/conf/sonar.properties</pre><p>Add jdbc user name and password:</p><pre>sonar.jdbc.username=sonar<br>sonar.jdbc.password=sonar_password</pre><p>Uncomment Postgres driver property.Remove current schema param if you are not using a custom schema for SonarQube database</p><pre>sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube</pre><p>set a path for dedicated volume with fast io for Elasticsearch data storage</p><pre>sonar.path.data=/path/to/fast/io/volume/data<br>sonar.path.temp=/path/to/fast/io/volume/temp</pre><p><strong>Start the server:</strong></p><pre>$/opt/sonarqube/bin/linux-x86–64/sonar.sh start</pre><p>If you want to log the server logs in the terminal go for:</p><pre>$/opt/sonarqube/bin/linux-x86–64/sonar.sh console</pre><p>Yikes !! Your sonar should be up now. You can access the sonarQube UI at</p><pre>http://&lt;&lt;EC2 instance public ip&gt;&gt;:9000/sonarqube</pre><h4><strong>Considerations</strong>:</h4><p>We have installed the PostgreSQL within the instance. This kind off makes it a stateful and introduces a single point of failure of the database if the EC2 instance goes down. Consider using AWS RDS to to configure in place of internal DB like we did here. This will help in persisting even if the instance goes down. Once the new instance connects back to db in AWS RDS, you have everything back.</p><h4><strong>Conclusion</strong>:</h4><p>Tried to cover all the mistakes which were encountered while installing SonarQube in EC2. Hope this helps someone not to recommit them!</p><p>God Speed!!!</p><p><strong>References used to write this:</strong></p><ul><li><a href="https://stackoverflow.com/questions/44658510/setting-up-sonarqube-on-aws-using-ec2">Setting up SonarQube on AWS using EC2</a></li><li><a href="https://medium.com/@mkaschke/how-to-install-sonarqube-on-aws-ec2-20e3a2fec634">How to install Sonarqube on AWS EC2</a></li><li><a href="https://medium.com/@harith.sankalpa/a-complete-guide-to-install-sonarqube-server-in-amazon-linux-ed1fee227d81">A Complete Guide to Install SonarQube Server in Amazon Linux</a></li><li><a href="https://docs.sonarqube.org/latest/setup/install-server/">Install the Server</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b287da39e42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/one-point-stop-for-installing-sonarqube-server-in-an-aws-ec2-instance-2b287da39e42">One point stop for Installing SonarQube server in an AWS EC2 instance</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Producer failure handling at multiple levels of Spring abstractions]]></title>
            <link>https://akhil-ghatiki.medium.com/kafka-producer-failure-handling-at-multiple-levels-of-spring-abstractions-e530edb02a6c?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/e530edb02a6c</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[failure]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[spring]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Sun, 15 Mar 2020 15:34:28 GMT</pubDate>
            <atom:updated>2020-03-16T06:27:13.701Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is co-authored by <a href="https://medium.com/u/946c3f2e0be3">swathi kurella</a>.</p><p>Excessive abstractions by frameworks can sometimes be painful for developers to utilize the underlying functionalities of core libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*VCNiUhJeQZ6W6J5ZPXevXw.png" /><figcaption><a href="https://www.google.com/search?q=spring+kafka&amp;client=firefox-b-d&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjP4Mb15ZzoAhVnyTgGHV_kCkMQ_AUoA3oECBAQBQ#imgrc=9okFBP6Us16_RM">source</a></figcaption></figure><p>In this write out, you are going to take away how to elegantly handle publisher failures while publishing to Kafka using Spring abstractions.</p><p>At ThoughtWorks, we have built a data ingestion platform as a part of a project. We chose Kafka as a streaming platform, Spring messaging framework to ingest payloads using spring-cloud-stream’s Kafka binder.</p><h4>First things first:</h4><p>The reader is assumed to have some basic knowledge of Spring framework, Kafka streaming platform, callbacks mechanism, asynchronous communication.</p><h4>Problem:</h4><p>While publishing the payloads to Kafka, we figured out that there were few false positives related to acknowledgement status on which we relied to assure the payload is published successfully. The application logs were showing that pay loads had been published successfully in spite of few failures and we never had access to failed payloads.</p><h4>What we did:</h4><p>To have some exception handling in place, we solely depended on the flag returned by <strong><em>“</em></strong><a href="https://github.com/spring-projects/spring-framework/blob/55418b256da906027cfcf7d3f1cb99cf6a56c93e/spring-messaging/src/main/java/org/springframework/messaging/support/AbstractMessageChannel.java#L121"><strong><em>send(…)</em></strong></a><strong><em>”</em></strong> method of Spring messaging to validate the status of payloads published to Kafka. <a href="https://github.com/spring-projects/spring-framework/blob/55418b256da906027cfcf7d3f1cb99cf6a56c93e/spring-messaging/src/main/java/org/springframework/messaging/support/AbstractMessageChannel.java#L121">Click here</a> to view the source code of “send()”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31de4380ff86cdbfa375a735963237fa/href">https://medium.com/media/31de4380ff86cdbfa375a735963237fa/href</a></iframe><p>Everything was smooth until we noticed a mismatch in published and consumed payloads count in Splunk logs. On digging deeper, we realized that few payloads failed to get published in spite of the success return from above <strong><em>send</em></strong> method.</p><h4>What this “<em>send”</em> does<strong>:</strong></h4><p>This method delegates the payload to the subscribed message handler which internally adds the message to a group. If this process succeeds, it returns success else an exception which means a failure. <a href="https://github.com/spring-projects/spring-framework/blob/3a0f309e2c9fdbbf7fb2d348be861528177f8555/spring-messaging/src/main/java/org/springframework/messaging/support/ExecutorSubscribableChannel.java#L96">Learn more</a>.</p><blockquote>Can we now rely just on this status to check if the message got published to Kafka ?</blockquote><p>The answer is NO. This method returns the success just on successful delegation of payload to handler but it helped us with a run time exception for non-recoverable errors.</p><h4>The tears of abstraction onion:</h4><p>The <strong><em>send</em></strong> method has abstracted the core implementation of the <strong><em>send method of KafkaTemplate </em></strong>in Spring<strong><em> </em></strong>Kafka which intern abstracted the <strong><em>send method of Producer</em></strong> in Apache Kafka. It was quite clear that there were three possibilities of using the <strong><em>send</em></strong> method. We observed that there were callbacks available at each inner layers i.,e in Spring Kafka and Apache Kafka and by now, we were sure that callbacks are going to be our rescuers.</p><h4>Handling producer failures in different frameworks:</h4><p><strong>Implementing Apache Kafka callback:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b261cb0a5120a20c01dd29c0ff632399/href">https://medium.com/media/b261cb0a5120a20c01dd29c0ff632399/href</a></iframe><p>This is the simplest implementation and its good enough to know that there is failure while producing the payload. It will not give the failed payload and impossible to investigate why it has failed. It just logs when ever there is a failure and one can get alerts from these logs using any logs analyzer tools.</p><p>This is best suited if the system is streaming behavioral data like clicks, logs, in-game player activity etc. from upstream systems as it is not a great loss in missing them for a moment. If it deals with data like inventory picture, air traffic data etc. which can potentially effect the near real time business, its big NO for this implementation as one cannot afford to loose investigating that data.</p><p><strong>Implementing Spring Kafka Producer Listener:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2e9fa3ea7136b77a23d92a44e61c7c8/href">https://medium.com/media/e2e9fa3ea7136b77a23d92a44e61c7c8/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7163278eddeb4978ff05f372ea1071ec/href">https://medium.com/media/7163278eddeb4978ff05f372ea1071ec/href</a></iframe><p>For the above reasons, we started looking more for another solution and we started looking into a higher level of abstraction. This implementation was decent enough to get the failure logs to get alerts along with the access to the failed payloads.</p><p>And we had one issue with this implementation. This demanded a lot of configuration in the existing code base as we have come a long way using the spring-messaging and spring cloud data stream frameworks. We cannot afford that now.</p><p><strong>Implementing in Spring cloud data stream:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ec5f6b1df30a77a65b8f69e50c6a49b/href">https://medium.com/media/6ec5f6b1df30a77a65b8f69e50c6a49b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a7b5c906524866276269a794cdf8ca3/href">https://medium.com/media/1a7b5c906524866276269a794cdf8ca3/href</a></iframe><p>We started looking into one more higher abstraction. We figured out that spring was using a producer callback. Thanks to Spring’s Auto-wiring by name. Creating the producer listener with the same name and overriding the implementation gave us what we wanted with minimal code changes.</p><p>Now the above created bean is injected directly to Kafka template as shown in below classes by the <strong>spring-cloud-stream-binder-kafka</strong> library</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01f808a053954e26f4e9bd6b51741e1a/href">https://medium.com/media/01f808a053954e26f4e9bd6b51741e1a/href</a></iframe><p><a href="https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/blob/master/spring-cloud-stream-binder-kafka/src/main/java/org/springframework/cloud/stream/binder/kafka/config/KafkaBinderConfiguration.java">See more for complete library source code</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/018773e7fe709380ca22759d91ef8710/href">https://medium.com/media/018773e7fe709380ca22759d91ef8710/href</a></iframe><p><a href="https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/blob/master/spring-cloud-stream-binder-kafka/src/main/java/org/springframework/cloud/stream/binder/kafka/KafkaMessageChannelBinder.java">See more for complete library source code</a></p><h4><strong>Conclusion</strong>:</h4><blockquote>It’s always helpful to be able to read framework code and be able to think like a framework developer.</blockquote><p>Thanks to <a href="https://medium.com/u/e6e95e5ec133">Unmesh Joshi</a> for the above line and his inspiration to dive into framework source codes.</p><h4>God Speed !!!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e530edb02a6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git hooks — The unsung rescuers]]></title>
            <link>https://akhil-ghatiki.medium.com/git-hooks-the-unsung-rescuers-44142c0f5c8c?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/44142c0f5c8c</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[git-hooks]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[unit-testing]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Sun, 12 Jan 2020 15:31:26 GMT</pubDate>
            <atom:updated>2020-01-12T15:36:21.243Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mM0-g_027EApr0jHxPVtGQ.jpeg" /><figcaption><a href="https://www.google.com/url?sa=i&amp;rct=j&amp;q=&amp;esrc=s&amp;source=images&amp;cd=&amp;ved=2ahUKEwjCyOnzr_7mAhXWzTgGHRGlBKwQjRx6BAgBEAQ&amp;url=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40chatie%2Fgit-scripts&amp;psig=AOvVaw1x4jLqobkWppH9-aqk3v58&amp;ust=1578929227805240">source</a></figcaption></figure><h3>Git hooks — The unsung rescuers</h3><p>What a git hook can do is constrained only by the developer’s imagination. Git hooks are one of the most productive hacks every software engineering team must possess.</p><h4><strong>First things first:</strong></h4><p>I assume the reader has some hands on experience with git and basic shell scripting. We will be going through only two types of git hooks in this write out. The intention is to introduced the reader to git hooks,not to explain details of different types of git hooks.</p><h4>The need:</h4><p>Keeping the application’s build pipeline always green is a skill. We all often see developers saying :</p><blockquote>“oh! the pull request build failed because of failing unit tests or integration tests”.</blockquote><blockquote>“Ahh !! damn I have push the data base password into git.”</blockquote><p>On a sincere note, committing passwords into git is a crime that a developer can potentially do unknowingly. I once read about a situation where a developer has committed the passwords of AWS instance which were eventually taken over by a hacker and whole AWS was compromised for the organization and used to mine cryptocurrency for 24hrs. We can stop this using a pre-commit git hook.</p><p>To fix the above scenarios developers have to revert that commit or push a new commit with fixed test suite or by removing the password that is pushed. This eventually disturbs the commit history of the application.And yes, there is a difference between a developer and a good developer. We can be good developers. We can handle these types of common mistakes even before we commit the code into our local machines.</p><h4>Dive in:</h4><p>We will be going through pre-commit hook and pre-push hooks only in this write out. If you are interested in knowing more about different types of git hooks, refer <a href="https://git-scm.com/docs/githooks">this link</a>.</p><p>A pre-commit hook script will be executed every time you commit the code in to the local machine where as a pre-push hook script will be executed every time you push the code to git.</p><h4>Pre-commit hook to detect passwords in staged files:</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e147379b49a4ab2fcaa535a013273ed/href">https://medium.com/media/6e147379b49a4ab2fcaa535a013273ed/href</a></iframe><p>We are using a regex here to detect any line of the type <strong>password:&lt;&lt;text&gt;&gt; </strong>in the staged files. (One needs to change this regex according to how passwords go into your code.)</p><h4>Pre-commit hook to abort commit for failing unit tests:</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b7b888e0b650fe3f7dc74ef2d7df376/href">https://medium.com/media/8b7b888e0b650fe3f7dc74ef2d7df376/href</a></iframe><pre>Of course, both the above implementations can go into one pre-commit git hook. Save the above file as pre-commit.sh</pre><h4>Pre-push hook to abort pushing for failing integration tests:</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef669390f2a7401a7356332224863f03/href">https://medium.com/media/ef669390f2a7401a7356332224863f03/href</a></iframe><pre>Save the above file as pre-push.sh</pre><p>Save these files in a folder “githooks” and run the below commands. So that any further changes can be made in these files and they will reflect in the git’s hooks.</p><pre>ln -s -f /Users/&lt;&lt;Path to your project folder&gt;&gt;/githooks/pre-commit ./.git/hooks/pre-commit</pre><pre>ln -s -f /Users/&lt;&lt;Path to your project folder&gt;&gt;/githooks/pre-push ./.git/hooks/pre-push</pre><p>God Speed !!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44142c0f5c8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debugging a low performance issue of a distributed system]]></title>
            <link>https://akhil-ghatiki.medium.com/debugging-a-low-performance-issue-of-a-distributed-system-c119b3018643?source=rss-f26fe154db02------2</link>
            <guid isPermaLink="false">https://medium.com/p/c119b3018643</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Akhil Ghatiki]]></dc:creator>
            <pubDate>Wed, 03 Jul 2019 18:07:02 GMT</pubDate>
            <atom:updated>2019-07-04T03:27:53.019Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XpXew5FWuUyfkpwkvvr1nw.png" /><figcaption><a href="https://www.google.com/search?q=distributed+systems&amp;client=firefox-b-d&amp;tbm=isch&amp;tbs=rimg:CbIKpufEvwbXIji1AhYmg9T9QzfrjUAdImo9i4idvTB5rGYZB3bDuaiFC8-q0I7kb9N6fFF9N7FPLBrr9aIQBfjcDyoSCbUCFiaD1P1DEW9OFKt0R16ZKhIJN-uNQB0iaj0Rx0u_1BUhXkYAqEgmLiJ29MHmsZhHBsf0pbn4KxioSCRkHdsO5qIULESnNMmaK2daiKhIJz6rQjuRv03oRiFoskpxMdWsqEgl8UX03sU8sGhGRheWzAZT4byoSCev1ohAF-NwPEc24PIqIIPSW&amp;tbo=u&amp;sa=X&amp;ved=2ahUKEwiMk5jmqpnjAhWSSH0KHRKxDXAQ9C96BAgBEBw&amp;biw=1680&amp;bih=870&amp;dpr=2#imgrc=z6rQjuRv03opMM:">image source</a></figcaption></figure><h3>Debugging the performance issue of a distributed system</h3><p>Debugging the performance of a distributed system is always a pain. But on the other side of this coin, you get a lot of learning, knowing the system better and this helps in building an architectural mindset.</p><p>Recently, I have spent close to 2 weeks on debugging a performance issue of a distributed system our team was developing. I am sharing my learning from this period in this blog post.</p><p><strong>First things first:</strong></p><p>I assume the reader has some knowledge of micro services, distributed systems and how critical would a performance issue of a distributed could be. From here on, “system” or “DS” represents a distributed system.</p><p><strong>Do not go gentle into that good night….. :</strong></p><p>Read this poem by Dylan Thomas <a href="https://poets.org/poem/do-not-go-gentle-good-night">here</a> (Its my favourite poem). But yeah, coming back to debugging, I would say DO GO gentle into “this” good night. Don’t just jump into the implementation and crave for the fix of the issue. This is the first mistake one can do. Without analyzing the existing implementation of the system, the developer just dives to fix it. When you know the system better and if you have a clear understanding of the performance issue, you can have a slight guess about where the issue could be and what could be the bottleneck of this system. So, first, analyze the existing system.</p><p><strong>Capture the metrics of the existing system:</strong></p><p>This is crucial. Unless you know how well/bad the existing system is performing, you can’t proceed further. Leverage the power of logs and profiles of the system to capture the metrics. These metrics could be about the I/O operations, thread pool profiling, publishing data to another system or subscribing data from another system or could be as simple as a small API call. Keep a note of memory usage and CPU usage of the system. It would be great if you are using a tool like Splunk to get insights from your logs. Dashboards built using the logs of a DS are helpful when you are dealing with this type of issue. They drastically reduce the time you put in. Invest time in developing dashboards (Ex: Splunk Dashboards) while developing itself. They will save a lot of time later. Proceed further once you are quite clear about the existing performance.</p><p><strong>Are you messing with default configurations ?</strong></p><p>When you are dealing with a distributed system, at times we configure a few parts of this system with default configurations. Let’s take a small use case. If you are using Kafka or RabbitMQ as a messaging broker in your system, in general, we go ahead with a few default configurations. For example, some configurations like fetch.min.bytes, max.poll.records, fetch.max.wait.ms etc of Kafka should be left to default configurations unless you are very sure that it is not your system that is the bottleneck for the performance. Default configurations are default for a reason, they are not just random figures. The team while developed Kafka might have tested it rigorously and came up with these defaults. So, think twice before changing the defaults.</p><p><strong>Chalk out the architecture:</strong></p><p>This helps a lot. Doesn’t your office have whiteboards? No big deal, grab a paper and sketch your DS. Try to include every minute detail of the system. The key aspects would be drawing out, which part of the system is communicating with which part. Include all the API calls happening, the messaging brokers if any. The more detailed your sketch is, the less time you are going to spend solving the issue. From this sketch mark out the points which could be affecting your system.</p><p><strong>It’s not always the game of instance count:</strong></p><p>Yes, when capturing the metrics of a DS. Stick on to one instance of the system . Spawning up multiple instances of the system to capture metrics might mislead the decision. Performance should always be talked per instance. So, fix the problem per instance before you scale up the count.</p><p><strong>Divide and Conquer:</strong></p><p>Here lies the key aspect of debugging the performance of a DS. Divide the whole system into smaller chunks (of course in your mind. That is why I asked to chalk it out in the previous section). Now, decommission one part at a time in your DS either by bringing down the service or just commenting out the code. Capture the metrics after that. Loop this in until you narrow down the problem to a particular part of your system. Now, conquer the issue.</p><p><strong>God Speed !!!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c119b3018643" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>