<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ruween Iddagoda on Medium]]></title>
        <description><![CDATA[Stories by Ruween Iddagoda on Medium]]></description>
        <link>https://medium.com/@ruweeniddagoda?source=rss-b082bd7b5f97------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TDX40EZVf4yu4toa_T82Jg.png</url>
            <title>Stories by Ruween Iddagoda on Medium</title>
            <link>https://medium.com/@ruweeniddagoda?source=rss-b082bd7b5f97------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:00:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ruweeniddagoda/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Proposal for Migrating to AWS EKS with Karpenter and KEDA for Cost Reduction and Improved…]]></title>
            <link>https://ruweeniddagoda.medium.com/proposal-for-migrating-to-aws-eks-with-karpenter-and-keda-for-cost-reduction-and-improved-2ea86456ebf6?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ea86456ebf6</guid>
            <category><![CDATA[ek]]></category>
            <category><![CDATA[solutions-architecture]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[ec]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 18:07:41 GMT</pubDate>
            <atom:updated>2026-08-08T18:07:41.639Z</atom:updated>
            <content:encoded><![CDATA[<h3>Proposal for Migrating to AWS EKS with Karpenter and KEDA for Cost Reduction and Improved Scalability</h3><blockquote>This is a proposal I came up with after having an existential crisis with myself one night, kinda looking back at the architecture we had at one of the companies I worked for, its shortcomings, and thinking about how I’d rebuild it if I were to design it again based on the requirements we have now.</blockquote><blockquote>It’s also somewhat of a self-critique of the architecture we had back then. Even this new proposed iteration probably has its own faults that I might realize later and end up revisiting and iterating on in the future. But I think that’s kind of the point; reflecting, reviewing, and rethinking things is what helps us grow.</blockquote><blockquote>I spent a whole night, all the way until morning, trying to come up with this setup. I even had to convince Claude to stop constantly suggesting “best practices” when there are clearly workarounds that are just as valid depending on the context. Anyways might be a good read.</blockquote><blockquote>Also, this only includes part of the proposal. While copying over the context from the Confluence document I wrote back then, I realized there are some sensitive details that I can’t really share, since they essentially reveal our entire network topology.</blockquote><blockquote>The summary, though, is that the original architecture had several unnecessary network hops that had been in place even before I joined, and we never properly addressed them. These were adding roughly 20–30ms of additional latency, and potentially even more depending on the context, without providing any meaningful benefit.</blockquote><h3>Project Greenfield</h3><blockquote>A clean-slate architecture for the workload we run today</blockquote><h3>TL;DR</h3><p>Our current ECS-on-EC2 platform has two structural limitations that cost us money continuously:</p><ol><li><strong>ECS cannot rebalance task placement after the initial scheduling decision.</strong> Instances drift into imbalanced utilization (some packed, others nearly empty) and stay that way until the under-utilized ones happen to drain naturally. We have a documented example of an m6a.large running a single task with 94% of CPU and 97% of memory sitting idle, while sibling nodes are at 88-94% utilization.</li><li><strong>Our queue-driven worker scaling chains a custom Lambda → custom CloudWatch metric → ECS target tracking policy.</strong> The pipeline works but is structurally slow (90s-3min response time) and target tracking cannot natively scale to zero, therefore we bypass it by the lambda function logic.</li></ol><p>Both problems disappear with EKS + Karpenter + KEDA. Karpenter continuously consolidates underutilized nodes. KEDA polls queue depth directly every 15 seconds and natively scales deployments to zero. Together with topology consolidation (one cluster, fewer load balancers, HTTP API v2) this is projected to reduce our infrastructure cost for these workloads by ~49%, from roughly <strong>$5,040/mo to $2,580/mo — a saving of ~$2,460/mo or ~$29,500/year.</strong></p><p>These savings assume our services are already correctly sized. We are not counting “right-size everything” as a savings vector. The reduction comes from architectural change.</p><h3>Current state</h3><p>We run 6 ECS clusters in eu-west-2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8CS9IOwc9yczYZo--p9JQ.png" /></figure><p>Supporting infrastructure for this fleet:</p><ul><li>4 ALBs (csm-bff, csm-bfp, csm-external, csm-internal)</li><li>9 REST API Gateway v1 instances (bff, bfp, webhooks, etc.)</li><li>2 WebSocket APIs (prod-websocket, canvas-collab)</li><li>3 NAT gateways</li><li>Custom Lambda + custom CloudWatch metric (taskUtilization) + ECS target tracking for worker service scaling</li></ul><p>Worker scaling currently uses a custom Lambda that runs on a schedule, reads CloudWatch metrics for each queue (ApproximateNumberOfMessagesVisible and, where relevant, ApproximateAgeOfOldestMessage), computes a custom utilization metric (taskUtilization) that accounts for both queue depth and message age, and publishes it back to CloudWatch. Each worker service then has an ECS Service Auto Scaling target tracking policy that scales the desired count against this custom metric. The approach is more sophisticated than simple queue-depth scaling — it accounts for stale messages, not just message count — but it inherits the latency of every CloudWatch metric hop in the chain.</p><h3>The case for Karpenter</h3><h3>How ECS places tasks today</h3><p>ECS task placement is a two-layer system designed for predictability:</p><ul><li><strong>The ASG</strong> defines what instance types exist. Our launch templates declare specific types (m6a.large, c6a.xlarge, etc.) and the ASG launches only those.</li><li><strong>The ECS Capacity Provider</strong> decides when the ASG should scale, based on a “capacity reservation” metric — essentially, what fraction of the cluster’s CPU/memory is reserved by running tasks.</li><li><strong>ECS task placement strategies</strong> pick a node when a task is launched, typically via binpack on CPU or memory.</li></ul><p>These three layers are deliberately decoupled. The consequence: <strong>ECS makes a placement decision once, when a task is launched, and never reconsiders.</strong></p><h3>Why this leaves money on the table</h3><p>Over time, as services scale up and down independently and tasks come and go, the distribution drifts. Some nodes become heavily packed, others become nearly empty, and ECS has no mechanism to rebalance.</p><p>Below is a real snapshot from our internal-prod-ecs-cluster capacity provider, showing 5 m6a.large nodes (each providing 2 vCPU and ~7.7 GB usable):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aD6S8lZ7zC3AdkyxeU4e4g.png" /></figure><p>Node 1 is running a single task with the rest of the box sitting idle. That instance costs us approximately $71/mo to hold one container. The task on node 1 could comfortably fit on node 2 (which has 0.75 vCPU and 5.1 GB free), but ECS doesn’t reconsider placement after initial scheduling. The instance will remain until the task on it terminates naturally — which could be hours, days, or weeks depending on the service’s deployment cadence.</p><p>This pattern repeats across the fleet — Not because the services need less compute, but because the placement has drifted and ECS won’t rebalance.</p><h3>What Karpenter does differently</h3><p>Karpenter is one component that owns both node provisioning and continuous bin-packing. Instead of “ASG launches a declared type, ECS asks for more nodes,” Karpenter watches pending pods and existing nodes, and makes placement and provisioning decisions in a single integrated loop.</p><p>For the scenario above, Karpenter would run a continuous consolidation loop (every ~30 seconds by default) that asks:</p><ol><li><strong>Can this node be terminated entirely?</strong> Karpenter looks at node 1, sees one pod consuming ~125m CPU and ~200 MB. It checks whether that pod could be rescheduled onto any other node. Node 2 has 0.75 vCPU and 5.1 GB free — easily fits. So Karpenter cordons node 1, evicts the pod gracefully (respecting any PodDisruptionBudget), Kubernetes places it on node 2, and node 1 terminates.</li><li><strong>Can this node be replaced with a cheaper one?</strong> If the workload on a node genuinely needed an m6a.large’s resources, Karpenter would still check whether a smaller instance type (t3.medium, t3.large) would fit and replace the node with that.</li></ol><p>Both checks run continuously on every node. Nodes that drift into low utilization get consolidated automatically.</p><p>Critically, Karpenter also makes better <strong>initial</strong> node choices because it picks instance types per-pod-batch rather than per-launch-template. Given a NodePool that allows (for example) m6a.large, m6a.xlarge, m6i.large, c6i.large, c6i.xlarge, t3.large, t3.medium across all three AZs, Karpenter picks the cheapest combination that fits the pending pods at provision time. No more ASGs locked to a single declared instance type.</p><h3>Cost impact of Karpenter alone</h3><p>Even with all services correctly sized:</p><ul><li><strong>Continuous consolidation</strong> reclaims the waste shown in the screenshot above. Our internal team estimate is that 25–35% of HTTP node-hours are attributable to placement drift that would consolidate away under Karpenter.</li><li><strong>Per-pod instance type selection</strong> improves match between workload and instance, typically 5–15% additional savings on top.</li><li><strong>Faster scale-down</strong> during off-peak hours — Karpenter consolidates eagerly when load drops, where ECS only scales down when the capacity reservation metric falls.</li></ul><p>Combined effect on HTTP compute: estimated <strong>30–40% reduction</strong> ($1,000–1,200/mo on the current $3,000/mo HTTP cluster compute spend), purely from architectural change, before any right-sizing.</p><h3>The case for KEDA</h3><h3>How we scale workers today</h3><p>We’ve built a thoughtful worker scaling pipeline:</p><ol><li>A custom Lambda runs on a schedule and queries CloudWatch for each queue’s ApproximateNumberOfMessagesVisible and (where relevant) ApproximateAgeOfOldestMessage.</li><li>The Lambda computes a custom taskUtilization metric per service that combines queue depth with message age — so we scale not just on how many messages are queued, but on how stale they&#39;re getting.</li><li>The Lambda publishes taskUtilization back to CloudWatch as a custom metric.</li><li>Each worker service has an ECS Service Auto Scaling target tracking policy configured against this custom metric. When taskUtilization exceeds the target value, target tracking calls UpdateService to increase the desired count.</li></ol><p>This is a smarter design than basic queue-depth scaling — accounting for message age catches the case where a queue with 10 messages that have been waiting 5 minutes is more urgent than a queue with 100 messages that just arrived. The Lambda + custom metric pattern lets us encode that domain knowledge.</p><p>But the approach inherits the latency of every CloudWatch metric hop in the chain, and it cannot cleanly scale to zero.</p><p><strong>The latency stack:</strong></p><ul><li><strong>SQS metric publishing latency.</strong> ApproximateNumberOfMessagesVisible is published at 1-minute granularity by SQS.</li><li><strong>Lambda invocation cadence.</strong> Our Lambda runs on a CloudWatch Events / EventBridge schedule, typically every 60 seconds. Worst case, a message arrives just after a Lambda run and waits 60+ seconds before the next evaluation.</li><li><strong>Custom metric publishing latency.</strong> Once the Lambda computes taskUtilization, it publishes to CloudWatch — there&#39;s a propagation delay before the metric is available for target tracking to read.</li><li><strong>Target tracking evaluation period.</strong> ECS target tracking policies typically evaluate metrics over a 1-minute period and require sustained breach of the target value before triggering. This adds another 60–180 seconds of decision latency.</li><li><strong>Task launch.</strong> Once the desired count changes, ECS still has to place the task and (if needed) trigger Capacity Provider to launch an EC2 instance. Another 60–90 seconds.</li></ul><p><strong>Total end-to-end latency from message arrival to first processing: typically 2–4 minutes.</strong></p><p><strong>The scale-to-zero limitation.</strong> Target tracking policies don’t work well at zero desired count. When desiredCount=0, there are no running tasks to evaluate utilization against — target tracking has no data to make a scaling decision. We work around this with separate “wake-up” Lambda logic that watch for any queue activity and bump desired count from 0 to 1, but that adds another moving piece and reintroduces race conditions.</p><p><strong>The operational tax.</strong> Every worker service needs:</p><ul><li>A Lambda function (or a multi-tenant Lambda with per-service configuration)</li><li>IAM roles for CloudWatch read + custom metric publish + ECS update</li><li>A target tracking policy configuration per service</li><li>Per-service tuning of the taskUtilization formula and target value</li></ul><p>This is real ongoing engineering work to maintain, and the configuration lives outside the service repos themselves.</p><h3>How KEDA changes this</h3><p>KEDA (Kubernetes Event-Driven Autoscaler) is a Kubernetes-native controller that replaces the entire Lambda + custom metric + target tracking chain with direct queue polling.</p><p>For each worker service, you write a ScaledObject CRD that lives alongside the service&#39;s Deployment in the same Git repo:</p><p>minReplicaCount: 0 maxReplicaCount: 30 pollingInterval: 15 cooldownPeriod: 300 triggers: - type: aws-sqs-queue metadata: queueURL: &lt;queue-url&gt; queueLength: &quot;5&quot; awsRegion: eu-west-2</p><p>KEDA’s SQS scaler polls the queue directly using the AWS SDK every 15 seconds (configurable down to 10s). When queue depth crosses the threshold, KEDA triggers the HPA, which scales the Deployment. Karpenter provisions a new node if needed (~30–60s for spot capacity).</p><p><strong>Replicating our </strong>taskUtilization<strong> logic in KEDA.</strong> KEDA&#39;s built-in SQS scaler only reads queue depth, not message age. If we want to preserve our domain-specific &quot;scale on staleness&quot; behavior, we have two clean options:</p><ol><li><strong>Use a KEDA </strong>aws-cloudwatch<strong> scaler against the existing </strong>taskUtilization<strong> metric.</strong> Our Lambda keeps running and publishing the metric, but KEDA reads it directly instead of target tracking. We get faster polling (15s vs target tracking&#39;s 60s evaluation) but still depend on the Lambda. Acceptable as a transition state.</li><li><strong>Use KEDA’s multiple-trigger composition.</strong> A single ScaledObject can have multiple triggers (e.g., one SQS scaler on queue depth, plus a CloudWatch scaler on age). KEDA scales to the maximum desired replicas across all triggers. We can encode &quot;scale up if either depth or age crosses threshold&quot; without a Lambda at all. This is the long-term clean target.</li></ol><p>Either way, we delete the target tracking complexity and gain native scale-to-zero.</p><p><strong>Total end-to-end latency from message arrival to first processing: 15–25 seconds typical, 60–90 seconds cold start.</strong> That’s roughly 10× faster than our current setup.</p><p><strong>Scale-to-zero is native.</strong> Setting minReplicaCount: 0 is all that&#39;s required. KEDA&#39;s controller watches the queue even when zero pods are running and brings up the first pod when activity appears. No race conditions, no companion mechanisms.</p><p><strong>Configuration lives with the service.</strong> The ScaledObject YAML sits in the service&#39;s Helm chart or Kustomize overlay, next to the Deployment definition. Developers can see and modify scaling behavior in the same place they modify the service. No separate Lambda repo, no separate IAM role per service, no out-of-band tuning workflow.</p><h3>Cost impact of KEDA</h3><p>For our worker clusters specifically:</p><ul><li><strong>event-handler cluster</strong> with KEDA scale-to-zero, services with empty queues actually drop to zero pods. Realistic reduction in worker compute: 40–60%. (These are guestimated by AI)</li><li><strong>custom-integration cluster</strong> KEDA does this systematically for all 270+ worker services with the safety guarantees we currently can’t get from target tracking. Realistic reduction: 50–70%. (These are guestimated by AI)</li></ul><p>Combined: estimated <strong>50–60% reduction in worker compute</strong> (~$300–350/mo savings on the current ~$590/mo combined worker compute), plus elimination of the custom Lambda + custom metric + target tracking pipeline.</p><p>The Lambda + custom metric infrastructure becomes unnecessary. We currently maintain a Lambda deployment, IAM roles for metric publish + ECS update, per-service target tracking policy configurations, and the taskUtilization formula tuning. That&#39;s recurring operational overhead that disappears — KEDA&#39;s per-service configuration lives in the service&#39;s Helm chart, version-controlled alongside the code.</p><p>Some of the stats given abrove on saving percentages are guestimates by AI which might not be accurate. Point of focus should be on the theory behind it.</p><h3>Proposed greenfield architecture</h3><p>Single EKS cluster running in eu-west-2 across three AZs, with two Karpenter NodePools and KEDA for all scaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4dRY7BXJSGZutZJw8eabQ.png" /></figure><h3>Compute topology</h3><ul><li><strong>HTTP NodePool</strong> — x86 on-demand only (per current platform constraints). Allowed instance families: m6a, m6i, m7i, c6i, c7i in sizes large, xlarge, 2xlarge. Karpenter picks per-pod-batch. Consolidation policy: WhenEmpty (conservative — don&#39;t shuffle live HTTP pods aggressively).</li><li><strong>Workers NodePool</strong> — x86, 80% spot / 20% on-demand split. Allowed instance families: m6a, m6i, c6i, t3, t3a in sizes large through 2xlarge. Karpenter picks per-pod-batch. Consolidation policy: WhenUnderutilized (aggressive — workers tolerate churn).</li></ul><p>A 1-year Compute Savings Plan covers the on-demand baseline (~$120/mo offset on committed capacity).</p><h3>Edge and traffic flow</h3><ul><li><strong>API Gateway</strong> — migrate the 9 v1 REST API Gateways (bff, bfp, webhook listeners, etc.) to HTTP API v2 where v1 features are not in use. Keep the 2 WebSocket APIs as v2 WebSocket (no change — that’s already the only WebSocket option).</li><li><strong>Internal NLB</strong> — single internal NLB serving as VPC Link target for the WebSocket APIs. Targets the websocket-bff pods directly via IP target type.</li><li><strong>Internal ALB</strong> — single internal ALB serving as VPC Link target for HTTP APIs. Host-based routing distributes traffic to all 80 HTTP services in the cluster. AWS Load Balancer Controller manages target group bindings.</li><li><strong>No public load balancers.</strong> API Gateway is the only public surface. All ALB and NLB endpoints are private.</li></ul><h3>Scaling and deployment</h3><ul><li><strong>KEDA</strong> scales all services. HTTP services use the aws-cloudwatch scaler against ALB RequestCountPerTarget metric. Worker services use the aws-sqs-queue scaler against queue depth.</li><li><strong>HPA</strong> runs underneath KEDA (KEDA creates HPAs to do the actual scaling).</li><li><strong>Argo Rollouts</strong> handles deployments using blueGreen strategy. Traffic switches at the target group level in sub-30-second windows. Rollback is a single metadata flip.</li></ul><h3>Supporting platform</h3><ul><li><strong>AWS Load Balancer Controller</strong> manages ALB/NLB target group sync with pod IPs.</li><li><strong>Pod Identity</strong> (via eks-pod-identity-agent) handles per-service IAM roles.</li><li><strong>VPC Endpoints</strong> for ECR, S3, SQS, Secrets Manager, CloudWatch Logs, STS, execute-api — reduces NAT data charges.</li><li><strong>Container Insights</strong> enabled cluster-wide (currently disabled on all our ECS clusters).</li><li><strong>ArgoCD or Flux</strong> for GitOps deployment.</li><li><strong>3 NAT gateways</strong> unchanged from current setup.</li></ul><h3>Cost comparison</h3><p>All figures monthly, eu-west-2 pricing, assuming all services are correctly sized in both scenarios (no rightsizing dividend counted).</p><pre>+--------------------------------------------------------------+---------------+---------------+----------+<br>| Category                                                     | Current (ECS) | Proposed (EKS)| Saved    |<br>+--------------------------------------------------------------+---------------+---------------+----------+<br>| HTTP compute (x86 on-demand)                                | $3,007        | $1,800        | $1,207   |<br>| Worker compute (80/20 spot, KEDA scale-to-zero)             | $590          | $280          | $310     |<br>| Karpenter system overhead (daemonsets, addons)              | —             | $80           | -$80     |<br>| 1-year Compute Savings Plan offset                           | —             | -$120         | $120     |<br>| Compute subtotal                                             | $3,597        | $2,040        | $1,557   |<br>| EBS (fewer nodes = less node-attached storage)              | $352          | $80           | $272     |<br>| Load balancers (4 ALBs → 1 ALB + 1 NLB)                     | $200          | $80           | $120     |<br>| NAT gateways + VPC endpoints                                 | $275          | $229          | $46      |<br>| ECS service charges (mostly Fargate in internal)            | $441          | $0            | $441     |<br>| EKS control plane                                            | $0            | $73           | -$73     |<br>| API Gateway (REST v1 → HTTP v2 + WebSocket unchanged)       | $175          | $75           | $100     |<br>| Total                                                        | $5,040        | $2,697        | $2,343   |<br>+--------------------------------------------------------------+---------------+---------------+----------+</pre><p><strong>Savings: ~$2,460/mo, ~$29,500/year, ~49% reduction.</strong></p><h3>Where the savings come from</h3><p>Breaking out the contribution by lever:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6nFG3nHbscYihL2jyMLDw.png" /></figure><p>Karpenter consolidation and the elimination of ECS service charges are the two largest contributors. Both are direct architectural effects, not behavioral changes.</p><h3>Variance bands</h3><p>A few line items have real uncertainty:</p><ul><li><strong>HTTP compute</strong> could swing ±$200/mo depending on how aggressively HPA + Karpenter consolidate during off-peak hours and traffic seasonality.</li><li><strong>KEDA effectiveness on workers</strong> could swing ±$80/mo depending on queue burst patterns. Very bursty queues with long-tail messages keep workers warmer than ideally idle ones.</li><li><strong>NAT data charges</strong> could swing ±$50/mo depending on container image sizes and pod restart frequency.</li></ul><p>Realistic range: <strong>$2,200–2,900/mo savings, $26k-$35k/year.</strong></p><h3>What we gain that isn’t a dollar figure</h3><ul><li><strong>True scale-to-zero on workers</strong> with sub-30-second cold start, vs our current 90s-3min latency.</li><li><strong>Blue/green deployments with instant rollback</strong> via Argo Rollouts — sub-30-second traffic switch, vs minute-scale rolling deploys today.</li><li><strong>Per-pod IAM via Pod Identity</strong> — currently every container on a node shares the instance’s IAM role.</li><li><strong>Zero public attack surface for backend services.</strong> API Gateway becomes the only public surface; backends are unreachable from the internet.</li><li><strong>Unified observability</strong> — Container Insights, Prometheus, and per-pod metrics, currently disabled on our ECS clusters. Enables per-service cost and performance attribution that we don’t have today.</li><li><strong>Operational consolidation</strong> — 13 capacity providers, 13 ASGs, 13 launch templates, 6 clusters, custom Lambda scaling scripts, and per-cluster CloudFormation stacks all collapse into one Terraform/Pulumi codebase managing one EKS cluster.</li><li><strong>Modern AMI lifecycle</strong> — Karpenter rolls nodes onto fresh Bottlerocket or AL2023 AMIs automatically.</li></ul><h3>Risks and tradeoffs</h3><p>Honest assessment of what we’re signing up for:</p><p><strong>Operational complexity increases.</strong> EKS is more complex to operate than ECS. We will need to manage:</p><ul><li>EKS version upgrades (annually, with ~14-month support windows per version)</li><li>Karpenter upgrades (every 3–6 months)</li><li>KEDA, Argo Rollouts, AWS Load Balancer Controller upgrades</li><li>CoreDNS, kube-proxy, VPC CNI addon lifecycle (mitigated by EKS Auto Mode if we opt in)</li><li>General Kubernetes housekeeping (certificate rotation, webhook health, etc.)</li></ul><p><strong>Mitigation:</strong> Consider <strong>EKS Auto Mode</strong> for the first phase. AWS manages Karpenter, addons, and node patching. We give up some flexibility but eliminate most of the ongoing ops burden. Once we have operational confidence, we can move to self-managed if needed.</p><p><strong>Migration complexity.</strong> ~250 services need their task definitions converted to Kubernetes Deployments (or Helm charts), IAM roles re-mapped via Pod Identity, networking switched from bridge mode to awsvpc-equivalent (Kubernetes native), and CI/CD pipelines updated to deploy manifests instead of calling aws ecs update-service.</p><p><strong>Mitigation:</strong> This is the “no migration concerns” greenfield design. If migration cost is on the table separately, it’s a multi-month project requiring its own planning document.</p><p><strong>KEDA SQS scaler limits.</strong> KEDA is generally reliable but it does poll AWS APIs, which counts against API rate limits. For our 270+ worker services each polling SQS every 15 seconds, we’d be making ~1,000 SQS API calls per minute. This is within SQS’s rate limits but worth monitoring.</p><p><strong>Mitigation:</strong> Configure pollingInterval per service tier — frequently-active queues at 15s, low-volume queues at 60s. Standard practice.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ea86456ebf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Scaling of ECS Services for FIFO Queues with MessageGroupIds: Leveraging In-flight Messages]]></title>
            <link>https://ruweeniddagoda.medium.com/dynamic-scaling-of-ecs-services-for-fifo-queues-with-messagegroupids-leveraging-in-flight-messages-f988bbb03b08?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/f988bbb03b08</guid>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[scaling]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 17:53:19 GMT</pubDate>
            <atom:updated>2026-08-06T17:56:45.126Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This is one of the proudest archivements from my time at Velaris before becoming a full-time Claude Maxxer; sharing this on my own blog for the public to see after years. Hope y’all find this useful. I’ll be dumping more write-ups from my company’s Confluence about the things I’ve built over the years, making them available for anyone who’d like to read and learn from them.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zRMU5fnevd8GLRx2UDX3AA.png" /></figure><h3>Introduction</h3><p>In a previous article, we examined the process of scaling ECS services according to message backlog. While this method is effective in many scenarios, it may not be optimal for scaling FIFO queues that involve MessageGroupIds.</p><p>MessageGroupId serves as a marker indicating that a message belongs to a specific message group. Messages within the same group are processed sequentially, maintaining strict order relative to the group. In cases where multiple messages share the same group Id, only a few workers will actively process messages, even if a considerable number of ECS workers, such as 40 or more, are available. This is due to the inability to process messages in parallel within the same group.</p><p>Implementing a scaling algorithm based solely on backlog in such scenarios would result in inefficiency. A significant portion of ECS tasks would be running without processing any messages, particularly when the queue depth remains consistent for messages with the same MessagegroupID.</p><h3>Solution</h3><p>In our proposed solution, we introduce a scaling technique that leverages in-flight messages. When a consumer task associated with an SQS (Simple Queue Service) receives a message, the queue marks it as an in-flight message. According to the documentation, an in-flight message is one that has been received from the queue by a consumer but has not yet been deleted from the queue.</p><p>The in-flight message count per queue aligns with the number of active workers (ECS tasks) processing messages, assuming a single consumer processes messages sequentially. For instance, if the FIFO queue contains 100,000 messages, but the consumers can only handle 2 messages concurrently, effectively maintaining 2 in-flight messages at a time, additional workers would be redundant.</p><p>Hence, our system dynamically adjusts the number of ECS tasks based on the number of in-flight messages present, ensuring optimal resource allocation.</p><h4>Process</h4><ol><li>Utilizing a custom Lambda function, the system systematically retrieves the ApproximateNumberOfMessagesNotVisible metric from CloudWatch, pertaining to the specified Amazon Simple Queue Service (SQS) queue, with a polling frequency of one minute.</li><li>This custom Lambda function is also responsible for computing the taskUtilization metric, derived from the ApproximateNumberOfMessagesNotVisible (representing the count of in-flight messages) and the current number of running tasks.</li><li>Subsequently, the calculated taskUtilization percentage value is propagated as a custom metric to CloudWatch.</li><li>A target tracking alarm is configured to monitor the custom metric “taskUtilization&quot; associated with the Amazon Elastic Container Service (ECS) service. This alarm scales the ECS service accordingly based on the observed metric values.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Twxl6YkDjNWaKUX6ZSyYZQ.png" /></figure><h3>Scale-In</h3><p>Understanding the scale-down mechanism implemented herein is straightforward. Primarily, the system evaluates the count of in-flight messages, and if this count consistently remains lower than the number of tasks deployed, a scale-down operation is initiated. The scale-down process is governed by a target tracking policy, which orchestrates a gradual reduction in the number of tasks. This policy monitors the taskUtilization percentage, aiming for it to remain below 90 for a duration encompassing 15 data points over a 15-minute period. It’s important to note that this threshold is predefined by the autoscaling API and cannot be further adjusted. However, this configuration adequately suits our specific requirements and ensures that scale-down actions are only taken when warranted, preventing unnecessary resource depletion.</p><h3>Scale-Out</h3><p>When a message is retrieved from an SQS queue by an ECS task but has not yet been deleted through the delete message API call, it is considered “in-flight.” In simpler terms, a message becomes in-flight when an ECS task is actively processing it. If there are no tasks in progress, the in-flight count will be zero, resulting in no scaling-out action taking place. This poses a challenge when scaling-out the queue processing capacity. However, the solution is simple: employ spare tasks.</p><p>Spare tasks are idle ECS tasks that are ready to process incoming messages. It’s essential to maintain a certain number of spare tasks that are continuously running and prepared to consume new messages as they become visible in the queue. The specific number of spare tasks required may vary depending on the desired latency, but typically 1 or 2 spare tasks are adequate for efficient scaling.</p><p>In our configuration, we utilize 2 spare tasks. This means that if there are currently 2 in-flight messages being processed, the total desired or running tasks should be 4. As new messages become visible in the queue, they can be immediately consumed by the ECS service, resulting in an increase in the in-flight message count to a new limit of 4. Consequently, this adjustment in the in-flight message count triggers the scaling-out of the ECS service, increasing the number of tasks to 6.</p><h3>TaskUtilization Percentage</h3><p>The TaskUtilization metric is computed through a custom process utilizing the ApproximateNumberOfMessagesNotVisible value and the current task count. A Lambda function is specifically developed for this purpose, invoking the CloudWatch Metrics API to retrieve five data points representing the one-minute average values for ApproximateNumberOfMessagesNotVisible of the designated FIFO queue. The maximum average value among these data points is then determined.</p><p>Subsequently, the desired_task_count is calculated by adding the maximum average for ApproximateNumberOfMessagesNotVisible to the number of spare tasks. The taskUtilization percentage is then derived by multiplying 100 with the ratio of desired_task_count to the number of currently running tasks. This calculated value is crucial for dynamically scaling the ECS task count.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*yrJ9-zO50TfkhoDIEXiRyg.png" /></figure><h3>Lambda Function</h3><p>As previously stated, the lambda function outlined in this document is designed to achieve the following objective:</p><ul><li><strong>Calculate the maximum average of Messages In-flight</strong></li></ul><p>To accomplish this, the function will perform the following steps:</p><ol><li>Poll the CloudWatch metric “ApproximateNumberOfMessagesNotVisible.&quot;</li><li>Retrieve the last 5 data points associated with this metric.</li><li>Determine the maximum value among these data points.</li></ol><p>This approach is implemented to mitigate the occurrence of null values and minimize error rates by obtaining a more accurate representation of the actual inflight message count. Given that this count tends to fluctuate significantly, this method helps in improving the precision of the estimation.</p><pre>def get_metric_sqs_cloudwatch(queue_name):<br><br>    end_time = datetime.datetime.utcnow()<br>    start_time = end_time - datetime.timedelta(minutes=5)<br><br>    # Define the metric query parameters<br>    metric_name = &#39;ApproximateNumberOfMessagesNotVisible&#39;<br>    namespace = &#39;AWS/SQS&#39;<br>    statistics = &#39;Average&#39;<br>    dimensions = [{&#39;Name&#39;: &#39;QueueName&#39;, &#39;Value&#39;: queue_name}]<br><br>    # Query the CloudWatch metric<br>    try:<br>        response = cloudwatch.get_metric_statistics(<br>            Namespace=namespace,<br>            MetricName=metric_name,<br>            StartTime=start_time,<br>            EndTime=end_time,<br>            Dimensions=dimensions,<br>            Period=60,  # 1 minute<br>            Statistics=[statistics]<br>        )<br>    except Exception as e:<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;body&#39;: f&#39;Error: {str(e)}&#39;<br>        }<br>    <br>    # Extract the max average value <br>    datapoints = response.get(&#39;Datapoints&#39;, [])<br>    if datapoints:<br>        max_average_value = max(datapoint[&#39;Average&#39;] for datapoint in datapoints)<br>        <br>    return max_average_value</pre><ul><li>Compute and transmit the taskUtilization metric value</li></ul><p>This procedure involves analyzing both the in-flight message count and the quantity of active tasks, employing the algorithm previously delineated to derive the taskUtilization percentage.</p><pre>approximate_number_of_messages_in_flight = float(get_metric_sqs_cloudwatch(os.environ.get(&#39;QUEUE_NAME&#39;)))<br>number_of_active_task_in_service = get_number_of_active_task_in_service(cluster_name, service_name)<br><br>desired_task_count = approximate_number_of_messages_in_flight + 2<br><br>task_utilization = (100 * desired_task_count)/number_of_active_task_in_service<br><br>metric_data = put_metric_data(task_utilization, cluster_name, service_name) </pre><p>Target Tracking Policy</p><p>Configure the target value to 100 and establish a target tracking policy based on the taskUtilization custom metric. When the taskUtilization exceeds this threshold, ECS will automatically increase the desired task count to maintain the target level. Conversely, if the taskUtilization falls below the threshold, ECS will decrease the desired count accordingly. To optimize scaling responsiveness, adjust the ScaleOutCoolDown and ScaleInCoolDown parameters to reduce the scaling latency.</p><pre>{<br>    &quot;TargetValue&quot;:100,<br>    &quot;ScaleOutCooldown&quot;:0,<br>    &quot;ScaleInCooldown&quot;:0,<br>    &quot;CustomizedMetricSpecification&quot;:{<br>        &quot;MetricName&quot;:&quot;taskUtilization&quot;,<br>        &quot;Namespace&quot;:&quot;CustomDemo/ECS/SQS&quot;,<br>        &quot;Dimensions&quot;:[<br>           {<br>              &quot;Name&quot;:&quot;ECSClusterName&quot;,<br>              &quot;Value&quot;:&quot;event-demo-dev-ecs-cluster&quot;<br>           },<br>           {<br>              &quot;Name&quot;:&quot;ECSServiceName&quot;,<br>              &quot;Value&quot;:&quot;dev-event-handler-dummy-listener-ecs-service&quot;<br>             }<br>        ],<br>        &quot;Statistic&quot;:&quot;Average&quot;<br>     }</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f988bbb03b08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adaptive ECS Autoscaling with Rate Limit Circuit Breaker]]></title>
            <link>https://ruweeniddagoda.medium.com/adaptive-ecs-autoscaling-with-rate-limit-circuit-breaker-dd3b97f820d4?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd3b97f820d4</guid>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 18:11:25 GMT</pubDate>
            <atom:updated>2025-12-11T18:24:03.238Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>This document outlines an architectural pattern for scaling AWS ECS services that consume SQS queues. While standard scaling relies on queue depth, real-world integrations often have downstream rate limits (e.g., Salesforce, 3rd Party APIs).</p><p><strong>The Problem:</strong> Standard auto-scaling blindly adds tasks when the queue is high, potentially flooding downstream APIs and causing critical failures.</p><p><strong>The Solution:</strong> A “Hybrid Controller” architecture. We use <strong>AWS Target Tracking</strong> for standard queue-based scaling, combined with an <strong>Event-Driven Lambda Circuit Breaker</strong>. When a rate limit is hit, the system instantaneously “clamps” the ECS Service capacity to a safe minimum, overriding the scaling policy without deleting it.</p><h3>2. High Level Architecture</h3><p>The system operates in two distinct modes: <strong>Normal Mode</strong> and <strong>Panic Mode</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H3N7ZzucO3VrZ4HGG5KU3Q.png" /></figure><h3>3. Design Logic</h3><h4>3.1. The “Metric Math” Target Tracking (Normal Mode)</h4><p>The system does not rely on simple queue depth triggers. Instead, it uses <strong>CloudWatch Metric Math</strong> to calculate <strong>Backlog Per Task</strong> using the formula:</p><p>ApproximateNumberOfMessagesVisible / RunningTaskCount</p><p>The target is set at <strong>1,000 messages per task</strong>.</p><p>To handle edge cases where RunningTaskCount is 0 (which would normally cause division errors), the math expression forces a <strong>high pseudo-value</strong>, ensuring the service scales from 0 to 1 task as needed.</p><h4>3.2. The “Clamp” Strategy (Panic Mode)</h4><p>Rather than updating scaling policies each time — which can be slow and error-prone — the system manipulates the <strong>ECS Service Auto Scaling limits</strong>.</p><ul><li>When the <strong>Rate Limit Alarm</strong> triggers, a Lambda function sets both <strong>MinCapacity</strong> and <strong>MaxCapacity</strong> to 1.</li><li>As a result, even if auto scaling attempts to scale out, the service is capped at 1 task, effectively holding the task count steady.</li><li>When the <strong>Rate Limit Alarm</strong> clears, the Lambda function restores <strong>MinCapacity</strong> and <strong>MaxCapacity</strong> to their default values.</li><li>The target tracking policy immediately resumes control, calculating the required task count based on the current queue size.</li></ul><h3>4. Implementation Guide</h3><p>This section outlines the step-by-step implementation of the hybrid scaling architecture. It details the configuration of the detection mechanism, the scaling engine, and the circuit breaker logic.</p><h4>Phase 1: The Rate Limit Signal</h4><p>The implementation begins by converting application logs into a numerical metric that AWS CloudWatch can monitor. This requires a <strong>Metric Filter</strong> to parse logs for specific error patterns (e.g., “exceeded API call threshold”) and a <strong>CloudWatch Alarm</strong> to trigger the response.</p><p><strong>A. Create Metric Filter</strong></p><p>The following command configures a metric filter that scans the log group /aws/ecs/my-app-logs. It increments the metric RateLimitExceeded by 1 whenever the specific error string is detected.</p><pre>aws logs put-metric-filter \<br>    --log-group-name &quot;/aws/ecs/my-app-logs&quot; \<br>    --filter-name &quot;RateLimitFilter&quot; \<br>    --filter-pattern &#39;&quot;has exceeded API call threshold&quot;&#39; \<br>    --metric-transformations \<br>        metricName=&quot;RateLimitExceeded&quot;,metricNamespace=&quot;MyApp/Limits&quot;,metricValue=1</pre><p><strong>B. Create the Alarm</strong></p><p>Once the metric exists, an alarm must be configured to define the “Panic” threshold. The command below sets the alarm to trigger if the cumulative count of rate limit errors exceeds 0 within a single one-minute period.</p><pre>aws cloudwatch put-metric-alarm \<br>    --alarm-name &quot;RateLimit-my-service-name&quot; \<br>    --metric-name &quot;RateLimitExceeded&quot; \<br>    --namespace &quot;MyApp/Limits&quot; \<br>    --threshold 0 \<br>    --comparison-operator &quot;GreaterThanThreshold&quot; \<br>    --evaluation-periods 1 \<br>    --alarm-actions &quot;arn:aws:events:region:account-id:rule/scaling-manager-rule&quot;</pre><h4>Phase 2: The Target Tracking Policy</h4><p>The system requires a persistent scaling policy to manage the service during normal operation. This policy utilizes CloudWatch Metric Math to calculate the backlog per task, ensuring accurate scaling even when the task count is zero.</p><p><strong>A. The Metric Configuration (JSON)</strong></p><p>The configuration file policy.json defines the metric math logic. It handles the edge case where RunningTaskCount (m2) is zero by forcing a high artificial value (1000000) to trigger an immediate scale-out event.</p><pre>{<br>    &quot;TargetValue&quot;: 1000.0,<br>    &quot;CustomizedMetricSpecification&quot;: {<br>        &quot;Metrics&quot;: [<br>            {<br>                &quot;Id&quot;: &quot;m1&quot;,<br>                &quot;Label&quot;: &quot;QueueSize&quot;,<br>                &quot;MetricStat&quot;: {<br>                    &quot;Metric&quot;: {<br>                        &quot;Namespace&quot;: &quot;AWS/SQS&quot;,<br>                        &quot;MetricName&quot;: &quot;ApproximateNumberOfMessagesVisible&quot;,<br>                        &quot;Dimensions&quot;: [{&quot;Name&quot;: &quot;QueueName&quot;, &quot;Value&quot;: &quot;my-queue-name&quot;}]<br>                    },<br>                    &quot;Stat&quot;: &quot;Average&quot;<br>                },<br>                &quot;ReturnData&quot;: false<br>            },<br>            {<br>                &quot;Id&quot;: &quot;m2&quot;,<br>                &quot;Label&quot;: &quot;TaskCount&quot;,<br>                &quot;MetricStat&quot;: {<br>                    &quot;Metric&quot;: {<br>                        &quot;Namespace&quot;: &quot;AWS/ECS&quot;,<br>                        &quot;MetricName&quot;: &quot;CPUUtilization&quot;,<br>                        &quot;Dimensions&quot;: [{&quot;Name&quot;: &quot;ServiceName&quot;, &quot;Value&quot;: &quot;my-service-name&quot;}, {&quot;Name&quot;: &quot;ClusterName&quot;, &quot;Value&quot;: &quot;my-cluster&quot;}]<br>                    },<br>                    &quot;Stat&quot;: &quot;SampleCount&quot;<br>                },<br>                &quot;ReturnData&quot;: false<br>            },<br>            {<br>                &quot;Id&quot;: &quot;e1&quot;,<br>                &quot;Expression&quot;: &quot;IF(m2==0, IF(m1&gt;0, 1000000, 0), m1/m2)&quot;,<br>                &quot;Label&quot;: &quot;MessagesPerTask&quot;,<br>                &quot;ReturnData&quot;: true<br>            }<br>        ]<br>    }<br>}</pre><p><strong>B. Apply the Policy</strong></p><p>The policy is applied to the ECS service using the AWS CLI. This command associates the JSON configuration with the specific ECS resource.</p><pre>aws application-autoscaling put-scaling-policy \<br>    --service-namespace ecs \<br>    --resource-id service/my-cluster/my-service-name \<br>    --scalable-dimension ecs:service:DesiredCount \<br>    --policy-name &quot;queue-depth-target-tracking&quot; \<br>    --policy-type TargetTrackingScaling \<br>    --target-tracking-scaling-policy-configuration file://policy.json</pre><h4>Phase 3: The Circuit Breaker (Lambda)</h4><p>The core logic resides within a Lambda function acting as a “Traffic Cop.” This function maintains a configuration map of services and their default capacity limits. It listens for state changes from the CloudWatch Alarm configured in Phase 1.</p><p><strong>Lambda Logic (Python/Boto3)</strong></p><p>The function executes the following logic:</p><ul><li><strong>Identify:</strong> Parses the event to identify which service triggered the alarm.</li><li><strong>Panic (ALARM State):</strong> If the rate limit is breached, it calls register_scalable_target with both MinCapacity and MaxCapacity set to <strong>1</strong>. This physically constricts the service, preventing the Target Tracking policy from adding tasks.</li><li><strong>Recover (OK State):</strong> When the alarm clears, it restores MinCapacity to <strong>0</strong> and MaxCapacity to the defined default (e.g., 10), allowing standard auto-scaling to resume.</li></ul><pre>import boto3<br>import json# Configuration Map: Maps service names to their cluster and healthy limits<br># service_config.json<br># {<br>#   &quot;my-service-name&quot;: { &quot;cluster&quot;: &quot;my-cluster&quot;, &quot;max_capacity&quot;: 10 }<br># }<br>client = boto3.client(&#39;application-autoscaling&#39;)<br>def lambda_handler(event, context):<br>    # 1. Detect Alarm State Change<br>    alarm_name = event[&#39;detail&#39;][&#39;alarmName&#39;] # e.g., RateLimit-my-service-name<br>    new_state = event[&#39;detail&#39;][&#39;state&#39;][&#39;value&#39;] # ALARM or OK<br>    <br>    # 2. Identify Service via helper function (logic omitted for brevity)<br>    service_name = parse_service_name(alarm_name)<br>    config = load_config(service_name)<br>    resource_id = f&quot;service/{config[&#39;cluster&#39;]}/{service_name}&quot;<br>    <br>    # 3. Apply Circuit Breaker Logic<br>    if new_state == &#39;ALARM&#39;:<br>        print(f&quot;Panic Mode: Locking {service_name} to 1 task.&quot;)<br>        # Hard Cap: Forces the service to 1 task regardless of queue depth<br>        client.register_scalable_target(<br>            ServiceNamespace=&#39;ecs&#39;,<br>            ResourceId=resource_id,<br>            ScalableDimension=&#39;ecs:service:DesiredCount&#39;,<br>            MinCapacity=1, <br>            MaxCapacity=1 <br>        )<br>        <br>    elif new_state == &#39;OK&#39;:<br>        print(f&quot;Recovery: Restoring {service_name} limits.&quot;)<br>        # Restore: Opens the limits so Target Tracking can function<br>        client.register_scalable_target(<br>            ServiceNamespace=&#39;ecs&#39;,<br>            ResourceId=resource_id,<br>            ScalableDimension=&#39;ecs:service:DesiredCount&#39;,<br>            MinCapacity=0, <br>            MaxCapacity=config[&#39;max_capacity&#39;] <br>        )</pre><p><strong>Infrastructure as Code (Serverless Framework)</strong></p><p>The deployment uses Amazon EventBridge to trigger the Lambda function only when specific alarm state changes occur. This minimizes cost and ensures immediate reaction times.</p><pre>functions:<br>  scalingManager:<br>    handler: lambda_function.lambda_handler<br>    events:<br>      - eventBridge:<br>          pattern:<br>            source: [&quot;aws.cloudwatch&quot;]<br>            detail-type: [&quot;CloudWatch Alarm State Change&quot;]<br>            detail:<br>              alarmName: [&quot;RateLimit-my-service-name&quot;]<br>              state:<br>                value: [&quot;ALARM&quot;, &quot;OK&quot;]</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd3b97f820d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ECS Graceful Shutdown and Managed Termination Protection]]></title>
            <link>https://ruweeniddagoda.medium.com/ecs-graceful-shutdown-and-managed-termination-protection-8c24fe1c3111?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c24fe1c3111</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[ec]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Fri, 28 Mar 2025 08:33:41 GMT</pubDate>
            <atom:updated>2025-03-28T08:33:41.024Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>When an ECS task is running on an EC2 instance that is being terminated, it is crucial to ensure that tasks exit gracefully to prevent service disruptions, request loss, or data inconsistency. ECS provides a mechanism for managing instance termination through <strong>Managed Termination Protection</strong> and <strong>Instance Draining</strong>, allowing running tasks to either complete their work or be rescheduled elsewhere before the instance shuts down.</p><p>This document explains how ECS manages graceful shutdowns and the steps required for applications to handle termination signals properly.</p><h3>How ECS Handles Instance Termination</h3><h3>1. Instance Marked for Termination</h3><p>When an EC2 instance in an <strong>Auto Scaling Group (ASG)</strong> is scheduled for termination (due to scaling policies, manual termination, or spot instance expiration), ECS detects this and marks the instance as <strong>DRAINING</strong>.</p><p><em>In </em><strong><em>DRAINING</em></strong><em> mode:</em></p><ul><li>No new ECS tasks are scheduled on the instance.</li><li>Existing tasks continue running but are scheduled for termination or rescheduling.</li></ul><h3>2. SIGTERM Signal Sent to Containers</h3><ul><li>ECS sends a SIGTERM signal to all running containers on the instance.</li><li>This allows the application inside the container to initiate a graceful shutdown process.</li><li>The container has a <strong>grace period</strong> (default: 30 seconds) to perform cleanup operations before ECS force-stops it.</li></ul><h3>3. Graceful Shutdown Process</h3><p><em>Applications should:</em></p><ul><li><strong>Stop accepting new requests</strong></li><li><strong>Complete any pending processes</strong></li><li><strong>Save state and logs</strong> before shutting down.</li><li><strong>Exit cleanly</strong> with exit(0) to indicate successful termination.</li></ul><h3>4. Forced Termination if Not Exited in Time</h3><ul><li>If the container does not exit within the grace period, ECS sends a <strong>SIGKILL</strong> signal, immediately terminating the process.</li><li>This may cause data loss if the application has not handled termination properly.</li></ul><h3>Handling SIGTERM in Applications</h3><p>To ensure graceful shutdown, applications running in ECS tasks <strong>must listen for SIGTERM signals</strong> and exit properly.</p><h4>Example: Handling SIGTERM in a Node.js Application</h4><pre>process.on(&#39;SIGTERM&#39;, () =&gt; {<br>    console.log(&#39;Received SIGTERM, shutting down gracefully...&#39;);<br>    server.close(() =&gt; {<br>        console.log(&#39;Closed server, exiting process.&#39;);<br>        process.exit(0);<br>    });<br>});</pre><p>Ensures the server <strong>stops accepting requests</strong> before exiting.</p><h3>Autoscaling Considerations and EC2 Infrastructure Sharing</h3><h3>Impact of Scale-In Actions on ECS Tasks</h3><ul><li>If ECS determines that a cluster is <strong>over-provisioned</strong>, it triggers <strong>instance termination</strong> as part of Auto Scaling.</li><li>This can lead to <strong>random instance terminations</strong>, affecting services running on the same infrastructure.</li><li>As a result, tasks belonging to different microservices <strong>may be killed unexpectedly</strong> during rebalancing.</li></ul><h3>Workarounds to Prevent Disruptions</h3><ol><li><strong>Move critical services to dedicated Auto Scaling Groups</strong></li></ol><ul><li>Services that should not be randomly terminated should be moved to their own autoscaling groups within the same cluster.</li></ul><p><strong>2. Separate non-critical services into a separate Auto Scaling Group</strong></p><ul><li>If the service has minimal impact on the application or end users, allow it to be terminated freely.</li><li>This also enables cost-saving measures such as using <strong>spot instances</strong>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c24fe1c3111" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Amazon ECS Services for Efficient SQS Queue Handling]]></title>
            <link>https://ruweeniddagoda.medium.com/scaling-amazon-ecs-services-for-efficient-sqs-queue-handling-d57c9c68b5e9?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/d57c9c68b5e9</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[autoscaling]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Sun, 04 Feb 2024 22:07:57 GMT</pubDate>
            <atom:updated>2024-02-06T07:54:43.696Z</atom:updated>
            <content:encoded><![CDATA[<p>This article serves as a continuation of the initial post authored by Ahmed Azzam and Sebastian Lee titled “<a href="https://aws.amazon.com/blogs/containers/amazon-elastic-container-service-ecs-auto-scaling-using-custom-metrics/">Amazon Elastic Container Service (ECS) Auto Scaling using custom metrics.</a>” It includes practical code segments and configurations to assist individuals seeking to implement a comparable solution.</p><h3>Introduction</h3><p>This post details our approach to scaling an Amazon Elastic Container Service (ECS) service dynamically based on the load changes in an Amazon Simple Queue Service (SQS) queue. Recent challenges in efficiently managing a growing queue with our event-handler microservices prompted us to devise a custom scaling solution. This is particularly crucial, as sustaining a high number of tasks consistently can lead to substantial costs.</p><h3>Solution</h3><p>While leveraging the SQS CloudWatch metric “ApproximateNumberOfMessages” provides a basic scaling mechanism, its limitation lies in its inability to adjust proportionally to the ECS service’s capacity. Merely relying on the number of messages in the SQS queue is insufficient for determining the required number of ECS tasks. Therefore, our solution incorporates additional factors, specifically the time it takes to process a message and the acceptable latency.</p><h3>Backlog Per Task Custom Metric</h3><p>Introducing a custom metric named “Backlog per Task,” we factor in both message processing time and acceptable latency to determine the optimal workload per task. For instance, if we have a queue with 1500 messages and the service runs 10 tasks, with an average processing time of 0.1 seconds per message and an acceptable latency of 10 seconds, the backlog per task would be set at 100 messages. Considering the current messages per task value is 150, configuring the target tracking policy to 100 ensures that the ECS service scales with an additional 5 tasks to balance the workload.</p><p>A more pragmatic approach would involve regarding the acceptable latency as half of the message retention time, a duration usually configured in the SQS queue attributes. For instance, if the message retention period is set to 1 day, the acceptable latency would be considered as 12 hours (43200 seconds). Assuming a message requires an average processing time of 6.5 seconds, the acceptable backlog per task would then be approximately 6600 messages. By setting an acceptable latency of 12 hours, we guarantee that tasks are consistently adequate to handle fluctuating workloads, even if the queue experiences a surge in messages. The average message processing time is a parameter we adjust to meet the desired messages per task count, providing flexibility in managing varying workloads.</p><p>When adjusting the target tracking policy, simply determine the desired number of messages a task should consume and set it accordingly. This approach ensures a dynamic and cost-effective scaling solution for ECS services, addressing the challenges posed by evolving SQS queue loads.</p><p>Diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JsGXixK2YzZiLUQRHoS28w.png" /></figure><h3>Components</h3><h4>Pre-requisites</h4><p>To implement this solution, ensure the following:</p><ul><li>An SQS queue and a corresponding consumer microservice are in place.</li><li>Infrastructure autoscaling is configured for the ECS cluster to automatically adjust the number of instances based on task count changes.</li></ul><p>A Lambda function is set up to periodically poll the SQS queue’s ApproximateNumberOfMessages and calculate the backlog_per_task (current) metric every 5 minutes.</p><h4>Lambda Function</h4><p>The custom Lambda function extracts key metrics, such as the number of messages in the SQS queue and the current tasks in the ECS service, then calculates the current backlog per task. This function runs at regular intervals to provide real-time data for scaling decisions.</p><pre>approximate_number_of_messages = float(get_queue_attributes(os.environ.get(&#39;QUEUE_URL&#39;))[&#39;ApproximateNumberOfMessages&#39;])<br>        number_of_active_task_in_service = get_number_of_active_task_in_service(cluster_name, service_name)<br>backlog_per_task = approximate_number_of_messages / number_of_active_task_in_service<br>        metric_data = put_metric_data(backlog_per_task, cluster_name, service_name)<br>        print({<br>            &#39;number_of_active_task_in_service&#39;: number_of_active_task_in_service,<br>            &#39;approximate_number_of_messages&#39;: approximate_number_of_messages,<br>            &#39;backlog_per_task&#39;: backlog_per_task<br>        })</pre><h4>Target Tracking Policy</h4><p>In the microservice’s target tracking policy, we set the acceptable_backlog_per_task value as our target. This is determined by dividing the longest acceptable latency by the average processing time of a message. CloudWatch alarms are configured based on these target values, triggering ECS to scale in or out to maintain the desired backlog_per_task.</p><p>To implement this, a CLI method is utilized due to the absence of support for custom target tracking policies via the console UI or boto3. The following steps are involved:</p><ol><li>Create a target_tracking_policy.json file for the custom backlog_per_task metric.</li></ol><pre>{<br>    &quot;TargetValue&quot;:6600,<br>    &quot;ScaleOutCooldown&quot;:15,<br>    &quot;ScaleInCooldown&quot;:15,<br>    &quot;CustomizedMetricSpecification&quot;:{<br>        &quot;MetricName&quot;:&quot;backlogPerTask&quot;,<br>        &quot;Namespace&quot;:&quot;CustomDemo/ECS/SQS&quot;,<br>        &quot;Dimensions&quot;:[<br>           {<br>              &quot;Name&quot;:&quot;ECSClusterName&quot;,<br>              &quot;Value&quot;:&quot;event-demo-dev-ecs-cluster&quot;<br>           },<br>           {<br>              &quot;Name&quot;:&quot;ECSServiceName&quot;,<br>              &quot;Value&quot;:&quot;dev-event-handler-dummy-listener-ecs-service&quot;<br>             }<br>        ],<br>        &quot;Statistic&quot;:&quot;Average&quot;<br>     }<br> }</pre><p>2. Run the following AWS CLI command to register the microservice as a scalable target</p><pre>CLUSTER_NAME=event-demo-dev-ecs-cluster<br>SERVICE_NAME=dev-event-handler-dummy-listener-ecs-service<br>aws application-autoscaling register-scalable-target \<br>--service-namespace ecs \<br>--scalable-dimension ecs:service:DesiredCount \<br>--resource-id service/$CLUSTER_NAME/$SERVICE_NAME \<br>--min-capacity 1 \<br>--max-capacity 10</pre><p>3. Run the following AWS CLI command to attach the custom target tracking policy</p><pre>CLUSTER_NAME=event-demo-dev-ecs-cluster<br>SERVICE_NAME=dev-event-handler-dummy-listener-ecs-service<br>POLICY_NAME=sqs-queue-tracking-policy<br>aws application-autoscaling put-scaling-policy \<br>--policy-name $POLICY_NAME \<br>--service-namespace ecs \<br>--resource-id service/$CLUSTER_NAME/$SERVICE_NAME \<br>--scalable-dimension ecs:service:DesiredCount \<br>--policy-type TargetTrackingScaling \<br>--target-tracking-scaling-policy-configuration file://policy.json</pre><p>Alternatively, the Lambda function can independently scale the task count by calculating the acceptable_backlog_per_task value. This approach eliminates the need for a target tracking alarm and allows manual scaling of the service.</p><pre>async def scale_ecs_task(approximate_number_of_messages):<br>    average_processing_time = float(os.environ.get(&#39;AVERAGE_PROCESSING_TIME&#39;))<br>    acceptable_latency = float(os.environ.get(&#39;ACCEPTABLE_LATENCY&#39;))<br>    acceptable_backlog_per_task = acceptable_latency / average_processing_time<br>    no_of_task_desired = approximate_number_of_messages / acceptable_backlog_per_task<br><br>  return await scale_no_of_tasks_in_service(<br>        os.environ.get(&#39;ECS_CLUSTER_NAME&#39;),<br>        os.environ.get(&#39;ECS_SERVICE_NAME&#39;),<br>        no_of_task_desired<br>      )</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d57c9c68b5e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synchronizer Token Pattern for Cross-Site Request Forgery Prevention]]></title>
            <link>https://ruweeniddagoda.medium.com/synchronizer-token-pattern-for-cross-site-request-forgery-prevention-8a9b0480fb3b?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a9b0480fb3b</guid>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 10:00:55 GMT</pubDate>
            <atom:updated>2022-02-21T10:00:55.233Z</atom:updated>
            <content:encoded><![CDATA[<p>Within the contemporary context in which most of the communication and transactions are taking place over the web, the data being transferred are susceptible to many forms of attacks. An eminent attack type out of many is the Cross-Site Request Forgery attack. therefore, defense mechanisms which include Synchronized Token sample and Double publish Cookies got here into mild.</p><p>Cross-Site Request Forgery (CSRF), also called session riding and XSRF, is a common application-layer vulnerability that allows the malicious attacker to use an active session of the target to perform tasks on his behalf with out his prior knowledge or consent. CSRF incidents are tough to detect as they may be disguised into everyday user requests.</p><h3>What is a CSRF attack?</h3><p>CSRF attacks take advantage on the web applications inability to authenticate user access. These type of attacks commonly take form as follows.</p><p>Assume a victim user logs in to a web application which is vulnerable to Cross-Site Request Forgery, it will initiate a session which will have a unique session ID as a token of identification, which then, allows them to interchange information with the server and perform tasks. An attacker can utilize a unsolicited email with some malicious code planted inside or any other social engineering technique (often times in tandem with other techniques ) to trick the user on clicking a link which runs a background code against the targeted web application. If successful, the web application will assume the commands are legitimate and will execute them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*o6UZHEYaeWMmowHEgZAj5w.png" /></figure><p><a href="https://www.imperva.com/learn/application-security/csrf-cross-site-request-forgery/">https://www.imperva.com/learn/application-security/csrf-cross-site-request-forgery/</a></p><p>As an example, my banking website, example.com, does not protect itself against CSRF. You, an unsuspecting example.com user, also happened to be logged in to example.com. Now, an Attacker might send you (and millions of other example.com users, of course) an HTML e-mail including the following tag.</p><pre>&lt;img src=&quot;https://www.example.com/transfer?amount=1000<strong>&amp;amp;</strong>destination=ruween&quot;&gt;</pre><p>If you have a webmail client that loads images automatically, the transfer request will be made from your browser using your IP address and <em>your example.com session cookies</em>, exactly as if you made the request yourself. My bank website, therefore, treats this like a legitimate request and sends $1000 from your account to Attacker’s account. All evidence suggests you legitimately made this transaction from your logged-in browser.</p><p>A successful CSRF attack can be devastating for both the business and user. It can result in damaged client relationships, unauthorized fund transfers, changed passwords and data theft — including stolen session cookies.</p><p>The Solution — Synchronizer Token Pattern.</p><h3>Synchronizer Token Patterns as a Prevention Mechanism</h3><p>Synchronizer Token Pattern, generally abbreviated as STP, is a technique where a token is embedded by the web application in all HTML forms, which is verified by the server side.</p><p>The token in particular is a unique arbitrarily generated secret value that cannot be predicted by an Attacker, which is used to authenticate legitimate users. Tokens are often time generated using cryptographic methods with multiple layers of encryption and encoding placed on top of it.</p><p>Although it introduces some level of complexity in the server side due to the burden associated with checking validity per every request, Synchronizer Token Patterns can be considered as the most compatible as it primarily relied on HTML.</p><p>The following section contains an explanation of a sample implementation of Synchronizer Tokens using PHP, AJAX and JQuery. (Source Code — <a href="https://github.com/ruween/sync-token-pattern-csrf"><strong>Github</strong></a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*T7-7URSRX89EiZjfYCdLRA.png" /></figure><p>Login Form</p><p>Users need to input their credentials to gain access, which verifies them using generated token. If successful, users are redirected to the main page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*2ax5YH05EfUwzFJP30u3fg.png" /></figure><ul><li>uniqid — Generates a unique identifier based on the current time in microseconds.</li><li>mt_rand — Generate a random value via the Mersenne Twister Random Number Generator</li><li>sha1 — Calculate the sha1 hash of a string, in this example, the function will calculate the hash of the string (unique(mt_rand))</li><li>base_convert — Convert a number between arbitrary bases, in this case, hash value will be converted into a number of base value 36</li><li>substr — Only return up until the 32 character.</li></ul><p>Once the token is generated, it will be placed inside the HTML form, as a hidden value, which then will be sent to the server as a part of the POST request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*74IjufgG1h86dzsg2Ueo_g.png" /></figure><p>Login Form with the hidden value field</p><p>Once the server receives the data, it will check for it’s validity and return an error message if the credentials are incorrect. If not, it will generate a new session cookie and update the <em>loggedin</em> status as true. Then the user will be redirected to the main page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*_ivnyheMvWA-gu8nSO4mTA.png" /></figure><p>main page</p><p>Verified users can send credits, although the server will check authenticity of each request using synchronizer tokens.</p><p>Inside the main page, an Ajax call will be placed to the PHP back end in order to generate another token for the requests per session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*cJQaIZTm1We_ugcfmXrxUA.png" /></figure><p>Ajax call to request the new Token</p><p>The back end will process the request and generate a new token using the <em>Token::generate() </em>function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*m8HKWZVgzGUB-CoXug761A.png" /></figure><p>Token Class :: generate function</p><p>Then it’ll echo that value back to the Ajax call, which in turn will be used inside the request form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*MfRvbnu3pkhMKhuImGBvEg.png" /></figure><p>echo the token and call the mapper function</p><p>The code will access the mapper function to concatenate the session id with the newly created token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*Jl4x5XLcOEb90cuDQ63JCw.png" /></figure><p>mapper function</p><p>Again, similar to the login form, a hidden field will include the freshly created token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*68b1rZmv_MA0jOvvPSOtsw.png" /></figure><p>the Empty value will be updated using the Ajax “data”</p><p>Once POST-ed back to the server, it will separate the <em>mappedCookie, </em>and compare the mapped CSRF token value and session id with the POST data to verify authenticity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*qEAFw5engXvSD-xkrQ37FA.png" /></figure><p>Comparison of the mappedCookie data with POST data</p><p>If the CSRF token is not kept sufficiently secret and is disclosed in some way, the application may still be vulnerable to CSRF. The token should also be invalidated after some time and after the user logs out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a9b0480fb3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Double Submit Cookie for Cross-Site Request Forgery Prevention]]></title>
            <link>https://ruweeniddagoda.medium.com/double-submit-cookie-for-cross-site-request-forgery-prevention-f5a7711a33e1?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5a7711a33e1</guid>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 09:59:58 GMT</pubDate>
            <atom:updated>2022-02-21T09:59:58.854Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the extensively used alternatives for <a href="https://medium.com/@aeosheus212/synchronizer-token-pattern-for-cross-site-request-forgery-prevention-2bc0846d5b47"><strong>Synchronizer Token Pattern</strong></a>, for protection against Cross-Site Request Forgery, is the Double Submit Cookie.</p><p>Double Submitting cookies, as the name suggest, is sending an arbitrary unique generated value, both as a cookie and a POST data, where the server compares the two to verify and differentiate between legitimate users and impersonators.</p><h3>Overview</h3><p>When a user logs into the system, a session is created and a session ID is set as a cookie, alongside another cookie will be created containing a CSRF token.</p><p>Then, the token value will be extracted from the cookie and will be placed in a hidden input field in the HTML, and will be used to verify submitted forms by comparing the POST data with the actual cookie data.</p><p>The reason behind using two separate cookies is that, it allows the main cookie to be marked as HttpOnly, thus making it inaccessible to JavaScript. Also making it arguably harder for an attacker to penetrate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*BgR7F2kkBcuPsYVd" /></figure><p>Double Submit Cookie mitigates CSRF</p><p>The Following is a implementation of a Double Submit Cookie in a Web Application. (Source Code — <a href="https://github.com/ruween/double-submit-cookies-csrf-implementation"><strong>GitHub</strong></a>)</p><h3>Explanation of the Code</h3><p>The main program is quite similar to previous example used in the Synchronizer Token Pattern. (More on that <a href="https://medium.com/@aeosheus212/synchronizer-token-pattern-for-cross-site-request-forgery-prevention-2bc0846d5b47"><strong>here</strong></a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/0*unbB4TH65ZkCEyyS" /></figure><p>Token::generate()</p><p>Token generation function is the same as my previous example, with the only difference being the fact that the generated token is passed into a cookie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*5YPZfT_gCD1JIo0G" /></figure><p>Ajax call to obtain the cookie</p><p><em>document.cookie.match</em> function will return the cookie with the name, “csrf”. And if that cookie matches cookie[2] then it’s passed into the element with the id of “csrf”, which is the hidden input filed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*EbkivaNFnv4B7EcS" /></figure><p>Obtaining CSRF cookie Flow Chart</p><p>Finally, In the <em>process.php</em> the POST data will be compared with the cookie data as seen below. If it matches, the session is valid and the requested task will execute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/0*4QdeNv-o_3tJPfVg" /></figure><p>comparing POST data with cookie values</p><p>Once completed make sure to empty out the csrf cookie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/0*S42Ccc0AnhJ7WGiO" /></figure><p>Cleaning the CSRF cookie</p><p>Even after implementing a Double Submit Cookie Protection to a Web Application, it can still be susceptible to CSRF attacks due to;</p><ul><li>Potential Cross-Site Scripting vulnerabilities in sub domains, which can introduce poisoned cookies in upper domains</li><li>Cookies with sensitive server metadata are not being flagged as HTTPOnly</li></ul><p>Therefore, it’s crucial to identify any other loop holes that may exist within the system and secure them as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5a7711a33e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is API security?]]></title>
            <link>https://ruweeniddagoda.medium.com/what-is-api-security-60169a38a7ab?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/60169a38a7ab</guid>
            <category><![CDATA[application-security]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[owasp]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 09:58:52 GMT</pubDate>
            <atom:updated>2022-02-21T09:58:52.270Z</atom:updated>
            <content:encoded><![CDATA[<h3>OWASP API Top 10</h3><p>An Application Programming Interface (API) can be defined as the middle man to a program’s backbone. Modern software is designed as stacks, more like sandwiches and APIs rest in the middle acting as a communicating relay between the program and the hardware drivers. Or else, from a web developer’s point of view, between one service and another. APIs are widely used in modern cloud services and it’s one the common ways how microservices and containers communicate.</p><p>Businesses use APIs to connect their services and transfer data. Take twitter for an example, twitter plugins which display chucks of a twitter feed in other websites use the twitter API to access that feature. With the use of API, twitter and other major companies can share their information and let third parties access their services without sharing their code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CsWpavWMijYRTBkcmoiRfA.png" /></figure><p><a href="https://developer.twitter.com/en/docs/twitter-for-websites/follow-button/overview">https://developer.twitter.com/en/docs/twitter-for-websites/follow-button/overview</a></p><p>Often times APIs work with sensitive data such as application logic and Personally Identifiable Information (PII). Thus, it is prone to become the target of attackers who seek to benefit from accessing and obtaining those data.</p><p>83% of the web traffic generated today are from API calls. Compared to user interfaces (UI), exposed APIs will form a larger attack surface and it’s estimated to become the number 1 attack target by 2022.</p><p>Since APIs have already become a foundational element of the web technologies that are thriving today, the need for securing APIs has never been a more paramount concern than now.</p><h3>What is the OWASP API Top 10?</h3><p>Open Web Application Security Project (OWASP) has released a number of (ten, to be precise) substantial vulnerabilities that exist within APIs. We’ll be taking a look at each of them in detail in the following sections and the prevention methods for each of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*eqJmAFx7sYuwpUsL" /></figure><p><a href="https://www.slideshare.net/QAFest/qa-fest-2019-api">https://www.slideshare.net/QAFest/qa-fest-2019-api</a></p><h3>1. Broken Object Level Authentication</h3><p>This is also known as the Insecure Direct Object Reference (IDOR). Traditionally your rest APIs based on paths to resources, rely on the client to maintain the state and figuring out what it needs. These APIs tend to expose the endpoints of object identifiers.</p><p>Imagine a situation where you are logged into your bank account and you request to view your financial information, if the back-end doesn’t properly check for authentication or authorization then an attacker can use a different account to mimic the API call to access your account information, using the exposed object identifier by modifying the user input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*XZuU1Z1Xa6xsMVAu.jpg" /></figure><p><a href="https://apisecurity.io/encyclopedia/content/owasp/api5-broken-function-level-authorization.htm">https://apisecurity.io/encyclopedia/content/owasp/api5-broken-function-level-authorization.htm</a></p><p>Object-level authorization checks must be in place for every function that accesses the data source using input from the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dpW8QKB9K2dUb7j-.png" /></figure><p><a href="https://medium.com/@sagarag/api-security-how-to-avoid-broken-object-level-authorization-broken-function-level-authorization-f93b9b5ac333">https://medium.com/@sagarag/api-security-how-to-avoid-broken-object-level-authorization-broken-function-level-authorization-f93b9b5ac333</a></p><p>Imagine another situation of a web application — that allows students to view their marks and the Teachers to add marks for their respective subjects. If the Oauth scopes aren’t defined properly, providing read/write access to the teacher and read-only access to the student, the web application is vulnerable to being exploited by one of its users, which might try to manipulate the sensitive data that’s stored in the system.</p><h3>How to prevent it?</h3><ul><li>perform authorization checks at API gateways, or in a remote policy evaluation point.</li><li>For locally running APIs, use traditional Role-Based Access Controls (RBAC) (i.e Java, Springboot)</li></ul><h3>2. Broken Authentication</h3><p>When software developers incorrectly implemented authentication controls it causes a case of broken authentication.</p><ul><li>User authentication credentials are not protected when stored</li><li>Users using predictable login credentials</li><li>The session IDs are exposed in the URLs</li><li>The session ID is vulnerable to session fixation attacks? tricking a user to authenticate using a session ID we chose.</li><li>Session valued do not time-out or get invalidated after logout</li><li>Session ID’s are not rotated after successful logins</li><li>Passwords, session IDs and other sensitive credentials are sent over unsecured, unencrypted connections</li></ul><p>are some of the common situations of broken authentication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*m2j6y4Jp5hRseRsE" /></figure><p>broken authentication and session management</p><h3>How to prevent it?</h3><ul><li>User credentials must be protected using hashing or encryption when stored</li><li>Don’t expose the session ID in the URL</li><li>Sessions should expire after a time</li><li>Renew session ID with every login</li><li>Do not send credentials over unsecured, unencrypted connections. Use secure transfer protocols as TLS with perfect forward secrecy (PFS) ciphers and secure parameters.</li><li>Demand users to use complex passwords</li><li>Enforce account disabling after a number of failed attempts to log in</li><li>Multi-factor Authentication</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/0*mIGQqkNR5duw1g1D" /></figure><h3>3. Excessive Data Exposure</h3><p>When APIs retrieve full data objects from the backend database and relies on clients to filter the data out, an attacker can use this to their advantage and make it reveal information that a UI would normally filter out.</p><h3>How to prevent it?</h3><ul><li>Never rely on clients to filter out data!</li><li>Define schemas for API responses carefully</li><li>Define all the sensitive data and justify its use, clarify what’s sensitive according to privacy laws, regulatory requirements or business needs</li><li>Enforce response checks to prevent accidental leaks</li><li>Discard sensitive data after it served its purpose or use PCI DSS compliant tokenization or even truncation.</li><li>Encrypt all sensitive data, use strong algorithms</li></ul><h3>4. Lack of Resources &amp; Rate Limiting</h3><p>APIs generally don’t put limitations on the no of requests that can be requested from a client or a server. This might result in attackers sending requested more than the API could handle, basically overloading it. It can affect the API’s processing speed essentially clogging it up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*-co81_vQvXtjAcANHxUbig.png" /></figure><p><a href="https://stripe.com/blog/rate-limiters">https://stripe.com/blog/rate-limiters</a></p><h3>How to prevent it?</h3><ul><li>Limit the payload size</li><li>Define proper rate limiting</li><li>Add checks on compression ratios</li><li>Define limits for container resources</li></ul><p>Imagine a limited API being spoken via a backend process, and you want to limit the API for 20 requests per second, then you can have 1 process to allow those 20 requests to pass through.</p><p>Below is an example of how a rate limiter works in Node.js.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*HpVMuwINB2yRvBro" /></figure><p><a href="https://apisyouwonthate.com/blog/what-is-api-rate-limiting-all-about">https://apisyouwonthate.com/blog/what-is-api-rate-limiting-all-about</a></p><h3>5. Broken Function Level Authorization</h3><p>Overly complicated hierarchies, groups and roles and unclear separation between administrative and regular functions lead to authorization flaws. When such administrative functions are exposed, non-privileged users can access these functions if they know how to.</p><p>Example:</p><pre>/api/users/v1/user/myinfo</pre><pre>/api/admins/v1/users/all</pre><h3>How to Prevent it?</h3><ul><li>Do not rely on the client to enforce admin access</li><li>Deny all access by default (Implicit Deny)</li><li>Group operations by roles</li><li>Test authorization to and use proper design principles</li></ul><h3>6. Mass Assignment</h3><p>The best way to explain this vulnerability is to demonstrate it in an example.</p><p>The concept of a Mass Assignment is that often times, a web form will implement a model binding, by taking the information sent in the POST request and binding it with the relevant model in the database.</p><p>The API takes the data the client provides and stores it without proper filtering or sanitization for whitelisted properties. Thus, blindly transforms the user-provided payload into an object of the API and stores it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*EKrFsA6-lIFiJq3OfoBtaA.png" /></figure><p>node.js takes the information provided in the request body and modifies the object</p><h3>How to prevent it?</h3><ul><li>Do not automatically bind user-provided data with internal objects</li><li>Explicitly define all parameters and payloads you are expecting</li><li>Use readOnly property on an object that doesn’t need to be modified</li><li>Precisely define the patterns and schemas you will accept in requests</li></ul><h3>7. Security Misconfiguration</h3><p>Poor configuration of security in servers causes attackers to exploit them. Also leaving security configs as defaults can cause the same. Imagine a scenario when a server’s directory listing is not disabled by default. An attacker can simply list directories to find any file and execute it and even get a hold of the source code of the application and find more vulnerabilities.</p><p>Another scenario is when unnecessary administration ports being open for an application, which exposes the application to remote attacks.</p><p>Also, un-updated legacy applications might contain vulnerabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*zH6QqMIJRWx1jGvk.jpg" /></figure><p><a href="https://www.tutorialspoint.com/security_testing/testing_security_misconfiguration.htm">https://www.tutorialspoint.com/security_testing/testing_security_misconfiguration.htm</a></p><h3>How to prevent it?</h3><ul><li>Establish a repeatable hardening and patching process</li><li>Automate locating configuration flaws</li><li>Disable unnecessary features/ ports.</li><li>Restrict administrative access</li><li>Define and enforce all outputs, including errors. (Make sure you don’t display the errors to clients)</li><li>Least privilege access</li></ul><h3>8. Injection</h3><p>Injection flaws occur when developers create dynamic database queries that include user-supplied input and attackers use that construct API calls that SQL, NoSQL, LDAP, OS or other commands that the API or the backend behind it blindly executes. Therefore, to prevent such attacks from happening, developers must either:</p><ol><li>Stop writing dynamic queries</li><li>Prevent user-supplied input which contains malicious SQL from affecting the logic of the executed query</li></ol><h3>How to prevent it?</h3><p><strong>Setup your primary defenses:</strong></p><ol><li>Use of prepared statements with parameterized queries</li><li>Use of stored procedures</li><li>Whitelist Input Validation</li><li>Escaping All user-supplied input</li></ol><p><strong>Additional Defenses:</strong></p><ol><li>Enforcing the least privilege</li><li>Performing whitelist input validation as a secondary defense</li></ol><pre><em>// Vulnerable to injection attacks — no pramertarization</em></pre><pre>export const <strong>one</strong> = async (id: number) =&gt; {</pre><pre>return new <strong>Promise</strong>((resolve, reject)) =&gt; {</pre><pre>Connection.<strong>query</strong>(‘SELECT * from blogs WHERE id = ${id}’, (err,results)) =&gt; {</pre><pre>if(err){</pre><pre>return <strong>reject</strong>(err)</pre><pre>}</pre><pre><strong>resolve</strong>(results);</pre><pre>});</pre><pre>});</pre><pre>}</pre><pre><em>// After implementing params</em></pre><pre>export const <strong>one</strong> = async (id: number) =&gt; {</pre><pre>return new <strong>Promise</strong>((resolve, reject)) =&gt; {</pre><pre>Connection.<strong>query</strong>(‘SELECT * from blogs WHERE id = ?’, [id] (err,results)) =&gt; {</pre><pre>if(err){</pre><pre>return <strong>reject</strong>(err)</pre><pre>}</pre><pre><strong>resolve</strong>(results);</pre><pre>});</pre><pre>});</pre><pre>}</pre><h3>9. Improper asset management</h3><p>APIs prefer to reveal more endpoints than traditional web apps, giving high priority to adequate and modified information. Adequate hosts and installed versions of API implementations still play an important role in minimizing problems such as obsolete variants of APIs and open debug endpoints.</p><p>Attackers may find API that may still be in its early stages (beta, testing, staging, etc) and take advantage of the fact that it not being well protected as a post-production API, and use it to deploy their attacks.</p><p>Many of the DevOps, cloud, Kubernetes environments used in development are vulnerable to this since they use API that is mostly in testing stages.</p><p>Desire to make their software backward compatible leaves it being vulnerable to being exploited since it still relies on older outdated software.</p><p>Not maintaining proper versions and not updating their APIs puts the Application in danger as well.</p><h3>How to prevent it?</h3><ul><li>Keep an up to date inventory of all APIS</li><li>Limit access to anything that should not be public</li><li>Limit access to production data, and segregate access to production and non-production data.</li><li>Implement additional external controls, such as API firewalls.</li><li>Properly retire old versions of APIs or backport security fixes to them.</li><li>Implement strict authentication, redirects, CORS, and so forth</li></ul><p><strong>Insufficient logging and monitoring</strong></p><p>Lack of proper logging of the processes that happen within the API, (the user logins, etc) and consistent monitoring of those logs to detect anomalies is the number 10 and the final vulnerability of OWASP top 10 for APIs.</p><p>Emphasis on the point monitoring since, regardless of how well you perform the logging of the daily tasks of your API/ Web Application, it is equally, if not more important to monitor them to find anomalies, if they may be one.</p><p>One of the most overlooked aspects of logging is not securing the log files. An attacker shouldn’t be able to access that file and modify it. Its integrity must be protected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*5QCreCubTAFbNQTi" /></figure><p>logging failed login attempts</p><h3>How to prevent it?</h3><ol><li>Log failed attempts, denied access, input validation failures, or any failures in security policy checks.</li><li>Ensure that logs are formatted so that other tools can consume them as well.</li><li>Protect logs like highly sensitive information.</li><li>Include enough detail to identify attackers.</li><li>Avoid having sensitive data in logs — if you need the information for debugging purposes, redact it partially.</li><li>Integrate with SIEMs and other dashboards, monitoring, and alerting tools.</li></ol><h3>Resources —</h3><p>OWASP API Security Project — <a href="https://www.owasp.org/index.php/OWASP_API_Security_Project">https://www.owasp.org/index.php/OWASP_API_Security_Project</a></p><p>API Security Articles — <a href="https://apisecurity.io/">https://apisecurity.io/</a></p><p>What is API security? Redhat — <a href="https://www.redhat.com/en/topics/security/api-security">https://www.redhat.com/en/topics/security/api-security</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60169a38a7ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Approach to Implementing ISO 27001 — summary]]></title>
            <link>https://ruweeniddagoda.medium.com/practical-approach-to-implementing-iso-27001-summary-744a6b99f50c?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/744a6b99f50c</guid>
            <category><![CDATA[ciso]]></category>
            <category><![CDATA[iso27001implementation]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 09:57:10 GMT</pubDate>
            <atom:updated>2022-02-21T09:57:10.291Z</atom:updated>
            <content:encoded><![CDATA[<h3>Practical Approach to Implementing ISO 27001 — summary</h3><h3>What is ISO 27001?</h3><ul><li>It is one of the standards introduced by the International Organization of Standardization (ISO) to assist in managing the security of assets such as intellectual property, financial information, employee details or information entrusted to you by third parties.</li><li>Basically, it provides the requirements for an Information Security Management System (ISMS).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*odLqB07y50beopGy" /></figure><p><a href="https://www.iso27001security.com/html/toolkit.html">https://www.iso27001security.com/html/toolkit.html</a></p><h3>Why do we need it?</h3><ul><li>For software development companies, it proves that the company has a baseline security perimeter implemented and also, that the company takes the security aspect seriously, ensuring the client’s trust.</li><li>Other companies prefer working with a company that has an independent assessment of security, which adds extra weight for your credibility.</li><li>Management support is required from end to end</li><li>To get that build a proper case study</li><li>management should be aware of the cost and be on-board.</li></ul><h3>Process Overview</h3><h4>Step 1:</h4><ul><li>The primary goal before implementing ISO standards should be to get the Management on board with the costs associated with it. In order to accomplish that, a proper case study must be presented.</li></ul><h4>Step 2:</h4><ul><li>Initially, define the scope of the implementation, whether it be for a single location, or for multiple server farms, etc. The scope can be expanded later as the requirements precede.</li></ul><h4>Step 3:</h4><ul><li>Identify the assets that are within the scope and have information value.</li><li>Figure out the high-value assets and calculate the risk value.</li></ul><h4>Step 4:</h4><ul><li>Give solutions to reduce the risks.</li><li>Select controls to mitigate risks.</li><li>Create the Statement of Applicability (SOA).</li></ul><h4>Step 5:</h4><ul><li>Implement the controls as defined in the SOA</li></ul><h4>Step 6:</h4><ul><li>Once implementation is done, an audit will be performed and the certification will be given.</li></ul><h3>Business Plan</h3><ul><li>The business case document will be created defining the costs and benefits of implementing ISO 27000. It’ll provide a structured approach to a comprehensive description of the risks associated with the assets and the benefits of securing them.</li><li>Each of the controls that are to be implemented for the purpose of securing the asset will be associated with cost. Some may cost you a lot secure, even more than the value of the asset itself. In such cases, you must evaluate the impact and determine what approach to be taken.</li><li>Ultimately, layout an explicit and comprehensive understanding of the ISMS for the management.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*TRBZEtmU9PmibEBFFg-67w.png" /></figure><p><a href="https://www.iso27001security.com/ISO27k_Generic_business_case_for_ISO_IEC_27001_ISMS_v2.docx">https://www.iso27001security.com/ISO27k_Generic_business_case_for_ISO_IEC_27001_ISMS_v2.docx</a></p><h3>Asset Inventory</h3><ul><li><strong>Asset Register </strong>— is a list of the assets owned by a business. It contains pertinent details about each fixed asset to track their value and physical location. The register shows the quantity and value of things like office equipment, motor vehicles, furniture, computers, communications systems, and equipment. People or the human capital of a company is also considered as assets. <strong>Although, only the assets with an information value will be considered in this particular situation.</strong></li></ul><ol><li>Asset ID — Every asset should have an identification code for accountability</li><li>Owner — The entity which owns the asset, should be accountable for the asset</li><li>Custodian — The entity which looks after the asset</li><li>User — The entity which actively uses the asset</li><li>Location — Where is the file located?</li><li>Storage — What type of encryption is used?</li><li>Confidentiality requirement? — High, Medium, Low (A numeric value will be allocated based on the input you give)</li><li>Integrity requirement? — High, Medium, Low (A numeric value will be allocated based on the input you give)</li><li>Availability requirement? — High, Medium, Low (A numeric value will be allocated based on the input you give)</li><li>Classification level — confidential or not? internal/public? can even be a custom classification.</li><li>Disposal method — What to do when the life cycle expires? how to dispose of the asset?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKLGwqEYKxxcbk0SO1fxRQ.png" /></figure><p>Asset Register (Inventory) example — isms.online</p><h3>Risk Assesment</h3><ul><li>Assume that your asset is a desktop PC, and it may have a couple of vulnerabilities (virus guard, not up-date, Operating System not patched with the latest security patches, etc). The threat in this situation is that the PC in context might get affected with a virus.</li><li>Risk is the probability of a given vulnerability of an asset is exposed to a threat.</li><li>Identify the assets with high-value and more risk and apply more controls to ensure it’s security.</li><li>In order to come up with a quantitative value for the risk, we use the following equation based on three factors.</li></ul><blockquote><em>Risk Value = Probability * Impact * Undetectability</em></blockquote><ol><li>Probability (0–5) — the chance of a vulnerability being exploited</li><li>Impact(0–5) — the impact it’ll have on the system</li><li>Undetectability(0–5) — Whether this particular vulnerability is detectable or not?</li></ol><h3>Risk Treatment</h3><ul><li>In this section, we discuss the treatments that we can apply for the risks we identified and calculated in the previous sections. The key idea is that there’s no single solution for every asset or scenario. Each situation requires a different tactic.</li></ul><h4>Risk Management</h4><ol><li>Risk Avoidance — If the risk can be avoided completely, no controls are required.</li><li>Risk Reduction — Fewer controls should be placed.</li><li>Risk transfer — We can insure the asset and transfer the risk.</li><li>Risk Accept — In some scenarios, we have no choice but to accept the risk.</li></ol><ul><li>Documenting the treatments and the planning for the risk management process, we generate one of the important documents of the ISO 27000 standard — Statement of Applicability (SOA). We use this as a blueprint to implement the controls.</li><li>In this document, we define why it’s needed to apply the controls and what requirements demand it.</li><li>LR — Legal Requirements, CO — Customer Obligations, BR — Business Requirements, RRA — Risk Assessment</li><li>Then create a risk treatment plan based on the controls you selected.</li><li>You can read about best practices for implementing ISO 27001 standard from the official ISO documentation — Although, it is required to be bought for any commercial use.</li><li>One the implementation process is completed, we collect three months of data to provide as evidence for the auditing process, which evaluated the efficiency of our ISMS. <em>Longer you’ve been practicing ISO 27000, the more secure you are.</em></li></ul><p>Resource : <a href="https://www.udemy.com/course/implementing-iso27001/">https://www.udemy.com/course/implementing-iso27001/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=744a6b99f50c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to setup Multi-factor Authentication with OpenVPN Community Edition]]></title>
            <link>https://ruweeniddagoda.medium.com/how-to-setup-multi-factor-authentication-with-openvpn-community-edition-3c2f1586ad0b?source=rss-b082bd7b5f97------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c2f1586ad0b</guid>
            <category><![CDATA[openvpn]]></category>
            <category><![CDATA[free]]></category>
            <category><![CDATA[mfa]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[google-authenticator]]></category>
            <dc:creator><![CDATA[Ruween Iddagoda]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 09:53:31 GMT</pubDate>
            <atom:updated>2022-02-21T09:53:31.325Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview of the Process</h3><ol><li>User executes google-authenticator to generate the QR code and verification code (secrets)</li><li>User connects a MFA enabled device by scanning the QR code presented.</li><li>Verfication code must be installed (stored) onto the VPN server itself</li><li>During the authentication, OpenVPN will call the PAM module to perform verification on the user submitted password (OTP code) against the secret shared by the user.</li><li>Upon verifying, users are able to connect to the VPN.</li></ol><h3>Initial Configurations</h3><p><strong>Pre-requisites</strong></p><ol><li>OpenVPN server community edition installed Instance with root privileges</li><li>Google Authenticator enabled device</li><li>OpenVPN client installed device</li></ol><p><strong>Server-side</strong></p><ol><li>Create a new directory called /etc/google-auth</li></ol><pre>$ mkdir /etc/google-auth</pre><p>2. Change permissions to 0400 since PAM does not prefer world/group readable objects</p><pre>$ chown -R root /etc/google-auth</pre><p>3. Install libpam-google-authenticator and libqrencode3 packages</p><pre>$ sudo apt install libqrencode3 libpam-google-authenticator</pre><p>4. Add the following line to your openvpn server config file to ask openvpn to authenticate against libpam , which has it’s own google-auth module</p><pre>plugin /usr/lib/openvpn/openvpn-plugin-auth-pam.so openvpn</pre><ul><li>This states openvpn will use the pam authentication method with pam auth id as <em>openvpn</em></li></ul><p>5. To create the pam setup, create a new file named /etc/pam.d/openvpn and add the following lines</p><pre>auth    requisite       /lib/security/pam_google_authenticator.so secret=/etc/google-auth/${USER}  user=root<br>account    required     pam_permit.so</pre><ul><li>By auth requisite we state that if unsuccessful the authentication will fail immediately.</li><li>By secret= we specify the location where the verification codes are stored and root user can access it.</li><li>In the last line we state that we allow the connection if successful</li></ul><p>6. Append the client .ovpn file with the following</p><pre>auth-user-pass<br>auth-nocache<br>reneg-sec 0</pre><ul><li>auth-user-pass — for openvpn-client to ask for username and password</li><li>auth-nocache — to not cache the password since it’s changing periodically</li><li>reneg-sec 0 — to not re-negotiate periodically</li></ul><p><strong>Client-side</strong></p><ol><li>Install libpam-google-authenticator and libqrencode3 packages</li></ol><pre>$ sudo apt install libqrencode3 libpam-google-authenticator</pre><p>2. Execute google-authenticator from the terminal</p><pre>$ google-authenticator</pre><p>3. Scan the generated QR code using Google Authenticator enabled device.</p><p>4. Proceed by providing y to all prompts. The secret file will be store in your $HOME directory as a hidden file.</p><p>5. Make sure to backup the verification codes and other secrets as well.</p><p>6. To view the the secret file use the command</p><pre>$ ls -a | egrep &#39;^\.&#39;</pre><pre>OUTPUT<br>.<br>..<br>.google-authenticator</pre><p>7. Rename the secret file to your openvpn username</p><pre>$ mv ~/.google-authenticator someuser</pre><p>8. Transfer the file to the VPN server.</p><ul><li>The server must store this file in the /etc/google-auth directory</li></ul><p><strong>How to connect?</strong></p><ul><li>Use the openvpn — config command to connect as usual. The openvpn-client will prompt for the username and password (OTP code).</li></ul><pre>$ sudo openvpn --config &lt;client_config&gt;.ovpn</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c2f1586ad0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>