<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prasenjit Roychoudhury on Medium]]></title>
        <description><![CDATA[Stories by Prasenjit Roychoudhury on Medium]]></description>
        <link>https://medium.com/@prasenjitrc?source=rss-f86efc65df6c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xmZJl1ysxjmoQXsn-9JMyA.jpeg</url>
            <title>Stories by Prasenjit Roychoudhury on Medium</title>
            <link>https://medium.com/@prasenjitrc?source=rss-f86efc65df6c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:02:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prasenjitrc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Claude Certificed Architect-Foundation- Learning Journey]]></title>
            <link>https://medium.com/@prasenjitrc/claude-certificed-architect-foundation-learning-journey-573a0ead270a?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/573a0ead270a</guid>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[claude-certified]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 23:22:46 GMT</pubDate>
            <atom:updated>2026-06-12T21:34:51.600Z</atom:updated>
            <content:encoded><![CDATA[<p>I have completed Anthropic Claude Certified Architect — Foundation certification which is available to partners. Following is my learning journey.</p><ol><li><strong>Why I did this certification?</strong></li></ol><ul><li>Because the certification is on Anthropic Claude that I think has immense and unique potential of solving many problems. Also the certification has limited access(only for partners) for now.</li><li>I am interested in prompt engineering concepts and working with Claude extensively for last 5 months.</li><li>I always look forward to encourage anyone who is interested in GenAI and participate on overall AI learning and adoption in right use case.</li></ul><p><strong>2. Preparation/My training</strong></p><p>a. The YouTube <a href="https://www.youtube.com/watch?v=VvKDoV39jAk&amp;list=PLmiDJB5zE0KGwFtDCSqHfQG7SV6VeQ_31">public video tutorial</a> I found very useful. These covers all concepts related to exam.</p><p>b. I have appeared for Udemy practices tests multiple time which I have found very close to actual exam. <a href="https://www.udemy.com/course/claude-certified-architect-y/?couponCode=KEEPLEARNING">This set</a> has 6 exam sets. My practice scores before exam had been 90% to 94%.</p><p>c. This <a href="https://www.udemy.com/course/anthropic-claude-certified-architect-3-full-practice-exams/">Udemy practice test </a>has 3 practice tests which also helped me to validate the concepts. The scores were on 84% to 90% range.</p><p>d. I appeared for Anthropic practice test which I found most akin to exam and I appreared one day before my final exam and score was 826.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*vOAghYKvenA14hvQh5K8wg.png" /><figcaption>Anthropic Claude Certification</figcaption></figure><p><strong>3. Actual Exam Experiece</strong></p><ul><li>Struggled on actual use cases for few shots during preparation and exam.</li><li>In exam the use cases on What not to do is more important than what to do. In practical word of Generative AI I think the same concept prevails.</li><li>Not much to keep in memory as exam is more of conceptual. There is only one question I received in exam for which I had not have concept from above test cases.</li><li>I used claude for wrong answers analysis during my preparation and practice exams. On final day I asked Claude to give summary of concepts on which I struggled during preparation. This strategy helped me.</li><li>The exam is not very stressful and majority of questions are not having too many paragraphs. But the questions were very conceptual.</li><li>In exam I was confident on 50 questions and 9 questions I did answer on my understanding and 1 was completely randomly selected.</li><li>I completed preparation in 5 weeks and I have fundamental knowledge but limited GenAI experience in production. I have very limited prompt engineering concepts which I learnt a lot in the process.</li><li>Final result is less relevant as in AI and always training matters and actual result is directly propertional to training. What went wrong is more important than what can go right. My final score is 825.</li><li>The exam score came up after 5 working days. After exam submission it was still showing to resume and resubmit. The exam can be given anytime 24*7 and doesn’t need pre booking slot.</li><li><em>Disclaimer — I deserve to be wrong on above like some cases of AI and above are my experiences and perception. I learnt a lot on Claude Certification. I shall like to pursue upcoming professional certification as well.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=573a0ead270a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating Amazon S3 Vector for RAG — Fast, Accurate, and Low-Latency Retrievals]]></title>
            <link>https://aws.plainenglish.io/evaluating-amazon-s3-vector-for-rag-fast-accurate-and-low-latency-retrievals-194c66df2e0a?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/194c66df2e0a</guid>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 11:46:18 GMT</pubDate>
            <atom:updated>2025-11-27T20:10:19.424Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Evaluating Amazon S3 Vector for RAG — Fast, Accurate, and Low-Latency Retrievals</strong></h3><h3>1. Introduction</h3><p>In<a href="https://aws.amazon.com/blogs/aws/introducing-amazon-s3-vectors-first-cloud-storage-with-native-vector-support-at-scale/"> July,25 AWS</a> released Amazon S3 Vector which can work in RAG for Generative AI while cost can reduce upto 90%. Retrieval-Augmented Generation (RAG) systems rely on fast and accurate retrieval from large document stores to provide grounded responses. Traditionally, databases like PostgreSQL with pgvector extension have been used for vector similarity search. However, Amazon S3 Vector introduces a new serverless retrieval paradigm that promises scale, low latency, and minimal management overhead.</p><p>This paper evaluates S3 Vector performance for various sizes for Knowledge Base size and queries. The experiments are based on realistic workloads using Amazon Bedrock embedding models and a synthetic knowledge base simulating planetary science content. Further the Proof of Concept is used for comparison with knowledge base created in Amazon Aurora ServerlessV2.</p><p>We followed a simple but very structured sequence to run this study. First, we <strong>generated a synthetic planetary science knowledge base</strong>, starting with a few dozen documents and then scaling up to <strong>50, 100, 500, and 1,000 text files</strong>, each containing realistic, fact-style content about planets, moons, or orbital characteristics. Next, we built a common ingestion pipeline that <strong>chunked each document (1200 and 1800 characters), embedded every chunk using Amazon Titan Text Embeddings v2, and wrote those vectors into two separate backends</strong>: Amazon S3 Vector and Amazon Aurora PostgreSQL ServerlessV2 with pgvector, using the same keys, metadata, and chunking rules so both indexes stayed perfectly aligned. In parallel, we <strong>curated a query set</strong> of domain-specific questions (e.g., “Which planet has the shortest orbital period?”) and a <strong>qrels file</strong> that mapped each question to the “correct” chunk, effectively defining our ground truth. With the data and indexes ready, we ran a dedicated evaluation Lambda that, for each query, <strong>re-embedded the question, performed top-K vector search on both S3 Vector and Aurora, and computed retrieval metrics</strong> such as Recall@10, nDCG@10, MRR, Grounded@1, Hallucination@10, and latency timings. Finally, we <strong>validated the outputs</strong> by checking that qrels were present in the retrieved sets, inspecting ranking positions, and confirming that the metrics behaved consistently across multiple runs and corpus sizes, giving us a clean, end-to-end view of how S3 Vector behaves as a RAG retrieval engine. Codes for embedding and retrieval are uploaded in <a href="https://github.com/PrasenjitRoychoudhury/s3VectorRAGValidation">GitHub</a>.</p><h3>2. Use Case</h3><p>To evaluate retrieval performance for RAG workloads, a synthetic knowledge base was constructed containing <strong>planetary science documents</strong> stored in Amazon S3. These documents were ingested twice using the <strong>same ingestion Lambda function</strong>, producing two independent vector stores:</p><ol><li><strong>Amazon S3 Vector Search</strong> (serverless vector index)</li><li><strong>Amazon Aurora Serverless v2 PostgreSQL with pgvector</strong> (database-managed vector index)</li></ol><p>Both ingestion pipelines use <strong>identical source files, chunking strategy (1800 chars, 200 overlap), and embeddings generated by <em>Amazon Titan Text Embeddings v2</em></strong>, ensuring a consistent and fair comparison.</p><p>The evaluation consists of two core workloads:</p><h3>A. Corpus Ingestion Workload</h3><ul><li>Reads documents from S3 (planet50, planet500, planet1000 test sets).</li><li>Chunks documents deterministically.</li><li>Embeds with Titan v2 (1024-dim cosine).</li><li>Writes vectors + metadata either to: <strong>S3 Vector</strong>, or <strong>Aurora pgvector</strong> table (kb_chunks)</li><li>Captures ingestion latency distribution (p50, p90, p95, p99). Uses KEY_MODE=stable to ensure deterministic vector keys across runs.</li></ul><p>This allows comparison of <strong>write path performance</strong> for both engines as the KB grows.</p><h3>B. Retrieval &amp; Evaluation Workload</h3><p>A second Lambda function performs RAG-style retrieval following this sequence:</p><p>a. Reads <strong>queries</strong> and <strong>qrels</strong> from S3.</p><p>b. Embeds each query using Titan v2.</p><p>c. Issues vector search requests against either:</p><ul><li><strong>S3 Vector</strong> (vector.search API)</li><li><strong>Aurora pgvector</strong> (ORDER BY embedding &lt;=&gt; query LIMIT K)</li></ul><p>d. Captures full retrieval metrics:</p><ul><li><strong>Recall@10</strong>, <strong>nDCG@10</strong>, <strong>MRR@10</strong></li><li><strong>Grounded@1</strong> and <strong>Hallucination@K</strong></li><li><strong>Avg First-Relevant Rank</strong></li><li><strong>Distance diagnostics</strong> (Top-1 distance, relevant-hit distances)</li><li><strong>Per-query latency (ms)</strong></li></ul><p>The retrieval workload mirrors real RAG systems that must quickly and accurately fetch the top-K relevant chunks for summary generation.</p><h3>3. Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ddbMo7Nd7vxsURRdWNvRsQ.png" /><figcaption><strong>Architecture</strong></figcaption></figure><h3>4. Analysis</h3><p>The experiments reveal that <strong>Amazon S3 Vector delivers remarkably strong retrieval quality</strong> for RAG systems, consistently outperforming expectations as the knowledge base scales. At medium and large corpus sizes (100–500 files), S3 Vector achieves near-perfect semantic search accuracy with <strong>Recall@10 between 0.95 and 1.00</strong>, backed by stable ranking, high groundedness, and almost zero hallucinations. The index architecture handles expansion smoothly, preserving precision even as vector count grows. Although latency remains comfortably low (~50–60 ms) in most runs, occasional spikes appear in large-file, large-chunk (1800) evaluations — an expected outcome for a serverless, partition-based engine under cold-start or heavy-load conditions. At the 1000-file scale, retrieval quality drops modestly due to chunking density and dataset complexity, but still remains usable. Overall, the results demonstrate that <strong>S3 Vector combines the accuracy of a purpose-built vector index with the simplicity and elasticity of a serverless design</strong>, making it a highly effective choice for enterprise-grade RAG workloads.</p><ul><li><strong>Near-perfect retrieval accuracy</strong> (0.95–1.00 Recall@10) for 100–500 file workloads.</li><li><strong>Highly grounded responses</strong>, with Grounded@1 frequently between <strong>0.90 and 1.00</strong>.</li><li><strong>Minimal hallucination</strong>, often <strong>0%–5%</strong>, even at large vector counts.</li><li><strong>Latency stable</strong> around <strong>50–60 ms</strong> in typical runs, with expected spikes for cold partitions or large chunks.</li><li><strong>Scales effectively</strong> with corpus size; accuracy grows from 50 → 500 files.</li><li><strong>Performance dip at 1000 files</strong> highlights need for tuned chunk sizes and warm indexing.</li><li><strong>Fully serverless</strong>, eliminating cluster management, vacuuming, indexing overhead, and scaling decisions.</li></ul><p><strong>Overall:</strong> S3 Vector is a robust, accurate, and operationally efficient vector store ideally suited for Retrieval-Augmented Generation systems at small-to-large scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UNfTJaOAPdWjcpDEDWkTgg.png" /><figcaption>Data Collection after run</figcaption></figure><h3>5. What Services Are Used</h3><ul><li><strong>Amazon S3</strong> — Used to host the synthetic knowledge-base text files (planet50, planet100, planet500, planet1000 test sets) and also to store ingestion logs, evaluation query sets, and result output artifacts.</li><li><strong>AWS Lambda</strong> — Two distinct Lambda functions are used: one for ingestion (continually reading files, chunking, embedding via Amazon Titan Embed Text v2, and writing vectors to S3 Vector or Aurora), and a second for retrieval/evaluation (reading queries and qrels, embedding each query, issuing search requests, computing metrics).</li><li><strong>Amazon S3 Vector</strong> — A serverless vector-search engine into which embeddings + metadata are written and from which retrieval queries are executed. This service supports high-scale similarity search without cluster management.</li><li><strong>Amazon Aurora PostgreSQL Serverless V2 with pgvector extension</strong> — A managed relational database augmented with vector search capability via the pgvector extension. Maintains an index of embedding vectors and supports similarity search (via ORDER BY embedding &lt;=&gt; query LIMIT K).</li><li><strong>Amazon Bedrock</strong> — Provides the Titan Text Embeddings v2 model used for encoding both document chunks and query embeddings, ensuring a consistent embedding platform across ingestion and retrieval phases.</li><li><strong>AWS CloudWatch / Logging Services</strong> — Used to capture ingestion latency (p50/p90/p95/p99), retrieval latency, ranking diagnostics, and error logs across both engines.</li><li><strong>AWS IAM &amp; Permissions</strong> — Proper IAM-roles and permissions ensure Lambda functions can read from S3, invoke Bedrock, write to S3 Vector and Aurora, and log to CloudWatch.</li></ul><h3>6. Clean up</h3><p>After the experiments and evaluations are completed, all resources created for the study are dismantled to avoid ongoing cost and to leave a clean account footprint. The clean-up steps include:</p><ul><li><strong>Amazon S3</strong> — Delete the test knowledge-base buckets/folders (planet50, planet100, planet500, planet1000), ingestion output logs, query/qrel files and result artifacts.</li><li><strong>AWS Lambda</strong> — Delete the ingestion and retrieval/evaluation Lambda functions (including alias/versions) and associated triggers/configurations.</li><li><strong>Amazon S3 Vector</strong> — Remove or purge the vector indexes created for each corpus size to free associated storage and indexing metadata.</li><li><strong>Amazon Aurora Serverless V2</strong> — Drop the pgvector-enabled database instances used for the evaluation and release any associated capacity.</li><li><strong>Amazon CloudWatch</strong> — Delete log groups or streams created by the ingestion and retrieval Lambdas, and optionally remove metrics dashboards or alarms created for the study.</li><li><strong>IAM</strong> — Remove or revoke any dedicated IAM roles, policies or permissions that were created solely for the study.</li></ul><p>By thoroughly cleaning up the resources, the study avoids persistent costs and retains a zero or minimal footprint in the account while still delivering the full evaluation results.</p><h3>7. Conclusion</h3><p>This study demonstrates that <strong>Amazon S3 Vector delivers the highest retrieval accuracy, best groundedness, and most reliable semantic relevance</strong> across all knowledge base sizes tested — 50, 100, 500, and 1,000 files. As the corpus grows, S3 Vector consistently improves, achieving <strong>near-perfect to perfect Recall@10 (0.95–1.00)</strong> in the 100- and 500-file workloads, with extremely low hallucination rates and stable ranking behavior. The engine shows strong robustness under repeated evaluations, maintaining high-quality retrieval even when vector counts increase and query loads become heavier.</p><p>Latency remains competitive in normal workloads, typically around <strong>50–60 ms</strong>, though larger chunk sizes and larger corpora occasionally introduce <strong>cold-start or partition-level latency spikes</strong>, especially in 1800-character chunk tests. Importantly, these latency variations do <strong>not</strong> degrade retrieval quality, which stays uniformly high — a key requirement for reliable grounding in RAG systems.</p><p>The 1000-file run highlights the natural limitations of chunk density and dataset overlap, where retrieval quality drops modestly, reinforcing the importance of optimized chunking strategies at very large scales. Despite this, the system continues to deliver usable results and predictable latency behavior.</p><p>Overall, the findings are clear: <strong>S3 Vector provides a high-accuracy, low-hallucination, serverless retrieval engine ideally suited for enterprise RAG workloads</strong>, combining the strengths of a purpose-built vector index with the simplicity of S3’s operational model. For teams building scalable, grounded, and cost-efficient GenAI applications, S3 Vector stands out as a powerful and production-ready choice.</p><p>The retrieval workload mirrors real RAG systems that must quickly and accurately fetch the top-K relevant chunks for summary generation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=194c66df2e0a" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/evaluating-amazon-s3-vector-for-rag-fast-accurate-and-low-latency-retrievals-194c66df2e0a">Evaluating Amazon S3 Vector for RAG — Fast, Accurate, and Low-Latency Retrievals</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster Starts, Predictable Stops: Measuring AWS Fargate Graviton vs x86 Performance Across Spot and…]]></title>
            <link>https://towardsaws.com/amazon-ecs-fargate-on-aws-graviton2-spot-boots-up-4-faster-1fbf08b0812b?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fbf08b0812b</guid>
            <category><![CDATA[graviton2]]></category>
            <category><![CDATA[fargate-spot]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Tue, 11 Nov 2025 20:58:17 GMT</pubDate>
            <atom:updated>2025-11-12T05:13:45.588Z</atom:updated>
            <content:encoded><![CDATA[<h3>Faster Starts, Predictable Stops: Measuring AWS Fargate Graviton vs x86 Performance Across Spot and On-Demand</h3><h3>1. Introduction</h3><p>This paper is simple, short and informative to calculate start up time on different AWS Fargate infrastructure. The start up time is calculated in our AWS account by running similar docker(for ARM64 and x86) container with s<strong><em>ame parameters and same code</em></strong> running for 1046 times over 3 days.</p><p>Preliminary observation:<br>Spot capacity did <strong>not</strong> introduce measurable latency penalties in this region during the experiment window. Minor differences between On-Demand and Spot start times are discussed in Section 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*Wj2VqEU574jbs4-ac7iZrg.png" /></figure><p>A very important note is the exercise is done on very tiny container in US-EAST-2 region and cost may vary based on code, execution time, other fargate parameters and AWS region.</p><p>This is a continuation of our other following write ups.</p><p>a. <a href="https://medium.com/@prasenjitrc/aws-ecs-fargate-spot-on-arm64-9083e6b7e9cb"><em>AWS ECS Fargate Spot on Graviton2</em></a></p><p>b. <a href="https://medium.com/@prasenjitrc/interruption-simulation-of-fault-tolerant-tasks-in-aws-fargate-spot-3a841dc43b8f"><em>Simulation of interruption on Fault tolerant tasks in AWS Fargate Spot</em></a></p><p>c. <a href="https://medium.com/towards-aws/aws-fargate-graviton-spot-76-cheaper-d27f8c570a19"><strong><em>Amazon ECS Fargate on AWS Graviton2 Spot 76% cheaper</em></strong></a></p><h3>2. Use Case</h3><p>There is one container created from same codebase which has built twice on docker using ARM64(Graviton) and x86 architecture. There are 2 task definitions created using the Amazon ECR each for Graviton and ARM64(Graviton) with rest all memory, CPU parameters same. The containers are started using AWS Lambda and various times are captured using Amazon EventBridge.</p><p><em>latencyMs <br>issuedAt (AWS Lambda time)</em></p><p><em>acknowledgedAt<br>pendingAt <br>provisioningAt <br>runningAt <br>stoppedAt</em></p><p>A Lambda function triggers an ECS RunTask request every 2 minutes for ~1,000 invocations over 3 days. The task definitions have identical CPU (0.25 vCPU) and memory (512 MB) settings across the variants; only architecture and capacity provider differ. The Lambda logs the timestamps of task state transitions (PROVISIONING → PENDING → RUNNING) and records any failed/timeout launches from Spot capacity shortages.</p><h3>3. Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*e9SFAmun-1KibLC92-kKfA.png" /><figcaption>Architecture Diagram for POC</figcaption></figure><p>The code for container and AWS Lambda, docker files and commands added in the GitHub <a href="https://github.com/PrasenjitRoychoudhury/AWSECSFargateStartupComparison">link</a>.</p><h3>4. Analysis</h3><h3>Executive Summary</h3><ul><li>Dataset size: 4,180 task invocations (1046 per variant).</li><li>Median end-to-end (Lambda invocation → Running) latency ≈ <strong>26–28 seconds</strong> across all variants.</li><li>The <strong>slowest phase</strong> is consistently <strong>Provisioning → Pending</strong>, averaging ≈ 13–17 seconds.</li><li>The <strong>Pending → Running</strong> stage typically takes ≈ 9–13 seconds.</li></ul><blockquote><strong><em>Graviton (ARM64)</em></strong><em> variants start slightly faster than x86 (~5–10 % lower median total latency).</em></blockquote><ul><li><strong>Spot</strong> capacity shows no significant latency penalty vs On-Demand — indicating stable Fargate Spot availability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rv2_eDyDmsyjTpG6DYL-5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKCFsDkJHlVSZFyEHq3KuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*1zXnJFxtI51BEfxJsROxhQ.png" /></figure><h3>Phase-Specific Observations</h3><ul><li><strong>Provisioning → Pending</strong> averaged lower on ARM64, suggesting faster task placement or image fetch on Graviton.</li><li><strong>Pending → Running</strong> was marginally faster on x86, likely reflecting cached image pulls or network timing.</li><li><strong>Running → Stopped</strong> (task runtime) was significantly shorter for Graviton, showing stronger compute efficiency rather than infrastructure latency.</li></ul><h3>Spot vs On-Demand</h3><p>Across all phases, <strong>Spot capacity</strong> showed no sustained slowdown relative to On-Demand. Median differences were within a few percent, and p95 tails showed only minor spread. This indicates <strong>stable Spot availability</strong> in this region during the test window but should not be assumed universal for all regions or larger containers.</p><h3>Interpretation</h3><ul><li>Startup latency across variants is <strong>comparable</strong>, with ARM64 Spot slightly ahead on median and lowest runtime variance.</li><li>Control-plane speed is largely uniform across capacity types; the main observable gain for Graviton appears during execution.</li><li>Outlier behavior is minimal but non-zero (occasional &gt; 60 s Issuance→Running in x86 Spot).</li><li>These results confirm that <strong>ARM64 containers run efficiently and predictably on Fargate Spot</strong> without introducing startup risk for short-lived workloads.</li></ul><h3>5. What Services Are Used</h3><p><strong>Amazon ECR</strong> — <br> Two repositories were created to host Docker container images:</p><p>One built for <strong>x86 (amd64)</strong> architecture, and another built for <strong>Graviton (ARM64)</strong>. Both images contain identical application logic to ensure consistent execution time across architectures.</p><p><strong>Amazon ECS (Fargate Launch Type)</strong> — <br> Used to register <strong>task definitions</strong> and run <strong>standalone ECS tasks</strong> under four capacity variants:</p><p>a. Fargate x86 (On-Demand)</p><p>b. Fargate x86 Spot</p><p>c. Fargate Graviton (On-Demand)</p><p>d. Fargate Graviton Spot</p><p>Each task executes a short-lived workload and emits ECS state-change events (PROVISIONING → PENDING → RUNNING → STOPPED) that are captured for timing analysis.</p><p><strong>Amazon EventBridge</strong> — <br> Configured as the <strong>scheduler</strong> to periodically trigger the <strong>AWS Lambda</strong> function responsible for invoking the ECS tasks. EventBridge ensures consistent invocation intervals and maintains experiment repeatability across variants.</p><p><strong>AWS Lambda</strong><br> Implements the orchestration logic:</p><p>a. Receives the scheduled trigger. Invokes four ECS tasks (one per variant) in parallel, tagging each with its capacity type and architecture.</p><p>b. Records timestamps such as issuedAt, acknowledgedAt, and the ECS state transitions for detailed phase-by-phase latency measurement.<br> In previous cost-analysis experiments, this Lambda also calculated cost per run; in this study, it primarily measures <strong>startup latency and unavailability window</strong>.</p><p><strong>Architecture Compatibility Considerations (Graviton / ARM64)</strong> — <br> When deploying containers on <strong>Graviton (ARM64)</strong> instances, the following best practices were applied:</p><ul><li>Ensure the <strong>application code, dependencies, and base images</strong> are ARM64-compatible. Build Docker images specifically for ARM64 or use <strong>multi-architecture manifests</strong> to support both x86 and ARM seamlessly.</li></ul><p>The dedicated <strong>Dockerfile for the Graviton build</strong> has been included in the GitHub repository linked in the paper.</p><h3>6. Clean up</h3><ul><li>Amazon ECR — Delete the repositories .</li><li>AWS ECS — Delete task definition and AWS ECS Cluster.</li><li>Amazon EventBridge — The scheduler to mark disable.</li><li>AWS Lambda — Delete the function.</li><li>Amazon CloudWatch Logs — Delete the Amazon CloudWatch loggroups.</li><li>AWS DynamoDB — Delete the NoSQL database table.</li></ul><h3>7. Conclusion</h3><p>This study demonstrates that <strong>AWS Fargate with Graviton Spot instances</strong> delivers the fastest and most consistent <strong>startup performance</strong> among the four variants tested — x86, x86 Spot, Graviton, and Graviton Spot.<br> Graviton-based tasks show comparable or slightly better end-to-end startup times than x86 in this test, with a clear advantage in execution time (Running→Stopped) for our workload. <strong>Also Spot instances boots up faster than on demand</strong>. Arm64 spot didn’t get invoked for capacity constrain 4 times out of 1046 times invocation and x86 Spot never fails to start.</p><p>The <strong>RUNNING → STOPPED</strong> phase highlights that Graviton instances not only start faster but also complete workloads significantly sooner than x86 tasks. Spot capacity shows <strong>no meaningful degradation or pre-emption impact</strong> on task runtime. This stability reinforces the broader conclusion: <strong>Graviton Spot provides both faster runtime and predictable shutdown performance</strong>, making it ideal for non-production and burst workloads that demand low startup-to-completion latency.</p><h3>8. Reference</h3><p>AWS Graviton2 Support <a href="https://aws.amazon.com/blogs/aws/announcing-aws-graviton2-support-for-aws-fargate-get-up-to-40-better-price-performance-for-your-serverless-containers/">blog </a>by Amazon</p><p>Amazon ECS support AWS Graviton for fargate — <a href="https://aws.amazon.com/about-aws/whats-new/2024/09/amazon-ecs-graviton-based-spot-compute-fargate/">blog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fbf08b0812b" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/amazon-ecs-fargate-on-aws-graviton2-spot-boots-up-4-faster-1fbf08b0812b">Faster Starts, Predictable Stops: Measuring AWS Fargate Graviton vs x86 Performance Across Spot and…</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon ECS Fargate on AWS Graviton2 Spot 76% cheaper]]></title>
            <link>https://towardsaws.com/aws-fargate-graviton-spot-76-cheaper-d27f8c570a19?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d27f8c570a19</guid>
            <category><![CDATA[aws-graviton]]></category>
            <category><![CDATA[ecs-fargate]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sun, 07 Sep 2025 12:09:03 GMT</pubDate>
            <atom:updated>2025-09-16T20:34:07.595Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>This paper is simple, short and informative about costs on different AWS Fargate infrastructure. The costs is calculated in our AWS account by running same container with s<strong><em>ame parameters and same code</em></strong> running for 200+ hours over <strong><em>30 days</em></strong> and check cost performance.</p><p>a. AWS ECS Fargate(x86)</p><p>b. AWS ECS Fargate(Graviton)</p><p>c. AWS ECS Fargate Spot(x86)</p><p>d. AWS ECS Fargate Spot(Graviton)</p><p>A very important note is the exercise is done on very tiny container in US-EAST-2 region and cost may vary based on code, execution time, other fargate parameters and AWS region.</p><p>This is a continuation of our other following write ups.</p><h4><a href="https://medium.com/@prasenjitrc/aws-ecs-fargate-spot-on-arm64-9083e6b7e9cb">AWS ECS Fargate Spot on Graviton2</a></h4><h4><a href="https://medium.com/@prasenjitrc/interruption-simulation-of-fault-tolerant-tasks-in-aws-fargate-spot-3a841dc43b8f">Simulation of interruption on Fault tolerant tasks in AWS Fargate Spot</a></h4><h3>2. Use Case</h3><p>There is one container created from same codebase which has built twice on docker using ARM64(Graviton) and x86 architecture. There are two task definitions created using the Amazon ECR each for Graviton and ARM64(Graviton) with rest all memory, CPU parameters same.</p><p>The containers are executed 1000 times over 27 days to check the cost analysis. Also, for spot it is checked how many times pre-emption happens and how many times AWS ECS Fargate capacity not available.</p><h3>3. Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SU1mFITLLQl0RejWHLBCgA.png" /><figcaption>Architecture Diagram for Exercise</figcaption></figure><p>The code for container and AWS Lambda, docker files and commands added in the GitHub <a href="https://github.com/PrasenjitRoychoudhury/AWSECSFargateCostComparison">link</a>.</p><h3>4. Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*tiWzwLbxRq9dMliT7965Ng.png" /><figcaption>Cost Analysis</figcaption></figure><ul><li><strong>Baseline (Fargate x86)</strong> → Highest cost, serves as 100%.</li><li><strong>Fargate Graviton</strong> → ~21% cheaper than Fargate x86, shows Graviton advantage.</li><li><strong>Fargate Spot (x86)</strong> → ~69% cheaper than On-Demand x86.</li><li><strong>Fargate Spot (Graviton)</strong> → The cheapest option, ~76% savings over x86 On-Demand.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7QIG51-LJZPpjJX_loopew.png" /><figcaption>AWS ECS Fargate Costs</figcaption></figure><p>There is 0 instances where AWS Fargate Capacity not available when invoked 1000+ times over 30 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDS3vbcOJnS0CJSlVFbGGg.png" /><figcaption>AWS Fargate Capacity Not available</figcaption></figure><p>The container has been scheduled every 5 minutes and tasks run for 5 minutes. The spot can be interrupted and code has been updated to print SIGTERM for interruption. The interruption has been 0 in our exercise however the interruption can depend on various parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tme0CIBRrZWxFDSoKVUv-A.png" /><figcaption>AWS Sigterm2 Invocation</figcaption></figure><h3>5. What Services Are Used</h3><ul><li>Amazon ECR — Two repositories used for docker container on X86 and Arm64.</li><li>AWS ECS — The service is used for task definitions, running standalone task.</li><li>Amazon EventBridge — The scheduler using to invoke the AWS Lambda.</li><li>AWS Lambda — The function is used to invoke 4 standalone tasks with capacity tags defined which is used to calculate the cost.</li></ul><p>When running containers on <strong>Graviton (ARM64)</strong>, it is important to ensure that the <strong>application code, dependencies, and base images</strong> are compatible with the ARM64 architecture. Docker images must be built specifically for ARM64, or use multi-arch manifests to support both x86 and ARM seamlessly. Certain libraries, SDKs, or binaries may require recompilation or replacement with ARM-compatible versions. We have added our docker file for graviton in the GitHub link.</p><h3>6. Clean up</h3><ul><li>Amazon ECR — Delete the repositories .</li><li>AWS ECS — Delete task definition and AWS ECS Cluster.</li><li>Amazon EventBridge — The scheduler to mark disable.</li><li>AWS Lambda — Delete the function.</li><li>Amazon CloudWatch Logs — Delete the Amazon CloudWatch loggroups.</li></ul><h3>7. Extension of work</h3><p>On 13 Sep’25 one service is deployed on AWS ECS Service with capacity provider Fargate Spot with cpu 0.5 and 1GB memory . The service ran for 6 hours 1 minute and first interruption happened after <strong>4 hours 50 minutes</strong>.</p><p>On 16 Sep’25 on service with two tasks are deployed on AWS ECS cluster with Fargate Spot capacity and service had run for 12 hours. The services were interrupted 4 times during the time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*kBdQYK9T5IwjDq4Bjdhwyg.png" /><figcaption>AWS ECS Graviton Fargate Spot- Interruption statistics</figcaption></figure><h3>8. Conclusion</h3><p>This study demonstrates that <strong>AWS Fargate with Graviton Spot instances offers the most cost-efficient execution option, reducing costs by up to 76% compared to Fargate on-demand x86 tasks</strong>. Graviton provides a clear efficiency advantage over x86 even in on-demand scenarios, and Spot pricing further compounds the savings when workload characteristics allow for potential pre-emption.</p><p>In our 30-day controlled exercise, no capacity shortages or Spot interruptions were observed, highlighting both the stability and the cost effectiveness of Spot Graviton in this specific region and workload profile. However, results may vary depending on workload size, execution duration, region, and Spot market dynamics.</p><p>Overall, organizations seeking to optimize compute costs for containerized workloads on ECS should strongly consider <strong>Graviton-based task definitions</strong> and, where workloads are tolerant to interruptions, <strong>Spot capacity</strong>. This combination delivers significant savings without compromising reliability in many scenarios.</p><p>Our recommendation is that to run the non-production environments on AWS ECS Spot capacity which offers 69% to 76% cost reduction.</p><h3>9. Reference</h3><p>AWS Graviton2 Support <a href="https://aws.amazon.com/blogs/aws/announcing-aws-graviton2-support-for-aws-fargate-get-up-to-40-better-price-performance-for-your-serverless-containers/">blog </a>by Amazon</p><p>Amazon ECS support AWS Graviton for fargate — <a href="https://aws.amazon.com/about-aws/whats-new/2024/09/amazon-ecs-graviton-based-spot-compute-fargate/">blog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d27f8c570a19" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/aws-fargate-graviton-spot-76-cheaper-d27f8c570a19">Amazon ECS Fargate on AWS Graviton2 Spot 76% cheaper</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time AWS Lambda Performance Monitoring with Machine Learning]]></title>
            <link>https://medium.com/@prasenjitrc/real-time-aws-lambda-performance-monitoring-with-machine-learning-862eff1dcad0?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/862eff1dcad0</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[amazon-kinesis-firehose]]></category>
            <category><![CDATA[amazon-sagemaker]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sun, 17 Aug 2025 10:28:01 GMT</pubDate>
            <atom:updated>2025-08-17T10:28:01.015Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>Serverless compute services such as AWS Lambda offers scalability and cost-efficiency in right use case. Observability and performance management are important for diagnosing cold starts, memory overuse, or function failures. While logs and metrics provide raw data, extracting actionable insights requires a unified data for observability for <strong>unknown unknown</strong> cases. This paper is built on use case for detecting unknown unknown anomalies like existing error surged or duration surged beyond the usual ones.</p><p>This paper explores an end-to-end approach to collecting, enriching, and analysing AWS Lambda performance logs using AWS services. By leveraging Amazon CloudWatch, AWS Data Firehose, AWS Lambda, and Amazon SageMaker, we have built a machine learning-driven pipeline that enables anomaly detection. This architecture demonstrates the use case of detection of anomaly in AWS Lambda duration using machine learning of RCF model.</p><p>The Amazon CloudWatch log<a href="https://aws.amazon.com/blogs/compute/aws-lambda-introduces-tiered-pricing-for-amazon-cloudwatch-logs-and-additional-logging-destinations/"> recently allowed</a> to send the log streams directly to Amazon Data Firehose which is used here.</p><p>We have detailed the steps, codes and Amazon Sagemaker Jupyter notebook implementation in our GitHub <a href="https://github.com/PrasenjitRoychoudhury/CloudWatchLogFirehoseML">link</a>.</p><h3>2. Use Case</h3><p>This use case is built using a machine learning model that can automatically detect performance anomalies in AWS Lambda functions. Common issues include:</p><p>· Cold start delay</p><p>· <em>Timeout risks for overrunning</em></p><p>· Function-level error patterns</p><p>To address this, detailed performance logs are extracted and analysed, enriched with metadata (e.g., function version, payload size), and used to train a model for outlier detection and failure forecasting. In the paper, the use case considers when a AWS Lambda runs longer/lesser than the historical value. The AWS Lambda can run longer for various reason and this use case gives a hint by detecting anomaly detection. This use case can be enhanced for various other use cases for anomaly detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*6FdmJdlwrURBT8qup88Dxg.png" /><figcaption>Amazon SNS Notification after Anomaly Detection</figcaption></figure><h3>3. Architecture</h3><p>The architecture integrates real-time log collection, transformation, partitioned storage, and machine learning in a AWS workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*COd6dcRslzsXB-Zb4T21Bw.png" /><figcaption>Architecture Diagram</figcaption></figure><p>a. Log Generation and Collection</p><blockquote>o AWS Lambda functions are invoked (via Amazon API Gateway) and automatically send execution logs to Amazon CloudWatch Logs.</blockquote><blockquote>o Logs contain key performance data: <strong>execution duration</strong>, memory used, initialization time, and error details.</blockquote><blockquote>o For experimentation the Apache Bench is used to initiate the traffic for AWS Lambda function. The Apache bench is installed in Amazon EC2.</blockquote><p>b. Log Streaming and Enrichment</p><blockquote>o A CloudWatch Logs forwards logs to Amazon Data Firehose.</blockquote><blockquote>o Firehose triggers a transformation Lambda function, which:</blockquote><blockquote>§ Parses logs to extract structured metrics (e.g., Duration, MemoryUsed, InitDuration)</blockquote><blockquote>§ Adds context such as invocation type, function version, and payload size</blockquote><p>c. Storage and Partitioning</p><blockquote>o The enriched logs are delivered to Amazon S3, partitioned by function name and timestamp (e.g., lambda_name/yyyy/mm/dd/).</blockquote><blockquote>o This structure supports efficient querying and scalable parallel processing during ML training.</blockquote><p>d. Machine Learning Pipeline</p><blockquote>o A SageMaker pipeline (e.g., using a Jupyter Notebook) ingests the partitioned logs from S3.</blockquote><blockquote>o Data is preprocessed and used to train an anomaly detection model using Random Cut Forest model.</blockquote><blockquote>o The trained model is deployed to a SageMaker real-time endpoint for ongoing inference.</blockquote><p>e. Real-time Inference and Notification</p><blockquote>o A downstream Lambda function queries the SageMaker endpoint with recent log data.</blockquote><blockquote>o Anomalies or failure predictions trigger alerts via Amazon SNS, enabling proactive response or automated mitigation.</blockquote><blockquote>This architecture exemplifies a scalable, automated observability pipeline that blends log analytics with machine learning to ensure resilient serverless applications.</blockquote><h3>4. <strong>How to Use (Deploying and Testing the Pipeline)</strong></h3><p>The entire codebase, docker files and setup screenshots are attached in my GitHub <a href="https://github.com/PrasenjitRoychoudhury/CloudWatchLogFirehoseML">link</a>.</p><p>To deploy and test the observability pipeline end-to-end, the following steps are performed:</p><p>a. AWS Lambda and Amazon API Gateway<br> A simple AWS Lambda function is created to serve as the microservice for test. The function is configured to log metrics such as execution time and memory usage to Amazon CloudWatch Logs. The AWS Lambda is exposed via an Amazon API Gateway HTTP API so it can be invoked externally. An API Gateway endpoint is created and integrated with the Lambda (no authentication is configured for this simple test).</p><p>b. Configure CloudWatch Logs Streaming to Firehose<br> The AWS Lambda logs are stored by default in a Amazon CloudWatch Logs log group (e.g., /aws/lambda/&lt;function-name&gt;). A CloudWatch Logs events forwards to new destination to an Amazon Data Firehose delivery stream. The configuration specifies the Firehose ARN and an IAM role that grants CloudWatch permission to send events to Firehose. A new IAM role is created with a policy allowing logs:PutRecord to the Firehose stream. Once the subscription is configured, any new log generated by the Lambda is automatically sent to Amazon Data Firehose for processing.</p><p>c. Set Up Amazon Data Firehose with a Transform AWS Lambda<br> A Data Firehose delivery stream is created with “Direct PUT” as the source and Amazon S3 as the destination. The Firehose is configured with:</p><p>· An Amazon S3 bucket (e.g., lambda-observability-logs).</p><p>· An Amazon S3 prefix for organized storage (e.g., function name and date).</p><p>· Record transformation enabled with a transformation Lambda function.</p><p>The transformation Lambda performs the following actions:</p><p>· Parsing — Decodes base64-encoded Amazon CloudWatch log events and extracts metrics such as Duration, Billed Duration, Memory Size, Max Memory Used, and Init Duration (if a cold start occurred).</p><p>· Enrichment — Adds metadata such as function name, version, and timestamp, and formats the output as structured JSON.</p><p>o The AWS Lambda timeout is adjusted to handle larger batches, and IAM permissions are configured appropriately. The Firehose service role is given s3:PutObject access to the bucket. Once configured, the delivery stream is activated.</p><p>d. Verify Log Delivery to S3<br> Load is generated against the API/Lambda using Apache Bench to produce logs. After several minutes, the S3 bucket is checked for objects appearing in the specified prefix (e.g., myLambdaFunction/2025/08/07/). The logs, typically delivered in JSON format, contain structured records with fields such as Lambda name, timestamp, and duration. Data is partitioned by date and function name to support efficient querying and machine learning.</p><p>e. Train an Anomaly Detection Model with SageMaker<br> Historical invocation data stored in S3 is used to train an anomaly detection model. An Amazon SageMaker Notebook instance (or SageMaker Studio) is launched to develop the model. Data is loaded from S3 using AWS Glue, Pandas, or similar tools. The built-in SageMaker Random Cut Forest (RCF) algorithm is selected for detecting outlier patterns in metrics such as execution duration and init duration. The training job is configured with:</p><p>· The S3 location of the dataset.</p><p>· Appropriate instance types.</p><p>· Hyperparameters such as the number of trees and sample size.</p><p>Upon completion, model artifacts are saved to S3. The model is deployed to a real-time SageMaker endpoint. The endpoint is tested with sample data to verify that anomaly scores are returned as expected.</p><p>f. Create a Lambda for Real-Time Anomaly Inference<br> An additional Lambda function (“AnomalyDetectorLambda”) is created to query the SageMaker endpoint upon the arrival of new log data. Amazon S3 Event Notifications are configured to trigger this Lambda when new objects are added to the logs bucket. When triggered, the Lambda:</p><p>· Retrieves the new log file from S3.</p><p>· Processes relevant records.</p><p>· Sends them to the SageMaker endpoint for anomaly scoring.</p><p>· Publishes alerts via Amazon SNS if anomaly scores exceed the threshold.</p><p>o IAM permissions are configured to allow the Lambda to read from S3 and invoke the SageMaker endpoint.</p><p>g. End-to-End Test<br> The primary API/Lambda is invoked again to generate new log files. When data lands in S3, the AnomalyDetectorLambda is triggered, anomaly detection is performed, and alerts are sent if anomalies are detected. CloudWatch Logs are checked to confirm successful execution, and SNS email notifications are verified where applicable.</p><h3>5. What Services Are Used</h3><p>This solution is composed of multiple AWS services combined within a single architecture. A summary of each service and its role within the pipeline is provided below:</p><p>· <strong>AWS Lambda</strong> — In this architecture, Lambda is utilized in three ways: (a) as the primary Lambda function invoked through Amazon API Gateway to simulate the application workload, (b) as the transformation Lambda triggered by Amazon Data Firehose to preprocess log data, and © as the anomaly detection Lambda triggered by Amazon S3 to perform inference.</p><p>· <strong>Amazon API Gateway</strong> — Amazon API Gateway is a fully managed service for creating and hosting APIs at scale. Within the observability pipeline, an HTTP endpoint is provided by API Gateway to trigger the primary Lambda function.</p><p>· <strong>Amazon CloudWatch Logs</strong> — Amazon CloudWatch Logs is used to centralize the storage and management of logs from AWS services. When the AWS Lambda function is executed, logs containing information such as execution time, memory usage, and custom log entries are automatically streamed to CloudWatch Logs.</p><p>· <strong>Amazon Data Firehose</strong> — Amazon Data Firehose is used to capture, transform, and deliver streaming data to storage destinations. In this solution, Amazon Data Firehose receives the log stream from Amazon CloudWatch Logs. The delivery stream is configured to use a Lambda function for data transformation and Amazon S3 as the destination. Incoming log data is buffered, transformed to parse and enrich metrics, and then delivered to S3 in JSON format. Amazon Data Firehose manages scaling, buffering, and retries without manual intervention, ensuring continuous delivery of logs even under high volume.</p><p>· <strong>Amazon S3</strong> — Amazon Simple Storage Service (S3) is used as the central data lake for the observability pipeline. Enriched logs are delivered from Firehose to an S3 bucket, organized by partition keys such as date and function name. This structure enables efficient analytics and machine learning processing. Amazon SageMaker also uses S3 for storing training data and model artifacts; training datasets are read from S3, and trained models are saved back to it. Due to S3’s high reliability, logs and models persist for long-term analysis and are easily accessible to other services.</p><p>· <strong>Amazon SageMaker</strong> — Amazon SageMaker is used to build, train, and deploy the anomaly detection model on the Lambda log data. The service manages the training of an unsupervised Random Cut Forest model and the deployment of the model to a real-time inference endpoint. This endpoint is later accessed by the anomaly detection Lambda to retrieve anomaly predictions for new log entries. By using SageMaker, the setup and management of ML infrastructure are avoided, and the entire workflow from training to serving is handled by the service.</p><p>· <strong>Amazon SNS (Simple Notification Service)</strong> — Amazon SNS is used to send alerts when anomalies are detected..</p><p>· <strong>AWS IAM (Identity and Access Management)</strong> — AWS IAM is used to enforce secure, least-privilege access between pipeline components. Roles and policies are created to allow Amazon CloudWatch Logs to write to Amazon Data Firehose.</p><p><strong>Amazon EC2</strong> — Amazon Elastic Compute Cloud (EC2) is used in the testing phase to generate load against the API Gateway endpoint using Apache Benchmark.</p><h3>6. Benefits</h3><p>The implementation of the machine learning-powered observability pipeline for AWS Lambda functions is associated with numerous operational and engineering benefits:</p><p>· <strong>Automated Anomaly Detection:</strong> Vast volumes of log data are automatically analyzed, and anomalies such as spikes in latency, memory leaks, and cold start durations are detected without human intervention. Predictive insights are generated, enabling issues to be identified before they manifest as outages. For example, if an abnormal upward trend in a function’s duration is detected (potentially due to an impending timeout), the relevant team can be alerted to investigate or scale resources proactively. As a result, the Mean Time to Detection (MTTD) is significantly reduced compared to manual monitoring.</p><p><strong>Reduced MTTR and Improved Resiliency:</strong> Integration with SNS ensures that the appropriate personnel or systems are notified immediately upon anomaly detection. This capability reduces the Mean Time to Response/Resolution (MTTR), allowing early intervention rather than reactive firefighting after an outage.</p><h3>7. Extensions</h3><ul><li>The log anomaly used here for anomaly detection for operation parameters like duration, memory usage etc. The model can be enhanced where semantic error can be checked for anomaly. After a deployment if any new error is reported or certain error is increased, anomaly detection can be used.</li></ul><h3>8. Clean up</h3><p>After testing the solution, it’s important to clean up resources to avoid ongoing costs. The following steps outline how to tear down the environment:</p><p>a. <strong>SageMaker Endpoint and Model Deletion</strong> — The real-time inference endpoint that had been deployed should be deleted using the SageMaker console or AWS CLI. We have used a script in SageMaker Jupyter notebook which has been attached in <em>NotebookREsourceCleanup.txt in GitHub link.</em></p><p>b. <strong>SageMaker Notebook Instance Shutdown</strong> — Any SageMaker Notebook instance created for training should be stopped and deleted.</p><p>d. <strong>Amazon Data Firehose Stream Deletion</strong> — The Firehose delivery stream that was set up should be selected and deleted in the Amazon Data Firehose console.</p><p>e. <strong>Test Resource (Amazon EC2) Termination</strong> — Amazon EC2 instance started for load testing should be terminated to prevent ongoing compute charges. Any allocated Elastic IPs should be released.</p><h3>9. Conclusion</h3><p>This paper presents a scalable, serverless, and automated observability pipeline for AWS Lambda that leverages AWS native services — Amazon CloudWatch, Amazon Data Firehose, Amazon S3, and Amazon SageMaker — to collect, enrich, and analyze detailed performance data in real time using machine learning for predictive anomaly detection. By proactively identifying issues such as cold start delays and memory exhaustion before they cause failures, the solution shifts operations from reactive troubleshooting to proactive optimization, reducing downtime, improving resiliency.</p><p>The purpose of the paper is to use anomaly detection on logs for operation parameters and texture of logs content.</p><h3>10. Reference</h3><ul><li>AWS Architecture <a href="https://aws.amazon.com/blogs/compute/aws-lambda-introduces-tiered-pricing-for-amazon-cloudwatch-logs-and-additional-logging-destinations/">blog </a>for AWS CloudWatch Log to Amazon Firehose</li><li>GitHub <a href="https://github.com/PrasenjitRoychoudhury/CloudWatchLogFirehoseML">link </a>for codes, setup &amp; execution screenshots</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=862eff1dcad0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Apache Bench and AWS X-Ray to Measure Tail Latency in Auto-Scaled ECS Services]]></title>
            <link>https://medium.com/@prasenjitrc/using-apache-bench-and-aws-x-ray-to-measure-tail-latency-in-auto-scaled-ecs-services-4a0ad086e1cb?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a0ad086e1cb</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[aws-x-ray]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 17:50:53 GMT</pubDate>
            <atom:updated>2025-07-04T19:21:48.107Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>This paper evaluates the performance of microservices deployed on Amazon ECS, utilizing both AWS Fargate and Amazon EC2 services with same autoscaling configuration for task level and same image from Amazon ECR. The study leverages AWS X-Ray for observability and Apache Bench for load testing to measure P99 and P95 latency levels, error rates etc. during task autoscaling in Amazon ECS.</p><p>In our earlier <a href="https://medium.com/@prasenjitrc/performance-analysis-of-auto-scaled-microservices-using-apache-bench-and-amazon-cloudwatch-2ae33f4defe2">paper </a>we have detailed how to measure microservices SLA parameters when service on auto scale enabled both at application and infrastructure layers. Here the experimentation is extended for comparison at infrastructure layer of Amazon EC2 and AWS Fargate and AWS X-Ray is used.</p><p>The exercise is done to evaluate performance of microservice with equal traffic and similar task and service auto scale level configurations in AWS Fargate and Amazon ECS EC2 services. The experimentation is done where similar number of traffic hit the same service running on AWS Fargate and Amazon EC2. Goal of experimentation is to evaluate the microservice performance when tasks auto scale in AWS Fargate and Amazon EC2(static) with traffic, image and task definition same.</p><p>The exercise, repeated multiple times on a particular use case, reveals that Fargate autoscaling scenarios exhibit P99 latencies ranging from 2073 ms to 2080 ms and P95 latencies from 2020 ms to 2029 ms across five tests, with failed requests varying between 62 and 164. In contrast, EC2-based ECS services show P99 latencies from 2058 ms to 2068 ms and P95 latencies from 2017 ms to 2018 ms, with fewer failed requests (27 to 53). These outcomes highlight the impact of serverless (Fargate) versus managed instance (EC2) autoscaling on performance, with EC2 demonstrating <strong>greater stability in error rates in a particular use case</strong>.</p><p><strong>The paper also details what steps are required to run AWS X-Ray as side car in Amazon ECS SERVICE.</strong> The paper details how to measure microservice based SLAs in different configurations of auto scaling in Amazon EC2 and AWS Fargate.</p><h3>2. Use Case</h3><p>In production, there are some use cases which have stringent SLAs regarding backend response time, error rate etc. for microservices running on auto scale enabled.</p><p>The primary goals are:</p><p>· Assessing P99 and P95 latency metrics using Apache Bench to evaluate microservice responsiveness during autoscaling. The exercise found AWS Fargate’s P99 latency averaging around 2074 ms and P95 around 2022 ms, while Amazon EC2 averaged 2063 ms and 2017 ms, respectively, indicating a slight edge in tail latency for Amazon EC2.</p><p>· Analysing error rates with AWS X-Ray to understand failure patterns in Fargate and EC2-based ECS services. Results show Fargate experiencing higher failure rates (62 to 164 failed requests) compared to EC2 (27 to 53), suggesting potential scalability challenges under load. The failure requests are generated by Amazon API Gateway for timeout error. This experiment tries to find out impact on backend response time which in turn results for failure response at client side.</p><p>The use case is to establish the optimal autoscaling configurations to meet stringent SLA requirements for latency and reliability. The data suggests Amazon EC2 may require less aggressive scaling policies due to lower error rates, while Fargate could benefit from enhanced error handling to maintain SLA compliance. The experiment highlights the trade-offs between AWS Fargate’s serverless efficiency and EC2’s consistent performance under the tested load of 10,000 requests with 50 concurrency.</p><h3>3. Architecture</h3><p>The architecture comprises tasks with containers(application and side car) deployed on Amazon ECS, with tasks running on AWS Fargate and Amazon EC2 instances, both configured for autoscaling with same . Traffic is initiated from Apache Bench on a dedicated Amazon EC2 instance, routed through REST api in Amazon API Gateway to an Application Load Balancer (ALB). The ALB distributes requests to ECS tasks, which connect to Amazon RDS. AWS X-Ray is integrated to trace latency and errors, operating within private subnets for security. The API Gateway is used with REST api where exercise is repeated when timeout is configured also. The exact similar setup is done where Amazon EC2 is replaced with AWS Fargate.</p><p>The details with code, task template and test results are uploaded in GitHub <a href="https://github.com/PrasenjitRoychoudhury/AWSXRayP99">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*LvXEDlsdi7Y1A-8gyKYyaA.jpeg" /><figcaption>Architecture Diagram</figcaption></figure><h3>4. How to use</h3><p>In the exercise AWS X-Ray used to find out the latency of service running on AWS ECS Service. For that the code is modified, task definition is updated to create side car for AWS X-Ray and AWS IAM role updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*13PMs2b-Ytvdd8VOODzt_Q.png" /></figure><p>This code snippet initiates an AWS X-Ray segment named MyECSPythonService to trace a request in a Flask application. Within a try block, it checks if a trace_id (from the X-Amzn-Trace-Id header) exists. If present, it logs the trace ID, joins the existing X-Ray segment using xray_recorder.current_segment(), and annotates the segment with the trace ID for tracking. Additionally, it enriches the segment with HTTP metadata, including the request URL, HTTP method, and user agent from the request headers, enabling detailed monitoring and debugging of the request’s lifecycle in the AWS X-Ray service.</p><p>Here the AWS X-Ray is used where sampling is swiched off and each request is traced. By this exercise each part of request can be traced to individual module/function if required. The code with details are attached in the Github link.</p><p>Task definition to be used to run the AWS X-Ray as sidecar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*8YCPEYa0H-qOj1nLPVnZ_g.png" /></figure><p><strong>The ECS task has following IAM Role attached with the trust policies attached with the role.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*yPjHPpgbxy1hERE8sv9-DA.png" /></figure><p><strong>In the exercise Apache Bench has been used to fire the traffic which has been explained in earlier paper. The syntax for using Apache Bench commands are following.</strong></p><blockquote><strong>for i in {1..1}; do</strong></blockquote><blockquote><strong>ab -n 10000 -c 50 https://&lt;Amazon apigateway stage URL&gt; /dev/next-prime?number=5</strong></blockquote><blockquote><strong>sleep 1</strong></blockquote><blockquote><strong>done</strong></blockquote><p>The Amazon X-Ray is used to create service map to check the service availability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*1oZ2uQZvUG1tIe9KKqfsSQ.png" /></figure><h3>5. What services are used</h3><p><strong>a.</strong> <strong>Amazon ECS Services: </strong>Here the services are placed on Amazon ECS Services on Amazon EC2 mode. The dynamic auto scaling is used which is scale out and in based TrafficPerTarget group basis. The reason for choosing Amazon ECS Service is to find out the latency and success rate based at P99 level. Here the scaling configuration is made simplistic and results depends on current configurations. The results can definitely vary based on auto scale in/out configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*J8pxqQFR9kfuNt8crlH2Qg.png" /><figcaption>Amazon ECS Service Task Definition scaling configuration</figcaption></figure><p><strong>b.</strong> <strong>Amazon API Gateway: </strong>For this paper, Amazon API Gateway is placed as a gateway before the traffic hits the Amazon ALB. For this use case, the http based api is used. One of the reasons for choosing the gateway is to measure the backend response and timeout rate before Amazon ALB. Also, rate limiting and throttling is used in the system to check the impact to overall latency and success rate. Here the REST api is used to define the backend response time out. The REST api configured to generate 50x response to client(Apache Bench here).</p><p><strong>c.</strong> <strong>Amazon EC2: </strong>The reason for using Amazon EC2 is to provide the infrastructure to Amazon ECS cluster where service is running. The purpose of the paper is to measure impact to overall latency and success rate at P99 level when task is auto scaling with fixed number of Amazon EC2. Here the Amazon EC2 is static and not on ASG mode. The result will vary if Amazon EC2 is on ASG mode. The purpose the paper is to identify when tasks scale in serverless AWS Fargate and infrastructure available in Amazon EC2.</p><p><strong>d.</strong> <strong>Amazon CloudWatch Matrices: </strong>The Amazon CloudWatch matrices are used measure the success rate and latency at P99 level and see the performance during task scale in/out and Amazon EC2 scale in/out conditions.</p><p><strong>e.</strong> <strong>AWS Fargate : T</strong>he reason for using AWS Fargate is to provide the infrastructure to Amazon ECS cluster where service is running. The purpose of the paper is to measure impact to overall latency and success rate at P99 level when task is auto scaling with same parameter as scaling implemented in service running on Amazon EC2.</p><h3>6. Analysis</h3><p>This exercise is done to analyse the microservice performance when tasks are on autoscaling with similar configuration in AWS Fargate and Amazon EC2. For this exercise, the 10000 service calls are made with 50 throughputs. This exercise has been repeated five times for services deployed in Amazon ECS service in AWS Fargate and Amazon EC2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*aLIlM8J8XL9Crys2xthw_Q.png" /></figure><p><strong>Performance Consistency</strong></p><ul><li><strong>Amazon EC2</strong>: Shows consistent performance across all five tests, with test durations ranging from 404.973s to 405.205s (standard deviation ~0.087s). Requests per second are stable at 24.68–24.69.</li><li><strong>AWS Fargate</strong>: Also consistent in test duration (405.185s–405.996s, standard deviation ~0.298s), but slightly slower than EC2. Requests per second range from 24.63–24.68, marginally lower than EC2.</li></ul><p><strong>Reliability (Failed Requests)</strong></p><ul><li><strong>Amazon EC2</strong>: Average of 45 failed requests (0.45% failure rate), with a range of 27–53. The lowest failure count (27) is in test 5, indicating better reliability in that run.</li><li><strong>AWS Fargate</strong>: Average of 112.6 failed requests (1.126% failure rate), significantly higher than EC2. Failure counts range from 62–164, with tests 1 and 3 showing the highest failures (164).</li></ul><p><strong>Outcome</strong></p><ul><li><strong>Amazon </strong>EC2 outperforms Fargate in reliability (fewer failed requests: 45 vs. 112.6) and data transfer efficiency (11.49 vs. 7.08 Kbytes/sec). It maintains slightly better throughput (24.69 vs. 24.67 requests/sec) and comparable latency.</li><li>AWS Fargate shows slightly higher latency across most percentiles and significantly more failed requests, indicating potential issues with autoscaling or resource allocation under load.</li></ul><h3>7. <strong>Extension of further work</strong></h3><p>Incorporation of Service Mesh (e.g., AWS App Mesh or Istio): Introducing service mesh capabilities will help trace fine-grained latency at the sidecar proxy level and improve routing decisions during autoscaling.</p><p>Enhanced Error Analytics Using AWS CloudTrail and X-Ray Annotations: Integration of deeper trace annotations and logs for failed requests will assist in pinpointing systemic errors and tuning application-level retry policies.</p><p>The purpose of the paper is how to use AWS X-Ray for service running on AWS ECS Service and measure the responses using Apache Bench. Apache Bench is used as synthetic tool with constant load which may not be production like situation always. For more production like situation JMeter like tools can be used.</p><h3>8. Clean Up</h3><p><strong>a</strong>. Delete the Service running on Amazon ECS Service</p><p><strong>b</strong>. Delete the Amazon Load Balancers created</p><p><strong>c</strong>. Delete the target groups</p><p>d. Delete the http REST api created on Amazon Api Gateway</p><p>e. Delete the ASG group created which is attached with Amazon ECS cluster</p><p>f. Delete the task definitions on Amazon ECS cluster</p><h3>9. Conclusion</h3><p>This paper presents a comparative analysis of microservice latency and reliability using Apache Bench and AWS X-Ray in environments backed by AWS Fargate and Amazon EC2. The findings highlight that:</p><ul><li>Amazon EC2-based ECS tasks consistently outperform Fargate in both P99 and P95 latency metrics, albeit by small margins.</li><li>Fargate exhibits a significantly higher error rate, with nearly 2.5 times more failed requests on average compared to EC2, suggesting limitations in its default resource provisioning or startup behaviour during scaling events.</li><li>Both platforms showed consistent performance across multiple test runs, indicating reliability in autoscaling mechanics. However, EC2 demonstrated marginally better stability in test durations and throughput.</li></ul><p>These observations suggest that while AWS Fargate offers operational simplicity and ease of scaling, Amazon EC2 provides more predictable performance under high-concurrency scenarios. Organizations aiming for stringent SLAs — especially around tail latency and error tolerance — may prefer EC2 for critical microservices.</p><p>Future work will expand into tracing complexity, service mesh integration, and cost-performance trade-offs to offer a holistic approach to scaling decisions in AWS environments. <strong><em>Most importantly, the performance of any service depends on various use cases.</em></strong> The configurations of auto scaling depend on various outcome parameters from testing. The exercise here is done how to get the parameters and analyse using various AWS services and Apache Bench.</p><h3>10. Reference</h3><p><a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html">https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a0ad086e1cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Analysis of Auto-Scaled Microservices Using Apache Bench and Amazon CloudWatch]]></title>
            <link>https://medium.com/@prasenjitrc/performance-analysis-of-auto-scaled-microservices-using-apache-bench-and-amazon-cloudwatch-2ae33f4defe2?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ae33f4defe2</guid>
            <category><![CDATA[apache-bench]]></category>
            <category><![CDATA[autoscaling]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 17:26:46 GMT</pubDate>
            <atom:updated>2025-07-04T19:22:39.332Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>Microservice architectures often rely on <strong>auto-scaling of tasks in the service </strong>to handle varying workloads and maintain performance Service Level Agreements (SLAs). While auto-scaling helps to maintain the throughput and availability, it can impact the SLAs like <strong>latency </strong>(especially tail latency such as 99th percentile latency, <em>P99</em>) and potentially impact error rates (e.g., <strong>HTTP 5xx </strong>responses).</p><p>We have done experimentation on impact of SLAs when service is on auto scaling mode running on Amazon ECS cluster on Amazon EC2.</p><p>This paper examines the performance of a sample microservice under different auto-scaling scenarios in AWS. We experimented on use cases of auto-scaling at both the <strong>application layer </strong>(scaling Amazon ECS service tasks) and the <strong>infrastructure layer </strong>(scaling Amazon EC2 instances) affects key performance metrics: — <strong>Latency </strong>at various percentiles (average, P50, P90, P99). — <strong>Throughput </strong>(requests per second). — <strong>Success rate </strong>(absence of 5xx errors).</p><p>This paper details how to use Apache Bench tool to check the P99, average backend etc parameters when service is autoscaling and underlying capacity infrastructure running on Amazon EC2 is also on auto scale model.</p><p>The paper does a hypothetical scenario to check on various scenarios on following scenarios.</p><p>a. When the Amazon ECS service tasks are running with defined desired count with no auto scale mode enabled.</p><p>b. When the Amazon ECS services tasks are on dynamic auto scale for scale out/in tasks without underlying infrastructure Amazon EC2 on auto-scale enabled.</p><p>c. When the Amazon ECS services tasks are on dynamic auto scale and underlying infrastructure Amazon EC2 is also auto scale enabled.</p><p>The purpose of the paper is identify the impact of auto-scaling on microservice performance and to provide guidance on what metrics to monitor (e.g., P99 latency) when tuning auto-scaling for microservices in production.</p><h3>2. Use Case</h3><p>The specific problem explored in this paper is: How does auto-scaling at the service level and at the infrastructure level affect the performance of a microservice, and how can we effectively measure and analyse this impact? We have explored impact on various NFRs while scaling out tasks at service level in Amazon ECS service and Amazon EC2 infrastructure level.</p><p>The goal of the experimentation is to show case following.</p><p>a. How to use Apache bench to check Microservices KPI. The paper details what are the parameters to read to check microservice level SLAs.</p><p>b. A use case for defining Amazon ECS Service Tasks auto scale with parameters and Amazon EC2 auto scale to support the business defined SLAs for microservices.</p><p>In the paper, we shall show how using Apache bench in the it was determined P99 level response time improves when Amazon ECS Services task is auto scaled. But the P99 level response time suffers when underlying infrastructure works with ASG by increasing number of Amazon EC2s.</p><p>The paper recommends what are parameters to be checked when Amazon ECS Service is on auto scale and Amazon EC2 is on ASG group enabled.</p><h3>3. Architecture</h3><p>For the experiment for this paper, the containers are deployed service running in Amazon ECS cluster. The Amazon ECS Service connects to Amazon RDS. The load balancing is done by AWS ALB. The is outside connection is done through Amazon API Gateway. The details of the experimentation is mentioned in my GitHub <a href="https://github.com/PrasenjitRoychoudhury/ApacheBenchSLAServices">link</a>.</p><p>For performance testing Apache Bench is deployed in Amazon EC2 in another subnet. The Apache Bench is sending the traffic to Amazon API Gateway which in turn connects through Application Load Balancer. The application Load balancer has target group which is attached to the service running on Amazon ECS cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*LgekIYB-wNcBmeIvETTmzg.jpeg" /><figcaption>Architecture Diagram</figcaption></figure><h3>4. How to use</h3><p>Apache Bench is used to initiate the traffic with defined parameters to evaluate the performance of microservice during peak traffic.</p><p>Standalone Amazon EC2 is used deploy the Apache Bench. The commands used are following.</p><blockquote>sudo apt-get update</blockquote><blockquote>sudo apt-get install apache2-utils</blockquote><blockquote>ab -V</blockquote><p>Following command is used to invoke the microservice</p><blockquote>for i in {1..2}; do</blockquote><blockquote>ab -n 200 -c 10 https://&lt;apigateway-url&gt;/next-prime?number=500</blockquote><blockquote>sleep 1</blockquote><p>In this command, a total of 200 HTTP requests are sent per iteration with a concurrency level of 10. The loop runs twice, resulting in a total of 400 requests. There is a 1-second delay between the two iterations.</p><h3>5. What services are used</h3><p><strong>a.</strong> <strong>Amazon ECS Services: </strong>Here the services are placed on Amazon ECS Services on Amazon EC2 mode. The dynamic auto scaling is used which is scale out and in based TrafficPerTarget group basis. The reason for choosing Amazon ECS Service is to find out the latency and success rate based at P99 level.</p><p><strong>b.</strong> <strong>Amazon API Gateway: </strong>For this paper, Amazon API Gateway is placed as a gateway before the traffic hits the Amazon ALB. For this use case, the http based api is used. One of the reasons for choosing the gateway is to measure the backend response and timeout rate before Amazon ALB. Also, rate limiting and throttling is used in the system to check the impact to overall latency and success rate.</p><p><strong>c.</strong> <strong>Amazon EC2 with ASG: </strong>The reason for using Amazon EC2 with ASG is to provide the infrastructure to Amazon ECS cluster where service is running. The purpose of the paper is to measure impact to overall latency and success rate at P99 level.</p><p><strong>d.</strong> <strong>Amazon CloudWatch Matrices: </strong>The Amazon CloudWatch matrices are used measure the success rate and latency at P99 level and see the performance during task scale in/out and Amazon EC2 scale in/out conditions.</p><h3>6. <strong>Analysis</strong></h3><p>Following is some parameters noted for 3 experimentation done on Apache Bench.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*-uIhuiQLofxiyn_l26mT8Q.png" /><figcaption>Table with readings</figcaption></figure><p><strong>Average Time per Request (Mean Latency)</strong>:</p><p>- 2 Microservices: 21018.285 ms</p><p>- 3 Tasks (Autoscaling): 21017.045 ms (improvement of ~1.24 ms)</p><p>- 2 EC2s + 4 Tasks (Autoscaling): 21018.977 ms (slightly worse than 3 tasks by ~1.93 ms, similar to 2 microservices)</p><p><strong>Interpretation</strong>:</p><p>- The <em>3 tasks with autoscaling</em> configuration shows a marginal improvement in average latency (~1.24 ms faster than 2 microservices). This suggests that adding a third task slightly improves resource availability for handling concurrent requests.</p><p>- The<em> 2 EC2s + 4 tasks</em> configuration performs similarly to the baseline (2 microservices), with a negligible increase in latency (~0.7 ms slower). <strong>This indicates that the additional Amazon EC2 instance and task did not significantly improve average latency, possibly due to overhead or inefficient load distribution.</strong></p><p><strong>P99 Latency (99th Percentile)</strong>:</p><p>- 2 Microservices: 20042 ms</p><p>- 3 Tasks (Autoscaling): 20033 ms (improvement of ~9 ms)</p><p>- 2 EC2s + 4 Tasks (Autoscaling): 20037 ms (improvement of ~5 ms over 2 microservices, but ~4 ms worse than 3 tasks)</p><p><strong>Interpretation</strong>:</p><p><strong>The <em>3 tasks with autoscaling </em>configuration has the best P99 latency, reducing tail latency by 9 ms compared to the baseline</strong>. This indicates better handling of worst-case scenarios, likely due to more tasks distributing the load effectively.</p><p>The 2 EC2s + 4 tasks configuration improves P99 latency over the baseline (by 5 ms) but is slightly worse than 3 tasks. <strong><em>This could suggest that the additional EC2 instance introduces some overhead (e.g., network latency or load balancer inefficiencies) that offsets the benefit of more tasks.</em></strong></p><p><strong>Requests per Second (Throughput</strong>):</p><p>- All configurations report ~0.48 requests per second, with no significant difference.</p><p><strong>Interpretation</strong>:</p><ul><li>Throughput remains unchanged across configurations, likely because the total number of requests (200) and concurrency level (10) are fixed, and the microservice’s processing time dominates the response time (~20 seconds per request). Autoscaling does not improve throughput in this case.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*aO9nTT7Ipkag8CE9oc6nQA.png" /></figure><p>The above graph shows the microservice latency when 2 task are running. When 3 Tasks are running latency improved gradually. When 4 tasks are running the latency is better as average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*B3_zabqJe7rqNxGM7IaLGw.png" /><figcaption>Latency MAX during Tasks Scaling</figcaption></figure><p>The above Amazon CloudWatch graph showing once the number of tasks increases, the latency gradually decreasing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*4SRfYGt6fBdkUKLREWBHJA.png" /><figcaption>CloudWatch dashboard</figcaption></figure><p>The above Amazon CloudWatch dashboard shows different graphs captured for various resources.</p><p><strong>3 Tasks (Autoscaling):- Why It Performs Better:</strong></p><p>- Adding a third task likely allows better load distribution across ECS tasks, reducing contention for resources and improving tail latency (P99).</p><p>- The lower standard deviation (4.2 ms) indicates more consistent performance, which is critical for user experience under load.</p><p>- This configuration strikes a balance between resource availability and minimal overhead, making it the most efficient in this test.</p><p><strong>2 EC2s + 4 Tasks (Autoscaling): Why It Doesn’t Outperform 3 Tasks:</strong></p><p>- While adding an EC2 instance and a fourth task increases capacity, it introduces slight overhead (e.g., load balancer routing, network latency, or task initialization), which may explain the marginally worse average and P99 latency compared to 3 tasks.</p><p>- The standard deviation (4.3 ms) is close to 3 tasks, suggesting similar consistency but no additional benefit from the extra EC2 instance.</p><p>- This setup may be over-provisioned for the tested load (10 concurrent users, 200 requests), as the computational bottleneck dominates.</p><p><strong>Autoscaling Benefits:</strong></p><p>- Autoscaling (both ECS tasks and EC2 instances) improves P99 latency and response time consistency (lower standard deviation) compared to the baseline.</p><p>- However, the benefits are limited because the microservice’s response time (~20 seconds) is likely dominated by a computational bottleneck (e.g., calculating the next prime number after 7) rather than infrastructure limitations.</p><h3>7. <strong>Extension of further work</strong></h3><p>In the next paper, the observability is done using AWS X-Ray service to monitor the latency, response time, success rate at P99 and P95 level. The exercise is carried out where the same service where scaled up with tasks running Amazon ECS cluster where infrastructure is provided by AWS Fargate and Amazon EC2.</p><p>In this analysis Apache Bench has been used as synthetic tool. For production like situation jMeter tool can be used. The Amazon CloudWatch has 1 minute latency for which granular observability may not be accurate. In next paper AWS X-Ray has been used for this purpose.</p><h3>8. Clean Up</h3><p><strong>a</strong>. Delete the Service running on Amazon ECS Service</p><p><strong>b</strong>. Delete the Amazon Load Balancer created</p><p><strong>c</strong>. Delete the target group</p><p>d. Delete the http api created on Amazon Api Gateway</p><h3>9. Conclusion</h3><p>In this paper, we have detailed a microservice’s performance under various auto-scaling conditions in Amazon ECS. Using Apache Bench to generate load and measure latency, we have compared performance when service has no auto-scale, service has auto-scale with and without Amazon EC2 auto scale in infrastructure attached to Amazon ECS cluster.</p><p>The auto scaled scenarios maintained similar or slightly better P99 latency compared to the baseline, indicating that scaling out prevented performance from degrading under load. The best configuration (in our test) was an Amazon ECS service scaling to 3 tasks, which achieved the lowest P99 latency and highest consistency. — <strong>Diminishing returns after a point</strong>: Scaling beyond 3 tasks to 4 tasks (on a new EC2 instance) did not yield further latency improvements, highlighting that more resources do not always equate to better performance for each request. The primary bottleneck was the application’s compute time, not the number of Amazon EC2s. There were no adverse effects observed on when service is on auto scale mode.</p><p>In summary, auto-scaling Amazon ECS tasks and Amazon EC2 instances can help a microservice handle higher load while keeping latency within SLA boundaries (here, P99 was kept around 20 seconds even as load increased). It provides <strong>resilience and consistency</strong>, which are crucial for meeting SLA targets in peak times. However, the <strong>absolute performance </strong>(latency per request) is largely determined by the application’s complexity and resources per instance. Future efforts should combine auto-scaling with code optimizations and possibly vertical scaling to achieve both scalability and low latency. This exercise demonstrated how to systematically use Apache Bench and AWS CloudWatch to assess microservice performance under different scaling strategies.</p><p><strong>Finally, the performance of any service depends on various use cases. The configurations of auto scaling depends on various outcome parameters from testing. </strong>The exercise here is done how to get the parameters and analyse using various AWS services and Apache Bench.</p><h3>11. Reference</h3><p><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html">https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ae33f4defe2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying AWS SSM Agent as a Sidecar for Task-Level Fault Injection :Chaos Engineering in Amazon…]]></title>
            <link>https://medium.com/@prasenjitrc/deploying-aws-ssm-agent-as-a-sidecar-for-task-level-fault-injection-chaos-engineering-in-amazon-dab49ec5a94d?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/dab49ec5a94d</guid>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[sidecar-pattern]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sat, 05 Apr 2025 18:55:07 GMT</pubDate>
            <atom:updated>2025-04-07T19:20:26.656Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploying AWS SSM Agent as a Sidecar for Task-Level Fault Injection :Chaos Engineering in Amazon ECS</h3><h3>1. Introduction</h3><p>This paper is for DevOps engineer who are responsible for validating micro services against probable operational issues in production in Amazon ECS. This the continuation of series of <a href="https://medium.com/@prasenjitrc/case-study-of-a-custom-circuit-breaker-for-saas-application-failures-in-aws-with-chaos-engineering-56d9583189d0">writings </a>on Chaos Engineering in Amazon ECS Service. In first paper we have explained the customized circuit breaker pattern in Amazon ECS Service. We intend to validate the pattern using AWS Fault Injection Service.</p><p>We have deployed our services on Amazon ECS Service using Amazon EC2. We want to validate the faults in single task level and not entire Amazon EC2. To validate the single task level interruption, <strong>AWS System Manager Agent (SSM) has to be deployed as sidecar of main container of service in the same task</strong>.</p><p><strong>While deploying the AWS SSM Agent container as sidecar there were few issues. This paper details what worked and what didn’t work while running AWS SSM Agent as sidecar.</strong></p><p>The section 4(Problem Statement), 5(What Worked) and 6(<strong><em>What didn’t work</em></strong>) are most important sections of the paper.</p><h3>2. Use Case</h3><p>The goal of the experimentation is to ensure that main application container is resilient with proper alerting against possible errors like SaaS application unavailability. The purpose of the experimentation is to use AWS FIS to simulate the interruption on the SAAS container running as single task on service in Amazon ECS using Amazon EC2. The use case demands AWS SSM as side car in running on same task where main application is running as service. The task must be registered as a managed instance in AWS Systems Manager.</p><h3>3. Architecture</h3><p>The SAAS container is running as a task on Amazon ECS using Amazon EC2. The container pulls the image from private Amazon ECR. The future work is to disrupt the SAAS application container using AWS FIS. To enable this disruption the SSM agent has to run on same task as sidecar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*b0ZnJ7zAlKN7-34uKtq44w.jpeg" /><figcaption>Architecture Diagram — SSM as Sidecar container</figcaption></figure><h3>4. Problem Statement</h3><p>The AWS SSM side car was crashing after start and the following error message was printing on Amazon CloudWatch loggroup.</p><p>The container is running on the task on public subnet having internet gateway. The underlying Amazon EC2 also have the public IP. The SSM agent was not able to start and following error was coming.</p><p><em>“- Curl </em><strong><em>error</em></strong><em> (28): Timeout was reached for </em><a href="https://cdn.amazonlinux.com/al2023/core/mirrors/2023.6.20250218/x86_64/mirror.list"><em>https://cdn.amazonlinux.com/al2023/core/mirrors/2023.6.20250218/x86_64/mirror.list</em></a><em> [Failed to connect to cdn.amazonlinux.com port 443 after 30000 ms: Timeout was reached.</em></p><p>The <strong>container was not able to connect with internet</strong> which is required to download the SSM agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*nYef3_QgNJ4JhjHkHtwv_g.png" /><figcaption>AWS console SSM Agent failure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*T8FxQlKHIkzuOTYHdepQnQ.png" /><figcaption>Error extracting Task ARN from ECS Container Metadata,exiting</figcaption></figure><h3>5. What worked — steps of deployment</h3><p>In the above section we have experienced that container was crashing even after it is started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TvZJN_zv76fZMoj-ZzqVew.png" /><figcaption>SSM Agent running as sidecar in same Amazon ECS task</figcaption></figure><p>a. Task Definition — The task definition has been added in <a href="https://github.com/PrasenjitRoychoudhury/SSMAgentSideCar">GitHub</a>. The task definition has been taken from AWS site <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Signals-ECS-Sidecar.html">link</a>.</p><p>The task definition provisions two containers in same task.</p><p>The SSM agent container is created as non-essential which means the main application container can run when SSM agent container fails.</p><p>The SSM agent has the very important command</p><p><strong>To install Necessary Utilities</strong> (dnf install jq procps awscli -y):</p><p>· These commands explicitly install additional packages required by the subsequent logic in your SSM agent’s startup script:</p><p>o <strong>jq</strong>: Used to parse JSON metadata returned from ECS task metadata endpoint.</p><p>o <strong>procps</strong>: Contains utilities like ps, needed by scripts to verify if the SSM agent process is already running.</p><p>o <strong>awscli</strong>: Needed to execute AWS CLI commands, like registering or deregistering the SSM agent activation.</p><p>For the task to start on Amazon ECS cluster on Amazon EC2 the internet connection is required which need NAT Gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qMwNA9HicHb1mpDm399krg.png" /><figcaption>Amazon CloudWatch log confirming SSM Agent Starting as side car</figcaption></figure><p>b. AWSVPC network mode</p><p>The task template has to have network mode as awsvpc when SSM agent is required by AWS FIS. This gives the task a ENI with private IP and not public IP. For container to connect with internet internet gateway doesn’t work and NAT gateway is required.</p><p>When using AWS Fault Injection Simulator (FIS) to interrupt or inject faults into Amazon ECS tasks running on Amazon EC2, ECS Task Definition <strong>must use the awsvpc networking mode</strong> because AWS FIS requires tasks to be individually addressable via <strong>Elastic Network Interfaces (ENIs)</strong>. When awsvpc is used as network mode the ENI created is private and not public. When an ECS task is launched with awsvpc mode, it receives its own dedicated Elastic Network Interface (ENI), attached directly to specified VPC subnet where by default the IP associated is private even if subnet/host Amazon EC2 is public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CNR8RObN79ihDUWimDLjOg.png" /><figcaption>ENI with private IP</figcaption></figure><p>c. NAT Gateway</p><p>Since task template requires both internet connection in network mode as awsvpc, the underlying container must need NAT gateway associated with the subnet.</p><p>So a NAT gateway is created on a public subnet and associated with customer Route Table.</p><p>d. Custom Route Table</p><p>A customer Route Table is created and attached with subnet which is attached to AWS ECS cluster and Amazon EC2 both.</p><p>The route of the customer route table is edited where connection is made with NAT gateway.</p><p>e. AWS IAM role</p><p>One custom role is created which has one AWS Managed role <a href="https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/policies/details/arn%3Aaws%3Aiam%3A%3Aaws%3Apolicy%2FAmazonSSMManagedInstanceCore">AmazonSSMManagedInstanceCore</a> and Custom Inline created role which has following entries.</p><blockquote><em>“Statement”: [</em></blockquote><blockquote><em>{</em></blockquote><blockquote><em>“Effect”: “Allow”,</em></blockquote><blockquote><em>“Action”: [</em></blockquote><blockquote><em>“ssm:DeleteActivation”,</em></blockquote><blockquote><em>“ssm:DeregisterManagedInstance”,</em></blockquote><blockquote><em>“ssm:RegisterManagedInstance”,</em></blockquote><blockquote><em>“ssm:UpdateInstanceInformation”,</em></blockquote><blockquote><em>“ssm:ListInstanceAssociations”,</em></blockquote><blockquote><em>“ssm:DescribeInstanceInformation”,</em></blockquote><blockquote><em>“ec2messages:*”,</em></blockquote><blockquote><em>“ssmmessages:*”,</em></blockquote><blockquote><em>“cloudwatch:PutMetricData”,</em></blockquote><blockquote><em>“ec2:DescribeInstanceStatus”</em></blockquote><blockquote><em>],</em></blockquote><blockquote><em>“Resource”: “*”</em></blockquote><blockquote><em>}</em></blockquote><h3>6. What didn’t work</h3><p>While doing the experimentation there were many things which didn’t work.</p><p>a. Running the Amazon EC2 with SSM agent preinstalled didn’t work.</p><p>The AWS FIS needs to interrupt the container running as task on Amazon EC2. The purpose is to interrupt only the container and not entire Amazon EC2. Containers running in Amazon ECS are isolated processes with their own namespaces and resources separate from the host Amazon EC2 instance. SSM Agent installed on Amazon EC2 has access only at the host OS level, and not directly inside the container namespaces.</p><p>The problem was resolved through running SSM agent as sidecar as task must be registered as a managed instance in AWS Systems Manager. A sidecar container runs alongside your application container in the same ECS task, sharing the same network and lifecycle.</p><p>b. Why Amazon ECS Task Definition with network modes bridge and host will not work</p><p>The SSM agent needs egress internet access to communicate with AWS System Manager service endpoint and download the image from Amazon ECR.</p><p>In network host mode, containers share the host Amazon EC2 instance’s network namespace and IP address space. Containers do not get isolated network interfaces or unique IP addresses separate from the host which is required by sidecar.</p><p>In the network bridge mode containers do not have individual IP addresses from the VPC subnet. This restricts AWS SSM agent that expect unique and routable VPC addresses, from directly interacting with them.</p><p>The problem was resolved when network mode is selected as AWSVPC as Each ECS task receives a unique ENI with its own private IP address, which is addressable by AWS services</p><p>c. Why internet gateway doesn’t work</p><p>After placing the SSM agent as sidecar with network mode as awsvpc in host Amazon EC2 with public subnet the container was not able to connect with internet. When Amazon ECS tasks use the awsvpc network mode, each task gets its own Elastic Network Interface (ENI). However, AWS attaches only a private IP to this ENI by default, even if placed in a public subnet in host Amazon EC2 having public IP.</p><p>The issue was resolved when Amazon NAT Gateway was connected with the subnet.</p><h3>7. How to use</h3><p>The setup can be used by AWS Fault Injection service to test the custom circuit breaker mentioned in previous <a href="https://medium.com/@prasenjitrc/case-study-of-a-custom-circuit-breaker-for-saas-application-failures-in-aws-with-chaos-engineering-56d9583189d0"><em>paper</em></a>.</p><h3>8. Which services are used</h3><p>a. Amazon ECS cluster — This is used for hosting the service with desired application task defined as essential and SSM agent as sidecar and non-essential.</p><p>b. Amazon EC2 services — Target group, ASG, Load Balancer</p><p>c. Amazon VPC services — NAT Gateway, Elastic IP, Subnet</p><p>All the services have been deployed using the AWS Console. The same thing can be done using cli command also.</p><h3>8. Extension</h3><p>a. Why SSM agent crashes after starting and gives start 1 error. A detailed study and</p><p>b. Circuit Breaker Metrics and Observability Enhancements: Integrate observability tools like CloudWatch, X-Ray, or Prometheus to monitor how circuit breaker patterns react during fault injections, and quantify metrics such as failover time, error rates, and recovery duration.</p><p>c. Testing Multi-Task and Service-Level Resilience: Expand the scope from single task interruption to simulate partial failures in services with multiple task replicas, validating ECS service-level circuit breaker behaviour and load balancer failover.</p><p>d. Custom Fault Injection with SSM Documents: Develop custom SSM documents to inject application-layer faults, simulate disk pressure, CPU throttling, memory leaks, or controlled crashes beyond just network-level disruptions.</p><p>e. Cross-Region and Cross-AZ Fault Injection: Explore FIS-based experiments that simulate failures across availability zones or regions to validate multi-AZ architecture resilience and traffic rerouting strategies.</p><p>SSM Agent Optimization with VPC Endpoints — How to replace the NAT Gateway with AWS PrivateLink and VPC endpoints for SSM, CloudWatch, and ECR services to eliminate outbound internet dependency.</p><h3>9. Clean Up</h3><p>To avoid the recurring costs in non-production after exercise is completed, following resources need to be deleted.</p><p>a. Delete the Amazon ECS Task Template</p><p>b. Delete the Amazon ECS Service</p><p>c. Delete the Amazon ECS Cluster</p><p>d. Delete Target Group and AWS Load Balancer created in EC2 Section</p><p>e. Delete in following sequence</p><p>i. De-associate the subject associated with custom in VPC section</p><p>j. Delete Custom Route table created for NAT gateway in VPC section</p><p>k. Delete NAT gateway</p><p>l. Release the Elastic IP created</p><p>f. Delete all services roles created on Amazon IAM</p><p>g. Delete the Amazon CloudWatch Loggroup created</p><h3>10. Conclusion</h3><p>This paper detailed the practical implementation and challenges of enabling AWS Fault Injection Simulator (FIS) to target individual ECS tasks running on Amazon EC2 by deploying the AWS Systems Manager (SSM) agent as a sidecar container. Through hands-on experimentation, we demonstrated why using awsvpc network mode is essential, how ENIs behave in ECS, and why a NAT Gateway is required to provide internet access for private task ENIs. We also clarified why traditional network modes like bridge and host are incompatible with FIS at the task level. By resolving common issues such as SSM agent start-up failures and container crashes due to lack of connectivity, we established a reliable architecture for running fault injection scenarios at the container level. These insights offer a repeatable framework for implementing chaos engineering in ECS environments and pave the way for more resilient service deployments.</p><h3>11. Reference</h3><p><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Signals-ECS-Sidecar.html">Sidecar SSM agent</a>, <a href="https://aws.amazon.com/blogs/containers/announcing-aws-fault-injection-simulator-new-features-for-amazon-ecs-workloads/">AWS Fault Injection</a>, <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/circuit-breaker.html">Cloud Design Pattern</a>, <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/hybrid-multicloud-service-role.html">Hybrid Multi-cloud environment</a>, <a href="https://aws.amazon.com/blogs/compute/using-the-circuit-breaker-pattern-with-aws-lambda-extensions-and-amazon-dynamodb/">Circuit Breaker</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dab49ec5a94d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case Study of a Custom Circuit Breaker for SaaS Application Failures in AWS with Chaos Engineering]]></title>
            <link>https://medium.com/@prasenjitrc/case-study-of-a-custom-circuit-breaker-for-saas-application-failures-in-aws-with-chaos-engineering-56d9583189d0?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/56d9583189d0</guid>
            <category><![CDATA[circuit-breaker]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sat, 05 Apr 2025 18:22:18 GMT</pubDate>
            <atom:updated>2025-04-06T08:29:08.233Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>This is foundational case study paper for chaos engineering for DevOps engineer who are responsible for building and maintaining highly available applications. The use case is on production grade problems where an application makes repetitive SaaS application when it fails intermittently. In a production environment handling 1000 transaction per minute and synchronously makes API call to SaaS application. A 5% of unresponsive traffic can increase repetitive calls to increase by 15–25% which can create 99th percentile latency increased by 500%.</p><p>We have conceptualized a solution using various AWS services for the above use case. The paper has details of code, deployment steps, observability process implemented in our personal cloud account.</p><p>In the development of a highly available application that interacts with a third-party SaaS application, we had encountered a critical failure scenario: the external SAAS application(API) became unresponsive, and the application — lacking a robust fault-tolerance mechanism — continued to make repeated requests. The retry mechanism was configured without proper limits, resulting in an effectively infinite loop of failed API calls. As the system waited for responses, downstream resources such as database connections were exhausted and left in a hung state. This cascading failure led to significant resource contention and operational noise across the Amazon ECS cluster hosting the main application.</p><p>To address this, we implemented a custom circuit breaker solution and auto healing using various AWS services. Application is designed to detect service degradation and prevent repeated calls to a failing dependency. This paper details the implementation including code, deployment steps, and observability practices, with all resources available in a <a href="https://github.com/PrasenjitRoychoudhury/CustomCircuitBreaker">GitHub </a>repository. It serves as the first in a series exploring resilience patterns and chaos engineering in AWS.</p><h3>2. Use Case</h3><p>Any SAAS application invoked through API call sometimes responds with 50x response for various reason. Any application invoking the SAAS application is supposed to re try in case of 5xx HTTP errors (503 Service Unavailable) response. If the SAAS application is down for indefinite time, prolonged initial API invocation and corresponding retry can put lots of pressure on the application and cluster.</p><p>We have designed and detailed a custom circuit breaker pattern with proper observability. The custom circuit breaker includes the following key features:</p><p>a. Exponential Backoff — Delays retries with increasing intervals to reduce load on failing services</p><p>b. Circuit breaker with closed, half-open and open states — Maintains closed, half-open, and open states to control API call behaviour</p><p>c. Observability in Amazon CloudWatch dashboard — Tracks circuit state changes and API call outcomes in Amazon CloudWatch</p><p>d. Triggering of Auto Heal when circuit transition to closed state from open state — Triggers asynchronous redrive of failed requests when the SaaS API recovers.</p><p>This use case lays the groundwork for chaos engineering experiments, enabling fault injection to validate system resilience. The assumption of the use case is the functionality implemented through SaaS API call is idempotent and can be tried asynchronously in case of failure.</p><h3>3. Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*y5fhMsU4r5loNHUSNDjP7Q.jpeg" /><figcaption>Architecture Diagram for custom circuit breaker with auto heal</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*Ijm-xX1auy8UJ65cB8thwQ.jpeg" /><figcaption>Sequence Diagram for custom circuit breaker with auto heal</figcaption></figure><h3>4. How to use</h3><p>We shall focus briefly on the use case implementation and what AWS services are used in next section to achieve.</p><p>a. Circuit Breaker</p><p>Here the customized circuit breaker is used so that consecutive calls are not made to SaaS API when SaaS API is down. The circuit states are maintained in container session where states are closed, half open or open. The states are changed based on customized logic. When the states are half open or open, the calls are not sent to SaaS API and stored in DLQ for later redrive and processing. The detailed code has been included in the <a href="https://github.com/PrasenjitRoychoudhury/CustomCircuitBreaker">GitHub</a> link.</p><p>b. How to redrive</p><p>There is an asynchronous process in the solution where the circuit breaker status is changed from open to closed. There is a parallel process in the container which makes health check calls to SaaS application to check the connectivity. If the connection is successful the state is changed to open and asynchronous calls are made to SaaS API to redrive failed messages.</p><p>c. Observability</p><p>Any observability needs a proper alerting and tracing. Here the logs are printed with proper messages for each request and state change. There is one dashboard implemented with service calls status history and circuit status in last 5/10 minutes. Details are depicted <a href="https://github.com/PrasenjitRoychoudhury/CustomCircuitBreaker/blob/main/CircuitBreakerObservabilityScreenShot.pdf">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*JpfAifR1sAafa_ZLRSOsvA.png" /><figcaption>Amazon CloudWatch Dashboard for circuit breaker state</figcaption></figure><h3>5. What services are used</h3><p>a. Amazon ECS cluster — The container is placed on Amazon EC2. The container is started using the task template defined in Amazon ECS service. The reason for selecting Amazon ECS service is that for future study AWS Fault Injection service is used for which sidecar required for which task template is used.</p><p>b. Amazon SQS — The Amazon SQS is used for storing the records which are used for storing messages which can’t be placed in SaaS API during circuit open/half open.</p><p>c. Amazon SNS — The circuit breaker makes a period health check call to SaaS API to check if the connection is made. If the connection is successful the circuit breaker makes an attempt to redrive which is triggered by Amazon SNS.</p><p>d. AWS Lambda — The AWS Lambda is used as redrive logic which gets invoked by Amazon SNS. The AWS Lambda code retrieves messages from Amazon SQS and post to SaaS API.</p><p>e. Amazon CloudWatch Loggroup and Dashboard — The observability is monitored through Amazon CloudWatch Loggroup where particular texts are printed when circuit states are changed. The corresponding matrices are created which are used to show the statistical data in graphs in dashboard.</p><p>All the services have been deployed using the AWS Console and using AWS cli commands. The detailed screenshots, code and task template are detailed in <a href="https://github.com/PrasenjitRoychoudhury/CustomCircuitBreaker">GitHub</a> link.</p><h3>6. Benefits</h3><p>The implementation of a custom circuit breaker pattern, integrated with AWS services such as Amazon ECS, SQS, SNS, and CloudWatch, delivers several key benefits to the application’s resilience while not overutilizing the cluster resources.</p><p>a. I<strong>mproved Resilience</strong>: Prevents cascading failures by blocking calls to an unresponsive SaaS API, reducing retry attempts by up to 20% in simulated tests.</p><p>b. <strong>Resource Efficiency</strong>: Minimizes ECS cluster CPU and memory utilization by avoiding infinite retry loops.</p><p>c. <strong>Auto-Healing</strong>: Enables asynchronous redrive of failed requests via AWS Lambda, ensuring no data loss during API outages.</p><p>d. <strong>Enhanced Observability</strong>: Provides real-time insights into circuit states and API performance through CloudWatch, enabling proactive issue detection.</p><p>e. <strong>Chaos Engineering Readiness</strong>: Prepares the system for fault injection experiments by isolating failures at the application level. These benefits align with chaos engineering principles, ensuring robust and cost-effective operations.</p><h3>7. Extension of further work</h3><p>The solution will be further enhanced in following papers.</p><p>a. The validation process when the % of API response is non responsive . The above scenario requires AWS Fault Injection Service.</p><p>b. The AWS Fault Injection Service requires AWS SSM agent running as sidecar in same task. This is detailed in <a href="https://medium.com/@prasenjitrc/deploying-aws-ssm-agent-as-a-sidecar-for-task-level-fault-injection-chaos-engineering-in-amazon-dab49ec5a94d">paper</a>.</p><p>There are further possibilities where existing application can be redesigned using following concepts.</p><p>a. Replace Custom Circuit Breaker with ECS Circuit Breaker</p><p>b. Enhanced Observability Using Grafana</p><p>c. Persistent Circuit State Storage</p><p>d. Application performance monitoring and distributed tracing</p><h3>8. Clean Up</h3><p>To avoid the recurring costs in non-production after exercise is completed, following resources need to be deleted.</p><p>a. Confirm ASG is updated with desired instance count as 0. This will ensure Amazon EC2 attached to Amazon ECS cluster is terminated.</p><p>b. Delete the Target Group and Load balancer from Amazon EC2 section. Both of these incur costs.</p><p>c. Delete the service running on Amazon ECS cluster.</p><p>d. Delete all services roles created on Amazon IAM once full exercise is completed.</p><p>e. Delete the Amazon CloudWatch Loggroup created once full exercise is completed</p><h3>9. Conclusion</h3><p>In the first paper we have detailed what is the use case of an application running on shared cluster call external SaaS API. We have briefed about neighbouring problem when SaaS API is unresponsive and application call makes repeated calls. We have used various AWS services to deploy custom circuit breaker. Finally, we have created observability for statistics on Amazon CloudWatch. This is foundational paper which will be further enhanced in future <a href="https://medium.com/@prasenjitrc/deploying-aws-ssm-agent-as-a-sidecar-for-task-level-fault-injection-chaos-engineering-in-amazon-dab49ec5a94d">paper </a>where chaos engineering concepts will be exercised.</p><h3>10. Reference</h3><p><a href="https://aws.amazon.com/blogs/containers/announcing-aws-fault-injection-simulator-new-features-for-amazon-ecs-workloads/">AWS Fault Injection</a>, <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/circuit-breaker.html">Cloud Design Pattern</a>, <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/hybrid-multicloud-service-role.html">Hybrid Multi-cloud environment</a>, <a href="https://aws.amazon.com/blogs/compute/using-the-circuit-breaker-pattern-with-aws-lambda-extensions-and-amazon-dynamodb/">Circuit Breaker</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56d9583189d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ECS Fargate Spot on Graviton2]]></title>
            <link>https://towardsaws.com/aws-ecs-fargate-spot-on-arm64-9083e6b7e9cb?source=rss-f86efc65df6c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9083e6b7e9cb</guid>
            <category><![CDATA[fargate-spot]]></category>
            <category><![CDATA[arm64]]></category>
            <category><![CDATA[graviton]]></category>
            <dc:creator><![CDATA[Prasenjit Roychoudhury]]></dc:creator>
            <pubDate>Sun, 22 Dec 2024 10:16:12 GMT</pubDate>
            <atom:updated>2025-09-08T05:23:17.915Z</atom:updated>
            <content:encoded><![CDATA[<p>The paper details how to deploy docker containers on AWS ECS Fargate Spot on Graviton chip for cost performance benefit.</p><h3>1. Introduction</h3><p>In Oct’2024, AWS published that <a href="https://aws.amazon.com/about-aws/whats-new/2024/09/amazon-ecs-graviton-based-spot-compute-fargate/">AWS ECS Fargate Spot </a>will be supported on AWS Graviton based compute. AWS ECS Fargate Spot on AWS Graviton is a cost-effective solution for running containerized applications on AWS.</p><p>This paper details on what are the use cases and cost-performance benefits for running services/tasks on AWS ECS Fargate Spot on ARM64. The paper also details how to build and deploy containers on ARM64 step by step.</p><p><strong>AWS Fargate Spot for Amazon ECS</strong></p><p>AWS Fargate Spot is a cost-saving option that allows to run containers at a significant discount of up to 70% compared to running on AWS ECS Fargate. AWS Fargate Spot is ideal for workloads that are fault-tolerant and can handle occasional interruptions, such as batch processing jobs, data analysis tasks, and development environments.</p><p><strong>Graviton Chip (ARM64)</strong></p><p>Graviton is a custom-built Arm-based processor designed by AWS to deliver high performance and cost-efficiency. Graviton-based containers offer up to 20% better price-performance compared to comparable x86-based counterparts. <a href="https://medium.com/@prasenjitrc/how-to-execute-failover-reports-in-fargate-spot-tasks-containers-6b00887b77ff">In earlier paper</a>, it is explained how application can be deployed on AWS Fargate spot capacity. Also it is explained how to do unit testing and schedule invocation through Amazon EventBridge Scheduler. The underlying assumption to run any application on AWS Fargate spot with Amazon ECS clusters on ARM64archicture is that the application has to support fault tolerance and idempotency.</p><p>A key consideration is underlying code has to support the ARM64 architecture.</p><h3>2. Use Cases</h3><p>Following are use-cases that can be deployed on AWS ECS Fargate Spot on ARM64. The combination of Spot capacity and ARM64 can lead to significant cost reductions for non-time-critical workloads.</p><p><strong>Cost-sensitive workloads:</strong></p><ul><li><strong>Non-production environments:</strong> Development, testing, and staging environments where cost efficiency is a priority.</li><li><strong>Batch processing jobs:</strong> Data processing, ETL jobs, and other tasks that can be scheduled and run intermittently.</li><li><strong>Machine learning and data analysis:</strong> Training models, running experiments, and analysing large datasets.</li><li><strong>Web scraping and crawling:</strong> Tasks that require frequent scaling and can tolerate occasional interruptions</li><li><strong>Periodic Scheduled Tasks</strong> Workloads that run at specific intervals and can be quickly restarted</li><li>Nightly data backups</li><li>Periodic report generation</li><li>System maintenance scripts</li><li>Log processing and archiving</li></ul><h3>3. Cost Benefit Analysis</h3><p>For cost benefit analysis the use case is considered where 5 tasks run continuously (24/7) with 2 vCPU and 4GB RAM across different AWS Fargate options.</p><p>Let’s break down the monthly costs for each scenario (using US East — N. Virginia region pricing as of December 2024).</p><p>a. <strong>Standard AWS Fargate (x86)</strong></p><ul><li>vCPU Price: $0.04048 per vCPU-hour</li><li>Memory Price: $0.004445 per GB-hour</li><li>Calculation:</li><li>vCPU Cost: (5 tasks × 2 vCPUs × $0.04048 × 24 × 30) = $726.64</li><li>Memory Cost: (5 tasks × 4GB × $0.004445 × 24 × 30) = $399.64</li><li><strong>Total Monthly Cost: $1,126.28</strong></li></ul><p><strong>b. AWS Fargate Spot (x86)</strong></p><ul><li>vCPU Price: $0.01214 per vCPU-hour (about 70% discount)</li><li>Memory Price: $0.001334 per GB-hour (about 70% discount)</li><li>Calculation:</li><li>vCPU Cost: (5 tasks × 2 vCPUs × $0.01214 × 24 × 30) = $218.52</li><li>Memory Cost: (5 tasks × 4GB × $0.001334 × 24 × 30) = $119.86</li><li><strong>Total Monthly Cost: $338.38</strong></li></ul><p><strong>c. AWS Fargate Spot (ARM64 Graviton)</strong></p><ul><li>vCPU Price: $0.01036 per vCPU-hour (about 74% discount)</li><li>Memory Price: $0.001134 per GB-hour (about 74% discount)</li><li>Calculation:</li><li>vCPU Cost: (5 tasks × 2 vCPUs × $0.01036 × 24 × 30) = $186.48</li><li>Memory Cost: (5 tasks × 4GB × $0.001134 × 24 × 30) = $102.24</li><li><strong>Total Monthly Cost: $288.72</strong></li></ul><blockquote><strong>Cost Comparison Summary:</strong></blockquote><blockquote>Standard Fargate: $1,126.28</blockquote><blockquote>Fargate Spot (x86): $338.38 (70% savings)</blockquote><blockquote>Fargate Spot (ARM64): $288.72 (74% savings)</blockquote><h3>4. Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*PpYmSsa0xpg_6Qmk3Q57IA.png" /><figcaption>Architecture Diagram for AWS ECS Fargate Spot on ARM64</figcaption></figure><h3>5. Steps of deployment</h3><p>Building a Docker container for ARM64 architecture can be achieved through two primary methods.</p><p>A. Build for ARM64: It has to be ensured that container images are built specifically for the ARM64 architecture. This involves using appropriate base images and cross-compiling application if necessary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*sU0aWtQRIvJrV19XAOfvaw.png" /><figcaption>Amazon EC2 for building container on ARM64</figcaption></figure><p>The following are commands to build container on Amazon EC2(Arm64).</p><pre>i. docker build -t hello-world-arm64 .<br>ii. docker run hello-world-arm64<br>iii. docker tag hello-world-arm64:latest &lt;accountid&gt;.dkr.ecr.us-east-1.amazonaws.com/helloworldfargategraviton<br>iv. docker push &lt;accounted&gt;.dkr.ecr.us-east-1.amazonaws.com/helloworldfargategraviton</pre><p>For this we have created an Amazon EC2 in ARM64 architecture where the docker container has been built. The <a href="https://github.com/PrasenjitRoychoudhury/AWSFargateSpotGraviton">github link</a> has details of each step.</p><p>B. Multi-Architecture Images: Here building multi-architecture images that can run on both x86 and ARM64 platforms. This provides flexibility and allows for easier deployment across different environments.</p><p>Docker Desktop, available for Windows and macOS, simplifies container development by bundling necessary tools and configurations. It employs a hypervisor to run a lightweight Linux kernel (LinuxKit) with QEMU emulation, enabling the execution of binaries for various architectures, including ARM. Arm actively supports Docker’s QEMU fork, ensuring up-to-date emulation capabilities. Docker Desktop Edge introduces the buildx command, which facilitates the creation of multi-architecture images, encompassing both native and emulated platforms. Buildx leverages BuildKit and Docker Desktop’s technology stack to seamlessly build images for different architectures, including ARM64, on x86 machines. By utilizing QEMU emulation, Docker Desktop empowers developers to build and run ARM64 container images directly on their x86 Windows or macOS systems.</p><p>Following are commands used for building docker on multi-architecture.</p><pre>i. docker buildx create --use<br>ii. docker buildx inspect --bootstrap<br>iii. docker buildx build --platform linux/arm64  --load -t hello-world-arm64:latest .<br>iv. docker inspect hello-world-arm64:latest<br>v. docker run hello-world-arm64:latest<br>vi. aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin &lt;ecr repot&gt;.dkr.ecr.us-east-1.amazonaws.com/helloworldfargategraviton<br>vii. docker tag hello-world-arm64:latest &lt;ecr repot&gt;.dkr.ecr.us-east-1.amazonaws.com/helloworldfargategraviton<br>viii. docker push &lt;ecr repot&gt;.dkr.ecr.us-east-1.amazonaws.com/helloworldfargategraviton  </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*mQ5uw5oqJuXoQ8PAKhHMYQ.png" /><figcaption>Container Image created by Emulsion</figcaption></figure><p>C. AWS ECS Fargate Task Definition</p><p>Once the container has been build and pushed to Amazon ECR container there has to be on AWS ECS Fargate Task Defination that has to support ARM64 architecture.</p><p>The task definition has to be created which runtimePlatform as ARM64.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*KAs-PzieGpVtGPCfnCzouQ.png" /><figcaption>AWS ECS Task Definition on ARM64</figcaption></figure><p>D. The AWS ECS cluster task creation should have capacity provider as Spot. This has been detailed in the earlier paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*P5Yn308vYmnJTHiCdTMWJA.png" /><figcaption>AWS ECS Task on cluster with capacity defined FARGATE_SPOT</figcaption></figure><p>Once the task runs on AWS ECS Fargate Spot on ARM64, the validation can be done to confirm container has run properly. Following is snippet from Amazon CloudWatch Loggroup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*Ptz4XfDIBD-lSSBWGxmzkg.png" /><figcaption>Amazon CloudWatch Loggroup for task on ARM64</figcaption></figure><p>E. The dockerFile should contain appropriate entries to support the build of Python file for build on ARM64 architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*jjOtmzGutjNfP4ZPglh5Dg.png" /><figcaption>DockerFile for ARM64 build for Python</figcaption></figure><h3>6. Benefits</h3><p>Running tasks/services on AWS Fargate Spot on Graviton offers a compelling combination of cost-effectiveness and performance advantages:</p><p><strong>Cost-Effectiveness:</strong></p><ul><li><strong>Spot Pricing:</strong> Fargate Spot leverages spare EC2 capacity, often resulting in significant cost savings (up to 70%) compared to on-demand pricing.</li><li><strong>Graviton Processor Efficiency:</strong> Graviton processors, based on Arm architecture, offer superior price-performance compared to traditional x86 processors. This further enhances cost savings.</li></ul><h3>7. Extension</h3><p><strong>Java:</strong></p><ul><li>If code is build using <strong>JVM</strong> it should support ARM64 (e.g., Amazon Corretto, OpenJDK).</li><li>Any JARs containing x86-only native code has to be recompiled for ARM64.</li><li>Some specific Java-based frameworks or third-party libraries might still depend on x86 binaries.</li></ul><p><strong>Python:</strong></p><ul><li>All dependencies in requirements.txt or Pipfile have ARM64 wheels available. Many popular Python libraries like NumPy, Pandas, and TensorFlow already support ARM64.</li></ul><p>Considerations and Best Practices:</p><p>· Design applications to be stateless and fault-tolerant</p><p>· Implement robust error handling and retry mechanisms</p><p>· Use container orchestration tools like Amazon ECS</p><p>· Combine with Auto Scaling groups for optimal resource management</p><h3>8. Clean Up</h3><p>To avoid the recurring costs in non-production after exercise is completed, following resources need to be deleted.</p><p>a. Confirm no running tasks running condition on Amazon ECS cluster</p><p>b. Delete the Amazon ECS Task Template</p><p>c. Delete the Amazon ECS Cluster</p><p>d. Delete Amazon CloudWatch loggroup</p><p>e. Amazon ECR</p><h3>8. Conclusion</h3><p>Following are key Observations:</p><ul><li>Fargate Spot provides significant cost reductions</li><li>ARM64 (Graviton) offers additional savings over x86 Spot instances</li></ul><p>In the paper, we have discussed how task/services can be deployed in AWS Fargate Spot in ARM64. To run container on Graviton, the container has to build to match with ARM64 architecture. This paper helps in building containers for appropriate use cases in non-production and production environment in AWS Fargate Spot in ARM64 architecture which help users to get better price benefit.</p><p>a. My <a href="https://medium.com/@prasenjitrc/how-to-execute-failover-reports-in-fargate-spot-tasks-containers-6b00887b77ff">paper </a>on AWS Fargate Spot</p><p>b. My <a href="https://medium.com/@prasenjitrc/interruption-simulation-of-fault-tolerant-tasks-in-aws-fargate-spot-3a841dc43b8f">paper </a>on AWS Fargate Spot interruption Simulation</p><p>c. My <a href="https://medium.com/@prasenjitrc/aws-ecs-fargate-spot-on-arm64-9083e6b7e9cb">paper </a>on step by step guide to deploy AWS Fargate Spot on Graviton</p><h3>9. Reference</h3><p>AWS <a href="https://aws.amazon.com/about-aws/whats-new/2024/09/amazon-ecs-graviton-based-spot-compute-fargate/">Reference </a>for supporting ARM64 for AWS ECS Fargate Spot</p><p>Docker <a href="https://www.docker.com/blog/multi-arch-images/#:~:text=This%20fast%20and%20lightweight%20container,run%20binaries%20of%20any%20supported">reference </a>for supporting multi-architecture for docker</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9083e6b7e9cb" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/aws-ecs-fargate-spot-on-arm64-9083e6b7e9cb">AWS ECS Fargate Spot on Graviton2</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>