<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Claude Ducharme on Medium]]></title>
        <description><![CDATA[Stories by Claude Ducharme on Medium]]></description>
        <link>https://medium.com/@claude.ducharme?source=rss-3f27460752dd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pMo6yKnzUjtoiVn25VG-rQ.jpeg</url>
            <title>Stories by Claude Ducharme on Medium</title>
            <link>https://medium.com/@claude.ducharme?source=rss-3f27460752dd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:08:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@claude.ducharme/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Resilient, Performant, and Scalable Message Processing System]]></title>
            <link>https://medium.com/@claude.ducharme/resilient-performant-and-scalable-message-processing-system-6d7e74706d75?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d7e74706d75</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[parallel-processing]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[idempotency]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Wed, 03 Jul 2024 13:59:56 GMT</pubDate>
            <atom:updated>2024-07-03T22:56:05.814Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbnPn2HdG0YDQ3QGbeg92g.png" /><figcaption>Resilient, Performant, and Scalable Message Processing System — Sequencer PoC</figcaption></figure><p>Recently I took up the challenge of designing and implementing a message processing PoC that is resilient, performant, and scalable. Key takeaways for reusing the solution are:</p><ul><li>Processing of messages in parallel in the first layer and Sequencing in the second layer (they can also be used independently)</li><li>Fast Leader-Follower failover (within 1 second) between using distributed lock (baton mechanism) on Redis. Three versions are provided with unit tests to validate their behaviors: NativeLock using LockTakeAsync leveraging WATCH command, SimpleCustomLock with Redis String, and AtomicCustomLock using Lua scripts</li><li>Implementation of idempotency with Redis List</li><li>Different groups can be configured in the system. In case performance is critical, deploying separate instances of the system is recommended</li></ul><h3>Challenge</h3><p>The system should be able to handle a high volume of messages while maintaining the original sequence order of the messages.</p><h4>Flow</h4><ol><li>Retrieve messages one by one from an input queue</li><li>Process messages in parallel</li><li>Update the system of record for each message</li><li>Write messages to an output queue</li></ol><h4>Constraints</h4><ol><li>Messages must be processed in their incoming order</li><li>The sequence order must be maintained when outgoing</li></ol><h4>Tests</h4><ol><li>Start multiple services for parallel processing</li><li>Prepare hundreds of messages in an input queue</li><li>Each message has at least an unique sequence number and a small payload</li><li>Long processing time will be simulated by waiting for 500 ms</li></ol><h4>Verifications</h4><ol><li>Order is preserved by confirming output sequence is the same as the input</li><li>System can recover fast for process and connection failures</li><li>No messages are lost or duplicated even when disruptions occur</li><li>Time elapsed when all messages are generated at once (burst mode / load test)</li><li>Time elapsed when messages are generated at a sustainable processing rate (soak test)</li></ol><h4>Implementation choices</h4><ol><li>ElastiCache Serverless Redis providing 99.99% availability SLA</li><li>Develop clients in C# .NET Core 8.0 referencing StackExchange.Redis and Npgsql libraries</li><li>Deploy multiple active-active instances of a Processing service reading from an input List and writing to a Stream</li><li>Deploy a few leader-follower instances of a Sequencing service reading from the Stream and writing to an output List</li><li>Implement leader-follower with a distributed lock in Redis having expiry time of 1 second allowing very fast switch</li><li>Keep the followers constantly ready to be the leader and continuously requesting ownership of the distributed lock</li><li>Utilize Redis Pub/Sub with channels to limit short polling requests</li><li>Leverage Lua scripts and transactions to ensure good performances</li><li>Implement idempotency in Redis for performance considerations</li><li>Use Redis as main system of record, although switching to PostgreSQL database is possible</li><li>On top of sequence number and payload, message format includes creating, processing, processed, sequencing, and sequenced datetimes to easily calculate performance statistics</li></ol><h4>Potential improvements</h4><ol><li>Processing service handling multiple messages in parallel</li><li>Partitioning/Sharding the input queue and channels based on the message sequence number</li><li>Enhance security (note this is a PoC and much more is required to release to production)</li><li>Implement robust monitoring to identify faster the bottlenecks</li></ol><h4>By relaxing the ordering constraint</h4><ol><li>Remove the Sequencing service to significantly increase performance</li><li>Create a Dead Letter Queue when processing fails a few time or encounter specific exceptions</li></ol><h3>Diagrams</h3><h4>Architecture</h4><ol><li>The input FIFO queue (Pending List) is where messages are stored and retrieved by the processors</li><li>The Processor services (Active-Active) perform parallel processing of messages from the input FIFO queue (can be scaled as needed) then write processed messages to a Processed Stream</li><li>The Sequencer services (Active-Standby/Leader-Follower) are responsible for maintaining the original sequence order of the messages. They read messages from the Processed Stream and write them to an output FIFO queue (Sequenced List), ensuring the original sequence order is preserved</li><li>Operations on Lists and Streams are follow by the publication of events on channels to avoid many unwanted short pollings requests by services</li><li>A database can be used as the system of record for storing processed messages (much slower than saving into Redis so it has been made optional)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y37Ta7LKvbqBWv2G" /><figcaption>Architecture Diagram</figcaption></figure><h4>Sequence</h4><p>This sequence diagram represents the interactions and flow of messages between services and Redis for storage and communication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xrsPxEqzC5WfNgv3" /><figcaption>Sequence Diagram</figcaption></figure><p>The diagram shows the flow of messages from the AdminService to the PendingList, then to the ProcessorService for processing. The processed messages are added to the ProcessedStream and then sequenced by the SequencerService. The SequencerService maintains the original order of messages and adds them to the SequencedStream and SequencedList.</p><p>The use of publish/subscribe channels allows for communication and coordination between the services. The ProcessorService subscribes to the PendingNewMessagesChannel/NewMessages channel to receive notifications about new messages, while the SequencerService subscribes to the ProcessedStreamChannel/ProcessedMessages and SequencedStreamChannel/HighestEntryIdAndSequence channels to receive notifications about processed messages and the highest sequenced entry ID, respectively.</p><h3>Project implementation</h3><p>Available under MIT license at <a href="https://github.com/Ducharme/sequencer">https://github.com/Ducharme/sequencer</a></p><h3>Next</h3><p>Dockerized services seems to be faster than launching the processes with dotnet command directly. Moving forward, I will try to create an AWS EKS cluster to host services as pods with eksctl, add monitoring capabilities to pinpoint performance bottlenecks faster, and provision the IaC with AWS CDK or terraform instead of using AWS CLI commands. Please let me know what you would like to see in future iterations!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d7e74706d75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to configure custom docker images for AWS lambda functions (ubuntu with bash/shell & nodejs)]]></title>
            <link>https://medium.com/@claude.ducharme/how-to-configure-custom-docker-images-for-aws-lambda-functions-ubuntu-with-bash-shell-nodejs-e1d6592c88b3?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1d6592c88b3</guid>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[aws-api-gateway]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 12:53:22 GMT</pubDate>
            <atom:updated>2023-08-28T12:53:22.621Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How to configure custom docker images for AWS lambda functions (ubuntu to run bash/shell scripts on linux and node nodejs)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iuwIgqyr4j19fW8kOdtJxw.png" /></figure><p>AWS provides a set of lambda runtimes and steps to implement your own. Nevertheless, setting up custom images to execute bash/shell scripts and very recent versions of nodejs via API Gateway v2 using json payloads proved to be more challenging than expected.</p><h3><strong>Whys and wherefores</strong></h3><p>There are multiple reasons behind building a <a href="https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html#runtimes-custom-build">custom runtime</a> for lambda</p><ul><li>Flexibility #1: The number of <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html">supported runtimes</a> by AWS is limited to a few languages and versions. You might want to use a newer version not yet supported or an older version not supported anymore.</li><li>Flexibility #2. Many frameworks, packages, libraries, tools, commands, or instructions are not available in the list of supported runtime languages. It could be much faster to develop a function with a runtime being very powerful at handling certain types of operations compared to other programming or scripting languages. For example, kubectl program to send commands to a kubernetes cluster has no equivalent in nodejs or python.</li><li>Simplicity. On last point, even when the binary program like kubectl is copied and called from a supported runtime, running the command can become much more complicated than anticipated when dealing with piping, loading profile, redirecting stdin/stdout/stderr, encoding/decoding data of special characters, managing async callbacks, handling output and errors</li><li>Familiarity. DevOps are typically more comfortable with bash/shell scripting compared to traditional programming languages.</li><li>Cheaper. Lambdas are billed for the duration they are running. Provisioning an EC2 instance with either commands to configure it and/or baked AMI to set up in advance might incur more costs. Also, when the project does not have long running instances, it would be overkill to think about RunCommand or kubernetes jobs for executing work that lasts only a few seconds/minutes.</li></ul><p>Side notes on Flexibility #1. It takes time for AWS to make newly released versions of nodejs to lambda runtimes and other frameworks. Here are my observations</p><ul><li>For major version: Around 7 months. The last upgrade was for nodejs 18.x which was announced on <a href="https://aws.amazon.com/blogs/compute/node-js-18-x-runtime-now-available-in-aws-lambda/">November 18, 2022</a> although nodejs 18 was released 7 months earlier on <a href="https://nodejs.org/en/blog/release/v18.0.0">April 19, 2022</a>. Current is 20.x (released on <a href="https://nodejs.org/en/blog/announcements/v20-release-announce">April 18, 2023</a>) but there is no nodejs20.x available for lambda yet.</li><li>For minor version: More than 6 weeks. Current LTS is 18.17.1 (released on <a href="https://nodejs.org/en/blog/release/v18.17.1">August 9, 2023</a>). AWS lambda created with nodejs18.x are now using version 18.16.1 (released on <a href="https://nodejs.org/en/blog/release/v18.16.1">June 20, 2023</a>).</li><li>New version might not be supported right away in AWS SAM, CloudFormation, and AWS CDK which could delay the adoption when your project CI/CD pipelines are leveraging by Infra as Code.</li></ul><h3><strong>Proof of Concept</strong></h3><p>Lambdas are awesome for running short-lived non real-time background work. They can be called ad-hoc via the API Gateway, directly with the Function URL, from the lambda API, and the Console. They can also be scheduled using the CloudWatch Events Rules or be triggered by a multitude of events. I created a minimal PoC with Custom Images using API Gateway v2 (<a href="https://github.com/Ducharme/awsCustomLambdas">awsCustomLambdas</a>). It includes 5 functions based on 3 docker images (public.ecr.aws/lambda/nodejs:18, node:20-bookworm, and ubuntu:22.10) and 2 Zip referencing different images (provided.al2 and nodejs18.x).</p><ul><li>bonjour: public.ecr.aws/lambda/nodejs:18 (image)</li><li>salut: node:20-bookworm-slim (image)</li><li>hello: ubuntu:22.10 (image)</li><li>coucou: provided.al2 (runtime &amp; zipped code)</li><li>howdy: nodejs18.x (runtime &amp; zipped code)</li></ul><h4>Straightforward architecture</h4><p>The user makes a POST request to the API Gateway which calls the lambda function. The lamba pulls the image from the ECR repository and is loaded with a dedicated IAM role to execute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*Fd1hk80BmtKnFl1CVn4CYA.png" /></figure><p>Here are the steps involved in setting up the above design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmCeAbDbXx4qOv2oQIzDHg.png" /></figure><p>There are 4 scripts to set up and 3 to test. It should take less than 5 minutes to have your first lambda working</p><ul><li><em>build_image.sh</em> takes 12 to 260 seconds (see details of timing below)</li><li><em>create_role.sh</em> takes 10s</li><li><em>create_lambda.sh</em> takes 6s</li><li><em>create_api.sh</em> takes 22s</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*oxIKpUH5evz7EXz8cHLhBg.png" /><figcaption>Timing and image details related to script build_image.sh</figcaption></figure><p>Only nodejs18.x runtime with zipped code has fast enough initialization and execution durations to be considered for customer facing apps. The high latency of others makes them only good candidates for handling non-interactive backend workloads — i.e. scheduled or event processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*uybnpyIr8C7G8fTI6U5s-w.png" /><figcaption>Durations and memory usage for each lambda types</figcaption></figure><h4><strong>Challenges encountered</strong></h4><ol><li>Defining which script should be configured as entrypoint and command in the Dockerfile. Lambda would report Runtime.InvalidEntrypoint, Runtime.ImportModuleError, or simply exit with code 2. Playing with Image configuration overrides in the Console saved me a lot of time debugging.</li><li>API Gateway v2 is leaner and more difficult to debug than its predecessor. The biggest problem I spent effort on was getting status code 500 when calling with curl even though the lamba returned 200 in the Console. The root cause was the encoding of the body response payload in json. Documentation does not provide examples more complex than returning a simple string. When dealing with a json response from a bash/shell script, a json as string will work when the lambda is invoked directly by fail when the call is coming from API Gateway v2. The quotes need to be escaped in the json or the payload encoded in Base64. An example of valid payload would be ‘{ “isBase64Encoded”: false, “statusCode”: 200, “body”: “{ \”allo\”: \”hehe\” }” }’. API Gateway v2 has only access logs which makes debugging a nightmare. Note the response will be encoded in the reply even if the response was not encoded. More on the topic in my re:Post question <a href="https://repost.aws/questions/QUXAb79a7gRtqDw2_T8aoRjQ/aws-custom-lambda-returns-status-ok-but-api-gateway-v2-fails-with-500">AWS Custom Lambda returns status OK but API Gateway v2 fails with 500</a> (see the accepted answer).</li><li>Finding values for the lambda Triggers and Permissions when set up as AWS_PROXY was not obvious.</li><li>Configuring the Function URL via Console is much simpler than figuring the related AWS CLI commands. The Console is able to put a StringEquals condition for lambda:FunctionUrlAuthType to the policy statement but this option is not available in the CLI command aws lambda add-permissions.</li><li>Vague documentation on AWS CLI commands like “<a href="https://awscli.amazonaws.com/v2/documentation/api/latest/reference/apigatewayv2/create-stage.html">aws apigatewayv2 create-stage</a>” with option “ — access-log-settings” which has a “Format=string” field without sample payload. When done with the Console it is simple to configure json format; unfortunately the same cannot be said about the CLI.</li></ol><h3><strong>Final thoughts</strong></h3><p>The PoC completion unleashed more power to lambda functions thanks to various custom runtimes. AWS has extensive documentation on their products and features and yet I wish there would be more examples of different commands and payloads. The API Gateway v2 has great potential despite needing a better user experience with debugging capabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1d6592c88b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring & Logging quickstart on AWS EKS]]></title>
            <link>https://medium.com/@claude.ducharme/monitoring-logging-quickstart-on-aws-eks-c0a6b13db1b2?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0a6b13db1b2</guid>
            <category><![CDATA[opensearch]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[fluentbit]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Thu, 12 Jan 2023 23:11:20 GMT</pubDate>
            <atom:updated>2023-01-12T23:11:20.047Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KsbJeMGj64k6nMU2OG7DHA.png" /></figure><p>This article aims to relieve some pain around setting up monitoring and logging from scratch with <a href="https://github.com/kubernetes-sigs/metrics-server">Kubernetes Metrics Server</a>, <a href="https://prometheus.io/">Prometheus Community</a>, <a href="https://opensearch.org/">OpenSearch</a> on Docker (not the AWS Managed Service), <a href="https://fluentbit.io/">fluentbit</a>, and <a href="https://grafana.com/">Grafana</a> (all being open-source software). If you already have a k8s cluster configured meeting basic requirements (IAM IODC provide &amp; EBS CSI driver) and common tools installed on your Linux (ubuntu) development desktop, it will take less than 2 minutes to have everything up and running by following instructions from this <a href="https://github.com/Ducharme/aws-eks-monitoring-logging-quickstart/README.md">README.md</a></p><h4>Background</h4><p>Few weeks ago I was looking at setting up an open source stack for monitoring and logging in my project <a href="https://github.com/Ducharme/LaFleet-Project">LaFleet</a>. The path was not a straight one, for example I explored <a href="https://www.eksworkshop.com/intermediate/230_logging/">eksworkshop logging</a> and faced many issues, so I’ve spent many hours figuring out what would be a simple but effective monitoring and logging architecture. Along the way there was a lot of debugging, unfortunately many articles describe a very precise component which usually requires much more research to get a comprehensive working solution. Disclaimer: this quickstart is for exploratory work only (not intended for production usage).</p><h3>Decisions made</h3><h4>ELK stack</h4><p>The new dual license of Elastic complicates things. Indeed, Server Side Public License v1 brings more restrictions than MIT or Apache License. Installing the stack with helm charts from Elastic (<a href="https://github.com/elastic/helm-charts">github repository</a>) was not complicated. Unfortunately, version 8.5.1 cannot run a single node cluster because of a <a href="https://github.com/elastic/helm-charts/issues/312">bug</a> so I forced the version to 7.17.3. Default CPU and memory requests/limits also had to be decreased in order to be scheduled on a single node. Since helm-charts repository is only maintained by the community and before getting deleted in May, using Elasticsearch (ECK) Operator is recommended. Using their <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html">quickstart</a> was challenging so I only did a small demo without Prometheus, Logstash and Grafana so far.</p><h4>OpenSearch stack</h4><p>Why not using the <a href="https://aws.amazon.com/opensearch-service/">AWS OpenSearch Service</a>? In my case the setup took more than 20 minutes and I had not a clear visibility over what needed to be done to ingest the logs I wanted and how much space they would take. Having full control over the configuration and deployment was helpful to understand how the engine was working and tweak it for my usage. Also, no version 2.x was available at that time since AWS lags integrating new releases. To illustrate the concern, according to the <a href="https://opensearch.org/docs/latest/version-history/">version history</a> (<a href="https://opensearch.org/lines/2x.html">2.x</a>), OpenSearch 2.0.0 was released on May 26th, 2022 and 2.3.0 was released on September 14th, 2022 but AWS first 2.x version was 2.3 launched almost 6 months later on November 15, 2022 (<a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/release-notes.html">notes</a> and <a href="https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-opensearch-service-supports-opensearch-version-2-3/">post</a>).</p><h3>Problems solved by the quickstart</h3><p>The quickstart tackles these issues</p><ul><li>Prometheus exporter services would not always start so its priorityClassName was increased;</li><li>Having 3 pods of OpenSearch master in a cluster, one in each availability zone, also caused trouble, so to simplify the deployment the configuration was set to singleNode (can easily be reverted with a bool);</li><li>OpenSearch demo keys and certificates were setup for easy usage by fluentbit to stream logs;</li><li>Grafana persistence was enabled with 1Gi of disk so dashboards can be saved and reloaded when pod restarts;</li><li>Grafana data source for prometheus is pre-configured and 6 dashboards are imported via their id from <a href="https://grafana.com/grafana/dashboards/">https://grafana.com/grafana/dashboards/</a>;</li><li>fluentbit’s input excludes verbose logs from non-app components like prometheus which makes it much easier to find the hello dummy logs;</li><li>fluentbit Replace_Dots is set to On to avoid ingestion errors due to labels or annotations clashes in OpenSearch;</li><li>Script eks/start_proxy.sh configures the port forwarding and displays the username/password to login Grafana and OpenSearch;</li><li>As using Grafana and OpenSearch for the first time can be daunting, screenshots with supporting steps are provided below.</li></ul><h3>Guide to using web UIs</h3><h4>Prometheus</h4><p>Server</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*EcCh_g0k2YHFPzmoazNjmw.png" /></figure><p>PushGateway</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*OMB_mmO0MPUSV5Y-F3kdMg.png" /></figure><p>AlertManager</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*WB0IF3EeDKIDwGu2YS2XBA.png" /></figure><h4>Grafana</h4><p>URL and credentials are displayed in the output of script eks/start_proxy.sh</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gwzY9YjMCJY_l-AHoIPB2w.png" /></figure><p>Data source prometheus was configured and six dashboards were imported. You can see the list of dashboards by clicking the <strong>Dashboards icon</strong> (4 squares) on the left bar then <strong>Browse</strong> link</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_YBFzV564YPHAgKLu0uQ9g.png" /></figure><p>Click on one of the dashboard to load it, for example Kubernetes Cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8utR5X2pCiBsEJ4yYpJY9A.png" /></figure><p>Dashboard is displayed…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*so8Q0ICRNHk-uB63CCtVDw.png" /></figure><p>When leaving the page it is recommended to click <strong>Save dashboard</strong> button so the <strong>Unsave changes</strong> popup does not show again next time</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rs7zJB546RnxVefhmNJyvQ.png" /></figure><h4>OpenSearch</h4><p>URL and credentials are displayed in the output of script eks/start_proxy.sh</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JsUqAUPxcNOUAb_92gLEbg.png" /></figure><p>Once logged in, click the hamburger on the upper-left corner to display the menu panel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEk7GG8kzWcfr1drOJQDiw.png" /></figure><p>Click <strong>Stack Management</strong> link at the bottom of the menu panel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bF0ZJWhRqtMPavZK_owuTQ.png" /></figure><p>Under Stack Management, click <strong>Index Patterns</strong> link</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xgPKyZZfmZW00BQkbX2EaQ.png" /></figure><p>Click <strong>Create index pattern</strong> button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0G6CNfCKzfH1mqHL6eJQw.png" /></figure><p>If logs were properly ingested, <strong>fluent-bit</strong> will be listed under sources as one index pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGXIRaoiZXy3DPR7FcfEIg.png" /></figure><p>Copy/paste “<strong>fluent-bit</strong>” into the textbox Index pattern name then click <strong>Next step</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3Xvdd1pvodfzzKN_y48vw.png" /></figure><p>Select <strong>@timestamp</strong> in the Time field combo drop down and click <strong>Create index pattern</strong> button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ir-gZh4rGD-Qu-v9i1t2tA.png" /></figure><p>Next page lists every fields of the index. Click the hamburger on the upper-left corner to display the menu panel again</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aj9YlSyzK55447iaBElXTA.png" /></figure><p>Click <strong>Discover</strong>, the second item, on the menu panel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GpobpdL8AFke2zDilhujew.png" /></figure><p>Logs should be shown in a table with row you can expand</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D_T3TlUEGY5fAM8kmKF_CQ.png" /></figure><h3>What’s next?</h3><p>Official documentation and a myriads of articles describe how to</p><ul><li>Add custom metrics to Prometheus</li><li>Add data sources in Grafana</li><li>Create Grafana and OpenSearch dashboards (Kibana is the underlying component of OpenSearch Dashboard)</li><li>Create alerts in Grafana and OpenSearch</li><li>Create and use your own keys and certificates for OpenSearch</li><li>Use fluentd to stream logs (legacy compared to fluentbit but it has more input and output plugins available so could be useful)</li><li>Use Logstash to centralize, transform &amp; stash your logs</li><li>Configure components to leverage multiple Availability Zones to improve resiliency and performance</li><li>Deploy on GKS (GCP) or AKS (Azure)</li></ul><p>Let me know what are your interests in comments and I will try covering popular topics in follow up articles. Meanwhile, feel free to watch my github repository for enhancements <a href="https://github.com/Ducharme/aws-eks-monitoring-logging-quickstart">https://github.com/Ducharme/aws-eks-monitoring-logging-quickstart</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0a6b13db1b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Selecting a geo-representation]]></title>
            <link>https://medium.com/@claude.ducharme/selecting-a-geo-representation-81afeaf3bf01?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/81afeaf3bf01</guid>
            <category><![CDATA[uber]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Mon, 03 Oct 2022 08:33:00 GMT</pubDate>
            <atom:updated>2022-10-03T08:33:00.204Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous post I shared information on <a href="https://medium.com/@claude.ducharme/building-a-geo-localisation-platform-db4841aac519">building a geo-localisation platform</a>. H3 was selected for the Proof of Concept without much justification. This article describes why this particular library was chosen for geospatial indexing.</p><h3>What are the options?</h3><p>Google S2 (<a href="https://s2geometry.io/">https://s2geometry.io/</a>) divides the globe into squares. Library was released in 2011 and was moved to github in 2015 . <a href="https://eng.uber.com/h3/">Uber H3</a> (<a href="https://h3geo.org/">Hexagonal hierarchical geospatial indexing system</a>) was uploaded to github in 2018. How do they compare? Unfortunately I quickly dropped S2 after a few hours as the documentation is nowhere close to what I found on H3 and the portability and maintenance are too limited (see some reported <a href="https://code.google.com/archive/p/s2-geometry-library/issues">issues</a> years back). On top of these, the math behind the algorithm are complicated and the world map Mercator projection has strong disformations. See by yourself with the following screenshots.</p><h4>Google S2 vs Uber H3</h4><p>Google S2 — Drawing rectangles on a 2D map <a href="https://s2.sidewalklabs.com/">https://s2.sidewalklabs.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*MLUEgsXSH6VBuGJldMVoeQ.png" /></figure><p>Google S2 — On a globe projection the lines are skewed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/0*20v0XcHiWBeuboLO.gif" /><figcaption>Picture credit: <a href="https://opensource.googleblog.com/2017/12/announcing-s2-library-geometry-on-sphere.html">https://opensource.googleblog.com/2017/12/announcing-s2-library-geometry-on-sphere.html</a></figcaption></figure><p>Uber H3 — Disformations at higher zoom level look very bad as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQI6Oa25A6CatOrsQSCsxA.png" /></figure><p>Uber H3 —On a globe projection the elongated hexagons look much better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KssFI5kEhpsRi2ekdnV6mA.png" /></figure><p>Feel free to check how polygons look at multiple zoom levels (0 to 15) in different part of the world at <a href="https://observablehq.com/@claude-ducharme/h3-map">https://observablehq.com/@claude-ducharme/h3-map</a>. Note how 12 pentagons (in red) are needed for every resolution to bridge gaps left by the usage of hexagons.</p><p>Uber also compared S2 to H3 <a href="https://h3geo.org/docs/comparisons/s2/">https://h3geo.org/docs/comparisons/s2/</a> and explained why they developed their own system <a href="https://www.uber.com/en-HK/blog/h3/">https://www.uber.com/en-HK/blog/h3/</a></p><h4>PoC implementation with H3</h4><p>Indexing in the PoC is done by leveraging polyfill on polygons drawn <a href="https://observablehq.com/@claude-ducharme/h3-sandbox-on-mapbox">https://observablehq.com/@claude-ducharme/h3-sandbox-on-mapbox</a></p><p>Shapes are made from polygons of multiple resolutions. The precision given to the shape highly depends on the number of resolutions used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Co0C1ZGan5TnSvtzMPoshA.png" /><figcaption>Drawing a shape and calling polyfill to get all polygons fitting inside</figcaption></figure><p>The screenshot below illustrates the merging of adjacent polygons having the same resolution together. <a href="https://h3geo.org/docs/3.x/api/regions#h3settolinkedgeo--h3settomultipolygon">Function</a> to perform the merge is built-in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dFKkmjDehx9KMxha-NQe7A.png" /><figcaption>Polygons having the same resolution are merged first</figcaption></figure><p>Last step is merging multi-polygons from different resolutions to create a single polygon as shown below. <a href="https://github.com/Turfjs">Turf</a> libraries are call for this operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ar3PfmPwhX3gJktBPKFzrQ.png" /><figcaption>Polygons merged from different resolutions are merged last</figcaption></figure><h3>Links</h3><p>Project overview <a href="https://github.com/Ducharme/lafleet">https://github.com/Ducharme/lafleet</a></p><p>PoC for device aggregation <a href="https://poc.lafleet.net/">https://poc.lafleet.net/</a></p><p>PoC for displaying shapes <a href="https://shape.lafleet.net/">https://shape.lafleet.net/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81afeaf3bf01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 AWS SDK for JavaScript v3 examples using TypeScript]]></title>
            <link>https://medium.com/@claude.ducharme/3-aws-sdk-for-javascript-v3-examples-using-typescript-c1e1ab209ec6?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1e1ab209ec6</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[aws-sdk]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Mon, 01 Aug 2022 17:52:29 GMT</pubDate>
            <atom:updated>2022-08-01T18:04:24.643Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXJfFcmCM6A5ivea59Pu7A.png" /><figcaption>TypeScript code samples for <a href="http://twitter.com/aws">@aws</a>-sdk/client-s3 and <a href="http://twitter.com/aws">@aws</a>-sdk/client-sqs (AWS SDK for JavaScript v3)</figcaption></figure><p>I love strongly typed languages as they leave less room for surprises at runtime. For my pet project <a href="https://github.com/Ducharme/infraAsCodeCdk">LaFleet</a> I decided to adopt TypeScript to get more familiar with its capabilities. Mastering JS control flow and its types can be daunting at times when introduced to async/await and Promises/Callbacks.</p><p>One of my recent struggles was with AWS SDK for JavaScript v3. AWS revamped interfaces in its 3rd version of the JavaScript client. It is often challenging to find good TypeScript examples using correct types instead of “any”. This article is my attempt to remediate the situation, a far better option than defaulting to v2 due to the lack of documentation/samples. For the code gists below, simply replace process.env.* variable values with yours to get them working. Gists were tested using version 3.131.0 of the @aws-sdk</p><h3>First example: S3 GetObjectCommand</h3><p>Most examples online only handle the type “NodeJS.ReadableStream” hidden under “any” but in fact there are three potential underlying types.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63474167f031ff3244244692702a2a78/href">https://medium.com/media/63474167f031ff3244244692702a2a78/href</a></iframe><h3>Second example: S3 SelectObjectContentCommand</h3><p>Online samples are still very rare for S3Select. SQL expression is the simplest possible, you may use a more complex statement once the code is running.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/068e6cc6e10d8cc1822dd212df257224/href">https://medium.com/media/068e6cc6e10d8cc1822dd212df257224/href</a></iframe><h3>Third example: SQS ReceiveMessageCommand</h3><p>Setup for this sample is an event notification on S3:ObjectCreated via PUT or POST sending a message to a SQS queue. Code collects the bucketName and objectKey into an array and displays it to the console (you should replace that output with your own program logic).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e7f2c7a2701b84bfdae6c7061f8534a/href">https://medium.com/media/5e7f2c7a2701b84bfdae6c7061f8534a/href</a></iframe><h3>What’s next?</h3><p>Were the gists useful? Would you have them implemented differently? Which methods of the SDK do you find unclear or difficult to use in TypeScript?</p><h3>References</h3><p>Official documentation <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html">S3 client</a> and <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-sqs/index.html">SQS client</a> (AWSJavaScriptSDK v3)</p><p>AWSJavaScriptSDK v3 source code <a href="https://github.com/aws/aws-sdk-js-v3">https://github.com/aws/aws-sdk-js-v3</a></p><p>Packages on npm <a href="https://www.npmjs.com/package/@aws-sdk/client-s3">@aws-sdk/client-s3</a> and <a href="https://www.npmjs.com/package/@aws-sdk/client-sqs">@aws-sdk/client-sqs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1e1ab209ec6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My experience on Infra as Code with AWS CDK + Tips & Tricks]]></title>
            <link>https://medium.com/@claude.ducharme/my-experience-on-infra-as-code-with-aws-cdk-tips-tricks-c967fe357133?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c967fe357133</guid>
            <category><![CDATA[infra-as-code]]></category>
            <category><![CDATA[cdk]]></category>
            <category><![CDATA[aws-cdk]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Mon, 13 Jun 2022 19:04:45 GMT</pubDate>
            <atom:updated>2022-06-13T19:04:45.513Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezbXbTJsv79ZkivHUUMwBA.png" /><figcaption>AWS Cloud Development Kit (CDK v2)</figcaption></figure><p>AWS CDK is a game changer in the IaC on AWS. It offers a decent level of abstraction saving time to DevOps by leveraging constructs compared to writing CloudFormation templates or Terraform HCL. Code leveraging the CDK is succinct, TypeScript and most other languages provide static type checking, and IDE like vscode have auto-complete and in-line documentation.</p><p>I started to develop with CDK a few weeks ago. My IoT pet project IaC first started with AWS CLI shell (<a href="https://github.com/Ducharme/infraAsCodeShell">github repo</a>) as it gives great flexibility in creating resources but it was tedious to maintain (adding new resources, updating existing ones, and deleting them). This is where CDK comes handy by quickly updating existing resources or destroying them, see AWS CDK (<a href="https://github.com/Ducharme/infraAsCodeCdk">github repo</a>).</p><h3>Challenges</h3><h4><strong>Finding the proper construct(s)</strong></h4><p>Discovering them is not always intuitive. You might start to use one which is old/deprecated before you stumble on code samples using a newer construct. An example for me was <a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudfront.CloudFrontWebDistribution.html">CloudFrontWebDistribution</a> vs <a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudfront.Distribution.html">Distribution</a>. Fortunately I found <a href="https://docs.aws.amazon.com/cdk/api/v1/docs/aws-cloudfront-readme.html#migrating-from-the-original-cloudfrontwebdistribution-to-the-newer-distribution-construct">Migrating from the original CloudFrontWebDistribution to the newer Distribution construct</a> to guide me through the process.</p><p>Other times you will realize constructs are missing in the CDK. AWS likely first focuses on CLI, then SDK, and finally CDK which means you might not find what you are looking for when you create resources with the latest framework(s). For example <a href="https://docs.aws.amazon.com/cli/latest/reference/iot/create-keys-and-certificate.html">aws iot create-keys-and-certificate</a> does not exists in the CDK so in this situation your options are</p><ul><li>Make calls with the CLI or SDK to generate output then upload values where you need them (meaning the DevOps running the stack will have all certificates/secrets on his computer)</li><li>Use <a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources.AwsCustomResource.html">AwsCustomResource</a> with AwsSdkCall. Finding the proper field to retrieve or get the payload out of a json output could be troublesome. The debugger does not help since types and methods are defined as strings. Invocations might hang then timeout if you make mistakes or will throw an exception if output is missing or too long. For example, accessing ‘Contents.0.Key’ when listing files from an empty S3 bucket will throw an error as the entry is missing but will succeed when there is at least one file present. More pain points are covered in this <a href="https://mechanicalrock.github.io/2021/12/20/cdk-cr.html">blog post</a>.</li><li>Create your own AwsFunction/NodejsFunction. Functions are difficult to develop. Changing the code won’t get the function re-deployed if it was previously successfully deployed. If you want to perform a change, I recommend to comment the block of code related, deploy, uncomment the code, do the change then deploy again. This is tedious!</li></ul><p>Note: Beware of search results on Google as most links point to CDK v1 with package “@aws-cdk”. A mention “<em>You are not viewing the latest version. </em><a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources.AwsCustomResource.html"><em>Click here to view the v2 documentation</em></a><em>.”</em> is present in yellow on top of the page. For CDK v2 the package is “aws-cdk-lib”. Most of the time you can simply add the “-lib” to convert from v1 to v2.</p><h4><strong>Naming convention and structure</strong></h4><p>There is no clear guidance on how to define a stack and name them. In my case I prefixed them with the project name and put them all at the same level after the app creation. The first stack is for setting up the core infrastructure, then there is one stack per microservice x5, and finally one stack to finalize the infrastructure configuration based on outputs from previous stacks.</p><h4><strong>Roles and Policies</strong></h4><p>Roles are mostly handled by the CDK. Policies are created inline in IAM roles. Often default policies will be enough to run the stack but sometimes failure will indicate you have to create a PolicyStatement then call addToRolePolicy or addToPolicy or addStatements or addToResourcePolicy to apply. As there are multiple ways to grant new statements, it is often not obvious how to assign them. You will end up searching code samples to copy/paste.</p><p>In my project I needed 13 roles</p><ul><li>1x CodeBuild per microservice (x5)</li><li>1x CodePipeline per microservice (x5)</li><li>3x for IoT: 2x for logging and 1x for a Rule</li></ul><p>CDK created 32 roles for doing the same work</p><ul><li>1x CodeBuild per microservice (x5)</li><li>1x CodePipeline per microservice + 3x roles each to be assumed (4x5)</li><li>5x for IoT: 1x for logging + 1x for a Rule, 1x to describe IoT endpoint, and 2x for one NodejsFunction</li><li>1x S3AutoDeleteObjects for deleting objects when destroying a S3 bucket otherwise the stack fails if files are present</li><li>1x LogRetention for deleting logs from CloudWatch after a specified retention period</li></ul><h4><strong>Destroying a stack</strong></h4><p>The operation might not delete all its resources. Some will remain like S3 buckets &amp; ECR repos unless explicitly mentioned in the code. Having a strong naming convention as suggested above will help you spot remaining resources after destroying a stack. Do not use suffixes as the CDK often truncates the name of the resources to add random characters at the end.</p><p>Some constructs do not offer the possibility to remove dependencies so destroying the stack will fail. For example, I had to use the CLI to list and delete all images from ECR repos before calling destroy on the stack. Other times the destroy will hang until it times out unexpectedly. You will have to go to the AWS console in CloudFormation / Events tab to find where the error comes from and manually delete resources before calling destroy again.</p><h4>Unexpected <strong>behaviors</strong></h4><p>I spent many hours debugging random issues. Here are my three biggest…</p><p>When destroying stacks I stumbled upon “<em>The bucket you tried to delete is not empty. You must delete all versions in the bucket.</em>” even though S3 bucket had “autoDeleteObjects: true”. It turns out that if you have “versioned: true” also set, the auto delete will not remove all Versions and DeleteMarkers. Similar story with ECR “<em>Resource handler returned message: “The repository with name ‘abc…’ in registry with id ‘123…’ cannot be deleted because it still contains images</em>”. Extra CLI scripts were written to cater for these cases.</p><p>Another one was CustomS3AutoDeleteObjects lambda encountering Access Denied when destroying a bucket. Message was “<em>MyStack: destroy failed Error: The stack named MyStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [S3XXX7DA36247, S3XXAutoDeleteObjectsCustomResource79E8]. ): Received response status [FAILED] from custom resource. Message returned: AccessDenied: Access Denied</em>”. For granting Origin Access Identity read access for CloudFront to an imported S3 bucket I used bucket.policy.document.addStatements(ps) but it had not effect (see issue <a href="https://github.com/aws/aws-cdk/issues/6548">S3 Bucket Policy Changes Not Recognized As A Change on CDK Deploy #6548</a>). In other words, when you add a policy statement to an existing bucket, the method will not do anything (<a href="https://github.com/aws/aws-cdk/tree/v1-main/packages/@aws-cdk/aws-s3#permissions">link</a>). Also checking an imported bucket policy will return an empty document even though it exists. As a workaround, I used the CDK to read the current policy and created a BucketPolicy with all statements inside.</p><p>When having multiple stacks in the same app, note that all the code from the declared stacks will execute even if not called SecondStack by <em>synth </em>and <em>deploy</em>. For example, pretend you have FirstStack and SecondStack declared in <em>bin/myapp.ts</em>, if you have a bug in SecondStack, FirstStack might fail to deploy even when deploy is not called on SecondStack.</p><p>The number of CloudWatch Log Groups created is astronomical. As they are not deleted by default and their name is dynamically generated with random alpha-numeric characters, try to clean the groups once in a while to avoid having too many CustomS3AutoDeleteObjects and LogRetention in the list.</p><p>Get used to seeing “<em>MyStack failed: Error: The stack named MyStack failed creation, it may need to be manually deleted from the AWS console</em>” as you will encounter it often. Error messages are not always helping much in quickly finding the real issue. The maturity of the CDK is at times questionable.</p><h4><strong>Deployment takes time</strong></h4><p>Time to deploy my 7 stacks with CDK is 20 minutes on average.<br>This is 2x longer than creating the same with AWS CLI.</p><h3><strong>Tips &amp; Tricks</strong></h3><ul><li>Keep in mind the code to create/update resources won’t execute anymore once a resource is successfully created/updated.</li><li>In relation to the point above, do not rely on the returned value of resources being created/updated; the first time the stack runs it will be fine, but the second time the value will be undefined. Relying on the CLI or SDK to get current values is the best way to avoid exceptions.</li><li>Remember that reading a value from a resource will yield a token instead of the real variable value. In many cases CLI and SDK are still needed.</li><li>Deploy your stacks often to avoid surprises</li><li>When you encounter a bug in your code during development, delete related resources then re-run the code with your fix. Otherwise, patching resources half created/deleted might lead you to a wild goose chase</li><li>Destroy your stacks just for the sake of studying the behaviors of deleting resources and re-creating them from scratch. Dependencies not defined explicitly can break the process or you might find issues when redeploying</li><li>Review automatically created roles and their inline policies to make sure they are not too permissive. Explicitly create roles with appropriate statements where default access rights are not secure enough</li><li>When you have resources shared across stacks it is often easier to import them by name or ARN instead of reusing them directly in other stacks. CDK dependency management might otherwise fail to synthetize.</li><li>Have a simple naming convention and structure</li><li>CI/CD might fail due to race conditions. For my microservices most stacks are doing the same: create an ECR (Elastic Container Registry), a CodeCommit repository, a CodeBuild project, and a CodePipeline. A build will automatically start but may fail randomly because not all underlying resources are ready. Relaunching the build a minute later will succeed.</li><li>Ask questions in aws-cdk channel of <a href="https://cdk-dev.slack.com/">cdk-dev</a> slack group if you get stuck</li></ul><h3><strong>What is your experience?</strong></h3><p>Have you faced similar challenges when adopting the CDK? Do you have other Tips &amp; Tricks to share with others? Has releasing multi-environment changes been easy with the framework? Can you deploy infra changes without downtime? Let me know the best practices you’ve learnt over time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c967fe357133" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a geo-localisation platform]]></title>
            <link>https://medium.com/@claude.ducharme/building-a-geo-localisation-platform-db4841aac519?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/db4841aac519</guid>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Mon, 18 Apr 2022 16:01:36 GMT</pubDate>
            <atom:updated>2022-04-18T16:01:36.256Z</atom:updated>
            <content:encoded><![CDATA[<p>Five years ago, my friend and co-founder Raphael Cohen and I, were on a journey to launch the first dockless bike-sharing platform in Hong Kong. In two months we kickstarted by pooling our money together, agreed on the brand and logo, selected a bike matching our criteria, bought a whitelabeled backend and apps, and employed our first staff while closing our seed round. In mid April we had 2,000 bikes at our official kick-off press conference. Three months later we closed our Series A to expand to Europe at the end of summer. Execution speed was critical to get the first mover advantage in targeted countries. With rapid growth in different markets came multiple requirements related to the location of the bikes, parking spots, no go zones, and city limits. These were all implemented using GPS coordinates directly as we did not have time to revamp the geo-localisation system. The screenshots below were screens within our app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*WADwdHrjMjGrszSJjPIy2g.png" /><figcaption>Available bikes to rent (green bike icons), free bikes to ride (purple gift icons), parking spots (blue P icons), and free ride when locking the bike in the area surrounding the flag (red circle)</figcaption></figure><p>Some of the drawbacks we experienced with the system implementation</p><ul><li>Big network payload in densely crowded bike areas. It made the mobile apps sluggish to load the data and more expensive too.</li><li>Clustering of bikes nearby was performed on the mobile apps so on top of using more data it was using CPU on the phone to perform aggregations.</li><li>There were no clustering capabilities on the backend console. After deploying a few thousands bikes in a city, the map would be clogged by bike icons at a point it was not possible to see the city anymore unless we zoom in extensively.</li><li>Parking spots were either loaded from the backend or embedded in the mobile apps. This increased the loading time or was not dynamic enough.</li><li>We needed bigger and more expensive infrastructure as we scaled our user base and bike fleet. At peak we had 250,000 users and 34,000 bikes.</li><li>We had a short period of downtime when releasing new versions of the application server where we could lose some updates from users or bikes.</li><li>Database tables for GPS location history tables would often timeout due to their massive size. It was challenging to run analytics on them.</li></ul><p>We had many improvements in our backlog to tackle these but never went ahead with the much needed redesign. As I was studying material for AWS, GCP, and CKA certifications I also wondered how the system could have been better to a point I decided to start a pet project to assess different approaches as Proof of Concept. Note: For the rest of the article “devices” will be used instead of bikes as the platform can easily handle any type of IoT device (non GPS related devices would need small improvements, nothing major).</p><h3>Proof of Concept</h3><p>There are many options in regards to selecting a map projection with distinct pros and cons. Capabilities of <a href="https://eng.uber.com/h3/">Uber H3</a> were more appealing than others for the job (clustering of nearby devices at different resolutions, finding devices in a radius around a point, defining zones and limits). Furthermore, when using hexagons to represent zone or position, the inherent imprecision is acceptable. Below is a representation of H3 at 3 different resolutions on the globe. Undoubtedly the hexagonal shape can be treated like a circle for radius.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*ZrmtHa3z-u-Uj6J5.png" /><figcaption>H3 representation at three different consecutive resolutions</figcaption></figure><p><strong>Demo Website</strong></p><p>Few devices have been placed statically on <a href="https://poc.lafleet.net/index.html">https://poc.lafleet.net/index.html</a> to demonstrate the display of aggregations returned by a lambda exposed by API Gateway. You can zoom in/out and drag the map to see how clustering works over 6 big cities. Montreal has 100 devices, Paris 50, Buenos Aires 45, Sydney 35, Tokyo 25, and Cape Town 15.</p><p><strong>First iteration</strong></p><p>AWS Managed Services were used as much as possible in this PoC. Code was mostly in TypeScript and JavaScript.</p><ul><li>Mock based on <a href="https://github.com/aws/aws-iot-device-sdk-js-v2">IoT Device SDK for JavaScript v2</a> running on AWS Lambda</li><li>IoT Core as entry point for devices sending messages to one SQS Queue</li><li>Node.js server in auto-scaling group (ASG) consuming messages from SQS</li><li>Server app code also saving results in RDS PostgreSQL instance and S3</li><li>API Gateway with AWS Lambda function to retrieve data</li><li>Single Page Application (SPA) on React hosted on S3</li><li>Performance analysis in Python was running on AWS Lambda</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*53Lunbpovl1Yo8XkQX5FyA.png" /><figcaption>Architecture of first iteration</figcaption></figure><p>What was observed when setting the frequency up to 800 updates/second</p><ul><li>Batching reads on SQS had a significant boost in performance</li><li>Database on t3.micro was getting very slow with half a million records</li><li>Saving files to S3 had to be done asynchronously to avoid slowing down</li><li>Setting up servers with scripts in Launch configuration or by baking AMI was time demanding. Indeed, a small change requires duplicating the Launch configuration, provisioning a new server, waiting for it to spin up, connecting via ssh, reading logs to ensure the change was effective, and sending test queries to ensure the expected behavior from the server).</li></ul><p><strong>Second iteration</strong></p><ul><li>Replacing ASG by Elastic Container Service (ECS) for the consumer server. ASG was doing better than ECS with similar instance type provisioned so ASG was used for all upcoming iterations unless specified otherwise.</li></ul><p><strong>Third iteration</strong></p><ul><li>Sending messages from IoT Core to a downstream HTTPS endpoint to be directly saved. It was slightly faster and processing times were statistically more predictable than using SQS queue yet a major caveat was the lack of buffering capabilities. Servers should in theory always be up to consume messages but in reality bugs or patching could cause mini downtime.</li></ul><p><strong>Fourth iteration</strong></p><ul><li>Sending messages to Managed Streaming for Apache Kafka (MSK). Idea was dropped due the inability to get IoT Core sending messages to MSK.</li></ul><p><strong>Fifth iteration</strong></p><ul><li>Storing data in DynamoDBv2 instead of RDS. Performance was good. Alas scaling proved to be quite expensive to meet write throughput (around 600 usd/month). Query capabilities to filter/aggregate data were limited.</li></ul><p><strong>Sixth iteration</strong></p><ul><li>ElastiCache for Redis was doing well for storing the data. Similar to DynamoDB, a daunting issue was about retrieving data. To improve query performance a lua script was added to the Redis server allowing reverse lookup. The performance drastically dropped with the script though.</li></ul><p><strong>Seventh iteration</strong></p><p>Using NoSQL to allow scaling of applications is a well-known pattern. Although <a href="https://redis.io/docs/stack/search/">RediSearch</a> is not an AWS Managed Service and the plugin cannot be installed on ElastiCache for Redis I decided to bench it. Performance was great for both reads and writes on top of having much needed extra features</p><ul><li>Primary and secondary indexing</li><li>Full-text indexing of multiple fields</li><li>Multi-fields queries</li><li>Aggregation</li></ul><p>In addition I was also getting <a href="https://redis.io/">Redis</a> basic features</p><ul><li>In-memory data structures like hashes (to store the latest information of the device) and streams (to store history of GPS coordinates over time).</li><li>Geo-filtering using Redis own geo commands</li><li>Persistence of all writes to permanent storage</li><li>Clustering for horizontal scalability with hash-based sharding</li><li>High availability with automatic failover</li><li>Publish/Subscribe messaging paradigm</li><li>And many more…</li></ul><p><strong>Eighth / Last iteration</strong></p><p>It was much bigger in size by leveraging Kubernetes orchestration</p><ul><li>Mock based on <a href="https://github.com/aws/aws-iot-device-sdk-js-v2">IoT Device SDK for JavaScript v2</a> running on EKS</li><li>IoT Core as entry point for devices sending messages to one SQS Queue</li><li>Node.js server consuming messages from SQS as deployment on EKS</li><li>Server app code also computing results from RedisSearch on EKS</li><li>New server app to query data as a service on EKS</li><li>Single Page Application (SPA) on React hosted on S3</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i202Byzf9VB1lcAIUD67Bw.png" /><figcaption>Architecture of the Proof of Concept</figcaption></figure><p>To speed up cadence simple CI/CD pipelines were put in place along Infra-as-Code via shell (POSIX) on ubuntu (IaC with CDK is under development)</p><ul><li>Source code in CodeCommit with CodeBuild and CodePipeline for CI</li><li>Docker images stored in Elastic Container Registry (ECR)</li><li>Manual deployment is done via kubectl when needed</li></ul><p><strong>Current features</strong></p><ul><li>Sidebar showing OpenStreetMap zoom level, H3 resolution, and GPS coordinates of the Center (CTR) / North-West (N-W) upper left corner / South-East (S-E) bottom right corner. Zoom value is between 0 and 22 and H3 resolution between 0 and 15.</li><li>Drawing of hexagons has a layer with count of devices below a marker. Number of hexagons is typically between 25 and 200 per screen.</li><li>Zoom will keep the layer in place until the next H3 resolution is reached.</li><li>Similarly on map resize or when the map is dragged to re-center the layer will refresh to adapt to the new view.</li><li>Basic security: CloudFront with Shield redirecting HTTP to HTTPS TLS 1.2, IAM roles and policy, NAT Gateway on the EKS cluster, and more.</li><li>Device payload include GPS coordinates (longitude, latitude, altitude), all 15 H3 resolution indexes, timestamps (device, IoT Core, SQS consumer, and approximate saved time to Redis), sequence number, plus extra fields (battery level and firmware version for example).</li><li>Supports 2,000 updates/seconds with 3 relatively small pods only.</li><li>Deployment with a few simple steps in your AWS account with shell scripts. Whole process should take less than 45 minutes in total.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*ibb4G44d_V7QtuJpLuhEsg.png" /><figcaption>At H3 resolution 0 there are 10 devices near New-York</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKEJQyGSbowHoHdqru5INQ.png" /><figcaption>Zooming in, at H3 resolution the 10 devices are showing over Montreal in 4 aggregations</figcaption></figure><h3>Performance benchmark</h3><p>The mock device publishes a message to IoT Core which processes according to defined actions, most important one being sending data to a SQS queue. Pods are listening to messages on the queue, calculate H3 indices, and save the payload in both a Hash and a Stream on RediSearch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*UcyyL8J-ayOwPhi8D_AGqw.png" /><figcaption>Flow of data for published messages</figcaption></figure><p><strong>Bare minimum — Single device publishing slowly</strong></p><p>One device was sending 1 update/second for almost 2 minutes</p><ul><li>Average time between the device publishing a location and the save into RedisSearch is 111 ms. The fastest was 72 ms and longest 204 ms.</li><li>IoT Core fastest time to ingest was 9 ms and max 102 ms.</li><li>Worker SQS Consumer fastest time was 60 ms and max 186 ms.</li><li>RedisSearch fastest save was less than 1 ms and longest was 8 ms.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*QsbbI_IwKw9j8sE4dQjpKA.png" /></figure><p><strong>Small load — Many devices at max publish rate per pod</strong></p><p>20 devices were sending 100 updates/second each for almost 2 minutes. Both node group instance types used have 2 vCPU and 4 GiB of memory.</p><ul><li>1x c6i.large for CPU demanding services (10x SQS consumer and 1x RedisSearch).</li><li>2x t3.medium for normal services (1x analytics, 2x haproxy-controller, and kube-system related services (aws-node, coredns, kube-proxy, metrics server)) plus 20x mock devices generating updates every 10 milliseconds. The 3x query services were deleted to let room for the mocks devices (max number of pods for this node type is limited to 17).</li><li>Results are trimmed in time so updates are only considered when the 20 mock devices are up simultaneously (for all devices the latest start time and earliest stop time were used to calculate throughput).</li><li>Average time between the device publishing a location and the save into RedisSearch is 95 ms. The fastest was 43 ms and longest 766 ms.</li><li>IoT Core fastest time to ingest was 5 ms and max 301 ms.</li><li>Worker SQS Consumer fastest time was 15 ms and max 751 ms.</li><li>RedisSearch fastest save was less than 1 ms and longest was 288 ms.</li><li>CPU might have been the limiting factor for the high latency on some records. Looking at p99 the number of outliers is very small. Further code optimizations could reduce the count and the latency.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*c6x-2_1oipxIcXpD3UHUPg.png" /></figure><p><strong>Analysis</strong></p><p>It is worth noting that 2000 updates/second, when applied to bikes with GPS like we had at GoBee.bike for example, represents approximately the load of over 2,000,000 devices assuming 25% of the fleet is being used by customers. When parked, a bike was reporting its coordinates every 2 hours and when riding the frequency was every 5 minutes. The lock also sent its position when locking and unlocking. For more accurate data when a customer was riding, his/her mobile phone was uploading GPS coordinates every 10 seconds.</p><p>Billing wise, with a fleet of 34,000 devices and an average lower than 35 updates/second, IoT Core would cost around 150 USD per month for connectivity plus an additional 100 USD for messaging. SQS would be around 36 USD, EKS 73 USD, NAT Gateway 43 USD, possibly a single on-demand EC2 c6i.large including its EBS would be enough at 75 USD, and the Load Balancer 21 USD. Expenditure on other services is negligible (CloudFront, CloudWatch, CodeCommit, CodeBuild, CodePipeline, and ECR. Total would be approximately <a href="https://calculator.aws/#/estimate?id=a7c581a25c661bd2452d2aa93c2ea2fb801455f5">500 USD/month</a> if hosted in Singapore.</p><p>For a system like Uber, having vehicule sending updates every 4 seconds, means tracking around 10,000 vehicles with 3 small servers. Most of the updates are processed within 1 second so optimization to reduce latency would not bring much value. In the PoC the most performant part is the unmanaged usage of RediSearch. Having HTTP ingress for IoT and replacing SQS with Kafka would likely reduce the latency and costs when scaling. Nevertheless, leveraging managed services is saving time and brings a lot of convenience for a small fleet with a system having few updates per second.</p><h3>Playground</h3><p>Demo website is available at <a href="https://poc.lafleet.net/index.html">https://poc.lafleet.net/index.html</a> with hundreds of devices over six cities as mentioned earlier. Locations have been hardcoded in a lambda for simplicity instead of calling the kubernetes cluster.</p><p><strong>Source Code</strong></p><p>Source code under MIT license is available at (Infra as Code repo is the only one required to deploy all the others in your AWS account)</p><ul><li><a href="https://github.com/Ducharme/reactFrontend">https://github.com/Ducharme/reactFrontend</a></li><li><a href="https://github.com/Ducharme/mockIotGpsDeviceAwsSdkV2">https://github.com/Ducharme/mockIotGpsDeviceAwsSdkV2</a></li><li><a href="https://github.com/Ducharme/sqsConsumerToRedisearchJs">https://github.com/Ducharme/sqsConsumerToRedisearchJs</a></li><li><a href="https://github.com/Ducharme/redisearchQueryClient">https://github.com/Ducharme/redisearchQueryClient</a></li><li><a href="https://github.com/Ducharme/redisPerformanceAnalyticsPy">https://github.com/Ducharme/redisPerformanceAnalyticsPy</a></li><li><a href="https://github.com/Ducharme/infraAsCodeShell">https://github.com/Ducharme/infraAsCodeShell</a></li></ul><h3>Conclusion</h3><p>I enjoyed exploring the different alternatives to setup a geo-localisation system. Most flaws from the backend implementation of GoBee.bike have been addressed with the new design. Given the new flexibility a feature not yet available in the PoC could be deployed in a day or two. What has been time consuming in this project was writing and testing Infra as Code in shell (many roadblocks have been encountered with CDK so I decided to fallback to simple scripts). Hopefully the project will be useful to many as a starting point for IoT related backend.</p><p><strong>What’s next?</strong></p><p>The backlog is growing fast! What would you like to see next? Performance optimizations, New features, Active-Active Resiliency, Auto scaling, Non GPS devices, topic revamp, Multiple streams per device, Authentication, etc. Let me know by leaving comments :-)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db4841aac519" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Services Performance Benchmark (EC2, ECS, API Gateway, Lambda, ELB, Redis, RDS, DynamoDB, CDN…]]></title>
            <link>https://medium.com/linux-academy/https-medium-com-claude-ducharme-aws-services-performance-benchmark-bbcf4e44c43?source=rss-3f27460752dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbcf4e44c43</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Claude Ducharme]]></dc:creator>
            <pubDate>Thu, 15 Nov 2018 11:21:02 GMT</pubDate>
            <atom:updated>2018-11-15T11:42:05.109Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Services Performance Benchmark (EC2, ECS, API Gateway, Lambda, ELB, Redis, RDS, DynamoDB, CDN, S3, Athena)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*RHEqwN1MhdJbnjNhMKa9PA.png" /><figcaption>Some of the AWS services bench-marked</figcaption></figure><p>Having low latency is very important when designing real-time systems or when smooth user experience is critical. Since users can notice interactions as quick as 50 milliseconds and the client code &amp; UI takes time to render on a device, it is crucial to have a very fast back-end, especially if users are far from the server or the network route is slow.</p><p>Many benchmarks are available to compare performance between coding languages and OS distributions but there is not much information about the AWS services. For example, what is the overhead of making an extra hop in the network, adding SSL to an ELB, or the latency between different Availability Zones of a same Region? I’ll go through this empirical process.</p><p>The goal here is to compare alternative architectures on a small project to have a rough idea about response times, not to gather precise and definitive timings. In order to collect data without noise, all tests consisted of 1000 synchronous calls each returning distinct simple html page to get an order of magnitude of the solutions. One thousand calls is enough to have meaningful statistics and get some outliers. Throttling was configured where needed and an initial warm up was conducted to avoid fluctuations. The average, minimum, maximum, and first call times where recorded then analysed.</p><h3>Acronyms</h3><p>AWS: Amazon Web Services<br>AZ: Availability Zone within a region<br>SSL: Secure Sockets Layer (encryption)<br>CA: Certificate Authority (cryptography)</p><p><strong>Hosting services</strong><br>EC2: Elastic Compute Cloud (a virtual server)<br>API Gateway: Configurable “front door” for applications<br>ECS: Elastic Container Service to run docker containers<br>Lambda: Serverless computing function</p><p><strong>Traffic handlers</strong><br>ELB: Elastic Load Balancer routing incoming requests<br>HA Proxy: High performance TCP/HTTP load balancer</p><p><strong>Data storage/service</strong><br>RDS: Relational Database Service<br>DynamoDB: NoSQL database service<br>Redis: In-memory data store for use as a database or cache<br>CDN: Content delivery network also known as CloudFront<br>S3: Simple Storage Service<br>Athena: Serverless SQL query service on files stored in S3</p><h3><strong>Configurations tested</strong></h3><p>1. EC2-RDS same AZ (A)<br>2. EC2-RDS other AZ (B)<br>3. EC2-RDS other AZ (C)<br>4. EC2-HAProxy-RDS (forward)<br>5. EC2-ELB-RDS (classic)<br>6. EC2-RDS [SSL] (256 bits, no CA)<br>7. EC2-ELB-RDS [SSL] (both sides)<br>8. EC2-Redis (ElastiCache)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*K6vEbJF1GdMFuLBx9zWySg.png" /><figcaption>Bench-marked services — First diagram</figcaption></figure><p>9. EC2-DynamoDB (1000 provisioned read capacity units)<br>10. EC2-CDN-S3 (2nd call when cache is loaded)<br>11. ECS Cluster-RDS<br>12. API Gateway-EC2-RDS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*9dHD2yPY-ip0laNOBRgKhw.png" /><figcaption>Bench-marked services — Second diagram</figcaption></figure><p>13. EC2-S3<br>14. EC2-CDN-S3 (1st call when cache is empty)<br>15. API Gateway-Lambda (128 MB)<br>16. API Gateway-Lambda (3 GB)<br>17. EC2-Athena-S3 (one flat csv file)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*ryL_lg-KfoxIEj84ftLFNA.png" /><figcaption>Bench-marked services — Third diagram</figcaption></figure><p>Note 1: Only two tests were done with the connection encrypted (SSL).<br>Note 2: Environment runtime was confined to Availability Zone A except where explicitly mentioned or when services are managed by AWS. Client calls were from an EC2 outside the app VPC.<br>Note 3: Same coding language was used with identical libraries. Program has few lines of code and was slightly adapted to run in the lambda method.<br>Note 4: SQL query takes an average of less than one millisecond to run.<br>Note 5: Database connection was re-used in the API Gateway Lambda test.<br>Note 6: Instance types used were t2.micro<br>Note 7: CPU and RAM were not a burden when running the server.<br>Note 8: Although network bandwidth of the instance family is likely impacting numbers it should be overall negligible.<br>Note 9: Full html page was stored in S3, Redis, and DynamoDB. Columns were used in RDS. Static pages have an average size of 420 bytes.</p><h3>Benchmark Results</h3><p>Servers making a hop is approximately 3 milliseconds whether it is a forward by a HA Proxy or a call to another Availability Zone. There is a small overhead when adding an ELB in front of a server but benefits justify it. Same logic applies to SSL, although the extra time spent encrypting/decrypting on an ELB is higher than on a server. Note the first call to the ELB was quite slow and is surely noticed by users. Pre-warming will mitigate this behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*GdawymQe_zFAeGLs0sdDhA.png" /><figcaption>Architecture benchmark — Low latency</figcaption></figure><p>Redis performed better than expected and CloudFront (CDN) is following very close in the fastest setups. API Gateway with Lambda and DynamoDB (6 to 36 ms) were slower than anticipated. The lambda itself executed as fast as 3 ms when launched from the management console but timing greatly varied between executions. Allocating more memory lead to the opposite effect than expected, 2 calls took over 9 seconds to execute. The API itself increased the latency by 5 ms average when placed in front of an EC2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*72_5blM9GBBMRevFIVBkgg.png" /><figcaption>Architecture benchmark — Higher latency</figcaption></figure><p>Athena is shown in a separate graph since it is much slower than other setups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*PpiumthxFWjh8O49Phd3Vg.png" /><figcaption>Architecture benchmark — Athena</figcaption></figure><p>The following table displays the timings collected. The Stability formula is “(MAX(FirstCall, Maximum) - Minimum) / Minimum” to demonstrate how performance of a configuration varies (lower the better). Another interest</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*lTA80ON1kvEQPZ0Vwgojrg.png" /><figcaption>Timings collected from different architecture</figcaption></figure><h3>Conclusion</h3><p>Once again the goal is to have an order of magnitude of solutions to take better architecture decisions. Are these numbers similar to what you have on your cloud? I am sure services performance varies between regions. Northern Virginia region is most likely faster as new offerings are normally tested there before being deployed globally. Which other configurations would you like to see benchmarked? For example Aurora vs RDS or the effect of a Placement Group… How other public cloud providers are performing against AWS?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbcf4e44c43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/linux-academy/https-medium-com-claude-ducharme-aws-services-performance-benchmark-bbcf4e44c43">AWS Services Performance Benchmark (EC2, ECS, API Gateway, Lambda, ELB, Redis, RDS, DynamoDB, CDN…</a> was originally published in <a href="https://medium.com/linux-academy">Linux Academy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>