<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeroen Reijn on Medium]]></title>
        <description><![CDATA[Stories by Jeroen Reijn on Medium]]></description>
        <link>https://medium.com/@jreijn?source=rss-61834bc2a27f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tquXEUA2xDYBQ3vMj3G2Fw@2x.jpeg</url>
            <title>Stories by Jeroen Reijn on Medium</title>
            <link>https://medium.com/@jreijn?source=rss-61834bc2a27f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:14:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jreijn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Amazon ECS Express Mode from an IaC perspective]]></title>
            <link>https://towardsaws.com/amazon-ecs-express-mode-from-an-iac-perspective-7ee94e69b465?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ee94e69b465</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[amazon-ecs]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Sat, 20 Dec 2025 21:10:19 GMT</pubDate>
            <atom:updated>2026-01-07T15:05:54.664Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Photo by Timelab on Unsplash" src="https://cdn-images-1.medium.com/max/1024/0*Q--ns2ivlJggf-lQ" /></figure><p>Amazon recently launched a new feature for Amazon ECS called ECS Express Mode. While reading the announcement, I got curious as I use ECS on a regular basis. In this post, we’ll look at what ECS Express Mode is, how you can use it from Infrastructure as Code by means of AWS CDK, and what the limitations are.</p><p>Most posts I’ve seen look at ECS Express Mode from an AWS Console (ClickOps) or AWS CLI perspective. The AWS Console is an easy way to get going, but in my experience, most engineers in larger organizations use Infrastructure as Code (IaC) tools like AWS CDK or Terraform for creating resources in AWS. Before we dive into the IaC part, let’s first take a closer look at ECS Express Mode.</p><h3>What is ECS Express Mode?</h3><p><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html">Amazon Elastic Container Service</a> (Amazon ECS) is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications. With ECS you can run all kinds of container based workloads. You can run container instances for data processing, web applications or spin up containers based on events.</p><p><a href="https://aws.amazon.com/blogs/aws/build-production-ready-applications-without-infrastructure-complexity-using-amazon-ecs-express-mode/">ECS Express Mode is a new feature</a> of Amazon ECS with the promise to go from container image to a fully operational ‘production grade’ web application without thinking too much about infrastructure. The intended promise is to let application development teams focus on the application and let AWS apply the best practices around container infrastructure, scaling, and deployments. It will simplify the process of getting started with ECS and all the different components outside of ECS you will need to set up to run, for instance, a public accessible web-application.</p><p>ECS Express Mode is focused on deploying web applications and APIs. It provides a fully integrated set of AWS resources out of the box. Resources that you would normally have to define, configure, and wire together manually using infrastructure as code. If you’re an experienced ECS user, you’ll immediately appreciate how much heavy lifting Express Mode handles for you.</p><p>When you create your first Express Mode service, ECS automatically provisions and configures the following resources:</p><ul><li><strong>ECS Cluster</strong> — A <strong>default</strong> cluster based on Fargate is created for you if you don’t provide an existing cluster.</li><li><strong>ECS Task Definition</strong> — Defines the container image, resource requirements, environment variables, and runtime configuration.</li><li><strong>ECS Service</strong> — Manages task lifecycle, placement, and health checks.</li><li><strong>Rollback alarm</strong> — When the service starts throwing a larger amount of 40X or 50X response codes after a new deployment, it can revert the deployment</li><li><strong>Canary Deployment</strong> strategy — By default, the service uses a canary deployment mechanism to deploy the service.</li><li><strong>Application Load Balancer (ALB)</strong> — Provides HTTP-based load balancing for your service. A single ALB will serve up to 25 ECS express services; when that limit is exceeded, Express Mode automatically provisions an additional ALB.</li><li><strong>ALB Target Groups</strong> — Route traffic from the load balancer to the running tasks of your service.</li><li><strong>Route 53 Records (AWS-managed)</strong> — Express Mode assigns an AWS-managed domain name that is not visible in your account but is automatically wired to your load balancer.</li><li><strong>ACM Certificates</strong> — An SSL/TLS certificate is created for each service to enable HTTPS. These certificates are managed in your account.</li><li><strong>Security Groups </strong>— Security groups are created for each layer of the network stack (load balancer, tasks, etc.). They follow the principle of least privilege-you only configure the application port once, and Express Mode manages the rest.</li><li><strong>Application Auto Scaling </strong>— A default auto scaling policy is created along with the necessary CloudWatch alarms. By default, scaling is based on CPU utilization, but you can switch to memory utilization or request count.</li><li><strong>Cloudwatch Logs</strong> — A new CloudWatch log group is created for your service. You can configure the log group and change the prefix if you want.</li></ul><p>One of the great benefits is that ECS deploys all these resources in <strong>your</strong> account, which means you can see all individual components, view the configuration, and make changes to the existing configuration if required.</p><p>As you can see from the above list, that’s quite a list of things to configure if you had to do this manually. If you are a user new to ECS and see a lot of unknown elements in the above list, ECS Express mode might be a great fit for you. To get started with ECS Express Mode, you only need three things:</p><ol><li><strong>An existing container image</strong></li><li><strong>An infrastructure role</strong> — an IAM role that lets the ECS provision (non-ECS) resources in your account.</li><li><strong>A task execution role</strong> — an IAM role that gives the container permissions to interact with other AWS services</li></ol><h3>Creating an ECS Express Mode service in AWS CDK</h3><p>ECS Express mode has support for CloudFormation and therefor also support in AWS CDK. The ECS Express Mode service takes care of provisioning all of the resources for you, so there is only a few CloudFormation resources for the express mode service itself. Let’s take a look at what using AWS CDK to provision an ECS Express Mode service looks like.</p><h3>IAM roles and permissions</h3><p>Let’s first start with creating a task executions role with the required permissions for our task. In this post, we will use a simple nginx service to show how the service works, so we don’t need a lot of permissions. You can get started with the default provided managed policy for ECS tasks: <strong>AmazonECSTaskExecutionRolePolicy</strong>.</p><pre>const taskExecutionRole = new Role(this, &#39;executionRole&#39;, {<br>    roleName: &#39;cdk-ecs-express-demo-execution-role&#39;,<br>    assumedBy: new ServicePrincipal(&#39;ecs-tasks.amazonaws.com&#39;),<br>    description: &#39;ECS Task Execution Role&#39;,<br>    managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName(&#39;service-role/AmazonECSTaskExecutionRolePolicy&#39;)]<br>});</pre><p>The next role we create will be for the ECS service to create the resources in our account. Luckily AWS, has provided a default managed policy which we can use: <strong>AmazonECSInfrastructureRoleforExpressGatewayServices</strong>.</p><pre>const infrastructureRole = new Role(this,&#39;infrastructureRole&#39;, {<br>    roleName: &#39;cdk-ecs-express-demo-infrastructure-role&#39;,<br>    assumedBy: new ServicePrincipal(&#39;ecs.amazonaws.com&#39;),<br>    description: &#39;ECS Task Infrastructure Role&#39;,<br>    managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName(&#39;service-role/AmazonECSInfrastructureRoleforExpressGatewayServices&#39;)]<br>});</pre><p>Upon closer inspection, you can imagine this policy contains quite a list of permissions as it needs to cover a lot of services. It’s worth inspecting the policy when your first start to use it. I won’t copy it here as it contains about 280 lines or JSON at the moment.</p><h3>Defining the ECS Gateway Service</h3><p>Now that we’ve set up the roles with the correct permissions, we can move to the final step, which is to create the ECS Express Mode Service. Currently (as of the beginning of December there is a level 1 CDK construct available, which maps directly to the CloudFormation resource. The resource is named <strong>CfnExpressGatewayService</strong>.</p><pre>const cfnExpressGatewayService = new CfnExpressGatewayService(this, &#39;ExpressServiceNginx1&#39;, {<br>    primaryContainer: {<br>        image: &#39;nginx:latest&#39;,<br>    },<br>    executionRoleArn: taskExecutionRole.roleArn,<br>    infrastructureRoleArn: infrastructureRole.roleArn,<br>});</pre><p>In general, that’s it. That’s all you need to run your web app container service in ECS, which I think is pretty great. Now, how can you reach the service?</p><h3>Accessing the service</h3><p>The ECS express mode service will handle DNS for you. The DNS records are the only thing that are not created within your own account, as far as I could tell (the ACM certificates are though). According to the <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/express-service-getting-started.html#express-service-access-application">documentation</a>, the <em>service name</em> and <em>region</em> should be enough to determine the URL of the service.</p><p>So, if we define the service as:</p><pre>let cfnExpressGatewayService = new CfnExpressGatewayService(this, &#39;ExpressServiceNginx1&#39;, {<br>    serviceName: &#39;demo-express-service&#39;<br>    primaryContainer: {<br>        image: &#39;nginx:latest&#39;,<br>    },<br>    executionRoleArn: taskExecutionRole.roleArn,<br>    infrastructureRoleArn: infrastructureRole.roleArn,<br>});</pre><p>and we follow the pattern mentioned in the docs:</p><pre>https://&lt;service-name&gt;.ecs.&lt;region&gt;.on.aws/</pre><p>should have resulted in the service being accessible from <a href="https://demo-express-service.ecs.eu-west-1.on.aws/">https://demo-express-service.ecs.eu-west-1.on.aws/</a>.</p><p>That made me wonder if the service name is unique, just like with an S3 bucket name, but when I tried specifying the <strong>serviceName</strong> property of the CfnExpressGatewayService the resulting URL would return a 404. The only way I’ve found to determine the correct URL for the service is by using the <strong>Endpoint</strong> return value from the CloudFormation resource.</p><pre>new CfnOutput(this,&quot;service1url&quot;, {<br>    key: &#39;service1url&#39;,<br>    value: &#39;https://&#39;+ cfnExpressGatewayService.getAtt(&quot;Endpoint&quot;).toString()<br>})</pre><p>Which results in the following output:</p><pre>https://ng-e6840bdfebb54e28bb5c3c9bd7c1f5fa.ecs.eu-west-1.on.aws</pre><p>As you can see there, is no mention of the service name in the generated endpoint. It does seem the first two letters of the domain are coming from the service name, but that might be a coincidence.</p><p>Once the service is up and running, you can inspect all the different components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T1z2KdCjCQFPOUIW.jpeg" /></figure><h3>Configuring alternative settings</h3><p>We’ve looked at the minimum configuration. Now let’s take a look as some more configuration options.</p><h3>Specifying your own ECS cluster</h3><p>By default, the service creates an ECS cluster in your default VPC with the cluster name <strong>default</strong>. If you don’t have a default VPC or want to have more control over your cluster, you can configure your own cluster.</p><pre>const cluster = new Cluster(this, &#39;Cluster&#39;, {<br>    vpc: vpc,<br>    clusterName: &#39;cdk-ecs-express-demo-cluster&#39;,<br>})<br><br>const cfnExpressGatewayService2 = new CfnExpressGatewayService(this, &#39;ExpressServiceNginx2&#39;, {<br>    cluster: cluster.clusterName,<br>    primaryContainer: {<br>      image: &#39;nginx:latest&#39;,<br>    },<br>    executionRoleArn: taskExecutionRole.roleArn,<br>    // other optional configuration options<br>    healthCheckPath: &quot;/&quot;,<br>    serviceName: &#39;express-service-nginx-2&#39;,<br>    infrastructureRoleArn: infrastructureRole.roleArn,<br>    // Autoscaling configuration<br>    scalingTarget: {<br>        minTaskCount: 1,<br>        maxTaskCount: 3,<br>    },<br>    // cpu: &#39;512&#39;,<br>    // memory: &#39;512&#39;,<br>});</pre><p>According to the <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-ecs-expressgatewayservice.html">CloudFormation documentation</a>, you can specify either the cluster name or the cluster ARN. However, when I tried the ARN, CloudFormation returned with an error.</p><pre>Resource handler returned message: &quot;Invalid request provided:<br>Invalid identifier: Unexpected number of separators (Service: Ecs, Status Code: 400, Request ID: ee074322-e841-4256-8bb7-2b2014b9c76e) (SDK Attempt Count: 1)&quot; (RequestToken: ec5bfa00-2c43-db0f-3b7c-6374d0a33f5a, HandlerErrorCode: InvalidRequest)</pre><p>I’m not sure if that’s a problem in the documentation or if I misconfigured something. I advise you to stick to the clusterName, which seems to work.</p><h3>Alternative configuration options</h3><p>The express mode service we created previously uses AWS best practices. If you have slightly different requirements, you can change specific parts of the configuration. For your container you can change for instance the cpu and memory requirements.</p><p>By default the ECS server will run in the default VPC and in a public subnet. You can specify alternative network settings for your service depending on your requirements. For instance your can switch to a different VPC and subnet. Because you can change the ECS cluster, you can probably also use Fargate Spot or EC2 based compute instead. I have not tested that though, and it might be something for another blog post.</p><p>Some settings can’t be configured right now. Canary deployment for instance is the only supported deployment strategy. I think it’s a great default, but I hope they will add linear or blue/green deployment as these strategies are now <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_service-options.html">ECS supported deployment strategies</a>.</p><p>ECS Express mode only allows you to configure a single container. If you’re currently using ECS and are using a sidecar container for for instance observability purposes, you won’t be able to do that during creation with ECS express mode. You might want to consider using an Open Telemetry library, which integrates with your application framework. If you really need a sidecar container you can import the task defintion into your stack, once it has been created. Express Mode will keep ‘manual’ changes during additional deployments / updates.</p><h3>Configuring a WAF for your ECS service</h3><p>In all cases where I use ECS with a public ALB, I always add a Web Application Firewall (WAF). I also tried this for the ECS Express mode service, but while trying this, I learned there things to consider. There is currently no out-of-the box experience for attaching the WAF to the load balancer created by the ECS service. The CloudFormation resource <strong>AWS::ECS::ExpressGatewayService</strong> does expose some <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-ecs-expressgatewayservice.html#aws-resource-ecs-expressgatewayservice-return-values">return values</a> after the resource has been created, which you can use to get the load balancer ARN.</p><p>With the return value for the ALB ARN, you can assign the WebACL to the created Load balancer.</p><pre>const cfnWebACLAssociation = new CfnWebACLAssociation(this, &#39;ALBWebACLAssociation&#39;, {<br>    resourceArn: cfnExpressGatewayService.getAtt(&quot;ECSManagedResourceArns.IngressPath.LoadBalancerArn&quot;).toString(),<br>    webAclArn: wafV2.attrArn,<br>});<br><br>// Don&#39;t forget to add a dependency, otherwise this association might happen before the service has been created.<br>cfnWebACLAssociation.addDependency(cfnExpressGatewayService);</pre><p>There is a tricky problem I foresee, which is when you hit the 25 services limit and a new Application Load Balancer is created. ECS Express Mode handles this by itself, and if you don’t have all 25 services defined in the same CDK stack, it gets hard to track if you’ll need to create a new WebACL Association.</p><h3>Resource state management</h3><p>After playing with ECS Express Mode for a while, it made me think. Who is in control of the resources created in your account? Who is responsible for its state? According to the ECS team they are not using CloudFormation internally, but are working with the direct APIs and use their own internal state for managing all resources. If you want to change specific parts of the default generated solution, you can import resources into CloudFormation/CDK and change them, but the ECS service will always be the owner of the resources. In an enterprise context where I see a lot of infrastructure as code, I wonder about the developer experience and aspects like compliance rules.</p><h3>CDK L3 constructs vs ECS Express Mode</h3><p>If you’re an experienced ECS and AWS CDK user you might question how Express Mode compares with, for instance, the <strong>ApplicationLoadBalancedFargateService</strong> construct. I think it comes quite close to what ECS express mode delivers. However, express mode also adds autoscaling, deployment rollbacks, canary deployment, smart load balancer sharing, default HTTPS, and domain management out of the box. If you’re an experienced CDK user, you can get very far with the CDK provided L3 construct(s). You might even consider creating one that does something similar. The advantage of that would be that you have all AWS resources defined in your in CDK/CloudFormation and you have full control over all your resources. The ExpressGatewayService is a single AWS resource, which means it’s tool agnostic and delivers the same level of business value if you use CDK, CloudFormation, Pulumi, Terraform, or any other kind of IaC tool. You’re not depending on Terraform modules, CDK L3 constructs, or any other kind of configuration elements when working with IaC tools.</p><h3>Closing Thoughts</h3><p>I like what the ECS team has created with ECS Express Mode. It’s really great to go from a container image to a production like environment in a matter of minutes. This is going to work great for demo’s and quick PoCs. Looking at this from an infrastructure as code perspective it feels a bit unnatural though as the ExpressGatewayService hides a lot of other AWS resources. If you’re already a heavy CDK user, you might feel you have too little control over your resources. You can get a similar experience with creating your own L3 construct or using an existing one like the <strong>ApplicationLoadBalancedFargateService.</strong> There are still some cases I want to explore further to see how ECS express mode compares to a set of L2/L3 constructs and how to handle changes to the individual components created by the express gateway service. The service is quite new, therefor I hope it will get some nice updates in the near future, like more deployment strategies, and perhaps custom domain support. If you’ve not looked at ECS Express Mode before I think it’s definitely worth to take a look and see if it works for your specific use case.</p><h3>Further resources</h3><ul><li><a href="https://www.youtube.com/watch?v=z9JUEQjpGgY">Build production-ready applications with Amazon ECS Express Mode</a> (Containers from the Couch)</li><li><a href="https://aws.amazon.com/blogs/aws/build-production-ready-applications-without-infrastructure-complexity-using-amazon-ecs-express-mode/">Build production-ready applications without infrastructure complexity using Amazon ECS Express Mode</a> (AWS Blog)</li><li><a href="https://dev.to/aws-builders/from-image-to-https-endpoint-in-one-step-with-ecs-express-mode-1oi2">From image to HTTPS endpoint in one step with ECS Express Mode</a></li><li><a href="https://github.com/jreijn/demos-aws-cdk/tree/develop/aws-cdk-ecs-express">GitHub repository</a> containing an ECS Express Mode sample CDK application with code mentioned in this blogpost.</li></ul><p><em>Originally published at </em><a href="https://www.jeroenreijn.com/2025/12/amazon-ecs-express-mode-from-an-iac-perspective.html"><em>https://www.jeroenreijn.com</em></a><em> on December 20, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ee94e69b465" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/amazon-ecs-express-mode-from-an-iac-perspective-7ee94e69b465">Amazon ECS Express Mode from an IaC perspective</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improve observability for Windows EC2 instances with the CloudWatch Agent]]></title>
            <link>https://jreijn.medium.com/improve-observability-for-windows-ec2-instances-with-the-cloudwatch-agent-1163592d7179?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1163592d7179</guid>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Sat, 04 Oct 2025 19:28:30 GMT</pubDate>
            <atom:updated>2026-01-05T12:32:06.657Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*367d4QbaLlbrO8-3.jpg" /></figure><p><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html">Amazon CloudWatch</a> monitors Windows EC2 instances out of the box, but only gives you basic host-level metrics like CPU and network usage. In this post, you’ll learn how to extend observability with the Amazon CloudWatch Agent. We will use AWS CDK in TypeScript to automatically deploy an example instance with an installed and configured CloudWatch Agent.</p><h3>Default collected metrics</h3><p>Elastic Compute Cloud (EC2) instances send <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html">a set of predefined metrics</a> to CloudWatch. Most of these metrics are system related and focus on CPU and Networking metrics. Under the monitoring tab in the AWS EC2 Console you can get a quick overview of some of out of the box available metrics for your EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AbMie-iJcutOPYyW.png" /></figure><p>It’s a great starting set, but what if you need more? What if you need metrics from inside of the VM?</p><h3>Gathering additional metrics</h3><p>If you need more system or application-level metrics, you can make use of the Amazon <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html">CloudWatch Agent</a>. The CloudWatch agent is service which you can run on the EC2 instance and it’s great for:</p><ul><li>Collecting system-level metrics (CPU, memory, disk, network)</li><li>Gathering custom metrics from your application(s)</li><li>Collecting and centralizing logs from various sources</li><li>Monitoring both AWS and on-premises environments with a single tool</li></ul><p>A high-level overview of the CloudWatch Agent and how it interacts with an EC2 instance and the CloudWatch service is shown in the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pJhLTWJpJnhzf3O7.png" /></figure><p>You can install the CloudWatch agent via EC2 user data, AWS Systems Manager, or via the AWS EC2 Console. In this post we will install the CloudWatch Agent and its configuration using the user data feature of an EC2 instance. Before we can actually install the agent, we first need to create a CloudWatch Agent configuration, which we can use later on during the installation.</p><h3>Create a CloudWatch Agent Configuration</h3><p>The CloudWatch Agent configuration file is a JSON file with four sections:</p><ul><li><strong>agent</strong> — includes fields for the overall configuration of the agent</li><li><strong>metrics</strong> — specifies the custom metrics for collection and publishing to CloudWatch</li><li><strong>logs</strong> — specifies what log files are published to CloudWatch Logs. This can include events from the Windows Event Log if the server runs Windows Server.</li><li><strong>traces</strong> — specifies the sources for traces that are collected and sent to AWS X-Ray</li></ul><p>As you can see there is a lot you can do with the CloudWatch agent. However, in this post we will focus only on metrics.</p><p>Let’s take a look at an example configuration:</p><pre>const cwAgentConfig = {<br>    agent: {<br>        metrics_collection_interval: 60<br>    },<br>    metrics: {<br>        namespace: &quot;CWAgent&quot;,<br>        append_dimensions: {<br>            InstanceId: &quot;\${aws:InstanceId}&quot;,<br>        },<br>        metrics_collected: {<br>            LogicalDisk: {<br>                measurement: [<br>                    {name: &quot;% Free Space&quot;, unit: &quot;Percent&quot;},<br>                    {name: &quot;Free Megabytes&quot;, unit: &quot;Megabytes&quot;},<br>                ],<br>                &quot;resources&quot;: [<br>                    &quot;*&quot;<br>                ]<br>            },<br>            Memory: {<br>                metrics_collection_interval: 60,<br>                measurement: [<br>                    {&quot;name&quot;: &quot;% Committed Bytes in Use&quot;, &quot;unit&quot;: &quot;Percent&quot;},<br>                ],<br>                &quot;resources&quot;: [<br>                    &quot;*&quot;<br>                ]<br>            }<br>        }<br>    }<br>};<br>// Stringify the JSON<br>const configJson = JSON.stringify(cwAgentConfig);</pre><p>In the above example configuration, we specify that the CloudWatch Agent needs to collect metrics every 60 seconds. If you gather more than just metrics you can also specify collection intervals per section (metrics, logs, traces).</p><p>For clarity the namespace is explicitly set to <strong>CWAgent</strong>, but if you omit the namespace all metrics will be stored in the <strong>CWAgent</strong> namespace by default. You can of course set your own custom namespace.</p><p>For each collected metric an additional dimension is added for the <em>InstanceId</em>. When collecting metrics for EC2 instances you can also think of other dimensions like InstanceType, AutoScalingGroupName or ImageId which would reflect in the JSON as:</p><pre>&quot;append_dimensions&quot;: {<br>    &quot;InstanceId&quot;: &quot;${aws:InstanceId}&quot;,<br>    &quot;InstanceType&quot;: &quot;${aws:InstanceType}&quot;,<br>    &quot;AutoScalingGroupName&quot;: &quot;${aws:AutoScalingGroupName}&quot;<br>}</pre><p>Looking at the metrics <strong>metrics_collected</strong> section you can see we want to collect metrics for the Windows Performance object <em>LogicalDisk</em> and <em>Memory</em>. If you’re doing this for the first time you might wonder what other performance objects and metrics are available. You can see the performance objects by running the following command on your Windows instance.</p><p>Powershell:</p><pre>Get-Counter -ListSet *</pre><p>Command-line:</p><pre>TypePerf.exe –q</pre><p>Running one of the above command will return quite a list of performance objects and corresponding metrics. You can expect to see all kinds of different objects. From SQL Server to Windows Events.</p><p>Now let’s take a closer look at the measurement section in which the metrics are defined that we want to store. For each metrics you can explicitly specify a metric name and unit the metric needs to be stored as (Percentage, Megabytes. etc). If you leave out the unit type the agent will try to pick the right one.</p><p>Due note that all metrics collected with the CloudWatch agent are considered custom metrics and are billed separately. A custom metric costs about $0.30 per metric per month (depending on your region), so <strong>costs can add up quickly</strong>.</p><h3>Storing the configuration in Systems Manager Parameter store</h3><p>While implementing this I ran into all kinds of issues with the JSON format of the configuration file. Creating the JSON file in typescript, passing it to PowerShell, writing it to disk, and reading the file took me multiple attempts to get right. An easier approach is to store the JSON configuration in Systems Manager Parameter Store and just pass the parameter when configuring the agent.</p><pre>const cwAgentConfigParam = new StringParameter(this, &#39;cw-agent-config&#39;, {<br>    parameterName: &#39;/ec2/cw-agent-config.json&#39;,<br>    stringValue: configJson<br>});</pre><p>For the instance to be able to fetch the parameter we will need to make sure we set the proper IAM permissions for the instance role. Let’s create a new IAM role and add the required permissions.</p><pre>const role = new Role(this, &quot;instance-role&quot;, {<br>    assumedBy: new ServicePrincipal(&quot;ec2.amazonaws.com&quot;)<br>});<br><br>// required for the CloudWatch agent to send data to cloudwatch<br>role.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName(&quot;CloudWatchAgentServerPolicy&quot;));<br><br>// required for SSM to manage the instance<br>role.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName(&quot;AmazonSSMManagedInstanceCore&quot;));<br><br>// Don&#39;t forget to add the permissions to read the agent configuration parameter<br>cwAgentConfigParam.grantRead(role);</pre><p>Now that we’ve configured IAM and stored our agent configuration, let’s create a Windows EC2 instance to attach and monitor.</p><h3>Creating a Windows Server EC2 instance</h3><p>When creating a Windows instance in AWS CDK you will need to place the instance in one of the subnets of your VPC. Pick a specific Windows AMI like Windows Server 2022 and make sure to reference our IAM instance role, otherwise a default role is created without the proper permissions.</p><pre>// Create a Windows based EC2 instance<br>const instance = new Instance(this, &quot;windows-demo-instance&quot;, {<br>    vpc: demoVpc,<br>    instanceType: InstanceType.of(InstanceClass.BURSTABLE3, InstanceSize.SMALL),<br>    machineImage: MachineImage.latestWindows(WindowsVersion.WINDOWS_SERVER_2022_ENGLISH_FULL_BASE),<br>    securityGroup: securityGroup,<br>    role: role,<br>    keyPair: keyPair,<br>    associatePublicIpAddress: false,<br>    vpcSubnets: {subnetType: SubnetType.PRIVATE_WITH_EGRESS},<br>    detailedMonitoring: true,<br>})<br><br>securityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(3389), &quot;Allow RDP Connections&quot;)</pre><p>Make sure to allow remote desktop access if you want to connect with remote desktop. I usually use the Systems manager Fleet manager to connect with Remote Desktop. Keep in mind that Amazon provided Windows AMI’s have the Systems Manager Agent installed. If you use your own AMI you will need to make sure you install the Systems Manager agent yourself.</p><h3>Installing the CloudWatch Agent on deployment</h3><p>As mentioned before in this post we will install the CloudWatch Agent, required for gathering our metrics, during the instance deployment by means of instance userdata. The latest version of the CloudWatch Agent is published by Amazon into an S3 bucket so all we need to do is request it and run the installer.</p><p>When starting the agent make sure you reference the configuration based on its SSM parameter name. We do so by specifying the ssm: prefix/protocol. The CloudWatch Agent, if given the permissions to fetch the parameter, will read the parameter from parameter store during startup.</p><pre>// Write config to disk and install agent<br>instance.userData.addCommands(<br>    // Install CW Agent via MSI<br>    `powershell.exe -Command &quot;Invoke-WebRequest https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/amazon-cloudwatch-agent.msi -OutFile C:\cwagent.msi&quot;`,<br>    `msiexec /i C:\cwagent.msi /qn /l*v C:\cwagent-install.log`,<br>    // Wait for installation to complete<br>    `timeout /t 30`,<br>    // Start the agent with proper PowerShell execution<br>    `powershell.exe -ExecutionPolicy Bypass -Command &quot;&amp; &#39;C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1&#39; -a fetch-config -m ec2 -c ssm:${cwAgentConfigParam.parameterName} -s&quot;`,<br>    // Wait for agent to start and load<br>    `timeout /t 30`,<br>    // check agent status<br>    `powershell.exe -ExecutionPolicy Bypass -Command &quot;&amp; &#39;C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1&#39; -a status -m ec2&quot;`,<br>);</pre><p>Once the deployment is finished you should be able to see more metrics in the monitoring tab for the EC2 instance. By default, it will check if there are metrics within the CWAgent namespace and if so add them to the monitoring screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HuQe9QDWiKTBAuKL.png" /></figure><p>If the agent doesn’t publish your custom metrics, login with remote desktop and check the CloudWatch Agent log file C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log for errors. If the file is not there it might be the case the CloudWatch Agent isn&#39;t running, so you will need to troubleshoot that first.</p><h3>Creating CloudWatch Alarms for proactive monitoring</h3><p>Adding metrics is great for trend analysis, but you really also want to have some alarms in place. Let’s look at the scenario in case the disk is running out of space. Let’s create an alarm for the “LogicalDisk % Free Space” metric.</p><pre>// Low diskspace alarm for C drive<br>let lowDiskSpaceCVolumeAlarm = new Alarm(this, &#39;alarm&#39;, {<br>    alarmDescription: &#39;This metric monitors the amount of free disk space on the instance. If the amount of free disk space falls below 10% for 5 minutes, the alarm will trigger&#39;,<br>    metric: new Metric({<br>        dimensionsMap: {<br>            InstanceId: instance.instanceId,<br>            objectname: &quot;LogicalDisk&quot;,<br>            instance: &quot;C:&quot;<br>        },<br>        statistic: &quot;Average&quot;,<br>        namespace: &quot;CWAgent&quot;,<br>        metricName: &quot;LogicalDisk % Free Space&quot;,<br>        unit: Unit.PERCENT,<br>    }),<br>    threshold: 10,<br>    evaluationPeriods: 5,<br>    datapointsToAlarm: 5,<br>    comparisonOperator: ComparisonOperator.LESS_THAN_OR_EQUAL_TO_THRESHOLD,<br>    treatMissingData: TreatMissingData.MISSING,<br>});<br>lowDiskSpaceCVolumeAlarm.addAlarmAction(new SnsAction(topic));<br>lowDiskSpaceCVolumeAlarm.addOkAction(new SnsAction(topic));</pre><p>This should result in an alarm that’s monitoring the amount of free disk space for the C: drive of our windows instance. It took me a couple of attempts to get the alarm working properly so it’s good to validate your alarm dimensions and check the alarm in the AWS Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4YMZENaqaUjDotxd.png" /></figure><p>Now that the alarm is in place, CloudWatch will notify the subscribers of the SNS topic in case the alarm state is triggered.</p><h3>Summary</h3><p>In this post, we explored how to create metrics and alarms for Windows based EC2 instances by leveraging the CloudWatch Agent and CloudWatch alarms. We’ve used AWS CDK for provisioning the infrastructure. Along the way, we covered some important topics like:</p><ul><li>CloudWatch agent configuration</li><li>Windows Performance objects and metrics.</li><li>CloudWatch alarms for custom metrics</li></ul><p>With these pieces in place, you’ll have a setup that alerts your team when your Windows metrics exceed their threshold. Next, you could extend this setup with other instance metrics, application-level metrics and log collection.</p><p>A fully working CDK application with the code mentioned in this blogpost can be found in the <a href="https://github.com/jreijn/demos-aws-cdk/tree/develop/aws-cdk-windows-diskspace-monitoring">following GitHub repo</a>.</p><p><em>Originally published at </em><a href="https://www.jeroenreijn.com/2025/10/improve-observability-for-windows-ec2-instances-with-the-cloudwatch-agent.html"><em>https://www.jeroenreijn.com</em></a><em> on October 4, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1163592d7179" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enabling AWS Budget Notifications with SNS using AWS CDK]]></title>
            <link>https://aws.plainenglish.io/enabling-aws-budget-notifications-with-sns-using-aws-cdk-139f617824ab?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/139f617824ab</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[aws-cdk]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 14:31:53 GMT</pubDate>
            <atom:updated>2025-09-15T01:25:41.002Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8VRGfgxcqgtdJoJg" /></figure><p><em>Learn how to set up AWS Budget notifications with SNS using AWS CDK in TypeScript, including gotchas around IAM and KMS policies.</em></p><p>Keeping track of AWS spend is very important. Especially since it’s so easy to create resources. You might forget to turn off an EC2 instance or container you started, or remove a CDK stack for a specific experiment. Costs can creep up fast if you don’t put guardrails in place.</p><p>Recently, I had to set up budgets across multiple AWS accounts for my team. Along the way, I learned a few gotchas (especially around SNS and KMS policies) that weren’t immediately clear to me as I started out writing AWS CDK code. In this post, we’ll go through how to:</p><ul><li>Create AWS Budgets with AWS CDK</li><li>Send notifications via email and SNS</li><li>Handle cases like encrypted topics and configuring resource policies</li></ul><p>If you’re setting up AWS Budgets for the first time, I hope this post will save you some trial and error.</p><h3>What are AWS Budgets?</h3><p><a href="https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html">AWS Budgets</a> is part of AWS Billing and Cost Management. It lets you set <strong>guardrails</strong> for spend and usage limits. You can define a budget around cost, usage, or even commitment plans (like Reserved Instances and Savings Plans) and trigger alerts when you cross a threshold. You can think of Budgets as your <strong>planned</strong> spend tracker. Budgets are great for:</p><ul><li>Alerting when costs hit predefined thresholds (e.g., 80% of your budgeted spend)</li><li>Driving team accountability by tying alerts to product or account owners</li><li>Enforcing a cap on monthly spend triggering an action and shutting down compute (EC2), if you go over budget (be careful with this)</li></ul><p>Keep in mind that budgets and their notifications are not instant. AWS billing data is processed multiple times a day, but you might trigger your budget a couple of hours after you’ve passed your threshold. This is clearly stated in the AWS <a href="https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-best-practices.html">documentation</a> as:</p><blockquote><em>AWS billing data, which Budgets uses to monitor resources, is updated at least once per day. Keep in mind that budget information and associated alerts are updated and sent according to this data refresh cadence.</em></blockquote><h3>Defining Budgets with AWS CDK</h3><p>You can create different kinds of budgets, depending on your requirements. Some examples are:</p><ul><li><strong>Fixed budgets</strong>: Set one amount to monitor every budget period.</li><li><strong>Planned budgets</strong>: Set different amounts to monitor each budget period.</li><li><strong>Auto-adjusting budgets</strong>: Set a budget amount to be adjusted automatically based on the spending pattern over a time range that you specify.</li></ul><p>We’ll start with a simple example of how you can create a budget in the CDK. We’ll go for a <strong>fixed</strong> budget of about $100. The AWS CDK currently only has <a href="https://docs.aws.amazon.com/cdk/v2/guide/constructs.html">Level 1 constructs</a> available for budgets, which means that the classes in the CDK are a 1 to 1 mapping to the CloudFormation resources. Because of this you will have to explicitly define all required properties (constructs, IAM policies, resource policies, etc), which otherwise could be taken care of by a CDK L2 construct. It also means your CDK code will be a bit more verbose. We’ll start by using the <strong>CfnBudget</strong> construct.</p><pre>new cdk.aws_budgets.CfnBudget(this, &#39;fixed-monthly-cost-budget&#39;, {<br>    budget: {<br>        budgetType: &#39;COST&#39;,<br>        budgetLimit: {amount: 100, unit: &#39;USD&#39;},<br>        budgetName: &#39;Monthly Costs Budget&#39;,<br>        timeUnit: &#39;MONTHLY&#39;<br>    }<br>}</pre><p>In the above example we’ve created a budget with a limit of <strong>$100 per month</strong>. A budget alone isn’t very useful. You’d still have to check into the AWS console manually to see what your spend is compared to your budget. The important thing is that we want to get notified in case we reach our budget or our forecasted budget will reach our threshold, so let’s add a notification and a subscriber.</p><pre>new cdk.aws_budgets.CfnBudget(this, &#39;fixed-monthly-cost-budget&#39;, {<br>  budget: {<br>    budgetType: &#39;COST&#39;,<br>    budgetLimit: {amount: 100, unit: &#39;USD&#39;},<br>    budgetName: &#39;Monthly Costs Budget&#39;,<br>    timeUnit: &#39;MONTHLY&#39;<br>  },<br>  notificationsWithSubscribers: [{<br>    notification: {<br>      comparisonOperator: &#39;GREATER_THAN&#39;,<br>      notificationType: &#39;FORECASTED&#39;,<br>      threshold: 100,<br>      thresholdType: &#39;PERCENTAGE&#39;<br>    },<br>    subscribers: [{<br>      subscriptionType: &#39;EMAIL&#39;,<br>      address: &#39;&lt;your-email-address&gt;&#39;<br>    }]<br>  }]<br>});</pre><p>Based on the notification settings, interested parties are notified when the spend is <strong>forecasted</strong> to exceed 100% of our defined budget limit. You can put a notification on forecasted or actual percentages. When that happens an email is send to the designated email address. Subscribers, at the time of writing, can be either email recipients or a Simple Notification Service (SNS) topic. In the above code example we use email subscribers for which you can add up to 10 recipients.</p><p>Depending on your team or organization it might be beneficial to switch to using an SNS topic. The advantage of using an SNS topic over a set of email subscribers is that you can add different kind of subscribers (email, chat, custom lambda functions) to your SNS topic. With an SNS topic you have a single place to configure subscribers and if you change your mind you can do so in one place instead of updating all budgets. Using an SNS Topic also allows you to push budget notifications to for instance a chat client like MS Teams or Slack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/0*TgB6OpHEDOsKkIdy.jpg" /></figure><p>In this case we wil make use of SNS in combination with email subscribers. Let’s start by defining an SNS topic with the AWS CDK.</p><pre>// Create a topic for email notifications<br>let topic = new Topic(this, &#39;budget-notifications-topic&#39;, {<br>      topicName: &#39;budget-notifications-topic&#39;<br>});</pre><p>Now let’s add an email subscriber, as this is the most simple way to receive budget notifications.</p><pre>// Add email subscription<br>topic.addSubscription( new EmailSubscription(&quot;your-email-address&quot;));</pre><p>This looks pretty straight forward and you might think you’re done, but there is one important step to take next, which I initially forgot. The AWS budgets service will need to be granted permissions to publish messages to the topic. To be able to do this, we will need to add a resource policy to the topic which allows the budgets service to call the <em>SNS:Publish</em> action for our topic.</p><pre>// Add resource policy to allow the budgets service to publish to the SNS topic<br>topic.addToResourcePolicy(new PolicyStatement({<br>  actions:[&quot;SNS:Publish&quot;],<br>  effect: Effect.ALLOW,<br>  principals: [new ServicePrincipal(&quot;budgets.amazonaws.com&quot;)],<br>  resources: [topic.topicArn],<br>  conditions: {<br>    ArnEquals: {<br>     &#39;aws:SourceArn&#39;: `arn:aws:budgets::${Stack.of(this).account}:*`,<br>   },<br>    StringEquals: {<br>      &#39;aws:SourceAccount&#39;: Stack.of(this).account,<br>    },<br>  },<br>}))</pre><p>Now let’s assign the SNS topic as a subscriber in our CDK code.</p><pre>// Define a fixed budget with SNS as subscriber<br>new cdk.aws_budgets.CfnBudget(this, &#39;fixed-monthly-cost-budget&#39;, {<br>    budget: {<br>        budgetType: &#39;COST&#39;,<br>        budgetLimit: {amount: 100, unit: &#39;USD&#39;},<br>        budgetName: &#39;Monthly Costs Budget&#39;,<br>        timeUnit: &#39;MONTHLY&#39;<br>    },<br>    notificationsWithSubscribers: [{<br>        notification: {<br>            comparisonOperator: &#39;GREATER_THAN&#39;,<br>            notificationType: &#39;FORECASTED&#39;,<br>            threshold: 100,<br>            thresholdType: &#39;PERCENTAGE&#39;<br>        },<br>        subscribers: [{<br>            subscriptionType: &#39;SNS&#39;,<br>            address: topic.topicArn<br>        }]<br>    }]<br>});</pre><h3>Working with Encrypted Topics</h3><p>If you have an SNS topic with <strong>encryption enabled</strong> (via KMS) you will need to make sure that the corresponding service has access to the KMS key. If you don’t, you will not get any messages and as far as I could tell you will see no errors (at least I could find none in CloudTrail). I actually wasted a couple of hours trying to figure this part out.</p><p>I should have <a href="https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-sns-policy.html#configure-kms-perm">read the documentation</a> as it is explicitly stated to do so. I guess I should start with the docs instead of diving right into the AWS CDK code.</p><pre>// Create KMS key used for encryption<br>let key = new Key(this,&#39;sns-kms-key&#39;, {<br>   alias: &#39;sns-kms-key&#39;,<br>   enabled: true,<br>   description: &#39;Key used for SNS topic encryption&#39;<br>});<br><br>// Create topic and assign the KMS key<br>let topic = new Topic(this, &#39;budget-notifications-topic&#39;, {<br>    topicName: &#39;budget-notifications-topic&#39;,<br>    masterKey: key<br>});</pre><p>Now let’s add the resource policy to the key and try to trim down the permissions as much as possible.</p><pre>// Allow access from budgets service<br>key.addToResourcePolicy(new PolicyStatement({<br>    effect: Effect.ALLOW,<br>    actions: [&quot;kms:GenerateDataKey*&quot;,&quot;kms:Decrypt&quot;],<br>    principals: [new ServicePrincipal(&quot;budgets.amazonaws.com&quot;)],<br>    resources: [&quot;*&quot;],<br>    conditions: {<br>        StringEquals: {<br>            &#39;aws:SourceAccount&#39;: Stack.of(this).account,<br>        },<br>        ArnLike: {<br>            &quot;aws:SourceArn&quot;: &quot;arn:aws:budgets::&quot; + Stack.of(this).account +&quot;:*&quot;<br>        }<br>    }<br>}));</pre><h3>Putting it all together</h3><p>If you’ve configured everything correctly and deployed your stack to your target account you should be good to go. Once you cross your threshold you should be notified by email that your budget is exceeding one of your thresholds (depending on the threshold set).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LD8vWt4shLt83iwg.png" /></figure><h3>Summary</h3><p>In this post, we explored how to create AWS Budgets with AWS CDK and send notifications through email or SNS. Along the way, we covered some important topics like:</p><ul><li>Budgets alone aren’t useful until you add notifications.</li><li>SNS topics need a resource policy so the Budgets service can publish.</li><li>Encrypted topics require KMS permissions for the Budgets service.</li></ul><p>With these pieces in place, you’ll have a setup that alerts your team when costs exceed thresholds via email, chat, or custom integrations.</p><p>A fully working CDK application with the code mentioned in this blogpost can be found in the <a href="https://github.com/jreijn/demos-aws-cdk/tree/develop/aws-cdk-budget-notifications">following GitHub repo</a>.</p><blockquote><em>Photo by </em><a href="https://unsplash.com/@towfiqu999999?utm_source=sveltia-cms&amp;utm_medium=referral"><em>Towfiqu barbhuiya</em></a><em> on </em><a href="https://unsplash.com/?utm_source=sveltia-cms&amp;utm_medium=referral"><em>Unsplash</em></a></blockquote><p><em>Originally published at </em><a href="https://www.jeroenreijn.com/2025/08/enabling-aws-budget-notifications-with-sns-using-aws-cdk.html"><em>https://www.jeroenreijn.com</em></a><em> on August 25, 2025.</em></p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=139f617824ab" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/enabling-aws-budget-notifications-with-sns-using-aws-cdk-139f617824ab">Enabling AWS Budget Notifications with SNS using AWS CDK</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-account DNS with AWS CDK]]></title>
            <link>https://towardsaws.com/multi-account-dns-with-aws-cdk-2cf525e53aa8?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cf525e53aa8</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[dns]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Mon, 10 Feb 2025 15:39:07 GMT</pubDate>
            <atom:updated>2025-02-27T05:32:59.897Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L8BMhD8WHj5Vld0e" /><figcaption>Photo by <a href="https://unsplash.com/@levifrey?utm_source=medium&amp;utm_medium=referral">Levi Frey</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I was recently tasked with setting up DNS within an AWS organisation structure. The idea was to use a single domain structure that would be able to support multiple environments (development, acceptance and production). In my actual use case it was a bit more complex, but in this post I’ll keep it simple and show some example code how you can implement the solution with <a href="https://github.com/aws/aws-cdk">AWS CDK</a>.</p><h3>Setting the stage</h3><p>In this post, we’ll look at setting up multi-account DNS for our AWS accounts. It starts with having a central (network) account that owns the DNS for the root domain. ( exampledomain.com ). To let teams create their applications, manage their subdomains, and request SSL certificates from ACM (AWS Certificate Manager), we wanted to let them own and manage the subdomain for a specific environment. In essence, we would have 4 AWS accounts with their own DNS management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAXJJ0FxtKmRaH9cubzlUQ.jpeg" /></figure><h3>Creating and Managing Hosted Zones</h3><p>Implementing DNS in AWS is handled by the Amazon Route 53 service. To implement DNS in Route 53 we need to use a Route 53 hosted zone. A hosted zone is a container within Route 53 where you store and manage the DNS records for a specific domain or subdomain. There are two types of hosted zones:</p><ul><li><strong>Public Hosted Zone</strong> : Used to manage the DNS records for a domain or subdomain that is publicly accessible on the internet. Example: A website like <a href="https://exampledomain.com/">exampledomain.com</a>.</li><li><strong>Private Hosted Zone</strong> : Used to manage DNS records for a domain within a Virtual Private Cloud (VPC). Example: Internal services accessed only within a private network, like <a href="https://internaldomain.com/">internaldomain.com</a>.</li></ul><p>In our case, we want to create a public hosted zone for our top-level domain. To do that with AWS CDK you can use the <strong>PublicHostedZone</strong> construct.</p><pre>const parentHostedZone = new PublicHostedZone(this, <br>  &#39;parent-hosted-zone&#39;, { zoneName: &#39;exampledomain.com&#39;,}<br>);</pre><p>The above code should be part of the stack deployed to the central (networking) account. Now lets look at what is being created when we created our public hosted zone.</p><h3>Types of DNS records in Route53</h3><p>When you create a public hosted zone, you will by default get two DNS records as part of the hosted zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_pdKs4hzohBB7g-OXXSmtw.png" /></figure><p>The hosted zone will contain both a NS record as well as a SOA record, but what do these records mean?</p><ul><li><strong>NS record</strong> — NS stands for ‘nameserver,’ and <strong>the nameserver record indicates which DNS server is authoritative for that domain</strong> (i.e. which server contains the actual DNS records). NS records tell the Internet where to go to find out a domain’s IP address.</li><li><strong>SOA record</strong> — The DNS start of authority (SOA) record stores important information about a domain or zone such as the email address of the administrator, when the domain was last updated, and how long the server should wait between refreshes.</li></ul><p>To tell the client (for instance a web browser) where to resolve the subdomain, we can explicitly define in route53 that the authoritative domain servers for the subdomain are located elsewhere by adding an explicit NS record to the hosted zone of the root/parent domain.</p><h3>Connecting the hosted zone in the central account with the hosted zone in the child account</h3><p>To manage the DNS for the subdomain in the environment-specific account, we will need to also create a route 53 hosted zone in the environment specific (dev, staging, prod) account. Our next would be to specify that the environment specific account is responsible for the sub-domain we will need to explicitly state the relationship in the Route53 hosted zone of the central account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBH61N6teWplMm18q0dsBA.jpeg" /></figure><p>The following table shows an example of how to define the authoritative NS servers for the subdomain dev.exampledomain.com. In the example we show a single value for the NS server, but when working with route53 you will always have four DNS servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*lca41mPGGmnjabi8LCZ0eg.png" /></figure><p>Now, when a client tries to resolve the DNS for <strong>dev.exampledomain.com</strong> , it will first try to find the NS servers for <strong>.com</strong> , after which it will try to resolve <strong>exampledomain.com,</strong> followed by resolving the <strong>dev.exampledomain.com</strong> before reaching the correct authoritative domain servers. Adding an additional NS server will cause one additional hop before it can resolve the DNS, so keep that in mind when implementing this solution.</p><p>We could create the NS record(s) for the dev or staging account in the public-hosted zones of the central account via the AWS console, but ideally, you would want to automate the entire process.</p><h3>Implementing Sub-zone DNS delegation</h3><p>As mentioned before we wanted teams to be able to manage sub-domain records for their accounts. This would allow them to create <strong>applicationname.dev.exampledomain.com</strong>. Having control over the DNS for the subdomain within the corresponding account will make it easy for them to create specific domain names for applications, or create DNS-validated HTTPS certificates (big plus for automation compared to email-validated certificates). To be able to do so in AWS CDK the easiest way is to create an IAM role that the Dev account can assume when trying to insert/update the NS records for its subdomain in the hosted zone of the parent domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-i_RWpiPqIgY4hXFuolcZA.jpeg" /></figure><p>To create such a role in the central account you can choose to do this by using an organization(al unit) id or by using an account id. Lets take a look at some examples.</p><pre>const role = new Role(this, &#39;ParentZoneOrganizationRole&#39;, { <br>    assumedBy: new OrganizationPrincipal(&#39;o-xxxxxxx&#39;),<br>    roleName: &#39;HostedZoneDelegationRole&#39;,<br>});</pre><p>In the above example, we specify that any account being part of a specific organization or organizational unit can assume the role to perform the changes on the hosted zone. However, if you want to be specific you can also opt for creating a specific role per account that is allowed to perform this kind of changes.</p><pre>const role = new Role(this, &#39;ParentZoneAccountRole&#39;, {<br>  assumedBy: new AccountPrincipal(&#39;123456789&#39;), <br>  roleName: &#39;dev-HostedZoneDelegationRole&#39;,<br>});</pre><p>For the role name in this example Ive used the prefix part of the subdomain, but you can of course also use the AWS account id or something that shows for which subdomain the role is meant. To allow the role to change the hosted zone in the networking account a <strong>grant</strong> can be given by means of the <strong>grantDelegation</strong> method on the HostedZone construct.</p><pre>parentHostedZone.grantDelegation(role);</pre><p>This will add the permissions to add or change records of type NS in the parent hosted zone. However, when inspecting policies I found that the IAM permissions were a bit too permissive for my liking. The following code snippet is taken from the AWS CDK v2 source code (at Feb 5th 2024):</p><pre>export function makeGrantDelegation(grantee: iam.IGrantable, hostedZoneArn: string): iam.Grant { <br><br>  const g1 = iam.Grant.addToPrincipal({ grantee, <br>      actions: [&#39;route53:ChangeResourceRecordSets&#39;],<br>      resourceArns: [hostedZoneArn], conditions: { <br>        &#39;ForAllValues:StringEquals&#39;: {<br>            &#39;route53:ChangeResourceRecordSetsRecordTypes&#39;: [&#39;NS&#39;], <br>            &#39;route53:ChangeResourceRecordSetsActions&#39;: [&#39;UPSERT&#39;, &#39;DELETE&#39;], <br>        }, <br>      }, <br>  }); <br><br>  const g2 = iam.Grant.addToPrincipal({ grantee, <br>    actions: [&#39;route53:ListHostedZonesByName&#39;], <br>    resourceArns: [&#39;*&#39;], <br>  }); <br><br>  return g1.combine(g2);<br>}</pre><p>As you can see the role allows all accounts (when used with an OrganizationPrincipal) to assume the role and allows these accounts to update NS records in the parent hosted zone without any strict validation on the subdomain (it does for record type). In my case, I want to limit that to a specific subdomain so that only the dev account can change the NS records in the parent zone for the <em>dev.exampledomain.com</em> subdomain and not for instance change the NS records for the <em>prod.exampledomain.com</em> subdomain. So how can we achieve this?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*gHP39CHkn-h0jqwGKX5vzQ.jpeg" /></figure><h3>Limiting the delegation scope</h3><p>To prevent this, a custom IAM policy needs to be created for the role that limits the scope to a specific subdomain.</p><pre>// Validate this can only happen for the dev.exampledomain.com sub-domain<br><br>const conditions = { <br>  &#39;ForAllValues:StringEquals&#39;: {<br>    &#39;route53:ChangeResourceRecordSetsRecordTypes&#39;: [&#39;NS&#39;],<br>    &#39;route53:ChangeResourceRecordSetsActions&#39;: [&#39;UPSERT&#39;,&#39;DELETE&#39;],<br>    &#39;route53:ChangeResourceRecordSetsNormalizedRecordNames&#39;: [&#39;dev.exampledomain.com&#39;] <br>  } <br>}; <br>// Allow the role to perform the GetHostedZone and ChangeResourceRecordSets methods on the recordset for the subdomain. <br><br>const recordSetPolicyStatement = new iam.PolicyStatement({<br>  actions: [&quot;route53:GetHostedZone&quot;, &quot;route53:ChangeResourceRecordSets&quot;], <br>  resources: [parentDomainHostedZone.hostedZoneArn], <br>  conditions: conditions <br>}); <br><br>// Allow the role to list hosted zones by name <br>const zoneListingPolicyStatement = new iam.PolicyStatement({  <br>  actions: [&quot;route53:ListHostedZonesByName&quot;], <br>  resources: [&quot;*&quot;] <br>}); <br><br>const policyDocument = new iam.PolicyDocument({ <br>  statements: [recordSetPolicyStatement, zoneListingPolicyStatement] <br>});</pre><p>Now that we’ve set up our policy all we need to do is assign that the IAM role.</p><pre>// Set explicit inline policies for the <br>const role = new Role(this, &#39;ParentZoneAccountRole&#39;, {<br>  assumedBy: new AccountPrincipal(&#39;123456789&#39;), <br>  roleName: &#39;dev-HostedZoneDelegationRole&#39;, <br>  inlinePolicies: { <br>    delegation: policyDocument <br>  }<br>});</pre><p>That it for the work we need to do in the central account. Now lets move on to what we need to do in the sub account.</p><h3>DNS in the sub-account</h3><p>In the sub accounts (dev, staging, prod) we will need to create a public hosted zone so we can manage subdomains for the applications living in the accounts.</p><pre>const subDomainHostedZone = new PublicHostedZone(this, &#39;subDomainHostedZone&#39;, { <br>  zoneName: &#39;dev.exampledomain.com&#39;<br>});</pre><p>Now that we’ve created the hosted zone we want to publish the NS record information into the central hosted zone, so we can perform the NS record delegation.</p><pre>// construct the ARN for our cross account role <br>const delegationRoleArn = Stack.of(this).formatArn({ <br>  account: rootAccountId, <br>  region: &#39;&#39;, <br>  resource: &#39;role&#39;, <br>  resourceName: &#39;dev-HostedZoneDelegationRole&#39;, <br>  service: &#39;iam&#39;, <br>}); <br>// Get the role by ARN <br>const delegationRole = Role.fromRoleArn(this, &#39;DelegationRole&#39;, delegationRoleArn); <br>// create a cross account hosted zone delegation record (NS) <br><br>new CrossAccountZoneDelegationRecord(this, &#39;DelegationRecord&#39;, { <br>  delegationRole, <br>  subDomainHostedZone, <br>  parentHostedZoneName: &#39;exampledomain.com&#39; <br>});</pre><p>The <a href="https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_route53.CrossAccountZoneDelegationRecord.html">CrossAccountZoneDelegationRecord</a> is a <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudformation-customresource.html">CloudFormation CustomResource</a> that will create assume the role and create the NS records in the hosted zone of the central account based on the NS servers of the hosted zone for the subdomain.</p><p>Now that we have the hosted zone in place for the sub-account it should be a matter of some simple lines of CDK code for generating DNS records and certificates for applications.</p><pre>const certificate = new Certificate(this, &#39;appSubDomainHostedZoneCert&#39;, { <br>  domainName: `applicationname.dev.exampledomain.com`,<br>  validation: CertificateValidation.fromDns(subDomainHostedZone),<br>});</pre><h3>Summary</h3><p>Implementing DNS management in a multi-account setup can be a bit challenging at first if you’ve never done this before. Using of AWS CDK you can add the required constructs that will allow you to perform sub-zone delegation. Limiting down the scope of what the cross-account role can do takes a bit more effort. While writing this post I learned there is an open issue for implementing a similar behaviour for permission limitation registered in the AWS CDK project <a href="https://github.com/aws/aws-cdk/issues/28078">https://github.com/aws/aws-cdk/issues/28078</a>. Lets hope it becomes part of the CDK, so it will save us some time.</p><p><em>Originally published at </em><a href="https://dev.to/aws-builders/multi-account-dns-with-aws-cdk-4dp2"><em>https://dev.to</em></a><em> on February 10, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cf525e53aa8" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/multi-account-dns-with-aws-cdk-2cf525e53aa8">Multi-account DNS with AWS CDK</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Journey with AWS CDK and Java: What You Need to Know]]></title>
            <link>https://towardsaws.com/my-journey-with-aws-cdk-and-java-what-you-need-to-know-885628ffbafb?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/885628ffbafb</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[cdk]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Mon, 02 Sep 2024 09:04:44 GMT</pubDate>
            <atom:updated>2024-09-06T11:19:48.047Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M8kL2MlLhd6A_41r" /><figcaption>Photo by <a href="https://unsplash.com/@neom?utm_source=medium&amp;utm_medium=referral">NEOM</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the first decisions you’ll need to make when working with the <a href="https://aws.amazon.com/cdk/">AWS Cloud Development Kit (CDK)</a> is choosing the language for writing your Infrastructure as Code (IaC). The CDK currently supports TypeScript, JavaScript, Python, Java, C#, and Go. Over the past few years, I’ve worked with the CDK in Typescript, Python and Java. While there is ample information available online for TypeScript and Python, this post aims to share my experience using Java as the language of choice for the AWS CDK.</p><h3>Wait…. what? Using Java with the AWS CDK?</h3><p>Some may say that Typescript is the most obvious language to use while working with the AWS CDK. The CDK itself is written in Typescript and it’s also the most used language <a href="https://matthewbonig.com/posts/community-survey-2023/">according to the 2023 CDK Community Survey</a>. Java is coming in 3rd place with a small percentage of use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*lB9Ue_2xHuGUf9Ee.png" /></figure><p>Image source <a href="https://matthewbonig.com/posts/community-survey-2023/">https://matthewbonig.com/posts/community-survey-2023/</a></p><p>I do wonder if this still holds true given the number of responses to the survey. I’ve worked with small businesses and large enterprise organizations over the last years, and I see more and more Java oriented teams move their workloads to AWS while adopting AWS CDK as their Infrastructure as Code tool. Depending on the type of service(s) being built by these teams they may or may not have any experience with either Python or Typescript and the Node.js ecosystem, which makes sticking to Java an easy choice.</p><h3>General observations</h3><p>From what I’ve seen, adopting the CDK in Java is relatively easy for most of these teams as they already understand the language and the ecosystem. Integrating the CDK with their existing build tools like <a href="https://maven.apache.org/">Maven</a> and <a href="https://gradle.org/">Gradle</a> is well documented, which leaves them with the learning curve of understanding how to work with infrastructure as code, how to structure a CDK project and when to use <a href="https://docs.aws.amazon.com/cdk/v2/guide/constructs.html">L1, L2 and L3 constructs</a>.</p><p>Compared to Typescript the CDK stacks and constructs written in Java contain a bit more boilerplate code and therefor might feel a bit more bloated if you come from a different language. I personally don’t feel this makes the code less readable and with modern IDE’s and coding assistants I don’t feel I’m less productive.</p><p>The CDK also seems to become more widely adopted in the Java community with more recent Java frameworks like <a href="https://micronaut.io/">Micronaut</a> even having built-in support for AWS CDK in the framework.</p><p>See for instance the following Micronaut launch configurations:</p><ul><li><a href="https://micronaut.io/launch?type=DEFAULT&amp;javaVersion=JDK_11&amp;features=aws-lambda&amp;features=aws-cdk&amp;features=amazon-api-gateway">Micronaut Application with API Gateway and CDK for Java runtime</a></li><li><a href="https://micronaut.io/launch?type=FUNCTION&amp;javaVersion=JDK_11&amp;features=aws-lambda&amp;features=aws-cdk&amp;features=amazon-api-gateway">Micronaut Function with API Gateway and CDK for Java runtime</a></li></ul><p>One of the advantages of Java is that it’s a statically typed language, which means it will catch most CDK coding errors during compile-time. There are still some errors which you will only see during an actual cdk synth or cdk deploy. For instance, some constructs have required properties which will only become visible if you try to synthesize the stack, but in my experience, you will have that in other languages as well.</p><p>Performance wise it feels like the CDK in Java is a bit slower compared to using it Typescript or any other interpreted language. I’ve not measured this, but it’s more of a gut feeling. This might have to do with the static nature of Java and its corresponding build tools and compile phase. On the other hand, it might be that the <a href="https://aws.github.io/jsii/overview/runtime-architecture/">JSII runtime architecture</a> also has an effect and how Java is interacting with a Javascript environment.</p><h3>Java Builders</h3><p>One of the biggest differences when using the AWS CDK with Java is the use of Builders. When creating constructs with Typescript you’re mainly using the <em>props</em> argument (map of configuration properties) while creating a construct. Let’s take a look at an example:</p><pre>const bucket = new s3.Bucket(this,&quot;MyBucket&quot;, { <br>  versioned: true, <br>  encryption: BucketEncryption.KMS_MANAGED <br>})</pre><p>The Java version of the above snippet uses a Builder class that follows the builder pattern for constructing the properties. If you’re unfamiliar with the Builder pattern in Java I recommend to checkout <a href="https://blogs.oracle.com/javamagazine/post/exploring-joshua-blochs-builder-design-pattern-in-java">this blog post</a> about using the Builder pattern. Depending on the CDK construct you might be able to define a CDK resource in two different ways.</p><p>In the first example you use the Builder for the Bucket properties.</p><pre>Bucket bucket = new Bucket(this, &quot;MyBucket&quot;, new BucketProps.Builder()<br>  .versioned(true) <br>  .encryption(BucketEncryption.KMS_MANAGED) <br>  .build()<br>);</pre><p>The alternative is that constructs can have their own builder class, which makes it a little less verbose and easier to read.</p><pre>Bucket bucket = Bucket.Builder <br>  .create(this, &quot;MyBucket&quot;) <br>  .versioned(true) <br>  .encryption(BucketEncryption.KMS_MANAGED) <br>  .build();</pre><h3>IDE support</h3><p>Overall, IDE support is really great when working with CDK in Java. I use IntelliJ IDEA on a daily basis and auto completion really helps when using the Builder objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z9STIheAOdmCnSxJ" /></figure><p>As the CDK documentation is also inside the CDKs Java source code, looking up documentation is really easy. It’s similar to how you would do it with any kind of other object or library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lUQu0CZqOBOqFUBs" /></figure><h3>Third party construct support</h3><p>The CDK itself is written in Typescript and for each supported programming language a specific binding is generated. This means that when a new resource or feature for an AWS service is added in the Typescript variant of the CDK it’s also available to developers using a Java based CDK.</p><p>Besides the default CDK constructs there are also a lot of community generated constructs. <a href="https://constructs.dev/">Construct Hub</a> is a great place to find them.</p><p>From what I’ve seen most constructs coming out of AWS will support Java as one of the default languages. Community supported constructs however might not. There are several popular constructs that only support Typescript and Python. Filtering on Construct Hub for AWS CDK v2 based constructs, sorted by programming languages results in the following data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0755ab820ec80e9e2712466ccb31f1ab/href">https://medium.com/media/0755ab820ec80e9e2712466ccb31f1ab/href</a></iframe><p>Depending on the type of infrastructure or third-party services you’re planning to use, you might not be able to use all available constructs. For instance, the constructs maintained by <a href="https://constructs.dev/packages/datadog-cdk-constructs-v2/">DataDog</a> are only available in Typescript, Python and Go. In my personal experience though, most construct developers are open to support Java. Third party constructs are based on <a href="https://projen.io/">projen</a> and <a href="https://aws.github.io/jsii/">jsii</a>, which means that adding a Java based version is most of the time a matter of configuration in the package.json file of the project.</p><pre> &quot;jsii&quot;: {<br>    &quot;outdir&quot;: &quot;dist&quot;,<br>    &quot;targets&quot;: {<br>      &quot;java&quot;: {<br>        &quot;package&quot;: &quot;io.github.cdklabs.cdknag&quot;,<br>        &quot;maven&quot;: {<br>          &quot;groupId&quot;: &quot;io.github.cdklabs&quot;,<br>          &quot;artifactId&quot;: &quot;cdknag&quot;<br>        }<br>      },<br>      &quot;python&quot;: {<br>        &quot;distName&quot;: &quot;cdk-nag&quot;,<br>        &quot;module&quot;: &quot;cdk_nag&quot;<br>      },<br>      &quot;dotnet&quot;: {<br>        &quot;namespace&quot;: &quot;Cdklabs.CdkNag&quot;,<br>        &quot;packageId&quot;: &quot;Cdklabs.CdkNag&quot;<br>      },<br>      &quot;go&quot;: {<br>        &quot;moduleName&quot;: &quot;github.com/cdklabs/cdk-nag-go&quot;<br>      }<br>    },<br>    &quot;tsc&quot;: {<br>      &quot;outDir&quot;: &quot;lib&quot;,<br>      &quot;rootDir&quot;: &quot;src&quot;<br>    }<br>  }</pre><p>(An example of how JSII is configured for the <a href="https://github.com/cdklabs/cdk-nag">CDK NAG project</a>)</p><p>Once the configuration is in place and the artifacts have been pushed to for instance Maven Central, you’re good to go.</p><p>When thinking about it, I once had a 3rd party construct I wanted to use that did not support Java (yet). It got added quite quickly and there was also an alternative solution for it, so I can’t remember having issues with the lower number of available constructs.</p><h3>Examples, tutorials and documentation</h3><p>I think it’s good to reflect on the fact that there are more CDK examples and tutorials available in Typescript and Python compared to Java. This reflects the findings in the usage chart from the CDK Community Survey. However, reading Typescript as a Java programmer is relatively easy (my personal opinion). If you’re new to the AWS CDK there is a ton of example code available on Github, Youtube, and in numerous blog posts and tutorials. If you’re already using the CDK in combination with Java, be sure to write some blog posts or tutorials, so others can see that and benefit from your knowledge!</p><h3>Summary</h3><p>Java is a very viable option when working with the AWS CDK, especially for workload teams already familiar with the language and its ecosystem. IDE support for the CDK is excellent with features like auto-completion and easy access to source code documentation.</p><p>All in all, the experience is really good. Keep in mind that picking Java for your infrastructure as code all depends on the context and the environment you’re in. I would suggest picking the language which is most applicable in your specific situation. If you still need to make the choice and are already working with Java, I would definitely recommend trying it out!</p><p><em>Originally published at </em><a href="https://www.jeroenreijn.com/2024/09/my-journey-with-aws-cdk-and-java-what-you-need-to-know.html"><em>https://www.jeroenreijn.com</em></a><em> on September 2, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=885628ffbafb" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/my-journey-with-aws-cdk-and-java-what-you-need-to-know-885628ffbafb">My Journey with AWS CDK and Java: What You Need to Know</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyze and debug Quarkus based AWS Lambda functions with X-Ray]]></title>
            <link>https://towardsaws.com/analyze-and-debug-quarkus-based-aws-lambda-functions-with-x-ray-94a386269f8d?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/94a386269f8d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[quarkus]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Tue, 06 Feb 2024 21:30:12 GMT</pubDate>
            <atom:updated>2024-02-07T05:07:23.333Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XYMp9i-BWsvkwIrU" /><figcaption>Photo by <a href="https://unsplash.com/@marilezhava?utm_source=medium&amp;utm_medium=referral">mari lezhava</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Using Quarkus and AWS X-Ray with Jakarta CDI Interceptors to keep your code clean</h3><p>Serverless architectures have emerged as a paradigm-shifting approach to building, fast, scalable and cost efficient applications. While Serverless architectures provide unparalleled flexibility, they also introduce new challenges in terms of monitoring and troubleshooting.</p><p>In this blog post, we’ll explore how Quarkus integrates with AWS X-Ray and how using a Jakarta CDI Interceptor can keep your code clean while adding custom instrumentation.</p><h3>Quarkus and AWS Lambda</h3><p><a href="https://quarkus.io/">Quarkus</a> is a Java based framework tailored for GraalVM and HotSpot, which results in an amazingly fast boot time while having an incredibly low memory footprint. It offers near instant scale up and high density memory utilization which can be very useful for container orchestration platforms like Kubernetes or Serverless runtimes like AWS Lambda.</p><p>Building AWS Lambda Functions can be as easy as <a href="https://code.quarkus.io/">starting a Quarkus project</a>, adding the quarkus-amazon-lambda dependency, and defining your AWS Lambda Handler function.</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.quarkus&lt;/groupId&gt;<br>    &lt;artifactId&gt;quarkus-amazon-lambda&lt;/artifactId&gt;<br>&lt;/dependency&gt;</pre><p>An extensive guide on how to develop AWS Lambda Functions with Quarkus can be found in the official <a href="https://quarkus.io/guides/aws-lambda">Quarkus AWS Lambda Guide</a>.</p><h3>Enabling X-Ray for your Lambda functions</h3><p>Quarkus provides out of the box support for X-Ray, but you will need to add a dependency to your project and configure some setting to make it work with GraalVM / native compiled Quarkus applications. Let’s first start with adding the quarkus-amazon-lambda-xray dependency.</p><pre>&lt;!-- adds dependency on required x-ray classes and adds support for graalvm native --&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.quarkus&lt;/groupId&gt;<br>    &lt;artifactId&gt;quarkus-amazon-lambda-xray&lt;/artifactId&gt;<br>&lt;/dependency&gt;</pre><p>Don’t forget to enable tracing for your Lambda function otherwise it won’t work. An example of doing that is by setting the tracing argument to <strong>active</strong> within your AWS CDK code.</p><pre>function = Function.Builder.create(this, &quot;feed-parsing-function&quot;)<br>      ...<br>      .memorySize(512)<br>      .tracing(Tracing.ACTIVE)<br>      .runtime(Runtime.PROVIDED_AL2023)<br>      .logRetention(RetentionDays.ONE_WEEK)<br>      .build();</pre><p>After the deployment of your function, and a function invocation, you should be able to see the X-Ray traces from within the Cloudwatch interface. By default it will show you some basic timing information for your function like the initialization and the invocation duration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*iOBjafgpO1W98bZe.png" /></figure><h3>Adding more instrumentation</h3><p>Now that the dependencies are in place and tracing is enabled for our function we can enrich the traces in X-Ray by leveraging the X-Ray SDKs TracingIntercepter . For instance for the SQS and DynamoDB client you can explicitly set the intercepter inside the <strong>application.properties</strong> file.</p><pre>quarkus.dynamodb.async-client.type=aws-crt <br>quarkus.dynamodb.interceptors=com.amazonaws.xray.interceptors.TracingInterceptor <br>quarkus.sqs.async-client.type=aws-crt <br>quarkus.sqs.interceptors=com.amazonaws.xray.interceptors.TracingInterceptor</pre><p>After putting these properties in place, redeploying and executing the function, the TracingIntercepter will wrap around each API call to SQS and DynamoDB and store the actual trace information along side the trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*mm-4sSXj-jCS6yn1.png" /></figure><p>This is very useful for debugging purposes as it will allow you to validate your code and check for any mistakes. Requests to AWS Services are part of the pricing model, so if you make a mistake in your code and you make too many calls it can become quite costly.</p><h3>Luc van Donkersgoed on Twitter: &quot;How a single-line bug cost us $2000 in AWS spend...We recently refactored a Lambda Function. We extensively tested its functionality and released it into production. And everything still worked as expected. But then the billing alarm went off..(repost with sanitized images) pic.twitter.com/CzR9LxLyxD / Twitter&quot;</h3><p>How a single-line bug cost us $2000 in AWS spend...We recently refactored a Lambda Function. We extensively tested its functionality and released it into production. And everything still worked as expected. But then the billing alarm went off..(repost with sanitized images) pic.twitter.com/CzR9LxLyxD</p><h3>Custom subsegments</h3><p>With the AWS SDK TracingInterceptor configured we get information about the calls to the AWS APIs, but what if we want to see information about our own code or remote calls to services outside of AWS?</p><p>The <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java.html">Java SDK for X-Ray</a> supports the concept of adding custom subsegments to your traces. You can add subsegments to a trace by adding a few lines of code to your own business logic as you can see in the following code snippet.</p><pre>public void someMethod(String argument)  {<br>  // wrap in subsegment<br>  Subsegment subsegment = AWSXRay.beginSubsegment(&quot;someMethod&quot;);<br>  try {<br>      // Your business logic<br>  } catch (Exception e) {<br>     subsegment.addException(e);<br>     throw e;<br>  } finally {<br>     AWSXRay.endSubsegment();<br>  }<br>}</pre><p>Although this is trivial to do, it will become quit messy if you have a lot of methods you want to apply tracing to. This isn’t ideal and it would be better of we don’t have to mix our own code with the X-Ray instrumentation.</p><h3>Quarkus and Jakarta CDI Interceptors</h3><p>The Quarkus programming model is based on the Lite version of the Jakarta Contexts and Dependency Injection 4.0 specification. Besides dependency injection the specification also describes other features like:</p><ul><li><strong>Lifecycle Callbacks</strong> — A bean class may declare lifecycle @PostConstruct and @PreDestroy callbacks.</li><li><strong>Interceptors</strong> — used to separate cross-cutting concerns from business logic.</li><li><strong>Decorators</strong> — similar to interceptors, but because they implement interfaces with business semantics, they are able to implement business logic.</li><li><strong>Events and Observers</strong> — Beans may also produce and consume events to interact in a completely decoupled fashion.</li></ul><p>As mentioned, CDI Interceptors are used to separate cross-cutting concerns from business logic. As tracing is a cross-cutting concern this sounds like a great fit. Let’s take a look at how we can create an interceptor for our AWS X-Ray instrumentation.</p><p>We start with defining our interceptor binding which we will call XRayTracing. Interceptor bindings are intermediate annotations that may be used to associate interceptors with target beans.</p><pre>package com.jeroenreijn.aws.quarkus.xray;<br><br>import jakarta.annotation.Priority;<br>import jakarta.interceptor.InterceptorBinding;<br><br>import java.lang.annotation.Retention;<br><br>import static java.lang.annotation.RetentionPolicy.RUNTIME;<br><br>@InterceptorBinding<br>@Retention(RUNTIME)<br>@Priority(0)<br>public @interface XRayTracing {<br>}</pre><p>The next step is to define the actual Interceptor logic, the code that will add the additional X-Ray instructions for creating the subsegment and wrapping it around our business logic.</p><pre>package com.jeroenreijn.aws.quarkus.xray;<br><br>import com.amazonaws.xray.AWSXRay;<br>import jakarta.interceptor.AroundInvoke;<br>import jakarta.interceptor.Interceptor;<br>import jakarta.interceptor.InvocationContext;<br><br>@Interceptor<br>@XRayTracing<br>public class XRayTracingInterceptor {<br><br>    @AroundInvoke<br>    public Object tracingMethod(InvocationContext ctx) throws Exception {<br>        AWSXRay.beginSubsegment(&quot;## &quot; + ctx.getMethod().getName());<br>        try {<br>            return ctx.proceed();<br>        } catch (Exception e) {<br>            AWSXRay.getCurrentSubsegment().addException(e);<br>            throw e;<br>        } finally {<br>            AWSXRay.endSubsegment();<br>        }<br>    }<br>}</pre><p>An important part of the interceptor is the @AroundInvoke annotation, which means that this interceptor code will be wrapped around the invocation of our own business logic.</p><p>Now that we’ve defined both our interceptor binding and our interceptor it’s time to start using it. Every method that we want to create a subsegment for, can now be annotated with the @XRayTracing annotation.</p><pre>@XRayTracing<br>public SyndFeed getLatestFeed() {<br>    InputStream feedContent = getFeedContent();<br>    return getSyndFeed(feedContent);<br>}<br><br>@XRayTracing<br>public SyndFeed getSyndFeed(InputStream feedContent) {<br>    try {<br>        SyndFeedInput feedInput = new SyndFeedInput();<br>        return feedInput.build(new XmlReader(feedContent));<br>    } catch (FeedException | IOException e) {<br>        throw new RuntimeException(e);<br>    }<br>}</pre><p>That looks much better. Pretty clean, if I say so myself.</p><p>Based on the hierarchy of subsegments for a trace, X-Ray will be able to show a nested tree structure with the timing information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*cHhB6SQ2Nk-aaUJU.png" /></figure><h3>Closing thoughts</h3><p>The integration between Quarkus and X-Ray is quite simple to enable. The developer experience is really good out of the box with defining the interceptors on a per client basis. With the help of CDI interceptors you can keep your code clean without worrying too much about X-Ray specific code inside your business logic.</p><p>An alternative to building your own Interceptor might be to start using <a href="https://docs.powertools.aws.dev/lambda/java/">AWS PowerTools for Lambda (Java)</a>. Powertools for Java is a great way to boost your developer productivity, but it can be used for more than X-Ray, so Ill save it for another post.</p><p><em>Originally published at </em><a href="https://dev.to/aws-builders/analyze-and-debug-quarkus-based-aws-lambda-functions-with-x-ray-3ke7"><em>https://dev.to</em></a><em> on February 6, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94a386269f8d" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/analyze-and-debug-quarkus-based-aws-lambda-functions-with-x-ray-94a386269f8d">Analyze and debug Quarkus based AWS Lambda functions with X-Ray</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution of the AWS Community Builders Blogs Twitter Bot Architecture]]></title>
            <link>https://towardsaws.com/the-evolution-of-the-aws-community-builders-blogs-twitter-bot-architecture-025c4098fa96?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/025c4098fa96</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Wed, 24 Jan 2024 06:22:03 GMT</pubDate>
            <atom:updated>2024-01-24T10:33:37.851Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A cup showing the twitter logo" src="https://cdn-images-1.medium.com/max/1024/0*MJxMVg6Fohjl-HMN" /><figcaption>Photo by <a href="https://unsplash.com/@ymoran?utm_source=medium&amp;utm_medium=referral">Yucel Moran</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In 2022, when I was accepted into the AWS Community Builders program, I was amazed by the abundance of high-quality content generated by the broader AWS community. While most community builders publish their blog content on personal websites, <a href="https://medium.com/">Medium</a>, or <a href="https://hashnode.com/">Hashnode</a>, it is common for this content to be (re)published on the <a href="http://dev.to/">dev.to</a> platform within the <a href="https://dev.to/aws-builders">AWS Community Builders organization</a>. It currently acts as a hub for blog content created by AWS community builders.</p><p>I didn’t frequently visit the <a href="https://dev.to/">dev.to</a> platform and also was not really using an RSS reader anymore, so I began exploring alternative solutions. Most of the interesting articles I usually find on Twitter/X. I sought to find an existing Twitter account sharing the community builders content but was unsuccessful. Being a builder myself, I decided to create a fresh Twitter account (<a href="https://twitter.com/aws_cb_blogs">@aws_cb_blogs</a>) and wanted to create a fully Serverless architecture. In this post I’ll explain how the architecture has evolved over time and what are some of the things to come.</p><h3>The initial idea</h3><p>As a first iteration I decided to put something together rather quickly. I needed three component:</p><ul><li>a component to read and parse the rss feed.</li><li>a component to turn posts into tweets.</li><li>a schedule to make sure new items are being picked up.</li></ul><p>To initiate the entire process I needed to have a solution similar to a <a href="https://en.wikipedia.org/wiki/Cron">cron</a> job (scheduled job). AWS offers this in the form of <a href="https://aws.amazon.com/blogs/compute/introducing-amazon-eventbridge-scheduler/">Amazon EventBridge Scheduler</a>. It’s a pretty straightforward service which allows you to easily create a scheduled call to more than 270 AWS services and over 6,000 API operations.</p><p>I decided to split the business logic of the application by responsibilities into different lambda functions and I ended up with the following architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UM9cClqGODvoW0HW.jpg" /></figure><p>The Twitter API requires authentication with keys and I needed to store them in a secure place. It’s better to not store them with(in) the Lambda, so I decided to put them into Systems Manager Parameter Store. For each cold start invocation of the Tweet function a call is made to parameter store to fetch the API credentials.</p><p>While building, I quickly realized I needed some persistent storage as the RSS feed could contain the same items during a consecutive fetch. I did not want the bot to repeat an already tweeted post so I needed some sort of storage for storing the tweetid for a specific post. I needed a simple persistent database and decided to use DynamoDB as the database for keeping track of the posts. The great thing about DynamoDB is that it’s a fully managed serverless databases and it offers a pay as you go model. It’s really cheap for most cases and I only pay $0.01 a month for using it for this application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CFELfUXwkf_KlWmx.jpg" /></figure><h3>Improving the design</h3><p>A few weeks later I decided that I wanted to improve the architecture a bit and decided to decouple the two Lambda components by means of an SQS queue. The advantage of using the SQS queue is that when the function fails the item(s) being processed is not removed from the queue and will be retried at a later moment in time. For consecutive failures I added a dead letter queue (DLQ), so in case the messages on the queue can’t be processed, they will be moved to the DLQ. If there is an error in the component or at the Twitter API I will be able to inspect the messages and re-drive them. You might think this is not necessary, but once Twitter announced their new <a href="https://www.theverge.com/2023/3/30/23662832/twitter-api-tiers-free-bot-novelty-accounts-basic-enterprice-monthly-price">API plans in March 2023</a> , the function that was responsible for tweeting was unable to send out tweets until I subscribed to the new usage plan. Because of the DLQ, it was a matter of re-driving the messages instead of trying to get the changes to propagate again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KlZKYWUWMwUBh9ya.jpg" /></figure><h3>New feature imposed more changes</h3><p>Some authors on the <a href="http://dev.to/">dev.to</a> platform have added their Twitter handle to their profile. I thought it would be great to be able to mention the author in the tweet when it’s send out. It gives the tweet a bit more body and the author is notified when the post hits the twitter platform.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/aws_cb_blogs/status/1749107205444444636%3Fs%3D20&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee691a06e56667cf16cb1cad1b694a4b/href">https://medium.com/media/ee691a06e56667cf16cb1cad1b694a4b/href</a></iframe><p>From the RSS feed I only get the <a href="https://dev.to/">dev.to</a> username of the author. The twitter handle is only available on the post itself, so I decided to add a third component to get the Twitter handle for the author.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6AHBPEkowZ7DM5ht.jpg" /></figure><h3>What’s next?</h3><p>This architecture has been running fine for quite some time now and I’ve even deployed the same stack for a bot for the posts by the AWS Heroes, but I still have some improvements in mind:</p><ol><li>Replacing the SQS queue for the posts by DynamoDB Streams. The author enrichment function would only have react to events in the stream. That would remove a bit of complexity and responsibility from the feed parser function.</li><li>If I look at the flow it looks a bit like a workflow. I’m considering to replace the entire thing and with a step functions workflow.</li><li>Integrate with Amazon Bedrock for generating summaries of the posts the bot is posting to Twitter. It would improve the Twitter posts a bit instead of just mentioning the author and the title of the post.</li><li>Extend the bot to also publish to other platforms. I’ve been looking into Threads, but it does not seem to have a public API yet. Not sure how big the AWS community is on other social platforms. Feel free to reach out if you prefer to follow all the great content on your social media platform of choice.</li></ol><p>In a follow up post I will dive into the technical details of how the functions work and will take a look at some of the best practices for building Lambda functions. If you have some comments/ideas/feedback, feel free to reach out!</p><p><em>Originally published at </em><a href="https://www.jeroenreijn.com/2024/01/evolution-of-the-aws-community-builders-blogs-twitter-bot-architecture.html"><em>https://www.jeroenreijn.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=025c4098fa96" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/the-evolution-of-the-aws-community-builders-blogs-twitter-bot-architecture-025c4098fa96">The Evolution of the AWS Community Builders Blogs Twitter Bot Architecture</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop and Test your Github Actions workflow locally with “act”]]></title>
            <link>https://jreijn.medium.com/develop-and-test-your-github-actions-workflow-locally-with-act-28d02dead87a?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/28d02dead87a</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[act]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 11:45:52 GMT</pubDate>
            <atom:updated>2023-07-11T06:37:49.818Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*65lgjYFJGoFhZrx2" /><figcaption>Photo by <a href="https://unsplash.com/@abzortu?utm_source=medium&amp;utm_medium=referral">Helio Dilolwa</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At work, I regularly train people on the subject of Continuous Integration and Continuous Delivery, where I predominantly utilize GitHub Actions for the workshop assignments. This choice is motivated by GitHub’s extensive adoption within the developer community and the generous offering of approximately 2000 minutes or 33 hours of free build time per month.</p><p>During one of my recent workshops, a participant raised a question regarding the possibility of locally testing workflows before pushing them to GitHub. They pointed out the inconvenience of waiting for a runner to pick up their pipeline or workflow, which negatively impacts the developer experience. At that time, I was unaware of any local options for GitHub Actions. However, I have since come across a solution called <a href="https://github.com/nektos/act">“act”</a> that addresses this issue.</p><h3>What is “act”?</h3><p><a href="https://github.com/nektos/act">“act”</a> is a command-line utility that emulates a Github Actions environment and allows you to test your Github Actions workflows on your developer laptop instead of in a Github Actions environment. You can install “act” by using for instance brew on the Mac.</p><pre>$ brew install act</pre><h3>Running Workflows Locally</h3><p>“act” enables you to execute and debug GitHub Actions workflows locally, providing a faster feedback loop during development. Running the “act” command line will pick up the workflows in your .github/workflows folder and try to execute them. Using &quot;act&quot; can be as simple as:</p><pre>$ act</pre><p>“act” uses Docker to create an isolated environment that closely resembles the GitHub Actions execution environment. This ensures consistency in the execution of actions and workflows. If you don’t have Docker installed you can use <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a> or use <a href="https://github.com/abiosoft/colima">Colima</a>, an easy way to run container runtimes on macOS.</p><h3>Runners</h3><p>While defining your workflow you can specify a runner based on a specific virtual machine/environment when performing your steps.</p><p>By default, “act” has a mapping to a specific docker image when you specify the ubuntu-latest runner. When running &quot;act&quot; for the first time it will ask you to pick a default image for ubuntu-latest. You can choose from 3 types of base images that can be mapped to ubuntu-latest:</p><ul><li>Micro Docker Image — <a href="https://hub.docker.com/_/buildpack-deps">node:16-buster-slim</a></li><li>Medium Docker Image — <a href="https://github.com/catthehacker/docker_images)">catthehacker/ubuntu:act-latest</a></li><li>Large Docker Image — <a href="https://github.com/catthehacker/docker_images">catthehacker/ubuntu:full-latest</a></li></ul><p>Don’t worry if you’re not happy with the one you selected, you can always change the default selection by changing the following file in your users home directory ~/.actrc.</p><p>The large docker image is around <strong>18GB 😲</strong>, so I initially picked the medium-sized image as it should contain most of the commonly used system dependencies. I soon learned that it contains quite some libraries, but when I tried to run a Java + Maven-based project I learned that it did not contain Apache Maven, while the normal ubuntu-latest on GitHub does have that.</p><pre>[CI/Build] ⭐ Run Main Build <br>[CI/Build] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir= | /var/run/act/workflow/2: line 2: mvn: command not found [CI/Build] ❌ Failure - Main Build <br>[CI/Build] exitcode &#39;127&#39;: command not found, please refer to https://github.com/nektos/act/issues/107 for more information</pre><p>I didn’t want to switch to an 18GB docker image to be able to just run Maven, so I ended up finding an existing image by <a href="https://github.com/jamezp">Jamez Perkins</a>. It simply takes the original “act” image <a href="http://ghcr.io/catthehacker/ubuntu:act-latest">ghcr.io/catthehacker/ubuntu:act-latest</a> and adds Maven version 3.x to it. You can easily specify running your workflow with custom images by providing the platform parameter.</p><pre>$ act -P ubuntu-latest=quay.io/jamezp/act-maven</pre><p>After using that image my workflow ran without any errors.</p><h3>Working with multiple jobs/stages</h3><p>Your GitHub Actions workflow usually consists of one or more jobs that separate different stages of your workflow. You might for instance have a Build, Test and Deploy stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xcwu92KZodry5FK-" /></figure><p>Usually, you build your application in the build job and use the resulting artifact in the deploy job. Jobs can run on different runners, so in a GitHub Actions environment, you will probably be using the <a href="https://github.com/actions/upload-artifact">upload</a>/<a href="https://github.com/actions/download-artifact">download</a> artifact action which will use centralized storage for sharing the artifacts between different runners. When using “act” and sharing artifacts you will need to be specific about where the artifacts need to be stored. You can do so by providing a specific parameter named --artifact-server-path.</p><pre>$ act -P ubuntu-latest=quay.io/jamezp/act-maven <br>--artifact-server-path /tmp/act-artifacts</pre><h3>Working with secrets</h3><p>It’s a good practice to always separate your secrets from your workflow definition and only reference them from a specific secret store. When using GitHub Actions you can store your secrets in the built-in secret management functionality.</p><p>To provide an action with a secret, you can use the secrets context to access secrets you&#39;ve created in your repository.</p><pre>jobs:<br>  staticanalysis:<br>    runs-on: ubuntu-latest<br>    steps:<br>    - uses: actions/checkout@v3<br>      with:<br>        # Disabling shallow clone is recommended for improving relevancy of reporting<br>        fetch-depth: 0<br>    - name: SonarQube Scan<br>      uses: sonarsource/sonarqube-scan-action@master<br>      env:<br>        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>        SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}</pre><p>“act” does not have a UI in which you can specify secrets, so you will need to provide those values explicitly from the command line or store them in a .env formatted file when testing your workflow. If you only have a few secrets you can easily add them by just providing the secret from the command line by using the -s option.</p><pre>$ act -s SONAR_TOKEN=somevalue <br>$ act --secret-file my.secrets</pre><h3>Working with environment variables</h3><p>Similar to secrets you sometimes make use of environment variables inside your workflow. For a single environment variable you can use --env myenv=foo or if you have a set of environment variables you can create a dotenv file and provide a reference to the file from the CLI by providing the --env-file parameter.</p><pre>$ act --env-file my.env</pre><p>The .env file is based on a simple standard file format which contains a set of key-value pairs divided by new lines.</p><pre>MY_ENV_VAR=MY_ENV_VAR_VALUE <br>MY_2ND_ENV_VAR=&quot;my 2nd env var value&quot;</pre><h3>Event simulation</h3><p>Events are a fundamental part of workflows. Workflows will start due to some specific event happening within Github like a push, creation of a pull request, etc. With “act” you can simulate such an event to trigger your workflow(s). You can provide the event as an argument.</p><pre>$ act pull_request</pre><p>Events are usually more complex than just a simple string so if you want to be specific you can provide a reference to an event payload:</p><pre>$ act --eventpath pull_request.json</pre><pre>{<br>  &quot;pull_request&quot;: {<br>    &quot;head&quot;: {<br>      &quot;ref&quot;: &quot;sample-head-ref&quot;<br>    },<br>    &quot;base&quot;: {<br>      &quot;ref&quot;: &quot;sample-base-ref&quot;<br>    }<br>  }<br>}</pre><p>By providing your events from the command line you can test different scenarios and observe how your workflows respond to those events.</p><h3>Summary</h3><p>Using “act” is straightforward and can significantly help in the initial phase of developing your workflow. “act” offers a significant advantage in terms of a swift feedback loop. It enables developers to perform tests locally and iterate rapidly until they achieve the desired outcome, eliminating the need to wait for GitHub’s runners to finish the workflow.</p><p>“act” additionally aids developers in avoiding resource wastage on GitHub’s runners. By conducting local tests, developers can ensure the proper functioning of their workflows before pushing code changes to the repository and initiating a workflow on GitHub’s runners.</p><p>If you’re working with GitHub Actions I would recommend to asses “act” as a tool for your development team.</p><p><em>Originally published at </em><a href="https://www.luminis.eu/blog/develop-and-test-your-github-actions-workflow-locally-with-act/"><em>https://www.luminis.eu/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28d02dead87a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improve AWS Security and Compliance with cdk-nag]]></title>
            <link>https://aws.plainenglish.io/improve-aws-security-and-compliance-with-cdk-nag-f451c54b456f?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f451c54b456f</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws-cdk]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cdk]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Thu, 13 Apr 2023 06:16:47 GMT</pubDate>
            <atom:updated>2023-04-13T06:16:47.089Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OKMGQim52aC4Erx3" /></figure><p><a href="https://aws.amazon.com/cdk/">AWS Cloud Development Kit (AWS CDK)</a> is a powerful tool that allows developers to define cloud infrastructure in code using familiar programming languages like TypeScript, Python, and Java. However, as with any infrastructure-as-code tool, it’s important to ensure that the resulting infrastructure adheres to security and compliance best practices. This is where cdk-nag comes in.</p><h3>What is cdk-nag ?</h3><p><a href="https://github.com/cdklabs/cdk-nag">cdk-nag</a> is an open-source tool that provides automated checks for AWS CDK code and the resulting Cloudformation templates to help ensure that they adhere to security and compliance best practices.</p><p>After adding cdk-nag to your project it checks for a variety of known security and compliance issues including overly-permissive IAM policies, missing access logs and unintended public s3 buckets. cdk-nag also checks for common mistakes that can lead to security vulnerabilities, such as the use of plain text passwords and the use of default security groups.</p><p>The great thing about cdk-nag is that it allows you to catch mistakes at a very early stage in the process. Ideally, you can catch them while developing your infrastructure as code in CDK on your local machine. As an alternative, you can add cdk-nag to your CI/CD pipeline and make the build fail in case of any issues.</p><h3>Adding cdk-nag to your project</h3><p>Using cdk-nag is simple. First, add it as a dependency to your AWS CDK project. If you’re using Java you can add it to your pom.xml file.</p><pre>&lt;dependency&gt;<br>  &lt;groupId&gt;io.github.cdklabs&lt;/groupId&gt;<br>  &lt;artifactId&gt;cdknag&lt;/artifactId&gt;<br>  &lt;version&gt;2.25.2&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>After you’ve added the dependency you will need to explicitly enable cdk-nag utilizing a <a href="https://docs.aws.amazon.com/cdk/v2/guide/aspects.html">CDK aspect</a>. You can apply cdk-nag in the scope of your entire CDK application or just in the scope of a single CDK stack.</p><p>cdk-nag works with rules which are defined in packs. Those packs are based on AWS Config conformance pack. If you’ve never looked at AWS Config, the <a href="https://docs.aws.amazon.com/config/latest/developerguide/operational-best-practices-for-hipaa_security.html">Operational Best Practices for HIPAA Security</a> page is a nice page to look at in the context of these cdk-nag conformance packs. By default, cdk-nag comes with several rule packs out of the box.</p><ul><li><a href="https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#awssolutions">AWS Solutions</a></li><li><a href="https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#hipaa-security">HIPAA Security</a></li><li><a href="https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#nist-800-53-rev-4">NIST 800–53 rev 4</a></li><li><a href="https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#nist-800-53-rev-5">NIST 800–53 rev 5</a></li><li><a href="https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#pci-dss-321">PCI DSS 3.2.1</a></li></ul><p>Based on your requirements you can enable one or more rule packs. Let’s take a look at how to apply such a rule pack.</p><pre>public class AwsCdkNagDemoApp {<br>    public static void main(final String[] args) {<br>        App app = new App();<br><br>        new AwsCdkNagDemoStack(app, &quot;AwsCdkNagDemoStack&quot;, <br>            StackProps<br>                .builder()<br>                .env(Environment.builder()<br>                .account(System.getenv(&quot;CDK_DEFAULT_ACCOUNT&quot;))<br>                .region(System.getenv(&quot;CDK_DEFAULT_REGION&quot;))<br>                .build())<br>            .build()<br>        );<br><br>         Aspects.of(app)<br>           .add(<br>                AwsSolutionsChecks.Builder<br>                .create()<br>                .verbose(true)<br>                .build()<br>           );<br>        app.synth();<br>    }<br>}</pre><p>As you can see in the above code fragment we’ve enabled the <strong>AwsSolutionsChecks</strong> rules for the scope of the entire CDK app. In this example, we’ve explicitly enabled <em>verbose</em> mode as it will generate more descriptive messages.</p><p>Now let’s take a look at an example stack and see how cdk-nag responds to that. The stack below is a very simple stack which contains an AWS Lambda function processing messages from an SQS queue.</p><pre>public AwsCdkNagDemoStack(final Construct scope, <br>  final String id, final StackProps props) {<br><br>  super(scope, id, props);<br><br>  final Queue queue = Queue.Builder.create(this, &quot;demo-queue&quot;)<br>                 .visibilityTimeout(Duration.seconds(300))<br>                 .build();<br><br>  final Function function = Function.Builder<br>    .create(this, &quot;demo-function&quot;)<br>    .handler(&quot;com.jeroenreijn.demo.aws.cdknag.FunctionHandler&quot;)<br>    .code(Code.fromAsset(&quot;function.jar&quot;))<br>    .runtime(Runtime.JAVA_11)<br>    .events(List.of(<br>      SqsEventSource.Builder.create(queue).build())<br>    )<br>    .build();<br><br>  queue.grantConsumeMessages(function);<br>}</pre><h3>Analyzing results</h3><p>Now when you run cdk synth from the command-line, it will trigger cdk-nag and it will automatically scan your resources in the resulting templates and check them for security and compliance issues. Once the scan is done, cdk-nag will either return successfully or return an error message and output a list of violations in a format that is easy to understand. After running cdk synth we will get the following messages in our output.</p><pre>[Error at /AwsCdkNagDemoStack/demo-queue/Resource] <br>AwsSolutions-SQS3: The SQS queue is not used as a dead-letter queue (DLQ) and does not have a DLQ enabled. <br>Using a DLQ helps maintain the queue flow and avoid losing data by detecting and mitigating failures and service disruptions on time.<br><br>[Error at /AwsCdkNagDemoStack/demo-queue/Resource] <br>AwsSolutions-SQS4: The SQS queue does not require requests to use SSL. <br>Without HTTPS (TLS), a network-based attacker can eavesdrop on network traffic or manipulate it, using an attack such as man-in-the-middle. Allow only encrypted connections over HTTPS (TLS) using the aws:SecureTransport condition in the queue policy to force requests to use SSL.<br><br>[Error at /AwsCdkNagDemoStack/demo-function/ServiceRole/Resource] <br>AwsSolutions-IAM4[Policy::arn:&lt;AWS::Partition&gt;:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole]: The IAM user, role, or group uses AWS managed policies. <br>An AWS managed policy is a standalone policy that is created and administered by AWS. Currently, many AWS managed policies do not restrict resource scope. Replace AWS managed policies with system specific (customer) managed policies.This is a granular rule that returns individual findings that can be suppressed with &#39;appliesTo&#39;. The findings are in the format &#39;Policy::&lt;policy&gt;&#39; for AWS managed policies. Example: appliesTo: [&#39;Policy::arn:&lt;AWS::Partition&gt;:iam::aws:policy/foo&#39;].<br><br>Found errors</pre><p>As you can see cdk-nag spotted some errors and explains what we can do to improve our infrastructure. Usually, it’s quite easy to fix these errors. <a href="https://aws.amazon.com/blogs/devops/leverage-l2-constructs-to-reduce-the-complexity-of-your-aws-cdk-application/">Level 2 CDK constructs</a> already incorporate some of the best practices, so when using them you will probably find fewer errors compared to using Level 1 constructs.</p><p>The messages depend on the rule pack you select. For instance, when we switch to the <strong>HIPAASecurityChecks</strong> rule pack we will get some duplicates but also some additional error messages.</p><pre>[Error at /AwsCdkNagDemoStack/demo-function/Resource] <br>HIPAA.Security-LambdaConcurrency: The Lambda function is not configured with function-level concurrent execution limits - (Control ID: 164.312(b)). <br>Ensure that a Lambda function&#39;s concurrency high and low limits are established. This can assist in baselining the number of requests that your function is serving at any given time.<br><br>[Error at /AwsCdkNagDemoStack/demo-function/Resource] <br>HIPAA.Security-LambdaDLQ: The Lambda function is not configured with a dead-letter configuration - (Control ID: 164.312(b)). <br>Notify the appropriate personnel through Amazon Simple Queue Service (Amazon SQS) or Amazon Simple Notification Service (Amazon SNS) when a function has failed.<br><br>[Error at /AwsCdkNagDemoStack/demo-function/Resource] <br>HIPAA.Security-LambdaInsideVPC: The Lambda function is not VPC enabled - (Control IDs: 164.308(a)(3)(i), 164.308(a)(4)(ii)(A), 164.308(a)(4)(ii)(C), 164.312(a)(1), 164.312(e)(1)). <br>Because of their logical isolation, domains that reside within an Amazon VPC have an extra layer of security when compared to domains that use public endpoints.<br><br>...</pre><p>The <strong>HIPAASecurityChecks</strong> also finds issues related to Lambda function concurrency and running your Lambda function inside a VPC. As you can see different packs look at different things, so it’s worthwhile to explore the different packs and see how they can help you improve. It’s worth mentioning that cdk-nag does not implement all rules defined in these AWS Config conformance packs. You can check which rules are excluded in the <a href="https://github.com/cdklabs/cdk-nag/blob/main/RULES.md#excluded-rules">cdk-nag excluded rules documentation</a>.</p><h3>Summary</h3><p>Overall, cdk-nag is a powerful tool for ensuring that your AWS CDK code and templates adhere to security and compliance best practices. By catching security issues early in the development process, cdk-nag can help you build more secure and reliable infrastructure. I’ve used it in many projects over the last couple of years and it’s adding value. Especially if you work in a team that does not have a lot of AWS experience it shines. If you’re using AWS CDK, I highly recommend giving cdk-nag a try. The <a href="https://github.com/jreijn/demos-aws-cdk/tree/develop/aws-cdk-cdk-nag">example code</a> in this post and a working project can be found on <a href="https://github.com/jreijn/demos-aws-cdk/tree/develop/aws-cdk-cdk-nag">GitHub</a>.</p><p><em>Originally published at </em><a href="https://www.luminis.eu/blog/cloud-en/improve-aws-security-and-compliance-with-cdk-nag/"><em>https://www.luminis.eu/blog/cloud-en/improve-aws-security-and-compliance-with-cdk-nag/</em></a><em>.</em></p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Interested in scaling your software startup</em></strong><em>? Check out </em><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f451c54b456f" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/improve-aws-security-and-compliance-with-cdk-nag-f451c54b456f">Improve AWS Security and Compliance with cdk-nag</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons learned from picking a Java driver for Amazon ElastiCache for Redis — Part 2]]></title>
            <link>https://aws.plainenglish.io/lessons-learned-from-picking-a-java-driver-for-amazon-elasticache-for-redis-part-2-69e5cebbdc21?source=rss-61834bc2a27f------2</link>
            <guid isPermaLink="false">https://medium.com/p/69e5cebbdc21</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[elasticache]]></category>
            <category><![CDATA[lettuce]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Jeroen Reijn]]></dc:creator>
            <pubDate>Wed, 05 Apr 2023 11:00:33 GMT</pubDate>
            <atom:updated>2023-04-08T11:07:20.738Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*20r0TU06oCKmU1LC" /><figcaption>Photo by <a href="https://unsplash.com/@jainath?utm_source=medium&amp;utm_medium=referral">Jainath Ponnala</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Lessons learned from picking a Java driver for Amazon ElastiCache for Redis — Part 2</h3><p>Exploring failover scenarios with Lettuce</p><p>In <a href="https://medium.com/aws-in-plain-english/lessons-learned-from-picking-a-java-based-driver-for-amazon-elasticache-for-redis-part-1-436ebb2f8f3">part one</a> of this series we looked at Jedis as the Java-based Redis driver when used in combination with ElastiCache for Redis in clustered mode. We learned that during a failover Jedis takes some time to figure out what the new primary is before it can restore the connection to the ElastiCache cluster. In this post, I’ll share some of the lessons learned from switching to Lettuce as the driver for our Java-based application.</p><h3>A different driver</h3><p><a href="https://lettuce.io/">Lettuce</a>, a Java-based driver for Redis, is the default driver for Redis when working with Spring Framework. Lettuce is a fully non-blocking Redis client built with Netty providing Reactive, Asynchronous and Synchronous Data Access.</p><p>As a driver, we found it quite similar to use compared to Jedis. Configuring the client was pretty easy to do as you can see in the code below.</p><pre>RedisURI redisUri = RedisURI.Builder.redis(REDIS_HOST)<br>                                    .withSsl(true)<br>                                    .build();<br><br>RedisClusterClient clusterClient = RedisClusterClient.create(redisUri);<br><br>StatefulRedisClusterConnection&lt;String, String&gt; connection = clusterClient.connect();<br><br>syncCommands = connection.sync();</pre><p>With Jedis, we configured a connection pool, but as you might notice from the above code, we’re not using a connection pool. Lettuce connections are designed to be thread-safe so one connection can be shared amongst multiple threads and Lettuce connections <a href="https://lettuce.io/core/release/reference/#client-options">auto-reconnect</a> by default. If needed you can still configure a connection pool, but we did not need it in our case.</p><p>Lettuce states to be <em>cloud-ready</em>, which sounds very promising.</p><blockquote><em>Battle-tested with major Cloud-based Redis services. Lettuce adopts the specifics of various cloud offerings to seamlessly integrate for the best performance.</em></blockquote><p>The AWS blog contains <a href="https://aws.amazon.com/blogs/database/optimize-redis-client-performance-for-amazon-elasticache/">a solid article on ElastiCache for Redis client performance</a>. In tests performed by the AWS team, we can see that Jedis was up to twice as fast as Lettuce. It’s an interesting fact if performance is the ‘only’ concern you have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z7YxzZZpj76Vlxiw.png" /></figure><p>Given the fact that Lettuce is built with <a href="https://netty.io/">Netty</a>, we also immediately noticed quite an impact on the initialization time (cold start) of our lambda function. Netty is really fast while executing, but takes a bit of time to initialize. The new <a href="https://aws.amazon.com/blogs/compute/reducing-java-cold-starts-on-aws-lambda-functions-with-snapstart/">Lambda Snapstart functionality</a> might help with that.</p><p>Instantiating an authorized connection with Lettuce took about <strong>1500 ms</strong>, which is almost twice the time compared to Jedis.</p><p>For Lettuce, we will run the same test scenarios and we will keep track of the number of nodes in the cluster by returning the list of cluster nodes from our test lambda function. Our lambda spits out the current information about the cluster and the roles of the different nodes.</p><pre>{<br>  &quot;nodes&quot;: <br>      &quot;254eaace08fafb42d26750b8721d1fd5b152f0bd test-cluster-0001-001.test-cluster.ifrpha.euw1.cache.amazonaws.com:6379@1122 master,fail - 1679323815172 1679323813195 1 connected<br>      5f41504ecd68d90956060e219ebf8ec32782c4e2 test-cluster-0001-003.test-cluster.ifrpha.euw1.cache.amazonaws.com:6379@1122 myself,slave fa0e9e04c33409b48a8d486e27277bd313e01c27 0 1679323902000 3 connected<br>      fa0e9e04c33409b48a8d486e27277bd313e01c27 test-cluster-0001-002.test-cluster.ifrpha.euw1.cache.amazonaws.com:6379@1122 master - 0 1679323903029 3 connected 0-16383&quot;<br>}</pre><h3>Failover scenarios</h3><p>Just like with Jedis, we wanted to test Lettuce in combination with the two following scenarios:</p><ol><li><strong>A primary node failure.</strong></li><li><strong>A maintenance upgrade that would rotate and update the nodes to a new version of Redis (from 6.0.x to 6.2.x).</strong></li></ol><h3>Primary node failover</h3><p>Just like in <a href="https://medium.com/aws-in-plain-english/lessons-learned-from-picking-a-java-based-driver-for-amazon-elasticache-for-redis-part-1-436ebb2f8f3">part one</a> we will test the impact of a primary node failover and see how the Lettuce client is handling the situation. Lettuce appeared to be quite configurable and flexible, so we tested a couple of different scenarios.</p><h3>Attempt 1: Simple settings</h3><p>Our first attempt was with the simple client configuration as previously shown. We do not use any connection pool and have a very simple synchronous client while connecting to the cluster. We connect with the cluster configuration endpoint, so that ElastiCache for Redis can give us a view of the cluster topology. To analyze the impact of the failover we started using Cloudwatch Logs Insights. It allowed us to scan for the number of exceptions during the failover. By using the following query on our lambda log group we could see all log messages that contained a Java Exception.</p><pre>fields @timestamp, @message, @logStream, @log<br>| filter @message like /(?i)(Exception)/<br>| sort @timestamp desc<br>| limit 2000</pre><p>From what we could see there are quite some Exceptions being thrown during the failover, however, the exceptions occurred for about <strong>2</strong> minutes, after which the client was able to properly handle the requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tkk0oaMVxgJsP2mJ.png" /></figure><p>Most of the exceptions fall in the category of timeouts and connection errors:</p><pre>io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)</pre><p>and</p><pre>io.lettuce.core.cluster.topology.DefaultClusterTopologyRefresh$CannotRetrieveClusterPartitions: Cannot retrieve cluster partitions</pre><p>Both seemed to be related to the fact that Lettuce was trying to fetch the values from the primary/master cluster node and trying to reconnect with the master node that just went ‘down’.</p><h3>Attempt 2: Topology refresh</h3><p>Lettuce has a feature called <a href="https://lettuce.io/core/release/reference/index.html#redis-cluster.refreshing-the-cluster-topology-view">topology refresh</a>, which allows the client to poll the server for an updated view of the cluster topology. We were wondering if enabling this would have a big impact on the fact that our client was losing its connection to the primary node.</p><pre>ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()<br>                // default refresh period = 60 seconds<br>                .enablePeriodicRefresh()<br>                .enableAllAdaptiveRefreshTriggers()<br>                .build();<br><br>clusterClient.setOptions(<br>    ClusterClientOptions<br>      .builder()<br>      .socketOptions(SocketOptions.builder().build())<br>      .timeoutOptions(TimeoutOptions.enabled())<br>      .topologyRefreshOptions(topologyRefreshOptions)<br>      .pingBeforeActivateConnection(true)<br>      .build()<br>);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wvGL-skM-hUkpvYA.png" /></figure><p>Based on the logs insights graph we could see that there was a certain positive impact as the number of errors produced during the failover is less and also less in volume, but our application was still having read issues when reading from the cluster. The logs were similar and still quite substantial in volume so we wanted to learn more about the other different configuration options.</p><h3>Attempt 3: Changing read preference</h3><p>By default Lettuce, just like Jedis, reads from the primary node / shard. However, Lettuce has an additional option, that Jedis does not have, called <a href="https://lettuce.io/core/release/reference/index.html#readfrom-settings"><strong>read preference</strong></a>. If your workload permits, you can tell Lettuce to also read from replica nodes when the primary node is down. In a read-heavy cluster, this can be a very valuable setting.</p><p>With the connection.setReadFrom you have the option to select a specific preference. Some of the available options include:</p><ul><li>UPSTREAM (primary node)</li><li>UPSTREAM_PREFERRED (primary/upstream and fall back to a replica if the upstream is not available)</li><li>REPLICA_PREFERRED (read preferred from replica and fall back to upstream if no replica is available)</li><li>REPLICA (only read from read replicas)</li></ul><p>In our case we configured it to use UPSTREAM_PREFERRED, so we would failover to the replicas in case of the primary node failover.</p><pre>StatefulRedisClusterConnection&lt;String, String&gt; connection = clusterClient.connect();<br>// removed topology refresh for readability (it&#39;s included in the actual configuration)<br><br>connection.setReadFrom(ReadFrom.UPSTREAM_PREFERRED);<br><br>syncCommands = connection.sync();</pre><p>We could see a clear impact on the number of error messages produced during the failover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*04WLSz_XFOwEYVJl.png" /></figure><p>From the logs, we could still see some error messages indicating that the cluster was down during the failover. Why were we still getting those messages?</p><h3>Attempt 4: Read preference and allow reads when the cluster is ‘down’</h3><p>After doing a bit more research we learned that Redis by default does not allow read operations from replicas when the primary node / shard is marked as ‘down’. However, for read-heavy workloads, this might not be an issue. If you want you can allow reads when the cluster is marked as down by changing a parameter.</p><ul><li><strong>cluster-allow-reads-when-down</strong> &lt;yes/no&gt;: If this is set to no, as it is by default, a node in a Redis Cluster will stop serving all traffic when the cluster is marked as failed, either when a node can’t reach a quorum of masters or when full coverage is not met. This prevents reading potentially inconsistent data from a node that is unaware of changes in the cluster. This option can be set to yes to allow reads from a node during the fail state, which is useful for applications that want to prioritize read availability but still want to prevent inconsistent writes. It can also be used when using Redis Cluster with only one or two shards, as it allows the nodes to continue serving writes when a master fails but automatic failover is impossible.</li></ul><p>In AWS CDK you can define a custom parameter group which you provide to your cluster. You can base your parameter group based on the default available groups; for instance “ <strong>default.redis6.x.cluster.on”</strong>.</p><pre>CfnParameterGroup parameterGroup = CfnParameterGroup.Builder<br>  .create(this, &quot;elasticache-param-group&quot;)<br>  .cacheParameterGroupFamily(&quot;redis6.x&quot;)<br>  .description(&quot;Custom Parameter Group for redis6.x cluster&quot;)<br>  .properties(<br>    Map.of(<br>      &quot;cluster-enabled&quot;, &quot;yes&quot;,<br>      &quot;cluster-allow-reads-when-down&quot;, &quot;yes&quot;)<br>    )<br>  .build();<br><br>CfnReplicationGroup replicationGroup = CfnReplicationGroup.Builder.create(this, &quot;replicationgroup-id&quot;)<br>                .cacheNodeType(props.getCacheNodeType())<br>                .engine(&quot;redis&quot;)<br>                .engineVersion(props.getEngineVersion())<br>                .autoMinorVersionUpgrade(true)<br>                .automaticFailoverEnabled(true)<br>                .cacheParameterGroupName(parameterGroup.getRef())<br>                .multiAzEnabled(true)<br>                .atRestEncryptionEnabled(true)<br>                .transitEncryptionEnabled(true)<br>...</pre><p>After configuring the above changes we noticed that while we were still doing 30 requests a second during the failover, there was a ‘seamless’ switch over to the replica nodes and our lambda function was operating as normal. No errors were produced during the fail-over. That looked promising!</p><h3>Cluster engine version upgrade</h3><p>The next scenario was running a cluster engine upgrade. To validate the impact of a version upgrade we migrated from version Redis 6.0.x to 6.2.x. We learned that just like with Jedis we did not see any obvious impact (at least after we configured all of the above options). The upgrade happened, while the service was serving a constant load of 30 requests per second, without producing any errors.</p><h3>Summary</h3><p>By doing this exercise with Lettuce we learned that Lettuce has some advanced capabilities when it comes to reading from an ElastiCache for Redis cluster. By tweaking the client and the server configuration we were able to perform a primary node failover without generating any errors for reads on our cluster. However, if performance is more important than availability you might want to stick with Jedis as it seems to be up to 2x as fast for certain operations.</p><p><em>Originally published at </em><a href="https://www.jeroenreijn.com/2023/04/lessons-learned-from-picking-a-java-based-driver-for-amazon-elasticache-for-redis-part-2.html"><em>https://www.jeroenreijn.com</em></a><em> on April 5, 2023.</em></p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Join our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><em> community and follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em> and</em><strong><em> </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Learn how to build awareness and adoption for your startup with </em></strong><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69e5cebbdc21" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/lessons-learned-from-picking-a-java-driver-for-amazon-elasticache-for-redis-part-2-69e5cebbdc21">Lessons learned from picking a Java driver for Amazon ElastiCache for Redis — Part 2</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>