<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[PostNL Engineering - Medium]]></title>
        <description><![CDATA[Excellent Logistic Intelligence - Medium]]></description>
        <link>https://medium.com/postnl-engineering?source=rss----94bc0bf5cd50---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>PostNL Engineering - Medium</title>
            <link>https://medium.com/postnl-engineering?source=rss----94bc0bf5cd50---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:08:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/postnl-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How a platform team can help application team to get a grip on their network costs]]></title>
            <link>https://medium.com/postnl-engineering/how-a-platform-team-can-help-application-team-to-get-a-grip-on-their-network-costs-41560c0fcea7?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/41560c0fcea7</guid>
            <category><![CDATA[aws-vpc]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Gideon Vrijhoeven]]></dc:creator>
            <pubDate>Thu, 22 Oct 2026 07:23:34 GMT</pubDate>
            <atom:updated>2026-10-22T07:50:30.735Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G3vS95eisk3z-0ROYto1KA.jpeg" /></figure><p>When organizations want to minimize the risk of data being compromised, it is a good idea to keep resources that don’t need to be publicly accessible private, by placing them in a private subnet of a VPC. However a lot of services offered by AWS are default via the internet.</p><p>The result is that when a private resource ( Lambda attached to a VPC, container of EC2 instance) connect to that service, it needs to use a NAT gateway.</p><p>AWS offers several endpoints to prevent this ( gateway endpoints and interface endpoints) Since the gateway endpoints are free, it’s a no brainer to add those endpoints to all VPCs that are created.</p><p>Interface endpoints are billed per hour, and there are a lot of services and thus <a href="https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html">interface endpoints</a> available. So it highly depends on the application and even the stage if it is beneficial to create those endpoints. When the NAT Gateway is centralized, a <a href="https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-tgw-dashboard.html">CloudWatch dashboard</a> could be a good starting point to identify the top talkers.</p><p>In my experience not all teams are aware of their network flows and the bandwidth requirements, especially for new applications. This blog describes a method to assist these teams to get insights into their flows.</p><p><strong>VPC Flow Logs and Route 53 Resolver Query Logs</strong></p><p>To get insights into the traffic from a VPC, AWS offers <a href="https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html">VPC Flow Logs</a> and <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-query-logs.html?refid=ea4f1bda-c329-47b3-a6d0-278045ca3aea">Route 53 Resolver Query Logs</a>. In VPC flow logs alone you can already specify that you want to get information regarding the AWS service that it tries to connect to, using the pkt-dst-aws-service field, however currently that is limited to these services:</p><p>AMAZON | AMAZON_APPFLOW | AMAZON_CONNECT | API_GATEWAY | AURORA_DSQL | CHIME_MEETINGS | CHIME_VOICECONNECTOR | CLOUD9 | CLOUDFRONT | CLOUDFRONT_ORIGIN_FACING | CODEBUILD | DYNAMODB | EBS | EC2 | EC2_INSTANCE_CONNECT | GLOBALACCELERATOR | IVS_LOW_LATENCY | IVS_REALTIME | KINESIS_VIDEO_STREAMS | MEDIA_PACKAGE_V2 | ROUTE53 | ROUTE53_HEALTHCHECKS | ROUTE53_HEALTHCHECKS_PUBLISHING | ROUTE53_RESOLVER | S3 | WORKSPACES_GATEWAYS</p><p>When the AWS service does not match one of these services the result will be listed as AMAZON</p><p>To get a better insight in what the exact services that are being called, we need to get an overview of all DNS queries that are being made from within the VPC.</p><p>Centralizing all the <a href="https://docs.aws.amazon.com/athena/latest/ug/vpc-flow-logs-parquet.html">VPC Flow Logs</a> and Route53 Resolver Query Logs into one account putting them in a parquet format makes it possible to query the logs. For the Route53 Query Logs, we used this design to put it in the parquet format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1mrnOvWwlwcbLNH-rrfTyA.png" /></figure><p><strong>Analyzing the data</strong></p><p>We start by creating an Athena query on the Route53 Resolver Query Logs for today:</p><pre>SELECT<br>  rtrim(query_name, &#39;.&#39;)  AS query_name,<br>  count(*)                AS query_count,<br>  array_distinct(<br>    flatten(<br>      array_agg(transform(answers, a -&gt; a.rdata))<br>        FILTER (WHERE answers IS NOT NULL AND cardinality(answers) &gt; 0)<br>    )<br>  )      <br>FROM &lt;data-source&gt;        <br>WHERE  account_id = &#39;&lt;accountid&gt;&#39;<br>  AND from_iso8601_timestamp(query_timestamp) &gt;= current_timestamp - INTERVAL &#39;1&#39; DAY<br>GROUP BY rtrim(query_name, &#39;.&#39;)<br>ORDER BY query_count DESC;</pre><p>This will result in all the individual DNS queries that are done from the VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r5ujGU_R6tJiOK-lXtMhAg.png" /></figure><p>The results will give back the ip-addresses that are returned by the DNS query.</p><p>We can start by filtering these results out:</p><ul><li>Private ip-addresses ( <a href="https://nl.wikipedia.org/wiki/RFC_1918">https://nl.wikipedia.org/wiki/RFC_1918</a>)</li><li>Results with “vpce-” since they all ready use a VPC Endpoint</li><li>Results for S3 and DynamoDB since they use the Gateway Endpoint to connect to the service based on prefix. This is a good blog that explains how they work: <a href="https://www.awscertificationhandbook.com/guides/aws-gateway-endpoint/">https://www.awscertificationhandbook.com/guides/aws-gateway-endpoint/</a></li><li>Results to non AWS services, not covered in this blog but could be use full to see if those could be changed to private link if supported.</li></ul><p>For the other results we can do a query on the VPC Flow Logs, using the returned ip-addresses as input. So save the ip-addresses!</p><p>Before we can do that, we identify the network interfaces of services that make a connection towards the NAT Gateway. This could be the Transit Gateway Attachments, CloudWan Attachments or just the NAT gateway addresses when is configured locally. To get these log info from the console of the account, go to EC2 and select the Network Interfaces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*4aDCjL9jFR_WNJr03NndoQ.png" /></figure><p>For Transit Gateway of CloudWan, the type is transit_gateway and for Nat Gateway it is nat_gateway.</p><p>Get the Network interface ID values and put them info the Athena query for the VPC Flow Logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aYh0LBLbZEl9dUtj5TsiYA.png" /></figure><p>Now its possible to do the query:</p><pre>WITH target_ips AS (<br>  SELECT ip FROM UNNEST(ARRAY[<br>  &#39;&lt;ip-from-dns-query&gt;&#39;, &#39;&lt;ip-from-dns-query&gt;&#39;, &#39;&lt;ip-from-dns-query&gt;&#39;, &#39;&lt;ip-from-dns-query&gt;&#39;<br>     ]) AS t(ip)<br>),<br>flows AS (<br>  SELECT f.*, t.ip AS matched_ip<br>  FROM &quot;&lt;data-source&gt;&quot; f<br>  JOIN target_ips t<br>    ON t.ip = f.pkt_srcaddr OR t.ip = f.pkt_dstaddr<br>  WHERE f.account_id = &#39;&lt;accountid&gt;&#39;<br>    AND f.interface_id IN (&#39;eni-&lt;&gt;&#39;,&#39;eni-&lt;&gt;&#39;,&#39;eni-&lt;&gt;&#39;)<br>    AND f.pkt_dst_aws_service IS NOT NULL<br>    AND f.pkt_dst_aws_service &lt;&gt; &#39;-&#39;<br>    AND f.bytes IS NOT NULL<br>    AND f.year = &#39;2026&#39; AND f.month = &#39;10&#39; AND f.day = &#39;08&#39;<br>    AND f.flow_direction = &#39;ingress&#39;<br>)<br>SELECT<br>  matched_ip,<br>  array_agg(DISTINCT pkt_dst_aws_service)      AS aws_services,<br>  array_agg(DISTINCT interface_id)             AS enis,<br>  count(DISTINCT IF(pkt_srcaddr = matched_ip, pkt_dstaddr, pkt_srcaddr)) AS distinct_peers,<br>  array_agg(DISTINCT dstport)                  AS dst_ports,<br>  array_agg(DISTINCT action)                   AS actions,<br>  count(*)                                     AS flow_count,<br>  sum(bytes)                                   AS total_bytes,<br>  sum(packets)                                 AS total_packets,<br>  round(sum(bytes) / 1073741824.0, 3)          AS total_gb<br>FROM flows<br>GROUP BY matched_ip<br>ORDER BY total_bytes DESC;</pre><p>Example for results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DZIbIsPQpIEhhOQWTG7Drw.png" /></figure><p>NOTE: Please match the day to reduce the error margin, since ip-addresses could be swapped between AWS services.</p><p><strong>Calculation</strong></p><p>Fixed costs for an interface endpoint per AZ are $7.30 so the break even traffic is ~209 GB/month so for 3 AZ’s its ~626 GB/month using a local NAT gateway, with a centralized egress configuration its ~133 GB/Month so ~398 GB/Month for 3 AZ’s, since we need to add $0,02 GB for the CloudWan traffic.</p><p>For the latest information about pricing always check the AWS <a href="https://calculator.aws/#/addService">pricing calculator</a>.</p><p><strong>Conclusion</strong></p><p>Here we made our calculations, in this example it seems logical to add an interface endpoint for kinesis.eu-west-1.amazonaws.com.</p><p>The total traffic towards Kinesis in one day is 76.67 GB using a centralized NAT gateway</p><p>76,67 * 365 / 12 = 2.331 GB per month</p><p>2.331 GB * $ 0,065 = $151,58 ( $0,02 CloudWan for Interface endpoint traffic and $0.045 NatGateway)</p><p>$7,30 * 3 AZ = $21,90 ( interface costs per month * number of AZs)</p><p>2331 GB * $ 0,01 = 23,31 traffic costs</p><p>Total costs interface endpoint $21,90 + 23,31 = $ 45,21</p><p>Savings $ 151,58 — $ 45.21 = $106,37 a month</p><p>So make sure you do the calculations per service that are returned in the DNS query, filtering out the flows that are not relevant. Check the bandwidth consumed via the flow logs query and start saving money!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41560c0fcea7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/how-a-platform-team-can-help-application-team-to-get-a-grip-on-their-network-costs-41560c0fcea7">How a platform team can help application team to get a grip on their network costs</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-On with AWS DevOps Agent: Changing DevOps Work Without Replacing Engineers]]></title>
            <link>https://medium.com/postnl-engineering/hands-on-with-aws-devops-agent-changing-devops-work-without-replacing-engineers-af0076b791cb?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/af0076b791cb</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Matheus das Mercês]]></dc:creator>
            <pubDate>Thu, 15 Oct 2026 07:26:41 GMT</pubDate>
            <atom:updated>2026-10-15T07:26:40.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tylYKJIjDXc2ezZp.png" /></figure><h3>Introduction</h3><p>When an AI agent can investigate incidents, read operational context, query systems, and suggest what might be wrong, it is natural to ask whether this kind of capability will eventually replace part of the work DevOps engineers do today.</p><p>In this article, I will share how we experimented with AWS DevOps Agent and why we wanted to bring the investigation flow closer to engineers through Slack. The goal is not to argue that AI agents should replace DevOps engineers, but to show how they can change the type of work engineers spend time on.</p><p>The most important parts are: what the engineer asks, what context the agent needs, which skills make the investigation useful, and where human judgment should remain in control.</p><h3>The turning point</h3><p>Generative AI focuses on creating new content based on an input. The output can be text, images, audio, video, software code, or any other artifact generated from a prompt. That is already useful in many engineering workflows, especially when the task is about explaining, summarizing, drafting, or creating something from existing context.</p><p>Agentic AI has a different goal. Instead of only generating content, an agent is expected to work toward an outcome. In the DevOps context, this means the agent needs to understand what is happening, use tools, inspect data, reason over the environment, and come back with findings that help the engineer decide what to do next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*44hTw6W-nCjY7Q6z.png" /></figure><p>This difference becomes important when the problem is not “write me a summary,” but something closer to “investigate why this service is failing.” In that case, the useful output is not only a nice explanation. The useful output is a hypothesis supported by evidence from logs, metrics, topology, recent changes, known incidents, and operational knowledge.</p><h3>The DevOps problem we wanted to solve</h3><p>In the <strong><em>AI Center of Excellence</em></strong> team at PostNL, we build and operate AI capabilities used by multiple teams and applications. These services do not live in a single cloud environment. A single user-facing capability can can depend not only on AWS services, but also on services in other cloud providers, external APIs, networking, authentication, model providers, and internal platform components</p><p>The complexity is not necessarily the number of services involved. <strong>The real complexity appears when something goes wrong and the engineer needs to understand where to look first</strong>. During an incident, this can easily become a manual process of opening dashboards, checking the AWS Console, searching logs, inspecting metrics, comparing timestamps, and trying to form a hypothesis from many disconnected signals.</p><p>This is especially painful during on-call situations. When an engineer is woken up at night, the first minutes are often spent collecting evidence instead of making decisions. The engineer still needs to own the decision, but there is a lot of repetitive investigation work before reaching the point where human judgment is most valuable.</p><h3>Why we chose AWS DevOps Agent</h3><p>When you look at this kind of problem, it is tempting to say that you should simply build an agent. However, building an agent is not only about sending a prompt to a model. You need to define, among other things:</p><ul><li>What the agent should achieve;</li><li>Which systems it can access;</li><li>Which tools it can use;</li><li>What information it should remember;</li><li>How much autonomy it should have;</li><li>Which actions require approval;</li></ul><p>In our case, we wanted the benefit of an operational investigation agent, but we did not want to become the team responsible for maintaining the entire investigation engine ourselves. We needed something that could understand resources, dependencies, operational patterns, and topology, while still allowing us to shape the workflow around our own environment.</p><p>This is where AWS DevOps Agent became interesting. Instead of building the investigation engine from scratch, we could use the managed capability as the core and focus on the thin layer around it. That layer was still important, but it was much closer to our actual problem: making the investigation accessible from where engineers already work and giving the agent the right operational context.</p><h3>What we built around it</h3><p>One of the decisions we made early was that we did not want another dashboard. During incidents, engineers are already communicating in Slack, so asking them to open another tool just to start the investigation did not feel like the right experience.</p><p>The idea was to create a Slack bot that could act as the entry point for investigations. An engineer could ask something like /watchdog investigate the 429s we are seeing, and the bot would trigger the investigation flow with AWS DevOps Agent. The result was not meant to remove the engineer from the process, but to reduce the amount of manual evidence gathering before the engineer could make a decision.</p><p>The architecture was intentionally thin around the managed capability:</p><ul><li><strong>Slack</strong> was the interface where engineers already worked.</li><li><strong>The </strong><strong>/watchdog command</strong> was the trigger for the investigation.</li><li><strong>A small Lambda function behind an API Gateway</strong> translated the Slack request into an investigation request.</li><li><strong>AWS DevOps Agent</strong> performed the investigation using the operational context available to it.</li><li><strong>The Slack thread</strong> became the place where findings and updates were shared back with the engineer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/0*jkrINsfHoXWwSvyM.png" /></figure><p>The agent itself was also defined as infrastructure. The important part was to define the AWS DevOps Agent space and the role it could assume to investigate the AWS account. The cool thing is that you can control exactly what resources the agent is allowed to investigate.</p><blockquote>One limitation we had to account for is that not everything in AWS DevOps Agent was available through CDK or CloudFormation. The agent space and AWS account association could be defined as infrastructure, but skills still had to be configured outside CDK at that point. That means the infrastructure could create the managed agent foundation, while the operational knowledge attached to the agent still required a separate setup step in the console.</blockquote><p>In a simplified version of the CDK code, the core definition of the AWS DevOps Agent space looked like this:</p><pre>import * as cdk from &#39;aws-cdk-lib&#39;;<br>import * as iam from &#39;aws-cdk-lib/aws-iam&#39;;<br>import { Construct } from &#39;constructs&#39;;<br><br>export class DevOpsAgentStack extends cdk.Stack {<br>  constructor(scope: Construct, id: string, props?: cdk.StackProps) {<br>    super(scope, id, props);<br><br>    const agentSpaceArnPattern = `arn:aws:aidevops:${this.region}:${this.account}:agentspace/*`;<br><br>    const monitorRole = new iam.Role(this, &#39;MonitorRole&#39;, {<br>      assumedBy: new iam.ServicePrincipal(&#39;aidevops.amazonaws.com&#39;, {<br>        conditions: {<br>          StringEquals: { &#39;aws:SourceAccount&#39;: this.account },<br>          ArnLike: { &#39;aws:SourceArn&#39;: agentSpaceArnPattern },<br>        },<br>      }),<br>      managedPolicies: [<br>        iam.ManagedPolicy.fromAwsManagedPolicyName(&#39;AIDevOpsAgentAccessPolicy&#39;),<br>      ],<br>    });<br><br>    monitorRole.addToPolicy(<br>      new iam.PolicyStatement({<br>        actions: [&#39;iam:CreateServiceLinkedRole&#39;],<br>        resources: [<br>          `arn:aws:iam::${this.account}:role/aws-service-role/resource-explorer-2.amazonaws.com/AWSServiceRoleForResourceExplorer`,<br>        ],<br>      })<br>    );<br><br>    const agentSpace = new cdk.CfnResource(this, &#39;AgentSpace&#39;, {<br>      type: &#39;AWS::DevOpsAgent::AgentSpace&#39;,<br>      properties: {<br>        Name: &#39;example-devops-agent&#39;,<br>        Description: &#39;DevOps investigation agent&#39;,<br>      },<br>    });<br><br>    new cdk.CfnResource(this, &#39;AwsMonitorAssociation&#39;, {<br>      type: &#39;AWS::DevOpsAgent::Association&#39;,<br>      properties: {<br>        AgentSpaceId: agentSpace.ref,<br>        ServiceId: &#39;aws&#39;,<br>        Configuration: {<br>          Aws: {<br>            AccountId: this.account,<br>            AccountType: &#39;monitor&#39;,<br>            AssumableRoleArn: monitorRole.roleArn,<br>          },<br>        },<br>      },<br>    });<br>  }<br>}</pre><p>This gave us a workflow where the agent could do the first investigation pass, but the engineer still owned the interpretation and the next step. That distinction was important because the goal was not to create a fully autonomous incident response system, but to reduce the repetitive work required to understand what was happening.</p><h3>The skills we defined for the agent</h3><p>The Slack bot request handler was only one part of the solution. The more important part was the context we provided to the AWS DevOps Agent space. A generic investigation agent can be useful, but real environments have details that are specific to the organization, the platform, the architecture, and the way teams operate.</p><p>For that reason, we defined custom skills and knowledge sources to help the agent understand our environment better. This can be easily created in the AWS DevOps Agent space web app. Some examples were:</p><ul><li><strong>sample-skip-scheduled-maintenance</strong>: used to skip low-priority incidents during a scheduled maintenance window and avoid unnecessary investigations for expected disruptions.</li><li><strong>azure-troubleshooting-guide</strong>: used to give the agent troubleshooting context for Azure-related incidents, performance issues, and dependencies.</li><li><strong>understanding-ai-toolkit-architecture</strong>: used to document the architecture on Azure, workloads, resource groups, Function Apps, Cognitive Services, networking, and related components.</li><li><strong>understanding-lpe-architecture</strong>: used to document the LandingZone PostNL Engineering infrastructure, including networking, IAM, monitoring, and alerting context.</li></ul><p>These are the kinds of details that an engineer usually learns over time, but that an agent also needs if we expect it to produce useful findings.</p><p>This is where DevOps engineers still play a critical role. The value is not only in connecting an agent to tools, but in deciding which context the agent needs, which operational assumptions are safe, which sources of information are reliable, and which parts of the investigation should remain read-only or require approval.</p><h3>AWS released native Slack support</h3><p>There is an interesting detail in this story. We built the Slack bot because we wanted to bring the investigation flow to where engineers already work, and at that moment this meant creating our own thin integration between Slack and AWS DevOps Agent.</p><p>Later, AWS released native bidirectional Slack communication for AWS DevOps Agent. According to the <a href="https://docs.aws.amazon.com/devopsagent/latest/userguide/whats-new.html">AWS DevOps Agent What’s New page</a>, on September 1, 2026, AWS announced that you can start and continue conversations with AWS DevOps Agent directly from a private Slack channel by enabling bidirectional communication when associating a channel.</p><p>At first glance, it might look like this makes the custom Slack bot unnecessary. In practice, it makes the communication layer simpler, which is a good thing. The bot solved the interface gap we had at the time, but the reusable part of the work was not only the bot implementation. It was the understanding of how engineers wanted to interact with an investigation agent during an incident.</p><p>We still needed to define what engineers ask during incidents, what the agent should know, which skills improve investigations, and where the boundary between recommendation and action should be. If the native Slack integration covers the communication layer, the surrounding concepts can still help with additional context, specialized knowledge, Bedrock-based extensions, knowledge bases, or other systems that provide the agent with better input for the investigation, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/0*y3v5i1TSenr-CBIi.png" /></figure><h3>Before and after</h3><p>The most important change is not that the engineer disappears from the workflow. The change is that the engineer starts from a better point in the investigation.</p><p>Before, the engineer had to manually collect evidence across multiple systems. That meant opening dashboards, checking cloud consoles, searching logs, comparing timestamps, and trying to understand whether the issue was related to AWS, Azure, an external dependency, networking, authentication, or something else. This work is important, but it is also repetitive and time-sensitive.</p><p>The old flow looked closer to this:</p><ol><li>Wake up because something might be wrong, or because of a phone call.</li><li>Open dashboards and cloud consoles.</li><li>Check logs, metrics, alarms, and recent signals.</li><li>Compare timestamps across systems.</li><li>Build a first hypothesis manually.</li><li>Decide what to investigate next.</li></ol><p>After introducing the agentic investigation flow, the engineer can ask the agent to investigate and then review the findings. The agent can search logs, query resources, correlate signals, and build an initial hypothesis. The engineer still evaluates the result, considers the business impact, and decides what should happen next.</p><p>The new flow became closer to this:</p><ol><li>Ask the agent to investigate from Slack.</li><li>Let the agent collect operational signals and context.</li><li>Review the findings and hypothesis.</li><li>Decide what action, if any, should be taken.</li></ol><p>This is the part of DevOps work that I think is worth changing. We should not try to automate judgment without understanding the risk, but we should reduce the amount of manual work required before judgment can happen.</p><h3>What should stay human</h3><p>Even if an agent can investigate an issue, not every action should be delegated without approval. There is a big difference between collecting evidence and changing production systems.</p><p>In a DevOps workflow, humans should remain responsible for decisions where context, risk, and business impact matter. For example, I would not put these actions in the same category as querying logs or summarizing metrics:</p><ul><li>restarting critical services</li><li>changing infrastructure</li><li>disabling alarms</li><li>modifying permissions</li><li>applying mitigations that affect customers</li></ul><p>These actions require a different level of control.</p><p>This is why I believe the most useful agentic workflows are not necessarily the ones with full autonomy. A workflow where the agent investigates, correlates, and proposes a hypothesis can already remove a lot of toil. The engineer can stay responsible for the decision while spending less time collecting the evidence manually.</p><h3>Conclusion</h3><p>AWS DevOps Agent is interesting to me because it can change the kind of DevOps work engineers spend time on. <strong>The goal is not to remove engineers from the process, but to reduce the repetitive investigation work that happens before engineers can make good decisions.</strong></p><p>In our experiment, the Slack bot was a way to bring the investigation flow <strong>closer to where engineers already work</strong>. With the native bidirectional Slack support now available in AWS DevOps Agent, that communication layer can become simpler, but the core lessons remain useful. The operational context, the skills, the knowledge sources, and the approval boundaries are still the parts that make the workflow valuable.</p><p>The opportunity is to let the agent help with logs, signals, topology, and hypotheses while <strong>keeping engineers responsible for context, risk, and business impact</strong>. In other words, use AWS DevOps Agent to change DevOps work, not to replace DevOps engineers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0076b791cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/hands-on-with-aws-devops-agent-changing-devops-work-without-replacing-engineers-af0076b791cb">Hands-On with AWS DevOps Agent: Changing DevOps Work Without Replacing Engineers</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Centralized, Multi-Backend Web Search Service for AI Agents]]></title>
            <link>https://medium.com/postnl-engineering/building-a-centralized-multi-backend-web-search-service-for-ai-agents-3809faa741b9?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/3809faa741b9</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Matheus das Mercês]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 07:30:46 GMT</pubDate>
            <atom:updated>2026-06-17T07:44:09.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JPzQaBkL-_Us7pfWvudnxg.jpeg" /></figure><h3>Introduction</h3><p>From small ones to more complex agentic architectures, <strong>agents are everywhere</strong>. As more teams build <strong>AI-powered</strong> solutions, web search is becoming a fundamental capability: access current information, verify facts, and gather external context.</p><p>In this article, I’ll walk through how we built a serverless, multi-backend web search service at PostNL, creating a foundation that can evolve from a single search backend to a <strong>centralized</strong>, provider-agnostic service on AWS.</p><h3>Why web search as a shared capability</h3><p>While integrating a search provider is relatively straightforward for a single app, the challenge becomes more interesting when multiple teams start building agents that need to <strong>access information beyond what the LLM already knows</strong>.</p><p>If teams have the freedom to implement their web search services independently, that can lead to **fragmentation**. Different providers, different APIs, some of them relying on public web search APIs, different cost models, etc.</p><p>At that point, web search is no longer just an application feature, but a **shared capability**. Providing a centralized web search service enables teams to consume a **consistent interface** while allowing platform owners to manage costs, operations, and future provider choices in a single place.</p><h3>Designing a centralized web search service</h3><p>At PostNL, the <em>AI Center of Excellence (AI CoE)</em> supports teams across the organization in adopting and scaling AI solutions. As more teams began experimenting with AI agents, the need for a web search service became clear.</p><p>Rather than recommending a specific provider, we decided to focus on a <strong>platform-oriented solution</strong>. Our goal was to provide a single web search capability that teams could easily consume, while keeping the underlying search implementation flexible and <strong>replaceable</strong>.</p><p>From the beginning, we defined a few key design principles:</p><ul><li>A single interface for all consumers;</li><li>Support for multiple search backends;</li><li>Low operational overhead;</li><li>Cost-efficient operation;</li><li>The ability to evolve without impacting consumers;</li></ul><p>These principles led us to an architecture centered around a <strong>lightweight routing layer</strong> that sits between AI applications and the underlying web search providers.</p><h3>Why the backend doesn’t matter to consumers</h3><p>Perhaps the most important aspect of the design is that consumers <strong>never interact directly with the underlying web search providers</strong>.</p><p>Because all requests flow through the routing layer, the backend can evolve independently of the applications using the service. Today, the router forwards requests to the primary search backend; tomorrow it could route traffic to additional providers, apply failover policies, or make routing decisions based on cost, geography, or freshness requirements.</p><h3>Initial architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/0*PIIyJlxkBwni4Pf1.png" /></figure><ul><li>The service is exposed through a private API Gateway, ensuring that only authorized consumers within the network can access it through a VPC endpoint.</li><li>Requests are routed to a lightweight <strong>Lambda-based router</strong>, which acts as the abstraction layer between clients and the underlying search backend.</li><li>The initial backend is a self-hosted <a href="https://github.com/searxng/searxng">SearXNG</a> deployment running on <strong>ECS Fargate behind an internal Application Load Balancer</strong>, providing a scalable and centrally managed web search capability while keeping the architecture flexible for future providers.</li></ul><h3>Building the Router</h3><p>The Router is the <strong>central component</strong> of the system. It enables future backend changes without impacting consumers.</p><p>The overall flow is fixed:</p><ol><li>Receive requests from API Gateway</li><li>Forward requests to the primary backend</li><li>Return responses from the backend</li><li>Perform basic request validation if required</li></ol><p>The Router follows a <strong>hexagonal architecture</strong>, also known as ports and adapters. The application core defines the search behavior and interfaces, while infrastructure concerns such as API Gateway events, HTTP clients, and provider-specific integrations are implemented as adapters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*UzZ5Kg4yUPlNTRXm.png" /></figure><p>This keeps the core logic independent from any specific backend. Today, the Router forwards requests to SearXNG. In the future, additional <strong>adapters</strong> can be introduced for other providers without changing the public API or affecting consumers.</p><h3>Runtime choice</h3><p>For the runtime, we selected <strong>Go</strong> because the Router is mostly a lightweight HTTP orchestration component. It does not perform heavy computation; it validates requests, applies routing logic, calls a backend, and returns a response.</p><p>Go is a good fit for this type of workload because it provides:</p><ol><li>Fast startup times</li><li>Low memory usage</li><li>Strong HTTP support in the standard library</li><li>Simple concurrency primitives</li><li>Easy deployment as a small Lambda binary</li></ol><h3>The reasoning behind Lambda</h3><p>AWS Lambda was selected to keep the operational footprint small. The Router does not need long-running infrastructure, local state, or complex runtime management. Keeping it <strong>stateless</strong> allows Lambda to scale horizontally as request volume changes.</p><p>At first glance, introducing a Lambda-based router in the request path may raise <strong>scalability concerns</strong>, especially for a service that could be consumed by multiple applications. However, the router intentionally remains lightweight, performing only request validation, routing, and protocol translation. By keeping the component stateless and focused on orchestration rather than search execution, Lambda provides automatic scaling with minimal operational overhead while maintaining low latency.</p><p>This <strong>separation</strong> keeps the Router simple today while allowing it to evolve later with routing policies, fallback logic, circuit breakers, and additional backend adapters.</p><h3>Deploying an open-source web search backend</h3><p>After establishing the Router layer, the next decision was selecting the initial web search backend.</p><p>Rather than immediately integrating <strong>commercial</strong> web search providers, we wanted a solution that was self-hosted, easy to experiment with, and replaceable in the future.</p><p>For the MVP, we selected <a href="https://github.com/searxng/searxng">SearXNG</a>, an open-source metasearch engine.</p><p>Why SearXNG?</p><ul><li>Self-hosted</li><li>Open source</li><li>Aggregates multiple engines</li><li>Easy container deployment</li><li>Vendor independence</li></ul><p>SearXNG runs as a containerized service on ECS Fargate and is exposed internally through an Application Load Balancer. For this workload, ECS Fargate offered the simplest path to a production-ready deployment.</p><h3>Consuming the service</h3><p>As the service is exposed through a private API Gateway, teams at PostNL can consume the service through a <strong>standard HTTP interface</strong>, where authentication, quotas, and usage tracking are handled centrally. This provides clear visibility into adoption and usage patterns while ensuring that the platform can be governed consistently as more teams onboard.</p><p>The service is intentionally focused on <strong>search retrieval</strong>. Responses consist of search results returned by the backend and are not interpreted or summarized by an LLM. Any reasoning or answer generation remains the responsibility of the consuming application.</p><h3>Preparing for a multi-backend future</h3><p>Although the initial implementation relies on a single backend, the service was designed from the beginning to support multiple search providers. The Router follows a hexagonal architecture, where the application core remains independent from backend-specific implementations through well-defined ports and adapters.</p><p>Potential future enhancements include:</p><ul><li>Additional search providers</li><li>Provider failover and fallback mechanisms</li><li>Health-based routing</li><li>Query-specific routing policies</li><li>Configurable provider selection</li></ul><p>By investing in the abstraction layer early, the service remains flexible and can evolve incrementally as requirements change, while continuing to provide a stable interface for consumers.</p><h3>Conclusion</h3><p>As AI agents become more common, capabilities such as web search are increasingly moving from individual applications into shared capabilities. By centralizing web search behind a consistent interface, teams can focus on building solutions rather than integrating and operating web search providers.</p><p>The specific technologies and providers will likely evolve over time, but the underlying principle remains the same: <strong>common capabilities</strong> are often most valuable when they are provided once and consumed by many.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3809faa741b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/building-a-centralized-multi-backend-web-search-service-for-ai-agents-3809faa741b9">Building a Centralized, Multi-Backend Web Search Service for AI Agents</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 0 to production: 10 lessons learned]]></title>
            <link>https://medium.com/postnl-engineering/from-0-to-production-10-lessons-learned-9dd62fa2024b?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/9dd62fa2024b</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[software-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <dc:creator><![CDATA[Paul van der Bijl]]></dc:creator>
            <pubDate>Thu, 23 Apr 2026 08:21:11 GMT</pubDate>
            <atom:updated>2026-04-23T08:21:10.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cEFmMgYlARiJNN9-" /><figcaption>Photo by <a href="https://unsplash.com/@bel2000a?utm_source=medium&amp;utm_medium=referral">Belinda Fewings</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Earlier this year the application that my team &amp; I at PostNL have been working on for a long time went live. It helps monitor the supply chain and offers help with steering in case of disruptions. It contains a back-end and Vue front-end that both use TypeScript and are deployed in AWS using CDK. As we built the application, we had to adapt to the challenges we encountered. Looking back there were many lessons that I learned by building an application from the ground up and getting it to production. In this blog I want to share those learnings summarized in ‘10 lessons learned’, so hopefully they also help you.</em></p><h3>1. Understanding the domain is sometimes more important than the technical solution</h3><p>When we started with the application, we literally started from 0. While it seemed that the problem we needed to solve was clear, we didn’t have a single line of code yet. Setting up a large project from scratch will force you to make a lot of choices early on. The hard thing is that you don’t have a reference yet for what works and what doesn’t work. In our case we kept choosing ‘the safe side’ during technical choices (what if … in the future), which resulted in an overengineered solution. It’s really tempting to pick a whiteboard and start coming up with very clever technical designs, but how do you know the added complexity is justified?</p><p>As we went along, we looked back at the parts of the application that we had to rebuild and tried to understand what caused it. Although we correctly implemented the often complex technical design, this didn’t allow us to adapt the application to the additional requests from the business. What we learned was that we didn’t understand the domain well enough and set up the architecture based on partial or wrong assumptions about the domain. Our lesson? When we start to work on a new feature we spend a few days (or weeks!) to better understand the domain. Only when we are confident and well versed in the domain we start with technical- and solution designs.</p><p><strong>In practice: </strong>before thinking about technical solutions, try to understand the domain by applying Domain Driven Design and embed it in your way of working and code. For us doing event storming sessions really helped in understanding the complex domain that we were working in. After we put this into practice, we didn’t have to rebuild any parts of our application anymore.</p><h3>2. Am I building a solution or solving a problem?</h3><p>In the beginning we also believed the requirements of each feature that got provided to us would fully cover the problem we had to solve. Of course most of it was correct, but the pitfall is what it missed. Often the requirements are written by someone who is not a developer, meaning they don’t look at a problem the way a developer looks at it. They might miss important context that is actually very relevant for a developer to build the correct solution.</p><p>When I went to the user and talked directly with them, I had some real eye-openers. It turned out that I had some unconscious assumptions about how they used our application. One of the assumptions was that we needed an advanced way to easily switch between different tabs. But when I visited different users, I saw that they used a tile manager to have multiple tabs with all their applications open side-by-side at the same time. This allowed them to easily monitor multiple things at the same time on their 4k screen without constantly having to switch. The solution we built had the assumption that they would have a single browser open in full screen and would switch between the tabs. If I hadn’t gone to the user, I would’ve never known this.</p><p><strong>In practice: </strong>I think a good way to validate whether you understand the user is asking yourself the question: how can/do we deliver value for a user? This really forces you to think from the perspective of the user. At some point I realized that we often got the solution provided to us by the business, while we didn’t fully understand the problem that we would solve for the user. Another question to ask is: which problem are we solving for the user? And: how will the user actually use our application?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EQpY8w6TJqgQdYp0" /><figcaption>Photo by <a href="https://unsplash.com/@nadineshaabana?utm_source=medium&amp;utm_medium=referral">Nadine E</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>3. Know what you should be good at (and thus what not)</h3><p>Starting with a new application always opens up many possibilities. I remember that in the beginning I brought all these ideas from other projects I worked on that would be really cool to also do for this project. Of course this knowledge helps, but it was also a pitfall for me. The question wasn’t whether we could build all these cool things, the question was whether we should. I think our job as an engineer is to build a car when a car fits with what’s being asked, although in theory we could build a rocket ship.</p><p>Next to knowing your products, knowing ‘what you should be good at’ is also about the available disciplines within your team. At some point we also explored adding data science to our application. We had a data scientist who was working on a model to predict the amount of incoming parcels. The idea was good and had potential, but unfortunately it was more complex than expected. Suprising, right? We learned that having to train and host a data science model was very different from the rest of our application. The code was also in Python instead of TypeScript, with which we had only limited experience. I could see that the added burden slowed down the team and created knowledge gaps between devs. When we decided to move the data science to another team, we could accelerate again and focus on the other features.</p><p><strong>In practice: </strong>a good way to know ‘what you should be good at’, is defining key challenges that you have to solve. In our case one of these was a focus on robustness, since our application was going to be used in operation. This also helps to know what you should not be good in and thus should think twice whether you also want to do that.</p><h3>4. When every added technique costs something, how do you not go bankrupt?</h3><p>When writing software we make use of many different techniques like frameworks, programming languages, libraries and more. Already for a simple application there are often tens of different ones underlying it. It’s often quite easy to keep adding new ones, since they make our lives much easier as developers. But it’s also important to be aware of the consequences of this. In some cases adding a new technique might result in more downsides than benefits.</p><p>Adding a new technique results in more cognitve load, requires more knowledge within the team and adds more complexity to the application. Often these only become clear in the long run, but then you’ve already embedded the technique in your application and switching away from it comes at a cost. In hindsight, it would’ve been easier upfront to make a different choice.</p><p>A good example of this is us choosing not to do websockets, although almost all devs who I talk to and explain a bit about our application suggest that as an idea. The application we build is a monitoring application that must show realtime updates similar to a stock market application. In our back-end we do receive all data realtime and could in theory show this in less than a second on the screen. The only thing is that this is not necessary for our use case. In the end we chose to build a simple polling mechanism in the front-end that would retrieve the latest data from the back-end every minute. Since the amount of users are at most a few hundred, this solution is good enough for our use-case. The effort and added complexity of building a websocket solution would definitely not outweigh the gains. Another example of just building the car and not a rocket ship!</p><p><strong>In practice:</strong> what helps is defining a toolset of techniques that we support within the team. When considering a new technique, it becomes a clear choice whether we want to include that in our toolset and carry the ‘weight’ that it brings. It’s also a great idea to embed the company wide mandatory techniques in this definition.</p><h3>5. Making technical decisions is hard; making the right one is even harder</h3><p>As I said before, there are a lot of technical choices to make when setting up an application from scratch. It’s really tempting to get these out of the way and embed them deep within your application. This is often the fastest way in the short term, but will you regret it later? If the choice turned out to be wrong, the cost is significant to move away from it since a lot has already been invested in it.</p><p>We also learned this the hard way when choosing a database. At first we opted for DynamoDB as our datastore and invested in building nice abstractions on top of it. As time went on, we kept extending the generic setup, making it more and more complex. At some point we hit a roadblock when we couldn’t add certain aggregating logic to our setup. We had to take a step back and had to reconsider the choice we made. As a result we had to also add a SQL database to our toolset and use this for grouping and aggregating data. In hindsight we shouldn’t have gone all-in on DynamoDB and created all these abstractions on top. Part of the time we spent on it was lost and it took quite a bit of effort to migrate a part of the existing, complex DynamoDB solution to a SQL solution.</p><p><strong>In practice: </strong>in every IT project technical choices like these are made, but I don’t always see teams being aware of the fact that they make them. I think it’s important to write them down in some way, for example in the form of Architectural Decision Records (ADRs). These always contain the problem that you’re trying to solve, the choices you considered and the choice you made including a reason why. This helps in the short term to make conscious choices, but also in the long run to look back at them and remember why a certain choice was made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SjIWzU3X3hn1sggW" /><figcaption>Photo by <a href="https://unsplash.com/@soymeraki?utm_source=medium&amp;utm_medium=referral">Javier Allegue Barros</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>6. Why I often prefer composition over inheritance</h3><p>As you’re reading this, I’m sure you agree that we have been learning more and more about the domain throughout the development of this application. During the beginning of this journey we often thought that we had a good enough understanding of the context. We spotted some patterns in the logic that seemed to be ideal for applying inheritance. We invested time in building some nice, generic inheritance solution and shipped it. Later on we unfortunately learned more about the domain and came to the conclusion that the abstractions underlying the inheritance weren’t fully correct. We had to refactor parts of the application that we already had in production, which was (especially for the infrastructure) painful.</p><p>Instead, we switched to using composition in certain places to still allow re-use for certain abstractions, but allowing the flexibility to add new learnings of the domain. Although composition can be less powerful than inheritance, it’s less rigid in its implementation. In the past, I’ve seen very impressive inheritance solutions, but to change or debug them would require complex surgery. A different team in our department provided us with a generic event-driven SDK built using inheritance that would make it incredibly easy to create event-driven Lambda’s containing a single function(store, translate, filter, etc.). We started to use this setup, but quite quickly had to be really creative in trying to add exceptions (aka hacks) to the generic setup for it to support use-cases that we had. The generic SDK approach was a cool tech solution that had many layers of ‘interesting’ complexity, but in the end it was more in the way than it helped.</p><p><strong>In practice: </strong>we chose with our infrastructure in CDK to switch from generic (CloudFormation) stacks that allow inheritance to composition via constructs. This allowed us to still re-use common patterns, while not having to redeploy infrastructure due to changes in the base stacks.</p><h3>7. How sharing problems can help you and others</h3><p>Depending on your situation this lesson might be more or less relevant, but in our case the department we were in luckily also used a very similar tech stack as us. This gave some nice opportunities to work together and help each other out. In a bigger company there is a big chance that there are other teams that have similar technical challenges as you. It’s good to be aware of this and consider solving these together. You could share knowledge or even share solutions. It’s good to be aware of the trade-off, since sharing a solution does mean that it comes at the cost of extra communication between the teams.</p><p><strong>In practice: </strong>within our department, we shared code in different ways and as we went on we learned that some ways work better than others. In general there are two ways to share code: via a shared package or using shared infrastructure. We tried both options, but learned that the first one seems to be much better suited to solve shared problems. Although shared infrastructure might be more efficient (no duplicate resources), it also comes at a risk. We learned the hard way that a rollout of one team can impact other teams since the infrastructure is shared. When teams have different requirements, the resources should always be compliant with the most strict set. In most cases code sharing via packages is the better option if there is no dedicated team to maintain the shared infrastructure.</p><h3>8. Build just big enough, but keep the possibility for growth</h3><p>While building the application, we understood the context better as we went. The code and infrastructure should support this. In the beginning, we overdesigned certain solutions and built a villa, while initially we only needed a simple house. After we had some partial redesigns of the architecture, we chose to change the core of our architecture. The focus would be on designing around the domain while keeping the technique as simple as possible. The domain is often the most unknown factor, but the one that’s key in giving value to the user.</p><p><strong>In practice: </strong>after our first few features we ran into challenges with scaling our architecture. We decided that we needed an architecture that would allow us to grow, while staying flexible and lean. We agreed to go with the <a href="https://medium.com/postnl-engineering/why-modular-monolith-is-a-great-starting-architecture-for-a-serverless-application-part-1-e69d139db719">modular monolith</a> and a layered architecture. Now that we’re live, these core choices allow us to house a complex application that we can easily extend. We also kept the <a href="https://evolutionaryarchitecture.com/">evolutionary architecture</a> principles in mind during the development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*esv-yqCXuYl9w3dx" /><figcaption>Photo by <a href="https://unsplash.com/@mimithecook?utm_source=medium&amp;utm_medium=referral">Milivoj Kuhar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>9. I like it when things fail</h3><p>Throughout the building of the platform and testing it with our users, we of course had plenty of failures. From small bugs to outages. I think it’s often not the case whether there will be a failure, but instead when it will happen. It might sound weird, but I like it when things fail, because this is also an opportunity to improve. When diving into these failures, the challenge is to find the root problem and try to address it. By applying this learning from failures over and over, we ended up with a solid application and team where each member understands how to effectively debug and apply fixes. It allowed us to go live almost two months earlier than planned and is an important factor in having almost no incidents afterwards.</p><p>During the development of our application we ran into many bugs that were all related to dates and timezones (who doesn’t like that?). The different daily processes that we support never stopped exactly at 0:00. We had to write logic where the process day didn’t align with the calendar day, while also in the Dutch timezone where we have summer and winter time. After doing a post mortem on one of the incidents, we learned that the root cause was how we tried to solve it. The time logic was scattered in a lot of places and sometimes resulted in contradictions. We decided to move all this ‘time magic’ to a single class and solved it properly there with a large set of unit tests around it. After this we barely had any bugs that were caused by this time magic.</p><p><strong>In practice: </strong>I think the key to learning from failures is doing post mortems. Sitting together with the team and diving into a problem make it possible to address the root cause of a problem. To do proper post mortems, it’s a must to make them blameless and try to learn from them as a team. Maybe one person did the deployment that caused an incident, it’s often a signal that the whole team didn’t do something right. Things like reviews, architecture choices and team processes are all shared responsibilities within the team.</p><h3>10. Trust in what you build, but also trust in each other</h3><p>One aspect from the previous lesson that I want to highlight is trust. I think this applies to both the application you build, but especially the team you work in. Let’s start with the first one. Within our team we had a big focus on frequent production deployments without the fear that they would not work. In the beginning, we didn’t have this trust yet and would ship code to production that would break. Instead of shipping less, we decided to apply the previous lesson and resolve the root causes. We’re now in a position where we deploy to production multiple times per week and sometimes even per day, with the benefit of being confident that it works. When we got the call on a Friday afternoon that the old application had issues, we were confident that we could put our application live and ask the users to use that one instead.</p><p>The trust within the team is also important. There are always unexpected curve balls being thrown at the team. A good example of this was the <a href="https://securitylabs.datadoghq.com/articles/shai-hulud-2.0-npm-worm/">Shai Hulud supply chain attack</a> that halted all our development. We had to quickly switch context and implement some security measures to reduce the risk of being hit by it. I was really happy to see how the team rallied together and solved it together.</p><p><strong>In practice: </strong>one way to measure the trust in what you build is by looking at the DORA metrics. The metric to measure the deployment frequency is a good indicator to reflect on how well you’re doing. If you only deploy once every two weeks, can you explain to yourself and your team why? Is it because of some external factor, or is it because you don’t have trust in what you build? Maybe it’s because you don’t slice up your stories well enough?</p><h3>Final thoughts</h3><p>This was a bit of a longer post, because I really wanted to give some depth to the lessons while trying to make them concrete enough to apply. There were probably many more things to share, but I think these 10 are a good summary of the most important lessons. One of the things to highlight here at the end is how domain-driven design guided us through many of the lessons. I hope that the above lessons will also help you in your journey to build and ship IT applications. Let me know what you think of these lessons and what you learned in bringing an application from 0 to production!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dd62fa2024b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/from-0-to-production-10-lessons-learned-9dd62fa2024b">From 0 to production: 10 lessons learned</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystify dual stack on AWS in combination with CloudWan and Network Function with send-to.]]></title>
            <link>https://medium.com/postnl-engineering/demystify-dual-stack-on-aws-in-combination-with-cloudwan-and-network-function-with-send-to-0408e8d1c726?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/0408e8d1c726</guid>
            <category><![CDATA[aws-cloud-wan]]></category>
            <category><![CDATA[nat-gateway]]></category>
            <category><![CDATA[dns64]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[ipv4-vs-ipv6]]></category>
            <dc:creator><![CDATA[Gideon Vrijhoeven]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 07:01:04 GMT</pubDate>
            <atom:updated>2026-04-01T07:01:04.336Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wQQQGkUyR807RlFitOGzdQ.jpeg" /></figure><p>In this blog we like to give a better understanding on the package flows and cost patterns in an enterprise dual stack environment. There are advantages of using IPv6 over IPv4, but since they are not compatible it is also needed to provide a NAT64 solution for teams for backwards compatibility.</p><p>The AWS NAT Gateway can provide NAT64. To use the centralized NAT Gateway, the traffic needs to be routed to the NAT Gateway and the subnet needs to support NAT64 by changing a setting.</p><p>When using CloudWan a service insertion can provide a centralized NAT Gateway solution, using a Network Function Group to inject a default route into segment routing tables, when the action specified is send to.</p><p><a href="https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-policy-service-insertion.html">https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-policy-service-insertion.html</a></p><p>This will work fine if no other route summarization is used, since it creates a default route towards the scanning VPC. But most companies have summarized routes pointing to VPN or Direct Connect and that could conflict with NAT64 functionality.</p><h3>The setup</h3><p>There are multiple accounts connected with a VPC towards CloudWan. Within CloudWan there are segments created and we will zoom in on only one segment called “nonprod”. CloudWan also has a Network Function group connection called “scanning” this contains a VPC that has an AWS NAT Gateway and a connection toward the internet. In the properties of CloudWan a “send-to” service insertion in the “nonprod” segment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XsrXbZY7cbe_OV4AhnHG7A.png" /></figure><p>The routing tables of the private subnet look like this and we enabled DNS64 on the subnets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wHvv1-0DVSn9oM3P4F_niA.png" /></figure><p>The next slide also shows the routing table of the “nonprod” segment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYah6GczKPrageGRvIg92w.png" /></figure><p>In order to achieve these routes it was needed to add the IPv4 route (10.79.192.0) of the scanning segment info in the “nonprod” using a segment action. This is needed so IPv4 NAT64 traffic can find its way back to the NAT Gateway in the scanning segment.</p><pre>{<br>      &quot;segment&quot;: &quot;nonprod&quot;,<br>      &quot;destinations&quot;: [<br>        &quot;attachment-01234567890&quot;<br>      ],<br>      &quot;action&quot;: &quot;create-route&quot;,<br>      &quot;destination-cidr-blocks&quot;: [<br>        &quot;10.79.192.0/20&quot;<br>      ]<br>    },</pre><h3>Packet walk for IPv4:</h3><p>In this scenario instance 10.64.32.16 wants to connect to Application LoadBalancer (ALB) 10.64.60.45. The first package will go to the core network following the default route in the routing table ( 0.0.0.0/0 core-nw)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yh2AXnbn77kodY6kXucBkQ.png" /></figure><p>The packet arrives on the “nonprod” segment routing table and follows the route towards the VPC of team B (10.68.32.0/22 VPC team B)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_hthzNjFrDeLgywHIGCqA.png" /></figure><p>The ALB sends the package back to 10.64.32.16 using the default route towards the core network ( 0.0.0.0/0 core-nw)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bz2YrjR8oePAJrebQvUllg.png" /></figure><p>And the “nonprod” segment routing table sends the packet to the VPC of team A ( 10.64.32.0/22 VPC team A)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTYKaLFgwwRycQj2CWBf5g.png" /></figure><p>The costs for this flow are $0.02 per GB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wu-czbvxbd2aNXhFZdA4g.png" /></figure><h3>Packet work towards the internet using IPv4</h3><p>Setup of the routing table of the scanning VPC and the Network Function group are provided here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0YwnOzrwkkgv4kO5lQ71zQ.png" /></figure><p>In the subnet of the scanning VPC we added an IPv6 route back toward 2a05:d010::/28 since we are creating this VPC in the Ireland region of AWS and we want to enable return traffic for the NAT64 requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_58Bfz6yNNwnbSNEo8Uw9Q.png" /></figure><p>The instance 10.64.32.16 in the account of team A wants to connect to the public website of PostNL 2.19.176.147. It uses again the default route to go the the core network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZxkf_Y_kJ3YWonrS8ZBVg.png" /></figure><p>In the “nonprod” segment is also uses the default route to go the the scanning VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xu_d2amatJcqjZ9XwyO4_Q.png" /></figure><p>The NAT Gateway creates a session using its public IP to connect to the PostNL website and receives a response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lh1PlJLy8nV787l-5Gv7qw.png" /></figure><p>The NAT Gateway maps the IP back to the private ip-address and uses its route 10.0.0.0/8 route to the core network to connect to the scanning Network Function Group routing table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xGEix-UQjtooa43qDNcww.png" /></figure><p>In the routing table of the Network Function Group there is a route 10.64.32.0/22 to the instance of team A.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nUni_ixNHmavIkXgSxJQ8A.png" /></figure><p>The costs of this traffic is $0,158 per GB ( data out)</p><h3>Packet towards the internet using IPv6</h3><p>In this scenario the instance in the account of team A is using IPv6 wants to connect to the IPv6 address (2a02:26f0:c900:b::5f65:4a3) of the website of PostNL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIt_bzrSTjg9kz_WKwDjqw.png" /></figure><p>Since the default route for IPv6 (::/0) is pointing to the EgressOnly Internet Gateway it can directly connect to the website, this way the traffic only costs $0.09 per GB for data out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1rSlGdr9aL5ZPkZVaaSm-A.png" /></figure><h3>Packet from IPv6 towards a local IPv4 destination</h3><p>To understand this flow it is needed to share all the routing tables again. In this scenario the IPv4 route is needed in the “nonprod” segment so it can reach the scanning VPC again. ( That is why we injected the IPv4 route to 10.79.192.0)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjErh1_qiU3xQ98xW0FsZg.png" /></figure><p>In this scenario the instance of team A (2a05:d018:1586:7b05:390a::/80) wants to connect to the ALB of team B. Team B did not create the ALB dual stack so it needs to connect to an IPv4 address. Since this is not possible the DNS64 setting in the subnet, transforms the IPv4 address into a special IPv6 address, a 64 address (64:ff96:::a40:3c22/96)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zlQ6o0XzUm0tKd-v-R0fA.png" /></figure><p>Lets follow the flow again, the IPv6 traffic is green and the IPv4 traffic is yellow. In the routing table of team a there is a 64:ff96::/96 towards the core network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y2vDFrGFNKBIFwcIXeRAEg.png" /></figure><p>The “nonprod” segment has the 64:ff96::/96 route toward the scanning VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*craxO9jTX67ni7hZEhGR2g.png" /></figure><p>The scanning VPC has the Nat Gateway and that translates the destination 64:ff96:::a40:3c22/96 to an IPv4 destination 10.64.60.34 and sends the package to the scanning function group following the 10.0.0.0/8 to the core network using a 10.79.192.12 address as source address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H47RLAWgVedLhhjHqaPYeA.png" /></figure><p>The core network routing table has a route to the VPC of team B (10.68.32.0/22 team B)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIRmkzxxp39RU8MIPrwbFw.png" /></figure><p>The ALB returns the traffic to source 10.79.192.12 using the 0.0.0.0/0 route towards the core network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ElhdAysr9IlgT-E-mEjdJQ.png" /></figure><p>The routing table of the “nonprod” segment has a route (10.79.192.0/20 scan vpc) towards the scanning VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PMogeh00R0xBCIoHg0mTKw.png" /></figure><p>The NAT Gateway of the scanning VPC now translates the packet to IPv6 and needs to return the response to 2a05:d018:1586:7b05:390a::/80. It uses the route 2a05:d010::/28 towards the core network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfXGkLIJqcB9dJ0vASSaAg.png" /></figure><p>The scanning Network Function Group route table has a route back to the VPC of team A ( 2a05:d018:821::/56 vpc team A)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKH8g7dDwmM9XMsitPuFFw.png" /></figure><p>The costs of this traffic will be $0,068 a GB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TcjNA22RpxB3Kn83Tyj2Rw.png" /></figure><p>Remarks on this flow:</p><ul><li>This only works if the ALB allows the NAT Gateway to connect to it so the security group needs to allow it.</li><li>If team B would have a IPv6 address on the ALB it would not be able to connect to it, since the ALB is in the private subnet and the default route is set towards the Egress Only Internet Gateway. So i should be in the public subnet of team B.</li><li>When you enable DNS64, IPv4 addresses will be resolved as 64 addresses adding complex flows and this can lead to unexpected flows.</li><li>We block the nonprod segment to talk to the prod segment. But using NAT64 this its still possible to connect from IPv6 nonprod to an cprod via the NAT Gateway. To prevent this we inserted blackhole routes into the segment routing tables.</li></ul><pre>{<br>      &quot;segment&quot;: &quot;nonprod&quot;,<br>      &quot;destinations&quot;: [<br>        &quot;blackhole&quot;<br>      ],<br>      &quot;action&quot;: &quot;create-route&quot;,<br>      &quot;destination-cidr-blocks&quot;: [<br>        &quot;10.64.0.0/14&quot;,<br>        &quot;64:ff9b::a40:0/110&quot;<br>      ]<br>    },</pre><h3>Take aways</h3><p>It is important to understand the flows of your application. AWS is adding a lot of IPv6 support on all sorts of resources. So if you are considering to start using the IPv6 addresses in your VPC please make sure your targets also support it, now a days most of them support it.</p><p><a href="https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html">AWS services that support IPv6</a></p><p>For the targets that do not support it there is the option of using DNS64 and NAT64 in your account.</p><p><a href="https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html">https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html</a></p><p>Please know that if you are using a centralized NAT Gateway the traffic towards the IPv4 resource will not come from your resource but from the Nat Gateway.</p><p>If you are using this in combination with CloudWan you need to add the return route towards the egress VPC if you use route summarization towards other destinations.</p><p>And mind that not sharing segments with others or denying segments to communicate with each other can be bypassed using NAT64 so add blackhole routes!</p><p><strong>Cost difference per flow:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3r39dbEnMMI23aDFWsg9sw.png" /></figure><p>Notes:</p><ul><li>All prices are per GB and may vary by region and over time, so always check <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/">the cost explorer</a>.</li><li>NAT Gateway costs are higher for internet-bound IPv4 traffic due to NAT processing and data out charges.</li><li>IPv6 direct egress is cheaper than IPv4 via NAT Gateway.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0408e8d1c726" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/demystify-dual-stack-on-aws-in-combination-with-cloudwan-and-network-function-with-send-to-0408e8d1c726">Demystify dual stack on AWS in combination with CloudWan and Network Function with send-to.</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overcoming the GitHub/CodeBuild limitations at scale and getting in control of GitHub actions]]></title>
            <link>https://medium.com/postnl-engineering/overcoming-the-github-codebuild-limitations-at-scale-and-getting-in-control-of-github-actions-7aa2232394f1?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/7aa2232394f1</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-codebuild]]></category>
            <dc:creator><![CDATA[Gideon Vrijhoeven]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 08:23:42 GMT</pubDate>
            <atom:updated>2026-03-27T09:12:34.978Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overcoming the GitHub/CodeBuild limitations at scale and getting in control of GitHub actions during supply chain attacks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rLNWbmAix7IWZ_zsXWaLiQ.jpeg" /></figure><h3>Introduction</h3><p>PostNL wanted to use CodeBuild as a self hosted runner solution to give teams more flexibility in compute, instance size, but also make it possible to do integration and end-to-end tests with private endpoints as explained in more detail in these blogs:</p><ul><li><a href="https://medium.com/postnl-engineering/securing-private-apis-with-security-group-referencing-6021a77f3f38">Securing Private APIs with Security Group Referencing</a></li><li><a href="https://medium.com/postnl-engineering/building-scalable-ci-cd-pipelines-with-self-hosted-github-actions-on-amazon-codebuild-6a82150a3eb2">Building Scalable CI/CD Pipelines with Self-Hosted GitHub Actions on Amazon CodeBuild</a></li></ul><p>GitHub has a ‘hard limit’ of 20 webhooks and CodeBuild throttles the webhook calls towards CodeBuild resulting in hanging jobs.</p><p><a href="https://docs.github.com/en/webhooks/types-of-webhooks">Types of webhooks - GitHub Docs</a></p><p>Last month the world has seen a few supply <a href="https://socket.dev/blog/ongoing-supply-chain-attack-targets-crowdstrike-npm-packages">chain attacks</a>. It is always a risk to use the newest code, as there are people that use source code to create worms that can cause a security leak and expose credentials.</p><p>When the full impact of the NPM worm became clear to us, we disabled the GitHub actions. This impact was big, since no teams were able to deploy new releases. So we started to think about a more permanent solution. Some of the teams were using JFrog as an intermediate, but we needed to force this on all repository configurations of the teams and all developers workstations. In this blog post we will focus only on the pipeline part of this issue.</p><h3>GitHub and CodeBuild challenges</h3><p>By default each CodeBuild project creates a webhook in GitHub. Doing this we bumped into a GitHub limitation that only 20 webhooks are supported. In addition, we noticed that AWS also throttles calls towards CodeBuild which results in cases where CodeBuild projects are not started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQfcp55hst-dJf4vTea-Hw.png" /></figure><p><strong><em>The fix</em></strong></p><p>To fix both issues, we designed a proxy webhook that listens to GitHub. Since we needed to initiate CodeBuild, we needed to manually create the CodeBuild project webhooks, save the URL and Secret and store that in a Secret. Create a specific Lambda that is allowed to call the secret. Since GitHub is public we wanted to put our CDN Akamai in front of the webhook calling an application loadbalancer that only allows traffic coming from Akamai triggering the proxy Lambda.</p><p>This puts an event on EventBridge where we create a EventRule for each repository. The event is put in a SQS Queue so we can redrive failed events, triggering the Lambda that calls the CodeBuild project. There are two possible flows. For specific repositories the filter triggers a lambda that starts a CodeBuild within the default VPC. This enables integration tests over the private network. When there is no rule that matches the repository a generic CodeBuild is triggered on the Firewall VPC. Since PostNL has multiple Github Organizations two different endpoints are exposed to the public internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MIsrbkgjWpwD2HMxFsJz7A.png" /></figure><p><strong><em>Observability</em></strong></p><p>Since we do this in a central account and we want teams to be able to view the status of the CodeBuild project, we also created a RAM share for the CodeBuild project towards the deployment account of the team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*Wz7YTbwDOf3yep6orunnjg.png" /></figure><p>For teams it is possible now to access information regarding the status of the projects from the RAM console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqJfLI-c_M10VnF5gFEXNw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wtoDcxVseAFIXB1FVuxC_w.png" /></figure><h3>Pipeline challenges to block unwanted traffic</h3><p>To be able to inspect traffic and block unwanted traffic on AWS, it is needed to put the resource in a VPC. This allows traffic to be routed via a firewall. Since PostNL is “serverless only” we used the AWS Firewall.</p><p>We wanted to make a distinction<strong> </strong>between the runners that need internal network connectivity to perform integration tests with private endpoints and the ones that don’t. So we created an additional VPC that is not connected to the Transit Gateway.</p><p>It is possible with AWS firewall to create endpoints in other VPC’s so the same rules are applied on both flows. Since we wanted to reduce the traffic towards the internet we added CloudFormation endpoints in the subnets the CodeBuild projects are created. In the standard VPC we needed to inject an additional scanning subnet to be able to add a default route from the private subnet towards the firewall endpoint and since we all-ready centralized our egress traffic towards the Transit Gateway we could use that default route to be able to connect to the internet.</p><p>So we created this design:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RSZCWY834vRuJfQ9NXHNTg.png" /></figure><h3>Issues</h3><p><em>Security</em></p><p>This left us with an issue that the webhooks created by CodeBuild would keep the secret for as long as the project exists. Since this is a public endpoint we created an EventRule schedule to recreate the webhook by triggering a Lambda.</p><p><em>Order of the firewall endpoints</em></p><p>When creating firewall endpoints they need to be alligned with the AZ they are created in to <a href="https://docs.aws.amazon.com/network-firewall/latest/developerguide/asymmetric-routing.html">avoid asymmetrical routing</a>. So we needed to implement a script that puts the endpoints in the correct order.</p><pre>import * as cdk from &#39;aws-cdk-lib&#39;;<br>import {<br>  aws_ec2 as ec2,<br>  aws_networkfirewall as networkfirewall,<br>} from &#39;aws-cdk-lib&#39;;<br><br>/** this is actually some voodoo magic here...<br> * we select firewall subnet from vpc, assuming (hoping...) vpc.availabilityZones are in order of AZs<br> * this means we require `firewall.attrEndpointIds` to be in order.<br> * however, `firewall.attrEndpointIds` is not in order for some inexplicable reasons.<br> * https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-networkfirewall/issues/15<br> * `firewall.attrEndpointIds` is a List-encoded tokens - https://docs.aws.amazon.com/cdk/v2/guide/tokens.html#tokens_list<br> * in order to sort `firewall.attrEndpointIds` in order of AZs, we need to use a combination of splits and joins.<br> * AWS AZs are ordered lexicographically starting at `a`, and &quot;assuming&quot; vpc.availabilityZones are returned in order of AZs<br> * to sort `firewall.attrEndpointIds` as a List-encoded tokens, the following steps are taken:<br> * 1. join `firewall.attrEndpointIds` as a string token with `,` as delimiter.<br> *   -&gt; &quot;eu-west-1c:vpce-111122223333,eu-west-1a:vpce-987654321098,eu-west-1b:vpce-012345678901&quot;<br> * 2. split the resulting string token `azWithFirewallVpcEndpointIds` with `a:` for the first AZ (`b:` and `c:` subsequently).<br> *   -&gt; [ &quot;eu-west-1c:vpce-111122223333,eu-west-1&quot;, &quot;vpce-987654321098,eu-west-1b:vpce-012345678901&quot; ]<br> * 3. select 2nd element as it contains the endpoint ID, along with some extra garbage and store it into var `temp`<br> *   -&gt; &quot;vpce-987654321098,eu-west-1b:vpce-012345678901&quot;<br> * 4. split `temp` by `,` as that was the joined list delimiter<br> *   -&gt; [ &quot;vpce-987654321098&quot;, &quot;eu-west-1b:vpce-012345678901&quot; ]<br> * 5. select 1st element as it contains the endpoint ID without any extra garbage now<br> *   -&gt; &quot;vpce-987654321098&quot;<br> * https://carriagereturn.nl/aws/cloudformation/firewall/endpoint/routing/2022/05/22/firewall-endpoints.html<br> * https://stackoverflow.com/questions/75204247/how-can-i-map-the-subnet-and-its-routing-table-to-the-firewall-endpoint-in-the-s<br> */<br>export function getFirewallEndpointsInOrder(<br>  firewall: networkfirewall.CfnFirewall,<br>  vpc: ec2.Vpc,<br>): string[] {<br>  const firewallVpcEndpointIds: string[] = [];<br>  const azWithFirewallVpcEndpointIds = cdk.Fn.join(&#39;,&#39;, firewall.attrEndpointIds);<br>  const aZs = [&#39;a&#39;, &#39;b&#39;, &#39;c&#39;];<br>  for (let i = 0; i &lt; vpc.availabilityZones.length; i += 1) {<br>    const temp = cdk.Fn.split(`${aZs[i]}:`, azWithFirewallVpcEndpointIds, 2)[1];<br>    const firewallVpcEndpointId = cdk.Fn.split(&#39;,&#39;, temp, 2)[0];<br>    firewallVpcEndpointIds.push(firewallVpcEndpointId);<br>  }<br>  return firewallVpcEndpointIds;<br>}</pre><h3>GitHub implementation:</h3><p>To start using Self Hosted Runners you have to alter the run-on in your jobs Like so:</p><pre>runs-on: codebuild-${{ vars.SELF_HOSTED_RUNNER_PROJECT }}-${{ github.run_id }}-${{ github.run_attempt }}</pre><p>In a workflow file:</p><pre>generic-ping-jfrog-job:         <br>  runs-on: codebuild-${{ vars.SELF_HOSTED_RUNNER_PROJECT }}-${{ github.run_id }}-${{ github.run_attempt }}          <br>  permissions:             <br>    id-token: write             <br>    contents: read         <br><br>steps:             <br>- name: Checkout gitgcode               <br>  uses: actions/checkout@v4             - <br>  name: Ping Jfrog               <br>  run: curl -I lpe.jfrog.io</pre><p>The SELF_HOSTED_RUNNER_PROJECT organization environment variabel is replaced with the Project Name of your Organization.</p><h3>Conclusion</h3><p>The standard GitHub Hosted Runners do not give the option to centrally control the access from the actions. To enhance the security it is possible to use the private hosted runners running on CodeBuild, since those runners can be attached to a VPC and it is possible to firewall the traffic.</p><p>In addition, private runners also allow flexibility in size, compute and allow you to do integration tests with private endpoints.</p><p>Due to webhook limitations this is limited to 20 projects and at scale you will hit the limits of the AWS CodeBuild API. By implementing our own webhook integrations with GitHub is is possible to bypass these limitations and centrally control the access from GitHub actions towards the internet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7aa2232394f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/overcoming-the-github-codebuild-limitations-at-scale-and-getting-in-control-of-github-actions-7aa2232394f1">Overcoming the GitHub/CodeBuild limitations at scale and getting in control of GitHub actions</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feedback Loops in the Age of AI and Cloud Development]]></title>
            <link>https://medium.com/postnl-engineering/feedback-loops-in-the-age-of-ai-and-cloud-development-54b380709f6e?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/54b380709f6e</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[feedback-loop]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Selcuk Sasoglu]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 10:52:43 GMT</pubDate>
            <atom:updated>2025-10-31T10:52:41.821Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4wmyLUzkf7CzOCzWVMRpA.png" /><figcaption>A roller coaster looping around</figcaption></figure><p>Even if you are not aware, you are using feedback loops to build software every day. Your team’s performance is actually highly correlated with how you utilize these feedback loops. During a recent development project, I once more saw firsthand how crucial feedback loops are in software development. In this blog post, I would also like to remind you of how important they are.</p><p>Every project has its own difficulties, and every team can make mistakes. This is normal in software development. What differentiates high-performing teams from the others generally boils down to what they do with the feedback they get. The feedback could be a failing test case from the pipeline, a monitoring alarm going off on production, or a sneaky bug that managed to pass all existing quality gates and made its way to production.</p><blockquote>Sneaky bugs require a post-mortem, an analysis to understand how all quality gates were bypassed. The questions that need to be asked here are “What is the possibility of this ever happening again?” and, if it is probable, “What can we do to prevent it ever happening again?</blockquote><h4>Cost of fixing defects</h4><p>Before anything else, I would like to show you this research from IBM, <a href="https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software_Development_Life_Cycle_SDLC">Integrating Software Assurance into the Software Development Life Cycle (SDLC)</a>. As you can see from the graphic below, noticing a design flaw in the earlier stages of development can be 100x cheaper than noticing it on production. If you notice it during implementation, it is 13x cheaper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8-ANjGp9DGbwVQZZYtm9A.jpeg" /><figcaption>Relative cost of fixing defects</figcaption></figure><p>I want you to analyze this graph as if you are investing money, time, and resources; “Think like an investor” here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wpi1cZ3RXRXN2X8G7Pna8w.jpeg" /><figcaption>Exponential pay off</figcaption></figure><p>If you can create a feedback loop that will find a bug, detect a design flaw or similar in the earlier stages of development, you can actually get an exponential payoff.</p><h4>Real world examples from a recent project</h4><p>With a group of highly skilled and experienced engineers, we worked on a high-pressure, hard-deadline project that would allow significant cost savings for our company. Since we had a hard deadline, time became a constant for us in the <a href="https://en.wikipedia.org/wiki/Project_management_triangle">Iron Triangle</a>, which also made time scarce for us. We had to make sure that every effort and every moment we spent mattered for the end result. We constantly balanced scope and quality to meet the deadline.</p><p>What I noticed during this period was the effectiveness of developer tools. Small investments we made in developer tools, created feedback loops which paid off so quickly that the time poured into these small developer tools became insignificant even under the high-pressure deadline.</p><p>Our configuration-heavy project had a few manual steps (copy/paste) to configure environments. Required time to completely automate this process was ~10000x more expensive than keeping it manual. However, an hour invested in comparing the configurations per environment in the CI/CD pipeline saved hours of work compared to the moment we realized that we actually deployed wrong configuration to production.</p><p>Another important advantage of working with experienced engineers was that most common feedback loops immediately found their place in our SDLC (Software Development Lifecycle). There have been many cases of active discussions to add more quality gates that will give us feedback earlier to notice problems before they occur on production. What are these common feedback loops? Where should you start? Are there specific tools that can help you create these feedback loops? Can you or should you take additional actions if you are working on the cloud, using Generative-AI? Let’s discuss these questions in the following sections.</p><h4>Where should you start?</h4><p>DevOps Infinity Cycle must be the starting point when working on feedback loops in your SDLC (Software Development Lifecycle). In the era of AI and cloud development, we have more tools to build on this construct. In the following sections let’s discuss <strong>some possibilities.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lp4kJmuyZzzJts7hsxfEBA.jpeg" /><figcaption>DevOps Infinity Cycle</figcaption></figure><p>Let’s go over the DevOps Infinity Cycle stages together and discuss one by one what could be good options.</p><h3>DevOps Infinity Cycle</h3><h4>Plan</h4><p>Everything starts with a good plan. If you check the <strong>Cost of fixing defects</strong> section, any bug, design flaw that we can notice in the Plan phase can be 100x cheaper to fix.</p><p>At this phase, it is imperative that you understand the domain and question Why/What/How clearly about the project. There are many possible approaches here. One I like to use is <a href="https://citoolkit.com/articles/5w1h/">5W1H</a> approach. 5W1H is a problem-solving approach that helps in obtaining fundamental information about a given problem or situation. This type of checklist is designed to guide the gathering of information and understanding of any current state using six key words: What/Who/Where/When/Why/How.</p><p>You should ask questions like:</p><ul><li>What is the problem statement?</li><li>Why are you building this particular project or feature? Is it a good investment?</li><li>Who will be the main audience?</li><li>How should you solve this problem? What kind of solution would strike the balance between cost and value generation?</li></ul><p>When the answers to above questions are clear, next step is to also have a <em>Testing Strategy</em>. Determine how you will validate your assumptions and solution.</p><p>If you are planning to build your solution on the cloud, you should check your assumptions against the best practices and frameworks provided by the cloud provider. For example, if you are building on AWS, you should check your assumptions against <a href="https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html">AWS Well Architected Framework</a>. Do your assumptions still hold or should you go back to the drawing board?</p><p><strong>Generative-AI</strong></p><p>Use Generative AI for documentation, note taking in meetings, summaries of your decisions, gap analysis or to create ADRs.</p><p>Create your own <a href="https://www.selcuksasoglu.com/2025/03/14/predictions-for-generative-ai/">context</a>.</p><p>Did you know that according to <a href="https://www.devops-research.com/research.html">DORA Report 2023</a>, quality of internal documentation has an almost 13-fold impact on organizational performance? Guess who is good at writing documentation? Yes, Generative-AI can do that for you once you give the correct template to it.</p><p><strong>Code</strong>, <strong>Build</strong> and <strong>Test</strong> phases will form a feedback loop by themselves and these phases will be the ones that you will be looping over and over again at the beginning of a project. This loop here should execute like a powerful engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iu3-gQddaiTxjm2Iqj69Pw.jpeg" /><figcaption>Code-Build-Test loop in DevOps Infinity Cycle</figcaption></figure><p>Try to make investments here to make this loop as seamless as possible! This is the place where your investments can really matter.</p><h4>Code</h4><p>Coding phase must be intertwined with automated testing. Automated testing creates fantastic feedback loops for us. We should use it to its full extend. Testable code has also high correlation with maintainable code. We should start with automated testing together with development.</p><p>There are also other tools here that can create valuable feedback loops for us. Let’s name a few.</p><p>Use a static code analysis tool (SAST — Static Application Security Testing) during development. Simple stuff that you can miss in code reviews can be caught by these tools. Some of them also helps you track your code coverage with tests. Use this as a feedback loop to improve your tests. Have thresholds.</p><p>You can already integrate observability (o11y) when you start development and trust me, it will make a huge difference in your development especially with code bases that have complex computations and interactions. We will touch on this again in <strong>Test</strong> phase.</p><p>Enforce mandatory <em>Code Reviews</em>, use AI where you can. Code Reviews can create bottlenecks in DevOps Team, but they are great for both as a quality gate and also collaboration moment within the team. AI code reviewers can help you with Code Reviews and create a faster feedback loop.</p><p>Invest in developer tools. As my example in <strong><em>Real world examples from a recent project</em></strong>, one hour of investment in a developer tool can save hundreds of hours. If you can prevent an error before a production deployment, your investment has already paid for itself. Or a production error can be a trigger to create a developer tool to prevent it ever happening again.</p><blockquote>Think like an investor.</blockquote><p>If you are working on the cloud, you can consider options that will allow you to build more on your local machine instead of deploying to the cloud. This approach will allow you to create a faster feedback loop and will also help you to reduce costs. For example, if you are working on AWS, you can use <a href="https://www.localstack.cloud/">LocalStack</a> to build your functionality faster.</p><p><strong>Generative-AI</strong></p><p>Use an AI assistant to help you with coding and Code Reviews. Push it to its limits to find bugs as early as possible.</p><h4>Build</h4><p>I hope I don’t need to write in detail to use a CI/CD pipeline to automate tasks for build/test/publish/deployment anymore. Hopefully you are already doing it.</p><p>What I can tell instead is to keep an eye on your pipeline runs and optimize them. You should set thresholds for how much time each step is supposed to take and take vigorous action if your thresholds are breached.</p><p>Integrate and scan for security at early stages. Heed to what Dependabot says about your package vulnerabilities. Adhere to best practices against supply chain attacks depending on your programming language.</p><h4>Test</h4><p>If you have already did your homework in <strong>Plan</strong> phase, you should already have a <em>Testing Strategy</em>. Create your <em>Testing Plan</em> in detail according to your <em>Testing Strategy</em>. You should layer your tests in such a way that crucial functionality is covered at least in one layer and overlap between testing layers (unit/integration/end-to-end) is as small as possible for efficiency.</p><p>Since we already have pipeline, we should also keep an eye on testing duration and optimize when necessary.</p><p>Using a DAST (Dynamic Application Security Testing) tool can help you find vulnerabilities in your application immediately during testing and you can prevent them from being deployed to production.</p><p>There will be parts in the codebase where you can test locally and there will be parts you can’t unless you make a deployment.</p><p>Here, it is crucial to make a good decision on what you can test locally. Because local testing will be faster, cheaper and ~13 times more efficient. It will give you a faster feedback loop on what doesn’t work. Maybe a code structure change just to be able to test locally is not a bad investment?</p><p>Guess what? O11y integration helps with testing as well. If you run a local Jaeger container and point your codebase to it, you can collect <em>OpenTelemetry Events</em> directly from your tests. This approach actually makes a huge difference in pinpointing bugs and understanding the behavior of your codebase with low cognitive overload. Switching to another <em>OpenTelemetry Backend</em> is just a configuration change when deployed to the cloud.</p><p><strong>Generative-AI</strong></p><p>You probably already know but Generative-AI is pretty good at writing tests once you establish standards. You can leverage this capability and increase your test coverage significantly.</p><p>Testing is one of the most powerful feedback loops that we have. We should use it to its full extend.</p><h4>Release</h4><p>During this phase, the most important thing you should do is to generate release notes and use semantic versioning so that you can refer back to the changes in the future.</p><p>The quality of the phases after this one will rely on versioning created in this phase.</p><h4>Deploy</h4><p>In deployment phase, it is imperative to deploy using a CI/CD pipeline. This automated process will allow you to:</p><ul><li>Keep an eye on how long does it take to deploy a new version.</li><li>Collect other metrics on your deployments (i.e. <a href="https://en.wikipedia.org/wiki/DevOps_Research_and_Assessment">DORA Metrics</a>).</li><li>Run your DAST (Dynamic Application Security Testing) tool to find vulnerabilities in your application before deployment to production.</li></ul><p>DORA Metrics are core to any DevOps team. Additionally, I personally like to track the duration of the deployments and use it as a feedback loop to optimize the deployment process.</p><h4>Operate</h4><p>Before we get to the Operate phase, we should have a proper incident management process to support our users.</p><p>We should also gather data about the usage of our application. This data can be used to feed back to the earlier stages of DevOps Infinity Cycle.</p><p>We should also add alerts to business metrics.</p><p><strong>Generative-AI</strong></p><p>Generative-AI can be used to pinpoint root causes of problems faster during incident management. Leverage this capability to create a feedback loop for your incident management process.</p><h4>Monitor</h4><p>For monitoring, health metrics for all your critical resources are crucial.</p><p>If you have been integrating o11y (Observability) during development, you should now have a good understanding of your applications behavior during normal and faulty operation. This will help significantly for incident management as well.</p><p>Keep running your DAST (Dynamic Application Security Testing) tool to find vulnerabilities in your application and fix them proactively.</p><p>Now that we have covered all phases of the DevOps Infinity Cycle, let’s discuss how we can create feedback loops for our team and organization. The feedback loops we create within DevOps Infinity Cycle will help us during software development. Feedback loops we create on top of them in “Team Level” and “Organization Level” will help us to improve the engineering excellence to the next level.</p><h3>Feedback Loops for your Team</h3><p>DevOps Teams must also invest in creating a <a href="https://en.wikipedia.org/wiki/PDCA">PDCA (Plan-Do-Check-Act) cycle</a> to improve the engineering excellence through continuous improvement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_WEdbWe9vWzXd3P6msiKpQ.png" /><figcaption><a href="https://en.wikipedia.org/wiki/PDCA">PDCA (Plan-Do-Check-Act) cycle</a></figcaption></figure><p>Retrospective meetings are the great to utilize for this activity. I know, you find them boring and you don’t want to talk about how you feel in front of your team members. I believe the way to decrease friction is to talk through numbers.</p><p>Use retrospective meetings effectively:</p><ul><li>Have a self assessment on team happiness and overall team performance.</li><li>Go over your DORA metrics to detect fluctuations and assign tasks to do root cause analysis.</li><li>Determine (new) quality gates and make it a standard.</li></ul><p>In my teams, I created a 5 statement anonymous survey to get feedback from the team at the end of every sprint. This allows us to assign numbers to the general feeling of the team and opens the door for more detailed feedback and discussions during the retrospective meeting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4Q6o8u_XLPk6pbeQ_aFgg.jpeg" /><figcaption>Team members’ survey to evaluate the sprint</figcaption></figure><p>What matters is the progress and visualizing the progress. Through Kaizen (small iterative improvements), this approach creates a feedback loop on team level and you can take action if you notice problems. A detected problem can be a trigger for a new quality gate and can become a new standard in the <a href="https://en.wikipedia.org/wiki/PDCA">PDCA (Plan-Do-Check-Act)</a> cycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*szpGtO5tODS2cCN4mZlgEQ.jpeg" /><figcaption>Team progress after PDCA cycles.</figcaption></figure><h4>Feedback Loops for your Organization</h4><p>Organisations also need a feedback loop to improve their decision making. In my experience, organizations who are working with engineering teams must also collect DORA Metrics per team.</p><p>We have many strategic decisions and from upper management it is extremely difficult to notice the effects of these decisions.</p><p>Beauty of DORA Metrics is that they are universal. It doesn’t matter what kind of application you are building. If all teams were to share their performance metrics on organizational level, the organization would also have its own metrics.</p><p>We already know that the shared data must not be used to compare teams. The teams are operating in different contexts. It would be comparing apples with oranges which doesn’t make any sense. But if say, all teams were reporting their DORA Metrics and we simply observe the trends of these metrics, it would allow possibility to:</p><ul><li>Visualize the performance of an individual team over a timeline.</li><li>This will allow you to self-reflect on your team’s performance.</li><li>Let the upper management to see the performance and decide better on where to invest in your team.</li><li>Create formulas on organizational level to visualize and reflect on upper management decisions looking at the overall context of all teams.</li></ul><h4>Conclusion</h4><p>I started this article showing you the cost of fixing defects in different stages of the development lifecycle. Then I gave you a real world examples of how feedback loops can help you save your time and resources. We went over the DevOps Infinity Cycle and discussed how you can create feedback loops for your team and organization. To highlight the most important points:</p><ul><li>Spend just enough time on planning, not too little, not too much. Try to find the sweet spot.</li><li>Focus on feedback loops where you iterate the most (Code/Build/Test). Any defect detected in these phases can be fixed quickly and with low cost.</li><li>“Think like an investor”. If you can create a feedback loop that will find a bug, detect a design flaw or similar in the earlier stages of development, you can actually get an exponential payoff.</li><li>Set thresholds for testing, deployments, and other metrics (DORA Metrics) and go after optimizations vigorously if thresholds are breached.</li><li>Create a PDCA cycle for your team. This will help you to improve the engineering excellence through continuous improvement.</li><li>Help your organization create feedback loops. Allow them to reflect on their decisions and make better investments.</li></ul><p>During this journey we added additional actions you can take if you are working on the cloud, and if your team is using Generative-AI.</p><p>I hope you enjoyed reading this article. If I managed to remind you the importance of feedback loops and spark a few ideas on how you can create a few for your team and organization, then my target is achieved. Thank you for reading.</p><p>Happy coding!</p><h4>References</h4><ul><li><a href="https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software_Development_Life_Cycle_SDLC">Integrating Software Assurance into the Software Development Life Cycle (SDLC)</a></li><li><a href="https://en.wikipedia.org/wiki/DevOps_Research_and_Assessment">DORA Metrics</a></li><li><a href="https://www.devops-research.com/research.html">DORA Report 2023</a></li><li><a href="https://citoolkit.com/articles/5w1h/">5W1H</a></li><li><a href="https://www.devops-research.com/infinity-cycle.html">DevOps Infinity Cycle</a></li><li><a href="https://en.wikipedia.org/wiki/PDCA">PDCA</a></li><li><a href="https://opentelemetry.io/">OpenTelemetry</a></li><li><a href="https://www.splunk.com/en_us/what-is-observability.html">Observability</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54b380709f6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/feedback-loops-in-the-age-of-ai-and-cloud-development-54b380709f6e">Feedback Loops in the Age of AI and Cloud Development</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Networking Costs in AWS VPCs with IPv4]]></title>
            <link>https://medium.com/postnl-engineering/optimizing-networking-costs-in-aws-vpcs-with-ipv4-b4349c35f865?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/b4349c35f865</guid>
            <category><![CDATA[aws-vpc]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Gideon Vrijhoeven]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 07:44:35 GMT</pubDate>
            <atom:updated>2025-09-15T07:54:00.078Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjAx1r3C_3K1-lXIzVe2Qw.jpeg" /></figure><p>This blog post aims to help you identify and address cost drivers for networking in your AWS environment. The AWS services provide a wide range of options for exposing and consuming services, but besides security and reliability, cost should also be a key factor in your architectural decisions.</p><p>We will focus on the basics of VPC connectivity and emphasize the importance of using the AWS Cost Explorer to understand your current networking costs, as they can change over time and vary by region.</p><p>Networking service costs can be divided into three main categories:</p><ol><li>Data transfer costs</li><li>Service usage costs</li><li>Service data processing costs</li></ol><p>We will explore each of these in detail and provide examples to help you understand the potential impact on your costs.</p><h3><strong>Data transfer costs</strong></h3><p>Data transfer costs are incurred when data packets cross certain boundaries, such as:</p><ul><li>Availability Zones (in and out)</li><li>Regions (out)</li><li>The internet (out)</li></ul><p>It’s important to note that traffic coming into a region from another region, as well as traffic coming into AWS from the internet, is free of charge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OdF-e4MwIS-bkm8cXW6GOg.png" /></figure><p>Zooming in on the costs:</p><p><strong>Data transfer costs cross AZ:</strong></p><ul><li>data out</li><li>data in</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRYMKU-Ss-PCkKqjiUsDrA.png" /></figure><p>Data transfer costs between Availability Zones can be reduced by using NAT Gateways and Gateway Load Balancers, depending on your traffic volumes. However, this approach may have availability trade-offs, so it’s essential to carefully consider your requirements. This saves money when you have less then 43.80 / 0.02 = 2190 GB of traffic, but it could accure hidden costs, like down time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mF0XqFsSxiSQ-5LoCZcZ4g.png" /></figure><h3>Service usage costs</h3><p>In addition to data transfer costs, you also need to account for the service usage costs associated with resources like NAT Gateways and Gateway Load Balancers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pHV1Yalx760Vn31sbPAsNw.png" /></figure><h3>Data processing costs</h3><p>Alongside data transfer and service usage costs, the data processing costs for these resources should also be considered. These costs are billed since the NAT Gateway and the Gateway Loadbalancer are using multiple high available EC2 resources. In some cases also capacity units ( e.g. number of connections) can be part of the costs. Let’s consider 100GB and 100TB scenarios.</p><p>100 GB scenario:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41srjBZpLhqN-7jQjbc9Vg.png" /></figure><p>100 TB scenario:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2gswG9wRXUi4D4b8nXvVg.png" /></figure><h3><strong>Summary</strong></h3><p>Combining all the costs will result in these total costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6TqI-PexXLBAmtWjMDlYw.png" /></figure><h3>Architectural examples</h3><p>When using <strong>Application Load Balancers</strong>, you need to factor in the service costs, data transfer costs, and data processing costs. To reduce <strong>IPv4</strong> costs, you can explore the use of Bring Your Own IP (BYOIP) addresses, which are free of charge. In a previous blog post, we shared our experience implementing this at <a href="https://medium.com/postnl-engineering/postnls-journey-to-byoip-on-aws-managing-costs-and-control-with-custom-cidr-ranges-3cd4c8d01019">PostNL</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4wflPgegHP6r5bkQzYJ1bQ.png" /></figure><p>For <strong>Network Load Balancers </strong>and <strong>Gateway Load Balancers</strong>, it’s essential to understand the impact of Availability Zone affinity configurations. If not configured correctly, cross-AZ data transfer costs can be incurred.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4Vq4aHUz3zXH3SNqA11QQ.png" /></figure><p><strong>Gateway Endpoints</strong> for services like <strong>DynamoDB</strong> and <strong>S3</strong> are a must-have, as they allow internal traffic to bypass the NAT Gateway, resulting in cost savings. Without the gateway endpoints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YdAQQB2UbyagyM9kKoJoQw.png" /></figure><p>The Gateway Endpoints are free of charge and the data processing is also free!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RZ5Zk40SonKeN--UJSSukg.png" /></figure><p>How to connect to the other 170+ AWS services?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lJsYxid7GZoT90Qw467CJQ.png" /></figure><p><a href="https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html"><strong>Interface Endpoints</strong></a> are another option for connecting to other the other &gt;170 AWS services or third-party providers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*38T0MKqpRJ5ntM6kLIN22Q.png" /></figure><p>So for 100 TB of traffic the savings are a factor 4.</p><p>When deciding where to deploy these interface endpoints (shared services account or local account), you need to weigh the service costs against the additional Transit Gateway (TGW) costs.</p><p>cost per hour * hrs per month * 3 AZ’s / transfer cost per GB =</p><p>0.011* 730 = $8,03 * 3 AZ-s= $24,09 / 0.02 = &gt; 1204,5 GB</p><p>Its cheaper to centralize it. Besides the costs benefits, it is private, faster, more secure!</p><h3><strong>RDS data transfer costs</strong></h3><p>RDS instances can have a Primary node and Standby/Read nodes. The traffic to sync the RDS data between the instances is free of charge, the connections towards the RDS instance from the same AZ is free of charge. Clients connection from an other AZ are charged with AZ in and out costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2Gqbb1X0kjz_9lMV_WDhw.png" /></figure><h3>VPC-to-VPC Connectivity Costs</h3><h3><strong>Transit Gateway (TGW)</strong></h3><p>Data processing costs are $0.02 per GB. When peering TGWs across regions, you only pay the TGW costs once, but you need to add the region-out costs of $0.01 per GB. The attachment costs are $0.05 per hour in the eu-west-1 region, amounting to $36.50 per month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2B_97k9tO-rwphehx7rSA.png" /></figure><h3><strong>CloudWAN</strong></h3><p>An unused Transit Gateway is free, when you create a core network edge it will cost you $0.50 per hour per edge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rgBjMt0EMLfztlikwIJXGA.png" /></figure><p>Data processing costs are also $0.02 per GB. The Core Network Edge attachment costs vary by region, with $0.065 per hour in eu-west-1, adding up to $47.45 per month. In the Asia Pacific region, the attachment costs are $0.09 per hour, totaling $65.70 per month per attachment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYc8A96zUOzryHnVOpftiw.png" /></figure><h3>VPC Peering</h3><p>This is free of charge within the same Availability Zone. For cross-AZ traffic, you’ll need to pay $0.01 per GB out and $0.01 per GB in to the AZ. VPC Peering can be a cost-effective alternative for chatty VPC scenarios. For cross region also Region out costs apply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wmGwypMDphx587BWV8N_eA.png" /></figure><h3>To Centralize or not</h3><p>The decision to centralize <strong>NAT Gateways</strong> depends on the volume of internet-bound traffic. We’ll provide cost examples based on 1 TB, 100 TB, and 500 TB of monthly traffic across 100 VPCs to help you assess the potential impact, based on only two AZ.s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rMsHeLZ06PJQ9ftIbIz_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KnVXOrJuCxmpb19ySzIcPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cygPTpXvwla_6aQ9u6XtZQ.png" /></figure><p>In the case of 500TB of data traffic it is wise from a cost perspective to create local NAT Gateways.</p><p>When setting up <strong>centralized inspection across VPCs</strong>, be mindful of the various cost components involved.</p><p>There are fixed costs, like TGW attachments per hour, Gateway Load Balancer ( GWLB) endpoints per hour, GWLB service per hour and the costs of the applicable that performs the scanning.</p><p>The flexible costs are TGW data processing costs, GLBW processing costs, Loadbalancer Capacity Unit ( LCU), GWLB endpoint data processing costs and the costs of and the costs of the appliance that does the scanning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3gQQEC9e0ukh8kPmASxaw.png" /></figure><p><strong>Centralized inspection for egress</strong> using GWLB using an appliance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L3NrQwJ0CZTrHEV51CKfYA.png" /></figure><p><strong>Centralized inspection</strong> using the AWS Firewall:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EL3Y7cy9WShGWGLhnSSFtw.png" /></figure><h3>VPC Lattice</h3><p>VPC Lattice is a feature that allows you to configure application networking within your Virtual Private Cloud (VPC). With VPC Lattice, you can create service networks, and then attach services and VPCs to these networks.</p><p>Pricing for VPC Lattice Services:</p><ul><li>Charge per hour: $0.0275</li><li>Charge per GB of data processed: $0.0275</li><li>Charge per HTTP request (for HTTP/HTTPS listeners) or TCP connection (for TLS listeners) after the Always Free Tier: $0.11 per 1 million requests/connections</li></ul><p>Note:</p><ul><li>AWS only charges for requests received by services over the Always Free Tier of 300,000 requests/connections per hour.</li><li>VPC association and Service Network Endpoints are available at no additional cost.</li><li>Pricing for additional Regions will be announced when they become available.</li></ul><p>Pricing for Shared VPC Resources:</p><ul><li>Charge per GB of data processed when sharing VPC resources: $0.006</li></ul><p>Charges for Accessing VPC Resources:</p><ul><li>Charge per VPC resource you add to a service-network: $0.11 per hour</li></ul><p>Charges per GB of data processed for accessing VPC resources:</p><ul><li>First 1 PB: $0.01</li><li>Next 4 PB: $0.006</li><li>Anything over 5 PB: $0.004</li></ul><h3>Conclusion</h3><p>When designing new applications in AWS, organizations have a wide range of networking options to choose from. The Well-Architected Framework provides guidance, but it is essential to consider all pillars, including cost optimization, in architectural decisions.</p><p>It is recommended to continuously monitor application flows and evaluate whether past decisions remain suitable for current requirements. New features, such as VPC Lattice, may offer cost-saving opportunities by reducing the need to connect VPCs using a Transit Gateway, eliminating the need for a load balancer, and resolving conflicting CIDR ranges.</p><p>Its always a good idea to use cost allocation tags or account boundaries and or separate accounts for clear tracking of costs, you can think of setting up automated budget alerts and anomaly repots, so actions can be taken based on changed behaviour.</p><p>When planning cloud infrastructure, it is crucial to refer to the AWS Price Calculator for the latest pricing information, as costs can change over time and vary by region.</p><h3>Additional usefull pricing info</h3><p>There is a blog post that describes a self-hosted calculator to help estimate your AWS networking costs.</p><p><a href="https://aws.amazon.com/blogs/networking-and-content-delivery/estimate-aws-networking-costs-with-a-self-hosted-calculator/">Estimate AWS networking costs with a self-hosted calculator | Amazon Web Services</a></p><h3>References</h3><p>This content is based on a routing loop session from November 14, 2024, which can be found on <a href="https://www.youtube.com/watch?v=5jfKQxbkjd4">YouTube</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4349c35f865" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/optimizing-networking-costs-in-aws-vpcs-with-ipv4-b4349c35f865">Optimizing Networking Costs in AWS VPCs with IPv4</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VPC Block Public Access at scale]]></title>
            <link>https://medium.com/postnl-engineering/vpc-block-public-access-at-scale-f3d2fb6532c4?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/f3d2fb6532c4</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-security]]></category>
            <category><![CDATA[aws-vpc]]></category>
            <dc:creator><![CDATA[Gideon Vrijhoeven]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 07:01:50 GMT</pubDate>
            <atom:updated>2025-09-15T07:52:53.610Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ya8qxL4S2a2yFH1ATUFY9w.jpeg" /></figure><p>At PostNL we only allow ‘serverless applications’ on the landingzone we created for our self development teams. This is a statement we were able to uphold for almost 5 years since we started with the AWS CoE. But this year we were asked to look at some use case where EC2 support was needed, but those workloads did not need internet connectivity at all. So we started looking at a simple way to provide the teams with accounts that meets that requirement. So we started looking at VPC Block Public Access as a solution.</p><p>Today, the <a href="https://aws.amazon.com/architecture/well-architected/">AWS Well-Architected Framework</a> describes <a href="https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_securely_operate_multi_accounts.html">a single account</a> with <a href="https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_network_protection_create_layers.html">a single VPC</a> as an anti-pattern. With a growing number of accounts and network paths in the AWS Cloud, customers and partners told us they wanted simple tools that help them understand and secure their cloud environments at scale.</p><p>This is part of the introduction of the blog that announces VPC Block Public Access, the blog post can be found here:</p><p><a href="https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-block-public-access/">Enhancing VPC Security with Amazon VPC Block Public Access | Amazon Web Services</a></p><p>The solution is simple and offers 3 modes:</p><ul><li>Off ( default)</li><li>Bidirectional: All traffic to and from Internet Gateways and Egress-Only Internet Gateways in this Region (except for excluded VPCs and subnets) is blocked.</li><li>Ingress-only: All internet traffic to the VPCs in this Region (except for VPCs or subnets which are excluded) is blocked. Only traffic to and from NAT Gateways and Egress-Only Internet Gateways is allowed because these gateways only allow outbound connections to be established.</li></ul><p>This implements a mode on the Internet Gateway and the Egress Only Internet Gateway. This works really well, but in an enterprise it is often the case that e.g. egress traffic is centralized via the Transit Gateway, CloudWan or VPC towards a centralized egress solution.</p><p>Traffic related to private connectivity, such as traffic for the following services and resources, is not blocked or impacted by VPC BPA:</p><ul><li>AWS Client VPN</li><li>AWS CloudWAN</li><li>AWS Outposts Local Gateway</li><li>AWS Site-to-Site VPN</li><li>Transit Gateway</li><li>AWS Verified Access</li></ul><p>This is a link to the AWS <a href="https://docs.aws.amazon.com/vpc/latest/userguide/security-vpc-bpa-basics.html">documentation</a>.</p><p>At PostNL we use centralized egress via the TGW so in our case VPC BPA does not do cover all scenario’s. We had a use case to create accounts that have VPC and only needed to be accessed from the private network. Working at the platform team that creates all the accounts and VPCs and network connectivity we came up with this solution.</p><h3>Solution</h3><p>We added a parameter in the accounts we create called “block-public-access”. This parameter can have three values:</p><ul><li>off</li><li>ingress</li><li>bidirectional</li></ul><p>When the value is <strong>off </strong>we deliver the default VPC.</p><p>The VPC stack default creates these resources:</p><ul><li>The public subnet has a the default route towards the IGW and for IPv6 towards the EOIGW</li><li>The private subnet has a default route towards the TGW ( that has a route towards the egress VPC)</li><li>The isolated subnet has only local routes</li><li>The TGW subnet has the default route towards the TGW, but has no other resources attached</li><li>Internet Gateway</li><li>Egress Only Internet Gateway</li><li>S3 and DynamoDB endpoints</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*ZlBr8R6X6452BboTjoO1Uw.png" /></figure><p>When the value is <strong>ingress </strong>we create a VPC like this:</p><ul><li>The private subnet has a default route towards the TGW ( that has a route towards the egress VPC)</li><li>The isolated subnet has only local routes</li><li>The TGW subnet has the default route towards the TGW, but has no other resources attached</li><li>Egress Only Internet Gateway</li><li>S3 and DynamoDB endpoints</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUiTiDMy2qkYQtYSHTX-lA.png" /></figure><p>When the value is <strong>bidirectional </strong>we create a VPC like this:</p><ul><li>The private subnet has only <a href="https://datatracker.ietf.org/doc/html/rfc1918">RFC1918</a> CIDRs towards the TGW</li><li>The isolated subnet has only local routes</li><li>The TGW subnet has the default route towards the TGW, but has no other resources attached</li><li>S3 and DynamoDB endpoints</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FbBKivZjrlBoOVIxOqrDNw.png" /></figure><h3>Request flow</h3><p>At PostNL, teams can request accounts and at the account request they can specify the type of public access they need. The request is done in the webshop, and when creating the account a parameter is created. When the account creation is finished we trigger the bootstrap orchestrator. This triggers a recipe that based on the account type creates a VPC by deploying the VPC stack into the account. At deployment time the parameter is read and the correct VPC is deployed. This is a simplified overview of this process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UBPMCo8Pw_eLQKCpKlJnA.png" /></figure><h3>Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*iFdJ_i4i_yPCGzGHl6bWaw.png" /></figure><h3>Conclusion</h3><p>The introduction of VPC Block Public Access (VPC BPA) by AWS provides a simple and scalable solution for customers to enhance the security of their cloud environments. By offering three different modes (off, bidirectional, and ingress-only), VPC BPA allows organizations to easily control and restrict public access to their VPCs as needed. Since enterprise customers centralize egress traffic, the VPC BPA does not affect these flows.</p><p>The blog post highlights how PostNL has implemented a custom solution using a “block-public-access” parameter during account creation process. This allows us to automatically deploy the appropriate VPC configuration based on the specified mode, ensuring that public access is restricted while still maintaining necessary private connectivity through services like Transit Gateway. The implementation does not use the AWS VPC BPA solution but is inspired by it and has the same effect. It closes the gate towards/from the internet when needed by just altering one parameter and triggering a redeploy of the VPC stack.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3d2fb6532c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/vpc-block-public-access-at-scale-f3d2fb6532c4">VPC Block Public Access at scale</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Don’t Need a Construct for That: Best Practices for Serverless Infrastructure with AWS CDK…]]></title>
            <link>https://medium.com/postnl-engineering/you-dont-need-a-construct-for-that-best-practices-for-serverless-infrastructure-with-aws-cdk-10f09e61cf9b?source=rss----94bc0bf5cd50---4</link>
            <guid isPermaLink="false">https://medium.com/p/10f09e61cf9b</guid>
            <category><![CDATA[aws-cdk]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Matheus das Mercês]]></dc:creator>
            <pubDate>Thu, 17 Jul 2025 07:27:51 GMT</pubDate>
            <atom:updated>2025-07-17T07:27:51.731Z</atom:updated>
            <content:encoded><![CDATA[<h3>You Don’t Need a Construct for That: Best Practices for Serverless Infrastructure with AWS CDK Blueprints</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OWV40GQlY_epRH5hQefyHw.jpeg" /></figure><p>If you use AWS CDK (Cloud Development Kit) to create infrastructure as code, you’re probably familiar with <strong>Constructs</strong> and <strong>Aspects</strong>. Have you heard about CDK Blueprints? You can inject properties at L2 Constructs and apply best practices to all your resources at scale.</p><p>In this article, we are going to define best practices for a few resources by understanding the real meaning of CDK building blocks, see how easy it is to define <strong>Blueprints</strong> in CDK, explore the benefits of property injection, and understand why <strong>you don’t need a Construct</strong> for applying best practices.</p><h3>Introduction</h3><p>“Everything in CDK is a Construct”, but not everything needs to be a Construct.</p><p>No, this article is not <em>only</em> about CDK Constructs. What you’re going to read briefly in the next session is a rough summary of the Constructs documentation, <strong>the way AWS wanted us to understand them</strong>, and understanding the <strong>semantics</strong> of it is going to make a big difference in your <strong>IaC</strong> (Infrastructure as Code) project.</p><h4>For what would you need a Construct?</h4><p>According to the documentation from CDK, Constructs are the basic building blocks of a CDK app, and they are <strong>a collection of one or more resources</strong> to be deployed via CloudFormation.</p><p>Moreover, Constructs are also defined in 3 levels (<strong>L1</strong> for CloudFormation definition, <strong>L2</strong> for a small layer of abstraction, and <strong>L3</strong> for higher abstraction or often a collection of AWS resources). The documentation also explains that you can create your own Constructs for a <strong>specific pattern</strong> (for example, S3 Bucket to SNS topic).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*grSJ2i0p68N2lFFQ.png" /></figure><p>The purpose of them is to help you reach the desired state of your infrastructure as code by grouping resources and logic into ~<strong>basically</strong>~ classes.</p><h4>Things you can do with Construct, but you don’t need to (or should not?)</h4><p>What you just read is all that CDK Constructs mean, <strong>according to AWS</strong>. But it’s 2025, and we are creative people. Let’s have a look at this example:</p><pre>import * as cdk from &#39;aws-cdk-lib&#39;;<br>import { Construct } from &#39;constructs&#39;;<br>import * as logs from &#39;aws-cdk-lib/aws-logs&#39;;<br><br>export interface ValidatedLogGroupProps extends logs.LogGroupProps { }<br><br>export class ValidatedLogGroup extends Construct {<br>    public readonly logGroup: logs.LogGroup;<br>    constructor(scope: Construct, id: string, props: ValidatedLogGroupProps) {<br>        super(scope, id);<br>        if (props.retention !== logs.RetentionDays.TWO_WEEKS) {<br>            throw new Error(&#39;Retention must be set to TWO_WEEKS&#39;);<br>        }<br>        if (props.removalPolicy !== cdk.RemovalPolicy.DESTROY) {<br>            throw new Error(&#39;Removal policy must be set to DESTROY&#39;);<br>        }<br>        this.logGroup = new logs.LogGroup(this, &#39;LogGroup&#39;, props);<br>    }<br>}</pre><p>Since you write CDK with, in this example, TypeScript, you can validate the Construct props and throw exceptions if they are not compliant with standards defined by you. In practice, this means:</p><ul><li>You created a new layer of abstraction upon the L2 Construct of LogGroup, which contains default props from CDK Lib that might change between versions. This can lead to operational overhead when bumping versions.</li><li>You will not allow LogGroups to be created without a two-week retention (in this hypothetical situation, this configuration is mandatory).</li><li>You write it, you maintain it. The extra logic is going to be maintained by you, even though it is a small class.</li></ul><p>Although creating a custom Construct to achieve this is possible, you need to ask yourself the questions: “Is it worth it? Are there other options? Am I <strong>misusing</strong> CDK Constructs?”</p><h3>Gladly, we have alternatives</h3><p>The beauty of <strong>imperative</strong> IaC is that you can do pretty much everything you want, preferably following the <strong>semantics</strong> of the tool.</p><h3>Aspects</h3><p>Hold on, we are almost there. Let’s first see if Aspects help us validate props values.</p><p>The documentation starts explaining that <strong>Aspects</strong> are “a way to apply an <strong>operation</strong> to all constructs in a given scope, or it could verify something about the <strong>state</strong> of the constructs, such as making sure that all buckets are encrypted”. Ok, things are getting more interesting.</p><p>The purpose of an Aspect is to modify resources at the CloudFormation level based on what was defined by a <strong>higher-level construct (L2)</strong> or throw exceptions if they deviate from a standard, pretty much what we tried to achieve with Constructs above. For example:</p><pre>class BucketVersioningChecker implements IAspect {<br>  public visit(node: IConstruct): void {<br>    // See that we&#39;re dealing with a CfnBucket<br>    if (node instanceof s3.CfnBucket) {<br>    // Check for versioning property, exclude the case where the property<br>      // can be a token (IResolvable).<br>      if (!node.versioningConfiguration<br>        || (!Tokenization.isResolvable(node.versioningConfiguration)<br>            &amp;&amp; node.versioningConfiguration.status !== &#39;Enabled&#39;)) {<br>        Annotations.of(node).addError(&#39;Bucket versioning is not enabled&#39;);<br>      }<br>    }<br>  }<br>}<br>// Later, apply to the stack<br>Aspects.of(stack).add(new BucketVersioningChecker());</pre><p>Seems like we are getting there, but here are a few important points from the Aspects <strong>semantics</strong>:</p><ul><li>You want to modify L1 Constructs (CloudFormation level)</li><li>You want to inspect (not modify) L2 Constructs props</li><li>You have to navigate the whole tree of resources and check one by one to inspect</li></ul><p>Since we want to leverage the existing L2 LogGroup construct from CDK and modify the props to be compliant with your standards, we could use Aspects and modify the CloudFormation template after the fact — again, because it is possible does not mean it is the best way.</p><h3>Blueprints</h3><p>You know the drill: Let’s see what the documentation says about CDK Blueprints.</p><blockquote><em>“</em>Use AWS CDK Blueprints to standardize and distribute L2 construct configurations across your organization. With Blueprints, you can ensure that AWS resources are configured consistently according to your organizational standards and best practices. For example, you can automatically enable encryption for all Amazon S3 buckets, apply specific logging configurations to all AWS Lambda functions, or enforce standard security rules for all security groups.<em>”</em></blockquote><p>This is exactly what we are trying to achieve, and this is not me saying, but how AWS wants us to apply resource best practices.</p><p>Blueprints are powered by <em>property injection</em>, and each Blueprint is responsible for applying default properties to a specific L2 Construct when they are instantiated. We can think about <strong>security standards</strong>, <strong>cost optimization</strong>, <strong>compliance requirements</strong>, etc.</p><p>Let’s get down to business and create some examples of Blueprints for a few resources:</p><p><strong>CloudWatch Log Group</strong></p><pre>export class LogGroupPropsInjector implements IPropertyInjector {<br>    public readonly constructUniqueId: string;<br><br>    constructor() {<br>            this.constructUniqueId = LogGroup.PROPERTY_INJECTION_ID;<br>        }<br>        public inject(originalProps: LogGroupProps, _context: InjectionContext): LogGroupProps {<br>            return {<br>                ...originalProps,<br>                removalPolicy: RemovalPolicy.DESTROY,<br>                retention: RetentionDays.TWO_WEEKS<br>            };<br>        }<br>    }</pre><p><strong>Lambda Function</strong></p><pre>export class MyNodejsFunctionPropsInjector implements IPropertyInjector {<br>    public readonly constructUniqueId: string;<br>    <br>    constructor() {<br>        this.constructUniqueId = NodejsFunction.PROPERTY_INJECTION_ID;<br>    }<br>    public inject(originalProps: FunctionProps, _context: InjectionContext): FunctionProps {<br>        return {<br>            ...originalProps,<br>            runtime: Runtime.NODEJS_22_X,<br>            memorySize: 128,<br>            timeout: Duration.seconds(30),<br>        };<br>    }<br>}</pre><p><strong>SQS Queue</strong></p><pre>export class MyQueuePropsInjector implements IPropertyInjector {<br>    public readonly constructUniqueId: string;<br><br>    constructor() {<br>            this.constructUniqueId = Queue.PROPERTY_INJECTION_ID;<br>        }<br>        public inject(originalProps: QueueProps, _context: InjectionContext): QueueProps {<br>            return {<br>                ...originalProps,<br>                visibilityTimeout: Duration.seconds(45),<br>                retentionPeriod: Duration.days(4),<br>            };<br>        }<br>    }</pre><p>Some extra remarks about CDK Blueprints:</p><ul><li>Place default properties before …​originalProps to allow overrides.</li><li>Place forced properties after …​originalProps to prevent overrides.</li><li>Use CDK context to enable/disable injectors for testing.</li></ul><h3>The combination of the two</h3><p>Since we are doing things “<strong>by the book</strong>” here, we also need to consider this sentence from the Blueprints docs:</p><blockquote><em>“</em>Blueprints are not a compliance enforcement mechanism. Developers can still override the defaults if needed. For strict compliance enforcement, consider using AWS CloudFormation Guard, Service Control Policies, or CDK Aspects in addition to Blueprints<em>”.</em></blockquote><p>Even though it is possible to prevent overrides with Blueprints only (but remember the <strong>semantics</strong>!), you could combine it with Aspects in the following way:</p><ol><li>With your Blueprints in place, your L2 constructs are going to have some properties injected during initialization.</li><li>If, for some reason, your Blueprints are not applied to the scope of a specific resource, you can have an Aspect as an extra layer of validation (Aspects inspection happens after construct initialization).</li><li>Your Aspect will then check if a specific property was not injected for a specific resource. You can throw an exception or add a warning message saying you could use a pre-existing Blueprint from your project.</li></ol><h3>Conclusion</h3><p>In the realm of AWS CDK, while <strong>Constructs</strong> are fundamental building blocks for defining infrastructure as code, they are not always necessary for <strong>enforcing best practices.</strong> The introduction of CDK <strong>Blueprints</strong> offers a more streamlined and efficient approach to standardizing configurations across your organization. By leveraging property injection, Blueprints allow you to apply default properties to L2 Constructs, ensuring consistency in resource configurations without the need for additional abstraction layers.</p><p>Ultimately, the combination of <strong>Blueprints and Aspects</strong> allows for a balance between flexibility and compliance, enabling you to maintain best practices at scale while still allowing for necessary customizations. This approach aligns with AWS’s vision of how infrastructure as code should be managed, providing a powerful toolkit for modern cloud infrastructure management.</p><h3>References</h3><ul><li><a href="https://docs.aws.amazon.com/cdk/v2/guide/constructs.html">CDK Constructs documentation</a></li><li><a href="https://docs.aws.amazon.com/cdk/v2/guide/aspects.html">CDK Aspects documentation</a></li><li><a href="https://docs.aws.amazon.com/cdk/v2/guide/blueprints.html">CDK Blueprints documentation</a></li><li><a href="https://github.com/aws/aws-cdk-rfcs/blob/main/text/0693-property-injection.md">Property Injection Implementation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10f09e61cf9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/postnl-engineering/you-dont-need-a-construct-for-that-best-practices-for-serverless-infrastructure-with-aws-cdk-10f09e61cf9b">You Don’t Need a Construct for That: Best Practices for Serverless Infrastructure with AWS CDK…</a> was originally published in <a href="https://medium.com/postnl-engineering">PostNL Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>