<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Usama Shujaat on Medium]]></title>
        <description><![CDATA[Stories by Usama Shujaat on Medium]]></description>
        <link>https://medium.com/@shujaatsscripts?source=rss-19c684175441------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*F_et_8MtAglu9da3AoYuug.jpeg</url>
            <title>Stories by Usama Shujaat on Medium</title>
            <link>https://medium.com/@shujaatsscripts?source=rss-19c684175441------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:58:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shujaatsscripts/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Attach EBS volume to AWS ECS Fargate]]></title>
            <link>https://medium.com/@shujaatsscripts/attach-ebs-volume-to-aws-ecs-fargate-e23fea7bb1a7?source=rss-19c684175441------2</link>
            <guid isPermaLink="false">https://medium.com/p/e23fea7bb1a7</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ecs-fargate]]></category>
            <category><![CDATA[ebs-volume]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Usama Shujaat]]></dc:creator>
            <pubDate>Mon, 29 Jan 2024 08:25:44 GMT</pubDate>
            <atom:updated>2024-01-29T08:25:44.292Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*zLGWIfMHtUiRT2q0TDxE3A.png" /></figure><p>AWS recently announced a new feature for ECS at the start of this year, 2024, that ECS can now leverage EBS volumes as a storage option for tasks. Before this feature, we could see many questions on Stackoverflow and other platforms on how to do this, but this was not supported before, we had to use an ephemeral storage option for the ECS tasks for data-intensive workloads.</p><p>There are multiple types of storage options available and let’s discuss what can be the use case for each.</p><ul><li><strong>Ephemeral Storage: </strong>If high throughput and IOPS disk are needed to store data and process locally. 20 GB is the default ephemeral storage available for any task which can be updated up to 200 GB.</li><li><strong>Elastic File System (EFS): </strong>If more than one task needs to access one dataset then EFS can be used as the common storage. EFS can be attached to many tasks at once and thus can be used for distributed computing on a single dataset. This is a persistent storage option in which data will be remain in EFS and can be used for other purposes.</li><li><strong>Simple Storage Service (S3): </strong>This is technically not a storage option for an ECS task, but many developer/ML engineers just use S3 to copy a small amount of data to process and then write back to S3. The only reason to add this as this is the cheapest option and instead of using EFS. S3 storage can be used via AWS SDK.</li><li><strong>[!New] Elastic Block Storage: </strong>This is the newest type, where now EBS volumes can be configured with the ECS tasks. We will discuss in detail below why anyone would use EBS over any other storage option.</li></ul><h3>EBS volume as an ECS Task storage</h3><p>Listed below are a few added benefits that can be leveraged.</p><ul><li>More than 200GB of storage is required for local processing for ML/AI workload or Big Data ETL.</li><li>Configurable throughput and IOPS are required, or different type of volumes is required.</li><li>If any solution is using Cold HDD for data storage which is cheaper than S3. Cold HDD can be used as a volume for ECS task.</li><li>Persistent storage is required. EBS volume can be configured to remain available even after the task is terminated. (Only for Standalone tasks)</li><li>Data is present in a snapshot. Volume can be created from a snapshot to access the data.</li></ul><h3>Attach EBS volume to ECS task</h3><p>For the demo, this article is using the ECS Fargate task to use EBS volume as a storage option.</p><h4>Pre-Requisites</h4><ul><li>As a prerequisite, please create an ECS cluster in advance.<br>Also, create an ECS volume service role, with the following policies for EBS volume creation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*4fYuaXbFlvoidDZdLqG7yw.png" /></figure><h4>Steps</h4><ul><li>Create a task definition with the required value. In the Storage section, click on <strong>Add volume</strong> to add the EBS volume.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9IspnoCqdGkGPrthiKnuxg.png" /></figure><ul><li>Then for EBS volume, select <strong>Configure at deployment. </strong>Also, select the container mount point for the volume</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4mF_0PNt8Uaa_gdto6moEQ.png" /></figure><ul><li>To create Task definition in JSON for Terraform or CloudFormation, these blocks can be added to add EBS volume.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOGp_Bwzm6MbEbVLum4ENg.png" /></figure><ul><li>Now create the service, there will be a <strong>Volume </strong>in the Service creation console. Configure all the required details in the service, especially the task definition, then the volume can be configured. The options available for configuration are volume type, size, IOPS, Throughput, Snapshot ID, File system, Termination Policy, IAM role, and Encryption.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B8q3LCNTa-I4r3Dy5qntFA.png" /></figure><p>Options available for each are given below:</p><ul><li>Volume Type</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*GgzV9me-ZLUmotQtKhsX2Q.png" /></figure><ul><li>File System Type</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*mzVOD53oftkOykv6o9Hbcg.png" /></figure><ul><li>Termination Policy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*o2Yo2m-oAf2VFbSPrX16bg.png" /></figure><ul><li>Encryption</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDSuISRqQmxzylpsxCv9MA.png" /></figure><ul><li>Create the service and monitor the task.</li></ul><p>To create a Service using JSON file, please use the following block to create volume. Please refer to the document below.</p><p><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specify-ebs-config.html">Configuring Amazon EBS volumes at deployment — Amazon Elastic Container Service</a></p><h3><strong>Observations</strong></h3><p>EBS volume configured can be viewed at the ECS Service console under <strong>Configurations and Network</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CpaeJ503VQDh7_4Vb3xvkQ.png" /></figure><p>To get the EBS volume ID, it can be viewed in the Task console under the Volume tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOvffUqYpL-gtUzJLCDDig.png" /></figure><h3>Conclusion</h3><p>There is still a grey area on whether to use EBS volume or just use the ephemeral storage, but having an option and flexibility is always good. Hence, it can be used if certain specific requirements mentioned above it required.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e23fea7bb1a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cross-Account AWS resources access using AWS EKS Pod Identity]]></title>
            <link>https://medium.com/@shujaatsscripts/cross-account-aws-resources-access-using-aws-eks-pod-identity-5035256a586c?source=rss-19c684175441------2</link>
            <guid isPermaLink="false">https://medium.com/p/5035256a586c</guid>
            <category><![CDATA[iam-roles]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[pod-identity]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[cross-account]]></category>
            <dc:creator><![CDATA[Usama Shujaat]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 04:17:35 GMT</pubDate>
            <atom:updated>2024-01-23T11:32:34.978Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*3gYZJUs75G1l0JjlqGEJ0Q.png" /></figure><p>This article will elaborate how to access the AWS resources of different AWS accounts from the pods running on AWS EKS by leveraging EKS Pod Identity</p><h3>EKS Pod Identity</h3><p>EKS Pod identity is announced in the AWS re-invent 2023 which is basically a new way of accessing AWS resources from inside the pods without creating or providing new AWS credentials. Just like EC2 uses IAM roles to access AWS API, EC2 Pod Identity association map an IAM role to a service account in a namespace which can be used by pods to access AWS resources.</p><h3>IRSA vs EKS Pod Identity</h3><p><strong>IAM Role for Service Account</strong> is an old way to access AWS resources. This uses OIDC identity Provider to authenticate the AWS API call. Even though the approach is the same as using an IAM role with a service account, there are some added benefits of using EKS pod identity.</p><ul><li>Only one role is required for one type of access and multiple clusters can use the same role with EKS Pod Identity. Unlike IRSA, we need to create separate IAM roles for each cluster as each cluster has a different OIDC ID</li><li>EKS Pod Identity Agent runs on each node and provides credentials to SDKs running in the pods. So, instead of each SDK running on pods assuming the temporary credentials, it is only assumed by the EKS Auth service in the pod identity, hence reducing the load.</li></ul><p>For a deep understanding of how EKS pod identity works please go through the below link</p><p><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html">Understand how EKS Pod Identity works</a></p><h3>Setup EKS Pod Identity</h3><p>This document gives you a brief guide on how to use pod identity if you already have an EKS cluster up and runing. For the demo purpose, two accounts are required with one S3 bucket in each account and one demo lambda function in a different account as of EKS cluster account.</p><p>Here, EKS cluster account is referred as Source account and Lambda function account as target account.</p><h3><strong>1. Worker Node IAM Role</strong></h3><p>Node group role should have the permission to peform <em>eks-auth:AssumeRoleForPodIdentity </em>action<em> </em>in the IAM role. If you are using AWS Managed policy <a href="https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEKSWorkerNodePolicy.html#AmazonEKSWorkerNodePolicy-json">AmazonEKSWorkerNodePolicy — AWS Managed Policy</a> , then no need for this step. If you are using a custom IAM node role then add the following policy as the inline in the IAM Role of nodegroup:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;eks-auth:AssumeRoleForPodIdentity&quot;,<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><h3>2. Install Add-on</h3><p>Install the Add-on using EKS console. The Add-On name is <strong>EKS Pod Identity Agent.</strong></p><p>or use AWS CLI to create the add-on by</p><pre>aws eks create-addon --cluster-name my-cluster --addon-name eks-pod-identity-agent --addon-version v1.0.0-eksbuild.1</pre><h3>3. Create source account IAM role</h3><p>Create an IAM Role in the same account as EKS cluster with the <strong>Trust Policy</strong> as below.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;AllowEksAuthToAssumeRoleForPodIdentity&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;Service&quot;: &quot;pods.eks.amazonaws.com&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;sts:AssumeRole&quot;,<br>                &quot;sts:TagSession&quot;<br>            ]<br>        }<br>    ]<br>}</pre><p>Now create a policy to access S3 bucket and attach it to the source account IAM role. Replace the bucket name in <strong>&lt;&gt;</strong></p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::&lt;source-account-bucket&gt;&quot;,<br>                &quot;arn:aws:s3:::&lt;source-account-bucket&gt;/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><blockquote><em>Copy the IAM role ARN to use in the next steps.</em></blockquote><h3>4. Create Service Account</h3><p>Connect your EKS cluster using <strong>kubectl</strong> and create a service account in the namespace. This step is not needed if you already have a service account.<br>Replace details in &lt;&gt;</p><pre>kubectl create sa -n &lt;namespace&gt;</pre><h3>5. Create Pod Identity Association</h3><p>Create a pod identity association with AWS CLI. Please run the command below.</p><p>Replace details in &lt;&gt;</p><pre>aws eks create-pod-identity-association --cluster-name &lt;eks-cluster-name&gt; --role-arn &lt;source-account-role-arn&gt; --namespace &lt;namespace&gt; --service-account &lt;service-account-name&gt;</pre><p>Now this service account can be used in <strong>Deployments</strong> or <strong>Pods</strong> to access AWS resources.</p><p>To test it, create a pod with AWS CLI and try to list s3 bucket objects. Use the pod definition below.</p><p>Replace details in &lt;&gt;</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: aws-cli-pod<br>spec:<br>  serviceAccountName: &lt;service-account-name&gt;<br>  containers:<br>  - name: aws-cli<br>    image: amazon/aws-cli<br>    command: [&quot;sh&quot;, &quot;-c&quot;, &quot;sleep 3600&quot;]</pre><p>Save this in a file and deploy it on the EKS cluster. Replace details in &lt;&gt;</p><pre>kubectl apply -f &lt;pod-definition-file&gt;.yaml</pre><p>Once the pod is running, exec into it:</p><pre>kubectl exec -it aws-cli-pod -- /bin/sh</pre><p>Inside the pod, use the AWS CLI to list objects in S3 bucket. Replace details in &lt;&gt;</p><pre>aws s3 ls &lt;source-account-bucket&gt;</pre><h3>Setup EKS Pod Identity for Cross account resources</h3><p>After setting up the EKS Pod identity, the same role can be used to access AWS resources in different accounts by using delegate access. Follow the procedure below for cross-account access. This article is gonna show a demo for invoking Lambda function on a different account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*3gYZJUs75G1l0JjlqGEJ0Q.png" /></figure><h3>6. Create a Target account IAM Role</h3><p>Open the target account and create an IAM role with the <strong>trust policy</strong> given below. Replace the source account ID in the policy.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::&lt;source-account-id&gt;:root&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>        }<br>    ]<br>}</pre><p>Now create a policy with the required permissions and attach it to the target account IAM role. As an example, lambda involve permission are used in the policy.</p><p>Replace the details in &lt;&gt;</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;lambda:InvokeFunction&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:lambda:&lt;region&gt;:&lt;target-account-id&gt;:function:&lt;target-account-function-name&gt;&quot;<br>        }<br>    ]<br>}</pre><blockquote>Copy the arn of the target account IAM role as it will be used in the next steps.</blockquote><h3>7. Configure source account IAM Role</h3><p>To assume the role in the target account, the source account role should also have permission. Create another policy provided below and attach it to source account role.</p><p>Replace the details in &lt;&gt;</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:iam::&lt;target-account-id&gt;:role/&lt;target-account-role-name&gt;&quot;<br>        }<br>    ]<br>}</pre><p>To test it use the same pod we created above and exc into the pod</p><pre>kubectl exec -it aws-cli-pod -- /bin/sh</pre><p>Run the provided AWS CLI command to assume the target account role. Replace the details in &lt;&gt;. Session name can be anything</p><pre>aws sts assume-role --role-arn arn:aws:iam::&lt;target-account-id&gt;:role/&lt;target-account-role-name&gt; --role-session-name &lt;session-name&gt;</pre><p>Set Temporary credentials by running the command below.</p><p>Replace the details in &lt;&gt; from the output of the above command</p><pre>export AWS_ACCESS_KEY_ID=&quot;&lt;AccessKeyId from assume-role output&gt;&quot;<br>export AWS_SECRET_ACCESS_KEY=&quot;&lt;SecretAccessKey from assume-role output&gt;&quot;<br>export AWS_SESSION_TOKEN=&quot;&lt;SessionToken from assume-role output&gt;&quot;</pre><p>Invoke the lambda function. Replace the details in &lt;&gt;</p><pre>aws lambda invoke --function-name &lt;lambda-fucntion-name&gt; output.txt</pre><p>This way of assuming roles seems tedious but this is just for the test. In real world scenario, most of the pods are using AWS SDKs and there we can easily assume role. Please see the example below</p><p><strong>Node JS</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/818f53583938a88cf3b59b86c1f6a8d1/href">https://medium.com/media/818f53583938a88cf3b59b86c1f6a8d1/href</a></iframe><p><strong>Python</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1edae9ad7aca08c68dc5e78feb08dd1/href">https://medium.com/media/d1edae9ad7aca08c68dc5e78feb08dd1/href</a></iframe><h3><strong>Cross-account S3 Bucket Access</strong></h3><p>To access the S3 bucket from the Target account we do not need to assume the role. We just need to provide the permission on the source account IAM role to access the S3 bucket and change the bucket policy in the target account.</p><p>Add the permission in the source account IAM role.</p><p>Replace the details in &lt;&gt;</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::&lt;target-account-bucket&gt;&quot;,<br>                &quot;arn:aws:s3:::&lt;target-account-bucket&gt;/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><p>Update the target S3 bucket policy to allow access from this Source account IAM role.</p><p>Replace the details in &lt;&gt;</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::&lt;source-account-id&gt;:role/&lt;source-account-role-name&gt;&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::&lt;target-account-bucket&gt;&quot;,<br>                &quot;arn:aws:s3:::&lt;target-account-bucket&gt;/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><h3>Limitation</h3><ul><li>Currently this service is not available for few regions, AWS Outposts , AWS EKS Anywhere and Kubernetes clusters created on EC2.</li><li>Pods running on fargate are also not supported.</li></ul><p>For complete details limitations please refer to link below</p><p><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations">EKS Pod Identities</a></p><h3>Conclusion</h3><p>EKS pod identity will be the new norm in the coming days for accessing AWS resources from the pods. It is easier to set up than ISRA and less complex to manage the permissions. By using the simple assume role feature, any AWS resource can be accessed and controlled from a different account.</p><p>Reach out to me at if you need any help with applying this solution.<br><a href="https://www.linkedin.com/in/usama-shujaat/">LinkedIn</a><br><a href="https://topmate.io/usama_shujaat">Topmate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5035256a586c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using loops in AWS CloudFormation with Fn::ForEach Intrinsic function]]></title>
            <link>https://medium.com/@shujaatsscripts/using-loops-in-aws-cloudformation-with-fn-foreach-intrinsic-function-172373392a88?source=rss-19c684175441------2</link>
            <guid isPermaLink="false">https://medium.com/p/172373392a88</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Usama Shujaat]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 04:22:24 GMT</pubDate>
            <atom:updated>2024-01-15T08:28:54.762Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ESEJGogJ_5jV_KGv-1SFZw.jpeg" /></figure><h3>Introduction</h3><p>Every programming language has loops to iterate to make repeated tasks easier, whether to print the progress percentage of a status bar or to iterate through the whole file. It is one of the basic parts of any programming language. When it comes to Infrastructure as Code, the two main tools that come to mind are CloudFormation and Terraform. The former is written in JSON/YAML and later is written in HashiCorp Configuration Language (HCL). Both are descriptive languages, hence a bit different from the traditional programming language. This article will discuss one of the challenges of iteration to create resources dynamically on CloudFormation and its comparison with Terraform.</p><h3>Fn::ForEach</h3><p>The solution for dynamic Infrastructure deployment is provided by AWS in the form of an Intrinsic function <strong>Fn::ForEach.</strong> Spoiler Alert: <strong>Fn::ForEach </strong>is like not a traditional loop. It iterates a list called <strong>Collection</strong> and applies the <strong>Values</strong> of the list on the template below to create multiple CloudFormation components with a single block. This function helps prevent code re-write and allows the deployment of resources dynamically.</p><h3>Syntax</h3><p>There are six components in the ForEach function listed below.</p><p>1. <strong>UniqeLoopName:</strong> This is the name of the loop which needs to be unique in the CF template.</p><p>2. <strong>Value:</strong> Components of a list</p><p>3. <strong>Collection: </strong>List of comma-delimited values</p><p>4. <strong>OutputKey:</strong> This will turn to logical ID after deployment of CF Template. This should be unique and should not match all the other Logical IDs in the template.</p><p>5. <strong>OutputValue:</strong> This the body of the template this will get repeated after deployment. This contains the configuration of the resources.</p><p><em>6.</em> <strong>Identifier</strong>:<strong> </strong>This is like a variable which value will change according to the <strong>values</strong> of <strong>Collections</strong>. Identifiers will be used in the <strong>OutputKey </strong>and <strong>OutputValue. </strong>It can be used like <em>${Identifier} </em>and <em>&amp;{Identifier} </em>in the <strong>OutputKey </strong>and <strong>OutputValue.</strong><em> <br> </em>Important: If the value contains alphanumeric letters then <em>&amp;{Identifier} </em>should be used in the <strong>OutputKey.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*otBYW3V-TNIClmnvIXMBaA.png" /></figure><h3>Dynamic deployment of Resources</h3><p>This is where you will use Fn::ForEach for most of the time. This article will explain the concept with the help of an example. Let’s take the example, that there is Veeam software that needs to install agents on every EC2 instance, and we need to allow a few ports in the security groups. The infrastructure has multiple VPCs but only a few of those VPCs will require the security group. <br>We will create a CF template in which a user can just select VPC in the parameter from the list and a security group will be created on each of these VPC.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2802a86d0baa612313e2aff8e52d1fe/href">https://medium.com/media/a2802a86d0baa612313e2aff8e52d1fe/href</a></iframe><p>First, we put the list of VPCs in the Parameters block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xktqWATRNO-E-8aYkMWpqQ.png" /></figure><p>Second, we use Fn::ForEach to deploy security groups dynamically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*6J21sRG7PE5sylkl6-yTWQ.png" /></figure><p>In this example the components of the functions are</p><ul><li><strong>UniqeLoopName:</strong> VPCIds</li><li><strong>Identifier:</strong> VPCID. This is used in the <strong>Output Key</strong> and security group name.</li><li><strong>Collection: </strong>List of VPCs in referencing from the Parameters</li><li><strong>OutputKey:</strong> VeeamSecurityGroup&amp;{VPCID}</li><li><strong>OutputValue:</strong> Body of the loop.</li></ul><p>So basically Fn::ForEach takes the value from the <strong>Collection</strong> and puts it in the <strong>Identifier</strong> and then you can reference the <strong>Identifier</strong> in both the <strong>Output Key</strong> and the <strong>Output Value</strong>. One important thing to note that its necessary to use the identifier in the Output Key because the Output Key should be unique for each iteration of the loop.</p><blockquote><strong>Why ?</strong></blockquote><p>Well, the reason is that this loop is not like a traditional loop, This function will just dynamically create the traditional CF template while running, which means it just creates multiple resources of SGs in the template with the Output Key as Logical ID and Output Value as body, As CF does not allow same Logical IDs in the template, so it should be dynamic to avoid error.</p><p>Also, only Alphanumeric characters are not allowed in the Logical IDs and VPC Ids contain a hyphen ‘-’ like many other resource Ids. To cater to this problem, AWS provided a solution to reference the identifier as <strong><em>&amp;{}</em></strong> instead of <em>${}.</em> This <strong><em>&amp;{}</em></strong><em> </em>will remove all non-alphanumeric characters in the identifier’s value.</p><p>Please refer to the resultant template below and verify the above two observations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7f3bc1f7e111133084491ca9507bcf0/href">https://medium.com/media/d7f3bc1f7e111133084491ca9507bcf0/href</a></iframe><h3>Prevention of Code Re-Write</h3><p>There are multiple examples provided in the AWS documentation to prevent code re-write.</p><p>Please view the AWS documentation for these simple examples</p><p><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-foreach-example-resource.html#intrinsic-function-reference-foreach-example-replicate-multiple-resources">Examples of Fn::ForEach in the Resources section</a></p><h3>Difference from Terraform</h3><p>There is a lot of difference in functionality from the <strong>for_each</strong> function of Terraform. <strong>for_each</strong> in terraform is used for many other purposes than just dynamic deployment of resources like</p><ul><li>reading lines from a file</li><li>reading variable from a map (instead of list)</li><li>tagging a resource</li><li>dealing with variable data</li><li>scaling Infrastructure</li><li>multiple environment deployments using one module block</li></ul><p>and much more. Foreach in terraform is more closed to real loops in a programming language than CF. CF Fn:ForEach can only be used in the Conditions, Outputs, and Resources block of the template.<br> Please refer to the link below for detailed documentation for <strong>for_each</strong> in Terraform.</p><p><a href="https://developer.hashicorp.com/terraform/language/meta-arguments/for_each">The for_each Meta-Argument - Configuration Language | Terraform | HashiCorp Developer</a></p><h3>References</h3><ul><li><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-foreach.html">Fn::ForEach — AWS CloudFormation (amazon.com)</a></li><li><a href="https://developer.hashicorp.com/terraform/language/meta-arguments/for_each">The for_each Meta-Argument — Configuration Language | Terraform | HashiCorp Developer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=172373392a88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Containerize Java App using AWS App2Container and Deploy to ECS on EC2]]></title>
            <link>https://medium.com/@shujaatsscripts/containerize-java-app-using-aws-app2container-and-deploy-to-ecs-on-ec2-9159ed631765?source=rss-19c684175441------2</link>
            <guid isPermaLink="false">https://medium.com/p/9159ed631765</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ecs-fargate]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Usama Shujaat]]></dc:creator>
            <pubDate>Mon, 03 Apr 2023 20:00:38 GMT</pubDate>
            <atom:updated>2024-01-08T13:31:54.905Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*Tbm4ZT2I9N73G2Sp.png" /></figure><p>This article is about the workaround to containerize and deploy Java applications to ECS on EC2 using AWS App2container. This feature is <strong>not provided by AWS</strong> by default, hence there are some configurations required before deploying, to make it work.</p><h3>Introduction</h3><h4>App2Contianer</h4><p><strong>AWS App2container (A2C)</strong> is a CLI tool provided by AWS to modernize Java and .NET applications by converting them into containers. Containerized applications are easy to deploy, scale, and shift using container orchestration services like Kubernetes and AWS ECS. Containerizing an old application requires experienced DevOps resources which cost time and money. In contrast, A2C analyzes the application and generates the container images without access to the source code. For detailed benefits and features please visit</p><p><a href="https://aws.amazon.com/app2container/features/">AWS App2Container Features | Containerize Applications | Amazon Web Services</a><br> <br>A2C also allows you to deploy the application on AWS ECS by generating the AWS CloudFormation deployment files and automatically deploying it. Both Java and .NET applications can be deployed on AWS ECS with just one CLI command by AWS CLI. Please refer to the link below to deploy the containerized application using A2C on ECS. <br> <a href="https://docs.aws.amazon.com/app2container/latest/UserGuide/a2c-integrations-ecs.html">Deploy application containers to Amazon Elastic Container Service with AWS App2Container — AWS App2Container</a></p><h4>AWS Elastic Container Service</h4><p>AWS Elastic Container Service is a container orchestration service fully managed by AWS. In this way, customers do not have to manage the underlying infrastructure and can deploy, manage, and scale the containerized application. AWS ECS can run the containers on both AWS EC2 and AWS Fargate. AWS Fargate is a serverless service that can run containers without managing the servers which makes it easy to manage the application without worrying about the servers, OS patching, OS security, and additional benefits. Please visit the link below to learn more about AWS Fargate<br> <a href="https://aws.amazon.com/fargate/">Serverless Compute — AWS Fargate — AWS (amazon.com)</a></p><h3>Limitation</h3><p>AWS App2Container <strong>does not support </strong>deploying containerized Java applications on ECS on EC2, only Fargate is allowed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*bb0Ga0fYrWdYRy9Zv6hRZA.png" /></figure><h3>Purpose</h3><p>So, there is a catch, as AWS Fargate provides additional benefits, these benefits come with greater cost, which makes AWS EC2 a viable option to deploy to save cost. Although AWS recommends using fully managed serverless AWS Fargate, customers that need cost-effective solutions should deploy applications to AWS ECS on EC2. As this feature is not provided by AWS, this document will provide a simple workaround to achieve this.</p><p>AWS Fargate is charged per vCPU/hour and per GB memory per hour</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*3LWBT3fUEfWwD30G85Hu4Q.png" /></figure><p>AWS EC2 prices are charged based on the instance type, which give you flexibility to use burstable the cheapest option for Dev Environments and General purpose for production environments and other instance type specific to the application type. Few pricing examples are given below, for more detailed and latest pricing please visit.</p><p><a href="https://aws.amazon.com/ec2/pricing/on-demand/">EC2 On-Demand Instance Pricing — Amazon Web Services</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*UvSZDAv-ek47ZJDkGYuoqw.png" /></figure><p>By analyzing the cost of above we can clearly identify that ECS Fargate is 1–4 times more expensive that t-series instances and almost 20 percent more expensive than m-series. <br> <br>Using AWS ECS on EC2 also comes with other benefits than just cost. Following are few benefits that can convince customers to use AWS ECS on EC2</p><ul><li>Enhanced Networking using EC2 instances network capabilities.</li><li>GPU Support</li><li>High throughput storage options on EBS</li><li>Control over the Operating System</li><li>Control over Instance type based on workload</li></ul><h3>Solution</h3><p>Let’s start with the simple Java application to containerize and deploy to AWS ECS. This article uses an AWS-provided lab on Workshop Studio to demonstrate the process. Please follow the step-by-step process. These steps need to perform on the Java server</p><h3>Prerequisite:</h3><p>AWS CLI should be installed, and the default profile should be configured for the account in which the activity is performed.</p><p>Please visit the link below to install and configure AWS CLI</p><p><a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">Install or update the latest version of the AWS CLI — AWS Command Line Interface (amazon.com)</a></p><p><a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html#getting-started-quickstart-new">Set up the AWS CLI — AWS Command Line Interface (amazon.com)</a></p><h4>Step 1: Download Docker Engine</h4><p>Please refer to the link below to install Docker engine based on you operating system.</p><p><a href="https://docs.docker.com/engine/install/">Install Docker Engine | Docker Docs</a></p><h4>Step 2: Download App2Container CLI</h4><p>For Linux, please run the commands below</p><pre>curl -o AWSApp2Container-installer-linux.tar.gz https://app2container-release-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/AWSApp2Container-installer-linux.tar.gz<br>sudo tar xvf AWSApp2Container-installer-linux.tar.gz<br>sudo ./install.sh</pre><p>For detailed installation on Linux and Windows. Please visit the link below</p><p><a href="https://docs.aws.amazon.com/app2container/latest/UserGuide/start-intro.html#start-step1-install">https://docs.aws.amazon.com/app2container/latest/UserGuide/start-intro.html#start-step1-install</a></p><h4>Step 3: Create an S3 bucket</h4><ul><li>Go to AWS S3 Console</li><li>Create a bucket and save its name. Make sure to create the bucket in the same region as the server.</li><li>Keep every other as default.</li><li>Provide full access of this bucket to the IAM role of the EC2 instance of Java application.</li></ul><h4>Step 4: Initialize App2Contianer</h4><ul><li>Run the following command to initialize A2C</li></ul><pre>sudo app2container init</pre><ul><li>If Java server is on AWS EC2 instance and step 3 has been completed. then enter ‘y’ for <em>User AWS EC2 instance Profile</em>.</li><li>Provide the region of the Java EC2 Instance.</li><li>Provide the S3 bucket name created in step 3.</li><li>Keep every other configuration as default.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BviaL0u1CdryCzVznrCOsg.png" /></figure><h4><strong>Step 5: Analyze Java Application</strong></h4><ul><li>Run the following command to list the Java applications running on server.</li></ul><pre>sudo app2container inventory</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_8vaJFEGVJBNn8Y2BpF0Ow.png" /></figure><ul><li>Copy the java application id.</li><li>Run the following command to analyze the application.</li></ul><pre>sudo app2container analyze --application-id &lt;java-app-id&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6B_ljt3Xp75dETueSJ_2Mw.png" /></figure><h4>Step 6: Containerize Java Application</h4><ul><li>Run the following command to containerize the Java application. This will create docker image and Dockerfile for the application.</li></ul><pre>sudo app2container containerize --application-id &lt;java-app-id&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cH-TRLSKJ4WATMfwTSZCng.png" /></figure><ul><li>Check the image created by running the command.</li></ul><pre>docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*NcCKuVIQvtr-Zu97ztZ6-A.png" /></figure><p>After this step your application is containerized and can be deployed on any of the container platforms. The steps 1 – 5 are explained in detail by AWS, please visit the link below for more details.</p><p><a href="https://docs.aws.amazon.com/app2container/latest/UserGuide/start-intro.html">Getting started with AWS App2Container — AWS App2Container (amazon.com)</a></p><h4>Step 7: Create Deployment Artifacts</h4><ul><li>Navigate to the app2container default directory, open the folder with the name of java-app-id and open the file <strong><em>deployment.json.</em></strong></li><li><strong>(!Important)</strong> Enter the “<strong>vpcId</strong>” under “<strong>ecsParameters</strong>” if the application is required to deploy in a pre-existing VPC.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*eFnC_TvpVr4DG90dfLydUg.png" /></figure><ul><li><strong>(!Important)</strong> This file contains all the required configuration to deploy to ECS, try changing the “<strong>deployTarget</strong>” under “<strong>ecsParameters</strong>” from “<strong>FARGATE</strong>” to “<strong>EC2</strong>”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*gXXNQL7_aFuFGAH90MP6sQ.png" /></figure><ul><li>Then run the following command to generate the deployment artifacts.</li></ul><pre>app2container generate app-deployment --application-id &lt;java-app-id&gt;</pre><p>This config throws an error stating that “Invalid ECS Deploy target provided.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wBxqmxkr8nG2kAPCKOBObw.png" /></figure><ul><li>Change back the “deployTarget” to “FARGATE” in<em> deployment.json</em></li><li>Run the command again to generate the deployment artifacts.</li></ul><pre>app2container generate app-deployment --application-id &lt;java-app-id&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C4cEI3J4fe5mQJSnqPfe2Q.png" /></figure><ul><li>Open the artifact bucket on S3 console and review the artifacts created by A2C</li><li>Download <em>ecs-master.yml</em> and <em>ecs-cluster.yml </em>in the sub-template folder.</li><li>Open <em>ecs-cluster.yml </em>and change the userdata in the launch configuration to the following lines.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0INextCf8djGHQ6bOJN2Aw.png" /></figure><pre>UserData: !Base64<br>        Fn::Join:<br>          - &#39;&#39;<br>          - - &quot;#!/bin/bash\n&quot;<br>            - &quot;echo &#39;ECS_CLUSTER=&quot;<br>            - !Ref Cluster<br>            - &quot;&#39; &gt;&gt; /etc/ecs/ecs.config\n&quot;<br>            - &quot;echo &#39;ECS_ENABLE_TASK_IAM_ROLE=true&#39; &gt;&gt; /etc/ecs/ecs.config\n&quot;<br>            - &quot;echo &#39;ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true&#39; &gt;&gt; /etc/ecs/ecs.config\n&quot;<br>            - !Sub &quot;cfn-init --verbose --stack ${AWS::StackId} --resource ContainerInstances --region ${AWS::Region} --configsets default\n&quot;<br>            - &quot;cfn-signal --exit-code $? --stack &quot;<br>            - !Ref &quot;AWS::StackId&quot;<br>            - &quot; --resource ECSAutoScalingGroup --region &quot;<br>            - !Ref &quot;AWS::Region&quot;<br>            - &quot;\n&quot;<br>            - &quot;instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)\n&quot;<br>            - &quot;aws ec2 create-tags --resources \&quot;$instance_id\&quot; --tags Key=Name,Value=$(hostname) Key=ObjectName,Value=$(hostname)$\n&quot;</pre><ul><li>Save <em>ecs-cluster.yml</em> , delete<em> ecs-cluster.yml</em> on the S3 bucket, and upload to the updated one on the same location.</li><li>Change the LaunchType parameter to EC2 in ecs-master.yml and save it.</li><li>Delete the <em>ecs-master.yml</em> on the S3 bucket and upload the updated one on S3.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BqAWMilBRqPs0TwgGETT5w.png" /></figure><pre>LaunchType:<br>    Type: String<br>    Default: EC2<br>    AllowedValues: [ FARGATE, EC2 ]</pre><ul><li>Change the healthcheck parameters according to the following</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*_yhplqt4w2kWJm_5eB0qJQ.png" /></figure><pre>  # Health Check Parameters for Target Groups<br>  HealthCheckIntervalSeconds:<br>    Type: Number<br>    Default: 300<br>  HealthyThresholdCount:<br>    Type: Number<br>    Default: 2<br>  HealthCheckTimeoutSeconds:<br>    Type: Number<br>    Default: 10</pre><h4>Step 8: Deploy the CloudFormation template.</h4><ul><li>Open CloudFormation console on AWS</li><li>Select Create Stack then select Create stack with new resources.</li><li>Provide the <em>ecs-cluster.yml</em> file location on s3 bucket.</li><li>Click next till you reach the last tab.</li><li>Select the Information checkboxes and then deploy.</li></ul><p>It will take few minutes to deploy all the stacks and wait for all the stacks to be completed successfully.</p><p>For details on how to deploy using cloudformation, please visit.</p><p><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html">Creating a stack on the AWS CloudFormation console — AWS CloudFormation (amazon.com)</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Adtkp9e8z2fMrEDvmJjUkg.png" /></figure><h4>Step 9: Access the Java Application</h4><ul><li>In the Output tab of the main stack, java load balancer DNS is given.</li><li>Open it in the browser to check if the application is working.</li><li>ALB DNS can also be found in the Load balancer section on the left in the EC2 console, open it click the java load balancer, and review the details below.</li><li>Use this DNS name as Cname and update your Web application DNS to move production traffic to new infrastructure.</li></ul><h3>Conclusion</h3><p>The workaround provided above enables another feature of App2Container to deploy to ECS on EC2 for Java applications. By the date of this post, AWS has not provided this feature so it’s good to have the flexibility to deploy infrastructure according to your needs.</p><p>Reach out to me at if you need any help with applying this solution.<br><a href="https://www.linkedin.com/in/usama-shujaat/">LinkedIn</a><br><a href="https://topmate.io/usama_shujaat">Topmate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9159ed631765" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>