<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudip Sarkar on Medium]]></title>
        <description><![CDATA[Stories by Sudip Sarkar on Medium]]></description>
        <link>https://medium.com/@aemdevopssudip?source=rss-8ac30dae8788------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*H8TGuxK6bo61pJFw</url>
            <title>Stories by Sudip Sarkar on Medium</title>
            <link>https://medium.com/@aemdevopssudip?source=rss-8ac30dae8788------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:36:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aemdevopssudip/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automate sending docker application logs to AWS Cloudwatch]]></title>
            <link>https://medium.com/@aemdevopssudip/automate-sending-docker-application-logs-to-aws-cloudwatch-5fc7b87ffdcc?source=rss-8ac30dae8788------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fc7b87ffdcc</guid>
            <category><![CDATA[aws-cloudwatch]]></category>
            <category><![CDATA[cloudwatch-agent]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Sudip Sarkar]]></dc:creator>
            <pubDate>Mon, 21 Aug 2023 08:24:44 GMT</pubDate>
            <atom:updated>2023-08-21T08:26:39.973Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*thEDaCW2AF8MmsllKwSnjw.png" /></figure><p>Here I will demonstrate a scenario where my application will be running on docker and on daily basis at a particular time my application log will be sent to cloudwatch automatically. So lets jump into the demo.</p><h3>1.</h3><p>First on a EC2 instance I have installed docker and there I am running two applications — one is product app and another is addition app. Both are written in Go language. One is running on port 8085 and another is on port 8080.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DdIArgnEsfVyYfCHg2kSog.png" /></figure><h3>2.</h3><p>You can see that I am receiving the logs when applications are running. But I want to store these logs on a daily basis. So that I can get access of previous logs when needed. For this we can use cloudwatch. I can create log groups in cloudwatch for every application and inside that, log streams will be created daily, named with date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*LHlLfo1Hz0a3Lx1YSBfrHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcHNUwmf2Sf-2c_53V4Gpg.png" /></figure><h3>3.</h3><p>For that first need to give EC2 instance such permissions so that from EC2 I can get access of cloudwatch. Create a role with permissions — AmazonSSMFullAccess and CloudwatchFullAccess. When role is being created then attach it with your EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*uu_054cK4R9o4MUgCFfLyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xs-bi7bOzyv-arOHOcrfwA.png" /></figure><h3>4.</h3><p>Then create a directory in /home/ubuntu named ‘cloudwatch’ and inside that directory create another one named ‘logs’. Inside cloudwatch dir, I will create a file named deploy.bash file where bash script will be written to create text files inside logs dir which containing respective container’s log.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*chybWwF07ZbxIxBq5WTePg.png" /></figure><h3>5.</h3><p>This is how deploy.bash file looks like —</p><pre>#!/bin/bash<br><br>arr=($(docker ps -a --format &#39;{{.ID}} {{.Image}}&#39;))<br><br>for ((i = 0; i &lt; $((${#arr[@]} / 2)); i++)) <br>do<br>#docker logs ${arr[2*$i]} --since 60m &gt;&gt; /home/ubuntu/cloudwatch/logs/${arr[2*$i+1]}-$(date +%d-%m-%Y).txt<br>docker logs ${arr[2*$i]} --since 1m &gt; /home/ubuntu/cloudwatch/logs/${arr[2*$i+1]}.txt<br>done</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rxhvvKw9bCWriBM-JdIzbQ.png" /></figure><p>First I am getting all docker image names and container ids in an array. Then run a for loop to create text files inside logs directory in the name of docker image name. These text files will contain logs of the respective docker containers. Here I am taking only those logs which are 1 minute old and every time this bash file will run, it will create a new log file with updated logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wSQcZl2DJXp9eBbfb7iXrg.png" /></figure><h3>6.</h3><p>Now I have to run this bash file automatically in every 1 minute interval. So I have to create a cron job in the server. For that run #crontab -e</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*h9vy8qYkyZLW_0NkkIfZeA.png" /></figure><p>Hit just enter when ask to choose between 1 to 4. Then add this line in the end of the file — #*<em> * * * * sudo bash /home/ubuntu/cloudwatch/deploy.bash</em></p><p>To know more about cron expression click <a href="https://crontab.guru/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*NfbpoYEJXI6fyR-Rpj3wrw.png" /></figure><h3>7.</h3><p>Now if you are continuously hitting your application then you can get log after 1 min interval. Bash file is written such way that after 1 min new log will replace the previous log. If you want to append all logs then in deploy.bash file, line no.6 change ‘&gt;’ to ‘&gt;&gt;’ . But before that change the permission of the cloudwatch dir —<em> #sudo chmod 777 /home/ubuntu/cloudwatch</em></p><p>Then we have to send this log to cloudwatch. On instance I am running these commands to setup awslogs agent—</p><p><em>#curl </em><a href="https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py"><em>https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py</em></a><em> -O</em></p><p><em>#sudo apt install python2</em></p><p><em>#sudo python2 ./awslogs-agent-setup.py — region ap-northeast-1</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*xdOra-2xMwm_ty0PhN8ysQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*KhU46_R9kK_G42cka3laaw.png" /></figure><h3>8.</h3><p>I need to give input when running the last command. AWS secret key id and access key can be blank as I have created a role regarding cloudwatch access, region name is your default one, output format is json, destination log group name is ‘add-app-log-group’, for log stream name I have chosen {instance_id}, for timestamp format I have chosen the third one, initial position ‘start of file’ means to upload everything in the data file and ‘end_of_file’ means to upload only newly appended data. If previously in deploy.bash file you are appending data in log file then choose the second one and if replace the log in log file then choose first one. Run <em>#sudo cat /var/awslogs/etc/awslogs.conf </em>to check the awslogs.conf file which has been setup recently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*76reltW8m7aOYDiD4dodSw.png" /></figure><h3>9.</h3><p>Run the application and check the awslog for sending log to cloudwatch — <em>#cat /var/log/awslogs.log</em><br> Go to console and in cloudwatch see that log group and log stream has been created or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oLsoqhT29Fja1xeTY_ZYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HcQY1Cs4BPbNVb8CAMhGKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8kJW7IgssUfmfIhS1g0W4A.png" /></figure><p>Inside the log group see that a log stream has been created in the name of your instance id. Inside the log stream you will find the application logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LrwOBLZR2thNoGM62UxueQ.png" /></figure><h3>10.</h3><p>I do not want to store all logs in a single log stream of the cloudwatch log group. That’s why I want to create log stream on daily basis and log stream name is current date. For that we will change the awslogs.conf file automatically and restart the awslogs service.</p><p>So, create a another bash file in home/ubuntu/cloudwatch named as awslog.bash . There add this script -</p><pre>#!/bin/bash<br>gen=&quot;[general]\nstate_file = /var/awslogs/state/agent-state&quot;<br>arr=($(docker ps -a - format &#39;{{.Image}}&#39;))<br>echo -e &quot;$gen&quot; &gt; /var/awslogs/etc/awslogs.conf<br>for img in &quot;${arr[@]}&quot; <br>do<br>stream=&quot;\n[&quot;$img&quot;]\ndatetime_format = %b %d %H:%M:%S\nfile = /home/ubuntu/cloudwatch/logs/&quot;$img&quot;.txt\nbuffer_duration = 5000\nlog_stream_name = &quot;$(date +%d-%m-%Y-%H-%M)&quot;\ninitial_position = start_of_file\nlog_group_name = &quot;$img<br>echo -e &quot;$stream&quot; &gt;&gt; /var/awslogs/etc/awslogs.conf<br>done<br>sudo systemctl restart awslogs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*IQ0Fah3ETR27tIZh4bFI0g.png" /></figure><h3>11.</h3><p>First manually run this file — <em>#sudo bash awslog.bash</em> .If successful then in crontab add this line — #<em>*/2 * * * * sudo bash /home/ubuntu/cloudwatch/awslog.bash</em> .To test you can write the cron expression such way that with 2 minutes interval it will run the bash file.</p><p>To run this on daily basis at 12.00 am, add this to crontab —<em> #0 0 * * * sudo bash /home/ubuntu/cloudwatch/awslogs.bash</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*hlvY5TXaoUAv0PHRf0gJ-w.png" /></figure><h3>13.</h3><p>Now in cloudwatch log group you can see that when your application is running, everyday at 12.00 am a new log steam is being created with date mentioned in the log steam name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9zN-pGXJPOIhsqKzca3BWg.png" /></figure><h3>14.</h3><p>I want my 30 days old log streams to be deleted automatically. So, select your log group and from actions dropdown select ‘Edit retention setting’. Then from retention setting you can choose your respective timeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*66aG7x4FCZqMutrxV2Mwgg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*4X4np17_cE2Wrhe53OL_tw.png" /></figure><p><strong><em>Hope you find this blog helpful. Please share :)</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fc7b87ffdcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CICD pipeline using Github, Jenkins, ECR & EKS (Amazon Elastic Kubernetes Service)]]></title>
            <link>https://medium.com/@aemdevopssudip/cicd-pipeline-using-github-jenkins-ecr-eks-amazon-elastic-kubernetes-service-95c4b35f96ed?source=rss-8ac30dae8788------2</link>
            <guid isPermaLink="false">https://medium.com/p/95c4b35f96ed</guid>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[jenkins-pipeline]]></category>
            <dc:creator><![CDATA[Sudip Sarkar]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 02:55:12 GMT</pubDate>
            <atom:updated>2023-09-27T19:39:55.779Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*1Y8fspu9XI1zg0bLrORogA.png" /></figure><p>Here I will create a CICD pipeline where developer pushes the code to github, jenkins picks up the commit from github using webhook, docker image will be built in slave node and pushed to ECR, lastly image will be pulled from ECR and deployed to kubernetes cluster. Instead of ECR and EKS you can use dockerhub and minikube. I will use a simple java project here and create a dockerfile for it. You can use any project with dockerfile.</p><h3>1</h3><p>First I have setup my own VPC, subnets, routetable, internet gateway and security group. My EKS cluster will be in public subnets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*FUEauBixDYvAfkaJ4BoT9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*KKubhOFZglzuSIynzYPDSA.png" /></figure><h3>2</h3><p>Next move to IAM console and create some necessary IAM roles. First create a role for EKS cluster where use case is EKS cluster and policy is AmazonEKSClusterPolicy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*UQmLa6cgz7_s5YCKlaaj0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*5OAe2la5QTYq1XRx-Jt7ow.png" /></figure><p>Create a role for EKS worker nodes where use case is EC2 and policies are AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*2v8zOMsbwQd0eWoRPdp-6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*AZuZ6iuSS0QdnfDMKC5faw.png" /></figure><p>Create a role for jenkins slave node where use case is EC2 and policy is AmazonContainerRegistryFullAccess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Qm5CNxG51i8AdRIztDR3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EiC0DzQB7Nqkxz9qqyrmsQ.png" /></figure><h3>3</h3><p>Now I have to setup master and slave node for jenkins. I have chosen both master and worker node type as t3a.medium and on master I will install jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2iyb_sfYMdYIW9DucA_lkw.png" /></figure><p>Attach the IAM role with jenkins-slave node. Select the slave node, go to Actions &gt; Security &gt; Modify IAM role , then select the IAM role(test-ec2-role) and select Update IAM role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tg82NZx2H2doOXsvQ3Qb-w.png" /></figure><p>On both nodes install maven. This will also install java on them. For this run <em>#sudo apt install maven -y</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*K48h55yD8YmzZpp_vpZkrQ.png" /></figure><p>Now on master node install jenkins by following these commands —</p><p>$ <em>sudo apt update</em></p><p>$ <em>curl -fsSL </em><a href="https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key"><em>https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key</em></a><em> | sudo tee \<br> /usr/share/keyrings/jenkins-keyring.asc &gt; /dev/null</em><br> <br>$ <em>echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \<br> </em><a href="https://pkg.jenkins.io/debian-stable"><em>https://pkg.jenkins.io/debian-stable</em></a><em> binary/ | sudo tee \<br> /etc/apt/sources.list.d/jenkins.list &gt; /dev/null</em></p><p>$ <em>sudo apt update</em></p><p>$ <em>sudo apt install jenkins</em></p><p>Open jenkins on browser by using master node’s public ip with port 8080. Setup jenkins step by step and login as user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ndjbo3Rj1Ct6GDZ1f2UySw.png" /></figure><h3>4.</h3><p>On jenkins install necessary plugins to run docker and kubernetes commands. Go to jenkins dashboard &gt; manage jenkins &gt; plugins. There click on ‘Available plugins’ and then search &amp; add these — Docker, Docker pipeline, Kubernetes CLI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9uTQgK97D9PEDoEnmGZZbw.png" /></figure><p>Setup maven on jenkins. Go to jenkins dashboard &gt; Manage jenkins &gt; Tools and there find Maven. Click on Add Maven, give name as Maven, uncheck Install automatically, MAVEN_HOME as /usr/share/maven, then add maven.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsgEf1qww98EPLHE5lNRAQ.png" /></figure><h3>5.</h3><p>Now I have to connect slave node with jenkins so that project can be built on jenkins slave node only. Go to jenkins dashboard and select Manage jenkins &gt; Nodes, select New Node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wf3_uajSu5dlINBSI9gp1Q.png" /></figure><p>Give any node name, select type as permanent agent and select create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KGVa4-cNxcNXj1_8A_Kqg.png" /></figure><p>Fill up the fields like shown in the images. Remote root directory path is where my project will be built on slave node. I have taken it as /opt/build but you can take any path, just make sure that given exists on the slave node. Given label name will be used in jenkins pipeline to recognise the specific slave node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2pgg-t4txdHJ-U-M5syk1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4xd712gNyz_kfJPdNZ47rQ.png" /></figure><p>When you save it you see that still your node is not connected and it is showing a red cross mark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCvjRvkrFu6SpWaqExKIbQ.png" /></figure><p>Click on the node, you will see commands to be run on slave node. One command is for downloading agent.jar file and second command is for using that agent.jar file your node will be connected with jenkins as a slave.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mChHBM1nGj8XZFoYusePlw.png" /></figure><p>Go to slave node and run them one by one. Remember when run the second command put <em>sudo </em>before that and at the end add ‘&amp;’ so that it will be connected in background and you can run other commands there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*yGPMkxL2iu2SReVeVch9tQ.png" /></figure><p>After that refresh the jenkins console and see that node is connected properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*D-tyV_1ygUksUK5sANzZFw.png" /></figure><h3>6.</h3><p>In AWS create a private repository in ECR to store our docker images. You can use dockerhub instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*k7FeyJTUcOPvTQKPfjYwIg.png" /></figure><p>Come to slave node and install docker — <em>#sudo apt install docker.io -y</em> and also run this command so that from jenkins pipeline docker commands can be executed on this node — <em>#sudo chmod 777 /var/run/docker.sock</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bfAqAF-trHGxP4vPWnRU2A.png" /></figure><h3>7.</h3><p>Now create EKS cluster on aws console. I have used latest version(1.27) of kubernetes, cluster service role is my previously created role for cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*YN4UEyA0QMyqjC3L6h09-Q.png" /></figure><p>VPC is my own created ; not default one, corresponding subnets are being selected, security group is my EC2’s one, then create cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*8Vw3jlHrpipdxMK6TSqMBQ.png" /></figure><p>Wait for sometime till its showing active. When it is active you will see a security group for EKS cluster has been automatically created. Go to compute section of EKS cluster, then create nodegroup. In node IAM role select previously created IAM role for EKS nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTTya-iduJuK3bFFH3SsrQ.png" /></figure><p>AMI type is Amazon linux 2, capacity type is On-Demand, I have taken instance type as t3.small which is enough for this demo, disk size is 20 GB, desired, minimum and maximum size is 1 because I don’t want autoscaling now, then create nodegroup. Wait for sometime till EKS worker node is being created and nodegroup is showing active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*Cg0qnNcB3w97t2FLSV_mWg.png" /></figure><h3>8.</h3><p>Go to jenkins slave node and install awscli there by following these commands -</p><p><em># sudo apt update<br># curl “</em><a href="https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"><em>https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip</em></a><em>&quot; -o “awscliv2.zip”<br># sudo apt install unzip<br># unzip awscliv2.zip<br># sudo ./aws/install<br># ./aws/install -i /usr/local/aws-cli -b /usr/local/bin<br># ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update<br># aws --version</em></p><p>Then run <em>#aws configure</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*bEfUbaKTHM_mrGENLZ4aNg.png" /></figure><p>Now install kubectl and eksctl by following these-</p><p><em># curl -o kubectl </em><a href="https://s3.us-west-2.amazonaws.com/amazon-eks/1.23.7/2022-06-29/bin/linux/amd64/kubectl"><em>https://s3.us-west-2.amazonaws.com/amazon-eks/1.23.7/2022-06-29/bin/linux/amd64/kubectl</em></a><em><br> <br># curl -o kubectl.sha256 </em><a href="https://s3.us-west-2.amazonaws.com/amazon-eks/1.23.7/2022-06-29/bin/linux/amd64/kubectl.sha256"><em>https://s3.us-west-2.amazonaws.com/amazon-eks/1.23.7/2022-06-29/bin/linux/amd64/kubectl.sha256</em></a></p><p><em># openssl sha1 -sha256 kubectl</em></p><p><em># sudo chmod +x ./kubectl</em></p><p><em># mkdir -p $HOME/bin &amp;&amp; cp ./kubectl $HOME/bin/kubectl &amp;&amp; export PATH=$PATH:$HOME/bin</em><br> <br><em> #cd $HOME/bin<br></em> <br><em> # echo ‘export PATH=$PATH:$HOME/bin’ &gt;&gt; ~/.bashrc</em></p><p><em># kubectl version --short --client</em></p><p>Recognise the kubectl where is my master-node or EKS-cluster. Using these commands we can access our EKS cluster from our local machine or anywhere — <br> <br><em> # aws eks --region us-west-2 describe-cluster --name test-eks-cluster --query cluster.status</em></p><p><em># aws eks --region us-west-2 update-kubeconfig --name test-eks-cluster</em></p><p>Test kubectl command by running —</p><p><em># kubectl get nodes</em></p><p><em># kubectl get ns</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kDISz5sNV2TVpA388_j6rw.png" /></figure><p>(Optional)Install eksctl to create EKS cluster using AWS CLI.</p><p><em># curl --silent --location “</em><a href="https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname"><em>https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname</em></a><em> -s)_amd64.tar.gz” | tar xz -C /tmp</em></p><p><em># sudo mv /tmp/eksctl /usr/local/bin<br># eksctl version<br># eksctl get cluster<br># eksctl get nodegroup --cluster=test-eks-cluster</em></p><ul><li>* In case you want to forcefully delete EKS nodegroup and cluster if any error happens — <br> <br><em># eksctl delete nodegroup --cluster=test-eks-cluster --name=test-nodegroup<br> <br># eksctl delete cluster test-eks-cluster</em></li></ul><h3>9.</h3><p>In slave node’s home path run <em>#sudo cat .kube/config</em></p><p>This will show you the content of the config file inside .kube folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*6mdadzfUJCa_Cw2QP32IVw.png" /></figure><p>You have to copy this content and paste it in a &lt;anyname&gt;.txt file. Then go to jenkins console on browser, go Manage Jenkins &gt; Credentials &gt; System &gt; Global credentials, there add a new credential. Change the kind as Secret file so that I will use the created .txt file, give any ID(this will be later used in jenkins pipeline to run kubectl commands), give any description and select create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hm6Xgtad-yBzhuxw-g-KIA.png" /></figure><h3>10.</h3><p>Now go to your project and create a yaml file for kubernetes deployment and service. You can follow this script and do necessary changes —</p><p>eks-deploy-k8s.yaml —</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  namespace: default<br>  labels:<br>    app: springboot-app<br>  name: springboot-deployment<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: springboot-app<br>  template:<br>    metadata:<br>      labels:<br>        app: springboot-app<br>    spec:<br>      containers:<br>      - name: my-springboot-app<br>        image: &lt;your_aws_account_id&gt;.dkr.ecr.ap-northeast-1.amazonaws.com/test-repo:$BUILD_ID<br>        imagePullPolicy: Always<br>        ports:<br>        - containerPort: 8085<br>          name: http<br>          protocol: TCP<br># service type nodeport       <br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  namespace: default<br>  labels:<br>    app: springboot-app<br>    k8s-app: springboot-app<br>  name: springboot-service<br>spec:<br>  ports:<br>  - name: http<br>    port: 80<br>    protocol: TCP<br>    targetPort: 8085<br>    nodePort: 32055<br>  type: NodePort<br>  selector:<br>    app: springboot-app</pre><p>I have used service as nodeport. You can use loadbalancer or whatever service type you want. <em>$BUILD_ID</em> in the image tag is actually the build id of jenkins pipeline. I will use ‘envsubst’ in k8s deploy stage of the pipeline script. It will change the image tag according to the build id of jenkins pipeline.</p><h3>11.</h3><p>Then I have to create jenkins pipeline. From jenkins dashboard select New item, give item name, select Pipeline and then ok. There at the bottom go to pipeline section, definition is Pipeline script, in script section I will directly use the pipeline script to test first and then put that script in github project repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zrDICC82CF2CVn3ogATkSg.png" /></figure><p>Run this pipeline script —</p><pre>pipeline {<br>   tools {<br>        maven &#39;Maven&#39;<br>    }<br>    agent {label &#39;slave-node-label&#39;}<br>    environment {<br>        registry = &quot;&lt;account_id&gt;.dkr.ecr.us-west-2.amazonaws.com/test-repo&quot;<br>    }<br>    <br>    stages {<br>        stage(&#39;Cloning Git&#39;) {<br>            steps {<br>                checkout([$class: &#39;GitSCM&#39;, branches: [[name: &#39;*/main&#39;]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: &#39;&#39;, url: &#39;&lt;your_project_github_repo_url&gt;&#39;]]])     <br>            }<br>        }<br>        stage (&#39;Build&#39;) {<br>          steps {<br>            sh &#39;mvn clean install&#39;           <br>            }<br>        }<br>        <br>            // Building Docker images<br>        stage(&#39;Building image&#39;) {<br>          steps{<br>            script {<br>              sh &#39;docker build -t $JOB_NAME:v1.$BUILD_ID .&#39;<br>              sh &#39;docker tag $JOB_NAME:v1.$BUILD_ID ${registry}:v1.$BUILD_ID&#39;<br>             }<br>           }<br>         }<br>   <br>    // Uploading Docker images into AWS ECR<br>       stage(&#39;Pushing to ECR&#39;) {<br>         steps{  <br>           script {<br>             sh &#39;aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin &lt;account_id&gt;.dkr.ecr.us-west-2.amazonaws.com&#39; <br>             sh &#39;docker push ${registry}:v1.$BUILD_ID&#39;<br>             sh &#39;docker rmi $JOB_NAME:v1.$BUILD_ID ${registry}:v1.$BUILD_ID&#39; // Delete docker images from server <br>           }<br>          }<br>         }<br>       <br>       stage(&#39;K8S Deploy&#39;) {<br>       steps{   <br>         script {<br>            withKubeConfig([credentialsId: &#39;kubeconfig&#39;, serverUrl: &#39;&#39;]) {<br>              sh &#39;&lt;put the below line marked with **&gt;&#39;<br>              sh &#39;chmod u+x ./kubectl&#39;<br>              sh &#39;envsubst &lt; eks-deploy-k8s.yaml &gt; eks-deploy-k8s1.yaml&#39;<br>              sh &#39;./kubectl apply -f eks-deploy-k8s.yaml&#39;<br>             }<br>           }<br>        }<br>       }<br>    }<br>}<br>    </pre><p>** curl -LO “<a href="https://storage.googleapis.com/kubernetes-release/release/v1.20.5/bin/linux/amd64/kubectl">https://storage.googleapis.com/kubernetes-release/release/v1.20.5/bin/linux/amd64/kubectl</a>&quot; **</p><p>Then apply &amp; save it. Click on Build now option. You will see that every stages have been executed successfully but kubectl command gets failed due to authorization error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFDmCvJnc-uD_P02Lc0ewg.png" /></figure><p>For this go to slave node and run <em>#sudo aws configure</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*za5Io9fkCjFc55DjN6tQzQ.png" /></figure><p>Now again build the pipeline and you will see that pipeline is successfully built.</p><p>** If you are building your project on jenkins-master node, then to solve this error you have to give permission to jenkins user. Follow these commands —</p><p><em># sudo su — jenkins</em></p><p><em># aws configure</em></p><h3>12.</h3><p>In slave node check pods and service using</p><p><em>#kubectl get po</em></p><p><em>#kubectl get svc</em></p><p>In inbound rule of eks-cluster security group add port 32055(nodeport) for anywhere so that for eks worker node port 32055 is allowed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nbt4-Ws1CQtzrJvhnA4Rfg.png" /></figure><p>Now get the public ip of eks worker node and run on browser with port 32055</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*twFPeaHe1kQWqiRwovzSPg.png" /></figure><p>If service type will be LoadBalancer instead of NodePort then in aws a classic loadbalancer will be created. You can test it with its DNS name.</p><h3>13.</h3><p>My requirement is that every time I do any changes on github, my pipeline will be built automatically and application will be deployed on kubernetes. For that first go to jenkins dashboard and from top right corner under name’s dropdown select configure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*y8etWvUqIGCvrMlN3ZkRVg.png" /></figure><p>And there you will find API Token section. Select Add new Token option, copy that and save somewhere. Click on apply &amp; save below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*BVO5bdsAnA23clPVk9RMog.png" /></figure><p>Now go to repository of your github and there Settings &gt; Webhook.</p><p>Payload URL is <em>https://&lt;jenkins_master_node_publicip&gt;:8080/github-webhook/</em></p><p>Content type is application/json, in Secret paste the previously copied token generated on jenkins and then select Add webhook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cC-qCFcwu82VecX2Q8doRQ.png" /></figure><h3>14.</h3><p>Go to jenkins project and in left side select configure. In build trigger section check ‘GitHub hook trigger for GITScm polling’ . As this option is enabled, jenkins will look into your github repo and checks that any commit is there or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*r8tow2K8YiTZf81j_XpNyQ.png" /></figure><p>Then come to pipeline section. In definition dropdown select ‘Pipeline script from SCM’, in SCM select Git, in Repository URL enter your github repository URL and enter your branch name. My github repository is public so I don’t need any credential here. If you are using private repository then go to Manage Jenkins &gt; Credentials &gt; System &gt; Global credentials, create new credential with kind username with password, enter username of github, enter github token as password, give any ID and description, then create. That credential id will be required here to authenticate your private repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kg8zXUNN2ReJxGsWAhBbJQ.png" /></figure><p>And most important that create a file in your repository named ‘Jenkinsfile’ and there keep the pipeline script previously defined. You can provide any filename but then go to jenkins <a href="http://35.88.53.113:8080/">Dashboard</a> &gt; <a href="http://35.88.53.113:8080/job/demo-pipeline/">demo-pipeline</a> &gt; Configuration and change the name in pipeline &gt; Script Path.</p><p><strong><em>Hope you find this blog helpful :)</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95c4b35f96ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate creating and deleting RDS snapshots by using AWS Lambda and Cloudwatch]]></title>
            <link>https://medium.com/@aemdevopssudip/automate-creating-and-deleting-rds-snapshots-by-using-aws-lambda-and-cloudwatch-924cc07e6db?source=rss-8ac30dae8788------2</link>
            <guid isPermaLink="false">https://medium.com/p/924cc07e6db</guid>
            <category><![CDATA[automate]]></category>
            <category><![CDATA[aws-rds-snapshot]]></category>
            <category><![CDATA[using-python]]></category>
            <category><![CDATA[boto3-python]]></category>
            <dc:creator><![CDATA[Sudip Sarkar]]></dc:creator>
            <pubDate>Sun, 23 Apr 2023 07:04:22 GMT</pubDate>
            <atom:updated>2023-04-23T07:04:22.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*W1IHF7yi5uHo8mbALCMTwA.png" /></figure><p>Here I will demonstrate a scenario where I will take take four snapshots of my RDS instance on daily basis from Mon to Fri only. And on saturday at a specific time all snapshots which are older than four days, will be deleted. This whole process will be automated using lambda function and cloudwatch event rule. So lets jump into the demo.</p><h3>1.</h3><p>First create a RDS instance in a private subnet and that will be accessed only via jump host(EC2) which is in public subnet. I hope you have already gone through my previous blog where I have designed this via cloudformation. If not then <a href="https://medium.com/@aemdevopssudip/aws-cloudformation-using-a-jump-host-to-access-an-rds-database-in-a-private-subnet-26f9eedb7bd1">click here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDmGWXAS5K7Bu6r5AIQY-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*FMeaNFKSBYVBTwvwDhHTRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5K5f5E0r35fs0zrpzWUZXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*j38H5-cV8l5u8Ua82XllkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gj3yrBQfR-c9B_mDqsKiBQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FTTdUDE_aWUw2-pi6Dvu8g.png" /></figure><h3>2.</h3><p>Now go IAM tab and create a IAM role for lambda where I will allow full access of RDS to lambda. So add these three policies in the role — AmazonRDSFullAccess, AWSLambdaBasicExecutionRole, AWSLambdaVPCAccessExecutionRole</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mjwCi3OF9HZU-8ygdMt3wA.png" /></figure><h3>3.</h3><p>Then come lambda tab and create a lambda function for creating snapshots first. I have written the function in python 3.10. Choose the previously created IAM role as execution role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*RZkorktTrINnZc_WNSFNnQ.png" /></figure><p>I am providing the code for lambda function-</p><pre>import json<br>import boto3<br>import datetime<br>from datetime import datetime<br><br>def lambda_handler(event, context):<br>        client = boto3.client(&#39;rds&#39;)<br>        now = datetime.now()<br>        date = now.strftime(&quot;%d-%m-%Y-%H-%M-%S&quot;)<br>        tagname = now.strftime(&quot;%d-%m-%Y&quot;)<br>        #print(&quot;date and time =&quot;, date)<br>        response = client.create_db_snapshot(<br>        DBSnapshotIdentifier=&#39;sudip-test-db-identifier-{}&#39;.format(date),<br>        DBInstanceIdentifier=&#39;sudip-test-db-identifier&#39;,<br>        Tags=[<br>               {<br>                 &#39;Key&#39;: &#39;backupon&#39;,<br>                 &#39;Value&#39;: tagname<br>               },<br>             ]<br>        )<br>        #return (response);</pre><p>Please change the db identifier name according to yours.</p><h3>4.</h3><p>You can test the function now. So first deploy your code, create a test event and test the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*fleu-4bO_5yPkvQzwtJ3Qw.png" /></figure><p>Now if your code has been run successfully then you will see that a snapshot of your db instance has been created successfully. Go to RDS tab and from snapshots you can check it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*oKts8ZInCc91hSRobbtY_Q.png" /></figure><h3>5.</h3><p>You have created the db snapshot using lambda function but by running the function manually. You have to automate this using cloudwatch event rule. So lets go to cloudwatch tab and from select Events &gt; Rules. There you create first rule for creating snapshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*L5unNjvu4zv1limA-y9kHw.png" /></figure><p>Next you have to define the cron expression by which you schedule your rule to be triggered. You can see that according to my cron expression my rule will be triggered four times a day from monday to friday only. You can change this expression according to your need. For more information about cron expression go thorough this <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*5rT8LghGbaubO0fBd_KRaQ.png" /></figure><p>Next you need to select your target. Here our target is lambda function for creating snapshots. So choose your lambda function as target and finally create the rule. Now wait for the scheduled trigger time and see that RDS snapshot has been created automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*YIjLfmcjOMewXxZulW4j7Q.png" /></figure><h3>6.</h3><p>So creating snapshot part is done. Now periodicallly we have to delete them also. Otherwise cost for the RDS snapshot will be increased. So come to lambda tab again and create a lambda function for deleting snapshot like same as creating snapshot function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*V4zS6hEYR7soHtddh5bV1Q.png" /></figure><p>Here I want to delete snapshots which are more than 4 days old. So at any given time I can access atleast total 16 snapshots(4 days of backup and per day 4 snaps) which are up-to-date.</p><pre>import json<br>import boto3<br>from datetime import datetime<br>from datetime import timedelta<br><br>def lambda_handler(event, context):<br>        client = boto3.client(&#39;rds&#39;)<br>        now = datetime.now() - timedelta(days=4)   #change the retention days as per your requirement.<br>        date = now.strftime(&quot;%d-%m-%Y&quot;)<br>        snapshots = client.describe_db_snapshots(DBInstanceIdentifier= &#39;sudip-test-db-identifier&#39;, SnapshotType=&#39;manual&#39;)<br>        print(&quot;These are the snapshots {} here&quot;,snapshots[&#39;DBSnapshots&#39;])<br>        for i in snapshots[&#39;DBSnapshots&#39;]:<br>            <br>            ID = i[&#39;DBSnapshotIdentifier&#39;]<br>            S_type = i[&#39;SnapshotType&#39;]<br>            S_date= i[&#39;SnapshotCreateTime&#39;]<br>            S_date = S_date.strftime(&quot;%d-%m-%Y&quot;)<br>            print(&quot;These are the snapshots here&quot;)<br>            print(ID, S_date)<br>            if S_type == &quot;manual&quot;:<br>                if S_date == date:<br>                    client.delete_db_snapshot( DBSnapshotIdentifier= ID )<br>                    print(&quot;We have deleted the {} RDS Snapshot&quot;.format(ID))</pre><p>Test the function and see that snapshots are being deleted or not.</p><h3>7.</h3><p>Lastly create a cloudwatch event rule for deleting snapshot sameway like creating snapshot rule. Here I have scheduled the date and time in such way that the delete rule will be triggered on every saturday at 12pm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*NRwGp6l8BpIMKRERdcbi9A.png" /></figure><p>Target is your lambda function for deleting snapshots and create the rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*0ThCNGYuQz7iP1kvzAezSg.png" /></figure><p>So finally these two rule will be triggered at their scheduled date &amp; time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*k7HXA7AMA-RPgrFFLBm6AQ.png" /></figure><h3>8.</h3><p>You can check the cloudwatch log from your lambda function when your rule is being triggered for that function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZV4oFxgR4UN9GGDGy2rI0Q.png" /></figure><p>In Cloudwatch &gt; Log groups you can see that log stream is being generated according to your event time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EKTuZvp7khSyveuM7YcphQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=924cc07e6db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS CloudFormation : Using a Jump host to access an RDS database in a private subnet]]></title>
            <link>https://medium.com/@aemdevopssudip/aws-cloudformation-using-a-jump-host-to-access-an-rds-database-in-a-private-subnet-26f9eedb7bd1?source=rss-8ac30dae8788------2</link>
            <guid isPermaLink="false">https://medium.com/p/26f9eedb7bd1</guid>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[private-subnet]]></category>
            <category><![CDATA[rds-mysql]]></category>
            <dc:creator><![CDATA[Sudip Sarkar]]></dc:creator>
            <pubDate>Thu, 13 Apr 2023 13:22:01 GMT</pubDate>
            <atom:updated>2023-04-16T11:36:23.583Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS CloudFormation : Using a Jump host to access an RDS database in a private subnet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*wbd1QlYZOYANwjxezknFgQ.png" /></figure><p>Hello guys !! Today we are going to learn about aws cloudformation and using that we also setup the mentioned architecture ; means we will create a mysql db instance in a private subnet and that db instance will be only accessible via a EC2 instance which is in public subnet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*EjaE2Ey0cq2MhJd2MjIriA.png" /></figure><p>First let’s have some basic knowledge about IaC and cloudformation.</p><h3>What is CloudFormation ?</h3><p>CloudFormation is a powerful Infrastructure as Code tool that can help automate and manage your aws deployments. Now the question is what Infrastructure as Code or IaC tool is.</p><p>Infrastructure as Code (IaC) is the managing and provisioning of infrastructure through code instead of through manual processes. Automating infrastructure provisioning with IaC means that devops don’t need to manually provision and manage servers, operating systems, storage, and other infrastructure components each time they deploy an application. Some popular IaC tools are Terraform, Saltstack, AWS CloudFormation etc.</p><p>CloudFormation templates can be created with YAML in addition to JSON. Or you can use AWS CloudFormation Designer to visually create your templates and see the interdependencies of your resources. CloudFormation takes a declarative approach to configuration, meaning that you tell it what you want your environment to look like, and it finds its way there.</p><p>During this configuration process, CloudFormation automatically manages dependencies between your resources. Thus, you don’t have to specify the order in which resources are created, updated, or deleted. CloudFormation automatically determines the correct sequence of actions to create your environment, though you can use the DependsOn attribute, wait condition handlers, and nested stacks to specify the order of operations, if necessary.</p><p>Sometimes updating an infrastructure stack can cause anxiety because you’re not sure what changes might break the environment. Not to fear! CloudFormation Change Sets allow you to preview how your resources will be impacted before any changes are executed. Only after you execute your change set will your stack be edited. Even if you execute a change set that has errors in it, CloudFormation has Rollback Triggers that allow you to monitor your stack creation or update process and roll back your environment to a previous state.</p><p>There will be a large yaml file to implement the whole architecture. But I will break that into small parts so that you can understand step by step what is happening. Here one thing to remember that there is no additional charge for using AWS CloudFormation. You only pay for those resources which are being created by cloudformation.</p><h3>1.</h3><p>First create .yaml file and there write a script to create your custom vpc. In ‘Resources’ you will define the resources and its configuration you want to create. Inside that logical id/resource name is being declared and inside that ‘Type’ is key and value is aws resource type. In ‘Properties’, the resource’s properties are being declared.</p><pre>AWSTemplateFormatVersion: 2010-09-09<br>Description: Deploy a VPC with public/private subnets<br><br>Resources:<br>  VPC:<br>    Type: AWS::EC2::VPC<br>    Properties:<br>      CidrBlock: 12.12.0.0/24<br>      EnableDnsHostnames: true<br>      Tags:<br>      - Key: Name<br>        Value: sudip-test-vpc</pre><p>Now log into your aws account and go to cloudFormation service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_xDr1kPjMtbFSSiP7Uy-w.png" /></figure><p>Select ‘Create Stack’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jsAIOgwSsiqpkDuPAzlKhA.png" /></figure><p>Here select Template is ready as I will use my own template. Template source can be S3 URL or upload your template file. I will upload my template file, then next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHDB1tuInIu7ogqA4-DBJQ.png" /></figure><p>Give your stack name. As there is no parameters for now select next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V1gKoBM7JvOoHM0SiLZQHA.png" /></figure><p>There is nothing to do here. So select next. Also in the next page nothing to do and select submit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zrCLvfHro-OxsB_8RHbBUg.png" /></figure><p>After submitting you will see that your vpc has been started to create. In events you will see the in_progress, create and failed status. In resources you will see what resources has been created by your yaml file.</p><h3>2.</h3><p>Now you need to edit the script for creating subnets, route table and internet gateway.</p><pre>  PublicSubnet1:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.0/26<br>      AvailabilityZone: ap-northeast-1a<br>      MapPublicIpOnLaunch: true  <br>      Tags:<br>        - Key: Name<br>          Value: 1a-sudip-test-subnet<br><br>  PublicSubnet2:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.64/26<br>      AvailabilityZone: ap-northeast-1c<br>      MapPublicIpOnLaunch: true<br>      Tags:<br>        - Key: Name<br>          Value: 1c-sudip-test-subnet<br><br>  PrivateSubnet1:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.128/26<br>      AvailabilityZone: ap-northeast-1d<br>      MapPublicIpOnLaunch: false<br>      Tags:<br>        - Key: Name<br>          Value: 1d-sudip-test-private-subnet1<br>  <br>  PrivateSubnet2:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.192/26<br>      AvailabilityZone: ap-northeast-1c<br>      MapPublicIpOnLaunch: false <br>      Tags:<br>        - Key: Name<br>          Value: 1c-sudip-test-private-subnet2<br><br>  TestInternetGateway:<br>    Type: AWS::EC2::InternetGateway<br>    Properties:<br>      Tags:<br>      - Key: Name<br>        Value: sudip-test-igw<br><br>  AttachGateway:<br>    Type: AWS::EC2::VPCGatewayAttachment<br>    Properties:<br>      VpcId: !Ref VPC<br>      InternetGatewayId: !Ref TestInternetGateway<br><br>  PublicRouteTable:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-routetable<br><br>  PublicRoute:<br>    Type: AWS::EC2::Route<br>    Properties:<br>      RouteTableId: !Ref PublicRouteTable<br>      DestinationCidrBlock: 0.0.0.0/0<br>      GatewayId: !Ref InternetGateway<br><br>  PublicSubnetRouteTableAssociation1:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PublicSubnet1<br>      RouteTableId: !Ref PublicRouteTable<br><br>  PublicSubnetRouteTableAssociation2:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PublicSubnet2<br>      RouteTableId: !Ref PublicRouteTable   <br>  <br>  PrivateRouteTable1:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-private-routetable1<br>  <br>  PrivateSubnetRouteTableAssociation1:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PrivateSubnet1<br>      RouteTableId: !Ref PrivateRouteTable1<br>  <br>  PrivateRouteTable2:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-private-routetable2<br>  <br>  PrivateSubnetRouteTableAssociation2:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PrivateSubnet2<br>      RouteTableId: !Ref PrivateRouteTable2</pre><p>Here with this script four subnets will be created- two public and two private subnets. Resources names for subnets are ‘PublicSubnet1’, ‘PublicSubnet2’, ‘PrivateSubnet1’ and ‘PrivateSubnet2’.</p><p>Resource for internet gateway has been declared which is named as ‘TestInternetGateway’. Then it is being attached with the custom vpc. Resource for igw attachment with vpc is named as ‘AttachGateway’.</p><p>Three routetable has been created where one is public routetable and another two are private routetables. Their resources names are ‘PublicRouteTable’, ‘PrivateRouteTable1’ and ‘PrivateRouteTable2’.</p><p>Then in public routetable our internet gateway will be attached in route via resource ‘PublicRoute’. In public routetable two public subnets are associated via resources ‘PublicSubnetRouteTableAssociation1’ and ‘PublicSubnetRouteTableAssociation2’. Also in private routetables, private subnet1 and private subnet2 will be associated via recources ‘PrivateSubnetRouteTableAssociation1’ and ‘PrivateSubnetRouteTableAssociation2’.</p><h3>3.</h3><p>Now go to cloudformation tab and replace your existing yaml file. In your stack, from stack actions select the option ‘Create change set for current stack’. Then upload your updated yaml file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MV9abz2Hu2udYN5wqD76ZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9wpBV66cwZEQi3hi5N-R5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*UrQjpHkjU7J_UlLEiqe2BA.png" /></figure><p>Then after next and submit you will land into a page where you can see what changes will be deployed. When the status is showing CREATE_COMPLETE, then select Execute change set from top right corner. After that a confirmation page will be shown where you need to check on ‘Roll back all stack resources’. It means that if the deployment got failed then it will rolled back the stack to the last known stable state. Then select Execute change set.</p><h3>4.</h3><p>Change the yaml file for creating security groups.</p><pre>  InstanceSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupDescription: Allow http to client host<br>      VpcId: !Ref VPC<br>      GroupName: sudip-test-ec2-sec<br>      SecurityGroupIngress:<br>        <br>        - IpProtocol: tcp<br>          FromPort: 80<br>          ToPort: 80<br>          CidrIp: 0.0.0.0/0<br><br>        - IpProtocol: tcp<br>          FromPort: 22<br>          ToPort: 22<br>          CidrIp: x.x.x.x/32<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-ec2-sec<br>  <br>  PrivateSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupDescription: Allow http to client host<br>      VpcId: !Ref VPC<br>      GroupName: sudip-test-private-sg<br>      SecurityGroupIngress:<br>        <br>        - IpProtocol: tcp<br>          FromPort: 3306<br>          ToPort: 3306<br>          SourceSecurityGroupId: !Ref InstanceSecurityGroup<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-private-sg</pre><p>Here I have defined two resources. ‘InstanceSecurityGroup’ for creating public/instance security group where port 80 is allowed for all and port 22/ssh is allowed for only your system’s public ip. ‘PrivateSecurityGroup’ is for creating private security group which will be used for db instance. Most importantly in private security group port for mysql 3306 has been added only for public security group’s id.</p><p>After replacing the yaml file in cloudformation, execute the changes.</p><h3>5.</h3><p>Now you need to write script for dbsubnet and dbinstance.</p><pre> <br>Parameters:<br>  <br>  DBInstanceName:<br>    Description: Enter your DB instance name<br>    Type: String<br>    MinLength: 3<br>    MaxLength: 30<br>    Default: DBInstance<br>    <br>  DBInstanceClass: <br>    Description: choose instance for db<br>    Type: String<br>    AllowedValues:<br>      - db.t3.small<br>      - db.t2.small<br>      - db.t3.medium<br>    Default: db.t2.small<br><br>  DBStorage:<br>    Description: enter storage in gb for db<br>    Type: String<br>    <br>  DBMasterUserName:<br>    Description: enter your db username<br>    Type: String<br><br>  DBMasterPass:<br>    Description: Enter db password<br>    Type: String<br><br>  DBName:<br>    Description: Enter database name<br>    Type: String<br><br><br>Resources:<br><br>  DBSubnetGroup:<br>    Type: AWS::RDS::DBSubnetGroup <br>    Properties:<br>      DBSubnetGroupName: sudip-12apr-dbsubnet <br>      DBSubnetGroupDescription: description<br>      SubnetIds: <br>        - !Ref PrivateSubnet1 <br>        - !Ref PrivateSubnet2<br>      Tags: <br>        - Key: Name<br>          Value: sudip-test-dbsubnet<br><br>  DBInstance:<br>    Type: AWS::RDS::DBInstance<br>    DeletionPolicy: Delete <br>    Properties:  <br>      DBName: !Ref DBName<br>      DBInstanceIdentifier: !Ref DBInstanceName<br>      MasterUsername: !Ref DBMasterUserName<br>      MasterUserPassword: !Ref DBMasterPass<br>      Engine: MySQL<br>      EngineVersion: &#39;8.0.32&#39;<br>      DBInstanceClass: !Ref DBInstanceClass<br>      StorageType: gp2<br>      AllocatedStorage: !Ref DBStorage<br>      DBSubnetGroupName: !Ref DBSubnetGroup<br>      PubliclyAccessible: False<br>      MultiAZ: False<br>      VPCSecurityGroups: <br>        - !Ref PrivateSecurityGroup</pre><p>Here two resources has been added. ‘DBSubnetGroup’ for creating dbsubnet group and ‘DBInstance’ for creating db instance.</p><p>Besides ‘Resources’ I have used ‘Parameters’ here. If you you want user input instead of hard coded in script, then you can use parameters. Here you can see that there are so many field which I have added in parameters section like DBInstanceName, DBInstanceClass, DBMasterUserName, DBMasterPass etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AAC8XCPWAkiXfK-qv8Bskw.png" /></figure><p>While executing the changes you need to enter all the details related to db instance like instance name, instance class, database name, username, password, storage. So again replace your yaml file in cloudformation and execute the change set. It will take some time to create the db instance.</p><h3>6.</h3><p>Now add resources to create EC2 instance which is our jump host.</p><pre>Parameters:<br><br>  InstanceTypeParam: # give instance type from user input<br>        Description: Select instance type<br>        Type: String<br>        AllowedValues:<br>          - dev<br>          - prod<br>          - qa<br>        Default: prod<br><br>Mappings:<br>  EnvMap: #logical id/name of mapping<br>    dev:<br>      instanceType: &quot;t2.micro&quot;<br>      name: &quot;dev&quot;<br>    prod:<br>      instanceType: &quot;t2.large&quot;<br>      name: &quot;prod&quot;<br>    qa:<br>      instanceType: &quot;t2.medium&quot;<br>      name: &quot;qa&quot;<br><br>Resources:<br>  EC2Instance: <br>    Type: AWS::EC2::Instance<br>    Properties:<br>      InstanceType: !FindInMap [EnvMap, !Ref InstanceTypeParam, instanceType]<br>      SecurityGroupIds:<br>        - !Ref InstanceSecurityGroup<br>      SubnetId: !Ref PublicSubnet2<br><br>      Tags: <br>        - Key: &quot;Name&quot;<br>          Value: !FindInMap [EnvMap, !Ref InstanceTypeParam, name]<br>      UserData:<br>        Fn::Base64:<br>          !Sub  |<br>            #!/bin/bash<br>            apt-get update -y<br>            apt-get install mysql-server -y</pre><p>Here Mappings has been used. Mappings is a separate section of the template like Parameters, Conditions, and Resources. It lets you map one value to another value that will be used in a template.</p><p>For EC2 instance a bash script has been added by which mysql-server will be preinstalled in your instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GGHv-p0N-RCJsayf7s81WA.png" /></figure><p>While executing your changes you will find three options for instance type. ‘dev’ for t2.micro, ‘qa’ for t2.medium and ‘prod’ for t2.large . Choose any one and execute the changes. You will see a instance with your desired type has been launched.</p><h3>7.</h3><p>Now ssh into the instance and check mysql-server is installed or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*GfHzzG9VhisqPZIuBvf6Ow.png" /></figure><p>Then login into your mysql server using your database endpoint and check your database — # sudo mysql -h &lt;endpoint&gt; -P 3306 -u admin -p</p><p>You can check that only using this instance you can access your database.</p><p>So, the whole yaml file is given below :</p><pre>AWSTemplateFormatVersion: 2010-09-09<br>Description: Deploy a VPC with public/private subnets<br><br><br>Parameters:<br>  <br>  InstanceTypeParam:<br>        Description: Select instance type<br>        Type: String<br>        AllowedValues:<br>          - dev<br>          - prod<br>          - qa<br>        Default: prod<br><br>  DBInstanceName:<br>    Description: Enter your DB instance name<br>    Type: String<br>    MinLength: 3<br>    MaxLength: 30<br>    Default: DBInstance<br>    <br>  DBInstanceClass: <br>    Description: choose instance for db<br>    Type: String<br>    AllowedValues:<br>      - db.t3.small<br>      - db.t2.small<br>      - db.t3.medium<br>    Default: db.t2.small<br><br>  DBStorage:<br>    Description: enter storage in gb for db<br>    Type: String<br>    <br>  DBMasterUserName:<br>    Description: enter your db username<br>    Type: String<br><br>  DBMasterPass:<br>    Description: Enter db password<br>    Type: String<br><br>  DBName:<br>    Description: Enter database name<br>    Type: String<br><br><br>Mappings:<br>  EnvMap: #logical id/name of mapping<br>    dev:<br>      instanceType: &quot;t2.micro&quot;<br>      name: &quot;dev&quot;<br>    prod:<br>      instanceType: &quot;t2.large&quot;<br>      name: &quot;prod&quot;<br>    qa:<br>      instanceType: &quot;t2.medium&quot;<br>      name: &quot;qa&quot;<br><br>Resources:<br>  VPC:<br>    Type: AWS::EC2::VPC<br>    Properties:<br>      CidrBlock: 12.12.0.0/24<br>      EnableDnsHostnames: true<br>      Tags:<br>      - Key: Name<br>        Value: sudip-test-vpc<br><br>  <br>  PublicSubnet1:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.0/26<br>      AvailabilityZone: ap-northeast-1a<br>      MapPublicIpOnLaunch: true  <br>      Tags:<br>        - Key: Name<br>          Value: 1a-sudip-test-public-subnet1<br><br>  PublicSubnet2:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.64/26<br>      AvailabilityZone: ap-northeast-1c<br>      MapPublicIpOnLaunch: true<br>      Tags:<br>        - Key: Name<br>          Value: 1c-sudip-test-public-subnet2<br><br>  PrivateSubnet1:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.128/26<br>      AvailabilityZone: ap-northeast-1d<br>      MapPublicIpOnLaunch: false<br>      Tags:<br>        - Key: Name<br>          Value: 1d-sudip-test-private-subnet1<br>  <br>  PrivateSubnet2:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      CidrBlock: 12.12.0.192/26<br>      AvailabilityZone: ap-northeast-1c<br>      MapPublicIpOnLaunch: false <br>      Tags:<br>        - Key: Name<br>          Value: 1c-sudip-test-private-subnet2<br><br>  TestInternetGateway:<br>    Type: AWS::EC2::InternetGateway<br>    Properties:<br>      Tags:<br>      - Key: Name<br>        Value: sudip-test-igw<br><br>  AttachGateway:<br>    Type: AWS::EC2::VPCGatewayAttachment<br>    Properties:<br>      VpcId: !Ref VPC<br>      InternetGatewayId: !Ref TestInternetGateway<br><br>  PublicRouteTable:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-routetable<br><br>  PublicRoute:<br>    Type: AWS::EC2::Route<br>    Properties:<br>      RouteTableId: !Ref PublicRouteTable<br>      DestinationCidrBlock: 0.0.0.0/0<br>      GatewayId: !Ref TestInternetGateway<br><br>  PublicSubnetRouteTableAssociation1:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PublicSubnet1<br>      RouteTableId: !Ref PublicRouteTable<br><br>  PublicSubnetRouteTableAssociation2:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PublicSubnet2<br>      RouteTableId: !Ref PublicRouteTable   <br>  <br>  PrivateRouteTable1:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-private-routetable1<br>  <br>  PrivateSubnetRouteTableAssociation1:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PrivateSubnet1<br>      RouteTableId: !Ref PrivateRouteTable1<br>  <br>  PrivateRouteTable2:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-private-routetable2<br>  <br>  PrivateSubnetRouteTableAssociation2:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      SubnetId: !Ref PrivateSubnet2<br>      RouteTableId: !Ref PrivateRouteTable2      <br>  <br>  InstanceSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupDescription: Allow http to client host<br>      VpcId: !Ref VPC<br>      GroupName: sudip-test-ec2-sec<br>      SecurityGroupIngress:<br>        <br>        - IpProtocol: tcp<br>          FromPort: 80<br>          ToPort: 80<br>          CidrIp: 0.0.0.0/0<br><br>        - IpProtocol: tcp<br>          FromPort: 22<br>          ToPort: 22<br>          CidrIp: 65.2.88.212/32<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-ec2-sec<br>  <br>  PrivateSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupDescription: Allow http to client host<br>      VpcId: !Ref VPC<br>      GroupName: sudip-test-private-sg<br>      SecurityGroupIngress:<br>        <br>        - IpProtocol: tcp<br>          FromPort: 3306<br>          ToPort: 3306<br>          SourceSecurityGroupId: !Ref InstanceSecurityGroup<br>      Tags:<br>        - Key: Name<br>          Value: sudip-test-private-sg<br><br><br>  DBSubnetGroup:<br>    Type: AWS::RDS::DBSubnetGroup <br>    Properties:<br>      DBSubnetGroupName: sudip-12apr-dbsubnet <br>      DBSubnetGroupDescription: description<br>      SubnetIds: <br>        - !Ref PrivateSubnet1 <br>        - !Ref PrivateSubnet2<br>      Tags: <br>        - Key: Name<br>          Value: sudip-12apr-dbsubnet<br><br>  DBInstance:<br>    Type: AWS::RDS::DBInstance<br>    DeletionPolicy: Delete <br>    Properties:  <br>      DBName: !Ref DBName<br>      DBInstanceIdentifier: !Ref DBInstanceName<br>      MasterUsername: !Ref DBMasterUserName<br>      MasterUserPassword: !Ref DBMasterPass<br>      Engine: MySQL<br>      EngineVersion: &#39;8.0.32&#39;<br>      DBInstanceClass: !Ref DBInstanceClass<br>      StorageType: gp2<br>      AllocatedStorage: !Ref DBStorage<br>      DBSubnetGroupName: !Ref DBSubnetGroup<br>      PubliclyAccessible: False<br>      MultiAZ: False<br>      VPCSecurityGroups: <br>        - !Ref PrivateSecurityGroup        <br><br>  EC2Instance: <br>    Type: AWS::EC2::Instance<br>    Properties:<br>      InstanceType: !FindInMap [EnvMap, !Ref InstanceTypeParam, instanceType]<br>      ImageId: ami-0d979355d03fa2522<br>      KeyName: mykey<br>      SecurityGroupIds:<br>        - !Ref InstanceSecurityGroup<br>      SubnetId: !Ref PublicSubnet2<br><br>      Tags: <br>        - Key: &quot;Name&quot;<br>          Value: !FindInMap [EnvMap, !Ref InstanceTypeParam, name]<br>      UserData:<br>        Fn::Base64:<br>          !Sub  |<br>            #!/bin/bash<br>            apt-get update -y<br>            apt-get install mysql-server -y      <br><br><br>Outputs:<br>  VPC:<br>    Description: VPC<br>    Value: !Ref VPC<br>  AZ1:<br>    Description: Availability Zone 1<br>    Value: !GetAtt <br>      - PublicSubnet1<br>      - AvailabilityZone</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26f9eedb7bd1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[REST AWS API Gateway with Custom Domain Name and EC2 back-end]]></title>
            <link>https://medium.com/@aemdevopssudip/rest-aws-api-gateway-with-custom-domain-name-and-ec2-back-end-e416844ed61e?source=rss-8ac30dae8788------2</link>
            <guid isPermaLink="false">https://medium.com/p/e416844ed61e</guid>
            <category><![CDATA[custom-domain-name]]></category>
            <category><![CDATA[aws-api-gateway]]></category>
            <dc:creator><![CDATA[Sudip Sarkar]]></dc:creator>
            <pubDate>Sun, 09 Apr 2023 15:44:25 GMT</pubDate>
            <atom:updated>2023-04-12T04:30:14.134Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS REST API Gateway with Custom Domain Name and EC2 back-end</h3><p>Here I will tell you how to create REST API gateway which back-end is EC2 instance ; means code will be written on EC2 instance. You can also write your code in AWS lambda function. Then the created api gateway will be hosted with your own custom domain name which you have bought from godaddy, freenom or anywhere. You can search free domain name in <a href="http://www.freenom.com/">www.freenom.com</a> .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*rq7ajzh0TzUDBc2DbklBjQ.png" /></figure><p>I will complete the whole process by following few steps -</p><ol><li>Create/Register a certificate in AWS Certificate Manager.</li><li>Create a hosted zone in AWS route53 and add necessary records there.</li><li>Add authoritative nameservers in your domain name server.</li><li>Create a EC2 instance in your own VPC or default VPC. Install node and create a node js application there.</li><li>Create and setup REST apigateway.</li><li>Check api using postman</li><li>Configure apigateway for custom domain name</li><li>Configure hosted zone for apigateway</li><li>Check your domain name in postman</li></ol><p>Before starting today’s agenda lets have some brief about some of the important AWS services used here.</p><p><strong>ACM</strong> : AWS Certificate Manager (ACM) allows you to provision, manage, and deploy public and private SSL/TLS certificates to be used with other AWS services and internally connected resources. ACM allows you to secure multiple domain names and multiple names within a domain, create wildcard SSL certificates to protect an entire domain and its subdomains, and install public ACM certificates via services integrated with ACM.</p><p><strong>Route53 :</strong> AWS Route 53 is an AWS Managed Service that references traditional DNS ports and is highly available, scalable, and fully managed. The AWS Route 53 service mainly uses the concept of records, allowing you to route traffic for a particular domain according to the requirements. Each record contains details such as domain or subdomain, record type, Value of record type, routing policy, and <a href="https://ns1.com/resources/understanding-ttl-values-in-dns-records">TTL</a>. AWS Route 53 also allows you to check the health of your resources. Furthermore, AWS Route 53 is also a <em>domain registrar</em>, so anyone can purchase the domain they wish to if it is available.</p><p><strong>API Gateway </strong>: As we all know Amazon Web Services is the world’s leading cloud provider and Amazon also provides its own variant of an API Gateway. AWS API Gateway is an AWS service for creating, publishing, maintaining, monitoring REST APIs, HTTP, and WebSocket APIs. API developers can create APIs that access <a href="https://intellipaat.com/blog/what-is-amazon-web-services-aws/">AWS</a> or other web services, as well as data stored in the AWS Cloud. As an API Gateway API developer, you can create APIs for use in your own client applications.</p><ol><li>First request for a certificate in ACM. Add your domain name there.It will be ‘Pending validation’ state. It will take time to issue. After some time certificate status will be shown as ‘Issued’.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RBX83FLZky1QB84c31cnzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LgPjHqfQOYZur4kxwUwc2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THyfFqBvBv_5hp2tnVLZdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CDnnf9TJ1dk3Ypz4Wi2CA.png" /></figure><p>2. Now create a hosted zone in aws Route53. Give your domain name and it will a public hosted zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*r9fjrI3K5Q-jHRsQekwVIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*myy9wA0obRLRJt7zdPSSjg.png" /></figure><p>3. After hosted is successfully created you will see that record type NS/Name servers has been created. You need to go to your domain server and replace the nameservers with the NS which are being created by your hosted zone. Though you will see that nameservers have been showing immediately but this will take some time to be replaced. To confirm this open your terminal and run —<em> # sudo nslookup -type=ns &lt;your domain name&gt; 1.1.1.1 . </em>If those are replaced successfully then in output those will be shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*L1_nIjVuC1l36rVN43Clpw.png" /></figure><p>Also you need to go to ACM and then your created certificate. There you will see that in domain section there are two CNAME records which are needed to add in your hosted zone of route53. You just have to click on the ‘Create records in Route53’ and then in the next page select ‘Create records’ . You will see that two CNAME records has been added in your hosted zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mvq4dT767fBytnOBWMY64A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oOH1uKeMxjbEiUugURFbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGgl-jtJeKrn0_n8QKFA4w.png" /></figure><p>4. Go to EC2 tab and create a t2.micro ubuntu 22.04 instance. In security group port 3000, 80(http) and 443(https) will be allowed for anywhere. There install node, npm and pm2 to run a helloworld node js project. To install those follow the below commands —</p><p># <em>sudo apt update</em><br> # <em>curl -fsSL </em><a href="https://deb.nodesource.com/setup_18.x"><em>https://deb.nodesource.com/setup_18.x</em></a><em> | sudo -E bash -</em><br> # <em>sudo apt-get install nodejs -y</em><br> # <em>sudo apt-get install gcc g++ make</em><br> # <em>sudo npm install -g npm@latest</em><br> # <em>sudo npm install pm2@latest -g</em><br> # <em>npm — versi</em>on<br> #<em> node -v</em><br> # <em>pm2 -v</em></p><p>Then create a file named hello.js . Inside that file add the following code block —</p><pre>//Load the required HTTP module<br>const http = require(&quot;http&quot;);<br>//Create an HTTP server that listens on port 3000<br>const server = http.createServer((req, res) =&gt; {<br><br>//Set the response HTTP header with HTTP status and Content type<br>res.statusCode = 200;<br>res.setHeader(&#39;Content-Type&#39;, &#39;text/plain&#39;);<br>res.end(&#39;Hello Sudip Sarkar\n&#39;);<br>});<br><br>//listen for requests on port 3000, and as a callback function have the port listened on logged. It can be any port like 9000<br>server.listen(3000, () =&gt; {<br>console.log(&#39;Server running at http://&lt;your EC2&#39;s public ip&gt;:3000/&#39;);<br>});</pre><p>Now run the command to start the application — #<em>sudo pm2 start hello.js</em></p><p>Go to browser and check by entering <em>http://public ip:3000 . </em>See that ‘Hello Sudip Sarkar’ is showing or not.</p><p>5. Now go to apigateway and create your REST apigateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WfdJMXHdzEwwA6m2w0b73Q.png" /><figcaption>Select ‘Build’ from REST API section</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZLNaMLv2s4htU6p-UOnpw.png" /><figcaption>Api name can be anything and Endpoint will be Regional</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XImxMfTxjXT_QBC6RYvEWA.png" /><figcaption>From top select Actions and then select Create Resource</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-HbhUV1yeiU90lWilaf61w.png" /><figcaption>Give Resource name and select Create Resource</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiDiMyRArEJD-oa-ceKa9Q.png" /><figcaption>Within in ‘node’ resource create another resource which is proxy</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKyltpSTedhtc1CwCVKpew.png" /><figcaption>Within proxy resource, method is ANY and as we are using EC2 as backend so select HTTP Proxy and enter your public ip with port 3000 and path proxy like shown in the image. Select save</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IqDTX5m_KjX8qGfEa-V8nQ.png" /><figcaption>Deploy your api by selecting Deploy API from Actions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*uGQyP-Mx1D6gjyAUfeavTg.png" /><figcaption>If no stage has been created then create a new stage and select Deploy</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Mb2NRo9YNsVRDq3wY4uvg.png" /><figcaption>After deployment you will find the invoke URL</figcaption></figure><p>6. I am using postman to check my api. Open postman and enter the apigateway invoke url with ‘/&lt;resource_name&gt;’ at the end. Here my resource name is node. As I am using proxy here it means that you don’t need to create separate resource for separate api. Suppose if you want to create api for upload image and user details then you don’t need to create two separate resources. By using proxy you just have to call https://&lt;invoke url&gt;/dev/&lt;resource_name&gt;/&lt;your_desired_path_name&gt; ; means for every api upto /resource_name the path is same and change is only after that. This is the benefit of using proxy in apigateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*LIXBGY3SJlKAloPacXtixA.png" /><figcaption>Check API on postman</figcaption></figure><p>7. Then in apigateway create custom domain names. There give your domain name, endpoint is regional, select your ACM certificate and create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ot5jmRpbrRPyCzGSp_0DLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugLDHjPu9mwNCTLzirPu_A.png" /></figure><p>In the created custom domain name select ‘API mapping’ and inside that select ‘Configure API mappings’. There select your apigateway ,choose stage and save it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gacOswcWuwWy7MxAybQFpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Z65gTvL6bcvIj-EQdt0_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qOlqVItvlmvpbkpj3xZkCA.png" /></figure><p>8. Later go to your hosted zone and create a record there. Record name is blank ; means record name is considered hellodevops.xyz , record type is A, swtich on the Alias option. Then for routing traffic select ‘Alias to API Gateway API’, select your region, select your created custom domain name. Then create record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BZ5SzjSqE0bj6yMEECchNQ.png" /></figure><p>9. Now go to postman again enter your api url like https://&lt;your_domain_name&gt;/&lt;resource_name&gt;/&lt;your_desired_path_name&gt;. Then send it and see the response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*pZpKImt_LsRhNnUIP-jR3Q.png" /></figure><p>Now for every time you change your code in EC2 and restart the pm2 server by running — <em># sudo pm2 restart hello.js</em> , you will see the change in postman response for that api.</p><h3>Hope you find it helpful. Thank you !!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e416844ed61e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>