<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ghazanfar Ali on Medium]]></title>
        <description><![CDATA[Stories by Ghazanfar Ali on Medium]]></description>
        <link>https://medium.com/@ghazanfaralidevops?source=rss-39137f9e988------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*q0doUE2sc2l-a3xxJHb7VA.jpeg</url>
            <title>Stories by Ghazanfar Ali on Medium</title>
            <link>https://medium.com/@ghazanfaralidevops?source=rss-39137f9e988------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:12:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ghazanfaralidevops/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Secure VPN Tunnelling Between AWS & GCP]]></title>
            <link>https://ghazanfaralidevops.medium.com/secure-vpn-tunnelling-between-aws-gcp-f9a4f3f533cc?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9a4f3f533cc</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 13:27:11 GMT</pubDate>
            <atom:updated>2024-09-29T13:27:11.221Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Scenario: <br></strong>Imagine your company has applications running in both AWS and GCP. These applications need to communicate securely and efficiently with each other, sharing sensitive data and resources across the different cloud environments.</p><h3>Why VPN Tunneling is Needed:</h3><ol><li>Secure Data Transmission:</li></ol><ul><li>Encryption: VPN tunnels encrypt the data traveling between different cloud environments, ensuring that sensitive information remains secure from eavesdropping or interception during transit.</li><li>Confidentiality: Ensures that only authorized users and systems can access the data, maintaining the confidentiality and integrity of your communications.</li></ul><ol><li>Consistent Network Policies:</li></ol><ul><li>Unified Security Policies: Implementing VPN tunnels allows you to enforce consistent security policies across different cloud environments, providing a standardized security posture.</li><li>Compliance: Helps in meeting regulatory and compliance requirements by ensuring secure data transfer and communication between different clouds.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vFPOGbuJOUZhheX0" /></figure><p><strong>AWS:<br></strong>After VPC and ec2 setup we will create the <br>1- VPN gateway and attach to vpc, in VPN gateway we will define the BGP ASN, <br>An <strong>AWS VPN Gateway</strong> (Virtual Private Gateway) allows you to securely connect your on-premises network or another cloud environment to your Amazon VPC (Virtual Private Cloud) via a <strong>VPN connection</strong>. It provides an encrypted tunnel over the public internet for secure communication between the two networks.</p><p>2- After this we will create customer gateway which will have the gcp vpn peering information means public ip from GCP + BGP ASN of AWS. <br>A <strong>Customer Gateway (CGW)</strong> is the on-premises (or external) VPN device or software that connects to an AWS <strong>Virtual Private Gateway (VGW)</strong> or <strong>Transit Gateway</strong> to establish a secure <strong>Site-to-Site VPN connection</strong> between your on-premises network and your AWS VPC.</p><p>While the <strong>Virtual Private Gateway (VGW)</strong> exists on the AWS side of the VPN connection, the <strong>Customer Gateway</strong> is the entity that exists on your side (your network or data center).</p><p>3- VPN site to site (means put the vpn gateway and customer gateway inside a box).<br><strong>GCP:<br></strong>1- First we will create cloud router in gcp which use to connect your vpc to outside network.</p><p>2- Then we will create vpn gateway as we did on aws.</p><p>3- Then we will create peer vpn which will has information of aws vpn site.<br>4- At last we will create vpn tunnel which will has configuration of BGP session.<br>Our target is to establish the ping connection between aws ec2 instance and GCP VM.<br>Lets start the hands on!</p><p>We will start the hand on with AWS first, we already have vpc and subnets in the AWS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/0*6JoxJPL8A8-GbjTq" /></figure><p>Then creates subnets, 2 private and 2 public with diff AZs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JpFR3WTQgIvlYI7_" /></figure><p>Now we will create one NAT gateway for internet access to public subnets, NAT gateway will be created in public subnet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p6O9ebY1ixHPe6Uf" /></figure><p>Now create the internet gateway so that public subnets and nat gateway will be able to access the internet, Also attach the IG with vpc:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h9ZR42qsp8xR2X52" /></figure><p>Now create the two route tables and attach the private subnets with private RT and public subnets with public RT, add one default route 0.0.0.0/0 of private RT toward NAT gateway , and one default route in public RT toward IG.<br><strong>Note: </strong>Enable the route propagation of each RT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y4lKmIy-3acRU4X_" /></figure><p>Below is the example of public RT default route:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sB36pfZ6WAPJcxnL" /></figure><p>Our AWS VPC part is completed. <br>Now we will create one ec2 instance in public subnet with public ip:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KCaAYY0ItTi7wIOw" /></figure><p>Now we will move to GCP to create the VPC there:<br>Create the VPC network there, put the network name and four subnets two private in us-east-1 and other in us-east-2 same goes for two public network, we will use below subnet ranges, also enable the private google access for private subnets:<br>10.10.1.0/24 private-01<br>10.10.2.0/24 private-02<br>10.10.3.0/24 public-01<br>10.10.4.0/24 public-02</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WUDuucCObYEV8qCk" /></figure><p>Google Cloud Router helps to dynamically exchange routes between your Virtual Private Cloud (VPC) and on-premises networks using Border Gateway Protocol (BGP):<br>Now we will create NAT gateways for our private subnets inside this cloud router:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/0*xHGKie17QWAh2Okr" /></figure><p>In our case we will have to create the the two cloud routers and two nat gateways as we have two private subnets in different regions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B1_W3dEenrTEMsyV" /></figure><p>In GCP, there is no separate component explicitly called an “Internet Gateway” like in AWS. Instead, GCP allows subnets to have direct access to the internet through the default routing and firewall rules. If a VM instance has an external IP address and the appropriate firewall rules are in place, it can access the internet directly.</p><p>Now we will create VPN on GCP, we will use HA because it provide two tunnels also support BGP dynamic routing, which is not supported by classic tunnel:<br>First it will ask u to create the cloud HA vpn gateway then it will create two vpn tunnels stop here copy the public ips of tunnel and came to aws:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c1tRGsyOmthahWjG" /></figure><p>At aws frist create the virtual private gateway and attach to VPC:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uIgbFT31W3Ib21G3" /></figure><p>Now we will create two customer gateways for each GCP tunnel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kk2HJIQfVFyypLTJ" /></figure><p>Now we will create site to site vpn connections:<br>Conn1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zmLoJr5VTpIDyV6j" /></figure><p>Create conn 2 with gateway02.<br>Now download the conf file of both vpn connections:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*Jdzx3_vzrolTZWvq" /></figure><p>We will require these file to check the conf information and configure GCP tunnel, lets come to the GCp and configure vpn tunnels:<br>First create the new vpn gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/0*4n3ZN5d2BhAKxMOs" /></figure><p>Public ips of aws tunnels.<br>Select the cloud router:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*UHuBCEktCZKvLGK9" /></figure><p>Create four tunnels at GCP use same preshared key that is in aws.<br>Nest step is to configure bgp sessions for each tunnel by using aws ASN and private ips of each tunnel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*QZOXQr1RFceypSF4" /></figure><p>All four tunnels are established and BGP is up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*INFWO_Bk8duphuAD" /></figure><p>Now create the google cloud compute engine instance and initiate ping to aws ec2</p><p>instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*sciOCCEmeyVTrUN0" /></figure><p>Thats all in this project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9a4f3f533cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing AWS WAF To Secure Web app against security vulnerabilities:]]></title>
            <link>https://ghazanfaralidevops.medium.com/implementing-aws-waf-to-secure-web-app-against-security-vulnerabilities-fe0041fd1663?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe0041fd1663</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Sat, 17 Aug 2024 15:33:13 GMT</pubDate>
            <atom:updated>2024-08-17T15:33:13.455Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:<br></strong>In this project we will not just explore the AWS WAF but also explore how to create VPC and its components, EC2 server and deploy app on it, implementation of WAF to protect our web app against cyber attacks and exploitation of security vulnerabilities.<br>Lets start the project!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*xiAVCKTjLvUjzKXO" /></figure><p>First we will create new vpc:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/0*KFVsm4zoK-nddyf4" /></figure><p>Then creates subnets, 2 private and 2 public with diff AZs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AKNAKGwPR3lyp_Lj" /></figure><p>Now we will create one NAT gateway for internet access to public subnets, NAT gateway will be created in public subnet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mNDEt7SwhEghR4mq" /></figure><p>Now create the internet gateway so that public subnets and nat gateway will be able to access the internet, Also attach the IG with vpc:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GaVzRocITdiu_bMt" /></figure><p>Now create the two route tables and attach the private subnets with private RT and public subnets with public RT, add one default route 0.0.0.0/0 of private RT toward NAT gateway , and one default route in public RT toward IG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KhkAA4ytKDeE2b-W" /></figure><p>Below is the example of public RT default route:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LhapN3rGHqs0rlBN" /></figure><p>Our VPC part is completed.<br>Now we will create one ec2 instance in public subnet with public ip:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*koI5YYjv7FFqfE5l" /></figure><p>Access the server and install apache server on it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/0*_eYLoYl30lsHm98S" /></figure><p>⇒ sudo apt update<br>⇒ sudo apt install apache2<br>After install apache open the port 80 of the SG of your server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wsrz6uvfDhMr6zHG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZQPir4d3DITnyhQZ" /></figure><p>Now we will create application load balancer which will sit between the waf and our app server:<br>Bit first create the target group and add our instance in that TG:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aFQnj6ELe6bmizcl" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/0*cNheB3Ok3KOzCaGK" /></figure><p>Add instance in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jFsgdDdc3LQzGgTq" /></figure><p>Now create the load balancer by using this TG:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/0*L0PCaKsSl956G20U" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tJIpjzkNCuvPKaVj" /></figure><p>Add TG in LB:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NM8fj66Feq-Ci_B_" /></figure><p>Also create new security group for this LB and allow port</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J1KcpWfDKbu9Zy7k" /></figure><p>Now paste the dns endpoint of your load balancer on browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jcxK750mwgaSIPY_" /></figure><p>Now we will create the application firewall, create the new web ACL rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BZB82sp4iH8iMHty" /></figure><p>Create the regional web acl and attach the LB in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/0*tHlt6YKyebXY61JP" /></figure><p>Now add the rules and rule groups, for testing we will block our own ip address:<br>So, first create the ip sets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/0*4kfgjJytApZ4AMSL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*KxaDoujztM-Z8PJ6" /></figure><p>Now add that in option add my own rule groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/0*TEfVSLXhD6hrysUo" /></figure><p>Once it is created, go to that LB endpoint again, it will block the ip:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QgHweT0Eo_cMMYby" /></figure><p>Thats all in this project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe0041fd1663" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All About AWS Lambda Service]]></title>
            <link>https://ghazanfaralidevops.medium.com/all-about-aws-lambda-service-7e5c9ac6c9bb?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e5c9ac6c9bb</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 17:12:02 GMT</pubDate>
            <atom:updated>2024-08-16T17:12:02.779Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS lambda is a compute service and fully serverless which allow to run your code without provisioning any server, it runs on pay as you go model. We do not need to manage the scaling according to the load, lambda itself manages the load and scaling. <strong><br>1-Concurrency:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*FVRpPdIH5I3YQTJQ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*gCMz2ZQp7KLuM4je" /></figure><p>There is two types of concurrency in lambda:<br>1- Reserved:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/0*3j9Y1AmXxVHkM5kM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*4GNsED3bRgqLahu8" /></figure><p>2- Provisioned:<br>In AWS lambda whenever it is invoked it runs the cold start, you can essentially tells AWS that i wanna respond immediately whenever request comes so that container will be already to run your lambda request it is called warm start:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*dDKl_1ETB6AVD045" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*-N6nlybaEuyf4JAL" /></figure><p>Here 90 means 90 containers are ready to provisioned my lambda request but i will also bear cost.<br>Let understand the lambda cold start:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/0*unarDqct_mJbWWQ8" /></figure><p>In this case a client is send the api request to api gateway then api gateway integrate with lambda that is invoked and update the dynamodb, here in this scenario lambda is not the server that is running all the time and respond immediately so it always takes the cold start means whenever it is invoked it spins up the new container and execute the lambda on it sometime it can cause delays as some request need immediate response.<br>Let suppose when first request is served and it comes the another request from another client in this case previous execution env will still be available so that it will skip the steps 2,3,4 in this case , it is called warm start. But without provisioned concurrency enabled if first request completed and then second comes there will be not warm start, it will again cold start so repeat all the 5 steps, other than this provisioned concurrency we can also use the aws cloud watch event role that will continuously ping the lambda at specific interval of time to keep our execution time keep active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*uFES8085CCWz-oqk" /></figure><p><strong>2- Layers:<br></strong>Whenever you work on multiple projects you create a library for multiple projects same goes for lambda:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*PTzevSImwZ7XRmVR" /></figure><p>Let suppose there is some common library that are common between many functions here comes the layer, so we do not need to copy the same code in every lambda we just create a layer (library) and use that in multiple lambdas in same AWS account. <br>We can upload the layer in zip file in lambda section in aws account:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/0*rr9SBxuCM2Yt_csB" /></figure><p><strong>3- Trigger:<br></strong>Let suppose you want to invoke your lambda function on some kind of interval or invoke by any other aws resources like api gateway, s3, dynamodb etc we use lambda triggers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VrtSkj-ZagxHHHBM" /></figure><p><strong>4- Environment Variables:<br></strong>AWS Lambda environment variables are simple key-value pairs that you can use to configure your Lambda function at runtime. They let you inject configuration settings, secrets, or any other data your function needs to run without changing the code. Here’s a straightforward explanation:</p><h3>What Are Environment Variables?</h3><ul><li>Key-Value Pairs: Environment variables are like a dictionary where each key (name) has a corresponding value. For example, API_KEY might be a key, and 12345 could be its value.</li><li>Configuration Settings: They help you provide different configurations or settings to your Lambda function based on the environment it’s running in (e.g., development, staging, production).</li></ul><p><strong>5- Step Functions with Lambda:<br></strong>AWS Step Functions is a service that helps you coordinate multiple AWS services into serverless workflows, making it easier to build and manage complex applications. It allows you to define a series of steps (or states) that execute in a specific order, using a visual workflow.</p><h3>Key Concepts</h3><ol><li>State Machine: This is the workflow definition. It includes the sequence of steps (states) that your application will execute.</li><li>States: Individual steps in the workflow. Each state can perform a task, make a choice, or pass data to the next state.</li><li>Tasks: The work that each state performs. In this context, tasks often involve invoking Lambda functions.</li></ol><h3>How AWS Step Functions Work with Lambda</h3><ol><li>Define a State Machine: You create a state machine that specifies the sequence of steps your application should follow. This definition is written in a JSON-based language called Amazon States Language (ASL).</li><li>Invoke Lambda Functions: Each state in your state machine can invoke a Lambda function to perform a specific task. For example, one state might call a Lambda function to process data, while another state calls a different Lambda function to send a notification.</li><li>Transition Between States: After a state completes its task, it transitions to the next state based on the outcome. This transition logic can include conditional branching, allowing your workflow to make decisions and follow different paths.</li><li>Handle Errors and Retries: Step Functions can handle errors and retries for you, ensuring that your workflow can recover from failures and continue executing.</li></ol><h3>Example Use Case</h3><p>Imagine you have an application that processes user uploads and sends a notification once processing is complete. Here’s how you could use AWS Step Functions with Lambda to achieve this:</p><ol><li>Upload State: When a user uploads a file, an initial Lambda function processes the file. This is the first state.</li><li>Processing State: After the upload is processed, the state machine transitions to the next state, which calls another Lambda function to analyze the file.</li><li>Notification State: Once the analysis is complete, the state machine transitions to a final state that calls a Lambda function to send a notification to the user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/0*8d4olHR9MSFs_Rs-" /></figure><ol><li>We can also use api gateway to invoke our step function like we do lambda integration.<br>Now we will see practical demo of step function with lambda:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7w7AjYRQpVwOTt26" /></figure><ol><li>State machine is like a work flow template, first we will create the design of out state machine, add choice and two rules in it, then paste the lambda there and update the names:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*l8JP62x45VQxQrrE" /></figure><ol><li>Condition example:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EbQUh-W1G3W583QY" /></figure><ol><li>Now we will write these lambdas and select them in state machine where we define the diagram:<br><strong>Purchase handler (pyhton 3.9):</strong><br>import json</li></ol><p>def lambda_handler(event, context):</p><p># TODO implement</p><p>return {</p><p>“result”: “Purchased proceed success”</p><p>}</p><p><strong>Refund Handler:<br></strong>import json</p><p>def lambda_handler(event, context):</p><p># TODO implement</p><p>return {</p><p>“result”: “Refund proceed success”</p><p>}</p><p><strong>3- Result Handler:<br></strong>import json</p><p>def lambda_handler(event, context):</p><p>step_function_result = event[‘result’]</p><p>return {</p><p>‘body’: step_function_result + “_RESULT”</p><p>}</p><p>Select the functions in state machine diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wk8rME6GNZRHWCX3" /></figure><p>Now its time to test things out!<br>Go to your state machine and click on start execution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7YY-ozn0E3mC9oLj" /></figure><p>Then give below input and click test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FQWGZUrekogJptQH" /></figure><p>It executed successfully executed:</p><p>1-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3qUjGK8kZ1zvIliz" /></figure><p>2-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LbhfEL8_pewvi1aP" /></figure><p>Thats all here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e5c9ac6c9bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction:
This project will help to deploy the real world containerized application on AWS…]]></title>
            <link>https://ghazanfaralidevops.medium.com/introduction-this-project-will-help-to-deploy-the-real-world-containerized-application-on-aws-f4984202feec?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4984202feec</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Tue, 13 Aug 2024 16:36:41 GMT</pubDate>
            <atom:updated>2024-08-13T16:36:41.182Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Deploying Docker Container on AWS Using ECS And ECR</strong></h3><p><strong>Introduction:<br></strong>This project will help to deploy the real world containerized application on AWS using their managed services so that we will not have to manage any server by own, ECS (elastic container service) is AWS managed service which offers you to deploy you containers on ec2 instances (you responsible for scaling, monitoring, patching of server) as well as using fargate (fully managed service). First we will create docker image and push on registry, we will use ECR for docker registry then deploy that image on ECS.<br>Lets start the practical demonstration!<br>Below is the link of our application:<br><a href="https://github.com/piyushgarg-dev/nodejs-docker-example.git">https://github.com/piyushgarg-dev/nodejs-docker-example.git<br></a>First create the docker image on local system using docker file in the repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EnuUlrkfpHZevklF" /></figure><p>Then run container on local system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/0*k3db_R8XV_Pgcspv" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*VobgDsEh1H4ZywQV" /></figure><p>Now we will push to image to AWS ECR, first create the repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_YjXzoUMlprsDPlL" /></figure><p>Follow the instruction to push the image to this repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aTdrOeKkZ5m5ppPz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h3X_FHVU2b7wnW9F" /></figure><p>Now we will go to ECS service for deployment of container using this image: <br>First we will create new cluster which can have many micro services:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7SyIDYNiXPeZcRb-" /></figure><p>Then we will create task, task is basically knows how to run image inside cluster, all the conf and variables will be in task:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*B646sH6AMDiphP9-" /></figure><p>Then we will ask service to get that task and run it. It will make sure scalability and re create if some thing crash etc, we can have multiple task also and one cluster can have multiple services also:<br>Create new task now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xiaqzczbh_i6NiEX" /></figure><p>Also provide image url and port in task:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1hqyMKwB27Ilqw9E" /></figure><p>Also define health check of container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/0*4KCD8soZyJ3z0EcY" /></figure><p>Now time to create a service that will take conf from task and run it:<br>First tell the service which task it will run and set the name of service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/0*AhD6BbCbuLwzgt8o" /></figure><p>Set desire task to 2 so there will load balancer which run two task parallel and divide load:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*FWyAB3dGb2f__kDo" /></figure><p>Now add load balancer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nSHvT2UJOi4Xww5v" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cJbRteVdMf2lgDbW" /></figure><p>Now our service is running with two task (container) as we define minimum value to 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_xrSJjTI4eRCCDQQ" /></figure><p>Now we can access our application through load balancer end point:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*rg1Ck2MC3qfLBIN5" /></figure><p>Thats all in this project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4984202feec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an AWS Serverless Web App with Amplify, Cognito, API-Gateway, Lambda and DynamoDB.]]></title>
            <link>https://ghazanfaralidevops.medium.com/build-an-aws-serverless-web-app-with-amplify-cognito-api-gateway-lambda-and-dynamodb-0f302a5ca978?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f302a5ca978</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Fri, 19 Jul 2024 13:48:02 GMT</pubDate>
            <atom:updated>2024-07-19T13:48:02.446Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:<br></strong>In this tutorial, you will create a serverless web application that enables users to request unicorn rides from the Wild Rydes fleet. The application will present users with an HTML-based user interface for indicating a pick-up location and a RESTful web service on the backend to submit the request for dispatching a unicorn. The application will also provide facilities for users to register with the service and log in before requesting rides.</p><p><strong>Prerequisites</strong></p><p>To complete this tutorial, you will need an AWS account, aws cli installed.<br><strong>Application Architecture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*ywsuZtBQ2ouoTqOG" /></figure><p>Amplify Console provides continuous deployment and hosting of the static web resources including HTML, CSS, JavaScript, and image files which are loaded in the user’s browser. JavaScript executed in the browser sends and receives data from a public backend API built using Lambda and API Gateway. Amazon Cognito provides user management and authentication functions to secure the backend API. Finally, DynamoDB provides a persistence layer where data can be stored by the API’s Lambda function.</p><p>Lets start the project!</p><p>First we will create a code commit repo for our project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eLvxFfWudgUiJoqb" /></figure><p>After creating the repo create the IAM user with git credentials:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MvA61xXvz1131Z49" /></figure><p>Also generate the https code commit creds for this user it will help to clone the repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/0*KDAkNLndZMAhOIzV" /></figure><p>Now we will clone our repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ym28o4kB3EWaA6cS" /></figure><p>On terminal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*WU9c6OifhEt0LL82" /></figure><p>Change directory into your repository and copy the static files from S3 using the following commands (make sure you change the Region in the following command to copy the files from the S3 bucket to the Region you selected at the beginning of this tutorial):<br>⇒cd wildrydes-site<br>⇒ aws s3 cp s3://wildrydes-us-east-1/WebApplication/1_StaticWebHosting/website ./ — recursive<br>Add, commit, and push the git files.</p><p>The following code block is an example of what you will see in your terminal window:</p><p>$ git add .</p><p>$ git commit -m “new files”</p><p>$ git push</p><p>Counting objects: 95, done.</p><p>Compressing objects: 100% (94/94), done.</p><p>Writing objects: 100% (95/95), 9.44 MiB | 14.87 MiB/s, done.</p><p>Total 95 (delta 2), reused 0 (delta 0)</p><p>To <a href="https://git-codecommit.us-east-1.amazonaws.com/v1/repos/wildrydes-site">https://git-codecommit.us-east-1.amazonaws.com/v1/repos/wildrydes-site</a></p><p>* [new branch] master -&gt; master</p><p>Now we will built the front end on AWS amplify:<br>Select the code commit → repo→ branch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e_jpyn2wdtZMLwks" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1NJx1hjBFIbVL8wE" /></figure><p>Once app create new IAM role with code commit full permission and assign this role to the amplify app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uie_OX1Uj_OdlkqI" /></figure><p>After deployment amplify will provide a link to access the website:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9q5Vonu1BxDr_xIn" /></figure><p>Amplify also manages the pipeline itself whenever you push any changes in the repo branch it will deploy the changes on environment itself.<br>Now we will move to second part which is creating the AWS cognito user pool for users authentication:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3YXJiz9QOEQdBUjl" /></figure><p>In MFA section pick no MFA, enable self registration, choose send email with cognito, put the userpool and app client name, leaves all setting default and create te user pool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*haf-yqB5hDutTa3y" /></figure><p>Now update the website config in config .js file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jsDaNIgjWKAb4X8x" /></figure><p>Save the file and push on the repo.</p><p>Now go to your website and click on “Giddy up” and register the user here it will available in the users section of your userpool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x_IAG9EpMxuIx-PW" /></figure><p>Now we will setup our serverless backend with lambda , dynamo and api gateway.<br>Create the dynamodb table first:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iiUm6D3Pqg7CgeLp" /></figure><p>And put all the settings as default, create table. Copy the ARN of table we will require it in lambda function.<br>Before creating the lambda we will create the IAM role of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xq7Z2qWWUBGdz_am" /></figure><p>Also add dynamodb permission to this role:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eLo0B7rVDT3H8f-s" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aDDgbOOdAtLy4eZw" /></figure><p>Put the dynamodb table ARN in resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*bgdZNdpns3ao0qFq" /></figure><p>Lets create the lambda function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*nr0pMZuevdFDNHIn" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/0*FE00f_x9e1rwH8rs" /></figure><p>Use the below code for lambda:<br>const randomBytes = require(‘crypto’).randomBytes;</p><p>const AWS = require(‘aws-sdk’);</p><p>const ddb = new AWS.DynamoDB.DocumentClient();</p><p>const fleet = [</p><p>{</p><p>Name: ‘Angel’,</p><p>Color: ‘White’,</p><p>Gender: ‘Female’,</p><p>},</p><p>{</p><p>Name: ‘Gil’,</p><p>Color: ‘White’,</p><p>Gender: ‘Male’,</p><p>},</p><p>{</p><p>Name: ‘Rocinante’,</p><p>Color: ‘Yellow’,</p><p>Gender: ‘Female’,</p><p>},</p><p>];</p><p>exports.handler = (event, context, callback) =&gt; {</p><p>if (!event.requestContext.authorizer) {</p><p>errorResponse(‘Authorization not configured’, context.awsRequestId, callback);</p><p>return;</p><p>}</p><p>const rideId = toUrlString(randomBytes(16));</p><p>console.log(‘Received event (‘, rideId, ‘): ‘, event);</p><p>// Because we’re using a Cognito User Pools authorizer, all of the claims</p><p>// included in the authentication token are provided in the request context.</p><p>// This includes the username as well as other attributes.</p><p>const username = event.requestContext.authorizer.claims[‘cognito:username’];</p><p>// The body field of the event in a proxy integration is a raw string.</p><p>// In order to extract meaningful values, we need to first parse this string</p><p>// into an object. A more robust implementation might inspect the Content-Type</p><p>// header first and use a different parsing strategy based on that value.</p><p>const requestBody = JSON.parse(event.body);</p><p>const pickupLocation = requestBody.PickupLocation;</p><p>const unicorn = findUnicorn(pickupLocation);</p><p>recordRide(rideId, username, unicorn).then(() =&gt; {</p><p>// You can use the callback function to provide a return value from your Node.js</p><p>// Lambda functions. The first parameter is used for failed invocations. The</p><p>// second parameter specifies the result data of the invocation.</p><p>// Because this Lambda function is called by an API Gateway proxy integration</p><p>// the result object must use the following structure.</p><p>callback(null, {</p><p>statusCode: 201,</p><p>body: JSON.stringify({</p><p>RideId: rideId,</p><p>Unicorn: unicorn,</p><p>Eta: ’30 seconds’,</p><p>Rider: username,</p><p>}),</p><p>headers: {</p><p>‘Access-Control-Allow-Origin’: ‘*’,</p><p>},</p><p>});</p><p>}).catch((err) =&gt; {</p><p>console.error(err);</p><p>// If there is an error during processing, catch it and return</p><p>// from the Lambda function successfully. Specify a 500 HTTP status</p><p>// code and provide an error message in the body. This will provide a</p><p>// more meaningful error response to the end client.</p><p>errorResponse(err.message, context.awsRequestId, callback)</p><p>});</p><p>};</p><p>// This is where you would implement logic to find the optimal unicorn for</p><p>// this ride (possibly invoking another Lambda function as a microservice.)</p><p>// For simplicity, we’ll just pick a unicorn at random.</p><p>function findUnicorn(pickupLocation) {</p><p>console.log(‘Finding unicorn for ‘, pickupLocation.Latitude, ‘, ‘, pickupLocation.Longitude);</p><p>return fleet[Math.floor(Math.random() * fleet.length)];</p><p>}</p><p>function recordRide(rideId, username, unicorn) {</p><p>return ddb.put({</p><p>TableName: ‘Rides’,</p><p>Item: {</p><p>RideId: rideId,</p><p>User: username,</p><p>Unicorn: unicorn,</p><p>RequestTime: new Date().toISOString(),</p><p>},</p><p>}).promise();</p><p>}</p><p>function toUrlString(buffer) {</p><p>return buffer.toString(‘base64’)</p><p>.replace(/\+/g, ‘-’)</p><p>.replace(/\//g, ‘_’)</p><p>.replace(/=/g, ‘’);</p><p>}</p><p>function errorResponse(errorMessage, awsRequestId, callback) {</p><p>callback(null, {</p><p>statusCode: 500,</p><p>body: JSON.stringify({</p><p>Error: errorMessage,</p><p>Reference: awsRequestId,</p><p>}),</p><p>headers: {</p><p>‘Access-Control-Allow-Origin’: ‘*’,</p><p>},</p><p>});</p><p>}</p><p>Now we will move to API gateway that will receive the traffic from front end and route to the backend using APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*4M4iTDT_IPOgvMXn" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*0Ix--GroWZnlI0D6" /></figure><p>Create new resource in the API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/0*PIHyryiqGNMMz_BV" /></figure><p>Now create the method for this resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*Wd_r06qGboG0eMIx" /></figure><p>Now we need to add user pool for authorization of this route, first create the authorizor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*37O2A92whXPVMK-E" /></figure><p>Now add it to the resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7NML95_yRoxFw2rx" /></figure><p>Add authorizer for this method:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*vs_4bi3TKBaj43A6" /></figure><p>At last deploy the API, and copy the URL and add it to the config file, so front end will generate request for backend using this url:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A-s1Jz9wxRuQ3nTB" /></figure><p>Also update the ride.html file:<br>Replace the argos version 4.3 to 4.6:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*L3xaHp_mx0WqLL97" /></figure><p>Now test the application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k9CITUAOWKfgxrKQ" /></figure><p>Thats all in the project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f302a5ca978" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating the Github project on Azure Devops (Azure Repo & Pipelines)]]></title>
            <link>https://ghazanfaralidevops.medium.com/migrating-the-github-project-on-azure-devops-azure-repo-pipelines-dcc4446dc34a?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcc4446dc34a</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Sun, 02 Jun 2024 14:00:58 GMT</pubDate>
            <atom:updated>2024-06-10T09:37:53.296Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/414/0*l69vkWUxRfGZL9UU" /></figure><h4>In this project we will be deploying the microservices application which is written in different programming languages, For CI part we will use pipelines in azure pipelines. This voting app is written in python and we have a memory data store called “Redis” and we have a worker which is written in .Net framework which takes the information from the redis and save in the database and finally there is another microservice called result which read the information from database and display it for you. To perform this task you should have a good understanding of CI and CD.</h4><h4>Below is the source code of the application:<br><a href="https://github.com/dockersamples/example-voting-app">https://github.com/dockersamples/example-voting-app</a></h4><h4>First we will clone the project locally and run the project on our local system through docker compose:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/0*ACSUgq0e1w3QiZf_" /></figure><h4>Once app is deployed in containers just open it on browser on port 5000 for voting service and port 5001 for result service as mentioned in docker compose file.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3myJ6gJSc6hAojw0" /></figure><h4>We will deploy each CI pipeline for each microservice like one for vote, result and one for worker. If you go through the each microservice folder you will see the separate docker file for each service.<br>Lets create new project on azure devops:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*OSOHxXsNKT1z8M1O" /></figure><h4>Although azure pipeline can also be use with github but we will fully do on azure devops so first go to repo section and import git hub repo there:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9DVVBT84BkzFA6xX" /></figure><h4>Select the main branch for azure pipeline and set it as default branch:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i7znuO3f00thwVm6" /></figure><h4>Now we have to create azure pipelines for our microservices, we will start with result services with template name “build and push docker image to azure container registry”:<br>For thai you have to create a container registry so create that first on azure portal:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2jV20d8ZBeuQHHQc" /></figure><h4>Now select this registry when creating the pipeline:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y_q9gCPwZTDdm596" /></figure><h4>When it create the yaml template for us we will modify some changes in it , like for trigger we will use path base triggering like result’s pipeline only trigger when there is commit in result folder only and same for other pipelines also:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*iG3ZiQGj2hgtGCdi" /></figure><h4>we will also update the agent details in yaml file because on free account azure does not provide you agent to run pipelines, so we will create our vm on azure portal and use that VM to run our pipelines:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*VieI41E_JpmLi519" /></figure><h4>We will create build and push stages separate so remove pool from here and below step click on setting and there it will give you options select there only build and give docker file path it will create the build step for you:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*_9fQ8mX53rn-C3P1" /></figure><h4>copy this stage add new one for push:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/0*KE81d7Q27ll_9h3J" /></figure><h4>Full pipeline:<br>trigger:</h4><h4>paths:</h4><h4>include:</h4><h4>- result/*</h4><h4>resources:</h4><h4>- repo: self</h4><h4>variables:</h4><h4># Container registry service connection established during pipeline creation</h4><h4>dockerRegistryServiceConnection: ‘7ca84e89–4b16–4eb1-bb52–62c5c450da13’</h4><h4>imageRepository: ‘resultapp’</h4><h4>containerRegistry: ‘microserviceregistryci.azurecr.io’</h4><h4>dockerfilePath: ‘$(Build.SourcesDirectory)/result/Dockerfile’</h4><h4>tag: ‘$(Build.BuildId)’</h4><h4>pool:</h4><h4>name: ‘azureagent’ # Ensure this pool name is correct and matches the one where your self-hosted agent is registered</h4><h4>stages:</h4><h4>- stage: Build</h4><h4>displayName: Build</h4><h4>jobs:</h4><h4>- job: Build</h4><h4>displayName: Build</h4><h4>steps:</h4><h4>- task: Docker@2</h4><h4>displayName: Build an image</h4><h4>inputs:</h4><h4>containerRegistry: ‘$(dockerRegistryServiceConnection)’</h4><h4>repository: ‘$(imageRepository)’</h4><h4>command: ‘build’</h4><h4>Dockerfile: ‘result/Dockerfile’</h4><h4>tags: ‘$(tag)’</h4><h4>- stage: Push</h4><h4>displayName: Push</h4><h4>jobs:</h4><h4>- job: Push</h4><h4>displayName: Push</h4><h4>steps:</h4><h4>- task: Docker@2</h4><h4>displayName: Push an image to container registry</h4><h4>inputs:</h4><h4>containerRegistry: ‘$(dockerRegistryServiceConnection)’</h4><h4>repository: ‘$(imageRepository)’</h4><h4>command: ‘push’</h4><h4>tags: ‘$(tag)’</h4><h4>Now create the azurevm then hit save and run:<br>VM:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uFgfRvXPuiraC6d6" /></figure><h4>ssh the VM and configure it as agent for our azure devops pipeline, for this below is the document:<br><a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops">https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops</a></h4><h4>Go in azure devops→ project setting → agent pool → add pool → self host pool and create new pool:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3fUEelarMNLOJXJ_" /></figure><h4>Inside agent pool create new agent and follow the steps:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*PJOBuLMM7jDKCpgA" /></figure><h4>Note: During this process it will ask for presonal access token you can generate it from azure devops→ user setting.</h4><h4>once these all command run, install the docker.io and assign permission to docker user:<br>⇒ sudo usermod -aG docker azureuser<br>⇒ sudo systemctl restart docker</h4><h4>Now check your agent should be in online state:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*rGEcVD0TJvEicek8" /></figure><h4>Now run the pipeline:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kn_JMSY5EZzN0WEu" /></figure><h4>We have done one pipeline for microservice called result now we will do the same for other two pipelines for worker and vote:</h4><h4>Our all three pipelines are created and ran:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yYZ0Dq1MjDa-X_fv" /></figure><h4>Check the docker registry there will be three docker images for each service:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yoE_-MhYCtQsVfET" /></figure><h4>So, we have completely migrated the github to azure devops pipelines ad our CI part of this project is completed.</h4><h3>Now we will start the CD part of this project, For continuous delivery we will use GitOps approach. <br>We will use thre stage in pipeline with name update basically its a shell script which clones the repo and update the manifest versions of our K8 yaml files then commit it:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>and then we will run argoCD it will detect changes in yaml files and fetch the changes and deploy on K8s cluster.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>Let start the configuration for this CD part:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>First we will create kubernetes service on azure kubernetes services, we will also need agent pool in AKS because to run pod or services in kubernetes AKS required compute power so that agent pool provides that in agent pool we can also select scale method and minimum maximum nodes we needed etc,<br>Note:<br>Also expose the public ip for node, and choose all default configurations.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>Cluster creation is in progress:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>Now we will install argocd in K8s cluster. First login to the cluster:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><h3>Setting up argocd:<br><a href="https://argo-cd.readthedocs.io/en/stable/getting_started/">https://argo-cd.readthedocs.io/en/stable/getting_started/</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*szTwyDLIdrx9ZtcO" /></figure><h3>argocd will create its pods:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*xljHcb7O7BZvXTPu" /></figure><p>Now we will configure argocd, first we will connect argocd with our git repo so it will its single source of truth and argocd will do continuous monitoring to this git repo whenever any change in the any K8s manifest it will deploy those changes on K8s cluster, and if someone will try to make any change directly in the cluster argocd will revert it on the basis on this git single source of truth.</p><p>First get the argocd credentials to login in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/0*slXpPhotDOUjTFZO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/0*_DbDAVaWNeDNEUyF" /></figure><p>secrets are base64 encoded so need to decode the secret first:<br>After decoding the password we can login on argocd UI, for this we have to expose argocd server in nodeport mode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*YGAtnjfx6ixXaNPv" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/0*yk9Ft-o3u6wVxu_c" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/0*q9sK1vBAfTIKRG2c" /></figure><p>Now access the argocd:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*owPByxU96Gl0cgwj" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/0*iPwNt-Euz0mehqRr" /></figure><p>But first open the node port in security group:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nJ_yMzJV7me_xAcs" /></figure><p>We have access argocd now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X0dHHDH4BpfdFEyQ" /></figure><p>Now we will connect argocd with our azure repository, we will create personal access token on repo and use that in argocd:</p><p>Now copy the https clone url of repo and put in the argocd but replace the first part of link with the token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lLZ9CXSiypKJC2_q" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hARuEfMaYY_mwUrb" /></figure><p>Now create new app on argocd:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DdIyaSCOa1nCm2cZ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*giIwVLXi2CLh6IfO" /></figure><p>It have started deploying app on kubernetes cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GKx5t9JkbKrjP2U1" /></figure><p>Now we will write the update script which will link our CI and CD, mean when CI runs in last stage we will run the update script which will modify the image tag of my manifests, so when CI pipelines runs this update script also update the image tag or number, so argos will fetch that changes and re deploy it on kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*m7lrEyPUAYYs1Vuv" /></figure><p>update the upper shared stage in all three microservices pipelines.</p><p>Now put the update script in new script folder at azure repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*omVy3VbFLVkppOuw" /></figure><p><strong>Below is the script:</strong><br>#!/bin/bash</p><p>set -x</p><p># Set the repository URL</p><p>REPO_URL=”https://&lt;ACCESS-TOKEN&gt;@dev.azure.com/&lt;AZURE-DEVOPS-ORG-NAME&gt;/voting-app/_git/voting-app”</p><p># Clone the git repository into the /tmp directory</p><p>git clone “$REPO_URL” /tmp/temp_repo</p><p># Navigate into the cloned repository directory</p><p>cd /tmp/temp_repo</p><p># Make changes to the Kubernetes manifest file(s)</p><p># For example, let’s say you want to change the image tag in a deployment.yaml file</p><p>sed -i “s|image:.*|image: &lt;ACR-REGISTRY-NAME&gt;/$2:$3|g” k8s-specifications/$1-deployment.yaml</p><p># Add the modified files</p><p>git add .</p><p># Commit the changes</p><p>git commit -m “Update Kubernetes manifest”</p><p># Push the changes back to the repository</p><p>git push</p><p># Cleanup: remove the temporary directory</p><p>rm -rf /tmp/temp_repo</p><p>Now run th each pipeline they will update the manifest file with image tag so that argocd will fetch the changes and deploy on cluster.</p><p>You will notice that argocd will deploy the changes but in the pod there will be an error of pulling image because we are now pulling the image from private container registry to resolve this we will use concept “ImagePullSecrets” to pull image from private registry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*4Vzk1tEcwylwuLVc" /></figure><p>enable the the admin user in your registry and copy the access keys from there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/0*9cU_dl2jPiJTX7KO" /></figure><p>Then at cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*vs1i0n4ERVJfSjhR" /></figure><p>Once secret created then come to the manifests file and update it also:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*eiH4PLa_GQzdIHa9" /></figure><p>Thats all in this project!<br> — — — — — — — — — — — — — — — — — — -</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcc4446dc34a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your First WebApp on Azure cloud]]></title>
            <link>https://ghazanfaralidevops.medium.com/build-your-first-webapp-on-azure-cloud-207fe9f381d2?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/207fe9f381d2</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Tue, 21 Nov 2023 07:01:22 GMT</pubDate>
            <atom:updated>2023-11-21T07:01:22.859Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>In this project we will use azure app service which is web hosting service like beanstalk in AWS. It supports all the popular programming languages that web developer use. Through azure app service we will get the benefit of security, high availability, scalability.</p><p>First create a new resource in azure cloud window and webapp instance in that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ni8F7_YQ8mzKld0I" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*Ay8uG2p9mvl6yYqN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*6rFnYX-hWoPSAWVn" /></figure><p>Then click on review+ create then app service instance will be created. You will get a url of your app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ErQjA2THMaVB6Bqv" /></figure><p>Now we will git clone the code from github to our webapp:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/0*-1wZ-UGiX8zESFIP" /></figure><p>We will deploy this index.html file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*aaPvLBxmFYLehb2V" /></figure><p>In order to deploy on your app service this need to be a git repo so initialize this git repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/0*hpOvXRkRKhqDpE33" /></figure><p>Lets add a files to it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*509wyIOcV9HpKW4e" /></figure><p>This is ready to deploy but app service does not get the git deployment by default so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/0*tDHh4LOm_9a9-yK3" /></figure><p>This command will output a url that we can use for git deployment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bGWy1WAhox5B4y0R" /></figure><p>We will use app level credentials for app deployment, another is user level credentials that can use for all apps in your subscription. To use app level creds find them first:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X8Ensyfq04I9gMiE" /></figure><p>This command printed the publishing username and password. It also gave the url that embedded your username and password we will add that as a remote in local github repository:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_ImhIr2_tXEYI2XH" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/0*3ykOmqwYTqAcK7i2" /></figure><p>Then push the code to this repo address the code will be deployed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/0*cRjmw40gUJKopaV7" /></figure><p>We have deployed it on azure app service,</p><p>But what if we do all these things with one azure cli command? Just go back to azure cloud shell and use below command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*CodhznzjqSvz3W1d" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=207fe9f381d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tetris Game Deployment on Azure Devops | Azure DevOps Project from Scratch | End to End Azure…]]></title>
            <link>https://ghazanfaralidevops.medium.com/tetris-game-deployment-on-azure-devops-azure-devops-project-from-scratch-end-to-end-azure-550cf8af1cfd?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/550cf8af1cfd</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Wed, 15 Nov 2023 14:54:08 GMT</pubDate>
            <atom:updated>2023-11-15T14:54:08.005Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Tetris Game Deployment on Azure Devops | Azure DevOps Project from Scratch | End to End Azure Project |</strong></h3><p><strong>Introduction:</strong></p><p>This azure devops project is divided into two parts:</p><p>1- First will be build pipeline.</p><p>2- Second will be Release pipeline.</p><p>In the build pipeline if the developer made any change in the github repo it will trigger the pipeline. It will pull the code from github repo and build the docker container image and push the image to the azure container registry (ACR) like ECR in AWS.</p><p>After the completion of build pipeline the release pipeline will be automatically triggered, the release pipeline will actually pull the artifact from the build pipeline which is the image on ACR, using that image it will deploy the azure container on azure app service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J05oCMikUAV8Ky47" /></figure><p>Lets start building our project!</p><p>First task is to create the azure container registry.</p><p>Go to container registry service and create new container registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CvkVp6Cn6oWQgiEg" /></figure><p>Create a new resource group for this project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*uzlCUg_RMmpyqOUW" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*ysn7CYB_fbZRhnkU" /></figure><p>Put all the remaining thighs to default then create the registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*6Xnu-uLt6A8-lM1g" /></figure><p>Now create new project in azure devops organization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CNNpeGfbR68XteVO" /></figure><p>Once project is created we will create two service connections one for ACR and second one is for github:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/0*IeKpOwq7w4BgnBN6" /></figure><p>go to project setting→ service connections → create service connection → select docker registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oqHm0gHU-nRzAeRH" /></figure><p>Then select azure container registry because we will save docker images on ACR here select your previously created docker registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*Y3CaOFof6Wgz7vd5" /></figure><p>Now create another service connection for github:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/0*PTTj2aOQ9rKRE1GE" /></figure><p>So we have two service connection now in our project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*IRC4hjHX-IWVPLxU" /></figure><p>Now we will create build pipeline in azure, go to pipeline section in your azure devops org’s project→ then click on create pipeline→ select github as source code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*KNfdsF1AbOj_hXGi" /></figure><p>Then choose the repo on github where your code is placed:</p><p><a href="https://github.com/devops-CloudComputing/Tetris-Game-App">https://github.com/devops-CloudComputing/Tetris-Game-App</a></p><p>In configure tab choose option : docker (build and push image to azure container registry):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IrMr5-MWE0qUHKFj" /></figure><p>Once validation complete click on save and run, it will start creating the pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/0*fQ8oKSMc84BSOTXj" /></figure><p>Build and push has been completed. Go to container registry there will be created docker image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*diFkvZSxJj82LhXP" /></figure><p><strong>Release to Azure Web APP</strong></p><p>Next task is ot create the linux web app for the container, Before that enable admin user in your container registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8lC4mwYCzZhjW2s5" /></figure><p>Now to go home→ ap services → webapp(instance):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/0*jcUD42k73X4wPan3" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*QrIxwQztV71cadQk" /></figure><p>Image source will be azure container registry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*_pOsA6REjWeADxg2" /></figure><p>So the linux web app for the containers has been created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qe7bAmShqaCe_Brd" /></figure><p>Next task is to create the release pipeline (CD part):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M1b6f2NAYUFfzXYF" /></figure><p>Also add artifact once stage is defined, select artifact from build pipeline on which we will run release:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*w_8lwouRe5t1pMOo" /></figure><p>So we have added artifact source and stage 1 of release pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/0*_5Gv7PFvqon7XB93" /></figure><p>Also enabled the continuous deployment trigger on artifact:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4gq8G0ldNM9rN1NY" /></figure><p>Now go to task option under stage 1, select your subscription and authorize it so that it will be able to create service connection with other apps like docker hub, github:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V3Rh_DaQMFyiHrki" /></figure><p>Click on save and then release and wait for deployment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/0*AQxe4wLyjHweRYiQ" /></figure><p>The app has been deployed on app service you can find the url in Deploy Azure App Service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BSZqWCC1Y-AqzzXh" /></figure><p>Before accessing this app go to app service→ setting→ configuration → application settings →new application settings and define the website port 3000 because our application is running on the port 3000:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*s4AC555CZeoeajDt" /></figure><p>Then click on save and access the app on browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GFAgLF4gzMEgjE8H" /></figure><p>Thats all in this Project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=550cf8af1cfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deployment of 10-Tier Microservices Application on Azure Kubernetes Cluster:]]></title>
            <link>https://ghazanfaralidevops.medium.com/deployment-of-10-tier-microservices-application-on-azure-kubernetes-cluster-778b078ea235?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/778b078ea235</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 13:49:06 GMT</pubDate>
            <atom:updated>2023-11-07T13:49:06.998Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>In this project we will use online shopping application which creates around 12 pods and same number of services when we creates the apps, so it likely to create 24 resources inside the kubernetes cluster when we deploy this application.</p><p><strong>What is microservices application and why we need it?</strong></p><p>Let consider the e commerce application like amazon where we do not write whole source code in a single folder but we write different source code in different folder for each service like payment, cart, shipping etc these all have different source code by keeping everything separate whenever need to change/update the source code of application it will easy to update the specific folder it will not effect the entire application. The best part of microservices application is that they can be written in different programming languages for each service.</p><p><strong>How the microservices communicate among each other?</strong></p><p>There are multiple ways for that one of the way is known as “Message Broker” any service want to communicate with other service first it communicate with the message broker then it forward the request to other service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*o9MDymFWrxTwHs8D" /></figure><p>Some of the common knows message broker is “RabbitMQ” and “RediCache”, Another way is like GRPC (Google remote procedure call), GRPC support too many variety of languages so we will also use it in our project.</p><p><strong>Architecture of our 10 Tier Application:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zf5YmijoxHJjuwRu" /></figure><p>Through browser we will access the application and <strong>front end</strong> will be the entry point of this application Then there will be service called <strong>product catalog service </strong>which is used when you search for different products on the application. <strong>Recommendation service </strong>will recommend you the items based on your search. <strong>Cart service </strong>will hold the item that we wish to buy. We will use <strong>redis database </strong>to store the cart item in it. <strong>AD service </strong>will manage the Ads runs on the application. <strong>Checkout service </strong>is used when you go to buy the item stored in cart service, this service also link with <strong>currency service </strong>which help us to select our desired payment currency, From checkout service we also have other three services that are link to it first is <strong>email service </strong>that is used to get the invoice or receipt of the shopping, payment will be managed bo another micro service called <strong>payment service </strong>and at last there is <strong>shipping service </strong>that is use for shipment fo the purchased items.</p><p>These all services are written in different programming languages that is the beauty of microservice applications.</p><p>Now we will create kubernetes cluster on azure cloud called (AKS azure kubernetes service)to deploy this application, we will use Dev/Test environment because it creates small cluster and very cheap in pricing as compare to production environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jwv5P7vKFzqwG1yZ" /></figure><p>Then define the node pool for your cluster, as we need just 2 cpus and 4gb of ram we will go with B2S node:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LQAbe9qLRFAWzo6L" /></figure><p>Go with all default setting in networking and integration, advanced, security tab and click on create cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rbx3hM9Y7gPSReA6" /></figure><p>Once cluster is created we will create <strong>Azure Devops Platform </strong>for deployment of the application, we can also use jenkins CICD but we will go with this azure service, Azure DevOps is a set of development tools and services provided by Microsoft for software development, including CI/CD (Continuous Integration and Continuous Deployment) capabilities. It offers a comprehensive platform for managing the entire software development lifecycle, from source code management to build and release automation.</p><p>In azure devops environment we first creates the organization then create a project in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OaBOLSlsrfcHSt8d" /></figure><p>Once project is created go to organization level(ghazanfarali9595) and enable pipeline build process there so that we will use azure devops pipeline service in our project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*yppPgitXmPCTpA_L" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/0*Bwnt0OfSnLPnfVGV" /></figure><p>Now go back inside our project and go in Repos option and import a github repo there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AnBjThPWBCnij89X" /></figure><p>Repo:</p><p><a href="https://github.com/GoogleCloudPlatform/microservices-demo">https://github.com/GoogleCloudPlatform/microservices-demo</a></p><p>It have successfully import the github repo to our project actually this repo has all the deployments and service files for each microservice application that will deploy on AKS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0huS768X3Rpg7vrz" /></figure><p>Now we will create pipeline for our project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5_WRmyoHERFHrm3y" /></figure><p>Since we have yaml file ready and we are not building the docker files from scratch so we will chose <strong>releases</strong> option in pipeline so that we can directly deploy to kubernetes cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/0*nc9enhcg080bw96h" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YsaVNcAQt2N2sUSO" /></figure><p>Here we created a new job then click on add artifact and and fetch the file from azure repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1z_SBliihebu58Qw" /></figure><p>We will use kubectl for to communicate with cluster so that add kubectl in agent section of your pipeline job and connect it with your newly created kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/0*ZVUphn7Dki1flo4c" /></figure><p>Use this connection in agent task, and in command tab use command “apply” and give the file path of kubernetes yaml file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*vR9lCTNGKAuUKgzO" /></figure><p>Once it saved, click on “create release” to do the deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/0*aojgbR3tj_rcdmiS" /></figure><p>It has created the new release:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*plpuWRgFAvVXjk9Y" /></figure><p>Now we will deploy the released pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*Lc4HnVIJ8v93T9Px" /></figure><p>It has started the deployment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IjSZtu1IRx7KPbKD" /></figure><p>This deployment can take little bit of time as it have to deployed 10 to 12 pods in our cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/0*zJgNSNAT1kLxXZGU" /></figure><p>Coming back to your cluster you will start seeing the deployed pods on your cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wQqHZGKArVSVK0fq" /></figure><p>Once these all pods created we will be able to access our application through services and ingress tab on our azure cluster:</p><p>There will be a service with name “front-external” in which we have used loadbalancer service type so that we can access our application from the outside world:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8TIbisrkbwryFQQ4" /></figure><p>Click on the service to check the application deployed status:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2tZPgB_60CVe-BKj" /></figure><p>Our app is deployed successfully on azure kubernetes cluster by using azure devops organization.</p><p>Thats all in this project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=778b078ea235" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deployment of a Netflix-Style Application on Kubernetes Using DevSecOps Practises:]]></title>
            <link>https://ghazanfaralidevops.medium.com/kubernetes-deployment-of-a-netflix-style-application-using-devsecops-practises-13c86961d765?source=rss-39137f9e988------2</link>
            <guid isPermaLink="false">https://medium.com/p/13c86961d765</guid>
            <dc:creator><![CDATA[Ghazanfar Ali]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 17:05:53 GMT</pubDate>
            <atom:updated>2023-11-01T10:29:42.015Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIoOnHIFXW1c9uc6QOnByQ.png" /></figure><p><strong>Introduction:</strong></p><p>We will start with creating an <strong>EC2</strong> instance and deploy our application locally using <strong>docker </strong>container once the application is running we are going to integrate security using <strong>sonarqube </strong>and <strong>trivy, </strong>Once it is done manually we will automate this process using <strong>jenkins CI </strong>and upload the image on <strong>docker hub.</strong> Then will integrate monitoring using <strong>prometheus</strong> and <strong>grafana</strong> for better visualization. On the basis of jenkins job success or failure we will get email notification using<strong> SMTP.</strong> Once we automate this whole process we will deploy the application on <strong>kubernetes </strong>using <strong>ArgoCD </strong>the GitOps tool which uses for CD process which tracks all the changes and records of deployment also. We will also have kubernetes monitoring using by using prometheus and grafana using <strong>helm charts.</strong></p><p>Lets start the project!</p><p>First we will install t2.large ubuntu based instance for jenkins, sonarqube and trivy.</p><p><strong>Note: </strong>Use the 25gb storage for this instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RwvJQBY7oQaXlgwn" /></figure><p>Also attach the elastic ip with with instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RBiOxq5h9D-ubrqO" /></figure><p>Access the newly created instance and update the packages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Mhb6F5H5D0TaKeV" /></figure><p>Once update is complete clone the below mentioned github repo in the instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wS-rsQv-hwq3hvgu" /></figure><p>We need to install docker our instance as we will create docker image from Dockerfile which is present in the cloned directory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O14o15IyKhoE5_Tq" /></figure><p>Lets create the docker image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_JCEnKacpNVoME7v" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ho5TXGZO0MNU5JKL" /></figure><p>Now we will create container form this image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qm91_-gFZSyZ2_gD" /></figure><p>Allow port 8082 in SG of instance as we are running this application on 8082 then check the status:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P6vo_j9lxASLhOV1" /></figure><p>It is not showing any movies here because we have to integrate our application with movies database with their API so that our application gets movies from there:</p><p>Create your account on <a href="https://themoviesdb.com/">https://themoviesdb</a>.org/login</p><p>In setting tab find the API option and request for new key then copy the API key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6o-5P7KQhVvnNf28" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*6iYZH1PikIm4Ffqo" /></figure><p>Now we will create new docker image with the help of this API key, But first stop the previous container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*zMMJTJYI0_m8nsm3" /></figure><p>Now create new image again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y64iScuBi-VdkQDe" /></figure><p>Now run the container from this new image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1ds3fD18s0TgZAg5" /></figure><p>Now check it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2dvpKecP2D7Jj-9U" /></figure><p>Now lets do the security part, First we will install sonarqube and trivy:</p><p><strong>sonarqube:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pgLJN0xT0bV-6XmM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x7a1zhu1me8mIMo-" /></figure><p><strong>Trivy:</strong></p><p>Trivy is the most popular open source security scanner, reliable, fast, and easy to use. Use Trivy to find vulnerabilities &amp; IaC misconfigurations, SBOM discovery, Cloud scanning, Kubernetes security risks,and more.</p><p>⇒ sudo apt-get install wget apt-transport-https gnupg lsb-release</p><p>⇒ wget -qO — <a href="https://aquasecurity.github.io/trivy-repo/deb/public.key">https://aquasecurity.github.io/trivy-repo/deb/public.key</a> | sudo apt-key add -</p><p>⇒ echo deb <a href="https://aquasecurity.github.io/trivy-repo/deb">https://aquasecurity.github.io/trivy-repo/deb</a> $(lsb_release -sc) main | sudo tee -a ⇒/etc/apt/sources.list.d/trivy.list</p><p>⇒ sudo apt-get update</p><p>⇒ sudo apt-get install trivy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZZOPepe_OwYRfL9C" /></figure><p>Now we will scan the filesystem using trivy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-N3EeXfQgYUzCBSu" /></figure><p>It also found the known CVE in our repository:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NKbb7vciTb3SUCA6" /></figure><p>We can also use trivy to find vulnerabilities in docker images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tZmtB1SWszCF8cGN" /></figure><p>It did not find any vulnerability in the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d-lmuXHMJkuYwNb8" /></figure><p>Now we will move toward automation by CICD we will use jenkins for this, install the jenkins on our instance, once jenkins is installed access it on port 8080 but first allow the port 8080 in security group of jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MT4xNju7TeuAkPR7" /></figure><p>Add few plugins in jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wegJPuP0Xy1PyyJW" /></figure><p>Once plugin added go to tools section and add these tools there:</p><p>1- JDK:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/0*bAkB5VyBOsX66GML" /></figure><p>2- NodeJs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*E9SAqYXT1V5Hqpi2" /></figure><p>Now configure the sonarqube:</p><p>Administration → security → users → update token → generate new token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y7Q_WH6scYWlD_lI" /></figure><p>Now copy this token and add in jenkins credentials tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D45RWy5qtcQrU4D2" /></figure><p>Now come to manage jenkins → systems then search for sonarqube and define the sonarqube location there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*2LR821R07pxpjwRX" /></figure><p>Add the sonar-scanner tool too because we will use it in the pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/0*9oPg23-bM__MAwok" /></figure><p>Our pipeline is ready to deploy our application, so create new pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gykyDJSXGUAw1XU5" /></figure><p>Now copy the jenkins script till stage “install dependencies” from github repo and paste in jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ihRpMZyiOv0goeca" /></figure><p>At stage sonarqube analysis there is project keyname “netflix” which we will add on sonarqube:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jc7EFNObEz44aEjS" /></figure><p>Then click on locally then generate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g3kQJfPlmupLGJZy" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PJGhFRIJTri_PwLF" /></figure><p>Now build the pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vgdWcw04qdpQDMSY" /></figure><p>Meanwhile pipeline is executing we can install more plugins in our pipeline:</p><p>1- OWASP Dependency check</p><p>2- Docker</p><p>3- Docker commons</p><p>4- Docker pipeline</p><p>5- Docker API</p><p>6- Docker-build-step</p><p>We will create docker image through jenkins and also add or push it on docker hub using jenkins so we will add docker hub credentials on jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*82jEHRtlUuNC3IaS" /></figure><p>Also add the owasp dependency check in tool section of jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*tU4A1zf4udlEfWON" /></figure><p>Also add docker:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/0*kviQJxr4mw6S6U39" /></figure><p>Now delete the docker container that we run on our instance because we will run the container through pipeline:</p><p>Now paste the full pipleline</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/0*rWQKc1VctDgqk1dz" /></figure><p>that also create docker image push it on docker hub, check the dependency check through OWASP and use trivy for docker image.</p><p><strong>Note: </strong>Do not forget to put the TMDB API key in the pipeline, add the jenkins user in docker group and replace the docker hub image name with your docker_hub name mine is “markhor1995”.</p><p>Now we will add monitoring which is the next part of our Ops section, create new t2.medium instance for monitoring purpose:</p><p>We will install prometheus and grafana as a monitoring tool:</p><p>First, create a dedicated Linux user for Prometheus and download Prometheus:</p><p>⇒ sudo useradd — system — no-create-home — shell /bin/false prometheus</p><p>⇒ wget <a href="https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz">https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz</a></p><p>Extract Prometheus files, move them, and create directories:</p><p>⇒ tar -xvf prometheus-2.47.1.linux-amd64.tar.gz</p><p>⇒ cd prometheus-2.47.1.linux-amd64/</p><p>⇒ sudo mkdir -p /data /etc/prometheus</p><p>⇒ sudo mv prometheus promtool /usr/local/bin/</p><p>⇒ sudo mv consoles/ console_libraries/ /etc/prometheus/</p><p>⇒ sudo mv prometheus.yml /etc/prometheus/prometheus.yml</p><p>Set ownership for directories:</p><p>⇒ sudo chown -R prometheus:prometheus /etc/prometheus/ /data/</p><p>Create a systemd unit configuration file for Prometheus:</p><p>⇒ sudo nano /etc/systemd/system/prometheus.service</p><p>Add the following content to the prometheus.service file:</p><p>[Unit]</p><p>Description=Prometheus</p><p>Wants=network-online.target</p><p>After=network-online.target</p><p>StartLimitIntervalSec=500</p><p>StartLimitBurst=5</p><p>[Service]</p><p>User=prometheus</p><p>Group=prometheus</p><p>Type=simple</p><p>Restart=on-failure</p><p>RestartSec=5s</p><p>ExecStart=/usr/local/bin/prometheus \</p><p>— config.file=/etc/prometheus/prometheus.yml \</p><p>— storage.tsdb.path=/data \</p><p>— web.console.templates=/etc/prometheus/consoles \</p><p>— web.console.libraries=/etc/prometheus/console_libraries \</p><p>— web.listen-address=0.0.0.0:9090 \</p><p>— web.enable-lifecycle</p><p>[Install]</p><p>WantedBy=multi-user.target</p><p>Enable and start Prometheus:</p><p>⇒ sudo systemctl enable prometheus</p><p>⇒ sudo systemctl start prometheus</p><p>Verify Prometheus’s status:</p><p>⇒ sudo systemctl status prometheus</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*4m3SqSdAu1r82iwp" /></figure><p>Prometheus runs on port 9090:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yr9npGn8M6sdw4GL" /></figure><p><strong>Installing Node exporter:</strong></p><p>Node Exporter is a software component used in the Prometheus monitoring and alerting ecosystem. It plays a crucial role in collecting various system-level metrics and statistics from a target machine and makes them available for scraping by Prometheus.</p><p>Create a system user for Node Exporter and download Node Exporter:</p><p>⇒ sudo useradd — system — no-create-home — shell /bin/false node_exporter</p><p>⇒ wget <a href="https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz">https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz</a></p><p>Extract Node Exporter files, move the binary, and clean up:</p><p>⇒ tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz</p><p>⇒sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/</p><p>⇒ rm -rf node_exporter*</p><p>Create a systemd unit configuration file for Node Exporter:</p><p>⇒ sudo nano /etc/systemd/system/node_exporter.service</p><p>Add the following content to the node_exporter.service file:</p><p>[Unit]</p><p>Description=Node Exporter</p><p>Wants=network-online.target</p><p>After=network-online.target</p><p>StartLimitIntervalSec=500</p><p>StartLimitBurst=5</p><p>[Service]</p><p>User=node_exporter</p><p>Group=node_exporter</p><p>Type=simple</p><p>Restart=on-failure</p><p>RestartSec=5s</p><p>ExecStart=/usr/local/bin/node_exporter — collector.logind</p><p>[Install]</p><p>WantedBy=multi-user.target</p><p>Enable and start Node Exporter:</p><p>⇒ sudo systemctl enable node_exporter</p><p>⇒ sudo systemctl start node_exporter</p><p>Verify the Node Exporter’s status:</p><p>⇒ sudo systemctl status node_exporter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/0*hWZAPYBbedz8fB6p" /></figure><p><strong>Configure Prometheus Plugin Integration:</strong></p><p>Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.</p><p><strong>Prometheus Configuration:</strong></p><p>To configure Prometheus to scrape metrics from Node Exporter and Jenkins, you need to modify the prometheus.yml file. Here is an example prometheus.yml configuration for your setup:</p><p>⇒ sudo nano /etc/prometheus/prometheus.yml</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*EGJGWAPrbOyaClTm" /></figure><p>Reload the Prometheus configuration without restarting:</p><p>⇒ curl -X POST <a href="http://localhost:9090/-/reload">http://localhost:9090/-/reload</a></p><p>Now we can see jenkins and node exporters in prometheus target:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PP_76i4PAO4DlMnH" /></figure><p>Now we will install grafana to get better monitoring visualization which is taken by prometheus:</p><p><strong>Step 1: Install Dependencies:</strong></p><p>First, ensure that all necessary dependencies are installed:</p><p>⇒ sudo apt-get update</p><p>⇒ sudo apt-get install -y apt-transport-https software-properties-common</p><p><strong>Step 2: Add the GPG Key:</strong></p><p>Add the GPG key for Grafana:</p><p>⇒ wget -q -O — <a href="https://packages.grafana.com/gpg.key">https://packages.grafana.com/gpg.key</a> | sudo apt-key add -</p><p><strong>Step 3: Add Grafana Repository:</strong></p><p>Add the repository for Grafana stable releases:</p><p>⇒ echo “deb <a href="https://packages.grafana.com/oss/deb">https://packages.grafana.com/oss/deb</a> stable main” | sudo tee -a /etc/apt/sources.list.d/grafana.list</p><p><strong>Step 4: Update and Install Grafana:</strong></p><p>Update the package list and install Grafana:</p><p>⇒ sudo apt-get update</p><p>⇒ sudo apt-get -y install grafana</p><p><strong>Step 5: Enable and Start Grafana Service:</strong></p><p>To automatically start Grafana after a reboot, enable the service:</p><p>⇒ sudo systemctl enable grafana-server</p><p>Then, start Grafana:</p><p>⇒ sudo systemctl start grafana-server</p><p><strong>Step 6: Check Grafana Status:</strong></p><p>Verify the status of the Grafana service to ensure it’s running correctly:</p><p>⇒ sudo systemctl status grafana-server</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/0*Yi3kfuorTRZKRFMK" /></figure><p><strong>Step 7: Access Grafana Web Interface:</strong></p><p>Open a web browser and navigate to Grafana using your server’s IP address. The default port for Grafana is 3000. For example:</p><p>http://&lt;your-server-ip&gt;:3000</p><p>Default user and pass is admin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nzsIJiJZFvetDdam" /></figure><p>In data source section choose prometheus as a data source for grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lx_yH-7r7thDl8Se" /></figure><p>Now we will import dashboard to display visualization on grafana use 1860 is that is for node exporter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oqYxvA749ckhCdrP" /></figure><p>Then select prometheus datasource and click on import then you will start seeing the data visualization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lJuih-iS7mS8to_R" /></figure><p>To monitor the jenkins in grafana we will install plugin the jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*W_FOgksycrD2RKXH" /></figure><p>Now you will see jenkins is also up in target of prometheus before it was down:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ET_0HvIuT5JplY0f" /></figure><p>Lets add a dashboard for jenkins with id 9964 at grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zVL6QmD2DxPYKAgZ" /></figure><p>Now we will add email notification, first make sure 2fa is enable in your google account:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nZLAGxZGsr5ig7pD" /></figure><p>Then search for app passwords:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/0*K_j0NO5I6rh1DcdW" /></figure><p>Once you create it will give you a password, copy this password we will use it in</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9J_kq6YMFdurMqtD" /></figure><p>jenkins to send us the notifications:</p><p>Now come in the system options and define the gmail smtp there:</p><p>Then in jenkins→ manage → credentials section create the new credentials with gmail email id and password that google shared with us:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ys8HzPL4zGB3u_nw" /></figure><p>Use this in extended email notification section at systems option:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/0*p6cnPEsUtIa96P_-" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/0*ZKY4l0dQL66pbJTH" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/0*WHavMuPpXsotr8SN" /></figure><p>Add a new stage in pipeline so that i sends the notification on our email:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*oC2sJ7G-KElyvcmB" /></figure><p>Now build the pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lw0bQrV-Kwz4j5Up" /></figure><p>It also created and push docker image on docker hub:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*ssozSQBr36xS_G3q" /></figure><p>Also received the email notification:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*z3kUA0uV6585mDa7" /></figure><p>Application is also running in docker container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HnJczR9ORXD9jmxo" /></figure><p>Now we will start argoCD part first install the argoCD it will help to keep the record in continuous deployment also, we will use minikube as kubernetes cluster you can also use AWS EKS for k8s cluster:</p><p>⇒ kubectl create namespace argocd</p><p>⇒ kubectl apply -n argocd -f <a href="https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml">https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*WcXYlDc_1JgGmZjO" /></figure><p>We will use helm chart for defining, installing and upgrading application in kubernetes cluster:</p><p>First install the helm through script:</p><p><strong>$</strong> curl -fsSL -o get_helm.sh <a href="https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3">https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3</a></p><p><strong>$</strong> chmod 700 get_helm.sh</p><p><strong>$</strong> ./get_helm.sh</p><h3><a href="https://github.com/N4si/DevSecOps-Project/tree/main#monitor-kubernetes-with-prometheus">Monitor Kubernetes with Prometheus</a></h3><p>Prometheus is a powerful monitoring and alerting toolkit, and you’ll use it to monitor your Kubernetes cluster. Additionally, you’ll install the node exporter using Helm to collect metrics from your cluster nodes.</p><p><a href="https://github.com/N4si/DevSecOps-Project/tree/main#install-node-exporter-using-helm"><strong>Install Node Exporter using Helm<br></strong></a>To begin monitoring your Kubernetes cluster, you’ll install the Prometheus Node Exporter. This component allows you to collect system-level metrics from your cluster nodes. Here are the steps to install the Node Exporter using Helm:</p><ol><li>Add the Prometheus Community Helm repository:</li><li>⇒ helm repo add prometheus-community <a href="https://prometheus-community.github.io/helm-charts">https://prometheus-community.github.io/helm-charts</a></li><li>Create a Kubernetes namespace for the Node Exporter:</li><li>⇒ kubectl create namespace prometheus-node-exporter</li><li>Install the Node Exporter using Helm:</li><li>⇒ helm install prometheus-node-exporter prometheus-community/prometheus-node-exporter — namespace prometheus-node-exporter</li></ol><p>This will install prometheus to monitor kuberntees through prometheus the same we are monitoring our jenkins server.</p><p>Now we will edit argoCD service file to change from cluster ip to nodeport type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*Z9emwWJP0rdmUQkT" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/0*8oHmgagA9BOvGwgg" /></figure><p>Now again check the service status it will show the http and https port also find the minikube cluster ip and access the argocd server on browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qrrzu1mueqjRQYHD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SnCLoXPNg8n8Haey" /></figure><p>Get the password of argocd through secret file and decode it using base64, default username is admin:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*DwFXg5peFS8ia11u" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/0*pkxKsfVxU_6gnq9c" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/0*ilQkmXO3Y05FkAg4" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/0*VlJb-fd9KkhZXRtK" /></figure><p>Now connect the repo to argocd so that it get the application and deploy on K8s cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r2LJLn5zfDCVNrU3" /></figure><p>Now create new app on argocd:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Neb0D8a3sLUsw_Ye" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tl37TipFgV7ih-HP" /></figure><p>Click on create app it will start syncing the application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V2c3g6VtcriVbzHx" /></figure><p>It have deployed the application on minikube cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*52bT4QegOh43fiKA" /></figure><p>Now we can access the application through nodeport and cluster ip:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PSOPSyBqj4FOFJSo" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ngxl5DlPTLJCZraS" /></figure><p>Our node exporter is running on 9100 port so we will create this job name in prometheus.yaml file so that it start the cluster monitoring also:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/0*ympgFJ-ks1SASPyo" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*8DgLT2EuhQ-tHhYF" /></figure><p>Check the validity of the configuration file:</p><p>⇒ promtool check config /etc/prometheus/prometheus.yml</p><p>Reload the Prometheus configuration without restarting:</p><p>⇒ curl -X POST <a href="http://localhost:9090/-/reload">http://localhost:9090/-/reload</a></p><p>Now you can see the kubernetes cluster as well in the prometheus monitoring:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u2bseuvVSdxD2Lb7" /></figure><p>Thats all in this DevSecOps project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13c86961d765" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>