<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinuja Khatode on Medium]]></title>
        <description><![CDATA[Stories by Vinuja Khatode on Medium]]></description>
        <link>https://medium.com/@vinujakhatode?source=rss-4c1c69284406------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uiPcZ_lHK3mTH_07aPn0Mw.jpeg</url>
            <title>Stories by Vinuja Khatode on Medium</title>
            <link>https://medium.com/@vinujakhatode?source=rss-4c1c69284406------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:15:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinujakhatode/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Amazon EKS and WordPress with EFS.]]></title>
            <link>https://medium.com/@vinujakhatode/amazon-eks-and-wordpress-45fe862b479c?source=rss-4c1c69284406------2</link>
            <guid isPermaLink="false">https://medium.com/p/45fe862b479c</guid>
            <category><![CDATA[amazon-eks]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vinuja Khatode]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 16:29:09 GMT</pubDate>
            <atom:updated>2020-07-17T15:22:02.313Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Back with another article!😎</strong></blockquote><p>Today, I’m here to share my new project based on awesome concepts of Hybrid Multi-Cloud i.e. Kubernetes on top of Cloud using which we can launch a front-end application with Load balancers.🤘</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*xWDp4pn6-w5MVy69.gif" /></figure><p><strong><em>Technologies which are going to be discussed in this blog are:</em></strong></p><ol><li>Amazon Elastic Kubernetes Service (EKS)</li><li>Kubernetes clusters</li><li>Amazon Web Services(AWS)</li></ol><h4><strong>Very important question! What is the Amazon elastic Kubernetes service?</strong></h4><p>Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed <a href="https://aws.amazon.com/kubernetes/">Kubernetes</a> service. Customers such as <strong>Intel, Snapchat, Intuit, GoDaddy, Autodesk, and many more </strong>trust EKS to run their most sensitive and mission-critical applications because of its security, reliability, and scalability. EKS is deeply integrated with services such as Amazon CloudWatch, Auto Scaling Groups, AWS Identity and Access Management (IAM), and Amazon Virtual Private Cloud (VPC), providing you a seamless experience to monitor, scale, and load-balance your applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HrkDC7n4FeKaNXQc.png" /><figcaption>Associated companies</figcaption></figure><h4><strong>Another Important question! How WordPress with MySQL backend can be hosted on AWS?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*5XvXG6_dpAU_b3V4.gif" /></figure><p><strong>WordPress</strong> is a common Content Management System for building websites and blogs. Scaling WordPress can be difficult, especially in the cloud due to the shared file system requirement for uploads, plugins, and themes. AWS publishes a <a href="https://d1.awsstatic.com/whitepapers/wordpress-best-practices-on-aws.pdf"><strong><em>document</em></strong></a> with best practices for running highly scalable WordPress installations on AWS.</p><p>This blog follows the WordPress deployment for putting the entire WordPress codebase in an <strong>Elastic File System (EFS)</strong> mount.</p><h4>You must be thinking, If we need storage, there are storage services offered by AWS like S3 and EBS then why we need EFS?</h4><ul><li>EFS is the storage option for EC2, that can be attached to <strong><em>more than one EC2</em></strong>, remember Volume can only be attached to one EC2.</li><li>EFS is Elastic, and it is scalable ( The <strong><em>storage capacity will increase or decrease as you add files to it</em></strong>, It is not like Volume)</li><li>You only pay for the <strong><em>storage you use</em></strong> (no pre-provisioning required)</li><li>Amazon EFS is a regional service storing data, within and across <strong>multiple Availability Zones (AZs)</strong> for high availability and durability.</li></ul><p><strong><em>Moving on, here are the prerequisites:</em></strong></p><ul><li><a href="https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html">AWS CLI</a> with Amazon AWS account</li><li><a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">Kubectl</a></li><li><a href="https://kubernetes.io/docs/tasks/tools/install-minikube/">Minikube</a></li><li><a href="https://eksctl.io/">Eksctl</a></li><li><a href="https://www.terraform.io/downloads.html">Terraform</a> (Optional)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*nx_eJTY5pmuKHqei.gif" /><figcaption>If you are Indian techie, you can definitely hear this gif!!!</figcaption></figure><h3>Our Objectives:</h3><ul><li>Create a <strong>Kubernetes cluster</strong> using AWS EKS.</li><li>Integrate <strong>EKS with EC2, ELB, EBS, EFS</strong>.</li><li>Deploying <strong>WordPress &amp; Mysql</strong> on top of <strong>AWS EKS.</strong></li></ul><p>🤗In this article, I am going to do everything from the command line <strong><em>(AWS CLI &amp; eksctl)</em></strong> So I recommend you First create an <strong>Admin power IAM User </strong>by following the step given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*38khemhwtUeYyLL8.jpeg" /><figcaption>AWS profile configuration</figcaption></figure><ol><li>Go to users and click on Add user from the IAM service in Amazon console.</li><li>Give some name and password to the user and don’t forget to check on both <strong>Programmatic access </strong>and<strong> AWS Management Console access.</strong></li><li>Select <strong>AdministratorAccess</strong> power, Check on it to create a user.</li><li>Please don’t forget to download<strong> credentials <em>(AWS Access Key ID and secret key ID)</em></strong> else you won’t be able to log in via the command line.</li></ol><blockquote><em>🎗Refer this blog for AWS configuration: </em><a href="https://medium.com/@vinujakhatode/terraform-aws-and-cloud-automation-b7d60cc6a3a2"><strong><em>Blog for IaC by Terraform</em></strong></a></blockquote><h4>Now, You have two choices launch the cluster i.e. using Terraform or using the eksctl command-line tool.</h4><ul><li>I will give you hands-on with both of them but let’s first build a cluster using <strong>eksctl CLI</strong>.</li><li>Create one <strong><em>.yml file</em></strong> to deploy a cluster on EKS at AWS, Which contains <strong>1 master and 2 slave nodes</strong> together 3 Instances at Amazon AWS.</li><li>The 1st node group will contain two ec2 instances of type <strong>t2.micro </strong>and The 2nd node group will contain <strong>min two and max three</strong> ec2 instances of <strong>type t2.micro</strong> and that node group is a <strong>spot instance</strong> type i.e it will automatically create the instance when a client tries to access the webserver.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7e854030e1d0985af906bc3c8b165e6/href">https://medium.com/media/d7e854030e1d0985af906bc3c8b165e6/href</a></iframe><ul><li>Create a file given above and save it in your workspace folder for this project, Now let’s run this file by using eksctl.io CLI commands.</li></ul><pre>eksctl create cluster -f cluster.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BSJIvwPaA1ilo-ma.jpeg" /></figure><p>Hence cluster is created, You can check it manually through <strong>Amazon AWS console.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9gLkdju5OYc2PyzM.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yWB7C5xtwXXW365q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RiLBCpz0luzVOai9.jpeg" /></figure><ul><li>To run any services on Kubernetes, We have to set the Kube config file in .kube folder in any system, Run below command for the same.</li></ul><pre>aws eks update-kubeconfig --name &lt;Your Cluster Name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-92YYSHLF-eHrEKI.jpeg" /><figcaption>kube config</figcaption></figure><ul><li>Now, Before moving ahead there is one step which you have to do manually but using terraform it can also be done automatically.</li></ul><h4>Check the steps below which you have to configure:</h4><p>→ Make one <strong>EFS file System</strong>, By choosing EFS service from Amazon AWS console.</p><p>→ Choose <strong>VPC where your instances are located</strong>, After choosing VPC you can see there three instances.</p><p>→ Change the Security group of each instance to a<strong> common security group</strong> you can find it by checking Amazon AWS running instances in EC2.</p><p>→ After the VPC and security groups are set, Now click to <strong>create.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RL_pTqRix-EbiSUf.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HldIyROuUAwinkhu.jpeg" /></figure><h4>Copy this DNS to use in your further WordPress deployment process.</h4><p><strong><em>The Above procedure can also be done using Terraform code, Check below is terraform code provided, Copy it in the same folder.</em></strong></p><p>Here is a bit of code in this gist, you can refer the whole code <a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/1cluster.tf">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92e48cd562e68ce44134cb5325bc5dac/href">https://medium.com/media/92e48cd562e68ce44134cb5325bc5dac/href</a></iframe><p>→ The above can build entire infrastructure discussed above using eksctl CLI, You just need to run this file using the command given below:</p><pre>terraform init<br>terraform validate<br>terraform apply -auto-approve</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qmz6IUQhd0TPBfn_.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-VKvMTd3eudyBZNU.jpeg" /></figure><h3>🚨You can face one error, During this Terraform IaC code building, Sharing because it’s a general error due to some typo mistakes.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aRbjz6Zeog6OfCZO.jpeg" /><figcaption>security group error</figcaption></figure><ul><li>To solve this error, Follow:</li></ul><p>→ You have to confirm that each subnet that you have provided here should be connected to the same VPC.</p><p>→ Also, those VPC should contain the same Security group attached.</p><p><strong>Congratulations🎉, Most of the part is already done… The cluster has been created!!</strong></p><blockquote><strong>Now Let’s work on the main part to deploy WordPress, Joomla, OwnCloud, NextCloud and many such Apps using EKS over Amazon Web Services</strong><em>.</em></blockquote><h4>♦ I made these 7 files to deploying WordPress over EKS:</h4><p><em>→ </em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/cluster.yml"><em>cluster.yml</em></a></p><p><em>→ </em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/efs-provisioner.yml"><em>efs-provisioner.yml</em></a></p><p><em>→ </em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/rbac.yml"><em>rbac.yml</em></a></p><p><em>→</em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/storageclass.yml"><em> storageclass.yml</em></a></p><p><em>→</em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/deploy%20mysql.yml"><em> deploy mysql.yml</em></a></p><p><em>→ </em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/deploy%20wp.yml"><em>deploy wp.yml</em></a></p><p><em>→ </em><a href="https://github.com/vinujakhatode/Amazon-EKS-Project/blob/master/kustomization.yml"><em>kustomization.yml</em></a></p><blockquote><em>Click on Above files, there are linked to my codes for the same. Do </em><strong><em>fork</em></strong><em> my repo and </em><strong><em>Star </em></strong><em>the repo for such exciting blogs in future. </em><strong><em>Many thanks</em></strong><em>✨</em></blockquote><ul><li>If you remember I told you above to copy one <strong>DNS website</strong>, Now we are going to use it in<strong><em> the efs-provisioner.yml</em></strong> file. Change the server website given in my code with your DNS website and also change the <strong><em>value for env</em> </strong>in the same file using <strong><em>DNS first 12 characters.</em></strong></li><li>Before deploying <strong>WordPress App</strong>, Do check everything is working fine by running below commands.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/0*TAwxxoGplFIc1z_j.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*_ibeipmIGYHFWSli.jpeg" /></figure><p>→ Hence Our cluster is ready🤩, Let’s deploy <strong><em>WordPress and MySQL</em></strong> and attach <strong><em>PVC with storage class from EBS.</em></strong></p><h4>Why should we attach PVC? 🙄</h4><ul><li>If we write anything in our pod and restart the pod, We will lose entire data, Hence we have to attach PVC to <strong><em>retrieve our data </em></strong>even if we <em>delete</em> or <em>restart </em>the pods.</li></ul><blockquote>Before launching WordPress &amp; MySQL pods we have to do a small set up. We have to launch EFS-PROVISIONER, STORAGE, RBAC pods!!🧵</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*vGRoYhtaq8vsUj3p.gif" /></figure><p><strong>EFS_PROVISIONER:-</strong></p><p>The <strong><em>EFS Provisioner </em></strong><em>is deployed as a Pod that has a container with access to an AWS EFS file system. </em>The container reads a ConfigMap containing the File system ID, Amazon Region of the EFS file system, and the name of the provisioner.</p><p><strong>STORAGE CLASS:-</strong></p><p>A <strong><em>StorageClass</em></strong> resource is defined as, whose<strong> <em>provisioner</em></strong> attribute determines which volume plugin is used for provisioning a <strong>PersistentVolume</strong> (PV). In this case, the StorageClass specifies the<strong><em> EFS Provisioner Pod</em></strong> as an external provisioner by referencing the value of the <em>provisioner.name</em> key in the ConfigMap.</p><p><strong>RBAC:-</strong></p><p><strong><em>Role-based access control (RBAC)</em></strong> is a method of regulating access to a computer or network resources based on the roles of individual users within your organization. <strong>RBAC</strong> authorization uses the rbac.authorization.k8s.io <a href="https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups">API group</a> to drive authorization decisions, allowing you to dynamically configure policies through the <strong>Kubernetes API</strong>.</p><p>→ <a href="https://github.com/rohitg00/EKS-Code/blob/master/kustomization.yml"><strong><em>kustomization.yml</em></strong></a> is the beautiful file, We have to just enter the name of the files in the same folder which we want to run and <strong><em>within just one-click</em></strong> entire deployment will be done.</p><p>→ Now run below command to deploy your WordPress App with MySQL as a backend to store the database.</p><pre>kubectl create -k .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/0*9bmUv9WtT3E2W_u1.jpeg" /></figure><p>→ Just by running a single command given above, Our WordPress App is launched. Also, Load balancers are launched to auto-scale the nodes which are launched using EKS. Run below command to check the running pods.</p><pre>kubectl get all</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AI2XP4F7726VfQzI.jpeg" /><figcaption>Everything is running fine….</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EUPmlkNKzBQn_mzx.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F96nhIULtuowhpNU.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YTf1Z77dGbLPZ2h9.jpeg" /></figure><h4>One step closer to launch our WordPress application</h4><ul><li>Login into our ec2 instances by WEBUI and run below command.</li></ul><pre>sudo su -<br>yum install amazon-efs-utils -y</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fGUZqSwDN-D7vneg.jpeg" /></figure><p>→ <strong>LoadBalancer</strong> gives a Public DNS so that the outside world can connect to our pods through the load balancer and isolate our pods. Based on the traffic &amp;load, It dynamically creates the nodes. I am Copying <strong>load balancer IP</strong> on my browser to get connected with <strong>WordPress.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ID-lu8Cs9eztYDoh.jpeg" /><figcaption>This will be the first page that will appear.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tvMOutpXrYAQZo69.jpeg" /><figcaption>Installation and account setup</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zIlIGg1mJPh2xBn2.jpeg" /><figcaption>Login page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BoOrvCzs84wsUN_S.jpeg" /><figcaption>Dashboard of WordPress</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tXvXlHFD3gZZHKBJ.jpeg" /><figcaption>The appearance of blog. (Final Output)</figcaption></figure><blockquote><strong>Congratulations Guys!!🎇</strong></blockquote><blockquote><strong>If you followed my steps correctly you must have got the same output as mine.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*99Gizt3Q4D9k6LeP.gif" /><figcaption>YOU!!</figcaption></figure><p><strong><em>Okay, Thanks for staying till the end! I hope you learned some new things. Keep Learning! 🤘</em></strong></p><p><strong><em>I hope this article helped. You can appreciate it by giving us applaud.</em></strong></p><p><strong><em>You can connect with me on </em></strong><a href="https://www.linkedin.com/in/vinujakhatode/"><strong><em>LinkedIn</em></strong></a><strong><em>.</em></strong></p><ul><li><a href="https://www.linkedin.com/in/vinujakhatode/">Vinuja Khatode - Hybrid Multi-Cloud Intern - LinuxWorld Informatics Pvt Ltd | LinkedIn</a></li><li><a href="https://github.com/vinujakhatode/Amazon-EKS-Project">vinujakhatode/Amazon-EKS-Project</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*jAe4PKopt_xsaMsl.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45fe862b479c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing the app using Docker, Jenkins,  and GitHub.]]></title>
            <link>https://medium.com/@vinujakhatode/testing-the-app-using-docker-jenkins-and-github-f983e9a1a5da?source=rss-4c1c69284406------2</link>
            <guid isPermaLink="false">https://medium.com/p/f983e9a1a5da</guid>
            <category><![CDATA[jenkins-pipeline]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[devops-training]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <dc:creator><![CDATA[Vinuja Khatode]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 20:51:31 GMT</pubDate>
            <atom:updated>2020-07-10T20:55:02.851Z</atom:updated>
            <content:encoded><![CDATA[<h3>Testing the app using Docker, Jenkins, and GitHub.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*qhNw99S0TcUJU2to" /></figure><p>In this article, we are going to do a simplistic integration of GitHub, Docker, and Jenkins to automate the testing process i.e., to check whether our app is working or not.🤘</p><p>Basically, we have to perform some jobs to automate this whole process. The jobs are listed as:</p><p><strong>Job1:</strong> <strong><em>Pull the Github repo</em></strong> automatically when some developers push the repository to Github.</p><p><strong>Job2:</strong> By looking at the code or program file, <strong><em>Jenkins should automatically start</em></strong> the respective language interpreter to deploy code ( eg. If code is of PHP, then Jenkins should start the container that has PHP already installed ).</p><p><strong>Job3: <em>Test your app</em></strong> if it is working or not. If the app is not working, then <strong><em>send an email</em></strong> to the developer with error messages.</p><p>We have to create one more extra job <strong><em>Job4 to monitor</em></strong>: If the container where the app is running, fails due to any reason then this job should automatically start the container again.</p><p>So, in order to begin with this task, you should know all the basics and setup of the Docker and Jenkins. You can refer to this <a href="https://github.com/vinujakhatode/AutomatedIntegration"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*El0d_JSlTdTg4TWjOZ7JoA.png" /></figure><p>Let&#39;s get started then,</p><p>First, we have to <strong>create one docker image </strong>that has Jenkins installed. So when we launch the container using that image, Jenkins should be automatically installed and get started. To build an image, we need<strong> Dockerfile</strong> that should look like this👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*BGGv5sJThXpYSl0B9g7exQ.png" /><figcaption>Dockerfile</figcaption></figure><p>After creating Dockerfile we have to <em>create an image</em> out of that file. To build the image, these commands are to be followed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*S7-yErhhyOgGkOEtj7VHVA.jpeg" /><figcaption>building image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*0o4_0Tg-fUboJop9x6jA6A.jpeg" /><figcaption>Image built</figcaption></figure><p>Here our image has been built. The next step is to launch a container using this image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*KJ3YXS91jJsRYmEfCNm05Q.jpeg" /><figcaption>Command to launch container</figcaption></figure><p>The next step is to launch a container using this image. Here we have launched the container, this will give <strong>one time password </strong>for login to the Jenkins.</p><p>Copy this password, get the IP of your BaseOS by doing<strong><em> ifconfig</em></strong>, and in browser, put your IP:9999. It will ask for the Initial password, paste that copied password there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*_uMzHGRgKefBhSw8YqYsMQ.png" /><figcaption>Jenkins password</figcaption></figure><p>Further, they will ask to download plugins, download all the recommended plugins, and in addition download these plugins<strong> SSH, Email Extension, GitHub Plugin</strong> for sure. They will ask to create <strong>First Admin User</strong>, there, provide the required info and account will be created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*1ErQZnRBw2YKz1WDEtKflQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*paEoTt24BzBUVyl189Dr9g.jpeg" /><figcaption>Jenkins setup</figcaption></figure><p><em>So moving towards jobs,</em></p><p>The<em> first job </em>is to <strong>pull the content from the GitHub</strong> repository. So first we have to make a repository in GitHub and upload all the files to that repository.</p><p>Here I’m using a web app for testing. So I’ve uploaded all the html files and folders to this repository. You can use any code, PHP, Python, etc to perform this task.</p><p>Also, you have to make one <strong>workspace </strong>for this task in the RedHat system, as I’ve made this task2 folder in my system. Where you can<strong> copy all the files</strong> from GitHub to this folder.</p><p>Now, go to the Jenkins, you also can access Jenkins from the windows browser. Click on ‘<em>New Item</em>’, Enter the name you want, configure that 1st Job as given.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-MRdoEXE0sEhktLGEQwUA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fiKsjsagA9fCzuE_e_4c5g.jpeg" /></figure><p>After the configuration, try building this Job, click on ‘<strong><em>Build now</em></strong>’. If configured properly, this job should be built successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*J_wvPKK4OaS4uUaP3tlLUA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*yRH6aAGreeNOvVeDuHti4A.jpeg" /></figure><p>Here, we have<strong> copied the content</strong> from the repository to our folder.</p><p>I’ve used this Repository here.👇</p><p><a href="https://github.com/vinujakhatode/Docker-and-Jenkins">vinujakhatode/Docker-and-Jenkins</a></p><p>On successful build, move on to the next job, this one is a bit tricky, here we have to <strong>check the type of code</strong> and perform actions accordingly. You can use any code, PHP, Python, etc. to perform this task. In the execute shell of this job, you can add more use cases like I’ve done here for html.</p><p>Configure this Job2 as given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*giIIzP-Cnk4rFOP8vJ7F9Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u0_xLxaNLyuAk96AVrfr3g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqN4AceRoM1ZHba1oqQQhg.jpeg" /></figure><p>After configuring job 2, build it. It should give output as below👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGWRE2hP06dU9XzmKJ5i9Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*Vc599kwrOZH55QyFvGk-Sw.jpeg" /><figcaption>Container launched</figcaption></figure><p>The next step is to check whether our <strong>site is working or not</strong>. For that, we use this curl command, which gives status as a number from that we can identify whether our site is working or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKGYUpNqtqydzHKTbYG7yA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJXcIyQBqaQnmxU0NdsPZQ.jpeg" /></figure><p>Also, in this job we have to configure the settings such as this job should send emails to the developer on failure. For that, we have to do a setup in the Jenkins settings → configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8skx6JoxhrwQjthrnMAow.png" /></figure><p>Put your email id here, to send the emails, and also enter the receiver&#39;s email. This will be enough to send an email. The next configuration is to be done in Job3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWwIVDm5Js4505svzPQ9Zg.jpeg" /></figure><p>Build the job check whether an email is being sent or not. If the emails are getting sent, the output of this Job will be something like this if the job found out that the website is not working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*95G3Zp5tyOhemPdZ30Q2uQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*Hoy5wSYt6uFH0onwNscEjA.jpeg" /></figure><p>You are almost done!🤘👌</p><p>The last job is to keep monitoring the whole setup, if the container fails or stops abruptly, this job will immediately launch a new container to carry on the process.</p><p>Configure this job as given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5MFlh3KlWRQAzP2LwYwjsg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UGnTET0Sba2F6ZakSayKA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2vL9FylK11w9Xww0_KLGcw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*4FI6VYe_ZJnQZUwEigSwUg.jpeg" /></figure><blockquote>Okay, now all the jobs are done.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/0*4uNrNZvCqKa8a8qx.png" /></figure><p>Try running the whole pipeline once again to make sure it is working properly.</p><p>We will check our website now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZdv1ulbEu8O2V6klcna9w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*urOR2xBMN2Nm0HEQYPSEKg.jpeg" /><figcaption>Site is working properly</figcaption></figure><p>Also, you can have a beautiful <strong>build view</strong> of the pipeline by downloading ‘<em>build pipeline</em>’ plugin from ‘<em>manage Jenkins</em>’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wPYriXwOK0w4jbwJPGnYg.jpeg" /></figure><p>Seeing all these jobs in green, we can say now that our is <strong><em>task is completed.</em></strong>😍</p><p>This task can be performed for a large scale using Kubernetes and all.</p><blockquote><strong><em>Congratulations, If you&#39;ve reached here! Kudos!✨</em></strong></blockquote><p>Also, <strong><em>Thanks for reading</em></strong>! You can connect me on LinkedIn and GitHub. <strong><em>Give CLAPS if you liked the content. 👏</em></strong></p><ul><li><a href="https://www.linkedin.com/in/vinujakhatode/">Vinuja Khatode - Hybrid Multi-Cloud Intern - LinuxWorld Informatics Pvt Ltd | LinkedIn</a></li><li><a href="https://github.com/vinujakhatode">vinujakhatode - Overview</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*7-ATYNjRzytYOHjb73XBjQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f983e9a1a5da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform, AWS, and Cloud Automation.‍]]></title>
            <link>https://medium.com/@vinujakhatode/terraform-aws-and-cloud-automation-b7d60cc6a3a2?source=rss-4c1c69284406------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7d60cc6a3a2</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloudops]]></category>
            <category><![CDATA[cloud-automation]]></category>
            <category><![CDATA[covid-19-india]]></category>
            <category><![CDATA[aws-cloud]]></category>
            <dc:creator><![CDATA[Vinuja Khatode]]></dc:creator>
            <pubDate>Sat, 13 Jun 2020 20:03:18 GMT</pubDate>
            <atom:updated>2020-06-13T20:03:18.552Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V71fNR8fDPTVGr2r.jpg" /></figure><p><em>Considering you are here, you must be having a slight idea about </em><strong><em>Terraform </em></strong><em>and its uses. So, if you want to build something good using it, you’ve landed on the right page.😉</em></p><p>It is always good, to begin with the basics.😬</p><p><a href="https://www.terraform.io/intro/index.html"><strong>Terraform </strong></a>is an intelligent tool for building, changing, and versioning infrastructure safely and efficiently. The simplest explanation of Terraform can be seen in the image below. We just have to write the <a href="https://stackify.com/what-is-infrastructure-as-code-how-it-works-best-practices-tutorials/"><strong>Infrastructure as Code (IAC)</strong></a> in the configuration files having <strong>.tf</strong> as an extension to build the whole infrastructure in just one click!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*ZQSRKEnVRPrXyhV4.jpg" /></figure><p><em>For this practical, we will need </em><strong><em>Terraform setup</em></strong><em>, </em><strong><em>Account on AWS Console</em></strong><em>, </em><strong><em>AWS CLI setup</em></strong><em>, and </em><strong><em>little curiosity to learn</em></strong><em> something new.😄 </em>Here we are launching an infrastructure to deploy websites.</p><blockquote><strong>Automating the Infrastructure setup using Terraform and AWS Cloud.</strong></blockquote><p>Here is the detailed problem statement:</p><ol><li>Create the <strong>key </strong>and <strong>security group </strong>which allows the port 80 and 22.</li></ol><p>2. Launch <strong>EC2 instance</strong>. In this EC2 instance use the key and security group created above.<br>3. Launch one <strong>Volume (EBS)</strong> and mount that volume into /var/www/html.<br>4. <strong>Copy </strong>the GitHub repo code that is uploaded by the developer into /var/www/html<br>5. Create an <strong>S3 bucket</strong>, and copy the images from GitHub repo into this s3 bucket and change the permission to public readable.<br>6. Create a <strong>Cloudfront using s3 bucket</strong>(which contains images) and use the Cloudfront URL to update in code in /var/www/html</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*12Elc3UTMZNvkRtf" /></figure><ul><li>The first thing to do is, <a href="https://s3.amazonaws.com/aws-cli/AWSCLI64PY3.msi">Download the <strong>AWS CLI</strong></a> and finish the installation. If done with this installation go the AWS Console, and under services go to the IAM and Add a new user. After adding the user, we will get access credentials, save that CSV file of credentials. Now in command prompt write the command as below: (put “<em>json</em>” in output format)*</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*QrNovjR2jRqfpLAVoBB6JA.jpeg" /><figcaption>configure profile</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*P9K811hVs3LiVCWVBCex4A.jpeg" /><figcaption>list profiles</figcaption></figure><ul><li><em>Done with the profile configuration!!</em></li><li>Now, Download the <a href="https://www.terraform.io/downloads.html"><strong>Terraform </strong></a>and extract it in a folder. <strong>Copy the path</strong> of the folder wherever you have extracted and add that path to your system’s <strong>environment variables.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*3QUuTMQ50NVGq7QX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*21y7ycykyZ2EikuM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/0*8Epq2MYpm3HEY1Pi" /></figure><p>That&#39;s all for the Terraform setup, for confirmation run this command in the command prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*6Y3NUvynwkm9SWzpGnLQWA.png" /></figure><p><em>Yesss, your Terraform setup is ready!!!</em></p><p><strong>Time to do some serious task now. 😁</strong></p><p><strong>Create a workspace</strong> for this practical. For reference see the above picture. There you can see the <strong>Task1 folder</strong>. That is the folder where we will be saving all the files and doing all the work!</p><p><em>Yeah I know that&#39;s not the serious work,😂 It starts now👇</em></p><ul><li>cd into your workspace, and <strong>start notepad</strong> by</li></ul><pre>notepad Task1.tf</pre><ul><li>Here Task1 is the name of the configuration file that we will be using to write code.</li><li>First, we have to tell Terraform that we are using <strong>AWS provider</strong>. The way to do this is:</li></ul><pre>provider &quot;aws&quot;{<br>   region=&quot;ap-south-1&quot;<br>   profile = &quot;VinujaKhatode&quot;<br>}</pre><ul><li>So, the next step is to <strong>generate keys </strong>and creating security groups using Terraform. For the same, the following code can be used:</li></ul><pre><strong>//Creating KEY</strong></pre><pre>resource &quot;tls_private_key&quot; &quot;tls_key&quot; {<br>   algorithm = &quot;RSA&quot;<br>}</pre><pre><strong>//Generating Key-Value Pair</strong></pre><pre>resource &quot;aws_key_pair&quot; &quot;generated_key&quot; {<br>   key_name = &quot;Task1keyvin&quot;<br>   public_key =&quot;${tls_private_key.tls_key.public_key_openssh}&quot;<br>   <br> depends_on = [<br>      tls_private_key.tls_key<br>   ]<br>}</pre><pre><strong>//Saving Private KEY PEM File</strong></pre><pre>resource &quot;local_file&quot; &quot;key-file&quot; {<br>   content  = &quot;${tls_private_key.tls_key.private_key_pem}&quot;<br>   filename = &quot;Task1keyvin.pem&quot;</pre><pre>   depends_on = [<br>      tls_private_key.tls_key,<br>      aws_key_pair.generated_key<br>  ]<br>}</pre><ul><li>The above code will generate the key named Task1keyvin.pem in the same folder.</li><li><strong><em>In order to execute and validate, you have to write code till launching the instance at once! So that you won’t have to face the error of key generation.</em></strong></li><li>Let&#39;s move forward to the<strong> Security group.</strong> Here we will allow SSH Port and HTTP Port and also the Localhost. Naming this Security group as Task1sec.</li></ul><pre><strong>//Creating Security Group</strong></pre><pre>resource &quot;aws_security_group&quot; &quot;Task1sec&quot; {<br>   name        = &quot;Task1sec&quot;<br>   description = &quot;Security Group for Task1 SSH and HTTPD&quot;</pre><pre><strong>//Adding Rules to Security Group</strong></pre><pre>   ingress {<br>      description = &quot;SSH Port&quot;<br>      from_port   = 22<br>      to_port     = 22<br>      protocol    = &quot;tcp&quot;<br>      cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>   }<br>   ingress {<br>      description = &quot;HTTP Port&quot;<br>      from_port   = 80<br>      to_port     = 80<br>      protocol    = &quot;tcp&quot;<br>      cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>   }<br>   ingress {<br>      description = &quot;Localhost&quot;<br>      from_port = 8080<br>      to_port = 8080<br>      protocol = &quot;tcp&quot;<br>      cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  egress {<br>     from_port   = 0<br>     to_port     = 0<br>     protocol    = &quot;-1&quot;<br>     cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  tags = {<br>      Name = &quot;Task1sec&quot;<br>  }<br>}</pre><ul><li>After configuring security groups, the next step is to <strong>launch the EC2 instance </strong>using the above-created security group and keys. We have to give <strong>ami-id and the instance type</strong> to launch that instance.</li></ul><pre><strong>// Creating instance with above created key and security group</strong></pre><pre>resource &quot;aws_instance&quot; &quot;Task1instance&quot; {<br>   ami = &quot;ami-005956c5f0f757d37&quot;<br>   instance_type = &quot;t2.micro&quot;<br>   key_name = &quot;${aws_key_pair.generated_key.key_name}&quot;<br>   security_groups = [&quot;${aws_security_group.Task1sec.name}&quot;]<br>   tags = {<br>      Name = &quot;Task1instance&quot; <br>   }<br>}</pre><ul><li>If we run this tf file now, this code will launch the instance named Task1instance with the private key Task1keyvin and Task1sec as a Security group.</li><li>So to run this code, <strong>Save</strong> the Task1.tf file and Open Command Prompt and give commands as follow:</li></ul><pre>terraform init</pre><ul><li>You always have to run this command <strong>every time you make changes </strong>in the configuration file. It installs all the<strong> required plugins</strong>. After the successful completion of the above command, the next command is:</li></ul><pre>terraform apply -auto-approve</pre><ul><li>This command will apply all the code written inside the tf file. The attribute <strong><em>-auto-approve</em></strong><em> </em>is optional. It basically gives approval for further procedures.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RwzuK7CLMJqgF1kHzarKoQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3wDFYz1yWX99J6LPq88W_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PXmJxfnW4P2U--oLRNrQg.png" /><figcaption>Instance, key, and security group created</figcaption></figure><p><em>Hope you are doing good till now! ✨</em></p><p><em>Okay, now moving towards the main steps.</em>😵</p><ul><li>In this step, we have to <strong>launch one volume </strong>and mount it to the var/www/html folder. Here we will create a volume of 1GB.</li></ul><pre><strong>// Creating new EBS Volume and attachin it to the above created instance</strong></pre><pre>resource &quot;aws_ebs_volume&quot; &quot;ebs1&quot; {<br>   availability_zone = aws_instance.Task1instance.availability_zone<br>   size = 1<br>   tags = {<br>      Name = &quot;Task1ebs&quot;<br>   }<br>}</pre><pre><strong>// To attach the Volume created</strong></pre><pre>resource &quot;aws_volume_attachment&quot; &quot;ebs_attach&quot; {<br>   device_name = &quot;/dev/sdh&quot;<br>   volume_id   = &quot;${aws_ebs_volume.ebs1.id}&quot;<br>   instance_id = &quot;${aws_instance.Task1instance.id}&quot;<br>   force_detach = true<br>}</pre><pre><strong>// Outputing the IP of the instance</strong></pre><pre>output &quot;myos_ip&quot; {<br>   value = aws_instance.Task1instance.public_ip<br>}</pre><p>Okay, so in order to use and mount the volume in the instance, we have to <strong>remote login to the ec2</strong> instance and then first <strong>partition, format, and then mount the volume.</strong></p><p>Also, as we are setting a web server, we need the <strong>httpd server running</strong> in the instance. And some <strong>HTML files </strong>to host/deploy on the server. So I have a website that I&#39;ve downloaded from the internet. I uploaded those files to Github and we will pull that files/code and copy it into the var/www/html folder.</p><p><strong><em>My GitHub Repo link:</em></strong></p><p><a href="https://github.com/vinujakhatode/Webserver-Terraform-AWS">vinujakhatode/Webserver-Terraform-AWS</a></p><p>Add the below code to your configuration file:</p><pre><strong>// In order to use Volume partition, format nad mounting is necessary</strong></pre><pre>resource &quot;null_resource&quot; &quot;nullremote&quot;  {</pre><pre> depends_on = [<br>     aws_volume_attachment.ebs_attach,<br>     aws_security_group.Task1sec,<br>     aws_key_pair.generated_key <br>  ]</pre><pre>connection {<br>    type = &quot;ssh&quot;<br>    user = &quot;ec2-user&quot;<br>    private_key = file(&quot;C:/Users/vinuja khatode/VWorkspace/Terraform/Task1/Task1keyvin.pem&quot;)<br>    host = aws_instance.Task1instance.public_ip<br>  }</pre><pre>provisioner &quot;remote-exec&quot; {<br>    inline = [<br>       &quot;sudo yum install httpd  php git -y&quot;,<br>       &quot;sudo service httpd start&quot;,<br>       &quot;sudo chkconfig httpd on&quot;, <br>       &quot;sudo mkfs.ext4  /dev/xvdh&quot;,<br>       &quot;sudo mount  /dev/xvdh  /var/www/html&quot;,<br>       &quot;sudo rm -rf /var/www/html/*&quot;,<br>       &quot;sudo git clone <a href="https://github.com/vinujakhatode/Webserver-Terraform-AWS.git">https://github.com/vinujakhatode/Webserver-Terraform-AWS.git</a>  /var/www/html/&quot;<br>    ]<br>  }<br>}</pre><ul><li>We used a “<strong>connection</strong>” block to remote log in and <strong>provisioner </strong>block to execute commands inside the instance. The above code will make <strong>partitions, format, and mount the volume </strong>to html folder and will also clone the GitHub repository there. It will also install <strong>httpd</strong>, <strong>PHP, </strong>and <strong>git </strong>services.</li><li><strong><em>Here is a fact about git command, it only clones into an empty folder, so make our operation successful we are forcefully removing the contents of the HTML folder.</em></strong></li></ul><blockquote>Till now, you can launch an instance with web hosting environment and volume attached to it in just one click!✨</blockquote><ul><li><strong>So the next step is to create an S3 Bucket:</strong></li></ul><pre><strong>// Creating S3 bucket</strong></pre><pre>resource &quot;aws_s3_bucket&quot; &quot;task1bucketvinuja00vinuja00&quot; {<br>  bucket = &quot;task1bucketvinuja00vinuja00&quot;<br>  acl    = &quot;private&quot;<br>  tags = {<br>  Name = &quot;task1bucketvinuja00vinuja00&quot;<br> }<br>}</pre><pre><strong>// Allow Public Access</strong></pre><pre>resource &quot;aws_s3_bucket_public_access_block&quot; &quot;S3PublicAccess&quot; {<br>  bucket = &quot;${aws_s3_bucket.task1bucketvinuja00vinuja00.id}&quot;<br>  block_public_acls = true<br>  block_public_policy = true<br>  restrict_public_buckets = true<br>}</pre><p>This script will create the S3 bucket named <strong>task1bucketvinuja00vinuja00. </strong>Make sure whatever name you choose must be in <strong>small letters and unique</strong> i.e there should not exist any other bucket with the same name. Here we are also allowing public access.</p><p>Now we have to fetch the<strong> images from S3 using CloudFront.</strong> For that, we first have to put images into the S3.</p><pre><strong>// Uploading files to S3 bucket</strong></pre><pre>resource &quot;aws_s3_bucket_object&quot; &quot;bucketObject&quot; {<br>   for_each = fileset(&quot;C:/Users/vinuja khatode/Desktop/girly/assets&quot;, &quot;**/*.jpg&quot;)<br>   bucket = &quot;${aws_s3_bucket.task1bucketvinuja00vinuja00.bucket}&quot;<br>   key = each.value<br>   source = &quot;C:/Users/vinuja khatode/Desktop/girly/assets/${each.value}&quot;<br>   content_type = &quot;image/jpg&quot;<br>}</pre><p>The images have been uploaded to the S3 bucket.</p><pre><strong>//Creating Cloudfront to access images from S3</strong></pre><pre>locals {<br> s3_origin_id = &quot;S3Origin&quot;<br>}</pre><pre><strong>// Creating Origin Access Identity for CloudFront</strong></pre><pre>resource &quot;aws_cloudfront_origin_access_identity&quot; &quot;origin_access_identity&quot; {<br> comment = &quot;task1bucketvinuja00vinuja00&quot;<br>}</pre><pre>resource &quot;aws_cloudfront_distribution&quot; &quot;Task1CF&quot; {</pre><pre>origin {<br> domain_name = &quot;${aws_s3_bucket.task1bucketvinuja00vinuja00.bucket_regional_domain_name}&quot;<br> origin_id = &quot;${local.s3_origin_id}&quot;<br> s3_origin_config {<br>  origin_access_identity = &quot;${aws_cloudfront_origin_access_identity.origin_access_identity.cloudfront_access_identity_path}&quot;<br> }<br>}</pre><pre>enabled = true<br> is_ipv6_enabled = true<br> comment = &quot;accessforTask1&quot;<br> default_cache_behavior {<br>  allowed_methods = [&quot;DELETE&quot;, &quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;, &quot;PATCH&quot;, &quot;POST&quot;, &quot;PUT&quot;]<br>  cached_methods = [&quot;GET&quot;, &quot;HEAD&quot;]<br>  target_origin_id = &quot;${local.s3_origin_id}&quot;<br>  forwarded_values {<br>   query_string = false<br>   cookies {<br>    forward = &quot;none&quot;<br>   }<br>  }<br>  viewer_protocol_policy = &quot;allow-all&quot;<br>  min_ttl = 0<br>  default_ttl = 3600<br>  max_ttl = 86400<br> }<br><strong>// Cache behavior with precedence 0</strong></pre><pre>ordered_cache_behavior {<br> path_pattern = &quot;/content/immutable/*&quot;<br> allowed_methods = [&quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;]<br> cached_methods = [&quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;]<br> target_origin_id = &quot;${local.s3_origin_id}&quot;<br> forwarded_values {<br>  query_string = false<br>  headers = [&quot;Origin&quot;]<br>  cookies {<br>   forward = &quot;none&quot;<br>  }<br> }</pre><pre>min_ttl = 0<br> default_ttl = 86400<br> max_ttl = 31536000<br> compress = true<br> viewer_protocol_policy = &quot;redirect-to-https&quot;<br>}<br><strong>// Cache behavior with precedence 1</strong></pre><pre>ordered_cache_behavior {<br> path_pattern = &quot;/content/*&quot;<br> allowed_methods = [&quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;]<br> cached_methods = [&quot;GET&quot;, &quot;HEAD&quot;]<br> target_origin_id = &quot;${local.s3_origin_id}&quot;<br> forwarded_values {<br>  query_string = false<br>  cookies {<br>   forward = &quot;none&quot;<br>  }<br> }<br>min_ttl = 0<br>default_ttl = 3600<br>max_ttl = 86400<br>compress = true<br>viewer_protocol_policy = &quot;redirect-to-https&quot;<br>}<br>price_class = &quot;PriceClass_200&quot;<br>restrictions {<br>geo_restriction {<br>restriction_type = &quot;whitelist&quot;<br>locations = [&quot;IN&quot;]<br>}<br>}<br>tags = {<br>Name=&quot;Task1CFDistribution&quot;<br>Environment = &quot;production&quot;<br>}<br>viewer_certificate {<br>cloudfront_default_certificate = true<br>}<br>retain_on_delete = true</pre><pre>depends_on=[<br> aws_s3_bucket.task1bucketvinuja00vinuja00<br>]<br>}</pre><p>This script will create a <strong>CloudFront </strong>distribution using an S3 bucket. In this bucket, we have stored all of the <strong>assets </strong>of our site like images, icons, etc. This CloudFront distribution will provide us one URL. By using this URL, we can access the objects inside the bucket.</p><p>Here we require that whenever the Infrastructure is destroyed, the CloudFront distribution should not get destroyed. Because, if we create new CloudFront distribution each time, we will be required to change the assets URLs every time. To overcome this problem we will set <strong><em>retain_on_delete </em>to<em> true</em></strong><em>. This will disable</em> the distribution instead of deleting it when destroying the resource through Terraform.</p><p><strong><em>WOAH! Too much huh? Just a little more to go!🤘</em></strong></p><ul><li>Creating Bucket Policy for CloudFront.</li></ul><pre>// AWS Bucket Policy for CloudFront<br>data &quot;aws_iam_policy_document&quot; &quot;s3_policy&quot; {<br>statement {<br>actions   = [&quot;s3:GetObject&quot;]<br>resources = [&quot;${aws_s3_bucket.task1bucketvinuja00vinuja00.arn}/*&quot;]<br>principals {<br>type        = &quot;AWS&quot;<br>identifiers = [&quot;${aws_cloudfront_origin_access_identity.origin_access_identity.iam_arn}&quot;]<br>}<br>}<br>statement {<br>actions   = [&quot;s3:ListBucket&quot;]<br>resources = [&quot;${aws_s3_bucket.task1bucketvinuja00vinuja00.arn}&quot;]<br>principals {<br>type        = &quot;AWS&quot;<br>identifiers = [&quot;${aws_cloudfront_origin_access_identity.origin_access_identity.iam_arn}&quot;]<br>}<br>}<br>}<br>resource &quot;aws_s3_bucket_policy&quot; &quot;s3BucketPolicy&quot; {<br>bucket = &quot;${aws_s3_bucket.task1bucketvinuja00vinuja00.id}&quot;<br>policy = &quot;${data.aws_iam_policy_document.s3_policy.json}&quot;<br>}</pre><p>This will create a Bucket Policy for the CloudFront.</p><h4><strong><em>So, FINALLY, Our code is ready to launch the whole infrastructure in just one click!!!</em></strong></h4><p>Why wait then?!🤔</p><h4>Perform <strong><em>terraform init</em></strong> and <strong><em>terraform apply!!</em></strong></h4><p><em>Let us see the outputs now:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*RIpwJky8zR7NZOAzTete2Q.jpeg" /><figcaption>terraform init</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WCbORfv1HoLR2YKhAzjNuA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*5fzoSv7csDRedCwKoWJz3Q.jpeg" /><figcaption>terraform apply</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4k7l54moRp2ebwkCvY9Njw.png" /><figcaption>S3 bucket</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2feDnBoGNOxHAHpWHkh-w.png" /><figcaption>CloudFront</figcaption></figure><p><strong><em>Now, Checking the IP whether the website is hosted or not!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*7Ek7tuCG0nqxqqoi6jjsFg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEV6v92KNbW4YAnG7HNAFg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*RieBFQH3K--DEzzm_pK_TA.jpeg" /><figcaption>This website is hosted properly and working right!</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*ubjY4wCHTENrU98wXmUuPw.jpeg" /><figcaption>Inside html folder of instance</figcaption></figure><blockquote><strong>Congratulations if your output is similar as that of above!🎇</strong></blockquote><p>This automation has enough power that can launch the whole infrastructure in a single click.</p><p>Also destroying this infrastructure is pretty easy. A single command and our whole infrastructure will be destroyed permanently.</p><pre>terraform destroy</pre><p>If you have hands-on Jenkins and Ansible you can use that to pull the code from Github and for many more purposes from the above code. That would be</p><blockquote>DevOps + Cloud = CloudOps</blockquote><p>But for now,👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*3PNdD6w9R-BsqCCM.jpg" /></figure><p><em>In this whole process, you may get many errors, but try to solve them, try to debug on your own with patience. In that way, you learn more. </em><strong><em>Patience is the key.</em></strong><em> You just have to keep calm and keep doing the work.✨</em></p><p>Okay, Thanks for staying till the end! I hope you learned some new things. <strong><em>Keep Learning! 🤘</em></strong></p><p><em>I hope this article helped. You can appreciate it by giving us applaud.</em></p><p><em>You can connect with me on </em><a href="https://www.linkedin.com/in/vinujakhatode/"><em>LinkedIn</em></a><em>.</em></p><p><a href="https://www.linkedin.com/in/vinujakhatode/">Vinuja Khatode - Hybrid Multi-Cloud Intern - LinuxWorld Informatics Pvt Ltd | LinkedIn</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*HmwhVzvZP8s-Am6T.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7d60cc6a3a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automation in Machine Learning and Increasing the accuracy of models using MLOps.]]></title>
            <link>https://medium.com/@vinujakhatode/automation-in-machine-learning-and-increasing-the-accuracy-of-models-using-mlops-9fe18313687?source=rss-4c1c69284406------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fe18313687</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[covid-19-testing]]></category>
            <category><![CDATA[x-rays]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Vinuja Khatode]]></dc:creator>
            <pubDate>Sat, 30 May 2020 11:54:16 GMT</pubDate>
            <atom:updated>2020-07-06T16:53:05.171Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automation of Machine Learning and Increasing the accuracy of models using MLOps.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*DTq5dU2QBsBjNxU3ni83YA.png" /><figcaption>A question to think about!</figcaption></figure><p>It was pointed out that nearly 90% of ML models cooked up by data scientists never actually make it into production, at <a href="https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/">Transform 2019</a> conference. Also, the deployed Machine Learning and AI models fail in real-world and most of the ML and DL models are not even deployed, due to the maintenance and inability to increase the accuracy of the model when deployed, the need for manual tweaking of the code, the manual changes to be made in the hyperparameters, etc., critical manual work is required which extremely time-consuming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*Kj8UosiWJbdhENuBz_qCUw.png" /><figcaption>The solution is here!</figcaption></figure><p>Yes, MLOps!! This integration of Machine Learning with the DevOps i.e., the Automation is the solution for such problems. It will help in increasing the accuracy of the model, handling the tweaking of the code by itself, so no human efforts are needed after setting up the system just once. And then,</p><p><em>Docker containers and Jenkins will do the Job!</em></p><p>If you are also going through the same problem and in search of the solution, stay here with this article till the end and you’ll find the solution to your problem.</p><p>So, here I will be talking about the project I did on the <strong>Integration of ML and DL models with Jenkins and Docker.</strong></p><p>This project is based on the integration of Machine Learning with DevOps. Also, before starting with the Automation part, make sure you have the basic knowledge of<em> </em><a href="https://github.com/vinujakhatode/OWnCloud-Docker-Compose-IIEC_RISE-Project"><strong><em>Docker</em></strong></a><strong><em>,</em></strong><em> </em><a href="https://github.com/vinujakhatode/AutomatedIntegration"><strong><em>Jenkins</em></strong></a><em>,</em><strong> RedHat, GitHub, and GitBash.</strong></p><p><strong><em>The problem statement of the project:</em></strong></p><blockquote>1. Create container image that has ML and DL environment using <strong>Dockerfile.</strong></blockquote><blockquote>2. Create a <strong>job chain</strong> of job1, job2, job3, job4 and job5 using build pipeline plugin in <strong>Jenkins.</strong></blockquote><blockquote>3. <strong>Job1 </strong>:<strong> Pull the Github repo</strong> automatically when some developers push repo to Github.</blockquote><blockquote>4. <strong>Job2 </strong>: Launch the respective<strong> docker container</strong>, by looking at the code or program file. ( eg. If code uses CNN, then Jenkins should start the container that satisfies the cnn processing).</blockquote><blockquote>5. <strong>Job3 </strong>: Train your model and <strong>predict accuracy.</strong></blockquote><blockquote>6. <strong>Job4 </strong>: If accuracy is less than 80% , then <strong>tweak the machine learning model </strong>architecture.</blockquote><blockquote>7. <strong>Job5</strong>: <strong>Retrain </strong>the model or <strong>notify </strong>that the best model is being created.</blockquote><blockquote>8. Create One extra job <strong>job6 </strong>tomonitor: If container where app is running, fails due to any reason then this job should automatically start the container again from where the last trained model left.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*Vj5FpRXjcLCQ6D9V.jpg" /></figure><p>First, let&#39;s talk about the model that is being used here.</p><p>Here, I’m using the <strong>CNN model</strong> which <strong>predicts the COVID-19 from Chest X-ray images.</strong> With an available small dataset of Chest X-ray images of COVID-19 patients and other normal Chest X-ray images, this model predicts whether a patient is COVID-19 positive or negative. One of the reasons for choosing this model is, it is found that the detection of Coronavirus can be done using the X-rays way before the patients start to show that symptoms. If more dataset is provided it can be used on large-scale since we are automating the model!</p><p>Here is an interesting question!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-kEiHS3kSh9Yzdv6BO1dQ.png" /></figure><p>Yeah! You would say, <strong>“I’m not Radiologist, how would I know!”</strong>. You’re right!</p><p><strong>But, with Artificial Intelligence and Machine Learning, you can…!!</strong></p><p>The Anwer is 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2cbZ9lmesCq03Y5hnpPweA.png" /></figure><p>How???? <strong><em>Using Classification!</em></strong> But, that is a whole lot of another topic! We’ll cover this in another blog.</p><p>You can find the code for the above model and dataset in my <a href="https://github.com/vinujakhatode/ML-Docker-Jenkins"><strong><em>Github repo.</em></strong></a></p><p>Coming back to our automation task, It requires a <strong>good amount of RAM</strong>. So, make sure whichever model you are using, it has <strong>fewer parameters.</strong></p><p>Start with writing the Python code and save it in .py format. Save your Dataset and these codes in one folder and <strong>Push </strong>all the files from that folder to the Github repo. <br>To do so, you’ll have to first create a Github repo from Github or if you have a grasp on GitBash, go with that! After creating the repo, clone it to your system in the same folder where you stored those files and follow these commands.</p><pre>git clone &lt;your repo link&gt;<br>git add .<br>git commit -m &quot;First commit&quot;<br>git push<br>git status</pre><p>Check the git status for confirmation.</p><p>Before starting with the Job1, start creating the Dockerfile as it takes time to build the Dockerfile. So that we can utilize that time configuring the Job1 instead of just sitting idle.</p><p>For that, go to your RedHat8 or Linux system, start the docker and Jenkins services.</p><pre>systemctl start docker<br>systemctl start jenkins<br>systemctl start httpd</pre><p>After being in RHEL8, create a separate workspace for this project where you can build Dockerfile. Make sure you use the file name as <strong>Dockerfile</strong>.</p><pre>cd /yourworkspace<br>gedit Dockerfile</pre><p>Enter the following commands in Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*7CExECWjASDlV5s8cS_FJw.jpeg" /><figcaption>Dockerfile</figcaption></figure><p>After you complete editing Dockerfile, run:<br>The syntax is: <em>docker build -t name:tag location</em></p><pre>docker build -t cnnos:v1 .</pre><p>This process will take time and patience too! Keep calm!</p><ul><li>We can now start with <strong>Job1:</strong><br>To automate the pull from Github first, you’ll have to configure the post-commit file, so that whenever a developer pushes the code to Github, Jenkins Job1 will pull that code.</li></ul><p>cd into the folder in windows where you have cloned Github repository. To configure the post-commit file,</p><pre>cd .git/hooks<br>notepad post-commit</pre><p>A notepad file will open, write these commands there.</p><pre>#!/bin/bash</pre><pre>echo “Welcome to the Automation”<br>git push -u origin master</pre><pre>curl --user &quot;admin:redhat&quot;  <a href="http://192.168.31.205:8080//job/MLOpsJob1/build?token=redhat">http://192.168.1.208:8080/job/MLJob1/build?token=redhat</a>8</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*o69MNaIoGBAeTdqwZBASRA.jpeg" /></figure><p>Save this file and check the git status if there is anything remaining to be pushed, Push it using the commands I showed earlier.</p><p>As you’ve started Jenkins in the Redhat, fetch your IP using <strong><em>ifconfig enp0s3</em></strong> and type IP:8080 in your browser to start the GUI of Jenkins. Log in with the username and password. To start building a new job, Click on the new item, select free-project, and click on okay. Configure this job referring to the photos below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2EPCBoLyQ0dE1ouDFONGaQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LXBv6XT2ao5vRlfSDVi7cA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20IKvSGE0MY_ZEtfO4-v1w.jpeg" /></figure><pre>sudo cp -rv * /root/mlops</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*LZ0FZFHZkjRL2zaBnFfz4w.png" /><figcaption>Output of Job1</figcaption></figure><p><strong>mlops</strong> is the folder that I created in the root of the Redhat system as the workspace of this project. This command will copy the files from Github to this folder. After completing the process, click on build now and test whether it is working or not. All the files from Github are copied here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*VqUIy1wqrA7jna79jfCfjA.jpeg" /></figure><p><strong>Tadaaa! Your Job1 is ready!</strong></p><p>Now, go to the terminal where your docker image was built, See, if it’s completed, if completed check it by running. So that we can move further towards Job2.</p><pre>docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*1R2BxU-FLn0LC6ugwY7FNg.jpeg" /></figure><p>If your image appears there, time to go for the next step.</p><ul><li><strong><em>Job2:</em></strong></li></ul><p>Go to Jenkins and start with a new job, name it Job2. Here, we have to launch the container of the respective environment. For example, if the model is of ML, launch the ML container and if the model is of DL launch DL container.</p><p>Here, I’ve installed all the libraries that are required for both ML and DL in a single Docker image. This job will be triggered after the Job1 is stable.</p><p>Configure the Job2 as given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S6vpqOTQRrqbDc_6HXhFwQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W1ihwDUlc7t99-YPqerbLA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vgM20IUp7eJHUOb3BUFysQ.jpeg" /></figure><p>Done with configuration? Check output in console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4uijoq0qvVM03qr2Xhf0w.png" /><figcaption>Output of Job2</figcaption></figure><p>Check whether the container is launched and running or not in the Redhat. For that, use</p><pre>docker ps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*T97U2Ro6VKme_-94NfOF5w.jpeg" /></figure><p>If it is running, well and good, let’s get to Job3. Also, here we have to trigger Job6 if this build is unstable, i.e., if the containers are not running Job6 will restart or launch the container.</p><ul><li><strong>Job3:</strong></li></ul><p>Again, start a new Jenkins job, name it Job3. Here, we have to train our model. This is a very simple job where you have to give a single command in the execute shell. This job will be triggered if Job2 is stable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyhzI2OFeKD-8A7Uq8m66Q.jpeg" /></figure><pre>sudo docker exec cnnos python3 mlops/Covid_Detection_using_Xrays.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fwl1gO3kgxdGWD1IAz9RrA.jpeg" /></figure><p>This job will start the training of the model and output will be shown in the console output of the job. This will be the output of your Job3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DH2v5bb4aOeGdTzEu2Aisg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oiDp4TVhU4JeXO-st3-syg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4oR4ciiJ0VYGbpcMHWXqzg.jpeg" /><figcaption>Output of Job3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYR8oFDyAcXvLaaLDeHiAA.jpeg" /><figcaption>Execution of job3</figcaption></figure><p><strong>Done with job3!</strong></p><p><em>Till now, you can train your model in just one click!✨</em></p><ul><li><strong>Job4:</strong></li></ul><p>Here, we check the accuracy given by the model in Job3. If the accuracy is equal to or greater than 98% it will send a <strong>success email</strong> to the developer. And if the accuracy is less than 98% it will trigger Job5 to retain the model with some extra hyperparameters that are passed using <strong>Execute shell</strong>. Also, it will notify the developer that we are tweaking the model because of the less accuracy! Configuration of Job4 is as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*BJQQya9Z3roO-EkJXcUqcQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dq9139FfjkDx-cNqbbeXag.jpeg" /></figure><pre>read=$( sudo docker exec cnn python3 mlops/check_accuracy.py )</pre><pre>echo $read</pre><pre>if (( $read &gt;= 98))</pre><pre>then</pre><pre>echo &quot;Achieved accuracy&quot;</pre><pre>sudo python3 /root/mlops/success_mail.py</pre><pre>exit 0</pre><pre>else</pre><pre>sudo python3 /root/mlops/fail_email.py</pre><pre>sudo sed -i &#39;/^epoch=.*/a epoch=epoch+5&#39; /root/mlops/Covid_Detection_using_Xrays.py</pre><pre>sudo sed -i &quot;24i model.add(Conv2D( filters=64, kernel_size=2, padding=&#39;same&#39;, activation=&#39;relu&#39;, input_shape=(224,224,3)))&quot; /root/mlops/Covid_Detection_using_Xrays.py</pre><pre>sudo sed -i &quot;25i model.add(MaxPooling2D( pool_size=2 ))&quot; /root/mlops/Covid_Detection_using_Xrays.py</pre><pre>fi</pre><p>It will read the accuracy from the check_accuracy.py file and compare it with the given value. If it is less than 98%, It will add a Convolution layer at line number 24 of the code and also a pooling layer at line number 25. It will also increase the number of epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ZBhefnBcFwTTUo-58JTjA.jpeg" /><figcaption>Output of Job4</figcaption></figure><p>Here, It will go the Job5 as accuracy is less than the required accuracy. The following are the examples of the emails sent by this job. I’ve used smtp Gmail server in the python script for sending the mails. If you want to mail using another way you can try <a href="https://medium.com/@ashishkumar_99/sending-mail-using-jenkins-plugin-linux-based-ef05ae141194">this one.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LMgJhAvB2XrZJ8tDz7iMOg.jpeg" /><figcaption>Mails sent by Job4</figcaption></figure><p><strong><em>Keep it up, You’re going good!!!👌</em></strong></p><ul><li><strong>As Job5 is triggered let’s get to the job5:</strong></li></ul><p>Job5 will retrain the model as new layers and hyperparameters are included in the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*nEWGuB-SRsWv0mBTYwrzYg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*riBzb7NyA7irqru7gsgtaQ.jpeg" /></figure><p>It will again trigger Job4 to check the accuracy. This is a kind of loop, isn’t it?</p><p>The Output of the Job5 will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PcorxT5bbJQd3gsdghgJsA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QPCW4wVum7_eFkprxvuMcg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2ljclNsRfAO1f5PduCGnA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEI9cD4Zf07OXpEYmkZBqA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKDi0AyDm3gut3DKXXwznQ.jpeg" /><figcaption>Output of Job5</figcaption></figure><p>Here you can see the <strong>additional layer and the number of epochs is also increased</strong>. It will go to the Job4 now. We can say that the configuration of Job5 is done.</p><p><strong><em>Congratulations that you’ve come this far!🎇</em></strong></p><p>Just one small thing is remaining and that is👇</p><ul><li><strong>Job6:</strong></li></ul><p>Job 6 is the job which there to monitor Job2 or you can say the whole process! Configure Job6 as given:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*cLc_fJCL6KQepfLch_wPUw.jpeg" /></figure><p>Poll SCM will monitor the job every minute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*3gq99y8OySRZABKTtk-HJw.jpeg" /></figure><p>If the container was not present and is launched by this Job then it will trigger Job3 for further processing. The 2 types of Outputs would be:</p><ol><li>If the Container is launched by this Job</li><li>The container is already present and running.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kfoBjwGjf-aHh99kP6NDOg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdOev09g18ohp4pNmQE7xA.jpeg" /><figcaption>Output of Job6</figcaption></figure><p>Here are some build pipeline views, in order to get this view you just have to download the <strong>Buil Pipeline Plugin</strong> of Jenkins from Manage Plugins. This a great visualization plugin provided by Jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R5ibJZAw2t3RDx2Kw2cblw.jpeg" /><figcaption>Build Pipeline view of whole process</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Orhs6YSTdOBcUugy_vddYg.jpeg" /></figure><p>Here we have automated the whole hectic manual process into just almost one click. Push the code to the Github and there you go! Your model will be giving 98–99% accuracy by automation. Also, you can add one more job to push the updated files to Github.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*-KbbQRBLP09gu3p_2NRksg.jpeg" /></figure><p>In this whole process, you may get many errors, but try to solve them, try to debug on your own with patience. In that way, you learn more. <strong>Patience is the key here. </strong>I also got many errors starting from the usage of memory to the wrong paths.😂 You just have to keep calm and keep doing the work.✨</p><p>Okay, Thanks for staying till the end! I hope you learned some new things. <strong><em>Keep Learning! 🤘</em></strong></p><p>I hope this article helped. You can appreciate it by giving us an applaud.</p><p><em>You can connect with me on </em><a href="https://www.linkedin.com/in/vinujakhatode/"><em>LinkedIn</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*vkSdDObgejEHmvIi.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fe18313687" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Market Basket Analysis using Association Rule-Mining]]></title>
            <link>https://medium.com/@vinujakhatode/market-basket-analysis-using-association-rule-mining-f9e479d966e9?source=rss-4c1c69284406------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9e479d966e9</guid>
            <category><![CDATA[apriori-algorithm]]></category>
            <category><![CDATA[association-rule-mining]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[market-basket-analysis]]></category>
            <dc:creator><![CDATA[Vinuja Khatode]]></dc:creator>
            <pubDate>Sun, 30 Jun 2019 09:04:06 GMT</pubDate>
            <atom:updated>2019-06-30T09:04:06.520Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I got an opportunity to work on a project based on Market Basket Analysis, and here I am sharing my experience.</p><p><strong>What is Market basket analysis?</strong></p><p>Market Basket Analysis is one of the key techniques used by large retailers to uncover associations between items. It works by looking for combinations of items that occur together frequently in transactions. To put it another way, it allows retailers to identify relationships between the items that people buy.</p><p><strong>What are association rules?</strong></p><p>Association Rules are widely used to analyze retail basket or transaction data, and are intended to identify strong rules discovered in transaction data using measures of interestedness, based on the concept of strong rules. The outcome of this type of technique is, in simple terms, a set of <strong>rules</strong> that can be understood as <strong>“if this, then that”.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/0*ZJ5RG0nAd4cxH4vJ.jpg" /></figure><p><strong>An example of Association Rules</strong></p><ul><li>Assume there are 100 customers</li><li>10 of them bought milk, 8 bought butter and 6 bought both of them.</li><li>bought milk =&gt; bought butter</li><li>support = P(Milk &amp; Butter) = 6/100 = 0.06</li><li>confidence = support/P(Butter) = 0.06/0.08 = 0.75</li><li>lift = confidence/P(Milk) = 0.75/0.10 = 7.5</li></ul><p>This example is extremely small. In practice, a rule needs the support of several hundred transactions, before it can be considered statistically significant, and datasets often contain thousands or millions of transactions.</p><p>The <strong>Apriori algorithm</strong> generates association rules for a given data set. An association rule implies that if an item A occurs, then item B also occurs with a certain probability. Let’s see an example,</p><pre>order 1: apple, egg, milk  <br>order 2: carrot, milk  <br>order 3: apple, egg, carrot<br>order 4: apple, egg<br>order 5: apple, carrot</pre><p>There are 4 key metrics to consider when evaluating association rules:</p><p><strong>1.Support :</strong><br>This is the percentage of orders that contains the item set. In the example above, there are 5 orders in total and {apple,egg} occurs in 3 of them, so:</p><pre>support{apple,egg} = 3/5 or 60%</pre><p>The minimum support threshold required by apriori can be set based on knowledge of your domain.</p><p><strong>2.Confidence :</strong><br>Given two items, A and B, confidence measures the percentage of times that item B is purchased, given that item A was purchased. This is expressed as:</p><pre>confidence{A-&gt;B} = support{A,B} / support{A}</pre><p>Confidence values range from 0 to 1, where 0 indicates that B is never purchased when A is purchased, and 1 indicates that B is always purchased whenever A is purchased. Note that the confidence measure is directional. This means that we can also compute the percentage of times that item A is purchased, given that item B was purchased:</p><pre>confidence{B-&gt;A} = support{A,B} / support{B}</pre><p>In our example, the percentage of times that egg is purchased, given that apple was purchased is:</p><pre>confidence{apple-&gt;egg} = support{apple,egg} / support{apple}<br>                                    = (3/5) / (4/5)<br>                                    = 0.75 or 75%</pre><p>A confidence value of 0.75 implies that out of all orders that contain apple, 75% of them also contain egg. Now, we look at the confidence measure in the opposite direction (ie: egg-&gt;apple):</p><pre>confidence{egg-&gt;apple} = support{apple,egg} / support{egg}<br>                                    = (3/5) / (3/5)<br>                                    = 1 or 100%</pre><p><strong>3.Lift :</strong></p><p>Unlike the confidence metric whose value may vary depending on direction (eg: confidence{A-&gt;B} may be different from confidence{B-&gt;A}), lift has no direction. This means that the lift{A,B} is always equal to the lift{B,A}:</p><pre>lift{A,B} = lift{B,A} = support{A,B} / (support{A} * support{B})</pre><p>In our example, we compute lift as follows:</p><pre>lift{apple,egg} = lift{egg,apple} = support{apple,egg} / (support{apple} * support{egg})<br>                  = (3/5) / (4/5 * 3/5) <br>                  = 1.25</pre><p>In summary, lift can take the following values:</p><ul><li>Lift = 1; implies no relationship between A and B (i.e., A and B occur together only by chance)</li><li>Lift &gt; 1; implies that there is a positive relationship between Aand B(i.e., A and B occur together more often than random)</li><li>Lift &lt; 1; implies that there is a negative relationship between A and B(i.e., A and B occur together less often than random)</li></ul><p>In our example, apple and egg occur together 1.25 times <em>more</em> than random, so we conclude that there exists a positive relationship between them</p><p><strong>4.Conviction :</strong></p><p>The <em>conviction</em> of a rule is defined as</p><pre>conv{apple,egg} = ( 1 - support{egg}) / <br>                  (1 - confidence{apple -&gt;egg})</pre><p>It is interpreted as the ratio of the expected frequency that A occurs without B (that is to say, the frequency that the rule makes an incorrect prediction) if A and B were independent divided by the observed frequency of incorrect predictions.</p><p><strong>Input Dataset :</strong></p><p><a href="https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/groceries.csv">https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/groceries.csv</a></p><p><strong>Let’s look at the code of market basket analysis using Python:</strong></p><pre><strong>CODE :</strong></pre><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>import pandas as pd</pre><pre># Data Preprocessing<br>dataset = pd.read_csv(‘groceries.csv’)</pre><pre>transactions = []</pre><pre>for i in range(0, 9835):<br> transactions.append([str(dataset.values[i,j]) for j in range(0, 32)])</pre><pre># Training Apriori on the dataset<br>from apyori import apriori<br>rules = apriori(transactions, min_support = 0.007, min_confidence = 0.5, min_lift = 3, min_length = 2)</pre><pre># Visualising the results<br>results = list(rules)</pre><pre>dataset.head()<br>dataset.shape</pre><pre>for a in results:<br>    print(&quot;------------------------------------------------------&quot;)<br>    print(a)</pre><pre><strong>OUTPUT :<br>        </strong>RelationRecord(items=frozenset({&#39;citrus fruit&#39;, &#39;other vegetables&#39;, &#39;root vegetables&#39;}), support=0.010371123538383325, ordered_statistics=[OrderedStatistic(items_base=frozenset({&#39;citrus fruit&#39;, &#39;root vegetables&#39;}), items_add=frozenset({&#39;other vegetables&#39;}), confidence=0.5862068965517241, lift=3.0296084222733612)])<br>-------------------------------------------------------------------------------------------------------------<br>RelationRecord(items=frozenset({&#39;tropical fruit&#39;, &#39;other vegetables&#39;, &#39;root vegetables&#39;}), support=0.012302999491611592, ordered_statistics=[OrderedStatistic(items_base=frozenset({&#39;tropical fruit&#39;, &#39;root vegetables&#39;}), items_add=frozenset({&#39;other vegetables&#39;}), confidence=0.5845410628019324, lift=3.020999134344196)])<br>-------------------------------------------------------------------------------------------------------------<br>RelationRecord(items=frozenset({&#39;nan&#39;, &#39;citrus fruit&#39;, &#39;other vegetables&#39;, &#39;root vegetables&#39;}), support=0.010269445856634469, ordered_statistics=[OrderedStatistic(items_base=frozenset({&#39;nan&#39;, &#39;citrus fruit&#39;, &#39;root vegetables&#39;}), items_add=frozenset({&#39;other vegetables&#39;}), confidence=0.5838150289017341, lift=3.0172468782178425)])<br>-------------------------------------------------------------------------------------------------------------<br>RelationRecord(items=frozenset({&#39;tropical fruit&#39;, &#39;nan&#39;, &#39;other vegetables&#39;, &#39;root vegetables&#39;}), support=0.012201321809862735, ordered_statistics=[OrderedStatistic(items_base=frozenset({&#39;tropical fruit&#39;, &#39;nan&#39;, &#39;root vegetables&#39;}), items_add=frozenset({&#39;other vegetables&#39;}), confidence=0.5825242718446603, lift=3.010576044977527)])<br>-------------------------------------------------------------------------------------------------------------<br>RelationRecord(items=frozenset({&#39;tropical fruit&#39;, &#39;whole milk&#39;, &#39;other vegetables&#39;, &#39;root vegetables&#39;}), support=0.007015760040671073, ordered_statistics=[OrderedStatistic(items_base=frozenset({&#39;tropical fruit&#39;, &#39;whole milk&#39;, &#39;root vegetables&#39;}), items_add=frozenset({&#39;other vegetables&#39;}), confidence=0.5847457627118644, lift=3.0220570553185424)])</pre><p><strong>CONCLUSION :</strong></p><p>From the output above, we see that the top associations are not surprising, with one flavor of an item being purchased with another flavor from the same item family . As mentioned, one common application of association rules mining is in the domain of recommendation systems. Once item pairs have been identified as having positive relationship, recommendations can be made to customers in order to increase sales. And hopefully, along the way, also introduce customers to items they never would have tried before or even imagined existed!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9e479d966e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>