<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashish Kumar on Medium]]></title>
        <description><![CDATA[Stories by Ashish Kumar on Medium]]></description>
        <link>https://medium.com/@schoolofsre?source=rss-30cd4672fbe3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dgU-6gEsmjMPotrksRAqWQ.jpeg</url>
            <title>Stories by Ashish Kumar on Medium</title>
            <link>https://medium.com/@schoolofsre?source=rss-30cd4672fbe3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:40:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@schoolofsre/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[K8s Tips — “how to secure your containers”]]></title>
            <link>https://faun.pub/k8s-tips-how-to-secure-your-containers-2d795d1f5581?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d795d1f5581</guid>
            <category><![CDATA[kubernetes-security]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[container-security]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 04:08:02 GMT</pubDate>
            <atom:updated>2024-01-16T20:27:43.761Z</atom:updated>
            <content:encoded><![CDATA[<h3>K8s Tips — “How to secure your containers”</h3><h4>Why did the container go to therapy? It had too many root issues 😅😂</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyc7K4TaTM678b_S7687-g.png" /></figure><p>Is your container operating with root user privileges? Please avoid it.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F8ccJRxcJexT8iSEyQL%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FSNL-snl-saturday-night-live-season-44-8ccJRxcJexT8iSEyQL&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExdXE4M3JwbTRrd2dpM3R3OGM5ZHc4azU0YnRmYWlzMXk2enhxa2d5dSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2F8ccJRxcJexT8iSEyQL%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="242" frameborder="0" scrolling="no"><a href="https://medium.com/media/b590ab75f34c87a3210397f42bc7957a/href">https://medium.com/media/b590ab75f34c87a3210397f42bc7957a/href</a></iframe><p>Let us quicky understand, why your container shouldn’t operate as a root user, before we move to the solution:</p><p>Running containers as the <strong>root user</strong> can introduce <strong>security risks</strong> and <strong>potential issues</strong>. Here are some reasons why running containers as the root user is discouraged:</p><ul><li><strong>Malicious actor</strong> —If a malicious actor gains control of a container running as root, they may have greater access to the underlying system.</li><li><strong>Filesystem Modification</strong> — Malicious actor can modify system files and directories, potentially disrupting the functioning of the host or other containers.</li><li><strong>Increased Attack Surface</strong> — Containers running as root have a larger attack surface, making them more susceptible to exploitation and compromising the overall security of the containerised environment.</li><li><strong>Network Manipulation </strong>— Containers running as root can make changes to the network configuration, potentially impacting other containers or the host system.</li></ul><p>I guess, above reasons are enough to convinced you that we shouldn’t run the containers as a root-user.</p><p><strong>TL;DR</strong></p><p>Run it as a <strong>non-root </strong>user 🙂</p><h4>How do I make my containers run as a non-root user?</h4><p>1. <strong>Create a non-root user in the Dockerfile:</strong></p><pre><br>FROM your-base-image<br><br># Create a non-root user<br>RUN useradd -m myuser<br><br># Set the user as the default user for subsequent commands<br>USER myuser<br><br># ... continue with your Dockerfile instructions</pre><p><strong>2. Specify User and Group IDs:</strong></p><pre># Create a non-root user with specific UID and GID<br><br>RUN useradd -m --uid 1000 --gid 1000 myuser</pre><p><strong>useradd</strong> is a command in Linux that is <strong>used to add user accounts to your system</strong>.</p><p><strong>3. Adjust File Permissions:</strong></p><p>By default, the new non-root user will not have permission to root files.<br>Ensure that any directories or files that the non-root user needs to access have the appropriate permissions. This may involve changing ownership or permissions during the Dockerfile build process.</p><pre># Change ownership of a specific directory<br>RUN chown -R myuser:myuser /path/to/some/directory</pre><p><strong>4. Now, Build and Run the Docker Image:</strong></p><pre>docker build -t your-image-name .<br><br>docker run -d your-image-name</pre><p>Voila! You’re done! If you go inside your container, you can validate that it is now running as a non-root user (the user which you specified in your dockerfile.</p><pre>docker exec &lt;mycontainer&gt; id</pre><p><strong>id</strong> command returns the id of the user, which is 1000 in our example as per the above dockerfile.</p><h3>BONUS 🎉🎉</h3><p>If you are running your container in Kubernetes, you can make use of a Kubernetes concept called <strong>securityContext</strong>.</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: mypod<br>spec:<br>  containers:<br>  - name: mycontainer<br>    image: your-image-name<br>    securityContext:<br>      runAsUser: 1000  # Replace with the UID of your non-root user<br>      runAsGroup: 1000 # Replace with the GID of your non-root user</pre><p>Make sure, your image has these users created well before using it. <br>How do we do that? Well, you better read my article again, I have just shown it while you create your dockerfile 😏</p><p>Kubernetes along with <strong>securityContext </strong>provides various other ways to make your containers more secure, please comment down below if you want to understand how.</p><p>Wishing you a secure and joyful 2024! 🎉🎉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*CWNoWBICGehf-lZ4.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d795d1f5581" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/k8s-tips-how-to-secure-your-containers-2d795d1f5581">K8s Tips — “how to secure your containers”</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I had no idea, preventing Configuration Drift in Terraform was so easy]]></title>
            <link>https://faun.pub/i-had-no-idea-preventing-configuration-drift-in-terraform-was-so-easy-b0e977aeb617?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0e977aeb617</guid>
            <category><![CDATA[configuration-management]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Fri, 05 Jan 2024 20:36:41 GMT</pubDate>
            <atom:updated>2024-01-06T17:17:15.380Z</atom:updated>
            <content:encoded><![CDATA[<h4>External changes in your remote infrastructure can cause the state file go out of sync — Find solutions in this article.</h4><p>Recently, my organisation implemented and enforced a stringent policy across Azure/AWS resources, requiring the addition of essential tags/labels to each individual resource.</p><p><strong>Let me explain</strong>: I used Terraform to make <strong>Azure</strong> and <strong>AWS</strong> resources. Later, because of an organisational policy, I noticed a tag/label got added to all the resources I made. Two tags were added — ‘<strong>owner</strong>’ and ‘<strong>purpose</strong>.’ <br>Although I wasn’t pleased with the external policy change, I recognized the importance of these tags, and the organisation likely has significant reasons for using them.</p><p>But this caused a <strong>drift </strong>in my state file, all the hours I spent on creating the resources, will lead to a conflict in my Terraform state file. How shall I avoid this?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Filkfz8Mn5Yz7O%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fpartner-put-ilkfz8Mn5Yz7O&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExMmZpdHlwNTVpMGJvYnh0aWwwaGVla2k1aTZ6cjA2aGU5MHk1cWN0biZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2Filkfz8Mn5Yz7O%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="472" frameborder="0" scrolling="no"><a href="https://medium.com/media/189af83419d3f5972249bc64bac3e0fe/href">https://medium.com/media/189af83419d3f5972249bc64bac3e0fe/href</a></iframe><p>Don’t stress about it. That’s where Terraform comes in handy with a cool feature called ‘<strong>lifecycle</strong>’.</p><h4><strong>What is lifecycle in Terraform?</strong></h4><p>Lifecycles is a block which can be used inside any resource block. Yes, you got me right, any terraform resource :) <br>It has 4 independent argument create_before_destroy, prevent_destroy, ignore_changes, and replace_triggered_by. Each of the argument has different purposes. In this article, <strong>I will only talk about </strong><strong>ignore_changes.</strong></p><p><strong>ignore_changes</strong>: As the name suggests, it is to tell Terraform, to ignore certain changes/update made in the infrastructure. <br>It takes a <strong>list</strong> of attributes which Terraform needs to ignore. See the below example:</p><pre>resource &quot;aws_instance&quot; &quot;example&quot; {<br> <br>  lifecycle {<br>    ignore_changes = [<br><br>      # Ignore changes to tags, because it was added by my organizational<br>      # policy.<br><br>      tags,<br>    ]<br>  }<br>}</pre><p>In the above example, Terraform will ignore any changes made in tags attribute, but allow to update other attribute present in the resource block.</p><h4><strong>Interesting. How can I make Terraform disregard ALL changes (not limited to a specific attribute like tags)?</strong></h4><p>Well, in that case, instead of a list, the special keyword allis used to instruct Terraform to ignore <em>all</em> attributes, which means that Terraform can create and destroy the remote object but will never propose updates to it.</p><pre>resource &quot;aws_instance&quot; &quot;example&quot; {<br> <br>  lifecycle {<br>    ignore_changes = all<br>  }<br>}</pre><h4><strong>How can I make Terraform overlook an attribute when it’s a list, map, or object type?</strong></h4><p>Maps/objects, tuple/list can be referenced using index notation, see the below example:</p><pre>resource &quot;aws_instance&quot; &quot;example&quot; {<br> <br>  lifecycle {<br>    ignore_changes = [<br><br>    tags[&quot;owner&quot;]<br>    list[0]<br><br>    ]<br>  }<br>}</pre><h4><strong>If this is very clear to you, let us also understand when to use </strong><strong>ignore_changes in your code?</strong></h4><ul><li>If someone is bound to make changes in your remote infrastructure (like my organisational policy)</li><li>If you already know that some attributes in your resource might change in the future.</li><li>Dealing with sensitive information like passwords or private keys, you might want to prevent Terraform from updating the resource whenever these values change. Instead, handle sensitive data separately and only update it when necessary.</li></ul><pre>resource &quot;example_resource&quot; &quot;example&quot; {<br>  name     = &quot;example&quot;<br>  password = &quot;sensitive_password&quot;<br><br>  lifecycle {<br>    ignore_changes = [password]<br>  }<br>}</pre><h4><strong>Important note to remember</strong></h4><ul><li>Only attributes defined by the resource type can be ignored. ignore_changes cannot be applied to itself or to any other meta-arguments.</li><li>Keep in mind that using ignore_changes should be done thoughtfully, as it might lead to Terraform not reflecting actual changes in your infrastructure. Always ensure that ignoring certain changes aligns with your overall infrastructure management strategy.</li></ul><p>Reference — <a href="https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#ignore_changes">https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#ignore_changes</a></p><h3>Dive into Hands-On</h3><p>I have already created a resource in Azure cloud using Terraform:</p><p><strong>Resource group</strong> : terraform-lifecycle-example<br><strong>Resource type</strong> : Azure container registry<br><strong>Resource name</strong> : lifecyleexampleacr<br><strong>Source code</strong> : <a href="https://github.com/iAshishHere/terraform-lifecycle-example">https://github.com/iAshishHere/terraform-lifecycle-example</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDOg_3REtE8FqU_p03rqjA.png" /></figure><p>Now, I will manually go and add some tags to the resource that I have already created (See the screenshot below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O57s75FV_SeZhsbXRBElvQ.png" /></figure><p>Now that I have manually added some tags, lets check, how terraform will react to this change.</p><p>As soon as I ran theterraform plan command, you can see that Terraform detects changes in the tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P4ytaDN38kOdzycDedUBKg.png" /></figure><p>I don’t want terraform to detect any changes made in the tags, hence, I will use Terraform lifecycle. See the example below:</p><pre>data &quot;azurerm_resource_group&quot; &quot;rg&quot; {<br>  name = &quot;terraform-lifecycle-example&quot;<br>}<br><br><br>resource &quot;azurerm_container_registry&quot; &quot;example&quot; {<br>  name                = &quot;lifecyleexampleacr&quot;<br>  resource_group_name = data.azurerm_resource_group.rg.name<br>  location            = &quot;east us&quot;<br>  sku                 = &quot;Premium&quot;<br><br>  lifecycle {<br>    ignore_changes = [tags]<br>  }<br>}</pre><p>As soon as I added the lifecycle block in my <a href="https://github.com/iAshishHere/terraform-lifecycle-example/blob/main/main.tf">resource block</a>, Terraform ignored the changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BxSdsjuSt6vq1XeonDSTUg.png" /></figure><p>Now, that you have got a glimpse of how to use the lifecycle (whenever needed), why not trying out yourself?</p><p>Happy terraforming! 🚀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*2MjYJt5Km6J2W6yF.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0e977aeb617" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/i-had-no-idea-preventing-configuration-drift-in-terraform-was-so-easy-b0e977aeb617">I had no idea, preventing Configuration Drift in Terraform was so easy</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The most awaited feature of Terraform]]></title>
            <link>https://faun.pub/the-most-awaited-feature-of-terraform-db0176482213?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/db0176482213</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[modules]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 13:42:30 GMT</pubDate>
            <atom:updated>2020-08-11T10:30:46.094Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article briefs you about one of the most awaited features of Terraform (version 0.13)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSiRatL4BQACRTKjD1ChOQ.png" /></figure><p>If you are working intensely with terraform, you guessed it right. Terraform rolled up with <strong>Module Dependencies in Terraform 0.13 </strong>👏👏👏</p><p>I was waiting for this feature for a long time. Now we don’t have to do any “workaround” to achieve dependencies among the module anymore.</p><p>For those who are still wondering what module dependencies are — It is similar to the resource dependency. See the below example</p><pre>resource &quot;resource-1&quot; &quot;alias1&quot; {<br>    ...<br>    ...<br>}</pre><pre>resource &quot;resource-2&quot; &quot;alias2&quot; {<br>    ...<br>    ...<br>    <strong>depends_on = [resource-1.alias1]</strong><br>}</pre><p>Similarly, terraform in its latest version (<strong>i.e. 0.13) </strong>has released depends_on meta-argument on modules to create dependencies among the module.</p><p>Interested to have a look at the workaround before you see how it works on Terraform 0.13? See this <a href="https://medium.com/@aniket10051994/inter-module-resource-dependency-in-terraform-9291070133f3">example</a>.</p><h3>0.13 Feature for Module Dependencies</h3><p><strong>Now let us understand this by an example.</strong></p><h4>About the example</h4><p>In this example, we are creating a bucket on GCP, for that, we have created a <a href="https://github.com/iAshishHere/Module-Depencency-Example/tree/master/Storage">module</a> where we are passing the name of the bucket, respective storage classes, locations, and Project.</p><p>But just to explain the module dependency, we will call the same module twice such that module1 should execute first followed by module2.</p><p>The storage module code is <a href="https://github.com/iAshishHere/Module-Depencency-Example/tree/master/Storage">here</a></p><p>The <a href="https://github.com/iAshishHere/Module-Depencency-Example/blob/master/main.tf">main.tf</a> of the code :</p><pre>module “Storage1” {<br>   source              = &quot;.//Storage&quot;<br>   names = [&quot;iashishhere-bucket1&quot;]<br>   locations = [ &quot;us-west2&quot;]<br>   storage_classes = [&quot;STANDARD&quot;]<br>   project = &quot;ashish-sandbox-2020&quot;<br>}</pre><pre>module “Storage2” {<br>   source              = &quot;.//Storage&quot;<br>   names = [&quot;iashishhere-bucket2&quot;]<br>   locations = [ &quot;us-west2&quot;]<br>   storage_classes = [&quot;COLDLINE&quot;]<br>   project = &quot;ashish-sandbox-2020&quot;<br>   <strong>depends_on = [module.Storage1]<br></strong>}</pre><pre>output &quot;StorageName&quot; {<br>  value = module.Storage2.names<br>}</pre><p>depends_on = [module.Storage1] — This is how syntactically we declare the dependency among the modules using the depends_on argument.</p><p>Observe the below <strong>output </strong>in the console :</p><pre>An execution plan has been generated and is shown below.<br>Resource actions are indicated with the following symbols:<br>  + create</pre><pre>Terraform will perform the following actions:</pre><pre># module.Storage1.google_storage_bucket.bucket[0] will be created<br>  + resource &quot;google_storage_bucket&quot; &quot;bucket&quot; {<br>      + bucket_policy_only = (known after apply)<br>      + force_destroy      = false<br>      + id                 = (known after apply)<br>      + location           = &quot;US-WEST2&quot;<br>      + name               = &quot;iashishhere-bucket1&quot;<br>      + project            = &quot;ashish-sandbox-2020&quot;<br>      + self_link          = (known after apply)<br>      + storage_class      = &quot;STANDARD&quot;<br>      + url                = (known after apply)<br>    }</pre><pre># module.Storage2.google_storage_bucket.bucket[0] will be created<br>  + resource &quot;google_storage_bucket&quot; &quot;bucket&quot; {<br>      + bucket_policy_only = (known after apply)<br>      + force_destroy      = false<br>      + id                 = (known after apply)<br>      + location           = &quot;US-WEST2&quot;<br>      + name               = &quot;iashishhere-bucket2&quot;<br>      + project            = &quot;ashish-sandbox-2020&quot;<br>      + self_link          = (known after apply)<br>      + storage_class      = &quot;COLDLINE&quot;<br>      + url                = (known after apply)<br>    }</pre><pre>Plan: 2 to add, 0 to change, 0 to destroy.</pre><pre>Do you want to perform these actions?<br>  Terraform will perform the actions described above.<br>  Only &#39;yes&#39; will be accepted to approve.</pre><pre><strong>Enter a value: yes</strong></pre><pre>module.Storage1.google_storage_bucket.bucket[0]: Creating...<br>module.Storage1.google_storage_bucket.bucket[0]: Creation complete after 2s [id=iashishhere-bucket1]<br>module.Storage2.google_storage_bucket.bucket[0]: Creating...<br>module.Storage2.google_storage_bucket.bucket[0]: Creation complete after 2s [id=iashishhere-bucket2]</pre><p>Note: This is a very simple example just for the explanation. You can try out yourself in some of your complex scripts.</p><h3>Conclusion</h3><p>So now you don’t have to look for any hack for using module dependency, because it is so easy to create dependencies among modules as we have already seen in the above example.</p><p>If you want to try yourself, try out this <a href="https://github.com/iAshishHere/Module-Depencency-Example">example</a></p><h4>BONUS</h4><p>Since already we were talking about the modules, there is one more cool feature available in the Terraform 0.13 version, now you can call multiple instances of the same module using the count and index meta-argument just like we do in resources.</p><p>Comment below if you want to see a separate example for that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Subscribe to </strong><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumprebanner"><strong>FAUN topics</strong></a><strong> and get your weekly curated email of the must-read tech stories, news, and tutorials </strong>🗞️</p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db0176482213" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/the-most-awaited-feature-of-terraform-db0176482213">The most awaited feature of Terraform</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Airflow in 5 minutes]]></title>
            <link>https://medium.com/swlh/apache-airflow-in-5-minutes-c005b4b11b26?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c005b4b11b26</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 09:05:22 GMT</pubDate>
            <atom:updated>2020-11-10T07:18:16.963Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Airflow in 10 minutes</h3><h4>A quick introduction to Apache Airflow (A beginners guide)</h4><h4>Introduction</h4><p>Apache Airflow is an open-source tool for orchestrating complex workflows and data processing pipelines. It is a platform to programmatically schedule, and monitor workflows for scheduled jobs.</p><p>Apache airflow makes your workflow simple, well organized, and more systematic which can be easily authored and schedules based on the requirement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJsMPN5kPMI7JuqhsaP7rA.png" /></figure><p><strong>Let’s start with the basics.</strong></p><p><strong>What do we mean by workflow?<br></strong>Workflow can be your simple calculation, creating infrastructure, perform some query in the database, bash command, python script, MySQL queries, Hive queries, etc. <br>Workflow is divided into one or more than one task which relates to each other and forms a <strong>DAG </strong>(Directed Acyclic Graph).</p><p><strong>What is DAG?<br></strong>In simple terms, DAG is a collection of all small task which joins together to perform a big task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*l2rG8ILtIrQl-h9JMbq4Bg.png" /></figure><p>Still, Confused? Let us take a better example.</p><p><strong>Let’s understand this by Phases of Compiler.</strong></p><p>For people who are unaware of phases of the compiler — Think of it as a process followed by your compiler to convert high-level language into the low-level language (which your machine understands)</p><p>This is what happens after compiling a piece of code. Our code gets converted into a character stream, Lexical Analyzer converts it into tokens, then syntax analyzer converts it into a syntax tree, then semantic analyzer, intermediate code generator, code optimization, target assembly code. Here each step is very crucial and very much dependent on the previous steps. <br> Error in any one of the steps will not compile our code successfully.</p><p>How these steps would look in DAG?</p><p>Here diagram of some DAG</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*NCJL_QPs9eI9e8r9P1g3HQ.png" /></figure><p>If your task is dependent on some other task, you can set dependencies based on your requirement. We will discuss this briefly in getting started with Apache-airflow.</p><p><strong>What are the scheduled jobs?</strong></p><p>Suppose your workflow must be run every Sunday, you can schedule it in such a way that it will only be trigger on Sundays. How cool is this?<br> In simple terms, you can automate your workflow. The best part of automation is you can avoid future human errors. <br> Your DAG can be easily monitored, controlled, and triggered.</p><p><strong>What if your workflow gets fail? What if it got completed successfully? What if it took more than the expected time to complete?</strong></p><p>Well, keeping all such things in mind, apache-airflow has given such features like If your workflow gets fail, you can set it as to send an Email alert, slack notification to the required person/team.<br> You can also set it to send an email when the DAG ran successfully. <br> <br> What if your task got successfully run, but took more than expected time? (In real case scenario this is a problem) — For this, you can set your SLA. Suppose your SLA time is 600 seconds. But your task took 900 seconds. Isn’t it something wrong? If this happens, it triggers the required team</p><p><strong>Advantages of Apache Airflow</strong></p><p>Apache-airflow has got quite a few advantages which makes it a better tool than comparing to other tools in the market. <br> First, we’ll discuss its advantages and then a few benefits of using airflow over other similar tools.</p><ul><li><strong>Open Source / Python</strong> — Airflow is developed in Python and it’s very easy to design your workflow. You can contribute your plugins in open source and also use other plugins based on your requirement.</li><li><strong>Monitoring</strong> — You can easily monitor your task status once it is running. Airflow provides all the necessary details of each task such as execution time, landing time, logs, etc.</li><li><strong>Scalable</strong> — Mostly all the data-driven companies prefer to use Airflow, so the complexity of workflow will grow as moving ahead. But airflow holds the advantage over other tools as it’s more scalable.</li><li><strong>Smart Scheduling</strong> — You can schedule your task however you want to. For example, if you want to schedule your task to run every Sunday at 4:00 PM, you can do it.</li><li><strong>Managing Dependency</strong> — One of the cool features which make airflow better than all other tool is proper management of dependency. If a task t2 is dependent on task t1 which is further dependent on task t0, you can set dependency across.</li><li><strong>Resilience</strong> — All workflow tools sometimes behave unexpectedly, may it be any reason such as network issues, human error, taking a longer time than expected, etc. Airflow has several features such as retry. If by any chance, your task failed to execute, airflow retires it after a certain minute, (If it might have failed due to other reasons such as network issue, then retry might make that task successful). <br> It even gives an alert message to the team via email/slack.</li><li><strong>Alerting</strong> — This is one of the coolest feature airflows have, if you want your team should get notified if any of your tasks get fail, it will notify via Mail, slack notification. <br> Even if your task gets successful, you may want to get notified, airflow has that feature too.</li><li><strong>Service Level Agreement (SLA) Timeout</strong>: This might be one of the most crucial features (Al least for some company) which airflow has provided. If your task takes generally 500 sec to finish, but one certain day, it took 1500 seconds, don’t you think something unexpected happened? <br>Wouldn’t you like to get notified? You may have your own reason to store these records, maybe for analytic purposes or any other research work. <br>Example: Suppose your company collects user registration data at the end of the day, but you have noticed that every Saturday your task is taking a lot of time that means people on Saturday are more to participate.</li><li><strong>User Interface</strong> — It has a good UI which makes it more approachable for the user. You can view the task in a well-formatted tree structure, you can view the logs details, airflow database, task duration, lading times, rich graph view in UI.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*1mnjl5D-WphcmJvn7gxDIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*1a_p_5vuuq27-HMv04L33Q.png" /></figure><ul><li><strong>Plugins and hooks</strong> — Airflow has got various pre-defined plugins and also user-defined plugins which makes your task easy.</li><li><strong>Custom plugins, hook, sensors </strong>— I will get back to you about these keywords such as plugins hook sensors, etc. In short, we can make our own operators in airflow and manipulate it to work as we like it to behave.</li><li><strong>Cloud services</strong> — Cloud platforms like Google cloud shows its support to apache- airflow. Cloud composer — is a google service that creates an environment of apache — airflow in google cloud. You can perform almost everything and use cloud services such as Google BigQuery, Dataproc, Dataflow, etc.</li></ul><p><strong>How about doing some hands-on?</strong></p><p>I will be using a cloud composer (a GCP based managed services) to create an airflow environment. We can also create a local environment.<br>As soon as you create a cloud composer, it creates a bucket in your cloud storage automatically which is eventually mounted with your composer environment. Similarly, you will have the same directory structure when you will install on your local environment.</p><pre>airflow_home</pre><pre>├── airflow.cfg</pre><pre>├── airflow.db</pre><pre>├── plugins</pre><pre>├── dags                &lt;- Your DAGs directory</pre><pre>│   └── hello_world.py  &lt;- Your DAG definition file</pre><pre>└── unittests.cfg</pre><p>In the DAG folder, you need to upload all your python script or DAG which will get rendered into the airflow server and show in the UI, and then you can trigger it manually or if scheduled, it will trigger automatically.</p><p>Below is the sample code!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4436660e6f327804d84428bf54d87dd5/href">https://medium.com/media/4436660e6f327804d84428bf54d87dd5/href</a></iframe><p>Let us understand this code line by line.</p><pre>from datetime import datetime<br>from airflow import models<br>from airflow.operators.bash_operator import BashOperator</pre><p>These are the import statements for the facility which we are using in our DAG. Since we are using BashOperator we need to import BashOperator from the airflow library.</p><pre>default_dag_args = {<br> ‘start_date’: yesterday,<br> ‘email_on_failure’: False,<br> ‘email_on_retry’: False,<br> ‘retries’: 1,<br> ‘retry_delay’: datetime.timedelta(minutes=5),<br> ‘project_id’: models.Variable.get(‘gcp_project’)<br>}</pre><p>These are the default argument which we can set for each task by setting the argument to each task’s constructor. We can define a dictionary of default parameters that we can use when creating tasks.</p><p><strong>Start_date</strong> as yesterday means start as soon as it loaded into the server. We can set it at any time.</p><p><strong>Email_on_failure</strong> as False, if it is true it will send an email to the specified person/team if any particular task gets fail.</p><p><strong>Retries</strong> as 1 mean the number of retries after the task get fails.</p><p><strong>Retry_delay</strong> is set as 5 minutes means, after any specific task gets fail, it should wait exactly 5 minutes to start a retry.</p><p><strong>Email_on_retry</strong> as False, if it is true, then after a task gets fail, after every retry it will send email to the specified person/team.</p><p>In DAG everything works as an operator. <br>Example t1=SomeOperator(arguments)</p><p>We are using Bash Operator in this example.</p><pre>t1 = BashOperator(</pre><pre>task_id=’Make directory’,</pre><pre>bash_command=’mkdir folder_name’,</pre><pre>dag=dag)</pre><p><strong>t1</strong> is a value that is calling the <strong>BashOperator</strong> class and sends all the required arguments to it.<br>Every task has a <strong>task_id</strong> which uniquely defines a task, and other required arguments based on what operator you are using.</p><p>In our DAG, we are running two different tasks such as t1 and t2, one is <strong>creating </strong>a directory and the other is <strong>deleting </strong>the directory. <br>So, its obvious <strong>t1</strong> has to be run before <strong>t2, </strong>so we have set dependency such as t1 must run before task t2.</p><p><strong>There are two ways to set dependency:</strong></p><ul><li>t1 &gt;&gt; t2 (This means t1 will run before t2)</li><li>t1.upstream(t2) (This also means the same as above)</li></ul><p>In order to do vice-versa the syntax would be like this:</p><ul><li>t1 &lt;&lt; t2</li><li>t1.downstream (t2)</li></ul><p>Finally, we need to place our .py file into the DAG folder, and then it will get loaded into the server automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*f3ZpmMlkewJoKRrrGTZ68g.png" /></figure><p>You can see the task graph view as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*X6EVEglrc4rxi-FfFrOheA.png" /></figure><p>The arrow denotes that make_directory is dependent on delete_directory.</p><p>Well, this was a very simple example of how we create tasks and run the workflow.</p><h3>Operators</h3><p>While DAGs describes how to run a workflow, Operators determine what actually gets done.</p><p>An operator describes a single task in a workflow. Operators are usually (but not always) atomic, meaning they can stand on their own and don’t need to share resources with any other operators.</p><p>Note: If two operators need to share information, like a filename or a small amount of data, you should consider combining them into a single operator. If it absolutely can’t be avoided, Airflow does have a feature for operator cross-communication called <a href="https://airflow.apache.org/docs/1.10.1/concepts.html?highlight=xcom">XCom</a>.</p><p>Airflow provides many operators some of them includes:</p><ul><li>BashOperator (Which we just saw)</li><li>PythonOperator — Use to call any python function in DAG</li><li>EmailOperator — Sends email</li><li>SimpleHttpOperator — Sends an HTTP request</li><li>MySqlOperator, SqliteOperator, PostgresOperator, OracleOperator, JdbcOperator — executes SQL queries.</li></ul><h3>Sensors</h3><p>Sensors are a special type of operator which runs behind the scene all the time. The sensor class is created by extending BaseSensorOperator.</p><p>It has a poke method, which executes the task over and over after every poke_interval seconds until it returns True and if it returns False it will be called again.</p><p><strong>Example:</strong> Sensor to check whether a file is present in a specified directory. After every poke_interval, the poke method of the sensor class will be executed, if the file is not present it will send <strong>False</strong>, once the file is present in the directory, it will return <strong>True</strong>.</p><h3>Trigger Rules</h3><p>All operators have a trigger_rule argument that defines the rule by which the generated task gets triggered. The default value for trigger_rule is all_success and can be defined as “trigger this task when all directly upstream tasks have succeeded”. All other rules described here are based on direct parent tasks and are values that can be passed to any operator while creating tasks:</p><ul><li>all_success: (default) all parents have succeeded</li><li>all_failed: all parents are in a failed or upstream_failed state</li><li>all_done: all parents are done with their execution</li><li>one_failed: fires as soon as at least one parent has failed, it does not wait for all parents to be done</li><li>one_success: fires as soon as at least one parent succeeds, it does not wait for all parents to be done</li><li>none_failed: all parents have not failed (failed or upstream_failed) i.e. all parents have succeeded or been skipped</li><li>dummy: dependencies are just for show, trigger at will<br>See more about <a href="https://airflow.apache.org/docs/stable/concepts.html#trigger-rules">Trigger rules</a></li></ul><h4>Are you curious to know how two operators communicate with each other?</h4><p>Apache-airflow provides a feature called XCom</p><h3>XCom (Cross — Communication):</h3><p>Communication among two operators. If any operator returns some value, it gets store in xcom, airflow provides a mechanism to pull xcom value using xcom_pull() and use it in some other operation and also to push value using xcom-push().</p><p><strong>Example: </strong>View code in GitHub <a href="https://github.com/iAshishHere/Apache-Airflow/blob/master/Xcom_example/Xcom_example.py"><strong>Xcom_example</strong></a><strong>.py</strong></p><p>Let’s end this article by listing other alternative tools in the market.</p><ul><li>Cron</li><li>Apache Oozie</li><li>Luigi</li><li>Azkaban</li></ul><p>Now that you have understood what DAG is, here’s a suggestion for you</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*-mvQD87V7cNUW9SaDWbBRw.png" /><figcaption>Picture Credit <a href="https://medium.com/u/ded51ddac369">Kaxil Naik</a></figcaption></figure><p><a href="https://airflow.apache.org/"><strong>Airflow Documentation</strong> </a><br><a href="https://github.com/airflow-plugins"><strong>Apache-Airflow GitHub</strong></a><strong><br></strong><a href="https://github.com/iAshishHere/Apache-Airflow"><strong>To see some example code visit my GitHub</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c005b4b11b26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/apache-airflow-in-5-minutes-c005b4b11b26">Apache Airflow in 5 minutes</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Variables Validations in Terraform]]></title>
            <link>https://schoolofsre.medium.com/variables-validations-in-terraform-95a12542aa3a?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/95a12542aa3a</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[infrastructure-automation]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[iac]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Tue, 12 May 2020 04:08:44 GMT</pubDate>
            <atom:updated>2020-06-05T16:45:02.185Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article guides you on how to do validation checks for your terraform variables.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*klM1FxKlLugg4LMLyzfUvQ.jpeg" /></figure><p>There have been scenarios, where you want to perform specific validations on some variables used by your Terraform script.<br>Terraform gives you the liberty to pass variables among one of the ways shown below :</p><pre>terraform apply -var-file=”file_location.tfvars” <br>terraform apply -var=”name=sample_name”</pre><p>But since its a manual task to write a variable, there can be a chance of human error that might lead to a disaster.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FzPOErRpLtHWbm%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fcommunity-donald-glover-zPOErRpLtHWbm&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2FzPOErRpLtHWbm%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="444" frameborder="0" scrolling="no"><a href="https://medium.com/media/83efb2d67fdb8840aa35acfadd03bb94/href">https://medium.com/media/83efb2d67fdb8840aa35acfadd03bb94/href</a></iframe><p><strong>Consider a scenario: </strong>You need a simple machine type as e2-standard-2 (which is of low cost), but someone from your team member is not aware of machine type specifications and gave a different high-end machine type called e2-highmem-2 (price is comparatively very very high)</p><p>So let’s quickly have look at how can we perform a simple validation.</p><p><strong>Let us consider you want to put a validation on a variable whose length should not exceed by 10 characters and it should be in all CAPS.</strong></p><p><strong>Case 1 : </strong>( name = “bob” )</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a72628cd38e8fb0db206dd4324a4f894/href">https://medium.com/media/a72628cd38e8fb0db206dd4324a4f894/href</a></iframe><p>There is an <strong>experimental feature</strong> available in terraform v.12.20 called variable_validation (It is still an experimental feature, not recommended for your final release)</p><p>So, that is how you have to implement validation on your variable.<br>What should happen according to our defined validation?</p><p>Yes, you have guessed it right. It will<strong> throw an error</strong>. As per our validations, the variable length must be more than 4 characters <strong>AND </strong>all should be of <strong>UPPER CASE</strong>. Therefore name = “bob” is invalid</p><pre>Error: Invalid value for variable</pre><pre>  on main.tf line 9:<br>   9: variable &quot;name&quot; {</pre><pre>Validation condition of the name variable did not meet.</pre><p>Case 2 : ( name = “BOB MARLEY” )</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3c6d5e73b802ae0b9c0c38e38ed03fb/href">https://medium.com/media/e3c6d5e73b802ae0b9c0c38e38ed03fb/href</a></iframe><p>Again you guessed it right. It will <strong>not throw any error</strong> as it satisfied all our conditions which we defined.</p><pre>$ terraform plan<br>Refreshing Terraform state in-memory prior to plan...<br>The refreshed state will be used to calculate this plan, but will not be<br>persisted to local or remote state storage.</pre><pre>--------------------------------------------------------------------</pre><pre>No changes. Infrastructure is up-to-date.</pre><p><strong>Another Scenario?</strong></p><p>Suppose you want your variable to have a specific regular expression. We can use some inbuilt functions like can() and regex() together.</p><p>Similarly, you can define various validations on all of your variables if needed.<br>You can also use some other terraform inbuild functions like for(), try() , upper() and whatnot.</p><p><strong>Key Points to Remember :</strong></p><ul><li>error_message must be in a specific format. It must be at least one full English sentence starting with an uppercase letter and ending with a period or question mark.</li><li><strong>Interpolation</strong> is <strong>NOT </strong>allowed in error_message</li><li>Validation is not mandatory in Terraform, but it is a good practice to do if required</li><li>This is just one of the few ways to perform validations, you can perform in many other ways depending on your need</li><li>There is an experimental feature available in terraform v.12.20 called variable_validation. It is still an experimental feature, not recommended for your final release.</li></ul><p><strong>Curious about what is Infrastructure as a code is?</strong></p><p><a href="https://medium.com/@iashishhere/infrastructure-as-code-917d245ce35e">Infrastructure as Code</a></p><p><strong>Contact me @ iashishhere@gmail.com<br>Linkedin: </strong><a href="https://www.linkedin.com/in/iashishhere/">https://www.linkedin.com/in/iashishhere</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95a12542aa3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How did I resolved pip package dependency issue in Apache Airflow?]]></title>
            <link>https://schoolofsre.medium.com/how-did-i-resolved-pip-package-dependency-issue-in-apache-airflow-8e0b1e5a067c?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e0b1e5a067c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Thu, 09 Apr 2020 15:01:57 GMT</pubDate>
            <atom:updated>2021-02-23T17:54:55.793Z</atom:updated>
            <content:encoded><![CDATA[<h3>How I resolve the pip package dependency issue in Apache Airflow?</h3><h4>Talks about the cyclic dependency issue with pip packages, how to resolve it using PythonVirtualenvOperator.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*mPkAJ9Gnmt_XRkPiDovohQ.png" /><figcaption>Airflow and pip packages dependency issue</figcaption></figure><p>Have you ever faced pip packages version issue while working in Apache airflow? Especially while working with multiple tasks or multiple Dags.</p><p>Let me explain the issue first before I go ahead and explain <strong>how I resolved this issue.</strong></p><p>Consider pip-packageA-v1 has a dependency with pip-packageC-v1<br>whereas, pip-packageB-v1 has a dependency with pip-packageC-v2<br>Now obviously, you can’t have two different versions of pip-packageC that is pip-packageC-v1 and pip-packageC-v2 in your Airflow environment. Therefore, this leads to<strong> cyclic package dependency</strong>.</p><p>And trust me, it’s a pain to find out which version of that package will work for both. Especially if your Apache airflow is hosted in Kubernetes.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fql4LidslabKpi%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Firritated-ql4LidslabKpi&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2Fql4LidslabKpi%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="245" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba57b857df89c9f6b342d1d1256179ce/href">https://medium.com/media/ba57b857df89c9f6b342d1d1256179ce/href</a></iframe><p>The good news, there is an operator known as PythonVirtualenvOperato can solve this issue.</p><p>Document for the PythonVirtualenvOperator<strong> </strong>can be found <a href="https://airflow.apache.org/docs/stable/_api/airflow/operators/python_operator/index.html"><strong>here</strong></a></p><p>And here is a snippet that shows how to use it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2b08bd9e9cd3c04c219a1d912b474b5/href">https://medium.com/media/e2b08bd9e9cd3c04c219a1d912b474b5/href</a></iframe><h4>Before using it, point to be noted</h4><ul><li>Allows one to run a function in a virtualenv that is created and <strong>destroyed automatically.</strong></li><li>The function must be defined using def and not be part of a class. All imports must happen inside the function and no variables outside of the scope may be referenced.</li><li>Besides, one can pass stuff through op_args and op_kwargs, and one can use a return value.</li><li>Note that if your virtualenv runs in a different Python major version than Airflow, you cannot use return values, op_args or op_kwargs,but can use string_args.</li></ul><pre><strong>requirements </strong>- need to provide all the pip packages which your script is using.</pre><pre><strong>python_version </strong>- Here you can provide your target python version based on your need.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*kuObUEN63fU0beg3i6lUAg.png" /></figure><p><strong>Is this the only way to deal with PyPI cyclic dependency in Airflow?<br> No, </strong>this is one of the ways to deal with it. <br>Airflow provides <a href="https://airflow.apache.org/docs/stable/_api/airflow/operators/docker_operator/index.html">DockerOperator</a>,<a href="https://airflow.apache.org/docs/stable/_modules/airflow/contrib/operators/kubernetes_pod_operator.html">KubernetesPodOperator </a>which have even more flexibility, and you can choose based on your requirements.</p><p>Comment down below if you want me to show a brief example for DockerOperator, KubernetesPodOperator or any other operators. I will try my level best to come up with another article.</p><p>Clap for appreciation :) Thanks</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e0b1e5a067c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infrastructure as Code]]></title>
            <link>https://faun.pub/infrastructure-as-code-917d245ce35e?source=rss-30cd4672fbe3------2</link>
            <guid isPermaLink="false">https://medium.com/p/917d245ce35e</guid>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Ashish Kumar]]></dc:creator>
            <pubDate>Mon, 06 Apr 2020 02:56:53 GMT</pubDate>
            <atom:updated>2022-05-25T08:31:50.018Z</atom:updated>
            <content:encoded><![CDATA[<h3>Infrastructure as Code — DevOps revolution.</h3><h4>Explained what is IaC, compared it with the traditional approach, explained with a famous tool of Hashicorp terraform</h4><p><strong>What is Infrastructure as code?</strong></p><p>Before understanding what Infrastructure as code is, let’s have a look at how the infrastructure was managed traditionally. Let’s consider a private data center, which has VMware virtual machines running. If someone wants to get a server or a resource provisioned, then they will have to follow slow and grinding traditional procedure that may look something like below</p><ul><li>Raise an IT ticket</li><li>Someone from the IT team will process the ticket, sent it to a review team</li><li>And finally, after reviewing all aspects, the operation team will provision that piece of infrastructure.</li></ul><p>The same goes for deleting/decommissioning that piece of infrastructure. Below diagram shows the flow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qe4jVgp1p3wrnf2vt7tQcw.png" /><figcaption>Explained the traditional approach</figcaption></figure><p>This approach was fine as the companies needed limited infrastructure, the provisioned server used to live for months and years, there was a relatively limited scale of deployments, and since it was a private data center it was working fine.</p><p><strong>When did this approach fail?</strong></p><ul><li>One, when the industry made a transition to the cloud.</li><li>Secondly, with the demand for elasticity in infrastructure, where instead of months to years, it’s nowadays to weeks in terms of how long a resource might live.</li></ul><p>The scale of infrastructure is much higher in a modern approach. Instead of having one large Instance (traditional approach), now we have many smaller instances. We might scale up the load during the peak days and then scale down at night to save our cost (unlike owning hardware, we are on the cloud and we are paying on hourly bases), so we pay for only what we use. So, it’s important to remove the piece of infrastructure immediately when we are not using it. Similarly, provision the piece of infrastructure as fast as possible based on our needs.</p><p>Now imagine raising thousands of tickets to spin up the infrastructure at peak time and file another for spin them back down. Will this not be detrimental to business agility?<br> <br> <strong>Another major problem </strong>was how to manage the vast infrastructure? <br> Imagine you have requested IT, team, to spin up infrastructure on the cloud which consists of various resources such as multiple VMs, Storage, SQL, etc. IT team has successfully reviewed and provisioned the needed infrastructure in a specified environment, say, Dev.<br> Now, you want the same infrastructure in QA and Prod environment as well, and you raised another ticket for the same, and the IT team has successfully reviewed and provisioned the infrastructure in QA and Prod. But soon after your application deployment, you noticed your application is not working as expected as it was working in the Dev environment, even though the deployment code was the same.</p><p>So, what was the problem? After spending hours, you noticed that the problem was in the infrastructure <strong>configuration drift</strong>, there was a minute change in the infrastructure in Dev and QA and prod by the IT team, a human error (as they manually setup the infrastructure). If they would have spun up the infra exactly with the same configuration as they did for the dev environment, the problem would have not occurred.<br> <br> So, we faced two major problems:</p><p>· The ticket raising system<br>· The <strong><em>configuration drift</em></strong> and management of the vast infrastructure</p><p><strong>How do we begin to operationalize this in a way that’s reliable and robust, and not prone to human error?</strong></p><p>Now, here Infrastructure as code plays its role to solve these two problems. So, the real idea behind infrastructure as code is: <strong><em>How do we take the process — in some sense, the things that we were pointing and clicking to achieve — how do we take that and capture that in a codified way</em></strong>? So, if I need to do it one time, ten times, or a thousand times, I can automate that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DXCqLYd5TJef-O0M_Fzcpw.png" /><figcaption>Explained the modern approach</figcaption></figure><p>Every morning, I can hit a script that brings up a thousand machines, and every evening, hit the same script to bring it back down to whatever our evening size is.</p><p>I can begin to both automate that, but also now that I’ve captured it as a codified form, I can check it into <strong>version control</strong>, and I get versioning. Now I can see an incremental history of who changed what, how is my infrastructure defined at any given point of time, and I have this transparency of documentation that I lack in a traditional point-and-click environment.</p><p><strong>Adopting IaC solves following problems</strong></p><ul><li>Infrastructure as Code evolved to solve the problem of <strong>environment drift</strong> in the release pipeline. Without IaC, teams must maintain the settings of individual deployment environments. As the resources increase over time, it becomes difficult to reproduce the same in all the environment manually. Inconsistency among environments leads to issues during deployments.</li><li><strong>Idempotence</strong> is a principle of Infrastructure as Code. Idempotence is the property that a deployment command always sets the target environment into the same configuration, regardless of the environment’s starting state. Idempotency is achieved by either automatically configuring an existing target or by discarding the existing target and recreating a fresh environment.</li><li>IaC automation dramatically <strong>speeds the process of provisioning infrastructure for development, testing, and production </strong>Because it codifies and documents everything, IaC can even automate the provisioning of legacy infrastructure, which might otherwise be governed by time-consuming processes (like pulling a ticket).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*AMkLtzp5AyuEOOHZWxZGog.png" /></figure><ul><li>In addition to dramatically reducing the time, effort, and specialized skill required to provision and scale infrastructure, IaC lets organizations take maximum advantage of cloud computing’s consumption-based cost structure. It also enables developers to spend less time on plumbing and more time developing innovative, mission-critical software solutions.</li><li><strong>Reusability</strong> — Can reuse the codified form/scripts in various environment, without creating a new one. To spin up the same resource multiple times, we can reuse the existing scripts.</li></ul><p>So IaC gives all the benefits such as <strong>consistency</strong>, <strong>accelerating</strong>, <strong>re-usability</strong>, <strong>extensibility</strong>, all were the major cause in the traditional approach.</p><p>Let’s formally define I<strong>nfrastructure as Code</strong></p><p>This is how Microsoft defines Infrastructure as Code (IaC) — It is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using the same versioning as DevOps team uses for source code. Like the principle that the same source code generates the same binary, the IaC model generates the same environment every time it is applied. IaC is a key DevOps practice and is used in conjunction with continuous delivery.</p><p><strong>Tools available for IaC</strong></p><ul><li>AWS CloudFormation</li><li>Azure Resource Manager</li><li>Google Cloud Deployment Manager</li><li>JUJU</li><li>Otter</li><li>Puppet</li><li>SaltStack</li><li>Vagrant</li><li>Terraform</li></ul><p><strong>Terraform </strong>is widely used and adopted by some of the leading cloud technologies like Azure, AWS, and GCP.</p><p>The below diagram explains the flow of IaC considering Terraform as a language for creating Infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQ-oZUj0lS5Xk1Jap_zOSA.png" /><figcaption>Terraform on GCP</figcaption></figure><p>Curious to know how does a sample Terraform code looks like? Here is my GitHub <a href="https://github.com/iAshishHere/Terraform_with_GCP_VPC_Script">repository</a>.</p><p>Source — Microsoft, Hashicorp, IBM</p><p><strong>Ever thought of validating Terraform variables?</strong></p><p><a href="https://medium.com/@iashishhere/variables-validations-in-terraform-95a12542aa3a">Variables Validations in Terraform</a></p><p><strong>Contact me @ iashishhere@gmail.com<br>Linkedin: </strong><a href="https://www.linkedin.com/in/iashishhere/">https://www.linkedin.com/in/iashishhere</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Subscribe to </strong><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumprebanner"><strong>FAUN topics</strong></a><strong> and get your weekly curated email of the must-read tech stories, news, and tutorials </strong>🗞️</p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=917d245ce35e" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/infrastructure-as-code-917d245ce35e">Infrastructure as Code</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>