<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Jimenez Garcia on Medium]]></title>
        <description><![CDATA[Stories by Daniel Jimenez Garcia on Medium]]></description>
        <link>https://medium.com/@danieljimgarcia?source=rss-1d47a9a98035------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Bc4m0hnMEKSNz74s</url>
            <title>Stories by Daniel Jimenez Garcia on Medium</title>
            <link>https://medium.com/@danieljimgarcia?source=rss-1d47a9a98035------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:30:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danieljimgarcia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Start the year by joining one of these 5 free online tech events]]></title>
            <link>https://medium.com/@danieljimgarcia/start-the-year-by-joining-one-of-these-5-free-online-tech-events-d6c4438df78e?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6c4438df78e</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[vuejs]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 09:24:44 GMT</pubDate>
            <atom:updated>2024-01-12T10:07:13.287Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B5Myk7iqqqDpmYn5" /><figcaption>Photo by <a href="https://unsplash.com/@matthewosborn?utm_source=medium&amp;utm_medium=referral">Matthew Osborn</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>What better way to start on 2024 than by joining your fellow colleagues on sharing insights, discussing hot topics and learning about the latest trends?</p><p>Okay, there are <em>many ways</em>. But if you are recharged after the holiday break and want to start the year head-on or simply check something new, take a look at these events. You might find something of your interest! And the best part is that they are both online and free to join.</p><p>And don’ forget the resources at the end to get an idea on what’s coming through the year!</p><h4>1. Rust day — Jan 24, 2024</h4><p>Online event organised by <a href="https://www.wearedevelopers.com/">WeAreDevelopers</a> as part of their <a href="https://www.wearedevelopers.com/live">LIVE</a> online events, dedicated to Rust.</p><blockquote>Discover the power and performance of this modern programming language through presentations including a Q&amp;A with Rust experts. Don’t miss out on this opportunity to expand your skills in one of the fastest-growing programming languages.</blockquote><p><a href="https://www.wearedevelopers.com/event/rust-day-january-2024">Save your spot: Rust Day</a></p><h4>2. Vue.js Nation Conference — Jan 24–25, 2024</h4><p>Full online event with some of the greatest minds within the Vue.js ecosystem, including creator Evan You. More than 20 talks over 2 days, plus the opportunity of live Q&amp;A. Check it out on:</p><p><a href="https://vuejsnation.com/">https://vuejsnation.com/</a></p><h4>3. Test Exchange — Jan 25, 2024</h4><p>Community event organized by <a href="https://www.ministryoftesting.com/">Ministry of Testing</a>, open to anyone with a career, or simply an interest, in Software Testing.</p><blockquote>The Test Exchange is an opportunity to gather, get to know each other and explore ways of helping us all on our journey of advancing our software testing career. […] You can participate through video, voice, text, polls and break-out rooms. Or simply sit back and observe. And there’s no need to gather all the information as we’ll make the recording, AI summary, notes, transcript and chat history available to explore afterwards.</blockquote><p><a href="https://www.ministryoftesting.com/events/test-exchange-january-2024">Test Exchange</a></p><h4>4. Conf42 DevOps — Jan 25, 2024</h4><p>Organised by Conf42, this is a 100% online event with 69 talks organised on multiple tracks such as AI, Cloud, Data, Kubernetes or Culture. Check it out!</p><p><a href="https://www.conf42.com/devops2024">Conf42: DevOps</a></p><h4>5. DevOps day — Feb 7, 2024</h4><p>Another event organised from by <a href="https://www.wearedevelopers.com/">WeAreDevelopers</a> as part of their <a href="https://www.wearedevelopers.com/live">LIVE</a> online events, this one focused around DevOps</p><blockquote>Get recent insights into a great combination of cultural philosophies, work practices, and tools that will increase your ability to deliver applications and services at a faster pace than anyone else.</blockquote><p><a href="https://www.wearedevelopers.com/event/devops-day-february-2024">Save your spot: DevOps Day</a></p><h4>Want more?</h4><p>Does the idea sound interesting, but none of the events above are really of your interest? Perhaps the time doesn’t suit?</p><p>Check out the pages below which either organise or track events through the entire year!</p><ul><li>Calendar with 800+ developer conferences: <a href="https://dev.events/">https://dev.events/</a></li><li>Another calendar tracking 250+ conferences: <a href="https://confs.tech/">https://confs.tech/</a></li><li>Python community calendar: <a href="https://pycon.org/#calendar">https://pycon.org/#calendar</a></li><li>WeAreDevelopers LIVE events: <a href="https://www.wearedevelopers.com/live">https://www.wearedevelopers.com/live</a></li><li>Conf42 online events: <a href="https://www.conf42.com/">https://www.conf42.com/</a></li></ul><p>Hope this was helpful and you find something of interest through the year!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6c4438df78e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform hidden gems! Secret rotation with time_rotating]]></title>
            <link>https://medium.com/cloud-native-daily/terraform-hidden-gems-secret-rotation-with-time-rotating-72ae8683ef7f?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ae8683ef7f</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 16:38:30 GMT</pubDate>
            <atom:updated>2023-08-29T12:17:48.258Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraform Hidden Gems! Secret Rotation with time_rotating</h3><h4>Secret management in IaC: Learn how to manage secrets using the time_rotating resource.</h4><p>Over the years Terraform amassed a large number of features across its language and providers. As teams leverage Terraform to solve typical and not-so-typical infrastructure as code problems, it’s easy to overlook some of those features. Let’s take a look at some of them, starting with the time_rotating resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wmn-wyg76VX2veAF" /><figcaption>Photo by <a href="https://unsplash.com/@neonbrand?utm_source=medium&amp;utm_medium=referral">Kenny Eliason</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Secret management is a topic that any team implementing an infrastructure as code strategy will come across. Sooner or later you will find some of the infrastructure managed by Terraform requires and/or generates secrets. And this brings its own set of considerations and challenges, like being able to rotate those secrets.</p><p>This is where the time_rotating <a href="https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/rotating.html">Terraform resource</a> can be of help.</p><h4>How the time_rotating resource works</h4><p>In itself, this is nothing but a simple resource that forces a recreation of itself whenever a given period has passed. Once recreated, it will stay put until the given period passes again, and so on.</p><p>For example, the following resource will be recreated every 10 days:</p><pre>resource &quot;time_rotating&quot; &quot;secrets_rotation&quot; {<br>  rotation_days = 10<br>}</pre><p>This means after creating the resource, there will be no changes on every terraform plan for the next 10 days. Once 10 days since its creation have passed, the next terraform plan will show the resource needs to be recreated.</p><p>This is easier to see with a shorter period, like rotating_minutes = 5.</p><ul><li>If you run terraform plan before 5 minutes since its creation, you will see no changes.</li><li>But if you run terraform plan after 5 minutes since its creation, you will see:</li></ul><pre>Terraform will perform the following actions:<br><br>  # time_rotating.secrets_rotation will be created<br>  + resource &quot;time_rotating&quot; &quot;secrets_rotation&quot; {<br>      + day              = (known after apply)<br>      + hour             = (known after apply)<br>      + id               = (known after apply)<br>      + minute           = (known after apply)<br>      + month            = (known after apply)<br>      + rfc3339          = (known after apply)<br>      + rotation_minutes = 5<br>      + rotation_rfc3339 = (known after apply)<br>      + second           = (known after apply)<br>      + unix             = (known after apply)<br>      + year             = (known after apply)<br>    }<br><br>Plan: 1 to add, 0 to change, 0 to destroy.</pre><h4>How you can use it to rotate secrets</h4><p>The power of the time_rotating resource comes when you combine with other resources, as a dependency, so recreating the time_rotating forces a recreation of the dependent resource.</p><p>This is easier seen with a couple of examples. Let’s get down to it.</p><p>Imagine you are creating an Azure AD application, with its own <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/resources/application_password">application password</a>. You can rotate that password after 10 days by combining a time_rotating resource with the azuread_application_password resource:</p><pre>resource &quot;time_rotating&quot; &quot;password_rotation&quot; {<br>  rotation_days = 10<br>}<br><br>resource &quot;azuread_application_password&quot; &quot;app_password&quot; {<br>  application_object_id = azuread_application.my_application.object_id<br>  rotate_when_changed = {<br>    rotation = time_rotating.password_rotation.id<br>  }<br>}</pre><p>We are forcing the password to be regenerated by setting the rotate_when_changed property of the azuread_application_password resource to the id of the time_rotating resource. <br>After the desired period, terraform recreates the time_rotating resource, which causes it to get a new id, which in turn forces the AD application password to be regenerated:</p><pre>Terraform will perform the following actions:<br><br>  # azuread_application_password.app_password must be replaced<br>-/+ resource &quot;azuread_application_password&quot; &quot;app_password&quot; {<br>      + display_name          = (known after apply)<br>      ~ end_date              = &quot;2025-08-28T15:05:32Z&quot; -&gt; (known after apply)<br>      ~ id                    = &quot;24478106-fee9-4b2d-bcba-ca01d49833f1/password/a3eda3cb-6c4b-4f0c-8d44-51c37a448eb6&quot; -&gt; (known after apply)<br>      ~ key_id                = &quot;a3eda3cb-6c4b-4f0c-8d44-51c37a448eb6&quot; -&gt; (known after apply)<br>      ~ rotate_when_changed   = { # forces replacement<br>          - &quot;rotation&quot; = &quot;2023-08-28T15:05:30Z&quot;<br>        } -&gt; (known after apply) # forces replacement<br>      ~ start_date            = &quot;2023-08-28T15:05:32Z&quot; -&gt; (known after apply)<br>      ~ value                 = (sensitive value)<br>        # (1 unchanged attribute hidden)<br>    }</pre><p>Let’s see a second example. Imagine you are generating a <a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/token">Databricks Token</a> using Terraform. You could similarly rotate that token by say 5 days, combining a time_rotating resource with a databricks_token resource:</p><pre>resource &quot;time_rotating&quot; &quot;token_rotation&quot; {<br>  rotation_days = 5<br>}<br><br>resource &quot;databricks_token&quot; &quot;cicd_token&quot; {<br>  comment = &quot;CICD environment token (created: ${time_rotating.token_rotation.rfc3339})&quot;<br>  # Token is valid for 6 days but is rotated after 5 days as per time_rotating above.<br>  # This assumes you run `terraform apply` very frequently, or at least daily!<br>  lifetime_seconds = 6 * 24 * 60 * 60<br>}</pre><p>In this case, we are setting the dependency by using time_rotating to build the comment of the token. Since a change in the comment forces a recreation of the token, after 5 days the time_rotating resource changes, which forces a change on the comment, which in turn forces terraform to recreate the token.</p><p>It’s also worth pointing out that the token lifetime needs to be a little longer than the time_rotating period. How much longer depends on how often you run terraform apply. For example, if the rotation is set as 5 days while the token expires in 6 days, you then have a window of 24h in which to run terraform apply and generate a new token before the old one expires.</p><blockquote>This is one of the reasons why tokens are troublesome from a security perspective (fixed lifetime that cant be invalidated once created), and why short-lived tokens are preferred when possible. <br>In the particular case of databricks tokens for CICD, you might want to consider a service principal whose password you rotate as part of your infra as code, generating a short-lived token on-demand using the principal credentials. See <a href="https://github.com/databricks/run-notebook/#use-the-service-principal-in-your-github-workflow">https://github.com/databricks/run-notebook/#use-the-service-principal-in-your-github-workflow</a></blockquote><p>As the examples demonstrate, the time_rotatingresource can be a simple and built-in way to automate the rotation of secrets managed by Terraform.</p><p>Happy coding!</p><h4>Further Reading:</h4><ul><li><a href="https://gethelios.dev/blog/serverless-observability/?utm_source=medium&amp;utm_medium=cloud+native+daily">Serverless observability, monitoring, and debugging explained</a></li><li><a href="https://gethelios.dev/opentelemetry-a-full-guide/?utm_source=medium&amp;utm_medium=cloud+native+daily">OpenTelemetry: A full guide</a></li><li><a href="https://aws.plainenglish.io/top-tools-for-aws-lambda-monitoring-8ea51b1f095c">Top Tools for AWS Lambda Monitoring</a></li><li><a href="https://gethelios.dev/blog/combinining-opentelemetry-traces-with-prometheus-metrics-to-build-a-powerful-alerting-mechanism/?utm_source=medium&amp;utm_medium=referral&amp;utm_campaign=cloud+native+daily">Combining OTel and Prometheus metrics for alerting machine</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ae8683ef7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloud-native-daily/terraform-hidden-gems-secret-rotation-with-time-rotating-72ae8683ef7f">Terraform hidden gems! Secret rotation with time_rotating</a> was originally published in <a href="https://medium.com/cloud-native-daily">Cloud Native Daily</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We save thousands every month by using incremental snapshots with our Velero backups]]></title>
            <link>https://medium.com/@danieljimgarcia/we-save-thousands-every-month-by-using-incremental-snapshots-with-our-velero-backups-c1a39dd64382?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1a39dd64382</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[velero]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[finops]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Thu, 10 Aug 2023 17:10:25 GMT</pubDate>
            <atom:updated>2023-08-10T17:11:37.101Z</atom:updated>
            <content:encoded><![CDATA[<h4>With a large enough number of Kubernetes workloads and volumes, the cost of your backups can easily get out of control. The ability to enable incremental snapshots as part of your Velero backups can have a huge impact on the cost of your backup strategy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kDmvZ8oWXy9dOyxb" /><figcaption>Photo by <a href="https://unsplash.com/@joshappel?utm_source=medium&amp;utm_medium=referral">Josh Appel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We use Kubernetes in Azure’s AKS as the basis of our development platform, and we have adopted tools such as <a href="https://velero.io/">Velero</a> in order to make it easier for teams and services to follow <a href="https://learn.microsoft.com/en-us/azure/aks/operator-best-practices-storage#secure-and-back-up-your-data">best practices</a> and be production ready. In the particular case of Velero, it allows us to ensure a consistent backup strategy that even allows for optional per-service/per-cluster customization.</p><h4>First some context…</h4><p>Typically, we will setup a minimum of 3 Velero <a href="https://velero.io/docs/v1.11/backup-reference/#schedule-a-backup">Backup Schedules</a> on each Kubernetes cluster:</p><ul><li>a weekly one</li><li>a nightly one</li><li>a frequent one, set to a configurable frequency in minutes/hours.</li></ul><p>There can be some customization done per cluster, but that’s roughly the baseline number of schedules. Depending on how critical a particular workload is, it will be included in some (or all) of these strategies. Any stateful workload that relies on Kubernetes Persistent Volumes (PV) will have the underlying Azure Disk backed up by Velero via <a href="https://velero.io/docs/v1.11/examples/#snapshot-example-with-persistentvolumes">Azure Disk Snapshots</a>.</p><h4>This is costing too much!</h4><p>In a cluster with a large number of critical workloads that rely on volumes, the number of disk snapshots can very quickly grow! Imagine 40 such workloads, with a couple of PVs each. Let’s say the frequent backups run every 2h and you keep them for 2 days, while you keep the weekly/nightly for a week. This means for each individual PV you have</p><ul><li>24 frequent snapshots</li><li>7 nightly snapshots</li><li>1 weekly snapshot</li></ul><p>That’s 32 snapshots per volume. Considering 40 workloads with 2 volumes each, thats 40*2*32 = <strong>2,560 disk snapshots!</strong></p><p>When each one of these snapshots is a full disk snapshot, their combined daily costs can be significant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rThJpHhE5ugp0rjLcBCa_A.png" /><figcaption>Daily costs of the snapshots created by the backup strategies when using full disk snapshots</figcaption></figure><p>We are burning ~174$ per day, more than 5,200$ per month!</p><h4>Lets enable incremental snapshots</h4><p>Velero allows you to use <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/disks-incremental-snapshots?tabs=azure-cli">incremental snapshots</a> in Azure when backing up the Azure disks associated with Kubernetes volumes. All you need is to enable the incremental flag within your <a href="https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/volumesnapshotlocation.md">Velero’s VolumeSnapshotLocation</a>.</p><pre>apiVersion: velero.io/v1<br>kind: VolumeSnapshotLocation<br>metadata:<br>  name: default<br>  namespace: velero<br>spec:<br>  provider: velero.io/azure<br>  config:<br>    resourceGroup: my-resource-group<br>    # Enable incremenal snapshots!<br>    incremental: &quot;true&quot;</pre><blockquote>Note by default full snapshots are taken unless incremental is explicitly enabled.</blockquote><p>The effects on the costs are significant, given you will pay just for the amount of data that changed since the last snapshot. This was the effect of enabling incremental backups on the same environment we saw the costs above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hqhoGm9oRCRQewUIJhYyCA.png" /><figcaption>Cost savings after enabling incremental snapshots</figcaption></figure><p>From a peak of ~174$ per day, costs plummeted to ~20$ per day. That’s <strong>88.5% savings </strong>compared to the cost of the full snapshots!</p><blockquote>As mentioned before, the savings depend on the rate of change of your data, ie how much data changes between each snapshot. But unless most of your data is constantly changing, you should see significant savings.<br>Of course, the total amount of these savings depends on the number of volumes and their size. Ie, if you have 2 volumes of 4Gb each, the cost of 32 full snapshots per volume is roughly 11$ per day and you might save 5-10$ per day.</blockquote><p>To top it all off, there are no other changes needed than the incremental flag! Backups and restores continued to <a href="https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/52#issuecomment-659008111">work like before</a>. It’s all handled behind the scenes by Velero and Azure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1a39dd64382" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Github actions for Databricks CI/CD]]></title>
            <link>https://medium.com/@danieljimgarcia/github-actions-for-databricks-ci-cd-ab6335bfd521?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab6335bfd521</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cicd]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Mon, 07 Aug 2023 15:57:53 GMT</pubDate>
            <atom:updated>2023-08-09T08:24:02.281Z</atom:updated>
            <content:encoded><![CDATA[<h3>Databricks CI/CD using Github Actions</h3><h4>Databricks recommends the usage of repos as part of their engineering best practices. And with repos comes the need to implement a development lifecycle that integrates and deploys code changes, <em>aka</em> CI/CD. This article shows how you can implement CI/CD using Github Actions and the Databricks CLI.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o3tqhPmo8Wq47m7b" /><figcaption>Photo by <a href="https://unsplash.com/@praveentcom?utm_source=medium&amp;utm_medium=referral">Praveen Thirumurugan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you read about the recommended engineering best practices from Databricks, you will soon realise leveraging Databricks repos is key to these best practices. For example, see the recommended best practices for <a href="https://docs.databricks.com/en/notebooks/best-practices.html">notebooks</a>, or the ones for <a href="https://www.databricks.com/blog/applying-software-development-devops-best-practices-delta-live-table-pipelines">DLT pipelines</a>. And the moment you add a Databricks repo, you need a CI/CD system in place to integrate and promote the changes!</p><p>For those of you using Github, it would be very attractive to build such CI/CD workflow using Github Actions. However you will find that the <a href="https://docs.databricks.com/en/dev-tools/ci-cd/ci-cd-github.html">official set of Github Actions</a> provided by Databricks is very limited! Luckily, you can implement you own actions by leveraging the Databricks CLI under the hood.</p><h3>Installing the Databricks CLI in your Github Actions runner</h3><p>The first step to interact with your Databricks workspace from your Github Actions workflow is to get the Databricks CLI installed in your runner.</p><p>Since the new Databricks CLI is a <a href="https://docs.databricks.com/dev-tools/cli/databricks-cli.html">standalone executable</a>, this is relatively straightforward, curl the executable and run the install script as part of your Github Actions workflow. See the “Install databricks CLI” step below:</p><pre>name: databricks-cicd<br>...<br>jobs:<br>  main:<br>    runs-on: ubuntu-latest<br>    ...<br>    steps:<br>      - name: Check out repository<br>        uses: actions/checkout@v3<br><br>      # Install databricks CLI as per https://docs.databricks.com/dev-tools/cli/databricks-cli.html<br>      - name: Install databricks CLI<br>        run: |<br>         curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh</pre><h3>Providing credentials to the Databricks CLI</h3><p>Now you need to provide the necessary credentials for the Databricks CLI to authenticate with your Databricks workspace, so you can interact with it.</p><p>There are multiple ways to <a href="https://docs.databricks.com/en/dev-tools/cli/databricks-cli.html#set-up-authentication">handle authentication</a>. To keep things simple:</p><ul><li>Let’s use a Databricks Personal Access Token (for either a user or a service principal), as per <a href="https://docs.databricks.com/en/dev-tools/auth.html#databricks-personal-access-token-authentication">the official docs</a>.</li><li>The token as well as the Databricks workspace URL would be stored as part of the Github Environment secrets and env variables, so they can be used within your Github Actions.</li></ul><blockquote>Note you could look at other <a href="https://docs.databricks.com/en/dev-tools/cli/databricks-cli.html#set-up-authentication">authentication strategies</a>. For example in Azure you could create a service principal with access to the workspace, storing the principal’s client id/secret as part of Github environment’s secret. Within your github actions, you could then create a short-lived Databricks token as in <a href="https://github.com/databricks/run-notebook/#use-the-service-principal-in-your-github-workflow">https://github.com/databricks/run-notebook/#use-the-service-principal-in-your-github-workflow</a></blockquote><p>The steps would be:</p><ol><li>Create a <strong>Github Environment</strong> within your Github repo.</li><li>Generate a <strong>Databricks token</strong> as per <a href="https://docs.databricks.com/en/dev-tools/auth.html#databricks-personal-access-token-authentication">the official docs</a>.</li><li>Save the Databricks token as a secret named DATABRICKS_TOKEN in the <a href="https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#environment-secrets">Github Environment</a>.</li><li>Get the <strong>Databricks workspace URL</strong> as per <a href="https://docs.databricks.com/en/workspace/workspace-details.html#workspace-url">the officinal docs</a>. Add the URL as an environment variable named DATABRICKS_HOST in the <a href="https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#environment-variables">Github Environment</a>.</li><li>Optionally add protection rules to the Github Environment. For example, you can <a href="https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#deployment-branches">limit which branches</a> are allowed to run Github Action workflows that use this environment.</li></ol><p>The end result for the Github Environment would look similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9970R9kPCC8X2jOwPrk9Zg.png" /><figcaption>Github Environment with the DATABRICKS_HOST and DATABRICKS_TOKEN needed for the Databricks CLI authentication</figcaption></figure><blockquote>Are you using Terraform? You can get all of this automated! Use the databricks_token resource of the <a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/token">Databricks provider</a> to generate (and even rotate) a token. Then use the github_repository_environment resource of the <a href="https://registry.terraform.io/providers/integrations/github/latest/docs/resources/repository_environment">Github provider</a> as well as the github_actions_environment_variable and github_actions_environment_secret to create and configure the Github Environment. It’s a great way of automating the setup of a Github Environment, codifying your security posture such as Databricks auth and/or Github protection rules.</blockquote><p>Assuming your Github Environment is named my-environment, you can then use it within a workflow in order to authenticate with the Databricks CLI. The key is to make sure that the step which runs the databricks CLI command has access to the DATABRICKS_HOST and DATABRICKS_TOKEN as environment variables.</p><p>See for example the “List databricks repos” step below:</p><pre>name: databricks-cicd<br>...<br>jobs:<br>  main:<br>    runs-on: ubuntu-latest<br>    # Name of the environment you created<br>    environment: my-environment<br>    ...<br>    steps:<br>      - name: Check out repository<br>        uses: actions/checkout@v3<br><br>      # Install databricks CLI as per https://docs.databricks.com/dev-tools/cli/databricks-cli.html<br>      - name: Install databricks CLI<br>        run: |<br>         curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh<br><br>      # List the databricks repos within the workspace whose credentials<br>      # are stored in the Github Environment &quot;my-environment&quot;<br>      - name: List databricks repos<br>        env:<br>          # Note the host is an &quot;environment variable&quot; in the Github Environment<br>          # while the token is a &quot;secret&quot; in the Github Environment<br>          DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}<br>          DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}<br>        run: |<br>          databricks repos list</pre><h3>Use the Databricks CLI in your Github Actions</h3><p>You now have everything you need in place to run any of the <a href="https://docs.databricks.com/en/dev-tools/cli/databricks-cli.html#cli-command-groups">Databricks CLI commands</a> as part of your Github Action workflows. Let’s see some typical examples next, and we will finish off by seeing how to build your own library of reusable custom actions</p><h4>Update a Databricks repo using Github Actions</h4><p>If you have a Databricks repo that points to your main branch, it’s very likely that you would want that repo to be automatically updated with the latest changes merged into your branch.</p><p>You could use the databricks repos update command and the path to the repo inside the Databricks workspace. For example, imagine you have a Databricks repo in a folder named main which points to the main branch of a Github repo named my-github-repo, so the path within your workspace looks like /Repos/main/my-github-repo.</p><blockquote>If you are not sure about the right path, run the databricks repos list command. The output contains the path to each repo</blockquote><p>You can then update such repo, pulling the latest changes committed in Github, adding the following step to your workflow:</p><pre>steps:<br>  ...<br>  # Update a databricks repo given its workspace path<br>  - name: Update databricks repo<br>    env:<br>      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}<br>      DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}<br>    run: |<br>      databricks repos update /Repos/main/my-github-repo --branch mainIf the databricks repo has dirty uncommitted files, the update will likely fail!</pre><blockquote>Note that if the Databricks repo has dirty uncommitted files, the update will fail! (Same as if you tried to pull latest changes from the Databricks UX)</blockquote><h4>Run a Databricks job using Github Actions</h4><p>Another typical example would be running a Databricks job after new code is merged to your main branch, probably after updating the Databricks repo as per the previous example.</p><p>You can run a job with the databricks jobs run-now command of the CLI.</p><ul><li>This is a little more complicated than the previous example, because the command expects the <em>job ID</em>, while most likely you would like to use the <em>job name</em>.</li><li>You can use the databricks jobs list command and extract the job ID from the JSON response using jq</li></ul><p>For example, if you have a job named my-job, you could run the job as part of your Github Action workflow by using a step like follows:</p><pre>steps:<br>  ...<br>  # Run a databricks job given its name<br>  - name: Run databricks job &quot;my-job&quot;<br>    env:<br>      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}<br>      DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}<br>      JOB_NAME: &#39;my-job&#39;<br>      WAIT_TIMEOUT: &#39;20m0s&#39;<br>    run: |<br>      JOB_ID=$(databricks jobs list | grep -w $JOB_NAME | awk -F &#39; &#39; &#39;{print $1}&#39;)<br>      databricks jobs run-now $JOB_ID --timeout $WAIT_TIMEOUT</pre><blockquote>You could “fire and forget”, without waiting for the job to be finished, by using the --no-wait parameter with the databricks jobs run-now command.</blockquote><h4>Build a library of reusable Custom Actions</h4><p>In most teams/companies, you will likely have multiple Databricks workspaces and/or many Github repos used by those workspaces. And this means multiple Github Action workflows as part of your overall Databricks CI/CD strategy.</p><p>As we have seen in the article, it is straightforward enough to run databricks CLI commands as part of the Github Actions workflow. However, with multiple workflows, the copy&amp;paste of those steps <em>could </em>become a headache. One option you can consider is to extract the most common steps into your own <a href="https://docs.github.com/en/actions/creating-actions/about-custom-actions">custom actions</a>, slowly building a small library of actions tailored for your team/company.</p><p>For example, let’s create an action to install the CLI. Within your repo, create a folder .github/actions/databricks-cli-install. Inside create a file action.yml with the following contents:</p><pre>name: &#39;databricks-cli-install&#39;<br>description: &#39;Install latest version of the databricks CLI&#39;<br>runs:<br>  using: &quot;composite&quot;<br>  steps:<br>    - shell: bash<br>      run: |<br>          curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh</pre><blockquote>For the sake of simplicity, we are creating these custom actions within the same repo. However you can create them on a separate repo, including private ones. See the <a href="https://docs.github.com/en/actions/creating-actions/about-custom-actions">official Github docs</a>.</blockquote><p>Then create an action to update a Databricks repo. Similarly create a folder .github/actions/databricks-repo-update and an action.yml file inside like:</p><pre>name: &#39;databricks-repo-update&#39;<br>description: &#39;Update a databricks repo given its path&#39;<br>inputs:<br>  databricksRepoPath:<br>    description: &#39;The path of the databricks repo within the databricks workspace&#39;<br>    required: true<br>  branch:<br>    description: &#39;The git branch the databricks repo should be updated to&#39;<br>    default: &#39;main&#39;<br>runs:<br>  using: &quot;composite&quot;<br>  steps:<br>    - shell: bash<br>      env:<br>        DATABRICKS_HOST: ${{ env.DATABRICKS_HOST }}<br>        DATABRICKS_TOKEN: ${{ env.DATABRICKS_TOKEN }}<br>        REPO_PATH: ${{ inputs.databricksRepoPath }}<br>        BRANCH: ${{ inputs.branch }}<br>      run: |<br>        databricks repos update $REPO_PATH --branch $BRANCH</pre><p>Finally, you can use these 2 custom actions in your workflow as in:</p><pre>steps:<br>  ...<br><br>  - name: Install databricks CLI<br>    uses: ./.github/actions/databricks-cli-install<br><br>  - name: Update databricks main repo<br>    uses: ./.github/actions/databricks-repo-update<br>    env:<br>      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}<br>      DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}<br>    with:<br>      databricksRepoPath: /Repos/main/my-github-repo<br></pre><blockquote>For a full workflow example, see the following gist: <a href="https://gist.github.com/DaniJG/b21170482545ad8c93874d164fd97a90">https://gist.github.com/DaniJG/b21170482545ad8c93874d164fd97a90</a></blockquote><p>As you have seen, you can get very far combining Github Actions with the Databricks CLI in order to implement your CI/CD strategy. Hope you found the article useful!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab6335bfd521" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can Microsoft Fabric complement your Databricks data platform?]]></title>
            <link>https://medium.com/@danieljimgarcia/can-microsoft-fabric-complement-your-databricks-data-platform-7afa6b873737?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/7afa6b873737</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Wed, 02 Aug 2023 10:07:07 GMT</pubDate>
            <atom:updated>2023-08-08T11:43:38.615Z</atom:updated>
            <content:encoded><![CDATA[<h3>Can Microsoft Fabric complement your Databricks platform?</h3><h4>This is one of the questions we made ourselves when Microsoft announced their new product! In this article, we explore what the launch of Microsoft Fabric meant for our Databricks-based data platform and the different options we considered in response to this new product.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F9C_nYgtDaiiLykF" /><figcaption>Photo by <a href="https://unsplash.com/@sortino?utm_source=medium&amp;utm_medium=referral">Joshua Sortino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>More than a year ago, we decided to leverage Databricks and the Delta Lake as the core for our data platform. Fast-forward to Microsoft Build 2023, when Microsoft <a href="https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/">announces their new data platform</a> Microsoft Fabric. We were positively surprised by Fabric’s highlighted key points and design choices, many of which overlap with the reasons why we decided to use Databricks in the first place, such as:</p><ul><li>Separation of compute and storage, underpinned by a Data Lake</li><li>Data Lakehouse supported by a centralized unified catalog, governance and security</li><li>Multiple ways of building and consuming data products: Spark, SQL, ML, PowerBI, …</li></ul><figure><img alt="Microsoft Fabric features. Source: Microsoft." src="https://cdn-images-1.medium.com/max/1024/1*XTUASzbGpY9ca4RleEa7pw.png" /><figcaption>Microsoft Fabric features. Source: Microsoft.</figcaption></figure><p>since we use Azure, our curiosity was piqued. Should we re-evaluate our platform design choices? Are they even compatible, given the similarities between Databricks and Fabric? Should this be another tool in our toolbox? We could see a number of options in which our platform could leverage Fabric, ranging from doing nothing (for now) to a full-on migration.</p><p>Let’s take a look at the options we considered, and the initial decision we made.</p><h3>Options considered</h3><h4>Doing nothing</h4><p>We have heard similar claims about past Microsoft products, such as Azure Synapse Analytics (which we tested and abandon).</p><p>Thus waiting seems a sensible response. Let’s give Fabric some time to mature, for its hype to fade away and for teams to share some testimonies. At some point in the future, we can then re-evaluate and decide if things are different enough so we should adopt another response.</p><h4>Integrate Fabric with our Delta Lake</h4><p>Fabric understands Delta Tables, and is able to query those in existing Azure’s ADLS Gen2 accounts. To do so, Fabric provides the concept of a <em>shortcut. </em>With a shortcut, users are able access a Delta Table in a separate ADLS Gen2 account as if this was stored within Fabric’s OneLake. (Very much like a file shortcut within your laptop)</p><p>The data will be available to use within the various Fabric services such as spark notebooks, SQL endpoints or <a href="https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts-adb-quickstart">PowerBI</a>.</p><figure><img alt="Fabric shortcuts to Delta Tables in ADLS Gen2. Source: Microsoft" src="https://cdn-images-1.medium.com/max/1024/0*dvpYuGgTn9Oltmo2.png" /><figcaption>Fabric shortcuts to Delta Tables in ADLS Gen2. Source: Microsoft</figcaption></figure><blockquote>For more info on Shortcuts, see <a href="https://blog.fabric.microsoft.com/en-US/blog/using-azure-databricks-with-microsoft-fabric-and-onelake/">https://blog.fabric.microsoft.com/en-US/blog/using-azure-databricks-with-microsoft-fabric-and-onelake/</a> and the official docs <a href="https://learn.microsoft.com/en-us/fabric/onelake/create-adls-shortcut">https://learn.microsoft.com/en-us/fabric/onelake/create-adls-shortcut</a></blockquote><p>This provides us with a nice and low-risk integration point. We can continue leveraging Databricks in the same way we do, while we slowly develop some use cases in Fabric or get some power users to start leveraging Fabric.</p><blockquote>It’s worth noting that each shortcut will auto-discover the Delta Table metadata and save it to Fabric’s Lakehouse catalog. However this is a new metadata/catalog that lives inside Fabric, and is fully disconnected from the metadata/catalog used by your Databricks instance.</blockquote><blockquote>For now, I am only aware of a 1-way catalog migration to Fabric, with no known way to sync/share the catalog. See <a href="https://murggu.medium.com/migrating-spark-catalog-to-fabric-lakehouse-cc8c14f0f0e1">https://murggu.medium.com/migrating-spark-catalog-to-fabric-lakehouse-cc8c14f0f0e1</a></blockquote><h4>Integrate Databricks with Fabric’s One Lake</h4><p>There is also the option to integrate in the opposite direction. Rather than having Fabric read the ADLS Gen2 storage used by Databricks, we can have Databricks read/write directly into Fabric’s OneLake.</p><p>This works thanks to both ADLS Gen2 and Fabric’s OneLake being compatible with the Azure Blob Filesystem (ABFS) driver used by Databricks.</p><figure><img alt="Databricks directly reading/writing to Fabric’s OneLake. Source: Microsoft" src="https://cdn-images-1.medium.com/max/1024/0*p4UbNmyrgmbi-_FA.png" /><figcaption>Databricks directly reading/writing to Fabric’s OneLake. Source: Microsoft</figcaption></figure><blockquote>For more info on using OneLake with Databricks, see <a href="https://blog.fabric.microsoft.com/en-US/blog/using-azure-databricks-with-microsoft-fabric-and-onelake/">https://blog.fabric.microsoft.com/en-US/blog/using-azure-databricks-with-microsoft-fabric-and-onelake/</a> and the official docs <a href="https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks">https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks</a></blockquote><p>This is a very similar approach to the one we saw in the previous section. However this would play better if you are considering Fabric for most of your use cases like your ETL and BI, while you use Databricks for additional use cases like ML.</p><p>By choosing this option, your main use cases will benefit from Fabric’s centralised features across its various services while you can still leverage Databricks in specific use cases. This could also be a good option if you are also planning to move some of your use cases from Databricks into Fabric.</p><h4>Migrate from Databricks to Fabric</h4><p>The last option would be to fully embrace Microsoft Fabric and rearchitect our data platform around it, abandoning Databricks in the process (or at least leaving Databricks for a very small number of non-critical use cases, using the OneLake integration we saw earlier)</p><p>Whether this makes sense or not, depends largely on your specific needs. There are a lot of similarities and overlap between Fabric and Databricks. But there are also significant differences in the features they offer, their pricing or their maturity. Let’s consider 2 sample scenarios:</p><ul><li>You are already leveraging ADF for ETLs and use Databricks mostly for ad-hoc analysis and data-science. In this case, you might want to consider moving everything into Fabric</li><li>You are a heavy user of Databricks’ Delta Lakehouse, relying on Delta Live Tables for data pipelines, using Unity as your centralized catalog/governance, orchestrating multiple Databricks jobs, etc. In this case, you will be less inclined to consider a full migration to Fabric!</li></ul><p>Of course your team/needs are unlikely to fit exactly in either of those 2 boxes, so you will need to do your own assessment.</p><h3>Hands on with Fabric</h3><p>Let’s take Fabric for a spin and start to get a sense on what it can do. This section is a small portion of what we tried and explored, but at least will give you a starting point!</p><h4>Enabling the trial</h4><p>Currently Microsoft offers a free trial for anyone to try out Fabric free of charge. You can start the trial by following these steps:</p><ul><li>An admin of the PowerBI tenant needs to enable Fabric, as per <a href="https://learn.microsoft.com/en-us/fabric/admin/fabric-switch#enable-for-your-tenant">Enable Microsoft Fabric for your organization </a>.</li><li>Login with your Azure user account and navigate to <a href="https://app.fabric.microsoft.com/">https://app.fabric.microsoft.com</a></li><li>Start the free trial as per <a href="https://learn.microsoft.com/en-us/fabric/get-started/fabric-trial">Fabric (preview) trial </a>. This creates a free trial Fabric capacity within the tenant. Your personal workspace “My workspace” is upgraded to use this free capacity</li></ul><blockquote>Note the trial is independent per user. Each user has 60 days and can use all Fabric resources</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jq71QHjexpiaAigB4GN4HA.png" /><figcaption>Microsoft’s Fabric home page</figcaption></figure><p>Now within your “My workspace” you should be able to create any fabric resources such as notebooks.</p><h4>Accessing Databricks Delta Tables via shortcuts</h4><p>Fabric is organized within workspaces and Lakehouses, see the <a href="https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview#organizational-structure-of-onelake-and-lakehouse">official docs</a>. If you don’t have one, you will need to create a Lakehouse within a workspace, for example the workspace created after enabling the trial.</p><p>Once you have a Lakehouse, you can then follow these docs in order to create a shortcut to a Delta Table within one of your ADLS Gen2 accounts, see <a href="https://learn.microsoft.com/en-us/fabric/onelake/create-adls-shortcut">https://learn.microsoft.com/en-us/fabric/onelake/create-adls-shortcut</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnvGsUO2t7Rz8A0lGprPvw.png" /><figcaption>Delta Table queried in Fabric via shortcut to ADLS Gen2</figcaption></figure><p>There are a few things worth noting though:</p><ul><li>When creating a shortcut, the path to the existing Delta Table folder is case sensitive and requires a trailing slash. Ie, in my case that&#39;s /gold/main/DimProduct/, meaning neither /gold/main/DimProduct nor /gold/main/dimproduct/ would work.</li><li>You can only create table shortcuts to Tables. Ie, while in Databricks/spark you might save as parquet using something like df.write.format(“parquet”).save(… , you are not able to import these as a table shortcut</li><li>However you can create a shortcut to raw files such as CSV or JSON files (or folders containing raw files), which will be available as blobs within Fabric. Each of these files is then available via ABFS within Fabric’s spark notebooks, and can also be ingested/transformed via Fabric’s ADF</li></ul><h4>Consuming imported Delta Tables</h4><p>Once you have data available as a table within Fabric’s Lakehouse, you can start using the various Fabric services to work with that data.</p><p>When you create a spark notebook, you can browse your Lakehouse and the tables within the Lakehouse on the sidebar. From that sidebar, you can even generate a spark.read command to read the table into a DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*poqmGF5CTP39lS4Re6ObPQ.png" /><figcaption>Querying inside a Spark notebook a Delta Table imported into Fabric via a shortcut</figcaption></figure><p>You can also get the ABFS path to any table by right clicking the table and checking its properties</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E4VMamALcWmiqvBb0g1NHA.png" /><figcaption>Checking the ABFS path of any table/blob within a Fabric Lakehouse</figcaption></figure><p>It’s also worth noting that every Fabric Lakehouse has a SQL endpoint associated that exposes the same tables via a SQL interface (including tables imported via a shortcut)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lZfuc3Sz4A32EXlxuA8P3A.png" /><figcaption>Querying the imported Delta Table through the Lakehouse SQL endpoint</figcaption></figure><p>To finish off this section, let’s take a quick look at PowerBI.</p><ul><li>Each Lakehouse has a “default dataset” you can use in PowerBI. This is the Dataset you can see when switching to the SQL endpoint in the Lakehouse.</li><li>A Dataset essentially makes Lakehouse tables available through a SQL endpoint, which tools like PowerBI can consume)</li><li>Using PowerBI you can create reports using any of the available datasets and their tables</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ynm0RS1FlDbhm0Zo9vZSkA.png" /><figcaption>Quick and dirty PowerBI report that uses a Delta Table imported into Fabric</figcaption></figure><h3>Good, bad and ugly highlights</h3><p>Good</p><ul><li>Ease of use! It is straightforward to query and analyse datasets using various tools (SQL, Spark, PowerBI, etc), even when these come from different storages and sources like multiple data lakes.</li><li>PowerBI can directly work with data from the Lakehouse, including data available via shortcuts. No need to copy into PowerBI or connect it in any other way.</li><li>Speaking about PowerBI, it <em>could</em> provide a lower entry-point cost, see <a href="https://data-mozart.com/microsoft-fabric-beyond-hype-and-marketing/">https://data-mozart.com/microsoft-fabric-beyond-hype-and-marketing/</a></li><li>The ability to share items and integrate with other MS applications could be very powerful. Easily share links to reports and notebooks, integrate with Excel and PowerPoint, etc could all be time-savers for certain users</li><li>The different integration options with existing data lakes allows teams to get the best of both worlds without having to fully commit to Fabric. And you can keep re-evaluating how much you use Fabric vs other tools such as Databricks, without ever having to pick just one.</li></ul><p>Bad</p><ul><li>While the no-code/low-code and UX-based approaches make it easy to get started, it still needs better support for automation. I really <em>need</em> to be able to create and manage elements such as Capabilities, Lakehouses, Shortcuts or SQL datasets using an infra as code tool such as Terraform.</li><li>Pricing and licensing can be confusing, particularly if you are not used to the PowerBI model. See <a href="https://learn.microsoft.com/en-gb/fabric/enterprise/licenses">https://learn.microsoft.com/en-gb/fabric/enterprise/licenses</a></li></ul><p>Ugly</p><ul><li>While expected given its preview state, some of the features would fail with little information or workaround options. A specific Delta Table would fail to be included in the SQL dataset, the PowerBI summary report would crash during generation, a report would fail to be embedded in PowerPoint, some items can’t be shared, …</li><li>Its very easy to access and consolidate data from various sources, but its all done through the UX and it defaults to creating connections using your own personal azure account. It can be changed later (also through the UX), but combined with the lack of automation support, this can easily become on a full time admin job.</li><li>The git integration is limited for now to git repos hosted within Azure DevOps</li></ul><h3>Our decision</h3><p>We took a step back to evaluate the main use cases we are catering for with our data platform, and how Databricks is currently enabling (or not) those use cases.</p><ul><li>We are quite happy using Delta Live Tables to build and orchestrate our ETLs</li><li>We have an automated and self-service data platform that uses Terraform so any team/project can create and manage their Databricks workspace(s)</li><li>We have enabled data analysis and data science within our data platform, primarily through notebooks and jobs</li><li>Our Data Mesh implementation is emerging around the Databricks ecosystem, using tools such as the Unity Catalog. BI tools and specifically PowerBI, are a common request for us.</li></ul><p>A full migration to Fabric is out of the table, given its preview bleeding-edge status and its limitations. But given the similarities and compatibilities between our Databricks-based data platform and Microsoft Fabric, we decided to cautiously add Fabric into our toolset.</p><p>To do so, we:</p><ul><li>will favour the integration via shortcuts</li><li>will specifically consider this path for use cases that would benefit from Fabric’s strengths like its PowerBI integration or the centralized access/governance, without being mission-critical use cases.</li><li>will work closely with the users benefitting from these use cases. They are going to help us better understand/learn Fabric!</li></ul><p>The idea is that we can continue largely using our data platform in the same way we do, while we add Fabric into the mix without becoming a strong dependency for us. Over time, feedback and learnings will be gathered that could lead us to re-evaluate our position either moving further from or closer to Fabric.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7afa6b873737" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use spark libraries and their Python API in Databricks]]></title>
            <link>https://medium.com/@danieljimgarcia/use-spark-libraries-and-their-python-api-in-databricks-e60b5977c00c?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/e60b5977c00c</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 15:55:56 GMT</pubDate>
            <atom:updated>2023-06-10T13:03:40.343Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YqRvJkTSGCjnJNdg" /><figcaption>Photo by <a href="https://unsplash.com/@glencarrie?utm_source=medium&amp;utm_medium=referral">Glen Carrie</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You have found a nice spark library such as <a href="https://github.com/G-Research/spark-extension">spark-extension</a>. Now you would like to use it in your Databricks project, where you use Python for your notebooks/jobs/pipelines.</p><p>While possible, there are a few moving pieces that are easy to miss. If this is the first time you are doing it, the process can be confusing and as a result end in unnecessary frustration</p><h4>TLDR</h4><p>The process can be resumed in 3 steps:</p><ul><li>Identify the right versions of the Maven/PyPI packages that match the scala and spark versions of your Databricks cluster(s). The library docs will usually give you a hint on how to identify these in their package versions</li><li>Install the Java library into the cluster(s) by adding it into the Databricks workspace from Maven and select which cluster(s) to install it on. See <a href="https://docs.databricks.com/libraries/workspace-libraries.html#maven-libraries">https://docs.databricks.com/libraries/workspace-libraries.html#maven-libraries</a></li><li>Install the Python API into either a notebook or the cluster(s) using PyPI. See <a href="https://docs.databricks.com/libraries/notebooks-python-libraries.html">https://docs.databricks.com/libraries/notebooks-python-libraries.html</a> or <a href="https://docs.databricks.com/libraries/workspace-libraries.html#pypi-package">https://docs.databricks.com/libraries/workspace-libraries.html#pypi-package</a> respectively.</li></ul><blockquote>Do you use Terraform? Congratulations, you can then use the databricks_library resource in order to add the library and install it into your cluster(s)! See t<a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/library#javascala-maven">heir official docs</a>.</blockquote><h4>Selecting the right Maven and PyPI versions of the library</h4><p>First, note which Scala and Spark version are used by your Databricks cluster(s) runtime.</p><p>If you are unsure, you can check the cluster details, for example this particular cluster uses Spark 3.3.2 and scala 2.12:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*uaJ8qnVOKU469JaU0qcQLA.png" /><figcaption>Cluster details, including Spark and Scala versions of the runtime</figcaption></figure><p>Armed with this data, you can then choose the right version of the library. For example, the <a href="https://github.com/G-Research/spark-extension#using-spark-extension">spark-extension</a> library describes how their Maven/PyPI packages link to the Scala and Spark versions it supports. Given the cluster in the image above, that would be:</p><ul><li>The Maven packageuk.co.gresearch.spark:spark-extension_2.12:2.4.0-3.3</li><li>The PyPI packagepyspark-extension==2.8.0.3.3</li></ul><blockquote>If you install the wrong version of the library Maven/PyPI packages, you will likely encounter errors at runtime!</blockquote><h4>Adding the Java library to the workspace as a Maven library</h4><p>You now need to follow the <a href="https://docs.databricks.com/libraries/workspace-libraries.html#maven-libraries">official docs</a> in order to create a new <em>Databricks workspace library</em> that references the Maven package you have identified.</p><p>Go into the Workspace page, expand the Workspace node and select “Shared”. Then on the right side, select Add -&gt; Library. You will see the following page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xps8MKsyLKjSNAGN4J01TA.png" /><figcaption>Adding a databricks library using Maving as the source</figcaption></figure><p>Here, make sure to select Maven as the library source. You can then either manually enter the package reference like uk.co.gresearch.spark:spark-extension_2.12:2.4.0-3.3 or use the search functionality.</p><blockquote>If you use the search, make sure the Maven coordinates point to a library version compatible with both your Scala and Spark versions! I have noticed the search window picking a version for Scala 2.13 while I needed 2.12. If so, manually tweak the selected package version.</blockquote><p>Click Create once ready.</p><h4>Installing the Maven library into a cluster(s)</h4><p>The step above has added the library into the workspace, but has not installed it into any cluster. Lets fix that.</p><p>Either right after creating the library or from the “Shared” node of the Workspace page, you can open the details page for your newly added library. In this page you can select on which cluster(s) to install the library, including automatically installing it in all clusters</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*foji2QPbPfiCOzCTPuT6Kw.png" /><figcaption>Installing the library into your Databrick cluster(s)</figcaption></figure><p>Select the options you want, then click Install.</p><blockquote>If you want to install into specific clusters, note that terminated clusters will not show up there. <br>However, if you install the library in a cluster and you later terminate it, the next time you start the cluster the library will be installed!</blockquote><h4>Installing the Python library from PyPI</h4><p>Unlike for Java packages from Maven, you can decide to install a PyPI package scoped to a single notebook. See the <a href="https://docs.databricks.com/libraries/notebooks-python-libraries.html">official docs</a>.</p><p>Doing so requires adding %pip commands to the first code cell of your notebook. For example:</p><pre>%pip install pyspark-extension==2.8.0.3.3</pre><blockquote>You need to add %pip commands to the top of your notebook because they will restart the Python interpreter, meaning all variables and methods defined so far will be gone. If you add them in the middle of your notebook, thats likely to break the rest of your notebook code.</blockquote><p>Alternatively, you can install PyPI packages into clusters following pretty much the same process than for the Java Maven package (except selecting PyPI as the source). See the <a href="https://docs.databricks.com/libraries/workspace-libraries.html#pypi-package">official docs</a>.</p><p>Regardless of the method used, you should now be all set. Happy coding!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WOUE7keasCeGdtG0kqCOFQ.png" /><figcaption>The library in action on a notebook</figcaption></figure><h4>Appendix. Using Terraform to add/install the library</h4><p>The D<a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/library">atabricks Terraform provider</a> contains the resource databricks_library which can be used to both add a library and install it into your cluster(s).</p><p>For example, you could add the spark-extension from Maven and install it into all clusters with:</p><pre>data &quot;databricks_clusters&quot; &quot;all&quot; {<br>  # Optionally, add explicit dependencies on any resources creating a cluster<br>  # like &quot;databricks_cluster&quot; or &quot;databricks_pipeline&quot;, so you are sure this<br>  # will indeed get all clusters<br>  depends_on = [...]<br>}<br><br>resource &quot;databricks_library&quot; &quot;spark_extension&quot; {<br>  for_each   = data.databricks_clusters.all.ids<br>  cluster_id = each.key<br>  maven {<br>    coordinates = &quot;uk.co.gresearch.spark:spark-extension_2.12:2.4.0-3.3&quot;<br>  }<br>}</pre><p>You could do the same for the PyPI package (if installing into your cluster(s)) or continue installing per notebook using %pip.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e60b5977c00c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatically stop and start AKS clusters using Azure Automation and Terraform]]></title>
            <link>https://towardsdev.com/automatically-stop-and-start-aks-clusters-using-azure-automation-and-terraform-4890ff945f73?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/4890ff945f73</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Mon, 08 May 2023 07:01:53 GMT</pubDate>
            <atom:updated>2023-05-12T15:00:00.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*glAy-3Ux23V1WBi7" /><figcaption>Photo by <a href="https://unsplash.com/@emiturbuc?utm_source=medium&amp;utm_medium=referral">Emanuel Turbuc</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There is one guiding principle to manage and optimise your Azure costs: only pay for <em>what</em> you need, <em>when</em> you need it. Applying this principle to AKS clusters, strategies such as leveraging the <a href="https://github.com/kubernetes/autoscaler">Cluster Autoscaler</a> or the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">Horizontal Pod Autoscaler</a> will jump to mind.</p><p>What about clusters needed only during a well-known schedule? For example, clusters used for dev/test purposes by team(s) located on the same or close-by time zones. You could optimise your costs by stopping those clusters during nights and weekends!</p><p>This is where <a href="https://learn.microsoft.com/en-us/azure/automation/overview">Azure Automation</a> can help. Using Azure Automation you can automate management tasks using PowerShell or Python scripts, and run them on a predefined schedule (as well as ad-hoc). Starting and Stopping AKS clusters is one of the many things you can do with it.</p><p>The excellent article linked below from Matthias Güntert explains how to setup Azure Automation using the Azure CLI and portal. Inspired by it, I wrote this article describing how to achieve the same using Terraform.</p><p><a href="https://www.azureblue.io/how-to-start-stop-your-aks-cluster-on-schedule/">How to start/stop your AKS cluster on schedule</a></p><h4>Creating an Automation Account</h4><p>Before you can run any PowerShell/Python management scripts using Azure Automation, you will need an Automation Account. Luckily, the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/automation_account">Azure provider</a> for Terraform defines the azurerm_automation_account resource.</p><pre>resource &quot;azurerm_automation_account&quot; &quot;aks_automation_account&quot; {<br>  name                = &quot;aa-aks-automation&quot;<br>  location            = var.location<br>  resource_group_name = var.resource_group_name<br>  sku_name            = &quot;Basic&quot;<br>  identity {<br>    type = &quot;SystemAssigned&quot;<br>  }<br><br>  tags = var.tags<br>}</pre><blockquote>By default, the automation account will get a managed identity assigned by Azure. This is important, since later we will need to grant permissions to this identity so it can start/stop the AKS cluster(s) you want to start/stop. Note it is also possible to provide your own identity if you prefer to do so.</blockquote><p>Make sure to grab the account id, name and managed identity’s principal as either outputs or local variables, we’ll need them later.</p><pre>output &quot;id&quot; {<br>  value = azurerm_automation_account.aks_automation_account.id<br>}<br><br>output &quot;name&quot; {<br>  value = azurerm_automation_account.aks_automation_account.name<br>}<br><br>output &quot;principal_id&quot; {<br>  value = azurerm_automation_account.aks_automation_account.identity.0.principal_id<br>}</pre><h4>Creating a runbook to start/stop AKS clusters</h4><p>With an Automation Account defined, the next step is to define a <em>runbook</em>, ie a PowerShell or Python script we want to run.</p><p>For this, the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/automation_runbook">Azure provider</a> for Terraform defines the azurerm_automation_runbook resource.</p><blockquote>The PowerShell script is from Gordon Byers, and can be found in the Azure Automation gallery as well as his <a href="https://github.com/Gordonby/aks-cluster-changestate">github account</a>.</blockquote><pre>resource &quot;azurerm_automation_runbook&quot; &quot;aks_start_stop_runbook&quot; {<br>  name                    = &quot;aks-start-stop&quot;<br>  location                = var.location<br>  resource_group_name     = var.resource_group_name<br>  automation_account_name = var.automation_account.name<br>  log_verbose             = &quot;false&quot;<br>  log_progress            = &quot;true&quot;<br>  description             = &quot;An Azure Automation Runbook for stopping and starting your AKS Cluster&quot;<br>  runbook_type            = &quot;PowerShell&quot;<br>  tags                    = var.tags<br><br>  # Inline powershell script<br>  content = &lt;&lt;-EOF<br>    &lt;#<br>        .SYNOPSIS<br>            This Azure Automation runbook automates the scheduled shutdown and startup of AKS Clusters in an Azure subscription.<br><br>        .DESCRIPTION<br>            This is a PowerShell runbook, as opposed to a PowerShell Workflow runbook.<br>      Note that the Automation Account will need RBAC permission on the Cluster (scoped directly or inherited) in order to<br>      perform the start/stop operation.<br><br>        .PARAMETER ResourceGroupName<br>            The name of the ResourceGroup where the AKS Cluster is located<br><br>        .PARAMETER AksClusterName<br>            The name of the AKS Cluster to<br><br>        .PARAMETER Operation<br>            Currently supported operations are &#39;start&#39; and &#39;stop&#39;<br><br>        .INPUTS<br>            None.<br><br>        .OUTPUTS<br>            Human-readable informational and error messages produced during the job. Not intended to be consumed by another runbook.<br>    #&gt;<br><br>    Param(<br>          [parameter(Mandatory=$true)]<br>      [String] $ResourceGroupName,<br>          [parameter(Mandatory=$true)]<br>      [String] $AksClusterName,<br>          [parameter(Mandatory=$true)]<br>      [ValidateSet(&#39;start&#39;,&#39;stop&#39;)]<br>          [String]$Operation<br>    )<br><br>    try<br>    {<br>      Disable-AzContextAutosave -Scope Process<br><br>      #System Managed Identity<br>      Write-Output &quot;Logging into Azure using System Managed Identity&quot;<br>      $AzureContext = (Connect-AzAccount -Identity).context<br>      $AzureContext = Set-AzContext -SubscriptionName $AzureContext.Subscription -DefaultProfile $AzureContext<br>    }<br>    catch {<br>      Write-Error -Message $_.Exception<br>      throw $_.Exception<br>    }<br><br>    Write-Output &quot;Performing $Operation&quot;<br>    switch -CaseSensitive ($Operation)<br>    {<br>      &#39;start&#39;<br>      {<br>      Write-Output &quot;Starting Cluster $AksClusterName in $ResourceGroupName&quot;<br>      Start-AzAksCluster -ResourceGroupName $ResourceGroupName -Name $AksClusterName<br>      }<br>      &#39;stop&#39;<br>      {<br>      Write-Output &quot;Stopping Cluster $AksClusterName in $ResourceGroupName&quot;<br>      Stop-AzAksCluster -ResourceGroupName $ResourceGroupName -Name $AksClusterName<br>      }<br>    }<br>  EOF<br>}</pre><p>There are a couple of element’s worth highlighting from the code above:</p><ul><li>The automation_account_name should be the name of the Automation Account we created earlier. Since I defined the account and runbook on separate modules, I exported as output and injected as input variable.</li><li>For the purposes of this article, I have directly defined the PowerShell script as a multiline string for the content argument. You might prefer to move the script to a separate .ps1 file as per the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/automation_runbook#example-usage---custom-content">Terraform docs</a>. It is also possible to directly <a href="https://stackoverflow.com/questions/51762668/import-azure-automation-runbook-using-terraform">import the script from its published source</a>, although I prefer to maintain control of management scripts.</li><li>Note the PowerShell script requires a number of arguments: the names of the AKS cluster and its resource group, as well as an operation indicating whether you want to start or stop the cluster. We will need to <strong>provide these later</strong> when running the runbook!</li></ul><h4>Creating a schedule that runs mornings and evenings</h4><p>With the runbook defined, it is time to define the schedule we want to follow. As you might have guessed, the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/automation_schedule">Azure provider</a> for Terraform defines an azurerm_automation_schedule resource that we can use for this.</p><p>Before we jump into it, lets clarify a couple of points.</p><p>Firstly, the azurerm_automation_schedule is well, <em>just a schedule</em>. What I mean is that the schedule defines on which dates and times should the Azure Automation <em>execute something</em>. But it doesnt define what runbook should be executed or which parameters to be provided. In the last section of the article we will see how to <strong>link a schedule with a runbook</strong>.</p><p>Secondly, to effectively define the schedule, we need to think about when we need to run the script in order to <em>stop an AKS cluster during nights and weekends</em>. This can be achieved with 2 schedules:</p><ul><li>A <strong>morning schedule</strong> can run monday to friday at 7am</li><li>An <strong>evening schedule</strong> can run monday to friday at 8pm</li></ul><p>Now that we know the schedules that have to be created, we can define them using Terraform:</p><pre>locals {<br>  current_time           = timestamp()<br>  current_wallclock_time = formatdate(&quot;h.mm&quot;, local.current_time)<br>  today                  = formatdate(&quot;YYYY-MM-DD&quot;, local.current_time)<br>  tomorrow               = formatdate(&quot;YYYY-MM-DD&quot;, timeadd(local.current_time, &quot;24h&quot;))<br>}<br><br>resource &quot;azurerm_automation_schedule&quot; &quot;monday_friday_morning&quot; {<br>  name                    = &quot;monday-friday-morning&quot;<br>  description             = &quot;Run Monday to Friday early morning&quot;<br>  resource_group_name     = var.resource_group_name<br>  automation_account_name = var.automation_account.name<br>  frequency               = &quot;Week&quot;<br>  interval                = 1<br>  timezone                = var.timezone<br>  start_time              = &quot;${local.current_wallclock_time &gt;= &quot;6.50&quot; ? local.tomorrow : local.today}T07:00:00Z&quot;<br>  week_days               = [&quot;Monday&quot;, &quot;Tuesday&quot;, &quot;Wednesday&quot;, &quot;Thursday&quot;, &quot;Friday&quot;]<br>}<br><br>resource &quot;azurerm_automation_schedule&quot; &quot;monday_friday_evening&quot; {<br>  name                    = &quot;monday-friday-evening&quot;<br>  description             = &quot;Run Monday to Friday late evening&quot;<br>  resource_group_name     = var.resource_group_name<br>  automation_account_name = var.automation_account.name<br>  frequency               = &quot;Week&quot;<br>  interval                = 1<br>  timezone                = var.timezone<br>  start_time              = &quot;${local.current_wallclock_time &gt;= &quot;19.50&quot; ? local.tomorrow : local.today}T20:00:00Z&quot;<br>  week_days               = [&quot;Monday&quot;, &quot;Tuesday&quot;, &quot;Wednesday&quot;, &quot;Thursday&quot;, &quot;Friday&quot;]<br>}</pre><p>You are probably wondering what is going on with the start_time and related locals in the code above! It’s worth stopping a few minutes on this so you understand what’s going on.</p><ul><li>The issue starts with a quirk on Azure’s API for Automation Schedules. To specify the time of the day you want a recurring schedule to happen, like monday-friday <em>at 7am</em>, Azure requires you to provide a <em>start date time</em>. Ie, you need to define the schedule recurring monday-friday, with a start time like 2023–05–08 07:00, and then it will repeat every following day at 07:00.</li><li>Additionally, Azure’s API requires you to provide a start_time in the future. This means, if you later make any changes to the schedule, you will need to provide a new start_time that happens in the future.</li></ul><p>To handle this quirk in Terraform you have 2 options:</p><ul><li>Whenever you need to make any changes to the schedule, you also manually update the start_time parameter so it is in the future.</li><li>You use Terraform’s time-related functions to build an ISO datetime that is always in the future. This is the approach taken by the code above, something suggested <a href="https://discuss.hashicorp.com/t/specifying-a-date-in-the-future/8359/2">as early as 2020</a> to deal with azurerm_automation_schedule resources. The downside of this option is that every time you run Terraform, you will update this resource, moving the start_time to the future.</li></ul><p>None is ideal! But you will need to choose one, unless Microsoft decides to change its API so the recurring time is defined on its own, separate from the start time…</p><h4>Linking the schedule and runbook</h4><p>Finally, we just need to link the 2 schedules we created with the runbook. Apart from linking both entities, we also need to provide the values for the runbook parameters.</p><p>The <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/automation_job_schedule">Azure provider</a> for Terraform defines an azurerm_automation_job_schedule resource that we can use for this.</p><pre>resource &quot;azurerm_automation_job_schedule&quot; &quot;aks_start&quot; {<br>  resource_group_name     = var.resource_group_name<br>  automation_account_name = var.automation_account.name<br>  schedule_name           = azurerm_automation_schedule.monday_friday_morning.name<br>  runbook_name            = azurerm_automation_runbook.aks_start_stop_runbook.name<br>  parameters = {<br>    resourcegroupname = var.resource_group_name<br>    aksclustername    = var.cluster.name<br>    operation         = &quot;start&quot;<br>  }<br>}<br><br>resource &quot;azurerm_automation_job_schedule&quot; &quot;aks_stop&quot; {<br>  resource_group_name     = var.resource_group_name<br>  automation_account_name = var.automation_account.name<br>  schedule_name           = azurerm_automation_schedule.monday_friday_evening.name<br>  runbook_name            = azurerm_automation_runbook.aks_start_stop_runbook.name<br>  parameters = {<br>    resourcegroupname = var.resource_group_name<br>    aksclustername    = var.cluster.name<br>    operation         = &quot;stop&quot;<br>  }<br>}</pre><p>As you can see, both pass the names of the resource group and AKS cluster. However the schedule that runs in the morning, will <em>start</em> the cluster while the schedule running in the evenings will <em>stop</em> the cluster.</p><blockquote>Due to <a href="https://github.com/Azure/azure-sdk-for-go/issues/4780">a bug in the implementation of Runbooks</a> in Azure, the parameter names need to be specified in lowercase only. This causes no issues since PowerShell variables are case insensitive, just makes them harder to read compared to their names in the script source.</blockquote><p>In case you want to start/stop <strong>multiple clusters</strong>, you would add a pair of azurerm_automation_job_schedule resources per cluster. Each of these pairs would be identical except for the aksclustername parameter.</p><h4>Giving Azure Automation permissions to start/stop an AKS cluster</h4><p>If you were to apply the Terraform resources defined so far, you will create all the Azure Automation resources you need. However whenever the runbook runs you would get an error due to the Azure Automation managed identity lacking permissions to start/stop the cluster.</p><blockquote>When we created the Automation Account, we could decide between the Account using a managed service principal or providing our own identity. In either case, the identity used by the Automation Account will need these AKS permissions</blockquote><p>All you need is to add the predefined Contributor role scoped to the AKS cluster(s) you want to start/stop using your automation account:</p><pre>resource &quot;azurerm_role_assignment&quot; &quot;automation_account_principal_aks_contributor&quot; {<br>  scope                = var.cluster.id<br>  role_definition_name = &quot;Contributor&quot;<br>  principal_id         = var.automation_account.principal_id<br>}</pre><p>The value for principal_id is the one we exported when we created the automation account. The cluster id you can get from the azurerm_kubernetes_cluster resource or data source.</p><p>That’s it! You have now fully automated with Terraform the configuration of an Azure Automation account that automatically stops a given AKS cluster(s) during nights and weekends.</p><h4>Appendix — Keeping the cluster running or stopped outside its scheduled time</h4><p>Sometimes you might find yourself needing a cluster that is already stopped. Or you might actually not need the cluster at all for a few days, so there is no point in keeping it running from 7am to 8pm.</p><p>In the first scenario, you could manually “run” the runbook (providing the resource group and AKS names, as well as start as operation). Or you could also just start the AKS cluster using the Azure portal or <a href="https://learn.microsoft.com/en-us/azure/aks/start-stop-cluster?tabs=azure-cli">Azure CLI</a>.</p><p>In the second scenario, you could update your Terraform with a variable to enable/disable the azurerm_automation_job_schedule . Removing these resources is effectively equivalent to disabling the automation.</p><pre>resource &quot;azurerm_automation_job_schedule&quot; &quot;aks_start&quot; {<br>  count = var.enable_cluster_morning_start ? 1 : 0<br>  ...<br>}<br><br>resource &quot;azurerm_automation_job_schedule&quot; &quot;aks_stop&quot; {<br>  count = var.enable_cluster_evening_stop ? 1 : 0<br>  ...<br>}</pre><p>Alternatively, you can <em>disable a schedule</em> using the <a href="https://learn.microsoft.com/en-us/azure/automation/shared-resources/schedules">Azure portal</a> or Azure CLI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4890ff945f73" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/automatically-stop-and-start-aks-clusters-using-azure-automation-and-terraform-4890ff945f73">Automatically stop and start AKS clusters using Azure Automation and Terraform</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to shoot yourself in the foot with Azure resource locks]]></title>
            <link>https://itnext.io/how-to-shoot-yourself-in-the-foot-with-azure-resource-locks-97cf1d2bfa75?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/97cf1d2bfa75</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Fri, 05 May 2023 06:01:52 GMT</pubDate>
            <atom:updated>2023-06-05T08:02:47.668Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*laPFwHohIl_Da1Gj" /><figcaption>Photo by <a href="https://unsplash.com/@davfts?utm_source=medium&amp;utm_medium=referral">David Pupăză</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You have heard of <a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources">Azure resource locks</a> as a tool to prevent accidental deletion and/or modification of a given resource.</p><p>However there is one aspect of how resource locks work that will likely catch you the first time. And the second. And the third…</p><h3>Resource locks do not only affect the resource they are scoped to</h3><p>The Azure docs dedicate an entire <a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources#lock-inheritance">section on locks inheritance</a>. Its first paragraph summarizes it pretty well:</p><blockquote>When you apply a lock at a parent scope, all resources within that scope inherit the same lock. Even resources you add later inherit the same parent lock. The most restrictive lock in the inheritance takes precedence.</blockquote><p>For example:</p><ul><li>If you add a resource lock to a resource group, then all resources inside such resource group will be affected by the lock.</li><li>If you add a resource lock to an AKS cluster, it will affect all the node pools created for that cluster.</li></ul><p>So far so good.</p><p>However when you start using resource locks, say to protect yourself from accidental deletions, you will soon begin to find scenarios where the lock scope wasnt obvious at all.</p><h3>Resource inheritance might not be obvious</h3><p>If you use Azure, its very likely you are leveraging Azure keyvaults as a secret vault. Therefore adding a resource deletion lock to the Azure keyvault might seem like a good idea given the value of the data it contains!</p><p>However, could you tell which one of the following resources will be affected by a lock added to the keyvault?</p><ul><li>A. private endpoint for the keyvault</li><li>B. role assignment scoped to the keyvault Id</li><li>C. secret created inside the vault</li></ul><p>The right answer is A and B!</p><p>Surprised? Not as surprised as I was when our Terraform pipeline failed :)</p><p>—</p><p>Let’s see another example. You are setting up a databricks environment which will connect to a data lake. You decide to add a deletion lock to the data lake’s storage account to avoid accidental deletion of the account and/or containers.</p><p>Some time later, the data engineering team start using <a href="https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/">databricks’ Autoloader</a> to automatically process new files as they are uploaded into the data lake. They decide to do so using the <a href="https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/file-notification-mode">File Notification Mode</a>, ie by Autoloader setting up an EventGrid subscription to the events emitted by the storage account…</p><p>See where this is going?</p><p>Eventually, the data engineering team starts hitting issues as they reach the hard limit of <a href="https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/file-notification-mode#--cloud-resources-used-in-auto-loader-file-notification-mode">500 subscriptions per storage account</a>! Why is that? Because the susbcriptions on a storage account are also covered by the storage account lock, so databricks’ Autoloader cannot clean them up!</p><p>—</p><p>In general, you can tell if a given resource will be affected by the lock if the resource ID is a sub-resource of the one where the lock is defined:</p><pre># keyvault ID<br>/subscriptions/***/resourceGroups/***/providers/Microsoft.KeyVault/vaults/my-keyvault<br># private endpoint connection proxy ID<br>/subscriptions/***/resourceGroups/***/providers/Microsoft.KeyVault/vaults/my-keyvault/privateEndpointConnectionProxies/pe-my-keyvault.abd7c451-c354-4579-8ead-130247e21c17</pre><p>The problem comes with resource such as the private endpoint, which themselves do not appear as sub-resources, but include some inner component which is a sub-resource. For example, in the private endpoint above example, the ID of the private endpoint isn&#39;t a sub-resource, but the inner connection proxy is…</p><p>How can you anticipate this? Test! Seriously, if you introduce a resource lock, test modifying/deleting most resources that seem tangentially related with such resource.</p><p>No harm reviewing the <a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources#considerations-before-applying-your-locks">official list of gotchas</a> when applying resource locks either!</p><h3>Resources might be deleted in unexpected situations</h3><p>Imagine this. You are using Terraform to create an <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/container_registry">ACR registry</a>, which you have configured with geo-replication enabled on a few regions. Your Terraform code might look like this:</p><pre>resource &quot;azurerm_container_registry&quot; &quot;acr&quot; {<br>  name = var.acr_registry_name<br>  ...<br><br>  dynamic &quot;georeplications&quot; {<br>    # Multiple georeplications blocks have to be defined in alphabetical order, hence the &quot;sort&quot;.<br>    # See https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/container_registry#georeplications<br>    for_each = toset(sort(var.geo_replicas))<br><br>    content {<br>      location = georeplications.value<br>      tags = var.tags<br>    }<br>  }<br>  ...<br>}</pre><p>Nothing out of the ordinary. You define with regions you want to replicate to as part of the ACR definition.</p><p>Later you decide to introduce a resource lock to prevent the accidental deletion of the ACR:</p><pre>resource &quot;azurerm_management_lock&quot; &quot;acr_lock&quot; {<br>  name       = &quot;...&quot;<br>  scope      = azurerm_container_registry.acr.id<br>  lock_level = &quot;CanNotDelete&quot;<br>  notes      = &quot;Locked for accidental deletion of Azure Container Registry&quot;<br>}</pre><p>Do you see the problem?</p><p>No? Don’t worry, you are not alone. I also learned this after a Terraform pipeline failure.</p><p>Notice the ACR resource definition defines some tags for each of the ACR geo-replicas. You might think that changing those tags results in an in-place update the of the ACR. And you would be <em>partially</em> right.</p><p>The ACR is updated in-place. But the geo-replicas will be <em>replaced</em> when the tags are changed. And since the geo-replicas are children resources of the ACR registry, they are affected by the deletion lock we created:</p><pre>Error: applying geo replications for Registry (Subscription: &quot;***&quot; Resource Group Name: &quot;...&quot; Registry Name: &quot;myRegistry&quot;): <br>deleting Replication (Subscription: &quot;***&quot; Resource Group Name: &quot;my-rg&quot; Registry Name: &quot;my0registry&quot; Replication Name: &quot;southeastasia&quot;): <br>performing Delete: unexpected status 409 with error: <br>ScopeLocked: The scope &#39;/subscriptions/***/resourceGroups/my-rg/providers/Microsoft.ContainerRegistry/registries/my0registry/replications/southeastasia&#39; <br>cannot perform delete operation because following scope(s) are locked: &#39;/subscriptions/***/resourceGroups/my-rg/providers/Microsoft.ContainerRegistry/registries/my0registry&#39;. <br>Please remove the lock and try again.</pre><p>This means you <strong>cannot update the tags on the ACR geo-replicas if the ACR has a deletion lock defined</strong>!</p><p>What can you do? In my case, I decided to ignore certain tags that change frequently in the geo-replicas. In the event some of the more permanent tags has to be changed, I will remove the lock, change the tags, and then reintroduce the lock.</p><pre>dynamic &quot;georeplications&quot; {<br>  for_each = toset(sort(var.geo_replicas))<br><br>  content {<br>    location = georeplications.value<br>    # NOTE: changing tags requires the georeplication to be recreated, which is then blocked by the resource deletion lock added to the ACR!<br>    tags = { for k, v in var.tags : k =&gt; v if !contains([&quot;FrequentlyChangingTag1&quot;, &quot;FrequentlyChangingTag2&quot;], k) }<br>  }<br>}</pre><h3>In summary</h3><p>Resource locks are very useful tools, giving you peace of mind against accidental deletions/modifications of key resources.</p><p>However, their scope behaviour can result in unexpected effects! We have seen a couple of examples, and the azure docs have a <a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources#considerations-before-applying-your-locks">long list of gotchas</a>.</p><p>If you are planning on leveraging resource locks:</p><ul><li>Testing is the best proactive strategy! Try to verify whether you can update/delete most resources somewhat related to the one where you define a lock!</li><li>Be ready to either adopt workarounds to allow certain changes, or even allow yourself to temporarily remove the lock during certain changes/refactorings.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97cf1d2bfa75" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-to-shoot-yourself-in-the-foot-with-azure-resource-locks-97cf1d2bfa75">How to shoot yourself in the foot with Azure resource locks</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting Databricks to Azure Keyvault with Terraform is surprisingly hard!]]></title>
            <link>https://medium.com/@danieljimgarcia/connecting-databricks-to-azure-keyvault-with-terraform-is-surprisingly-hard-efef414130da?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/efef414130da</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Tue, 02 May 2023 23:06:09 GMT</pubDate>
            <atom:updated>2023-08-07T16:02:19.492Z</atom:updated>
            <content:encoded><![CDATA[<h3>Connecting Databricks to an Azure Keyvault with Terraform is surprisingly hard!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NbN9RLyP2JpCF9Ko" /><figcaption>Photo by <a href="https://unsplash.com/@flyd2069?utm_source=medium&amp;utm_medium=referral">FLY:D</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You have been setting up Azure Databricks following infrastructure as code principles using Terraform. Now you want to create a databrick’s secret scope connected to an Azure keyvaults.</p><p>It should be easy but it is far from it!</p><p>Not only you are unlikely to automate with Terraform (more on it later). Even setting up the necessary permissions to manually create the secret stores can be harder than it seems.</p><blockquote><strong>Update</strong>: It is finally possible to <a href="https://learn.microsoft.com/en-us/azure/databricks/release-notes/product/2023/april#create-an-azure-key-vault-backed-secret-scope-with-a-service-principal">use a service principal</a> to create keyvault backed secret scopes! Note you still need to assign the necessary permissions to the Azure user/principal you use with Terraform as well as the predefined AzureDatabricks service principal, where the docs can still be confusing.</blockquote><h3>Why automating with Terraform wont be an option for many of you</h3><p>As it turns out, the databricks terraform provider includes a resource to create a secret scope connected to an Azure keyvault: <a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret_scope#keyvault_metadata">https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret_scope#keyvault_metadata</a></p><p>The problem lies in the following warning in the databricks provider docs:</p><blockquote>Currently, it’s only possible to create Azure Key Vault scopes with Azure CLI authentication and not with Service Principal.</blockquote><p>This means the secret_scope resource cannot be used when authenticating with Azure via a Service principal. In other words, you need to authenticate with Azure via the azure CLI and a real user account.</p><p>If you have been using Terraform to manage your Azure infrastructure, this might be surprising. The reason is that the Azure provider <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure">explicitly recommends</a> to setup your automation using a service principal or managed identity:</p><blockquote>We recommend using either a Service Principal or Managed Service Identity when running Terraform non-interactively (such as when running Terraform in a CI server) — and authenticating using the Azure CLI when running Terraform locally.</blockquote><p>If you are using the azure CLI authentication, congratulations because you will be able to use the secret_scope resource (Even then, you still need to setup the necessary permissions as in the following section).</p><p>However, I suspect many of you will be following the recommended authentication approach, and use a service principal to automate Terraform. Then you can’t automate the secret scope creation unless you are willing to configure some user credentials as part of your automation.</p><h3>Setting up the necessary permissions to create the databricks secret scopes connected to a keyvault</h3><p>Ok, so to create secret scopes connected to a keyvault you need to authenticate with Azure using a real user account, regardless of whether you use the databricks Terraform provider or not.</p><p>But what permissions does that user account needs to have?</p><p>If you read the documentation, you will uncover the first permission you need:</p><ul><li>According to the <a href="https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#--create-an-azure-key-vault-backed-secret-scope-using-the-databricks-cli">Azure docs</a> “<em>the Azure AD user who creates the secret scope must have </em><a href="https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#permissions-required-for-registering-an-app"><em>permission to create service principals</em></a><em> in the key vault’s tenant.</em>”</li><li>The docs on <a href="https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#check-azure-subscription-permissions">creating service principals</a> further clarify: “<em>your account must have </em>Microsoft.Authorization/*/Write<em> access to assign a role to an AD app. This action is granted through the </em><a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#owner"><em>Owner</em></a><em> role or </em><a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#user-access-administrator"><em>User Access Administrator</em></a><em> role</em>“.</li></ul><p>However these docs fail to mention that if you just have theUser Access Administrator instead of theOwner role then you ALSO need to have the role Owner assigned on both the databricks workspace and keyvault!</p><p>In addition to that, if your Azure keyvault has RBAC enabled instead of access policies, then you need to add read permissions to the predefined AzureDatabricks service principal.</p><ul><li>For example, you can assign the role Key Vault Secrets User to the AzureDatabricks service principal, scoped to the keyvault you want to access via the secret scope.</li><li>There was an <a href="https://github.com/MicrosoftDocs/azure-docs/issues/84380">issue logged to update/fix the Azure docs</a>, but it was closed without the docs being updated.</li></ul><p>As an aside, given the AzureDatabricks service principal is created by azure for each tenant, this means the AzureDatabricks service principal will have access to all the keyvaults you eventually connect to a databricks secret scope <em>across the entire tenant</em>. If you think this could/should be improved, you can vote on the <a href="https://feedback.azure.com/d365community/idea/71f94069-f024-ec11-b6e6-000d3a4f0da0">feedback</a> opened 3 years ago.</p><h4>In summary…</h4><p>The user authenticating with Azure that creates the databricks secret scope needs to have either of:</p><ul><li>Owner role granted over the entire subscription(s)</li><li>User Access Administrator role over the subscription, AND Owner over the databricks workspace AND Owner over the keyvault</li></ul><p>The predefined AzureDatabricks service principal needs to be assigned at a minimum the role Key Vault Secrets User so it can read secrets. Assign other roles if you also want databricks code to create/update secrets.</p><h3>Creating the databricks secret scopes</h3><p>So you have assigned the necessary permissions. If you happen to use Terraform authenticating with the azure CLI and a real user account, then you can use the secret_scope resource.</p><p>But as we mentioned earlier, that might not be the case for many of you, givent the recommended approach is to use an Azure service principal to automate Terraform.</p><p>So how do you create the secret scope? You have 2 options</p><h4>Using the databricks UX</h4><p>There is a “secret” form in the databricks workspace that can be used to create the secret scopes</p><ul><li>Open the databricks workspace in the browser</li><li>Navigate to https://&lt;databricks-instance&gt;#secrets/createScope. (NOTE: exact casing is needed!)</li><li>Fill in the form. You can get the “DNS Name” and “Resource ID” from the properties tab of the keyvault page in the Azure portal.</li></ul><h4>Using the databricks CLI</h4><p>Follow these steps:</p><ul><li>Login with your user (or the user you want to use to create the secret scope) in the azure CLI:<br>az login</li><li>Get the AAD token to authenticate with the databricks CLI. (NOTE: the --resource is always that specific value, which represents the programmatic ID for Azure Databricks)<br>export DATABRICKS_AAD_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq .accessToken --raw-output)</li><li>Authenticate with the databricks CLI and configure access to the workspace where you want to create the secret scope<br>databricks configure — aad-token</li><li>Create the secret scope using the following command<br>databricks secrets create-scope --scope &lt;scope-name&gt; --scope-backend-type AZURE_KEYVAULT --resource-id &lt;keyvault-resource-id&gt; --dns-name &lt;keyvault-dns&gt; --initial-manage-principal users. As in the databricks UX option, you can get the --resource-id and --dns-name values from the properties tab of the keyvault page in the Azure portal.</li></ul><h3>Verify everything is working</h3><p>If you have followed all the steps, you will likely want to make sure everything is working and that you can read keyvault secrets in databricks using the created secret scope.</p><p>You can add a code block like the following to a notebook in order to test you can read a keyvault secret via the secret scope:</p><pre>foo = dbutils.secrets.get(&quot;&lt;name-of-secret-scope&gt;&quot;, &quot;&lt;name-of-a-secret-in-keyvault&gt;&quot;)<br>for char in foo:<br>    print(char, end=&quot; &quot;)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efef414130da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Application Gateway Ingress Controller is broken]]></title>
            <link>https://medium.com/@danieljimgarcia/the-application-gateway-ingress-controller-is-broken-6aa9eb229881?source=rss-1d47a9a98035------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa9eb229881</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ingress]]></category>
            <dc:creator><![CDATA[Daniel Jimenez Garcia]]></dc:creator>
            <pubDate>Fri, 30 Sep 2022 11:44:09 GMT</pubDate>
            <atom:updated>2024-06-09T08:44:41.600Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Update Jun’24: </strong>The redesigned implementation called <strong>Application Gateway for Containers</strong> has been GA since late Feb’24, see <a href="https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/overview">https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/overview</a>. Amongst its goals, it addressed thechallenges called out in the article. While I haven’t personally tried it (yet), I am happy to see listed amongst it features the <em>near real-time updates to add or move pods, routes, and probes</em>.</p><blockquote>Please note the article applies to the older service called Application Gateway Ingress Controller!</blockquote><p><strong>Update Sep’23: </strong>Microsoft has announced a public preview of a new Application Gateway tier designed specifically for AKS: <a href="https://learn.microsoft.com/en-gb/azure/application-gateway/for-containers/overview">https://learn.microsoft.com/en-gb/azure/application-gateway/for-containers/overview</a>. I have only briefly looked at the docs, but it appears to address the flaw discussed in this article.</p><h3>TLDR</h3><p>The Application Gateway Ingress Controller is a great idea. It allows exposing applications hosted in Kubernetes to the outside world via Azure’s native Application Gateway, taking care of most of the necessary configuration and rules.</p><p><a href="https://github.com/Azure/application-gateway-kubernetes-ingress">GitHub - Azure/application-gateway-kubernetes-ingress: This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.</a></p><p>However, it has important design flaws, which can cause <a href="https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1124">minutes of downtime</a> when updating your workloads! These flaws are still present at the time of writing, Sept’ 22.</p><p>If you are planning on using it for production workloads, I would advise to read through the article, check the status of Github issues and run your own tests/benchmarks.</p><blockquote>Note the article describes issues with the Application Gateway Ingress Controller. These do not necessarily translate to the Application Gateway itself. You can still leverage the Application Gateway and avoid these flaws, as long as you take care of the configuration that the Ingress Controller would manage for you.</blockquote><h3>A fundamental design flaw causes downtime when updating Pods</h3><h4>Understanding the issue</h4><p>The design of the Application Gateway Ingress Controller is based around the idea of sending traffic from the gateway directly to the Pods. Per se not a bad idea and not too different from other solutions.</p><p>But this means the Ingress Controller needs to create a backend pool per Ingress. And it needs to keep it up to date with the IPs of the Pods behind that Ingress.</p><p>What happens when you upgrade/redeploy a workload?</p><ul><li>The current<strong> </strong>Pods are terminated and new Pods replace them.</li><li>The Ingress Controller updates the Gateway’s backend pool for the affected Ingress, replacing the IPs of the old Pods with the new Pod IPs<strong>.</strong></li></ul><p>So far so good. Here is the catch, the <strong>Application Gateway does not guarantee fast updates</strong> of the IP addresses. In my experience, <strong>10 minutes were normal</strong>, sometimes even longer. Other users have reported up to 40m! And according to Azure support, these times are within the expectations of the Application Gateway, which does not make any guarantees regarding update times.</p><p>This is the main design flaw. The failure to account for the long update times means users experience minutes of downtime. <strong>Once the old Pods are terminated, your application won’t be reachable until the Application Gateway update completes</strong>. While the update completes, the Application Gateway rules will still point to the old Pods IP, sending traffic to Pods that no longer exist.</p><p>The following diagram shows this sequence, assuming the usual 10m update times <em>I have experienced</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QVivrjVI-i70JpNbqDE3bA.png" /><figcaption>downtime sequence after updating a deployment workload</figcaption></figure><p>As you can see, several minutes of downtime can be experienced. Let’s assume a rolling upgrade process, whereby all your replicas have disappeared after X minutes. If the Application Gateway update takes 10m then you could experience 10 - X minutes of downtime.</p><blockquote>This issue isn’t recent. It <a href="https://github.com/Azure/application-gateway-kubernetes-ingress/issues/75#issuecomment-441979234">was noticed</a> as early as 2018, and explicitly called out multiple times like in <a href="https://github.com/Azure/application-gateway-kubernetes-ingress/issues/725">Jan’20</a> or <a href="https://medium.com/r?url=https%3A%2F%2Fgithub.com%2FAzure%2Fapplication-gateway-kubernetes-ingress%2Fissues%2F1124">Feb’21</a>. Users noted the flaws of the current design, as it depends on fast gateway updates which are not guaranteed.</blockquote><h4>The official guidelines to avoid downtime are not enough</h4><p>The Application Gateway Ingress Controller docs publishes some guidelines to <em>minimize downtime</em> during deployments.</p><p><a href="https://azure.github.io/application-gateway-kubernetes-ingress/how-tos/minimize-downtime-during-deployments/">Minimizing Downtime During Deployments</a></p><p>Note the word <em>minimize</em>. The docs admit you cannot guarantee avoiding downtime, not even when following these guidelines:</p><blockquote>This document offers a method to alleviate this problem. Since the method described in this document relies on correctly aligning the timing of deployment events it is not possible to guarantee 100% elimination of the probability of running into a 502 error. Even with this method there will be a non-zero chance for a period of time where Application Gateway backends could lag behind the most recent updates applied by a rolling update to the Kubernetes pods.</blockquote><p>Here are the guidelines:</p><ul><li>Add a preStop lifecycle hook to the Pods where you execute a sleep of at least 90 seconds</li><li>Add terminationGracePeriodSeconds to the Pod, with a value greater than in the preStop hook. Example uses 101 seconds</li><li>Decrease thePod’s probes period/timeout, so they are marked unhealthy sooner</li><li>Setup connection draining annotations, for in-flight connections to be gracefully handled before removing Pod from Gateway backend</li></ul><p>The issue with them is that they are betting on the update to the Application Gateway to finish <em>before</em> the terminationGracePeriodSeconds . Ie, we are keeping the old Pods for a while longer before they are finally removed, hoping this gives enough extra time for the Application Gateway update to finish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L0kNiFSLBmCI8nkwDXVzzA.png" /><figcaption>suggested mitigations are not enough</figcaption></figure><p>As you can see in the diagram above, for users experiencing updates that consistently take 10 minutes or even longer, this won’t be enough. <em>The guidelines make the same mistake, they assumes updates to the Application Gateway are fast!</em></p><ul><li>Also be aware that preStop isn’t called in situations like a Pod crashing or OOM killed. In these situations the workarounds won’t be able to mitigate the issue at all</li></ul><p>Ok, let’s say you are fortunate and are experiencing the updates taking 60 seconds instead of 10 minutes. Or that you are willing to keep your old Pods around for 10m via the preStop lifecycle and theterminationGracePeriod. Even then, we still need to account for updates/redeployments of <strong>multiple workloads</strong>!</p><p>This is when avoiding downtimes gets way more complicated. <strong>The updates to the Application Gateway have to be done one at a time</strong>. Once a workload starts the update process, any other updated workload will need to wait for the first update to complete:</p><ol><li>Imagine you update/redeploy workload A, which begins an update to the gateway. We know the update can take a long time, 10 minutes or longer.</li><li>Imagine after 2 minutes, you update/redeploy another workload B. This will require updating the Application Gateway, but it will first need to wait 8 minutes for the first update the complete, then another 10 minutes for this new update</li><li>Meanwhile Kubernetes begins the process to replace the Pods in workload B.</li><li>Even if you were prepared to keep old pods around for 10 minutes, in the case of workload B this won’t be enough. You had an extra 8 minutes wait that you couldn’t account for!</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FGDfhQzQ6feSeiMey5kFCA.png" /><figcaption>updates to multiple workloads result in even longer downtimes</figcaption></figure><p>There are many users reporting updates that take frequently 10, 20 or more minutes. With these slow updates, downtimes can become quite severe once multiple workloads are updated/redeployed.</p><h4>It can get <em>even worse</em> in some scenarios</h4><p>Knowing that multiple updates to workloads causes even longer downtimes, think about these scenarios:</p><ul><li><strong>Horizontal Pod Autoscaler (HPA)</strong>. While you configured HPA for scalability and resiliency, in combination with the issues you have seen, the automated removal/addition of Pods might end up causing intermittent 502 errors</li><li><strong>A node needs to be drained</strong>. Given all pods in that node have to be terminated, there will be a long queue of updates to the Application Gateway!</li><li><strong>All the cluster nodes have to be updated</strong>. A not so uncommon scenario to process OS updates in the cluster nodes, by cordoning, draining and rebooting each in sequence. This will eventually see all of your Pods being terminated as the update cycles through all the nodes. Just imagine the queue of pending Application Gateway updates!</li></ul><p>I cannot but agree with <a href="https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1124#issuecomment-824595041">this comment</a>:</p><blockquote>The AGIC design requires AppGW to make guarantees that it simply doesn’t. The design is in itself broken.</blockquote><h3>What can you do instead</h3><p>As it is today, I would not recommend relying on the Application Gateway Ingress Controller for any production workloads.</p><p>Without deviating much from an architecture based on it, you could consider:</p><ul><li>An Application Gateway (managed by you) in front of a load balancer managed by the NGINX ingress. This way you expose the AppGw to the outside world and keep WAF happening before traffic reaches your cluster. The downsides are that you need to be the one updating rules/certificates/etc in the AppGw, and you have extra network hoops plus cpu/memory consumed by NGINX ingress</li><li>Just a load balancer managed by the NGINX ingress. The load balancer is directly exposed to the outside world, and <a href="https://systemweakness.com/nginx-ingress-waf-with-modsecurity-from-zero-to-hero-fa284cb6f54a">WAF moves to the NGINX ingress via ModSecurity</a>. Everything is managed for you, but now WAF happens within your cluster, so you might want to account for extra cpu/memory and perhaps use dedicated/separate nodes from your actual workloads.</li></ul><p>Of course, these are mere pointers in directions to explore, and not the only alternatives. However, I wanted to close the article by providing a way out for anyone who based its design in the Application Gateway Ingress Controller!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa9eb229881" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>