<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lalit Adithya on Medium]]></title>
        <description><![CDATA[Stories by Lalit Adithya on Medium]]></description>
        <link>https://medium.com/@lalitadithya?source=rss-f395fcfb2897------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*w9jmB1kEgzEooFnyw7HEAw.jpeg</url>
            <title>Stories by Lalit Adithya on Medium</title>
            <link>https://medium.com/@lalitadithya?source=rss-f395fcfb2897------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:35:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lalitadithya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Git based CMS for Jekyll using GitHub Actions]]></title>
            <link>https://medium.com/@lalitadithya/git-based-cms-for-jekyll-using-github-actions-ec73889187a?source=rss-f395fcfb2897------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec73889187a</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cms]]></category>
            <category><![CDATA[jekyll]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[website]]></category>
            <dc:creator><![CDATA[Lalit Adithya]]></dc:creator>
            <pubDate>Wed, 09 Sep 2020 11:31:01 GMT</pubDate>
            <atom:updated>2020-09-09T11:31:01.264Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JF6gEN3kmTiCTNDj" /><figcaption>Photo by <a href="https://unsplash.com/@richygreat?utm_source=medium&amp;utm_medium=referral">Richy Great</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A git-based CMS allows users to leverage git as a single source of truth for all their content, i.e., blog posts, website theme, plugins, assets etc. Every time a user pushes a change (a change can be anything from adding a new post to updating a older post to adding a plugin or changing the theme) to the master/main branch, a pipeline will be triggered that will build and deploy the website.</p><h3>Advantages of a git-based CMS</h3><ol><li><strong>Version control for everything</strong> — Images, scripts, plugins, content, theme etc are version controlled out of box</li><li><strong>Easy rollback</strong> — Just rollback the bad commit and the pipeline will take care of the rest</li><li><strong>Ability to leverage git branches to work on multiple things at the same time</strong> — When one post or update is complete, just merge that branch to master/main and the pipeline will take care of the rest</li><li><strong>Ability to leverage the pull request feature for an easy review process</strong> — A pipeline will deploy the changes made in the pull request to a non-public website to make it easy for reviewers to view changes</li><li><strong>No vendor lock in</strong> — Don’t want to use GitHub? You can use GitLab or an on-premises git server. Don’t want to use Azure? You can make use of AWS or GCP or IBM cloud. Even switching from one cloud provider to the other is as simple as updating the pipeline(s)</li></ol><h3>Disadvantages of git-based CMS</h3><ol><li>Not easy to use for non-technical users</li><li>Might not be the best if your content is changing rapidly</li></ol><h3>What is Jekyll?</h3><p>Jekyll is a simple, blog-aware, static site generator. It is written in Ruby and it is distributed under the open source MIT license.</p><blockquote>Jekyll does what you tell it to do — no more, no less. It doesn’t try to outsmart users by making bold assumptions, nor does it burden them with needless complexity and configuration. Put simply, Jekyll gets out of your way and allows you to concentrate on what truly matters: your content.</blockquote><blockquote>– Jekyll readme file</blockquote><h3>Why not WordPress like 38% of the internet?</h3><p>Unlike WordPress that needs to be run on servers (that we must manage, scale, maintain, and secure), Jekyll generates good old HTML and CSS files which can be hosted on any object store such as Azure Storage Accounts (or Amazon Web Services’ S3). We do not have to manage, scale, maintain and secure any compute infrastructure or databases.</p><p>We can leverage the “only pay for what you use” model to lower our costs. We only need to pay for the storage we use and the amount of data that we transfer to our users. In the case of WordPress, we will need pay a fixed monthly cost irrespective of the amount of data that we store or the amount of data we serve to our users.</p><h3>Interested to learn more?</h3><p>Read the complete post on my blog:</p><ol><li><a href="https://lalitadithya.com/blog/git-based-cms-for-jekyll-using-github-actions-part-1/">https://lalitadithya.com/blog/git-based-cms-for-jekyll-using-github-actions-part-1/</a></li><li><a href="https://lalitadithya.com/blog/git-based-cms-for-jekyll-using-github-actions-part-2/">https://lalitadithya.com/blog/git-based-cms-for-jekyll-using-github-actions-part-2/</a></li><li><a href="https://lalitadithya.com/blog/git-based-cms-for-jekyll-using-github-actions-part-3/">https://lalitadithya.com/blog/git-based-cms-for-jekyll-using-github-actions-part-3/</a></li></ol><p>Follow me on social media using the links below to get updates for new content:</p><p><a href="https://www.facebook.com/thelalitadithya/">Facebook</a>, <a href="https://twitter.com/thelalitadithya">Twitter</a>, <a href="https://www.instagram.com/lalitadithya/">Instagram</a>, <a href="https://www.linkedin.com/in/lalitadithya/">LinkedIn</a>, <a href="https://github.com/lalitadithya/">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec73889187a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Cloud Native Task Execution Framework on Azure]]></title>
            <link>https://medium.com/@lalitadithya/building-a-cloud-native-task-execution-framework-on-azure-bcb5d0acaa77?source=rss-f395fcfb2897------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcb5d0acaa77</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Lalit Adithya]]></dc:creator>
            <pubDate>Fri, 19 Jun 2020 11:31:01 GMT</pubDate>
            <atom:updated>2020-06-19T11:31:01.202Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YkoXA6UNGA29zYjn" /><figcaption>Photo by <a href="https://unsplash.com/@airfocus?utm_source=medium&amp;utm_medium=referral">airfocus</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this post we are going to be looking at how to build a consumption-based task execution framework that can scale elastically as demand rises and falls.</p><p>The architecture diagram for the complete framework is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*k-7AvouemBOdsOM5" /><figcaption>Architecture Diagram</figcaption></figure><p>In this post we will only be exploring the core components of the framework, namely:</p><ol><li>Task Runners — Docker containers running on Azure Container Instances.</li><li>Task Orchestrator — Azure Functions App</li></ol><h3>Task Runners</h3><p>At a very high level, a task runner is a docker container that will download and execute a task.</p><h3>What is a task?</h3><p>A task can be anything ranging from powering off a virtual machine to deploying a new service to a Kubernetes cluster.</p><p>For our framework, a task is any *nix executable that is stored in an Azure storage container. This executable will read input from environment variables.</p><p>To keep things simple, let us take the example of turning off a virtual machine as our task. The bash script for this task shown below:</p><pre>#! /bin/bash<br>az login --identity<br>az vm stop --resource-group &quot;${ResourceGroupName}&quot; --name &quot;${VMName}&quot;</pre><pre>az vm stop --resource-group &quot;${ResourceGroupName}&quot; --name &quot;${VMName}&quot;</pre><p>The script logs in to Azure using the assigned identity and uses the “az vm stop” command to stop the virtual machine. For this script to run, we will need the Azure CLI.</p><h3>What are Task Runners?</h3><p>A task runner is a docker container that contains all the dependencies needed for a task to execute and it will download and launch the task executable.</p><p>For our example of turning off a virtual machine, the only dependency we have is the Azure CLI. Let us create a docker image that contains the Azure CLI and call it “lalitadithya/az-cli-image”.</p><p>The Dockerfile for this image is shown below:</p><pre>FROM mcr.microsoft.com/azure-cli<br>WORKDIR /scripts<br>COPY script-runner.sh .<br>ENV PATH=&quot;/scripts:${PATH}&quot;<br>RUN chmod a+x script-runner.sh<br>CMD [&quot;./script-runner.sh&quot;, &quot;&quot;]</pre><p>The docker file copies a bash script called “script-runner” from the host, sets the executable permissions for the script and then executes the script. The “script-runner” is shown below:</p><pre>#! /bin/bash<br>wget -O script.sh $1<br>chmod a+x script.sh<br>./script.sh</pre><p>The script runner downloads the task script/executable from the URL provided in the command line and executes the task script/executable.</p><h3>What are Task Definitions?</h3><p>A task definition is a JSON document that contains metadata associated with a task such as the name of the executable for the task, the docker image that should be used to execute the task and the list of input parameters for the task. The task definitions will be stored along with the task executables.</p><p>The task definition for our turning off a virtual machine task is shown below:</p><pre>{<br>  &quot;taskName&quot;: &quot;StopVM&quot;,<br>  &quot;dockerImage&quot;: &quot;lalitadithya/az-cli-image:latest&quot;,<br>  &quot;scriptName&quot;: &quot;stop-vm-azure.sh&quot;,<br>  &quot;scriptParameters&quot;: [<br>    {<br>      &quot;parameterName&quot;: &quot;VMName&quot;,<br>      &quot;parameterType&quot;: &quot;string&quot;,<br>      &quot;required&quot;: true<br>    },<br>    {<br>      &quot;parameterName&quot;: &quot;ResourceGroupName&quot;,<br>      &quot;parameterType&quot;: &quot;string&quot;,<br>      &quot;required&quot;: true<br>    }<br>  ]<br>}</pre><h3>Task Orchestrator</h3><p>Let us now look at how we can make use of an Azure Functions app to automatically spawn Docker containers to execute tasks.</p><p>Here are the tasks that needs to be accomplished by our task orchestrator for every request</p><ol><li>Determine the container that needs to be spawned using the task definition</li><li>Create a SAS for the task executable</li><li>Spawn the docker container</li></ol><p>The task orchestrator will contain one POST endpoint that will be used to perform the above-mentioned tasks. The request body will contain the name of the task and the input parameters for the task as shown below:</p><pre>{<br>  &quot;taskName&quot;: &quot;StopVM&quot;,<br>  &quot;parameters&quot;: [<br>    {<br>      &quot;name&quot;: &quot;VMName&quot;,<br>      &quot;value&quot;: &quot;a3etest&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;ResourceGroupName&quot;,<br>      &quot;value&quot;: &quot;rg-a3e-test_resources-dev&quot;<br>    }<br>  ]<br>}</pre><h4>Determine the container that needs to be spawned using the task definition</h4><p>To determine the container that needs to be spawned we will have to pull the task definition from the storage account.</p><p>The code for pulling the task definition from storage account is shown below:</p><pre>var cloudStorageAccount = CloudStorageAccount.Parse(configuration[&quot;StorageConnetionString&quot;])<br>var blobClient = cloudStorageAccount.CreateCloudBlobClient();<br>var storageContainer = blobClient.GetContainerReference(“task-container”);<br>var blob = storageContainer.GetBlobReference(“stop-vm.json”); <br> <br>using MemoryStream blobContents = new MemoryStream();<br>await blob.DownloadToStreamAsync(blobContents);<br>string jsonTaskDefinition = Encoding.UTF8.GetString(blobContents.ToArray());<br> <br>TaskDefinition taskDefinition = JsonConvert.DeserializeObject&lt;TaskDefinition&gt;(jsonTaskDefinition);<br>string containerImage = taskDefinition.DockerImage<br>string taskExecutableName = taskDefinition.ScriptName</pre><h4>Create a SAS for the task executable</h4><p>In the previous step we found the name of task executable, now let us create a SAS for the executable so that we can pass the URI to task runner so that it can download the executable.</p><p>The code for generating a SAS is shown below:</p><pre>var cloudStorageAccount = CloudStorageAccount.Parse(configuration[&quot;StorageConnetionString&quot;])<br>var blobClient = cloudStorageAccount.CreateCloudBlobClient();<br>var storageContainer = blobClient.GetContainerReference(“task-container”);<br>var blob = storageContainer.GetBlobReference(taskDefinition.ScriptName);<br> <br>SharedAccessBlobPolicy sharedAccessBlobPolicy = new SharedAccessBlobPolicy<br>{<br>        SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),<br>        Permissions = SharedAccessBlobPermissions.Read<br>};<br>var blobToken = blob.GetSharedAccessSignature(sharedAccessBlobPolicy);<br>string uri = blob.Uri + blobToken;</pre><h4>Spawn the docker container</h4><p>Now that we have a SAS URI for the task executable, let us now spawn a docker container on Azure Container Instances.</p><p>The code for this is shown below:</p><pre>var environmentVariables = taskDefinition.ScriptParameters<br>                             .ToDictionary(x =&gt; x.ParameterName,<br>                               x =&gt; task.Parameters.FirstOrDefault(y =&gt; x.ParameterName == y.Name)?.Value);<br> <br>await azure.ContainerGroups<br>.Define(SdkContext.RandomResourceName(configuration[&quot;ContainerGroupNamePrefix&quot;], 20))<br> .WithRegion(Region.USWest)<br> .WithExistingResourceGroup(configuration[&quot;ResourceGroupName&quot;])<br> .WithLinux()<br> .WithPublicImageRegistryOnly()<br> .WithoutVolume()<br>.DefineContainerInstance(SdkContext.RandomResourceName(configuration[&quot;ContainerInstanceNamePrefix&quot;], 20))<br>  .WithImage(containerImage)<br>  .WithoutPorts()<br>  .WithCpuCoreCount(1)<br>  .WithMemorySizeInGB(1)                                 .WithEnvironmentVariables(environmentVariables)<br>  .WithStartingCommandLine(&quot;script-runner.sh&quot;, scriptLocation)<br>.Attach()                                .WithExistingUserAssignedManagedServiceIdentity(await azure.Identities.GetByIdAsync(configuration[&quot;UserAssignedManagedServiceIdentityId&quot;]))  .WithRestartPolicy(ContainerGroupRestartPolicy.Never)<br>.CreateAsync();</pre><h3>Recap</h3><p>In this post we looked at how we can build a consumption-based task execution framework on Azure using Azure Container Instances and Azure Functions.</p><p>You can view the source for all the components explained at my GitHub repository <a href="https://github.com/lalitadithya/automate-anything-and-everything">here</a>.</p><p>Liked what you just read? Please subscribe to my mailing list on my website <a href="http://lalitadithya.com/">http://lalitadithya.com/</a> to learn about the latest and greatest in technology (No spam, I promise). Additionally, you can follow me on social media using the links below:</p><p><a href="https://www.facebook.com/thelalitadithya/">Facebook</a>, <a href="https://twitter.com/thelalitadithya">Twitter</a>, <a href="https://www.instagram.com/lalitadithya/">Instagram</a>, <a href="https://www.linkedin.com/in/lalitadithya/">LinkedIn</a>, <a href="https://github.com/lalitadithya/">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcb5d0acaa77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Save Money by “Turning Off” your Azure PaaS Resources]]></title>
            <link>https://medium.com/@lalitadithya/how-to-save-money-by-turning-off-your-azure-paas-resources-7209a5c5fbb4?source=rss-f395fcfb2897------2</link>
            <guid isPermaLink="false">https://medium.com/p/7209a5c5fbb4</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-container-instances]]></category>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Lalit Adithya]]></dc:creator>
            <pubDate>Wed, 27 May 2020 15:31:01 GMT</pubDate>
            <atom:updated>2020-05-27T15:31:01.463Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UOTVy6AkqoTz1niX" /><figcaption>Photo by <a href="https://unsplash.com/@spanic?utm_source=medium&amp;utm_medium=referral">Damir Spanic</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Azure resources can be broadly divided into the following categories:</p><ol><li><strong>Platform as a Services (PaaS): </strong><em>Consumption based</em> — e.g., Azure Functions, Azure Logic Apps, Azure Container Instances etc and <em>Hour based</em> — e.g., Azure SQL Server, Azure Database for Postgres, Azure Kubernetes Service etc</li><li><strong>Infrastructure as a Service (IaaS)</strong> — e.g., Azure Virtual Machines</li><li><strong>Software as a Service (SaaS)</strong> — e.g., Microsoft 365</li></ol><p>In an ideal world, all our workloads will be entirely consumption based and we only pay for what we use. Often that is not the case our workloads consist of a mix of consumption-based PaaS, hour-based PaaS and sometimes sprinkled with IaaS for good measure. One of the major downsides of the hour-based PaaS is that they do not have a “stop” or a “start” button like IaaS resources generally do. In this post we are going to be looking at how we can save more than just a few bucks on our Azure bill by turning off the hour-based PaaS services when we are not using them.</p><p>A lot of folks save on their Azure bill by opting for the 1-year reserved or the 3-year reserved option for production use, but that kind of commitment might not be needed for dev/test or side projects. In most cases the only option we have would be to delete the resources after we are done with them and then re-create them when we need them. That is a viable approach, but we lose the “state” of our resources. “State” of a resource means many things. It can be the tables and/or stored procedures in case of a database, or it could be deployments in case of Kubernetes.</p><p>We can export the state before deleting the resource and then delete the resource. When we want to start the resource back up, we can create the resource and import the state back. This is exactly what I used to do.</p><p>This got me thinking: the delete/create is an extremely redundant and boring process that I do at least twice a week. Can this be automated? Afterall a wise man once said</p><blockquote><em>If you must do something more than twice, stop and automate it</em></blockquote><blockquote><em>A wise man</em></blockquote><h3>Ferb, I know what we’re gonna do today</h3><p>We are going to be building two docker containers that will give us the ability to “turn off” PaaS resources when we are not using them.</p><p>We will be making use Azure Container Instances to run these containers and we will use Azure Logic Apps to trigger the docker containers. We will also be making use of a storage media such as Azure Storage to store state and finally we will leverage Azure’s ARM templates to create our PaaS resources on demand.</p><h4>Side node</h4><p>I am taking Azure SQL server as an example for this post, but the method described here is extremely generic and it can be used for any PaaS resource on Azure. I also make use of Azure storage and Azure container registry for storing state as I have those running 24*7, but you can make use of any other storage media like Google Drive or OneDrive and you can make use of Docker Hub for storing your container images if you don’t want to use Azure container registry.</p><h3>Azure Container Instances</h3><p>First, let us talk about Azure’s container instances (ACI). ACI lets you build container-based event driven applications. If you want a short-lived container to run in response to (sporadic) events you can do that with ACI. The icing on the cake is that you will only be billed for the time your container is running. Pretty cool, right?</p><p>Wait, why do we need ACI? I can do almost anything with Azure Functions and/or Azure Logic Apps, right?</p><p>Yes and no.</p><p>If you just want to destroy/create resources you can probably do that using the Azure management SDK and Azure functions, but if you want to make use of other tools like sqlcmd, kubectl etc for saving/restoring your resource’s state you most likely can not use Azure Functions or Azure Logic Apps.</p><h3>Let us drive right into it, shall we?</h3><p>Let us start by dividing our resources into three separate resources groups, namely:</p><ol><li><em>rg-acme-rebootable</em> — This resource group will contain the PaaS resources that we would like to “start” and “stop”</li><li><em>rg-acme-non-rebootable</em> — This resource group will contain all our other resources that we need to keep running for our application and are consumption based</li><li><em>rg-acme-automation</em> — This resource group will contain all the resources that we will be using for automation</li></ol><p>If we have an Azure Functions app that connects to an Azure SQL backend. We should put the functions app in the “rg-acme-non-rebootable” and the SQL server in the “rg-acme-rebootable” resource group.</p><p>To “start” and “stop” the SQL server we are going to be making use of PowerShell scripts which will be run inside docker containers on ACI.</p><p>Let us now look at the scripts.</p><h3>“Stop”ing the SQL database</h3><p>To stop the database, we can make use of a super cool tool called <a href="https://github.com/microsoft/mssql-scripter">mssql-scripter</a>. For those of you who are familiar with SQL server you know that SSMS provides an option called “generate scripts” which can be used to export schema and data to an SQL file. mssql-scripter is the command line version of that option.</p><p>A snippet from the script is shown below. You can find the complete script <a href="https://github.com/lalitadithya/AzureAutomation/blob/master/StopDatabase/stop-database.ps1">here</a>.</p><pre>Write-Host &quot;Attempting Azure login&quot;</pre><pre>az login --identity</pre><pre>Write-Host &quot;Database export started&quot;</pre><pre>mssql-scripter -S $DatabaseServerName -d $DatabaseName -U $AdminUsername -P $AdminPassword --schema-and-data --target-server-version &quot;AzureDB&quot; --display-progress &gt; backup.sql</pre><pre>Write-Host &quot;Uploading script to blob&quot;</pre><pre>az storage blob upload -f backup.sql -c $StorageContainerName -n backup.sql --connection-string &quot;$StorageConnectionString&quot;</pre><pre>Write-Host &quot;Stopping database in resource group $ResourceGroupName&quot;</pre><pre>az group delete -n $ResourceGroupName --yes</pre><pre>Write-Host &quot;Database stopped&quot;</pre><p>The script first logs in to Azure using the “az login” command. It then makes use of the mssql-scripter tool to export the database to an SQL file. After the export is complete, it uploads the file to Azure storage and then it deletes the resource group. You can customize the upload target to any other cloud storage by making use of a utility like <a href="https://rclone.org/">rclone</a>.</p><p>As mentioned earlier, the script will be running inside a docker container. The DockerFile for the container is shown below:</p><pre>FROM ubuntu:18.04</pre><pre># install azure cli<br>RUN apt-get update &amp;&amp; \<br>    apt-get install -y ca-certificates curl apt-transport-https lsb-release gnupg &amp;&amp; \<br>    curl -sL <a href="https://packages.microsoft.com/keys/microsoft.asc">https://packages.microsoft.com/keys/microsoft.asc</a> | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg &gt; /dev/null &amp;&amp; \<br>    AZ_REPO=$(lsb_release -cs) &amp;&amp; \<br>    echo &quot;deb [arch=amd64] <a href="https://packages.microsoft.com/repos/azure-cli/">https://packages.microsoft.com/repos/azure-cli/</a> $AZ_REPO main&quot; | tee /etc/apt/sources.list.d/azure-cli.list &amp;&amp; \<br>    apt-get update &amp;&amp; \<br>    apt-get install -y azure-cli</pre><pre># install pwsh<br>RUN apt-get -y install wget software-properties-common &amp;&amp; \<br>    wget -q <a href="https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb">https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb</a> &amp;&amp; \<br>    dpkg -i packages-microsoft-prod.deb &amp;&amp; \<br>    apt-get update &amp;&amp; \<br>    add-apt-repository universe &amp;&amp; \<br>    apt-get install -y powershell</pre><pre># install mssql-scripter<br>RUN apt-get -y install python python-pip &amp;&amp; \<br>    pip install mssql-scripter</pre><pre>COPY stop-database.ps1 /stop-database.ps1</pre><pre>CMD [&quot;pwsh&quot; ,&quot;.\\stop-database.ps1&quot;, \<br>    &quot;-ResourceGroupName&quot;, &quot;&quot;, \<br>    &quot;-DatabaseServerName&quot;, &quot;&quot;, \<br>    &quot;-DatabaseName&quot;, &quot;&quot;, \<br>    &quot;-AdminPassword&quot;, &quot;&quot;, \ <br>    &quot;-AdminUsername&quot;, &quot;&quot;, \ <br>    &quot;-StorageConnectionString&quot;, &quot;\&quot;&lt;&lt;connection string goes here&gt;&gt;\&quot;&quot;, \ <br>    &quot;-StorageContainerName&quot;, &quot;&quot;]</pre><p>In the DockerFile, we first install Azure CLI, PowerShell and mssql-scripter. The script shown above is copied into the container and the script is executed using the “CMD” command. This container will run the script and then it will terminate.</p><p>You can find the DockerFile <a href="https://github.com/lalitadithya/AzureAutomation/blob/master/StopDatabase/DockerFile">here</a>.</p><h3>“Start”ing the database</h3><p>To start the database, we make use of Azure’s ARM template deployment model to deploy SQL server and then we can make use of Microsoft’s sqlcmd to run the exported SQL file after downloading it from the storage media.</p><p>A snippet from the script is shown below. You can find the complete version of the script <a href="https://github.com/lalitadithya/AzureAutomation/blob/master/StartDatabase/start-database.ps1">here</a>.</p><pre>Write-Host &quot;Attempting Azure login&quot;</pre><pre>az login --identity</pre><pre>Write-Host &quot;Fetching ARM template&quot;</pre><pre>az storage blob download -c $TemplateStorageContainerName --file template.json --name template.json  --connection-string &quot;$StorageConnectionString&quot;</pre><pre>Write-Host &quot;Fetching template parameters&quot;</pre><pre>az storage blob download -c $TemplateStorageContainerName --file parameters.json --name parameters.json  --connection-string &quot;$StorageConnectionString&quot;</pre><pre>Write-Host &quot;Creating resource group&quot;</pre><pre>az group create --name $ResourceGroupName --location eastus</pre><pre>Write-Host &quot;Database create started&quot;</pre><pre>$deploymentName = Get-Date -Format &quot;MMddyyyyHHmmss&quot;</pre><pre>az group deployment create --name $deploymentName --resource-group $ResourceGroupName --template-file template.json --parameters `@parameters.json</pre><pre>Write-Host &quot;Database restore started&quot;</pre><pre>az storage blob download -c $DatabaseStorageContainerName --file backup.sql --name backup.sql  --connection-string &quot;$StorageConnectionString&quot;</pre><pre>$content = (Get-Content &#39;.\backup.sql&#39;)<br>$content[105..($content.Length-3)] | Set-Content backup.sql</pre><pre>/opt/mssql-tools/bin/sqlcmd -U $AdminUsername -S $DatabaseServerName -P $AdminPassword -i &quot;./backup.sql&quot; -d $DatabaseName</pre><pre>Write-Host &quot;Deleting backup&quot;</pre><pre>az storage blob delete -c $DatabaseStorageContainerName --name backup.sql --connection-string &quot;$StorageConnectionString&quot;</pre><p>The script logs in to Azure using the “az login” command. It creates a resource group for the resources and deploys the SQL server from an ARM template. After the deployment is complete, the script fetches the SQL export from the storage media and makes use of the sqlcmd command to run the script. Once done, the script deletes the SQL file from the storage media. As mentioned previously, you can make use of any cloud storage media of your choice you do not have to stick to Azure Storage.</p><p>This script will also be run in a Docker container like the previous one. The DockerFile for this one is extremely similar to the previous script and it can be found <a href="https://github.com/lalitadithya/AzureAutomation/blob/master/StartDatabase/DockerFile">here</a>.</p><p>Now that we know the scripts that will be used to “start” and “stop” the SQL server, let us now look at how we can put all of this together.</p><h3>Putting it all together</h3><p>Push the two docker images to a container registry of your choice and create a two ACI instances from the Azure portal as shown in the screenshots below. Make sure to clear out the “Command Override” text box in the advanced tab and set the “Restart Policy” to never.</p><figure><img alt="Page 1 ACI" src="https://cdn-images-1.medium.com/max/1024/0*IvkjoJwfnvlaoHBb" /><figcaption>Page 1 ACI</figcaption></figure><figure><img alt="Page 2 ACI" src="https://cdn-images-1.medium.com/max/1024/0*MvqJ_e7-jYrnGL-F" /><figcaption>Page 2 ACI</figcaption></figure><figure><img alt="Page 3 ACI" src="https://cdn-images-1.medium.com/max/1024/0*TsN1G4iODENZMSAL" /><figcaption>Page 3 ACI</figcaption></figure><p>After the ACI instances have been provisioned, assign a system managed identity, and provide it with enough access to create/destroy your PaaS resources. Restart your ACI containers and verify that the RBAC is working as expected.</p><p>We now have two containers running on ACI which can “start” and “stop” our PaaS resources (in my case an SQL server). All we need to do now is to figure out how to trigger the ACI instances.</p><h3>Enter, Azure Logic Apps</h3><p>Azure Logic Apps will be perfect for our trigger due to the large number of connectors and triggers that it supports. Go ahead and create two new Azure Logic Apps and configure the trigger of your choice (in my case I have chosen the recurrence trigger) and add a step to start the container instance as shown in the screenshot below:</p><figure><img alt="Azure Logic Apps Designer" src="https://cdn-images-1.medium.com/max/1024/0*fP3mJVH4YiDWsSSY" /><figcaption>Azure Logic Apps Designer</figcaption></figure><h3>Conclusion</h3><p>In this post we had a look at how we can “turn off” Azure PaaS resources which do not have a Microsoft provided start/stop button. To achieve this, we made use of Azure Container Register (ACI) to run the Docker containers which run PowerShell scripts to save/restore the “state” of the PaaS resource and then delete/create resource. For triggering the ACI, we made use of Azure Logic Apps due to the large number of connectors and triggers that it supports.</p><p>Liked what you just read? Please subscribe to my mailing list on my website <a href="http://lalitadithya.com/">http://lalitadithya.com/</a> to learn about the latest and greatest in technology (No spam, I promise). Additionally, you can follow me on social media using the links below:</p><p><a href="https://www.facebook.com/thelalitadithya/">Facebook</a>, <a href="https://twitter.com/thelalitadithya">Twitter</a>, <a href="https://www.instagram.com/lalitadithya/">Instagram</a>, <a href="https://www.linkedin.com/in/lalitadithya/">LinkedIn</a>, <a href="https://github.com/lalitadithya/">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7209a5c5fbb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything I’ve Learnt About Distributed Locking So Far]]></title>
            <link>https://medium.com/@lalitadithya/everything-ive-learnt-about-distributed-locking-so-far-1f1569e6df5?source=rss-f395fcfb2897------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f1569e6df5</guid>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[locks]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Lalit Adithya]]></dc:creator>
            <pubDate>Fri, 15 May 2020 12:01:01 GMT</pubDate>
            <atom:updated>2020-05-15T12:01:01.212Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MXT1lZkboUHFresk" /><figcaption>Featured Photo by <a href="https://unsplash.com/@mr_williams_photography?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Micah Williams</a> on <a href="https://unsplash.com/s/photos/lock-fence?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post we are going to be looking at locking in a distributed system. But, first let us go back to the basics and remind ourselves what is a lock is and why we need locks.</p><h3>What is a lock? Why do we need it?</h3><p>In computer science, a lock is a synchronization mechanism that is used to control access to a resource when there are multiple threads of execution. Locks are used to guarantee mutual exclusion, i.e., only one thread of execution can be inside a critical section at any given point in time. At a very high level we make use of locks to achieve correctness in a multithreaded environment.</p><p>In order to understand locks further let us look at an example: Consider a singly linked list that contains four items, namely “i — 1, “i”, “i + 1” and “i + 2” as shown in the diagram below:</p><figure><img alt="Linked list with 4 items" src="https://cdn-images-1.medium.com/max/1024/0*pxcsNtaoa69INXpG" /><figcaption>Linked list with 4 items</figcaption></figure><p>This linked list is being shared by two different threads of execution. At time t0, the first thread wants to remove node “i”, so it changes the “next” pointer of node “i — 1” to point to node “i +1” as shown in red. At the same time instant, t0, the second thread of execution wants to remove the node “i + 1” and hence it changes the “next” pointer of node “i” to point to node “i + 2” as shown in blue.</p><figure><img alt="Two threads attempting to remove two different elements" src="https://cdn-images-1.medium.com/max/1024/0*CF-_vnblQ3xv5Y8k" /><figcaption>Two threads attempting to remove two different elements</figcaption></figure><p>At time instant t1 &gt; t0, the state of the linked list will be:</p><figure><img alt="State of the linked list after the removal" src="https://cdn-images-1.medium.com/max/1024/0*jdLXSUX9UoAz5fpK" /><figcaption>State of the linked list after the removal</figcaption></figure><p>Even though the remove operation was performed twice on the linked list, we ended up removing only one element. Our desired state of the linked list has not been achieved. This problem is commonly known as a race condition. To solve the race conditions, we need to make use to mutual exclusion (by extension locks) to ensure that simultaneous updates cannot occur.</p><h3>Distributed Locking</h3><p>In the previous section we looked at why locks are important to ensure correctness of results. We also looked at an example of a linked list in a multithreaded environment in a single system, now let us look at how can we make use of a locks in a loosely coupled distributed systems to ensure correctness of results. But, first let us remind ourselves of the CAP theorem.</p><h3>CAP Theorem</h3><p>When designing distributed services, there are three properties that are commonly desired: consistency, availability, and partition tolerance. It is impossible to achieve all three. [1]</p><ol><li><strong>Consistency</strong>: Every read receives the most recent write or an error</li><li><strong>Availability</strong>: Every request receives a (non-error) response, without the guarantee that it contains the most recent write</li><li><strong>Partition tolerance</strong>: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes</li></ol><p>In a distributed system, a network partition is inevitable. Therefore, according to the CAP theorem, we must pick among:</p><ol><li>Cancel the operation and thus decrease the availability but ensure consistency</li><li>Proceed with the operation and thus provide availability but risk inconsistency</li></ol><h3>Are networks really that unreliable? Yes.</h3><p>A team from the University of Toronto and Microsoft Research studied the behaviour of network failures in several of Microsoft’s datacentres. They found an average failure rate of 5.2 devices per day and 40.8 links per day with a median time to repair of approximately five minutes (and up to one week). While the researchers note that correlating link failures and communication partitions is challenging, they estimate a median packet loss of 59,000 packets per failure. Perhaps more concerning is their finding that network redundancy improves median traffic by only 43%; that is, network redundancy does not eliminate many common causes of network failure. [2]</p><p>A joint study between researchers at University of California, San Diego and HP Labs examined the causes and severity of network failures in HP’s managed networks by analysing support ticket data. “Connectivity”-related tickets accounted for 11.4% of support tickets (14% of which were of the highest priority level), with a median incident duration of 2 hours and 45 minutes for the highest priority tickets and a median duration of 4 hours 18 minutes for all priorities. [2]</p><p>Google’s paper describing the design and operation of Chubby, their distributed lock manager, outlines the root causes of 61 outages over 700 days of operation across several clusters. Of the nine outages that lasted greater than 30 seconds, four were caused by network maintenance and two were caused by “suspected network connectivity problems.” [2]</p><h3>Can other things go wrong in a distributed environment? Absolutely.</h3><p>A network partition is only a small part of what can go wrong in a distributed environment. Some of the other things that can go wrong in a distributed environment include:</p><ol><li>A node can crash, or be rebooted by the orchestrator</li><li>The service might become unresponsive for an extended period due to a super long GC pause</li><li>Time synchronization issues</li></ol><h3>Goals</h3><p>We have established that it is not possible for us to build a prefect distributed system and we must pick between availability and consistency. Let us now look at what is the main purpose of a distributed lock.</p><figure><img alt="Purpose of a distributed lock" src="https://cdn-images-1.medium.com/max/640/0*_fBj-vhQiTHbbxpc" /><figcaption>Purpose of a distributed lock</figcaption></figure><p>Both Alice (shown in red) and Bob (shown in blue) want access to Service X and a bunch of other services (not shown), so they first request a lock from one of the many lock service nodes and then they access “Service X” using the lock they acquired from the lock service. The lock service will ensure mutual exclusion for Alice and Bob for Service X and all the other services that would like access. Note: all the services must participate in the distributed locking mechanism.</p><p>Let us now start looking at how we can get distributed lock with a single instance of the lock service and then we can look at how we can horizontally scale the lock service.</p><h3>Fenced locks</h3><p>A fenced lock is a lock that includes a fencing token. A fencing token is a monotonically increasing token that increments whenever a client acquires the lock.</p><p>To understand why fencing tokens are necessary, let us look at what will happen when we do not have fencing tokens in a distributed system:</p><figure><img alt="What can go wrong when we do not use a fenced lock?" src="https://cdn-images-1.medium.com/max/640/0*yJMlGRFqhx_ytX_E" /><figcaption>What can go wrong when we do not use a fenced lock?</figcaption></figure><p>In the above scenario, Bob has acquired the lock, but due to some reason (bug in code, super long GC), Bob stopped responding for an extended period. When Bob was in a “hung” state, Bob’s lock has expired due to a timeout (Note: It is always good to have timeout for locks otherwise, a crashed client will lock out every other client). Alice acquired the lock the lock for service ‘X’ and made a change to service ‘X’. When Bob is back from the “hung” stage, he will now make an unsafe change to service ‘X’, because he still thinks he has the lock for the service. But, wait why can’t Bob just check if he still has access to the lock just before accessing the service? Yes, he can but that will also not work as Bob can go into an unexpected “hung” state at any time (It can be after he has made the check if has the lock).</p><p>Now, let us look at how we can solve this by making use of fencing tokens.</p><figure><img alt="Using fencing tokens" src="https://cdn-images-1.medium.com/max/640/0*lT3UAXFbJQ5VEdni" /><figcaption>Using fencing tokens</figcaption></figure><p>Now that we have fencing tokens, Bob acquires a lock and he gets a fencing token of 1. Just like in the previous scenario, he goes into a “hung” state. While Bob is in this “hung” state, his token times out. Alice comes online and acquires the lock. She is given a fencing token of 2. Alice passes her token (i.e., 2) to service ‘X’ along with her write request. The service stores Alice’s token as the last known token and it honours Alice’s request. Now, when Bob comes back from his “hung” state, he attempts to make a write request to service ‘X’ by passing his fencing token. The service checks the token passed by Bob against its’ last known token, finds out that the Bob holds an old token (remember, fencing tokens are monotonically increasing) and it will not honour Bob’s request.</p><p>Notice that it is no longer possible to make unsafe changes to service ‘X’ due to the monotonically increasing property of fencing tokens. And yes, this has made service ‘X’ linearized.</p><p>Now that we have a fair idea of a distributed lock, let us now look at how to make our lock service horizontally scalable.</p><h3>Horizontally scaling the lock service</h3><p>To horizontally scale the lock service itself we need to ensure that the service is always consistent while tolerating network partitions. This would mean that we are going to be sacrificing the availability of the lock service, i.e., in the face of a network partition, the lock service may decide to stop issuing locks till the partition is resolved. This is done so that we do not end up with a split-brain scenario.</p><p>Split-brain is a computer term, based on an analogy with the medical Split-brain syndrome. It indicates data or availability inconsistencies originating from the maintenance of two separate data sets with overlap in scope, either because of servers in a network design, or a failure condition based on servers not communicating and synchronizing their data to each other. This last case is also commonly referred to as a network partition. [3]</p><p>In the case of distributed locking, a split-brain scenario will become a nightmare because it will give two parties a “lock” on the same resource. If two parties can hold the same lock, we have fundamentally broken the definition of a lock. To avoid this, we pick consistency over availability.</p><p>To develop a consistent locking service that can tolerate network partitions each node should maintain the state of the lock along with the fencing tokens and it must achieve consensus using a consensus algorithm. You can either make use of the Raft consensus algorithm or the Paxos consensus algorithm. Raft consensus algorithm is easier to understand and is described below.</p><h3>Raft consensus algorithm</h3><p>Raft is a consensus algorithm that is designed to be easy to understand. It is equivalent to Paxos in fault-tolerance and performance. The difference is that it is decomposed into relatively independent subproblems, and cleanly addresses all major pieces needed for practical systems. [4]</p><p>I had planned on explaining the Raft consensus algorithm, but I found this beautiful visualization of the raft consensus algorithm <a href="http://thesecretlivesofdata.com/raft/">here</a>. I highly recommend that you check it out before proceeding. It very clearly explains how the algorithm behaves in a normal situation and during a network partition.</p><h3>Conclusion</h3><p>In this blog post, we had a look at one of the ways in which we can design a horizontally scalable distributed locking system that provides consistency (while sacrificing availability) while tolerating network partitions using fencing tokens and the Raft Consensus algorithm. This exact approach has been implemented by the Hazelcast team in their latest version of their in-memory data grid, please do check that our as well.</p><p>Liked what you just read? Please subscribe to my mailing list on my website <a href="http://lalitadithya.com/">http://lalitadithya.com/</a> to learn about the latest and greatest in technology (No spam, I promise). Additionally, you can follow me on social media using the links below:</p><p><a href="https://www.facebook.com/thelalitadithya/">Facebook</a>, <a href="https://twitter.com/thelalitadithya">Twitter</a>, <a href="https://www.instagram.com/lalitadithya/">Instagram</a>, <a href="https://www.linkedin.com/in/lalitadithya/">LinkedIn</a>, <a href="https://github.com/lalitadithya/">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f1569e6df5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Confidential Computing: Securing data while it is in use]]></title>
            <link>https://medium.com/@lalitadithya/confidential-computing-securing-data-while-it-is-in-use-82207aad6ff1?source=rss-f395fcfb2897------2</link>
            <guid isPermaLink="false">https://medium.com/p/82207aad6ff1</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Lalit Adithya]]></dc:creator>
            <pubDate>Fri, 08 May 2020 15:43:50 GMT</pubDate>
            <atom:updated>2020-05-08T15:43:50.006Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-Ekii5kyJJ7DiMQM" /><figcaption>Photo by <a href="https://unsplash.com/@chrispanas?utm_source=medium&amp;utm_medium=referral">chris panas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>According to the Confidential Computing Consortium, a Linux Foundation project, computing is moving to span multiple environments, from on premises to public cloud to edge. As companies move to these environments, they need protection controls for sensitive IP and workload data and are increasingly seeking greater assurances and more transparency of these controls. Current approaches address data at rest and in transit; confidential computing will address data in use.</p><h3>Confidential computing</h3><p>Confidential computing is a term defined by the Confidential Computing Consortium (CCC). The Confidential Computing Consortium is a Linux Foundation Project brings together hardware vendors, cloud providers, and software developers to accelerate the adoption of Trusted Execution Environment (TEE) technologies and standards. [1] The founding premiere members include Alibaba, Arm, Google Cloud, Huawei, Intel, Microsoft and Red Hat. General members include Baidu, ByteDance, decentriq, Fortanix, Kindite, Oasis Labs, Swisscom, Tencent and VMware. [2]</p><blockquote><em>Confidential computing is defined as the protection of data in use by performing computation in a hardware-based Trusted Execution Environment.</em></blockquote><blockquote><em>Confidential Computing Consortium, “FAQ — Confidential Computing Foundation,”</em></blockquote><p>Let us take a brief detour to understand Trusted Execution Environment (TEE).</p><h3>Trusted Execution Environment</h3><p>TEEs are built on top of separation kernels.</p><blockquote><em>A separation kernel is defined as “hardware and/or firmware and/or software mechanisms whose primary function is to establish, isolate and separate multiple partitions and control information flow between the subjects and exported resources allocated to those partitions”</em></blockquote><blockquote><em>Information Assurance Directorate of the US National Security Agency (NSA) in the publication titled “U.S. Government Protection Profile for Separation Kernels in Environments Requiring High Robustness”</em></blockquote><p>At its core, the separation kernel should provide:</p><ul><li>Protection of all resources (including CPU, memory and devices) from unauthorized access</li><li>Partitioning and isolation of exported resources</li><li>Mediation of information flows between partitions and between exported resources</li></ul><p>The separation kernel is what provides us with two isolated environments, i.e., the secure environment (TEE) and the real world (REE). It controls the information flow between the two environments and protects all resources in the secure environment.</p><figure><img alt="Figure showing TEE, REE and the separation" src="https://cdn-images-1.medium.com/max/820/0*TwLbOz_38S0OytjM" /><figcaption>Figure showing TEE, REE and the separation</figcaption></figure><p>Now that we know what a separation kernel is, let us look at TEEs. There are several definitions for TEEs that are available on the internet, the most complete among them is given below:</p><blockquote><em>Trusted Execution Environment (TEE) is a tamper resistant processing environment that runs on a separation kernel. It guarantees the authenticity of the executed code, the integrity of the runtime states (e.g. CPU registers, memory and sensitive I/O), and the confidentiality of its code, data and runtime states stored on a persistent memory. In addition, it shall be able to provide remote attestation that proves its trustworthiness for third parties. The content of TEE is not static; it can be securely updated. The TEE resists against all software attacks as well as the physical attacks performed on the main memory of the system. Attacks performed by exploiting backdoor security flaws are not possible.</em></blockquote><blockquote><em>M. Sabt, M. Achemlal and A. Bouabdallah in their paper titled “Trusted Execution Environment: What It is, and What It is Not”</em></blockquote><p>At the core of any system that offers security is trust — Users must trust the system enough to use it, Engineers must trust the system enough to build applications around it.</p><h4>Trust</h4><p>In an ideal world, you would probably find a lab that is certified by other labs (or maybe even a Government agency) who can build and test a TEE. You would then take that TEE to a device manufacturer (that you trust) who can reproduce the sample TEE in production devices. But, the world we live in very far from ideal. Moreover, if one attempts the solution proposed above it will become very expensive, and hence it would become a limiting factor.</p><p>In the real world, we trust the manufacturer of the TEE and we make use of lot and lots of signature checks to establish trust during run time.</p><p>TEE establishes trust during the boot process using “hardware chain of trust”. In “hardware chain of trust” every image that is loaded and executed is cryptographically verified by the image loaded in the previous stage.</p><p>The first image that is loaded and executed (also known as the primary boot loader) is loaded from immutable read-only memory. This immutable read-only memory is burned into the chip by the manufacturer of the chip and it cannot be altered physically or otherwise.</p><p>The primary boot loader cryptographically verifies the signature of the next image that needs to be loaded and executed and this chain continues. All the signature checks are done using a set of private keys that are embedded directly onto the chip during the manufacturing process (hence the name “hardware chain of trust”). These private keys are stored using read only memory and they cannot be altered physically or otherwise. If during some stage of the boot, a signature check fails, then the TEE will not offer the services for which the signature check failed.</p><p>Once the boot is complete, the TEE can prove its validity by using a process called attestation. After the TEE has proved its validity, we trust the secure firmware that is loaded to execute trusted applications (TAs) in isolation, i.e., one TA can not affect other TAs that are running.</p><p>TEEs rely on the underlying separation kernel to prevent tampering of the executing TAs. The separation kernel employs various hardware-based mechanisms to protect the trusted applications from tampering. It prevents all resources (including CPU, memory and devices) from unauthorized access while ensuring partitioning and isolation of the exported resources and mediating the information flows between secure word (TEE) and the real world (REE).</p><p>Now that have a fair understanding of TEEs, let us jump back into confidential computing.</p><h3>Applications of Confidential Computing available right now</h3><p>Some of the many applications of confidential computing that you can try out right now include:</p><ol><li>Multiparty machine learning</li><li>Trusted networks</li><li>SQL Server Always Encrypted Technology</li></ol><h4>Multiparty machine learning</h4><p>In many cases, multiple parties will benefit from sharing of data, but currently they are not able to share data while preserving privacy.</p><p>Consider the example of hospitals that are using machine learning models to diagnose diseases early. Every hospital will have its own silo of data and it will train and run machine models on that silo. We know that the accuracy of any machine learning model depends entirely on the amount and accuracy of the data that is fed to it. With confidential computing, all hospitals can confidentially run a machine learning model on the combined dataset, thereby improving the model itself and benefiting all hospitals and by extension their patients.</p><p>Similarly, if a bank is using machine learning models to detect fraud and money laundering; multiple banks can make use of confidential computing to build a combined machine learning model benefiting everyone.</p><p>Olga Ohrimenko, Felix Schuster, Cedric Fournet, Aastha Mehta, Sebastian Nowozin, Kapil Vaswani and Manuel Costa published a paper titled “<em>Oblivious Multi-Party Machine Learning on Trusted Processors</em>” where they propose data oblivious machine learning algorithms for support vector machines, matrix factorization, neural networks, decision trees, and k-means clustering.</p><h4>Trusted networks</h4><p>Another application of confidential computing is the creation of trusted networks. The Confidential Consortium Framework (CCF), a joint project with Azure Engineering, is an open-source framework for building a new category of secure, highly available, and performant applications that focus on multi-party compute and data. While not limited just to blockchain applications, CCF can enable high-scale, confidential blockchain networks that meet key enterprise requirements — providing a means to accelerate production enterprise adoption of blockchain technology. [3]</p><p>CCF enables enterprise-ready computation or blockchain networks that deliver [3]:</p><ol><li>Throughput and latency approaching database speeds</li><li>Richer, more flexible confidentiality models</li><li>Network and service policy management through non-centralized governance</li><li>Improved efficiency versus traditional blockchain networks</li></ol><p>You can read more about CCF in the paper titled “<em>CCF: A Framework for Building Confidential Verifiable Replicated Services</em>” by Mark Russinovich, Edward Ashton, Christine Avanessians, Miguel Castro, Amaury Chamayou, Sylvan Clebsch, Manuel Costa, Cedric Fournet, Matthew Kerner, Sid Krishna, Julien Maffre, Thomas Moscibroda, Kartik Nayak, Olga Ohrimenko, Felix Schuster, Roy Schuster, Alex Shamis, Olga Vrousgou and Christoph M. Wintersteiger.</p><p>Code for CCF can be found on GitHub <a href="https://github.com/Microsoft/CCF">here</a></p><h4>SQL Server Always Encrypted Technology</h4><p>People familiar with SQL Server would have heard of the “Always Encrypted” technology that was introduced with SQL Server 2016. This technology allowed users to control the access of encrypted data, but, it had some limitations. Some of the limitations include: the only operations that SQL server could perform on a “Always Encrypted” column was equality comparisons (only available with deterministic encryption). All the other operations must be performed on the client side and not on the database server. Say, if the user wanted to perform pattern matching, the user would have to move all the data outside the database and then perform these operations on the client side.</p><p>With SQL server 2019 and later, Microsoft announced the “Always Encrypted with Secure Enclaves”. This makes use of confidential computing to address the limitations described above. Read more about it <a href="https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-enclaves?view=sql-server-ver15">here</a>.</p><h3>Who is offering Confidential Computing?</h3><p>Earlier this week, Mark Russinovich from Microsoft announced the general availability of the DCsv2-series VMs that has support for confidential computing. You can read the announcement <a href="https://azure.microsoft.com/en-us/blog/dcsv2series-vm-now-generally-available-from-azure-confidential-computing/">here</a>. IBM also has a similar offering that you can read about <a href="https://www.ibm.com/cloud/blog/data-use-protection-ibm-cloud-using-intel-sgx">here</a>.</p><h3>How do I develop trusted apps that make use of Confidential Computing?</h3><p>Microsoft and Google are offering SDKs that help developers get started with developing application that make use of confidential computing.</p><p>You can check out the SDKs <a href="https://openenclave.io/sdk/">here</a> and <a href="https://asylo.dev/">here</a>.</p><h3>Conclusion</h3><p>With more and more confidential workloads moving to the public cloud and edge devices, it is becoming more and more important to secure data not only while it is in rest or transit, but also while it is in use. At the time of writing this article, confidential computing is gaining momentum. It shows a lot of promise for securing data while in use and has a tremendous potential to do good.</p><h3>References</h3><p>[1] Confidential Computing Consortium, “FAQ — Confidential Computing Foundation,” [Online]. Available: <a href="https://confidentialcomputing.io/faq/">https://confidentialcomputing.io/faq/</a>.<br>[2] Linux Foundation, “Confidential Computing Consortium Establishes Formation with Founding Members and Open Governance Structure,” [Online]. Available: <a href="https://www.linuxfoundation.org/uncategorized/2019/10/confidential-computing-consortium-establishes-formation-with-founding-members-and-open-governance-structure/">https://www.linuxfoundation.org/uncategorized/2019/10/confidential-computing-consortium-establishes-formation-with-founding-members-and-open-governance-structure/</a>.<br>[3] Microsoft, “Confidential Consortium Framework,” [Online]. Available: <a href="https://www.microsoft.com/en-us/research/project/confidential-consortium-framework/">https://www.microsoft.com/en-us/research/project/confidential-consortium-framework/</a>.</p><p>Liked what you just read? Please subscribe to my mailing list on my website <a href="http://lalitadithya.com/">http://lalitadithya.com/</a> to learn about the latest and greatest in technology (No spam, I promise). Additionally, you can follow me on social media using the links below:</p><p><a href="https://www.facebook.com/thelalitadithya/">Facebook</a>, <a href="https://twitter.com/thelalitadithya">Twitter</a>, <a href="https://www.instagram.com/lalitadithya/">Instagram</a>, <a href="https://www.linkedin.com/in/lalitadithya/">LinkedIn</a>, <a href="https://github.com/lalitadithya/">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82207aad6ff1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>