<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sarath Sasidharan on Medium]]></title>
        <description><![CDATA[Stories by Sarath Sasidharan on Medium]]></description>
        <link>https://medium.com/@sarathsasidharan_92863?source=rss-16a2b00a345e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QA6bWOHpEesLkOlTxxn46Q.jpeg</url>
            <title>Stories by Sarath Sasidharan on Medium</title>
            <link>https://medium.com/@sarathsasidharan_92863?source=rss-16a2b00a345e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:33:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sarathsasidharan_92863/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Lake in Azure — Part 2]]></title>
            <link>https://medium.com/@sarathsasidharan_92863/data-lake-in-azure-part-2-fd1f6f3cb5ed?source=rss-16a2b00a345e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd1f6f3cb5ed</guid>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Sarath Sasidharan]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 07:05:47 GMT</pubDate>
            <atom:updated>2019-03-12T07:05:47.555Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Lake in Azure — Part 2</h3><p>This blog is a follow up of the <a href="https://medium.com/@sarathsasidharan_92863/data-lake-creation-in-azure-df195f52d75b">previous</a> one which focused on infrastructure as code ,</p><p>This blog focuses on :</p><ul><li>Data loading / Extraction</li><li>Data Preparation / ETL</li><li>Data Sink</li></ul><h4>Prerequisites</h4><p>Assuming that you have executed data lake in azure part 1 of the blog series.</p><ul><li>Azure Data Lake Store Gen2</li><li>Azure Data Factory Gen 2</li><li>Azure Databricks</li><li>Azure SQL DB</li><li>Azure SQL Data Warehouse</li></ul><p>Since this blog is focused on integration we have a simple use case of joining product data from a company to the product category data ( assuming this comes from an external source). The data set being used is the one which is from the sample database which was created in SQL DB.</p><p><strong>Data Loading / Extraction</strong></p><p>This part of the blog focuses on :</p><ol><li>Dump Data From SQL DB ( Dimension) data onto Azure Data Lake Store using an orchestrator ( Azure Data Factory)</li><li>Dump Data from an external Data Source into the same storage layer .</li></ol><p>Click on the resource group blade on your left panel at portal.azure.com. This should list the resource group which was created in the previous blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rx1YJ13tAzP2yi-SqfP9lw.png" /></figure><p>Next, click on the azure data factory which was created , and then under this click on Author and Monitor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZLeFiEBvfabJCLz038wMWw.png" /></figure><p>This opens up a the new UI for Azure Data Factory. To learn more about this service , you can find extensive documentation here.</p><p><a href="https://docs.microsoft.com/en-us/azure/data-factory/">Azure Data Factory Documentation - Azure Data Factory</a></p><p>To fake data on an external storage , you need to create a new storage account (ADLS Gen2) and then use the copy activity in data factory to copy the data from SQL DB to this new storage account which you have created. The table which needs to be copied here is the “Product Category” Table.</p><p>To Create a new pipleine click on … beside the pipeline blade , this should launch a new canvas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*7diBqIn2AeAX5Mp83W7j3w.png" /></figure><p>Once this new canvas is launched , add a new name to the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObqYlDhdeCK2S9HMKW9e4g.png" /></figure><p>Add a new activity which will the first step in the pipeline , this copies data from external source to the central ADLS Gen2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o13eGI99LaaH51JW6581zA.png" /></figure><p>Drag this copy data activity on to the canvas and rename the activity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3dPnn4WmbH-s9EsH8G2_A.png" /></figure><p>Some work for you to explore , complete this exercise to :</p><ol><li>Copy the data from SQL DB (Product Category) to the external storage which was created in this blog , store this on a folder inside the central store under product_category folder.</li><li>copy the data from the SQL DB (Product Table) to the central data lake store under product folder.</li></ol><p>The end state for this pipeline is , make sure after each step of the pipeline select the validate and publish only then the changes will be saved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bF27ykfBX5VAFyHtViP2mA.png" /></figure><p>Trigger this step manually to see if the pipeline works. This pipeline :</p><ul><li>copies data from External store to the central data store</li><li>copies data from SQL DB (Relational) to the central data store for processing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nZXWLOmzVdm3ZmiY_Td3Yw.png" /></figure><p><strong>ETL Processing</strong></p><p>This step combines the external data and the SQL DB data which has been copied to the central data lake store.</p><p>Azure databricks is used for this.</p><p><a href="https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks">What is Azure Databricks?</a></p><p>Click on the resource group created , and then on the databricks instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpeY5ukPqz1hM3CLEDZ64Q.png" /></figure><p>After login , Create a new new cluster on databricks.</p><p><a href="https://docs.azuredatabricks.net/user-guide/clusters/create.html">Create a Cluster - Databricks Documentation</a></p><p>Create a new notebook</p><p><a href="https://docs.azuredatabricks.net/user-guide/notebooks/notebook-manage.html#create-a-notebook">Managing Notebooks - Databricks Documentation</a></p><p>Once this is done , copy this snippet of code into your notebook.</p><pre>spark.conf.set(<br>  &quot;fs.azure.account.key.&lt;<strong>NAME_OF_STORAGE</strong>&gt;.dfs.core.windows.net&quot;,<br>  &#39;&lt;<strong>ENTER_STORAGE_KEY</strong>&gt;&#39;)<br>spark.conf.set(&quot;fs.azure.createRemoteFileSystemDuringInitialization&quot;, &quot;true&quot;)<br>dbutils.fs.ls(&quot;abfss://<a href="/service/mailto:datalake@datalakecentral.dfs.core.windows.net">&lt;<strong>NAME_OF_FILESYSTEM</strong>&gt;@</a>&lt;<strong>NAME_OF_STORAGE</strong>&gt;<a href="/service/mailto:datalake@datalakecentral.dfs.core.windows.net">.dfs.core.windows.net</a>/&quot;)<br>spark.conf.set(&quot;fs.azure.createRemoteFileSystemDuringInitialization&quot;, &quot;false&quot;)</pre><p>This piece of code allows you to read from ADLS Gen2 using keys , this is not a recommended approach but in later blogs we will talk about best practices , security and secrets.</p><pre>product          = spark.read.csv(&quot;abfss://<a href="/service/mailto:product@datalakecentral.dfs.core.windows.net">&lt;<strong>NAME_OF_CONTAINER</strong>&gt;@</a>&lt;<strong>NAME_OF_STORAGE</strong>&gt;<a href="/service/mailto:product@datalakecentral.dfs.core.windows.net">.dfs.core.windows.net</a>/product&quot;,header=True)<br>product_category = spark.read.csv(&quot;abfss://<a href="/service/mailto:product@datalakecentral.dfs.core.windows.net">&lt;<strong>NAME_OF_CONTAINER</strong>&gt;@</a>&lt;<strong>NAME_OF_STORAGE</strong>&gt;<a href="/service/mailto:product@datalakecentral.dfs.core.windows.net">.dfs.core.windows.net</a>/product_category&quot;,header=True)</pre><p>This reads the data into 2 data frames and loads headers.</p><pre>productTab         = product.createOrReplaceTempView(&#39;product&#39;)<br>productCategoryTab = product_category.createOrReplaceTempView(&#39;product_category&#39;)<br>result_val = spark.sql(&#39;select a.*,b.ParentProductCategoryID from product a join product_category b on a.ProductCategoryID=b.ProductCategoryID&#39;)</pre><p>The next step is to store the data back into the data lake. This stores it inside a folder product_category_out.</p><pre>result_val.write.csv(&quot;abfss://<a href="/service/mailto:product@datalakecentral.dfs.core.windows.net">&lt;<strong>NAME_OF_CONTAINER</strong>&gt;@</a>&lt;<strong>NAME_OF_STORAGE</strong>&gt;<a href="/service/mailto:product@datalakecentral.dfs.core.windows.net">..dfs.core.windows.net</a>/product_category_out&quot;,header=True)</pre><p>Once this notebook is saved , this can be called from Azure Data Factory and appended to the pipeline which was created earlier.</p><p>Refer to this documentation to get this done.</p><p><a href="https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook">Run a Databricks Notebook with the activity - Azure Data Factory</a></p><p>Once its complete , you should have a pipeline similar to this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*upB0MLvay0z2QF14LGBEPw.png" /></figure><p>Trigger this pipeline and then go to the monitor tab to see the progress of this job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BECm8fSV4qHLrG5Ypz2Chw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd1f6f3cb5ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Lake Creation In Azure]]></title>
            <link>https://medium.com/@sarathsasidharan_92863/data-lake-creation-in-azure-df195f52d75b?source=rss-16a2b00a345e------2</link>
            <guid isPermaLink="false">https://medium.com/p/df195f52d75b</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <dc:creator><![CDATA[Sarath Sasidharan]]></dc:creator>
            <pubDate>Mon, 11 Mar 2019 18:49:47 GMT</pubDate>
            <atom:updated>2019-03-13T18:17:35.429Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog focuses on how to create a data lake on Azure using a script. This enhances developer productivity and quick deployment of data lakes using the concept of IaC ( Infrastructure as Code).</p><p>This would cover :</p><ul><li>Creation of storage</li><li>Creation of an Orchestrator / Worklow Engine</li><li>Creation of relational database (OLTP)</li><li>Creation of an ETL / Compute Engine</li><li>Creation of an Massively Parallel Processing Data warehouse (OLAP)</li><li>Single shell script when executed creates all of the above resources and bundles them in a resource group.</li></ul><h4>Prerequisites</h4><p>IDE , in this case <a href="https://code.visualstudio.com/?wt.mc_id=DX_841432">visual studio code</a>.</p><p><a href="https://azure.microsoft.com/en-us/free/">Azure account</a></p><p><strong>Create a new project</strong></p><p>Create a new folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*iNtNmVFZwO7G0IPIjq9i7g.png" /></figure><p>Open Visual Studio Code by typing in :</p><pre>code .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*k5MPXRIoyGHIOOhbMm5ycw.png" /></figure><p>Create a folder inside this project named infra , create a script inside this named data_lake.sh , this script will contain the steps required to setup the infrastructure needed for a data lake.</p><p>In this tutorial we are using az commands to create the resources , there are other ways on how resources could be created in azure which includes ,</p><p><a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates">ARM</a></p><p><a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy">Powershell</a></p><p><a href="https://www.terraform.io/docs/providers/azurerm/">Terraform</a></p><p><a href="https://docs.microsoft.com/en-us/rest/api/azure/">Azure REST APIs</a></p><p>First step in the data lake creation is to create a data lake store.</p><p><strong>Creation of Storage</strong></p><p>The qualities of the storage layer need to be :</p><ul><li>Easily Scalable</li><li>Cheap Storage</li><li>Easy integration with Services (Both Open Source and Proprietary Services)</li><li>Folder Restrictions based on POSIX to restrict access on folder levels.</li><li>Easy sink for all types of data sources ( Streaming as well as Batch)</li></ul><p>For this blog , we have Azure Data Lake Store Gen 2 as the raw storage layer for all our data flowing into our data lake.</p><p>An Introduction to this can be found here</p><p><a href="https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction">Azure Data Lake Storage Gen2 Introduction</a></p><p>This data lake supports POSIX level file systems permission which can be set using Active Directory Entities , which can help control access in a very scalable and elegant manner.</p><p>Create a resource group which contains logical grouping of resources which are spun up on azure, click to learn more about <a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview">resource group</a>.</p><p>Copy the snippet of code as below and paste it inside the data_lake.sh file , replace the variable names with the correct values.</p><pre>#!/bin/bash</pre><pre>DATA_LAKE_RG=&#39;&lt;<strong>REPLACE_WITH_DATA_LAKE_RESOURCE_GROUP_NAME</strong>&gt;&#39;<br>DATA_LAKE_LOCATION=&#39;&lt;<strong>REPLACE_WITH_DATA_LAKE_REGION</strong>&gt;&#39;</pre><pre># Create a new Resource Group<br>az group create \<br>--name $DATA_LAKE_RG \<br>--location $DATA_LAKE_LOCATION</pre><p>This command uses <a href="https://docs.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest">az cli</a> to request creation of a new resource group with name specified with the “- -name” flag and also specify the location of the resource group using the “- -location” flag. Location is the the region where the data center of Azure is put up. This would request the resources to be spun up in that particular region during the creation.</p><p>To run this on azure , you can :</p><ol><li>Run it from your terminal by using az login , and then executing the shell script (If you have a linux machine)</li><li>Login to the azure portal (portal.azure.com) and use the cloud shell. There are two options in the cloud shell , one is bash shell and the other is a powershell.</li></ol><p>This blog we use option 2 with bash shell, but feel free to play around with option 1 if you have a linux machine.</p><p>Go to the portal and then click on the bash shell to open a new window , where you can execute bash commands. Follow the steps as mentioned below to request a new cloud shell from the portal.</p><p><a href="https://docs.microsoft.com/en-us/azure/cloud-shell/overview">Azure Cloud Shell overview</a></p><p><strong>Script Execution</strong></p><p>Once you have the shell import the script (data_lake.sh) using the selection on the window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xb5aBEMHifpsqJ4aDXjpxw.png" /></figure><p>Upload the file data_lake.sh.</p><p>If you use a windows machine , issue the dos2unix command and also grant execute rights on the script , to do this run the following command.</p><pre>dos2unix data_lake.sh<br>chmod +x data_lake.sh</pre><p>Run the shell script , this will create a new resource group with the name specified in the azu<a href="https://azure.microsoft.com/en-us/global-infrastructure/locations/">re location</a>.</p><pre>./data_lake.sh</pre><p>After this script succeeds , execute the list command to see if the resource group created is present.</p><pre>az group list</pre><p>This can also be viewed in the azure portal from portal.azure.com.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*LfTHNNj1fi5SKl32M5sy3A.png" /></figure><p>Setup a storage account within this resource group , this would be the raw storage layer for the data lake. Add this snippet to the data_lake.sh.</p><p>Before running this on the cloud shell run the following command on the cloud shell.</p><pre>az extension add --name storage-preview</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*WELfVh2gEl6vwfBdyrtWmA.png" /></figure><p>This adds the extension for Azure Cli needed to install ADLS Gen2 .</p><p>Copy this snippet into data_lake.sh script , replace storage names with the right values.</p><pre>DATA_LAKE_STORAGE=&#39;&lt;<strong>REPLACE_WITH_DATA_LAKE_STORAGE_NAME</strong>&gt;&#39;</pre><pre># Create a new storage (ADLS Gen2) Account</pre><pre>az storage account create \<br>--name $DATA_LAKE_STORAGE  \<br>--resource-group $DATA_LAKE_RG  \<br>--location $DATA_LAKE_LOCATION  \<br>--sku Standard_LRS \<br>--kind StorageV2  \<br>--hierarchical-namespace true</pre><p>Repeat step “<strong>Script Execution</strong>” from above .</p><p>This should create a new storage account. View the resource using the following command.</p><pre>az resource list -g &lt;<strong>NAME_OF_RESOURCE_GROUP</strong>&gt;</pre><p>The new storage should be visible here , this can also be viewed from the azure portal under storage accounts blade on the left side of the portal.</p><p><strong>Creation of an Orchestrator / Pipeline / Workflow builder</strong></p><p>Once the storage account is ready the next step is to create a tool which can help in loading data to the storage layer.</p><p>What we need to keep in mind for this are :</p><ul><li>Connectors available from varying sources of input can range from legacy like mainframes to latest sources like snowflake for ex.</li><li>Ease of usage</li><li>Quick / Rapid pipeline development</li><li>Lineage</li><li>Availability of Triggers ( Time Based / Event Based / File Trigger /Rolling window ..etc)</li><li>Support for Steaming and Batch data flows</li><li>Connectors to push back data to different sinks</li><li>Integration with version control</li></ul><p>For this blog we use Azure Data Factory Gen 2 . Read through the documentation before your proceed further to familiarize what ADF Gen2 can and cannot do , this will help during the selection of the right tool.</p><p><a href="https://docs.microsoft.com/en-us/azure/data-factory/">Azure Data Factory Documentation - Azure Data Factory</a></p><p>Az cli commands for creating the azure data factory are not yet available so we will be using ARM template to create this resource.</p><p>To know more about ARM , this is the documentation :</p><p><a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/">Resource Manager on Azure documentation - Tutorials, samples, reference, and resources</a></p><p>Create a new folder inside the project named “arm” inside the infra folder. The folder structure is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*_94eR1ihnoq6Z4uoeQQD-A.png" /></figure><p>This folder holds the ARM template used to create the data factory instance. Create a new file (json) inside arm folder and then copy the contents into the new file.</p><pre>{<br>    &quot;$schema&quot;: &quot;<a href="https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#">https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#</a>&quot;,<br>    &quot;contentVersion&quot;: &quot;1.0.0.0&quot;,<br>    &quot;parameters&quot;: {<br>        &quot;name&quot;: {<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;defaultValue&quot;: &quot;&lt;<strong>NAME_OF_DATA_FACTORY</strong>&gt;&quot;<br>        },<br>        &quot;location&quot;: {<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;defaultValue&quot;: &quot;&lt;<strong>REPLACE_WITH_DATA_FACTORY_REGION</strong>&gt;&quot;<br>        },<br>        &quot;apiVersion&quot;: {<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;defaultValue&quot;: &quot;2018-06-01&quot;<br>        }<br>    },<br>    &quot;resources&quot;: [<br>        {<br>            &quot;apiVersion&quot;: &quot;[parameters(&#39;apiVersion&#39;)]&quot;,<br>            &quot;name&quot;: &quot;[parameters(&#39;name&#39;)]&quot;,<br>            &quot;location&quot;: &quot;[parameters(&#39;location&#39;)]&quot;,<br>            &quot;type&quot;: &quot;Microsoft.DataFactory/factories&quot;,<br>            &quot;identity&quot;: {<br>                &quot;type&quot;: &quot;SystemAssigned&quot;<br>            }<br>        }<br>    ]<br>}</pre><p>For explanation of this json and the elements in it , refer to the documentation link above.</p><p>Replace the variable values in the json and then save this file. This json also refers to a properties file where the values of the parameters are obtained.</p><p>Create a new folder named conf at the root level and create a new file inside this folder.</p><p>Copy these contents into the properties file.</p><pre>{<br>    &quot;$schema&quot;: &quot;<a href="https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#">https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#</a>&quot;,<br>    &quot;contentVersion&quot;: &quot;1.0.0.0&quot;,<br>    &quot;parameters&quot;: {<br>        &quot;name&quot;: {<br>            &quot;value&quot;: &quot;&lt;<strong>NAME_OF_DATA_FACTORY</strong>&gt;&quot;<br>        },<br>        &quot;location&quot;: {<br>            &quot;value&quot;: &quot;&lt;<strong>REPLACE_WITH_DATA_FACTORY_REGION</strong>&gt;&quot;<br>        },<br>        &quot;apiVersion&quot;: {<br>            &quot;value&quot;: &quot;2018-06-01&quot;<br>        }<br>    }<br>}</pre><p>These values get substituted into the run time values for the parameters in the main ARM template.</p><p>Once the templates are ready we need to add this deployment instruction to the script , <strong>infra/data_lake.sh</strong></p><pre>DATA_LAKE_WORKFLOW=&#39;datalakeorchestrator&#39;<br>ARM_LOCATION=&#39;arm/data_factory.json&#39;<br>ARM_PROPS_LOCATION=&#39;../conf/data_factory_prop.json&#39;</pre><pre># Create Data Factory Version 2 <br>az group deployment create \<br> --name $DATA_LAKE_WORKFLOW \<br> --resource-group $DATA_LAKE_RG \<br> --template-file $ARM_LOCATION \<br> --parameters $ARM_PROPS_LOCATION</pre><p>This command deploys a Azure Data Factory Gen 2 using the template referred by the “- -template-file” argument , the properties file is referred by “- -parameters”.</p><p>The best way to transport this to the cloud shell is to commit this to a git repository and clone this to your azure cloud shell.</p><p>Check if the data factory has been created. This can also be viewed in the portal.</p><pre>az resource list -g &lt;<strong>NAME_OF_RESOURCE_GROUP</strong>&gt;</pre><p><strong>Create a Relational Database</strong></p><p>Source systems could be Relational Databases which can release data to the data lake.</p><p>Orchestrators / workflow engines could pull data from these systems onto the data lake storage to create a enterprise wide data sharing platform aka enterprise data lake.</p><p>For this part we use SQL DB (Managed SQL Server) on azure as our Relational Database.</p><p><a href="https://docs.microsoft.com/en-us/azure/sql-database/">Azure SQL Database Documentation - Tutorials, API Reference</a></p><p>In order to create this add this to the <strong>infra/data_lake.sh</strong> script , under the infra folder.</p><p>Replace the variable names with appropriate names. Ideally these variables could be stored in a configuration file , secrets can be stored and retrieved from azure <a href="https://docs.microsoft.com/nl-nl/azure/key-vault/key-vault-overview">keyvault</a>.</p><pre>DATA_LAKE_SERVER_NAME=&#39;&lt;<strong>SQL_SERVER_NAME</strong>&gt;&#39;<br>DATA_LAKE_DATABASE_NAME=&#39;&lt;<strong>SQL_DB_NAME</strong>&gt;&#39;<br>ADMIN_USERNAME=&#39;&lt;<strong>ENTER_USERNAME</strong>&gt;&#39;<br>ADMIN_PASSWD=&#39;&lt;<strong>ENTER_PASSWORD</strong>&gt;&#39;</pre><pre>## Create a logical server in the resource group<br>az sql server create \<br> --name $DATA_LAKE_SERVER_NAME \<br> --resource-group $DATA_LAKE_RG \<br> --location $DATA_LAKE_LOCATION  \<br> --admin-user $ADMIN_USERNAME \<br> --admin-password $ADMIN_PASSWD</pre><pre>## Create a database in the server <br>az sql db create \<br> --resource-group $DATA_LAKE_RG \<br> --server $DATA_LAKE_SERVER_NAME \<br> --name $DATA_LAKE_DATABASE_NAME \<br> --sample-name AdventureWorksLT \<br> --service-objective S0</pre><pre># Configure a firewall rule for the server<br>az sql server firewall-rule create \<br> --resource-group $DATA_LAKE_RG \<br> --server $DATA_LAKE_SERVER_NAME \<br> -n AllowYourIp \<br> --start-ip-address 0.0.0.0 \<br> --end-ip-address 0.0.0.0</pre><p>Once this is ready execute , to see the resources.</p><pre>az resource list -g &lt;<strong>NAME_OF_RESOURCE_GROUP</strong>&gt;</pre><p><strong>Create an ETL Engine</strong></p><p>Once the storage and orchestrator have been defined the next core component in the data lake is an ETL / ELT tool.</p><p>Crucial topics in choosing an ETL engine :</p><ul><li>Should be scalable / On Demand</li><li>Should provide flexibility to process both structured and unstructured data</li><li>Should be easy to develop / script jobs</li><li>Integration with source control systems</li><li>Should be able to handle stream and batch flow of data</li></ul><p>For this blog , we go with Azure Databricks. Follow this for documentation.</p><p><a href="https://docs.microsoft.com/en-us/azure/azure-databricks/">Azure Databricks documentation</a></p><p>Similar to Azure data factory we will use ARM (Azure Resource Manager ) Template to create an Azure Databricks instance.</p><p>Under the <strong>/infra/arm</strong> folder create a new file of your choice for example , databricks.json.</p><p>Copy the following snippet into the file.</p><pre>{<br>    &quot;$schema&quot;: &quot;<a href="https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#">https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#</a>&quot;,<br>    &quot;contentVersion&quot;: &quot;1.0.0.0&quot;,<br>    &quot;parameters&quot;: {<br>      &quot;workspaceName&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;metadata&quot;: {<br>          &quot;description&quot;: &quot;The name of the Azure Databricks workspace to create.&quot;<br>        }<br>      },<br>      &quot;pricingTier&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;defaultValue&quot;: &quot;premium&quot;,<br>        &quot;allowedValues&quot;: [<br>          &quot;standard&quot;,<br>          &quot;premium&quot;<br>        ],<br>        &quot;metadata&quot;: {<br>          &quot;description&quot;: &quot;The pricing tier of workspace.&quot;<br>        }<br>      },<br>      &quot;location&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;defaultValue&quot;: &quot;[resourceGroup().location]&quot;,<br>        &quot;metadata&quot;: {<br>          &quot;description&quot;: &quot;Location for all resources.&quot;<br>        }<br>      }<br>    },<br>    &quot;variables&quot;: {<br>      &quot;managedResourceGroupName&quot;: &quot;[concat(&#39;databricks-rg-&#39;, parameters(&#39;workspaceName&#39;), &#39;-&#39;, uniqueString(parameters(&#39;workspaceName&#39;), resourceGroup().id))]&quot;<br>    },<br>    &quot;resources&quot;: [<br>      {<br>        &quot;type&quot;: &quot;Microsoft.Databricks/workspaces&quot;,<br>        &quot;name&quot;: &quot;[parameters(&#39;workspaceName&#39;)]&quot;,<br>        &quot;location&quot;: &quot;[parameters(&#39;location&#39;)]&quot;,<br>        &quot;apiVersion&quot;: &quot;2018-04-01&quot;,<br>        &quot;sku&quot;: {<br>          &quot;name&quot;: &quot;[parameters(&#39;pricingTier&#39;)]&quot;<br>        },<br>        &quot;properties&quot;: {<br>          &quot;ManagedResourceGroupId&quot;: &quot;[concat(subscription().id, &#39;/resourceGroups/&#39;, variables(&#39;managedResourceGroupName&#39;))]&quot;<br>        }<br>      }<br>    ],<br>    &quot;outputs&quot;: {<br>      &quot;workspace&quot;: {<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;value&quot;: &quot;[reference(resourceId(&#39;Microsoft.Databricks/workspaces&#39;, parameters(&#39;workspaceName&#39;)))]&quot;<br>      }<br>    }<br>  }</pre><p>This is the code which requests a new databricks workspace from azure. There is also a properties file which needs to be added , this is under <strong>/conf/ </strong>directory. Create a new file for ex : databricks_prop.json and add the following snippet. Substitute the name of the databricks instance.</p><pre>{<br>    &quot;$schema&quot;: &quot;<a href="https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#">https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#</a>&quot;,<br>    &quot;contentVersion&quot;: &quot;1.0.0.0&quot;,<br>    &quot;parameters&quot;: {<br>        &quot;workspaceName&quot;: {<br>          &quot;value&quot;: &quot;&lt;<strong>ENTER_NAME_OF_DATABRICKS_INSTANCE</strong>&gt;&quot;<br>        }<br>    }<br>}</pre><p>Once this is done , add the trigger for these into our <strong>/infra/data_lake.sh</strong> script</p><p>Copy this snippet of code and append it to the script.</p><pre>DATA_LAKE_WORKFLOW_DB=&#39;&lt;<strong>ENTER_ANY_DEPLOYMENT_NAME</strong>&gt;&#39;<br>ARM_LOCATION_DB=&#39;arm/&lt;<strong>NAME_OF_ARM_TEMPLATE</strong>&gt;&#39;<br>ARM_PROPS_LOCATION_DB=&#39;../conf/&lt;<strong>NAME_OF_PROPERTY_FILE</strong>&gt;&#39;</pre><pre># Create Azure Databricks <br>az group deployment create \<br> --name $DATA_LAKE_WORKFLOW_DB \<br>        --resource-group $DATA_LAKE_RG \<br> --template-file $ARM_LOCATION_DB \<br> --parameters $ARM_PROPS_LOCATION_DB</pre><p>If you use git the commit this code to git and then pull it from cloud shell and execute this script. On successful execution , check the resources using. Newly created databricks instance should be visible now , this can also be seen via the portal.</p><pre>az resource list -g &lt;<strong>NAME_OF_RESOURCE_GROUP</strong>&gt;</pre><p><strong>Creation of an Massively Parallel Processing Data warehouse (OLAP)</strong></p><p>Once the data is prepared and ready to be served an MPP data warehouse needs to be provisioned to expose the data to visualization tools.</p><p>The core requirements would be :</p><ul><li>Highly scalable Data Warehouse</li><li>Easy to tune up performance and tune down when not needed to avoid extra costs</li><li>Easy connectors to Raw storage</li><li>SQL based query engine</li></ul><p>For this part of the blog , SQL Data warehouse is considered.</p><p><a href="https://docs.microsoft.com/en-us/azure/sql-data-warehouse/">Azure SQL Data Warehouse Documentation - Tutorials, Concepts</a></p><p>This can be created using the az command , append the following snippet to the script data_lake.sh.</p><pre>DATA_LAKE_DWH_NAME=&#39;&lt;<strong>ENTER_SQL_DWH_NAME</strong>&gt;&#39;</pre><pre># Create a SQL Datawarehouse<br>az sql dw create \<br> --resource-group $DATA_LAKE_RG \<br> --server $DATA_LAKE_SERVER_NAME \<br> --name $DATA_LAKE_DWH_NAME</pre><p>Commit this back into git and execute the script again , check the list of resources inside the resource group. This should include the newly created SQL Data Warehouse.</p><p>Adjust the script accordingly and do the required cleanup , refer a sample git repository which has been used by me for this blog.</p><p><a href="https://github.com/sarathsasidharan/datalake">sarathsasidharan/datalake</a></p><p>This script <strong>/infra/data_lake.sh , </strong>would be the single click deploy script for the data lake. It would deploy a storage account , an Azure Data Factory instance, Azure Databricks Instance, SQL Database and a SQL Data warehouse.</p><p>This script can be included in a Build pipeline (<a href="https://azure.microsoft.com/en-us/services/devops/">Azure Devops</a>) , which can enable automated rollouts to Test / Acceptance / Production or also as a templates to teams within the organization to start building their data lakes.</p><p>The next blog focuses on :</p><ul><li>Data loading / Extraction</li><li>Data Preparation</li><li>Data Sink</li></ul><p>The series of blogs to follow will build up on this blog with other factors like Monitoring / controls / nomenclatures / DevOps practices for data lake / Data Management etc. As a follow up of this blog you can go <a href="https://medium.com/@sarathsasidharan_92863/data-lake-in-azure-part-2-fd1f6f3cb5ed">here</a> for part 2 of this blog series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df195f52d75b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enable CI/CD for Data Science Projects]]></title>
            <link>https://faun.pub/enable-ci-cd-for-data-science-projects-4f5b61acbfc9?source=rss-16a2b00a345e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f5b61acbfc9</guid>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Sarath Sasidharan]]></dc:creator>
            <pubDate>Tue, 18 Dec 2018 21:47:56 GMT</pubDate>
            <atom:updated>2019-07-17T20:05:27.246Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial will show you how can you create an end to end CI/CD project for data science.</p><p>We will create a sample project which showcases how to :</p><ol><li>Define a Data Science project Structure</li><li>Train the models on remote compute</li><li>Deploy the generated artifacts onto containers for consumption by other parties all using automated build and release pipelines.</li></ol><p>You can use the tools of your choice , since this is a concept. For this particular tutorial we will use :</p><h4><strong>Prerequisites</strong></h4><p>IDE , in this case visual studio code</p><p><a href="https://azure.microsoft.com/en-us/free/">Azure account</a></p><p><a href="https://visualstudio.microsoft.com/team-services">Azure DevOps Account</a></p><p><a href="https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-workspace">Azure Machine Learning Service Workspace</a></p><p>Python</p><h4><strong>Create a Data Science Project</strong></h4><p>We are going to use the famous MNIST dataset to train a model to predict numbers within images and then later deploy this to a container.</p><p>Create a new directory .</p><p>Clone a project template which you could use as a base template for your project the other directory is the one which will point to your own repository. Replace the &lt;&gt; with your directory names.</p><pre>md csu-geekster\template<br>md csu-geekster\&lt;name_project_directory&gt;<br>cd csu-geekster\template<br>git clone <a href="https://github.com/sarathsasidharan/devopsai.git">https://github.com/sarathsasidharan/devopsai.git</a><br>cd ..</pre><p>Create a new project in Azure DevOps.</p><p><a href="https://docs.microsoft.com/en-us/azure/devops/organizations/projects/create-project?view=vsts&amp;tabs=new-nav">Create a project - Azure DevOps</a></p><p>Click on the repository folder on the left pane as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*hqjmj2w2emDtiPZLEffRCw.png" /></figure><p>Create a new directory on your machine and then clone this new project into your local machine.</p><pre>cd &lt;name of your azure devops project directory&gt;<br>git clone &lt;remote repository&gt;<br>cd &lt;name of your devops project&gt;<br>notepad except.txt</pre><p>Add .git to the except.txt and save the file.</p><p>Copy the files from the template folder to the devops project folder . Add and commit the files to the azure devops project</p><pre>xcopy c:\Users\sasasid\csu-geekster\template\devopsai c:\Users\sasasid\csu-geekster\dsproject\csu-geekster /E /exclude:except.txt</pre><pre>git commit -am &quot;project template updated&quot;<br>git push</pre><p>After the project structure is ready , we are ready to start developing our data science project.</p><p>Open the project in the IDE of your choice.</p><p>Install the azure ml libraries using pip install , this assumes python to be installed on the machine</p><pre>pip install azure-cli==2.0.46<br>pip install --upgrade azureml-sdk[notebooks,automl]</pre><p>Edit the project structure if need be or feel free to use the structure as such or tweak based on your company standards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*Pb2d3qfi0bWezbbF49ur8w.png" /></figure><p>Open a new terminal and commit the changes into the azure dev-ops repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z3O-iOSiOKfbxha_zmQmnw.png" /></figure><pre>git add .<br>git commit - m &quot;Changed folder names according to comp standards&quot;<br>git push</pre><p>Open the config directory and fill in the values corresponding to the keys within the config file.</p><p><em>Subscription id </em>: <a href="https://blogs.msdn.microsoft.com/mschray/2016/03/18/getting-your-azure-subscription-guid-new-portal/">Link to find this</a></p><p><em>Resource Group Name</em> : Name of the resource group where AML Service Workspace was created</p><p>Workspace Name : Name of the workspace which was created in AML Service Workspace.</p><p>Location : Location of where the aml resource has been created , ex: westeurope</p><p>Once this is done the first script to edit is the data preparation script. This is under the <strong><em>dataprep</em></strong> folder. This script downloads the data from a remote store onto a storage account (azure).</p><p>This script uses a service principal to connect to azure ml service , first we need to create a service principal on azure.</p><p>Click on Azure Active Directory as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*Hu9QS07OCONJ2yH1hfrNhQ.png" /></figure><p>Select App Registrations and create a new Service Principal. After this is done create a new key which will be used in this script to connect to azure ml service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEoW_NcvHrEOf_vbRLtWgA.png" /></figure><p>Select the keys tab and copy the key which is generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6f2TFY2lMntjZ6ogFuFnLg.png" /></figure><p>Close this tab , copy the application id from this page , this will be used in the script .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WAQ6SIeO5etIJxeqAu5JCg.png" /></figure><p>Inside dataPrep.py replace the values in the variables with the values you have collected in the previous steps , namely app_id needs to be replaced with the application id , app_key needs to be replaced by the application key generated in the previous step.</p><pre>tenant_id=&quot;&lt;Enter Your Tenant Id&gt;&quot;<br>app_id=  Replace this with your application Id<br>app_key= Replace with the key generated in the previous step<br>workspace=&quot;&lt;Name of your workspace&gt;&quot;<br>subscription_id=&quot;&lt;Subscription id&gt;&quot;<br>resource_grp=&quot;&lt;Name of your resource group where aml service is created&gt;&quot;<br>experiment_name = &#39;&lt;Name of your experiment&gt;&#39;</pre><p>Next , provide contributor rights for this Service Principal on the resource group which has the aml service.</p><p>In order to do this , first go to resource group tab and then click on the resource group which holds aml service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SmwzuzorI_3ZM6_9SHLckA.png" /></figure><p>Click the IAM tab and add the service principal created in the previous step and give contributor rights on this resource group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QZrnzR-7zetqTmG5tfoWkA.png" /></figure><p>Once this has been updated , go back to your project code.</p><p>Next script to edit is under <strong><em>services , </em></strong>edit the file<strong><em> </em></strong>EstimatorTrigger.py. This script triggers the training process and also registers and saves the model output into azure ml service for later retrieval (during deployment). Aspects like accuracy , auc , data set used to train can all be logged using the azure ml sdk.</p><p>This feature is crucial for <strong>Model Management and Model tracking.</strong></p><p>Replace the variables with the values generated in the previous steps just as you have performed in the dataPrep.py script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqXzpAU2OeOeyhcXbtvENw.png" /></figure><p>The next script which needs to be edited is under the <strong><em>deploy</em></strong> folder.</p><p>Edit the deploy.py repeat the step which was done earlier , replace the variable with the corresponding values. This script does multiple things.</p><ol><li>Connect to aml service and downloads the model artifact</li><li>Creates as a conda dependency file which specifies which packages i need to include in my container</li><li>Creates a docker Image and pushes this to Azure Container Registry</li><li>Creates an Azure container instance.</li><li>Creates a Web service serving the model we created , hosted on Azure Container Instance.</li></ol><p>The next script to edit is named test.py tests. This tests the web service after it has been deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqXzpAU2OeOeyhcXbtvENw.png" /></figure><p>Push all your changes to azure devops.</p><pre>git add .<br>git commit -m &quot;Edited scripts for CI/CD&quot;<br>git push</pre><p>Next up we need to create our Continuous Integration pipeline in azure devops.</p><p>Go to <a href="https://visualstudio.microsoft.com/">https://visualstudio.microsoft.com/</a> and then click on the project which you have created. Click on Pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*uKvKNXJozA62vSxsXVVi-g.png" /></figure><p>Create a new build pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*jXSbLT1ulnwti81TsHAvdA.png" /></figure><p>Select the visual designer option , you could also prefer to use yaml , in this tutorial we stick to the visual option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*QHqDMivmLBbyamxOcs0Gqw.png" /></figure><p>In the next page click on continue , make sure you see your project repo selected.</p><p>In the next step select an empty job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*2--m4TSV9gQCXcl_00QU9A.png" /></figure><p>Give a name to your build pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QJjS-JxW5XXWzxgHQpT1g.png" /></figure><p>Click on the Agent Job and Change the name to something meaningful and also in this demo we choose the agent pool to be Hosted VS2017.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aGHVWeTxt42hvgzBUdNktQ.png" /></figure><p>Click on the + icon Under the Selected Agent Job , first step in our build is to create a conda environment on the agent.</p><p>Search for Create Conda in the serach box and click on add.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVlBfaAdNu1Xm24yT_DPOA.png" /></figure><p>Click on the task which was added and change the display name and also check Create a custom environment and name it. Also specify python version you want to use in this case 3.6</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*j954TeXZT4v-t5k6TvatVg.png" /></figure><p>Next click on the + again on the agent to add a new task and this time search for a bash script . This script is going to install dependencies required by our data preparation and training scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zznndl2kb08DJuMxAuGh9Q.png" /></figure><p>Select the script to be executed and also under advanced set the directory to the setup folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*U_QmxO05VGDdOrFrbe3-yg.png" /></figure><p>Add next task , again select + and then search for python script task. This task triggers our Data Preparation script which creates our remote compute and also prepares the data for model training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fGYrhkSr_8a5wX0d0dhNXA.png" /></figure><p>Under the script path select the script under data preparation and then also set the working directory under advanced option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A4He6tpisiyBsH3_r9uq7A.png" /></figure><p>Add a new python task and then select the script under service/EstimatorTrigger.py and set the current working directory to service directory. This script triggers our training step and when it has been successfully executed then stores the model artifact into Azure ML Service and also the best model is registered in Azure ML service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xAZvrTqO5HLeAgb8qvNiw.png" /></figure><p>After this is done add the task to copy the artifacts . This step packages our artifacts which can be used for deployment.</p><p>Add a new task and then search for copy files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KyIlEkPreQV1hQl62Paosg.png" /></figure><p>Add this task and then set the properties as shown.</p><p>Source Folder as : $(Build.SourcesDirectory)</p><p>Target Folder as: $(Build.ArtifactStagingDirectory)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OENvPPh8k2_2whRkW368nQ.png" /></figure><p>In the next step add a new task , search for publish build.This</p><p>Give it a meaningful name on what you want to name the artifact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJOeZlMyrA9Bky7CV_vBSw.png" /></figure><p>Next click on save an queue to trigger this build pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8UKC9iTvOkhsnR6xKOfeiA.png" /></figure><p>Once this is done you should be able to see the build job and click on builds to see the list of builds running and completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1RVK_cFjt14jqOBkDi6oQ.png" /></figure><p>You can also enable Continuous Integration by going back to the build definition , editing it and under triggers check the continuous integration tick box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JhbIWo5xDT2o-3zy1f8GOw.png" /></figure><p>This will trigger a build as soon as you commit anything new into the code base.</p><p>The next part is about deploying the Machine Learning Model as a Container on a Azure Container Instance.</p><p>Click on the Releases tab and then on new release pipeline</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSeJgGp3x4M4iJ-QZHfeXQ.png" /></figure><p>Select an empty job and click next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EvsLLQDQ4Zc4eKYg_CVnYg.png" /></figure><p>Enter a descriptive stage name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ol5-dCgjJ3ZQ1o2EWYrvLg.png" /></figure><p>Add an artifact which we would like to release in this pipeline. In this case select the artifact created from our previous build pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6x0vj8vKSXv43Tg3YC_jg.png" /></figure><p>Add a conda env , just like in the build</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVlBfaAdNu1Xm24yT_DPOA.png" /></figure><p>Next step is to add dependencies just like step 2 of build pipe. Make sure to set the script to be triggered and also the working directory under advanced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_j71vpcUAU3xUl0yNiU1Yw.png" /></figure><p>Next Step is to add a new python script task. Point the script path to a file named deploy.py and also set the working directory in advanced options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VzpzL0rsvnhP1RG7aEApTw.png" /></figure><p>The previous step deploys the model on an Azure Container Instance and then we need to test if the endpoint responds to a test call. That is what we do in the last step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCPA3DPj1WHqTx87NZVvSA.png" /></figure><p>Save and close the release pipeline , now release the pipeline.</p><p>After a successful run check the logs to see the response of our newly created web service.</p><p>After the runs are successful you can view them on azure ml service dashboard.</p><p>Once this trigger has been executed, the experiment and the runs are visible , csu-image-recognition is the experiment created using the script.</p><p>Click on the experiment to look at the runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUhyBxEl9rKmIAOYTgRlBQ.png" /></figure><p>Inside the experiment you can see the run and also compare different runs using interactive filters in the dashboard , all metrics logged inside the script against this run are available on this dashboard .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZvYtTlH-m4Vya0DZ5fosRA.png" /></figure><p>If you want to change the metrics being displayed against a run it is possible using the edit button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jnmz-IXqma6LWXpka2h1-A.png" /></figure><p>More documentation on this can be found at : <a href="https://docs.microsoft.com/en-us/azure/machine-learning/service/">https://docs.microsoft.com/en-us/azure/machine-learning/service/</a></p><p>The Compute used for this is also created ,you can view that under the Compute blade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*uwfi0f5FxwP_LKcY36637A.png" /></figure><p>The model registered along with the versions could also be viewed under the Models blade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*ekWTFnTYu61zZr93qPNVoA.png" /></figure><p>Image which was created is visible under Images blade each new image published by the script gets a newer version number , this can help to track released artifacts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QF9X0pNFS8s-bMWl1m48iQ.png" /></figure><p>Deployments on Azure Container instances / Azure Kubernetes Service / Edge can all be visible from here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7rPwjsurucIrDiCjHklSA.png" /></figure><p>To be covered next , is unit testing. Will be updating this post with the tests as soon as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Join our community Slack and read our weekly Faun topics ⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oSdFkACJxs5iy1oR" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f5b61acbfc9" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/enable-ci-cd-for-data-science-projects-4f5b61acbfc9">Enable CI/CD for Data Science Projects</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Process Semi Structured Data (JSON) using Azure Data Lake Analytics (USQL) and combine with…]]></title>
            <link>https://medium.com/@sarathsasidharan_92863/connect-azure-sql-data-warehouse-to-azure-data-lake-store-214aa14aef76?source=rss-16a2b00a345e------2</link>
            <guid isPermaLink="false">https://medium.com/p/214aa14aef76</guid>
            <category><![CDATA[azure-data-lake]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Sarath Sasidharan]]></dc:creator>
            <pubDate>Mon, 11 Jun 2018 07:15:32 GMT</pubDate>
            <atom:updated>2018-06-13T09:32:01.398Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Process Semi Structured Data (JSON) using Azure Data Lake Analytics (USQL) and combine with relational data on SQL Database</strong></p><p>In this blog we are going to cover on how you can combine structured and unstructured data using Azure Tooling.</p><p><strong>Use Case</strong></p><p>This scenario is about creating a dataset for a sports campaign by filtering out products from the pre-computed recommendation set stored as a JSON on azure data lake store and later combining it with the customer data present on a SQL DB.</p><p>The final data set will be the input for the application which needs to push recommendations to the customers</p><p>The data set used in this tutorial is from the Adventure works sample database in the Azure SQL Data DB and a Sample JSON file curated for this.</p><p><strong>Pre-requisites :</strong></p><p><strong>Azure SQL Data DB</strong></p><p>Follow this link to setup a new SQL Data DB. Make sure during the creation to select the sample database ( See the screenshot below for reference)</p><p><a href="https://docs.microsoft.com/en-us/azure/sql-database/sql-database-get-started-portal">Azure portal: Create a SQL database</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJvg_6QOM12pqEj8NcqQ6Q.png" /><figcaption>Select the Source as Sample Database</figcaption></figure><p><strong>Azure Data Lake Store</strong></p><p>Follow this link to setup a new Azure Data Lake Store</p><p><a href="https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-portal">Use Azure portal to get started with Data Lake Storage Gen1</a></p><p><strong>Azure Data Lake Analytics</strong></p><p>Follow this link to setup a new Azure Data Lake Analytics Account</p><p><a href="https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-get-started-portal">Get Started with Azure Data Lake Analytics using the Azure portal</a></p><p><strong>Next Steps :</strong></p><p>The JSON document needs to be processed using Azure Data Lake Analytics (U-SQL). The output from the U-SQL script will be stored back on the Data Lake Store in a relational format.</p><p>The next part of this tutorial will focus on combining this data with structured data which is already on the SQL DB.</p><p><strong>Process JSON Documents using U-SQL</strong></p><p>There are 2 libraries which needs to be uploaded to your Azure Data Lake Store in order to query JSON files using USQL, since this is not out of the box functionality , for now.</p><p>Download this from <a href="http://aka.ms/usql-libs">aka.ms/usql-libs</a></p><p>Unpack this zip file , there are 2 dll’s in these namely :</p><ul><li><strong>Newtonsoft.Json.dll</strong> (Documentation: <a href="https://www.newtonsoft.com/json">Json.NET</a>)</li><li><strong>Microsoft.Analytics.Samples.Formats.dll</strong> (Documentation: <a href="https://github.com/Azure/usql/tree/master/Examples/DataFormats/Microsoft.Analytics.Samples.Formats">GitHub</a>)</li></ul><p>Return to Azure portal and then click on the Azure Data Lake Store instance which you have created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p8HdUOecYLmv1vGWvhGqSw.png" /><figcaption>Click on All Resources link on the left pane</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZI_15-v7cpRKUfXR6emkYg.png" /><figcaption>Click on the data lake store account</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S0V2cMH42AkP9EYq8tVjUQ.png" /><figcaption>Select Data Explorer in order to browse the data folders</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WoaVm4yG1gk9CpZC3Dx1WA.png" /><figcaption>Click Upload and select the 2 Libraries which was downloaded previously</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1Lh4QKiDNeRTKTYviZ12g.png" /><figcaption>Upload the libraries</figcaption></figure><p>Upload the input files for the USQL script , in order to do this execute the following steps listed below :</p><ol><li>Navigate to Azure Data Lake Storage account</li><li>Click on the Data Explorer tab</li><li>Create a new Folder (Name this as data)</li><li>Click on this new Folder and then Upload Data</li><li>Upload the sample JSON Files which are provided at <a href="http://aka.ms/usql-data">aka.ms/usql-data</a></li></ol><p>Navigate to your <strong>Azure Data Lake Analytics</strong> resource (e.g. Azure Portal &gt; All Resources &gt; “Your Azure Data Lake Analytics”).</p><p>Click <strong>New Job.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4WGQ_MTXzszN9fp6NLDKwQ.png" /></figure><p>Copy the code into the canvas.</p><p>This piece of code does the following :</p><ol><li>loads 2 customer documents .</li><li>Uses the Libraries we uploaded to explode the fields</li><li>Filters out product recommendations which are only relevant to our sports campaign.</li></ol><p>Update the code by replacing the text “NAME_DATALAKE_STORE” with the name of your Azure Data Lake Store account before you submit the job</p><pre><em>// A. CREATE ASSEMBLY: Register assemblies (if they do not already exist).</em></pre><pre>CREATE ASSEMBLY IF NOT EXISTS [Newtonsoft.Json] FROM @&quot;adl://&lt;<strong>NAME_DATALAKE_STORE</strong>&gt;.azuredatalakestore.net/Newtonsoft.Json.dll&quot;;</pre><pre>CREATE ASSEMBLY IF NOT EXISTS [Microsoft.Analytics.Samples.Formats] FROM @&quot;adl://&lt;<strong>NAME_DATALAKE_STORE</strong>&gt;.azuredatalakestore.net/Microsoft.Analytics.Samples.Formats.dll&quot;;<br> </pre><pre><em>// B. REFERENCE ASSEMBLY: Load assemblies for compile time and execution.</em><br>REFERENCE ASSEMBLY [Newtonsoft.Json];<br>REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats];</pre><pre>// C. USING: Specify namespace to shorten function names (e.g. Microsoft.Analytics.Samples.Formats.Json.JsonExtractor)<br>USING Microsoft.Analytics.Samples.Formats.Json;</pre><pre>// 1. Initialise variables for Input (e.g. JSON) and Output (e.g. CSV).</pre><pre>DECLARE <a href="http://twitter.com/InputFile">@InputFile</a> string = @&quot;adl://&lt;<strong>NAME_DATALAKE_STORE</strong>&gt;.azuredatalakestore.net/data/{*}.json&quot;;</pre><pre>DECLARE <a href="http://twitter.com/OutputFile">@OutputFile</a> string = @&quot;adl://&lt;<strong>NAME_DATALAKE_STORE</strong>&gt;.azuredatalakestore.net/data/customer_filtered.csv&quot;;<br></pre><pre>// 2. Extract string content from JSON document (i.e. schema on read).<br><a href="http://twitter.com/json">@json</a> =<br>EXTRACT<br>    firstName string,<br>    lastName string,<br>    id int,<br>    products string<br>FROM<br>    <a href="http://twitter.com/InputFile">@InputFile</a><br>USING new Microsoft.Analytics.Samples.Formats.Json.JsonExtractor();</pre><pre><a href="http://twitter.com/products">@products</a> =<br>    SELECT firstName,<br>           lastName ,<br>           id ,<br>           Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(products).Values AS product_recom<br>    FROM <a href="http://twitter.com/json">@json</a>;</pre><pre><a href="http://twitter.com/final_products">@final_products</a> =<br>    SELECT firstName,<br>           lastName,<br>           id,<br>           Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(t_value) [&quot;type&quot;]AS recommendation_type,<br>           Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(t_value) [&quot;item&quot;]AS recommendation_item<br>    FROM <a href="http://twitter.com/products">@products</a><br>          CROSS APPLY<br>             EXPLODE(product_recom) AS T(t_value);</pre><pre><a href="http://twitter.com/filtered_campaign">@filtered_campaign</a> =<br>     SELECT firstName,<br>           lastName,<br>           id,<br>           recommendation_item <br>    FROM <a href="http://twitter.com/final_products">@final_products</a> WHERE recommendation_type==&quot;sport&quot;;</pre><pre>// 3. Write values to CSV<br>OUTPUT <a href="http://twitter.com/filtered_campaign">@filtered_campaign</a><br>TO <a href="http://twitter.com/OutputFile">@OutputFile</a><br>USING Outputters.Csv(outputHeader:true,quoting:false);</pre><p>Click <strong>Submit</strong>.</p><p>After the job is complete check the output by selecting the Data tab and then the output , click on the output file to view the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNsCEL0P_mr20P5jW9VE_Q.png" /></figure><p>Now the data from SQL DB needs to be sourced as an input for USQL and combined with the recommendations from our previous step to create the enriched data set.</p><p>In order to get this up and running , first create a authentication procedure between your Azure Data Lake Analytics Account and your SQL Data Warehouse.</p><p>Keep in mind you need to replace the variable name with your corresponding instance names.</p><p>Navigate to your <strong>Azure Data Lake Analytics</strong> resource (e.g. Azure Portal &gt; All Resources &gt; “Your Azure Data Lake Analytics”).</p><p>Click <strong>New Job.</strong></p><pre>CREATE DATABASE &lt;<strong>DATABASE_INSIDE_ANALYTICS_ACCOUNT</strong>&gt;;</pre><p>After the database has been created in your analytics account open a powershell prompt and execute the following commands.</p><p>Follow this link in case you do not have the azure powershell Cmdlets installed . If you do have it installed then you can skip this step</p><p><a href="https://docs.microsoft.com/en-us/powershell/azure/install-azurerm-ps?view=azurermps-6.2.0">Install Azure PowerShell with PowerShellGet</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*N_9dBZL7gOOQoFOVI9JaDg.png" /><figcaption>Open a Power Shell prompt and execute the code snippet below</figcaption></figure><p>Execute the code below sequentially , so first Login-AzureRmAccount. After this is executed a JSON is returned with the SubscriptionId which needs to be added as an input in the second command. Make sure that the SQLDB username and password are entered in the pop up when prompted for a username and password</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*wHtczSf6L4hfVW7bgltK2A.png" /><figcaption>The Subscription Id is found in the box which is highlighted in white</figcaption></figure><pre>Login-AzureRmAccount;<br>Set-AzureRMContext -SubscriptionId &lt;SubscriptionId&gt;;</pre><pre>New-AzureRmDataLakeAnalyticsCatalogCredential -AccountName &quot;&lt;<strong>NAME_DATALAKE_ANALYTICS</strong>&gt;&quot; -DatabaseName &quot;&lt;<strong>DATABASE_INSIDE_ANALYTICS_ACCOUNT</strong>&gt;&quot; -CredentialName &quot;SQLDBCredential&quot; -Credential (Get-Credential) -DatabaseHost &quot;&lt;<strong>NAME_OF_SQL_DB</strong>&gt;.database.windows.net&quot; -Port 1433;</pre><p>Launch a new job under data lake analytics account. The code snippet below creates a new connection token which helps USQL connect to the SQL DB</p><pre>USE DATABASE &lt;<strong>DATABASE_INSIDE_ANALYTICS_ACCOUNT</strong>&gt;;</pre><pre>CREATE DATA SOURCE IF NOT EXISTS SQLDBDataSource<br>FROM AZURESQLDB<br>WITH<br>(<br>    PROVIDER_STRING = &quot;Database=<strong>&lt;DATABASE_INSIDE_SQL_DB&gt;</strong>;Trusted_Connection=False;Encrypt=True&quot;,<br>    CREDENTIAL = SQLDBCredential,<br>    REMOTABLE_TYPES = (bool, byte, sbyte, short, ushort, int, uint, long, ulong, decimal, float, double, string, DateTime)<br>);</pre><p>Final part of this blog is how you can combine the data from SQL DB and the recommendation data which we have computed.</p><p>Launch a new Job under Azure Data Lake Analytics and paste the code below</p><pre>USE DATABASE &lt;<strong>DATABASE_INSIDE_ANALYTICS_ACCOUNT</strong>&gt;;</pre><pre><a href="http://twitter.com/customerDim">@customerDim</a> =<br>    SELECT *<br>    FROM EXTERNAL SQLDBDataSource LOCATION &quot;SalesLT.Customer&quot;;</pre><pre><a href="http://twitter.com/customerRecommendati">@customerRecommendati</a>on = <br>    EXTRACT firstName             string,<br>            lastName              string,<br>            id                    int?,<br>            recommendation_item   string<br>    FROM &quot;/data/customer_filtered.csv&quot;<br>    USING Extractors.Csv(silent:true);</pre><pre><a href="http://twitter.com/result">@result</a> =<br>SELECT     a.firstName             AS firstName, <br>           a.lastName              AS lastName, <br>           b.EmailAddress          AS Email,<br>           b.Phone                 AS Phone,<br>           b.CompanyName           AS Company,<br>           a.recommendation_item   AS recommendation<br>    FROM <a href="http://twitter.com/customerRecommendati">@customerRecommendati</a>on  AS a<br>         JOIN<br>             <a href="http://twitter.com/customerDim">@customerDim</a> AS b<br>         ON a.id == b.CustomerID;</pre><pre>CREATE VIEW CustomerRecommendationSQLDB  <br>    AS  <br>SELECT * <br>  FROM <a href="http://twitter.com/result">@result</a>;</pre><pre>OUTPUT <a href="http://twitter.com/result">@result</a><br>TO &quot;/data/output.csv&quot;<br>USING Outputters.Csv(outputHeader:true,quoting:false);</pre><p>Check the View after this has been executed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4FINJIG65tnBkWQq3ZonCA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=214aa14aef76" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>