<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by shashi on Medium]]></title>
        <description><![CDATA[Stories by shashi on Medium]]></description>
        <link>https://medium.com/@shashiwalker?source=rss-dd3967a5b7fa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VFnuXFxozfSOFRa2aCUciA.jpeg</url>
            <title>Stories by shashi on Medium</title>
            <link>https://medium.com/@shashiwalker?source=rss-dd3967a5b7fa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:07:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shashiwalker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Azure CLI: Installation to Interaction]]></title>
            <link>https://shashiwalker.medium.com/azure-cli-installation-to-interaction-faba0ee7108d?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/faba0ee7108d</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Sun, 21 Aug 2022 18:43:21 GMT</pubDate>
            <atom:updated>2022-08-21T18:43:21.845Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*LzwfRqx5B3nZdoOGjZRjMg.png" /></figure><h3>Interacting with Azure Cloud Platform</h3><p>Azure offers three different ways to interact with the resource on the cloud:</p><blockquote><strong><em>Azure Web Portal (UI)</em></strong></blockquote><blockquote><strong><em>Azure SDKs or APIs</em></strong></blockquote><blockquote><strong><em>Azure CLIs</em></strong></blockquote><p>To access resources programmatically, generally, one of the two approaches is used:</p><blockquote>Azure SDKs</blockquote><blockquote>Azure CLI</blockquote><h3><strong>Installation of Azure CLI</strong></h3><p>The Azure CLI is available to install in Windows, macOS, and Linux environments. It can also be run in a Docker container and Azure Cloud Shell.</p><p><strong>Installation on Windows</strong></p><p>Azure CLI can be installed on Windows using an MSI(Microsoft Installer), which then gives access to the CLI via Command Prompt or PowerShell.</p><p>The MSI distributable is used to install and update the azure CLI, if you have an existing version, then the installer will automatically update your CLI to the latest version. You can easily download and install the Azure CLI for windows using this <a href="https://aka.ms/installazurecliwindows">link</a>.</p><p><strong>Installation on macOS</strong></p><p>Azure CLI is also available for installation on macOS. It offers the same commands to run through a terminal or via scripts. The installation of Azure CLI on macOS is available via the <a href="https://brew.sh/">homebrew package manager</a>. If you don’t have homebrew on your macOS, you can install the package via the following command</p><pre>/bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)&quot;</pre><p>Once the homebrew package is installed on your macOS, you can proceed to install Azure CLI on your macOS. You can install Azure CLI on macOS via the following command: brew install azure-cli</p><h3>Interacting with Azure Resources using CLIs</h3><p>Azure SDKs and CLIs offer entirely the same functionalities to access their cloud resources, but you get more granular control using Azure CLI. Azure CLI is a multi-platform tool that helps the developer or the end user to interact with Azure Resources. One can install Azure CLI on Mac, Windows, Linux, and any other OS out there. It can also be used from a browser through the Azure Cloud Shell or run from inside a Docker container. It allows the execution of commands through a terminal using interactive command-line prompts or a script.</p><p>Once you have installed the Azure CLI on your system, you can even automate a particular task using Azure CLI through a script. We can use multiple azure CLI to form a chain of command which then forms a script that in turn interacts with the cloud resources.</p><h4>Authentication First</h4><p>Before running any azure CLI command, you need to authenticate your system with azure. You can do so by, signing in with: az login . This will open your default browser and will prompt you to log in with your account to the Azure portal. After successful authentication, you may close the browser page or it will be redirected to the docs page after 10 secs.</p><p>If your browser is not available you can alternatively use the<strong><em> sign-in via device code</em></strong> option using the command : az login --use-device-code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*ZUf_Kbcn3Vq6O-03nE2JUQ.png" /><figcaption>az login</figcaption></figure><h4>Interaction Next</h4><p>Now you are all set to play around with Azure CLI and interact with your resource in the cloud. To interact with Azure resources, CLI offers various commands specific to those resources. Each command in Azure CLI starts with a az keyword, followed by the name of the resources. The next keyword in the CLI is the action that you want the CLI to perform. One basic command that you can run is to retrieve the list of accounts in azure subscription using the following command : az account list . This command will give you the following result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*RjKgILyXp8Zh4ORHCxHz9g.png" /></figure><p>You can find the entire reference to the Azure CLI <a href="https://docs.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest">here</a>. There are a ton of resources on Azure that is useful for various different use cases. All these resources have their own CLI actions. But all these commands follow a general pattern. The Azure CLI syntax is a combination of groups, references, commands, and parameters. Often the full reference command is referred to as command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*Pj8qWlQp-uVucgwchWLfnw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=faba0ee7108d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Cloud Platform: Getting Started]]></title>
            <link>https://shashiwalker.medium.com/azure-cloud-platform-getting-started-9c2634d545b1?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c2634d545b1</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[sdk]]></category>
            <category><![CDATA[cli]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Sat, 06 Aug 2022 17:06:27 GMT</pubDate>
            <atom:updated>2022-08-06T17:11:28.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*zrmE0rQZgbhU2oLgU108LQ.png" /></figure><p>Cloud Computing as we all know is an ocean of various services for various different use cases. There are multiple cloud service providers such as <strong><em>Amazon Web Services, Google Cloud Platform, Oracle Cloud Infrastructure, IBM Cloud, and Microsoft Azure</em></strong><em>.</em></p><p><strong><em>Microsoft</em></strong><em> </em>offers various resources for different use cases through its cloud platform which is commonly known as <strong><em>Azure Cloud Platform</em></strong>. It offers services ranging from storage, computing, networking, and services related to AI/ML.</p><p>To use these resources you need a subscription which can generally be understood as an account on the Azure Portal. Remember one azure account can have multiple subscriptions. Once you get an Azure subscription you can go ahead and create various azure resources directly from the portal.</p><p>Each Azure resource sits in a logical container which is known as a resource group. A resource group is just a collection of resources basis certain use cases or types. Though it can contain resources of different use cases and types. A single Azure subscription can contain multiple resource groups and a resource group can contain multiple resources.</p><p>The following diagram explains the architecture of resources in the Azure Cloud Platform</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*H0f2koBwzSeBX_c02MHbHA.png" /><figcaption>Azure Resource Hierarchy</figcaption></figure><h3>Interacting with Azure Cloud Platform</h3><p>Azure offers three different ways to interact with the resource on the cloud:</p><blockquote><strong>Azure Web Portal (UI)</strong></blockquote><blockquote><strong>Azure SDKs or APIs</strong></blockquote><blockquote><strong>Azure CLIs</strong></blockquote><h4>Accessing Azure using Web Portal</h4><p>Accessing Azure resources using the Web portal is one of the easiest ways to access all the resources you have. The simple UI has everything you need in one place. The following is the landing page of the Azure Portal. You can access the portal<strong><em> </em></strong><a href="https://portal.azure.com/#home"><strong><em>here</em></strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4nrjGSE97yBFKE0YMf6pzQ.png" /><figcaption>Azure Web Portal</figcaption></figure><p>The azure web portal offers everything as per your need, you can easily create and delete resources, upscale and downscale any resource capacity based on your need, add and configure any firewall rules, and various different things.</p><h4>Accessing Azure Resources using Azure SDKs</h4><p>Azure offers programmatic access to all its resources via the use of software development kits. These SDKs comes in various language and can be used as per the developer&#39;s choice. The full list of SDKs and tools can be found<strong><em> </em></strong><a href="https://azure.microsoft.com/en-in/downloads/"><strong><em>here</em></strong></a>.</p><p>All the SDKs offer all the functionalities to manage and configure resources on azure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*70WT35TMsC-XGzX2X3uo8Q.png" /><figcaption>Azure SDKs</figcaption></figure><p>Some of the most common SDKs include libraries compiled in the following languages:</p><ul><li>Python</li><li>JavaScript/TypeScript</li><li>Java</li><li>.Net/C#</li></ul><h4>Accessing Azure Resources using CLIs</h4><p>Another programmatic way to access azure resources is through the use of CLIs. Azure offers a wide range of CLIs for interacting with its resources. Using the CLIs we can configure all the parameters for resources in a more easy and granular way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*5Ly_US2Irjlebb9KeHtJSQ.png" /><figcaption>Azure CLI</figcaption></figure><blockquote><em>Each and every azure CLI command starts from</em> <strong>az </strong><em>keyword followed by the name of the resources.</em></blockquote><p>Here’s an example command to interact with Azure Resources using Azure CLI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*HYPEmREcURPfcIQLYt28Sw.png" /></figure><p>Azure CLIs hold much more power than offered by SDKs in general. Though we can achieve the same using SDKs, Azure CLIs offer much more granular access for interaction with its resources. To learn more about the azure CLI click <a href="https://docs.microsoft.com/en-us/cli/azure/"><strong><em>here</em></strong></a></p><p>Overall <strong>Azure Cloud Platform</strong> provides a wide range of tools and services for various use cases, along with the tools to create and interact with them.</p><p>In my next blog, I will explain in brief the working and interaction of Azure CLIs with the Azure Cloud Platform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c2634d545b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysis and Visualization using KQL and Azure Dashboards]]></title>
            <link>https://shashiwalker.medium.com/analysis-and-visualization-using-kql-and-azure-dashboards-678ec9f28de5?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/678ec9f28de5</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[kql]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 10:52:59 GMT</pubDate>
            <atom:updated>2022-03-18T10:52:59.902Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*6VV4fqq5PHVIntcF6g7dDw.png" /></figure><p>Everyone has heard of SQL at some point in their career. Though SQL is simple and efficient, KQL is designed especially for Big Data Analysis.</p><h3>Analyzing Data using KQL</h3><p><strong>KQL </strong>stands for <strong>Kusto Query Language</strong> and is used to analyze tabular data available through Azure Monitor and Azure Application Insights. KQL is fairly simple and resembles SQL in many ways.</p><p>A Kusto query is a read-only request to process data and return results. Kusto queries are made of one or more query statements. The most common statements include the <strong>Tabular </strong>statements, which comprise of one or many <strong>operators</strong>. These operators are separated by pipe (<strong>|</strong>) operators.</p><p>Every KQL query results in a Tabular output which can then be converted into visuals depending upon your choices.</p><blockquote>Some of the visuals present at the explorer are: <strong>Pie Chart, Donut Chart, Bar Graphs, Area and Line Charts.</strong></blockquote><p>These visuals can also be obtained using the KQL query using the “<strong>render</strong>” operator followed by the type of graph needed, viz. bar, line, area.</p><h3>Visualize Data using Azure Dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W3Ndtq-Ff9v9wwWq" /><figcaption>Photo by <a href="https://unsplash.com/@firmbee?utm_source=medium&amp;utm_medium=referral">Firmbee.com</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The main purpose of Data Visualization is to convey the hidden information in the underlying data.</p><p><em>The more you hammer the data the more it will yield the result, showing the never seen before results.</em></p><p>The interactive dashboards allow the end-users to explore the data and understand certain areas that might help drive the business and also identify those areas that need special attention. There are a lot of different visualizer tools that meet the demand of the business needs. One such tool is the <strong>Azure Dashboards.</strong></p><p>The tabular data that forms the result of the analysis using KQL can be converted into visuals for storytelling. Azure Dashboards provide lots of different customization and allow the addition of different types of visuals. Some of these visuals include:</p><ul><li>Markdown Cells</li><li>Alerts Metrics</li><li>Resource Counters</li><li>KQL Visuals</li></ul><p>Once the tabular data is generated, it can easily be converted into visuals, and then it can be pinned to the Azure Dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZoGB4fdyTYQ9qc67" /><figcaption>Photo by <a href="https://unsplash.com/@hostreviews?utm_source=medium&amp;utm_medium=referral">Stephen Phillips - Hostreviews.co.uk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every account in azure has a default private dashboard, which has some default visuals in it. Once the visuals from the KQL explorer are pinned to the dashboard, the dashboard can be shared with any specific resource group and the members of the resource group will then have the access to it.</p><p>Azure Dashboards offer some customizations to their visuals, including fonts sizing, labeling, and adding links to different dashboards. Currently, navigation through dashboards is not possible in Azure Dashboards unlike PowerBI and Tableau, but we can achieve this by adding a markdown cell.</p><p>The markdown cell is a simple visual to represent any form of HTML text. Add the link to the destination dashboards and you are good to go. The markdown link opens the dashboard on a new page by default.</p><p>Filters in Azure Dashboards as similar to any other BI tools filter, with visual level customization. The filter at the tile level can overwrite the filter at the level of the dashboard. Some of the filters include time filters, text filters, category filters.</p><p>Azure Dashboards are a really powerful tool for any business to track its productivity. KQL offers far more Data handling capabilities as compared to SQL and is often preferred over SQL for Big Data Analysis on Azure.</p><p>Overall Data Analysis with KQL is simple and easy and Data Visualization with Azure Dashboards offers lots of flexibility.</p><p>For more info follow these links:</p><ul><li><a href="https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/">KQL reference</a></li><li><a href="https://docs.microsoft.com/en-us/azure/azure-portal/azure-portal-dashboards">Azure Dashboards</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=678ec9f28de5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Interview Experience with Tiger Analytics]]></title>
            <link>https://shashiwalker.medium.com/my-interview-experience-with-tiger-analytics-e634d98ada4d?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/e634d98ada4d</guid>
            <category><![CDATA[experience]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[interview-experience]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Sun, 13 Mar 2022 09:18:55 GMT</pubDate>
            <atom:updated>2022-03-13T09:18:55.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*GqVKHNSTNigzZ9r0SatSwQ.png" /></figure><pre>Privet, drug|Hola amigo|Hallo, Freund</pre><p>In this blog, I’ll be sharing my interview experience with some of the top tech companies, in which I was hired.</p><p>So, In the fall of January and the start of February, I got two offers. One as a <strong>Senior Analyst — Data Science</strong> from <strong>Tiger Analytics</strong> and one as a <strong>Software Engineer</strong> from <strong>Optum</strong>, <strong>UnitedHealth Group</strong></p><h3>Let us start with Tiger Analytics.</h3><p>I applied for tiger analytics somewhere around December last year, and after around 1 week time I got a call from the HR team. They explained to me the hiring process at Tiger Analytics. There are a total of four stages at Tiger to get selected.</p><pre><strong>- Telephonic Interview<br>- Coding Round<br>- 1st Round of Interview<br>- 2nd Round of Interview</strong></pre><h4>Telephonic Interview</h4><p>Once you get the first call from HR after your resume got selected, they will ask some general questions like <strong>current organization, CTC, expectation, and YOE</strong>. Stay calm and answer the question with a normal tone. Once they are satisfied with your behavior, then you will be invited for a technical round which is typically held on <strong>HackerEarth.</strong></p><h4>Coding Round</h4><p>The coding round that I appeared had</p><pre>- 10 MCQs related to Data Science, and <br>- 3 Coding Questions</pre><p>The MCQ is generally focused on data science, and you can expect questions on Probability, Confusion Matrix, Statistics. Some questions from ML algorithms. The level of questions is from beginner to intermediate only. Some questions may be hard tough.</p><p>Then comes the coding part which had three questions in total. You can code in any language of your choice(i.e Python 😝). The level of the questions is also from beginner to intermediate. Once you clear the coding round you can expect a call from the HR team for the next rounds.</p><h4>1st Round of Interview</h4><p>If you are here then take a deep breath, you are through to half the line. Cheers(🍻). Relax and don’t panic before the interview. Stay calm and focused.</p><p>In this round of interview, you may be asked to turn on your camera. This round generally focuses on your Python/R skills. You will be required to write codes by sharing your screen. You will be given 3 – 4 coding questions that you have to solve. You don&#39;t have to worry about the time complexity at first. First, understand the question clearly and then explain to the interviewer your approach on how you are going to solve it. Then gets your hands dirty.</p><p>Once you are done with the code and 2 – 3 test cases have passed, the interviewer may ask you to optimize it, and may also give you hints. It&#39;s completely<strong> OK </strong>to ask for any hints. You can also pass on the questions stating that you don’t have an answer to it, or you are not able to derive an answer for it.</p><p>Once the live coding is done, they will ask some questions related to your projects and what you worked on previously. So it&#39;s always a plus point if you have some projects by your side. Since the company is focused more on Data Science, it is better to have some projects related to ML/AI in your hands. Then they will ask some questions on probability, and statistics which are generally easy. Some questions will be from intermediate python.</p><p>Example questions that I remember:</p><pre>- Probability of even/odd when 1/2/3 dice is/are rolled.<br>- Bayes theorem probability question.<br>- Intermediate Python concepts</pre><p>Answer/write all the answers carefully staying calm and confident throughout the interview. Don’t just stay quiet in the interview, explain your answers and how you arrived at those results. It&#39;s all about impressing the interviewer with your skills and knowledge.</p><p>Now it&#39;s up to the interviewer to process it. Sit back and relax...😴</p><h4>2nd Round of Interview</h4><p>You are almost there….!!!</p><p>This round of interview is more focused on the profile you have applied to or the profile they have considered. You can expect questions related to ML algorithms, Pandas, and DF functions. They may also ask you to share some light on your past experience and projects. How/What you have implemented in your projects.</p><p>The second half of the interview is related to behavior and managerial kinds of stuff.</p><p>You can expect questions like</p><pre>- Why us...?<br>- What have you accomplished using your previous projects?<br>- Why are you a good fit for our organization?<br>- Past experiences and learning.</pre><p>Again stay calm, focused, and confident. You got this…!!!</p><p>Lying on the bed, scrolling Instagram, applying for different jobs, reading, coding, watching movies…cell phone rings….</p><blockquote>Hello, this is….from Tiger Analytics, We would like to offer you the position of ……</blockquote><p>Congratulations, you did it🎊🎉.</p><p>Finally, after 3–4 rounds of the hiring process, you will be relieved when you receive the offer. Don’t give up if you haven&#39;t received the offer at any stage. There are lots of other opportunities out there…</p><p>Keep applying and skilling yourself. Stay focused.</p><p>For any guidance on your resume and interview process reach out to me on <a href="https://www.linkedin.com/in/shashi-kumar-soni/">LinkedIn</a>.</p><p>In the next blog, I’ll share my interview experience with UnitedHealth Group. Stay connected…!!!</p><pre>Poka druz&#39;ya|Adios amigos|Tschüss, Freunde</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e634d98ada4d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Data Integration using Azure Data Factory]]></title>
            <link>https://shashiwalker.medium.com/serverless-data-integration-using-azure-data-factory-fb34ffaf5dff?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb34ffaf5dff</guid>
            <category><![CDATA[etl-tool]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[data-integration]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Thu, 10 Mar 2022 10:04:51 GMT</pubDate>
            <atom:updated>2022-03-10T10:04:51.786Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*ch2MnniCMfySb-QKhAEwcg.jpeg" /></figure><p>Data is indeed the building block of any business work. Data is needed to drive the business in a proposed direction. Data comes from multiple different directions as well as in multiple different forms. The need of the business is always to process the data in order to get the insights that help in achieving different heights with the product/projects. But getting the data is the actual tedious part not to mention the wrangling which actually takes up 70% of the time.</p><h3>Data Acquisition</h3><p>Data integration generally involves acquiring structured or unstructured data in one place followed by any data transformation or data wrangling. There are lots of different tools and packages which allow integrating data from various different sources. One such tool is the Azure Data Factory.</p><h4>Azure Data Factory: Serverless Data Integrator</h4><p>Azure data factory or ADF is the new age data integration tool and an ETL service offered by Microsoft Azure. ADF offers lots of different sources and sinks for extracting data and loading it to the destination.</p><p>Some of the popular connectors for ADF are listed below:</p><ul><li>Amazon S3,</li><li>Azure SQL,</li><li>Mongo DB,</li><li>Amazon RDS,</li><li>SQL Server,</li><li>Azure Blob Storage,</li><li>Azure Cosmos DB…and many more.</li></ul><p>The complete details of the connectors can be found <a href="https://docs.microsoft.com/en-us/azure/data-factory/connector-overview">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yrUJlK8ApBGzrlqMHzvlxQ.jpeg" /></figure><p>ADF the data is integrated to the sink from the source via pipeline. These pipelines perform the extraction of the data from the source and transform them if necessary. This is then followed by loading or inserting the data into the sink target. This entire process can be termed an ETL process.</p><p>ADF provides different methods to build and deploy the pipeline.</p><blockquote>- SDK</blockquote><blockquote>- REST APIs</blockquote><blockquote>- az CLI</blockquote><p>Here are some of the commands that are commonly used:</p><h4><strong>Create a data factory</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*_OCZUCbD5S_43hU63Ic_Fg.png" /><figcaption>ADF create command</figcaption></figure><h4><strong>Create a pipeline with a for each activity</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*ANWzp-zcMxXqCGa1daeEsw.png" /><figcaption>for each activity</figcaption></figure><p>The creation of activity in ADF requires a JSON object which can be passed as a file. The enclosed <strong>forEach.json</strong> looks as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*H9IJHhDOdTLF2D-WZlZYpA.jpeg" /></figure><h4>Deleting a Data factory ( -y for direct consent : optional)</h4><figure><img alt="ADF delete command" src="https://cdn-images-1.medium.com/max/549/1*FNLngUmIFEkhwDfBgejNHw.png" /><figcaption>ADF delete command</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb34ffaf5dff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accessing Data from SAS Code]]></title>
            <link>https://shashiwalker.medium.com/accessing-data-from-sas-code-829499633c1d?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/829499633c1d</guid>
            <category><![CDATA[sass]]></category>
            <category><![CDATA[csv-import]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Tue, 24 Nov 2020 09:37:07 GMT</pubDate>
            <atom:updated>2020-11-24T09:40:59.449Z</atom:updated>
            <content:encoded><![CDATA[<h4>Accessing structured and unstructured data in SAS</h4><p>In the previous blog, I have discussed some of the basics of the SAS. As discussed the SAS table has an extension of <strong><em>.sas7bdat.</em></strong></p><p>In this blog, I am going to discuss how we can access SAS data and tables from the SAS code.</p><h3>Accessing Structured Data</h3><p>SAS tables can be accessed using a proc contents step, and by specifying the path of the table to the data parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*CppBIPN_PuxDXalRkBJfvw.jpeg" /><figcaption>Accessing Table from SAS code</figcaption></figure><p>This step creates a summary of the SAS table, with the table name, number of observations, variables name, sort information, file access permissions, and a lot more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*aVzyDnUR5eOhceNapRCrsw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*Nmz_YjiKYfVj8wlYbhV3FA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*lLqr4B5sHuOrc4XdMqgHTg.jpeg" /><figcaption>Summary and Properties</figcaption></figure><p>Using the hardcoded full path for the table might cause some issues, for eg. When the path changes sometimes then we have to change the entire codebase to accommodate the new file path. This is taken care of by including libraries.</p><h4>SAS Libraries</h4><p>Accessing a table requires tow information:</p><ul><li>Path</li><li>Type</li></ul><p>But this is tedious sometimes. Libraries, on the other hand, gives us a way to specify this information, in a very simple and efficient way. A library is a collection of similar data files in the same location and of the same type.</p><p>First, we specify what library we want to create using libname which is a global variable. After the libname, we specify librefwhich is the name of the library we want to create with and then we specify the engine parameter, which tells what type of data is to be imported. By default, the engine is set to base which is the SAS table. Other parameters are xlsx , hadoop , teradata .After that we specify the path of the file. The library includes only those files for which the engine is specified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*6mBvFGD2zjre5TpRA2OjOQ.jpeg" /></figure><p>These libraries get destroyed when you close the SAS session, but the data are not destroyed. There are several libraries that are created automatically. One such library is the work library, which is also the default name when creating a library using libname step. This library is temporary, and hence the data is deleted after each SAS session. Some libraries are also created by the SAS administrator, hence we can use it directly in the code.</p><p>Once the library is created we can access the SAS table just by referencing to the table we want to access as: “ <em>libname.tablename</em>”. To clear the library after use, we use libname libref clear.</p><p>To create an Excel library, in SAS we use the following code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*Jy6WscGQWJSC5pxUREYeww.jpeg" /><figcaption>Excel Library Creation</figcaption></figure><p>We use the <strong><em>validvarname,</em></strong> to specify that the spaces in the columns name be replaced by the underscore.</p><h4>Accessing Unstructured Data</h4><p>To use and play around with the unstructured data in the SAS, we first have to import it into the SAS table. proc import statement is used to import the unstructured data into the SAS table. It has various different options, including DATAFILE which is the file path, DBMS which is the filetype, and the OUT which specifies the output table in SAS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*p9aKrzns4tPcf8GHhLUKsQ.jpeg" /><figcaption>Accessing Tab Separated file</figcaption></figure><p>Notice the keyword replace after the OUT option. This is done to replace any existing SAS table or to run the same code twice without any conflicts.</p><p>To avoid data truncation, use the guessingrows option, and set it to MAX. This is by default set to 20.</p><h4>Importing Excel File</h4><p>Sometimes we want to import an excel file into a SAS table, this can be done using the proc import statement. As follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*5RW74HlWvhQZWIiYuHGS0A.jpeg" /><figcaption>Importing an Excel file</figcaption></figure><p>While importing the Excel workbook, the DBMS option must be specified as xlsx. By default the import reads only the first sheet of the excel file, this can be dealt with by including the sheet parameter, with the sheet name specified.</p><p>SAS assumes that the headers are in the first line and the data portion starts after that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=829499633c1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with SAS]]></title>
            <link>https://shashiwalker.medium.com/getting-started-with-sas-ec91e8d95?source=rss-dd3967a5b7fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec91e8d95</guid>
            <category><![CDATA[data-science-training]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[sass]]></category>
            <category><![CDATA[processing]]></category>
            <dc:creator><![CDATA[shashi]]></dc:creator>
            <pubDate>Sun, 19 Jul 2020 16:34:26 GMT</pubDate>
            <atom:updated>2022-03-10T07:37:37.946Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHRG2QeMyWrT57W16LPRNw.jpeg" /></figure><p>Hello Everyone,</p><p>Welcome to my blog on Getting Started with SAS Programming. In this and upcoming blog I will try to explain everything related to SAS from basic to advanced. So stay tuned.</p><p>Let’s get started.</p><h3>SAS Overview</h3><p>SAS Program consists of two steps</p><ol><li><strong>DATA</strong> Step and</li><li><strong>PROC</strong> Step or Procedure Steps</li></ol><p>A SAS program can contain any number of DATA and PROC steps depending on the tasks performed.</p><p>The data steps end with the run statement and proc statements end with quit and others with run statements. If the run statement is not used, then the beginning of the new step continues with the data or proc statement.</p><p>The steps are counted as the number of data or proc segments.</p><h3>Steps in SAS</h3><p>The DATA step is the input source and it reads the data, processes it, generates the SAS table, and also performs data manipulations.</p><p>A PROC or procedure step processes a SAS table in a specific and predefined way. SAS has dozens of procedures that generate reports and graphs, managed data, or perform complex statistical analyses.</p><p>All statement ends with semicolons;</p><figure><img alt="Steps Overview of SAS" src="https://cdn-images-1.medium.com/max/785/1*Lo87ZaQAY8L-HFHeRzhF3g.png" /><figcaption>Steps in SAS</figcaption></figure><h3>Global Statements</h3><p>The Global statements are kept outside the data and proc statements, they generally define the options for that SAS sessions. They don&#39;t need ‘run’ statements after them.</p><h3>Writing code in SAS</h3><p>The Spacing doesn&#39;t matter to SAS, as long as there is a semicolon between statements. We can write any keyword in any case as SAS is not case-sensitive.</p><h4>Comments</h4><p>Single line comments are written using “*” and end with a semicolon. Multiline comments are written as “/* comments*/”.</p><h3>Accessing Data</h3><pre>1. Structured Data: Defined Rows and Columns, can be SAS Table, SQL table, MS Excel.<br>2. Unsturctured Data: JSON, weblogs, csv files</pre><h3>SAS Table</h3><p>SAS Tables have defined rows and columns and have an extension of .sas7bdat There is a two-part to the SAS table:</p><ol><li>The descriptor portion often known as the header, contains information about the table metadata like table name, number of rows, date of creation, column name and attributes,</li><li>The Data portion which is the rest of the row, stores data in columns.</li></ol><h4>SAS Table Terminology:</h4><pre>1. Table is also called a dataset.<br>2. Columns is also called a variable,<br>3. Row is also called an obseravtion.</pre><p>These terms are interchangeable.</p><h4>Required Column Attributes</h4><p>A SAS column must have three attribute</p><p>1. <strong><em>Name</em></strong>:</p><blockquote>- SAS Column names can be 1–32 characters long.</blockquote><blockquote>- Must start with a letter or underscore.</blockquote><blockquote>- Can contain any number of combination of letters, numbers and, underscores.</blockquote><p>The Column name is stored in the same way you specify when you create the column. After the column is created we can reference them using the name in any case &amp; even mixed cases are allowed.</p><p>2. <strong><em>Type</em></strong>:</p><p>- There are two types of columns in SAS: <strong><em>Character</em></strong> &amp; <strong><em>Numeric</em></strong></p><ul><li><strong>Numeric</strong> Columns can store only numeric values, which can include:</li></ul><blockquote>- A number from 0–9,</blockquote><blockquote>- A minus sign,</blockquote><blockquote>- A single decimal point and</blockquote><blockquote>- E for scientific notation.</blockquote><p>Missing Numeric Characters are stored as a period</p><p>SAS Dates are a particular kind of numeric value. SAS stores date values as the number of days between January 1, 1960, and a specific date. Dates before January 1, 1960, are stored as negative values. This way of storing date values makes calculations and sorting possible. There are ways to display these values so they look like dates we can understand.</p><ul><li><strong>Character</strong> Columns can store letters, numbers, special characters, and blanks. Missing character columns are stored as space.</li></ul><p>3. <strong><em>Length</em></strong></p><blockquote>- The column length is the length in bytes allocated to store the column values.</blockquote><blockquote>- The length is related to the column type.</blockquote><blockquote>- Numeric columns, by default, are always stored as 8 bytes, which is enough for about 16 significant digits.</blockquote><blockquote>- Character columns can be any length between 1 and 32,767 bytes, and one byte stores one character.</blockquote><p>A Column length must have a length at least as long as the longest entry in the column.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec91e8d95" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>