Question :- What is Kubernetes ?

Question :- How do we check the Kubectl version ?

Question :- What is the usual way of deploying to Kubernetes ?

Question :- Show the process of creating a Kubernetes Deployment from scratch ?

Step #1.) We shall be using this specific repository which is already present in the DockerHub :-

Step #2.) For this specific repository, we are going to use this image :-

Step #3.) Using the aforementioned docker-image, we shall be creating a Kubernetes Deployment :-

Step #4.) Now, we can also expose this deployment as a service, so that we can access the endpoint of this microservice :-

Now, if we go to the dashboard, we can see that the service has been exposed by our afore-mentioned command :-

Using the afore-mentioned Endpoint, we can access our micro-service :-

Step #5.) Now, under the hoods, the aforementioned command (as mentioned in step #3) shall be creating :- Deployments → ReplicaSet → Pods.

Concept of Pod

Question :- Here is how the relationship between Nodes, Pods and Containers looks like :-

Part #1.) We can also see the detailed information about all of the existing pods :-

• Note that, every Pod have got a unique IP-Address with it.
• Number of specific containers present inside this specific Pod are : 1/1 i.e. Only ONE container is present inside of this Pod.

On a concept side, One Pod can have multiple containers inside it, but here there is only a single container inside this Pod. Within a sindle Pod, the containers can talk to each other via the localhost.

Part #2.) Here is what a Pod means, in words of Kubernetes itself as well :-

Part #3.) We can also see the detailed information about a single pod, by copying the ID of the Pod from the Part #1 above. We can see a lot of details about the Pod through this command :-

• It tells us that, this Pod is running in the default namespace. Namespace is a very important concept, as it provides the isolation to one part of the cluster from another part of the cluster.
• It also shows us the Node, inside which, this Pod is running.

Concept of ReplicaSet

Question :- Demonstrate the benefit of ReplicaSet here ?

Step #1.) Here, we can see that, we have ReplicaSet as shown below.

• There are 3 ways to get the information about the ReplicaSet as shown in the screenshot below.
• Purpose of ReplicaSet → It is to make sure that, if some pods go un-healthy OR are not well, then these pods can be brought back asap. The ReplicaSet ensures that, specific number of pods are running always.

Step #2.) Here, let’s now intentionally kill one of the existing pods. What we shall observe is that → The very next moment, we can see that, ReplicaSet spawns a new Pod with a different name as shown in below screnshot :-

Thus, in this way, the application would continue to be available, up & running and available. ReplicaSet always keeps on monitoring the count of Pods. As soon as it finds that, count of current pods is less than the count of desired pods, it immediately kicks the new pod creation process.

Here, is the meaning of ReplicaSet in the words of the Kubernetes documentation itself :-

Question :- Now, if load rises on the application, then how can we ensure that our deployment scales effectively ?

Answer → We can effectively scale the Kubernetes-Deployment as shown below, which shall add the new pods to the K8S deployment :-

==> This work of adding the new pods, is being done by the ReplicaSet. From the above screenshot, we can see that, the 2 pods are launched just very latest.

==> These pods are also doing the load-distribution now, amongst themselves.

The same can be verified by opening the link to the App now. Here for the first time, we can see that, the output from the App looks like this :-

Here for the second time, we can see that, the output from the App looks like this (Note that, we are intentionally printing the #podId along with the Hello World in output) :-

Also, we have done the up-scaling of Deployment above but we can also verify from our ReplicaSet as the same has also being scaled-up :-

In the background, there are a lot of events which would had happenned. We can see them like this :-

Concept of Deployment

Question :- Demonstrate the benefit of Deployment here ?

Step #1.) Recall that, we created a Deployment earlier, in the beginning of this blog :-

Now, let’s try to set the image version to some version-number, which doesn’t exists, basically we want to see the behaviour :-

Step #2.) Although it shows that, the image has been successfully updated, but the application doesn’t goes down.

• We can see that, the application is running perfectly fine.
• Even though, we made a mistake while updating the deployment, but still the application has not yet gone down.

Step #3.) Let’s see what happens in the background for this situation :-

• Here, we have our original ReplicaSet which is perfectly fine and it has 3 pods desired as well as available.
• We also observe that, there is another ReplicaSet which have got desired-pod-count as ONE, but Ready state Pods are ZERO.

We can also verify that, the status of ONE pod shows as “InvalidImageName”, just because this pod can’t be started because, an invalid image has been specified :-

Step #4.) Let’s now proceed ahead to set the right image name now. First, let’s verify the right image that we have :-

Now, we go ahead and set this specific version : 0.0.2.RELEASE :-

As soon as we set the right version and execute this command, we would observe that, the pods are changing now :-

Also, after few moments, we shall observe that, the replicaSet is also changed now to a new replicaSet.

We can also observe that, the updated version of the application is now up & available with us :-

Step #5.) Let’s now verify the sequence of events that happenned with this change of replicaSet :-

Strategy adopted by the Deployment → Rolling Updates :- A deployment is very very important to make sure that you are able to update new releases of applications without downtime. :-

• What it does is, “Let’s say I have five instances of V1 and I want to update to V2.”
• The rolling update strategy, it updates one pod at a time. So it launches up a new pod for V2. Once it’s up and running, it reduces the number of pods for V1.
• Next, it increases the number of pods for V2 and so on and so forth until the number of pods for V1 becomes zero. And the all the traffic goes then to V2.

Concept of Service

Question :- What is the need of a Service ?

Step #1.) See that, when we try to access our application, it serves via one of the available pods :-

Next time, we hit this endpoint, it serves via some another pod maybe :-

Step #2.) We can also observe that, each Pod have its own IP-Address :-

Step #3.) Now, if we delete one pod, the replicaset would bring up another pod immediately as shown in the below screenshot : The age of one of the Pod is merely 12 seconds whereas for other pods, it is 32 minutes.

Step #4.) Similarly, if we delete another pod, we can observe that a yet another pod comes-up but with a different IP-Address.

Thus, if we want to avoid the IP-address of our service to change frequently, then we can have it fixed with the help of a Service. That’s the main objective of having a service :-

Learnings So Far :-

• When we created a Kubernetes Deployment, K8S automatically creates for us → Deployment, ReplicaSet, Pod.
• When we expose a Kubernetes Deployment, K8S automatically creates for us → Service.

==> Each of these concepts have one important responsibility to offer.

1.) A Container always lives inside the Pod. The container can’t exist independently on its own.

2.) A Deployment is very important, in order to ensure the release of new application without any downtime.

That’s all in this blog. We shall meet you in next blog for this series.

Question: Talk about the Azure-Synapse-Analytics ?

Part #1.) The Azure Synapse is an enterprise analytics service that accelerates time to insight across data warehouses and big data systems.

• “Time to Insight” refers to the amount of time it takes to derive meaningful and actionable insights from the data.
• “Big data systems” are the software and hardware solutions that are designed to store, process, and analyze large, complex data sets.

And with the help of Azure Synapse Analytics, you can accelerate time to insight across data warehouses and big data systems.

Part #2.) The Azure-Synapse brings together the best SQL-Technologies used in enterprise data warehousing, Spark technologies used for the Big Data Data Explorer for log and time series analytics, pipelines for data integration and ETL and processes and deep integration with other Azure services like power-BI, cosmos DB, and Azure Machine Learning.

Part #3.) Let us understand more about the Azure Synapse with the help of Data Engineering pipeline :-

1.) In the data engineering pipeline, we have the data sources, for example, data source one, data source two up to data source n. The data sources can be on premises data, cloud data, or the streaming data.

2.) These are the three important steps in the data engineering pipeline ETL operations, big data analytics, and storing the data in data warehouses.

• We have to build the ETL operations on that data using the tools like Azure Data Factory.
• After building the ETL operations, we have to do big data analytics. In big data analytics, we have to use the tools like Databricks.
• After that, we have to store the data in data warehouses like SQL DB.

3.) The Azure Data Factory is also connected to the ADLs, that is Azure Data Lake storage. After ETL operations, data is also stored in ADLs.

4.) At the end, we are taking the data from data warehouses and applying the business intelligence tools like power BI. With the help of these BI tools, we can visualize the data and at the end, we are providing this data to business decision makers.

==> The Azure Synapse Analytics brings all these capabilities at a single place. In Azure Synapse, you can do ETL operations, big data analytics and data warehouses. So, this is a single tool to do all these capabilities as well as you can use the business intelligence tools in Azure Synapse Analytics.

Question: Show the steps to launch the Azure-Synapse-Analytics ?

Step #1: Go-to the Azure-Synapse-Analytics at Azure Portal and hit the “Create New Workspace” button :-

Step #2: Once, we specify the name of resource-group, storage-workspaceName, storageAccountName and storageFileSystemName, we would be able to see the Syanspe-Workspace being created :-

• Resource-Group (newly created) → “aditya-synapse-project”.
• Synapse-Workspace (newly created) → “adisynapse”.
• Storage Account (newly create) → “adiprojectstoragesynapse”.
• File System (newly create) → “adifilesystem”.

Question: Show the project inside the Azure-Synapse-Analytics ?

Step #1: Let’s now go inside our storage-account & container and create the directory inside it with the name as : “Data” :-

We shall now upload the 2 files, with which we shall work throughout this project :-

Once the files are successfully uploaded inside the storage-account, we can see these 2 files :-

Step #2: Since this storage-account is linked to the Synapse-Studio, we can also see these 2 files inside the synapse-studio as well :-

Step #3: Now, we can see the SQL-Pool inside the Synapse-Studio :-

• Formally it is known as SQL Data Warehouse. It is a component of Azure Synapse Analytics used for high performance data warehousing and structured data analytics.
• SQL pool provides massively parallel processing architecture for the query execution. So, this is all about the SQL pool.

There are two types of SQL-Pools :-

• Serverless SQL Pool.
• Dedicated SQL Pool.

The SQL-Pool is created when we have created the synapse workspace itself.

• Type → “Built-In”.
• Status → “Online”.
• Size → “Auto”.

If we can go into the details of this SQL-Pools, we can see it’s details :-

PART #1 :: SQL-POOL

Example #1 → Let’s now see the URL of the file that we uploaded above :-

Let’s now perform some analytics now:- Let’s head to the “Develop” pane and write a SQL-Script. This SQL-Script would query to our data-set that we just uploaded above.

SELECT
*
FROM 
OPENROWSET (
    BULK 'https://adiprojectstoragesynapse.dfs.core.windows.net/adifilesystem/Data/employee.csv',
    FORMAT = 'csv',
    HEADER_ROW = TRUE,
    PARSER_VERSION = '2.0'
) AS [result]

That’s how we can write our First SQL-Query to view the data present in the ADLS-Gen2. We can now “Publish-All” all of these changes.

Example #2 → Let’s now perform the GroupBy operation on our data now :- Let’s head to the “Develop” pane and write this new SQL-Script :-

SELECT
dept, SUM(salary) AS TotalSalary
FROM 
OPENROWSET (
    BULK 'https://adiprojectstoragesynapse.dfs.core.windows.net/adifilesystem/Data/employee.csv',
    FORMAT = 'csv',
    HEADER_ROW = TRUE,
    PARSER_VERSION = '2.0'
) AS [result]
GROUP BY dept

That’s how have written our Second SQL-Query to perform the aggregation on the data present in the ADLS-Gen2.

==> We can now “Publish-All” all of these changes. Let’s press on the Publish button.

==> We can also see the Chart of this data from the “Views” :-

Example #3 → Let’s now analyse the NYC Taxi Data (in the PARQUET data format) with the serverless SQL Pool :-

Example #4 → Let’s now see the Top 100 Records of this data-set (NYC Taxi Data (in the PARQUET data format)) in the another way i.e. from Storage-Account itself :-

• Head to “Data” Pane → “Linked” → “ADLS Gen2” → Storage-Account → Container.
• Right click on the File → “Select TOP 100 rows”.

This would generate the code automatically :-

PART #2 :: SERVERLESS-SPARK-POOL

Question → Explain about the serverless spark pool ?

Answer → The serverless spark pool is a way of indicating how a user wants to work with the spark.

• When you start using a pool, a spark session is created if needed.
• The spark pool controls how many spark resources will be used by that session.
• It also controls how long the session will last before it automatically pauses.
• We pay for the spark resources used during that session and not for the pool itself, and this is because it is a serverless spark pool.

This way, a spark pool lets you use Apache Spark without managing the clusters, and it is similar to how a serverless SQL pool works.

Example #1 → Let’s now create the Apache-Spark-Pool :-

Step #1 :- First head to “Manage” section → “Apache Spark Pools” → “New”.

Step #2 :- Now, here we get a enough room for customisations, as in what type of Spark-Pool cluster that we wanted to create under “Basics” section :-

Step #3 :- Now, proceed to “Additional Settings” section, to select some more types of configurations :- Here note that, after 15 mniutes of Inactivity, the Spark-Pool would pause.

Step #4 :- Finally, under “Review + create” section, we get to review what exactly we shall be creating :-

Step #5 :- Once we hit the “Create” Button, we shall be able to see that, we have a new Spark-Pool created :-

To see the created spark-pool, click on this name and it would show us our newly created spark-pool inside this Synapse-workspace :-

• Here, we can also see the URL of this newly launched SparkPool.
• We can also modify the settings of this Spark-Pool, in case we wish to do so.

Example #2 → Let’s now show the example of using the Spark-Pool ?

Step #1 :- First head to the “Develop” section → “Add New Resource” → “NoteBook” :-

Step #2 :- Now, if you click at the Configure-Session button, you would observe that, there doesn’t exist any session yet :-

Let’s now write a simple print statement, so that the session can be established :-

• We can see here that, we wrote a simple print statement.
• We can see the session has now been created.

Example #2 → Let’s now show the example of Analysing the data (Let’s see top 10 records) with the Spark-Pool ?

Step #1 :- First head to the “Data” pane → “Linked” → “ADLSGen2” → Refer to the file and click on Properties :-

Now, note the ABFSS Path of the Data-File (to which we want to read) from the Properties section:-

Step#2:- Now head to the “Develop” pane → Write the NoteBook code to load the data from this parquet file into a Spark-DataFrame named “df” :-

Step#3:- Now , we can also see the top 10 rows of the DataFrame :-

Example #3 → Let’s now show bring this entire data from this DataFrame to the Database ?

Step #1 :- First head to the “Data” pane and from there we can see that, there doesn’t exists yet any database here :-

Step #2 :- We shall not create the SQL-Database from here directly, rather we shall get the Database created with the help of the Spark-Code :-

spark.sql("CREATE DATABASE IF NOT EXISTS nyctaxi")
df.write.mode("overwrite").saveAsTable("nyctaxi.trip")

• Here, we shall create the database with the name of “nyctaxi”.
• Here, we shall create the Table with the name of “nyctaxi.trip”.
• Also, whatever the data we are currently holding in the DataFrame, we shall be writing the same to a Table that we just created.

And we can now verify from the “Data” from the Database :-

newDFFromDB = spark.sql("SELECT * FROM nyctaxi.trip")
display(newDFFromDB)

• Here, we have created a new dataframe, but this time, this new dataframe is being loaded from the Table that we just recently created.
• And then, we are displaying this dataframe.

Another verification that we can do is :- If we go to the “Data” pane again, we can very well see that, there do exists the database, table and records inside this table :-

Example #4 → Let’s now perform some simple analysis from this data which is now present in our ?

Step #1 :- First, we shall write Spark-Script to fetch the data in aggregated format from the Spark-Database :-

analysisDataFrame = spark.sql("""
   SELECT PassengerCount,
       SUM(TripDistanceMiles) as SumTripDistance,
       AVG(TripDistanceMiles) as AvgTripDistance
   FROM nyctaxi.trip
   WHERE TripDistanceMiles > 0 AND PassengerCount > 0
   GROUP BY PassengerCount
   ORDER BY PassengerCount
""") 

display(analysisDataFrame)

analysisDataFrame.write.saveAsTable("nyctaxi.passengercountstats")

• We are performing the desired aggregation here with PassengerCount and total sum of distance travelled by them.
• We then display this aggregated data in this dataframe.
• We then write this data into a new table, which is also created on the fly here now.

Step #2 :- We can also verify that, whether this new table got created OR Not from the “Data” Section :-

• We need to first refresh the database, so as to see the newly created table.
• We now right click on the new table-name and write an auto generated sql-script to see the top 100 records.
• The same data can be seen here in the newly created Table as well.

Step #3 :- We can now bring the data from this table (passengercountstats) into a Spark-DataFrame :-

df = spark.sql("SELECT * FROM nyctaxi.passengercountstats")
display(df)

Example #5 → Let’s now bring this data into the Storage and perform some validations ?

Step #1 :- Now, whatever data we have loaded into the DataFrame (as shown in above steps), we can write this data into the Storage Account. Let’s first see the “Container”. It contains only one folder right now :: “Data”.

Step #2 :- Let’s execute this script now, which shall save the data in the 2 different formats i.e. CSV & PARQUET :-

df.write.mode("overwrite").csv("/NYCTaxi/csv_format")
df.write.mode("overwrite").parquet("/NYCTaxi/parquet_format")

Step #3 :- As soon as we execute this script, we can see the new directory getting created inside the Container :-

We can now go inside this new folder named : “NYCTaxi”. There is sub-folder named “csv_format” created inside it, as per the instructions we gave in the aforementioned script :-

There is another folder “parquet_format” inside the folder “NYCTaxi” :-

Step #4 :- Similarly, we can also create the file with the same name inside the folder. Let’s execute the following script now :-

df = spark.sql("SELECT * FROM nyctaxi.passengercountstats")
df = df.repartition(1)
df.write.mode("overwrite").csv("/NYCTaxi_2/csv_format")
df.write.mode("overwrite").parquet("/NYCTaxi_2/parquet_format")

This shall also create a new folder named “NYCTaxi_2” inside our container :-

Inside this folder, we can see 2 hierarchies i.e. 2 different folders :-

Inside each one of these sub-folders, we can see the files which has now been auto-created. The first sub-folder is “csv_format”.

Step #5 :- If we want, we can also see the data from this file as shown above screenshot, by right clicking on this file and “Select TOP 100 rows”. This shall auto-generate the code :-

Step #6 :- There is one another approach through which we can see the contents inside the folder.

As shown in above screenshot, Right click on the file and click on : “Load to DataFrame”. This would generate a small script :-

Copy this script & we shall bring this to our Note-Book where we were executing all the queries earlier. We can very well see the data from this Storage-Container now :-

Finally, in order to save our work, we can “Publish-All” our changes now as well :-

Example #6 → Let’s also show the approach to analyse the data from Storage-Account from PARQUET-format ?

Step #1 :- Now, let’s head back to our Linked Azure Data Lake Storage Gen2 Storage-Account. We can see the folder named : “parquet_format” here :-

Step #2 :- Now, go inside this folder : “parquet_format” and we would see this file. Now, right click on this file and click on “Load to Dataframe” :-

Step #3 :- This shall generate a script, as shown below :-

Step #4 :- We copy this script and bring it onto our main notebook, where we were writing all the queries. Once we execute this script, we can see the data from this parquet-file as well :-

That’s all that we have in this blog. We shall meet you in next blog in this series.

Question → Explain about the Virtual-Machine ?

Step #1.) This is the first step towards hosting an application & making it available to the users :-

Step #2.) Now, here are the repercussions to this situation :-

Step #3.) Here is the alternative to all this :-

Step #4.) Following are the steps involved in deploying a Virtual-Machine :-

Question → Assuming that, we have spinned-up a Windows based VM, how do we connect to the same ?

Let’s talk about the Architecture first :

Question → Setup a Project (based in .Net language) at our local machine and run it ?

Step #1.) Open the folder : “tmp2” in Visual-Studio-Code :-

Step #2.) Now, go inside the Visual-Studio-Code and click on : “Show and Run Commands” :-

Step #3.) Next, we click on “New Project” :-

Step #4.) Next, we select : “ASP.NET Core Web App” from the options :-

Step #5.) Let’s give the name of this app as : “learningapp” :-

With this, the entire project skeleton shall be formed inside the IDE :-

We can also see the Solution-Explorer view for this project :-

This project can now also be seen in the folder view :-

Step #6.) Let’s now go ahead and run this project by right clicking on the project :: “Debug” → “Start New Instance” :-

Once we Start the new Instance of this Project, we can see the logs flowing in the terminal as well :-

Step #7.) Here is now we can see the output, in the form of this WebApp being accessible from the Browser :-

Step #8.) Now, we publish the .NET based application with the help of this command :- “dotnet publish -c Release -o ./bin/Publish”

After the above step is succesful, we can see that, build artefacts are created in the below shown folder :-

Question → Now imagine that, I want to host my ASP.NET based application on the afore-created Windows based Virtual-Machine. Show steps, what all things we need to do there ?

Step #1.) We first need to install a IIS Web-Server on our Virtual-Machine :-

Let’s login (We shall login via the Windows RDP based mode) to this VM now and open the “Server Manager” at this VM :-

Let’s now configure the Web-Server on our VM :-

Let’s select the Roles now :-

Let’s select the Server-Roles now :: Click on check-box for : “Web Server (IIS)” :-

Here is we can see the Installation-Progress of Web-Server :-

Once done, we can see the status here :-

Step #2.) We also need to enable the Inbound Port for this VM. So, we go to this VM on dashboard :-

Here, we go to the Network-Settings :-

Once we scroll to the bottom, we can add this Inbound-Rule :- Here, we are adding a rule which allows Incoming traffic from any source at Port 80 :-

Step #3.) Once both of the above steps are done, we can check that, the default view of web-server is visible :-

Step #4.) Let’s now modify the Settings of our IIS Web-Server within our VM : “IE Enhanced Security Configuration” :-

Now, for this specific setting, turn it off :-

Step #5.) Next, in order for us to be able to run our .NET based application, we need to install the ASP.NET runtime (Hosting Bundle) as well :-

Inside our VM, let’s first download the .NET Core Hosting Bundle :-

Inside our VM, let’s download the Runtime :-

After the Hosting Bundle is installed, let’s go ahead and create a new folder inside our VM, from which we shall be hosting our App :-

Step #6.) Let’s now create the Website inside this Web-Server :-

Now, click on “Basic Settings” :-

Here, now click on “Edit” Settings for this site :-

We now change the port from 80 to 8080 :-

After we have changed the port, here are the settings :-

Now, we create the another Site like as shown in the screenshot below :-

Here, we select the physical path as the same one which we created above in Step #5 :-

Finally, we have the “LearningApp” website created now :-

Step #7.) Now, let’s connect to our VM, but before that, we modify the settings little bit, in such a manner that, the local-disk C: from our local computer is being shared on the VM :-

Now, login inside the VM and we can see that, the C-drive from my local computer is now visible inside the Virtual-Machine :-

Let’s now copy the publish folder (of our App) that we created in above step from this C-Drive to the Virtual-Machine :-

Let’s start the copy operation :-

Here is now, we can see the folder being successfully copied on the Virtual-Machine :-

Once the publish folder is done copying, we can see see that, the same application is now being reflected on the web-browser :-

That’s all in this blog. We shall meet you in next blog. if you liked it, please do clap on this page.

Let’s talk about the Architecture, that we are going to discuss in this blog :-

• Users wants to upload the Videos to the Application.
• Application saves those videos into the DataStore.
• Module processes these videos from the DataStore.

Question → What are the problems associated with the afore-mentioned design & what’s the solution ?

Answer → If the Processing module crashes, then there is no way of keeping track of what all videos got processed and what all videos were not processed ? Therefore, the solution for this problem is to have an Storage-Queue in between of the following two components :-

• Video-Store.
• Video-Processing-Module → This module is the one, which actually process the video.

Question → Show the demo of using the Storage-Queue ?

Step #1.) We first open the Azure-Storage-Account :- In the left hand side pane, we can see the various options for Data-Storage within the Storage-Account :-

• Containers.
• File Shares.
• Queues.
• Tables.

Step #2.) We now go to the Queues option here and create a new queue with name as “appqueue” :-

Step #3.) We now go inside this Queues option and perform :-

• Add a new Message to this Queue.
• The Message sent to the Queue would expire in 7 days.
• The message sent is encoded in Base64 format.

Step #4.) We can now see that, the message is now successfully arrived into the Queue. We can also see the properties of the message which have been saved to the Queue.

Step #5.) Similarly, let’s push another message to the same Queue :-

Step #6.) Now, we can consume/dequeue the message from this Queue :- As soon as the message is being de-queued from the Queue, that message is gone forever.

After the dequeue operation is successful, we can see that, only one message is being left in the queue :-

Step #6.) Now, we can also clear the queue and this option shall finish all the messages from queue :

After we perform this operation, we can see that the queue is empty now :-

Question → Show the demo of using the Storage-Queue through the C-Sharp Code for pushing the messages to the Queue ?

Step #1.) We first go to our Visual-Studio-Code and Add New Project :-

Step #2.) We now create a Console-App :-

Step #3.) We now create the name of project as “storagequeue”.

Step #4.) We now add the package to our environment by this command :-

dotnet add package Azure.Storage.Queues

Step #4.) We now go to Azure Portal and take the ConnectionString for our Queue :-

Step #5.) We now can refer to the Azure documentation for pushing the messages to the Queue :-

Now, we write the code in C-Sharp language, which would connect to this Storage-Queue and push the messages to it :-

Step #6.) We now execute this code-piece & can see the messages being pushed to Queue :-

Step #7.) We now can verify from the Azure Portal that, our logs are getting pushed to the Azure Storage Queue :-

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for peeking (i.e. Just Reading) into the messages ?

Step #1.) We first write the code for Peeking the message from line no. 19 to 28 as shown below :-

Step #2.) Now, we execute our code and we can see that messages are being read from queue. Please note here that, these messages shall not be deleted from the Storage-Queue i.e. they would still be present into the Queue.

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for Reading & Deleting the messages ?

Step #1.) We first write the code for ReceiveMessage from the Queue. Here note that, we have not deleted the message after reading the same :-

Step #2.) We now execute this code and can see that, the messages are being read from the queue :-

Step #3.) Initially, there were around 5 messages in our Storage-Queue. We read the three messages, so we are now left with 2 messages in the queue :-

But note that, since we didn’t explicitly deleted the messages from the Queue, they shall re-appear into the queue after 30 seconds of Invisibility Timeout :-

This is because :-

• The code that we wrote above would only make the message Invisible for 30 seconds (default) so that no other consumer/instance should be able to read this message during this 30 second period.
• As a good practice, the consumer have to manually delete the message from the queue, after the message is being processed succesfully.

Step #4.) Now, we explicitly delete the message, after the processing is done successfully :-

Now, we re-run this code again :-

And we can see that, the messages which has been consumed shall not re-appear into the queue now. So, Message 2, 3, 4 are consumed successfully and Message 0, 1 are now remaining in the queue now.

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for knowing the length of the Queue ?

Step #1.) We first write the code for checking the length of the queue :-

Step #2.) If we execute this code now, we can see the output :-

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for pushing the object into the Queue ?

Step #1.) We first write the blueprint for Object (“Order”) that we wanted to push to the Queue :-

Step #2.) We now write the code to initialise the instances of this class and push to the queue :-

Step #3.) Now, if we execute this code, we can see, it is successfully pushing the messages into the Queue :-

Step #4.) We can also verify that, the messages are now visible into the Azure Dashboard :-

Here, we can see a sample message by clicking upon it :-

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for receiving the object from the Queue, using Azure Functions ?

Step #1.) We first create an Azure Function inside our Workspace :-

Next, We now create a new Project :-

Now, we select our Runtime as .NET :-

Here, we now select the : “Azure Queue Storage Trigger” :-

For this very purpose, we shall need a Storage Account and we shall be using the Azurite Emulator for Local Storage :-

Next, we specify the Queue Name :-

Step #2.) We now write the code for consuming from the Queue i.e. QueueTrigger. Note here that, we have consumed the message in the format of : QueueMessage :-

Step #3.) We now check the ConnectionString for our StorageAccount which we shall be configuring in our codebase for QueueTrigger :-

In below code-snippet, we configure the connection-url that we got from above dashboard :-

Step #4.) Now, let’s go ahead and consume from the Queue. Here we can see the logs which indicates something fishy. Our consumption have failed.

We can see that, there are no messages remaining in our queue :-

Also, these messages have automatically landed to an another queue which is also known as Poison-Queue. This Poison-Queue is something which got automatically created, because the messages could not be processed here :-

In order to debug this, let’s add logging to our code with mode as Trace :-

Now, if we try to consume the message again, we can see the error clearly that : The message decoding has failed.

Step #5.) Now, let’s go ahead and fix this issue at the time when the messages are being sent to the Queue. So, we perform the encoding of the message to Base64 format :-

Let’s run this code, so as to see whether messages are being sent to the queue :-

We can also verify the messages being received at the Azure dashboard :-

If we now re-run the QueueTrigger, we can see the messages consumed clearly :-

Also, we can see that, there are no messages remaining in our queue :-

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for receiving the object from the Queue, using Azure Functions , in the same format/Type as that of Object rather than Generic message format ?

Answer → Note that, we have already re-pushed the messages to this queue, before proceeding to consume from these messages :-

Step #1.) We first modify the code of our Azure Function, so as to receive the Object as it is, while consuming from the Queue :-

Step #2.) We now consume the message from this Queue and can see that, the messages started to be consumed :-

Question → Show the demo of using the Storage-Queue through the C-Sharp Code, for receiving the object from the Queue, using Azure Functions and writing the same into the Storage — Table ?

Step #1.) We first create a Table with name as : “Order” :-

Once the Table is created, we can see this on the dashboard :-

We can see that, as of now, there are no records in this Table :-

Step #2.) We now write the code of this Class, whose object shall be saved to the Table :-

Now, we write the code for consuming the messages from the Queue :-

Step #3.) We can see that, on line no. 20 above, we can see that, there is an error. Here, we install the Azure Extension for Tables using this command :-

dotnet add package Microsoft.Azure.Functions.Worker.Extensions.Tables

As we install the aforesaid extension, we can see that, this error is gone from the codebase :-

Step #4.) Finally, we can see that, while we execute the code of our Consumer again, we can see that the messages are being consumed successfully from the Queue and are being saved to the Table effectively.

We can also verify from the Azure Dashboard that, these records are now visible at the Azure Portal Dashboard as well :-

That’s all in this Blog. We shall see you in next blog. If you gained something from this blog, I encourage you to clap on this page.

References :- https://learn.microsoft.com/en-us/azure/storage/queues/storage-quickstart-queues-dotnet?tabs=passwordless%2Croles-azure-portal%2Cenvironment-variable-windows%2Csign-in-visual-studio-code

If you are landing first time, this is the link to go for you. In this blog, we shall learn about Data-Engineering from Microsoft-Azure :-

Background → The resources shown in the above slide are the most commonly used resources by any data engineers to build any kind of data engineering projects.

Step #1.) So, the data source that we are going to use in this particular use case is : On-Premises-SQL-Server-Database. Imagine that, inside the database we are having like 6 or 7 tables. And here, we are going to migrate this database completely to cloud.

• So, as part of this the first step, would be using a tool called Azure Data Factory.
• The Azure Data Factory is a ETL tool, which is mainly used for data ingestion.
• What we are going to do is :: The data factory will be used to connect to On-Premises SQL server database, in order to copy all the tables from the database and move all the tables to the cloud.

Step #2.) For Data Storage in the cloud, we are going to make use of Azure Data Lake Gen2.

• ADLS2 is the storage solution in Azure.
• The data which is stored in Azure Data Lake is pretty cheap.

Step #3.) Once the data has been added into the ADLS2, we’ll be using a tool called Azure Databricks to transform all the raw data into the most curated data.

• Azure Databricks is a big data analytics tool which is mainly used for high end data analytics workloads.
• In simpler words, we can say that using Azure Databricks we can actually write the code either in SQL or PySpark or Python.

Step #4.) There is a concept called Lake House architecture. Usually, In this architecture, the Azure Data Lake will be divided into multiple layers. We have different layers such as bronze layer, silver layer and gold layer.

1.) Bronze layer → ADF connects to the on premise server database, copies the data and put the data into the Azure Data lake.

• So inside Azure Data Lake, the Azure Data Factory puts the data first into the bronze layer, which means that the bronze layer has an exact copy of what the data looks like in this data source.
• Usually, we are not going to touch any data inside the bronze layer, and we are not going to change the format or anything inside the bronze layer. So, this is going to be the source of truth.
• One of the main advantages of this is, for example, in the subsequent data transformations that we are going to do if something goes wrong, you can come back to this bronze layer and get all the data to be retrieved, which is going to be the same as the data source.

2.) Silver Layer → So once the data has been copied to this bronze layer by Azure Data Factory, we then use Azure Data Bricks to connect to this bronze layer, do some data transformation and load the transformed data into the silver layer.

• These silver layer and gold layer is kind of a different levels of transformation.
• For example the first level of transformation is silver layer And the next layer of transformation is the gold layer.
• So in bronze to silver layer the transformation might be kind of simple, like changing the column names or changing the data types, because a lot of times the on premise kind of data types and the cloud data types is not really compatible.
• So we may need to change few things based on how it is going to support it into the cloud. So those kind of minimal transformation can be done in this silver layer.

3.) Gold Layer → And once the data has been transformed using Azure Data Bricks and the final data has been loaded into the silver layer, we then do another set of transformation using same Azure Databricks.

• And this data is going to be loaded finally to this gold layer.
• So this gold layer is the final cleanest form of data. So we all need to have a clean data right.
• So one of the main task of data engineers is to clean the raw data into the most curated data which means that all this data transformation is done by the data engineers using Azure Databricks in different zones.

So that’s called Lake House Architecture.

Step #5.) So once the data has been loaded to the gold layer, now we are using another tool called Azure Synapse Analytics.

• Azure Synapse analytics is kind of similar to what is on premise SQL server database.
• We can also create the same kind of database, similar to how we can create using the on premise SQL database.
• We can also create all these database and tables using Azure Synapse Analytics and once these database and tables have been set up in Azure Synapse Analytics, all the data that is present in the gold layer will be loaded to all the tables that we have created in Azure Synapse Analytics.
• So now at the end of this step, the Azure Synapse Analytics will have a kind of a similar data warehousing model of how it looks like in On-Premise-SQL-database.

Step #6.) Next, we are going to do further analysis on the data that has been stored inside the Azure Synapse Analytics using a tool called power BI.

• PowerBI will get all the data that has been loaded into the Azure Synapse Analytics and we could create a different kind of reports like the charts, bar charts or whatever the reports that we want in power BI.
• Usually, either the Data-Analyst OR Data-Engineers (and sometimes Devs) are the one who will be creating reports in power BI.

Question → Showcase the process of copying a particular table from On-Premise Database Table to Azure ?

Solution → Here, we have first created the following resources :-

Here is our local database on our own server :-

We now create a User, using which we shall be logging-in to our database.

On the user “mrk”, we give the permission of “db_datareader”, so that our user have necessary permissions :-

Here, we create the “username” as a Secret in the KeyVault :-

Next, we also create “password” as an another secret in the KeyVault :-

Finally, we are having 2 secrets in the KeyVault :-

Now, we go to “Azure Data Factory” and we shall be setting-up the Data-Copy operation from our On-Premise-Database-Table to the Azure ADLS. So, let’s go ahead & click on the “Launch Studio” here :-

Now, there is a concept of “Integration-Runtimes” :-

• AutoResolveIntegrationRuntime → Used for the scenarios where we want to connect to the Azure resources itself.
• SelfHostedIntegrationRuntime → Used for the scenarios where we want to connect to customer’s On-Premise resources.

Now, we go ahead and create a new SHIR (Self Hosted Integration Runtime) → We now choose Azure, Self-Hosted :-

Next, we choose here Self-Hosted as the option :-

We now specify the properties of IR Setup like Name, Description and Hit the Create Button →

For the setup, we have 2 options :-

• Express Setup → This is used when we want to setup for our own computer.
• Manual setup → This is used when we want to setup at multiple other computers in a company.

Let’s go ahead with Express Setup for now. Once the express setup is done, we can open the “IR” from our computer and this would look something like this :-

Next, we can see here that, SHIR is now up & running :-

Part A. Copying a single Table from On-Premise Database to Azure Cloud

Next, we come in the “Author” Section, where we can lay down a Pipeline. Note that, a Pipeline would have multiple Activities.

We specify the name of this pipeline as : “copy_pipeline” :-

Inside this pipeline, we can create the Activities. One such activity that we would demonstrate is to copy one of the Table from On-Premise to Azure.

Part-1 : Source

Let’s now go to the Source and configure a new DataSet. We shall choose “SQL-Server” because we want to connect to the On-Premise-DB :-

Here, we specify the name of the property and then have to attach a Linked-Service :-

We shall create a new Linked-Service :-

The properties specified above have been copied from the On-Premise SQL Server itself. For example, we can go to our On-Premise Database Engine :-

We now copy the Server-name as “localhost” :-

We copied the Database Name from our On-Premise :-

Now, we choose the AuthenticationType as “SQL-Authentication”. Here the userName is being entered as manually :--

For password, we would use it from KeyVault’s secret. Let’s first create Azure-KeyVault-Linked Service :-

After we create the Linked-Service, the Secrets would not load yet :-

For this to work, we need to allocate the permission in the keyVault :-

Next, we select our Principal (Azure Data Factory in this case) to which the permissions have to be allocated :-

After the permissions are allocated, we can see that, the secrets would be loading easily without any concerns :-

Now, we choose the Secret Name :-

Once the Linked-Service for the SQL-Server (On-Premise Database Server) is created, we shall be able to see the Tables (These are the tables which are visible in the On-Premise Database) loading :-

Next, let’s choose the Table-Name (“SalesLT.Address”) to which we wanted to copy from On-Premise-Database to the Azure :-

Here, we can also see the preview of the data from the “Preview data” option :-

Here is how the data look like :-

Part-2 : Sink

Now, we go and configure the Sink DataSource where finally the data shall be saved :- We shall choose ADLSGen2 from here :-

We choose the format of the data as Parquet :-

Next, we provide the properties of this configuration. Here, we first create the Linked-Service :-

We specify the properties of the Linked-Service here like Name, IR type, Authentication Type, etc..

• Note here that, we are choosing the AutoResolveIntegrationRuntime because the sink datasource is present in none other than Azure itself.
• AuthenticationType being chosen here is AccountKey.

We choose the Storage Account Name into which the data has to be copied :-

Here, we also choose the Folder into which data has to be dropped :-

Here, once we select this path, we can see the properties section :-

Part3 : Testing the Pipeline

Now, we can debug our Pipeline that we have just laid down by “Debug” button :-

Once we started the debug Pipeline option, we can see that, we get here the Output as follows :-

After sometime the process would be finished :-

And we can now verify from the Storage-Account as well, whether the data has been copied OR not :-

Finally, we click on the Publish-All button to publish these changes.

Part B. Copying the entire DataBase from On-Premise Database to Azure Cloud

First & foremost, we come in the “Author” Section, where we can lay down a Pipeline. Note that, a Pipeline would have multiple Activities.

We specify the name of this pipeline as : “copy_all_tables” :-

Before, we create the activities here under this pipeline :-

• Let’s have a look at all the tables (present in the SalesLT schema), to which we want to copy from On-Premise to Azure-Cloud.
• We have overall 10 Tables, which has to be copied.

Now, we head back to ADF and create our first activity “Lookup”.

• This activity shall lookup for all the tables in our On-Premise Database.
• We have specified the name of this activity as : “Look for all tables”.

Let’s configure a fresh/new Source-DataSet here :-

We choose SQL-Server as new data-source :-

We name this new SQL-Server as “SqlDBTables” and utilise our existing LinkedService named “onpremsqlserver” (This was created in Part-A above) :-

As a part of this LinkedService :-

• We can see that, the we are using SHIR, which shall be installed on the On-Premises Machine where our SQL-Server is present.
• Also, we would not select any specific table here in the “Lookup” step, because we wanted to perform the look-up for all the tables.

After we configure this Activity, here is how it looks at Azure-Dashboard :-

Because, we wanted to look-up for all the tables present in the SalesLT schema, therefore we would write the query here in the “Lookup” activity, in order to fetch all the tables :-

We can also check the tables fetched by clicking upon the “Preview Data” button :-

So far finally, We can look for the final page of the pipeline : “copy_all_tables”, where there exists one activity named “Look for all tables” :-

Now, we perform the “Debug” step for this pipeline :-

We can see that, under “Output” tab, there exists a Job which is queued up :-

Once this pipeline is succeeded, the status can be seen below :-

In order to see the output, we have to click on the special button under the “Output” folder :-

First, we can look at Input here, which is nothing but the query that we configured in the Lookup activity :-

And next, we can see the output, which is nothing but the list of all the tables that we fetched from this Pipeline :-

Next, we want to iterate over all the table-names that we have got and copy them one by one. So, we create a new activity called “ForEach” and attach it to Lookup activity

We name this activity as : “ForEach Schema Table” :-

Now, we go to Settings tab to explore some configurations for this new activity :-

Now, we click on this button : “Add Dynamic Content” for this activity under settings and build this activity of this pipeline : Basically, we are reading from the previous Activity and reading its output.value →

Next, we go to “Activities” section of this specific Activity :-

Here, we specify the “Copy Data” as activity inside this “ForEach” activity :-

We change it’s name first of all to : “Copy Each Table” :-

Now, we go inside the Source configuration for this activity and make use of the “SqlDBTables” Source DataSet :-

Next, we make use of the existing Linked-Service and SHIR :-

And we also configure the below shown Pipeline Expression Builder, because for every table there, I want to read all the rows from that table :-

Now, we move on to Sink portion of this Activity :-

We want to choose the Sink as ADLSGen2 :-

We also choose the format of the data that we wanted to store in the ADLSGen2 as “Parquet” :-

Once we choose this format, we specify its properties like its name and file-path. We click on “Browse” button on this file-path :-

Here, we select the folder that we have created already in our storage account :-

And after we select this folder, this is how dashboard look :-

We want to organise our data in the ADLSGen2 like this :-

Now, we go to Sink part of this Activity & open the Sink DataSet :-

Here, we specify following 2 parameters :-

Here, finally, the sink portion of this dataset looks like :-

We also build the pipeline through expression builder, with the help of 2 parameters that we created above :-

And our sink property looks like this :-

We finally now go to Publish the Pipeline :-

And once the pipeline is published succesfully, we can see the same from Dashboard :-

We now run this Pipeline with the help of Trigger on the ADF :-

We can see the Triggered Runs as well for this pipeline :-

Here, we can see the details of all the activities involved in this pipeline :-

As we have 10 tables and all of these tables have some data, after sometime we can see the status would change to Success :-

And finally we can see the output here i.e. Inside the ADLSGen2 Storage, we can see that, a new folder has been created for each individual table in the “bronze” folder of ADLSGen2 Storage :-

That’s all in this blog. We shall meet you in next blog. If you enjoyed learning about ADF, please do clap on this page. See you soon.

Part 1. Consistency in Cosmos-DB

Question → Talk about the Data Consistency Levels in Cosmos DB ?

Answer → There are many consistency levels which are supported by Cosmos-DB :-

Here is the understanding about each one of these levels :-

• If we are in Banking Domain OR any other Financial Domain, the consistency level should be kept as Strong.
• If we are in Social-Media Domain, the consistency level can be kept as Eventual, because it hardly matters if some likes OR comments comes out of order.

Strong Consistency → For example if writes are happening in Region : “West US 3” and reads happens from “West US” & “North Central US”, then the reads would see the latest data.

Bounded Staleness → For example if writes are happening in Region : “West US 3” and reads happens from “West US” & “North Central US”, then the reads would be visible after sometime. They shall not be immediately visible.

Eventual Consistency → For example if writes are happening in Region : “West US 3” and reads happens from “West US” & “North Central US”, then the reads may not be in order AND they would be visible late i.e. not instantaneous.

Question → Give one end to end demo about the Data Consistency Levels in Cosmos DB, through codebase ?

Step #1.) Here we have coded the Model, which shall be used to ingest/save the documents into Cosmos-DB :-

Step #2.) Assuming that, we have some records already present into our Container :-

• Let’s try to fetch the documents, with the help of this codebase as shown below.
• Note that, by default the Strong consistency Level shall be applied.

Step #3.) Here, we can see the charge in terms of RUs in order to fetch the record on the basis of Id + PartitionKey.

Part 2. Writing Data to Multiple Regions in Cosmos-DB

Question → Can we have an option, through which we can write the data to multiple regions simultaneously ?

Step #1.) We can enable the Multi-Region writes, from Cosmos-DB Dashboard UI :-

If we scroll to the right, we can see that, the write has been now enabled in the multiple regions. In the below screenshot, we have 3 regions where writes shall happen :-

• West US.
• West Central US.
• Canada Central.

Step #2.) We can also see here on the Overview page that, there are 3 Write-Locations as well as there are 3 Read-Locations :-

Step #3.) Now, let’s first note down the connection-strings, in order to connect to this Cosmos-Database :-

Step #4.) Now, let’s try to now code for the same :- We first define the endpoint and key, the ones that we copied from above :-

Now, we try to insert the record into the Cosmos-DB.

Note that, since Multi-Region-Writes are enabled, there are 20.57 RUs consumed, which are quite high, even for a simple write of a record. This is because, we are writing in 3 regions at a single point in time.

Step #5.) Now, we can go to the Cosmos-Dashboard and see the same record that we have inserted through the code :-

Part 3. Querying from Cosmos-DB

Question :- Give one end to end Demo to play with Cosmos-DB API of NoSQL-API, especially from querying prospective ?

Answer → Let’s imagine that, we already have a Container named “products” and it already contains some Documents.

Step #1.) This is how, we can fire the Select Query on our Cosmos-DB :-

Step #2.) For the afore-said select-query, we can also see the number of RUs being consumed :-

Step #3.) We can also perform selective projection from the table i.e. we can only fetch some limited columns :-

Step #4.) We can also query from our Container on the basis of some where clauses i.e. query on some column as shown in below query.

• In the below screenshot, we can see the RUs charged to us for this query.
• Please note that, since Cosmos had to do little extra work, therefore the count of RUs being consumed are little higher.

Part 4. Ingesting & Querying the data to Cosmos-DB

Question :- Give one end to end Demo to play with Cosmos-DB API of NoSQL-API, especially from Documents-Ingestion prospective ?

Answer → Now, we shall be Ingesting some records/documents into the Container named “products” :-

Step #1.) This is how, we can save/ingest a fresh/new document into our Container in Cosmos-DB :-

Step #2.) Similarly, we can also ingest some another document.

• While we are ingesting a document into Cosmos, the fields that we ingested specifically are : “categoryId”, “categoryName”, “price”.
• While we are ingesting a document into Cosmos, the fields that get auto-created (i.e. System Generated) by themselves are : “id”, “_rid”, “_self”, “_etag”, “_attachments”, “_ts”.

Step #3.) Similarly, we can also ingest some another document.

Step #4.) Similarly, we can also ingest some another document.

Step #5.) Similarly, we can also ingest some another document.

Step #6.) Now, we fetch the documents from Cosmos through SQL Query :-

In this case, the total RUs being charged for querying all these documents are :-

Step #6.) Now, let’s fetch some selective data only from the Containers :-

Note that, in the above query, considering the Record #1 :-

• Key is “categoryName”.
• Value is : “Apple”

Similarly note that, in the above query, considering the Record #2 :-

• Key is “categoryName”.
• Value is : “Banana”.

If we only want to fetch the values out of the K,V pairs, then we can fetch the same through this query :-

Step #7.) Now, if we only want to fetch unique values of a particular attribute :-

Now, if we only want to fetch unique values i.e. only names of a particular attribute :-

Part 5. Another demo for Querying the data to Cosmos-DB

Step #1.) Using the “cosmicworks” tool, let’s insert some data to our database and container which already exists :-

• Database → cosmosdb255
• Container → product

Step #2.) We can now verify that, the records are now visible in our container :-

Step #3.) By default, Cosmos creates the Indexes on all the fields :-

Step #4.) Now, let’s query on this Container and note the RUs consumed are 8.41 :-

Step #5.) Now, let’s query on this Container and additionally ask for ordering of data too. So, here cosmos have to do some extra bit of work. Therefore we can observe that, the count of RUs being consumed are little higher :-

Part 6. Playing with Gremlin-API of Cosmos-DB

Part 6.1 Creating a Database :-

Question :- Give one end to end Demo to work with Cosmos-DB API of Gremlin Graph ?

Answer → We shall show the demo for Cosmos-GraphDB :-

Step #1.) We here create the Cosmos-DB-Account first :-

Step #2.) We now choose the API along with which, we shall work in Cosmos-DB-Account :-

Step #3.) We now choose the capacity mode from following 2 options :-

• Provisioned Throughput.
• Serverless.

Here, we shall be choosing the Serverless Mode :-

Step #4.) Finally, we hit the Create Button here :-

Once the account is created, we can see our deployments :-

Here is how, our Cosmos-DB-Account would look like :-

Step #5.) Let’s now create a Database :-

Part 6.2 Creating Container inside the Database :-

Step #6.) Now, we create a Container/Table/Graph inside this particular Database :-

Here, we can see that, we now have :-

• Database Name → “graph-database”.
• Table/Container → “persons-graph”.

Inside this Container, there can be multiple entries OR Rows OR Records.

Part 6.3 Inserting Vertices into the Graph :-

Step #7.) Here is how we can insert the Vertices into this Container :- g.addV(‘person’).property(‘name’, ‘Ravi’).property(‘age’, 39).property(‘userid’, 2)

Inserting another record :- g.addV(‘person’).property(‘name’, ‘Ranga’).property(‘age’, 40).property(‘userid’, 1)

Part 6.4 Inserting Edges into the Graph :-

Step #8.) Here is how we can insert the Edges into this Container :- g.V().hasLabel('person').has('name', 'Ranga').addE('knows').to(g.V().hasLabel('person').has('name', 'Ravi'))

Inserting another Edge:- g.V().hasLabel('person').has('name', 'Ranga').addE('knows').to(g.V().hasLabel('person').has('name', 'Ranjith'))

Part 6.5 Fetching Vertices from the Graph :-

Step #9.) Here is how we can fetch the graph’s vertices :-

So, we can see that :

• We have 3 vertices i.e. 3 persons that we added to this Graph : [Ravi, Ranga, Ranjith].
• For every person, there are some properties that do exists.

Part 7. Indexing with Cosmos-DB

Step #1.) We assume here that, our Database already exists here. Let’s go and check the Connection-Strings and Keys here :-

Step #2.) Let’s create a new container here with the name as “products” :-

Step #3.) By default, Cosmos creates indexes on all the fields/columns of a table. We can see the same from here :-

Step #4.) Next, lets go to the code. As shown in below snapshot, we are showing the mechanism to insert the records into the Cosmos-Containers :-

We are generating some records ourselves, which we shall be ingesting to our Container. Each record (that we generate here) shall have following fields :-

• Name.
• Price.
• CategoryId.

This is how the final leg of our code looks like :-

Once we execute this code snippet, we can see here :-

We can verify the documents from our Cosmos-Dashboard as well :-

Step #5.) Let’s now query our Table on the basis of the field : “name”.

• Note that, we dont have any records for name = “Licensed”. This is an intentional empty query that we have fired.
• Also, as of now, the name field is indexed, because this field was indexed from Container creation time itself.

Note that, the RUs being charged for this query are : 2.8 RUs.

Similarly, we now query our table on the basis of “price” which is again the indexed field. Here as well, note down the RUs that are there :-

Step #6.) Now, let’s create the index on only “price” field and no other field :-

Step #7.) Here, if we now query on the “name” field (which is no more a indexed field in Cosmos Container), we can see the RUs consumed shall be too much.

Part 8. Indexing Optimisation with Cosmos-DB

Step #1.) Let’s assume that, we already have a Database (named “cosmicworks”) and a Container (named “products”) inside it. By default, Cosmos creates indexes on each and every field as shown here :-

What we are going to do here is : To read through our sample file and create the document into our Cosmos-Container :-

We are using the Library from Cosmos, in order to Upsert the Item into Cosmos-DB :-

Step #2.) Let’s now Build our Program : dotnet build

We now execute our program and interesting thing to note here is that, the RUs consumed to insert this record into our Container are 48.38. This is because of the reason that, indexing on all the fields is by-default ON (as we saw in the Step #1 above) :-

And hence we can verify that, we can see our Record into our Container :-

Step #3.) Let’s now go ahead and modify the settings in such a manner that, we only create the Indexes on only 2 below mentioned fields & remove indexes from any other field :

• name.
• categoryName.

We now again insert our record into Container. Couple of things to note here that :-

• Our RUs consumed have reduced from 4838 to 8.38.
• This is because, we are now NOT indexing on all the fields, but rather just indexing on only 2 fields.

Learning here is that →

• If the indexing is ON on all the fields, then during write operation, much-higher RUs shall be consumed.
• If the indexing is ON on all the fields, then during read operation, much-lesser RUs shall be consumed.

Therefore, we should only create the Index on the fields which are really required to be queried later.

Now, let’s do one more experiment here by removing indexing from even these 2 fields as well :-

Now, if we try to ingest a record once again, we can note that, the count of RUs consumed have further reduced :-

That’s all in this blog. If you liked reading it, do clap on it. We shall see you in next blog.

Question → What is Cosmos-DB ?

Question → How the data is organised inside the Cosmos-DB ?

Question → What are various APIs exposed by Cosmos-DB ?

This is how the dashboard interface of the Cosmos-DB homepage looks like :-

Question → How is Cosmos-DB different from other databases ?

Question → What are the naming conventions used with Cosmos-DB for various APIs ?

Question → Explain about the concept of Request-Unit (“RU”) in Cosmos-DB ?

Answer → RU stands for Request-Unit.

• The usage in Cosmos-DB is expressed in terms of RU.
• Each Database Operation (like Read, Insert, Delete, etc.) consumes a variable number of RUs.

Question → At what level, do we configure the RUs i.e. whether we do at Container Level OR Database level ?

Answer → Yes, this is very much possible and its a good practice.

If we have to configure the RU at Database level, this is how it would look like :-

If we have to configure the RU at Container level, this is how it would look like :-

If we have to configure the RU at mixed level, this is how it would look like :-

Question → Talk through example about the computation for RUs ?

Question → How do we compute, for our particular workload, how much of RUs shall we need ?

Answer → Azure have exposed a Capacity-Calculator where we can compute the number of RUs that we shall need for our specific use-case based upon combination of all the below factors :-

• Size of Item in our workload.
• Read Traffic per second.
• Create Traffic per second.
• Update Traffic per second.
• Delete Traffic per second.

Here, Cosmos-Capacity-Calculator have automatically computed : How many RUs shall be needed for the given CRUD load (Assuming that the size of our Item is 1 KB) :-

• Create Traffic would be 10 RPS → 50 RUs/sec.
• Read Traffic would be 50 RPS → 50 RUs/sec.
• Updates Traffic would be 99 RUs/sec.
• Delete Traffic would be 0 RUs/sec.

Similarly, if we know increase the item-size to 2 KB, we can see the updated costing :-

Question → Talk about the concept of Logical & Physical Partitions available with Cosmos-DB ?

In simpler words, we can say that, each container may physically exist at multiple Physical Machines. But this all is abstracted from the User i.e. User would never come to know, at which all machines my container is being stored ?

Question → Talk more about the concept of PartitionKey in Cosmos-DB ?

Answer → Imagine that :-

• We choose the PartitionKey as “City” for our use-case.
• In this case, whenever we insert any records / documents in the container, all those records with partitionKey as “London” shall be stored in one single logical partition.
• Similarly, all those records with partitionKey as “Paris” shall be stored in one single different logical partition.

Question → What shall happen, if there are too many records for a given city i.e. London ?

Answer → Usually a Container is divided into logical partitions on the basis of the partitionKey.

• Imagine, we get comparatively higher count of records for city : “London”.
• In this case, it would lead to problem of Hot Partition, because the logical-partition (corresponding to city: London), shall be too much populated as compared to the other partitions.

Therefore, our strategy to choose the partitionkey should be such that, the records should get distributed equally amongst all the Logical-Partitions, so that we can avoid the Hot-Partitions.

Question → Talk through an example of Single Partition Query ?

Answer → In below example, we have chosen userName as the partitionKey. Therefore, all the records which have userName as ‘Sara’ shall be saved in one single logical partition.

Question → Show where we configure the RUs ?

Answer → Since ours is a free account, we get 1000 RU per second and 25 GB of storage amount. See in below screenshot, this is how, it is configured.

Question → Let’s talk about various scenarios to talk about How Cosmos-DB should be used ?

Question → What if we want the items to be auto-deleted from the Cosmos-DB-Container, after a certain time-period ? Do we have some TTL like feature in Cosmos-DB ?

We shall use TTL feature of Cosmos-DB in following scenarios :-

We can define the TTL at Container level. This shall automatically apply the TTL to all the items inside this container.

In case, we want to apply TTL at Record/Document level, we choose the option : “On (no default)” and define the TTL for the document through code, while ingesting.

Question → What is the difference between two available ways to of configuring the Cosmos-DB in terms of Capacity i.e. “Provisioned” and “Serverless” ?

Here is the snippet from the MSFT documentation itself :-

Question :- Talk more about the Pricing Calculator for Cosmos-DB Account ?

Answer → We can check for the pricing from Azure website as well. Imagine that we are checking for API choice as : NoSql. Now there are 3 options available to us :-

• Standard Provisioned Throughput (Manual).
• Automatic Provisioned Throughput.
• Serverless.

1.) Let’s check first for Standard-Provisioned-Throughput (Manual) →

• We have chosen the RUs as 400 per second.
• Imagine we are going to run this for a month, because we are computing for Monthly cost. In a month approx, we have 730 Hours.
• The costing may vary from Region to Region.
• So, we can see in below screenshot that, the costing shall be : 4 X 730 X 0.008 = 23.36 USD per month for 400 RUs per second.

• Also, imagine our dataset would come under the 10 GB, so costing for the same shall be : 2.50 USD per month, as shown below :-

2.) Let’s now check first for Automatic Provisioned Throughput :-

• We have chosen the RUs as 1000 per second, again for a month with 70% utilisation.
• The costing comes to be 61.32 USD per month.

Question :- Show by creating an Non-Serverless Account for Cosmos-DB ?

Step #1.) We start to create the Cosmos-DB-Account :-

• AccountName → cosmosdbact255
• Location → (US) West US
• Capacity Mode → Provisioned Throughput

Step #2.) Now here, we choose the Global-Distribution. There are 3 options available here :-

Geo-Redundancy → It means, the data shall be automatically backed-up to the Paired-Region.

Recall that, In step #1, we chosen our Region as : (US) West US. We can see from the publicly-available Paired-Region list that, for this region the paired region is : East US

Therefore, the data from our Cosmos-DB-Account shall be automatically backed-up in paired region :-

If we would had chosen some other region, the paired region would had been different :-

Please note one thing about this option that :-

• Imagine Main Region is 1 and Paired Region is 2.
• Now, our application is writing to Region #1, then in this case, although the data shall be replicated to Region #2, but the data would only be available for Reads in Region #2 and No Direct Write shall happen to the Region #2 directly.

Step #3.) Next is the Networking Option for Cosmos-DB-Account :-

Step #4.) Next is the BackUp Policy :-

4.a) Option #1 → Continuous BackUp →

• This option is available for free of cost for 7 days. For 7 Days, no additional cost shall be incurred on our Subscription.
• This option is also available for 30 days of backup and in this variant, there shall be some modest cost applicable.

4.b) Option #2 → Periodic BackUp → Here we need to configure 2 things :-

• Backup Interval → Can be configured in Hours OR Minutes.
• Backup Retention → Can be configured for last number of days.

Note that, it is showing us the number of copies would be 2 in this case.

Now, if we perform the backup every 6 hours for 15 days, in this case the number of copies shall be 60 :-

Next, we have three options on where do we want to have the Backup :-

• Locally Redundant Backup Storage → It means we want to backup in the same Datacentre, as that where our primary data resides.
• Zone Redundant Backup Storage
• Geo Redundant Backup Storage

Question :- For Cosmos-DB, can we enable to Geo-Redundancy after the Account Creation ?

Answer → Yes, this is very much possible. Imagine our Primary Region is : “US West”, then we can configure the other regions as Replica-Regions. For example, we have shown in below screenshot :

• UK South
• Japan East.

We can also enable the Geo-Redundancy at Account level, even after the Account creation as well :-

Once we enable the Geo-Redundancy, from here we can also choose all regions, to which we want our data to be replicated. In below example, we have chosen two additional regions as our Read-Regions :

• South Central US.
• West US 3.

We can also see this configuration also reflecting at our Cosmos-DB-Account Page :-

• Read Locations → West US, South Central US, West US3
• Write Locations → West US.

There shall also be the COGS implications for enabling the Replicas for Cosmos-DB :-

Question :- Talk about the Service managed Fail-Over ?

Answer → Now, imagine something goes wrong with Cosmos-DB in the West-US-Region (Our Primary region for Writes), in that case we can enable the Automatic-FailOver that should happen to the Primary-Replica-Region. This is how, we get the Reliability with Cosmos-DB.

Step #1.) So, let’s first go ahead and Enable-Service-Managed-FailOver. We can also change the priorities for the Replica-Region from the same dashboard.

Step #2.) Finally, once we save our Settings, we can see that the Automatic-Service-Managed-FailOver has been switched-off.

Now, this would be helpful in a scenario, when the Primary Region gets into some trouble from Cosmos-DB side. This is something that we can’t replicate our own-selves.

Question :- Talk about the Manual Service Fail-Over ?

Answer → Now, imagine that we want to manually switch-over to some replica-region.

Step #1.) We enable the Failover by manually choosing the Region where we want the Failover to happen :-

Step #2.) We can now see that, Manual-Failover is being triggered now :-

We can also see that, the changed configuration is also reflecting at our Cosmos-DB-Account Page now :-

• Read Locations → West US, South Central US, West US3
• Write Locations → West US3

This is how the response looks like, if we try to read via the SDK :-

That’s all that we wanted to cover in this blog. Pl clap, if you liked reading it. Do check out the Part #2 for this Blog as well.

Example #1

Question → How the permissions are checked for the Users ?

Answer → We can basically assign different Roles to a particular User.

Question → Showcase how can our Application, can access some resources which are present in our Azure Storage Account, using the very basic approach ?

Answer → Here is how we can access the files present in our Storage-Container :-

Step #1.) Let’s first see the files which are present inside our container :-

• Storage Name → “appstore55455344243”
• Container Name → “data”

Step #2.) Let’s now navigate to “Access Keys” section for our Storage Account. Below screenshot, we can see the ConnectionString and Key that we can use, in order to access this storage-account :-

Step #3.) Here is how our code looks like in order to access this file. We make use of following things :-

• BlobServiceClient
• BlobContainerClient
• BlobClient

In the above approach, there is a problem — We have manually entered the credentials (i.e. ConnectionString) inside our code.

Question → Showcase how can our Application, can access some resources which are present in our Azure Storage Account without having to mention the connectionString ?

Answer → This can be done with the help of “Application-Object”.

Step #1.) Let’s first see navigate inside our EntraId first.

Step #2.) We would register an Application now :-

Step #3.) We would now supply the name of the Application :- In our case, we are specifying the name as “blobapp”.

Step #4.) Once our Application is registered successfully, we take note of following things :-

• TenantId.
• ClientId

Step #5.) We would now go inside the : “Certificates and Secrets” as shown below and create the “Client-Secrets”.

Here is we can see the “Client Secrets” :-

Step #6.) Now, so far we registered an Application and now, we need to allocate the permission to this Application-Object. So, let’s first go inside the Storage-Account → Access-Control → Add-Role-Assignment.

Next :-

• We have selected the role as : “Storage Blob Data Reader”.
• Under the option : “Assign access to” :-we choose : “User, Group, or Service Principal” :-
• When we click on Select-Members → we can select our Application-Object i.e. “blobapp”.

Finally, once the Role is assigned to this ApplicationObject, this is ready to be used in our application-codebase.

Step #7.) Now, we shall directly access the files, which are present inside our Storage-Account → Container → Files using :-

• ClientSecretCredential
• BlobClient

Now, when we execute our code at our local-personal-On-Premise-machine, we shall be able to download the file successfully now :-

Question → What’s the main dis-advantage of using Application-Object mechanism of accessing the Azure-Resources from the Application codebase ?

Answer → The main problem here is :- Even though, we have made use of RBAC, we still needed to embed the credentials of the Application-Object in our codebase :-

Example #2

Question → What is the alternative to the Application-Objects ?

Answer → The alternative to the “Application Object” is to use : “Managed Entities”. This gives us a way for our Applications to authenticate to Azure Resources without the need of embedding the credentials.

Question → What is the main requirement in order to use the “Managed Entities” ?

Answer → The main requirement is :

• Our application-code needs to be hosted on a service that supports Managed-Entities like VM, App-Engine, etc.
• Under the hoods, the “Managed Entities” shall create a Service-Principal for that resource.

Step #1.) After our first is successfully created at our Azure Account, Under the “Identities” section, we enable the System Managed Entity :-

Once we enable the “System Assigned Managed Entity”, here is how it would look :-

Step #2.) Now, let’s assign the RBACs. Here, we are saying , we shall allocate the permissions of accessing the Storage-Account to our VM.

• So, for this, we go to the Storage-Account → “Access Control (IAM)”.
• Please note that, this is the Account, that has to be accessed by our application-codebase.

Step #3.) We shall choose the Role as : “Storage Blob Data Reader” and move forward :-

• We choose : “Managed Identity”.
• Under Members, we choose our VM, to which the access to Storage-Account has to be assigned.

So, here we choose our VM that we created in aforementioned-step :-

After the Role has been assigned to the afore-shown VM, here is how the UI would look like. Now, we hit the : “Review and Assign” button.

Step #4.) The class that we shall use to access the Storage-Account from the Application-codebase is : “DefaultAzureCredential” :-

Step #5.) The code to access the Storage-Account & objects in the container , through the DefaultAzureCredential looks like as follows :-

• BlobClient.

Please note that :-

• The aforementioned code-base can’t be executed at our local-on-premise-machine, rather this code can only be executed at VM, which is present inside our Azure Ecosystem.
• In the above mentioned steps, we have created the VM (at which we shall be running the afore-shown code-piece).
• So, we first build the code at our local and would take the executables at our VM.

Question → How does DefaultAzureCredential works under the hoods ?

Answer → Under the hoods, it generates a token, which VM shall be used to hit the Storage-Accounts :-

Step #1.) Here, Azure exposes this endpoint, which we shall copy from here :-

Step #2.) Let’s now write this code, to access the token for the Storage-Account :-

Step #3.) We build this code at our local machine and take this executable at the VM. Once the executable is there is at the VM, we shall execute it and we can now see the token, which is what being used under the hoods of ManagedEntity :-

Step #4.) Now, we can also fetch, some more informations from this Token :-

Step #5.) Now, we again build the code at our local :-

We now take the executable to VM :-

We now execute the same DLL at VM and we can see some additional information from this Token :-

• client_id
• token_type

Step #6.) Here is now, we can execute our code to access the Storage Account too :-

The procedure would remain same. Build at Local → Transfer to VM → Execute at VM.

Example #3

Question:- Demonstrate the concept of Managed-Identities with some different example ?

Step #1.) Imagine that we have a Virtual-Machine.

• First, we go and enable the System-Assigned Identity at this VM.
• Once we enable to System-Assigned Managed-Identity for this Resource (VM), it would be able to call other Azure services.

Once we enable the System-Assigned-Managed-Identity for this VM, we can see it’s Principal-Id :-

Step #2.) Imagine that our application is running at the afore-said VM and it wants to access the below shown Storage-Account (“tempfunction4567grob884”), then we shall have to : “Add the Role Assignment” on the storage-Account :-

Here, we have to choose the Job-Function-Role : “Storage Blob Data Contributor”, because we want our VM to be able to contribute to the Storage Blob Data :-

Under the Members section, we can choose “Managed-Identity” and under the MI, we can choose the VM (for which we enabled the MI in step #1) :-

We shall be able to see the VM (for which we enabled the MI in step #1) :-

Here, we can see the final step :-

So, finally now, our application (running at our VM) would now be able to access the Storage-Account automatically.

Example #4

Question:- Demonstrate the concept of UAMI with some different example ?

Step #1.) Let’s first create an User-Assigned-managed-Identity :-

This is how our UAMI would look like, once it is succesfully created :-

Step #2.) Now, we go to our Storage-Account and attach this UAMI to this Storage Account :-

From the Role, we shall assign the : “Storage Blob Data Delegator” role to the Storage-Account :-

Now, we associate the Managed-Identity to this Storage-Account :-

Here is how, it would look, after we attach the MI to this Storage-Account :-

And we can see here that, UAMI has been associated with the Storage-Account succesfully :-

Step #3.) Now, we associate the afore-created UAMI with our Virtual-Machine.

• Note that, our intention here is that, our application running on this VM, should be able to access the afore-shown Storage-Account.
• We can attach this UAMI to as many resources as we want. Say tomorrow some App-Function wants to access this Storage-Account, then we can associate this same UAMI to that App-Function as well. We don’t need to enable the MI again & again inside the Storage-Account.

Let’s add the Identity to our VM now :-

Question :- What is a Node ?

Answer :- A node is a machine, physical or virtual on which Kubernetes is installed.

• A node is a worker machine and that is where containers will be launched by Kubernetes.
• It was also known as minions in the past.

Question :- What if the node on which your application is running fails?

Answer :- Well obviously our application goes down.

• So you need to have more than one nodes. A Cluster is a set of nodes grouped together.
• This way even if one node fails you have your application still accessible from the other nodes.
• Moreover having multiple nodes helps in sharing load as well.

Question :- Now we have a cluster. But who is responsible for managing the cluster? Where is the information about members of the cluster stored? How are the nodes monitored? When a node fails how do you move the workload of the field node to another worker node?

Answer :- That’s where the Master comes in.

• The Master is another node with Kubernetes installed in it and is configured as a Master.
• The master watches over the nodes in the cluster and is responsible for the actual orchestration of containers on the worker nodes.

Question :- What are the various components of Kubernetes ?

Answer :- When you install Kubernetes on a system, you’re actually installing the following components.

1.) The API server acts as the front end for Kubernetes. The users, management devices, command line interfaces, all talk to the API server to interact with Kubernetes cluster.

2.) The etcd key store is a distributed reliable key value store used by Kubernetes to store all data used to manage the cluster.

• When you have multiple nodes and multiple Masters in your cluster, etcd stores all that information on all the nodes in the cluster in a distributed manner.
• etcd is responsible for implementing locks within the cluster to ensure that there are no conflicts between the Masters.

3.) The Scheduler is responsible for distributing work or containers across multiple nodes.

• It looks for newly created containers and assigns them to nodes.
• The controllers are the brain behind orchestration.
• They are responsible for noticing and responding when nodes, containers or end points goes down.

4.) The controllers make decisions to bring up new containers in such cases.

5.) The Container Runtime is the underlying software that is used to run containers.

• In our case it happens to be Docker, as we shall demo below.
• But there are other options as well like VirtualBox.

6.) Kubelet is the agent that runs on each node in the cluster.

• The agent is responsible for making sure that the containers are running on the nodes as expected.
• So far, we saw two types of servers. Master and Worker and a set of components that make up Kubernetes.

Question :- How are these components distributed across different types of servers? In other words, how does one server become a master and the other the slave?

Answer :-

• The worker node or minion as it is also known is — where the containers are hosted. For example, Docker containers.
• And to run containers on a system, we need container runtime installed and that’s where the container runtime falls. In this case it happens to be Docker. This doesn’t have to be Docker. There are other container runtime alternatives available such as Rocket or Cryo.
• The master server has the kube API server and that is what makes it a master.
• Similarly, the worker nodes have the kubelet agent that is responsible for interacting with a master to provide health information of the worker node and carry out actions requested by the Master on the worker nodes.
• All the information gathered are stored in a key value store on the master.
• The master also has the control manager and the Scheduler.

Question → What is our goal with Kubernetes ?

Answer → Our ultimate aim is to deploy our application in the form of containers on a set of machines that are configured as worker nodes in a cluster.

• However, Kubernetes does not deploy containers directly on the worker nodes.
• Rather, the containers are encapsulated into a Kubernetes object known as pods.

Question → What is a Pod ?

Answer → A pod is a single instance of an application. A pod is the smallest object that you can create in Kubernetes.

==> Here we see the simplest of simplest cases where you have a single node Kubernetes cluster with a single instance of your application running in a single Docker container encapsulated in a pod.

==> What if the number of users accessing your application increase and you need to scale your application?

==> Solution is : you need to add additional instances of your web application to share the load. Now, where would you spin up additional instances?

• Do we bring up new container instance within the same pod? The answer is No.
• We create new pod altogether with a new instance of the same application.

As you can see in above diagram, we now have two instances of our web application running on two separate pods on the same Kubernetes system or Node.

==> What if the user base further increases and your current node has no sufficient capacity?

==> Well then you can always deploy additional pods on a new node in the cluster. You will have a new node added to the cluster to expand the cluster’s physical capacity as shown below.

So what I’m trying to illustrate is that :-

• Pods usually have a 1 to 1 relationship with containers running your application to scale up.
• You create new pods and to scale down you delete existing pods.
• You do not add additional containers to an existing pod to scale your application.

Question → Are we restricted to having a single container in a single pod?

Answer → No. A single pod can have multiple containers except for the fact that they’re usually not multiple containers of the same kind.

• There might be a scenario where you have a helper container that might be doing some kind of supporting task for our web application, such as processing a user entered data, processing a file uploaded by the user, etc.
• And you want these helper containers to live alongside your application container?

In that case, you can have both of these containers part of the same pod so that :-

• When a new application container is created, the helper is also created.
• And when it dies, the helper also dies since they are part of the same pod.
• The two containers can also communicate with each other directly by referring to each other as local host since they share the same network space Plus they can easily share the same storage space as well.

Question → Explain some important Kube commands ?

1.) Starting the Kubernetes Server at our machine :- minikube start --driver=docker

• You asked Minikube to start a single-node Kubernetes cluster using Docker as the machine.
• It pulled a lightweight base image (kicbase).
• Created a Docker container acting as your Kubernetes control-plane.
• Booted up Kubernetes 1.32.0 inside that Docker container.
• It also configured kubectl automatically to connect to your new cluster.

2.) Checking the Status of Kubernetes Server :- minikube status

• You checked the status. Output says:
• host: Running
• kubelet: Running
• apiserver: Running
• kubeconfig: Configured

==> This means Minikube is successfully running a working K8s cluster inside Docker!

3.) To see the Nodes of our K8S Cluster :- kubectl get nodes

• You asked Kubernetes to show the cluster’s nodes.
• Output:
• minikube — your single node.
• STATUS = Ready

==> Your control-plane node is healthy and ready for workloads.

4.) To launch a new Pod containing Docker Container :- kubectl run nginx — image=nginx :-

• What this command really does is it deploys a Docker container by creating a pod. So, it first creates a pod automatically.
• Inside this pod, it will run one container and then deploys an instance of the Nginx docker image. In this case, the Kubelet pulls the Nginx image from the Docker Hub Repository & starts the container.
• Docker Hub, as we discussed, is a public repository where latest Docker images of various applications are stored. You could configure Kubernetes to pull the image from the public Docker hub or a private repository within the organization.

5.) Now that we have a pod created, how do we see the list of pods available? → kubectl get pods

• The kube control get Pods Command helps us see the list of pods in our cluster.
• In this case, we see the pod is in a container creating state and soon changes to a running state when it is actually running.

6.) Now that we wanted to look into the details of this specific pod → kubectl describe pod <POD_NAME>

• We can see from below screenshot that, the Image being used to launch this pod is : “nginx”.
• It’s current status is : “Running”.
• We can also see the hierarchy of the Events, which shows the lifecycle of this Pod.

Here is yet another example of one Pod, in which we have two different containers running :-

• nginx → For this container, there is a Image from which it is being powered.
• agentx → For this container, it’s status is Waiting.

7.) Now that we wanted to see the Node, on which this particular Pod is running → kubectl get pods -o wide

8.) Now that we wanted to delete a particular Pod → kubectl delete pod webapp

Question → How do we launch a Pod in Kubernetes using the YAML file ?

Answer → Here we have defined the configuration of a Pod :-

• Name of this Pod shall be “myapp-pod”.
• Two labels have been given to it as shown below.
• The value of kind is Pod.
• Image being used to launch the container (inside the Pod) is : “nginx”.

Question → Show the way to generate the YAML file automatically in the Kubernetes ?

Step #1.) First, we write this command to generate the YAML File : kubectl run redis — image=redis123 — dry-run=client -o yaml

• Name of this Pod shall be “redis”.
• The container (inside this Pod) shall be started from the ImageName : “redis123”.
• The kind of this resource is a POD.

Step #2.) Now, we save this YAML file into a file.

Step #3.) Now, we view the contents of this file and finally launch a Pod i.e. Create a Pod from this YAML File : kubectl create -f redis.yaml

• Since we intentionally supplied the incorrect ImageName, thats why we can see that, status of the Redis Pod is : “ErrImagePull”.
• Pl see below, where we have fetched all the pods using command : kubectl create -f redis.yaml

Step #4.) Let’s now go ahead and modify this file “redis.yaml” through VI Editor :-

Step #5.) Now, we again try to start the Pod from this YAML file : kubectl apply -f redis.yaml

• Finally we can see in below screenshot that, the “redis” pod has been successfully created.
• Since, we already create the Pod earlier, but earlier command didn’t worked and therefore we are now doing the apply operation.

Question → Explain the concept of Replication-Controller ?

Step #1.) If a particular POD dies OR is unhealthy, then our customers shall be suffering because of this.

Step #2.) That’s where the concept of Replication-Controller comes into picture. It would automatically create a fresh & healthy Pod, if existing Pod goes down.

Step #3.) Say, the load on our application increases further more, then the Replication-Controller can help in scaling the Pods even further as well.

Note that, the concept of “Replication-Controller” & “Replica-Set” are almost same.

Question → Showcase the way of creating a ReplicaSet ?

• We specify the apiVersion here as : “apps/v1”.
• We specify the kind as “ReplicaSet”.
• We specify the replica as 3, which means that we shall have 3 pods inside this ReplicationController.
• We specify the spec in which we specify the template of the Pod (the one that has to be created inside this ReplicaSet).

Please note that, the matchLabels are very important to be mentioned here, so that ReplicationController can manage/identify the Pods of same type through the help of these labels.

Hence, In our above example of creating a ReplicaSet, we shall be creating these 3 Pods and all of them shall be having the label as : “front-end”.

Important commands used above are :-

1. To create the ReplicationController → kubectl create -f rc-definition.yaml
2. To see the existing ReplicationControllers that exists → kubectl get replicationcontroller
3. To see the pods overall → kubectl get pods

Question → Say, we wanted to upscale the ReplicaSet, then how shall we do it ?

Approach #1 → We first edit the YAML file as shown below and run the command : kubectl replace -f replicaset-definition.yaml

Approach #2 → In case we dont wanted to modify the file, we can also upscale our ReplicaSet on the fly with this command : kubectl scale — replicas=6 -f replicaset-definition.yaml

Approach #3 → We can also upscale our ReplicaSet on the fly with the help of the ReplicaSetName too: kubectl scale — replicas=6 -f myapp-replicaset

Question → Show another example of creating a ReplicaSet containing Nginx Pods ?

Step #1 → Here is the definition of our replicaset.yaml file looks like :-

Step #2 → Here is how, we execute this file :- kubectl create -f replicaset.yaml

Step #2 → Here is how, we can fetch the replicaset :- kubectl get replicaset

Step #3 → Under the hoods, this ReplicaSet created the 3 Pods and we can see the Pods as : kubectl get pods

Step #4 → We can also see the details of a particular ReplicaSet : kubectl describe replicaset <NAME_OF_REPLICASET>

Step #5 → Let’s now try to delete one of the Pod and we shall see the magic that, ReplicaSet automatically shall be creating a fresh new pod: kubectl delete pod <POD_NAME> Please note the names of the pods in below screenshot and above screenshot.

Question → Now that, we have a ReplicaSet of 3 pods with label as (app: myapp) and say we launch the Pod independently and this Pod also have the label as (app: myapp), what shall happen ?

Step #1 → Here is the definition of our YAML file with the help of which we shall a fresh Pod. Note that, so far we already have 3 pods running already.

Step #2 → If we see the Pods now, we would still see the 3 pods only, because the ReplicaSet shall be terminating this 4th pod, even if it is running independently.

And finally we can see that, we have 3 pods running :-

Summarise the commands learned so far :-

That’s all in this Blog. We shall see you in another blog with some other concepts.

Conceptual Time

Question → What is Glossary in Apache-ATLAS ?

Answer → It basically associates the business meaning to the Entities in the ATLAS.

Question → Show, what is Glossary in general sense ?

In the General Sense: “Glossary” → A glossary is a list of terms and their definitions — like a dictionary, but usually specific to a domain. For example:

A glossary helps people across teams, projects, and companies understand the same meaning for business terms. It avoids confusion and ensures consistency. See this is how, a sample glossary looks like :-

In Apache Atlas: “Glossary” → A Glossary in Atlas is a formal way to capture and manage business vocabulary in your Data Catalog. It lets you define:

• Terms (example: Customer, Order, Revenue)
• Categories (grouping of terms)
• Relationships (link terms to data assets like Tables, Columns, Files).

Example in Atlas:

Demo Time

Glossary

Step #1.) We create the Glossary with the name as “User” from the ATLAS-UI :-

This is how it looks like, after it is created :-

Category

Step #2.) Inside the Glossary, we can now create the Category.

We wanted to create a Category named “Customer” inside this particular Glossary :-

Let’s create another Category named “Seller” inside this same Glossary named “User” :-

Step #3.) Inside a particular Category, we can also create the sub-categories too, but we shall not create any sub-category for now.

Step #4.) Let’s now create another Glossary as well with name as : “Sales” :-

Term

Step #5.) Now, we can also create the Terms, inside a particular Glossary :-

Step #6.) We create a Term named as “CustomerName” :-

After the Term is successfully created, here is how it looks like :-

Assigning Category to the Term

Step #7.) We can now associate the category (Customer) to the term (CustomerName) as well :-

After the category is being associated to the Term, here is how it looks like :-

Adding Classification to the Term

Step #8.) We can also associate the Classification (CustomerPII) to the term (CustomerName) as well. Note below that, we have chosen the type of classification as “Propagate” which means that, this classification shall be propagated.

After the classification is being attached to the Term, here is how it looks like :-

Let’s also associate one more Classification (“Dimension”) to this Term (“CustomerName”) but this would not be propagated :-

After this specific classification is being attached to this Term, here is how it looks like :-

Adding Terms to the Entities

Step #9.) We shall now associate the Term to the existing Entities in our system.

• Here, the “name” is one entity (i.e. Instance i.e. Object) of the Type: Column, that exists in our system.
• We shall associate the Term to this Column.

So far, we have only one Term named “CustomerName” and let’s associate this term to the afore-shown entity.

Once this term (CustomerName) is assigned successfully to this Entity (name) :-

• Below is how the view of the entity looks like.
• Also note that, this entity have also got a propagated-classification with name as “CustomerPII”. Visit to Step #8 where we associated the classification (CustomerPII) with the Term (CustomerName).

Ease of Search for this Entity

Step #10.) Now that, the classification (CustomerPII) exists with this entity (name), therefore it is quite easy to search from the ATLAS-UI.

Additional Term

Step #11.) Now, let’s create a new additional Term, this time, inside the “Sales” Glossary :-

Step #12.) We create a Term named as “salesPrice” :-

After the Term is successfully created, here is how it looks like :-

That’s all about the glossary in this blog that we wanted to show. If you liked reading it, please do clap on it.