<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Houssem Dellai on Medium]]></title>
        <description><![CDATA[Stories by Houssem Dellai on Medium]]></description>
        <link>https://medium.com/@HoussemDellai?source=rss-20f6b842a7cd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V5kVbxFTxOkzbIQul6RUpw.png</url>
            <title>Stories by Houssem Dellai on Medium</title>
            <link>https://medium.com/@HoussemDellai?source=rss-20f6b842a7cd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 09:11:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@HoussemDellai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introduction to Kubernetes Service Catalog]]></title>
            <link>https://medium.com/@HoussemDellai/introduction-to-kubernetes-service-catalog-37317b15670?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/37317b15670</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[open-service-broker-api]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ak]]></category>
            <category><![CDATA[osba]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Mon, 06 Jan 2020 08:09:22 GMT</pubDate>
            <atom:updated>2020-01-12T06:27:50.836Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog/">Service Catalog</a> is what allows Kubernetes to provision and connect to resources in the cloud. Let me explain through a demo.</p><p>In Kubernetes, we can deploy ‘almost’ anything. Anything that is a container. We can deploy SQL/NoSQL Databases, API Gateways, Queueing systems, monitoring tools, etc… But, it is still hard manage them. For example, running a database server in a container in Kubernetes is not always the best option. Because, if we will need a High Availability (HA), SLA of 99.99%, Failover plans… then we will need to do a lot of work. It is doable, for sure. But we still need to have a dedicated team with deep knowledge about k8s, working only on this.</p><p>On the other side, the Cloud providers have some nice managed PaaS offers. They have HA, SLA of four or five nines, failover plans…</p><blockquote>Why not using PaaS offerings to complete apps running on Kubernetes ? Well, that is the idea behind a Service Catalog.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBnzIbhRgO4E%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBnzIbhRgO4E&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBnzIbhRgO4E%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c50400bd4fcf22d5156716f0fbae3ca7/href">https://medium.com/media/c50400bd4fcf22d5156716f0fbae3ca7/href</a></iframe><h4>1. What is Service Catalog ?</h4><p>Service Catalog is an extension to the Kubernetes API. It enables k8s to provision and connect to external managed services like databases and messaging systems.</p><p>We know that a managed Kubernetes cluster have some extension API to create resources in the cloud like a managed storage disk/file, Load Balancer, or even a Node/VM. These resources are vital and essential for k8s. Each cloud provider implements this API into its managed k8s offering. If we are using AKS, we would benefit from this API to use an Azure Disk or Azure Load Balancer. The same applies for EKS, GKE, etc…</p><p>Service Catalog, however, extends even more these capabilities. It can create and connect to ‘almost’ any service on the cloud. We said ‘almost’, because it is up to the cloud provider to implement the API that enable this integration.</p><p>How Service Catalog works ?</p><blockquote>To provision and connect to a cloud service, Kubernetes should talk to Service Catalog which in its turn should talk to Open Service Broker specific implementation for Azure or AWS or GCP etc…</blockquote><h4>2. What is Open Service Broker API (OSBA)</h4><p><a href="https://www.openservicebrokerapi.org/">Open Service Broker API</a> defines a specification for listing, provisioning, accessing and deprovisioning cloud services. It have implementations for many cloud providers. Each implemenetation have a list of services that can be managed:</p><p><a href="https://osba.sh/"><strong>OSBA for Azure</strong></a>: Azure SQL Database, Azure Database for PostgresSQL / for MySQL, Azure CosmosDB, Azure Redis Cache, Azure Service Bus, Azure Storage, Azure Event Hubs, Azure Key Vault.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dODrm-7TMq0EcbBUOHGtQg.png" /><figcaption>Azure PaaS services available for Service Catalog</figcaption></figure><p><a href="https://aws.amazon.com/partners/servicebroker/"><strong>Service Broker for AWS</strong></a>: Amazon Relational Database Service (Amazon RDS), Amazon EMR, Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), and Amazon Simple Queue Service (Amazon SQS).</p><p><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/google-cloud-platform-service-broker"><strong>GCP Service Broker</strong></a>: BigQuery, Cloud Bigtable, Pub/Sub, Cloud Spanner, Cloud SQL, Cloud Storage.</p><h4>3. Deploying Service Catalog into Kubernetes</h4><p>Service Catalog is an extension API. This means that is not installed by default in Kubernetes. And because it depends on the OSBA, we need to deploy both. We will use an AKS cluster with Azure’s OSBA implementation.</p><h4>3.1. Deploying the Service Catalog with Helm</h4><p>The easisest way to deploy into k8s is using <a href="http://helm.sh/">Helm</a>. Lets first install Helm into k8s, then add Helm repository for Service Catalog:</p><pre>$ <strong>helm init</strong><br>$ <strong>helm repo add</strong> <strong>svc-cat</strong> <a href="https://svc-catalog-charts.storage.googleapis.com/">https://svc-catalog-charts.storage.googleapis.com</a></pre><p>Now, we can deploy the Service Catalog from the Helm repo:</p><pre>$ <strong>helm install svc-cat/catalog</strong> <br>       --name catalog <br>       --namespace catalog<br>       --set rbacEnable=false<br>       --set apiserver.storage.etcd.persistence.enabled=true<br>       --set apiserver.healthcheck.enabled=false<br>       --set controllerManager.healthcheck.enabled=false<br>       --set apiserver.verbosity=2<br>       --set controllerManager.verbosity=2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BvNS01uE-f5t0QzGucNMJQ.png" /><figcaption>The objects deployed with Service Catalog</figcaption></figure><p>One of the objects created is the APIService (<em>v1beta1.servicecatalog.k8s.io</em>). We can check if it was successfully created:</p><pre><strong>$ kubectl get apiservices</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WdibuPLvFt1eH5d9vLkVUw.png" /></figure><h4>3.2 Deploying Open Service Broker for Azure with Helm</h4><p>Now we need to install OSBA into Kubernetes. Again, we’ll be using Helm charts from the Azure repository.</p><pre><strong>$ helm repo add azure </strong><a href="https://kubernetescharts.blob.core.windows.net/azure">https://kubernetescharts.blob.core.windows.net/azure</a></pre><p>Then, we’ll proceed to the deployment. But, before that, we need to understand that OSBA will communicate to Azure API, thus it needs authentication and authorisation. For that, we need to create a Service Principal to get the access right.</p><pre><strong>$ az ad sp create-for-rbac</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYO8L3yxqUgBMilyqeRIiQ.png" /></figure><pre><strong>$ helm install azure/open-service-broker-azure</strong> <br>       --name osba <br>       --namespace osba <br>       --set azure.subscriptionId=$AZURE_SUBSCRIPTION_ID <br>       --set azure.tenantId=$AZURE_TENANT_ID # tenant<br>       --set azure.clientId=$AZURE_CLIENT_ID # appId<br>       --set azure.clientSecret=$AZURE_CLIENT_SECRET # password</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjelYI_qbzulGOYpw3RwKg.png" /><figcaption>Objects deployed with Open Service Broker for Azure</figcaption></figure><p>It will take a few minutes before the service broker gets ready. We can check its status here:</p><pre><strong>$ kubectl get ClusterServiceBrokers</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgoTch-Pbjauxp1GyPL-VA.png" /></figure><h4>3.3 Exploring Service Catalog CLI: svcat</h4><p>Kubectl is what we typically use to interact with Kubernetes cluster including some Service Catalog API. But, for even more flexibility, Service Catalog had created its own CLI tool, that is <a href="https://svc-cat.io/docs/cli/"><strong>svcat</strong></a>. We can install it using <a href="https://svc-cat.io/docs/install/#installing-the-service-catalog-cli">this link</a>.</p><p>Lets explore the available commands with svcat:</p><pre><strong>$ svcat --help</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_feIAZ0fv1HxNvgB_66S0A.png" /><figcaption>Commands available with svcat CLI</figcaption></figure><p>To get the list of different brokers, as we have done before with command: <em>kubectl get ClusterServiceBrokers</em></p><pre><strong>$ svcat get brokers</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gf742IvQ3dcbSGk_IvbZMA.png" /></figure><p>The broker defines the list of cloud services we can create through the command marketplace. Note here the services mentioned earlier like SQL databases, CosmosDB, etc.</p><pre><strong>$ svcat marketplace </strong># or<strong> </strong>svcat get classes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RwvHRTYzjPS8H-D-tkIiCg.png" /></figure><p>Note that each offer have different plans like Basic, Standard, Premium, etc. We can get the list of all these plans with the command <strong>svcat get plans</strong>.</p><p>Later, we will be using the class ‘azure-sql-12–0’.</p><h4>4. Provisioning SQL Azure Database with Service Catalog</h4><p>The goal here is to use Service Catalog to create an Azure SQL database then connect it to a web app. We’ll be using yaml files with app’s source code which are available here: <a href="https://github.com/HoussemDellai/OSBA-Azure-AKS/blob/master/sql-instance.yaml">github.com/HoussemDellai/OSBA-Azure-AKS</a></p><h4>4.1 Provisioning SQL Azure Server and Database</h4><p>Service Catalog API uses the same structure of Kubernetes manifest files. We need to use the object type <strong>ServiceInstance </strong>and specify the class name with<strong> clusterServiceClassExternalName </strong>with the plan through<strong> clusterServicePlanExternalName. </strong>Note how we could specify the resource location and the resource group name.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d45c4c5dbd38f465600748d3d7bc642/href">https://medium.com/media/8d45c4c5dbd38f465600748d3d7bc642/href</a></iframe><p>To deploy this object, we’ll use the command:</p><pre><strong>$ kubectl create -f sql-instance.yaml</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxAsK2vH00ff3fXMQ3Kz-Q.png" /></figure><p>Then we can check if the creation was successful:</p><pre><strong>$ svcat get instances -n osba <br></strong>$ kubectl get ServiceInstances -n osba</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rBqR8QH3DFurbnOr9-IUkA.png" /></figure><p>Now if we go to the Azure portal, we can see that there is a new resource group created with the two resources: Azure SQL server and database, great!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uuV9Rk0eDSrY1nlpgGjWnw.png" /><figcaption>Azure SQL database created by Service Catalog</figcaption></figure><p>Now we want to connect to this database from the app. We can go to the database properties to retrieve the connection string from Azure portal. This is bad from a security perspective. Because we will be sharing sensitive data across email, slack, postit, etc. Fortunately, there is a another better solution. Service Catalog solves this prolem.</p><h4>4.2 Binding Database secrets with the cluster</h4><p>Service Catalog can get the Database information like name, username, password, host URL and port number. Then it will share these information in a Secret object in Kubernetes. To ask it to do that, we create a <strong>ServiceBinding</strong> object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bac92ed606624857a4a2deaebde19383/href">https://medium.com/media/bac92ed606624857a4a2deaebde19383/href</a></iframe><pre><strong>$ kubectl create -f sql-binding.yaml</strong></pre><p>The database credentials will be saved into a secret file called ‘my-sql-secret’. Lets view its content:</p><pre><strong>$ kubectl describe secret my-sql-secret -n osba</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u97XpOlIBQWB5uq_rmDoyQ.png" /></figure><p>We can also view these credentials within the k8s dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MdsCzgza28OFX6KQsiVIWg.png" /></figure><h4>5. Connecting a sample web app to the database instance</h4><p>Now that the credentials are exposed through a Secret object, we can use it to connect a web app. We have a sample .NET Core web app configured to read the connection string different parts from environment variables.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e1379c983aa56cc19026b7f4304de5d/href">https://medium.com/media/6e1379c983aa56cc19026b7f4304de5d/href</a></iframe><p>And we need to pass these environment variables through the Deployment file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c30e3afe3afc13952c50e943fbc0d30c/href">https://medium.com/media/c30e3afe3afc13952c50e943fbc0d30c/href</a></iframe><pre><strong>$ kubectl create -f app-deploy-svc.yaml</strong></pre><p>If all works fine, we should be able to open the web app URL on a browser and navigate to web-app-url/Products to see an empty table for products. That means the web app successfully connected to the SQL database.</p><h4>5. Deleting the created resources</h4><p>To delete all the created resources:</p><pre><strong>$ helm delete osba --purge<br>$ helm delete catalog --purge<br>$ svcat delete instance my-sql-instance -n osba<br>$ svcat delete binding my-sql-binding -n osba</strong></pre><h4>To learn more</h4><p>I run a youtube channel where I post videos about DevOps, Docker and Azure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7HLsaPycVMis0HBIXjR_zg.png" /><figcaption><a href="https://youtube.com/houssemdellai">https://youtube.com/houssemdellai</a></figcaption></figure><p>Take a look at <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Kubernetes Operator</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37317b15670" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup SonarQube in a Docker Container]]></title>
            <link>https://medium.com/@HoussemDellai/setup-sonarqube-in-a-docker-container-3c3908b624df?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c3908b624df</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[sonarcloud]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[sonarqube]]></category>
            <category><![CDATA[sonar]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Sat, 04 Jan 2020 09:52:37 GMT</pubDate>
            <atom:updated>2020-01-16T14:43:02.581Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s see how we can quickly setup a SonarQube environment using Docker container to run a code analysis for a .NET Core application.</p><ol><li>Run SonarQube on Docker</li><li>Install SonarScanner for .NET Core</li><li>Start the code analysis</li></ol><h4>1. Running SonarQube on Docker</h4><pre>$ <strong>docker run</strong> -d --name sonarqube -p 9000:9000 -p 9092:9092 <strong>sonarqube</strong></pre><h4>2. Installing SonarScanner for .NET Core</h4><p>Download SonarScanner from <a href="https://github.com/SonarSource/sonar-scanner-msbuild/releases/download/4.7.1.2311/sonar-scanner-msbuild-4.7.1.2311-netcoreapp2.0.zip">this link</a>. Extract it, then look for the file ‘SonarQube.Analysis.xml’ and add the following:</p><pre>&lt;SonarQubeAnalysisProperties       xmlns:xsi=&quot;<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>&quot; xmlns:xsd=&quot;<a href="http://www.w3.org/2001/XMLSchema">http://www.w3.org/2001/XMLSchema</a>&quot; xmlns=&quot;<a href="http://www.sonarsource.com/msbuild/integration/2015/1">http://www.sonarsource.com/msbuild/integration/2015/1</a>&quot;&gt;<br>&lt;Property Name=&quot;<strong>sonar.host.url</strong>&quot;&gt;<a href="http://localhost:9000/"><strong>http://localhost:9000</strong></a>&lt;/Property&gt;<br>&lt;Property Name=&quot;<strong>sonar.login</strong>&quot;&gt;<strong>[my-user-token]</strong>&lt;/Property&gt;<br>&lt;/SonarQubeAnalysisProperties&gt;</pre><p>Then, Add the path to the extracted SonarScanner folder to your PATH environment variable.</p><p>Run the command:</p><pre>$ dotnet tool install --global dotnet-sonarscanner --version 4.7.1</pre><p>More about this setup could be found here: <a href="https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/">https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/</a></p><h4>3. Starting the code analysis</h4><p>Let’s first clone a sample project fom Github to analyse it:</p><pre>$ git clone https://github.com/HoussemDellai/WebAppWithDatabaseDemo</pre><p>Then, from the current folder, we run:</p><pre>$ dotnet sonarscanner begin /k:&quot;project-key&quot; <br>$ dotnet build &lt;path to solution.sln&gt;<br>$ dotnet sonarscanner end</pre><h4>4. Running Sonar Scanner on a Container</h4><p>Instead of installing the CLI tools in the host machine, we can use a Docker container. The container here will start the code analysis using the CLI tools already installed inside. We just need to start the container, tell it the path to the source code and the url of sonarqube.The project for this container is open source and available here: <a href="https://github.com/newtmitch/docker-sonar-scanner">https://github.com/newtmitch/docker-sonar-scanner</a>. This is useful in CI pipeline because we won’t need to install additional dependencies into the build agent.</p><p>To use it we can run the following command:</p><pre>$ <strong>docker run</strong> -ti -v ${pwd}:/root/src — link sonarqube newtmitch/sonar-scanner:4 -D <strong>sonar.host.url=http://sonarqube:9000</strong> -D sonar.scm.provider=git -D <strong>sonar.project<br>BaseDir=/root/src</strong> -D sonar.sources=. -D sonar.projectName=”Web App with Database Demo”</pre><p>At the end, we should be able to go to localhost:9000 and see this nice generated dashboard with results of the code analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WiDDzDFsnr_yAHx8stT6TQ.png" /><figcaption>SonarQube dashboard</figcaption></figure><h4>5. SonarQube on Kubernetes</h4><p>We can also deploy SonarQube on Kubernetes. There is a Helm chart for that <a href="https://github.com/helm/charts/tree/master/stable/sonarqube">available here</a>.</p><h4>More resources</h4><p><a href="https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/">https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-msbuild/</a></p><p><a href="https://github.com/skilld-labs/sonar-scanner">https://github.com/skilld-labs/sonar-scanner</a></p><p>One of my videos on Sonar Cloud (the SaaS version of SonarQube):</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8dETmUcAl8w%3Fstart%3D1091%26feature%3Doembed%26start%3D1091&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8dETmUcAl8w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8dETmUcAl8w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbe7d94284ebe3365d2d8bf14ab09a10/href">https://medium.com/media/fbe7d94284ebe3365d2d8bf14ab09a10/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c3908b624df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run Selenium UI tests in Docker containers]]></title>
            <link>https://medium.com/@HoussemDellai/run-selenium-ui-tests-in-docker-containers-f41ae2796b8d?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f41ae2796b8d</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[ui-testing]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Mon, 30 Dec 2019 19:49:42 GMT</pubDate>
            <atom:updated>2019-12-31T15:07:51.473Z</atom:updated>
            <content:encoded><![CDATA[<p>Along with Unit and Integration Tests, UI tests are very important in the software development lifecycle. They can detect bugs and anomalies before going to Production. But, UI tests could be so expensive to setup: we need a full VM with all the dependencies. And sometimes we need more than one VM if we want to scale out to run tests in less time.</p><p><strong><em>What would be a light and easy solution to run UI tests ?</em></strong></p><h4>The short answer</h4><p>Instead of relying on VMs, we can launch UI Tests in Docker containers !</p><h4>The answer in a video</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fq-HQ_jDURyk%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dq-HQ_jDURyk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fq-HQ_jDURyk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/740becd0f73913833ac8779372e157cd/href">https://medium.com/media/740becd0f73913833ac8779372e157cd/href</a></iframe><p>Let’s see how that works through a demo.</p><p>We will take a sample .NET Core UI tests written using Selenium. Then, we will setup and deploy the Selenium containers to run the tests.</p><p>The principles still apply to other programming languages and frameworks.</p><h4>What is Selenium ?</h4><blockquote><a href="https://github.com/SeleniumHQ">Selenium</a> is an opensource framework for developing UI tests can run on Chrome, Firefox, IE and Edge.</blockquote><p>When we deploy an app, we want to test it manually. A tester will open the browser, paste the URL, type his login and password, click login button and check if the home page shows up correctly. Selenium comes with the promise to automate this process. A robot will do that instead of a human ! The following video explains it in details.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmxDG4rzfEOI%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmxDG4rzfEOI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmxDG4rzfEOI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfce629d8f4dec7bd8ca5553f5ae423c/href">https://medium.com/media/cfce629d8f4dec7bd8ca5553f5ae423c/href</a></iframe><h4>1. Setting up the UI tests and switching to RemoteWebDriver</h4><p>We typically use the <a href="https://selenium.dev/documentation/en/webdriver/">WebDriver</a> class in the code to configure parameters like the required browser, incognito mode, download directory, browser driver, etc… By default, WebDriver assumes that we want to run the UI tests inside the host machine. But, here, we want to run them inside a docker container. So, we need to connect to the container IP address. For that, we need to use the <a href="https://selenium.dev/documentation/en/remote_webdriver/">RemoteWebDriver</a> and specify the IP address and port number for the container (something like: http:localhost:4444/wd/hub).</p><pre><strong>var driver = new RemoteWebDriver(new Uri(&lt;RemoteUrl&gt;), options);</strong></pre><p>A sample UI test that opens a web page and checks that it contains a certain label would look like the following:</p><pre>public void PageLive_Test()<br>{<br>var options = new ChromeOptions();<br>var driver = new <strong>RemoteWebDriver</strong>(new Uri(&lt;RemoteUrl&gt;), options);<br>driver.Manage().Window.Maximize();<br>driver.Navigate().GoToUrl(&quot;https://www.youtube.com/houssemdellai&quot;);<br>Assert.IsTrue(_driver.PageSource.Contains(&quot;Houssem Dellai&quot;));<br>}</pre><h4>2. Setting up Selenium docker containers</h4><p>We can run Selenium on single/standalone container or on a Grid of multiple containers. Let’s focus in more details in each approach.</p><h4>2.1 Running Selenium in a standalone container</h4><p>This option means that we can use only one single container to launch all the tests. This container is <a href="https://hub.docker.com/r/selenium/standalone-chrome">selenium/standalone-chrome</a> or <a href="https://hub.docker.com/r/selenium/standalone-firefox">selenium/standalone-firefox</a>. We are limited to only the resources available for one container and we should handle targeting Chrome or Firefox browsers. This option is the easiest, still good for small projects but it doesn’t scale.</p><p>To run the above test, we just need to run the container exposed on port 4444 which is the default port that connects to the web driver:</p><pre><strong>$ docker run</strong> -d <strong>-p 4444:4444</strong> -v /dev/shm:/dev/shm   <br>             <strong>selenium/standalone-chrome</strong>:3.141.59-yttrium</pre><p>Then we configure the RemoteWebDriver’s URL:</p><pre>var remoteUrl = &quot;http://localhost:4444/wd/hub&quot;;<br>var driver = new <strong>RemoteWebDriver</strong>(new Uri(remoteUrl), options);</pre><p>And we run the tests using the .NET Core CLI from within the project folder:</p><pre><strong>$ dotnet test</strong> </pre><p>At this point, we should see the results about the tests failure or success in the console.</p><blockquote>To be able to visually view what is happening inside the container, we could use an image that have VNC preinstalled: <em>selenium/standalone-chrome-debug</em>.</blockquote><h4>2.2 Running Selenium in a Grid of containers</h4><p>If the first option doesn’t scale, this one solves this problem. We can create a <a href="https://selenium.dev/documentation/en/grid/">Grid of multiple and different containers</a>, something like 10 instances of Chrome v79, 20 instances of Chrome v60, 30 instances of Firefox latest version. As a result, we can run tests in parallel against different versions of different browsers. They will be controlled by selenium/hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*YW4jY7dUxgNnpGou.png" /><figcaption>Selenium Grid architecture</figcaption></figure><blockquote>Selenium could be used to run not only web UI tests, but, also mobile UI tests.</blockquote><p>The Hub becomes the central part where we should connect to. It will then dispatch the requests on its nodes depending on the requirements for each test.</p><h4>2.2.1 Setting up Selenium Grid using Docker Compose</h4><p>Because the Grid is a composition of multiple docker images, it is easy to configure it using Docker Compose:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fef1e13c3311111049e8e17ee2a67b57/href">https://medium.com/media/fef1e13c3311111049e8e17ee2a67b57/href</a></iframe><p>Then we need to run the docker compose command:</p><pre><strong>$ docker-compose up</strong></pre><p>This will spin up the Grid. Now, we can connect to it through the <em>remoteUrl and run the tests as in the previous section (2.1).</em></p><blockquote>We are using <a href="https://github.com/HoussemDellai/SeleniumTestsOnDocker/blob/master/.runsettings"><em>.runsettings</em></a> file in order to change the remoteUrl through environment variable without recompiling the app.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dha8nOWKxLQH-or330HlPA.png" /><figcaption>Test runned inside Selenium Grid containers</figcaption></figure><p>We can view the Grid dashboard showing the different connected nodes, as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQh-T6VOHqu_e47xdenDRg.png" /><figcaption>Selenium Grid console showing connected browser nodes</figcaption></figure><h4>2.2.2 Setting up Selenium Grid in Kubernetes</h4><p>Docker compose is useful for development in local machine. But, when it comes to releasing the system into production, then we will typically need Kubernetes. What was achieved with docker compose is also achievable with k8s manifest files. So, we create the Deployments for the Hub, Chrome and Firefox, along with the Service that will point to the Hub.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4793a3bc57c0968c097d6c6b10fb8cc7/href">https://medium.com/media/4793a3bc57c0968c097d6c6b10fb8cc7/href</a></iframe><blockquote>We’ve exposed the Service type as LoadBalancer to get a public IP address.</blockquote><p>We need to connect to a Kubernetes cluster and run the command:</p><pre><strong>$ kubectl apply -f </strong>selenium-k8s-deploy-svc.yaml</pre><p>To get the IP address for the service, we need to run : <em>kubectl get svc</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OuBGRSedF0d_sNWcXRVdFw.png" /></figure><p>Then, again, we should edit the remoteUrl to take the external IP address of the Hub. And finally we can run the tests with the same <em>dotnet test</em> command.</p><h4>3. How to make it even better</h4><p>There is always different ways to make things even better. For example, here we are still launching the UI tests from the local machine, even if they will run on a remote web driver on a container. So, it would be interesting to run it all in containers. We can also configure the Horizontal Pod Autoscaling in Kubernetes to scale automatically the number of replicas.</p><h4>More resources</h4><p>Source code: <a href="https://github.com/HoussemDellai/SeleniumTestsOnDocker">https://github.com/HoussemDellai/SeleniumTestsOnDocker</a></p><p>Sources: <a href="https://github.com/SeleniumHQ/docker-selenium">https://github.com/SeleniumHQ/docker-selenium</a></p><p>My youtube channel: <a href="https://www.youtube.com/houssemdellai">https://www.youtube.com/houssemdellai</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHz8LLA2M3wJHspTSU6FWw.png" /><figcaption>Videos about Kubernetes, Docker &amp; DevOps</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f41ae2796b8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring with Prometheus and Grafana in Kubernetes]]></title>
            <link>https://faun.pub/monitoring-with-prometheus-and-grafana-in-kubernetes-42727866562c?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/42727866562c</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Sat, 09 Nov 2019 07:40:43 GMT</pubDate>
            <atom:updated>2019-11-11T14:27:34.890Z</atom:updated>
            <content:encoded><![CDATA[<p>This workshop will get you started with installing Prometheus and Grafana into Kubernetes to monitor applications to view metrics. We’ll go through these 3 steps:</p><ol><li>Export metrics for Prometheus from an ASP.NET Core app</li><li>Install Prometheus into Kubernetes using Helm charts</li><li>Add Grafana dashboard to view metrics</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE7-HuslXKj1MB10ncorfzEGa&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbErGEHf6GCc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbErGEHf6GCc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e749c5f4b06f292670dd03b50d67495/href">https://medium.com/media/4e749c5f4b06f292670dd03b50d67495/href</a></iframe><h4>Setting up the lab</h4><p>You will need to have the following to start the lab:</p><ul><li>Kubernetes cluster: AKS/GKE/EKS… or <a href="https://github.com/kubernetes/minikube">Minikube</a> would be enough !</li><li>Helm CLI tool installed (<a href="https://helm.sh/docs/using_helm/#quickstart">guide</a>)</li><li>Dotnet CLI tool installed (<a href="https://docs.microsoft.com/en-us/dotnet/core/tools/">here</a>)</li></ul><h4>1. Exporting metrics for Prometheus from an ASP.NET Core app</h4><p>We want to configure the app to start exporting metrics. These metrics will be scraped later by Prometheus. We’ll create a new ASP.NET Core app and we’ll enable exporting predefined metrics. Then we’ll create a custom metric.</p><h4>1.1 Exporting predefined metrics</h4><p>Let’s start by creating a simple ASP.NET Core app. We can use Visual Studio or VS Code as we’ll do here. Then, we need to install the <a href="https://www.nuget.org/packages/prometheus-net/">Prometheus nuget package for .net core</a>. This package was developed by the community and not by Prometheus dev team.</p><blockquote>Extensions/packages are also available for Java, Python, Javascript, Go… Each extension/package defines different metrics.</blockquote><pre>$ mkdir WebApp # create a new directory called WebApp<br>$ <strong>dotnet new mvc </strong># create a new ASP.NET Core MVC project<br>$ dotnet <strong>add package</strong> <strong>prometheus-net </strong># install prometheus nuget pckg<br>$ dotnet add package <strong>prometheus-net.AspNetCore</strong></pre><p>The package is already configured to export some pre-defined metrics. We just need to tell our app to expose these on a public endpoint. From Startup.cs class, at the beginning of the Configure method, add the following code:</p><pre><strong>app.UseMetricServer(url: &quot;/metrics&quot;); </strong>// enable metrics in C#</pre><p>The app is now ready to export metrics. Lets run it using this command:</p><pre><strong>$ dotnet run </strong># restore, build and run the ASP.NET Core project</pre><p>Navigate to the web app to view the metrics <a href="https://localhost:5001/metrics">https://localhost:5001/metrics</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jyTM5jmJ3T2lyInNKUPUpA.png" /><figcaption>Prometheus metrics exposed on a public endpoint</figcaption></figure><h4>1.2 Creating custom Prometheus metrics</h4><p>In addition to the predefined metrics, we can also create custom metrics. But how to do that ? The Metrics object can create different <a href="https://prometheus.io/docs/concepts/metric_types/">metric types</a>: Counter, Gauge, Histogram and Summary. Lets create a counter to count the number of times users open the index page inside the HomeController. We’ll create the counter called <strong>my_counter</strong> as an attribute in the controller. Then we’ll increment its value wihin the Index method.</p><pre>Counter counter =<strong> Metrics.CreateCounter(&quot;my_counter&quot;,</strong>&quot;Metrics counter&quot;<strong>);<br>counter.Inc(); </strong>// Increment the counter</pre><p>Run again the app, navigate to the Index page to generate the metric and then go to the metrics endpoint. We’ll see our metric publicly exposed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V7-myckKbmToJhcVkqjuDQ.png" /><figcaption>Prometheus custom metric displaying number of page views</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE7-HuslXKj1MB10ncorfzEGa&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEAQOr8FAs44&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEAQOr8FAs44%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5be9471ff92702c1a5c22b09d9c6c662/href">https://medium.com/media/5be9471ff92702c1a5c22b09d9c6c662/href</a></iframe><h4>1.3 Deploying the app into a Docker container in Kubernetes</h4><p>Until now, the app is not running in a Docker container! To dockerize it, we need a Dockerfile to create a container, we’ll use the one <a href="https://github.com/HoussemDellai/WebAppMonitoring/blob/master/WebApp/Dockerfile">available here</a>. Then we need the Kubernetes YAML manifest file for creating a Deployment and a Service, we’ll use <a href="https://github.com/HoussemDellai/WebAppMonitoring/blob/master/web-deploy-svc.yaml">this one</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE7-HuslXKj1MB10ncorfzEGa&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEC92qSp8sx8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEC92qSp8sx8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/18d0797ace49164aa38c9eee88f06cb8/href">https://medium.com/media/18d0797ace49164aa38c9eee88f06cb8/href</a></iframe><p>In this first part, we’ve exported the metrics on a public endpoint. Next, we’ll configure Prometheus to scrape these metrics.</p><h4>2. Install Prometheus into Kubernetes using Helm charts</h4><p>We’ll install Prometheus into Kubernetes using this <a href="https://github.com/helm/charts/tree/master/stable/prometheus">Helm chart</a>. We need to name the release and customise some values in order to publicly expose Prometheus server dashboard through a LoadBalancer. A better solution could be using port forwarding. Then we disable RBAC for simplicity, otherwise you need to create Role and RoleBinding.</p><pre><strong>$ helm install stable/prometheus --name my-prometheus <br>  --set server.service.type=LoadBalancer --set rbac.create=false</strong></pre><p>In a few seconds, Prometheus will be deployed and the Load Balancer will create the public IP address. Use the following command to get the external IP:</p><pre><strong>$ kubectl get services</strong></pre><p>Open that IP address in a new browser window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNAdhXwOupQxF1hdhQtkGw.png" /><figcaption>Prometheus dashboard</figcaption></figure><p>With this basic install, Prometheus will start scraping data from the k8s cluster which have default configuration to export metrics. From the dashboard, you can explore those metrics on a graph.</p><h4>2.2. Configure Prometheus to scrape metrics from the app</h4><p>We need to configure Prometheus to scrape metrics from another source: our app. Prometheus uses a yaml configuration file to manage alerts and rules. This file is “prometheus.yml”. We can edit that file either directly inside the container or through the Helm chart values. We’ll continue with the latter option as it is better for documentation and versioning.</p><p>Helm deploys charts with default configuration described in <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml">values.yaml</a>. We can override those values on the command arguments. That is what we had already done with “<em>— set server.service.type=LoadBalancer</em>”. Actually, <em>server.service.type</em> is a variable created inside <em>values.yaml</em> file and used in the chart. Helm will assign its value on deployment.</p><p>If you take a look at the content of <a href="https://github.com/helm/charts/blob/master/stable/prometheus/templates">Prometheus Helm template</a>, you won’t find <em>prometheus.yml</em>. Helm uses a container image that had a <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/">default config</a> inside. Then Helm can override its content.</p><p>But, how ?</p><p>It is interesting to understand this. Helm defines a <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L1041">variable</a> that will be used to set the content for prometheus.yml (and also for the <a href="https://github.com/helm/charts/blob/master/stable/prometheus/templates/server-configmap.yaml">server-configmap.yaml</a>). This means we can setup indirectly the content of prometheus.yml through values.yaml.</p><p>We can override <em>values.yaml</em> with another file. The new one is a copy, but with edited values. Create a new file as copy of <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml">values.yaml</a> and name it <em>prometheus.values.yaml.</em> Then override the values in <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L1046"><em>scrape_configs</em></a><em> </em>section to add a new target which is the name of your app’s Kubernetes Service:</p><pre><strong>scrape_configs</strong>:<br>- job_name: prometheus<br>  <strong>static_configs</strong>:<br>  - <strong>targets</strong>:<br>    - localhost:9090<br>    - webapp-service # add your app’s k8S Service name to be scraped</pre><p>Now, let’s deploy the new configuration using Helm:</p><pre><strong>$ helm upgrade</strong> my-prometheus stable/prometheus <br>  --set server.service.type=LoadBalancer --set rbac.create=false <br>  <strong>-f prometheus.values.yaml</strong></pre><p>Then, we should check the values of the Prometheus server ConfigMap. First, we need to get its name :</p><pre><strong>$ kubectl get configmaps<br>$ kubectl describe configmap &lt;your-prometheus-server-configmap&gt;</strong></pre><p>It should show the updated targets with both localhost and app’s service name.</p><p>In addition to that, we can also check for the new targets in the prometheus.yml :</p><pre><strong>$ kubectl exec</strong> -it &lt;your-prometheus-server&gt; <br>  -c prometheus-server -- <strong>cat /etc/config/prometheus.yml</strong></pre><p>Now go to refresh the Prometheus dashboard and expand the list of metrics to see our custom metric counter <em>my_counter</em> scraped from the app !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WVxQIJ87ccTbp390mprMcg.png" /><figcaption>Prometheus scraped a custom metric from the app</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE7-HuslXKj1MB10ncorfzEGa&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhuR8NpYNBrc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhuR8NpYNBrc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b44961f8f456c5c3f9c9d191d80ed9c/href">https://medium.com/media/7b44961f8f456c5c3f9c9d191d80ed9c/href</a></iframe><p>Now that we have the metrics data in Prometheus, we want to create a pretty nice dashboard to view these metrics live. Prometheus can create graphs that are so simple and minimalistic. But, we are looking for a more powerful tool that can create nice looking graphs that make you proud to share with your team :) . Fortunately, this tool exists, it is called Grafana.</p><h4><strong>3. Add Grafana dashboard to view metrics</strong></h4><p>We’ll install <a href="https://grafana.com/">Grafana</a> in Kubernetes cluster using <a href="https://github.com/helm/charts/tree/master/stable/grafana">Helm charts</a>, as we had done with Prometheus. We’ll expose publicly the dashboard, disable RBAC and enable data persistence.</p><pre><strong>$ helm install</strong> --name my-grafana <strong>stable/grafana</strong> <br>  --set rbac.create=false --set service.type=LoadBalancer <br>  --set persistence.enabled=true</pre><p>In a few seconds, once the deployment completes, follow the instructions in the output to get the password for the dashboard.</p><pre><strong>$ kubectl get secret</strong> --namespace default my-grafana -o jsonpath=&quot; <br>  {<strong>.data.admin-password</strong>}&quot; | base64 --decode ; echo</pre><p>And we should be able to get the external IP adress for Grafana:</p><pre><strong>$ kubectl get services</strong></pre><p>Open the IP adress in a new browser window and login with the default created user login (admin) and the retrieved password. We’ll get a web page with no dashboard configured. In the next section, we’ll create one.</p><h4>3.1. Import Grafana dashboard</h4><p>Grafana offers lots of <a href="https://grafana.com/grafana/dashboards">official and community dashboards</a>. These dashboards are pre-configured to view metrics coming many systems like Prometheus, but also Elasticsearch, Stackdriver, CloudWatch, Azure Monitor… Click on “Add data source” on Grafana web app, then select Prometheus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ely417h5No76CgZJv_egxg.png" /></figure><p>After that, we need to add the URL for the metrics coming from Prometheus. Here, we’ll use the Prometheus Service name. We could had used the IP address but it is a better practise to use service names in Kubernetes to not worry about the case where IP address might change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ANuQfqZsGM4O5J0gKukZw.png" /><figcaption>Adding the URL to Prometheus Service name</figcaption></figure><p>Validate this config, if everything well setup, we’ll get a success notification.</p><p>At this point, we can create a dashboard, or to make things easier, let’s import an existing one. To do that, go to Dashboards -&gt; Manage menu item and click Import to add the dashboard with ID: <strong>1621</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uiCQdM91PsMYbmQn4OCsVA.png" /></figure><p>Click Load and in the next page select Prometheus from the drop down list. Immediately, we’ll see our beautiful dashboard :))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QyJVZopsoC2GYPcHxtge8A.png" /><figcaption>Grafana dashboard viewing data from Prometheus</figcaption></figure><p>Note all the metrics we get from Prometheus like cpu, memory and network usage for both Kubernetes cluster nodes and pods.</p><h4>3.2. Add view for the custom metric counter</h4><p>The dashboard wes pre-configures to view certain <em>standard</em> metrics from k8s cluster. That is why we don’t see the custom metric we created in the app <em>my_counter. </em>Let’s add it. We can customize the imported dashboards by adding another panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iAXCTV_kb2RZhWTgl0Wg-w.png" /></figure><p>Then click <em>Add Query.</em> A new window will show up, enter the name of the metric (my_counter). After that, select Visualisation and choose the graph format you want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSsaJ0QRV-eoePCbLh6Ekw.png" /><figcaption>Grafana offers multiple graph types</figcaption></figure><p>By coming back to the dashboard, we’ll notice that a new panel was added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2d4oBS13d1H-q8t5agJ3Wg.png" /><figcaption>Grafana panel showing the metrics counter (my_counter)</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE7-HuslXKj1MB10ncorfzEGa&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DgDX3HtoRta4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FgDX3HtoRta4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/73c647f6b05ad311580b2280546f2fb1/href">https://medium.com/media/73c647f6b05ad311580b2280546f2fb1/href</a></iframe><p>Great! In a few steps, we added a custom metric into the app, configured Prometheus for scraping metrics and Grafana for visualisation.</p><p>I hope you like it :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook Group</strong></a><strong> </strong>💬<strong>.</strong></p><p><strong>To join our community Slack </strong>🗣️ <strong>and read our weekly Faun topics </strong>🗞️,<strong> click here⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oSdFkACJxs5iy1oR" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42727866562c" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/monitoring-with-prometheus-and-grafana-in-kubernetes-42727866562c">Monitoring with Prometheus and Grafana in Kubernetes</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating page templates for Xamarin Forms]]></title>
            <link>https://medium.com/@HoussemDellai/creating-page-templates-for-xamarin-forms-ec8cbc336d25?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec8cbc336d25</guid>
            <category><![CDATA[template]]></category>
            <category><![CDATA[xamarin-forms]]></category>
            <category><![CDATA[design]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Mon, 19 Aug 2019 17:24:36 GMT</pubDate>
            <atom:updated>2019-08-19T18:11:06.883Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem</h3><p>Templates are a good solution for reusing pages’ design. In fact, when designing pages, we have a specific code dedicated for the design. Think of the page layout, colors, font sizes, title style, margins, etc… In a traditional approach, we need to rewrite this design’s code for every component that uses it. This results in code duplication, thus less maintainable code!</p><h3>The solution</h3><p>Fortunately, as we could refactor and reuse code for business logic, we could apply the same principal for UI code. This what is known under the name Application or Page Template. This approach could be used in almost all UI frameworks and platforms. In this article, we’ll be focusing on Xamarin Forms.</p><h3>The implementation</h3><p>In Xamarin Forms, the code consists of a UI component that defines the design of a page or any kind of view. This is the template. Then, we need to apply this template into our pages or views through a specific syntax.</p><p>Let’s first create a sample template. We’ll define one for page layout. We want to have title label at the top of the page with font size 22. Then we want to have a body section that defines the main content for the page. Finally, at the bottom, we want to have a footer section where we can put views like action buttons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1K2aT_sFafciZ_DdqkToQ.png" /></figure><p>In Visual Studio, create/open a new project, add a new page or component, call it MainTemplate and copy paste the following XAML code in MainTemplate.xaml. Note here the three sections. Note also that the title section is a Label so we can only change its text property. While the body and footer are ContentView so we can change their content to put Grid, StackLayout, Button...</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?&gt;<br>&lt;ContentView xmlns=&quot;http://xamarin.com/schemas/2014/forms&quot;<br>             xmlns:x=&quot;http://schemas.microsoft.com/winfx/2009/xaml&quot;<br>             x:Class=&quot;AppTemplate.MainTemplate&quot;&gt;<br>    &lt;ContentView.Content&gt;<br>        &lt;StackLayout Padding=&quot;20&quot; Spacing=&quot;20&quot;&gt;<br>            &lt;Label x:Name=&quot;<strong>HeaderLabel</strong>&quot; FontSize=&quot;22&quot; /&gt;<br>            &lt;ContentView x:Name=&quot;<strong>BodyContent</strong>&quot;     <br>                         VerticalOptions=&quot;FillAndExpand&quot;/&gt;<br>            &lt;ContentView x:Name=&quot;<strong>FooterContent</strong>&quot;/&gt;      <br>         &lt;/StackLayout&gt;<br>    &lt;/ContentView.Content&gt;<br>&lt;/ContentView&gt;</pre><p>To be able to access and change the content of title, body and footer, we need to expose these UI components through public properties in code behind. We create three public properties in MainTemplate.xaml.cs as following:</p><pre>[ContentProperty(&quot;Child&quot;)]<br>public partial class MainTemplate : ContentView<br>{<br>    public MainTemplate()<br>    {<br>        InitializeComponent();<br>    }<br>    public string <strong>PageTitle</strong><br>    {<br>        get =&gt; HeaderLabel.Text;<br>        set =&gt; HeaderLabel.Text = value;<br>    }<br>    public View <strong>Body</strong><br>    {<br>        get =&gt; BodyContent.Content;<br>        set =&gt; BodyContent.Content = value;<br>    }<br>    public View <strong>Footer</strong><br>    {<br>        get =&gt; FooterContent.Content;<br>        set =&gt; FooterContent.Content = value;<br>    }<br>}</pre><p>Now, we have created the template and we can start applying it to a page. To do that, we’ll apply it to the MainPage.</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?&gt;<br>&lt;ContentPage xmlns=&quot;http://xamarin.com/schemas/2014/forms&quot;<br>             xmlns:x=&quot;http://schemas.microsoft.com/winfx/2009/xaml&quot;<br>             xmlns:appTemplate=&quot;clr-namespace:AppTemplate&quot;<br>             x:Class=&quot;AppTemplate.MainPage&quot;<br>    &lt;appTemplate:MainTemplate <strong>PageTitle</strong>=&quot;Login page&quot;&gt;   <br>        &lt;appTemplate:MainTemplate.<strong>Body</strong>&gt;<br>            &lt;StackLayout&gt;<br>                &lt;Entry Placeholder=&quot;email&quot; /&gt;<br>                &lt;Entry Placeholder=&quot;password&quot; /&gt;<br>            &lt;/StackLayout&gt;<br>        &lt;/appTemplate:MainTemplate.Body&gt;<br>        &lt;appTemplate:MainTemplate.<strong>Footer</strong>&gt;<br>            &lt;StackLayout Orientation=&quot;Horizontal&quot;&gt;<br>                &lt;Button Text=&quot;Cancel&quot; /&gt;<br>                &lt;Button Text=&quot;Login&quot; /&gt;<br>            &lt;/StackLayout&gt;<br>        &lt;/appTemplate:MainTemplate.Footer&gt;<br>    &lt;/appTemplate:MainTemplate&gt;<br>&lt;/ContentPage&gt;</pre><p>Note here how we are setting the title Label with text, body and footer content with StackLayout.</p><blockquote>Application source code is available on <a href="https://github.com/HoussemDellai/Xamarin-Forms-Template">Github</a>.</blockquote><h3>Conclusion</h3><p>Now, we can re-use the same template design for multiple pages without redefining the title, body and footer. This makes it really easy to have a unique design and unique ‘look and feel’ for the entire application.</p><h3>More resources</h3><p>If you are looking for more resources about Xamarin or anything Cloud, DevOps and Kubernetes, I’m pretty sure you will find useful content on my youtube channel: <a href="https://youtube.com/houssemdellai">https://youtube.com/houssemdellai</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HkO8RhlHy0ET6hcoLNX2GQ.png" /><figcaption><a href="https://youtube.com/houssemdellai">youtube.com/houssemdellai</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec8cbc336d25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Change by listening to inspiring friends]]></title>
            <link>https://medium.com/@HoussemDellai/change-by-listening-to-inspiring-friends-8cd40997ba27?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cd40997ba27</guid>
            <category><![CDATA[change]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[career-paths]]></category>
            <category><![CDATA[changemaker]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Sun, 31 Mar 2019 07:58:06 GMT</pubDate>
            <atom:updated>2019-04-01T09:23:26.113Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Do you think that change is hard? Here is what could make it easy and natural. My short answer is “listen to people around you!”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3au4JSV2zABCwoVfgnyvSQ.png" /></figure><p>Next, I’ll share my own experience and explain how my friends inspired me to achieve three major career changes in few years.</p><p><strong>First career change</strong></p><p>The first career change started when I was a software engineering student, 8 years ago. My university used to focus heavily on technologies like Java and Oracle. However, one of my friends had a different experience. Indeed, he studied in a different university and used different technologies. Ones I hardly heard of before: .NET/C# and SQL Server. Both of us were working on almost the same project, each using what he learned. My work was satisfactory, nevertheless, he showed me what he had done. His application’s user interface was way so much better than mine. In addition to that, he had done the work in half the time it took me! I was inspired and I kept that in mind.</p><p>A few months later, I joined a university group to learn about these new technologies. The group’s leader was really inspiring and seemed mastering the subject. Even though he didn’t study it. Again, I was inspired. But this time, the inspiration was enough that just three months later, I found myself leading the group! Not only I was learning these new technologies, but also, I was teaching other students!</p><p>What happened here is just amazing. I just saw what could be done with that technology. And I felt myself surrounded by people who seemed to me smart. Thus, I had made the move. I decided to invest my free time learning technologies I did not need to graduate. That is because I was inspired.</p><p><strong>Second career change</strong></p><p>I worked for Microsoft to develop apps for Windows Phone and Windows to defend Android and iOS. I had spent two years working intensively on that technology, until there were rumours these technologies were going to be deprecated. I strongly believed a tech giant like Microsoft would never let that happen. I thought that would happen only to Silverlight, and this time it is different. I hadn’t enough experience to expect what would happen.</p><p>I was proudly going on the wrong direction, until I was inspired to go in a different one. My surrounded friends started telling me and arguing again and again these technologies will die. They didn’t suggest any other technology, but to leave the ones I was using. I had no choice but to change. I didn’t like to lose all what I learned in C#/.NET. So, I decided I’m will invest in learning a new technology: Xamarin which uses C#/.NET. What was different here, is that the change was not triggered by inspiration, but by fear and force. I was surrounded by people who pushed me away from the wrong direction.</p><p><strong>Third career change</strong></p><p>My third career change was the most magnificent and the most complicated. I was a Xamarin consultant for more than three years. I was investing all my time on that platform so much that I had created more than <a href="https://www.youtube.com/HoussemDellai">200 videos on Youtube</a> with 15,000 subscribers. Gladly, my channel was the preferred for Xamarin beginners. The platform itself seemed really good. But it was not evolving fast enough as its competitors, take a look at Flutter. It seemed wise to me to leave it and move to something else, before it dies. But this was not enough to make the decision.</p><p>I was still using it, until my friends came to rescue.</p><p>At that time, I was surrounded by three friends who were working on DevOps, Docker and Kubernetes. I always was a developer and I hated the command line. My friends loved what they were doing. They have authored 4 books on these topics. So, my friends were inspiring me, again, at the right time! Despite I didn’t know these technologies before, and even considered them as not interesting at all, I decided to make the next change. That is just because I was close to inspiring people.</p><p><strong>Conclusion</strong></p><p>What I have learned from the previous experiences, in one sentence, is:</p><blockquote>Make inspiring friends that could influence you to change and achieve your target.</blockquote><p>If you are interested in learning what I’m already doing now, you will be happy to check my videos on youtube about things DevOps, Docker and Kubernetes.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE79s_9ZFNQpSRl8F0iSVhcM2&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dje5WRKxOkWQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fje5WRKxOkWQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/536fe5158ec01db8d94c24e2b3342b94/href">https://medium.com/media/536fe5158ec01db8d94c24e2b3342b94/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cd40997ba27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playing with AKS & AAD]]></title>
            <link>https://faun.pub/playing-with-aks-aad-568bf7bc5ae8?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/568bf7bc5ae8</guid>
            <category><![CDATA[ak]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure-active-directory]]></category>
            <category><![CDATA[rbac]]></category>
            <category><![CDATA[roles]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Sat, 02 Mar 2019 23:06:14 GMT</pubDate>
            <atom:updated>2019-07-19T16:40:22.633Z</atom:updated>
            <content:encoded><![CDATA[<p>This quick workshop will walk you through configuring AKS with AAD to use RBAC to authenticate users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E8GniSqDM48IKH2R7sztGQ.jpeg" /></figure><h4>Introduction</h4><p>The development team members needs to access Kubernetes cluster to deploy apps and configure the cluster. But, not all members needs the same level of access rights. It is even a best practice or a must have to grant members different access levels. For example, developers can get, create and delete deployments inside their own namespace. And, Ops guys can configure the cluster to add or remove users, scale the cluster or upgrade it. This will secure the k8s cluster and reduce the risk of bad manipulations.</p><p>You will need an Azure subscription, az and kubectl cli installed.</p><p>This workshop will start by creating Azure AD client and server apps. Then we’ll create an AKS cluster configured with AAD. And we’ll finish by creating the Role and RoleBinding k8s manifest files.</p><h4>1. Create Azure AD server and client apps</h4><p>We need to create a server and client apps, those will be used to authenticate the users connecting to AKS through AAD. This means the user should be first created in AAD to have access to the cluster. This is the authentication part. For the authorisation part, it will be managed by Role and RoleBinding k8s objects in section 3.</p><p>Please follow the steps described in the link here (just before section Create Cluster) to create client and server apps: <a href="https://docs.microsoft.com/en-us/azure/aks/aad-integration">https://docs.microsoft.com/en-us/azure/aks/aad-integration</a></p><p>At the end, we should have the AAD tenant Id, client app Id, server app Id and server app secret. We’ll use those to configure AKS with AAD.</p><h4>2. Create AKS cluster configured with Azure Active Directory</h4><p>Azure uses Resource Group to group resources related to the same application or service. All resources in Azure must live inside a resource group, including AKS. Let’s start by creating a resource group using the az command line.</p><pre><strong>$ az group create -name </strong>aks-aad-rg <strong>-location </strong>westeurope</pre><blockquote>If you have multiple Azure subscriptions, you can set a default one so that az commands points always to it, by using:</blockquote><blockquote><strong><em>az account set “Your_Azure_Subscription_Id_Or_Name”</em></strong></blockquote><p>Then we can create the AKS cluster. Kubernetes uses OpenId Connect (OIDC) to authenticate users. And, AKS could be configured to use AAD as an implementation of OIDC. The az cli tool provide the options to make this configuration during (but not after) the creation of the cluster. Make sure to replace the values from your AAD. This will take about 5 minutes to run.</p><pre><strong>$ az aks create <br>     -resource-group </strong>aks-aad-rg \<strong><br>     -name </strong>aks-aad \<strong><br>     -generate-ssh-keys </strong>\<strong><br>     -aad-server-app-id </strong>cbb2efcb-9b3c-4441-b58f-9c6cca37 \<strong><br>     -aad-server-app-secret </strong>VBnbw1gEhxtRVfKTv8dYD+MyraLF5jn= \<strong><br>     -aad-client-app-id </strong>4c2a05a5–7ed1–4a2c-b4aa-b78bc \<strong><br>     -aad-tenant-id </strong>72f988bf-0000–0000–0000–2d7cd011db47</pre><p>Now, all users from the AAD account can authenticate to the AKS cluster. But, they can not do any operation as they don’t have yet any permission to do whatever. We need to grant them access to k8s resources through Role and RoleBinding. Before that, we need to access the cluster as admin, in order to assign the roles.</p><h4><strong>3. Connect to AKS cluster</strong></h4><p>We need to connect to AKS in order to run kubectl commands against the new cluster, as an admin.</p><pre><strong>$ az aks get-credentials \<br>     -resource-group </strong>aks-aad-rg \<strong><br>     -name </strong>aks-aad<br>  <strong>   --admin<br></strong>Merged &quot;aks-aad-admin&quot; as current context in /Users/houssem/.kube/config</pre><p>By the way, the source code for all the commands are here: <a href="https://github.com/HoussemDellai/rbac-aks-aad">github.com/HoussemDellai/rbac-aks-aad</a></p><p>And this workshop is available as a video.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FsKwe1jsllt4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsKwe1jsllt4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FsKwe1jsllt4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f2a90c09e6c3bc385326b805f726488/href">https://medium.com/media/8f2a90c09e6c3bc385326b805f726488/href</a></iframe><h4>4. Create the Role and RoleBinding</h4><p>Let’s create the Role which will define access to certain resources. For example, only reading information from pods.</p><pre><strong>kind: </strong>Role<br><strong>apiVersion: </strong>rbac.authorization.k8s.io/v1<br><strong>metadata:<br>  namespace: </strong>default<br>  <strong>name: </strong>pod-reader<br><strong>rules:<br>- apiGroups: </strong>[“”] # “” indicates the core API group<br>  <strong>resources: </strong>[“pods”]<br>  <strong>verbs: </strong>[“get”, “watch”, “list”]</pre><p>To assign the role to a user, we need to use <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RoleBinding</a>.</p><pre><strong>kind: </strong>RoleBinding<br><strong>apiVersion: </strong>rbac.authorization.k8s.io/v1<br><strong>metadata:<br>  name: </strong>read-pods<br>  <strong>namespace: </strong>default<br><strong>subjects:<br>- kind: </strong>User<br>  <strong>name: </strong>&quot;houssem.dellai@live.com&quot; # Name is case sensitive<br>  <strong>apiGroup: </strong>rbac.authorization.k8s.io<br><strong>roleRef:<br>  kind: </strong>Role #this must be Role or ClusterRole<br>  <strong>name: </strong>pod-reader # must match the name of the Role<br>  <strong>apiGroup: </strong>rbac.authorization.k8s.io</pre><blockquote>We can use a Group instead of User, to bind to all users within the AAD group.</blockquote><p>Then we deploy both files using kubectl:</p><pre><strong>$ kubectl apply -f role.yaml<br></strong>role.rbac.authorization.k8s.io/pod-reader created<br><strong>$ kubectl apply -f role-binding.yaml<br></strong>rolebinding.rbac.authorization.k8s.io/read-pods created</pre><p>We can check if the Role and RoleBinding were successfully created:</p><pre><strong>$ kubectl get roles<br></strong>NAME       AGE<br>pod-reader 2m<br><strong>$ kubectl get rolebindings<br></strong>NAME        AGE<br>read-pods   2m</pre><h4>5. Test authentication and authorisation to AKS</h4><p>Now that everything is configured, we will pretend to be the user used in RoleBinding. We’ll connect to the AKS cluster from another machine. This is using login to Azure, not to to the cluster.</p><pre><strong>$ az aks get-credentials \<br>     -resource-group </strong>aks-aad-rg \<strong><br>     -name </strong>aks-aad<strong><br></strong>Merged &quot;aks-aad&quot; as current context in /Users/houssem/.kube/config</pre><p>Then we’ll try the allowed command:</p><pre><strong>$ kubectl get pods<br></strong>To sign in, use a web browser to open the page <a href="https://microsoft.com/devicelogin">https://microsoft.com/devicelogin</a> and enter the code CL7DWV3P2 to authenticate.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2Sw909o7eygC3KgRkR5hA.png" /></figure><p>Because we added authentication, it is asking to login to the cluster. We’ll login as the user specified in RoleBinding. We need to launch the browser in the specified url address and use the given code.</p><p>Then we press continue.</p><blockquote>Make sure the user used in RoleBinding exists also in the Azure AD.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QX4-dMd-sLpQO-1qYXpt4g.png" /></figure><p>And here we come to the most exciting part, the login to the AKS cluster!</p><p>We can login with the email and password for the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58zne_7trFCnwWtE_uSXxg.png" /></figure><p>This web page means that you are logged in successfully !</p><p>We run again the allowed command:</p><pre><strong>$ kubectl get pods</strong><br>No resources found. # we didn&#39;t created a Pod yet !</pre><p>If we try to run one of the non allowed operations, then it will fail!</p><pre><strong>$ kubectl get deployments</strong><br>Error from server (Forbidden): deployments.extensions is forbidden: User “f97fc9ba-ebda-439e-825a-..” cannot list deployments.extensions in the namespace “default”</pre><p>If we try to get the pods from all the namespaces, that will also fail. Because we configured the user in Role to get pods from only the default namespace.</p><pre><strong>$ kubectl get pods --all-namespaces</strong><br>Error from server (Forbidden): pods is forbidden: User &quot;f97fc9ba-ebda-439e-825a-0e25801cb32c&quot; cannot list pods at the cluster scope</pre><h4>Conclusion</h4><p>We have configured a user to login and get access to specific AKS resources. We could also do this for an entire AAD group.</p><p>To learn more about Kubernetes:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE79s_9ZFNQpSRl8F0iSVhcM2&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DiST-gFFF3xE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FiST-gFFF3xE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f5017b0652f8f698fcda3ed8474fc49/href">https://medium.com/media/8f5017b0652f8f698fcda3ed8474fc49/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook Group</strong></a><strong> </strong>💬<strong>.</strong></p><p><strong>To join our community Slack </strong>🗣️ <strong>and read our weekly Faun topics </strong>🗞️,<strong> click here⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oSdFkACJxs5iy1oR" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=568bf7bc5ae8" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/playing-with-aks-aad-568bf7bc5ae8">Playing with AKS &amp; AAD</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RBAC with Kubernetes in Minikube]]></title>
            <link>https://medium.com/@HoussemDellai/rbac-with-kubernetes-in-minikube-4deed658ea7b?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4deed658ea7b</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[roles]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[rbac]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Sat, 23 Feb 2019 18:55:27 GMT</pubDate>
            <atom:updated>2019-02-28T16:01:33.636Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial will guide you on how to use RBAC with Minikube. The objective is to create users with limited and controlled access to kubernetes resources.</p><p>RBAC or Role Based Access Control is a security approach to grant access for specific resources to assigned users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2LXiArRKdxleP4fqdOavg.jpeg" /></figure><p><strong>Introduction</strong></p><p>Minikube, by default, gives you full access to manage the resources in kubernetes. That is because it assumes that you are using it for learning and it is only you who is using it. Though, it is still possible to create multiple accounts with different access levels using RBAC. For example, we can create a user (e.g developers &amp; Ops) who can create and delete resources and another user (e.g developers from another department) who can only read resources. The same thing could be done with groups. In addition to that, the access could be managed by namespace. So you can create users or groups who can access a specific namespace and not others. This is useful for scenarios where you isolate the dev and production environments through namespaces. So you give access to production namespace only to a few users.</p><p>To authenticate API requests, k8s uses the following options: <strong>client certificates, bearer tokens, authenticating proxy, or HTTP basic auth</strong>.</p><blockquote>Kubernetes doesn’t have an API for creating users. Though, it can authenticate and authorize users.</blockquote><p>We will choose the client certificates as it is the simplest among the four options.</p><p>Please refer to this Github repo to find all the commands and files we use in this workshop: <a href="https://github.com/HoussemDellai/rbac-kubernetes-minikube">github.com/HoussemDellai/rbac-kubernetes-minikube</a></p><h4>1. Create a client certificate</h4><p>We’ll be creating a key and certificate sign request (CSR) needed to create the certificate. Let’s create a directory where to save the certificates. I’ll call it cert:</p><pre><strong>$ mkdir cert &amp;&amp; cd cert</strong></pre><p>1.1. Generate a key using OpenSSL:</p><pre><strong>$ openssl genrsa -out user1.key 2048</strong></pre><p>1.2. Generate a Client Sign Request (CSR)</p><pre><strong>$ openssl req -new -key user1.key -out user1.csr -subj “/CN=user1/O=group1”</strong></pre><p>ls ~/.minikube/ # check that the files ca.crt and ca.key exists in this location.</p><ol><li>3. Generate the certificate (CRT)</li></ol><pre><strong>$ openssl x509 -req -in user1.csr -CA ~/.minikube/ca.crt -CAkey ~/.minikube/ca.key -CAcreateserial -out user1.crt -days 500</strong></pre><p>Now, that we have the .key and the .crt, we can create a user.</p><p>By the way, this article is now available as a video on my channel:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fky4b1HLs3w8%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dky4b1HLs3w8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fky4b1HLs3w8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/40ff03b5048b4e57dd53c00778ce3a7b/href">https://medium.com/media/40ff03b5048b4e57dd53c00778ce3a7b/href</a></iframe><h4><strong>2. Create a user</strong></h4><p>2.1. Set a user entry in kubeconfig</p><pre><strong>$ kubectl config set-credentials user1 --client-certificate=user1.crt --client-key=user1.key</strong></pre><p>2.2. Set a context entry in kubeconfig</p><pre><strong>$ kubectl config set-context user1-context --cluster=minikube --user=user1</strong></pre><p>You can check that it is successfully added to kubeconfig:</p><pre><strong>$ kubectl config view</strong></pre><p>2.3. Switching to the created user</p><p>Now, instead of using the minikube context, we want to use user1-context:</p><pre><strong>$ kubectl config use-context user1-context<br>$ kubectl config current-context </strong># check the current context<br>user1-context</pre><p>But, now, user1 doesn’t have any access privileges to the cluster. For that we’ll have access denied if we try to create any resource:</p><pre><strong>$ kubectl create namespace ns-test<br></strong>Error from server (Forbidden): namespaces is forbidden: User &quot;user1&quot; cannot create resource &quot;namespaces&quot; in API group &quot;&quot; at the cluster scope</pre><h4>3. Grant access to the user</h4><p>To give access to manage k8s resources to user1, we need to create a Role and a BindingRole.</p><p>3.1. Create a Role</p><p>Let’s create a Role object in role.yaml file. The Role will define which resources could be accessed and which operations/verbs could be used.</p><pre><strong>kind: </strong>Role<br><strong>apiVersion: </strong>rbac.authorization.k8s.io/v1<br><strong>metadata:<br>  namespace: </strong>default<br>  <strong>name: </strong>pod-reader<br><strong>rules:<br>- apiGroups: </strong>[“”] # “” indicates the core API group<br>  <strong>resources: </strong>[“pods”]<br>  <strong>verbs: </strong>[“get”, “watch”, “list”]</pre><blockquote>Resources: <strong>pod, deployment, namespace, secret, configmap, service, persistentvolume…</strong></blockquote><blockquote>Verbs: <strong>get, list, watch, create, delete, update, edit, exec.</strong></blockquote><p>3.2. Create a BindingRole</p><p>We want to match the User1 to the role created above named : pod-reader. To do that, we need to use RoleBinding.</p><pre><strong>kind: </strong>RoleBinding<br><strong>apiVersion: </strong>rbac.authorization.k8s.io/v1<br><strong>metadata:<br>  name: </strong>read-pods<br>  <strong>namespace: </strong>default<br><strong>subjects:<br>- kind: </strong>User<br>  <strong>name: </strong>user1 # Name is case sensitive<br>  <strong>apiGroup: </strong>rbac.authorization.k8s.io<br><strong>roleRef:<br>  kind: </strong>Role #this must be Role or ClusterRole<br>  <strong>name: </strong>pod-reader # must match the name of the Role<br>  <strong>apiGroup: </strong>rbac.authorization.k8s.io</pre><p>Deploy both the role.yaml and role-binding.yaml to k8s, and don’t forget to use the context of minikube:</p><pre><strong>$ kubectl config use-context minikube<br></strong>Switched to context &quot;minikube&quot;.<br><strong>$ kubectl apply -f role.yaml<br></strong>role.rbac.authorization.k8s.io/pod-reader created<br><strong>$ kubectl apply -f role-binding.yaml<br></strong>rolebinding.rbac.authorization.k8s.io/read-pods created</pre><p>We check that the Role and BindingRole was created successfully:</p><pre><strong>$ kubectl get roles<br></strong>NAME       AGE<br>pod-reader 2m<br><strong>$ kubectl get rolebindings<br></strong>NAME        AGE<br>read-pods   2m</pre><blockquote>We used Role to scope the rules to only one namespace, but we can also use <strong>ClusterRole</strong> to define more than one namespace. RoleBinding is used to bind to Role and <strong>ClusterRoleBinding</strong> is used to bind to ClusterRole.</blockquote><h4>4. Testing the allowed operations for user</h4><p>Switch again to user1 and try one of the non allowed operations like to create a namespace. This will fail, because user1 is not allowed to do so.</p><pre><strong>$ kubectl config use-context user1-context<br>$ kubectl create namespace ns-test</strong> # won&#39;t succeed, Forbidden</pre><p>But, when try one of the allowed operations, like getting the pods:</p><pre><strong>$ kubectl get pods </strong># this will succeed !</pre><h4>Summary</h4><p>At the end, we have created a user with limited access to k8s resources. User1 will be authenticated through his certificate and will be granted access through the Role and RoleBinding.</p><p>I have used the following resources to create this workshop, (thank you the writers !):</p><ul><li><a href="https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/">Configure RBAC in your Kubernetes Cluster</a></li><li><a href="https://www.cncf.io/blog/2018/08/01/demystifying-rbac-in-kubernetes/">Demystifying RBAC in Kubernetes - Cloud Native Computing Foundation</a></li></ul><p>Learn more about Kubernetes on my youtube channel:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE79s_9ZFNQpSRl8F0iSVhcM2&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DeHvKc6hNmhI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FeHvKc6hNmhI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec875dda47263ff51cb9262755646e23/href">https://medium.com/media/ec875dda47263ff51cb9262755646e23/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4deed658ea7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PreProd for Database]]></title>
            <link>https://faun.pub/preprod-for-database-288aaec97a1?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/288aaec97a1</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[preprod]]></category>
            <category><![CDATA[data-migration]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Mon, 31 Dec 2018 13:20:25 GMT</pubDate>
            <atom:updated>2019-07-17T20:34:06.895Z</atom:updated>
            <content:encoded><![CDATA[<p>Deploying changes to the production database is really risky. It is enough to forget about one single simple detail and the migration will fail. Customers will start complaining and even worst they will go to the competitors.</p><p>In this article, we’ll solve this issue by creating a PreProd environment to run migrations and tests. Then, we’ll detail the challenges of this approach.</p><p>In a previous article, we have detailed the main options on how to <a href="https://medium.com/@HoussemDellai/devops-for-database-4e442abfd939"><strong>apply migrations to the database from the CI/CD pipeline</strong></a>. We recommend you go through it first, as this one is a sort of continuity to it.</p><blockquote>Our goal is to make sure database migration will always work on production.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*urDRl-wh3OXsz1shBmTTRg.png" /><figcaption>The PreProd environment pipeline for database</figcaption></figure><h4>1. Create the PreProd environment</h4><p>To make sure database migrations will work on production, we’ll create a similar environment. We’ll call it PreProd. Similar here means the same RDBMS version, settings and offer. Because migrations might work for one setting but not for another. We want to be as close as possible to the prod settings. This makes our tests more efficient.</p><p>Technically speaking, this means creating a new Resource Group in Azure dedicated for this environment. Then creating a new instance of SQL Azure similar to the one running in production.</p><blockquote>We can automate the creation of PreProd using IaC through ARM templates or Cloud Formation or Terraform…</blockquote><p>So that we can create and delete it on the flight. Because we need this env temporarily, just the time to run migrations and tests.</p><h4>2. Import Database from Production to PreProd</h4><p>We can test database migrations on a new database containing fake simple data. But, again migrations might succeed here and fail on prod. That is because sometimes the data itself can fail the migrations.</p><p>Again, the key point here, is to be close as much as possible to the prod. So, we’ll import the same database from prod to PreProd. We could do this through cli tools. On Azure, that could be done through the following powershell script: <a href="https://docs.microsoft.com/en-us/powershell/module/azurerm.sql/new-azurermsqldatabasecopy">more details here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gw4lNMNRt9CnaimYRz0nAQ.png" /><figcaption>Azure PS script to copy database</figcaption></figure><blockquote>We can copy the entire database to PreProd. And we can also import only the data and sanitise it on the flight.</blockquote><h4>3. Sanitise data (GDPR obligation)</h4><p>We copied the prod database with all our sensitive customers data. Allo GDPR :) GDPR have some rules here. So we need to make our data anonymous, i.e we won’t be able to read certain customer’s data. In addition to that, we need to hide sensitive data like credit cards.</p><p>Masking data could be done using SQL scripts for simple scenarios like just hiding the data, like replacing email addresses with static characters. But, in other cases, we need customers emails, we might have queries using them in our tests. Or may be email is used as foreign key. In this case, we can use a Data Masking tool to substitute our data. Instead of having xxxxx shown on all email properties, we’ll have values like email01@email.com.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LOdk4whl8Ki2LbFoLclmg.png" /><figcaption>Data masking/sanitisation</figcaption></figure><h4>4. Run database migrations</h4><p>At this stage, we have a database server with same settings as in prod and a database with same/sanitised data as in prod. We now have an env really similar to prod, but totally isolated. If the changes will fail on prod, they will first fail in PreProd. We can now run the migrations securely and confidently.</p><h4>5. Run unit tests and integration tests</h4><p>Even if migration runs successfully, we still want to run tests to make sure the database behaves as the app expects. For this case, we can run unit and integration tests. Frameworks like tSqlt makes it easy to write unit tests against the database. <a href="https://github.com/asagues/CaveTester">Cave Tester</a> makes sure you have the required data for each unit/integration test. It guarantees that we will end up with the exact same data as we started with before running the tests. It uses snapshots to save data before each test and restore it after each test.</p><h4>6. Delete PreProd environment</h4><p>After running migrations and tests, if one of them fails, then we can investigate the problem in PreProd and we can clap because we detected it before it happens in prod. Then, we can delete the entire environment to save money. We use IoC, so we can recreate the infrastructure within seconds when needed.</p><blockquote>Typically, PreProd needs to run just before going to production.</blockquote><h4>Optimising PreProd</h4><p>The steps described above are the main lines. But, they can be optimised. For example, copying the database could take too long if it is too big. In addition to that, the database performance will degrade during the copying. A solution for this might be to run a job at the time we have lowest traffic, to copy the database to a cloud storage. So that the PreProd will run faster as the database will be ready.</p><h4>This article as video</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FnCIqLh4WsNs%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnCIqLh4WsNs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnCIqLh4WsNs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ab77974fb2ae9a42d83fb3bfec4a914/href">https://medium.com/media/1ab77974fb2ae9a42d83fb3bfec4a914/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Join our community Slack and read our weekly Faun topics ⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oSdFkACJxs5iy1oR" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=288aaec97a1" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/preprod-for-database-288aaec97a1">PreProd for Database</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevOps for the Database]]></title>
            <link>https://faun.pub/devops-for-database-4e442abfd939?source=rss-20f6b842a7cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e442abfd939</guid>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Houssem Dellai]]></dc:creator>
            <pubDate>Sun, 30 Dec 2018 22:51:46 GMT</pubDate>
            <atom:updated>2019-07-17T20:33:41.262Z</atom:updated>
            <content:encoded><![CDATA[<p>Most talks about DevOps focuses on the application’s source code and dismiss the database. In this article, we’ll see how to add the database to the same CI/CD pipelines used for the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uKT1pJnYqcrLdqHJ-z2ZLg.png" /><figcaption>CI/CD for the app (Dev), database (DBA) and infrastructure (Ops)</figcaption></figure><h4>Why adding the Database to the CI/CD pipeline ?</h4><p>The objective of DevOps is to create a process for safe, secure and even more frequent delivery for the app. By saying the app, we mean the entire app including the database. Because, in a new app update, we might also change the database schema. For example, we need to add a property Color to our products. This requires 2 changes: updating the entity in our app’s code and also updating the SQL script for the product table.</p><blockquote>The point is that both changes to the app and the database must be deployed together during the same deployment process.</blockquote><p>Because, if we update only the app’s entity, the app will try to find the corresponding column in the database, it won’t find it, thus it will fail.</p><p>A bad practice is being done today is that DBAs are working in a separate team than working with developers. They use their own tools and manages the database as it’s their own property. And if developers ask them for an update, they will take so long to do it. It’s not they are they are lazy, but it is because they have their own process (in most cases not agile) and they need to make sure their SQL scripts are really optimised and well tested.</p><p>In addition to that, DBAs deploys the updates to the database manually, ignoring the CI/CD process, even when there’s one. That is because the lack of communication with developers.</p><p>DBAs and developers (and the Ops) are working on the same project, so they need to work as one team, in the same open-space, as one team, attend together the DSM, commit to the same repository, plan the updates and impacts together and most of all: deliver together. Here is when DevOps process and practices comes in to play. We’ll have processes that unites not only developers and operations, but also DBAs <em>(Oh shouldn’t we call it DevDbaOps !).</em></p><p>In one hand, integrating the DBAs to the dev &amp; ops team requires a structural change to the organisation. Agile/DevOps experts can handle that as they already got a similar challenge to integrate dev and ops together.</p><p>In the other hand comes the technical details: the CI/CD process. That is what we’ll be explaining next.We’ll be trying to integrate the DBAs work (SQL scripts) into the CI/CD pipelines.</p><blockquote>The technical solution to apply DevOps to the database is to add or generate SQL scripts in the CI pipeline and deploy them during the CD.</blockquote><p>We have 2 main options: one using ORM migrations and another one using Database project. We’ll be building the sample using SQL Server, ASP.NET Core, EF Core and Azure DevOps. But, the same concept could be applied to other similar technologies.</p><h4>1. ORMs can enable database migrations transparently</h4><p>ORMs like Entity Framework (EF), Doctrine, Hibernate (with Liquibase or Flyway) have built-in features for code-first automatic migrations. They can detect changes to the object entities, generate the corresponding SQL script and deploy it to the database.</p><p>Automatic migrations could be enabled in EF by cli command <em>‘</em>dotnet ef migrations add AddedColor<em>’</em> and adding the following code: <strong>Context.Database.Migrate(); </strong>// <a href="https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/">find out more.</a></p><p>EF will create a Migrations folder inside our app source code. Then it will create migration objects containing the new change through the Up method written using the ORM objects. Note in the following picture, the AddColumn method which will be translated later, by EF, during the deployment, into a SQL operation ALTER TABLE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSzbn8BRjKB4WZ0u2X0Azw.png" /><figcaption>Entity Framework Core Code First Migrations</figcaption></figure><p>Note also, the use of Down method which is used to revert back the changes made by the Up method. That is useful for Rollback scenarios.</p><p>With that in place, you don’t need to change anything in your CI/CD pipelines. The ORM will generate and deploy the migration SQL scripts transparently. Our Build/CI pipeline will look like the following in Azure DevOps: all the tasks are for restoring, building, unit testing and publishing the app’s source code, nothing special for the database. Note the last task used to publish the ARM template created by the Ops to create the infrastructure needed to run the app on Azure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KdGZQ3gthLkPD2odmYEsZQ.png" /><figcaption>CI pipeline when using EF migrations</figcaption></figure><p>And our Release/CD pipeline will look like the following, where we are just publishing the generated package into Azure App Service. During that operation, the migrations will run at the app restart. You can notice also the first task which will deploy the ARM template to Azure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_FD6XaJavECaYYFA4fBNRA.png" /><figcaption>CD pipeline when using EF migrations</figcaption></figure><p>However, this approach is well suited only for small projects where there’s no DBA, database schema is pretty simple and most of all: we give full control and trust to the ORM to manage the database !</p><h4>2. Running the migration script generated by the ORM</h4><blockquote>Well, we can get back some control. We’ll let the ORM generate the SQL scripts. But then we’ll go to read, edit, validate and deploy it.</blockquote><p>For that, we need to run a command line to intercept the generated SQL scripts: <strong>dotnet ef migrations script -o outputfile.sql</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFNuFyJZ-SrNn82GZ_rCuA.png" /><figcaption>CI pipeline that intercepts generated EF SQL scripts</figcaption></figure><p>We’ll get the generated script and we’ll upload it to drop/artifact folder so we can access it during the CD pipeline. That is done using Publish Artifact task. When the build is done, we can view and edit the generated scripts.</p><p>Now, from the CD pipeline, we’ll deploy the script to the database directly. For that, we need to connect to the db hosted on Azure. That is the role of the task Azure SQL Publish. It will use the <strong>sqlcmd </strong>cli tool under the hood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LigUgG-Dzhqep-aU2yCIWw.png" /><figcaption>CD pipeline for deploying SQL scripts into database</figcaption></figure><h4>3. Using the Database project</h4><p>Still, for larger projects, DBAs creates separate project types for managing the database. These projects contains, generate and validate SQL queries. We are talking here about the Database project that could be opened from Visual Studio, but also from SQL Server Management Studio (SSMS) which is the favourite tool for DBAs. With this approach, DBA will have full control over generating the SQL scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSO0sngelIn8NHD7ZrFmzQ.png" /><figcaption>Database project in Visual Studio</figcaption></figure><p>Now, the generation of the SQL scripts will be done manually by DBAs, not anymore managed by ORM. We are fine with that as we believe the DBA are more intelligent than ORM especially for preventing and resolving conflicts.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE78FEvDjD9zfzSGvsdkvkkrj&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBgEP5U2YA2E&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBgEP5U2YA2E%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab140ee4c18221536cce35a8775bf18c/href">https://medium.com/media/ab140ee4c18221536cce35a8775bf18c/href</a></iframe><p>We just need to build the project to generate a DacPac file containing the new schema built using the provided scripts. Note in the following picture we could build the database project in a separate agent/phase as it doesn’t depend from the app’s projects. That is useful if we want to run them in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOH6ZtlWaPcA9iCYCZshGQ.png" /><figcaption>CI pipeline for using the database project</figcaption></figure><p>For the CD pipeline, we’ll deploy the DacPac to the database. We’ll use the same task as before: Azure SQL Database Deployment, but instead of publishing a SQL script, we’ll tell it to publish the DacPac.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJg9FssW9nmXjj2vlsrV7Q.png" /><figcaption>CD pipeline to deploy DacPac to database</figcaption></figure><h4>Problems with these approaches</h4><p>All the above approaches will work in most of the cases. But in some minor cases, they can put us in serious problems. In fact, in some cases where we have a complex schema to update, the script might stop running at the middle. This means it deployed some changes but not all. Imagine the case where we want to add two columns to two different tables where we have dependency like foreign keys between them. These are the cases where we need to deploy all the changes or nothing.</p><blockquote>It’s a good practice to have a Rollback strategy for both the app and database updates.</blockquote><p>But for this specific case, the already created revert-back scripts might not work. Because we did expected the case to rollback all the changes, but not just some of the changes. So, we’ll have DBA connecting directly to the database to fix the issue manually.</p><p>For the application side, the rollback strategies are easier, quicker and efficient. One solution would be to delete the package v2 and redeploy the v1. Another solution would be deploying v2 on another application server and route traffic to it only when deployment is successful. These solutions are not applicable to the database.</p><p>It is in order to avoid such problems that one of the DevOps practices is to release more frequently small deployments instead of less frequently big ones. We already apply this principle in Agile. So why not applying the same principle for the database too.</p><p>This principle help to reduce these problems but not avoiding them all. The solution would be through testing the migrations on a PreProd environment with a similar database schema and data to the one on Prod. This way, we can detect earlier any migration failures that might happen in Prod.</p><h4>Unit tests and integration tests for the database</h4><p>It’s a common practice to have unit and integration tests for the application as these tests helps raise the quality of the app and gives more confidence to deploy more frequently. The same type of tests could be also applied to the database.</p><blockquote>Yes, we can develop unit tests and integration tests for the database.</blockquote><p>Database unit tests typically tests the stored procedures. And they have a dedicated framework too ! And it uses Assert ! It is <a href="https://tsqlt.org/">tSQLt</a>.</p><p>For app’s integration tests, we use mocks or in-memory database to not change the real data. It would be really valuable to be able to run the integration tests directly against a real database. It makes the tests more real and helps to detect more failures especially the ones that are related to the data itself more than the schema. Think of the cases like foreign keys with null values.</p><p>The problem here is that every test might change or even delete the data that might be required by the next test. <a href="https://github.com/asagues/CaveTester">Cave Tester</a> addresses this issue. It creates a database snapshot before every test. Then we can change the data as we want, we can even delete all the data. After each test, it will restore the snapshot. So that after the tests, we’ll end up with the same exact data we started with.</p><blockquote>Cave Tester ensures we can change the data during the tests and end up with the same exact data we started with. No data loss or change.</blockquote><h4>Conclusion</h4><p>Applying DevOps to the database is really valuable to help innovate by deploying faster and securely. It still have its challenges but small steps makes a difference. We recommend to start with encouraging the DBAs to add their scripts to the app’s repository. This way they will be dependant to developers, thus they’ll start talking to them and collaborate together.</p><p>Before wiring this article, we created the following video series that walks through more details on building and releasing the database.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLpbcUe4chE78FEvDjD9zfzSGvsdkvkkrj&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DF8Ng_vimd2k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FF8Ng_vimd2k%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae9bf25db9ab2a2f34bd700e1aba4c86/href">https://medium.com/media/ae9bf25db9ab2a2f34bd700e1aba4c86/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FV7XKEPh2oLk%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DV7XKEPh2oLk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FV7XKEPh2oLk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f947be97710e8a4fe69ec3b24f19603d/href">https://medium.com/media/f947be97710e8a4fe69ec3b24f19603d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Join our community Slack and read our weekly Faun topics ⬇</strong></p><figure><a href="https://www.faun.dev/join/?utm_source=medium.com%2Ffaun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oSdFkACJxs5iy1oR" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e442abfd939" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/devops-for-database-4e442abfd939">DevOps for the Database</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>