<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nevin Cansel Efendioglu on Medium]]></title>
        <description><![CDATA[Stories by Nevin Cansel Efendioglu on Medium]]></description>
        <link>https://medium.com/@nevincansel?source=rss-853bf1b442b9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ExJhHLq1m7fNmFy2kJfKfA@2x.jpeg</url>
            <title>Stories by Nevin Cansel Efendioglu on Medium</title>
            <link>https://medium.com/@nevincansel?source=rss-853bf1b442b9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:05:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nevincansel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cross-Account Backup on AWS with Terraform]]></title>
            <link>https://medium.com/@nevincansel/cross-account-backup-on-aws-with-terraform-83fbd5310f03?source=rss-853bf1b442b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/83fbd5310f03</guid>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cross-account]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Nevin Cansel Efendioglu]]></dc:creator>
            <pubDate>Sun, 11 Dec 2022 16:49:33 GMT</pubDate>
            <atom:updated>2024-05-16T08:49:13.850Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*3a86AB4Ayil4Nm_JYhle_A.png" /></figure><p>Backup is essential when we have critical systems. We never forget to enable backup option to protect our data.</p><p><strong>But, what if suddenly we cannot reach our account and need to create whole infrastructure from the beginning?</strong></p><p>We need the data we have backed up and we have to backup our backups to another account to be able to access the data in any disaster scenario. AWS provides a solution for that with AWS Backup.</p><blockquote>AWS Backup is a fully-managed service that makes it easy to centralize and automate data protection across AWS services, in the cloud, and on premises. Using this service, you can configure backup policies and monitor activity for your AWS resources in one place.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*SVSZCTVx8oibbBwjhW0uOA.gif" /><figcaption>Backup of Backup?</figcaption></figure><p>Let’s see how to backup a backup to another account.</p><p>We will take EFS as an example.</p><blockquote>Amazon Elastic File System (Amazon EFS) provides serverless, fully elastic file storage so that you can share file data without provisioning or managing storage capacity and performance.</blockquote><p>Let’s begin!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*B0J5vGBl_9fhlzyo-FYQBw.gif" /></figure><p><strong><em>Resources should be created in destination account first because destination vault will be used in source account’s backup rule.</em></strong></p><h4>Required Resources for AWS Backup in Destination Account</h4><ul><li>KMS Key</li><li>KMS Alias</li><li>Backup Vault</li><li>Backup Vault Policy</li></ul><p><strong>KMS Key</strong></p><p>The policy below is needed to share KMS with other account.</p><pre>resource &quot;aws_kms_key&quot; &quot;aws_kms_key&quot; {<br>  description         = &quot;KMS Key for Backup&quot;<br>  enable_key_rotation = true<br>  policy              = &lt;&lt;POLICY<br>  {<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Id&quot;: &quot;key-consolepolicy-3&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;Enable IAM User Permissions&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: [<br>                    &quot;arn:aws:iam::${local.source_account_number}:root&quot;,<br>                    &quot;arn:aws:iam::${local.target_account_number}:root&quot;<br>                ]<br>            },<br>            &quot;Action&quot;: &quot;kms:*&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Allow access for Key Administrators&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;${local.kms_admin_role_arn}&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;kms:Create*&quot;,<br>                &quot;kms:Describe*&quot;,<br>                &quot;kms:Enable*&quot;,<br>                &quot;kms:List*&quot;,<br>                &quot;kms:Put*&quot;,<br>                &quot;kms:Update*&quot;,<br>                &quot;kms:Revoke*&quot;,<br>                &quot;kms:Disable*&quot;,<br>                &quot;kms:Get*&quot;,<br>                &quot;kms:Delete*&quot;,<br>                &quot;kms:TagResource&quot;,<br>                &quot;kms:UntagResource&quot;,<br>                &quot;kms:ScheduleKeyDeletion&quot;,<br>                &quot;kms:CancelKeyDeletion&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Allow use of the key&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: [<br>                    &quot;arn:aws:iam::${local.source_account_number}:root&quot;,<br>                    &quot;arn:aws:iam::${local.target_account_number}:root&quot;,<br>                    &quot;${local.kms_admin_role_arn}&quot;<br>                ]<br>            },<br>            &quot;Action&quot;: [<br>                &quot;kms:Encrypt&quot;,<br>                &quot;kms:Decrypt&quot;,<br>                &quot;kms:ReEncrypt*&quot;,<br>                &quot;kms:GenerateDataKey*&quot;,<br>                &quot;kms:DescribeKey&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Allow attachment of persistent resources&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: [<br>                    &quot;arn:aws:iam::${local.source_account_number}:root&quot;,<br>                    &quot;arn:aws:iam::${local.target_account_number}:root&quot;,<br>                    &quot;${local.kms_admin_role_arn}&quot;<br>                ]<br>            },<br>            &quot;Action&quot;: [<br>                &quot;kms:CreateGrant&quot;,<br>                &quot;kms:ListGrants&quot;,<br>                &quot;kms:RevokeGrant&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;Bool&quot;: {<br>                    &quot;kms:GrantIsForAWSResource&quot;: &quot;true&quot;<br>                }<br>            }<br>        }<br>    ]<br>}<br>POLICY<br>}</pre><p><strong>KMS Alias</strong></p><p>KMS Alias is used for naming the KMS.</p><pre>resource &quot;aws_kms_alias&quot; &quot;aws_kms_alias&quot; {<br>  name          = &quot;alias/aws-backup-kms&quot;<br>  target_key_id = aws_kms_key.aws_kms_key.key_id<br>}</pre><p><strong>Backup Vault</strong></p><p>Backup vault will be created with KMS.</p><pre>resource &quot;aws_backup_vault&quot; &quot;aws_backup_vault&quot; {<br>  name        = &quot;aws_backup_vault&quot;<br>  kms_key_arn = aws_kms_key.aws_kms_key.arn<br>}</pre><p><strong>Backup Vault Policy</strong></p><p>It will be attached to vault which is defined above to allow copying backups into destination vault.</p><pre>resource &quot;aws_backup_vault_policy&quot; &quot;aws_backup_vault_policy&quot; {<br>  backup_vault_name = aws_backup_vault.aws_backup_vault.name<br>  policy = &lt;&lt;POLICY<br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Id&quot;: &quot;default&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;Allow ${local.source_account_number} to copy into aws_backup_vault backup vault&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;backup:CopyIntoBackupVault&quot;,<br>      &quot;Resource&quot;: &quot;*&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;AWS&quot;: &quot;arn:aws:iam::${local.source_account_number}:root&quot;<br>      }<br>    }<br>  ]<br>}<br>POLICY<br>}</pre><h4>Required Resources for AWS Backup in Source Account</h4><ul><li>KMS Key</li><li>KMS Alias</li><li>Backup Vault</li><li>Backup Plan</li><li>Backup Selection</li></ul><p><strong>KMS Key</strong></p><p>KMS Key will be used while creating a vault to encrypt the data. It has its policy inside to define who can access and use the key. Furthermore, if it is wanted to share the key with another account, policy should be adjusted for sharing.</p><p>The policy below is needed to share KMS with other account.</p><pre>resource &quot;aws_kms_key&quot; &quot;aws_kms_key&quot; {<br>  description         = &quot;KMS Key for Backup&quot;<br>  enable_key_rotation = true<br>  policy              = &lt;&lt;POLICY<br>  {<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Id&quot;: &quot;key-consolepolicy-3&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;Enable IAM User Permissions&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: [<br>                    &quot;arn:aws:iam::${local.source_account_number}:root&quot;,<br>                    &quot;arn:aws:iam::${local.target_account_number}:root&quot;<br>                ]<br>            },<br>            &quot;Action&quot;: &quot;kms:*&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Allow access for Key Administrators&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;${local.kms_admin_role_arn}&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;kms:Create*&quot;,<br>                &quot;kms:Describe*&quot;,<br>                &quot;kms:Enable*&quot;,<br>                &quot;kms:List*&quot;,<br>                &quot;kms:Put*&quot;,<br>                &quot;kms:Update*&quot;,<br>                &quot;kms:Revoke*&quot;,<br>                &quot;kms:Disable*&quot;,<br>                &quot;kms:Get*&quot;,<br>                &quot;kms:Delete*&quot;,<br>                &quot;kms:TagResource&quot;,<br>                &quot;kms:UntagResource&quot;,<br>                &quot;kms:ScheduleKeyDeletion&quot;,<br>                &quot;kms:CancelKeyDeletion&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Allow use of the key&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: [<br>                    &quot;arn:aws:iam::${local.source_account_number}:root&quot;,<br>                    &quot;arn:aws:iam::${local.target_account_number}:root&quot;,<br>                    &quot;${local.kms_admin_role_arn}&quot;<br>                ]<br>            },<br>            &quot;Action&quot;: [<br>                &quot;kms:Encrypt&quot;,<br>                &quot;kms:Decrypt&quot;,<br>                &quot;kms:ReEncrypt*&quot;,<br>                &quot;kms:GenerateDataKey*&quot;,<br>                &quot;kms:DescribeKey&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Allow attachment of persistent resources&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: [<br>                    &quot;arn:aws:iam::${local.source_account_number}:root&quot;,<br>                    &quot;arn:aws:iam::${local.target_account_number}:root&quot;,<br>                    &quot;${local.kms_admin_role_arn}&quot;<br>                ]<br>            },<br>            &quot;Action&quot;: [<br>                &quot;kms:CreateGrant&quot;,<br>                &quot;kms:ListGrants&quot;,<br>                &quot;kms:RevokeGrant&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;Bool&quot;: {<br>                    &quot;kms:GrantIsForAWSResource&quot;: &quot;true&quot;<br>                }<br>            }<br>        }<br>    ]<br>}<br>POLICY<br>}</pre><p><strong>KMS Alias</strong></p><p>KMS Alias is used for naming the KMS.</p><pre>resource &quot;aws_kms_alias&quot; &quot;aws_kms_alias&quot; {<br>  name          = &quot;alias/aws-backup-kms&quot;<br>  target_key_id = aws_kms_key.aws_kms_key.key_id<br>}</pre><p><strong>Backup Vault</strong></p><p>Backup vault will be created with KMS.</p><pre>resource &quot;aws_backup_vault&quot; &quot;aws_backup_vault&quot; {<br>  name        = &quot;aws_backup_vault&quot;<br>  kms_key_arn = aws_kms_key.aws_kms_key.arn<br>}</pre><p><strong>Backup Plan</strong></p><p>With this rule, file system will be backed up and copied into destination vault.</p><pre>resource &quot;aws_backup_plan&quot; &quot;aws_backup_plan&quot; {<br>  name = &quot;backup_plan&quot;<br><br>  rule {<br>    rule_name         = &quot;backup_rule&quot;<br>    target_vault_name = aws_backup_vault.aws_backup_vault.name<br>    schedule          = &quot;cron(0 5 ? * * *)&quot;<br>    start_window      = 480<br>    completion_window = 10080<br>    copy_action {<br>      destination_vault_arn = local.destination_vault<br>    }<br><br>    lifecycle {<br>      delete_after = 90<br>    }<br>  }<br>}</pre><p><strong>Backup Selection</strong></p><p>Backup selection is required to select the specific file system in AWS EFS.</p><pre>resource &quot;aws_backup_selection&quot; &quot;aws_backup_selection&quot; {<br>  iam_role_arn = &quot;arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/service-role/AWSBackupDefaultServiceRole&quot;<br>  name         = &quot;backup_selection&quot;<br>  plan_id      = aws_backup_plan.aws_backup_plan.id<br>  resources    = [local.efs_arn]<br>}</pre><p>After all resources are created in source and destination accounts, backups that are stored in source vault will be copied into destination vault. Data are safe!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*JMNYk9gHYO3gg1vDuW8r4Q.gif" /></figure><p>Hope you enjoy the article. Feel free to contact me for any questions or contributions!</p><p>🌿 If you want to support me, you can click the picture 👇 Have a wonderful day!</p><figure><a href="https://www.buymeacoffee.com/nevincansel"><img alt="" src="https://cdn-images-1.medium.com/max/545/0*7QlAaLgHkW01P5E7.png" /></a></figure><ul><li><a href="https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html">What is Amazon Elastic File System?</a></li><li><a href="https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html">What is AWS Backup?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83fbd5310f03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EKS Cluster with Terraform]]></title>
            <link>https://medium.com/@nevincansel/eks-cluster-with-terraform-589fed43dc22?source=rss-853bf1b442b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/589fed43dc22</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Nevin Cansel Efendioglu]]></dc:creator>
            <pubDate>Thu, 18 Aug 2022 14:16:15 GMT</pubDate>
            <atom:updated>2023-01-16T22:42:41.786Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*FPEn-aGMXTM3hhNa.jpg" /></figure><h3><strong>TL;DR</strong></h3><ul><li><strong>Amazon Elastic Kubernetes Service</strong> (Amazon EKS) is a managed <a href="https://aws.amazon.com/kubernetes/">Kubernetes</a> service that makes it easy for you to run Kubernetes on AWS and <a href="https://aws.amazon.com/eks/eks-anywhere/">on-premises</a>.</li><li><strong>HashiCorp Terraform</strong> is an infrastructure as code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share.</li><li><strong>GitLab CI/CD</strong> is the part of GitLab that you use for all of the continuous methods (Continuous Integration, Delivery, and Deployment).</li></ul><p>Let’s see how to create EKS Cluster with Terraform using GitLab CI.</p><p><strong>Prerequisites</strong></p><ul><li>AWS Account</li><li>Terraform</li><li>GitLab</li></ul><h3>Terraform</h3><blockquote>A <em>module</em> is a container for multiple resources that are used together.</blockquote><blockquote>Every Terraform configuration has at least one module, known as its <em>root module</em>, which consists of the resources defined in the .tf files in the main working directory.</blockquote><p>We can create modules by ourselves but we also can use modules which are created in Terraform Registry.</p><p>While we create the cluster repository, we will create some modules by ourselves and use some modules from cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/1*SiWs1gNbS__mfUUBGhSv3g.png" /><figcaption>Structure of Terraform Folder</figcaption></figure><p>First, we will create modules that the cluster needs which are <strong>eks</strong>, <strong>vpc </strong>and <strong>security group</strong>.</p><p>To add modules, we need to create a folder named <strong>modules </strong>and subfolders for all modules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*l_yAwsJyMDU07XhyXNTSaA.png" /><figcaption>Modules Folder</figcaption></figure><p>Let’s start with <strong>VPC module</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*aXNDWAj6AsQ749SFkf7SHQ.png" /><figcaption>Structure of VPC Module</figcaption></figure><ul><li>We have a main.tf which is <strong>vpc.tf</strong> here to create <strong>VPC </strong>as a resource.</li><li>We also have <strong>variables.tf</strong> because we will give some values as variables.</li><li>We need to expose some information for cluster so we have <strong>outputs.tf</strong>.</li></ul><pre>module &quot;vpc&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version = &quot;3.2.0&quot;<br><br>  name                 = var.vpc_name<br>  cidr                 = &quot;172.40.0.0/16&quot;<br>  azs                  = var.azs<br>  private_subnets      = [&quot;172.40.1.0/24&quot;, &quot;172.40.2.0/24&quot;, &quot;172.40.3.0/24&quot;]<br>  public_subnets       = [&quot;172.40.4.0/24&quot;, &quot;172.40.5.0/24&quot;, &quot;172.40.6.0/24&quot;]<br>  enable_nat_gateway   = true<br>  single_nat_gateway   = true<br>  enable_dns_hostnames = true<br><br>  tags = {<br>    &quot;kubernetes.io/cluster/${var.cluster_name}&quot; = &quot;shared&quot;<br>  }<br><br>  public_subnet_tags = {<br>    &quot;kubernetes.io/cluster/${var.cluster_name}&quot; = &quot;shared&quot;<br>    &quot;kubernetes.io/role/elb&quot;                      = &quot;1&quot;<br>  }<br><br>  private_subnet_tags = {<br>    &quot;kubernetes.io/cluster/${var.cluster_name}&quot; = &quot;shared&quot;<br>    &quot;kubernetes.io/role/internal-elb&quot;             = &quot;1&quot;<br>  }<br>}</pre><p>As you can see, while creating <a href="https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest">VPC module</a>, we also use <strong>module as resource</strong> which was created by AWS on Terraform Cloud.</p><p><em>While creating a resource, declaring version is very important! Otherwise you can be affected by any changes that will come with new versions.</em></p><p>With this method, we do not need to create a module from zero and we can use all inputs and outputs which were defined in <a href="https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest">Terraform Registry Documentation</a>.</p><p>Now, let’s check <strong>EKS module</strong>.</p><p>Again, we will use a module from cloud for our EKS module.</p><p><em>To access the cluster from outside, we need to add our user to </em><strong><em>aws_auth_users</em></strong><em>.</em></p><pre>module &quot;eks&quot; {<br>  source                    = &quot;terraform-aws-modules/eks/aws&quot;<br>  version                   = &quot;18.26.0&quot;<br>  cluster_name              = var.cluster_name<br>  cluster_version           = &quot;1.22&quot;<br>  subnet_ids                = var.subnet_ids<br>  vpc_id                    = var.vpc_id</pre><pre>eks_managed_node_group_defaults = {<br>    disk_size      = 50<br>    min_size       = var.desired_nodes<br>    max_size       = 10<br>    desired_size   = var.desired_nodes<br>    capacity_type  = &quot;SPOT&quot;<br>  }</pre><pre>eks_managed_node_groups = {<br>    &quot;eks_cluster&quot; = {<br>      instance_types = [var.instance_type_nodes]<br>      desired_capacity   = var.desired_nodes<br>    }    <br>  }</pre><pre>aws_auth_users = [<br>    {<br>      userarn  = var.eks_userarn<br>      username = var.eks_username<br>      groups   : [<br>        &quot;system:nodes&quot;, &quot;system:masters&quot;<br>      ]<br>    }<br>  ]</pre><pre>cluster_enabled_log_types = [<br>    &quot;api&quot;,&quot;audit&quot;,&quot;authenticator&quot;,&quot;controllerManager&quot;,&quot;scheduler&quot;<br>  ]<br>  tags = var.tags<br>}</pre><p>As you can see, some parameters are hard coded and some parameters are defined as variables. We can define parameters, which probably will not change, hard coded but for others, we need to define them as variables to be able to change them easily from <strong>main.tf</strong>.</p><p>This is the <strong>main.tf</strong> to create necessary resources which are eks, vpc and security group.</p><pre>module &quot;eks&quot; {<br>    source = &quot;./modules/eks&quot;<br>    cluster_name = local.cluster_name<br>    subnet_ids = module.vpc.private_subnets<br>    vpc_id = module.vpc.vpc_id<br>    instance_type_nodes = var.instance_type_nodes<br>    desired_nodes = var.desired_nodes<br>    eks_userarn = local.eks_userarn<br>    eks_username = local.eks_username<br>    tags = local.common_tags<br>}</pre><pre>module &quot;vpc&quot; {<br>    source = &quot;./modules/vpc&quot;<br>    vpc_name = local.vpc_name<br>    azs = data.aws_availability_zones.available.names<br>    cluster_name = local.cluster_name<br>}</pre><pre>module &quot;sg&quot; {<br>    source = &quot;./modules/sg&quot;<br>    sg_name = &quot;eks_cluster_sg&quot;<br>    vpc_id = module.vpc.vpc_id<br>}</pre><h3>Pipeline</h3><p>We will call terraform steps from bash script, that’s why we need to add a bash file to repository which you can see below.</p><pre>#!/bin/bash</pre><pre>function init(){<br>  export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}<br>  export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}<br>  export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)<br>  export AWS_REGION=&quot;eu-central-1&quot;<br>  export CLUSTER_NAME=&quot;eks_cluster&quot;<br>}</pre><pre>function associate_oidc(){<br>  curl --silent --location &quot;<a href="https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname">https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname</a> -s)_amd64.tar.gz&quot; | tar xz -C /tmp<br>  mv /tmp/eksctl /usr/local/bin<br>  eksctl version<br>  <br>  eksctl utils associate-iam-oidc-provider \<br>      --region $AWS_REGION \<br>      --cluster $CLUSTER_NAME \<br>      --approve<br>}</pre><pre>function initTerraform(){<br>  tfenv use 1.0.0<br>  cd terraform || exit 1</pre><pre>export STATEKEY_FILENAME=&quot;EKS.state&quot;<br>  export TF_VAR_pipeline_tag=$CI_PIPELINE_URL<br>  export TF_VAR_environment=$ENVIRONMENT<br>  terraform init -backend-config=&quot;bucket=eks-cluster-state-bucket&quot;  \<br>                 -backend-config=&quot;key=$STATEKEY_FILENAME&quot; \<br>                 -backend-config=&quot;region=$AWS_REGION&quot;<br>}</pre><pre>function plan(){<br>  initTerraform;<br>  terraform plan<br>}</pre><pre>function apply(){<br>  initTerraform;<br>  terraform apply -auto-approve=true<br>}</pre><pre>function destroy(){<br>  initTerraform;<br>  terraform destroy -auto-approve<br>}</pre><pre>function main() {<br>  init;<br>  if [ &quot;$1&quot; == &quot;plan&quot; ];then<br>    plan;<br>  elif [ &quot;$1&quot; == &quot;apply&quot; ];then<br>    apply;<br>  elif [ &quot;$1&quot; == &quot;destroy&quot; ];then<br>    destroy;<br>  fi<br>}</pre><pre>main $1</pre><p>Since we need a bucket for state file and we do not need to create it every time, let’s create the S3 bucket manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*CqFFLu4W0d1eBkyeqGpOHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*hIq0pqdjezCj3dUSN5dKVQ.png" /></figure><p>To access AWS, we need to create a user with administrator access.</p><p><em>Using a role can be a better option if you have security concerns.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6gh329bUvZlqy6hYQSrkA.png" /></figure><p>We will add Access Key ID and Secret Access Key to Gitlab Variables because these values are used in bash script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*HYEwm7aokVZYQUMhnmbUYw.png" /></figure><p>Pipeline with terraform steps can be seen below:</p><pre>image: &lt;runner_image&gt;</pre><pre>stages:<br>  - plan<br>  - apply<br>  - destroy</pre><pre>.terraform_plan: &amp;terraform_plan<br>  when: manual<br>  script:<br>    - chmod +x ./run.sh<br>    - ./run.sh plan<br>  tags:<br>    - &lt;runner_tag&gt;</pre><pre>.terraform_apply: &amp;terraform_apply<br>  when: manual<br>  script:<br>    - chmod +x ./run.sh<br>    - ./run.sh apply<br>  tags:<br>    - &lt;runner_tag&gt;</pre><pre>.terraform_destroy: &amp;terraform_destroy<br>  when: manual<br>  script:<br>    - chmod +x ./run.sh<br>    - ./run.sh destroy<br>  tags:<br>    - &lt;runner_tag&gt;</pre><pre>1-terraform-plan:<br>  stage: plan<br>  &lt;&lt;: *terraform_plan</pre><pre>2-terraform-apply:<br>  stage: apply<br>  &lt;&lt;: *terraform_apply</pre><pre>3-terraform-destroy:<br>  stage: destroy<br>  &lt;&lt;: *terraform_destroy</pre><p>Now, we are ready to run the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*qT4dtuJ2Qrd81uum1msSUQ.png" /></figure><p>After pipeline finishes successfully, we can check our nodes with cli.</p><p>First, run<strong> “aws configure”</strong> command to login.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*LpKqvVy1yUkC33vLD-fF8w.png" /></figure><p>Then, to add context to your local environment, run the command below.</p><pre>aws eks --region &lt;AWS_REGION&gt; update-kubeconfig --name &lt;CLUSTERNAME&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*dS8ExG1nGMoiQqNk5mwiyg.png" /></figure><p>Now, we can list our nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*CNo5OLRCzk-mGg8XE78GMA.png" /></figure><p>Our EKS cluster is ready!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*BZLTgMpul8dNdg9_C-Bj2Q.gif" /></figure><p>Cluster logs that we defined in EKS module can be seen in <strong>CloudWatch</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kO2a5dGSmZeuKqRKi420dw.png" /></figure><h3>Sample Applications</h3><p>Let’s deploy <strong>Prometheus</strong>, <strong>Grafana </strong>with helm and <strong>Fluent-Bit</strong> with kubectl to our cluster.</p><p><em>We will deploy Fluent-Bit to send logs to </em><strong><em>CloudWatch</em></strong><em>.</em></p><p>First, we will create namespace manually because it is needed only <strong>once</strong>.</p><pre>kubectl create ns monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*pU0nKFTidm72PeY5DkDx2g.png" /></figure><h4><strong>Pipeline</strong></h4><pre>image: &lt;runner_image&gt;</pre><pre>stages:<br>  - install<br>  - update<br>  - destroy</pre><pre>.install_app: &amp;install_app<br>  when: manual<br>  script:<br>    - chmod +x ./run.sh<br>    - ./run.sh install<br>  tags:<br>    - &lt;runner_tag&gt;</pre><pre>.update_app: &amp;update_app<br>  when: manual<br>  script:<br>    - chmod +x ./run.sh<br>    - ./run.sh update<br>  tags:<br>    - &lt;runner_tag&gt;</pre><pre>.destroy_app: &amp;destroy_app<br>  when: manual<br>  script:<br>    - chmod +x ./run.sh<br>    - ./run.sh destroy<br>  tags:<br>    - &lt;runner_tag&gt;</pre><pre>1-install_app:<br>  stage: install<br>  &lt;&lt;: *install_app</pre><pre>2-update_app:<br>  stage: update<br>  &lt;&lt;: *update_app</pre><pre>3-destroy_app:<br>  stage: destroy<br>  &lt;&lt;: *destroy_app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*icprHehvjmpEuBq5aPqxFQ.png" /></figure><p>After pipeline finishes, we can check our pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*x83UssVTYAHgLIIfcWZ5AA.png" /></figure><p><strong>Grafana</strong></p><p>Let’s expose Grafana locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*cDIkQfbG1NzXVtr8c7Ak2Q.png" /></figure><p>Now, we can reach Grafana from <strong>localhost:3000</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORATPvPTUMZjCRjxRG-tqw.png" /></figure><p><strong>Fluent-Bit</strong></p><p>Let’s check AWS Cloudwatch if our logs exist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*-BJ98g0PBSlLhmVk2Uawsw.png" /></figure><p>Here they are!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*VoBZ1boLiXdMGpWvfmdaMQ.gif" /></figure><p>I hope this article is useful and you like it! Feel free to contact me for any questions or contributions!</p><ul><li><a href="https://github.com/nevincansel/eks_cluster_terraform">GitHub - nevincansel/eks_cluster_terraform</a></li><li><a href="https://github.com/nevincansel/eks_monitoring">GitHub - nevincansel/eks_monitoring</a></li><li><a href="https://registry.terraform.io/browse/modules">Terraform Registry</a></li><li><a href="https://docs.gitlab.com/ee/ci/">Get started with GitLab CI/CD | GitLab</a></li><li><a href="https://aws.amazon.com/">Cloud Computing Services - Amazon Web Services (AWS)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=589fed43dc22" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mirroring Git Repositories]]></title>
            <link>https://medium.com/kocsistem/mirroring-git-repositories-1376bd5cdfb2?source=rss-853bf1b442b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/1376bd5cdfb2</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[repositories]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[mirror]]></category>
            <dc:creator><![CDATA[Nevin Cansel Efendioglu]]></dc:creator>
            <pubDate>Sun, 13 Mar 2022 19:25:51 GMT</pubDate>
            <atom:updated>2022-03-13T19:49:36.392Z</atom:updated>
            <content:encoded><![CDATA[<p>In the development world, repositories are treasures that we have because we keep all of our works in there. However, sometimes we need to use more than one repository and there might be a requirement that synchronizing two repositories.</p><p>We had the exact same situation for the one of our projects and this question popped into my mind. How can i synchronize my repositories which are in different locations?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*mxL9Z2I7qWhVEHOyWPsZyg.gif" /></figure><p>I wrote a Powershell script to mirror two repositories and it worked but what if it would be an extension and everyone who needs could use that?</p><p>That would be awesome and here is the result!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*1qq-T35xaSpKkO3_RyM63Q.png" /></figure><h4>What does this extension do?</h4><p>This extension synchronizes repositories by running on Azure DevOps Pipelines.</p><p>You can find the <a href="https://marketplace.visualstudio.com/items?itemName=NevinCansel.mirror-repos"><strong>extension </strong></a>in Visual Studio Marketplace.</p><h4>How does it work for Cloud?</h4><p>These are source and destination repositories which are located in Github and Azure DevOps Cloud.</p><p><strong>Source: Github</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*vPRdtL-GWzrN-3tmNs8D3g.png" /></figure><p><strong>Destination: Azure Cloud</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4_gTnOZw_slbZjb_6uFqrw.png" /></figure><p>Fields of the extension is set to <strong>Cloud </strong>as default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CExS4vOAN-y9aH0OoKUrBQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*csMHpPeTWpGztzx1ziA2BQ.png" /></figure><p>I added personal access tokens as <strong>variables</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*Aq3l3SA9Ild-sTp82vAlZw.png" /></figure><p>Now, I can run the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*PFWRcA4E_ebR4L8zY2Ja5g.png" /></figure><p><strong><em>Warning:</em></strong><em> Since the extension is written in Powershell, it must be run in </em><strong><em>Windows Agents</em></strong><em>.</em></p><p>My pipeline finished successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*APD1B7JJ8F4jw6N0Fujnlg.png" /></figure><p>Now, let’s check the <strong>destination </strong>repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*hYaNBANva0th_oXJmg3cWQ.png" /></figure><p>It worked!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*3yH4NZBM93yeVu8dYzjaGA.gif" /></figure><h4>How does it work for on-prem?</h4><p>Create personal access token for destination and source repositories. I will synchronize Azure Repos on-prem and cloud as an example.</p><p>These are source and destination repositories.</p><p><strong>Source: Azure On-Prem</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*wUddHQotDNPLtPtpGPB8ZQ.png" /></figure><p><strong>Destination: Azure Cloud</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*-N_M4PlDZV9nT17XzoI5Cg.png" /></figure><p>This time, I will use classic pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*XfIFASWCr0jGWgms5elbMw.png" /></figure><p>I deselected the first option because my source is not on <strong>Cloud</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*dBau7bpLHT8sH2Eud0LHDw.png" /></figure><p>Username, password and PAT are saved in <strong>variables</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*pBFjLU-nFlXMMIAA4SbjQQ.png" /></figure><p>Now, I can run the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*rgfvILJv3BFwJ3LJ3m0Ffw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*mRupC27ouTvNgjuzDo3iMg.png" /></figure><p>Now, let’s check the <strong>destination </strong>repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*dWyY_SYpZlsZNHpx33kGsg.png" /></figure><p>It worked again!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*3yH4NZBM93yeVu8dYzjaGA.gif" /></figure><p><strong>Recommendations</strong></p><ul><li>If you create the pipeline in your source repository, you can add <strong>trigger </strong>to pipeline to synchronize the destination repository if there is a new commit.</li><li>You can <strong>schedule </strong>your pipeline to sync your repositories in a designated period.</li><li><strong>Force Push</strong> and<strong> Create Branch</strong> permissions for repositories might be necessary in Azure DevOps.</li></ul><p>You can check the article below if you have an extension and need an easy way to publish it!</p><p><a href="https://medium.com/kocsistem/publish-azure-devops-extension-with-azure-devops-pipeline-10b2e3ae2c91">Publish Azure DevOps Extension with Azure DevOps Pipeline</a></p><p>Hope you find my article and extension useful! I would appreciate for any comments or contributions that you make!</p><p><a href="https://github.com/nevincansel/azuredevops-mirror-repositories">GitHub - nevincansel/azuredevops-mirror-repositories</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1376bd5cdfb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kocsistem/mirroring-git-repositories-1376bd5cdfb2">Mirroring Git Repositories</a> was originally published in <a href="https://medium.com/kocsistem">KoçSistem</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ELK Stack in Kubernetes Using Helm]]></title>
            <link>https://medium.com/@nevincansel/elk-stack-in-kubernetes-using-helm-52398564f7fc?source=rss-853bf1b442b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/52398564f7fc</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[elk]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Nevin Cansel Efendioglu]]></dc:creator>
            <pubDate>Wed, 05 Jan 2022 09:53:46 GMT</pubDate>
            <atom:updated>2024-01-10T12:29:15.911Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is ELK Stack?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*qcwAb1b2i-aK6_f49a0rwg.png" /></figure><blockquote>“ELK Stack is the leading open-source IT log management solution for companies who want the benefits of a centralized logging solution without the enterprise software price. Elasticsearch, Logstash, and Kibana when used together, form an end-to-end stack (ELK Stack) and real-time data analytics tool that provides actionable insights from almost any type of structured and unstructured data source.”</blockquote><h3>What are these tools and what are they doing?</h3><p><strong>Elasticsearch: </strong>Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.</p><p><strong>Logstash:</strong> Logstash is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination.</p><p><strong>Kibana: </strong>Kibana is an free and open frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch.</p><p><strong>Filebeat: </strong>Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> or <a href="https://www.elastic.co/products/logstash">Logstash</a> for indexing.</p><h3>Why do we use ELK?</h3><p>As benefits of ELK Stack, we can have a list as below.</p><ul><li>It is open-source and free.</li><li>It can securely pull, analyze and visualize data, in real time, from any source and format.</li><li>It is simple to set up and user friendly.</li><li>It can perform centralized logging to help identify any server and application-related issues across multiple servers and correlate the logs in a particular time frame.</li><li>It is a total log-analysis platform for search, analyses and visualization of log-generated data from different machine.</li></ul><h3>Configuration of ELK Stack</h3><p>Firstly, we will import <a href="https://github.com/elastic/helm-charts">helm charts</a> to our repository.</p><p><em>Helm is a Kubernetes package and operations manager and Helm Charts are used to deploy an application, or one component of a larger application.</em></p><p><strong>Elasticsearch</strong></p><p>Since we have a helm chart, we do not need to do many things but if you want to use elastic search without username and password like me, you need to add following line under elastic.yml in values.yaml</p><pre>esConfig:<br>  elasticsearch.yml: |<br>    xpack.security.enabled: false</pre><p>Also, I wanted to access elasticsearch from browser so I configured services as NodePort.</p><p>To do that, you need to change the configuration of service in values.yaml as following.</p><pre>service:<br>  enabled: true<br>  labels: {}<br>  labelsHeadless: {}<br>  type: NodePort<br>  nodePort: &quot;30234&quot;<br>  annotations: {}<br>  httpPortName: http<br>  transportPortName: transport<br>  loadBalancerIP: &quot;&quot;<br>  loadBalancerSourceRanges: []<br>  externalTrafficPolicy: &quot;&quot;</pre><p>Now, we can deploy it to our cluster. I used Azure Pipelines for deployment and here is my azure-pipelines.yml</p><pre>- task: HelmDeploy@0<br>  inputs:<br>    connectionType: &#39;Kubernetes Service Connection&#39;<br>    kubernetesServiceConnection: &#39;Kubernetes-Cluster&#39;<br>    namespace: &#39;monitoring&#39;<br>    command: &#39;install&#39;<br>    chartType: &#39;FilePath&#39;<br>    chartPath: &#39;$(Build.SourcesDirectory)/elasticsearch&#39;<br>    releaseName: &#39;elasticsearch&#39;<br>    valueFile: &#39;$(Build.SourcesDirectory)/elasticsearch/values.yaml&#39;</pre><p>After pipeline is finished, we can check resources in the cluster and its endpoint from browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*UVAsvaWU3amBTYQFqLmnXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VR-54E0H66lFOyiFiD0tQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*UVEGkeop5cFSuFTPHVjqyA.png" /></figure><p><strong>Logstash</strong></p><p>We need more touch for logstash because we will create pipeline for not only filebeat but also an external server.</p><p>Since we need to reach logstash from the external server, logstash service also must run as NodePort. To do that, we will edit service-headless.yaml under templates folder.</p><p><em>The reason why we edit the service-headless.yaml is logstash service will be created as headless by helm chart.</em></p><p>The only changes in service-headless.yaml is under spec.</p><pre>spec:<br> type: NodePort<br> selector:<br> app: “{{ template “logstash.fullname” . }}”<br> ports:<br> — name: http<br> port: {{ .Values.httpPort }}<br> nodePort: 31123<br> — name: tcp<br> port: 9500 <br> nodePort: 30123</pre><p><em>You may notice that I added another name, tcp. This will be used for external server’s pipeline.</em></p><p>There are some important changes in values.yaml.</p><p>First of all, we need to create separate pipelines for two different sources under logstashConfig.</p><pre>logstashConfig:<br>  logstash.yml: |<br>    http.host: 0.0.0.0<br>    pipeline.ecs_compatibility: disabled<br>  pipelines.yml: |<br>    - pipeline.id: logstash<br>      path.config: &quot;/usr/share/logstash/pipeline/logstash.conf&quot;<br>    - pipeline.id: devopsdashboard<br>      path.config: &quot;/usr/share/logstash/pipeline/devopsdashboard.conf&quot;<br>      pipeline.workers: 3<br>  log4j2.properties: |<br>    logger.logstashpipeline.name = logstash.inputs.beats<br>    logger.logstashpipeline.level = error</pre><p>(Optional) In the configuration above, I added a line for ecs_compability because there were too many logs as warning. Also edited log4j properties to collect only error logs from beats.</p><p>Then, we will create our pipelines under logstashPipeline.</p><pre>logstashPipeline:<br>  logstash.conf: |<br>    input {<br>      beats {<br>        port =&gt; 5044<br>      }<br>    }<br>    filter {<br>    }<br>    output {<br>      elasticsearch {<br>        index =&gt; &quot;%{[<a href="http://twitter.com/metadata">@metadata</a>][beat]}-%{+YYYY.MM.dd}&quot;<br>        hosts =&gt; [ &quot;elasticsearch-master:9200&quot; ]<br>      }<br>    }<br>  devopsdashboard.conf: |<br>    input {<br>      tcp {<br>        port =&gt; 9500<br>        codec =&gt; json<br>      }<br>    }<br>    filter {<br>        }<br>    }<br>    output {<br>      elasticsearch {<br>        index =&gt; &quot;devopsdashboard-%{+YYYY.MM.dd}&quot;<br>        hosts =&gt; [ &quot;elasticsearch-master:9200&quot; ]<br>      }<br>    }</pre><p>Filebeat will send logs to port 5044 and the external server will send logs to port 30123 which routes to 9500 in the cluster.</p><p>And finally I added extraPorts for logstash under extraPorts in values.yaml.</p><pre>extraPorts:<br>  - name: tcpport<br>    containerPort: 9500</pre><p>Now, we can deploy it to the cluster.</p><pre>- task: HelmDeploy@0<br>  displayName: Helm Install<br>  inputs:<br>    connectionType: &#39;Kubernetes Service Connection&#39;<br>    kubernetesServiceConnection: &#39;Kubernetes-Cluster&#39;<br>    namespace: &#39;monitoring&#39;<br>    command: &#39;install&#39;<br>    chartType: &#39;FilePath&#39;<br>    chartPath: &#39;$(Build.SourcesDirectory)/logstash&#39;<br>    releaseName: &#39;logstash&#39;<br>    valueFile: &#39;$(Build.SourcesDirectory)/logstash/values.yaml&#39;</pre><p>Let’s check our resources and endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*x_yC8W5UAcf2KYubjcYacA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AII6klJEZIAmj5vUPXGwcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKsYCRuOTfu5u_awDxPgcQ.png" /></figure><p><strong>Filebeat</strong></p><p>We need to change output with logstash in values.yaml. Here is filebeatConfig.</p><pre>filebeatConfig:<br>    filebeat.yml: |<br>      filebeat.inputs:<br>      - type: container<br>        paths:<br>          - /var/log/containers/*.log<br>        processors:<br>        - add_kubernetes_metadata:<br>            host: ${NODE_NAME}<br>            matchers:<br>            - logs_path:<br>                logs_path: &quot;/var/log/containers/&quot;<br>      logging.level: error<br>      output.logstash:<br>        hosts: [&quot;logstash-logstash-0.logstash-logstash-headless.monitoring.svc.cluster.local:5044&quot;]</pre><p>Let’s deploy it.</p><pre>- task: HelmDeploy@0<br>  displayName: Helm Install<br>  inputs:<br>    connectionType: &#39;Kubernetes Service Connection&#39;<br>    kubernetesServiceConnection: &#39;Kubernetes-Cluster&#39;<br>    namespace: &#39;monitoring&#39;<br>    command: &#39;install&#39;<br>    chartType: &#39;FilePath&#39;<br>    chartPath: &#39;$(Build.SourcesDirectory)/filebeat&#39;<br>    releaseName: &#39;filebeat&#39;<br>    valueFile: &#39;$(Build.SourcesDirectory)/filebeat/values.yaml&#39;</pre><p>After pipelines is done, we can check our pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*fxtmCMZn7H_XEIKBlqbUkQ.png" /></figure><p><strong>Kibana</strong></p><p>To access Kibana through browser with a secure endpoint, we will configure ingress in values.yaml</p><pre>ingress:<br>  enabled: false<br>  className: &quot;nginx&quot;<br>  pathtype: ImplementationSpecific<br>  annotations:<br>   kubernetes.io/ingress.class: nginx<br>  hosts:<br>    - host: your_host<br>      paths:<br>        - backend:<br>      serviceName: kibana-kibana<br>   servicePort: 5601<br>  tls: <br>    - secretName: your_secret<br>      hosts:<br>        - your_host</pre><p>Now, we can deploy Kibana.</p><pre>- task: HelmDeploy@0<br>  inputs:<br>    connectionType: &#39;Kubernetes Service Connection&#39;<br>    kubernetesServiceConnection: &#39;KubeOps-Cluster&#39;<br>    namespace: &#39;monitoring&#39;<br>    command: &#39;install&#39;<br>    chartType: &#39;FilePath&#39;<br>    chartPath: &#39;$(Build.SourcesDirectory)/kibana&#39;<br>    releaseName: &#39;kibana&#39;<br>    valueFile: &#39;$(Build.SourcesDirectory)/kibana/values.yaml&#39;</pre><p>Here our Kibana is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XiF0zqFx1u4jNs3CG7QbvA.png" /></figure><h3>Send Logs from External Server to Logstash</h3><p>In the external server, I use NLog for logging and configured it for my new Logstash intance. You can edit the following line to send logs to Logstash via TCP.</p><pre>&lt;target name=&quot;logstashException&quot; xsi:type=&quot;Network&quot; address=&quot;tcp://your_cluster_ip:30123&quot; newLine=&quot;true&quot;&gt;</pre><p><strong><em>Warning</em></strong><em>: Do not forget to define firewall rules to reach Logstash.</em></p><h3><strong>View Logs in Kibana</strong></h3><p>We need to define index patterns to be able to see logs in Kibana.</p><p>From homepage, write Kibana / Index Patterns to search bar. Go to Index patterns page and click to Create index pattern on the right corner. You will see the list of index patterns here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Er1O_pggwr5t227DEk5FA.png" /></figure><p>Add the following patterns four your indices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCISIcBoH_afDc6yWC_tOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQJQvRWxr-BFvf99EN-S9Q.png" /></figure><p>Now, I have two index patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K5PwmkapuNP9m_MoJYFivQ.png" /></figure><p>From left side menu, click to discover and check your logs :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0CFnI658V0QL3j20y5nAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6dE-b8orbebCEu8beVz0Q.png" /></figure><p>🌿 Hope you find my article helpful! If you want to support me, you can click the picture 👇 Have a wonderful day!</p><figure><a href="https://www.buymeacoffee.com/nevincansel"><img alt="" src="https://cdn-images-1.medium.com/max/545/1*9R5ZBFeoBqwUsHqTALB6EQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52398564f7fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Containerized Build Agents with Azure DevOps Pipelines]]></title>
            <link>https://medium.com/@nevincansel/create-containerized-build-agents-with-azure-devops-pipelines-e9384b3307de?source=rss-853bf1b442b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9384b3307de</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Nevin Cansel Efendioglu]]></dc:creator>
            <pubDate>Mon, 20 Sep 2021 06:42:36 GMT</pubDate>
            <atom:updated>2024-01-10T13:36:01.875Z</atom:updated>
            <content:encoded><![CDATA[<p>New Agents with different requirements are needed if you work with different projects. What if you could create an agent without connecting to the server itself? I think that would be easy and fast. That is why I wrote this article to show how we can create a containerized build agent with Azure DevOps Pipelines. Let’s begin.</p><p>We have an empty server (Ubuntu 20.04 LTS) and docker-compose is needed to create an agent. To install docker-compose, docker is required.</p><p><strong>Install Docker:</strong></p><p>Add Docker’s official GPG key:</p><pre>curl -fsSL <a href="https://download.docker.com/linux/ubuntu/gpg">https://download.docker.com/linux/ubuntu/gpg</a> | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg</pre><p>Use the following command to set up the <strong>stable</strong> repository.</p><pre>echo \<br>  &quot;deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \<br>  $(lsb_release -cs) stable&quot; | sudo tee /etc/apt/sources.list.d/docker.list &gt; /dev/null</pre><p>Install Docker Engine and Docker Compose:</p><pre>sudo apt-get update<br>sudo apt-get install docker-ce docker-ce-cli containerd.io<br>sudo apt-get install docker-compose</pre><p>We also will install Azure-CLI to login Azure and you will see why we need this later.</p><p><strong>Install Azure-CLI:</strong></p><pre>sudo apt install azure-cli</pre><p>We need a Docker image to create our first Agent. Here is a sample:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/665e8fb595f176981d795d75dfef1fd1/href">https://medium.com/media/665e8fb595f176981d795d75dfef1fd1/href</a></iframe><p>You can create and push your image to your registry manually but I will use pipeline to do it.</p><p>If you don’t have a service connection to your registry, create it first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Mzs0MP2jhTQsyilJlP6RLA.jpeg" /></figure><p><strong>Pipeline for Docker Image:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X0pHzIcPoXbn2s1BMUqEAA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2df5eecbd9963d18f50821c6f54c0550/href">https://medium.com/media/2df5eecbd9963d18f50821c6f54c0550/href</a></iframe><p>When the pipeline is finished, check your repository to see if your image is there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-nHR_qsWHm2AiQlfBcOAQ.png" /></figure><p>The next step is creating an agent with this image. But first, create a new Agent Pool to locate your new agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DON43ByHuZJr463-QVxFKg.png" /></figure><p>Now, we can connect to the server and create a directory for volumes.</p><pre>mkdir Azure-AgentPool<br>cd Azure-AgentPool<br>mkdir Agent-1<br>cd Agent-1<br>mkdir data<br>sudo vi docker-compose.yml</pre><p>Sample of docker-compose.yml:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fd41a0643c46be39c919f230160db05/href">https://medium.com/media/9fd41a0643c46be39c919f230160db05/href</a></iframe><p>To pull image from Azure Container Registry, we need to connect it from our server.</p><pre>az login<br>az acr login -n &lt;your registry&gt;</pre><p>If you get an error while logging in to <strong>acr </strong>like below, run sudo chmod 666 /var/run/docker.sock</p><blockquote>An error occurred: DOCKER_COMMAND_ERROR<br>Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get “<a href="http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json">http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json</a>&quot;: dial unix /var/run/docker.sock: connect: permission denied</blockquote><p>After you logged in successfully, it is time to create the first build agent.</p><pre>docker-compose up -d</pre><p>Check if your container is healthy.</p><pre>docker ps</pre><p>Now, you can check the Agent Pools to see your first agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kMIrO044fYvW8DROIw8BTg.png" /></figure><p>Since we have an agent in our server, we can create a pipeline to create another agent automatically.</p><p>You will need a <strong>start.sh</strong> file to create agent within the pipeline. Add this file to your repository.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d8aceebd14d7e3f796a1ed6d52a3b11/href">https://medium.com/media/2d8aceebd14d7e3f796a1ed6d52a3b11/href</a></iframe><p>We already have a docker-compose.yml. Add that in your repository too.</p><p><em>Do not forget to change Container Name, Agent Name and Folder under volumes.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fd41a0643c46be39c919f230160db05/href">https://medium.com/media/9fd41a0643c46be39c919f230160db05/href</a></iframe><p><strong>Pipeline for Containerized Build Agent:</strong></p><p><em>Attention to what is written as pool. We use our new Agent Pool to create another agent.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fbf40b0d7ff112ce6050154a5fda97d/href">https://medium.com/media/2fbf40b0d7ff112ce6050154a5fda97d/href</a></iframe><p>After pipeline is finished, check your Agent Pool to see if the new agent is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*hsZmzXc7yvbRdXmeVfLcWA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PPQMJEI8aCbnqHyzt2ih7w.png" /></figure><p>It took only 3 seconds to create another agent! How easy and fast!</p><p><strong>— — — — — — — — — — — — — — BONUS— — — — — — — — — — — — — —</strong></p><p>If you do not want to write your token into your docker-compose.yml, you can use variables instead.</p><p><strong>Pipeline:</strong></p><pre>trigger:<br>- master</pre><pre>pool:<br>  name: AzureAgents</pre><pre>steps:</pre><pre>- task: Bash@3<br>    displayName: Set Variables<br>    inputs:<br>      targetType: &#39;inline&#39;<br>      script: |<br>        sed -i -r &quot;s/AZP_TOKEN/AZP_TOKEN=$(AZP_TOKEN)/g&quot; $(Build.SourcesDirectory)/docker-compose.yml</pre><pre>- task: DockerCompose@0<br>    inputs:<br>      containerregistrytype: &#39;Azure Container Registry&#39;<br>      azureContainerRegistry: &lt;your registry&gt;<br>      dockerComposeFile: &#39;**/docker-compose.yml&#39;<br>      action: &#39;Run a Docker Compose command&#39;<br>      dockerComposeCommand: &#39;up -d&#39;</pre><p><strong>docker-compose.yml:</strong></p><p><em>Do not assign any values to AZP_TOKEN.</em></p><pre>version: &#39;3.3&#39;<br>services:<br>  linux-agent3:<br>    image: azureagent.azurecr.io/azureagent:6<br>    container_name: AzureAgent2<br>    restart: always<br>    networks:<br>      - agentnet<br>    environment:<br>      - AZP_URL=<a href="https://dev.azure.com/nevincansel5517">https://dev.azure.com/nevincansel5517</a><br>      - AZP_TOKEN<br>      - AZP_AGENT_NAME=AzureAgent2<br>      - AZP_POOL=AzureAgents<br>    volumes:<br>      - /var/run/docker.sock:/var/run/docker.sock<br>      - /home/cansel/Azure-AgentPool/Agent-2/data:/azp/agent/<br>networks:<br>  agentnet:<br>    driver: bridge</pre><p><strong>Variables:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZxkV90d2--PoyjxJIXTvQ.png" /></figure><p><a href="https://github.com/nevincansel/azureagent/tree/main">https://github.com/nevincansel/azureagent/</a></p><p>🌿 Hope you enjoy the article! If you want to support me, you can click the picture 👇 Have a wonderful day!</p><figure><a href="https://www.buymeacoffee.com/nevincansel"><img alt="" src="https://cdn-images-1.medium.com/max/545/1*9R5ZBFeoBqwUsHqTALB6EQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9384b3307de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Publish Azure DevOps Extension with Azure DevOps Pipeline]]></title>
            <link>https://medium.com/kocsistem/publish-azure-devops-extension-with-azure-devops-pipeline-10b2e3ae2c91?source=rss-853bf1b442b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/10b2e3ae2c91</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[extension]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[azure-devops]]></category>
            <dc:creator><![CDATA[Nevin Cansel Efendioglu]]></dc:creator>
            <pubDate>Tue, 27 Jul 2021 14:32:49 GMT</pubDate>
            <atom:updated>2021-07-27T14:32:49.827Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*qJI-rUtYPtBV47h8ObN4qw.png" /></figure><p>Azure DevOps Extensions can be packaged and published by using Azure Pipelines. There is a free extension that provides build and release tasks for packaging and publishing Azure DevOps Extensions available in Visual Studio Marketplace.</p><p><a href="https://marketplace.visualstudio.com/items?itemName=ms-devlabs.vsts-developer-tools-build-tasks">Azure DevOps Extension Tasks</a> are made by “Microsoft DevLabs” for constructing Azure DevOps extensions.</p><p>First, we need to <a href="https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?WT.mc_id=DOP-MVP-5001511&amp;view=azure-devops&amp;tabs=preview-page">create a personal access token</a> to publish our extension to Visual Studio Marketplace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*buzyuJ3DoG56mIWM4vlU0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*YsP4TBARr3Zh5AHan-GnaA.png" /></figure><p><em>Personal access token must be created in Azure DevOps cloud even though you have an Azure DevOps Server. Otherwise, publish extension step will fail.</em></p><p>I assume that you already pushed your code to your repository but do not forget to configure your trigger so that when you push a change, new version of your extension will be automatically published.</p><p>Let’s start to construct our pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*qJI-rUtYPtBV47h8ObN4qw.png" /></figure><p>We will have four steps to publish our extension.</p><p><strong>1. Use Node CLI for Azure DevOps (tfx-cli)</strong></p><p>This step installs the Node CLI for Azure DevOps (tfx-cli) on your agent. We need this step to package and publish our extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*kiGqiWacc4zAvaGMkOq6gQ.png" /></figure><p><strong>2. Query Extension Version</strong></p><p>This step will query the current version from the Visual Studio Market Place. We need this step to increase our version number for each build.</p><p>To connect to Visual Studio Marketplace, we will create a connection by clicking new.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*C5iRA0ZhheFOSGksHW6FuA.png" /></figure><p>We will enter our PAT we created before and name our connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*lgxqxOtEMGBvvTuI5AaOpw.png" /></figure><p>Then, we will enter other information which is needed.</p><ul><li>You can choose Major, Minor or Patch for <strong><em>Increase version</em></strong> field. We prefer to increase our version as patch. If our current version 1.0.1, it will be 1.0.2.</li><li><strong><em>Reference name</em></strong> under Output Variables is important! If you don’t give a reference name, variable will not be used. We used adx as AzureDevOpsExtension. You can use whatever you want.</li></ul><p>We increased our version to 1.0.2 above and assign it to our variable <em>adx.Extension.Version</em> to use it while packaging. Why do we need this? Look at the Package Extension step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*LBX_HRGzMoCz9KATTzBKDQ.png" /></figure><p><strong>3. Package Extension</strong></p><p>This step package an Azure DevOps extension into a VSIX file by using our manifest file vss-extension.json.</p><p>We use our variable which we created in the second step to declare our new version number. Otherwise package will be created with the version number in vss-extension.json and there will be a conflict with the current version in the marketplace so that our pipeline will fail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*W5m2_5QEIi1blojixjtozA.png" /></figure><p><strong>4. Publish Extension</strong></p><p>We will use the same connection we created in the second step to connect Visual Studio Marketplace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*grvPJNl24O3PEdw6R98kdg.png" /></figure><p>If your extension is private and you want to share it with other organizations, you can write organization names in the <strong><em>Share with</em></strong> field. This field will be ignored if you choose public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJS6OH5MScYlVr-lrfSdNg.png" /></figure><p>Now, our pipeline is ready to publish!</p><p>Go, make a change and commit! See how fast your extension will be published in the marketplace.</p><p>Hope you enjoy the article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10b2e3ae2c91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kocsistem/publish-azure-devops-extension-with-azure-devops-pipeline-10b2e3ae2c91">Publish Azure DevOps Extension with Azure DevOps Pipeline</a> was originally published in <a href="https://medium.com/kocsistem">KoçSistem</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>