<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Walaa Elgenidy on Medium]]></title>
        <description><![CDATA[Stories by Walaa Elgenidy on Medium]]></description>
        <link>https://medium.com/@walaaelgenidy?source=rss-90eb4fdc372d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FUIlRgt6lMZJ1mTL6D800A.jpeg</url>
            <title>Stories by Walaa Elgenidy on Medium</title>
            <link>https://medium.com/@walaaelgenidy?source=rss-90eb4fdc372d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:10:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@walaaelgenidy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Configure Alertmanager Email Notifications Using AWS SNS Without Putting SMTP Credentials in…]]></title>
            <link>https://medium.com/@walaaelgenidy/how-to-configure-alertmanager-email-notifications-using-aws-sns-without-putting-smtp-credentials-in-8d8f3e1a6dee?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d8f3e1a6dee</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[alertmanager]]></category>
            <category><![CDATA[push-notification]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[aws-sns]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Sat, 26 Sep 2026 18:47:39 GMT</pubDate>
            <atom:updated>2026-09-26T21:08:19.501Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Configure Alertmanager Email Notifications Using AWS SNS Without Putting SMTP Credentials in Kubernetes .</h3><p>How we connected production Prometheus Alertmanager to AWS SNS, Microsoft Teams, and SMS while keeping email noise under control.</p><p>There is a deceptively simple request that appears in almost every operations team:</p><p>“Can we also send only the important alerts by email?”</p><p>At first, this sounds like a small configuration change. Add SMTP settings, add a recipient list, reload Alertmanager, and move on.</p><p>In a real Kubernetes environment, the difficult part is not sending an email. The difficult part is deciding which alerts deserve email, managing recipients without creating configuration drift, and giving Alertmanager AWS access without putting long-lived credentials in a pod.</p><p>This article explains the design we used for a production EKS tooling cluster. Alertmanager continues to send notifications to Microsoft Teams and SMS. Critical alerts also go to an AWS SNS topic, and SNS distributes them to confirmed email subscribers.</p><pre>Prometheus / Grafana alert rules<br>              |<br>              v<br>         Alertmanager<br>          /    |    \<br>         /     |     \<br>        v      v      v<br> Teams   AWS SNS    SMS adapter<br>           |<br>           v<br>   Confirmed email subscribers</pre><p>The important policy is simple:</p><blockquote><strong>Only severity=critical alerts generate email.</strong></blockquote><p>That rule turned out to be more important than the email integration itself.</p><h3>Why SNS instead of SMTP?</h3><p>We considered the usual Alertmanager SMTP configuration. It would have worked, but it would also have introduced another credential and another piece of infrastructure that the Kubernetes team would need to operate.</p><p><strong>Using AWS SNS gave us three useful properties:</strong></p><blockquote>1.Alertmanager only needs permission to publish to one SNS topic.</blockquote><blockquote>2.The Operations recipient list is managed centrally with Terraform.</blockquote><blockquote>3.SNS handles email delivery and subscription confirmation.</blockquote><p>The Kubernetes workload does not store an SMTP password or an AWS access key. It obtains AWS permissions through Amazon EKS Pod Identity. That keeps the trust boundary small: the Alertmanager ServiceAccount can publish only to the one topic used for these notifications.</p><p>This is the resulting design:</p><pre>Alertmanager<br>    |<br>    | sns:Publish<br>    v<br>SNS topic: alertmanager-email<br>    |<br>    +--&gt; Confirmed Operations email subscriber 1<br>    +--&gt; Confirmed Operations email subscriber 2<br>    +--&gt; Confirmed Operations distribution list</pre><h3>The environment used in this example</h3><p>The original implementation ran in a production tooling environment. The identifiers below are examples from that environment. Replace them with your own values before applying anything.</p><pre>AWS Region:          eu-central-2<br>AWS Account:         123456789012<br>EKS Cluster:         tooling<br>Kubernetes Namespace: monitoring<br>Alertmanager release: alertmanager<br>ServiceAccount:      alertmanager</pre><p>The account number above is deliberately a dummy value for publication. Do not copy it into your own deployment.</p><h3>The first design decision: email only critical alerts</h3><p><strong>Email is a high-attention channel. If every warning and informational event creates an email, people quickly stop reading the mailbox.</strong></p><p>We kept the routing policy explicit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*uavhzW5uLlj_Iwx_DnZYkQ.png" /><figcaption><strong>The SNS route must match critical severity exactly. Do not make the SNS route a catch-all route.</strong></figcaption></figure><h3>Terraform: create the SNS topic and subscriptions</h3><p>The first part of the implementation creates the topic and its email subscriptions.</p><h3>Variables</h3><p>Create a variable for the recipients instead of embedding addresses directly in the SNS resource:</p><pre>variable &quot;aws_region&quot; {<br>  description = &quot;AWS region where the Alertmanager SNS topic is deployed.&quot;<br>  type        = string<br>  default     = &quot;eu-central-2&quot;<br>}<br><br>variable &quot;cluster_name&quot; {<br>  description = &quot;EKS cluster that runs Alertmanager.&quot;<br>  type        = string<br>}<br><br>variable &quot;alertmanager_namespace&quot; {<br>  description = &quot;Kubernetes namespace that contains Alertmanager.&quot;<br>  type        = string<br>  default     = &quot;monitoring&quot;<br>}<br><br>variable &quot;alertmanager_service_account&quot; {<br>  description = &quot;Kubernetes ServiceAccount used by Alertmanager.&quot;<br>  type        = string<br>  default     = &quot;alertmanager&quot;<br>}<br><br>variable &quot;alertmanager_email_destinations&quot; {<br>  description = &quot;Email addresses that must confirm their SNS subscriptions.&quot;<br>  type        = set(string)<br>  default     = []<br>}</pre><p>Keep actual recipient addresses in a protected Terraform variable file or your normal secret/configuration management process. For example:</p><pre># terraform.tfvars<br>cluster_name = &quot;tooling&quot;<br><br>alertmanager_email_destinations = [<br>  &quot;operations-example-1@example.com&quot;,<br>  &quot;operations-example-2@example.com&quot;<br>]</pre><p>Use distribution lists where your organization owns the membership centrally. That reduces the number of individual subscriptions and makes team ownership clearer.</p><p><strong>SNS topic and subscriptions:</strong></p><pre>resource &quot;aws_sns_topic&quot; &quot;alertmanager_email&quot; {<br>  name = &quot;tooling-alertmanager-email&quot;<br>}<br><br>resource &quot;aws_sns_topic_subscription&quot; &quot;alertmanager_email&quot; {<br>  for_each = var.alertmanager_email_destinations<br><br>  topic_arn = aws_sns_topic.alertmanager_email.arn<br>  protocol  = &quot;email&quot;<br>  endpoint  = each.value<br>}<br><br>output &quot;alertmanager_sns_topic_arn&quot; {<br>  description = &quot;SNS topic used by Alertmanager email notifications.&quot;<br>  value       = aws_sns_topic.alertmanager_email.arn<br>}</pre><p>Every email subscription starts in a pending state. SNS sends a confirmation message to the address. The recipient must click the confirmation link before receiving alerts.</p><p>That detail is easy to miss during testing. A Terraform apply can succeed while the mailbox still receives nothing because the subscription has not been confirmed.</p><h3>IAM: give Alertmanager one permission</h3><p>Alertmanager does not need broad AWS permissions. It needs one action against one resource:</p><pre>sns:Publish on the Alertmanager SNS topic</pre><h3>Trust policy for EKS Pod Identity</h3><p>EKS Pod Identity uses the pods.eks.amazonaws.com service principal:</p><pre>data &quot;aws_iam_policy_document&quot; &quot;alertmanager_pod_identity_trust&quot; {<br>  statement {<br>    sid    = &quot;AllowEksPodIdentity&quot;<br>    effect = &quot;Allow&quot;<br><br>    actions = [<br>      &quot;sts:AssumeRole&quot;,<br>      &quot;sts:TagSession&quot;<br>    ]<br><br>    principals {<br>      type        = &quot;Service&quot;<br>      identifiers = [&quot;pods.eks.amazonaws.com&quot;]<br>    }<br>  }<br>}</pre><p>IAM role and publish policy</p><pre>resource &quot;aws_iam_role&quot; &quot;alertmanager&quot; {<br>  name               = &quot;tooling-alertmanager&quot;<br>  assume_role_policy = data.aws_iam_policy_document.alertmanager_pod_identity_trust.json<br>}<br><br>data &quot;aws_iam_policy_document&quot; &quot;alertmanager_sns_publish&quot; {<br>  statement {<br>    sid    = &quot;PublishAlertmanagerNotifications&quot;<br>    effect = &quot;Allow&quot;<br><br>    actions = [<br>      &quot;sns:Publish&quot;<br>    ]<br><br>    resources = [aws_sns_topic.alertmanager_email.arn]<br>  }<br>}<br><br>resource &quot;aws_iam_role_policy&quot; &quot;alertmanager_sns_publish&quot; {<br>  name   = &quot;publish-alertmanager-sns&quot;<br>  role   = aws_iam_role.alertmanager.id<br>  policy = data.aws_iam_policy_document.alertmanager_sns_publish.json<br>}</pre><p>The effective permission is equivalent to:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;sns:Publish&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:sns:eu-central-1:123456789012:tooling-alertmanager-email&quot;<br>    }<br>  ]<br>}</pre><blockquote>The topic ARN is generated from Terraform rather than copied into multiple places.</blockquote><h3>EKS Pod Identity association</h3><p>Connect the IAM role to the exact Kubernetes workload:</p><pre>resource &quot;aws_eks_pod_identity_association&quot; &quot;alertmanager&quot; {<br>  cluster_name    = var.cluster_name<br>  namespace       = var.alertmanager_namespace<br>  service_account = var.alertmanager_service_account<br>  role_arn        = aws_iam_role.alertmanager.arn<br>}</pre><p>This is the part that prevents static AWS keys from appearing in the Alertmanager deployment.</p><p>The association must match all three values:</p><pre>Cluster<br>Namespace<br>ServiceAccount</pre><blockquote>A mismatch in any one of them can produce a confusing credential error inside the pod.</blockquote><h3>Alertmanager: configure the SNS receiver</h3><p>Alertmanager supports an SNS receiver with AWS Signature Version 4 authentication. The following example is written as a Helm values fragment. Adapt the surrounding route, receivers, and existing Teams/SMS configuration to your deployment.</p><pre>alertmanager:<br>  config:<br>    global:<br>      resolve_timeout: 5m<br><br>    route:<br>      group_by:<br>        - alertname<br>        - cluster<br>        - service<br><br>      group_wait: 30s<br>      group_interval: 5m<br>      repeat_interval: 4h<br>      receiver: teams-info<br><br>      routes:<br>        # Existing critical Teams route. continue allows the alert to<br>        # continue to the SNS route below.<br>        - matchers:<br>            - severity=&quot;critical&quot;<br>          receiver: teams-critical<br>          continue: true<br><br>        # Email is intentionally exact: critical only.<br>        - matchers:<br>            - severity=&quot;critical&quot;<br>          receiver: sns-email<br>          continue: true<br><br>        # SMS remains independent of the email policy.<br>        - matchers:<br>            - sms=&quot;true&quot;<br>          receiver: sms<br><br>        # Existing lower-severity Teams routes can remain below this point.<br>        - matchers:<br>            - severity=~&quot;warning|info&quot;<br>          receiver: teams-info<br><br>    receivers:<br>      - name: teams-critical<br>        # Keep the existing Microsoft Teams configuration here.<br><br>      - name: teams-info<br>        # Keep the existing informational Teams configuration here.<br><br>      - name: sms<br>        # Keep the existing SMS adapter configuration here.<br><br>      - name: sns-email<br>        sns_configs:<br>          - api_url: https://sns.eu-central-1.amazonaws.com<br>            sigv4:<br>              region: eu-central-1<br>            topic_arn: arn:aws:sns:eu-central-1:123456789012:tooling-alertmanager-email<br>            subject: &#39;[{{ .Status | toUpper }}] Alertmanager critical alert&#39;<br>            message: |-<br>              Alertmanager notification<br><br>              Status: {{ .Status | toUpper }}<br>              Alerts in group: {{ len .Alerts }}<br><br>              {{ range .Alerts }}<br>              Alert: {{ .Labels.alertname }}<br>              Severity: {{ .Labels.severity }}<br>              Summary: {{ .Annotations.summary }}<br>              Message: {{ .Annotations.message }}<br>              Generator URL: {{ .GeneratorURL }}<br>              {{ end }}<br>            send_resolved: true</pre><blockquote><strong>There are two details worth calling out.</strong></blockquote><p><strong>First,</strong> <strong><em>continue: true</em></strong> on the critical Teams route allows a critical alert to continue to the SNS route. Without it, Alertmanager may stop at the first matching route and email will never be sent.</p><p><strong>Second<em>, send_resolved: true</em></strong> means that recovery notifications also produce email. That is useful for operational follow-up, but you can change it to false if your team wants email only while an alert is firing.</p><h3>A complete test alert</h3><p>Before testing a real alert rule, send a manual alert to Alertmanager. This confirms the route, AWS permissions, SNS topic, and subscriber confirmation independently of Prometheus evaluation.</p><pre>kubectl -n monitoring port-forward \<br>  svc/alertmanager \<br>  9093:9093</pre><p>In another terminal:</p><pre>curl -X POST http://127.0.0.1:9093/api/v2/alerts \<br>  -H &#39;Content-Type: application/json&#39; \<br>  --data-raw &#39;[{<br>    &quot;labels&quot;: {<br>      &quot;alertname&quot;: &quot;ProdCriticalEmailTest&quot;,<br>      &quot;severity&quot;: &quot;critical&quot;,<br>      &quot;type&quot;: &quot;manual-test&quot;,<br>      &quot;sms&quot;: &quot;true&quot;,<br>      &quot;cluster&quot;: &quot;tooling-prod&quot;,<br>      &quot;service&quot;: &quot;alertmanager-test&quot;<br>    },<br>    &quot;annotations&quot;: {<br>      &quot;summary&quot;: &quot;Critical email test&quot;,<br>      &quot;message&quot;: &quot;This is a manual validation of Alertmanager SNS email delivery.&quot;<br>    }<br>  }]&#39;</pre><p><strong>Expected behavior:</strong></p><blockquote>•An email arrives at each confirmed SNS subscription.</blockquote><blockquote>•The critical Teams notification is delivered.</blockquote><blockquote>•The SMS adapter receives the alert because sms=true.</blockquote><p>The first notification may take at least 30 seconds because of the group_wait setting.</p><h3>Test that non-critical alerts do not send email</h3><p>This test is just as important as the critical test. A routing rule is not complete until it proves that unwanted alerts are excluded.</p><pre>curl -X POST http://127.0.0.1:9093/api/v2/alerts \<br>  -H &#39;Content-Type: application/json&#39; \<br>  --data-raw &#39;[{<br>    &quot;labels&quot;: {<br>      &quot;alertname&quot;: &quot;ProdInfoEmailExclusionTest&quot;,<br>      &quot;severity&quot;: &quot;info&quot;,<br>      &quot;type&quot;: &quot;manual-test&quot;,<br>      &quot;cluster&quot;: &quot;tooling-prod&quot;,<br>      &quot;service&quot;: &quot;alertmanager-test&quot;<br>    },<br>    &quot;annotations&quot;: {<br>      &quot;summary&quot;: &quot;Informational email exclusion test&quot;,<br>      &quot;message&quot;: &quot;This alert must not generate an SNS email.&quot;<br>    }<br>  }]&#39;</pre><p>Expected behavior:</p><pre>Teams informational notification: delivered<br>SNS email: not delivered<br>SMS: not delivered</pre><h3>Practical lessons from the implementation</h3><p>This design surfaced a few practical lessons that are easy to miss when the task is described as “just add email.”</p><h3>A successful Terraform apply does not mean email works</h3><p>Terraform can create an SNS email subscription successfully, but the subscription remains inactive until the recipient confirms it. Always verify the subscription status, not only the Terraform output.</p><h3>A broad route creates an expensive inbox problem</h3><p>It is tempting to write a route that sends every Alertmanager event to SNS. That makes the first demo look impressive, but it turns email into another noisy monitoring channel. The exact severity=critical matcher is a small line with a large operational effect.</p><h3>IAM problems often look like application problems</h3><p>When Alertmanager says that it cannot find credentials, the problem may not be in the SNS receiver at all. Check the EKS Pod Identity association, the ServiceAccount name, the namespace, and whether the pod was restarted after the association was created.</p><h3>Route order matters</h3><p>Alertmanager routes are evaluated in order. A critical Teams route without continue: true can prevent the same alert from reaching the SNS receiver. Whenever a new receiver is added, read the entire route tree instead of editing one line in isolation.</p><h3>Recipient ownership matters as much as technical delivery</h3><p>A list of individual addresses is easy to create and hard to maintain. A team-owned distribution list gives Operations a clear owner for membership and SNS confirmation.</p><h3>Adding and removing recipients</h3><p>Recipients should be changed through Terraform:</p><pre>alertmanager_email_destinations = [ &quot;operations-example-1@example.com&quot;, &quot;operations-example-2@example.com&quot;, &quot;new-oncall-list@example.com&quot; ]<br>Review the plan:<br>Bash<br>terraform plan</pre><p>Only apply the change when the plan contains the expected subscription additions or removals and no unrelated destructive changes.</p><p>After adding a recipient, ask the owner to confirm the SNS email. After removing one, verify that Terraform plans to remove only the intended subscription.</p><p>Avoid manually changing subscriptions in the AWS console. Manual changes can create Terraform drift unless the resource is imported and managed consistently afterward.</p><h3>Final configuration summary</h3><p>The implementation is complete when the following is true:</p><pre>Critical alert -&gt; Microsoft Teams -&gt; SNS email subscribers -&gt; SMS only when sms=true Warning / info / error alert -&gt; Existing Teams behavior -&gt; No SNS email -&gt; SMS only when sms=true</pre><p>The most important design choice was not AWS, Terraform, or the Alertmanager receiver. It was the decision to make email critical-only and to keep recipient ownership outside the pod.</p><p>That gave us a notification path that was useful without becoming another source of alert fatigue.</p><h3>References</h3><p><a href="https://prometheus.io/docs/alerting/latest/alertmanager/">[1] Prometheus Alertmanager documentation</a></p><p><a href="https://docs.aws.amazon.com/sns/latest/dg/sns-email-notifications.html">[2] Amazon SNS email notifications</a></p><p><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html">[3] Amazon EKS Pod Identities</a></p><p><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_pod_identity_association">[4] Terraform AWS provider EKS Pod Identity association</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d8f3e1a6dee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Kubernetes Costs with OpenCost and Grafana]]></title>
            <link>https://medium.com/@walaaelgenidy/monitoring-kubernetes-costs-with-opencost-and-grafana-e39a9c33659d?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e39a9c33659d</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[kubecost]]></category>
            <category><![CDATA[opencost]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 14:05:27 GMT</pubDate>
            <atom:updated>2025-10-10T14:05:27.117Z</atom:updated>
            <content:encoded><![CDATA[<h3>Because “Why is our cloud bill so high?” shouldn’t be a monthly surprise</h3><p>If your team runs workloads on Kubernetes, you’ve probably faced the dreaded <em>cost visibility</em> problem. Engineers spin up pods, forget to downscale, and suddenly your CFO is asking why last month’s EKS bill doubled.</p><p>Enter <strong>OpenCost</strong> — an open-source tool for real-time Kubernetes cost monitoring — paired with <strong>Grafana</strong> for dashboards that even your finance team will love.</p><p>Here’s how to set it up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JbMghWPiLfNRNHRf8Cwpg.png" /></figure><h3>Why OpenCost?</h3><ul><li><strong>Real-time cost allocation</strong>: Track spend by namespace, deployment, or even pod.</li><li><strong>Open-source</strong>: No vendor lock-in (unlike proprietary tools like Kubecost’s paid tier).</li><li><strong>AWS/EKS-friendly</strong>: Works natively with AWS pricing data.</li></ul><h3>What You’ll Need</h3><ul><li>A running Kubernetes cluster (EKS, AKS, GKE, or on-prem)</li><li>Helm installed and configured</li><li>Prometheus already running in your cluster</li><li>AWS credentials (if using EKS)</li></ul><h3>Step 1: Install OpenCost with Helm</h3><p>First, let’s set up the OpenCost Helm chart:</p><pre># Chart.yaml<br>apiVersion: v2<br>appVersion: 1.114.0<br>name: opencost<br>description: OpenCost and OpenCost UI<br>type: application<br>keywords:<br>  - cloud-costs<br>  - cost-optimization<br>  - finops<br>  - monitoring<br>  - opencost<br>version: 1.0.0<br>dependencies:<br>- name: opencost<br>  version: 2.1.2<br>  repository: https://opencost.github.io/opencost-helm-chart</pre><p>This chart defines our OpenCost deployment with the latest stable version and includes the OpenCost UI for visualization.</p><h3>Step 2: Configure OpenCost Values</h3><p>Here’s the complete configuration that makes OpenCost work with AWS EKS:</p><pre># values.yaml<br>opencost:<br>  # Use default pricing instead of custom rates<br>  customPricing:<br>    enabled: false<br>  <br>  # Simple name for our deployment<br>  fullnameOverride: &quot;opencost-dev&quot;<br>  <br>  # Container image configuration<br>  image:<br>    repository: ghcr.io/opencost/opencost<br>    tag: v1.114.0<br>    pullPolicy: IfNotPresent<br><br>  # Main exporter configuration<br>  exporter:<br>    # Unique identifier for this cluster<br>    defaultClusterId: &quot;dev-cluster&quot;<br>    <br>    # Environment variables for OpenCost<br>    env:<br>      - name: PROMETHEUS_SERVER_ENDPOINT<br>        value: &quot;http://prometheus-dev-kube-promet-prometheus.monitoring.svc.cluster.local:9090&quot;<br>    <br>    # Additional configuration<br>    extraEnv:<br>      # Enable CSV export for data processing<br>      EXPORT_CSV_FILE: &quot;true&quot;<br>      # Debug logging for troubleshooting<br>      LOG_LEVEL: debug<br>      # Namespace where OpenCost runs<br>      KUBECOST_NAMESPACE: &quot;monitoring&quot;<br>      # API port for OpenCost<br>      API_PORT: &quot;9003&quot;<br>      # Cluster identifier<br>      CLUSTER_ID: &quot;dev-cluster&quot;<br><br>      # Metrics Configuration<br>      # Use only KSM v1 metrics for compatibility<br>      EMIT_KSM_V1_METRICS: &quot;false&quot;<br>      EMIT_KSM_V1_METRICS_ONLY: &quot;true&quot;<br><br>      # AWS Configuration<br>      AWS_REGION: &quot;eu-central-1&quot;<br>      AWS_ACCOUNT_ID: &quot;01234567890&quot;<br>      # Use regional STS endpoints for better performance<br>      AWS_STS_REGIONAL_ENDPOINTS: &quot;regional&quot;<br>      # IAM role for AWS API access<br>      AWS_ROLE_ARN: &quot;arn:aws:iam::01234567890:role/OpenCost-IRSA-Role&quot;<br>      # Token file for IAM authentication<br>      AWS_WEB_IDENTITY_TOKEN_FILE: &quot;/var/run/secrets/eks.amazonaws.com/serviceaccount/token&quot;<br><br>      # EKS Integration<br>      AWS_EKS_CLUSTER_ID: &quot;dev-cluster&quot;<br>      # Enable EKS-specific pricing<br>      AWS_EKS_PRICING_ENABLED: &quot;true&quot;<br>      # Disable Athena for simplicity<br>      AWS_ATHENA_ENABLED: &quot;false&quot;<br><br>      # CloudWatch Export - send metrics to CloudWatch<br>      AWS_CLOUDWATCH_METRICS_ENABLED: &quot;true&quot;<br>      AWS_CLOUDWATCH_METRICS_NAMESPACE: &quot;OpenCost/EKS&quot;<br><br>  # OpenCost UI configuration<br>  ui:<br>    enabled: true<br>    uiPort: 9091<br>    service:<br>      port: 9091<br>      targetPort: 9091<br>    containerPort: 9091<br>    extraEnv:<br>      - name: UI_PORT<br>        value: &quot;9091&quot;<br>      - name: API_PORT<br>        value: &quot;9003&quot;<br><br>  # Prometheus configuration<br>  prometheus:<br>    internal:<br>      enabled: true<br>      serviceName: prometheus-dev-kube-promet-prometheus<br>      # Namespace of in-cluster Prometheus<br>      namespaceName: monitoring<br>      # Service port of in-cluster Prometheus<br>      port: 9090<br>    external:<br>      enabled: false<br>      url: &quot;http://prometheus-dev-kube-promet-prometheus.monitoring.svc.cluster.local:9090&quot;<br>  <br>  # ServiceMonitor for Prometheus operator<br>  metrics:<br>    serviceMonitor:<br>      enabled: true<br>      namespace: monitoring<br><br>  # Service Account with IAM permissions<br>  serviceAccount:<br>    create: true<br>    name: &quot;opencost-serviceaccount&quot;<br>    annotations:<br>      eks.amazonaws.com/role-arn: &quot;arn:aws:iam::01234567890:role/OpenCost-IRSA-Role&quot;<br><br>  # Volume mounts for storage and IAM tokens<br>  extraVolumeMounts:<br>    - name: cost-data<br>      mountPath: /var/opencost<br>    - name: aws-iam-token<br>      mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount<br>      readOnly: true<br><br>  extraVolumes:<br>    - name: cost-data<br>      emptyDir: {}<br>    - name: aws-iam-token<br>      projected:<br>        defaultMode: 420<br>        sources:<br>          - serviceAccountToken:<br>              audience: sts.amazonaws.com<br>              expirationSeconds: 86400<br>              path: token</pre><h3>Step 3: Deploy OpenCost</h3><pre># Add the OpenCost Helm repository<br>helm repo add opencost https://opencost.github.io/opencost-helm-chart<br>helm repo update<br><br># Install OpenCost in the monitoring namespace<br>helm install opencost opencost/opencost -n monitoring --create-namespace -f values.yaml</pre><h3>Key Configuration Sections Explained</h3><h3>1. Prometheus Connection</h3><pre>prometheus:<br>  internal:<br>    enabled: true<br>    serviceName: prometheus-dev-kube-promet-prometheus</pre><p>OpenCost reads metrics from Prometheus to calculate costs. This configuration tells OpenCost where to find your Prometheus instance and how to connect to it.</p><h3>2.Cost Data Export</h3><pre>extraEnv:<br>  EXPORT_CSV_FILE: &quot;true&quot;<br>  AWS_CLOUDWATCH_METRICS_ENABLED: &quot;true&quot;</pre><p>This enables multiple output formats:</p><ul><li>CSV files for data processing and custom reports</li><li>CloudWatch metrics for AWS-native monitoring and alerting</li></ul><h3>3.Multi-Cluster Support</h3><pre>defaultClusterId: &quot;dev-cluster&quot;<br>CLUSTER_ID: &quot;dev-cluster&quot;</pre><p>These identifiers allow you to track costs across multiple clusters. If you have staging, production, and development clusters, each gets a unique ID for consolidated reporting.</p><h3>4.Integrate with Grafana</h3><p>Now let’s connect OpenCost to Grafana for beautiful dashboards.</p><h3># Import the OpenCost Dashboard</h3><ol><li>Open Grafana and navigate to Dashboards → Import</li><li>Use dashboard ID: 22208</li><li>Or download from: <a href="https://grafana.com/grafana/dashboards/22208-opencost-overview/">OpenCost Overview Dashboard</a></li><li>Add OpenCost as a data source in Grafana</li></ol><h3>Dashboard Features You’ll Love</h3><p>The OpenCost dashboard provides:</p><ul><li><strong>📊 Cost Overview: </strong>Total cluster costs by namespace, deployment, and service</li><li><strong>📈 Trend Analysis:</strong> Cost trends over time to spot anomalies</li><li><strong>🔍 Resource Breakdown:</strong> CPU, memory, and storage costs separated</li><li><strong>🚨 Alerting: </strong>Set up alerts when costs exceed thresholds</li><li><strong>💰 Efficiency Metrics: </strong>Cost per pod, cost per user, and resource utilization</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wf8Bl0G4JTrqMC-bec2PQ.png" /></figure><h3>Conclusion</h3><p>With OpenCost and Grafana, you’ve transformed Kubernetes cost management from a monthly surprise into a real-time, actionable dashboard. You can now:</p><p><strong>✅ See exactly where your money is going — by namespace, deployment, or team<br>✅ Catch cost spikes before they become problems — with real-time monitoring<br>✅ Make data-driven decisions — about resource allocation and optimization<br>✅ Speak the same language as finance — with clear, business-friendly dashboards</strong></p><p>The best part? This entire setup is open-source and runs for a fraction of the cost of commercial alternatives.</p><p>No more waiting for monthly bills — you’re now in control of your Kubernetes spending.</p><p><strong>Next steps:</strong> Set up alerts in Grafana for when costs exceed thresholds, and consider integrating OpenCost data into your CI/CD pipeline to catch expensive deployments before they reach production.</p><p><strong>Happy cost monitoring! 🎯</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e39a9c33659d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up Apache Airflow with Helm, Keycloak Integration, and EFS , S3 DAG Syncing]]></title>
            <link>https://medium.com/@walaaelgenidy/setting-up-apache-airflow-with-helm-keycloak-integration-and-efs-s3-dag-syncing-ef68ed697b50?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef68ed697b50</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[dataops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 13:46:29 GMT</pubDate>
            <atom:updated>2025-10-10T14:28:30.479Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting Up Apache Airflow with Helm, Keycloak Integration, and EFS , S3 DAG Syncing</h3><p>In this article, we’ll explore how to set up Apache Airflow on Kubernetes using Helm, integrate Keycloak for authentication, and sync DAGs from EFS. This comprehensive guide includes the necessary configuration file and explanations for each section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QOgNCCeeYKE7rbFOmSUvCA.png" /></figure><h3>Prerequisites</h3><p>Before you start, ensure you have the following:</p><ul><li>A running Kubernetes cluster.</li><li>Helm installed and configured.</li><li>Access to AWS services, specifically S3 and EFS.</li></ul><p>Step 1: Install Apache Airflow:</p><pre>helm repo add apache-airflow https://airflow.apache.org<br>helm repo update<br>helm install airflow apache-airflow/airflow --namespace airflow --create-namespace</pre><h3>Step 2: Configuration Overview</h3><p>Here’s the complete configuration file for setting up Apache Airflow, including Keycloak integration and EFS syncing.</p><pre>#  Chart.yaml<br><br>apiVersion: v2<br>name: airflow<br>description: A Helm chart for Apache Airflow<br>type: application<br>version: 1.1.0<br>appVersion: &quot;2.10.5&quot;<br>dependencies:<br>- name: airflow<br>  version: &quot;1.15.0&quot;<br>  repository: https://airflow.apache.org</pre><pre># values.yaml<br># Airflow configuration for Helm deployment<br>airflow:<br>  uid: 50000  # User ID for Airflow containers<br>  gid: 0  # Group ID for Airflow containers<br><br>  securityContexts:<br>    pod:<br>      runAsUser: 50000  # Run pods as user ID 50000<br>      fsGroup: 0  # File system group ID<br>      runAsGroup: 0  # Run pods as group ID 0<br>    containers:<br>      runAsUser: 50000  # Run containers as user ID 50000<br>      runAsGroup: 0  # Run containers as group ID 0<br><br>  config:<br>    logging:<br>      remote_logging: &#39;True&#39;  # Enable remote logging<br>      logging_level: &#39;INFO&#39;  # Set logging level to INFO<br>      remote_base_log_folder: &#39;s3://airflow-dags-sync-s3/logs&#39;  # S3 bucket for log storage<br>      remote_log_conn_id: &#39;aws_conn&#39;  # Connection ID for AWS logs<br>      delete_worker_pods: &#39;False&#39;  # Do not automatically delete worker pods<br><br>  useStandardNaming: true  # Use standard naming conventions for resources<br>  AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags  # Directory for storing DAG files<br>  AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: &quot;30&quot;  # Interval for checking new DAGs (in seconds)<br>  AIRFLOW__CORE__STORE_DAG_CODE: &quot;True&quot;  # Store DAG code in the metadata database<br>  rbac:<br>    create: true  # Enable Role-Based Access Control (RBAC)<br><br>#                   -----------##############-----------<br><br>  # Environment variables for Airflow<br>  env:<br>    - name: &quot;AIRFLOW__API__ENABLE_EXPERIMENTAL_API&quot;<br>      value: &quot;True&quot;  # Enable experimental API features<br>    - name: &quot;AIRFLOW__API__AUTH_BACKENDS&quot;<br>      value: &quot;airflow.api.auth.backend.session&quot;  # Use session-based authentication<br>    - name: &quot;AIRFLOW__WEBSERVER__RBAC&quot;<br>      value: &quot;True&quot;  # Enable RBAC for the webserver<br>    - name: &quot;AIRFLOW__WEBSERVER__AUTHENTICATE&quot;<br>      value: &quot;True&quot;  # Enable authentication for the webserver<br>    - name: &quot;AIRFLOW__WEBSERVER__AUTH_BACKEND&quot;<br>      value: &quot;airflow.www.security.manager.CustomSecurityManager&quot;  # Custom security manager for authentication<br>    - name: AWS_DEFAULT_REGION<br>      value: &quot;eu-central-1&quot;  # Set your AWS region for S3 access<br><br>    # Environment variables for metrics collection<br>    - name: AIRFLOW__METRICS__STATSD_ON<br>      value: &quot;True&quot;<br>    - name: AIRFLOW__METRICS__STATSD_HOST<br>      value: &quot;airflow-dev-statsd&quot;<br>    - name: AIRFLOW__METRICS__STATSD_PORT<br>      value: &quot;9125&quot;<br>    - name: AIRFLOW__METRICS__STATSD_PREFIX<br>      value: &quot;airflow&quot;<br>    - name: AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK<br>      value: &quot;True&quot;<br>    - name: AIRFLOW__SCHEDULER__ENABLE_METRICS<br>      value: &quot;True&quot;<br>    - name: AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX<br>      value: &quot;True&quot;<br>    - name: AIRFLOW__METRICS__STATS_ENABLED<br>      value: &quot;True&quot;<br>    - name: AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK<br>      value: &quot;True&quot;<br>    - name: AIRFLOW__SCHEDULER__ENABLE_METRICS<br>      value: &quot;True&quot;<br><br>#                   -----------##############-----------<br><br>  postgresql:<br>    enabled: false  # Disable PostgreSQL (use an external database if needed)<br>  data:<br>    metadataSecretName: airflowdbsecret  # Secret name for Airflow metadata storage<br>  pgbouncer:<br>    enabled: true  # Enable PgBouncer for database connection pooling<br><br>  executor: &quot;KubernetesExecutor&quot;  # Use the Kubernetes executor for task execution<br>  fernetKeySecretName: fernet-secret-key  # Secret for Fernet encryption key<br>  webserverSecretKeySecretName: webserver-secret-key  # Secret for webserver&#39;s secret key<br>                  <br>               <br>  #                   -----------##############-----------<br><br>  # Airflow Workers Configuration<br>  workers:<br>    replicas: 1  # Number of worker replicas to run<br>    safeToEvict: true  # Allow Kubernetes to safely evict the worker pod<br>    serviceAccount:<br>      annotations:<br>        eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/AirflowS3DAGAccessRole  # IAM role for service account<br>      automountServiceAccountToken: true  # Automatically mount the service account token<br><br><br>#                   -----------##############-----------<br><br><br>  # DAG Processor Configuration<br>  dagProcessor:<br>    enabled: false  # Disable separate DAG processor (KubernetesExecutor handles it)<br>    replicas: 1  # Number of replicas for the DAG processor<br><br>  # DAGs Configuration<br>  dags:<br>    persistence:<br>      enabled: true  # Enable persistence for DAGs (syncing with EFS)<br>      size: 10Gi  # Size of the EFS volume<br>      storageClassName: efs  # Storage class for EFS<br>      accessMode: ReadWriteMany  # Access mode for the EFS volume<br><br>    gitSync:<br>      enabled: false  # Disable Git sync for DAGs<br><br>#               -----------##############-----------<br><br>  # StatsD Configuration for metrics<br>  statsd:<br>    enabled: true  # Enable StatsD for metrics collection<br>    resources:<br>      requests:<br>        memory: &quot;100Mi&quot;  # Memory request for StatsD<br>        cpu: &quot;100m&quot;  # CPU request for StatsD<br>      limits:<br>        memory: &quot;256Mi&quot;  # Memory limit for StatsD<br>        cpu: &quot;500m&quot;  # CPU limit for StatsD<br><br>#               -----------##############-----------<br><br>  # Prometheus Metrics Configuration<br>  metrics:<br>    serviceMonitor:<br>      enabled: true  # Enable Prometheus service monitor<br>      labels:<br>        release: &quot;prometheus-dev&quot;  # Label for Prometheus release<br>        app.kubernetes.io/part-of: &quot;airflow&quot;  # Label for part of Airflow app<br>      interval: &quot;30s&quot;  # Scrape interval for Prometheus<br>      scrapeTimeout: &quot;10s&quot;  # Scrape timeout for Prometheus<br><br>#               -----------##############-----------<br><br>  # Ingress Configuration<br>  ingress:<br>    enabled: false  # Disable built-in Ingress for Airflow<br><br>  # Triggerer Configuration<br>  triggerer:<br>    enabled: true  # Enable triggerer for task management<br>    replicas: 1  # Number of replicas for the triggerer<br>    safeToEvict: true  # Allow safe eviction of the triggerer pod<br>    resources:<br>      requests:<br>        cpu: &quot;700m&quot;  # CPU request for triggerer<br>        memory: &quot;700Mi&quot;  # Memory request for triggerer<br>      limits:<br>        cpu: &quot;1400m&quot;  # CPU limit for triggerer<br>        memory: &quot;1.4Gi&quot;  # Memory limit for triggerer<br><br><br>#               -----------##############-----------<br><br>  # Scheduler Configuration<br>  scheduler:<br>    replicas: 1  # Number of scheduler replicas<br>    serviceAccount:<br>      create: true  # Create a service account for the scheduler<br>      name: airflow-dev-s3-access  # Name of the service account<br>      annotations:<br>        eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/AirflowS3DAGAccessRole  # IAM role for service account<br>      automountServiceAccountToken: true  # Automatically mount the service account token<br>    extraContainers:<br>      - name: s3-sync  # Container for syncing DAGs from S3<br>        image: amazon/aws-cli:2.13.4  # Docker image for AWS CLI<br>        command: [&quot;/bin/sh&quot;, &quot;-c&quot;]  # Command to execute<br>        args:<br>          - |<br>            while true; do  # Infinite loop to sync DAGs<br>              aws s3 sync s3://airflow-dags-sync-s3/ /opt/airflow/ --delete  # Sync DAGs from S3<br>              sleep 30  # Wait for 30 seconds before the next sync<br>            done<br>        env:<br>          - name: AWS_DEFAULT_REGION  # Environment variable for AWS region<br>            value: &quot;eu-central-1&quot;  # Set your AWS region for S3 access<br>        volumeMounts:<br>          - name: dags  # Mount point for DAGs<br>            mountPath: /opt/airflow/dags  # Path in the container where DAGs will be accessed<br>    <br>#                  -----------##############----------- <br><br> # Webserver Configuration<br>  webserver:<br>    replicas: 1  # Number of webserver replicas<br>    defaultUser:<br>      enabled: false  # Disable default user creation<br>    serviceAccount:<br>      annotations:<br>        eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/AirflowS3DAGAccessRole  # IAM role for webserver service account<br>      automountServiceAccountToken: true  # Automatically mount the service account token<br>    env:<br>      - name: KEYCLOAK_CLIENT_SECRET  # Keycloak client secret<br>        valueFrom:<br>          secretKeyRef:<br>            name: keycloak-client-secret-airflow-dev  # Reference to the secret containing Keycloak client secret<br>            key: KEYCLOAK_CLIENT_SECRET  # Key within the secret<br><br>    # Keycloak Webserver Configuration<br>    webserverConfig: |<br>      import logging  <br>      import os  <br>      from base64 import b64decode  <br>      import jwt  <br>      import requests  <br>      from airflow.www.security import AirflowSecurityManager  <br>      from cryptography.hazmat.primitives import serialization  <br>      from flask_appbuilder import expose  <br>      from flask_appbuilder.security.manager import AUTH_OAUTH  <br>      from flask_appbuilder.security.views import AuthOAuthView  <br><br>      basedir = os.path.abspath(os.path.dirname(__file__))  # Base directory for the application<br><br>      WTF_CSRF_ENABLED = True  # Enable CSRF protection<br>      WTF_CSRF_TIME_LIMIT = None  # Disable CSRF time limit<br><br>      log = logging.getLogger(__name__)  # Initialize logger<br><br>      AUTH_TYPE = AUTH_OAUTH  # Set authentication type to OAuth<br>      AUTH_ROLES_SYNC_AT_LOGIN = True  # Sync roles at each login<br>      AUTH_USER_REGISTRATION = True  # Allow automatic user registration<br>      AUTH_DEFAULT_ROLE = os.getenv(&quot;AUTH_DEFAULT_ROLE&quot;, &quot;Public&quot;)  # Default role for new users<br>      AUTH_USER_REGISTRATION_ROLE = AUTH_DEFAULT_ROLE  # Role assigned upon registration<br><br>      # Keycloak Configuration<br>      KEYCLOAK_ISSUER = &quot;https://keycloak.example.com/realms/example&quot;  # URL for Keycloak issuer<br>      KEYCLOAK_CLIENT_ID = &quot;airflow-dev&quot;  # Client ID for Airflow in Keycloak<br>      KEYCLOAK_CLIENT_SECRET = os.getenv(&quot;KEYCLOAK_CLIENT_SECRET&quot;)  # Fetch Keycloak client secret from environment variable<br><br>      # Keycloak URLs<br>      KEYCLOAK_SERVER_METADATA_URL = &quot;https://keycloak.example.com/realms/example/.well-known/openid-configuration&quot;  # Metadata URL for Keycloak<br>      KEYCLOAK_API_BASE_URL = &quot;https://keycloak.example.com/realms/example/protocol/&quot;  # API base URL for Keycloak<br>      KEYCLOAK_ACCESS_TOKEN_URL = &quot;https://keycloak.example.com/realms/example/protocol/openid-connect/token&quot;  # Access token URL for Keycloak<br>      KEYCLOAK_AUTHORIZE_URL = &quot;https://keycloak.example.com/realms/example/protocol/openid-connect/auth&quot;  # Authorization URL for Keycloak<br><br>      AUTH_ROLES_MAPPING = {  # Mapping of roles between Airflow and Keycloak<br>          &quot;Admin&quot;: [&quot;Admin&quot;],<br>          &quot;Op&quot;: [&quot;Op&quot;],<br>          &quot;User&quot;: [&quot;User&quot;],<br>          &quot;Viewer&quot;: [&quot;Viewer&quot;],<br>          &quot;Public&quot;: [&quot;Public&quot;], <br>      }  <br><br>      OAUTH_PROVIDERS = [  # Configuration for OAuth providers<br>          {  <br>              &quot;name&quot;: &quot;keycloak&quot;,  # Name of the OAuth provider<br>              &quot;icon&quot;: &quot;fa-key&quot;,  # Icon for the OAuth provider<br>              &quot;token_key&quot;: &quot;access_token&quot;,  # Key in the response for the access token<br>              &quot;remote_app&quot;: {  <br>                  &quot;client_id&quot;: KEYCLOAK_CLIENT_ID,  # Client ID for Keycloak<br>                  &quot;client_secret&quot;: KEYCLOAK_CLIENT_SECRET,  # Client secret for Keycloak<br>                  &quot;server_metadata_url&quot;: KEYCLOAK_SERVER_METADATA_URL,  # Metadata URL for Keycloak<br>                  &quot;api_base_url&quot;: KEYCLOAK_API_BASE_URL,  # Base URL for Keycloak API<br>                  &quot;client_kwargs&quot;: {&quot;scope&quot;: &quot;email profile&quot;},  # OAuth scopes<br>                  &quot;access_token_url&quot;: KEYCLOAK_ACCESS_TOKEN_URL,  # Access token URL for Keycloak<br>                  &quot;authorize_url&quot;: KEYCLOAK_AUTHORIZE_URL,  # Authorization URL for Keycloak<br>                  &quot;request_token_url&quot;: None,  # No request token URL<br>              },  <br>          },  <br>      ]  <br><br>      try:  <br>          response = requests.get(KEYCLOAK_ISSUER)  # Fetch Keycloak issuer information<br>          response.raise_for_status()  # Raise an error for bad responses<br>          key_der_base64 = response.json().get(&quot;public_key&quot;)  # Get public key from Keycloak<br>          key_der = b64decode(key_der_base64.encode())  # Decode the public key<br>          public_key = serialization.load_der_public_key(key_der)  # Load the public key<br>      except requests.RequestException as e:  <br>          log.error(f&quot;Failed to fetch Keycloak public key: {e}&quot;)  # Log error if fetching fails<br>          raise  <br><br>      class CustomAuthRemoteUserView(AuthOAuthView):  <br>          @expose(&quot;/logout/&quot;, methods=[&quot;GET&quot;, &quot;POST&quot;])  # Custom logout view<br>          def logout(self):  <br>              return super().logout()  # Call the parent logout method<br><br>      class CustomSecurityManager(AirflowSecurityManager):  <br>          authoauthview = CustomAuthRemoteUserView  # Use custom auth view<br><br>          def oauth_user_info(self, provider, response):  <br>              if provider == &quot;keycloak&quot;:  # Check if the provider is Keycloak<br>                  token = response.get(&quot;access_token&quot;)  # Get access token from response<br>                  if not token:  <br>                      log.warning(&quot;No access token found in the response.&quot;)  # Log warning if no token<br>                      return {}  <br><br>                  try:  <br>                      user_info = jwt.decode(  <br>                          token,  <br>                          public_key,  # Verify token using the public key<br>                          algorithms=[&quot;RS256&quot;],  <br>                          audience=KEYCLOAK_CLIENT_ID,  # Check audience<br>                      )  <br>                  except jwt.ExpiredSignatureError:  <br>                      log.error(&quot;Token has expired.&quot;)  # Log error if token has expired<br>                      return {}  <br>                  except jwt.InvalidTokenError as e:  <br>                      log.error(f&quot;Invalid token: {e}&quot;)  # Log error if token is invalid<br>                      return {}  <br><br>                  roles = (  <br>                      user_info.get(&quot;resource_access&quot;, {})  <br>                      .get(KEYCLOAK_CLIENT_ID, {})  <br>                      .get(&quot;roles&quot;, [AUTH_DEFAULT_ROLE])  # Fetch roles from user info<br>                  )  <br><br>                  return {  # Return user info<br>                      &quot;username&quot;: user_info.get(&quot;preferred_username&quot;),  <br>                      &quot;email&quot;: user_info.get(&quot;email&quot;),  <br>                      &quot;first_name&quot;: user_info.get(&quot;given_name&quot;),  <br>                      &quot;last_name&quot;: user_info.get(&quot;family_name&quot;),  <br>                      &quot;role_keys&quot;: roles,  <br>                  }  <br><br>              return {}  # Return empty dictionary if provider is not Keycloak<br><br>      SECURITY_MANAGER_CLASS = CustomSecurityManager  # Set the custom security manager<br><br>## this part is customized not exsit in the official chart <br>airflowSecrets:<br>  - keycloak-client-secret-airflow-dev  # List of secrets for Airflow<br><br>dagSecrets:<br>  - airflow-dag-secrets-dev  # List of secrets for DAG configurations<br>  - airflow-dag-secrets-preprod<br>  - ...etc <br></pre><h3>Key Sections Explained</h3><p>Now that we’ve seen the complete configuration, let’s break down the most critical parts to understand how everything works together in our production-ready Airflow deployment.</p><h4>1. Security Context &amp; User Management</h4><p>The security configuration ensures Airflow runs with proper user permissions while maintaining access to shared file systems:</p><p>yaml</p><pre>airflow:<br>  uid: 50000  # Dedicated user ID for Airflow containers<br>  gid: 0  # Root group for file system access</pre><p>This setup uses a specific non-root user (UID 50000) for running containers, which follows security best practices by avoiding root privileges. The group ID 0 (root) ensures Airflow can read and write to shared volumes like EFS, which often require root group permissions for multi-user access.</p><h4>2. Kubernetes Executor Configuration</h4><p>The choice of executor is crucial for scalability and resource management:</p><p>yaml</p><pre>executor: &quot;KubernetesExecutor&quot;  # Each task runs in its own pod</pre><p>The Kubernetes Executor provides significant advantages over other executors:</p><ul><li><strong>Task Isolation:</strong> Each task runs in its own dedicated pod, preventing resource conflicts</li><li><strong>Dynamic Resource Allocation: </strong>Kubernetes automatically handles resource requests and limits</li><li><strong>Cost Efficiency: </strong>Pods are created on-demand and destroyed after task completion</li><li><strong>Scalability: </strong>Easily handle workloads from small to enterprise-scale</li></ul><h4>3. EFS Persistence for DAGs</h4><p>The EFS integration solves the challenge of shared DAG storage across multiple pods:</p><p>yaml</p><pre>dags:<br>  persistence:<br>    enabled: true<br>    storageClassName: efs<br>    accessMode: ReadWriteMany</pre><h3><strong>EFS provides:</strong></h3><ul><li><strong>Multi-Pod Access:</strong> Scheduler, webserver, and workers all access the same DAG files</li><li><strong>Data Durability:</strong> DAGs persist through pod restarts and cluster updates</li><li><strong>Automatic Scaling:</strong> Storage capacity grows with your needs</li><li><strong>Cross-AZ Availability: </strong>EFS volumes are accessible across multiple Availability Zones</li></ul><h4>4. S3 Sync for DAG Deployment</h4><p>The S3 sync sidecar container enables continuous DAG deployment:</p><pre>extraContainers:<br>  - name: s3-sync<br>    args:<br>      - |<br>        while true; do<br>          aws s3 sync s3://airflow-dags-sync-s3/ /opt/airflow/ --delete<br>          sleep 30<br>        done</pre><p><strong>This approach provides:</strong></p><ul><li><strong>GitOps-style Deployment: </strong>Deploy DAGs by uploading to S3, perfect for CI/CD pipelines</li><li><strong>Automatic Updates: </strong>Changes in S3 automatically propagate to Airflow</li><li><strong>Version Control:</strong> Leverage S3 versioning for DAG rollbacks</li><li><strong>Team Collaboration: </strong>Multiple teams can deploy DAGs without direct cluster access</li></ul><h4>5. Keycloak OAuth Integration</h4><p>The custom security manager enables enterprise authentication:</p><pre>AUTH_TYPE = AUTH_OAUTH<br>AUTH_USER_REGISTRATION = True<br>AUTH_DEFAULT_ROLE = &quot;Public&quot;</pre><p><strong>This integration offers:</strong></p><ul><li><strong>Single Sign-On:</strong> Users authenticate with existing corporate credentials</li><li><strong>Centralized User Management:</strong> Admins manage users and roles in Keycloak</li><li><strong>Automatic Role Mapping: </strong>Keycloak roles automatically translate to Airflow permissions</li><li><strong>Enhanced Security: </strong>No local password storage or management required</li></ul><h4>6. IAM Roles for Service Accounts (IRSA)</h4><p>AWS IAM integration provides secure cloud access:</p><pre>serviceAccount:<br>  annotations:<br>    eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/AirflowS3DAGAccessRole</pre><p><strong>IRSA delivers:</strong></p><ul><li><strong>Credential Security:</strong> No AWS keys stored in environment variables or config files</li><li><strong>Fine-Grained Permissions: </strong>Different components can have tailored AWS permissions</li><li><strong>Automatic Rotation: </strong>AWS handles temporary credential management</li><li><strong>Audit Trail:</strong> Clear mapping between Kubernetes identities and AWS API calls</li></ul><h4>7. Monitoring &amp; Metrics</h4><p>Comprehensive observability ensures operational excellence</p><pre>statsd:<br>  enabled: true<br>metrics:<br>  serviceMonitor:<br>    enabled: true</pre><p><strong>The monitoring stack provides:</strong></p><ul><li><strong>Performance Insights:</strong> Track DAG execution times, task success rates, and resource usage</li><li>Alerting: Configure Prometheus alerts for scheduler health, queue backlogs, or resource constraints</li><li>Capacity Planning: Use metrics to right-size resource requests and limits</li><li>Troubleshooting: Detailed logs and metrics for debugging production issues</li></ul><h4>8. External Secrets Management</h4><p>Secure handling of sensitive information:</p><pre>airflowSecrets:<br>  - keycloak-client-secret-airflow-dev<br>dagSecrets:<br>  - airflow-dag-secrets-dev</pre><p>This approach ensures:</p><ul><li>Security: Secrets never touch version control systems</li><li>Centralized Management: All secrets stored in AWS Parameter Store</li><li>Access Control: Fine-grained permissions for who can access which secrets</li><li>Auditability: Track secret access and modifications</li></ul><h3>Conclusion</h3><p>This Apache Airflow deployment represents a production-ready, enterprise-grade workflow orchestration platform. By combining Kubernetes, Keycloak, EFS, and AWS services, we’ve created a system that is:</p><p><strong>Highly Available: </strong>Multiple replicas and persistent storage ensure continuous operation<br><strong>Secure:</strong> Keycloak integration, IRSA, and external secrets provide robust security<br><strong>Scalable: </strong>Kubernetes executor and EFS storage allow seamless growth<br><strong>Maintainable:</strong> Clear separation of concernsFeel free to adapt the <strong><em>configurations to meet your specific requirements, and happy orchestrating!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef68ed697b50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow on EKS: Syncing DAGs from S3 to Airflow on EKS]]></title>
            <link>https://medium.com/@walaaelgenidy/syncing-dags-from-s3-to-airflow-on-eks-c97afc9a6444?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c97afc9a6444</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 11:35:20 GMT</pubDate>
            <atom:updated>2025-07-16T11:37:09.250Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re running Apache Airflow on Amazon EKS, you know how crucial it is to keep your DAGs synchronized without manual intervention.</p><p>While Airflow can read DAGs directly from an S3 bucket, triggering an automatic reload when files change isn’t built-in.</p><p>In this guide, I’ll walk you through setting up an automated DAG sync system using S3 Event Notifications— ensuring that your Airflow instance reloads DAGs the moment they’re updated.</p><p><strong>Why This Approach?</strong><br>Manually reloading DAGs or restarting Airflow workers every time a change is made isn’t scalable. Instead, we’ll leverage AWS S3 Event Notifications to detect file changes and trigger Airflow’s DAG refresh mechanism.</p><h3><strong>Implementation Steps:</strong></h3><p>Step1: Adjust airflow deployment:</p><pre>airflow:<br>  config:<br>    logging:<br>      remote_logging: &#39;True&#39;<br>      logging_level: &#39;INFO&#39;<br>      remote_base_log_folder: &#39;s3://airflow-dags-sync-s3/logs&#39;<br>      remote_log_conn_id: &#39;aws_conn&#39;<br>      delete_worker_pods: &#39;False&#39;<br>  useStandardNaming: true<br>  AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags<br>  AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: &quot;30&quot;  # Check every 30s<br>  AIRFLOW__CORE__STORE_DAG_CODE: &quot;True&quot;  # Ensure UI access<br><br><br><br>  # Airflow Dag Processor Config<br>  dagProcessor:<br>    enabled: false # KubernetesExecutor handles DAG parsing in workers<br>    replicas: 1<br><br>  dags:<br>    persistence:<br>      enabled: true  # Enable persistence for S3 sync<br>      size: 10Gi<br>      storageClassName: ebs<br><br>    gitSync:<br>      enabled: false # this need to be disabled <br><br>  webserver:<br>    replicas: 1<br>    defaultUser:<br>      enabled: false<br>    serviceAccount:<br>      create: true<br>      name: airflow-dev-s3-access<br>      annotations:<br>        eks.amazonaws.com/role-arn: arn:aws:iam::1234556788:role/AirflowS3DAGAccessRole<br>      automountServiceAccountToken: true<br>    extraContainers:<br>      - name: s3-sync<br>        image: amazon/aws-cli:2.13.4<br>        command: [&quot;/bin/sh&quot;, &quot;-c&quot;]<br>        args:<br>          - |<br>            export HOME=/tmp<br>            echo &quot;Starting S3 sync...&quot;<br>            while true; do<br>              aws s3 sync s3://airflow-dags-sync-s3/ /opt/airflow/ \<br>                --exclude &quot;*.pyc&quot; \<br>                --exclude &quot;*.log&quot; \<br>                --delete<br>              sleep 30<br>            done<br>        env:<br>          - name: AWS_DEFAULT_REGION<br>            value: &quot;eu-central-1&quot;<br>        volumeMounts:<br>          - name: dags<br>            mountPath: /opt/airflow/dags<br>    extraVolumes:<br>      - name: dags<br>        persistentVolumeClaim:<br>          claimName: airflow-dev-dags</pre><p><strong>Step 2: Set Up the S3 Bucket</strong><br>First, create an S3 bucket dedicated to storing your Airflow DAGs:</p><p>1. Go to the AWS Management Console → S3 → Create bucket. <br>2. Name it something recognizable, like `airflow-dags-sync-s3`. <br>3. Select your preferred region (e.g., `eu-central-1`). <br>4. Enable versioning (optional but recommended for rollback capabilities).</p><p><strong>Step 3: Configure S3 Event Notifications <br></strong>We’ll set up S3 to notify us whenever a DAG file is added, modified, or deleted.</p><p>1. Navigate to your bucket → Properties tab. <br>2. Scroll to Event notifications → Create event notification. <br>3. Configure: <br> — Event name: `AirflowDAGsUpdate` <br> — Event types: <br> — All object create events<br> — All object delete events <br> — All object restore events (optional) <br> — Prefix: `dags/` (to filter only DAG-related changes)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJYGM-q0hrNY5kSGyjJSsw.png" /></figure><p><strong>Step 4: Create AWS Roles:<br></strong> Create GitHubActionsS3SyncRole, should have permissions to access s3.</p><p><strong>Step 5: Configure GitHub Actions for Automated DAG Uploads:<br></strong>To ensure DAG changes are automatically pushed to S3, set up a GitHub Actions workflow:</p><ol><li>Create .github/workflows/sync-dags-to-s3.yaml: ( in your airflow dag repos)</li></ol><pre>name: Sync DAGs to S3<br><br>on:<br>  pull_request:<br>    branches:<br>      - main<br>    paths:<br>      - &#39;dev/dags/**&#39;<br><br>jobs:<br>  sync-dags:<br>    runs-on:<br>      - self-hosted<br>    permissions:<br>      id-token: write<br>      contents: read  <br>    steps:<br>      - name: Checkout Code<br>        uses: actions/checkout@v4<br>      <br>      - name: Install AWS CLI<br>        run: |<br>          sudo apt update<br>          sudo apt install -y awscli<br><br>      - name: Configure AWS Credentials<br>        uses: actions/actions-configure-aws-credentials@main<br>        with:<br>          role-to-assume: arn:aws:iam::123456788910:role/GitHubActionsS3SyncRole<br>          aws-region: eu-central-1<br>          audience: need to set<br><br>      - name: Debug AWS CLI Configuration<br>        run: |<br>          aws sts get-caller-identity<br><br>      - name: Sync DAGs to S3<br>        run: |<br>          aws s3 sync ./dev/dags/ s3://airflow-dags-sync-s3/dags/  --delete</pre><p>2. Store AWS credentials in GitHub Secrets (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`).</p><p><strong>Step 6: Verify the Setup:</strong><br>1. Make a change to a DAG file and push to `main`. <br>2. Check GitHub Actions to confirm the sync ran successfully. <br>3. Check Airflow UI: — your DAG should reload automatically!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-1IiqCQDB0ul18ypNsI3_A.png" /></figure><p><strong>Final Thoughts:</strong><br>By using S3 Event Notifications + CI/CD Workflow , we’ve eliminated manual DAG refreshes while keeping the setup simple.</p><p>No need for SQS or complex polling mechanisms — just a clean, event-driven approach.</p><p>Now, your Airflow DAGs stay in sync effortlessly— letting you focus on writing workflows instead of managing deployments. 🚀</p><p>Have questions or run into issues? Drop a comment below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c97afc9a6444" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Security Seamlessly: DevSecOps Tools to Fortify Your Software Delivery]]></title>
            <link>https://medium.com/@walaaelgenidy/integrating-security-seamlessly-devsecops-tools-to-fortify-your-software-delivery-5baf1f46fcbd?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5baf1f46fcbd</guid>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[devops-tool]]></category>
            <category><![CDATA[security-services]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Wed, 17 Jul 2024 22:12:56 GMT</pubDate>
            <atom:updated>2024-07-17T22:12:56.507Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFIG-MesTR7CAUNjRDKYNw.png" /></figure><p>Embrace the future of software development with DevSecOps tools. In a rapidly evolving digital landscape, ensuring security while maintaining agility is paramount. Explore how the integration of development, security, and operations leads to more secure, efficient, and resilient software applications. Discover the essential tools that empower organizations to build and deploy secure software at speed without compromising on quality or safety.</p><h3>1- Aqua Security:</h3><p><strong>Benefit</strong>: Offers a robust platform for container security, including vulnerability scanning, image assurance, runtime protection, and compliance features, ensuring comprehensive security across the container lifecycle.</p><p><strong>Features</strong>:</p><ul><li><strong>Vulnerability scanning</strong>: Detects vulnerabilities in container images during development and deployment.</li><li><strong>Image assurance</strong>: Ensures that only compliant and secure images are deployed.</li><li><strong>Runtime protection</strong>: Monitors and protects running containers against threats.</li><li><strong>Compliance and auditing</strong>: Helps meet regulatory requirements and provides detailed audit trails.</li></ul><p><strong>Pricing</strong>: Tailored pricing based on specific requirements.</p><p><strong>Security Enhancements</strong>:</p><ul><li><strong>Continuous vulnerability scanning</strong>: Identifies and mitigates security risks in real-time.</li><li><strong>Image integrity</strong>: Ensures images are secure and have not been tampered with.</li><li><strong>Behavioral monitoring</strong>: Detects and responds to anomalous activities in container runtime.</li><li><strong>Compliance automation</strong>: Simplifies adherence to security standards and regulations.</li></ul><p><strong>Estimated Benefit</strong>: Aqua Security can reduce the risk of security breaches by up to 40%, enhancing our security posture and compliance efforts.</p><p><a href="https://www.aquasec.com/">Aqua Cloud Native Application Security</a></p><h3>2- Sysdig Secure:</h3><p><strong>Benefits</strong>: Provides a comprehensive container security platform with advanced features for vulnerability management, runtime security, compliance, and forensics, enhancing the security posture of containerized environments.</p><p><strong>Features</strong>:</p><ul><li><strong>Vulnerability management</strong>: Scans and manages vulnerabilities across the container lifecycle.</li><li><strong>Runtime security</strong>: Protects running containers from threats.</li><li><strong>Compliance and auditing</strong>: Ensures compliance with security standards and regulations.</li><li><strong>Forensics and incident response</strong>: Provides tools for investigating and responding to security incidents.</li></ul><p><strong>Pricing</strong>: Customized pricing based on specific needs.</p><p><strong>Security Enhancements</strong>:</p><ul><li><strong>Comprehensive vulnerability management</strong>: Reduces the attack surface by identifying and fixing vulnerabilities.</li><li><strong>Advanced runtime protection</strong>: Guards against real-time threats and suspicious activities.</li><li><strong>In-depth compliance support</strong>: Automates compliance checks and provides detailed reports.</li><li><strong>Enhanced forensics capabilities</strong>: Facilitates thorough investigation and rapid response to security incidents.</li></ul><p><strong>Estimated Benefit</strong>: Sysdig Secure can enhance our overall security posture by 35–50%, significantly reducing the likelihood of successful cyberattacks.</p><p><a href="https://sysdig.com/products/secure/">Sysdig Platform</a></p><h3>3- Falco:</h3><p><strong>Benefits</strong>: An open-source runtime security project that detects abnormal or malicious activities within containers and Kubernetes clusters, offering real-time threat detection and alerting.</p><p><strong>Features</strong>:</p><ul><li><strong>Real-time threat detection</strong>: Monitors container and host activities to detect threats.</li><li><strong>Abnormal activity monitoring</strong>: Identifies suspicious behavior patterns.</li><li><strong>Integration with Kubernetes and other container platforms</strong>: Seamlessly integrates into existing environments.</li></ul><p><strong>Pricing</strong>: Free to use.</p><p><strong>Security Enhancements</strong>:</p><ul><li><strong>Immediate threat detection</strong>: Provides instant alerts on suspicious activities.</li><li><strong>Behavioral analysis</strong>: Continuously monitors for abnormal behaviors in containers.</li><li><strong>Customizable rules</strong>: Allows the creation of custom security policies and detection rules.</li></ul><p><strong>Estimated Benefit</strong>: Falco can improve runtime security by 25–35%, providing an additional layer of protection for containerized applications.</p><p><strong>Key Difference</strong>: Aqua Security and Sysdig Secure are commercial products offering a broad range of container security features, while Falco provides real-time runtime security as an open-source project.</p><p><a href="https://falco.org/">Falco</a></p><h3>4- KubeClarity and Dependency-Track:</h3><p><strong>Benefits:</strong></p><ul><li><strong>Enhanced Security:</strong></li></ul><p>KubeClarity helps identify misconfigurations and vulnerabilities in Kubernetes deployments, reducing the risk of security breaches.</p><p>Dependency-Track identifies known vulnerabilities in our software dependencies, enabling proactive patching and mitigation.</p><ul><li><strong>Improved Efficiency:</strong></li></ul><p>KubeClarity streamlines troubleshooting and debugging by providing a comprehensive view of the Kubernetes environment.</p><p>Dependency-Track automates the process of identifying and managing software dependencies, saving time and effort.</p><ul><li><strong>Increased Compliance:</strong></li></ul><p>KubeClarity helps ensure compliance with security standards and regulations by providing visibility into our Kubernetes deployments.</p><p>Dependency-Track allows us to track and manage vulnerabilities in our software dependencies, meeting compliance requirements.</p><p><strong>Features:</strong></p><p><strong>KubeClarity:</strong></p><ul><li><strong>Real-time Kubernetes resource monitoring:</strong> Provides insights into the status of deployments, pods, services, and other resources.</li><li><strong>Deployment history tracking:</strong> Offers visibility into past deployments and changes, facilitating rollbacks and troubleshooting.</li><li><strong>Security and configuration analysis:</strong> Identifies potential vulnerabilities and misconfigurations in Kubernetes deployments.</li></ul><p><strong>Dependency-Track:</strong></p><ul><li><strong>Dependency analysis and tracking:</strong> Automatically identifies and tracks software dependencies across our projects.</li><li><strong>Vulnerability detection:</strong> Scans dependencies for known vulnerabilities and provides detailed information.</li><li><strong>Policy enforcement:</strong> Allows defining and enforcing policies for dependency management, including vulnerability thresholds.</li><li><strong>Reporting and dashboards:</strong> Provides comprehensive reports and dashboards for visualizing dependency information and vulnerabilities.</li></ul><p><strong>Cost:</strong></p><ul><li><strong>KubeClarity:</strong> [usage-based pricing].</li><li><strong>Dependency-Track:</strong> [open-source, free to use].</li></ul><ul><li><a href="https://github.com/openclarity/kubeclarity">GitHub - openclarity/kubeclarity: KubeClarity is a tool for detection and management of Software Bill Of Materials (SBOM) and vulnerabilities of container images and filesystems</a></li><li><a href="https://github.com/DependencyTrack/dependency-track">GitHub - DependencyTrack/dependency-track: Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain.</a></li></ul><h3>Policy Enforcement tools:</h3><h3>1- Open Policy Agent (OPA)</h3><p><strong>Benefit</strong>: Enables defining and enforcing custom security policies for Kubernetes resources, providing a flexible and extensible framework for policy management.</p><p><strong>Features</strong>:</p><ul><li><strong>Policy as code</strong>: Write policies using a high-level language.</li><li><strong>Integration with Kubernetes, Envoy, and other platforms</strong>: Wide compatibility for policy enforcement.</li><li><strong>Real-time policy enforcement and decision-making</strong>: Ensures policies are enforced instantly.</li></ul><p><strong>Pricing</strong>: Open-source project, free to use.</p><p><strong>Security Enhancements</strong>:</p><ul><li><strong>Customizable policy enforcement</strong>: Allows precise control over security policies.</li><li><strong>Dynamic decision-making</strong>: Adapts to changing conditions and requirements in real-time.</li><li><strong>Centralized policy management</strong>: Simplifies policy administration across multiple environments.</li></ul><p><a href="https://www.openpolicyagent.org/">Open Policy Agent - Homepage | Open Policy Agent</a></p><h3>2- Kubernetes Admission Controllers</h3><p><strong>Benefit</strong>: Built-in Kubernetes features that enforce security policies and validate incoming requests, ensuring that only compliant resources are deployed.</p><p><strong>Features</strong>:</p><ul><li><strong>Built-in to Kubernetes</strong>: No additional installation required.</li><li><strong>Policy enforcement at admission time</strong>: Validates resources before they are deployed.</li><li><strong>Validation and mutation of resources</strong>: Ensures resources comply with security and operational policies.</li></ul><p><strong>Pricing</strong>: Free to use.</p><p><strong>Security Enhancements</strong>:</p><ul><li><strong>Immediate policy enforcement</strong>: Blocks non-compliant resources from being deployed.</li><li><strong>Simplified compliance</strong>: Ensures adherence to organizational policies and standards.</li><li><strong>Enhanced security posture</strong>: Prevents misconfigurations and unauthorized changes.</li></ul><p><strong>Key Difference</strong>: OPA provides a customizable and flexible policy enforcement framework, while Kubernetes Admission Controllers offer built-in capabilities for policy enforcement.</p><p><a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/">Admission Control in Kubernetes</a></p><h3>Conclusion</h3><p><strong>By harnessing the power of these free and commercial DevOps tools, your organization can unlock significant benefits in security and operational efficiency.</strong></p><ul><li><strong>Security</strong>: Aqua Security and Sysdig Secure offer comprehensive container security features, ensuring robust protection across the container lifecycle. Falco provides real-time monitoring and security, detecting abnormal activities within containers and Kubernetes clusters.</li></ul><p>Collectively, these tools can enhance our security posture by 25–50%, significantly reducing the risk of security breaches.</p><ul><li><strong>Policy Enforcement</strong>: Open Policy Agent allows us to define and enforce custom security policies, while Kubernetes Admission Controllers ensure compliance through built-in policy enforcement capabilities.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5baf1f46fcbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing Your Applications with Keycloak on AWS — EKS Using Helm]]></title>
            <link>https://medium.com/@walaaelgenidy/securing-your-applications-with-keycloak-90936f1ce71c?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/90936f1ce71c</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[security-token]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 13:36:57 GMT</pubDate>
            <atom:updated>2024-07-16T13:37:47.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*0XF6JxrHhdi5APYC_8u-8Q.png" /></figure><p>Keycloak is a feature-rich IAM solution that provides a wide range of functionalities, including user authentication, authorization, and federation. It supports various authentication protocols, such as OpenID Connect, SAML, and OAuth 2.0, making it a versatile choice for integrating with a diverse range of applications.</p><p>Keycloak offers several key benefits for organizations looking to implement a robust identity and access management solution:</p><ol><li><strong>Single Sign-On (SSO)</strong>: Keycloak’s primary feature is its ability to provide SSO, allowing users to authenticate once and access multiple applications within the organization. This improves user experience, reduces login fatigue, and enhances overall security.</li><li><strong>Identity Federation</strong>: Keycloak supports a wide range of identity providers, including LDAP, Active Directory, and social media platforms. This enables organizations to leverage existing user credentials, reducing the need for maintaining separate user accounts.</li><li><strong>Centralized User Management</strong>: Keycloak provides a centralized user management system, allowing administrators to create, update, and manage user accounts from a single interface. This simplifies user lifecycle management and ensures consistent user data across all applications.</li><li><strong>Access Control and Authorization</strong>: Keycloak offers advanced access control features, including role-based access control (RBAC), fine-grained permissions, and policy-based authorization. This allows organizations to enforce and manage access policies consistently across their application landscape.</li><li><strong>Multi-Factor Authentication</strong>: Keycloak supports various multi-factor authentication (MFA) options, such as one-time passwords, SMS, and biometric authentication. This enhances the security of user accounts and protects against unauthorized access.</li><li><strong>Audit and Logging</strong>: Keycloak provides comprehensive auditing and logging capabilities, enabling organizations to monitor user activities, track access patterns, and investigate security incidents effectively.</li><li><strong>Scalability and High Availability</strong>: Keycloak is designed to be scalable and highly available, supporting clustering and load balancing. This ensures that the identity management solution can handle large user populations and remain resilient to failures.</li><li><strong>Open-Source and Extensible</strong>: Keycloak is an open-source project, allowing organizations to customize, extend, and integrate it with their existing IT infrastructure and applications. This flexibility helps organizations tailor the identity management solution to their specific requirements.</li><li><strong>Cost-Effective</strong>: As an open-source solution, Keycloak offers a cost-effective alternative to proprietary identity management products, making it accessible to organizations of all sizes, including small and medium-sized enterprises.</li><li><strong>Standards-Compliant</strong>: Keycloak adheres to industry-standard protocols, such as OpenID Connect, SAML, and OAuth 2.0, ensuring seamless integration with a wide range of applications and platforms.</li></ol><h3>Deploying Keycloak on Amazon EKS</h3><p>To install Keycloak from Bitnami Helm chart on AWS EKS with an external RDS database, you can follow these steps:</p><ol><li><strong>Create an EKS cluster:</strong></li></ol><ul><li>Use the AWS Management Console, AWS CLI, or AWS CloudFormation to create an EKS cluster.</li></ul><p><strong>2. Install the Keycloak Helm chart on the EKS cluster:</strong></p><p>To install Keycloak using Helm, follow these steps:</p><pre>helm repo add bitnami https://charts.bitnami.com/bitnami  <br>helm repo update  <br>helm install my-release oci://registry-1.docker.io/bitnamicharts/keycloak</pre><p><a href="https://github.com/bitnami/charts/tree/main/bitnami/keycloak">charts/bitnami/keycloak at main · bitnami/charts</a></p><h3>Keycloak Concepts (Realms, Users, Roles, Clients):</h3><ul><li><strong>Realms:</strong></li></ul><p>Realms in Keycloak act as security contexts, separating different sets of users, applications, and configurations. Each realm has its own users, roles, settings, and clients. Realms help in isolating and managing different parts of your system securely.</p><ul><li><strong>Users:</strong></li></ul><p>Users in Keycloak represent individuals who can log in to applications secured by Keycloak. User accounts can have various attributes like username, email, and custom attributes. Keycloak provides features for user registration, password management, and more.</p><ul><li><strong>Roles:</strong></li></ul><p>Roles in Keycloak define the permissions granted to users within a realm. Role-based access control (RBAC) is a common practice for managing access to resources. Users can be assigned different roles, which determine what actions they can perform within an application.</p><ul><li><strong>Clients:</strong></li></ul><p>Clients in Keycloak are applications or services that can use Keycloak for authentication and authorization. Keycloak supports various client types such as web applications, single-page applications, and microservices. Clients are registered within a realm and can be configured with specific settings.</p><h3>Creating Realms and Configuring Clients</h3><p><strong>To create a new realm and configure clients, follow these steps:</strong></p><ol><li><strong>Log in to the Keycloak admin console.</strong></li><li><strong>Create a new realm:</strong></li></ol><ul><li>Go to the “Realms” section and click “Create Realm.”</li><li>Provide a name for the new realm and click “Create.”</li></ul><p><strong>3. Configure clients:</strong></p><ul><li>Navigate to the “Clients” section within the realm.</li><li>Click “Create Client” to add a new client.</li><li>Provide the necessary configuration details, such as the client ID, access type, and redirect URIs.</li><li>Save the client configuration.</li></ul><p>Once the realm and clients are set up, you can integrate your applications with Keycloak for user authentication and authorization.</p><h3>Authentication and Authorization</h3><p><strong>Authentication Mechanisms:</strong> Keycloak provides a wide range of authentication mechanisms, including username/password authentication, social login (e.g., Google, Facebook), multi-factor authentication (MFA), and more. You can configure different authentication flows based on your application’s needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z78MQA713yly1rPvEIy2aw.png" /></figure><p><strong>Role-Based Access Control (RBAC):</strong> Keycloak enables you to implement RBAC by assigning roles to users and restricting access to resources based on these roles. By defining roles and associating them with permissions, you can control what users can do within your application.</p><h3>To configure your service to use Keycloak for authentication and authorization with access tokens, you need to perform the following steps:</h3><p><strong>Configure the Keycloak client:</strong></p><ol><li>Create a new client in your Keycloak realm specifically for your service.</li><li>Configure the client to use the appropriate access type (e.g., confidential or public) based on your requirements.</li><li>Set the redirect URI for your service to handle the authentication callback.</li><li>Obtain the Keycloak client credentials:</li><li>Keycloak uses client credentials (client ID and client secret) to authenticate the service.</li><li>Retrieve the client ID and client secret from the Keycloak administration console for your service client.</li><li>Integrate Keycloak with your service:</li><li>Use a Keycloak client library or an OpenID Connect (OIDC) client library in your code to integrate Keycloak with your service.(Developer job)</li><li>Configure the library with the Keycloak server URL, realm name, client ID, and client secret.</li></ol><h4><strong>Note :</strong></h4><p><strong>Client authentication</strong></p><p><strong>confidential : on</strong></p><p><strong>public : off</strong></p><p><strong>The type of OIDC client.</strong></p><p><strong><em>1- ON</em></strong></p><ul><li>For server-side clients that perform browser logins and require client secrets when making an Access Token Request. This setting should be used for server-side applications.</li></ul><p><strong><em>2- OFF</em></strong></p><ul><li>For client-side clients that perform browser logins. As it is not possible to ensure that secrets can be kept safe with client-side clients, it is important to restrict access by configuring correct redirect URIs.</li></ul><h4>The authentication flow:</h4><ul><li>When a user attempts to access a protected resource in your service, redirect them to the Keycloak server for authentication.</li><li>Upon successful authentication, Keycloak will issue an access token to the user.</li><li>Your service should receive the access token in the authentication callback.</li><li>Validate the token’s signature, expiration, issuer, and other necessary claims using the Keycloak public key .</li><li>Once the token is verified, consider the user authenticated and proceed with serving the requested resource.</li></ul><h3>Here’s a high-level overview of the mechanism Keycloak uses to work with tokens:</h3><p>Keycloak uses the OAuth 2.0 and OpenID Connect protocols for authentication and authorization</p><ol><li><strong>User authentication:</strong><br>When a user tries to access a protected resource, Keycloak’s authentication server handles the authentication process, including username/password validation, social login, etc…</li></ol><p><strong>2. Token issuance:</strong><br>After successful authentication, Keycloak’s authorization server issues tokens to the user. The primary tokens used are:</p><ul><li><strong>Access Token:</strong> This token represents the user’s identity and contains information about the user and their granted permissions. It is typically a JSON Web Token (JWT).</li><li><strong>Refresh Token:</strong> This token allows the user to obtain a new access token without re-authenticating, provided the refresh token is still valid.</li></ul><p><strong>3. Token verification:</strong><br>The service, upon receiving an access token, verifies its authenticity and validity.<br>It checks the token’s signature, expiration time, issuer, audience, and any other required claims.</p><p><strong>4. Authorization enforcement:</strong><br>Once the token is verified, the service uses the information in the token to perform authorization checks.<br>It can access the user’s roles, permissions, and other relevant details to determine if the user is authorized to access the requested resource.</p><p>By following these steps, you can easily deploy Keycloak on your Amazon EKS cluster and leverage its robust identity and access management capabilities to secure your applications.</p><p>Whether you’re managing a standalone application or a complex, distributed system, Keycloak’s flexibility and powerful features can help you address your identity and access management needs effectively.</p><ul><li><a href="https://www.keycloak.org/">Keycloak</a></li><li><a href="https://artifacthub.io/packages/helm/bitnami/keycloak">keycloak 21.6.2 · bitnami/bitnami</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90936f1ce71c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring GitOps and Traditional CICD Approaches]]></title>
            <link>https://medium.com/@walaaelgenidy/exploring-gitops-and-traditional-cicd-approaches-46a028947ded?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/46a028947ded</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Mon, 15 Jul 2024 23:49:29 GMT</pubDate>
            <atom:updated>2024-11-18T21:11:31.907Z</atom:updated>
            <content:encoded><![CDATA[<h3>GitOps vs. Traditional CI/CD: Choosing the Right Path for Modern Software Delivery</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GHm0mgXI-Cb4-5ioZstifg.png" /></figure><p>In the world of software development, the continuous integration and continuous deployment (CICD) pipeline has become a fundamental component of modern application delivery.</p><p>Two distinct approaches have emerged in this realm: the GitOps model and the traditional CICD approach. Each model offers unique benefits and is suited for specific use cases within an organization’s software development lifecycle.</p><h4>The GitOps Model:</h4><p>The GitOps model is an approach to CICD that leverages Git as the single source of truth for both the application code and the infrastructure configuration. In this model, developers manage the desired state of the entire system, including both the application and its supporting infrastructure, in a Git repository.</p><h4>Key Characteristics of the GitOps Model:</h4><p><strong>- Git-Centric Workflow: </strong>All changes to the application and infrastructure are managed through Git, where developers commit and push their changes.</p><p><strong>- Declarative Infrastructure Configuration: </strong>Infrastructure resources are defined declaratively, using tools like Terraform, Ansible, or Kubernetes manifests, and stored in the Git repository.</p><p><strong>- Automated Deployment Pipeline: </strong>The CICD pipeline automatically detects changes in the Git repository and applies them to the production environment, ensuring the desired state is consistently maintained.</p><p><strong>- Self-Healing Infrastructure:</strong> The GitOps model enables the continuous reconciliation of the production environment with the desired state defined in the Git repository, ensuring the system is always in the correct state.</p><h4>The Traditional CICD Approach:</h4><p>The traditional CICD approach is the classic method of implementing a CICD pipeline, where the application code, infrastructure configuration, and deployment scripts are managed separately.</p><h4><strong>Key Characteristics of the Traditional CICD Approach:</strong></h4><p><strong>- Separation of Concerns: </strong>The application code, infrastructure configuration, and deployment scripts are typically stored in different repositories or locations.</p><p><strong>- Imperative Deployment Process:</strong> The deployment process is defined using scripts or automation tools, which execute a series of steps to deploy the application and configure the infrastructure.</p><p><strong>- Centralized CICD Pipeline:</strong> The CICD pipeline is typically managed and maintained by a central team or platform, such as Jenkins or Travis CI.</p><p><strong>- Manual Intervention:</strong> The traditional CICD approach often requires manual intervention, such as approvals or manual trigger of the deployment process.</p><h4>Applying GitOps and Traditional CICD with ArgoCD and GitHub Actions:</h4><p><strong>GitOps with ArgoCD:</strong><br>ArgoCD is a popular open-source tool that enables the GitOps model for Kubernetes-based applications. With ArgoCD, organizations can:<br>1. Define their desired state in Git repositories using Kubernetes manifests.<br>2. Automatically reconcile the production environment with the desired state defined in Git.<br>3. Leverage ArgoCD’s built-in features for self-healing, rollbacks, and multi-cluster deployment.</p><p><strong>Traditional CICD with GitHub Actions:</strong></p><p>GitHub Actions is a CICD platform that can be used to implement the traditional CICD approach. With GitHub Actions, organizations can:<br>1. Manage application code, infrastructure configuration, and deployment scripts in separate repositories.<br>2. Define their deployment process using YAML-based workflow files, which can include steps for building, testing, and deploying the application.<br>3. Integrate with various tools and services to customize the CICD pipeline to their specific needs.</p><h4>Use Cases and Benefits:</h4><p><strong>GitOps Model:</strong><br><strong>- Complex, Distributed Systems:</strong></p><p>The GitOps model is well-suited for managing the deployment and infrastructure of complex, distributed systems, such as microservices-based applications running on Kubernetes.</p><p><strong>- Improved Reliability and Consistency:</strong></p><p>The declarative nature of the GitOps model ensures that the production environment is always in sync with the desired state defined in the Git repository, reducing the risk of configuration drift and ensuring consistent deployments.</p><p><strong>- Simplified Debugging and Rollbacks:</strong></p><p>The Git commit history provides a clear audit trail, making it easier to identify and roll back problematic changes.</p><p><strong>Traditional CICD Approach:</strong><br><strong>- Legacy Systems and Customization:</strong></p><p>The traditional CICD approach may be better suited for organizations with legacy systems or complex infrastructure that may not fit well within the declarative GitOps model.</p><p><strong>- Faster Deployment of Simple Applications:</strong></p><p>For simple applications with minimal infrastructure requirements, the traditional CICD approach can provide a more streamlined and faster deployment process.</p><p><strong>- Broader Ecosystem of Tools and Integrations:</strong></p><p>The traditional CICD approach has a more extensive ecosystem of tools and integrations, providing more options for organizations to choose from.</p><p>The choice between the GitOps model and the traditional CICD approach will depend on the organization’s specific requirements, existing infrastructure, and the complexity of the software development lifecycle.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46a028947ded" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the AWS Multi-Account Strategy: Shared VPCs vs.]]></title>
            <link>https://medium.com/@walaaelgenidy/navigating-the-aws-multi-account-strategy-shared-vpcs-vs-136b4da4592c?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/136b4da4592c</guid>
            <category><![CDATA[vpc]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[shared-vpc]]></category>
            <category><![CDATA[strategy]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Sat, 13 Jul 2024 00:29:34 GMT</pubDate>
            <atom:updated>2024-07-13T00:31:00.011Z</atom:updated>
            <content:encoded><![CDATA[<h3>Navigating the AWS Multi-Account Strategy: Shared VPCs vs. Default VPCs for Optimized Cloud Networking</h3><p>In the dynamic world of cloud computing, the architectural choices we make for our virtual network environments can significantly impact the performance, security, and manageability of our applications. AWS, the leading cloud provider, offers two primary models for virtual private cloud (VPC) implementation: <em>the Shared VPC and the Default VPC.</em></p><p>Understanding the trade-offs between these models is crucial for organizations looking to optimize their cloud infrastructure and stay ahead of the curve.</p><h4>Shared VPCs: Centralizing Networking for Collaboration and Efficiency</h4><p>The Shared VPC model in AWS allows multiple AWS accounts to share a single VPC, enabling seamless communication and resource sharing across these accounts. This approach offers several benefits, making it a compelling choice for organizations with complex networking requirements.</p><h4>Use Cases:</h4><p><strong>- Empowering Multi-Tenant Environments:</strong>Shared VPCs are particularly useful for organizations with multiple teams or business units that require isolated but interconnected virtual networks.</p><p><strong>- Centralizing Network Management:</strong> By sharing a VPC, organizations can centralize the management of networking resources, such as subnets, routing tables, and security groups, across their AWS accounts.</p><p><strong>- Maintaining Consistent Networking Policies: </strong>With a Shared VPC, organizations can apply consistent network policies and configurations across their different environments, ensuring a cohesive and secure infrastructure.</p><h4>Key Benefits:</h4><p><strong>- Reduced Operational Complexity:</strong>Shared VPCs simplify network management by allowing teams to collaborate on a single VPC, reducing the need to maintain and configure multiple VPCs.</p><p><strong>- Optimized Resource Utilization:</strong> By sharing a VPC, organizations can maximize the use of network resources, such as IP address ranges and routing tables, leading to more efficient resource utilization.</p><p><strong>- Enhanced Security through Centralization: </strong>Shared VPCs enable centralized control over network security, allowing for the implementation of consistent security policies and access controls across all connected accounts.</p><h4>Potential Drawbacks:</h4><p><strong>- Increased Governance Complexity:</strong> Sharing a VPC requires careful coordination and governance among the participating accounts, as changes in one account can potentially impact others.</p><p><strong>- Limited Isolation:</strong> While Shared VPCs provide logical isolation between accounts, there is a higher level of trust required among the participating teams, as they share the same network infrastructure.</p><p>Example arch for shared VPC model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jTcxVyoWYwBHSjcoBgZQeQ.png" /></figure><h4>Default VPCs: Simplicity and Isolation for Quick Deployment</h4><p>The Default VPC model is the standard VPC setup that AWS provides for new AWS accounts. It offers a quick and easy way to get started with networking in the cloud, without the need for extensive configuration.</p><h4>Use Cases:</h4><p><strong>- Rapid Deployment of Resources: </strong>The Default VPC is ideal for quickly launching resources in the cloud, particularly for small-scale or proof-of-concept applications.</p><p><strong>- Isolating Workloads and Environments: </strong>Default VPCs provide a dedicated virtual network for each AWS account, ensuring a high degree of isolation between different workloads and environments.</p><h4>Key Benefits:</h4><p><strong>- Simplicity and Ease of Use: </strong>The Default VPC model requires minimal configuration, making it a straightforward choice for users who are new to AWS or have limited networking requirements.</p><p><strong>- Isolation and Separation: </strong>Each AWS account has its own Default VPC, providing a clear separation between different workloads and environments.</p><h4>Potential Drawbacks:</h4><p><strong>- Limited Customization Capabilities: </strong>The Default VPC model offers limited flexibility in terms of network configuration, such as the ability to customize subnets, routing tables, and security groups.</p><p><strong>- Potential Resource Duplication: </strong>In scenarios where multiple Default VPCs are utilized, there may be a higher risk of resource duplication and inefficient resource utilization.</p><h4>Selecting the Optimal VPC Model for Your Organization:</h4><p>When deciding between the Shared VPC and Default VPC models, organizations should consider the following factors:</p><ol><li><strong>Networking Complexity:</strong></li></ol><p>If your organization has complex networking needs, such as multi-tenant environments or the need for centralized network management, the Shared VPC model may be more suitable. Conversely, if your requirements are simpler, the Default VPC model may be a better fit.</p><p><strong>2. Security and Governance:</strong></p><p>The Shared VPC model requires more robust security controls and governance mechanisms to ensure the integrity of the shared network infrastructure. The Default VPC model, on the other hand, provides a higher degree of isolation, but may lack the same level of centralized security management.</p><p><strong>3. Operational Efficiency:</strong></p><p>The Shared VPC model can simplify network management by allowing teams to collaborate on a single VPC, but it also introduces additional coordination and governance challenges. The Default VPC model is generally more straightforward to set up and manage.</p><p><strong>4. Resource Optimization:</strong></p><p>The Shared VPC model can potentially lead to more efficient resource utilization by allowing for the optimization of IP address ranges and routing tables. The Default VPC model may result in resource duplication in scenarios where multiple VPCs are utilized.</p><h4>Let’s dive deeper into the security considerations when implementing a Shared VPC model across multiple AWS accounts:</h4><p><strong>1- Access Control and Identity Management:</strong></p><p>- Carefully manage IAM permissions and roles to ensure that each account has the appropriate level of access to the shared VPC resources.<br>- Implement least-privilege principles and follow the principle of “need-to-know” when granting permissions.<br>- Regularly review and audit IAM policies to ensure they align with your security best practices.</p><p><strong>2- Network Segmentation and Isolation:</strong></p><p>- Utilize subnets and security groups to logically isolate resources within the Shared VPC, ensuring proper network segmentation.<br>- Implement robust access control lists (ACLs) and network ACLs to control inbound and outbound traffic at the subnet level.<br>- Carefully manage route tables to ensure proper routing between the connected accounts and external networks.</p><p><strong>3- Security Groups and Network ACLs:</strong><br>- Establish consistent security group and network ACL rules across all connected accounts to enforce a unified security policy.<br>- Regularly review and update these rules to adapt to changing security requirements and threat landscapes.<br>- Ensure that security groups and network ACLs are configured with the principle of “least privilege” in mind.</p><p><strong>4- Logging and Monitoring:</strong></p><p>- Enable comprehensive logging and monitoring across the Shared VPC, including VPC Flow Logs, CloudTrail, and Amazon GuardDuty.<br>- Centralize the logging and monitoring data to facilitate cross-account analysis and incident response.<br>- Regularly review the logs and monitoring data to detect and respond to potential security incidents or anomalies.</p><p><strong>5- Shared Services and Resources:</strong></p><p>- Carefully manage the sharing of services and resources (e.g., databases, application servers) between the connected accounts to minimize the risk of unauthorized access or data leakage.<br>- Implement robust access controls and data encryption mechanisms to protect the shared services and resources.<br>- Establish clear ownership and responsibility models for the shared services and resources.</p><p><strong>6- Incident Response and Disaster Recovery:</strong><br>- Develop a comprehensive incident response plan that addresses scenarios specific to the Shared VPC model.<br>- Ensure that the incident response plan includes clear communication and escalation processes across the connected accounts.<br>- Regularly test and update the incident response plan to maintain its effectiveness.</p><p>By carefully evaluating these factors and understanding the trade-offs between the Shared VPC and Default VPC models, organizations can make an informed decision that aligns with their specific requirements and helps them maximize the benefits of their cloud infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=136b4da4592c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crossplane and DevOps: Streamlining Infrastructure Provisioning — Episode 3]]></title>
            <link>https://medium.com/@walaaelgenidy/crossplane-and-devops-streamlining-infrastructure-provisioning-episode-3-7c0c8152ecdf?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c0c8152ecdf</guid>
            <category><![CDATA[resource-management]]></category>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[iac]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Thu, 11 Jul 2024 21:49:49 GMT</pubDate>
            <atom:updated>2024-07-24T00:41:55.274Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crossplane and DevOps: Streamlining Infrastructure Provisioning — Episode 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4i4o21Jq2G1cWpYEPm8Gg.jpeg" /></figure><p>Integrating Crossplane with DevOps practices can significantly streamline your infrastructure provisioning process. Crossplane, a Kubernetes-native open-source project, allows you to define your infrastructure as Kubernetes custom resources, promoting consistency and portability across different environments.</p><p>When working with Crossplane in a DevOps setup, one of the key challenges is maintaining Infrastructure as Code (IaC) consistency across development, testing, and production environments. Here’s how you can approach this:</p><ol><li><strong>Separation of Environments:</strong><br>- Create a directory structure that separates your Crossplane resource definitions by environment, such as:</li></ol><pre>infrastructure/app/<br> ├── dev/<br> │ ├── database.yaml<br> │ ├── storage.yaml<br> │ └── network.yaml<br> ├── staging/<br> │ ├── database.yaml<br> │ ├── storage.yaml<br> │ └── network.yaml<br> └── prod/<br> ├── database.yaml<br> ├── storage.yaml<br> └── network.yaml</pre><p>— This organization makes it easy to manage and apply the correct resource definitions for each environment.</p><p><strong>2. Environment Variables:</strong><br> — Utilize environment variables within your Crossplane resource definitions to make them more reusable and adaptable.</p><p>— For example, you can use environment variables for database connection strings, storage bucket names, and network configurations.</p><p>— This allows you to maintain a single set of resource definitions while customizing the values for each environment.</p><p><strong>3. Crossplane Composition :</strong><br> — Leverage Crossplane’s Composition feature to create reusable and modular resource definitions.</p><p>— This allows you to define common resource patterns and then compose them together to create environment-specific configurations.</p><p>Let’s expand the concepts of claims, definitions, and compositions in a more accessible way:</p><p>In the world of Crossplane, there are a few key concepts to understand:</p><p><strong>1.Claims:</strong></p><p>These are the resources that your application or service needs, such as an S3 bucket, a database, or a Kubernetes cluster. Essentially, a claim is a request for a specific type of infrastructure resource.</p><p><strong>2. Definitions:</strong></p><p>These are the actual infrastructure resources that Crossplane can provision, such as an AWS S3 bucket, an RDS instance, or a Kubernetes cluster. Definitions describe the desired state of the infrastructure resource.</p><p><strong>3. Compositions:</strong></p><p>Compositions allow you to combine multiple definitions into a single, more complex resource. This helps you create reusable and modular infrastructure configurations.</p><h4><strong><em>Let’s break down these concepts with a simple example:</em></strong></h4><p>Imagine you have a web application that needs to store user data. Your application would create a “Database Claim” that requests a database resource.</p><p>Crossplane would then look at the available “Database Definitions” (e.g., AWS RDS, Google Cloud SQL, etc.) and choose the appropriate one based on your configuration.</p><p>Crossplane would then provision the necessary infrastructure, including the database instance, networking, and security settings, and connect it to your application.</p><p><strong>Example: Provisioning an RDS Instance with Crossplane:</strong></p><p>Now, let’s look at an example of how you can use Crossplane to provision an RDS instance:</p><ol><li><strong>Define the RDS Instance(definition):</strong></li></ol><pre><br>apiVersion: apiextensions.crossplane.io/v1<br>kind: CompositeResourceDefinition<br>metadata:<br>spec:<br>  names:<br>    kind: XRdsInstance<br>    plural: xrdsinstances<br>  claimNames:<br>    kind: RdsInstance<br>    plural: rdsinstances<br>  defaultCompositionRef:<br>    name: rds-composition<br>  versions:<br>    - name: v1alpha1<br>      served: true<br>      referenceable: true<br>      schema:<br>        openAPIV3Schema:<br>          type: object<br>          properties:<br>            spec:<br>              type: object<br>              properties:<br>                parameters:<br>                  type: object<br>                  properties:<br>                    engine:<br>                      type: string<br>                      enum: [&quot;mysql&quot;, &quot;mariadb&quot;, &quot;postgres&quot;]<br>                    storageGB:<br>                      type: integer<br>                    region:<br>                      type: string<br>                      default: &quot;eu-central-1&quot;<br>                    instanceClass:<br>                      type: string<br>                    databaseName:<br>                      type: string<br>            status:<br>              type: object<br>              properties:<br>                instanceId:<br>                  type: string<br>                dbPassword:<br>                  type: string<br>                dbEngine:<br>                  type: string<br>                dbEndpoint:<br>                  type: string<br>                dbAddress:<br>                  type: string<br>                dbPort:<br>                  type: integer<br>                dbName:<br>                  type: string<br>                dbUsername:<br>                  type: string</pre><p>This definition describes the desired RDS instance, including the database engine, version, instance class, and master username and password.</p><p><strong>2. Define the necessary resources and configurations to provision an RDS instance ( composition):</strong></p><pre>apiVersion: apiextensions.crossplane.io/v1<br>kind: Composition<br>metadata:<br>  name: rds-composition<br>  labels:<br>    crossplane.io/xrd: rds.crossplane.io<br>    provider: aws<br>spec:<br>  compositeTypeRef:<br>    apiVersion: crossplane.io/v1alpha1<br>    kind: XRdsInstance<br>  resources:<br>    - name: subnet-group<br>      base:<br>        apiVersion: rds.aws.upbound.io/v1beta1<br>        kind: SubnetGroup<br>        spec:<br>          providerConfigRef:<br>            name: upbound-provider-family-aws<br>      readinessChecks:<br>        - matchCondition:<br>            status: &#39;True&#39;<br>            type: Ready<br>          type: MatchCondition<br><br>    - name: rds-instance<br>      base:<br>        apiVersion: rds.aws.upbound.io/v1beta1<br>        kind: Instance<br>        spec:<br>          forProvider:<br>            tags:<br>              backup: &quot;true&quot;<br>            autoMinorVersionUpgrade: true<br>            applyImmediately: false<br>            username: &quot;dbadmin&quot;<br>            publiclyAccessible: false<br>            multiAz: false<br>            caCertIdentifier: &quot;rds-ca-2019&quot;<br>            autoGeneratePassword: true<br>            manageMasterUserPassword: false<br>            passwordSecretRef:<br>              key: password<br>              name: rds-password-secret<br>              namespace: my-namespace<br>            skipFinalSnapshot: true<br>            iamDatabaseAuthenticationEnabled: true<br>            storageEncrypted: true<br>          providerConfigRef:<br>            name: upbound-provider-family-aws<br>      readinessChecks:<br>        - matchCondition:<br>            status: &#39;True&#39;<br>            type: Ready<br>          type: MatchCondition<br><br>    - name: grab-db-password-from-secret<br>      base:<br>        apiVersion: kubernetes.crossplane.io/v1alpha1<br>        kind: Object<br>        metadata:<br>          name: observesecret<br>        spec:<br>          managementPolicy: Observe<br>          forProvider:<br>            manifest:<br>              apiVersion: v1<br>              kind: Secret<br>              type: opaque<br>              metadata:<br>                name: rds-password-secret<br>                namespace: my-namespace<br>          providerConfigRef:<br>            name: kubernetes-provider<br>      readinessChecks:<br>        - type: None</pre><p><strong>3. Define the AWS Provider Configuration:</strong></p><pre>apiVersion: aws.crossplane.io/v1beta1<br> kind: ProviderConfig<br> metadata:<br> name: aws-provider<br> spec:<br> credentials:<br> source: Secret<br> secretRef:<br> namespace: crossplane-system<br> name: aws-provider-creds<br> key: credentials</pre><p>This defines the AWS provider configuration, which includes the credentials for Crossplane to access your AWS account.</p><p><strong>4. Provision the RDS Instance (claim):</strong></p><pre>apiVersion: database.crossplane.io/v1alpha1<br>kind: RDSInstance<br>metadata:<br>  name: rds<br>spec:<br>  compositionRef:<br>    name: rds-composition<br>  parameters:<br>    engine: postgres<br>    region: eu-central-1<br>    databaseName: database<br>    storageGB: 20<br>    instanceClass: db.t3.micro<br>    storageType: gp2<br>    username: dbadmin<br>    passwordSecretRef:<br>      name: rds-password-secret<br>      namespace: my-namespace<br>      key: password<br></pre><p>After applying these resource definitions, Crossplane will provision the RDS instance in your AWS account, and your application can then connect to the database.</p><p>This example demonstrates how Crossplane allows you to define your infrastructure resources as Kubernetes custom resources, making it easier to manage and deploy them across different environments.</p><p>By using claims, definitions, and compositions, you can create more complex and reusable infrastructure configurations, ensuring consistency and portability across your development, testing, and production environments.</p><p>In conclusion, the marriage of Crossplane and DevOps practices presents a robust framework for optimizing infrastructure provisioning processes. By adopting a structured approach, embracing environment variables, and harnessing Crossplane’s composition capabilities, teams can unlock enhanced consistency and portability in managing infrastructure resources across diverse environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c0c8152ecdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow Deployment on Amazon EKS: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@walaaelgenidy/airflow-deployment-on-amazon-eks-a-step-by-step-guide-c1a9ae3703c3?source=rss-90eb4fdc372d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1a9ae3703c3</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[airflow-installation]]></category>
            <dc:creator><![CDATA[Walaa Elgenidy]]></dc:creator>
            <pubDate>Fri, 05 Jul 2024 22:23:37 GMT</pubDate>
            <atom:updated>2025-07-16T12:56:03.387Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Airflow is an open-source platform used for orchestrating complex workflows and data pipelines. It allows users to define workflows as directed acyclic graphs (DAGs), where each node represents a task to be executed. Airflow facilitates scheduling, monitoring, and managing these workflows efficiently, providing features such as task dependencies, retries, alerts, and logging. With its extensible architecture and rich ecosystem of plugins, Airflow has become a popular choice for managing ETL (Extract, Transform, Load) processes, data processing pipelines, and other automation tasks in various industries.</p><p>This will guide you to <em>deploy airflow</em> on an existing <em>EKS cluster</em></p><p>The key highlights of this deployment include:</p><ul><li><strong>Namespace Creation</strong>: We started by creating a dedicated namespace for the Airflow deployment, ensuring a clean and organized Kubernetes environment.</li><li><strong>Helm Chart Configuration</strong>: By customizing the Airflow Helm chart, we were able to integrate our existing AWS RDS database, allowing for a smooth transition from your current infrastructure.</li><li><strong>Deployment and Accessibility</strong>: With the Airflow Helm chart installed, we were able to access the Airflow web UI through a Kubernetes service and Ingress setup, making it readily available to your team.</li><li><strong>Troubleshooting and Best Practices</strong>: We also addressed common issues, such as the Fernet key mismatch, and provided guidance on managing user permissions and connections within the Airflow platform.</li></ul><h4>Prerequiste:</h4><ul><li>EKS cluster</li><li>IAM admin role or enough permission to EKS node group to have access to RDS .</li><li>Helm-chart installed</li><li>Need to to open a tunnel between the cluster and the RDS , add the cluster SG to the inbound in the RDS SG.</li></ul><p><strong>Airflow main components</strong>: Worker, scheduler, web server , database and redis .</p><h3>Step-by-step guide</h3><ul><li>First make sure that you are in the correct context.</li><li>To deploy Airflow on Kubernetes, the first step is to create a namespace .</li></ul><pre>-   kubectl create namespace airflow</pre><ul><li>You need to fetch the official Helm of Apache Airflow that will get deployed on your cluster.</li></ul><pre>-   helm repo add apache-airflow https://airflow.apache.org</pre><pre>   -   helm repo update</pre><ul><li>you need to configure your values.yaml based on your needs for example we need to use our rds with airflow not it’s own database as it comes with a very basic setup. to do that configure this part:</li></ul><pre><br># Airflow database &amp; redis config<br>data:<br> # you can create a secret and pass it&#39;s name here<br>metadataSecretName: airflowexternaldbsecret<br>  # When providing secret names and using the same database for metadata and<br>  # result backend, for Airflow &lt; 2.4.0 it is necessary to create a separate<br>  # secret for result backend but with a db+ scheme prefix.<br>  # For Airflow &gt;= 2.4.0 it is possible to not specify the secret again,<br> <br>  # Otherwise pass connection values in<br>metadataConnection:<br>  user: postgres<br>  pass: postgres<br>  protocol: postgresql<br>  host: example.eu-central-1.rds.amazonaws.com<br>  port: &quot;5432&quot;<br>  db: postgres # name of the database you want to use<br>  sslmode: enable</pre><p>Select certain nodes for airflow worker pods:</p><pre>nodeSelector:<br>key: value</pre><p>also add tolerations:</p><pre><br>tolerations:<br>      - key: key<br>        operator: Equal<br>        value: value<br>        effect: NoSchedule</pre><p>then install:</p><pre>-  helm install airflow apache-airflow/airflow --namespace airflow --debug</pre><p>To access the Airflow UI, open a new terminal and execute the following command (locally)</p><pre>- kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow</pre><p>Make sure that everything is fine , most common issues happens due to database connection so go back to perquisite part and configure everything first .</p><p>After making sure that everything is fine create your ingress to expose this app to public:</p><pre><br>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: airflow-ingress<br>  namespace: airflow<br>spec:<br>  ingressClassName: traefik<br>  rules:<br>  - host: airflow.example.com<br>    http:<br>      paths:<br>      - backend:<br>          service:<br>            name: airflow-webserver<br>            port:<br>              number: 8080<br>        path: /<br>        pathType: Prefix</pre><p>Now you have your airflow online and available to users on <strong>airflow.example.com</strong></p><p><strong>NOTE:</strong></p><ul><li>you can add users from terminal to airflow with roles like this :</li></ul><pre>kubectl exec PODNAME  -n NAMESPACE -- airflow users create -u USEENAME -f FIRSTNAME -l LASTNAME -r ROLE -e EXAMPLE@example.com -p PASS</pre><ul><li>If you faced this common issue when trying to access connections</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QIMTR9UXNeuhrj3-VQ5Gg.png" /></figure><h4>Root cause :</h4><p>leaving the default values for fernetKey in values.yaml so a random key was generated. We have the postgres DB in a persistent volume, but we don&#39;t want to reset the DB and clear all existing dag run history. so we want to fix the mismatch between the newly generated key in the airflow web-server and the existing one in the db.</p><p><strong>please follow these steps :</strong></p><p>resolved it without loosing any history data or whatever. The point is that you need to delete all of your existing airflow connections</p><ul><li>First exec into your webserver container:</li></ul><pre>kubectl exec PODNAME  -n NAMESPACE -- /bin/bash</pre><ul><li>After getting into your container hit this command to access PostgreSQL RDS:</li></ul><pre><br>airflow db shell</pre><ul><li>You will find them here:</li></ul><pre>SELECT conn_id FROM connection;</pre><ul><li>Then iterate over them and delete:</li></ul><pre>SELECT * FROM connection;<br>DELETE FROM connection WHERE id = &lt;id from above&gt;;</pre><ul><li>After deleting the connections it will work fine directly.</li></ul><h4>Conclusion</h4><p>In this article, we’ve explored the step-by-step process of deploying Apache Airflow on an Amazon EKS cluster. By leveraging the power of Kubernetes and the flexibility of Airflow, you can now seamlessly integrate your data pipelines and workflows into a scalable and dynamic environment.</p><blockquote><a href="https://medium.com/@walaaelgenidy/syncing-dags-from-s3-to-airflow-on-eks-c97afc9a6444">https://medium.com/@walaaelgenidy/syncing-dags-from-s3-to-airflow-on-eks-c97afc9a6444</a><br><strong>check this for dags syncing</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1a9ae3703c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>