<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Madhav Prajapati on Medium]]></title>
        <description><![CDATA[Stories by Madhav Prajapati on Medium]]></description>
        <link>https://medium.com/@MadhavPrajapati?source=rss-bd161dccc754------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*o5_yopb9zi1WCWDtAcdFZw.jpeg</url>
            <title>Stories by Madhav Prajapati on Medium</title>
            <link>https://medium.com/@MadhavPrajapati?source=rss-bd161dccc754------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:59:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@MadhavPrajapati/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Security Headers from your NGINX Ingress configuration.]]></title>
            <link>https://medium.com/@MadhavPrajapati/security-headers-from-your-nginx-ingress-configuration-54c795915abd?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/54c795915abd</guid>
            <category><![CDATA[security-header]]></category>
            <category><![CDATA[http-headers]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[nginx-configuration]]></category>
            <category><![CDATA[nginx]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 07:04:53 GMT</pubDate>
            <atom:updated>2025-08-18T08:06:20.700Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TE5TF_jVX0cGFVKJLKdSDw.png" /></figure><h3>Why security headers are important?</h3><p>many websites and systems can be hacked due to a lack or bad configuration or lack of protection. Besides improving your site’s SEO score, Security headers will protect your website from some common attacks like Code injection, XSS, click-jacking, and more.</p><h3>1. X-Content-Type-Options</h3><h4>Purpose</h4><p>Prevents browsers from MIME-sniffing a response away from the declared Content-Type header.</p><h4>Possible Values</h4><ul><li><strong>nosniff</strong> (only valid value<br> — Instructs browsers to strictly follow the declared content type instead. of guessing the type.</li></ul><h4>Recommended Usage</h4><ul><li>Always use nosniff to prevent drive-by download attacks and reduce XSS risks.</li><li>Use this header on all responses serving scripts, styles, or any content with strict MIME types.</li></ul><h4>Example</h4><pre>X-Content-Type-Options: nosniff</pre><h3>2. X-Frame-Options</h3><h4>Purpose</h4><p>Controls whether your site can be embedded in &lt;iframe&gt;, protecting against clickjacking attacks.</p><h4>Possible Values</h4><ul><li><strong>DENY<br> — </strong>No framing allowed; your site cannot be embedded in any iframe.</li><li><strong>SAMEORIGIN<br> — </strong>Only allow framing by pages from the same origin.</li><li><strong>ALLOW-FROM URI</strong> (less supported)<br> — Allow framing only by a specific trusted origin.</li></ul><h4>Recommended Usage</h4><ul><li>Use SAMEORIGIN if your site or app uses frames internally.</li><li>Use DENY if you want to disallow framing totally and ensure maximum protection.</li><li>Avoid using ALLOW-FROM due to poor browser support.</li></ul><h4>Example</h4><pre>X-Frame-Options: SAMEORIGIN</pre><h3>3. Referrer-Policy</h3><h4>Purpose</h4><p>Controls what information is sent in the Referer header when navigating from your site, protecting user privacy.</p><h4>Common Possible Values</h4><ul><li><strong>no-referrer<br> — </strong>No Referer header is sent at all.</li><li><strong>no-referrer-when-downgrade<br> — </strong>Sends the Referer header except when navigating from HTTPS to HTTP.</li><li><strong>origin<br> — </strong>Sends only the origin (scheme + host + port), omitting path and query.</li><li><strong>strict-origin<br> — </strong>Sends origin but only for secure-to-secure navigation.</li><li><strong>origin-when-cross-origin<br> — </strong>Full URL for same-origin, only origin for cross-origin.</li><li><strong>strict-origin-when-cross-origin</strong> (recommended)<br> — Full URL for same-origin, origin for HTTPS→HTTPS cross-origin, no referer for HTTPS→HTTP.</li><li><strong>unsafe-url<br> — </strong>Always sends full URL (insecure for privacy).</li></ul><h4>Recommended Usage</h4><ul><li>Prefer strict-origin-when-cross-origin for good privacy and usability balance.</li><li>Use no-referrer or same-origin for higher privacy needs.</li></ul><h4>Example</h4><pre>Referrer-Policy: strict-origin-when-cross-origin</pre><h3>4. Strict-Transport-Security (HSTS)</h3><h4>Purpose</h4><p>Forces browsers to only use HTTPS when connecting to your domain for a set period.</p><h4>Flags and Values</h4><ul><li><strong>max-age=SECONDS<br> — </strong>Duration in seconds browsers should enforce HTTPS (e.g., 31536000 equals 1 year).</li><li><strong>includeSubDomains<br> — </strong>Applies the policy to all subdomains.</li><li><strong>preload<br> — </strong>Requests inclusion in browser preload lists (sites hardcoded to use HTTPS).</li></ul><h4>Recommended Usage</h4><ul><li>Set a long max-age (1 year or more).</li><li>Include includeSubDomains if you control all subdomains and want full HTTPS enforcement.</li><li>Use preload only after testing your site supports strict HTTPS.</li><li>Important: Only use on sites fully configured with SSL.</li></ul><h4>Example</h4><pre>Strict-Transport-Security: max-age=31536000; includeSubDomains; preload</pre><h3>5. Cross-Origin-Resource-Policy (CORP)</h3><h4>Purpose</h4><p>Restricts which origins can load your resources like images, fonts, scripts.</p><h4>Possible Values</h4><ul><li><strong>same-origin<br> — </strong>Only same-origin documents can load the resource.</li><li><strong>same-site<br> — </strong>Only documents from same site (including subdomains) can load resource.</li><li><strong>cross-origin<br> — </strong>No restrictions; any origin can load.</li></ul><h4>Recommended Usage</h4><ul><li>Use same-site to enable resource sharing within your site and subdomains securely.</li><li>Use same-origin for maximum restriction.</li><li>Use cross-origin only when resources are intentionally public.</li></ul><h4>Example</h4><pre>Cross-Origin-Resource-Policy: same-site</pre><h3>6. Content-Security-Policy (CSP)</h3><h4>Purpose</h4><p>Configures granular control over sources of executable scripts, styles, images, fonts, frames, and other resources — plus inline code execution.</p><h4>Key Directives and Common Values</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*pZL8wawN1roHEKpMZPTB0Q.png" /></figure><p>Inline Scripts and Styles Handling</p><ul><li>Avoid &#39;unsafe-inline&#39; as it weakens CSP.</li><li>Use <strong>nonces</strong>: Random tokens generated on each response, applied both in the CSP header and inline elements.</li><li>Use <strong>hashes</strong>: Cryptographic hashes of inline code to allow specific trusted blocks.</li><li>Use &#39;unsafe-hashes&#39; with script-src to allow hashes for inline event handlers.</li></ul><h4>Recommended CSP / Best Practice</h4><p>A secure starting point balancing functionality and security might look like:</p><pre>Content-Security-Policy: <br>  default-src &#39;self&#39;; <br>  script-src &#39;self&#39; &#39;nonce-&lt;random&gt;&#39;; <br>  style-src &#39;self&#39; &#39;nonce-&lt;random&gt;&#39;; <br>  img-src &#39;self&#39; data:;<br>  object-src &#39;none&#39;;<br>  frame-ancestors &#39;none&#39;;</pre><ul><li>Deny plugins with object-src &#39;none&#39;.</li><li>Deny framing with frame-ancestors &#39;none&#39; (configurable).</li><li>Allow images from your origin and inline base64 images.</li><li>Use nonces to allow inline scripts and styles securely.</li></ul><h3>Summary Table of Recommended Headers for Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*Foz4kmVjkeT79J_pZ5bhcw.png" /></figure><h3>Here is my nginx config</h3><pre>upstream madhav_app_servers {<br>    server 192.168.1.29:5000;<br>}<br>server {<br>    listen 80;<br>    listen [::]:80;<br><br>    server_name madhav.com;<br>    return 301 https://$host$request_uri;<br>}<br>server {<br>    listen 443 ssl;<br>    listen [::]:443 ssl;<br>    server_name madhav.com;<br><br>    ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;<br>    ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;<br><br>    location / {<br>      # Pass requests to the upstream group defined above<br>        proxy_pass http://madhav_app_servers;<br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>        proxy_read_timeout 90;<br>        autoindex off;<br><br># add_header Content-Security-Policy &quot;default-src &#39;self&#39;; script-src &#39;self&#39;;&quot; always;<br> add_header Content-Security-Policy &quot;default-src &#39;self&#39;; script-src &#39;self&#39; &#39;unsafe-eval&#39; https://unpkg.com/babel-standalone@; style-src &#39;self&#39; &#39;unsafe-inline&#39;; img-src &#39;self&#39; data:; font-src &#39;self&#39;; connect-src &#39;self&#39;; frame-ancestors https://onecompiler.com;&quot; always;<br><br> add_header Strict-Transport-Security &quot;max-age=31536000; includeSubDomains; preload;&quot; always;<br> add_header X-Frame-Options &quot;ALLOW-FROM https://onecompiler.com&quot;;<br> add_header Access-Control-Allow-Origin &quot;https://www.w3schools.com&quot; always;<br><br>    }<br>    location ~ \.tsx$ {<br>        add_header Content-Type application/javascript;<br>    }<br>}</pre><p>Absolutely, let’s break down your given NGINX configuration and explain each section step-by-step in simple terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WGX4Jy09LTGtG-yLXtJi6w.png" /></figure><h3>1. Upstream Block</h3><pre>upstream madhav_app_servers {<br>    server 192.168.1.29:5000;<br>}</pre><ul><li><strong>Purpose</strong>: This creates a group called madhav_app_servers pointing to a backend server at IP 192.168.1.29, port 5000.</li><li><strong>Usage</strong>: The group makes it easier to scale (add more backend servers here for load balancing). In your current setup, there’s only one server.<a href="https://www.hostmycode.in/tutorials/configure-nginx-for-multiple-backend-on-ubuntu">hostmycode+2</a></li></ul><h3>2. HTTP-to-HTTPS Redirect Block</h3><pre>server {<br>    listen 80;<br>    listen [::]:80;<br>    server_name madhav.com;<br>        return 301 https://$host$request_uri;<br>    }</pre><ul><li><strong>Purpose</strong>: Listens on HTTP (port 80).</li><li><strong>Behavior</strong>: All incoming HTTP requests to madhav.com are redirected to the secure HTTPS equivalent (301 = permanent redirect).</li><li><strong>Benefit</strong>: Makes sure users always use secure SSL connections to your website.</li></ul><h3>3. HTTPS Serving Block</h3><pre>server {<br>    listen 443 ssl;<br>    listen [::]:443 ssl;<br>    server_name madhav.com;<br>    ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;<br>    ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;</pre><ul><li><strong>Purpose</strong>: Listens on HTTPS (443) for the domain madhav.com.</li><li><strong>SSL</strong>: Uses self-signed SSL certificate for encrypting data.</li></ul><h3>Proxy and Headers</h3><pre>location / {<br>        proxy_pass http://madhav_app_servers;<br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>        proxy_read_timeout 90;<br>        autoindex off;</pre><ul><li><strong>proxy_pass</strong>: Forwards all client requests to the backend group defined above (madhav_app_servers).</li><li><strong>proxy_set_header</strong>: These lines forward client details (host, real IP, protocol) to your backend app. They help your backend know the original requestor.</li><li><strong>proxy_read_timeout</strong>: Limits waiting time on response from backend to 90 seconds.</li><li><strong>autoindex off</strong>: Directory indices (listing directory contents) are disabled for security.</li></ul><h3>Security Headers</h3><pre>add_header Content-Security-Policy &quot;...&quot;;<br>add_header Strict-Transport-Security &quot;max-age=31536000; includeSubDomains; preload;&quot; always;<br>add_header X-Frame-Options &quot;ALLOW-FROM https://onecompiler.com&quot;;<br>add_header Access-Control-Allow-Origin &quot;https://www.w3schools.com&quot; always;</pre><ul><li><strong>Content-Security-Policy</strong>: Blocks scripts/styles/images from unwanted sources. The given policy allows scripts from self, some unsafe-eval, and certain external domains for development.</li><li><strong>Strict-Transport-Security</strong>: Tells browsers to always use HTTPS for 1 year across all subdomains.</li><li><strong>X-Frame-Options</strong>: Controls which sites can embed your site in an iframe (ALLOW-FROM is only honored by certain browsers).</li><li><strong>Access-Control-Allow-Origin</strong>: Allows cross-origin requests from the URL specified (w3schools.com in this case).</li></ul><h3>Special File Type Handling</h3><pre>location ~ \.tsx$ {<br>    add_header Content-Type application/javascript;<br>}</pre><ul><li><strong>Purpose</strong>: Forces .tsx files to be served with a Content-Type of application/javascript, which may help in certain development workflows with React/TypeScript.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54c795915abd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to set up Node-exporter, cadvisor, and kube-state-metrics?]]></title>
            <link>https://medium.com/@MadhavPrajapati/how-to-set-up-node-exporter-cadvisor-and-kube-state-metrics-1dcca593c581?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dcca593c581</guid>
            <category><![CDATA[kube-state-metrics]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[cadvisor]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[node-exporter]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Tue, 27 May 2025 04:07:24 GMT</pubDate>
            <atom:updated>2025-05-27T04:07:24.130Z</atom:updated>
            <content:encoded><![CDATA[<p>Monitoring is critical to maintaining the health, performance, and stability of your Kubernetes clusters. Tools like <strong>Node Exporter</strong>, <strong>cAdvisor</strong>, and <strong>kube-state metrics</strong> play essential roles in collecting different types of metrics that together give a full picture of your system.</p><p>In this article, we’ll explore what these tools do, why they’re important, and what kind of data they provide.</p><h3>Node-Exporter</h3><h4>What is Node Exporter?</h4><p><strong>Node Exporter</strong> is a Prometheus exporter for hardware and operating system metrics. It runs on each Kubernetes node (virtual machine or physical server) and collects system-level metrics.</p><h4><strong>How it collects data:</strong></h4><ul><li><strong>Runs as an Agent:</strong> Node Exporter is a lightweight program that runs directly on each server/node you want to monitor.</li><li><strong>Accesses System Files:</strong> It doesn’t look at Kubernetes-specific information. Instead, it interacts directly with the operating system’s kernel and various system files (like /proc and /sys on Linux) to gather data about CPU, memory, disk, network, filesystem, and more. Think of it as reading the health vitals of the machine itself.</li><li><strong>“Collectors”:</strong> It has various “collectors” (plugins) for different types of metrics (e.g., cpu collector for CPU stats, diskstats for disk I/O, filesystem for disk usage). You can enable or disable these collectors to get specific data.</li></ul><h4><strong>How it exposes data:</strong></h4><ul><li><strong>HTTP Endpoint:</strong> Node Exporter opens an HTTP endpoint (typically on port 9100) at the /metrics path.</li><li><strong>Prometheus Format:</strong> It exposes all the collected metrics in a plain-text format that Prometheus understands. Each metric has a name, value, and optional labels.</li></ul><h4>Why Use Node Exporter?</h4><p>Node Exporter helps you monitor the <strong>health of the underlying infrastructure</strong> that runs your Kubernetes workloads. Without it, you won’t know if your nodes are low on memory, running out of disk space, or experiencing CPU bottlenecks.</p><h4>What Metrics Does It Provide?</h4><ul><li>CPU usage</li><li>Memory usage</li><li>Disk I/O</li><li>Filesystem usage</li><li>Network I/O</li><li>System uptime</li></ul><p>These metrics are essential for <strong>infrastructure monitoring</strong> and alerting on node-level issues.</p><h3>Install the Node-exporter</h3><h4>Create Namespace:</h4><p>It’s recommended to create a dedicated namespace for monitoring components.</p><pre>kubeclt create namespace metrics</pre><h4>Download Helm chart:</h4><p>Download the Node Exporter Helm chart and extract its contents.</p><pre>helm pull prometheus-community/prometheus-node-exporter --untar</pre><h4>Custom-values.yaml:</h4><p>Create or modify a custom-values.yaml file to define resource requests and limits for the Node Exporter pods. This helps in managing resource consumption within your cluster.</p><pre>resources:<br>  limits:<br>    cpu: 100m<br>    memory: 200Mi<br>  requests:<br>    cpu: 500m<br>    memory: 100Mi</pre><h4>Install Chart:</h4><p>Deploy Node Exporter using Helm, specifying your custom values and the target namespace.</p><pre>helm install node-exporter prometheus-node-exporter/ --values prometheus-node-exporter/custom-values.yaml --namespace metrics</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1_PIwVVx-19jsFHozPhACQ.png" /></figure><h3>Cadvisor</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CY3NAXdtmRXt2eTF.png" /></figure><p>You don’t need to install cAdvisor separately because <strong>it’s already built into every Kubelet.</strong> Kubelet is the agent that runs on each node in your Kubernetes cluster, and it automatically collects container resource metrics using its embedded cAdvisor. So, the metrics you need are already there without any extra installation.</p><p>But if you want, then you ca,n or when you are using<strong> Very Old Kubernetes Versions</strong></p><h4>What is cAdvisor?</h4><p><strong>cAdvisor</strong> is a tool developed by Google that provides <strong>container-level resource usage</strong> and performance metrics. It is often built into the <strong>Kubelet</strong> and runs on every node.</p><h4>Why Use cAdvisor?</h4><p>Kubernetes runs workloads in containers, so it’s important to see what’s happening <strong>inside those containers</strong>. cAdvisor provides visibility into how much CPU, memory, and disk I/O each container uses.</p><h4>What Metrics Does It Provide?</h4><ul><li>Container CPU and memory usage</li><li>Network and disk I/O stats</li><li>Container start and stop times</li><li>Resource limits and usage over time</li></ul><p>These are crucial for <strong>performance monitoring</strong>, capacity planning, and troubleshooting pod-level issues.</p><h3>Install the cadvisor</h3><h4>Create Namespace:</h4><p>Ensure the metrics namespace exists.</p><pre>kubeclt create namespace metrics</pre><h4>Download Helm chart:</h4><p>Download the Bitnami cAdvisor Helm chart.</p><pre> helm pull bitnami/cadvisor  --untar  </pre><h4>Custom-values.yaml:</h4><p>Define resource requests and limits for the cAdvisor pods.</p><pre>resources:<br>  requests:<br>    cpu: 60m<br>    memory: 200Mi<br>  limits:<br>    cpu: 100m<br>    memory: 300Mi</pre><h4>Install Chart:</h4><p>Deploy cAdvisor using Helm. helm upgrade --install ensures that if you run the command again, it will update the existing release instead of trying to create a new one.</p><pre>helm upgrade --install cadvisor cadvisor --values cadvisor/custom-values.yaml -n metrics</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qylRBylxXT5ucrOVWFITsw.png" /></figure><h3>kube-state-metrics</h3><h4>What is kube-state-metrics?</h4><p><strong>kube-state-metrics</strong> is a service that listens to the <strong>Kubernetes API</strong> and generates Prometheus metrics about the <strong>state of Kubernetes objects</strong>.</p><h4><strong>How it collects data:</strong></h4><ul><li><strong>Listens to Kubernetes API:</strong> kube-state-metrics runs as a separate service (typically a Deployment) within your cluster. It has read-only access to the Kubernetes API server.</li><li><strong>Translates API State to Metrics:</strong> It continuously watches for changes in the Kubernetes API. When it detects a change (e.g., a pod changes its status from Pending to Running, or a Deployment’s desired replicas change), it translates that internal Kubernetes state into a simple, flat metric format that Prometheus can understand. It acts as a “state-to-metric” translator.</li><li><strong>No Aggregation/Modification:</strong> It exposes the raw, unmodified state data. It doesn’t perform any calculations or aggregations; it just mirrors the Kubernetes API state into metrics.</li></ul><h4><strong>How it exposes data:</strong></h4><ul><li><strong>HTTP Endpoint:</strong> Like other exporters, it exposes its collected metrics on an HTTP endpoint (defaulting to port 8080 or 8081 on the /metrics path).</li><li><strong>Prometheus Format:</strong> All metrics are provided in the standard Prometheus exposition format.</li></ul><h4>Why Use kube-state-metrics?</h4><p>It tells you what’s happening in your <strong>Kubernetes control plane</strong>. Unlike Node Exporter and cAdvisor, it doesn’t provide CPU or memory data — it provides <strong>status data</strong> about pods, deployments, nodes, and more.</p><h4><strong>What Metrics Does It Provide?</strong></h4><ul><li><strong>Deployment Status:</strong> Number of desired vs. available replicas, updated replicas, etc. (kube_deployment_status_replicas_available)</li><li><strong>Pod Status:</strong> Lifecycle phases (Running, Pending, Failed, Succeeded), readiness, restarts (kube_pod_status_phase, kube_pod_container_status_ready, kube_pod_container_status_restarts_total)</li><li><strong>Node Readiness:</strong> Status of nodes (Ready, NotReady, etc.) (kube_node_status_condition)</li><li><strong>Resource Requests and Limits:</strong> For pods and containers (kube_pod_container_resource_requests_cpu_cores, kube_pod_container_resource_limits_memory_bytes)</li><li><strong>PersistentVolumeClaim (PVC) Status:</strong> Binding status and usage (kube_persistentvolumeclaim_status_phase)</li><li><strong>Job Status:</strong> Completion status of Kubernetes Jobs.</li><li>And many more, covering Services, Ingresses, ConfigMaps, Secrets, etc.</li></ul><p>This information is indispensable for comprehensive cluster health monitoring, ensuring service availability, and automating operational tasks based on cluster state.</p><h4>Install kube-state-metrics</h4><p><strong>1. Create Namespace:</strong> Ensure the metrics namespace exists.</p><pre>kubectl create namespace metrics</pre><p><strong>2. Download Helm Chart:</strong> Download the kube-state-metrics Helm chart.</p><pre>helm pull prometheus-community/kube-state-metrics --untar</pre><p><strong>3. Custom </strong><strong>custom-values.yaml:</strong> Define resource requests and limits for the kube-state-metrics pod.</p><pre>resources:<br>  limits:<br>    cpu: 50m    # Maximum CPU kube-state-metrics pod can consume<br>    memory: 64Mi  # Maximum memory kube-state-metrics pod can consume<br>  requests:<br>    cpu: 10m    # Minimum CPU requested for kube-state-metrics pod<br>    memory: 32Mi  # Minimum memory requested for kube-state-metrics pod</pre><p><strong>4. Install Chart:</strong> Deploy kube-state-metrics using Helm.</p><pre>helm upgrade --install kube-state-metrics prometheus-community/kube-state-metrics --values kube-state-metrics/custom-values.yaml --namespace metrics</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pr2bfTnKkusvbaHJ4Y6bHw.png" /></figure><h3>Verify Installation</h3><p>After installing all three components, you can verify that their pods are running in the metrics namespace:</p><pre>kubectl get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cUcmP_U1tosldmDgdOl3uQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dcca593c581" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Set Up Grafana Mimir in Kubernetese ?]]></title>
            <link>https://medium.com/@MadhavPrajapati/how-to-set-up-grafana-mimir-in-kubernetese-207b8693d1b5?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/207b8693d1b5</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[mimir]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Tue, 27 May 2025 03:40:15 GMT</pubDate>
            <atom:updated>2025-05-27T03:40:15.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Set Up Grafana Mimir in Kubernetese ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A7CaWTxEVxzeisSX.png" /></figure><h4>What is Grafana Mimir?</h4><p>Mimir is an open-source, horizontally scalable, highly available, multi-tenant TSDB for long-term storage for Prometheus.</p><h4>Deployment Modes:</h4><p>There are 3 Mimir deployment modes as of now, and they are:</p><ol><li><a href="https://grafana.com/docs/mimir/latest/references/architecture/deployment-modes/#monolithic-mode">Monolithic Mode</a> (All Mimir components are deployed as a single instance/pod/container).</li><li><a href="https://grafana.com/docs/mimir/latest/references/architecture/deployment-modes/#microservices-mode">Microservices/Distributed Mode</a> (Each Mimir Component is deployed as its instance/pod/container).</li><li><a href="https://grafana.com/docs/mimir/latest/references/architecture/deployment-modes/#read-write-mode">Read-Write Mode</a> (Experimental Phase).</li></ol><h4>Architecture of Mimir</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mTmYSnQUvyZumnZZ.png" /></figure><p><strong>Reads Path:</strong> When you view metrics (Getting data out)<br><strong>Writes Path:</strong> When Prometheus sends metrics (Putting data in)</p><h3>Compactor:</h3><p>The compactor increases query performance and reduces long-term storage usage by combining blocks.</p><p>The compactor is the component responsible for:</p><ul><li>It keeps the storage clean and organized.</li><li>It merges small pieces of data into bigger ones to make queries faster and reduce storage space.</li><li>Deleting blocks that are no longer within a configurable retention period.</li></ul><h3>Distributor:</h3><p>The distributor is a stateless component that receives time-series data from Prometheus or Grafana Alloy. The distributor validates the data for correctness and ensures that it is within the configured limits for a given tenant.</p><ul><li>This is the entry point for all new data.</li><li>It receives incoming data (logs, metrics, etc.) and forwards it to the right <strong>ingester</strong>.</li></ul><h3>Ingester:</h3><p>The ingester is a stateful component that writes incoming series to <a href="https://grafana.com/docs/mimir/latest/get-started/about-grafana-mimir-architecture/#long-term-storage">long-term storage</a> on the write path and returns series samples for queries on the read path.</p><p>all series are written to disk and periodically uploaded (by default every two hours) to the long-term storage.</p><h4>Ingesters&#39; failure and data loss</h4><p>If an ingester process crashes or exits abruptly, any in-memory time series data that have not yet been uploaded to long-term storage might be lost. There are the following ways to mitigate this failure mode:</p><ul><li>Replication</li><li>Write-ahead log (WAL)</li><li>Write-behind log (WBL), only used if out-of-order ingestion is enabled.</li></ul><h4>Replication:</h4><p>Writes in Mimir succeed if a <strong>majority of ingesters</strong> receive the data. With the default replication factor of 3, <strong>2 out of 3 ingesters</strong> must store the data. So, if one ingester fails, <strong>data isn’t lost</strong>, as it’s still held in the others.</p><h4>Write-ahead log (WAL):</h4><p>The WAL stores incoming data on disk until it’s saved to long-term storage. If an ingester fails, it can <strong>replay the WAL</strong> on restart to <strong>recover lost data</strong>.</p><h3>Querier:</h3><p>The querier is a stateless component that evaluates <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/">PromQL</a> expressions by fetching time series and labels on the read path.</p><p>The querier uses the <a href="https://grafana.com/docs/mimir/latest/references/architecture/components/store-gateway/">store-gateway</a> component to query the <a href="https://grafana.com/docs/mimir/latest/get-started/about-grafana-mimir-architecture/#long-term-storage">long-term storage</a> and the <a href="https://grafana.com/docs/mimir/latest/references/architecture/components/ingester/">ingester</a> component to query recently written data.</p><p>The <strong>main worker</strong> who processes the query. It gathers data from two places</p><ul><li><strong>Ingesters</strong> for recent metrics</li><li><strong>Store-gateway</strong> for older metrics in storage</li></ul><h3>Query-Frontend:</h3><p>The query-frontend is a stateless component that provides the same API as the <a href="https://grafana.com/docs/mimir/latest/references/architecture/components/querier/">querier</a> and can be used to accelerate the read path. Although the query-frontend is not required, we recommend that you deploy it. When you deploy the query-frontend, you should make query requests to the query-frontend instead of the queriers. The queriers are required within the cluster to execute the queries.</p><h3><strong>Store-Gateway:</strong></h3><p>The store-gateway component, which is stateful, queries blocks from <a href="https://grafana.com/docs/mimir/latest/get-started/about-grafana-mimir-architecture/#long-term-storage">long-term storage</a>. On the read path, the <a href="https://grafana.com/docs/mimir/latest/references/architecture/components/querier/">querier</a> and the <a href="https://grafana.com/docs/mimir/latest/references/architecture/components/ruler/">ruler</a> use the store-gateway when handling the query, whether the query comes from a user or from when a rule is being evaluated.</p><p>Like a <strong>librarian</strong> that reads old metric blocks from object storage (like S3). It helps querier Get historical data (older than what&#39;s in memory).</p><h3>Deploying Grafana Mimir on Kubernetes</h3><h3>1. Create a Kubernetes Namespace</h3><p>First, create a dedicated namespace for Mimir to ensure resource isolation and organization within your Kubernetes cluster.</p><pre>kubectl create namespace monitoring-mimir</pre><h3>2. Create Kubernetes Secret for S3 Credentials</h3><p>Mimir utilizes S3-compatible storage for its blocks. This step involves creating a Kubernetes Secret to securely store your S3 access key ID and secret access key. <strong>Replace </strong><strong>&lt;access_key&gt; and </strong><strong>&lt;secrete_aky&gt; with your actual S3 credentials.</strong></p><pre>kubectl create secret generic mimir-s3-credentials \<br>  --from-literal=access_key_id=&lt;access_key&gt; \<br>  --from-literal=secret_access_key=&lt;secrete_aky&gt; \<br>  -n monitoring-mimir</pre><h3>3. Download Helm Chart</h3><p>Download the official Grafana Mimir Helm chart and extract its contents. This will provide you with the necessary files to configure and deploy Mimir.</p><pre>helm pull grafana/mimir-distributed --untar</pre><h3>4. Customize custom-values.yaml</h3><p>A custom-values.yaml file is used to override default Helm chart values and configure Mimir according to your specific requirements.</p><p>The provided custom-values.yaml includes the following key configurations:</p><pre># Place extraEnvFrom under global<br>global:<br>  extraEnvFrom:<br>    - secretRef:<br>        name: mimir-s3-credentials<br>mimir:<br>  structuredConfig:<br>    memberlist:<br>      cluster_label: &quot;mimir&quot;<br>      cluster_label_verification_disabled: false<br>    server:<br>      log_level: info<br>      grpc_server_max_recv_msg_size: 104857600<br>      grpc_server_max_send_msg_size: 104857600<br>      grpc_server_max_concurrent_streams: 1000<br>    limits:<br>      max_label_names_per_series: 120<br>      max_global_series_per_user: 12000000<br>      ingestion_burst_size: 200000<br>      ingestion_rate: 150000<br>      max_global_series_per_metric: 200000000<br>      compactor_blocks_retention_period: 24h<br>    blocks_storage:<br>      backend: s3<br>      s3:<br>        endpoint: &quot;s3.ap-southeast-1.amazonaws.com&quot;<br>        region: &quot;ap-southeast-1&quot;<br>        access_key_id: &quot;${access_key_id}&quot;<br>        secret_access_key: &quot;${secret_access_key}&quot;<br>        bucket_name: &quot;solvei8-stateful-backups&quot;<br>        insecure: false<br>    compactor:<br>      compaction_interval: 24h<br># Writes incoming series to long-term storage on the write path and returns series samples for queries on the read path.<br>ingester:<br>  zoneAwareReplication:<br>    enabled: false<br>  persistentVolume:<br>    enabled: false<br>  replicas: 2<br># Receives data from alloy, corrects, devides data to batches and send to ingesters<br>distributor:<br>  extraArgs:<br>    auth.multitenancy-enabled: false<br>    log.level: info<br>  replicas: 1<br># Increases query performance and reduces long-term storage usage by combining blocks.<br>compactor:<br>  replicas: 1<br># Evaluates PromQL expressions by fetching time series and labels on the read path. Uses store-gateway fron minio and ingester for latest data<br>querier:<br>  replicas: 2<br># &quot;frontend&quot; for querier<br>query_frontend:<br>  replicas: 1<br># comuniacation with minio<br>store_gateway:<br>  zoneAwareReplication:<br>    enabled: false<br>  replicas: 1<br>## Disabled services:<br>rollout_operator:<br>  enabled: false<br>gateway:<br>  enabledNonEnterprise: false<br>ruler:<br>  enabled: false<br>query_scheduler:<br>  enabled: false<br>overrides_exporter:<br>  enabled: false<br>alertmanager:<br>  enabled: false<br>minio:<br>  enabled: false</pre><p><strong>Key configurations explained:</strong></p><ul><li><strong>global.extraEnvFrom</strong>: This section ensures that the S3 credentials from the mimir-s3-credentials secret are injected as environment variables into Mimir pods.</li><li><strong>mimir.structuredConfig.memberlist</strong>: Configures the Mimir memberlist for inter-component communication.</li><li><strong>mimir.structuredConfig.server</strong>: Sets general server-side configurations like log level and gRPC message sizes.</li><li><strong>mimir.structuredConfig.limits</strong>: Defines various limits for Mimir&#39;s operation, such as maximum label names per series, global series per user, and ingestion rates. These are crucial for performance and resource management.</li><li><strong>mimir.structuredConfig.blocks_storage</strong>: Configures Mimir to use S3 as its backend storage for blocks.</li><li>endpoint: Specifies the S3 service endpoint (e.g., s3.ap-southeast-1.amazonaws.com).</li><li>region: The AWS region for your S3 bucket.</li></ul><p>access_key_id and secret_access_key: These will be populated from the mimir-s3-credentials secret via environment variables.</p><ul><li>bucket_name: The name of your S3 bucket for Mimir data.</li><li>insecure: Set to false for secure communication.</li><li><strong>mimir.structuredConfig.compactor.compaction_interval</strong>: Sets how frequently the compactor service runs to optimize storage.</li></ul><p><strong>Ingester (</strong><strong>ingester)</strong>:</p><ul><li>zoneAwareReplication: enabled: false: Zone-aware replication is disabled.</li><li><strong>persistentVolume: enabled: true</strong>: The ingester will now use <strong>persistent storage</strong>. This is a significant improvement as it allows the ingester to store its write-ahead log (WAL) and in-memory series on disk, improving data durability and recovery across pod restarts.</li><li>size: 2Gi: Allocates a 2 GiB persistent volume.</li><li>storageClass: &quot;ebs-sc-delete&quot;: Specifies the storage class to use, likely an AWS EBS volume that will be deleted when the PVC is deleted.</li><li>replicas: 2: Two replicas for high availability and load distribution.</li></ul><p><strong>Distributor (</strong><strong>distributor)</strong>:</p><ul><li>extraArgs:</li><li><strong>auth.multitenancy-enabled: true</strong>: <strong>(Updated)</strong> Multi-tenancy is now <strong>enabled</strong> for the distributor. This allows Mimir to handle data from multiple tenants or users.</li><li>log.level: info: Informational logging.</li><li>replicas: 1: One replica for the distributor.</li></ul><p><strong>Compactor (</strong><strong>compactor)</strong>:</p><ul><li>replicas: 1: One replica.</li><li><strong>persistentVolume: enabled: true</strong>: <strong>(New)</strong> The compactor will now also use <strong>persistent storage</strong>. This is beneficial for storing temporary data during compaction operations, improving efficiency and robustness.</li><li>size: 2Gi: Allocates a 2 GiB persistent volume.</li><li>storageClass: &quot;ebs-sc-delete&quot;: Specifies the storage class.</li></ul><p><strong>Querier (</strong><strong>querier)</strong>:</p><ul><li>replicas: 2: Two replicas for query processing.</li></ul><p><strong>Query Frontend (</strong><strong>query_frontend)</strong>:</p><ul><li>replicas: 1: One replica, acting as a frontend for queriers.</li></ul><p><strong>Store Gateway (</strong><strong>store_gateway)</strong>:</p><ul><li>zoneAwareReplication: enabled: false: Zone-aware replication is disabled.</li><li>replicas: 1: One replica.</li><li><strong>persistentVolume: enabled: true</strong>: The store gateway will now use <strong>persistent storage</strong>. This helps in caching blocks from S3 locally, which can significantly improve query performance by reducing the need to fetch data repeatedly from object storage.</li><li>size: 2Gi: Allocates a 2 GiB persistent volume.</li><li>storageClass: &quot;ebs-sc-delete&quot;: Specifies the storage class.</li></ul><p><strong>Disabled services</strong>: Several Mimir components are explicitly disabled in this configuration as they might not be needed for a basic setup. This helps in reducing resource consumption.</p><h3>5. Install Helm Chart</h3><p>Finally, install the Mimir Helm chart using the customized custom-values.yaml file in the monitoring-mimir namespace.</p><pre>helm install mimir mimir-distributed --values mimir-distributed/custom-values.yaml --namespace monitoring-mimir</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=207b8693d1b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch Snapshot Backup — With AirFlow DAG]]></title>
            <link>https://medium.com/@MadhavPrajapati/airflow-dag-elasticsearch-snapshot-backup-4294ebd65b71?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/4294ebd65b71</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[snapshot]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 04:20:52 GMT</pubDate>
            <atom:updated>2025-04-11T04:26:49.775Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*vnSuzSE-Q_F5Ss5cUmAn1w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*3xtV3okEI4b21bHMFfl0bg.png" /></figure><h4>Prerequesit:-</h4><blockquote><a href="https://medium.com/@MadhavPrajapati/elasticsearch-backup-and-restore-a-comprehensive-guide-8a4d79ed8a48">https://medium.com/@MadhavPrajapati/elasticsearch-backup-and-restore-a-comprehensive-guide-8a4d79ed8a48</a></blockquote><h3>📌 Purpose</h3><p>This Airflow DAG is designed to <strong>automatically create and manage Elasticsearch snapshots</strong> using the built-in HTTP API. It performs:</p><ol><li><strong>Cluster health check</strong></li><li><strong>Snapshot creation</strong></li><li><strong>Old snapshot cleanup (retention policy)</strong></li></ol><p>Snapshots are stored in an <strong>S3-compatible repository</strong> (configured in Elasticsearch).</p><h3>📋 Task Breakdown</h3><h3>1. check_es_health – Check Cluster Health</h3><ul><li><strong>Function:</strong> check_es_cluster_health()</li><li><strong>What it does:</strong><br> Queries the cluster’s _cluster/health endpoint and ensures the status is <strong>green</strong> or <strong>yellow</strong>.<br> If the cluster is <strong>red</strong>, it aborts the DAG run.</li></ul><pre>response = requests.get(f&quot;{ELASTICSEARCH_HOST}/_cluster/health&quot;)<br>if response.json().get(&quot;status&quot;) not in [&quot;green&quot;, &quot;yellow&quot;]:<br>    raise Exception(&quot;Cluster not healthy&quot;)</pre><h3>2. create_es_snapshot – Create Snapshot</h3><ul><li><strong>Function:</strong> create_snapshot()</li><li><strong>What it does:</strong><br> Uses the _snapshot API to create a full snapshot of all indices.<br> Snapshot is named using the current timestamp.</li></ul><pre>SNAPSHOT_NAME = f&quot;es-incr-snap-{datetime.now().strftime(&#39;%Y-%m-%d-%H-%M-%S&#39;)}&quot;<br>requests.put(<br>    f&quot;{ELASTICSEARCH_HOST}/_snapshot/{S3_REPO}/{SNAPSHOT_NAME}?wait_for_completion=true&quot;,<br>    json={&quot;indices&quot;: &quot;*&quot;, &quot;ignore_unavailable&quot;: True, &quot;include_global_state&quot;: True}<br>)</pre><h3>3. cleanup_snapshots – Delete Old Snapshots</h3><ul><li><strong>Function:</strong> cleanup_old_snapshots()</li><li><strong>What it does:</strong><br> Retrieves all existing snapshots, checks their end time, and deletes any snapshot <strong>older than 7 days</strong>.</li></ul><pre>cutoff = (datetime.utcnow() - timedelta(days=7)).timestamp()<br>for snapshot in snapshots:<br>    if snapshot[&quot;end_time_in_millis&quot;] / 1000 &lt; cutoff:<br>        requests.delete(f&quot;{ELASTICSEARCH_HOST}/_snapshot/{S3_REPO}/{snapshot[&#39;snapshot&#39;]}&quot;)</pre><h3>📈 DAG Dependency Flow</h3><pre>┌────────────────────────────┐<br>│   check_es_health          │<br>└────────────┬───────────────┘<br>             ▼<br>┌────────────────────────────┐<br>│   create_es_snapshot       │<br>└────────────┬───────────────┘<br>             ▼<br>┌────────────────────────────┐<br>│   cleanup_snapshots        │<br>└────────────────────────────┘</pre><p>Full Code:-</p><pre>from airflow import DAG<br>from airflow.providers.http.sensors.http import HttpSensor<br>from airflow.providers.http.operators.http import SimpleHttpOperator<br>from airflow.operators.python import PythonOperator<br>from datetime import datetime, timedelta<br>import requests<br>import json<br><br># DAG Configurations<br>ELASTICSEARCH_CONN_ID = &quot;elasticsearch_http&quot;  # Set up in Airflow Connections<br>ELASTICSEARCH_HOST = &quot;http://ip address:9200&quot;<br>S3_REPO = &quot;test-s3&quot;<br>SNAPSHOT_PREFIX = &quot;es-incr-snap&quot;<br>RETENTION_DAYS = 7<br><br># Generate snapshot name<br>SNAPSHOT_NAME = f&quot;{SNAPSHOT_PREFIX}-{datetime.now().strftime(&#39;%Y-%m-%d-%H-%M-%S&#39;)}&quot;<br><br>def check_es_cluster_health():<br>    &quot;&quot;&quot;Checks Elasticsearch cluster health and ensures it is &#39;green&#39; or &#39;yellow&#39;.&quot;&quot;&quot;<br>    url = f&quot;{ELASTICSEARCH_HOST}/_cluster/health&quot;<br>    response = requests.get(url)<br>    data = response.json()<br><br>    cluster_status = data.get(&quot;status&quot;)<br>    if cluster_status not in [&quot;green&quot;, &quot;yellow&quot;]:<br>        print(f&quot;Cluster health is {cluster_status}. Exiting.&quot;)<br>        raise Exception(f&quot;Cluster health is {cluster_status}, not safe for snapshot.&quot;)<br><br>    print(f&quot;Cluster health is {cluster_status}. Proceeding with snapshot.&quot;)<br><br>def create_snapshot():<br>    &quot;&quot;&quot;Creates an snapshot and verifies its success.&quot;&quot;&quot;<br>    url = f&quot;{ELASTICSEARCH_HOST}/_snapshot/{S3_REPO}/{SNAPSHOT_NAME}?wait_for_completion=true&quot;<br>    payload = {<br>        &quot;indices&quot;: &quot;*&quot;,<br>        &quot;ignore_unavailable&quot;: True,<br>        &quot;include_global_state&quot;: True<br>    }<br><br>    response = requests.put(url, headers={&quot;Content-Type&quot;: &quot;application/json&quot;}, json=payload)<br>    data = response.json()<br>    <br>    snapshot_status = data.get(&quot;snapshot&quot;, {}).get(&quot;state&quot;, &quot;FAILED&quot;)<br>    if snapshot_status != &quot;SUCCESS&quot;:<br>        print(f&quot;Snapshot {SNAPSHOT_NAME} failed: {json.dumps(data, indent=2)}&quot;)<br>        raise Exception(f&quot;Snapshot {SNAPSHOT_NAME} failed. Check logs.&quot;)<br><br>    print(f&quot;Snapshot {SNAPSHOT_NAME} completed successfully.&quot;)<br><br>def cleanup_old_snapshots():<br>    &quot;&quot;&quot;Deletes snapshots older than the retention period.&quot;&quot;&quot;<br>    url = f&quot;{ELASTICSEARCH_HOST}/_snapshot/{S3_REPO}/_all&quot;<br>    response = requests.get(url)<br>    snapshots = response.json().get(&quot;snapshots&quot;, [])<br><br>    retention_timestamp = (datetime.utcnow() - timedelta(days=RETENTION_DAYS)).timestamp()<br><br>    for snapshot in snapshots:<br>        if snapshot.get(&quot;end_time_in_millis&quot;, 0) / 1000 &lt; retention_timestamp:<br>            snapshot_name = snapshot[&quot;snapshot&quot;]<br>            delete_url = f&quot;{ELASTICSEARCH_HOST}/_snapshot/{S3_REPO}/{snapshot_name}&quot;<br>            del_response = requests.delete(delete_url)<br>            <br>            if del_response.status_code == 200:<br>                print(f&quot;Deleted old snapshot: {snapshot_name}&quot;)<br>            else:<br>                print(f&quot;Failed to delete snapshot {snapshot_name}: {del_response.text}&quot;)<br><br># DAG Default Arguments<br>default_args = {<br>    &quot;owner&quot;: &quot;airflow&quot;,<br>    &quot;depends_on_past&quot;: False,<br>    &quot;start_date&quot;: datetime(2024, 4, 2),<br>    &quot;retries&quot;: 1,<br>    &quot;retry_delay&quot;: timedelta(minutes=5),<br>}<br><br># Define DAG<br>with DAG(<br>    &quot;elasticsearch_snapshot_dag&quot;,<br>    default_args=default_args,<br>    schedule_interval=&quot;@daily&quot;,<br>    catchup=False<br>) as dag:<br><br>    check_es_health = PythonOperator(<br>        task_id=&quot;check_es_health&quot;,<br>        python_callable=check_es_cluster_health<br>    )<br><br>    create_es_snapshot = PythonOperator(<br>        task_id=&quot;create_es_snapshot&quot;,<br>        python_callable=create_snapshot<br>    )<br><br>    cleanup_snapshots = PythonOperator(<br>        task_id=&quot;cleanup_snapshots&quot;,<br>        python_callable=cleanup_old_snapshots<br>    )<br><br>    # DAG dependencies<br>    check_es_health &gt;&gt; create_es_snapshot &gt;&gt; cleanup_snapshots</pre><h3>📝 Summary</h3><p>Task Description <br>check_es_health Ensure cluster is ready for backup <br>create_es_snapshot Create full snapshot to S3 repository cleanup_snapshots Enforce 7-day retention by deleting old snapshots</p><p>I hope this article will be useful for you and that you learned something new.</p><p>Thank you : )</p><p>#keeplearning &amp; #keepsharing <a href="https://medium.com/u/bd161dccc754">Madhav Prajapati</a></p><h3>keep learning, keep sharing (❁´◡`❁)</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4294ebd65b71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB Incremental Backups to S3 — With AirFlow DAG]]></title>
            <link>https://medium.com/@MadhavPrajapati/automating-mongodb-incremental-backups-to-s3-using-airflow-and-kubernetes-10b85ea5049f?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/10b85ea5049f</guid>
            <category><![CDATA[mongodb-backup]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 03:51:57 GMT</pubDate>
            <atom:updated>2025-04-11T04:26:02.651Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0eIRDIOtJTo2jDNCto96BA.png" /></figure><h3>🎯 Purpose</h3><p>This workflow (DAG) helps you <strong>automatically backup only the changes</strong> in your MongoDB database every 6 hours and stores them in <strong>Amazon S3</strong>. It’s more efficient than full backups, as it saves time and space.</p><h4>📌 Why This is Important</h4><ul><li><strong>MongoDB Oplog (oplog.rs)</strong> contains changes in your DB.</li><li>Incremental backups are lighter and faster than full dumps.</li><li>S3 provides a reliable and low-cost storage location.</li><li>Tracks the last backup timestamp using an Airflow Variable.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mtw7wWXAvqNsiqdv" /></figure><p>Here’s</p><h3>🧠 Step-by-step Explanation (Simplified)</h3><h4><strong>Create PVC (Persistent Volume Claim)</strong></h4><ul><li>A temporary volume is created using Kubernetes to store backup data.</li></ul><pre>    # Create PVC task<br>    create_pvc = KubernetesPodOperator(<br>        task_id=&quot;create_temp_pvc&quot;,<br>        name=&quot;create-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>        arguments=[&quot;&quot;&quot;<br>            cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: mongo-backup-pvc<br>  namespace: airflow<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 2Gi<br>  storageClassName: ebs-sc-delete<br>EOF<br>        &quot;&quot;&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>    )</pre><h4><strong>Run Incremental Backup</strong></h4><ul><li>A MongoDB container runs mongodump using the last stored oplog timestamp.</li><li>The timestamp helps capture only the new changes.</li><li>The new timestamp is saved for next backup.</li><li>Use password, username and host as secret</li></ul><pre>kubectl create secret generic airflow-mongodb-secret \<br>  --namespace airflow \<br>  --from-literal=MONGO_HOST=your-mongo-host \<br>  --from-literal=MONGO_PORT=27017 \<br>  --from-literal=MONGO_USER=your-username \<br>  --from-literal=MONGO_PASS=your-password</pre><p>Here is the task for taking incremental backup</p><pre> # Incremental backup task<br>    incremental_backup = KubernetesPodOperator(<br>        task_id=&quot;run_incremental_backup&quot;,<br>        name=&quot;mongo-incremental-backup&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;mongo:4.4&quot;,<br>        cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>        env_vars=[<br>            {<br>                &quot;name&quot;: &quot;MONGO_HOST&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_HOST&quot;<br>                    }<br>                }<br>            },<br>            {<br>                &quot;name&quot;: &quot;MONGO_PORT&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_PORT&quot;<br>                    }<br>                }<br>            },<br>            {<br>                &quot;name&quot;: &quot;MONGO_USER&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_USER&quot;<br>                    }<br>                }<br>            },<br>            {<br>                &quot;name&quot;: &quot;MONGO_PASS&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_PASS&quot;<br>                    }<br>                }<br>            }<br>        ],<br>        arguments=[f&quot;&quot;&quot;<br>            set -ex<br>            echo &quot;Starting backup process...&quot;<br>            <br>            BACKUP_DIR=&quot;{BACKUP_BASE_DIR}&quot;<br>            <br>            echo &quot;Creating backup directory...&quot;<br>            mkdir -p &quot;$BACKUP_DIR&quot;<br>            <br>            if [ -z &quot;$LAST_TIMESTAMP&quot; ]; then<br>                echo &quot;Getting initial timestamp...&quot;<br>                LAST_TS=$(mongo --host $MONGO_HOST --port $MONGO_PORT \<br>                    --username $MONGO_USER --password $MONGO_PASS \<br>                    --authenticationDatabase {MONGO_AUTH_DB} \<br>                    --quiet \<br>                    --eval &#39;db.getSiblingDB(&quot;local&quot;).oplog.rs.find().sort({{ $natural:1 }}).limit(1).toArray()[0].ts&#39;)<br>            else<br>                LAST_TS=&quot;$LAST_TIMESTAMP&quot;<br>            fi<br>            <br>            echo &quot;Running backup with timestamp: Timestamp({t_value}, {i_value})&quot;<br>            BACKUP_PATH=&quot;$BACKUP_DIR/incremental_{TIMESTAMP}&quot;<br>            <br>            mongodump --host $MONGO_HOST --port $MONGO_PORT \<br>                --username $MONGO_USER --password $MONGO_PASS \<br>                --authenticationDatabase {MONGO_AUTH_DB}  \<br>                --db {MONGO_DB} --collection {MONGO_COLLECTION} \<br>                --query &#39;{{&quot;ts&quot;: {{&quot;$gt&quot;: {{&quot;$timestamp&quot;: {{&quot;t&quot;: {t_value}, &quot;i&quot;: {i_value}}}}}}}}}&#39; \<br>                --out &quot;$BACKUP_PATH&quot;<br>            <br>            # Update timestamp after successful backup<br>            NEW_TS=$(mongo --host $MONGO_HOST --port $MONGO_PORT \<br>                --username $MONGO_USER --password $MONGO_PASS \<br>                --authenticationDatabase {MONGO_AUTH_DB} \<br>                --quiet \<br>                --eval &#39;db.getSiblingDB(&quot;local&quot;).oplog.rs.find().sort({{ $natural:-1 }}).limit(1).toArray()[0].ts&#39;)<br>            <br>            # Write the new timestamp to a file that will be processed by the next task<br>            echo &quot;$NEW_TS&quot; &gt; &quot;$BACKUP_PATH/last_timestamp.txt&quot;<br>            <br>            echo &quot;Backup completed to $BACKUP_PATH&quot;<br>            &quot;&quot;&quot;<br>        ],<br>        volumes=[volume],<br>        volume_mounts=[volume_mount],<br>        is_delete_operator_pod=True,<br>        startup_timeout_seconds=300,<br>        get_logs=True,<br>    )</pre><h4><strong>Update Timestamp</strong></h4><ul><li>The new oplog timestamp from the last backup is stored in an Airflow Variable.</li></ul><pre>def update_timestamp(**context):<br>        &quot;&quot;&quot;Update the Airflow variable with the new timestamp&quot;&quot;&quot;<br>        ts = context[&#39;ts_nodash&#39;]<br>        backup_path = f&quot;{BACKUP_BASE_DIR}/incremental_{ts}&quot;<br>        timestamp_file = f&quot;{backup_path}/last_timestamp.txt&quot;<br>        <br>        with open(timestamp_file, &#39;r&#39;) as f:<br>            new_ts = f.read().strip()<br>        <br>        Variable.set(LAST_TS_VAR_KEY, new_ts)<br>        print(f&quot;Updated timestamp to: {new_ts}&quot;)<br><br>    update_timestamp_task = PythonOperator(<br>        task_id=&quot;update_timestamp&quot;,<br>        python_callable=update_timestamp,<br>        executor_config={<br>            &quot;pod_override&quot;: k8s.V1Pod(<br>                spec=k8s.V1PodSpec(<br>                    containers=[<br>                        k8s.V1Container(name=&quot;base&quot;, volume_mounts=[volume_mount])<br>                    ],<br>                    volumes=[volume]<br>                )<br>            )<br>        },<br>    )</pre><h4><strong>Upload to S3</strong></h4><ul><li>The incremental backup files are uploaded to an S3 bucket using S3Hook.</li></ul><pre>def upload_incremental_to_s3(**context):<br>    &quot;&quot;&quot;Uploads incremental backup directory to S3.&quot;&quot;&quot;<br>    ts = context[&#39;ts_nodash&#39;]<br>    backup_path = f&quot;{BACKUP_BASE_DIR}/incremental_{ts}&quot;<br>    s3_hook = S3Hook(aws_conn_id=&#39;s3_logging&#39;)<br><br>    if not os.path.exists(backup_path):<br>        raise FileNotFoundError(f&quot;{backup_path} not found&quot;)<br><br>    for root, dirs, files in os.walk(backup_path):<br>        for file in files:<br>            local_path = os.path.join(root, file)<br>            relative_path = os.path.relpath(local_path, backup_path)<br>            s3_key = f&quot;{S3_PREFIX}/mongo_incremental_backup_{ts}/{relative_path}&quot;<br>            s3_hook.load_file(<br>                filename=local_path,<br>                key=s3_key,<br>                bucket_name=S3_BUCKET,<br>                replace=True,<br>            )<br>            print(f&quot;Uploaded: s3://{S3_BUCKET}/{s3_key}&quot;)  <br><br>  upload_to_s3_task = PythonOperator(<br>        task_id=&quot;upload_to_s3&quot;,<br>        python_callable=upload_incremental_to_s3,<br>        executor_config={<br>            &quot;pod_override&quot;: k8s.V1Pod(<br>                spec=k8s.V1PodSpec(<br>                    containers=[<br>                        k8s.V1Container(name=&quot;base&quot;, volume_mounts=[volume_mount])<br>                    ],<br>                    volumes=[volume]<br>                )<br>            )<br>        },<br>    )</pre><h4><strong>Delete PVC</strong></h4><ul><li>The temporary volume is deleted to save storage.</li></ul><pre>        # Delete PVC task<br>    delete_pvc = KubernetesPodOperator(<br>        task_id=&quot;delete_temp_pvc&quot;,<br>        name=&quot;delete-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;kubectl&quot;],<br>        arguments=[&quot;delete&quot;, &quot;pvc&quot;, pvc_name, &quot;-n&quot;, &quot;airflow&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>    )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMhk7cJ3EuLcTdcaQLKsEA.png" /></figure><h3>⚙️ Tech Stack Used</h3><p><strong>Airflow — — — — — — — — — — — →</strong>Task orchestration <br><strong>MongoDB</strong> <strong>— — — — — — — — — — →</strong>Source DB (oplog)<br><strong>S3</strong> <strong>— — — — — — — — — — — — — — →</strong>Backup destination <br><strong>PVC</strong> <strong>— — — — — — — — — — — — — -→</strong>Temporary file storage in cluster <br><strong>KubernetesPodOperator</strong> <strong>— — —→</strong>Run each job in isolated containers</p><h3>📂 Folder &amp; File Naming</h3><ul><li>Backup folder: /shared_data/backup/incremental/incremental_&lt;timestamp&gt;</li><li>Timestamp file inside backup: last_timestamp.txt</li><li>S3 Key: mongo-incremental-backup/mongo_incremental_backup_&lt;timestamp&gt;/</li></ul><h3>💡 Notes</h3><ul><li>Timestamp is stored and tracked using airflow.models.Variable (mongo_last_oplog_timestamp).</li><li>Uses Extended JSON query format for filtering oplog by timestamp:</li></ul><pre>{&quot;ts&quot;: {&quot;$gt&quot;: {&quot;$timestamp&quot;: {&quot;t&quot;: &lt;t&gt;, &quot;i&quot;: &lt;i&gt;}}}}</pre><ul><li>All tasks run in Kubernetes using Airflow’s KubernetesPodOperator.</li></ul><h4>Full Code:-</h4><pre>from datetime import datetime, timedelta<br>from airflow import DAG<br>from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator<br>from airflow.operators.python import PythonOperator<br>from airflow.operators.bash import BashOperator<br>from kubernetes.client import models as k8s<br>from airflow.hooks.S3_hook import S3Hook<br>from airflow.models import Variable<br>import os<br><br># Default DAG args<br>default_args = {<br>    &quot;owner&quot;: &quot;airflow&quot;,<br>    &quot;depends_on_past&quot;: False,<br>    &quot;start_date&quot;: datetime(2023, 1, 1),<br>    &quot;retries&quot;: 1,<br>    &quot;retry_delay&quot;: timedelta(minutes=5),<br>    &quot;email_on_failure&quot;: False,<br>}<br><br># Constants<br>SHARED_VOLUME_PATH = &quot;/shared_data&quot;<br>BACKUP_BASE_DIR = f&quot;{SHARED_VOLUME_PATH}/backup/incremental&quot;<br>LAST_TS_VAR_KEY = &quot;mongo_last_oplog_timestamp&quot;  # Airflow Variable key<br>S3_BUCKET = &quot;madhav-airflow-logging-bucket&quot;<br>S3_PREFIX = &quot;mongo-incremental-backup&quot;<br>MONGO_DB = &quot;local&quot;<br>MONGO_COLLECTION = &quot;oplog.rs&quot;<br>MONGO_AUTH_DB = &quot;admin&quot;<br>TIMESTAMP = &quot;{{ ts_nodash }}&quot;  <br>pvc_name = &quot;mongo-backup-pvc&quot;<br><br><br># Volume for shared data<br>volume = k8s.V1Volume(<br>    name=&quot;shared-data&quot;,<br>    persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(<br>        claim_name=pvc_name  # This should match the PVC name in your manifest<br>    )<br>)<br>volume_mount = k8s.V1VolumeMount(<br>    name=&quot;shared-data&quot;,<br>    mount_path=SHARED_VOLUME_PATH<br>)<br><br>def upload_incremental_to_s3(**context):<br>    &quot;&quot;&quot;Uploads incremental backup directory to S3.&quot;&quot;&quot;<br>    ts = context[&#39;ts_nodash&#39;]<br>    backup_path = f&quot;{BACKUP_BASE_DIR}/incremental_{ts}&quot;<br>    s3_hook = S3Hook(aws_conn_id=&#39;s3_logging&#39;)<br><br>    if not os.path.exists(backup_path):<br>        raise FileNotFoundError(f&quot;{backup_path} not found&quot;)<br><br>    for root, dirs, files in os.walk(backup_path):<br>        for file in files:<br>            local_path = os.path.join(root, file)<br>            relative_path = os.path.relpath(local_path, backup_path)<br>            s3_key = f&quot;{S3_PREFIX}/mongo_incremental_backup_{ts}/{relative_path}&quot;<br>            s3_hook.load_file(<br>                filename=local_path,<br>                key=s3_key,<br>                bucket_name=S3_BUCKET,<br>                replace=True,<br>            )<br>            print(f&quot;Uploaded: s3://{S3_BUCKET}/{s3_key}&quot;)<br><br># DAG Definition<br>with DAG(<br>    &quot;mongodb_incremental_backup_s3&quot;,<br>    default_args=default_args,<br>    schedule_interval=&quot;0 */6 * * *&quot;,  # Every 6 hours (adjust as needed)<br>    catchup=False,<br>    tags=[&quot;mongodb&quot;, &quot;incremental&quot;, &quot;s3&quot;],<br>    description=&quot;Incremental MongoDB backup DAG with oplog timestamp tracking&quot;,<br>) as dag:<br>    # Create PVC task<br>    create_pvc = KubernetesPodOperator(<br>        task_id=&quot;create_temp_pvc&quot;,<br>        name=&quot;create-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>        arguments=[&quot;&quot;&quot;<br>            cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: mongo-backup-pvc<br>  namespace: airflow<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 2Gi<br>  storageClassName: ebs-sc-delete<br>EOF<br>        &quot;&quot;&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>    )<br><br>    # Get the last timestamp from Airflow variables<br>    last_ts = Variable.get(LAST_TS_VAR_KEY, default_var=None)<br>    # Parse the timestamp string into t and i components<br>    if last_ts:<br>        # Extract t and i values from Timestamp(t, i) format<br>        import re<br>        match = re.match(r&quot;Timestamp\((\d+),\s*(\d+)\)&quot;, last_ts)<br>        if match:<br>            t_value, i_value = match.groups()<br>        else:<br>            t_value, i_value = &quot;0&quot;, &quot;1&quot;  # Default values if parsing fails<br>    else:<br>        t_value, i_value = &quot;0&quot;, &quot;1&quot;  # Default values if no timestamp exists<br><br>    # Incremental backup task<br>    incremental_backup = KubernetesPodOperator(<br>        task_id=&quot;run_incremental_backup&quot;,<br>        name=&quot;mongo-incremental-backup&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;mongo:4.4&quot;,<br>        cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>        env_vars=[<br>            {<br>                &quot;name&quot;: &quot;MONGO_HOST&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_HOST&quot;<br>                    }<br>                }<br>            },<br>            {<br>                &quot;name&quot;: &quot;MONGO_PORT&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_PORT&quot;<br>                    }<br>                }<br>            },<br>            {<br>                &quot;name&quot;: &quot;MONGO_USER&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_USER&quot;<br>                    }<br>                }<br>            },<br>            {<br>                &quot;name&quot;: &quot;MONGO_PASS&quot;,<br>                &quot;valueFrom&quot;: {<br>                    &quot;secretKeyRef&quot;: {<br>                        &quot;name&quot;: &quot;airflow-mongodb-secret&quot;,<br>                        &quot;key&quot;: &quot;MONGO_PASS&quot;<br>                    }<br>                }<br>            }<br>        ],<br>        arguments=[f&quot;&quot;&quot;<br>            set -ex<br>            echo &quot;Starting backup process...&quot;<br>            <br>            BACKUP_DIR=&quot;{BACKUP_BASE_DIR}&quot;<br>            <br>            echo &quot;Creating backup directory...&quot;<br>            mkdir -p &quot;$BACKUP_DIR&quot;<br>            <br>            if [ -z &quot;$LAST_TIMESTAMP&quot; ]; then<br>                echo &quot;Getting initial timestamp...&quot;<br>                LAST_TS=$(mongo --host $MONGO_HOST --port $MONGO_PORT \<br>                    --username $MONGO_USER --password $MONGO_PASS \<br>                    --authenticationDatabase {MONGO_AUTH_DB} \<br>                    --quiet \<br>                    --eval &#39;db.getSiblingDB(&quot;local&quot;).oplog.rs.find().sort({{ $natural:1 }}).limit(1).toArray()[0].ts&#39;)<br>            else<br>                LAST_TS=&quot;$LAST_TIMESTAMP&quot;<br>            fi<br>            <br>            echo &quot;Running backup with timestamp: Timestamp({t_value}, {i_value})&quot;<br>            BACKUP_PATH=&quot;$BACKUP_DIR/incremental_{TIMESTAMP}&quot;<br>            <br>            mongodump --host $MONGO_HOST --port $MONGO_PORT \<br>                --username $MONGO_USER --password $MONGO_PASS \<br>                --authenticationDatabase {MONGO_AUTH_DB}  \<br>                --db {MONGO_DB} --collection {MONGO_COLLECTION} \<br>                --query &#39;{{&quot;ts&quot;: {{&quot;$gt&quot;: {{&quot;$timestamp&quot;: {{&quot;t&quot;: {t_value}, &quot;i&quot;: {i_value}}}}}}}}}&#39; \<br>                --out &quot;$BACKUP_PATH&quot;<br>            <br>            # Update timestamp after successful backup<br>            NEW_TS=$(mongo --host $MONGO_HOST --port $MONGO_PORT \<br>                --username $MONGO_USER --password $MONGO_PASS \<br>                --authenticationDatabase {MONGO_AUTH_DB} \<br>                --quiet \<br>                --eval &#39;db.getSiblingDB(&quot;local&quot;).oplog.rs.find().sort({{ $natural:-1 }}).limit(1).toArray()[0].ts&#39;)<br>            <br>            # Write the new timestamp to a file that will be processed by the next task<br>            echo &quot;$NEW_TS&quot; &gt; &quot;$BACKUP_PATH/last_timestamp.txt&quot;<br>            <br>            echo &quot;Backup completed to $BACKUP_PATH&quot;<br>            &quot;&quot;&quot;<br>        ],<br>        volumes=[volume],<br>        volume_mounts=[volume_mount],<br>        is_delete_operator_pod=True,<br>        startup_timeout_seconds=300,<br>        get_logs=True,<br>    )<br><br>    def update_timestamp(**context):<br>        &quot;&quot;&quot;Update the Airflow variable with the new timestamp&quot;&quot;&quot;<br>        ts = context[&#39;ts_nodash&#39;]<br>        backup_path = f&quot;{BACKUP_BASE_DIR}/incremental_{ts}&quot;<br>        timestamp_file = f&quot;{backup_path}/last_timestamp.txt&quot;<br>        <br>        with open(timestamp_file, &#39;r&#39;) as f:<br>            new_ts = f.read().strip()<br>        <br>        Variable.set(LAST_TS_VAR_KEY, new_ts)<br>        print(f&quot;Updated timestamp to: {new_ts}&quot;)<br><br>    update_timestamp_task = PythonOperator(<br>        task_id=&quot;update_timestamp&quot;,<br>        python_callable=update_timestamp,<br>        executor_config={<br>            &quot;pod_override&quot;: k8s.V1Pod(<br>                spec=k8s.V1PodSpec(<br>                    containers=[<br>                        k8s.V1Container(name=&quot;base&quot;, volume_mounts=[volume_mount])<br>                    ],<br>                    volumes=[volume]<br>                )<br>            )<br>        },<br>    )<br><br>    upload_to_s3_task = PythonOperator(<br>        task_id=&quot;upload_to_s3&quot;,<br>        python_callable=upload_incremental_to_s3,<br>        executor_config={<br>            &quot;pod_override&quot;: k8s.V1Pod(<br>                spec=k8s.V1PodSpec(<br>                    containers=[<br>                        k8s.V1Container(name=&quot;base&quot;, volume_mounts=[volume_mount])<br>                    ],<br>                    volumes=[volume]<br>                )<br>            )<br>        },<br>    )<br>        # Delete PVC task<br>    delete_pvc = KubernetesPodOperator(<br>        task_id=&quot;delete_temp_pvc&quot;,<br>        name=&quot;delete-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;kubectl&quot;],<br>        arguments=[&quot;delete&quot;, &quot;pvc&quot;, pvc_name, &quot;-n&quot;, &quot;airflow&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>    )<br><br>    create_pvc &gt;&gt; incremental_backup &gt;&gt; update_timestamp_task &gt;&gt; upload_to_s3_task  &gt;&gt; delete_pvc</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10b85ea5049f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consul KV Backup to S3 — With AirFlow DAG]]></title>
            <link>https://medium.com/@MadhavPrajapati/consul-kv-backup-to-s3-using-airflow-ab75919db12b?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab75919db12b</guid>
            <category><![CDATA[consul]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[key-value-store]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 02:31:29 GMT</pubDate>
            <atom:updated>2025-04-11T04:27:34.551Z</atom:updated>
            <content:encoded><![CDATA[<h3>📘 What is Consul?</h3><p>Consul is a <strong>service mesh and service discovery tool</strong> by HashiCorp.<br> It’s widely used in <strong>DevOps</strong>, <strong>microservices</strong>, and <strong>infrastructure orchestration</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*8PTqGL2RIvTh6ccJ.png" /></figure><h3>Goal</h3><p>Automatically <strong>backup all key-value (KV) pairs from Consul</strong> every day and <strong>upload the backup file to an S3 bucket</strong>.</p><h3>How it works (Simple Explanation)</h3><ol><li><strong>Airflow DAG runs daily</strong>.</li><li>It connects to the <strong>Consul server</strong> and pulls all the <strong>KV data</strong>.</li><li>The KV data is saved into a <strong>JSON file</strong> on the local machine.</li><li>That file is then <strong>uploaded to Amazon S3</strong> using Airflow’s built-in AWS support.</li></ol><h3>File: consul_kv_backup_to_s3.py</h3><h3>🔧 Imports</h3><p>The DAG uses:</p><ul><li>Airflow core (DAG, PythonOperator)</li><li>S3Hook to upload to AWS S3</li><li>requests to get data from Consul</li><li>json, os, logging for data and file handling</li></ul><h3>🔢 Constants (Configuration)</h3><pre>CONSUL_HOST = &quot;ip address&quot;<br>CONSUL_PORT = &quot;8500&quot;<br>S3_BUCKET = &quot;madhav-airflow-logging-bucket&quot;<br>S3_KEY_PREFIX = &quot;consule_backup/&quot;</pre><ul><li><strong>CONSUL_HOST</strong> and <strong>PORT</strong>: Address of your Consul server</li><li><strong>S3_BUCKET</strong>: Where the backup file will be saved</li><li><strong>S3_KEY_PREFIX</strong>: Folder path inside the bucket</li></ul><h3>backup_kv_raw_and_upload() - Main Task Function</h3><p>Here’s what it does step by step:</p><ol><li><strong>Get KV Data from Consul</strong></li></ol><pre>        # 1. Get data from Consul<br>        url = f&quot;http://{CONSUL_HOST}:{CONSUL_PORT}/v1/kv/?recurse=true&quot;<br>        response = requests.get(url, timeout=30)<br>        response.raise_for_status()<br>        kv_data = response.json()<br>        logging.info(f&quot;Retrieved {len(kv_data)} KV pairs from Consul&quot;)</pre><p>Calls Consul’s API to fetch all KV entries.</p><ol><li><strong>Save Data to Local File</strong></li></ol><pre>       # 2. Save to local file<br>        timestamp = datetime.utcnow().strftime(&quot;%Y-%m-%d_%H-%M-%S&quot;)<br>        local_path = f&quot;/tmp/consul-kv-backup-{timestamp}.json&quot;<br>        with open(local_path, &quot;w&quot;) as f:<br>            json.dump(kv_data, f, indent=2)</pre><ul><li>Stores the data in a file like: /tmp/consul-kv-backup-2025-04-09_05-30-00.json.</li></ul><ol><li><strong>Upload File to S3</strong></li></ol><pre>        # 3. Upload to S3<br>        s3 = S3Hook(aws_conn_id=&#39;s3_logging&#39;)<br>        s3_key = f&quot;{S3_KEY_PREFIX}consul-kv-backup-{timestamp}.json&quot;<br>        s3.load_file(<br>            filename=local_path,<br>            key=s3_key,<br>            bucket_name=S3_BUCKET,<br>            replace=True<br>        )<br>        logging.info(f&quot;Backup uploaded to S3: {s3_key}&quot;)</pre><ul><li>Uses the Airflow S3Hook to upload the file to S3.</li></ul><h3>DAG Configuration</h3><pre>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;retries&#39;: 2,<br>    &#39;retry_delay&#39;: timedelta(minutes=5),<br>    &#39;start_date&#39;: days_ago(1),<br>}</pre><ul><li>Will retry the task twice if it fails, with a 5-minute wait.</li><li>Starts running from <strong>yesterday</strong> (good for testing or first runs).</li></ul><pre>with DAG(<br>    dag_id=&quot;consul_kv_raw_backup_to_s3&quot;,<br>    schedule_interval=&quot;@daily&quot;,<br>    ...<br>) as dag:</pre><ul><li>DAG will run <strong>once per day</strong>.</li></ul><pre>backup_task = PythonOperator(<br>    task_id=&quot;backup_kv_raw_and_upload&quot;,<br>    python_callable=backup_kv_raw_and_upload,<br>)</pre><ul><li>Runs the backup function as a task in the DAG.</li></ul><h3>✅ Requirements to Run</h3><ol><li>Your <strong>Airflow environment</strong> must have:</li></ol><ul><li>Network access to Consul (&lt;host&gt;:8500)</li><li>AWS credentials set up with aws_conn_id=&#39;s3_logging&#39;</li></ul><ol><li>Your <strong>S3 bucket</strong> must exist:</li></ol><pre>madhav-airflow-logging-bucket/consule_backup/</pre><ol><li>Airflow must have the <strong>apache-airflow-providers-amazon</strong> and requests packages installed.</li></ol><h4>Full Code:-</h4><pre>&quot;&quot;&quot;<br>DAG to backup Consul KV store to S3 daily.<br>&quot;&quot;&quot;<br><br>from airflow import DAG<br>from airflow.operators.python import PythonOperator<br>from airflow.providers.amazon.aws.hooks.s3 import S3Hook<br>from airflow.utils.dates import days_ago<br>from datetime import datetime, timedelta<br>import json<br>import logging<br>import os<br>import requests<br><br># Constants<br>CONSUL_HOST = &quot;ip address&quot;<br>CONSUL_PORT = &quot;8500&quot;<br>S3_BUCKET = &quot;madhav-airflow-logging-bucket&quot;<br>S3_KEY_PREFIX = &quot;consule_backup/&quot;<br><br>def backup_kv_raw_and_upload():<br>    &quot;&quot;&quot;Backup Consul KV store to S3.&quot;&quot;&quot;<br>    try:<br>        # 1. Get data from Consul<br>        url = f&quot;http://{CONSUL_HOST}:{CONSUL_PORT}/v1/kv/?recurse=true&quot;<br>        response = requests.get(url, timeout=30)<br>        response.raise_for_status()<br>        kv_data = response.json()<br>        logging.info(f&quot;Retrieved {len(kv_data)} KV pairs from Consul&quot;)<br><br>        # 2. Save to local file<br>        timestamp = datetime.utcnow().strftime(&quot;%Y-%m-%d_%H-%M-%S&quot;)<br>        local_path = f&quot;/tmp/consul-kv-backup-{timestamp}.json&quot;<br>        with open(local_path, &quot;w&quot;) as f:<br>            json.dump(kv_data, f, indent=2)<br><br>        # 3. Upload to S3<br>        s3 = S3Hook(aws_conn_id=&#39;s3_logging&#39;)<br>        s3_key = f&quot;{S3_KEY_PREFIX}consul-kv-backup-{timestamp}.json&quot;<br>        s3.load_file(<br>            filename=local_path,<br>            key=s3_key,<br>            bucket_name=S3_BUCKET,<br>            replace=True<br>        )<br>        logging.info(f&quot;Backup uploaded to S3: {s3_key}&quot;)<br><br>        # 4. Cleanup local file<br>        os.remove(local_path)<br><br>    except Exception as e:<br>        logging.error(f&quot;Backup failed: {str(e)}&quot;)<br>        raise<br><br># DAG definition<br>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;retries&#39;: 2,<br>    &#39;retry_delay&#39;: timedelta(minutes=5),<br>    &#39;start_date&#39;: days_ago(1),<br>}<br><br>with DAG(<br>    dag_id=&quot;consul_kv_raw_backup_to_s3&quot;,<br>    schedule_interval=&quot;@daily&quot;,<br>    catchup=False,<br>    default_args=default_args,<br>    tags=[&#39;consul&#39;, &#39;backup&#39;],<br>    description=&quot;Daily Consul KV backup to S3&quot;<br>) as dag:<br><br>    backup_task = PythonOperator(<br>        task_id=&quot;backup_kv_raw_and_upload&quot;,<br>        python_callable=backup_kv_raw_and_upload,<br>    )</pre><h3>📌 Summary</h3><p>Step Action Tool Used 1 Get KV data from Consul requests 2 Save to a local JSON file json module 3 Upload to S3 S3Hook 4 Delete local backup file os.remove() 5 Run daily in Airflow DAG</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab75919db12b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch Index Rotation — With AirFlow DAG]]></title>
            <link>https://medium.com/@MadhavPrajapati/elasticsearch-index-rotation-with-airflow-dag-d27c820240df?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/d27c820240df</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[rotation]]></category>
            <category><![CDATA[dağ]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 10:15:25 GMT</pubDate>
            <atom:updated>2025-04-09T10:15:25.599Z</atom:updated>
            <content:encoded><![CDATA[<h3>Elasticsearch Index Rotation — With AirFlow DAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VS02L38GZsI9A3NT" /></figure><h3>Goal of This DAG</h3><p>This Airflow DAG is designed to:</p><ol><li><strong>Fetch</strong> all indices from an Elasticsearch cluster.</li><li><strong>Close</strong> indices that are older than a certain number of days.</li><li><strong>Delete</strong> older indices then 2 days</li><li>Use a <strong>temporary PVC (Persistent Volume Claim)</strong> to store index data during processing.</li></ol><h3>Key Concepts</h3><ul><li><strong>Index</strong>: A collection of documents in Elasticsearch (like a database table).</li><li><strong>Closing an index</strong>: Makes it inactive but still stored in Elasticsearch.</li><li><strong>Deleting an index</strong>: Permanently removes it from Elasticsearch.</li><li><strong>PVC</strong>: Temporary storage used by tasks to save data across multiple steps.</li></ul><h3>Configuration Settings</h3><pre>CLOSE_DAYS = 1   # Close indices older than 1 day<br>DELETE_DAYS = 2  # Delete indices older than 2 days</pre><p>These values mean:</p><ul><li>If an index is <strong>1+ day old</strong>, we try to <strong>close</strong> it.</li><li>If an index is <strong>2+ days old</strong>, we <strong>delete</strong> it.</li></ul><h3>DAG Tasks Explained</h3><h3>1️⃣ create_temp_pvc — Create Temporary Storage</h3><ul><li>Uses KubernetesPodOperator to run kubectl and create a <strong>1Gi PVC</strong>.</li><li>The PVC is mounted at /shared_data and used to temporarily store index data.</li></ul><pre>create_pvc = KubernetesPodOperator(<br>        task_id=&quot;create_temp_pvc&quot;,<br>        name=&quot;create-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>        arguments=[&quot;&quot;&quot;<br>            cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: es-temp-pvc<br>  namespace: airflow<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 1Gi<br>  storageClassName: ebs-sc-delete<br>EOF<br>        &quot;&quot;&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>    )</pre><p>In this, we require serviceAccount , Role , RoleBinding</p><h4>ServiceAccount:-</h4><p>Means:<br> “Run this pod using the Kubernetes <strong>ServiceAccount</strong> named airflow.”</p><p>A <strong>ServiceAccount</strong> is like an identity for your pod inside the Kubernetes cluster, and it controls <strong>what that pod is allowed to do</strong>.</p><pre>serviceAccount:<br>  ## if a Kubernetes ServiceAccount is created<br>  ## - if `false`, you must create the service account outside this chart with name: `serviceAccount.name`<br>  ##<br>  create: true<br><br>  ## the name of the ServiceAccount<br>  ## - by default the name is generated using the `airflow.serviceAccountName` template in `_helpers/common.tpl`<br>  ##<br>  name: &quot;airflow&quot;<br>  annotations:<br>    eks.amazonaws.com/role-arn: arn:aws:iam::185889327143:role/KarpenterControllerRole-staging-solvei8-eks</pre><h4><strong>Role:-</strong></h4><p>This <strong>Role</strong> grants permissions within the airflow namespace to:</p><ul><li>✅ Fully manage PVCs: create, delete, get, update, etc.</li><li>✅ View PVs (PersistentVolumes): read-only access.</li></ul><p>Think of this as <strong>“what the pod is allowed to do inside the cluster.</strong></p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  namespace: airflow<br>  name: airflow-pvc-role<br>rules:<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;persistentvolumeclaims&quot;]<br>  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;persistentvolumes&quot;]<br>  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br></pre><h4>RoleBinding:-</h4><p>This connects the dots:</p><ul><li>It binds the <strong>airflow ServiceAccount</strong> to the <strong>airflow-pvc-role</strong> Role.</li><li>In plain English: <strong>“Hey Kubernetes, allow any pod using the </strong><strong>airflow ServiceAccount to use the permissions defined in </strong><strong>airflow-pvc-role.&quot;</strong></li></ul><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: airflow-pvc-rolebinding<br>  namespace: airflow<br>subjects:<br>- kind: ServiceAccount<br>  name: airflow<br>  namespace: airflow<br>roleRef:<br>  kind: Role<br>  name: airflow-pvc-role<br>  apiGroup: rbac.authorization.k8s.io</pre><h4>🔁 Flow Summary (TL;DR)</h4><ol><li>The DAG creates a Kubernetes pod to run kubectl apply and make a PVC.</li><li>This pod runs using the airflow <strong>ServiceAccount</strong>.</li><li>The airflow ServiceAccount has <strong>RBAC permissions</strong> (via Role and Role Binding) to manage PVCs and PVs.</li><li>The PVC is used as <strong>temporary shared storage</strong> for the rest of the DAG.</li></ol><h3>2️⃣ fetch_indices — Fetch All Indices</h3><ul><li>A Python task that:</li><li>Calls the Elasticsearch API to get a list of all indices.</li><li>Saves the result as a JSON file to /shared_data/indices.json.</li></ul><h3>3️⃣ close_old_indices — Close Old Indices</h3><ul><li>Loads the JSON file created in the previous step.</li><li>For each index:</li><li>Checks if it’s <strong>open</strong> and <strong>older than 1 day</strong>.</li><li>Sends a request to <strong>close</strong> it if both conditions are true.</li></ul><h3>4️⃣ delete_old_indices — Delete Very Old Indices</h3><ul><li>Loads the same JSON file.</li><li>For each index:</li><li>Checks if it’s <strong>older than 2 days</strong>.</li><li>Sends a request to <strong>delete</strong> it from Elasticsearch.</li></ul><h3>5️⃣ delete_temp_pvc — Clean Up</h3><ul><li>Deletes the temporary PVC we created in step 1.</li><li>Prevents wasting resources by cleaning up after the DAG finishes.</li></ul><h3>🔁 Task Flow</h3><pre>  create_temp_pvc<br>        ↓<br>  fetch_indices<br>        ↓<br> close_old_indices<br>        ↓<br>delete_old_indices<br>        ↓<br>  delete_temp_pvc</pre><h3>📌 Important Notes</h3><ul><li>All tasks run in Kubernetes pods.</li><li>Shared data between tasks is passed through the mounted PVC at /shared_data.</li><li>Only indices named in this format will be processed: myapp-ops-YYYY-MM-DD.</li></ul><h3>📅 Scheduling</h3><p>This DAG runs <strong>every day</strong> (@daily) and performs the rotation based on the age of the indices.</p><h3>CODE:-</h3><pre>from datetime import datetime, timedelta<br>from airflow import DAG<br>from airflow.operators.python import PythonOperator<br>import requests<br>import json<br>import os<br>from kubernetes.client import models as k8s<br>from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator<br><br># Elasticsearch Configuration<br>ES_HOST = &quot;&lt;host IP&gt;:9200&quot;<br>INDEX_FILE = &quot;/shared_data/indices.json&quot;<br># Index Retention Policy<br>CLOSE_DAYS = 1   # Close indices older than 1 day<br>DELETE_DAYS = 2  # Delete indices older than 2 days<br># Volume and PVC configuration for temporary storage<br>pvc_name = &quot;es-temp-pvc&quot;<br># Kubernetes PVC Pod Override<br><br>pod_override = k8s.V1Pod(<br>    spec=k8s.V1PodSpec(<br>        containers=[<br>            k8s.V1Container(<br>                name=&quot;base&quot;,<br>                volume_mounts=[<br>                    k8s.V1VolumeMount(<br>                        name=&quot;shared-data&quot;,<br>                        mount_path=&quot;/shared_data&quot;<br>                    )<br>                ]<br>            )<br>        ],<br>        volumes=[<br>            k8s.V1Volume(<br>                name=&quot;shared-data&quot;,<br>                persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(<br>                    claim_name=pvc_name<br>                )<br>            )<br>        ]<br>    )<br>)<br><br># Default DAG arguments<br>default_args = {<br>    &quot;owner&quot;: &quot;airflow&quot;,<br>    &quot;start_date&quot;: datetime(2025, 3, 26),<br>    &quot;retries&quot;: 1,<br>    &quot;retry_delay&quot;: timedelta(minutes=5),<br>}<br><br>dag = DAG(<br>    &quot;elasticsearch_index_rotation&quot;,<br>    default_args=default_args,<br>    schedule_interval=&quot;@daily&quot;,<br>    catchup=False,<br>)<br>    <br>    # Create PVC task<br>create_pvc = KubernetesPodOperator(<br>        task_id=&quot;create_temp_pvc&quot;,<br>        name=&quot;create-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>        arguments=[&quot;&quot;&quot;<br>            cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: es-temp-pvc<br>  namespace: airflow<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 1Gi<br>  storageClassName: ebs-sc-delete<br>EOF<br>        &quot;&quot;&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>    )<br>    <br>def fetch_indices():<br>    &quot;&quot;&quot;Fetch all Elasticsearch indices and save to a file.&quot;&quot;&quot;<br>    print(&quot;[FETCH] Fetching indices from Elasticsearch...&quot;)<br>    <br>    try:<br>        response = requests.get(f&quot;{ES_HOST}/_cat/indices?v&amp;format=json&quot;, timeout=10)<br>        response.raise_for_status()<br>    except requests.exceptions.RequestException as e:<br>        print(f&quot;[FETCH] ERROR: Failed to fetch indices. Error: {e}&quot;)<br>        raise<br>    <br>    indices = response.json()<br>    print(f&quot;[FETCH] Retrieved {len(indices)} indices from Elasticsearch.&quot;)<br><br><br>    with open(INDEX_FILE, &quot;w&quot;) as f:<br>        json.dump(indices, f, indent=2)<br>    <br>    print(f&quot;[FETCH] Saved indices data to {INDEX_FILE}&quot;)<br><br>def extract_index_date(index_name):<br>    &quot;&quot;&quot;Extracts the date from an index name formatted as &#39;myapp-ops-YYYY-MM-DD&#39;.&quot;&quot;&quot;<br>    try:<br>        date_str = index_name.split(&quot;-&quot;)[-3:]  # Extract last three parts (YYYY-MM-DD)<br>        index_date = datetime.strptime(&quot;-&quot;.join(date_str), &quot;%Y-%m-%d&quot;).date()<br>        return index_date<br>    except ValueError:<br>        print(f&quot;[ERROR] Skipping {index_name}: Invalid date format&quot;)<br>        return None<br><br>def close_old_indices():<br>    &quot;&quot;&quot;Close indices older than CLOSE_DAYS and not already closed.&quot;&quot;&quot;<br>    print(&quot;[CLOSE] Checking if index file exists...&quot;)<br>    if not os.path.exists(INDEX_FILE):<br>        print(f&quot;[CLOSE] ERROR: Indices file not found: {INDEX_FILE}&quot;)<br>        raise FileNotFoundError(f&quot;Indices file not found: {INDEX_FILE}&quot;)<br><br>    with open(INDEX_FILE, &quot;r&quot;) as f:<br>        indices = json.load(f)<br><br>    today = datetime.utcnow().date()<br>    <br>    for index in indices:<br>        index_name = index[&quot;index&quot;]<br>        index_status = index[&quot;status&quot;]  # Check if already closed<br>        index_date = extract_index_date(index_name)<br>        <br>        if not index_date:<br>            continue  # Skip if the date couldn&#39;t be extracted<br><br>        days_old = (today - index_date).days<br>        print(f&quot;[CLOSE] Checking {index_name}: {days_old} days old (Status: {index_status})&quot;)<br><br>        if days_old &gt;= CLOSE_DAYS and index_status == &quot;open&quot;:<br>            print(f&quot;[CLOSE] Closing index {index_name}...&quot;)<br>            try:<br>                response = requests.post(f&quot;{ES_HOST}/{index_name}/_close&quot;, timeout=10)<br>                response.raise_for_status()<br>                print(f&quot;[CLOSE] Successfully closed index: {index_name}&quot;)<br>            except requests.exceptions.RequestException as e:<br>                print(f&quot;[CLOSE] ERROR: Failed to close {index_name}. Error: {e}&quot;)<br><br><br>def delete_old_indices():<br>    &quot;&quot;&quot;Delete indices older than DELETE_DAYS.&quot;&quot;&quot;<br>    print(&quot;[DELETE] Checking if index file exists...&quot;)<br>    if not os.path.exists(INDEX_FILE):<br>        print(f&quot;[DELETE] ERROR: Indices file not found: {INDEX_FILE}&quot;)<br>        raise FileNotFoundError(f&quot;Indices file not found: {INDEX_FILE}&quot;)<br><br>    with open(INDEX_FILE, &quot;r&quot;) as f:<br>        indices = json.load(f)<br><br>today = datetime.utcnow().date()<br>    <br>    for index in indices:<br>        index_name = index[&quot;index&quot;]<br>        index_date = extract_index_date(index_name)<br>        if not index_date:<br>            continue  # Skip if the date couldn&#39;t be extracted<br>        days_old = (today - index_date).days<br>        print(f&quot;[DELETE] Checking {index_name}: {days_old} days old&quot;)<br>        if days_old &gt;= DELETE_DAYS:<br>            print(f&quot;[DELETE] Deleting index {index_name}...&quot;)<br>            try:<br>                response = requests.delete(f&quot;{ES_HOST}/{index_name}&quot;, timeout=10)<br>                response.raise_for_status()<br>                print(f&quot;[DELETE] Successfully deleted index: {index_name}&quot;)<br>            except requests.exceptions.RequestException as e:<br>                print(f&quot;[DELETE] ERROR: Failed to delete {index_name}. Error: {e}&quot;)<br>        # Delete PVC task<br>delete_pvc = KubernetesPodOperator(<br>        task_id=&quot;delete_temp_pvc&quot;,<br>        name=&quot;delete-pvc&quot;,<br>        namespace=&quot;airflow&quot;,<br>        image=&quot;bitnami/kubectl:latest&quot;,<br>        cmds=[&quot;kubectl&quot;],<br>        arguments=[&quot;delete&quot;, &quot;pvc&quot;, pvc_name, &quot;-n&quot;, &quot;airflow&quot;],<br>        service_account_name=&#39;airflow&#39;,<br>        is_delete_operator_pod=True,<br>        dag=dag<br>)<br># Task 1: Fetch all indices<br>fetch_task = PythonOperator(<br>    task_id=&quot;fetch_indices&quot;,<br>    python_callable=fetch_indices,<br>    dag=dag,<br>    executor_config={&quot;pod_override&quot;: pod_override}<br>)<br># Task 2: Close indices older than CLOSE_DAYS<br>close_task = PythonOperator(<br>    task_id=&quot;close_old_indices&quot;,<br>    python_callable=close_old_indices,<br>    dag=dag,<br>    executor_config={&quot;pod_override&quot;: pod_override}<br>)<br># Task 3: Delete indices older than DELETE_DAYS<br>delete_task = PythonOperator(<br>    task_id=&quot;delete_old_indices&quot;,<br>    python_callable=delete_old_indices,<br>    dag=dag,<br>    executor_config={&quot;pod_override&quot;: pod_override}<br>)<br># Define task dependencies<br>create_pvc &gt;&gt;fetch_task &gt;&gt; close_task &gt;&gt; delete_task &gt;&gt; delete_pvc</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d27c820240df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB Full Backups to S3 — With Airflow DAG]]></title>
            <link>https://medium.com/@MadhavPrajapati/automating-mongodb-backups-to-s3-using-apache-airflow-fa61ecdc197d?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa61ecdc197d</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[mongo]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Mon, 31 Mar 2025 07:55:56 GMT</pubDate>
            <atom:updated>2025-04-11T04:28:42.425Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Regular backups of databases are critical for disaster recovery and data protection. This article explains how to automate <strong>MongoDB backups to Amazon S3</strong> using <strong>Apache Airflow</strong> with Kubernetes integration. The process involves:</p><p>✅ Running <strong>mongodump</strong> to create a backup<br>✅ Verifying the backup files<br>✅ Uploading the backup to an <strong>S3 bucket</strong></p><h3>What is Apache Airflow?</h3><p><strong>Apache Airflow</strong> is an open-source platform for orchestrating complex workflows. It allows scheduling, monitoring, and managing tasks in a structured way.</p><p><strong>Key Features:</strong></p><ul><li>DAG-based task execution</li><li>Task dependencies and retries</li><li>Integration with Kubernetes and cloud platforms</li><li>Python-based extensibility</li></ul><h3>What is MongoDB Backup?</h3><p>A <strong>MongoDB backup</strong> involves taking a snapshot of the database to restore it in case of failures. The <strong>mongodump</strong> tool is commonly used for this purpose.</p><p><strong>Why is MongoDB Backup Important?</strong></p><ul><li>Prevents <strong>data loss</strong> due to failures</li><li>Enables <strong>disaster recovery</strong></li><li>Facilitates <strong>data migration</strong> between environments</li></ul><h3>DAG for MongoDB Backup and S3 Upload</h3><p>Below is the Airflow <strong>DAG (Directed Acyclic Graph)</strong> that automates the MongoDB backup and uploads it to S3.</p><h3>Step 1: Import Required Libraries</h3><pre>from datetime import datetime, timedelta<br>from airflow import DAG<br>from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator<br>from airflow.operators.python import PythonOperator<br>from kubernetes.client import models as k8s<br>from airflow.hooks.S3_hook import S3Hook<br>import os</pre><p>📌 <strong>Why?</strong></p><ul><li>KubernetesPodOperator → Runs tasks inside Kubernetes pods.</li><li>PythonOperator → Runs Python functions in Airflow.</li><li>S3Hook → Handles file uploads to <strong>Amazon S3</strong>.</li></ul><h3>Step 2: Define DAG Configuration</h3><pre>default_args = {<br>    &quot;owner&quot;: &quot;airflow&quot;,<br>    &quot;depends_on_past&quot;: False,<br>    &quot;start_date&quot;: datetime(2025, 3, 28),<br>    &quot;retries&quot;: 1,<br>    &quot;retry_delay&quot;: timedelta(minutes=5),<br>}<br><br>dag = DAG(<br>    &quot;mongodb_backup_s3&quot;,<br>    default_args=default_args,<br>    schedule_interval=&quot;@daily&quot;,<br>    catchup=False,<br>    tags=[&#39;mongodb&#39;, &#39;backup&#39;, &#39;s3&#39;],<br>    description=&#39;MongoDB backup DAG using mongodump with S3 upload&#39;<br>)</pre><p>📌 <strong>Key Parameters:</strong></p><ul><li>start_date → Sets the DAG start date.</li><li>retries → Retries a failed task <strong>once</strong>.</li><li>schedule_interval=&quot;@daily&quot; → Run the DAG Daily.</li></ul><h3>Step 3: Define Storage Configuration</h3><pre>BACKUP_PATH = &quot;/shared_data/mongo_backup-1&quot;<br>S3_BUCKET = &quot;mongo-backups-bucket&quot;  # Replace with your S3 bucket name<br>S3_KEY_PREFIX = &quot;mongodb-backups&quot;</pre><p>📌 <strong>Explanation:</strong></p><ul><li>BACKUP_PATH → Where MongoDB backups will be stored.</li><li>S3_BUCKET → The S3 bucket where backups will be uploaded.</li></ul><h3>Step 4:- Define Kubernetes Volumes</h3><pre>volume = k8s.V1Volume(<br>    name=&quot;shared-data&quot;,<br>    persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(<br>        claim_name=&quot;airflow-shared-pvc&quot;<br>    )<br>)<br><br>volume_mount = k8s.V1VolumeMount(<br>    name=&quot;shared-data&quot;,<br>    mount_path=&quot;/shared_data&quot;<br>)</pre><p>📌 <strong>Why?</strong></p><ul><li>Uses a <strong>Persistent Volume Claim (PVC)</strong> to store backups.</li><li>Ensures that multiple tasks can access the same backup files.</li></ul><h3>Task 5: Run MongoDB Backup Using KubernetesPodOperator</h3><pre>mongodump_task = KubernetesPodOperator(<br>    task_id=&quot;run_mongodump&quot;,<br>    name=&quot;mongo-backup-job&quot;,<br>    namespace=&quot;airflow&quot;,<br>    image=&quot;mongo:4.4&quot;,<br>    cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>    arguments=[<br>        f&quot;mongodump --host=mongodb-service.airflow.svc.cluster.local --port=27017 --out={BACKUP_PATH}&quot;<br>    ],<br>    volumes=[volume],<br>    volume_mounts=[volume_mount],<br>    is_delete_operator_pod=True,<br>    dag=dag,<br>)</pre><p>📌 <strong>Why?</strong></p><ul><li>Runs a MongoDB container (mongo:4.4).</li><li>Executes <strong>mongodump</strong> to create a database backup.</li><li>Stores the backup in the shared volume (/shared_data).</li></ul><h3>Task 6: Verify Backup Files</h3><pre>==verify_backup_task = KubernetesPodOperator(<br>    task_id=&quot;verify_backup&quot;,<br>    name=&quot;verify-backup&quot;,<br>    namespace=&quot;airflow&quot;,<br>    image=&quot;busybox&quot;,<br>    cmds=[&quot;sh&quot;, &quot;-c&quot;],<br>    arguments=[<br>        f&quot;ls -lah {BACKUP_PATH}&quot;<br>    ],<br>    volumes=[volume],<br>    volume_mounts=[volume_mount],<br>    is_delete_operator_pod=True,<br>    dag=dag,<br>)</pre><p>📌 <strong>Why?</strong></p><ul><li>Uses a <strong>lightweight busybox container</strong> to list backup files.</li><li>Ensures that the backup was successfully created.</li></ul><h3>Task 7: Upload Backup to S3</h3><pre>def upload_to_s3(**context):<br>    &quot;&quot;&quot;Upload MongoDB backup to S3&quot;&quot;&quot;<br>    s3_hook = S3Hook(aws_conn_id=&#39;&#39;)  # Use your AWS connection ID<br>    <br>    # Get current date for backup folder name<br>    backup_date = datetime.now().strftime(&#39;%Y-%m-%d_%H-%M-%S&#39;)<br>    <br>    # Walk through the backup directory and upload all files<br>    for root, dirs, files in os.walk(BACKUP_PATH):<br>        for file in files:<br>            local_file_path = os.path.join(root, file)<br>            # Create S3 key maintaining directory structure<br>            relative_path = os.path.relpath(local_file_path, BACKUP_PATH)<br>            s3_key = f&quot;{S3_KEY_PREFIX}/{backup_date}/{relative_path}&quot;<br>            <br>            # Upload file to S3<br>            s3_hook.load_file(<br>                filename=local_file_path,<br>                key=s3_key,<br>                bucket_name=S3_BUCKET,<br>                replace=True<br>            )<br>            print(f&quot;Uploaded {local_file_path} to s3://{S3_BUCKET}/{s3_key}&quot;)<br><br>upload_s3_task = PythonOperator(<br>    task_id=&#39;upload_to_s3&#39;,<br>    python_callable=upload_to_s3,<br>    executor_config={&quot;pod_override&quot;: k8s.V1Pod(<br>        spec=k8s.V1PodSpec(<br>            containers=[<br>                k8s.V1Container(<br>                    name=&quot;base&quot;,<br>                    volume_mounts=[volume_mount]<br>                )<br>            ],<br>            volumes=[volume]<br>        )<br>    )},<br>    dag=dag,<br>)</pre><p>📌 <strong>Why?</strong></p><ul><li>Uses S3Hook to upload files to <strong>Amazon S3</strong>.</li><li>Organizes backups using a <strong>timestamped folder structure</strong>.</li></ul><h3>Step 8:- Define Task Dependencies</h3><pre>mongodump_task &gt;&gt; verify_backup_task &gt;&gt; upload_s3_task</pre><p>📌 <strong>Why?</strong></p><ul><li><strong>Step 1:</strong> Run mongodump to take a backup.</li><li><strong>Step 2:</strong> Verify that the backup files exist.</li><li><strong>Step 3:</strong> Upload the backup to <strong>Amazon S3</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3hneEXdNVT24nIiipYl_CA.png" /></figure><h3>Conclusion</h3><p>🚀 <strong>This DAG automates the MongoDB backup process, making it easy to store backups securely in S3.</strong></p><ul><li>✅ Uses <strong>KubernetesPodOperator</strong> for containerized execution.</li><li>✅ Stores backups on a <strong>Persistent Volume (PVC)</strong>.</li><li>✅ Uploads backups to <strong>Amazon S3</strong> using S3Hook.</li></ul><p>🔥 <strong>Now you have a fully automated MongoDB backup pipeline!</strong> 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa61ecdc197d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into Passing Data Between Tasks Using XCom in Apache Airflow]]></title>
            <link>https://medium.com/@MadhavPrajapati/deep-dive-into-passing-data-between-tasks-using-xcom-in-apache-airflow-21d719b71098?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/21d719b71098</guid>
            <category><![CDATA[dağ]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[xcom]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[apache]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Mon, 31 Mar 2025 02:41:30 GMT</pubDate>
            <atom:updated>2025-03-31T02:41:30.633Z</atom:updated>
            <content:encoded><![CDATA[<p>In Apache Airflow, tasks often need to share data. <strong>XComs (Cross-Communication)</strong> are a powerful feature that allows tasks to push and pull data dynamically. Let’s break down how data flows between tasks using <strong>XComs</strong> step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n9OQJWPbHbBmCQHv.png" /></figure><h3>Understanding XComs in Depth</h3><h3>🔹 What is XCom in Airflow?</h3><p>XCom stands for <strong>Cross-Communication</strong> and is used to exchange small amounts of data between tasks in an Airflow DAG.<br>It works like a key-value store, where:</p><ul><li>A task can <strong>push</strong> data into XCom using xcom_push().</li><li>Another task can <strong>pull</strong> that data using xcom_pull().</li></ul><h3>🔹 Where is XCom Data Stored?</h3><p>Airflow stores XCom data in its <strong>metadata database</strong> (e.g., PostgreSQL, MySQL).<br> Every XCom entry consists of:<br> ✅ key → A unique identifier for the stored data<br> ✅ value → The actual data being stored<br> ✅ task_id → The task that pushed the data<br> ✅ dag_id → The DAG where the XCom was created</p><h3>Step-by-Step Breakdown of Task-to-Task Communication Using XComs</h3><p>Let’s analyze the following <strong>Python functions</strong> that push and pull data in an Airflow DAG.</p><pre>import json<br># Function for Task 1 - Push data to XCom<br>def task1(**kwargs):<br>    data = {&quot;message&quot;: &quot;Hello from Task 1!&quot;}  # Step 1: Create a dictionary<br>    kwargs[&#39;ti&#39;].xcom_push(key=&#39;data_key&#39;, value=json.dumps(data))  # Step 2: Convert dictionary to JSON and push to XCom<br># Function for Task 2 - Pull data from XCom<br>def task2(**kwargs):<br>    ti = kwargs[&#39;ti&#39;]  # Step 3: Get the task instance object<br>    received_data = ti.xcom_pull(task_ids=&#39;task1&#39;, key=&#39;data_key&#39;)  # Step 4: Pull data from XCom<br>    data = json.loads(received_data)  # Step 5: Convert JSON back to a dictionary<br>    print(f&quot;Received in Task 2: {data[&#39;message&#39;]}&quot;)  # Step 6: Print the message</pre><h3>Step 1: Creating a Dictionary in Task 1</h3><p>Inside task1, we define a dictionary containing a message:</p><pre>data = {&quot;message&quot;: &quot;Hello from Task 1!&quot;}</pre><p>🔹 This is the <strong>data that Task 1 wants to send</strong> to Task 2.<br> 🔹 Since XCom can only store <strong>serializable</strong> data (JSON, strings, lists, etc.), we later convert it into a JSON string.</p><h3>Step 2: Pushing Data to XCom</h3><p>Next, Task 1 stores this data in XCom using:</p><pre>kwargs[&#39;ti&#39;].xcom_push(key=&#39;data_key&#39;, value=json.dumps(data))</pre><ul><li>kwargs[&#39;ti&#39;]:</li><li>This gets the <strong>task instance (</strong><strong>ti)</strong>, which is used to interact with XComs.</li><li>.xcom_push(key, value):</li><li>key=&#39;data_key&#39; → Acts like a variable name to store the data.</li><li>value=json.dumps(data) → Converts the dictionary into a JSON string before storing it.</li></ul><p>✅ <strong>Now, the data is stored in Airflow’s metadata database.</strong></p><h3>Step 3: Getting the Task Instance in Task 2</h3><p>In task2, the first step is to access the task instance:</p><pre>ti = kwargs[&#39;ti&#39;]</pre><p>🔹 This lets us interact with <strong>XCom functions</strong> in Task 2.</p><h3>Step 4: Pulling Data from XCom</h3><p>Task 2 retrieves the data stored by Task 1:</p><pre>received_data = ti.xcom_pull(task_ids=&#39;task1&#39;, key=&#39;data_key&#39;)</pre><ul><li>task_ids=&#39;task1&#39; → Specifies which task&#39;s XCom data we want.</li><li>key=&#39;data_key&#39; → The same key that was used in xcom_push().</li></ul><p>✅ <strong>Now, Task 2 has retrieved the stored JSON string from XCom.</strong></p><h3>Step 5: Converting JSON String Back to a Dictionary</h3><pre>data = json.loads(received_data)</pre><ul><li>json.loads(received_data) → Converts the JSON string back into a <strong>Python dictionary</strong>.</li></ul><p>Now, data contains:</p><pre>{&quot;message&quot;: &quot;Hello from Task 1!&quot;}</pre><h3>Step 6: Printing the Data</h3><pre>print(f&quot;Received in Task 2: {data[&#39;message&#39;]}&quot;)</pre><p>🔹 Task 2 extracts the &quot;message&quot; from the dictionary and prints:</p><pre>Received in Task 2: Hello from Task 1!</pre><h3>Full DAG Implementation</h3><p>Now, let’s integrate this into a full DAG.</p><pre>from airflow import DAG<br>from airflow.operators.python import PythonOperator<br>from datetime import datetime<br>import json<br># Function for Task 1 - Push data to XCom<br>def task1(**kwargs):<br>    data = {&quot;message&quot;: &quot;Hello from Task 1!&quot;}<br>    kwargs[&#39;ti&#39;].xcom_push(key=&#39;data_key&#39;, value=json.dumps(data))  # Push to XCom<br># Function for Task 2 - Pull data from XCom<br>def task2(**kwargs):<br>    ti = kwargs[&#39;ti&#39;]<br>    received_data = ti.xcom_pull(task_ids=&#39;task1&#39;, key=&#39;data_key&#39;)  # Pull from XCom<br>    data = json.loads(received_data)<br>    print(f&quot;Received in Task 2: {data[&#39;message&#39;]}&quot;)<br># Default DAG arguments<br>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;start_date&#39;: datetime(2024, 3, 30),<br>    &#39;retries&#39;: 1,<br>}<br># Define DAG<br>with DAG(&#39;xcom_example_dag&#39;, default_args=default_args, schedule_interval=&#39;@daily&#39;, catchup=False) as dag:<br><br>    # Task 1 - Push data to XCom<br>    t1 = PythonOperator(<br>        task_id=&#39;task1&#39;,<br>        python_callable=task1,<br>        provide_context=True<br>    )<br>    # Task 2 - Pull data from XCom<br>    t2 = PythonOperator(<br>        task_id=&#39;task2&#39;,<br>        python_callable=task2,<br>        provide_context=True<br>    )<br>    # Task execution order<br> t1 &gt;&gt; t2  # Task 1 runs before Task 2</pre><h3>How to Verify XCom Data in Airflow UI</h3><ol><li><strong>Run the DAG</strong> in Airflow.</li><li>Go to <strong>Admin → XComs</strong> in the Airflow UI.</li><li>You’ll see an entry with:</li></ol><ul><li>key = &quot;data_key&quot;</li><li>value = &#39;{&quot;message&quot;: &quot;Hello from Task 1!&quot;}&#39;</li></ul><ol><li>Task 2 pulls this value and prints it.</li></ol><h3>Key Takeaways</h3><p>✅ <strong>XComs store task outputs for later use.</strong><br> ✅ <strong>Use </strong><strong>xcom_push() to send data</strong> and <strong>xcom_pull() to retrieve it.</strong><br> ✅ <strong>XComs are great for passing small data between tasks.</strong><br> ✅ <strong>Data is stored in Airflow’s metadata database.</strong></p><p>🚀 Now, you can build more complex DAGs with <strong>dynamic task communication using XComs!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21d719b71098" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch Backup and Restore: A Comprehensive Guide]]></title>
            <link>https://medium.com/@MadhavPrajapati/elasticsearch-backup-and-restore-a-comprehensive-guide-8a4d79ed8a48?source=rss-bd161dccc754------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a4d79ed8a48</guid>
            <category><![CDATA[snapshot]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[restore]]></category>
            <category><![CDATA[backup-and-restore]]></category>
            <category><![CDATA[kibana]]></category>
            <dc:creator><![CDATA[Madhav Prajapati]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 06:38:04 GMT</pubDate>
            <atom:updated>2025-02-28T06:38:04.042Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This article details the process of setting up an Elasticsearch cluster in Kubernetes, performing backups to an Amazon S3 bucket, and restoring data from snapshots. The setup includes YAML configurations for deploying Elasticsearch and Kibana, as well as step-by-step instructions for creating and restoring snapshots.</p><h3>1. Setting Up the Elasticsearch Cluster</h3><p>We can manually install the plugin and access key and secret key for AWS authentication for accessing the s3 bucket for storing the backup.</p><blockquote><strong>only for knowladge</strong></blockquote><ul><li><strong>Install the S3 Plugin in Elasticsearch</strong></li></ul><pre>/usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3</pre><ul><li><strong>Add AWS Credentials to Elasticsearch Keystore</strong></li></ul><pre>/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key<br>/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key</pre><p><strong>Restart Elasticsearch again:</strong></p><pre>systemctl restart elasticsearch</pre><blockquote><strong>as we are using the k8s so we are not going to use manual approch</strong></blockquote><h3>Kubernetes YAML Configuration</h3><h3>1.1 Deploying the Headless Service for Discovery</h3><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: elasticsearch-headless<br>  labels:<br>    app: elasticsearch<br>spec:<br>  clusterIP: None<br>  selector:<br>    app: elasticsearch<br>  ports:<br>    - name: transport<br>      port: 9300<br>    - name: http<br>      port: 9200</pre><h3>1.2 Deploying the StatefulSet for Elasticsearch</h3><p>Here we are going to deploy the Elasticsearch cluster with tree node and with minimum_master_nodes 2 for production and with the required configuration</p><pre>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: elasticsearch<br>  labels:<br>    app: elasticsearch<br>spec:<br>  serviceName: elasticsearch-headless<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: elasticsearch<br>  template:<br>    metadata:<br>      labels:<br>        app: elasticsearch<br>    spec:<br>      securityContext:  <br>        fsGroup: 1000<br>      initContainers:<br>      - name: fix-permissions<br>        image: 185889327143.dkr.ecr.ap-southeast-1.amazonaws.com/zl_busybox:latest<br>        command: [&quot;sh&quot;, &quot;-c&quot;, &quot;chown -R 1000:1000 /usr/share/elasticsearch/data&quot;]<br>        volumeMounts:<br>          - name: elasticsearch-data<br>            mountPath: /usr/share/elasticsearch/data<br>      containers:<br>        - name: elasticsearch<br>          image: 185889327143.dkr.ecr.ap-southeast-1.amazonaws.com/zl_elasticsearch:6.5.2<br>          command: [&quot;/bin/bash&quot;, &quot;-c&quot;]<br>          args:<br>            - |<br>              /usr/share/elasticsearch/bin/elasticsearch-keystore create<br>              echo &quot;&lt;access_key&gt;&quot; | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key<br>              echo &quot;&lt;secret_key&gt;&quot; | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key<br>              /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3<br>              exec /usr/local/bin/docker-entrypoint.sh eswrapper<br>          ports:<br>            - containerPort: 9200<br>              name: http<br>            - containerPort: 9300<br>              name: transport<br>          env:<br>            - name: cluster.name<br>              value: &quot;elasticsearch-cluster&quot;<br>            - name: discovery.zen.ping.unicast.hosts<br>              value: &quot;elasticsearch-headless&quot;<br>            - name: discovery.zen.minimum_master_nodes<br>              value: &quot;2&quot;<br>            - name: network.host<br>              value: &quot;0.0.0.0&quot;<br>            - name: bootstrap.memory_lock<br>              value: &quot;false&quot;<br>            - name: ES_JAVA_OPTS<br>              value: &quot;-Xms512m -Xmx512m&quot;<br>          volumeMounts:<br>            - name: elasticsearch-data<br>              mountPath: /usr/share/elasticsearch/data<br>  volumeClaimTemplates:<br>    - metadata:<br>        name: elasticsearch-data<br>      spec:<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        resources:<br>          requests:<br>            storage: 2Gi</pre><h4>Init Container: Fix Permissions</h4><ul><li>Before the main container starts, this <strong>initContainer</strong> runs a chown command to fix permissions on the Elasticsearch <strong>data directory</strong>.</li><li>Mounts the elasticsearch-data volume (persistent storage) to /usr/share/elasticsearch/data.</li></ul><h4>Keystore Setup &amp; Plugin Installation</h4><ul><li>Initializes an <strong>Elasticsearch keystore</strong> for storing sensitive credentials.</li><li>Adds <strong>AWS S3 credentials</strong> (access key &amp; secret key) for <strong>S3 repository snapshots</strong>.</li><li>Installs the <strong>repository-s3 plugin</strong>, allowing Elasticsearch to store snapshots in <strong>AWS S3</strong>.</li><li>Runs the <strong>default Elasticsearch entrypoint</strong>.</li></ul><h4>Environment Variables (Cluster Configuration)</h4><p>if we want to give more configuration then we can provide those configuration in the environment variable</p><ul><li>cluster.name: Names the Elasticsearch cluster as <strong>&quot;elasticsearch-cluster&quot;</strong>.</li><li>discovery.zen.ping.unicast.hosts: Uses elasticsearch-headless service for cluster <strong>discovery</strong>.</li><li>discovery.zen.minimum_master_nodes: 2: Ensures the cluster requires <strong>at least two master-eligible nodes</strong>.</li><li>network.host: 0.0.0.0: Binds Elasticsearch to <strong>all network interfaces</strong>.</li><li>bootstrap.memory_lock: false: Disables memory locking.</li><li>ES_JAVA_OPTS: Sets <strong>JVM heap size</strong> to <strong>512MB</strong>.</li></ul><h4>Ports for Communication</h4><ul><li><strong>9200</strong> → Used for REST API requests (HTTP).</li><li><strong>9300</strong> → Used for internal node-to-node communication (transport protocol).</li></ul><h3>1.3 Deploying Kibana</h3><p>Kibana is used as a <strong>visualization and management</strong> tool for Elasticsearch here we can use all the rest endpoints of Elasticsearch with Dev Tool. It helps:<br>✅ <strong>Monitor cluster health</strong> and node status.<br>✅ <strong>Search and analyze logs</strong> with real-time querying.<br>✅ <strong>View and restore snapshots</strong> from S3 easily.<br>✅ <strong>Troubleshoot issues</strong> with logs and dashboards.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: kibana<br>  labels:<br>    app: kibana<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: kibana<br>  template:<br>    metadata:<br>      labels:<br>        app: kibana<br>    spec:<br>      containers:<br>        - name: kibana<br>          image: docker.elastic.co/kibana/kibana:6.5.2<br>          ports:<br>            - containerPort: 5601<br>          env:<br>            - name: ELASTICSEARCH_HOSTS<br>              value: &quot;http://elasticsearch-headless:9200&quot;</pre><h3>1.4 Deploying the Kibana Service</h3><p>for access to the Kibana UI for managing the Elasticsearch cluster</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: kibana<br>  labels:<br>    app: kibana<br>spec:<br>  selector:<br>    app: kibana<br>  ports:<br>    - protocol: TCP<br>      port: 5601<br>      targetPort: 5601<br>  type: ClusterIP</pre><h3>1.5 Deploying the Kibana ingress for External Access(not require)</h3><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: kibana-ingress<br>  namespace: testns <br>spec:<br>  ingressClassName: external-nginx<br>  rules:<br>  - host: kibana.stage-k8s.test.com<br>    http:<br>      paths:<br>      - backend:<br>          service:<br>            name: kibana<br>            port:<br>              number: 5601<br>        path: /<br>        pathType: Prefix</pre><pre>http://kibana.stage-k8s.test.com</pre><h3>2. Elasticsearch Backup and Restore</h3><p>open the Kibana dashboard in <strong>Dev Tools</strong> and<strong> </strong>run the following command to take a backup</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*burQx4r1IDyUOStzBJJIZg.png" /></figure><h3>2.1 Configuring the S3 Repository</h3><pre>PUT /_snapshot/s3_repository<br>{<br>  &quot;type&quot;: &quot;s3&quot;,<br>  &quot;settings&quot;: {<br>    &quot;bucket&quot;: &quot;test-mongo-buckets&quot;,<br>    &quot;region&quot;: &quot;ap-southeast-1&quot;,<br>    &quot;base_path&quot;: &quot;elasticsearch-backups-k8s&quot;,<br>    &quot;compress&quot;: true<br>  }<br>}</pre><ul><li>/_snapshot/s3_repository → Registers a repository named <strong>s3_repository</strong>.</li><li>&quot;type&quot;: &quot;s3&quot; → Specifies that the repository will store snapshots in an <strong>Amazon S3 bucket</strong>.</li><li>&quot;bucket&quot;:Specifies the <strong>Amazon S3 bucket</strong> where snapshots will be stored. The bucket <strong>must already exist</strong> in your AWS account.</li><li>&quot;region&quot;:Defines the <strong>AWS region</strong> where the S3 bucket is located (<strong>Asia Pacific — Singapore</strong> in this case).</li><li>&quot;base_path&quot;:Specifies a <strong>directory inside the S3 bucket</strong> where snapshots will be stored.</li><li>All backups will be saved under the <strong>elasticsearch-backups-k8s/</strong> path within the bucket.</li><li>&quot;compress&quot;:<strong>Enables compression</strong> to reduce the snapshot size and <strong>optimize storage</strong> in S3.</li></ul><h3>2.2 Taking a Snapshot</h3><pre>PUT /_snapshot/s3_repository/full_backup_24-02-2025?wait_for_completion=true<br>{<br>  &quot;indices&quot;: &quot;*&quot;,<br>  &quot;ignore_unavailable&quot;: true,<br>  &quot;include_global_state&quot;: true<br>}</pre><ul><li>/_snapshot/s3_repository: This specifies the repository where the snapshot will be stored. In this case, it&#39;s an S3 repository named s3_repository.</li><li>full_backup_24–02–2025: This is the name of the snapshot.</li><li>?wait_for_completion=true: This parameter ensures that the request waits for the snapshot operation to complete before returning a response.</li><li>&quot;indices&quot;: &quot;*&quot;: This specifies that all indices in the cluster should be included in the snapshot.</li><li>&quot;ignore_unavailable&quot;: true: If any indices are unavailable (e.g., due to a node being down), this setting allows the snapshot to proceed without them, preventing the entire operation from failing.</li><li>&quot;include_global_state&quot;: true: This includes the cluster&#39;s global state in the snapshot, which covers settings like <strong>cluster settings, node information, index settings, index templates, and shard allocation</strong>. This is useful for restoring the cluster to its exact state if needed.</li></ul><p>Take Reference:- <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/create-snapshot-api.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/create-snapshot-api.html</a></p><h3>Restore Snapshot In Your Second Elasticsearch Cluster</h3><h3>2.1 Configuring the S3 Repository</h3><p>in the second cluster with the same bucket and base_path and the same region</p><pre><br>PUT /_snapshot/s3_repository<br>{<br>  &quot;type&quot;: &quot;s3&quot;,<br>  &quot;settings&quot;: {<br>    &quot;bucket&quot;: &quot;test-mongo-buckets&quot;,<br>    &quot;region&quot;: &quot;ap-southeast-1&quot;,<br>    &quot;base_path&quot;: &quot;elasticsearch-backups-k8s&quot;,<br>    &quot;compress&quot;: true<br>  }<br>}</pre><h3>2.3 Restoring a Snapshot</h3><pre>POST /_snapshot/s3_repository/full_backup_24-02-2025/_restore<br>{<br>   &quot;indices&quot;: &quot;*&quot;,<br>   &quot;ignore_unavailable&quot;: true,<br>   &quot;include_global_state&quot;: true<br>}</pre><p>Reference: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html</a></p><h3>3. Validation Steps</h3><ul><li>Ensure that all indices are restored with correct replicas.</li><li>Restore snapshots in different cluster sizes (1-node, 2-node, and 4-node) to observe distribution behavior.</li><li>Validate templates: all templates are also restore in the new cluster</li><li>Debug any shard allocation issues using the explain API.</li></ul><h3>4. Some Commands that help you</h3><pre>GET _cat/health<br>GET _cat/nodes?v<br>GET /_cat/indices?v<br>GET /_snapshot/my_backup/_all<br>GET _cluster/stats?human&amp;pretty</pre><p><strong><em>If we restore the multi-replica index to one node cluster then we will see that cluster health is yellow or red so for that we have to change the replica size based on the cluster size</em></strong></p><pre>PUT /_all/_settings<br>{<br> “index”: {<br> “number_of_replicas”: 0<br> }<br>}<br># Or Particular index<br><br>PUT /test2/_settings<br>{<br>  “index”: {<br>  “number_of_replicas”: 0<br>  }<br>}</pre><p>Or we can increase and decrease the replica size based on the cluster size</p><h3>Conclusion</h3><p>This guide covers the full Elasticsearch backup and restore process using Kubernetes. By following these steps, you can ensure reliable data recovery and scalability within your Elasticsearch cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a4d79ed8a48" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>