<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harshvijaythakkar on Medium]]></title>
        <description><![CDATA[Stories by Harshvijaythakkar on Medium]]></description>
        <link>https://medium.com/@harshvijaythakkar?source=rss-bcd900396e5b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*jpGOODK1Jw_RvxGkzMgecw.png</url>
            <title>Stories by Harshvijaythakkar on Medium</title>
            <link>https://medium.com/@harshvijaythakkar?source=rss-bcd900396e5b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:02:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harshvijaythakkar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Securely Distribute Millions of S3 Files with CloudFront Signed Cookies + a Single Signed URL]]></title>
            <link>https://aws.plainenglish.io/securely-distribute-millions-of-s3-files-with-cloudfront-signed-cookies-a-single-signed-url-dd91212d6e17?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd91212d6e17</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cdn]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Thu, 01 Jan 2026 11:59:53 GMT</pubDate>
            <atom:updated>2026-03-18T07:40:13.664Z</atom:updated>
            <content:encoded><![CDATA[<h4>Scale secure downloads from S3 without revealing bucket names. Use CloudFront signed cookies + one signed URL to a manifest, with OAC and SSE‑KMS.</h4><h3>Introduction and Problem Statement</h3><h4>Introduction</h4><p>When you need to share <strong>millions of files stored in Amazon S3</strong> with external users, security and scalability become critical challenges. The naive approach — using <strong>S3 presigned URLs</strong> — works for small datasets but quickly becomes impractical for large-scale distribution. Beyond operational complexity, presigned URLs expose your <strong>S3 bucket name</strong>, which can be a security and compliance risk for many organizations.</p><p>In this post, we’ll explore a <strong>production-grade solution</strong> that leverages <strong>Amazon CloudFront signed cookies and signed URLs</strong>, combined with a <strong>manifest file strategy</strong>, to securely distribute large volumes of files without exposing sensitive details. This approach is designed for scenarios where:</p><ul><li>You have <strong>millions of files</strong> in S3.</li><li>The bucket is encrypted using a <strong>customer-managed KMS key</strong>.</li><li>You need <strong>external users</strong> to download files securely.</li><li>You want to avoid exposing <strong>bucket names</strong> or internal paths.</li><li>You need a <strong>single entry point</strong> for users instead of thousands of URLs.</li></ul><h4>Why S3 Presigned URLs Don’t Scale</h4><p>Presigned URLs are great for single-file access, but they fail in large-scale scenarios because:</p><ul><li><strong>Operational Overhead</strong>: Generating millions of URLs is inefficient and error-prone.</li><li><strong>Security Risk</strong>: Each URL exposes the bucket name (bucket.s3.amazonaws.com), which may violate compliance or security policies.</li><li><strong>Poor UX</strong>: Users must manage thousands of links, making the experience cumbersome.</li><li><strong>Limited Control</strong>: Fine-grained access control (like IP restrictions or wildcard access) is hard to implement with presigned URLs.</li></ul><h4>Our Goal</h4><p>We needed a solution that:</p><ul><li>Scales to <strong>millions of files</strong>.</li><li>Keeps the <strong>bucket private</strong> and encrypted with <strong>SSE-KMS</strong>.</li><li>Provides <strong>secure, time-bound access</strong>.</li><li>Offers a <strong>single entry point</strong> for users.</li><li>Works seamlessly with <strong>CloudFront CDN</strong> for performance.</li></ul><h3>Solution Overview and Architecture</h3><h4>Sequence Diagram</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nROfvbz3HDo5z2FNBt_ow.png" /><figcaption>Sequence Diagram for Securely Downloading File using CloudFront</figcaption></figure><h4>Architecture Diagram</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XeiXbsdfma8JGdSkDpUOQg.png" /><figcaption>Securely Download File using CloudFront</figcaption></figure><h4>Design Principles</h4><p>To meet the requirements — <strong>scale</strong>, <strong>security</strong>, <strong>privacy</strong>, and <strong>great user experience</strong> — the solution adheres to the following principles:</p><ul><li><strong>CloudFront-first delivery</strong>: All end‑user traffic flows through CloudFront. S3 is never exposed directly.</li><li><strong>Least privilege at the origin</strong>: S3 bucket access is restricted to CloudFront via Origin Access Control (OAC). Objects remain private and encrypted with a <strong>customer-managed KMS key</strong>.</li><li><strong>Two-layer authorization</strong>: <br><strong>a. Signed cookies</strong> grant access to a <strong>prefix</strong> (wildcard scope) for bulk downloads.<br><strong>b. Signed URL</strong> grants access to the <strong>single entry point</strong>: manifest.json.</li><li><strong>Bootstrap via a manifest</strong>: Users fetch a single metadata file, then their browser/app follows the listed paths — automatically authorized by the signed cookies.</li><li><strong>Short-lived artifacts</strong>: Both cookies and URLs are <strong>time-bound</strong>. All expirations are computed in <strong>UTC</strong> to avoid timezone ambiguity at the edge.</li><li><strong>Operational separation</strong>: The system that assembles the dataset (and produces the raw manifest) is decoupled from the system that signs and publishes artifacts, simplifying security boundaries and auditing.</li></ul><h4>High-Level Components</h4><ul><li><strong>Amazon S3 (Private, SSE‑KMS)</strong><br>Stores the files and the manifest(s). Bucket encryption uses a <strong>customer-managed KMS key</strong>. Public access is blocked. The bucket policy allows <strong>only</strong> CloudFront (via OAC) to read objects.</li><li><strong>Amazon CloudFront (CDN)</strong><br>Serves content over your <strong>custom domain</strong>, hides any S3 details, and enforces <strong>signed URLs/cookies</strong> at the edge. Two relevant cache behaviors:<br><strong>a.</strong> <strong>Files behavior</strong>: Restricted to viewers with <strong>signed cookies</strong>, scoped to your folder/prefix.<br><strong>b.</strong> <strong>Manifest behavior</strong>: Also restricted, but accessed via a <strong>signed URL</strong> for that single file.</li><li><strong>Manifest File Strategy</strong><br>A manifest.json (stored in S3, delivered via CloudFront) contains the list of file paths and any metadata your client needs. It becomes the <strong>single source of truth</strong> the user downloads first.</li><li><strong>Lambda (Publishing &amp; Signing)</strong><br>Triggered when a raw manifest is uploaded. It reads the manifest, generates <strong>signed cookies</strong> for the relevant prefix, writes a <strong>processed manifest</strong> (enriched with cookie metadata and base path), and creates a <strong>signed URL</strong> to that processed manifest. The Lambda then updates your internal system (for example, sending a notification with the single link the user should open).</li><li><strong>Internal Notification/API</strong><br>Receives the signed manifest URL and forwards it to the end user (email, UI, webhook, etc.). End users never see bucket names or S3 URLs — only your <strong>CloudFront domain</strong>.</li><li><strong>Key Group &amp; Private Key Management</strong><br>CloudFront verifies signatures using a <strong>trusted key group</strong> (public keys). Your private key is stored in a secure vault (for example, a secrets manager) and used only by the signing component (Lambda) to generate cookies and URLs.</li></ul><h4>Request Flow</h4><ul><li><strong>Dataset ready → Raw manifest uploaded</strong><br><strong>a.</strong> Your data pipeline assembles a dataset and writes a <strong>raw</strong> manifest.json to S3.</li><li>Lambda trigger → Signing and publishing<br><strong>a.</strong> Reads the raw manifest.<br><strong>b.</strong> Creates <strong>signed cookies</strong> with a <strong>custom policy</strong> for the target prefix (wildcard).<br><strong>c.</strong> Writes a <strong>processed</strong> manifest.json that includes:<br> <strong><em>i.</em></strong> The three CloudFront cookie names/values,<br><strong><em>ii.</em></strong> The cookie <strong>expiry</strong> (UTC epoch),<br><strong><em>iii.</em></strong> A basePath (the CloudFront path prefix to the files),<br><strong><em>iv.</em></strong> Any other metadata required by the client.<br><strong>d.</strong> Generates a <strong>CloudFront signed URL</strong> for the processed manifest (canned policy with expiry).<br><strong>e.</strong> Notifies the internal system with the single <strong>signed URL</strong> to be shared externally.</li><li><strong>User journey<br>a. </strong>The user receives <strong>one link</strong> (CloudFront signed URL to manifest.json).<br><strong>b. </strong>Their browser/app downloads the manifest and applies the <strong>signed cookies</strong> (via your app’s Set‑Cookie or equivalent bootstrap).<br><strong>c. </strong>Subsequent file requests to CloudFront automatically include the cookies and succeed if within scope and time.</li></ul><h4>Trust Boundaries &amp; Security Controls</h4><ul><li><strong>Edge gateway (CloudFront)</strong> is the <strong>enforcement point</strong>: access decisions are made nearest to the user, reducing origin exposure and latency.</li><li><strong>Origin (S3)</strong> is <strong>never public</strong>. Access is limited to CloudFront via <strong>OAC</strong>; direct S3 URLs do not work.</li><li><strong>Key security</strong>:<br><strong>a. </strong>Private key for signing is <strong>never</strong> embedded in code or client-side.<br><strong>b. </strong>Rotation is handled by maintaining multiple public keys in the key group and phasing out old private keys.</li><li><strong>KMS considerations</strong>:<br><strong>a.</strong> The bucket uses a <strong>customer-managed KMS key</strong>.<br><strong>b. </strong>IAM and KMS key policies allow CloudFront (via OAC) to read objects.<br><strong>c. </strong>Application roles that create/modify objects must have kms:Encrypt and appropriate S3 permissions; viewers never interact with KMS.</li></ul><h4>Time &amp; Policy Semantics</h4><ul><li><strong>Signed cookies (custom policy)</strong>:<br><strong>a. </strong>Use a <strong>UTC epoch</strong> in DateLessThan.<br><strong>b. </strong>Optionally add IpAddress conditions (e.g., customer WAN ranges) for tighter security.<br><strong>c. </strong>Scope the Resource to your <strong>CloudFront URL</strong> + /* (wildcard).</li><li><strong>Signed URL (canned policy)</strong>:<br><strong>a. </strong>Use a <strong>UTC datetime</strong> for Expires.<br><strong>b. </strong>Short TTL (minutes) recommended because this URL is only the bootstrap to the session.</li><li><strong>Precedence:<br>a. </strong>Signed URL takes precedence when used for that resource.<br><strong>b. </strong>Cookies apply to <strong>all subsequent requests</strong> under the wildcard prefix.</li></ul><h4>Behavior &amp; Path Strategy</h4><p>Use clear, separate paths to limit overexposure:</p><ul><li><strong>Files prefix</strong> (bulk access via cookies):<br>/data/&lt;tenant&gt;/&lt;request&gt;/*<br>Restrict viewer access = <strong>Yes</strong>, trusted key group = <strong>Enabled</strong>.</li><li><strong>Manifest path</strong> (single signed URL):<br>/manifest/&lt;tenant&gt;/&lt;request&gt;/processed/manifest.json<br>Restrict viewer access = <strong>Yes</strong>. Users receive only <strong>this</strong> CloudFront URL.</li></ul><p>This split ensures the cookie policy grants <strong>just enough</strong> access for the intended dataset, and the manifest URL remains a tight, single resource authorization.</p><h4>ASCII Diagram (End‑to‑End)</h4><pre>                +-----------------------------+<br>                | External User (Browser/App) |<br>                +--------------+--------------+<br>                               |<br>                               | 1) Open signed URL to manifest<br>                               v<br>+-------------------+   +------+------+<br>| Internal System   |   | CloudFront |<br>| (Notification/UI) |--&gt;|  (CDN)     |<br>+---------+---------+   +------+------+<br>          |                   |<br>          | (single link)     | 2) Manifest download (signed URL)<br>          |                   v<br>          |            +------+------+<br>          |            |    S3       |  &lt;--- Private, SSE-KMS<br>          |            |  (Origin)   |       Accessible only by CloudFront (OAC)<br>          |            +------+------+<br>          |                   ^<br>          |                   |<br>          | 4) Files via CF   | 3) Cookies authorize prefix<br>          |    (bulk)         |<br>          v                   |<br>+-------------------+   +-----+-------+   +-----------------------------+<br>| Raw Manifest      |--&gt;|  Lambda     |--&gt;| Processed Manifest (S3)     |<br>| Upload (S3 event) |   | (Signer)    |   | + Cookie info + basePath    |<br>+-------------------+   +-------------+   +-----------------------------+<br><br>Legend:<br>(1) Internal system gives user one CloudFront signed URL to manifest<br>(2) User downloads manifest via CloudFront (no bucket exposure)<br>(3) Manifest includes CloudFront signed cookies for a prefix (wildcard)<br>(4) User downloads millions of files through CloudFront using cookies</pre><h4>Why this architecture works at scale</h4><ul><li><strong>Single entry point</strong> reduces complexity: share one link, not millions.</li><li><strong>Wildcard cookie policy</strong> scales linearly with datasets; the edge enforces authorization.</li><li><strong>Privacy by design</strong>: users only see the <strong>CloudFront domain</strong>; S3 paths and bucket names remain internal.</li><li><strong>Performance</strong>: edge caching and global distribution lower latency and origin load.</li><li><strong>Compliance</strong>: SSE‑KMS at rest, TLS in transit, and strict origin policies via OAC support strong governance.</li></ul><h4>Failure Modes &amp; Resilience</h4><ul><li><strong>Expired artifacts</strong>:<br><strong>a.</strong> Users clicking old links receive clean denials at the edge.<br><strong>b. </strong>Regenerate a fresh signed URL (manifest) and cookies on demand.</li><li><strong>Key rotation</strong>:<br><strong>a. </strong>Maintain at least two active public keys in the key group; deploy new private key, update Lambda, and phase out old keys without downtime.</li><li><strong>Partial dataset or path drift</strong>:<br><strong>a. </strong>The manifest acts as a declarative contract. If a path is missing, only the affected object fails while the rest of the session remains valid.</li><li><strong>Origin overload</strong>:<br><strong>a. </strong>Tune cache behaviors and object TTLs. Consider background warming or tiered caching if needed.</li></ul><h3>Detailed Implementation Decisions</h3><h4>Cookie Policy Design (Prefix Access)</h4><p><strong>Goal:</strong> Grant a viewer temporary, scoped access to <em>many</em> files without issuing a URL per object.</p><ul><li><strong>Resource scope:</strong> Use a <strong>prefix policy</strong> with a wildcard (e.g., /&lt;dataset-prefix&gt;/*). This gives the browser access to all files under that prefix and nothing beyond it.</li><li><strong>Policy type:</strong> Choose a <strong>custom policy</strong> (not canned) so you can include:<br><strong>a. </strong>DateLessThan (absolute UTC expiry),<br><strong>b. </strong>Optional IpAddress restriction (e.g., customer WAN CIDR),<br><strong>c. </strong>Optional DateGreaterThan (valid-after window) if you want to delay availability</li><li><strong>Time semantics:</strong> Always compute times in <strong>UTC</strong>. Convert to <strong>epoch seconds</strong> for cookies.</li><li><strong>Cookie names:</strong> Three values must be set for the viewer: CloudFront-Policy, CloudFront-Signature, and CloudFront-Key-Pair-Id.</li><li><strong>Path scoping:</strong> Prefer Path=/ on cookies unless you have a strong reason to scope further, to ensure the browser sends cookies to all dataset requests under your domain.</li><li><strong>Attributes:</strong> Use Secure (HTTPS only). Use HttpOnly to mitigate XSS cookie theft if downloads are triggered server-side; omit HttpOnly if a pure client-side flow needs programmatic cookie reads. Select SameSite=Lax or Strict according to your UX (Lax is typically sufficient for download flows initiated from your site).</li></ul><p><strong>Trade-offs:</strong></p><ul><li>A broader prefix simplifies user experience but increases blast radius if leaked.</li><li>Narrow prefixes reduce risk but may require per-tenant or per-request cookie issuance.</li></ul><h4>Signed URL Design (Single Entry Point)</h4><p><strong>Goal:</strong> Provide a <strong>single</strong> bootstrap link to the manifest without exposing origin details.</p><ul><li><strong>Object:</strong> Only the <strong>processed</strong> manifest is shared via <strong>signed URL</strong>.</li><li><strong>Policy type:</strong> A <strong>canned policy</strong> is adequate for the manifest — short, expiring access without IP restriction.</li><li><strong>TTL:</strong> Keep <strong>short</strong> (minutes). Users will obtain cookies from the manifest (or your app’s Set‑Cookie bootstrap) and then use them for bulk file access.</li><li><strong>Collision avoidance:</strong> Do not reuse the same URL for long periods; generate per-session or per-request signed URLs to reduce link sharing risk.</li></ul><h4>TTL Strategy</h4><p>Design cookie and URL lifetimes as complementary:</p><ul><li><strong>Signed URL TTL (manifest):</strong> Short — <strong>15 to 60 minutes</strong>. This limits exposure of your entry point if an email or notification is forwarded.</li><li><strong>Signed cookie TTL (prefix):</strong> Session-aligned — <strong>hours to days</strong> depending on business needs, data size, and user behavior.</li><li><strong>Grace margins:</strong> Consider small buffer windows to account for client clock skew. Ensure <strong>edge validation</strong> is based on UTC and your artifacts are all generated with UTC timestamps.</li><li><strong>Revocation model:</strong> If you must revoke mid‑session, issue a new prefix and new cookies, and invalidate the previous manifest to prevent re-bootstrap.</li></ul><h4>SSE‑KMS and IAM Nuances</h4><p><strong>Goal:</strong> Ensure private, encrypted content is retrievable <strong>only</strong> through your CDN.</p><ul><li><strong>Bucket encryption:</strong> Use a <strong>customer-managed KMS key</strong> for the S3 bucket.</li><li><strong>Bucket access:</strong> Block public access. Permit <strong>read</strong> operations <em>only</em> from the CDN via <strong>Origin Access Control (OAC)</strong>.</li><li><strong>KMS permissions:<br>a. </strong>The roles/services that <strong>write</strong> or <strong>modify</strong> objects need kms:Encrypt and appropriate S3 permissions.<br><strong>b. </strong>The CDN’s origin access must be allowed to <strong>decrypt/GET</strong> objects. Align the KMS <strong>key policy</strong> and IAM so that decryption through the CDN path is permitted, but direct access outside the CDN path is not</li><li><strong>Object ownership:</strong> Use the bucket owner‑enforced model unless you require ACLs. Keep ownership consistent to avoid permission surprises.</li></ul><h4>CloudFront Behaviors &amp; Domain Strategy</h4><p><strong>Goal:</strong> Make restrictive access enforceable and observable.</p><ul><li><strong>Custom domain:</strong> Serve content under your domain to avoid ever showing origin identifiers.</li><li><strong>Behaviors:<br>a. Files behavior</strong>: path pattern matching your dataset prefixes; restrict viewer access; trust your key group; tune cache and origin settings.<br><strong>b. Manifest behavior</strong>: path pattern for processed manifest locations; restrict viewer access; also trust your key group; typically minimal cache (or validation) to avoid stale metadata.</li><li><strong>HTTPS enforcement:</strong> Viewer protocol policy should deny or redirect HTTP to HTTPS. Set modern TLS configuration and cipher preferences.</li></ul><h4>Manifest Schema Recommendations</h4><p>Design the manifest to be the <strong>contract</strong> between producer and consumer. Consider:</p><ul><li><strong>Header block: <br>a. </strong>datasetId, tenantId, requestId, generatedAt (UTC), expiresAt (UTC ISO 8601).</li><li><strong>Access block:<br>a. </strong>basePath: the <strong>CDN prefix</strong> for file requests (no origin identifiers),<br><strong>b. </strong>cookie: { CloudFront-Policy, CloudFront-Signature, CloudFront-Key-Pair-Id, expiry_epoch }.</li><li><strong>Files array:<br>a. </strong>Each entry with relativePath (under basePath), sizeBytes, checksum (e.g., SHA‑256), optional content type, and any domain-specific metadata.</li><li><strong>Client hints:<br>a. </strong>Concurrency guidance, retry/backoff parameters, recommended chunk size for range requests if applicable.</li><li><strong>Versioning:<br>a. </strong>Include a manifest schema version to enable non-breaking client upgrades.</li></ul><p><strong>Principle:</strong> Never embed bucket names, regional endpoints, or origin URLs in the manifest. Keep it <strong>CDN-only</strong>.</p><h4>Signing Key Management</h4><p><strong>Goal:</strong> Maintain trust while enabling rotation without downtime.</p><ul><li><strong>Key storage:</strong> Store the <strong>private key</strong> only in a secure secret vault. Do not place it in environment variables or code repositories.</li><li><strong>Key group:</strong> Maintain multiple <strong>public keys</strong> in the CDN key group to support <strong>hot rotation</strong>.</li><li><strong>Rotation plan:<br>a. </strong>Introduce a new key pair; add the public key to the key group.<br><strong>b. </strong>Update the signing Lambda to use the new private key.<br><strong>c. </strong>Reissue cookies/URLs signed with the new key while the old key remains temporarily trusted.<br><strong>d. </strong>Remove the old public key after all active sessions naturally expire.</li></ul><h4>Security Hardening Checklist</h4><ul><li><strong>Strict origin isolation:</strong> Confirm the origin is accessible <strong>only</strong> through the CDN.</li><li><strong>Minimal data leakage:</strong> Audit manifests to ensure no origin identifiers or bucket names appear anywhere.</li><li><strong>Short entry-point TTLs:</strong> Keep signed URL TTL minimal.</li><li><strong>IP restriction (optional):</strong> Add IpAddress condition to cookies for customers with stable corporate ranges.</li><li><strong>TLS posture:</strong> Require modern TLS and strong cipher suites.</li><li><strong>Key hygiene:</strong> Rotate keys periodically; monitor use and access to signing materials.</li><li><strong>Least privilege:</strong> Ensure the signing Lambda’s role has only the permissions it needs (read manifest, write processed manifest, read secrets, send notifications).</li></ul><h3>Implementation Walkthrough</h3><p>This section outlines a practical, step‑by‑step path to stand up the solution from scratch, with emphasis on <strong>security</strong>, <strong>repeatability</strong>, and <strong>operability</strong>. It avoids revealing any origin identifiers and focuses entirely on CloudFront‑facing integration.</p><h4>Create KMS Key</h4><ul><li>Go to KMS page</li><li>Click on Customer-managed keys</li><li><strong>Configure key<br>a. Key Type:</strong> Symmetric<br><strong>b. Key usage:</strong> Encrypt and decrypt<br><strong>c. </strong>click on next</li><li><strong>Add labels<br>a. Alias: </strong>Type Name for Key<br><strong>b. Description:</strong> write descriotion if needed<br><strong>c. Tags: </strong>add tags if needed<br><strong>d. </strong>click on next</li><li><strong>Define key administrative permissions<br>a. Key administrators: </strong>select any IAM user / Role<br><strong>b. </strong>click on next</li><li><strong>Define key usage permissions<br>a. Key users: </strong>select any IAM user / Role<br><strong>b. </strong>click on next</li><li>For Now don’t edit Key policy we will update this later.</li><li>Click on finish</li></ul><h4>Create S3 Bucket</h4><ul><li>Go to S3 console</li><li>Click on Create Bucket</li><li>Give bucket name</li><li>Under <strong>Default encryption<br>a. </strong>Select “Server-side encryption with AWS Key Management Service keys (SSE-KMS)” <br><strong>b.</strong> Enter AWS KMS Key ARN which you created</li><li>Keep other settings as it is and click on Create Bucket</li></ul><h4>Create Key Pair</h4><p>We need to generate public and private key pair which will be used for generating signed url and signed cookie.</p><p>Run following command on your terminal (Linux OS):</p><pre># linux<br>openssl genrsa -out private_key.pem 2048<br>openssl rsa -pubout -in private_key.pem -out public_key.pem</pre><p>Run following command on your terminal (Mac Apple Silicon OS):</p><pre># mac apple silicon<br>openssl genrsa -out private_key.pem -traditional 2048<br>openssl rsa -pubout -in private_key.pem -out public_key.pem</pre><ul><li>Make Sure you keep “private_key.pem” and “public_key.pem” files securly.</li></ul><h4>Create Cloudfront Key Group</h4><ul><li>Go to cloudfront console</li><li>Under Key Management click on Public Keys<br><strong>a. </strong>Click on Create public key<br><strong>b. Name: </strong>Name of public key<br><strong>c. Description: </strong>write descriotion if needed<br><strong>d. Key: </strong>In this you need to copy content of “public_key.pem” generted in above step.</li><li>Click on Create public key</li><li>Note the ID of Public Key, we need to store this in AWS Secrets Manager</li><li>Under Key Management Click on Key groups<br><strong>a. C</strong>lick on Create key group<br><strong>b. Name: </strong>Name of key group<br><strong>c. Description: </strong>write description if needed<br><strong>d. Public keys: </strong>Select the public key</li><li>Click on Create key group</li></ul><h4>Create CloudFront Distribution</h4><ul><li>Go to CloudFront console</li><li>Click On Create Distribution<br><strong>a. </strong>Choose a plan — I am selecting Free<br><strong>b. </strong>Click On next<br><strong>c. Distribution name: </strong>Name of Distribution<br><strong>d. Description: </strong>write description if needed<br><strong>e. Domain:</strong> Write you domain name<br><strong>f. Origin type: </strong>Amazon S3<br><strong>g. Origin: </strong>Select S3 bucket<br><strong>h. Settings: </strong>Keep everything default only change <strong>Cache settings, </strong>Select “Customize cache settings” and in <strong>Cache policy </strong>select “CachingDisabled” policy<br><strong>i.</strong> Click on next<br><strong>j. Enable security</strong> keep everything default and click on next<br><strong>k. Get TLS certificate: </strong>For this ACM certificate should be present the us-east-1 Region that covers the domains you specified<br><strong>l.</strong> Click on next and review all the settings<br><strong>m. </strong>Click on Create distribution</li><li>Once distribution status is “Enabled” we need to edit our distribution:<br><strong>a.</strong> Click on your distribution<br><strong>b. </strong>Go to Behaviors tab and click on behavior and click on “Edit”<br><strong>c. Restrict viewer access: </strong>Click on Yes<strong><br>i. Trusted authorization type : </strong>Select “Trusted key groups (recommended)” <br><strong>ii. Add key groups: </strong>Select Key group<br><strong>d. </strong>Click on Save changes</li></ul><h4>Update KMS Key Policy</h4><ul><li>Go to KMS console</li><li>Click on key which you have used in S3 bucket</li><li>Click Edit in Key Policy and add following statement</li></ul><pre>{<br>      &quot;Sid&quot;: &quot;AllowCloudFrontUseOfTheKey&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;cloudfront.amazonaws.com&quot;<br>      },<br>      &quot;Action&quot;: [<br>        &quot;kms:Decrypt&quot;,<br>        &quot;kms:GenerateDataKey&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;*&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;AWS:SourceArn&quot;: &quot;&lt;cloudfront_distribution_arn&gt;&quot;<br>        }<br>      }<br>    }</pre><ul><li>Click on Save Changes</li></ul><h4>Manifest Schema (Suggested Fields)</h4><p>Design the manifest as a <strong>contract</strong> between publisher and consumer. Below is a template you can adapt:</p><pre><br>{<br>  &quot;schemaVersion&quot;: &quot;1.0&quot;,<br>  &quot;dataset&quot;: {<br>    &quot;tenantId&quot;: &quot;X&quot;,<br>    &quot;requestId&quot;: &quot;Y&quot;,<br>    &quot;datasetId&quot;: &quot;Z&quot;,<br>    &quot;generatedAt&quot;: &quot;&quot;,<br>    &quot;expiresAt&quot;: &quot;2025-12-15T13:34:56Z&quot;<br>  },<br>  &quot;access&quot;: {<br>    &quot;basePath&quot;: &quot;/data/&lt;tenant&gt;/&lt;request&gt;/&quot;,<br>    &quot;cookie&quot;: {}<br>  },<br>  &quot;downloadHints&quot;: {<br>    &quot;recommendedConcurrency&quot;: 8,<br>    &quot;retry&quot;: {<br>      &quot;maxAttempts&quot;: 5,<br>      &quot;backoffMs&quot;: [250, 500, 1000, 2000, 4000]<br>    },<br>    &quot;rangeRequests&quot;: {<br>      &quot;enabled&quot;: true,<br>      &quot;chunkBytes&quot;: 10485760<br>    }<br>  },<br>  &quot;files&quot;: [<br>    &quot;a/b/c/file-001.bin&quot;,<br>    // ... other file entries<br>  ]<br>}</pre><p><strong>Key points:</strong></p><ul><li><strong>No origin identifiers</strong>; only <strong>relative paths</strong> under basePath (CDN path).</li><li>Cookie block holds the <strong>three CloudFront cookie values</strong> plus UTC expiry in <strong>milliseconds</strong>.</li><li>Include <strong>download hints</strong> so clients are consistent and gentle on networks/CDN.</li></ul><h4>IAM Role for Lambda Function</h4><ul><li>Go to IAM console</li><li>Click on policies and click on create policy and add following policy</li></ul><pre>{<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucketMultipartUploads&quot;,<br>                &quot;s3:AbortMultipartUpload&quot;,<br>                &quot;s3:ListBucket&quot;,<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:GetBucketLocation&quot;,<br>                &quot;s3:ListMultipartUploadParts&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::&lt;s3_bucket_name&gt;&quot;,<br>                &quot;arn:aws:s3:::&lt;s3_bucket_name&gt;/*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Action&quot;: [<br>                &quot;kms:ListKeys&quot;,<br>                &quot;kms:Decrypt&quot;,<br>                &quot;kms:GenerateDataKey&quot;,<br>                &quot;kms:DescribeKey&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:kms:us-west-2:&lt;account_id&gt;:key/*&quot;<br>        },<br>        {<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogGroup&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;,<br>                &quot;ec2:CreateNetworkInterface&quot;,<br>                &quot;ec2:DescribeNetworkInterfaces&quot;,<br>                &quot;ec2:DescribeSubnets&quot;,<br>                &quot;ec2:DeleteNetworkInterface&quot;,<br>                &quot;ec2:AssignPrivateIpAddresses&quot;,<br>                &quot;ec2:UnassignPrivateIpAddresses&quot;,<br>                &quot;secretsmanager:GetSecretValue&quot;,<br>                &quot;secretsmanager:DescribeSecret&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ],<br>    &quot;Version&quot;: &quot;2012-10-17&quot;<br>}</pre><ul><li>Click on next and give name to policy and click on create policy</li><li>Click on Roles and click on Create role</li><li>Select “Lambda” in <strong>Service or use case </strong>and Click on next</li><li>Search Policy which you created and Click on next</li><li>Type role name and click on Create role</li></ul><h4>Secret in AWS Secrets Manager</h4><ul><li>Go to AWS Secrets Manager console and click on Store a new secret</li><li>Select other type of secret</li><li>In <strong>Key/value pairs:<br>a. Key: </strong>cf_private_key<br> <strong>Value: </strong>copy the content of “private_key.pem” and paste it<br><strong>b. Key: </strong>cf_public_key_id<br><strong> Value: Public keys ID </strong>created earlier</li><li>Click on next</li><li>Type secret name and description and click on next</li><li>Click on next</li><li>Review and click on Store</li></ul><h4>Lambda Function Code</h4><p>Create folder named cf-secure-download and create file named lambda_function.py</p><pre>cf-secure-download<br>|<br>|_ _ lambda_function.py</pre><p>You can use following code as reference:</p><pre>import json<br>import boto3<br>import datetime<br>import base64<br>import rsa<br>import os<br>import urllib3<br>import logging<br>import time<br>from botocore.exceptions import BotoCoreError, ClientError<br>from botocore.config import Config<br>from urllib.parse import unquote<br>from botocore.signers import CloudFrontSigner<br><br># === CONFIGURATION ===<br>REGION = os.environ.get(&quot;REGION&quot;, &quot;us-west-2&quot;)  # Default region fallback<br>CF_SECRET_NAME = os.environ.get(&quot;CF_SECRET_NAME&quot;)<br>CLOUDFRONT_URL = os.environ.get(&quot;CF_URL&quot;)<br>PRESIGNED_URL_EXPIRY_IN_SECONDS = int(os.environ.get(&quot;PRESIGNED_URL_EXPIRY_IN_SECONDS&quot;, &quot;300&quot;))<br>COOKIE_EXPIRY_IN_DAYS = int(os.environ.get(&quot;COOKIE_EXPIRY_IN_DAYS&quot;, &quot;1&quot;)) <br><br># === INIT AWS CLIENTS ===<br>s3 = boto3.client(&quot;s3&quot;, region_name=REGION, config=Config(signature_version=&quot;s3v4&quot;))<br>secretsmanager = boto3.client(&quot;secretsmanager&quot;, region_name=REGION)<br><br># === SETUP LOGGING ===<br>logger = logging.getLogger()<br>logger.setLevel(logging.INFO)<br><br>def normalize_pem_key(pem_str):<br>    if &quot;-----BEGIN RSA PRIVATE KEY-----&quot; in pem_str and &quot;\n&quot; not in pem_str:<br>        # Remove header and footer<br>        header = &quot;-----BEGIN RSA PRIVATE KEY-----&quot;<br>        footer = &quot;-----END RSA PRIVATE KEY-----&quot;<br>        body = pem_str.replace(header, &quot;&quot;).replace(footer, &quot;&quot;).replace(&quot; &quot;, &quot;&quot;)<br>        # Insert newlines every 64 characters<br>        body_lines = [body[i:i+64] for i in range(0, len(body), 64)]<br>        return f&quot;{header}\n&quot; + &quot;\n&quot;.join(body_lines) + f&quot;\n{footer}&quot;<br>    return pem_str  # Already formatted correctly<br><br>def lambda_handler(event, context):<br>    try:<br>        logger.info(&quot;Lambda triggered with event: %s&quot;, json.dumps(event))<br><br>        # === PARSE S3 EVENT ===<br>        record = event[&quot;Records&quot;][0]<br>        bucket_name = record[&quot;s3&quot;][&quot;bucket&quot;][&quot;name&quot;]<br>        raw_key = record[&quot;s3&quot;][&quot;object&quot;][&quot;key&quot;]<br>        decoded_key = unquote(raw_key)<br>        logger.info(&quot;Processing S3 object: %s from bucket: %s&quot;, decoded_key, bucket_name)<br><br>        # === LOAD PRIVATE KEY FROM SECRETS MANAGER ===<br>        try:<br>            cf_secret_response = secretsmanager.get_secret_value(SecretId=CF_SECRET_NAME)<br>            secret_data = json.loads(cf_secret_response[&quot;SecretString&quot;])<br>            public_key_id = secret_data[&quot;cf_public_key_id&quot;]<br>            # private_key_pem = secret_data[&quot;SECURE_DOWNLOAD_PRIVATE_KEY&quot;]<br>            private_key_pem = secret_data[&quot;cf_private_key&quot;]<br>            normalized_key = normalize_pem_key(private_key_pem)<br>            private_key = rsa.PrivateKey.load_pkcs1(normalized_key.encode(&quot;utf-8&quot;))<br>            logger.info(&quot;Successfully loaded signing key from Secrets Manager&quot;)<br>        except (BotoCoreError, ClientError, KeyError) as e:<br>            logger.error(&quot;Error retrieving secret: %s&quot;, str(e))<br>            raise<br><br>        # === TIMESTAMP SETUP ===<br>        now = datetime.datetime.now(datetime.timezone.utc)<br>        expires_at_dt = now + datetime.timedelta(days=COOKIE_EXPIRY_IN_DAYS)<br>        expires_at_iso = expires_at_dt.isoformat()<br>        expiry_epoch = int(expires_at_dt.timestamp())<br>        # Convert to milliseconds<br>        expiry_epoch_ms = int(expires_at_dt.timestamp() * 1000)<br><br>        # === DOWNLOAD manifest.json ===<br>        try:<br>            response = s3.get_object(Bucket=bucket_name, Key=decoded_key)<br>            manifest = json.loads(response[&quot;Body&quot;].read())<br>            logger.info(&quot;Successfully read manifest.json from S3&quot;)<br>        except (BotoCoreError, ClientError, KeyError) as e:<br>            logger.error(&quot;Error reading manifest.json: %s&quot;, str(e))<br>            raise<br><br>        # === CREATE CUSTOM POLICY ===<br>        resource_path = f&quot;{CLOUDFRONT_URL}/&lt;your_folder_path&gt;/*&quot;<br>        policy = {<br>            &quot;Statement&quot;: [<br>                {<br>                    &quot;Resource&quot;: resource_path,<br>                    &quot;Condition&quot;: {<br>                        &quot;DateLessThan&quot;: {&quot;AWS:EpochTime&quot;: expiry_epoch}<br>                    }<br>                }<br>            ]<br>        }<br>        policy_json = json.dumps(policy).encode(&quot;utf-8&quot;)<br>        signature = rsa.sign(policy_json, private_key, &quot;SHA-1&quot;)<br><br>        encoded_policy = base64.b64encode(policy_json).decode(&quot;utf-8&quot;)<br>        encoded_signature = base64.b64encode(signature).decode(&quot;utf-8&quot;)<br><br>        # === UPDATE manifest.json ===<br>        manifest[&quot;access&quot;][&quot;cookie&quot;] = {<br>            &quot;CloudFront-Policy&quot;: encoded_policy,<br>            &quot;CloudFront-Signature&quot;: encoded_signature,<br>            &quot;CloudFront-Key-Pair-Id&quot;: public_key_id,<br>            &quot;expiry_epoch&quot;: expiry_epoch_ms<br>        }<br><br>        manifest[&quot;dataset&quot;][&quot;expiresAt&quot;] = expires_at_iso<br>        manifest[&quot;access&quot;][&quot;basePath&quot;] = f&quot;{CLOUDFRONT_URL}/&lt;your_folder_path&gt;&quot;<br><br>        updated_manifest = json.dumps(manifest, indent=2)<br>        manifest_key = f&quot;manifest/&lt;your_folder_path&gt;/processed/manifest.json&quot;<br>        <br>        s3.put_object(Bucket=bucket_name, Key=manifest_key, Body=updated_manifest.encode(&quot;utf-8&quot;))<br>        logger.info(&quot;Successfully updated manifest.json with cookie and timestamps&quot;)<br><br><br>        # === Build CloudFront URL for processed manifest and sign it ===<br>        # Ensure the CloudFront distribution serves the &#39;manifest/&#39; prefix.<br>        manifest_cf_url = f&quot;{CLOUDFRONT_URL}/manifest/&lt;your_folder_path&gt;/processed/manifest.json&quot;<br><br>        # CloudFrontSigner requires: key_pair_id and a signer function that returns RSA-SHA1 signature bytes.<br>        def rsa_signer(message: bytes) -&gt; bytes:<br>            return rsa.sign(message, private_key, &quot;SHA-1&quot;)<br>        <br>        cf_signer = CloudFrontSigner(public_key_id, rsa_signer)<br><br>        cf_url_expires = datetime.datetime.fromtimestamp(int(time.time()) + PRESIGNED_URL_EXPIRY_IN_SECONDS, tz=datetime.timezone.utc)<br>        cf_signed_url = cf_signer.generate_presigned_url(https://medium.com/url=manifest_cf_url,%20date_less_than=cf_url_expires)<br><br>        logger.info(&quot;Generated CloudFront signed URL for processed manifest.json&quot;)<br>        logger.info(&quot;CloudFront Signed URL: %s&quot;, cf_signed_url)<br><br>        # ==== You can make your internal API call if required and store cf_signed_url in database ====<br></pre><p>Now we need to package our lambda function code into zip along with all the required dependencies.</p><pre># make sure you are in &quot;cf-secure-download&quot; dirctry<br><br># Install required dependencies in &quot;cf-secure-download&quot; folder<br>pip3 install rsa urllib3 -t .<br><br># zip the files<br>zip -r cf-secure-download.zip .</pre><h4>Create Lambda Function</h4><ul><li>Go to Lambda console</li><li>Click on Function and click on Create function</li><li>Give name to function</li><li>Runtime: Python3.14</li><li>In <strong>Change default execution role: </strong>Select Use an exising role and select the role which we created</li><li>Click on Create function</li><li>Click on Code tab and click on Upload from dropdown and select “.zip file” and upload a zip file which we created</li><li>Click on Add trigger</li><li>Trigger Configuration:<br><strong>a.</strong> Select S3<br><strong>b.</strong> Select your bucket<br><strong>c.</strong> Event Types: Only select “PUT”<br><strong>d. </strong>Prefix: Enter “manifest/” (Folder name where manifest file will be uploaded)<br><strong>e. </strong>Suffix: raw/manifest.json (We will be reading manifest file from raw folder)<br><strong>f. </strong>Click on Add</li><li>Click on Configuration and click on Environment variables and add following environment variables<br><strong>a. </strong>REGION: &lt;your-s3-bucket-region&gt;<br><strong>b. </strong>CF_SECRET_NAME: &lt;secret-name wich we created earlier&gt;<br><strong>c. </strong>CF_URL: https://&lt;your-cloud-front-domain&gt;<br>You can add other environment variables as required<br><strong>d.</strong> Click on save</li></ul><h3>Performance Tuning, Cost Optimization, and Security Enhancements</h3><h4>Caching Strategy (Edge and Origin)</h4><p><strong>Objective:</strong> Reduce origin load and bandwidth while ensuring correct, fresh content for users.</p><ul><li><strong>Immutable content rule:</strong> If file payloads under the dataset prefix are immutable once published, set aggressive cache TTLs on those objects. This dramatically improves edge hit ratios and cuts both latency and cost. Couple this with object versioning at the producer (e.g., publish content under unique release folders) to avoid cache‑staleness concerns.</li><li><strong>Manifest vs. payload split:</strong> Treat manifest.json differently from the payload files. The manifest is a control plane artifact—keep its effective TTL low (or “validate on access”) so updates propagate quickly. Files can remain highly cacheable.</li><li><strong>Edge locality:</strong> Large, regionally concentrated audiences benefit from higher TTLs and fewer invalidations. If your audience is geographically diverse, consider moderate TTLs to balance freshness across regions.</li><li><strong>Invalidations sparingly:</strong> Invalidate only when strictly necessary (e.g., manifest corrections or critical hotfixes). Prefer versioned paths over invalidating large prefixes — invalidations are operationally noisy and can be costly at scale.</li></ul><h4>Cookie Scope, Partitioning, and Multi‑Tenant Isolation</h4><p><strong>Objective:</strong> Ensure each user sees only what they are entitled to, at minimum blast radius.</p><ul><li><strong>Strict prefix partitioning:</strong> Issue cookies per tenant/request‑specific prefix rather than broad, cross‑tenant scopes. Partitioning limits the consequence of cookie leaks and simplifies audit trails.</li><li><strong>Folder hygiene:</strong> Keep directory boundaries clean and predictable. Avoid shared folders that mix public and private content; any overlap increases policy complexity and accidental exposure risk.</li></ul><h4>Cost Optimization</h4><p><strong>Objective:</strong> Reduce data transfer, origin egress, and operational overhead.</p><ul><li><strong>Edge cache hit ratio is king:</strong> Everything that increases edge hit ratio reduces origin egress and cost. Immutable payloads, correct cache headers, and avoiding frequent invalidations are your primary levers.</li><li><strong>Right‑sizing behaviors:</strong> Split behaviors thoughtfully (payload vs. manifest, static vs. dynamic) so you can attach precise cache policies. Over‑broad behaviors can force conservative settings that hurt efficiency.</li><li><strong>Compression:</strong> If your payloads are text‑based (manifests, metadata), ensure compression is enabled. For binary payloads, compression may yield no benefit or even be counterproductive — measure before enabling.</li><li><strong>Object granularity:</strong> Extremely small files incur proportionally more overhead per request. Where sensible, bundle tiny assets to reduce request count (but don’t break resumability by producing giant monoliths).</li><li><strong>Range request economics:</strong> Chunking helps resume and stability but increases per‑request overhead. Choose chunk sizes that strike a cost/performance balance.</li></ul><h4>Security Enhancements Beyond the Baseline</h4><p><strong>Objective:</strong> Layer defenses to meet stricter threat models without compromising usability.</p><ul><li><strong>IP scoping:</strong> Add an IP range condition to cookie policies for customers with stable, known egress ranges. This materially reduces the utility of leaked cookies outside the customer’s network.</li><li><strong>Strict TLS posture:</strong> Enforce modern TLS and minimum protocol versions. Strong cipher suites help reduce downgrade risks.</li><li><strong>Key hygiene:</strong> Practice routine signing‑key rotation. Keep multiple public keys in the trust set and rotate private keys without downtime. Audit all access to signing materials.</li></ul><h4>Putting It All Together</h4><p>The performance of this architecture hinges on three pillars:</p><ol><li><strong>Edge‑centric delivery</strong> with disciplined cache policies and minimal invalidations.</li><li><strong>Session‑aware access control</strong>, using short‑lived signed URLs for bootstrap and carefully scoped, longer‑lived signed cookies for bulk data.</li><li><strong>Operational excellence</strong> in signing key management, logging, and incident playbooks.</li></ol><p>Done right, you’ll serve massive catalogs securely, privately, and efficiently — without ever exposing your S3 bucket or internal pathing, and with a user experience that scales cleanly from thousands to millions of files</p><h3>FAQ and Troubleshooting</h3><h4>General Access &amp; Policy Questions</h4><p><strong>Q: Why did we choose signed cookies for files and a signed URL for the manifest?</strong><br><strong>A:</strong> Signed cookies are ideal for granting access to <strong>many</strong> objects under a <strong>single prefix</strong> (wildcard scope) without minting individual URLs. The manifest is a <strong>single resource</strong> and makes sense as a short‑lived <strong>signed URL</strong>. This split gives you a simple bootstrap link and a robust, scalable authorization model for bulk downloads.</p><p><strong>Q: Can we use only signed URLs for everything?</strong><br><strong>A:</strong> Technically yes, but it’s inefficient at scale. You’d be generating and distributing a URL for <strong>each</strong> file. Signed cookies cover entire prefixes, reducing operational overhead and improving UX.</p><p><strong>Q: Do signed URLs conflict with signed cookies?</strong><br><strong>A:</strong> Signed URLs take precedence <strong>for the specific resource</strong> they authorize. Cookies apply to <strong>subsequent requests</strong> under their wildcard scope. Using both is expected: URL for the manifest; cookies for files.</p><h4>Time, TTLs, and Expiry</h4><p><strong>Q: How should we set cookie and URL TTLs?</strong><br><strong>A:</strong></p><ul><li><strong>Manifest signed URL:</strong> short TTL (e.g., 15–60 minutes).</li><li><strong>Cookie policy:</strong> session‑aligned TTL (hours or days) to cover long downloads.<br> Compute times in <strong>UTC</strong> (epoch seconds for cookies, UTC datetime for signed URLs). Avoid overly generous buffers; keep strict control.</li></ul><p><strong>Q: Users report “expired” errors even though the link looks fresh — what’s going on?</strong><br><strong>A:</strong> Common causes include <strong>client clock skew</strong>, the link being <strong>cached</strong> in email previews, or the manifest being opened <strong>after</strong> TTL. Add a small grace window to cookie TTL, reduce URL TTL, and advise users to open the link directly in a browser instead of preview panes.</p><h4>403/Access Denied &amp; Edge Failures</h4><p><strong>Q: We get 403 Access Denied for files under the prefix — how to debug?</strong><br><strong>A:</strong></p><ol><li><strong>Check cookie presence</strong>: Ensure all three cookies are present (CloudFront-Policy, CloudFront-Signature, CloudFront-Key-Pair-Id).</li><li><strong>Verify scope</strong>: Confirm the Resource in your policy matches the <strong>exact CDN domain + path</strong> and includes the <strong>correct wildcard</strong>.</li><li><strong>Key group alignment</strong>: Make sure your Key-Pair-Id matches a <strong>trusted public key</strong> in the distribution.</li><li><strong>Behavior restrictions</strong>: The cache behavior serving those paths must have <strong>Restrict viewer access</strong> enabled and trust your key group.</li><li><strong>IP restriction</strong>: If you used IpAddress, confirm the viewer’s source IP is inside the allowed range.</li></ol><p><strong>Q: A file inside the prefix returns 404; others work.</strong><br><strong>A:</strong> Verify the file actually exists at the origin and the <strong>relative path</strong> in the manifest matches the CDN path. If the file is missing or the path is incorrect, you’ll get 404 even with valid cookies. The manifest enables partial success — download what’s available and report the missing entries.</p><p><strong>Q: Manifest URL opens but the file downloads still fail.</strong><br><strong>A:</strong> The manifest’s signed URL grants access only to the manifest. Ensure your client sets the cookies <strong>before</strong> attempting file requests. If you rely on a Set‑Cookie bootstrap, confirm it completes successfully and the cookies are scoped to your domain and path.</p><h4>OAC, S3, and SSE‑KMS</h4><p><strong>Q: Direct S3 links work internally, but fail externally — even with cookies. Why?</strong><br><strong>A:</strong> That’s by design. The bucket is private and should be accessible <strong>only through the CDN</strong> with <strong>Origin Access Control</strong>. External users must use the CDN domain; direct S3 URLs should be blocked.</p><p><strong>Q: We get decryption or permission errors on KMS‑encrypted objects.</strong><br><strong>A:</strong> Validate the <strong>KMS key policy</strong> and IAM. The CDN’s origin access must be allowed to <strong>decrypt</strong> through the origin path. Also ensure objects are owned appropriately (bucket owner enforced is typically recommended) and that producers who write content have kms:Encrypt plus the necessary S3 permissions.</p><h4>Cookie Mechanics &amp; Client Behavior</h4><p><strong>Q: What are the exact cookies required?</strong><br><strong>A:</strong></p><ul><li>CloudFront-Policy</li><li>CloudFront-Signature</li><li>CloudFront-Key-Pair-Id<br> All three must be present. Attributes typically include Secure; consider HttpOnly if set server‑side, and SameSite=Lax unless your UX requires otherwise.</li></ul><p><strong>Q: Can we scope cookies more tightly than </strong><strong>Path=/?</strong><br><strong>A:</strong> Yes, but be cautious. Over‑scoping can cause the browser <strong>not</strong> to send cookies for some file requests under your prefix. Use Path=/ unless there’s a strong need to narrow it.</p><h4>Manifest Content &amp; Integrity</h4><p><strong>Q: What should the manifest include — at minimum?</strong><br><strong>A:</strong></p><ul><li>Identifiers (tenantId, requestId, dataset references).</li><li>Timestamps (generatedAt, expiresAt in UTC).</li><li>basePath under the CDN domain (no origin identifiers).</li><li>The cookie block (three values + UTC expiry in ms).</li><li>files[] with relativePath, sizeBytes, and checksum.</li><li>Optional client hints (concurrency, backoff, chunk size).</li></ul><p><strong>Q: Can we change the manifest after publishing?</strong><br><strong>A:</strong> Prefer <strong>versioned</strong> manifests if updates are required. Keep TTL low for manifests so clients pick up changes quickly. Avoid invalidating large prefixes; publish a new processed manifest path when practical.</p><h4>Security &amp; Privacy</h4><p><strong>Q: How do we prevent bucket name exposure?</strong><br><strong>A:</strong> Never share S3 URLs. Use only your <strong>custom CDN domain</strong> in manifests, cookies, and user‑visible content. Ensure origin access is <strong>restricted</strong> to the CDN and bucket public access is <strong>blocked</strong>.</p><p><strong>Q: What about users forwarding links or cookies?</strong><br><strong>A:</strong></p><ul><li>Keep the <strong>manifest URL TTL</strong> short.</li><li>Consider <strong>IP restrictions</strong> in the cookie policy where appropriate.</li><li>For highly sensitive datasets, issue <strong>ephemeral prefixes</strong> per session so cookies aren’t reusable across sessions.</li></ul><h4>Frequently Seen Misconfigurations</h4><ul><li>Cookies set with the wrong <strong>domain</strong> or <strong>path</strong> → not sent on file requests.</li><li>Key-Pair-Id not in the <strong>trusted key group</strong> attached to the behavior.</li><li>Policy Resource missing the <strong>full CDN URL</strong> or incorrect <strong>wildcard</strong>.</li><li>Cache behavior not set to <strong>Restrict viewer access</strong> while expecting signatures to be enforced.</li><li>Attempting to use <strong>direct S3 links</strong> in clients (will fail by design).</li><li>Manifest contains <strong>origin identifiers</strong> (bucket names or regions) instead of CDN paths.</li></ul><h3>Conclusion and Key Takeaways</h3><h4>What We Achieved</h4><p>This design delivers a secure, scalable, and user‑friendly way to distribute <strong>millions of files</strong> without revealing any origin details. By separating <strong>bootstrap</strong> (a single signed URL to manifest.json) from <strong>bulk access</strong> (signed cookies for a wildcard prefix), we:</p><ul><li>Eliminated the need to mint one URL per file.</li><li>Prevented <strong>bucket name</strong> and <strong>origin path</strong> exposure entirely.</li><li>Preserved strong security with <strong>SSE‑KMS at rest</strong> and <strong>TLS in transit</strong>.</li><li>Moved authorization decisions to the <strong>edge</strong>, improving performance and reducing origin risk.</li></ul><h4>Core Principles to Remember</h4><ol><li><strong>CloudFront-first delivery</strong><br> All user traffic goes through the CDN. The origin remains private, reachable only via <strong>Origin Access Control</strong>.</li><li><strong>Two-layer authorization<br>i. Signed URL</strong>: short‑lived, single-object access for the manifest.<br><strong>ii. Signed cookies</strong>: longer‑lived, prefix‑scoped access for bulk files.</li><li><strong>Privacy by design</strong><br> Users see only the <strong>custom domain</strong> and CDN paths. No bucket names or regional endpoints appear in any user‑visible artifact.</li><li><strong>Time semantics and discipline</strong><br> <strong>UTC</strong> everywhere. Short TTLs for entry points; session‑appropriate TTLs for cookies. Prefer <strong>re‑bootstrap</strong> over extending long-lived tokens.</li><li><strong>Operational excellence</strong><br> Strong <strong>key hygiene</strong> (rotation without downtime), minimal IAM, structured logging, and tested incident playbooks ensure reliability at scale.</li></ol><h4>Key Architectural Benefits</h4><ul><li><strong>Scale</strong>: Wildcard cookie policies authorize entire datasets without per‑file work.</li><li><strong>Security</strong>: Strict origin isolation, KMS encryption, and edge enforcement minimize exposure.</li><li><strong>Performance</strong>: Edge caching, range requests, and tuned behaviors keep downloads fast and resilient.</li><li><strong>Maintainability</strong>: Manifest acts as a declarative contract, supporting audits, partial success, and schema evolution.</li></ul><h4>Final Thoughts</h4><p>The pattern of <strong>CloudFront signed cookies + a single signed URL to a manifest</strong> is simple, powerful, and battle‑tested. It gives teams a repeatable way to serve massive catalogs securely, privately, and efficiently — whether you’re handling archival data dumps, media libraries, or multi‑tenant exports. The result is a system that <strong>scales gracefully</strong>, <strong>respects compliance</strong>, and <strong>delights users</strong> with a straightforward download experience.</p><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd91212d6e17" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/securely-distribute-millions-of-s3-files-with-cloudfront-signed-cookies-a-single-signed-url-dd91212d6e17">Securely Distribute Millions of S3 Files with CloudFront Signed Cookies + a Single Signed URL</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing OKTA SSO in Apache Superset]]></title>
            <link>https://aws.plainenglish.io/implementing-okta-sso-in-apache-superset-848f44e1f4da?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/848f44e1f4da</guid>
            <category><![CDATA[sso]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[apache-superset]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Fri, 03 Oct 2025 11:44:26 GMT</pubDate>
            <atom:updated>2026-03-18T07:41:31.423Z</atom:updated>
            <content:encoded><![CDATA[<h4>Step-by-step instructions for enabling secure authentication in Superset using OKTA</h4><h3>🧭 <strong>Introduction</strong></h3><p>Apache Superset is a powerful open-source data visualization and exploration platform that’s increasingly used in modern data stacks. In my <a href="https://medium.com/aws-in-plain-english/installing-apache-superset-in-kubernetes-1aec192ac495">previous blog post</a>, I walked through how to set up a production-grade Superset deployment on Kubernetes. That setup used basic username/password authentication, which works well for internal use but doesn’t scale securely for larger teams or organizations.</p><p>In this post, I’ll walk you through how to integrate <strong>OKTA Single Sign-On (SSO)</strong> with Superset using OAuth2. This guide is based on a real-world implementation and is designed to be generic so that anyone using Superset — whether on Kubernetes or another platform — can follow along.</p><p>We’ll cover everything from setting up a test Superset instance, configuring OKTA, handling secure login and logout flows, and planning a smooth migration from password-based authentication to SSO. By the end of this guide, you’ll have a secure, scalable authentication setup for Superset that aligns with modern identity management practices.</p><h3>🧰 <strong>Prerequisites</strong></h3><p>Before diving into the integration steps, make sure you have the following components in place:</p><h4>1. OKTA Developer Account</h4><p>You’ll need access to an OKTA account with permissions to create and configure applications. A free OKTA Developer account works perfectly for testing and prototyping.</p><h4>2. OAuth2 Knowledge (Basic)</h4><p>While you don’t need deep expertise, understanding the basics of OAuth2 — especially concepts like client_id, client_secret, redirect_uri, and id_token — will help you follow along more easily.</p><h4>3. Ingress Controller (if on Kubernetes)</h4><p>If you’re using Kubernetes, ensure your Superset instance is exposed via an ingress controller (e.g., NGINX). You’ll need to configure it to forward headers like X-Forwarded-Host and X-Forwarded-Proto for Superset to correctly handle redirects and user sessions.</p><h4>4. Superset Configuration Access</h4><p>You should be able to modify superset_config.py, which is Superset’s main configuration file. This is where you’ll define the OAuth settings and customize the security manager.</p><h3>🏗️ Architecture Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRiC19hcj-862LfxJZRMTg.jpeg" /></figure><h3>🔀 Sequence Diagram Login</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTGEdd2AoRe_U4zRLHlIaQ.png" /><figcaption>Apache Superset with OKTA SSO Login Sequence Diagram</figcaption></figure><h3>🔀 Sequence Diagram Logout</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*w5_ilTLEp7VqQWTI-KsTCQ.png" /><figcaption>Apache Superset with OKTA SSO Logout Sequence Diagram</figcaption></figure><h3>🧪 Step 1: Setting Up a Test Superset Instance</h3><p>Before making any changes to your production Superset deployment, it’s highly recommended to set up a <strong>test instance</strong>. This allows you to experiment with SSO integration safely, without affecting existing users or dashboards.</p><p>In my case, I deployed a separate Superset instance in a non-production environment using the default username/password authentication. This gave me a clean baseline to validate the SSO flow, debug issues, and iterate on configuration changes.</p><p>Here’s what I did:</p><h4>✅ Why Use a Test Instance?</h4><ul><li>Avoids disruption to active users.</li><li>Allows isolated testing of login/logout flows.</li><li>Helps validate user mapping strategies before rollout.</li></ul><h4>⚙️ How to Set It Up</h4><ul><li>Deploy Superset using Helm or Docker in a staging namespace/environment.</li><li>Create test database and redis cluster</li><li>Configure basic authentication (username/password) to verify initial access.</li><li>Ensure the instance is exposed via ingress or a public endpoint for OKTA to redirect back after authentication.</li></ul><p>Once the test instance was up and running, I moved on to configuring OKTA and integrating it with Superset using OAuth2.</p><h3>🔐 Step 2: Configuring OKTA</h3><p>To enable Single Sign-On (SSO) in Superset, you’ll need to configure an application in OKTA that supports the OAuth2 protocol. This application acts as the identity provider (IdP) and handles authentication and token issuance.</p><h4>🛠️ Create an OKTA OIDC Application</h4><ul><li><strong>Log in to your OKTA dashboard</strong>.</li><li>Navigate to <strong>Applications → Create App Integration</strong>.</li><li>Choose <strong>OIDC — OpenID Connect</strong> and select <strong>Web Application</strong>.</li><li>Click <strong>Next</strong>.</li></ul><h4>🔧 Configure Application Settings</h4><ul><li><strong>Sign-in redirect URIs</strong>:<br> Add the Superset callback URL, typically:</li></ul><pre>https://&lt;your-superset-domain&gt;/oauth-authorized/okta</pre><ul><li><strong>Sign-out redirect URIs</strong>:<br> Add the Superset logout URL, e.g.:</li></ul><pre>https://&lt;your-superset-domain&gt;/logout/</pre><ul><li><strong>Assignments</strong>:<br> Assign the app to users or groups who should have access.</li></ul><p>Once the app is created, note down the following values:</p><ul><li><strong>Client ID</strong></li><li><strong>Client Secret</strong></li><li><strong>Issuer URL</strong> (e.g., <a href="https://dev-xxxx.okta.com/oauth2/default)">https://dev-xxxx.okta.com/oauth2/default)</a></li></ul><p>These will be used in Superset’s configuration file.</p><h4>🧩 Optional: Enable Front-Channel Logout</h4><p>To ensure complete session termination, enable <strong>Front-Channel Logout</strong> in the OKTA app settings:</p><ul><li>Check the box to <strong>Include user session details</strong>.</li><li>Add Superset’s logout endpoint as the <strong>logout redirect URI</strong>.</li></ul><p>This ensures that when a user logs out from Superset, their OKTA session is also terminated.</p><h3>⚙️ Step 3: Updating Superset Configuration</h3><p>Once your OKTA application is ready, the next step is to configure Superset to use it for authentication. Superset supports OAuth2 providers like OKTA through its extensible security framework, which can be customized via superset_config.py . Add following configs in your existing file.</p><p><strong>Note:</strong> This configuration file should be tailored to your specific use case. During the Helm chart installation, we will pass the necessary values to ensure proper integration.</p><pre>import os<br>import requests<br>from flask import session, redirect<br>from flask_appbuilder.security.manager import AUTH_OAUTH<br>from superset.security import SupersetSecurityManager<br>from flask_appbuilder.security.views import AuthOAuthView<br>from flask_appbuilder.security.views import expose<br>import logging<br><br>logger = logging.getLogger(&quot;superset.auth&quot;)<br><br>def env(key, default=None):<br>    return os.getenv(key, default)<br><br><br># Inform superset to use OAUTH for user login<br>AUTH_TYPE = AUTH_OAUTH<br><br># To securely terminate sessions without exposing sensitive tokens, override the logout view:<br>class CustomAuthOAuthView(AuthOAuthView):<br>    @expose(&quot;/logout/&quot;, methods=[&quot;GET&quot;])<br>    def logout(self):<br>        logger.info(&quot;Custom logout triggered from CustomAuthOAuthView&quot;)<br>        id_token = session.get(&quot;okta_id_token&quot;)<br>        session.clear()<br><br>        logout_redirect_uri = env(&quot;OKTA_LOGOUT_REDIRECT_URI&quot;, &quot;/login/&quot;)<br>        end_session_endpoint = env(&quot;OKTA_END_SESSION_ENDPOINT&quot;)<br><br>        # Backend logout: server-side call to Okta<br>        if id_token and end_session_endpoint:<br>            backend_logout_url = (<br>                f&quot;{end_session_endpoint}&quot;<br>                f&quot;?id_token_hint={id_token}&quot;<br>                f&quot;&amp;post_logout_redirect_uri={logout_redirect_uri}&quot;<br>            )<br>            try:<br>                response = requests.get(backend_logout_url)<br>                logger.info(f&quot;Backend logout to Okta: {response.status_code}&quot;)<br>            except Exception as e:<br>                logger.error(f&quot;Backend logout failed: {e}&quot;)<br><br>        # Frontend redirect (no token exposed)<br>        return redirect(logout_redirect_uri)<br><br><br># To ensure existing users are mapped correctly and avoid duplicate accounts, override the SupersetSecurityManager:<br>class CustomSecurityManager(SupersetSecurityManager):<br>    # Map oauth_user_info from OKTA response into superset expected fields<br>    def oauth_user_info(self, provider, response=None):<br>        logger.debug(&quot;In oauth_user_info method&quot;)<br>        if provider == &#39;okta&#39;:<br>            session[&#39;okta_access_token&#39;] = response.get(&#39;access_token&#39;)<br>            session[&#39;okta_id_token&#39;] = response.get(&#39;id_token&#39;)<br><br>            provider_config = next((p for p in OAUTH_PROVIDERS if p[&#39;name&#39;] == provider), None)<br>            userinfo_url = provider_config[&#39;remote_app&#39;].get(&#39;userinfo_url&#39;)<br><br>            if not userinfo_url:<br>                logger.error(&quot;userinfo_url not defined in OAUTH_PROVIDERS&quot;)<br>                raise ValueError(&quot;userinfo_url not defined in OAUTH_PROVIDERS&quot;)<br><br>            headers = {&#39;Authorization&#39;: f&#39;Bearer {response[&quot;access_token&quot;]}&#39;}<br>            resp = requests.get(userinfo_url, headers=headers)<br>            resp.raise_for_status()<br>            user_data = resp.json()<br><br>            email = user_data.get(&#39;email&#39;)<br>            existing_user = self.find_user(email=email) # check is user already exists in DB<br><br>            if existing_user:<br>                logger.info(f&quot;SSO login for existing user: {existing_user.username} ({email})&quot;)<br>                return {<br>                    &#39;username&#39;: existing_user.username,<br>                    &#39;email&#39;: existing_user.email,<br>                    &#39;first_name&#39;: existing_user.first_name,<br>                    &#39;last_name&#39;: existing_user.last_name,<br>                    &#39;name&#39;: f&quot;{existing_user.first_name} {existing_user.last_name}&quot;,<br>                }<br><br>            logger.info(f&quot;Creating new user via SSO: {email}&quot;)<br>            return {<br>                &#39;username&#39;: user_data.get(&#39;preferred_username&#39;) or email,<br>                &#39;email&#39;: email,<br>                &#39;first_name&#39;: user_data.get(&#39;given_name&#39;),<br>                &#39;last_name&#39;: user_data.get(&#39;family_name&#39;),<br>                &#39;name&#39;: user_data.get(&#39;name&#39;),<br>            }<br><br>    # Optional: Sync superset DB user&#39;s info with OKTA&#39;s response.<br>    def update_user_auth_stat(self, user, login=True):<br>        logger.debug(f&quot;update_user_auth_stat called for user: {user.username}, login={login}&quot;)<br><br>        # Only run SSO-related logic if access token is present<br>        access_token = session.get(&#39;okta_access_token&#39;)<br>        if not access_token:<br>            logger.debug(&quot;No access token found in session — skipping SSO update logic.&quot;)<br>            return<br><br>        provider_config = next((p for p in OAUTH_PROVIDERS if p[&#39;name&#39;] == &#39;okta&#39;), None)<br>        userinfo_url = provider_config[&#39;remote_app&#39;].get(&#39;userinfo_url&#39;)<br><br>        if access_token and userinfo_url:<br>            headers = {&#39;Authorization&#39;: f&#39;Bearer {access_token}&#39;}<br>            resp = requests.get(userinfo_url, headers=headers)<br>            if resp.ok:<br>                user_data = resp.json()<br>                updated = False<br><br>                if user.first_name != user_data.get(&#39;given_name&#39;):<br>                    logger.info(f&quot;Updating first_name for {user.username}: {user.first_name} → {user_data.get(&#39;given_name&#39;)}&quot;)<br>                    user.first_name = user_data.get(&#39;given_name&#39;)<br>                    updated = True<br>                if user.last_name != user_data.get(&#39;family_name&#39;):<br>                    logger.info(f&quot;Updating last_name for {user.username}: {user.last_name} → {user_data.get(&#39;family_name&#39;)}&quot;)<br>                    user.last_name = user_data.get(&#39;family_name&#39;)<br>                    updated = True<br><br>                if updated:<br>                    self.get_session.commit()<br>                    logger.info(f&quot;User profile updated for {user.username}&quot;)<br><br>    # Override logout method with custom view<br>    authoauthview = CustomAuthOAuthView<br><br><br># Inform superset to use custom security manager<br>CUSTOM_SECURITY_MANAGER = CustomSecurityManager<br><br><br># OAUTH Providers info<br>OAUTH_PROVIDERS = [{<br>    &#39;name&#39;: &#39;okta&#39;,<br>    &#39;token_key&#39;: &#39;access_token&#39;,<br>    &#39;icon&#39;: &#39;fa-circle-o&#39;,<br>    &#39;remote_app&#39;: {<br>        &#39;client_id&#39;: env(&#39;OKTA_CLIENT_ID&#39;),<br>        &#39;client_secret&#39;: env(&#39;OKTA_CLIENT_SECRET&#39;),<br>        &#39;api_base_url&#39;: &#39;&lt;your-okta-issuer-url&gt;&#39;,<br>        &#39;access_token_url&#39;: &#39;&lt;your-okta-issuer-url&gt;/v1/token&#39;,<br>        &#39;authorize_url&#39;: &#39;&lt;your-okta-issuer-url&gt;/v1/authorize&#39;,<br>        &#39;userinfo_url&#39;: &#39;&lt;your-okta-issuer-url&gt;/v1/userinfo&#39;,<br>        &#39;jwks_uri&#39;: &#39;&lt;your-okta-issuer-url&gt;/v1/keys&#39;,<br>        &#39;client_kwargs&#39;: {<br>            &#39;scope&#39;: &#39;openid profile email&#39;,<br>        },<br>    }<br>}]<br><br># Inform superset to Auto register user in superset DB<br>AUTH_USER_REGISTRATION = True<br><br># Inform superset to assign &quot;admin&quot; role for new user<br>AUTH_USER_REGISTRATION_ROLE = &quot;Admin&quot;  # or a custom role like &quot;Gamma&quot;<br><br># Inform superset to sync users role at login<br>AUTH_ROLES_SYNC_AT_LOGIN = True</pre><h3>🐳 Step 4: Building and Pushing a Custom Docker Image</h3><p>We will create a custom Dockerfile that builds on top of the official Apache Superset base image.</p><p>This approach allows us to include our own configurations — such as the superset_config.py file—and install any additional dependencies required for connecting to external services like Amazon RDS and ElastiCache.</p><pre>FROM apache/superset:4.1.2-py311<br><br>USER root<br><br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    gcc \<br>    libpq-dev \<br>    python3-dev \<br>&amp;&amp; pip install --no-cache-dir psycopg2-binary sqlalchemy-redshift Authlib<br><br>COPY --chown=superset superset_config.py /app/ <br><br>ENV SUPERSET_CONFIG_PATH /app/superset_config.py<br><br>USER superset</pre><p>Then build and push the image to your container registry (e.g., Amazon ECR):</p><pre>docker build -t &lt;your-ecr-repo&gt;:&lt;tag&gt; .<br>docker push &lt;your-ecr-repo&gt;:&lt;tag&gt;</pre><h3>🚀 Step 5: Deploying Superset with Custom Helm Values and Docker Image.</h3><p>To support the OKTA SSO integration and custom configuration, I deployed Superset using Helm with a custom Docker image and values file. This approach gives you full control over the Superset setup, including authentication, ingress, and environment-specific overrides.</p><p>Run following command to get default values</p><pre># Add Superset chart Repo<br>helm repo add superset http://apache.github.io/superset/<br><br># Get default values<br>helm show values superset/superset &gt; superset-values.yaml</pre><p>Update following in superset-values.yaml file</p><pre># -- Extra environment variables that will be passed into pods<br>extraEnv:<br>  SUPERSET_CONFIG_PATH: /app/superset_config.py<br>  OKTA_END_SESSION_ENDPOINT: &quot;&lt;your-okta-issuer-url&gt;/v1/logout&quot;<br>  OKTA_LOGOUT_REDIRECT_URI: &quot;https://&lt;your-superset-domain&gt;/login/&quot;<br>  OKTA_CLIENT_ID: &quot;&lt;your-okta-client-id&gt;&quot; # ideally you should read this from secret, refer my previous blog for mounting custom secret with superset<br>  OKTA_CLIENT_SECRET: &quot;&lt;your-okta-client-secret&gt;&quot; # ideally you should read this from secret, refer my previous blog for mounting custom secret with superset<br>  <br><br>image:<br>  repository: &lt;your-ecr-repo&gt;<br>  tag: &lt;tag&gt;<br>  pullPolicy: Always<br><br>ingress:<br>  enabled: true<br>  annotations:<br>    nginx.ingress.kubernetes.io/proxy-buffer-size: &quot;128k&quot;<br>    nginx.ingress.kubernetes.io/configuration-snippet: |<br>      proxy_set_header X-Forwarded-Proto https;<br>      proxy_set_header X-Forwarded-Port 443;<br>  hosts:<br>    - host: &lt;your-superset-domain&gt;<br>      paths:<br>        - path: /<br>          pathType: ImplementationSpecific</pre><blockquote>Ingress configuration is crucial because Superset dynamically generates redirect URLs during the OAuth2 flow. In many setups, the ingress controller defaults to the HTTP protocol, which can cause Superset to construct redirect URIs using http:// instead of https://. To avoid this, we explicitly set the X-Forwarded-Proto and X-Forwarded-Port headers so that Superset correctly builds secure HTTPS redirect URLs.</blockquote><h4><strong>Deploy Superset</strong></h4><p>Run following command to genertae Helm template</p><pre># Generate template<br>helm template my-superset superset/superset -f superset-values.yaml</pre><p>Run following command to deploy superset. I am using “0.14.2” version</p><pre># Deploy superset version 0.14.2<br>helm upgrade --install my-superset superset/superset --namespace superset --create-namespace --version 0.14.2 -f superset-values.yaml</pre><p>If everything works well you should see all pods in running state.</p><h3>🧪 Step 6: Testing the OKTA SSO Integration</h3><p>Once the SSO configuration is complete and deployed, it’s crucial to <strong>validate the setup thoroughly before rolling it out to production</strong>. Here’s a checklist to guide your testing:</p><h4><strong>Login Flow</strong></h4><ul><li>Access Superset via the browser and confirm you’re redirected to the OKTA login page.</li><li>After successful authentication, ensure you’re redirected back to Superset and logged in as the correct user.</li></ul><h4><strong>User Mapping</strong></h4><ul><li>Verify that existing Superset users are correctly mapped via email (as configured in CustomSecurityManager).</li><li>Check role assignments and permissions for mapped users.</li></ul><h4><strong>Logout Behavior</strong></h4><ul><li>Test the logout flow to ensure it terminates both Superset and OKTA sessions.</li><li>Confirm that the browser does not retain any active session cookies.</li></ul><h4><strong>Session Expiry and Re-authentication</strong></h4><ul><li>Simulate session expiry and verify that re-authentication works as expected.</li></ul><h4><strong>Error Handling</strong></h4><ul><li>Try logging in with an unauthorized user and confirm that access is denied gracefully.</li></ul><h4><strong>Audit Logs</strong></h4><ul><li>Check OKTA and Superset logs to ensure authentication events are recorded properly.</li></ul><blockquote>💡 <strong>Tip:</strong> Perform these tests in a staging or INT environment before migrating your DEMO or production instances.</blockquote><h3>✅ Strategy for Seamless Migration</h3><h4><strong>Email-Based Mapping</strong></h4><ul><li>Superset users are typically identified by their usernames.</li><li>OKTA provides user identity via email in the OAuth response.</li></ul><h4><strong>Pre-Migration Checklist</strong></h4><ul><li>Ensure all existing Superset users have their email field populated correctly.</li><li>Validate that email addresses in Superset match those in OKTA.</li></ul><h4><strong>Testing Before Full Rollout</strong></h4><ul><li>Deploy the SSO setup in an non prod environment first.</li><li>Test login and logout flows, user mapping, and role assignments.</li><li>Once validated, migrate the production environment.</li></ul><h3>🛠️ Troubleshooting &amp; Debugging Common SSO Issues</h3><p>Even with a well-documented setup, SSO integrations can be tricky. Here are some common issues you might encounter and how to resolve them:</p><h4>Login Redirect Loop</h4><p><strong>Symptoms:</strong> Browser keeps redirecting between Superset and OKTA without logging in.</p><p><strong>Fix:</strong></p><ul><li>Ensure X-Forwarded-* headers are correctly set in your ingress.</li><li>Confirm that Superset is aware of the external URL via ENABLE_PROXY_FIX = True in superset_config.py.</li></ul><h4>User Not Found or Duplicate Users</h4><p><strong>Symptoms:</strong> OKTA login succeeds, but Superset creates a new user instead of mapping to an existing one.</p><p><strong>Fix:</strong></p><ul><li>Check that CustomSecurityManager maps users by email.</li><li>Ensure existing Superset users have their email field populated and match OKTA emails exactly.</li></ul><h4>Logout Doesn’t End OKTA Session</h4><p><strong>Symptoms:</strong> Logging out of Superset doesn’t log the user out of OKTA.</p><p><strong>Fix:</strong></p><ul><li>Override authoauthview in CustomSecurityManager with a custom CustomAuthOAuthView class.</li><li>Call OKTA’s end_session_endpoint and configure front-channel logout URLs in OKTA.</li></ul><h4>403 or 500 Errors After Login</h4><p><strong>Symptoms:</strong> Login succeeds but Superset throws an error.</p><p><strong>Fix:</strong></p><ul><li>Check Superset logs for stack traces.</li><li>Validate that required fields (email, name) are present in the OAuth response.</li><li>Ensure roles and permissions are correctly assigned.</li></ul><h3>📚 Lessons Learned</h3><p>Implementing OKTA SSO in Superset was not just a technical integration — it was a journey through Superset’s internals, OAuth flows, and deployment nuances. Here are some key lessons we learned:</p><ol><li><strong>Superset’s OAuth implementation is flexible but under-documented.</strong><br> Deep-diving into BaseSecurityManager and community discussions helped uncover override strategies that aren’t obvious from the docs.</li><li><strong>Logout flow requires both backend and frontend coordination.</strong><br> OKTA’s end_session_endpoint alone isn’t enough — front-channel logout URLs and Superset overrides are essential.</li><li><strong>Testing in a controlled environment is invaluable.</strong><br> Deploying a test instance with username/password auth allowed us to validate SSO without risking production access.</li><li><strong>CustomSecurityManager is powerful but sensitive to placement.</strong><br> The order of class definitions in superset_config.py can affect behavior — especially when overriding views.</li><li><strong>Ingress headers are critical for OAuth redirect behavior.</strong><br> Missing X-Forwarded-* headers can break login flows silently.</li></ol><h3>🏁 Final Thoughts</h3><p>Integrating OKTA SSO with Apache Superset transforms your analytics platform into a secure, enterprise-ready solution. While the process involves multiple layers — from Docker and Helm to OAuth and Superset internals — each step builds toward a seamless user experience.</p><p>Whether you’re running Superset on EKS or another Kubernetes platform, this guide should help you navigate the complexities of SSO integration with confidence. And remember: <strong>test thoroughly, debug patiently, and document everything</strong> — your future self (and your team) will thank you.</p><h3>✅ Conclusion &amp; Key Takeaways</h3><p>Integrating OKTA SSO with Apache Superset not only enhances security but also streamlines user management across your organization. By following a structured approach — from setting up a test environment to customizing the security manager and handling logout flows — you can achieve a seamless and production-ready SSO experience.</p><h3>🔑 Key Takeaways:</h3><ul><li><strong>Start with a test environment</strong> using username/password auth to validate SSO integration safely.</li><li><strong>Customize Superset’s security manager</strong> to map users by email for smooth migration.</li><li><strong>Configure ingress headers and session handling</strong> to avoid redirect and login issues.</li><li><strong>Test thoroughly</strong> before rolling out to production — including login, logout, user mapping, and session behavior.</li><li><strong>Use Helm and custom Docker images</strong> to manage deployment cleanly and securely.</li><li><strong>Troubleshoot proactively</strong> using logs and error patterns to resolve common issues.</li></ul><blockquote>With this setup, your Superset deployment becomes enterprise-ready, secure, and scalable — all while maintaining a smooth user experience.</blockquote><p><strong>Congratulations!!! You have successfully Implemented OKTA SSO in Apache Superset.</strong></p><p><strong>Thank You!!!</strong></p><h3>📚 References</h3><ul><li><a href="https://superset.apache.org/docs/configuration/configuring-superset/">https://superset.apache.org/docs/configuration/configuring-superset/</a></li><li><a href="https://github.com/apache/superset/discussions/33812">https://github.com/apache/superset/discussions/33812</a></li><li><a href="https://stackoverflow.com/questions/67076134/apache-superset-with-okta-integration">https://stackoverflow.com/questions/67076134/apache-superset-with-okta-integration</a></li><li><a href="https://gist.github.com/ktmud/2475282a166893e5d17039c308cbe50d">https://gist.github.com/ktmud/2475282a166893e5d17039c308cbe50d</a></li><li><a href="https://github.com/apache/superset/issues/13948">https://github.com/apache/superset/issues/13948</a></li><li><a href="https://blog.elest.io/how-to-secure-apache-superset-with-oauth2/">https://blog.elest.io/how-to-secure-apache-superset-with-oauth2/</a></li><li><a href="https://stackoverflow.com/questions/44952750/oauth-authentication-in-apache-superset">https://stackoverflow.com/questions/44952750/oauth-authentication-in-apache-superset</a></li></ul><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=848f44e1f4da" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/implementing-okta-sso-in-apache-superset-848f44e1f4da">Implementing OKTA SSO in Apache Superset</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cross-Account S3 Replication Demystified]]></title>
            <link>https://aws.plainenglish.io/cross-account-s3-replication-demystified-d9566185bca1?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9566185bca1</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[replication]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 13:04:48 GMT</pubDate>
            <atom:updated>2026-03-18T07:41:54.494Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Step-by-Step Guide with SSE-KMS and IAM Best Practices</h4><p>Setting up cross-account S3 replication in AWS sounds straightforward — until encryption and permissions come into play. In a recent project, I faced a puzzling replication failure that turned out to be a missing IAM permission in the replication role. This blog is a deep dive into how I architected a secure, scalable S3 replication setup using SSE-KMS encryption across AWS accounts. Whether you’re a cloud engineer looking to implement this for the first time or someone troubleshooting a broken replication pipeline, this guide walks you through the entire process — from bucket policies and IAM roles to KMS key sharing and debugging real-world issues.</p><h3>🏗️ Architecture Overview</h3><p>This cross-account S3 replication setup is designed for secure, compliant, and scalable data synchronization between two AWS accounts. Both the source and destination buckets are configured with:</p><ul><li>SSE-KMS encryption</li><li>Versioning enabled</li><li>Object Lock for data immutability</li><li>Blocked public access</li><li>Controlled ACLs with specific grantee permissions</li></ul><p>The replication process uses an IAM role in the source account with permissions to replicate objects to the destination bucket. The KMS key used for encryption must be shared across accounts with appropriate grants to allow decryption and re-encryption during replication.</p><p>Here’s a high-level architecture diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWSFUF7lysI2rMQrC_fuvA.png" /><figcaption>Image Source: <a href="https://aws.plainenglish.io/aws-s3-cross-account-cross-region-replication-with-kms-encryption-a-comprehensive-guide-81fbbb4e6a0d">https://aws.plainenglish.io/aws-s3-cross-account-cross-region-replication-with-kms-encryption-a-comprehensive-guide-81fbbb4e6a0d</a></figcaption></figure><h3>🔐 IAM Role and KMS Key Configuration</h3><p>Setting up cross-account S3 replication with SSE-KMS encryption requires careful coordination between IAM roles, bucket policies, and KMS key permissions. Here’s a breakdown of the key components:</p><h3>IAM Role in Source Account</h3><p>Create an IAM role that S3 can assume to perform replication. This role must have:</p><ul><li>Trust relationship allowing S3 to assume the role.</li><li>Permissions to read from the source bucket.</li><li>Permissions to write to the destination bucket in the target account.</li><li>Permissions to use both the source and destination KMS keys.</li></ul><h3>🛠️ Step-by-Step: Creating IAM Role and Policy via AWS CLI</h3><h4>Create the IAM Role for S3 Replication</h4><p>Create a trust policy file trust-policy.json:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;s3.amazonaws.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>    }<br>  ]<br>}</pre><p>Run the CLI command to create the role:</p><pre>aws iam create-role \<br>  --role-name S3ReplicationRole \<br>  --assume-role-policy-document file://trust-policy.json</pre><h4>Create the IAM Policy for Replication</h4><p>Create a policy file replication-policy.json :</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:GetObjectVersion&quot;,<br>                &quot;s3:GetBucketVersioning&quot;,<br>                &quot;s3:ListBucket&quot;,<br>                &quot;s3:GetReplicationConfiguration&quot;,<br>                &quot;s3:GetObjectVersionForReplication&quot;,<br>                &quot;s3:GetObjectVersionAcl&quot;,<br>                &quot;s3:GetObjectVersionTagging&quot;,<br>                &quot;s3:GetObjectRetention&quot;,<br>                &quot;s3:GetObjectLegalHold&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::source-bucket-name&quot;,<br>                &quot;arn:aws:s3:::source-bucket-name/*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:ReplicateObject&quot;,<br>                &quot;s3:ReplicateDelete&quot;,<br>                &quot;s3:ReplicateTags&quot;,<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::destination-bucket-name/*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;kms:Decrypt&quot;,<br>                &quot;kms:GenerateDataKey&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:kms:region:source-account-id:key/source-kms-key-id&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;kms:Encrypt&quot;,<br>                &quot;kms:ReEncrypt*&quot;,<br>                &quot;kms:GenerateDataKey*&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:kms:region:destination-account-id:key/destination-kms-key-id&quot;<br>        }<br>    ]<br>}</pre><p>Create and attach the policy:</p><pre>aws iam put-role-policy \<br>  --role-name S3ReplicationRole \<br>  --policy-name S3ReplicationPolicy \<br>  --policy-document file://replication-policy.json</pre><h3>🔐 Updating KMS Key Policy via AWS Console (Destination Account)</h3><h4>Steps</h4><ol><li>Sign in to the AWS Console in the destination account.</li><li>Navigate to Key Management Service (KMS).</li><li>In the left sidebar, click Customer managed keys.</li><li>Select the KMS key used for encrypting the destination S3 bucket.</li><li>Go to the Key policy tab.</li><li>Click Edit.</li><li>Scroll to the Key policy editor section and click “Switch to policy view” if you’re in the visual editor.</li><li>Add a new statement to the existing policy (do not modify existing ones). Paste the following snippet inside the Statement array:</li></ol><pre>{<br>  &quot;Sid&quot;: &quot;AllowCrossAccountReplication&quot;,<br>  &quot;Effect&quot;: &quot;Allow&quot;,<br>  &quot;Principal&quot;: {<br>    &quot;AWS&quot;: &quot;arn:aws:iam::&lt;source-account-id&gt;:role/&lt;replication-role-name&gt;&quot;<br>  },<br>  &quot;Action&quot;: [<br>    &quot;kms:Encrypt&quot;,<br>    &quot;kms:ReEncrypt*&quot;,<br>    &quot;kms:GenerateDataKey*&quot;<br>  ],<br>  &quot;Resource&quot;: &quot;*&quot;<br>}</pre><p>9. Click Save changes.</p><h3>🪣 Updating S3 Bucket Policy via AWS Console (Destination Account)</h3><h4>Steps</h4><ol><li>Sign in to the AWS Console in the destination account.</li><li>Navigate to S3 and open the destination bucket.</li><li>Go to the Permissions tab.</li><li>Scroll down to Bucket policy and click Edit.</li><li>In the policy editor, add the following statement to the existing policy (inside the Statement array):</li></ol><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;AllowReplicationFromSourceAccount&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::&lt;source-account-id&gt;:role/&lt;replication-role-name&gt;&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;s3:ReplicateObject&quot;,<br>                &quot;s3:ReplicateDelete&quot;,<br>                &quot;s3:ReplicateTags&quot;,<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::&lt;destination-bucket-name&gt;/*&quot;<br>        }<br>    ]<br>}</pre><p>6. Click Save changes.</p><h3>🔁 Configuring S3 Replication via AWS Console (Source Account)</h3><h4>Step 1: Open the Source Bucket</h4><ul><li>Sign in to the AWS Console in the source account.</li><li>Navigate to S3 and open the source bucket.</li></ul><h4>Step 2: Enable Versioning</h4><p>Ensure versioning is enabled:</p><ul><li>Go to the Properties tab.</li><li>Scroll to Bucket Versioning.</li><li>If not enabled, click Edit and enable it.</li></ul><h4>Step 3: Configure Replication</h4><ul><li>Go to the Management tab.</li><li>Scroll down to Replication rules and click Create replication rule.</li></ul><h4>Step 4: Define Rule Details</h4><ul><li>Rule name: e.g., CrossAccountReplicationToTarget</li><li>Status: Enabled</li><li>Source: Choose either the entire bucket or a prefix filter.</li></ul><h4>Step 5: Destination Configuration</h4><ul><li>Destination bucket: Choose A bucket in another account.</li><li>Enter the Account ID of the destination account.</li><li>Bucket name: Enter destination bucket name</li></ul><h4>Step 6: IAM Role</h4><ul><li>Choose Enter IAM role ARN manually.</li><li>Select the IAM Role Created earlier</li></ul><h4>Step 6: Encryption</h4><ul><li>Select Check box: Replicate objects encrypted with AWS Key Management Service (AWS KMS)</li><li>AWS KMS key for encrypting destination objects: Enter AWS KMS key ARN of destination bucket KMS Key</li></ul><h4>Step 6: <strong>Additional replication options</strong></h4><ul><li>Select Chec box: Delete marker replication</li></ul><h4>Step 7: Review and Save</h4><ul><li>Review all settings.</li><li>Click Save to create the replication rule.</li></ul><h3>🧾 Conclusion &amp; Key Takeaways</h3><p>Setting up cross-account S3 replication with SSE-KMS encryption is a powerful way to securely synchronize data across AWS accounts, but it requires careful attention to detail. From IAM roles and KMS key policies to bucket configurations and replication rules, each component must be precisely aligned for replication to succeed.</p><h4>🔑 Key Takeaways:</h4><ul><li>Versioning and SSE-KMS encryption are mandatory for replication.</li><li>IAM roles must be correctly trusted and permissioned to access both buckets and KMS keys.</li><li>KMS key policies must explicitly allow cross-account access for encryption and decryption.</li><li>Bucket policies must permit replication actions from the source account.</li><li>Monitoring tools like CloudTrail and S3 metrics are essential for validating and troubleshooting replication.</li></ul><p>By following this guide, you can build a secure, scalable, and compliant replication pipeline that supports even the most tightly locked-down S3 buckets.</p><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9566185bca1" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/cross-account-s3-replication-demystified-d9566185bca1">Cross-Account S3 Replication Demystified</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing Apache Superset in Kubernetes]]></title>
            <link>https://aws.plainenglish.io/installing-apache-superset-in-kubernetes-1aec192ac495?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1aec192ac495</guid>
            <category><![CDATA[apache-superset]]></category>
            <category><![CDATA[superset]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[bi-analytics]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 08:03:52 GMT</pubDate>
            <atom:updated>2026-03-18T07:42:34.955Z</atom:updated>
            <content:encoded><![CDATA[<h4>Production Grade Apache Superset Deployment</h4><p>Superset is a user-friendly data visualization and exploration tool designed to help business analysts gain insights quickly. It offers:</p><ul><li>A no-code interface to build charts and dashboards</li><li>A SQL editor for deeper data analysis</li><li>Easy creation of custom metrics and dimensions</li><li>A wide variety of visualizations, including maps and time series</li><li>Fast performance with built-in caching</li><li>Secure, role-based access controls</li><li>Seamless integration with most SQL databases</li></ul><p>Apache Superset is a powerful, open-source platform for data exploration and visualization. What makes it scalable, flexible, and production-ready is its modular, service-oriented architecture.</p><h3>Inside Apache Superset: A Look at Its Modular Architecture</h3><h4>Frontend: React-Based Single-Page Application (SPA)</h4><p>Superset’s user interface is built with React, delivering a smooth, responsive experience. As a single-page application (SPA), it allows users to interact with dashboards, charts, and data sources without full page reloads — making data exploration fast and intuitive.</p><h4>Backend: Flask-Based Python Application</h4><p>The backend is powered by Flask, a lightweight Python web framework. It handles:</p><ul><li>API requests from the frontend</li><li>User authentication and authorization</li><li>Business logic and data processing</li></ul><p>This layer acts as the brain of Superset, orchestrating interactions between users, data sources, and internal services.</p><h4>Metadata Database</h4><p>Superset uses a relational database (like PostgreSQL or MySQL) to store its internal metadata. This includes:</p><ul><li>Dashboard and chart configurations</li><li>Saved queries</li><li>User roles and permissions</li><li>Data source definitions</li></ul><p>This database is essential for persisting user-generated content and system settings.</p><h4>Celery Workers</h4><p>To keep the UI responsive, Superset offloads time-consuming tasks to Celery workers. These background workers handle:</p><ul><li>Scheduled dashboard refreshes</li><li>Long-running SQL queries</li><li>Alerts and report generation</li></ul><p>This asynchronous processing ensures that heavy operations don’t block the main application.</p><h4>Redis</h4><p>Redis serves two key purposes:</p><ol><li>Message broker for Celery, enabling task queuing and communication between services.</li><li>Caching layer to store query results and metadata, reducing load on the database and speeding up performance.</li></ol><h4>Web Server: Gunicorn</h4><p>Superset typically uses Gunicorn, a Python WSGI HTTP server, to serve the Flask backend. It handles concurrent requests efficiently and is well-suited for production deployments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*XkGuQU6xjtgPLr2FqKKOCQ.png" /></figure><h3>Let’s get started!</h3><p>In this blog post, I’ll guide you through setting up a customized deployment of Apache Superset on Kubernetes, tailored for scalability, security, and performance in a cloud-native environment.</p><p>We’ll be using a combination of AWS services and custom configurations to enhance Superset’s capabilities. Here’s what the setup includes:</p><ul><li>A custom multi-architecture Docker image built on top of the official Apache Superset base image</li><li>A custom superset_config.py file to fine-tune Superset’s behavior</li><li>Secrets managed via AWS Secrets Manager for secure credential handling</li><li>An AWS RDS PostgreSQL instance to store Superset’s metadata</li><li>AWS ElastiCache for caching and background task management</li><li>The official Superset Helm chart, customized with a tailored values.yaml for Kubernetes deployment</li></ul><p>By the end of this guide, you’ll have a production-ready Superset instance running on Kubernetes, fully integrated with AWS services and optimized for real-world use.</p><h3>Pre-requisites:</h3><ol><li>EKS Cluster</li><li>OIDC Provider should be configured for EKS</li><li>Kubectl</li><li>AWS Cli</li><li>Helm</li><li>PostgreSQL DB</li><li>Redis ElastiCache</li><li>Ingress Controller</li></ol><p>In Kubernetes, you cannot directly access secrets stored in AWS Secrets Manager. To bridge this gap, I’m using the Secrets Store CSI Driver along with the AWS Secrets and Configuration Provider (ASCP). This setup allows Kubernetes to securely retrieve secrets from AWS Secrets Manager and automatically create corresponding Kubernetes Secrets.</p><p><em>For a step-by-step guide on installing and configuring ASCP, you can refer to my blog post. Refer — Step 2: Install and configure the ASCP</em></p><p><a href="https://aws.plainenglish.io/kubernetes-auto-restart-deployments-when-k8s-configmap-or-secret-is-updated-ad2c042a745f">Kubernetes: Auto Restart Deployments when K8s ConfigMap or Secret is Updated</a></p><h3>Create IAM Role for Superset to Access secret from aws secret manager</h3><ol><li>Create IAM policy Document.<br>create a file with name “<strong>secrets_policy”</strong> and add following content</li></ol><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;secretsmanager:DescribeSecret&quot;,<br>                &quot;secretsmanager:GetSecretValue&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>     ]<br>}</pre><p>2. Run following command to create IAM policy.<br>Take a Note of policy ARN, will need it while attaching policy with IAM role</p><pre>aws iam create-policy \<br>    --policy-name my-secret-manager-policy \<br>    --policy-document file://secrets_policy</pre><p>3. Create Trust Policy for IAM Role<br>Create file with name <strong>“trust_policy” </strong>and add following content. Make sure you replace correct values. I will be using “superset” namespace and “superset-sa” as service account name.</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Federated&quot;: &quot;arn:aws:iam::&lt;AWS_ACCOUNT_ID&gt;:oidc-provider/oidc.eks.&lt;AWS_REGION&gt;.amazonaws.com/id/&lt;OIDC_ID&gt;&quot;<br>      },<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;oidc.eks.&lt;AWS_REGION&gt;.amazonaws.com/id/&lt;OIDC_ID&gt;:aud&quot;: &quot;sts.amazonaws.com&quot;,<br>          &quot;oidc.eks.&lt;AWS_REGION&gt;.amazonaws.com/id/&lt;OIDC_ID&gt;:sub&quot;: &quot;system:serviceaccount:superset:superset-sa&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>4. Run following command to create IAM Role</p><pre>aws iam create-role --role-name my-secret-manager-role<br>--assume-role-policy-document file://trust_policy</pre><p>5. Attach IAM policy to IAM Role</p><pre>aws iam attach-role-policy --policy-arn &lt;your_policy_arn&gt;<br>--role-name my-secret-manager-role</pre><p>We have created all the required IAM Roles and Policies.</p><h3>Writing custom config file for superset:</h3><p>We will use a custom configuration file named superset_config.py to enable connectivity between Apache Superset and our external Amazon RDS and ElastiCache clusters.</p><p><strong>Note:</strong> This configuration file should be tailored to your specific use case. During the Helm chart installation, we will pass the necessary values to ensure proper integration.</p><pre>import os<br>from flask_caching.backends.rediscache import RedisCache<br><br>def env(key, default=None):<br>    return os.getenv(key, default)<br><br># Redis Base URL<br>REDIS_BASE_URL=f&quot;{env(&#39;REDIS_PROTO&#39;)}://{env(&#39;REDIS_HOST&#39;)}:{env(&#39;REDIS_PORT&#39;)}&quot;<br><br># Redis URL Params<br>REDIS_URL_PARAMS = &quot;&quot;<br><br># Build Redis URLs<br>CACHE_REDIS_URL = f&quot;{REDIS_BASE_URL}/{env(&#39;REDIS_DB&#39;, 1)}{REDIS_URL_PARAMS}&quot;<br>CELERY_REDIS_URL = f&quot;{REDIS_BASE_URL}/{env(&#39;REDIS_CELERY_DB&#39;, 0)}{REDIS_URL_PARAMS}&quot;<br><br>MAPBOX_API_KEY = env(&#39;MAPBOX_API_KEY&#39;, &#39;&#39;)<br>CACHE_CONFIG = {<br>      &#39;CACHE_TYPE&#39;: &#39;RedisCache&#39;,<br>      &#39;CACHE_DEFAULT_TIMEOUT&#39;: 300,<br>      &#39;CACHE_KEY_PREFIX&#39;: &#39;superset_&#39;,<br>      &#39;CACHE_REDIS_URL&#39;: CACHE_REDIS_URL,<br>}<br>DATA_CACHE_CONFIG = CACHE_CONFIG<br><br>SQLALCHEMY_DATABASE_URI = f&quot;postgresql+psycopg2://{env(&#39;DB_USER&#39;)}:{env(&#39;DB_PASS&#39;)}@{env(&#39;DB_HOST&#39;)}:{env(&#39;DB_PORT&#39;)}/{env(&#39;DB_NAME&#39;)}&quot;<br>SQLALCHEMY_TRACK_MODIFICATIONS = True<br><br>class CeleryConfig:<br>  imports  = (&quot;superset.sql_lab&quot;, )<br>  broker_url = CELERY_REDIS_URL<br>  result_backend = CELERY_REDIS_URL<br><br>CELERY_CONFIG = CeleryConfig<br>RESULTS_BACKEND = RedisCache(<br>      host=env(&#39;REDIS_HOST&#39;),<br>      port=env(&#39;REDIS_PORT&#39;),<br>      key_prefix=&#39;superset_results&#39;,<br>)<br><br>ROW_LIMIT = 1000<br>SUPERSET_WEBSERVER_TIMEOUT = 60</pre><h3>Building Custom Docker Image:</h3><p>We will create a custom Dockerfile that builds on top of the official Apache Superset base image.</p><p>This approach allows us to include our own configurations—such as the superset_config.py file—and install any additional dependencies required for connecting to external services like Amazon RDS and ElastiCache.</p><pre>FROM apache/superset:4.1.2-py311<br><br>USER root<br><br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    gcc \<br>    libpq-dev \<br>    python3-dev \<br>&amp;&amp; pip install --no-cache-dir psycopg2-binary sqlalchemy-redshift<br><br>COPY --chown=superset superset_config.py /app/ <br><br>ENV SUPERSET_CONFIG_PATH /app/superset_config.py<br><br>USER superset</pre><p>Here’s a simple shell script that automates the process of building a Docker image and pushing it to Amazon ECR:</p><pre>#!/bin/bash<br><br># Variables - replace these with your actual values<br>AWS_REGION=&quot;&lt;your_aws_region&gt;&quot;<br>ACCOUNT_ID=&quot;&lt;your_aws_account_id&gt;&quot;<br>REPO_NAME=&quot;&lt;your_repo_name&gt;&quot;<br>IMAGE_TAG=&quot;&lt;your_tag&gt;&quot;<br><br># Full ECR repository URI<br>ECR_URI=&quot;${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${REPO_NAME}:${IMAGE_TAG}&quot;<br><br># Authenticate Docker to ECR<br>echo &quot;Logging in to Amazon ECR...&quot;<br>aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com<br><br># Create ECR repository if it doesn&#39;t exist<br>echo &quot;Ensuring ECR repository exists...&quot;<br>aws ecr describe-repositories --repository-names $REPO_NAME --region $AWS_REGION &gt;/dev/null 2&gt;&amp;1 || \<br>aws ecr create-repository --repository-name $REPO_NAME --region $AWS_REGION<br><br># Build the Docker image<br>echo &quot;Building Docker image...&quot;<br>docker build -t $REPO_NAME .<br><br># Tag the image<br>echo &quot;Tagging image as $ECR_URI...&quot;<br>docker tag $REPO_NAME:latest $ECR_URI<br><br># Push the image to ECR<br>echo &quot;Pushing image to ECR...&quot;<br>docker push $ECR_URI<br><br>echo &quot;Docker image pushed to: $ECR_URI&quot;</pre><p>Run script</p><pre>chmod +x build-and-push.sh<br><br>./build-and-push.sh</pre><h3>Create Superset Admin Credentials Secret in AWS</h3><p>We will be creating secret with name “superset-secret” in AWS Secret Manager whihc will store: superset admin username, superset admin password and superset secret key.</p><pre>aws secretsmanager create-secret \<br>    --name superset-secret \<br>    --description &quot;Superset Secret&quot; \<br>    --secret-string &quot;{\&quot;superset_admin_username\&quot;:\&quot;superset_admin\&quot;,\&quot;superset_admin_password\&quot;:\&quot;EXAMPLE-PASSWORD\&quot;,\&quot;superset_secret_key\&quot;:\&quot;EXAMPLE-STORING-SECRET-KEY\&quot;}&quot;</pre><p><strong>Note:</strong> Recommended way to generate “superset_secret_key” is using openssl rand -base64 42 command.</p><h3>Create Superset Namespce</h3><p>Run following command to create superset namespace</p><pre>kubectl create ns superset</pre><h3>Create SecretProviderClass Object</h3><p>We will create SecretProviderClass Object to map secres from AWS Secret Manager with Kubernetes Secret</p><pre>apiVersion: secrets-store.csi.x-k8s.io/v1<br>kind: SecretProviderClass<br>metadata:<br>  name: superset-secrets-store # name of SecretProviderClass object<br>  namespace: superset<br>spec:<br>  provider: aws<br>  secretObjects:<br>  - secretName: superset-secret-csi-opaque # name of k8s secret<br>    type: Opaque<br>    data:<br>    - objectName: host<br>      key: host<br>    - objectName: username<br>      key: username<br>    - objectName: password<br>      key: password<br>    - objectName: port<br>      key: port<br>    - objectName: superset_secret_key<br>      key: superset_secret_key<br>    - objectName: superset_admin_username<br>      key: superset_admin_username<br>    - objectName: superset_admin_password<br>      key: superset_admin_password<br>  parameters:<br>    objects:  |<br><br>      - objectName: &lt;your_db_master_creds_secret_arn&gt;<br>        jmesPath:<br>          - path: host<br>            objectAlias: host<br>          - path: username<br>            objectAlias: username<br>          - path: password<br>            objectAlias: password<br>          - path: port<br>            objectAlias: port<br><br>      - objectName: &lt;your_superset_secret_arn&gt;<br>        jmesPath:<br>          - path: superset_secret_key<br>            objectAlias: superset_secret_key<br>          - path: superset_admin_username<br>            objectAlias: superset_admin_username<br>          - path: superset_admin_password<br>            objectAlias: superset_admin_password</pre><p>Run following command to create object</p><pre># Create SecretProviderClass Object<br>kubectl apply -f secretproviderclass.yaml<br><br># Verfiy SecretProviderClass Object<br>kubectl get secretproviderclass</pre><h3>Writing custom values file for superset:</h3><p>We will override default values for superset helm chart by providing our own custom values file.</p><p>Run following command to get default values</p><pre># Add Superset chart Repo<br>helm repo add superset http://apache.github.io/superset/<br><br># Get default values<br>helm show values superset/superset &gt; superset-default-values.yaml</pre><p>Create a file named “superset-values.yaml” and replace placeholders.</p><pre># -- Specify rather or not helm should create the secret described in `secret-env.yaml` template<br>secretEnv:<br>  # -- Change to false in order to support externally created secret (Binami &quot;Sealed Secrets&quot; for Kubernetes or External Secrets Operator)<br>  # note: when externally creating the secret, the chart still expects to pull values from a secret with the name of the release defaults to `release-name-superset-env` - full logic located in _helpers.tpl file: `define &quot;superset.fullname&quot;`<br>  create: false<br><br># -- Specify service account name to be used<br>serviceAccountName: superset-sa # This name should match with name of IAM trust policy<br>serviceAccount:<br>  # -- Create custom service account for Superset. If create: true and serviceAccountName is not provided, `superset.fullname` will be used.<br>  create: true<br>  annotations:<br>    eks.amazonaws.com/role-arn: &lt;your_IRSA_role_arn&gt;<br><br># -- The name of the secret which we will use to populate env vars in deployed pods<br># This can be useful for secret keys, etc.<br># envFromSecret: &#39;{{ template &quot;superset.fullname&quot; . }}-env&#39;<br>envFromSecret: &#39;{{ template &quot;superset.fullname&quot; . }}-config&#39; # we need to use &#39;-config&#39; prefix because we are setting secretEnv.create to false<br><br><br># -- Extra environment variables that will be passed into pods<br>extraEnv:<br>  REDIS_HOST: &quot;&lt;your_redis_db_endpoint&gt;&quot;<br>  REDIS_PORT: &quot;6379&quot;<br>  REDIS_DB: &quot;1&quot;<br>  REDIS_CELERY_DB: &quot;0&quot;<br>  REDIS_PROTO: &quot;redis&quot;<br>  SUPERSET_LOG_LEVEL: &quot;info&quot;<br>  DB_NAME: &quot;&lt;your_superset_db&gt;&quot; # This should be name of existing database<br>  SUPERSET_CONFIG_PATH: /app/superset_config.py<br><br><br># -- Extra environment variables in RAW format that will be passed into pods<br>extraEnvRaw:<br>  - name: DB_PASS<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: password<br>  - name: DB_USER<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: username<br>  - name: DB_HOST<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: host<br>  - name: DB_PORT<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: port<br>  - name: SUPERSET_SECRET_KEY<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: superset_secret_key<br>  - name: SUPERSET_ADMIN_USERNAME<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: superset_admin_username<br>  - name: SUPERSET_ADMIN_PASSWORD<br>    valueFrom:<br>      secretKeyRef:<br>        name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        key: superset_admin_password<br><br><br>extraVolumes:<br>  - csi:<br>      driver: secrets-store.csi.k8s.io<br>      readOnly: true<br>      volumeAttributes:<br>        secretProviderClass: superset-secrets-store # name of SecretProviderClass object<br>    name: awareintegrationsecretsmanager<br><br>extraVolumeMounts:<br>  - name: awareintegrationsecretsmanager<br>    mountPath: /mnt/secrets<br>    readOnly: true<br><br><br>image:<br>  repository: &lt;your_ecr_url&gt;<br>  tag: &lt;your_image_tag&gt;<br>  pullPolicy: IfNotPresent<br><br><br>ingress:<br>  enabled: true<br>  ingressClassName: nginx # your ingress class name<br>  annotations:<br>    # kubernetes.io/tls-acme: &quot;true&quot;<br>    ## Extend timeout to allow long running queries.<br>    nginx.ingress.kubernetes.io/proxy-connect-timeout: &quot;300&quot;<br>    nginx.ingress.kubernetes.io/proxy-read-timeout: &quot;300&quot;<br>    nginx.ingress.kubernetes.io/proxy-send-timeout: &quot;300&quot;<br>  path: /<br>  pathType: ImplementationSpecific<br>  hosts:<br>    - &lt;your_host&gt;<br>  tls: []<br>  extraHostsRaw: []<br>  #  - secretName: chart-example-tls<br>  #    hosts:<br>  #      - chart-example.local<br><br><br># Superset node configuration<br>supersetNode:<br>  replicas:<br>    enabled: true<br>    replicaCount: 2<br>  autoscaling:<br>    enabled: true<br>    minReplicas: 2<br>    maxReplicas: 5<br>    targetCPUUtilizationPercentage: 60<br>    targetMemoryUtilizationPercentage: 60<br>  <br>  # -- Init containers<br>  # @default -- a container waiting for postgres<br>  initContainers:<br>    - name: wait-for-postgres<br>      image: &quot;{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}&quot;<br>      imagePullPolicy: &quot;{{ .Values.initImage.pullPolicy }}&quot;<br>      env:<br>        - name: DB_HOST<br>          valueFrom:<br>            secretKeyRef:<br>              key: host<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        - name: DB_PORT<br>          valueFrom:<br>            secretKeyRef:<br>              key: port<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>      envFrom:<br>        - secretRef:<br>            name: &quot;{{ tpl .Values.envFromSecret . }}&quot;<br>      command:<br>        - /bin/sh<br>        - -c<br>        - dockerize -wait &quot;tcp://$DB_HOST:$DB_PORT&quot; -timeout 120s<br>      volumeMounts:<br>        - name: awareintegrationsecretsmanager<br>          mountPath: /mnt/secrets<br>          readOnly: true<br><br><br># Superset Celery worker configuration<br>supersetWorker:<br>  replicas:<br>    enabled: true<br>    replicaCount: 2<br>  autoscaling:<br>    enabled: true<br>    minReplicas: 2<br>    maxReplicas: 5<br>    targetCPUUtilizationPercentage: 60<br>    targetMemoryUtilizationPercentage: 60<br>  # -- Init container<br>  # @default -- a container waiting for postgres and redis<br>  initContainers:<br>    - name: wait-for-postgres-redis<br>      image: &quot;{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}&quot;<br>      imagePullPolicy: &quot;{{ .Values.initImage.pullPolicy }}&quot;<br>      env:<br>        - name: DB_HOST<br>          valueFrom:<br>            secretKeyRef:<br>              key: host<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        - name: DB_PORT<br>          valueFrom:<br>            secretKeyRef:<br>              key: port<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        - name: REDIS_HOST<br>          value: &quot;{{ tpl .Values.extraEnv.REDIS_HOST . }}&quot;<br>        - name: REDIS_PORT<br>          value: &quot;{{ tpl .Values.extraEnv.REDIS_PORT . }}&quot;<br>      envFrom:<br>        - secretRef:<br>            name: &quot;{{ tpl .Values.envFromSecret . }}&quot;<br>      command:<br>        - /bin/sh<br>        - -c<br>        - dockerize -wait &quot;tcp://$DB_HOST:$DB_PORT&quot; -wait &quot;tcp://$REDIS_HOST:$REDIS_PORT&quot; -timeout 120s<br>      volumeMounts:<br>        - name: awareintegrationsecretsmanager<br>          mountPath: /mnt/secrets<br>          readOnly: true<br><br># Superset beat configuration (to trigger scheduled jobs like reports)<br>supersetCeleryBeat:<br>  # -- This is only required if you intend to use alerts and reports<br>  enabled: true<br>  # -- List of init containers<br>  # @default -- a container waiting for postgres<br>  initContainers:<br>    - name: wait-for-postgres-redis<br>      image: &quot;{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}&quot;<br>      imagePullPolicy: &quot;{{ .Values.initImage.pullPolicy }}&quot;<br>      env:<br>        - name: DB_HOST<br>          valueFrom:<br>            secretKeyRef:<br>              key: host<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        - name: DB_PORT<br>          valueFrom:<br>            secretKeyRef:<br>              key: port<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        - name: REDIS_HOST<br>          value: &quot;{{ tpl .Values.extraEnv.REDIS_HOST . }}&quot;<br>        - name: REDIS_PORT<br>          value: &quot;{{ tpl .Values.extraEnv.REDIS_PORT . }}&quot;<br>      envFrom:<br>        - secretRef:<br>            name: &quot;{{ tpl .Values.envFromSecret . }}&quot;<br>      command:<br>        - /bin/sh<br>        - -c<br>        - dockerize -wait &quot;tcp://$DB_HOST:$DB_PORT&quot; -wait &quot;tcp://$REDIS_HOST:$REDIS_PORT&quot; -timeout 120s<br>      volumeMounts:<br>        - name: awareintegrationsecretsmanager<br>          mountPath: /mnt/secrets<br>          readOnly: true<br><br>init:<br>  loadExamples: false<br>  createAdmin: true<br>  adminUser:<br>    username: $SUPERSET_ADMIN_USERNAME<br>    firstname: Superset<br>    lastname: Admin<br>    email: &lt;admin_email&gt; # your admin email id<br>    password: $SUPERSET_ADMIN_PASSWORD<br>  # -- List of initContainers<br>  # @default -- a container waiting for postgres<br>  initContainers:<br>    - name: wait-for-postgres<br>      image: &quot;{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}&quot;<br>      imagePullPolicy: &quot;{{ .Values.initImage.pullPolicy }}&quot;<br>      env:<br>        - name: DB_HOST<br>          valueFrom:<br>            secretKeyRef:<br>              key: host<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>        - name: DB_PORT<br>          valueFrom:<br>            secretKeyRef:<br>              key: port<br>              name: superset-secret-csi-opaque # name of k8s secret, this should match with name provided in SecretProviderClass Object<br>      envFrom:<br>        - secretRef:<br>            name: &quot;{{ tpl .Values.envFromSecret . }}&quot;<br>      command:<br>        - /bin/sh<br>        - -c<br>        - dockerize -wait &quot;tcp://$DB_HOST:$DB_PORT&quot; -timeout 120s<br>      volumeMounts:<br>        - name: awareintegrationsecretsmanager<br>          mountPath: /mnt/secrets<br>          readOnly: true<br><br>postgresql:<br>  ##<br>  ## Use the PostgreSQL chart dependency.<br>  ## Set to false if bringing your own PostgreSQL.<br>  enabled: false<br><br>redis:<br>  ##<br>  ## Use the redis chart dependency.<br>  ##<br>  ## If you are bringing your own redis, you can set the host in supersetNode.connections.redis_host<br>  ##<br>  ## Set to false if bringing your own redis.<br>  enabled: false<br><br></pre><h3>Deploy Superset</h3><p>Run following command to genertae Helm template</p><pre># Generate template<br>helm template my-superset superset/superset -f superset-values.yaml</pre><p>Run following command to deploy superset. I am using “0.14.2” version</p><pre># Deploy superset version 0.14.2<br>helm upgrade --install my-superset superset/superset --namespace superset --create-namespace --version 0.14.2 -f superset-values.yaml</pre><p>Installation will take couple of minutes, because it creates init job which initialises superset.</p><p>Run following command to check the pods</p><pre># Check pod status<br>kubectl get po -n superset</pre><p>If everything works well you should see all pods in running state.</p><p>Run following command to check pod logs, In pod logs we will see that superset is using our cutom superset_config.py file</p><pre># Check pod logs<br>kubectl logs &lt;your_pod_name&gt; -f<br><br># OUTPUT<br>Defaulted container &quot;superset&quot; out of: superset, wait-for-postgres (init)<br>[2025-06-15 11:19:43 +0000] [8] [INFO] Starting gunicorn 22.0.0<br>[2025-06-15 11:19:43 +0000] [8] [INFO] Listening at: http://0.0.0.0:8088 (8)<br>[2025-06-15 11:19:43 +0000] [8] [INFO] Using worker: gthread<br>[2025-06-15 11:19:43 +0000] [9] [INFO] Booting worker with pid: 9<br>Loaded your LOCAL configuration at [/app/superset_config.py]<br>2025-06-15 11:19:47,792:INFO:superset.initialization:Setting database isolation level to READ COMMITTED<br>/usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitl<br>y specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring<br>the storage backend.<br>  warnings.warn(<br>2025-06-15 11:19:48,436:INFO:superset.utils.screenshots:No PIL installation found<br>2025-06-15 11:19:49,688:INFO:superset.utils.pdf:No PIL installation found</pre><p>You can see that it has this line<br><strong>“Loaded your LOCAL configuration at [/app/superset_config.py]” </strong><br>which is our custom superset_config.py file path.</p><h3>Port Forwarding superset UI</h3><p>If you haven’t configured Ingress you can access the superset UI by port forwarding superset service</p><p>Run following commands</p><pre># Get service name<br>kubectl get svc -n superset<br><br># you will see clusterIp service<br>NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE<br>my-superset   ClusterIP   10.100.80.33   &lt;none&gt;        8088/TCP   8m42s<br><br># Port Forward service<br>kubectl port-forward svc/my-superset --address 0.0.0.0 -n superset 8088:8088</pre><p>Open your browser and go to “http://localhost:8088” OR “http://&lt;your_ec2_public_IP&gt;:8088”</p><p>If you are using EC2 Instance make sure your EC2 Instance security group allows inbound connection on port “8088”</p><p>Login with superset admin username and password.</p><p><strong>Congratulations!!! You have successfully installed production grade apache superset.</strong></p><p><strong>Thank You!!!</strong></p><h4>References:</h4><ol><li><a href="https://github.com/apache/superset/tree/master/helm/superset">https://github.com/apache/superset/tree/master/helm/superset</a></li><li><a href="https://superset.apache.org/docs/configuration/configuring-superset">https://superset.apache.org/docs/configuration/configuring-superset</a></li></ol><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1aec192ac495" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/installing-apache-superset-in-kubernetes-1aec192ac495">Installing Apache Superset in Kubernetes</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes: Auto Restart Deployments when K8s ConfigMap or Secret is Updated]]></title>
            <link>https://aws.plainenglish.io/kubernetes-auto-restart-deployments-when-k8s-configmap-or-secret-is-updated-ad2c042a745f?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad2c042a745f</guid>
            <category><![CDATA[aws-secret-manager]]></category>
            <category><![CDATA[configmap]]></category>
            <category><![CDATA[reloader]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Sun, 19 May 2024 12:19:24 GMT</pubDate>
            <atom:updated>2026-03-18T07:42:58.891Z</atom:updated>
            <content:encoded><![CDATA[<h4>Auto Rollout Restart your Deployments when K8s ConfigMap or Secret is Updated</h4><p>Kubernetes Deployments are one of the most common resource in any Kubernetes cluster. We all run our pods using K8s Deployments to make sure it is highly available, pods are created automatically if it gets deleted and so on.</p><p>It is very common that Application always needs a config to run it smoothly across multiple environments. It may also need sensitive information such as database username, password etc. In Kubernetes you can use configmaps and secrets to store your application specific data and inject it into pod as env variable so that application can consume it.</p><p>What if you updated the value in configmap or secret? You need to restart your pod’s to take the latest value right? Or Rollout restart your deployments so that it creates new pods. Now, imagine you have hundred’s of deployments using some common configmap or secret and you updated the value and you need to make sure it uses latest value.</p><p>In our case we store our secrets in AWS Secrets Manager and we create kubernetes secrets from the AWS Secrets and Configuration Provider (ASCP) for the <a href="https://secrets-store-csi-driver.sigs.k8s.io/">Kubernetes Secrets Store CSI Driver</a></p><p>In this blog post I will explain how can you Automatically Rollout Restart your Kubernetes Deployment when Secret or ConfigMap is Updated.</p><h3>Architecture Diagram:</h3><figure><img alt="Kubernetes Auto Restart Deployment" src="https://cdn-images-1.medium.com/max/926/1*-LGDGlU3vmLqcFOYP_HKDA.png" /><figcaption>Kubernetes Auto Restart Deployment</figcaption></figure><h4>Let’s get started!</h4><h3>Pre-requisites:</h3><ol><li>EKS Cluster</li><li>OIDC Provider should be configured for EKS</li><li>Kubectl</li><li>AWS Cli</li></ol><h3>Step 1: Create IAM Roles and Policies for ASCP</h3><p>The ASCP retrieves the Amazon EKS pod identity and exchanges it for an IAM role. We set permissions in an IAM policy for that IAM role. When the ASCP assumes the IAM role, it gets access to the secrets we authorized. Other containers can’t access the secrets unless we also associate them with the IAM role.</p><ol><li>Create IAM policy Document. <br>create a file with name “<strong>secrets_policy”</strong> and add following content</li></ol><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;secretsmanager:DescribeSecret&quot;,<br>                &quot;secretsmanager:GetSecretValue&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>     ]<br>}</pre><p>2. Run following command to create IAM policy. <br>Take a Note of policy ARN, will need it while attaching policy with IAM role</p><pre>aws iam create-policy \<br>    --policy-name my-secret-manager-policy \<br>    --policy-document file://secrets_policy</pre><p>3. Create Trust Policy for IAM Role<br>Create file with name <strong>“trust_policy” </strong>and add following content. Make sure you replace correct values. The <strong>&lt;SERVICE_ACCOUNT_NAME&gt; </strong>can be anything but we need to use same name when we create actual service account in Kubernetes.</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Federated&quot;: &quot;arn:aws:iam::&lt;AWS_ACCOUNT_ID&gt;:oidc-provider/oidc.eks.&lt;AWS_REGION&gt;.amazonaws.com/id/&lt;OIDC_ID&gt;&quot;<br>      },<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;oidc.eks.&lt;AWS_REGION&gt;.amazonaws.com/id/&lt;OIDC_ID&gt;:aud&quot;: &quot;sts.amazonaws.com&quot;,<br>          &quot;oidc.eks.&lt;AWS_REGION&gt;.amazonaws.com/id/&lt;OIDC_ID&gt;:sub&quot;: &quot;system:serviceaccount:&lt;K8S_NAMESPACE&gt;:&lt;SERVICE_ACCOUNT_NAME&gt;&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>4. Run following command to create IAM Role</p><pre>aws iam create-role --role-name my-secret-manager-role<br>--assume-role-policy-document file://trust_policy</pre><p>5. Attach IAM policy to IAM Role</p><pre>aws iam attach-role-policy --policy-arn &lt;your_policy_arn&gt;<br>--role-name my-secret-manager-role</pre><p>We have created all the required IAM Roles and Policies.</p><h3>Step 2: Install and configure the ASCP</h3><p>Now we need to install 2 Helm charts.</p><ol><li>Install AWS Secrets and Configuration Provider (ASCP) chart</li></ol><pre># Add ASCP Helm Chart Repo<br>helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws<br><br># Install ASCP Helm Chart<br>helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws</pre><p>2. Install Secrets Store CSI Driver chart</p><ul><li>Add helm repo for Secrets Store CSI Driver chart</li></ul><pre># Add Secrets Store CSI Driver chart Repo<br>helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts</pre><ul><li>Get the default values</li></ul><pre># Get the default values<br>helm show values secrets-store-csi-driver/secrets-store-csi-driver &gt; secrets-store-csi-driver.yaml</pre><ul><li>Update following values in <strong>secrets-store-csi-driver.yaml </strong>file</li></ul><pre>## Install RBAC roles and bindings required for K8S Secrets syncing if true<br>syncSecret:<br>  enabled: true<br><br>## Enable secret rotation feature [alpha]<br>enableSecretRotation: true</pre><p>The above configuration will allow “secrets-store-csi-driver” to fetch the latest value from AWS Secret Manager and Update those values in Kubernetes Secrets Object.</p><p>The rotation-poll-interval is set to 2 minutes by default, however, it can be changed by setting up the property rotationPollInterval</p><ul><li>Install Helm Chart</li></ul><pre># Install Helm Chrt<br>helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver -f secrets-store-csi-driver.yaml</pre><h3>Step 3: Create Test Secret in AWS Secret Manager</h3><p>We will create a test secret in AWS Secret Manager</p><pre>aws secretsmanager create-secret \<br>    --name my-test-secret \<br>    --description &quot;My test secret created with the CLI.&quot; \<br>    --secret-string &quot;{\&quot;user\&quot;:\&quot;my-user\&quot;,\&quot;password\&quot;:\&quot;EXAMPLE-PASSWORD\&quot;}&quot;</pre><h3>Step 4: Create Kubernetes ServiceAccount</h3><p>Now we can create a ServiceAccount to allow the pods to assume the IAM role. This Service Account will be used by K8s Deployment.</p><p>Create a file with name <strong>serviceaccount.yaml</strong></p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: &lt;your_service_account_name&gt; # This name should match with the name you have given while creatig IAM trust policy<br>  annotations:<br>    eks.amazonaws.com/role-arn: &lt;IAM_ROLE_ARN&gt;</pre><p>Run following command to create service account in K8s.</p><pre>kubectl apply -f serviceaccount.yaml</pre><h3>Step 5: Create Test Objects</h3><p>Now we have deployed all the required resources. Let’s create test objects</p><ol><li>Create file with name “<strong>my-test-secret-manifest.yaml</strong>”</li></ol><pre>---<br>apiVersion: secrets-store.csi.x-k8s.io/v1<br>kind: SecretProviderClass<br>metadata:<br>  name: aws-secrets-providerclass<br>spec:<br>  provider: aws<br>  secretObjects:<br>    - secretName: my-test-k8s-secret<br>      type: Opaque<br>      data:<br>        - objectName: user<br>          key: user<br>        - objectName: password<br>          key: password<br>  parameters:<br>    objects: |<br>      - objectName: arn:aws:secretsmanager:&lt;AWS_REGION&gt;:&lt;AWS_ACCOUNT_ID&gt;:secret:my-test-secret<br>        jmesPath:<br>          - path: user<br>            objectAlias: user<br>          - path: password<br>            objectAlias: password<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: secret-rotation-test-ubuntu-deployment<br>  labels:<br>    app: ubuntu<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: ubuntu<br>  template:<br>    metadata:<br>      labels:<br>        app: ubuntu<br>    spec:<br>      serviceAccountName: &lt;your_service_account_name&gt; # This name should match with the service account name you created in step4<br>      volumes:<br>      - name: mount-secrets-access<br>        csi:<br>          driver: secrets-store.csi.k8s.io<br>          readOnly: true<br>          volumeAttributes:<br>            secretProviderClass: &quot;aws-secrets-providerclass&quot;<br>      containers:<br>      - name: ubuntu<br>        image: ubuntu<br>        command: [&quot;sleep&quot;, &quot;123456&quot;]<br>        env:<br>        - name: USER<br>          valueFrom:<br>            secretKeyRef:<br>              name: my-test-k8s-secret<br>              key: user<br>        - name: PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: my-test-k8s-secret<br>              key: password<br>        volumeMounts:<br>        - name: mount-secrets-access<br>          mountPath: &quot;/mnt/aws-secrets&quot;<br>          readOnly: true</pre><p>2. Apply the manifest</p><pre>kubectl apply -f my-test-secret-manifest.yaml</pre><p>3. It will Create following resources</p><ul><li>SecretProviderClass Resource -&gt; It will get the data from AWS Secret Manager and create K8s Secret</li><li>Deployment with Volume mount -&gt; We have to mount SecretProviderClass as a volume</li><li>Kubernetes Secret -&gt; It is injected as a env variable inside a pod</li></ul><p>The following diagram is a good visualization of what is happening behind the scenes when you apply the above YAML</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/0*fg9Z2iA0HiWYwEML" /><figcaption><a href="https://blog.bootlabstech.com/aws-secrets-manager-in-kubernetes-secret-rotation-and-reloader">https://blog.bootlabstech.com/aws-secrets-manager-in-kubernetes-secret-rotation-and-reloader</a></figcaption></figure><p>4. Verify that everything is deployed</p><pre># verify SecretProviderClass<br>kubectl get SecretProviderClass aws-secrets-providerclass -o yaml<br><br># Verify Deployment<br>kubectl get deploy secret-rotation-test-ubuntu-deployment -o yaml<br><br># Verify pod<br>kubectl get po &lt;pod_name&gt; -o yaml<br><br># Get secret<br>kubectl get secret my-test-k8s-secret -o yaml</pre><h3>Step 6: Install Reloader</h3><p><a href="https://github.com/stakater/Reloader">Reloader</a> can watch changes in ConfigMap and Secret and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, Daemonsets Statefulsets and Rollouts</p><ol><li>Add Reloader Helm Repo</li></ol><pre># Add Helm Repo<br>helm repo add stakater https://stakater.github.io/stakater-charts</pre><p>2. Get Default values</p><pre># Get Default values<br>helm show values stakater/reloader &gt; reloader.yaml</pre><p>3. Update <strong>reloader.yaml</strong> file</p><pre>reloader:<br>  # Set to true to enable leadership election allowing you to run multiple replicas<br>  enableHA: true<br>  deployment:<br>    # If you wish to run multiple replicas set reloader.enableHA = true<br>    replicas: 2</pre><p>4. Install Helm Chart</p><pre># Install Helm Chart<br>helm install reloader -f reloader.yaml stakater/reloader -n kube-system</pre><h3>Step 7: Test with Secret Update</h3><ol><li>Reloader works on the annotation. <br>The default annotation reloader.stakater.com/auto should be present to main metadata of our Deployment Use following command to annotate our deployment.</li></ol><pre># Add annotation on deployment<br>kubectl annotate deployment secret-rotation-test-ubuntu-deployment &quot;reloader.stakater.com/auto=true&quot;</pre><p>OR you can also edit our deployment file with following block and apply it.</p><pre>metadata:<br>  annotations:<br>    reloader.stakater.com/auto: &quot;true&quot;</pre><p>2. Update Secret in AWS Secret Manager</p><pre>aws secretsmanager put-secret-value \<br>      --secret-id my-test-secret \<br>      --secret-string &quot;{\&quot;user\&quot;:\&quot;diegor\&quot;,\&quot;password\&quot;:\&quot;SAMPLE-PASSWORD\&quot;}&quot;</pre><p>3. Once the secret is updated in AWS secrets-store-csi-driver will get the latest value and update the K8s secret as soon as K8s secret is updated Reloader will trigger the rollout restart of deployment.</p><pre># Check K8s Secret, it should have new value<br>kubectl get secret my-test-k8s-secret -o yaml<br><br># Check the pod, it should have started few seconds ago<br>kubectl get po<br><br># Check logs of reloader pod<br>kubectl logs &lt;reloader-pod-name&gt; -n kube-system<br><br># exec into pod and verify new values<br>kubectl exec -it &lt;pod_name&gt; -- bash<br><br># Once you are into pod run `env` command, it will print all env variables available in pod<br></pre><h3>Step 8: Test with ConfigMap Update</h3><ol><li>Create a file with name “<strong>my-test-cm-manifest.yaml</strong>”</li></ol><pre>---<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: my-test-k8s-cm<br>data:<br>  myvalue: &quot;Hello World&quot;<br>  drink: coffee<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: reloader-poc-ubuntu-deployment<br>  labels:<br>    app: ubuntu<br>  annotations:<br>    reloader.stakater.com/auto: &quot;true&quot;<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: ubuntu<br>  template:<br>    metadata:<br>      labels:<br>        app: ubuntu<br>    spec:<br>      containers:<br>      - name: ubuntu<br>        image: ubuntu<br>        command: [&quot;sleep&quot;, &quot;123456&quot;]<br>        env:<br>        - name: DRINK<br>          valueFrom:<br>            configMapKeyRef:<br>              name: my-test-k8s-cm<br>              key: drink<br>        - name: MYVALUE<br>          valueFrom:<br>            configMapKeyRef:<br>              name: my-test-k8s-cm<br>              key: myvalue<br></pre><p>2. Apply manifest</p><pre>kubectl apply -f my-test-cm-manifest.yaml</pre><p>3. Update the configmap in my-test-cm-manifest.yaml</p><pre>---<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: my-test-k8s-cm<br>data:<br>  myvalue: &quot;Hello World&quot;<br>  drink: tea<br></pre><p>4. Re-apply the file</p><pre>kubectl apply -f my-test-cm-manifest.yaml</pre><p>5. Verify</p><pre># Check configmap<br>kubectl get cm my-test-k8s-cm -o yaml<br><br># Check Pod<br>kubectl get po<br><br># exec into pod and verify new values<br>kubectl exec -it &lt;pod_name&gt; -- bash<br><br># Once you are into pod run `env` command, it will print all env variables available in pod</pre><p><strong>Congratulations!!! You have successfully configured secret-store-csi-driver and reloader.</strong></p><p><strong>Thank You!!!</strong></p><h4>References:</h4><ol><li><a href="https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html">https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html</a></li><li><a href="https://blog.bootlabstech.com/aws-secrets-manager-in-kubernetes-secret-rotation-and-reloader">https://blog.bootlabstech.com/aws-secrets-manager-in-kubernetes-secret-rotation-and-reloader</a></li><li><a href="https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation">https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation</a></li><li><a href="https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/charts/secrets-store-csi-driver/values.yaml">https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/charts/secrets-store-csi-driver/values.yaml</a></li><li><a href="https://github.com/stakater/Reloader/tree/master">https://github.com/stakater/Reloader/tree/master</a></li><li><a href="https://github.com/stakater/Reloader/blob/master/docs/Verify-Reloader-Working.md">https://github.com/stakater/Reloader/blob/master/docs/Verify-Reloader-Working.md</a></li><li><a href="https://github.com/stakater/Reloader/blob/master/docs/How-it-works.md">https://github.com/stakater/Reloader/blob/master/docs/How-it-works.md</a></li></ol><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>Tired of blogging platforms that force you to deal with algorithmic content? Try <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad2c042a745f" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/kubernetes-auto-restart-deployments-when-k8s-configmap-or-secret-is-updated-ad2c042a745f">Kubernetes: Auto Restart Deployments when K8s ConfigMap or Secret is Updated</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KEDA: Kubernetes HPA Based on External Metrics from Prometheus]]></title>
            <link>https://aws.plainenglish.io/keda-kubernetes-hpa-based-on-external-metrics-from-prometheus-3bb334c0ccf0?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bb334c0ccf0</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[keda]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 13:56:36 GMT</pubDate>
            <atom:updated>2026-03-18T07:43:29.107Z</atom:updated>
            <content:encoded><![CDATA[<h4>Smart way of scaling kubernetes resources like deployments, statefulset etc.</h4><p>Horizontal Pod Autoscaler (HPA) is a key feature in Kubernetes that automates the scaling of pods in a deployment based on predefined metrics. While Kubernetes HPA supports scaling based on built-in metrics like CPU and Memory utilization, there is a need for effective utilization of HPA with external metrics.</p><p><strong>In this blog post we will see how can we use KEDA to scale our Deployments, StatefulSet etc based on external metrics from Prometheus.</strong></p><h3>Architecture Diagram:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*io2Uqs9pLEUBZ-i-gBZeeQ.png" /><figcaption>KEDA: HPA based on External Metric from prometheus</figcaption></figure><h4>AWS metrics into prometheus</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RpcMIIX_qfpuyGmG" /><figcaption>CloudWatch Exporter data into prometheus</figcaption></figure><p>As you can see in the First diagram that we will use prometheus as an external trigger source. Data in prometheus can come from two sources</p><ol><li>Application specific metric</li><li>Data from AWS cloudwatch.</li></ol><p>In this blog post we will deploy <strong>YACE CloudWatch Exporter</strong> to get data from <strong>AWS CloudWatch</strong> which exports data into <strong>prometheus format</strong> and once we have data in prometheus we can use any available metrics to scale our deployment.</p><h3>Create EKS Cluster:</h3><p>we will be using AWS EKS cluster for this POC. If you already have kubernetes cluster running you can use it. We will be creating EKS cluster using eksctl utility. I have installed eksctl, kubectl , awscli and helm on EC2 instance and my EC2 Instance has Admin permissions.</p><pre>eksctl create cluster --name harsh-test-cluster --region ap-south-1 --vpc-cidr 10.0.0.0/16 --version 1.27 --instance-types t3.medium --enable-ssm --node-volume-size 30 --nodes 3 --full-ecr-access --alb-ingress-access --asg-access</pre><p>Once EKS cluster is created make sure you are able to connect to it, you can use following command to make sure you can connect to EKS cluster</p><pre>kubectl get svc</pre><pre>#Output:<br>NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE<br>kubernetes   ClusterIP   172.20.0.1   &lt;none&gt;        443/TCP   6h</pre><h3>IAM OIDC Provider for EKS:</h3><h4>To create an IAM OIDC identity provider for your cluster with eksctl</h4><ol><li>Determine whether you have an existing IAM OIDC provider for your cluster.</li></ol><pre>export cluster_name=harsh-test-cluster<br>oidc_id=$(aws eks describe-cluster --name $cluster_name --query &quot;cluster.identity.oidc.issuer&quot; --output text | cut -d &#39;/&#39; -f 5)</pre><p>2. Determine whether an IAM OIDC provider with your cluster’s ID is already in your account.</p><pre>aws iam list-open-id-connect-providers | grep $oidc_id | cut -d &quot;/&quot; -f4</pre><p>If output is returned, then you already have an IAM OIDC provider for your cluster and you can skip the next step. If no output is returned, then you must create an IAM OIDC provider for your cluster.</p><p>3. Create an IAM OIDC identity provider for your cluster with the following command.</p><pre>eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve</pre><h3>IAM Role permissions for YACE</h3><p>In 2nd Architecture diagram you can see that YACE CloudWatch exporter will be collecting data from AWS CloudWatch service. We will be using <strong>IAM Role for Service Account (IRSA) </strong>for allowing YACE CloudWatch exporter to get data from AWS CloudWatch.</p><p>Set your AWS account ID to an environment variable with the following command.</p><pre>account_id=$(aws sts get-caller-identity --query &quot;Account&quot; --output text)</pre><p>Set your cluster’s OIDC identity provider to an environment variable with the following command. Replace <em>harsh-test-cluster</em> with the name of your cluster.</p><pre>export AWS_REGION=&#39;ap-south-1&#39; # Your EKS cluster region<br>oidc_provider=$(aws eks describe-cluster --name harsh-test-cluster --region $AWS_REGION --query &quot;cluster.identity.oidc.issuer&quot; --output text | sed -e &quot;s/^https:\/\///&quot;)</pre><p>Set variables for the namespace and name of the service account. We will deploy YACE in default namespace and we will configure YACE to create service account named yace-cw-exporter</p><pre>export namespace=default<br>export service_account=yace-cw-exporter</pre><p>Run the following command to create a trust policy file for the IAM role.</p><pre>cat &gt;trust-relationship.json &lt;&lt;EOF<br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Federated&quot;: &quot;arn:aws:iam::$account_id:oidc-provider/$oidc_provider&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;$oidc_provider:aud&quot;: &quot;sts.amazonaws.com&quot;,<br>          &quot;$oidc_provider:sub&quot;: &quot;system:serviceaccount:$namespace:$service_account&quot;<br>        }<br>      }<br>    }<br>  ]<br>}<br>EOF</pre><p>Create the role.</p><pre>aws iam create-role --role-name cloudwatch-exporter-role --assume-role-policy-document file://trust-relationship.json --description &quot;cloudwatch-exporter-role&quot;</pre><p>Attach managed IAM policy to role.</p><pre># Attach cloudwatch exporter read only policy<br>aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess --role-name cloudwatch-exporter-role<br><br># Attch Resource Group and tagging read only policy<br>aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/ResourceGroupsandTagEditorReadOnlyAccess --role-name cloudwatch-exporter-role</pre><h3>Install kube-prometheus-stack:</h3><p>We will use <strong>kube-prometheus-stack</strong> helm chart to install prometheus. it used prometheus operator to install prometheus and it has custom CRD’s to read data exported in prometheus format.</p><p>Create monitoring namespace</p><pre>kubectl create ns monitoring</pre><p>Add prometheus-community repository</p><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts</pre><p>Get default values file</p><pre>helm show values prometheus-community/kube-prometheus-stack &gt; kube-prometheus-stack-values.yaml</pre><p>Update following in kube-prometheus-stack-values.yaml file</p><pre>prometheus:<br>  prometheusSpec:<br>    ruleSelectorNilUsesHelmValues: false<br>    serviceMonitorSelectorNilUsesHelmValues: false<br>    podMonitorSelectorNilUsesHelmValues: false<br>    probeSelectorNilUsesHelmValues: false</pre><p>default value for these 4 entries is: true which adds the “release”:”prometheus” as a selector as default.</p><p>Install Prometheus on Kubernetes</p><pre>helm install -f kube-prometheus-stack-values.yaml prometheus prometheus-community/kube-prometheus-stack -n monitoring</pre><p>Verify helm chart is deployed</p><pre>helm list -n monitoring<br><br># Output<br>NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION<br>prometheus      monitoring      1               2023-10-14 12:05:37.8830938 +0000 UTC   deployed        kube-prometheus-stack-51.4.0    v0.68.0</pre><p>Access prometheus UI using port-forwarding</p><pre>kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring</pre><p>Congratulations!!! You have successfully deployed prometheus.</p><h3>Create SQS Queue</h3><p>In this blog we will scale our deployment based on the number of messages in AWS SQS queue.</p><p>Run following command to create test SQS queue</p><pre>aws sqs create-queue --queue-name &lt;your_queue_name&gt;</pre><p>Send some sample messages</p><pre>aws sqs send-message --queue-url &lt;your_queue_url&gt; --message-body &quot;my test message 0&quot;<br><br>aws sqs send-message --queue-url &lt;your_queue_url&gt; --message-body &quot;my test message 1&quot;<br><br>aws sqs send-message --queue-url &lt;your_queue_url&gt; --message-body &quot;my test message 2&quot;</pre><h3>Install KEDA</h3><p>KEDA is a Kubernetes-based event-driven autoscaling framework that allows you to scale workloads based on external events, such as messages from a message queue or events from an event source. KEDA provides a wide range of event sources and supports custom metrics, making it a flexible option for autoscaling based on external metrics. KEDA can work in conjunction with HPA, allowing you to use external metrics from event sources to trigger scaling actions with HPA.</p><p>We will installing KEDA using Helm. Add kedacore helm repo</p><pre>helm repo add kedacore https://kedacore.github.io/charts</pre><p>Update Helm repo</p><pre>helm repo update</pre><p>Install keda Helm chart</p><pre>helm install keda kedacore/keda --namespace keda --create-namespace</pre><p>Verify KEDA Operator is deployed</p><pre>kubectl get po -n keda<br><br>kubectl get svc -n keda</pre><p>You can list the installed helm chart using following command</p><pre>helm list -n keda<br><br># Output<br>NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION<br>keda    keda            1               2023-10-14 13:52:07.0941042 +0000 UTC   deployed        keda-2.12.0     2.12.0</pre><p>Congratulations!!! You have successfully installed KEDA Operator.</p><h3>Install YACE CloudWatch Exporter</h3><p>We will use YACE CloudWatch Exporter to get data from AWS CloudWatch and it will export data into prometheus format. We will enable servicemonitor custom resource so that prometheus can read exported metrics from the application.</p><p>Add Helm Repo</p><pre>helm repo add nerdswords https://nerdswords.github.io/helm-charts</pre><p>Now we will get the default chart values and we need to do some modification to give <strong>AWS CloudWatch access using IRSA</strong> and create <strong>servicemonitor</strong> custom resource</p><p>Get the default chart values</p><pre>helm show values nerdswords/yet-another-cloudwatch-exporter &gt; yace-cw-exporter-values.yaml</pre><p>Open yace-cw-exporter-values.yaml and make following changes</p><pre>serviceAccount:<br>  # -- Specifies whether a service account should be created<br>  create: true<br>  # -- Labels to add to the service account<br>  labels: {}<br>  # -- Annotations to add to the service account<br>  annotations:<br>    # add annotation for IRSA<br>    eks.amazonaws.com/role-arn: arn:aws:iam::&lt;your_aws_account_id&gt;:role/&lt;your_role_name&gt;<br>  # -- The name of the service account to use.<br>  # If not set and create is true, a name is generated using the fullname template<br>  name: &quot;yace-cw-exporter&quot;</pre><pre>serviceMonitor:<br>  # When set true then use a ServiceMonitor to configure scraping<br>  enabled: true</pre><pre>config: |-<br>  apiVersion: v1alpha1<br>  sts-region: ap-south-1<br>  discovery:<br>    exportedTagsOnMetrics:<br>      AWS/SQS:<br>        - Name<br>        - env<br>    jobs:<br>    - type: AWS/SQS<br>      regions:<br>        - ap-south-1<br>      period: 60<br>      length: 60<br>      metrics:<br>        - name: NumberOfMessagesSent<br>          statistics: [Sum,Average]<br>        - name: NumberOfMessagesReceived<br>          statistics: [Sum,Average]<br>        - name: NumberOfMessagesDeleted<br>          statistics: [Sum,Average]<br>        - name: ApproximateAgeOfOldestMessage<br>          statistics: [Sum,Average]<br>        - name: NumberOfEmptyReceives<br>          statistics: [Sum,Average]<br>        - name: SentMessageSize<br>          statistics: [Sum,Average]<br>        - name: ApproximateNumberOfMessagesNotVisible<br>          statistics: [Sum,Average]<br>        - name: ApproximateNumberOfMessagesDelayed<br>          statistics: [Sum,Average]<br>        - name: ApproximateNumberOfMessagesVisible<br>          statistics: [Sum,Average]</pre><p><strong>The above config is very important. The YACE CloudWatch Exporter will get the metrics of the service defined in the config. So, you have to add all the required AWS services and their metrics in the config.</strong></p><p>For the purpose of this blog I am only getting metrics of AWS SQS queues in ap-south-1 region. All the configuration options are available <a href="https://github.com/nerdswords/yet-another-cloudwatch-exporter/tree/master/docs">here</a> and example configs are available <a href="https://github.com/nerdswords/yet-another-cloudwatch-exporter/tree/master/examples">here</a></p><p>You can change scrape interval default is <strong>300s (5 min) </strong>and enable some extra features.</p><pre>extraArgs:<br>  scraping-interval: 120<br>  enable-feature: max-dimensions-associator, list-metrics-callback</pre><p>Deploy the YACE CloudWatch Exporter Helm Chart</p><pre>helm install -f yace-cw-exporter-values.yaml yace-cw-exporter nerdswords/yet-another-cloudwatch-exporter</pre><p>Verify that pods are running</p><pre># get the pod name<br>kubectl get po<br><br># get pod logs<br>kubectl logs &lt;your_yace_pod_name&gt;<br><br># You should see logs something like this<br>{&quot;caller&quot;:&quot;main.go:211&quot;,&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;Parsing config&quot;,&quot;ts&quot;:&quot;2023-10-14T15:24:40.270520567Z&quot;,&quot;version&quot;:&quot;v0.55.0&quot;}<br>{&quot;caller&quot;:&quot;main.go:290&quot;,&quot;feature_flags&quot;:&quot;&quot;,&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;Yace startup completed&quot;,&quot;ts&quot;:&quot;2023-10-18T15:24:40.271390901Z&quot;,&quot;version&quot;:&quot;v0.55.0&quot;}</pre><p>Verify YACE CW Exporter is able to get metrics</p><pre>kubectl port-forward &lt;your_yace_pod_name&gt; 5000:5000<br><br># Open 127.0.0.1:5000 in your browser and click on Metrics</pre><p>Verify metrics are available in prometheus</p><pre>kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring<br><br># Open browser and type 127.0.0.1:9090 and you should see prometheus UI</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6wL5BnpfyXbdY9lmjJ5Ng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rnkn8UwQPkEQFhfQPcg7dw.png" /></figure><p>Congratulations!!! You have successfully installed YACE CloudWatch Exporter and AWS metrics are now available into prometheus.</p><h3>Sample Helm Chart</h3><p>Now that our AWS metrics are available into prometheus we can use KEDA to scale our deployment based on <strong>ApproximateNumberOfMessagesVisible </strong>(The number of messages to be processed)</p><p>Let’s create basic Kubernetes resources with Helm.</p><p>Create a folder for storing Helm related files</p><pre>mkdir keda-sample<br><br>cd keda-kample<br><br>mkdir helmfile<br><br>cd helmfile</pre><p>Create Chart.yaml file and add following content</p><pre>apiVersion: v2<br>name: keda-sample<br>description: A Helm chart for Kubernetes<br>type: application<br>version: 1.0.0<br>appVersion: &quot;1.0.0&quot;<br></pre><p>Create values.yaml file</p><pre># Default values for keda-sample.<br># This is a YAML-formatted file.<br># Declare variables to be passed into your templates.<br><br>replicaCount: 1<br><br>image:<br>  repository: nginx<br>  pullPolicy: Always<br>  # Overrides the image tag whose default is the chart appVersion.<br>  tag: &quot;latest&quot;<br><br>service:<br>  type: ClusterIP<br>  port: 80<br><br>resources:<br>  limits:<br>    memory: 256Mi<br>  requests:<br>    cpu: 100m<br>    memory: 128Mi<br><br>keda:<br>  enabled: true<br>  pollingInterval: 30 # This is the interval to check each trigger on. By default KEDA will check each trigger source on every ScaledObject every 30 seconds.<br>  cooldownPeriod: 60 # The period to wait after the last trigger reported active before scaling the resource back to 0<br>  minReplicaCount: 1 # Minimum number of replicas KEDA will scale the resource down to.<br>  maxReplicaCount: 6 # This setting is passed to the HPA definition that KEDA will create for a given resource and holds the maximum number of replicas of the target resource.<br>  fallback:<br>    failureThreshold: 3<br>    replicas: 4<br>  triggers:<br>    prometheus:<br>      serverAddress: &quot;http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090&quot; # Address of prometheus server<br>      metricName: aws_sqs_approximate_number_of_messages_visible_average # This can be any name. # Note: query must return a vector/scalar single element response<br>      query: aws_sqs_approximate_number_of_messages_visible_average{dimension_QueueName=&quot;harsh-test-queue&quot;} # This will be your prometheus query<br>      threshold: &quot;10.00&quot;</pre><p>In this values file we have added few variables which we will use while creating <strong>ScaledObject</strong> custom resource provided by KEDA to trigger our deployment when <strong>threshold </strong>is crossed.</p><p>Create templates folder to store Kubernetes resource related files</p><pre>mkdir templates<br><br>cd templates</pre><p>Create file _helpers.tpl</p><pre>{{/*<br>Expand the name of the chart.<br>*/}}<br>{{- define &quot;keda-sample.name&quot; -}}<br>{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix &quot;-&quot; }}<br>{{- end }}<br><br>{{/*<br>Create a default fully qualified app name.<br>We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).<br>If release name contains chart name it will be used as a full name.<br>*/}}<br>{{- define &quot;keda-sample.fullname&quot; -}}<br>{{- if .Values.fullnameOverride }}<br>{{- .Values.fullnameOverride | trunc 63 | trimSuffix &quot;-&quot; }}<br>{{- else }}<br>{{- $name := default .Chart.Name .Values.nameOverride }}<br>{{- if contains $name .Release.Name }}<br>{{- .Release.Name | trunc 63 | trimSuffix &quot;-&quot; }}<br>{{- else }}<br>{{- printf &quot;%s-%s&quot; .Release.Name $name | trunc 63 | trimSuffix &quot;-&quot; }}<br>{{- end }}<br>{{- end }}<br>{{- end }}<br><br>{{/*<br>Create chart name and version as used by the chart label.<br>*/}}<br>{{- define &quot;keda-sample.chart&quot; -}}<br>{{- printf &quot;%s-%s&quot; .Chart.Name .Chart.Version | replace &quot;+&quot; &quot;_&quot; | trunc 63 | trimSuffix &quot;-&quot; }}<br>{{- end }}<br><br>{{/*<br>Common labels<br>*/}}<br>{{- define &quot;keda-sample.labels&quot; -}}<br>helm.sh/chart: {{ include &quot;keda-sample.chart&quot; . }}<br>{{ include &quot;keda-sample.selectorLabels&quot; . }}<br>{{- if .Chart.AppVersion }}<br>app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}<br>{{- end }}<br>app.kubernetes.io/managed-by: {{ .Release.Service }}<br>{{- end }}<br><br>{{/*<br>Selector labels<br>*/}}<br>{{- define &quot;keda-sample.selectorLabels&quot; -}}<br>app.kubernetes.io/name: {{ include &quot;keda-sample.name&quot; . }}<br>app.kubernetes.io/instance: {{ .Release.Name }}<br>{{- end }}<br><br>{{/*<br>Create the name of the service account to use<br>*/}}<br>{{- define &quot;keda-sample.serviceAccountName&quot; -}}<br>{{- if .Values.serviceAccount.create }}<br>{{- default (include &quot;keda-sample.fullname&quot; .) .Values.serviceAccount.name }}<br>{{- else }}<br>{{- default &quot;default&quot; .Values.serviceAccount.name }}<br>{{- end }}<br>{{- end }}</pre><p>Create file deployment.yaml</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: {{ include &quot;keda-sample.fullname&quot; . }}<br>  labels:<br>    {{- include &quot;keda-sample.labels&quot; . | nindent 4 }}<br>spec:<br>  replicas: {{ .Values.replicaCount }}<br>  selector:<br>    matchLabels:<br>      {{- include &quot;keda-sample.selectorLabels&quot; . | nindent 6 }}<br>  template:<br>    metadata:<br>      labels:<br>        {{- include &quot;keda-sample.labels&quot; . | nindent 8 }}<br>    spec:<br>      containers:<br>        - name: {{ .Chart.Name }}<br>          securityContext:<br>            {{- toYaml .Values.securityContext | nindent 12 }}<br>          image: &quot;{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.Version }}&quot;<br>          imagePullPolicy: {{ .Values.image.pullPolicy }}<br>          ports:<br>            - name: http<br>              containerPort: {{ .Values.service.port }}<br>              protocol: TCP<br>          resources:<br>            {{- toYaml .Values.resources | nindent 12 }}</pre><p>Create file service.yaml</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: {{ include &quot;keda-sample.fullname&quot; . }}<br>  labels:<br>    {{- include &quot;keda-sample.labels&quot; . | nindent 4 }}<br>spec:<br>  type: {{ .Values.service.type }}<br>  ports:<br>    - port: {{ .Values.service.port }}<br>      targetPort: http<br>      protocol: TCP<br>      name: http<br>  selector:<br>    {{- include &quot;keda-sample.selectorLabels&quot; . | nindent 4 }}</pre><p>Now the most important file scaledobject.yaml</p><pre>{{- if .Values.keda.enabled -}}<br>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: {{ include &quot;keda-sample.fullname&quot; . }}<br>  labels:<br>    {{- include &quot;keda-sample.labels&quot; . | nindent 4 }}<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: {{ include &quot;keda-sample.fullname&quot; . }}<br>  pollingInterval: {{ .Values.keda.pollingInterval }}<br>  cooldownPeriod: {{ .Values.keda.cooldownPeriod }}<br>  minReplicaCount: {{ .Values.keda.minReplicaCount }}<br>  maxReplicaCount: {{ .Values.keda.maxReplicaCount }}<br>  {{- with .Values.keda.fallback }}<br>  fallback:<br>    {{- toYaml . | nindent 4 }}<br>  {{- end }}<br>  triggers:<br>    - type: prometheus<br>      metadata:<br>        serverAddress: {{ .Values.keda.triggers.prometheus.serverAddress }}<br>        metricName: {{ .Values.keda.triggers.prometheus.metricName }}<br>        query: {{ tpl .Values.keda.triggers.prometheus.query . }}<br>        threshold: {{ .Values.keda.triggers.prometheus.threshold | quote }}<br>{{- end -}}</pre><p><strong>Important configurations:</strong></p><ul><li>spec.scaleTargetRef.name — Name of the deployment to scale</li><li>spec.triggers.metadata.serverAddress — Address of prometheus server</li><li>spec.triggers.metadata.query — Prometheus Query (query must return a vector/scalar single element response)</li></ul><p>You can read <a href="https://keda.sh/docs/2.12/concepts/scaling-deployments/">here</a> for more configuration options of ScaledObject and configuration options for prometheus trigger <a href="https://keda.sh/docs/2.12/scalers/prometheus/">here</a></p><p>Generate local helm template</p><pre>helm template &lt;path_to_Chart.yaml&gt;</pre><p>Install keda-sample helm chart</p><pre>helm install keda-sample &lt;path_to_Chart.yaml&gt;</pre><p>Verify required resources are created</p><pre># Get pod<br>kubectl get po<br><br># Get service<br>kubectl get svc<br><br># Get scaledobject<br>kubectl get scaledobject<br><br># Get HPA<br>kubectl get hpa</pre><p>You might have observed in Architecture diagram that KEDA internally creates native Kubernetes HPA resource.</p><p>Verify that scaledobject is able to connect to prometheus server</p><pre>kubectl describe scaledobject keda-sample</pre><pre># If you see following events it means that scaledobject is able to get data from prometheus<br>Status:<br>  Conditions:<br>    Message:  ScaledObject is defined correctly and is ready for scaling<br>    Reason:   ScaledObjectReady<br>    Status:   True<br>    Type:     Ready<br>    Message:  Scaling is performed because triggers are active<br>    Reason:   ScalerActive<br>    Status:   True<br>    Type:     Active<br>    Message:  No fallbacks are active on this scaled object<br>    Reason:   NoFallbackFound<br>    Status:   False<br>    Type:     Fallback<br>    Status:   Unknown<br>    Type:     Paused</pre><p>Verify that HPA is able to successfully calculate a replica count from external metric</p><pre>kubectl get hpa keda-hpa-keda-sample -o yaml</pre><pre># If you see following events it means that hpa is able to calculate a replica count from external metric<br>status:<br>  conditions:<br>  - lastTransitionTime: &quot;2023-10-19T15:31:21Z&quot;<br>    message: recommended size matches current size<br>    reason: ReadyForNewScale<br>    status: &quot;True&quot;<br>    type: AbleToScale<br>  - lastTransitionTime: &quot;2023-10-19T15:31:21Z&quot;<br>    message: &#39;the HPA was able to successfully calculate a replica count from external<br>      metric s0-prometheus(&amp;LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name:<br>      keda-sample,},MatchExpressions:[]LabelSelectorRequirement{},})&#39;<br>    reason: ValidMetricFound<br>    status: &quot;True&quot;<br>    type: ScalingActive<br>  - lastTransitionTime: &quot;2023-10-19T15:31:21Z&quot;<br>    message: the desired count is within the acceptable range<br>    reason: DesiredWithinRange<br>    status: &quot;False&quot;<br>    type: ScalingLimited</pre><p>One more observation is that HPA is owned by keda.sh/v1alpha1 API.</p><pre>  ownerReferences:<br>  - apiVersion: keda.sh/v1alpha1<br>    blockOwnerDeletion: true<br>    controller: true<br>    kind: ScaledObject<br>    name: keda-sample</pre><p><strong>Note: Any changes related to HPA configuration should be done in “ScaledObject” resource.</strong></p><h3>Test ScaledObject</h3><p>Now that everything is ready it is time to test the <strong>scaledObject</strong>. So, we will be sending some more sample messages in our SQS queue so that it crosses the threshold of “10.00” (in our example) and we should see new pod.</p><p>Send some test messages into queue</p><pre>for i in $(seq 1 15); do aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue --message-body &quot;Information about the largest city in Any Region.&quot;; done;</pre><p>Once <strong>Data is available in Prometheus (because for KEDA trigger source is prometheus) </strong>you will see that HPA will be in action and it will scale-out the deployment. You can describe the events to verify scale-out.</p><pre>kubectl describe hpa keda-hpa-keda-sample</pre><pre>  Type            Status  Reason              Message<br>  ----            ------  ------              -------<br>  AbleToScale     True    ReadyForNewScale    recommended size matches current size<br>  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-prometheus(&amp;LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: keda-sample,},MatchExpressions:[]LabelSelectorRequirement{},})<br>  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range<br>Events:<br>  Type    Reason             Age   From                       Message<br>  ----    ------             ----  ----                       -------<br>  Normal  SuccessfulRescale  95s   horizontal-pod-autoscaler  New size: 2; reason: external metric s0-prometheus(&amp;LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: keda-sample,},MatchExpressions:[]LabelSelectorRequirement{},}) above target</pre><pre>kubectl get po <br><br># Output<br>NAME                                                              READY   STATUS    RESTARTS   AGE<br>keda-sample-766fd46d8-c26pv                                       1/1     Running   0          2m57s<br>keda-sample-766fd46d8-dsq6s                                       1/1     Running   0          35m</pre><p>Let’s delete few messages from SQS queue.</p><pre>QUEUE_URL=&lt;your_queue_url&gt;<br><br>(MSG=$(aws sqs receive-message --queue-url $QUEUE_URL) --max-number-of-messages 10; \<br> [ ! -z &quot;$MSG&quot;  ] &amp;&amp; echo &quot;$MSG&quot; | jq -r &#39;.Messages[] | .ReceiptHandle&#39; | \<br>  (xargs -I {} aws sqs delete-message --queue-url $QUEUE_URL --receipt-handle {}) &amp;&amp; \<br> echo &quot;$MSG&quot; | jq -r &#39;.Messages[] | .Body&#39;)</pre><p>Once <strong>Data is available in Prometheus (because for KEDA trigger source is prometheus)</strong> you will see that HPA will not immediately scale-in the deployment. This is because default value of <strong>stabilizationWindowSeconds </strong>is 300. You can modify this setting in scaledObject configuration.</p><pre>kubectl describe hpa keda-hpa-keda-sample<br><br>#Output:<br>Conditions:<br>  Type            Status  Reason              Message<br>  ----            ------  ------              -------<br>  AbleToScale     True    ReadyForNewScale    recommended size matches current size<br>  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-prometheus(&amp;LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: keda-sample,},MatchExpressions:[]LabelSelectorRequirement{},})<br>  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range<br>Events:<br>  Type    Reason             Age   From                       Message<br>  ----    ------             ----  ----                       -------<br>  Normal  SuccessfulRescale  25m   horizontal-pod-autoscaler  New size: 2; reason: external metric s0-prometheus(&amp;LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: keda-sample,},MatchExpressions:[]LabelSelectorRequirement{},}) above target<br>  Normal  SuccessfulRescale  48s   horizontal-pod-autoscaler  New size: 1; reason: All metrics below target</pre><p><strong>Congratulations!!! You have successfully scaled your Kubernetes Deployment based on the external metric from Prometheus using KEDA.</strong></p><p><strong>Thank You!!!</strong></p><h3>References:</h3><ol><li><a href="https://keda.sh/docs/2.12/concepts/scaling-deployments/">https://keda.sh/docs/2.12/concepts/scaling-deployments/</a></li><li><a href="https://keda.sh/docs/2.12/scalers/prometheus/">https://keda.sh/docs/2.12/scalers/prometheus/</a></li><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></li><li><a href="https://github.com/nerdswords/yet-another-cloudwatch-exporter/tree/master/docs">https://github.com/nerdswords/yet-another-cloudwatch-exporter/blob/master/docs/configuration.md</a></li><li><a href="https://github.com/nerdswords/helm-charts">https://github.com/nerdswords/helm-charts</a></li><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html</a></li><li><a href="https://github.com/prometheus-operator/kube-prometheus/issues/1392">https://github.com/prometheus-operator/kube-prometheus/issues/1392</a></li></ol><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bb334c0ccf0" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/keda-kubernetes-hpa-based-on-external-metrics-from-prometheus-3bb334c0ccf0">KEDA: Kubernetes HPA Based on External Metrics from Prometheus</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Backup: Automate Database Backup and Cross Region Copy]]></title>
            <link>https://medium.com/analytics-vidhya/aws-backup-automate-database-backup-and-cross-region-copy-31403eb236a4?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/31403eb236a4</guid>
            <category><![CDATA[aws-aurora]]></category>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws-backup]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 10:47:30 GMT</pubDate>
            <atom:updated>2026-03-18T07:43:52.633Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS Backup is a fully-managed service that makes it easy to centralize and automate data protection across AWS services.</p><h4>Some Key Features of AWS Backup:</h4><ol><li>Centralized backup management: — Using AWS Backup You can centrally manage backup of all supported resources</li><li>Policy-based backup: — You create backup policy known as <em>backup plan </em>to define your backup requirements</li><li>Tag-based backup policies: — You can filter your resource based on the tags applied on the resources</li><li>Lifecycle management policies: — You can configure lifecycle policies that automatically transition backups from warm storage to cold storage according to a schedule that you define</li><li>Cross-Region backup: — Using AWS Backup, you can copy backups to multiple different AWS Regions on demand or automatically as part of a scheduled backup plan</li><li>Incremental backups: — AWS Backup efficiently stores your periodic backups incrementally. The first backup of an AWS resource backs up a full copy of your data. For each successive incremental backup, only the changes to your AWS resources are backed up</li><li>Secure your data in backup vaults: — The content of each AWS Backup backup is immutable, meaning that no one can alter that content. AWS Backup further secures your backups in backup vaults, which separates them safely from their source instances</li></ol><p>In this Blog Post I will explain how AWS Backup Service can help you to automate the backup of your database clusters / instances.</p><h3>Architecture Diagram:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p47_DHBJdJthjJxqXfp7wg.png" /><figcaption>AWS Backup: Automate RDS Snapshot</figcaption></figure><ol><li>AWS Backup Plan will run based on the schedule and it will take RDS Snapshot specified in Resource assignments</li><li>After the Job is complete it will store recovery point in Source Region’s Backup Vault</li><li>Once recovery point is stored in source region’s backup vault it will initiate a copy job and it will copy the recovery point in target region’s backup vault</li></ol><h3>Steps:</h3><ol><li>Go to AWS Backup Console and click on “<strong>Backup plans</strong>”</li><li>Click Create Backup plan</li><li>Start options</li></ol><ul><li>Select Build a new plan</li><li>Give name (e.g. my-db-backup-plan)</li></ul><p>4. Backup rule configuration</p><ul><li>Give name (e.g. my-db-backup-rule)</li><li>Backup vault -&gt; default</li><li>Backup frequency -&gt; daily (You can change this based on your requirement. You can also have custom cron expression)</li><li>Backup window <br>a. Select customize backup window<br>b. Enter your preferred Backup window start time<br>c. Start within -&gt; Select 1 hr<br>d. Complete within -&gt; leave default value</li><li>Transition to cold storage -&gt; Never (You can select based on your need)</li><li>Retention period -&gt; 3 days (You can select based on your need)</li><li>Copy to destination -&gt; Select your destination region (Optional, if you don’t select any region it will not copy your recovery point)<br>If you have selected destination region then Expand Advanced settings and select Transition to cold storage, Retention period based on your requirements. You can add more regions if required</li><li>Click on create plan</li></ul><p>5. Assign resources</p><ul><li>Give name to group resources</li><li>IAM role -&gt; Select default Role, It has all the required permissions</li><li>Resource selection<br>a. Select Include specific resource types (since we want to backup only Aurora / RDS Databases)<br>b. In Select specific resource types -&gt; Select Aurora (if you have AWS Aurora Cluster) or Select RDS (if you have RDS Instance). You can also select both the options<br>c. Once Resources are selected, based on it you can select if you want to include all Aurora / RDS databases OR you can also select specific Aurora / RDS databases<br>d. If you want to select resources based on “<strong>tags</strong>” then you can add your Key / Value in Refine selection using tags section</li><li>Click on assign resources</li></ul><p>Once first job is completed you can view the snapshot in AWS RDS Console under Snapshots -&gt; Backup service Tab</p><p><strong>Congratulations!!! 🥳 </strong>You have successfully configured AWS Backup to take backup of your Aurora / RDS databases.</p><h4>Important Points to Consider</h4><ol><li>If your database is encrypted using <strong>default KMS Key (aws/rds)</strong> than in the destination region(if configured) the snapshot will be encrypted with destination region’s <strong>default (aws/rds) KMS Key</strong></li><li>If your database is encrypted using <strong>custom KMS Key</strong> than in the destination region(if configured) the snapshot will be encrypted with destination region’s <strong>default (aws/rds) KMS Key</strong></li><li>AWS Backup doesn’t guarantee that Backup Job will start on the exact timing, depending on your “<strong>Start within</strong>” time configuration during rule configuration it will start the backup job. <br>Example: — If your backup frequency is at <strong>03:30 PM UTC</strong> every day and Start within is <strong>1 hour</strong> than backup job will start anytime between 03:30 PM UTC to 04:30 PM UTC</li><li>Once the recovery point(snapshot) is stored in source region’s vault there is no fixed start time of Copy Job to store recovery point(snapshot) in destination region’s vault</li><li>If by mistake recovery point from backup vault is deleted from one region it will be available in another region</li></ol><h4>Thank You!!</h4><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31403eb236a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/aws-backup-automate-database-backup-and-cross-region-copy-31403eb236a4">AWS Backup: Automate Database Backup and Cross Region Copy</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Autoscale GitLab CI on AWS ECS Fargate]]></title>
            <link>https://harshvijaythakkar.medium.com/autoscale-gitlab-ci-on-aws-ecs-fargate-b63cd0621827?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b63cd0621827</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[aws-ecs-fargate]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Thu, 05 Jan 2023 13:30:32 GMT</pubDate>
            <atom:updated>2026-03-18T07:44:17.179Z</atom:updated>
            <content:encoded><![CDATA[<h4>Autoscale your GitLab CI</h4><p><a href="https://medium.com/u/6e46453333d1">Ayushigupta</a> and me were trying to run Gitlab Runner to execute our CI Jobs, and we explored multiple options to run Gitlab Runner on AWS Cloud and found that running Gitlab Runner on ECS Fargate is optimal solution</p><p>Amazon ECS is a fully managed container orchestration service that makes it easy for you to deploy, manage, and scale containerized applications. Run and scale your container workloads across availability zones, in the cloud, and on-premises, without the complexity of managing a control plane or nodes.</p><p>In this blog post we are going to discuss how you can Autoscale your GitLab CI by running your CI jobs on AWS ECS Fargate. I will take an example of how you can use terraform to create AWS Infra when a commit is made to a branch. GitLab Runner will execute our job(s) from .gitlab-ci.yml file by running container on AWS ECS Fargate Cluster and once the work is done for particular job the container will be terminated.</p><p>AWS Fargate pricing is calculated based on the vCPU, memory, Operating Systems, CPU Architecture, and storage resources used from the time you start to download your container image until the Amazon ECS Task Pod terminates, rounded up to the nearest second.</p><h4>Architecture Diagram:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4iGJAxsvKwTd5y_ssW1eRg.png" /><figcaption><a href="https://docs.gitlab.com/runner/configuration/img/runner_fargate_driver_ssh.png">https://docs.gitlab.com/runner/configuration/img/runner_fargate_driver_ssh.png</a></figcaption></figure><p>In the Architecture Diagram you can see that Gitlab Runner along with the Fagate Driver (custom executor) is running on EC2 Instance (you can run it on Container as well). The SSH keys are automatically managed by the Fargate driver. The container must be able to accept keys from the SSH_PUBLIC_KEY environment variable.</p><p>The GitLab <a href="https://docs.gitlab.com/runner/executors/custom.html">custom executor</a> driver for <a href="https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/fargate">AWS Fargate</a> automatically launches a container on the Amazon Elastic Container Service (ECS) to execute each GitLab CI job.</p><h3>Step 1: Create IAM role for EC2 Instance</h3><ol><li>Go IAM Console and click on Create Role</li><li>Trusted entity type -&gt; <strong>EC2</strong> and Use case -&gt; <strong>EC2</strong> and click on Next</li><li>Permissions -&gt; <strong>AmazonECS_FullAccess</strong> and click on Next</li><li>Give a meaningful name and click on Create Role</li></ol><h3>Step 2: Create Security Group</h3><ol><li>Go to EC2 Console, click on Security groups and click on create new security Group</li><li>Give name and description</li><li>Select the <strong>VPC</strong> (This should be same as your VPC in which you will be creating EC2 Instance)</li><li>Click Add Inbound Rule and add SSH rule (You can allow access from anywhere but as a best practice you should restrict the access to particular IP or IP ranges or VPC CIDR)</li><li>Click on Create Security Group</li></ol><h3>Step 3: Create EC2 Instance for GitLab Runner and Fargate Driver</h3><ol><li>Go to EC2 Console and click on Launch Instance</li><li>Give Name and add additional tags if required</li><li>Select <strong>Amazon Linux 2</strong> AMI</li><li>Select <strong>t2.micro</strong> Instance Type</li><li>Select Your Key Pair or create new Key Pair</li><li>Edit Networking section and Select VPC, Subnet and Existing Security Group (created in step 2), Assign IAM Role (created in step 1)</li><li>Click Launch Instance</li></ol><h3>Step 4: Build Docker Image and push to ECR</h3><p>Docker Image should have GitLab Runner, which handles artifacts and caching along with our required dependency and packages.</p><ul><li>Our GitLab Repo Structure</li></ul><pre>terraform-test<br>|_ custum-fargate-executor<br>   |_ Dockerfile<br>   |_ docker-entrypoint.sh<br>|_ modules<br>   |_ EC2<br>   |_ RDS<br>   |_ S3<br>|_ .gitlab-ci.yml<br>|_ .gitignore<br>|_ provider.tf<br>|_ backend.tf<br>|_ s3.tf</pre><ul><li>Open file terraform-test/custum-fargate-executor/Dockerfile and add following content</li></ul><pre>FROM debian:buster<br><br># ---------------------------------------------------------------------<br># Install https://github.com/krallin/tini - a very small &#39;init&#39; process<br># that helps processing signalls sent to the container properly.<br># ---------------------------------------------------------------------<br>ARG TINI_VERSION=v0.19.0<br><br>RUN apt-get update &amp;&amp; \<br>    apt-get install -y curl &amp;&amp; \<br>    curl -Lo /usr/local/bin/tini https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-amd64 &amp;&amp; \<br>    chmod +x /usr/local/bin/tini<br><br># --------------------------------------------------------------------------<br># Install and configure sshd.<br># https://docs.docker.com/engine/examples/running_ssh_service for reference.<br># --------------------------------------------------------------------------<br>RUN apt-get install -y openssh-server &amp;&amp; \<br>    # Creating /run/sshd instead of /var/run/sshd, because in the Debian<br>    # image /var/run is a symlink to /run. Creating /var/run/sshd directory<br>    # as proposed in the Docker documentation linked above just doesn&#39;t<br>    # work.<br>    mkdir -p /run/sshd<br><br>EXPOSE 22<br><br># ----------------------------------------<br># Install GitLab CI required dependencies.<br># ----------------------------------------<br>ARG GITLAB_RUNNER_VERSION=v12.9.0<br><br>RUN curl -Lo /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/${GITLAB_RUNNER_VERSION}/binaries/gitlab-runner-linux-amd64 &amp;&amp; \<br>    chmod +x /usr/local/bin/gitlab-runner &amp;&amp; \<br>    # Test if the downloaded file was indeed a binary and not, for example,<br>    # an HTML page representing S3&#39;s internal server error message or something<br>    # like that.<br>    gitlab-runner --version<br><br>RUN apt-get install -y bash ca-certificates git git-lfs &amp;&amp; \<br>    git lfs install --skip-repo<br><br># ----------------------------------------<br># Install Terraform required dependencies.<br># ----------------------------------------<br><br>RUN apt update -y &amp;&amp; \<br>    apt install  software-properties-common gnupg2 curl -y &amp;&amp; \<br>    curl https://apt.releases.hashicorp.com/gpg | gpg --dearmor &gt; hashicorp.gpg &amp;&amp; \<br>    install -o root -g root -m 644 hashicorp.gpg /etc/apt/trusted.gpg.d/ &amp;&amp; \<br>    apt-add-repository &quot;deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com focal main&quot; &amp;&amp; \<br>    apt update -y &amp;&amp; \<br>    apt install terraform=1.3.6 -y &amp;&amp; \<br>    apt-get install awscli -y<br><br>RUN apt update -y &amp;&amp; \<br>    apt-get install binutils -y &amp;&amp; \ <br>    strings -help<br><br>RUN terraform --version   <br><br># -------------------------------------------------------------------------------------<br># Execute a startup script.<br># https://success.docker.com/article/use-a-script-to-initialize-stateful-container-data<br># for reference.<br># -------------------------------------------------------------------------------------<br>COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh<br><br>RUN chmod +x /usr/local/bin/docker-entrypoint.sh<br><br>ENTRYPOINT [&quot;tini&quot;, &quot;--&quot;, &quot;/usr/local/bin/docker-entrypoint.sh&quot;]</pre><p><strong>Note:-</strong> We are using (<a href="https://gitlab.com/tmaczukin-test-projects/fargate-driver-debian/-/tree/master">https://gitlab.com/tmaczukin-test-projects/fargate-driver-debian/-/tree/master</a>) as base dockerfile and added dependency to install terraform (you can modify dockerfile based on your requirements)</p><ul><li>Open file terraform-test/custum-fargate-executor/docker-entrypoint.sh and add following content</li></ul><pre>#!/bin/sh<br><br># Create a folder to store user&#39;s SSH keys if it does not exist.<br>USER_SSH_KEYS_FOLDER=~/.ssh<br>[ ! -d ${USER_SSH_KEYS_FOLDER} ] &amp;&amp; mkdir -p ${USER_SSH_KEYS_FOLDER}<br><br># Copy contents from the `SSH_PUBLIC_KEY` environment variable<br># to the `$USER_SSH_KEYS_FOLDER/authorized_keys` file.<br># The environment variable must be set when the container starts.<br>echo ${SSH_PUBLIC_KEY} &gt; ${USER_SSH_KEYS_FOLDER}/authorized_keys<br><br># Clear the `SSH_PUBLIC_KEY` environment variable.<br>unset SSH_PUBLIC_KEY<br><br># Start the SSH daemon<br>exec /usr/sbin/sshd -D</pre><p><strong>Note:-</strong> Do not do any modification in docker-entrypoint.sh file it might break the things.</p><ul><li>Build docker image using following command:</li></ul><pre>docker build -t &lt;name_of_docker_image&gt; &lt;path_to_Dockerfile&gt;</pre><ul><li>Push Image to ECR:</li></ul><ol><li>Create Repository on ECR</li><li>Click on Repository name and click on view push commands (follow the steps and push your docker image to ECR)</li><li>Click on Image tag and note down the URI (This URI will be used in Fargate Task Creation)</li></ol><h3>Step 5: Create Fargate Task</h3><ol><li>Go ECS Console and then click Task Definitions</li><li>Click <strong>Create new Task Definition</strong></li><li>Choose <strong>FARGATE</strong> and click <strong>Next step</strong></li><li>Give Name (Note: The name will be used in fargate.toml file)</li><li>Assign Task Role (This will be used by containers to call AWS API calls) — <strong>Mandatory</strong> <br>a. This role should have <strong>AmazonEC2ContainerServiceRole </strong>Policy</li><li>Operating system family -&gt; <strong>Linux</strong></li><li>Assign Execution Role (This will be used by ECS container agent to make AWS API calls)</li><li>Select values for <strong>Task memory (GB)</strong> and <strong>Task CPU (vCPU)</strong></li><li>Click <strong>Add container</strong>. Then:</li></ol><ul><li>Name it ci-coordinator(do not change this name), so the Fargate driver can inject the <strong>SSH_PUBLIC_KEY</strong> environment variable.</li><li>Define image (Created in Step 4, write URI of ECR)</li><li>Define port mapping for <strong>22/TCP</strong></li><li>Click <strong>Add</strong></li></ul><p>8. Click <strong>Create</strong></p><h3>Step 6: Sample Terraform Files</h3><p>Terraform will require permission to create AWS resources on our behalf. There are different ways in which you can provide credentials to terraform, I will be using assume_role to supply AWS Credentials to terraform.</p><ol><li>Create IAM role for terraform to assume and provision AWS Resources</li></ol><ul><li>Go to IAM console and click on Roles</li><li>Click on Create Role</li><li>Trusted entity type -&gt; <strong>AWS Account</strong> and, An AWS account -&gt; <strong>This Account</strong> (your_account_number will be displayed) and click Next</li><li>Add permissions -&gt; <strong>AdministratorAccess </strong>Policy (We need to give Admin access to terraform, you can select your own custom policy if required)</li><li>Give name to role (Example: terraform-assume-role)</li><li>Click on Create Role</li></ul><p>Once the role is created any role/user in your AWS account can assume this role and has admin permissions, if you want to limit the access then update the role <strong>trust relationships</strong> as per your need. (For Example you can allow only ECS Task Role to assume this role by updating Principal to: arn:aws:iam::&lt;your_aws_account_id&gt;:role/ecsTaskExecutionRole)</p><p>2. Open file terraform-test/backend.tf and add following content</p><pre>terraform {<br>  required_version = &quot;&gt;= 0.13&quot;<br><br>  backend &quot;s3&quot; {<br>    region         = &quot;&lt;s3_bucket_region&gt;&quot;<br>    encrypt        = true<br>    bucket         = &quot;&lt;s3_bucket_name&gt;&quot;<br>    key            = &quot;&lt;terraform_state_file_name&gt;&quot;<br>    role_arn       = &quot;&lt;role_arn_created_in_step_6.1&gt;&quot;<br>  }<br>}</pre><p>3. Open file terraform-test/provider.tf and add following content</p><pre>terraform {<br>  required_version = &quot;&gt;= 1.0.0&quot;<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>    region       = &quot;&lt;your_aws_region&gt;&quot;<br>    assume_role {<br>        role_arn     = &quot;&lt;role_arn_created_in_step_6.1&gt;&quot;<br>        session_name = &quot;Terraform_session&quot;<br>    }<br>}</pre><p>4. Open file terraform-test/s3.tf and add following content</p><pre>data &quot;aws_caller_identity&quot; &quot;current&quot; {}<br><br>resource &quot;aws_s3_bucket&quot; &quot;test_bucket&quot; {<br>  bucket = &quot;&lt;unique_bucket_name&gt;&quot;<br>  tags = {<br>    Environment = &quot;Dev&quot;<br>  }<br>}</pre><h3>Step 7: Create ECS Cluster</h3><ol><li>Go to ECS Console</li><li>Click <strong>Create Cluster</strong></li><li>Choose <strong>Networking only</strong> type. Click <strong>Next step</strong></li><li>Give Name (Note: The name will be used infargate.toml)</li><li>Click <strong>Create</strong></li><li>Click <strong>View cluster</strong>. Click <strong>Update Cluster</strong> button</li><li>Next to Default capacity provider strategy, click <strong>Add another provider</strong> and choose FARGATE. Click <strong>Update</strong></li></ol><h3>Step 8: Configure EC2 Instance</h3><ol><li>SSH into EC2 Instance and become root user</li></ol><pre>ssh -i &lt;path_to_key_pair&gt; ec2-user@&lt;IP_address_of_EC2&gt;<br><br>sudo su -</pre><p>2. Run following command to install gitlab-runner</p><pre>sudo mkdir -p /opt/gitlab-runner/{metadata,builds,cache}<br><br>sudo curl -L &quot;https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh&quot; | sudo bash<br><br>sudo yum install gitlab-runner -y</pre><p>3. Go to your GitLab project’s <strong>Settings &gt; CI/CD</strong> and expand the <strong>Runners</strong> section. Under Set up a specific Runner manually, note the registration token</p><p>4. Register runner using following command</p><pre>sudo gitlab-runner register --url &lt;your_gitlab_url&gt; --registration-token &lt;your_registration_token&gt; --name fargate-test-runner --run-untagged --executor custom -n</pre><p>This command will create config.toml file and shown below is created by the registration command. Do not change it.</p><pre>concurrent = 1<br>check_interval = 0</pre><pre>[session_server]<br>  session_timeout = 1800</pre><pre>name = &quot;fargate-test-runner&quot;<br>url = &quot;&lt;your_gitlab_url&gt;&quot;<br>token = &quot;&lt;your_registration_token&gt;&quot;<br>executor = &quot;custom&quot;</pre><p>5. Run sudo vim /etc/gitlab-runner/config.toml and add the following content but do not modify pre written key/values by gitlab-runner:</p><pre>log_level = &quot;debug&quot;<br>concurrent = 1<br>check_interval = 0<br><br>[session_server]<br>  listen_address = &quot;[::]:8093&quot;<br>  session_timeout = 1800<br><br>[[runners]]<br>  name = &quot;fargate-test-runner&quot;<br>  url = &quot;&lt;your_gitlab_url&gt;&quot;<br>  token = &quot;&lt;your_registration_token&gt;&quot;<br>  token_obtained_at = &quot;&quot;<br>  token_expires_at = &quot;&quot;<br>  executor = &quot;custom&quot;<br>  builds_dir = &quot;/opt/gitlab-runner/builds&quot;<br>  cache_dir = &quot;/opt/gitlab-runner/cache&quot;<br>  clone_url = &quot;&lt;if_you_have_custom_gitlab_url_or_else_remove_clone_url_property&gt;&quot;<br>  [runners.custom_build_dir]<br>    enabled = true<br>  [runners.custom]<br>    config_exec = &quot;/opt/gitlab-runner/fargate&quot;<br>    config_args = [&quot;--config&quot;, &quot;/etc/gitlab-runner/fargate.toml&quot;, &quot;custom&quot;, &quot;config&quot;]<br>    prepare_exec = &quot;/opt/gitlab-runner/fargate&quot;<br>    prepare_args = [&quot;--config&quot;, &quot;/etc/gitlab-runner/fargate.toml&quot;, &quot;custom&quot;, &quot;prepare&quot;]<br>    run_exec = &quot;/opt/gitlab-runner/fargate&quot;<br>    run_args = [&quot;--config&quot;, &quot;/etc/gitlab-runner/fargate.toml&quot;, &quot;custom&quot;, &quot;run&quot;]<br>    cleanup_exec = &quot;/opt/gitlab-runner/fargate&quot;<br>    cleanup_args = [&quot;--config&quot;, &quot;/etc/gitlab-runner/fargate.toml&quot;, &quot;custom&quot;, &quot;cleanup&quot;]</pre><p><strong>Note:-</strong> In our case we have used clone_url (Overwrite the URL for the GitLab instance. Used only if the runner can’t connect to the GitLab URL.) property because our gitlab has different url for cloning the repo.</p><p>6. Run sudo vim /etc/gitlab-runner/fargate.toml and add the following content:</p><pre>LogLevel = &quot;info&quot;<br>LogFormat = &quot;text&quot;<br><br>[Fargate]<br>  Cluster = &quot;&lt;your_ECS_cluster_name&gt;&quot;<br>  Region = &quot;&lt;ECS_cluster_region&gt;&quot;<br>  Subnet = &quot;&lt;subnetId_in_which_you_want_to_place_fargate_task&gt;&quot;<br>  SecurityGroup = &quot;&lt;securityGroupId_which_you_want_to_attach_to_fargate_task&gt;&quot;<br>  TaskDefinition = &quot;&lt;task_definition_name&gt;:&lt;revision&gt;&quot;<br>  EnablePublicIP = false<br><br>[TaskMetadata]<br>  Directory = &quot;/opt/gitlab-runner/metadata&quot;<br><br>[SSH]<br>  Username = &quot;root&quot;<br>  Port = 22</pre><p><strong>Notes:</strong></p><ul><li>Give exact name of Cluster, TaskDefinition and revision number (If a revision number is not specified, the latest <strong>active</strong> revision is used)</li><li>You can use same subnet ID which you have used while creating EC2 Instance in Step 3</li><li>You can use same security group Id which you have used for EC2 (If you have enabled public SSH access or You can have SG rule to allow access on port 22 from VPC CIDR) or Create new security group and make sure that EC2 Security group is allowed on port 22 for new security group. (Fargate Driver running on EC2 will requires SSH access to your container i.e fargate task, so make sure to add correct SG Id).</li></ul><p>7. Install the Fargate driver:</p><pre>sudo curl -Lo /opt/gitlab-runner/fargate &quot;https://gitlab-runner-custom-fargate-downloads.s3.amazonaws.com/latest/fargate-linux-amd64&quot;<br>sudo chmod +x /opt/gitlab-runner/fargate</pre><p>8. Check status of gitlab-runner service</p><pre>sudo systemctl status gitlab-runner.service</pre><h3>Step 9: Test</h3><p>Your configuration should now be ready to use.</p><ol><li>In your GitLab project, create a simple .gitlab-ci.yml file and add following content:</li></ol><pre>before_script:<br>  - ps -ef<br>  - xargs --null --max-args=1 echo &lt; /proc/1/environ<br>  - cat /root/.profile<br>  - export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)<br>  - echo $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI<br>  - aws sts get-caller-identity<br>  - terraform --version<br>  - terraform init<br><br>stages:<br>  - validate<br>  - plan<br>  - apply<br><br>validate:<br>  stage: validate<br>  script:<br>    - terraform validate<br><br>plan:<br>  stage: plan<br>  script:<br>    - terraform plan -out &quot;planfile&quot;<br>  dependencies:<br>    - validate<br>  artifacts:<br>    paths:<br>      - planfile<br><br>apply:<br>  stage: apply<br>  script:<br>    - terraform apply -input=false &quot;planfile&quot;<br>  dependencies:<br>    - plan<br>  </pre><p><strong>Note:- </strong>export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)<strong> </strong>Command written in before_script section is very important. Do not remove it.</p><p>When a container starts in <strong>ECS Fargate</strong> only process running with <strong>PID 1</strong> has permissions to make AWS API calls, in our case the docker-entrypoint.sh will be executed with <strong>PID 1</strong>(you can see in output of ps -ef command) and as per the architecture diagram you can see that <strong>Fargate Custom Executor driver</strong> will SSH into container first (so its PID will be different, because it will bestarting new shell) and then it will send commands to container hence, we need to export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable from PID 1 so that every process running in container can make AWS API calls from container.</p><p>As mentioned above <strong>GitLab Runner </strong>will create container for every job(stage) we have added few lines in before_script section of gitlab-ci.yml file. So when a job will be executed it will execute commands written in before_script first and then it will execute commands written in each job(stages)</p><h3>Congratulations!!! 🥳</h3><p>You have successfully configured your CI pipeline to Autoscale and it will run CI Jobs on AWS ECS Fargate Cluster.</p><h3>Co-Authors:</h3><ol><li><a href="https://medium.com/u/6e46453333d1">Ayushigupta</a></li><li><a href="https://medium.com/u/bcd900396e5b">Harshvijaythakkar</a></li></ol><h3>Reference:</h3><ol><li><a href="https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/">https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/</a></li><li><a href="https://gitlab.com/tmaczukin-test-projects/fargate-driver-debian/-/tree/master">https://gitlab.com/tmaczukin-test-projects/fargate-driver-debian/-/tree/master</a></li><li><a href="https://docs.gitlab.com/runner/executors/custom.html">https://docs.gitlab.com/runner/executors/custom.html</a></li><li><a href="https://docs.gitlab.com/runner/configuration/advanced-configuration.html">https://docs.gitlab.com/runner/configuration/advanced-configuration.html</a></li><li><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html">https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html</a></li><li><a href="https://stackoverflow.com/questions/57078607/ecs-fargate-task-not-applying-role">https://stackoverflow.com/questions/57078607/ecs-fargate-task-not-applying-role</a></li></ol><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b63cd0621827" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seamless Major Version Upgrade of Aurora PostgreSQL RDS]]></title>
            <link>https://awstip.com/seamless-major-version-upgrade-of-aurora-postgresql-rds-6061efd00d3a?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6061efd00d3a</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[database-upgrade]]></category>
            <category><![CDATA[aurora-postgresql]]></category>
            <category><![CDATA[aws-dms]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Sat, 31 Dec 2022 07:29:57 GMT</pubDate>
            <atom:updated>2026-03-18T07:44:43.460Z</atom:updated>
            <content:encoded><![CDATA[<h4>Upgrade from version 9.x to 12.x using Bucardo</h4><p>We are using Aurora PostgreSQL RDS clusters as our Database for our Applications. We were using a 9.x version of Aurora PostgreSQL RDS clusters for some of our services. We have successfully and Seamlessly upgraded our clusters from 9.x to 12.x version with Minimal downtime. In this blog post I will try to explain how we have achieved this goal.</p><p>There are different ways in which you can upgrade your DB Cluster Version</p><ol><li><strong>In-Place Upgrade: —</strong> In-Place Upgrade time totally depends on the size of the database, if your database size is small you can use In-Place Upgrade. Remember one thing that during In-Place Upgrade you can not use your database cluster, your application can not connect to your database</li><li><strong>Use some third party tool: —</strong> We have used Bucardo and I will explain this in the blog</li><li>AWS Database Migration Service</li></ol><p>One of the major disadvantages of the 9.x version is that it doesn’t support Logical Replication and because of this you can not create replication slots and migrate your data, so to solve this problem we have used the Bucardo tool to migrate our data from source to target cluster. Bucardo uses a Trigger Based Approach to capture the changes on the source database and then migrates it to the target database. There are some limitations of bucardo as well, I will also discuss this in our blog and how we have used an alternate approach to migrate the data that also I will explain.</p><p>We have upgraded around 20 databases in single downtime window and we have used Bucardo for 90% of our databases, for 10% of special cases we have used either pg_dump and pg_restore because of small database size or we have used AWS Database Migration Service + Upsert Query (query to perform insert + update together based on conflict of primary key).</p><p>Let me tell you some of the characteristics across our databases for different tables:</p><ol><li>Records not too frequently updated/inserted</li><li>Table is Update heavy (more updates then inserts)</li><li>Table with the huge size (in Tb’s) and records inserted/modified</li><li>Tables can have LOB columns</li></ol><h3><strong>Architecture Diagram</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*YAySXtfzHblQ9NxeUXKWrA.png" /><figcaption>Major Version Upgrade Architecture Diagram</figcaption></figure><p>In the above diagram you can see that we have a source database cluster with a 9.x version where the application is reading / writing the data. We have created a new cluster with our target version as 12.x and we have migrated data from source cluster to target cluster using Bucaro.</p><p>The Bucardo process runs on an EC2 Instance, which is responsible for reading the data from the source and migrating it into the target cluster.</p><h3><strong>Prerequisite</strong></h3><ol><li>Source Cluster</li><li>Target Cluster with new Version</li><li>All the tables must have Primary Key</li><li>Schema is already present in Target cluster</li><li>Ubuntu OS EC2 Instances for Bucardo</li></ol><p>We collected some statistics to plan the Number of EC2 required and we have divided 1 database into multiple Bucardo Pipelines, responsible for migrating the tables based on the statistics we collected.</p><h3><strong>Let’s Get Started</strong></h3><h4>Get the total size of the database</h4><pre>select datname, pg_size_pretty(pg_database_size(datname)) <br>from pg_database order by pg_database_size(datname) desc limit 10;</pre><p>Based on the output of the above query if database size is in a few MB’s We have used <strong>pg_dump</strong> and <strong>pg_restore</strong> to migrate the tables during the downtime window. We had written bash scripts to take a dump and restore so that it saves time and chances of human error.</p><p>This approach doesn’t require table schema to be present in the target.</p><h4>Get the size of each tables</h4><pre>SELECT table_name,<br>pg_size_pretty(pg_relation_size(quote_ident(table_name))),<br>FROM<br>information_schema.tables<br>WHERE<br>table_schema = &#39;public&#39;<br>ORDER BY<br>pg_total_relation_size(quote_ident(table_name)) DESC;</pre><ul><li>Example Output</li></ul><pre>table_name             | pg_size_pretty |<br>----------------------------------------+---------------------------<br>table1                 | 476  GB        |       <br>table2                 | 28   GB        |          <br>table3                 | 8551 MB        |           <br>table4                 | 8649 MB        |           <br>table5                 | 1627 MB        |           <br>table6                 | 1207 MB        |</pre><p>Now assume that “<strong>table2”</strong> and “<strong>table4”</strong> are update heavy tables and based on the above output we have decided to create 4 EC2 Instances and we have used “<strong>c5.2xlarge”</strong> instance type</p><ol><li>EC2 Instance1 — Bucardo Pipeline only for table1</li><li>EC2 Instance2 — Bucardo Pipeline only for table2</li><li>EC2 Instance3 — Bucardo Pipeline for table3, table5 and table6</li><li>EC2 Instance4 — Bucardo Pipeline only for table4</li></ol><blockquote><strong>Note: </strong>— The reason for dividing it into 4 EC2 Instances is that we are aware that the “<strong>table2</strong>” and “<strong>table4</strong>” are update heavy tables and “<strong>table1</strong>” has ~500 Gb of size and if something goes wrong during the migration it will not affect the migration of other tables.</blockquote><p>Before starting the bucardo process we have updated the “<strong>shared_buffers”</strong> parameter value because we have observed that during the migration of update heavy tables or tables with a large size the default value doesn’t work and it throws an error during the migration.</p><p>In a similar approach we have planned Migration of the other tables.</p><p>Now let’s talk about how we have planned the migration of tables which has millions of records. We have used “<strong>AWS Database Migration Service”</strong> to migrate the data and to capture “<strong>inserts/updates”</strong> from the source database. We have used a column<strong> which has a timestamp datatype along with an id column</strong> and we have used UpSert Query to do the changes in the target database.</p><p>It is very important to plan how many DMS Tasks will be required, it totally depends on the number of records in the table. In our database we have <strong>~705 million records</strong> and based on this number we created 5 DMS Tasks to migrate this table. Each Task has “<strong>49 parallel”</strong> threads. We have used features provided by DMS such as <strong>“Parallel-load”</strong> and <strong>“filters”</strong> to divide tables into multiple DMS tasks and migrate the records in parallel.</p><blockquote><strong>Note:-</strong> It is recommended to have a larger instance type for source and target clusters, so that during migration our database doesn’t get affected because of the DMS task. Use of the above feature puts load on the source and target database clusters and it is recommended to have a large number of sessions.</blockquote><p>If your table contains LOB columns then Full Load Migration can take longer time, so we have identified the size of LOB column and while creating DMS Tasks we set <strong>‘InlineLobMaxSize’: &lt;max_lenght_of_lob_column&gt;</strong> , to get this value run the following query</p><pre>select max((octet_length(&lt;column_name&gt;::varcher))/(1024.0)) as &quot;Size in KB&quot; from &lt;table_name&gt;;</pre><p>We have an “<strong>id”</strong> column in our table and based on the id column we have divided table into 5 DMS Tasks. Now, we know that each DMS Task can have maximum “<strong>49 parallel threads”</strong> so, <strong>49 * 5 = 245</strong> threads. We have <strong>~705 million records,</strong> so “<strong>705 million/245 = 2877551”</strong> per thread. We have migrated <strong>~2.8 million records per thread </strong>using AWS DMS. In other words in DMS Task — 1, thread — 1 migrate records till “<strong>id &lt;= 2877551”</strong> for the DMS Task — 1, thread — 2 migrate records “<strong>id &gt; 2877551 and id &lt;= 5755102”</strong> (2877551 + 2877551) for DMS Task — 1, thread — 3 migrate records “id &gt; <strong>5755102 and id &lt; 8632653</strong>” (5755102 + 2877551) and so on. In DMS terms it is called a boundary. To get the value of all 245 boundaries use following python code</p><pre>x = 2877551<br>for i in range(245):<br>    print(x)<br>    x = x+2877551</pre><p>If you want to understand the calculations of these boundaries please go through this blog. In this I have tried to explain in detail how to divide the table and how to identify boundaries.</p><p><a href="https://harshvijaythakkar.medium.com/upgrade-migrate-aurora-postgresql-with-minimal-downtime-fd00cee5c086">Upgrade/Migrate Aurora PostgreSQL with Minimal Downtime</a></p><h4><strong>While Configuring DMS Tasks remember the following points:</strong></h4><ol><li>Migration type — Migrate existing data</li><li>Target table preparation mode — Do nothing</li><li>Include LOB columns in replication — Full LOB mode</li><li>Enable CloudWatch logs (Optional)</li><li>Maximum number of tables to load in parallel — 49</li><li>Commit rate during full load — 50000</li><li>Since I have 5 Tasks:<br>a. For the Task — 1 use filter-operator “<strong>less than equal to</strong>” and write &lt;49th id&gt; <br>b. For the Task — 2 use filter-operator “<strong>between</strong>” and <strong>start value</strong> will be &lt;49th id&gt; + 1 and <strong>end value</strong> will be &lt;98th id&gt;<br>c. For the Task — 3 use filter-operator “<strong>between</strong>” and <strong>start value</strong> will be &lt;98th id&gt; + 1 and <strong>end value</strong> will be &lt;147th id&gt;<br>d. For the Task — 4 use filter-operator “<strong>between</strong>” and <strong>start value</strong> will be &lt;147th id&gt; + 1 and <strong>end value</strong> will be &lt;196th id&gt;<br>e. For the Task — 5 use filter-operator “<strong>between</strong>” and <strong>start value</strong> will be &lt;196th id&gt; + 1 and <strong>end value</strong> will be &lt;245th id&gt;</li><li>“InlineLobMaxSize”: &lt;max_lenght_of_lob_column&gt; (Parameter setting is available in JSON mode only and only available if you have enabled full LOB mode)</li><li>Use <strong>Parallel-load </strong>feature of DMS to load data in parallel</li><li>Source and Target database cluster size should be large</li></ol><p>You can refer above blog to understand how to write Task definitions baed on the above calculations, depending on the number of Tasks your settings will be different for each Tasks.</p><p>Now, once we have prepared the DMS Tasks and stated it we need to note down the “<strong>Start Time of the DMS Task”</strong> which will help us to capture insert/updates in the source database.</p><p>For Example we started DMS Tasks at “<strong>8 PM IST”</strong> on the day of downtime and let’s say above 5 Tasks takes 4 Hrs to complete (time after doing test in lower environment) then whatever transactions happens after starting DMS Tasks to capture those transactions we used DMS Start Time.</p><h4>Capture Delta Data from Source and apply it on Target database in Downtime Window</h4><p>During the downtime we run a few queries to capture data from the source and apply it to the target.</p><ul><li><strong>Run the following query on source database to add new/updated transactions into new table</strong></li></ul><pre>create table table_delta as select * from table where modified_on &gt;= [DMS start time - 05:30];</pre><p>We have used DMS start time — 05:30 because we noted DMS task start time in <strong>IST timezone</strong> and in our tables the values are stored in <strong>UTC timezone</strong></p><ul><li><strong>Export delta table data from source</strong></li></ul><pre># Export using pg_dump<br>pg_dump -h $SOURCE_HOSTNAME -U $USERNAME -d $DATABASE -t table_delta &gt; table_delta.sql<br><br># Export using CSV<br>psql -h $SOURCE_HOSTNAME -U $USERNAME $DATABASE -c &quot;\copy table_delta to &#39;~/table_delta.csv&#39; DELIMITER &#39;,&#39; CSV HEADER;&quot;</pre><p>You can use any one query and run it on EC2 Instance to get the data from source and save data on EC2</p><ul><li><strong>Import delta table data into target</strong></li></ul><pre># Import command if you have exported using pg_dump<br>psql -h $TARGET_HOSTNAME -U $USERNAME -d $DATABASE &lt; table_delta.sql</pre><p>Run above command on EC2</p><pre># Import commands if you have exported using CSV<br>1. Create Table in target database cluster (Run this on database to create empty table)<br>somedb =&gt; create table table_delta as select * from &lt;table_name&gt; where 1=2;<br><br>2. Run Import command (Run this on EC2)<br>psql -h $TARGET_HOSTNAME -U $USERNAME $DATABASE -c &quot;\copy table_delta from &#39;~/table_delta.csv&#39; DELIMITER &#39;,&#39; CSV HEADER;&quot;</pre><p>In above methods we are storing exported data (delta data) from source table into a temp table on target database.</p><ul><li><strong>Run UpSert query</strong></li></ul><pre>psql -h $TARGET_HOSTNAME -U $USERNAME $DATABASE &lt;&lt; EOF<br>insert into &lt;table_name&gt;<br>select * from table_delta<br>on conflict(&lt;primary_key_column&gt;)<br>do<br>update set col1=EXCLUDED.col1<br>, col2=EXCLUDED.col2<br>, col3=EXCLUDED.col3<br>, col4=EXCLUDED.col4<br>, col5=EXCLUDED.col5<br>, col6=EXCLUDED.col6<br>, col7=EXCLUDED.col7;<br>EOF</pre><p>Using above command we are selecting records from temp table (table_delta) inserting it into actual table and on conflict of primary_key we are updating records in actual table.</p><p><strong>Note:-</strong> While Preparing UpSert Query don’t include “<strong>primary_key_column”</strong> in update statement.</p><p>So, In this way after learning from multiple failures in lower environments we have Seamlessly and Successfully Completed Major Version Upgrade of Aurora PostgreSQL RDS clusters from 9.x to 12.x version.</p><p>I would also like to thank <strong>Kasi Dogga, Soumyadip Roy ,Yogish Kutkunje Pai and Prashant Agarwal </strong>for continuous help and support towards achieving this goal.</p><h4>Thank You!</h4><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6061efd00d3a" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/seamless-major-version-upgrade-of-aurora-postgresql-rds-6061efd00d3a">Seamless Major Version Upgrade of Aurora PostgreSQL RDS</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Data Archival and Purging: PostgreSQL]]></title>
            <link>https://awstip.com/automate-data-archival-and-purging-postgresql-f4dd08c6fb6f?source=rss-bcd900396e5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4dd08c6fb6f</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Harshvijaythakkar]]></dc:creator>
            <pubDate>Mon, 12 Dec 2022 14:29:20 GMT</pubDate>
            <atom:updated>2026-03-18T07:45:24.733Z</atom:updated>
            <content:encoded><![CDATA[<h4>Serverless Solution for Data Archiving and Purging</h4><h3>Purge Process</h3><p>Purging is the process of freeing up space in the database or of deleting unused/obsolete data that is not required by the system. The purge<br>process can be based on the age of the data or the type of data.</p><h3>Archive Process</h3><p>Archiving is the process of backing up the historical/obsolete data that will be deleted during the purge process. During the archive process, data<br>will be moved from the main transactional table.</p><h3>Why do we need purge/archive processes?</h3><ul><li><strong>Quality</strong></li></ul><ol><li>Keeps relevant data of the business in an active database</li><li>Allows for access to historical records that must be kept</li><li>Destroys data that is no longer needed</li></ol><ul><li><strong>Time</strong></li></ul><ol><li>Optimizes access to relevant operational data</li><li>Reduces the duration of housekeeping efforts. (E.g. backups, re-indexing, resizing, etc.)</li></ol><ul><li><strong>Costs</strong></li></ul><ol><li>Reduces the Cluster scale significantly thus computing, memory needs reduces cascadingly the cost.</li><li>Reduces the need to invest in larger, faster resources to handle ever-growing data stores.</li><li>Allows for lower-cost long-term storage options</li></ol><h3>What is needed to implement the archive/purge process?</h3><p>There are 2 important tasks for the Archive/Purge process:</p><ol><li>Each candidate Table with a Criteria is the critical requirement of the archive/purge process. Once we identify the Table and a where<br>condition (Criteria can be either a date/timestamp or condition to get the range of records to be either purged or archived)</li><li>Reorganising the data after the Archive/Purge process.</li></ol><p>Once we understand the requirement, the Table to be archived/purged, and the Criteria, we need to schedule/set up scripts one-time that are<br>required to be executed as per the schedule for the archive and purge process. Also, a vacuum analyze function to reorganize the data and<br>compact tables by reclaiming the space.</p><h3>What are different ways to archive historical data?</h3><p>There are 2 ways by using which we can archive historical data and store it in an S3 bucket.</p><ol><li>Using <strong>aws_s3</strong> extension</li><li>Using AWS DMS</li></ol><p>In this blog post we will use first approach for Archival and Purging of data</p><h3>Architecture Diagram:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*V5HiEVTpSHQxxFTpXsS8Pg.png" /><figcaption>Automate Data Archival and Purging: PostgreSQ</figcaption></figure><ol><li>CloudWatch Event Rule will trigger lambda function on a particular interval (Daily / Weekly / Bi-Weekly / Monthly) as per business needs</li><li>Lambda Function will execute query on RDS and It will send the results into S3 Bucket</li><li>Once Query Execution is success and data is stored in S3 we can execute another query to purge the records from Database</li></ol><h3>Let’s Get Started:</h3><h4>Pre-requisite</h4><ol><li>EC2 Instance</li><li>RDS PostgreSQL Cluster / Instance</li><li>Empty Database should be present in RDS (If you have your own data then no need to create empty database)</li><li>EC2 Instance should be able to make connection with RDS</li><li>S3 Bucket</li><li>Python3.9</li></ol><h4>Verify</h4><ul><li>Run the following command to verify that RDS for PostgreSQL version supports exports to Amazon S3</li></ul><pre>aws rds describe-db-engine-versions --region &lt;your_region&gt; --engine postgres --engine-version &lt;your_postgres_engine_version&gt;| grep s3Export</pre><p>If the output includes the string &quot;s3Export&quot;, then the engine supports Amazon S3 exports. Otherwise, the engine doesn&#39;t support them.</p><h4>Load Sample Data (Optional)</h4><p>I will be using <a href="https://www.gov.uk/government/statistical-data-sets/banana-prices">Banana prices</a> data which is available on UK Gov Website.</p><p><strong>You can use your own data / sample data.</strong></p><ul><li>Connect to RDS using following command:</li></ul><pre>psql -h &lt;your_db_endpoint&gt; -U &lt;your_user_name&gt; -d &lt;your_database_name&gt;</pre><ul><li>Run following command to create table</li></ul><pre>create table price (origin varchar(500), date date, price numeric, units varchar(100));</pre><ul><li>Run following command to load downloaded sample data</li></ul><pre>\copy price from &#39;&lt;downloaded_dataset_file_complete_path&gt;&#39; with (format csv, header true, null &#39;&#39;, quote &#39;&quot;&#39;);</pre><ul><li>Run following command to verify the data</li></ul><pre>select * from price limit 10;</pre><h4>Install <strong>the “</strong><strong>aws_s3” extension</strong></h4><p>Make sure you are connected to your database using psql CLI client</p><ul><li>Run the following command to install the extension</li></ul><pre>CREATE EXTENSION aws_s3 CASCADE;</pre><ul><li>verify that the extension is installed</li></ul><pre>\dx</pre><h4>SetUp Access to Amazon S3 Bucket</h4><p>To export data to Amazon S3, give your PostgreSQL DB instance permission to access the Amazon S3 bucket that the files are to go in.</p><ul><li>Create IAM Policy</li></ul><pre>aws iam create-policy  --policy-name rds-s3-policy  --policy-document &#39;{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;S3:PutObject&quot;,<br>                &quot;S3:ListBucket&quot;,<br>                &quot;S3:GetObject&quot;,<br>                &quot;s3:AbortMultipartUpload&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::&lt;your_bucker_name&gt;&quot;,<br>                &quot;arn:aws:s3:::&lt;your_bucker_name&gt;/*&quot;<br>            ]<br>        }<br>    ]<br>}&#39;</pre><p>After you create the policy, note the Amazon Resource Name (ARN) of the policy. You need the ARN for a subsequent step when you attach the policy to an IAM role</p><ul><li>Create IAM Role</li></ul><pre>aws iam create-role  \<br>    --role-name rds-s3-role  \<br>    --assume-role-policy-document &#39;{<br>     &quot;Version&quot;: &quot;2012-10-17&quot;,<br>     &quot;Statement&quot;: [<br>  {<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Principal&quot;: {<br>      &quot;Service&quot;: &quot;rds.amazonaws.com&quot;<br>    },<br>    &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>    <br>  }<br>] <br>}&#39;</pre><ul><li>Attach the IAM policy that you created to the IAM role that you created</li></ul><pre>aws iam attach-role-policy  --policy-arn &lt;your_policy_arn&gt;  --role-name &lt;your_role_name&gt;</pre><ul><li>Add IAM Role to your PostgreSQL DB Cluster <br>If you have “<strong>Aurora PostgreSQL Cluster”</strong> then use below command</li></ul><pre>aws rds add-role-to-db-cluster \<br>   --db-cluster-identifier &lt;your_rds_db_cluster_name&gt; \<br>   --feature-name s3Export \<br>   --role-arn &lt;your_role_arn&gt;   \<br>   --region &lt;your_region&gt;</pre><ul><li>Add IAM Role to your PostgreSQL DB Instance <br>If you have “<strong>PostgreSQL Instance”</strong> then use below command</li></ul><pre>aws rds add-role-to-db-instance \<br>   --db-instance-identifier &lt;your_rds_db_cluster_name&gt; \<br>   --feature-name s3Export \<br>   --role-arn &lt;your_role_arn&gt;   \<br>   --region &lt;your_region&gt;</pre><h4>Create VPC Endpoint for S3</h4><ol><li>Go to VPC console and click on Endpoints</li><li>Click on Create Endpoint</li><li>Give Name to Endpoint and Service category -&gt; AWS services</li><li>Search for “S3” and Select <strong>Gateway</strong> Type Endpoint in Services</li><li>Select Your VPC and Route Tables</li><li>Click on Create Endpoint</li></ol><h4>Test the SetUp</h4><p>We will test our current setup by manually running few commands to Archive data from PostgreSQL to S3 Bucket, We will not delete any data from database because we are just testing the current setup.</p><p>In our sample database if you run the following query we will get oldest data stored in database</p><pre>select min(date) from price;</pre><p>You will find that the oldest record date is 1995–01–13</p><p>If you run the following query you will find that in our sample database we have hardly 3–5 records on daily basis</p><pre>banana_prices=&gt; select * from price order by date asc limit 20;<br>     origin     |    date    | price |       units<br>----------------+------------+-------+-------------------<br> windward_isles | 1995-01-13 |   0.6 | pound sterling/kg<br> guadeloupe     | 1995-01-13 |   0.6 | pound sterling/kg<br> surinam        | 1995-01-13 |  0.61 | pound sterling/kg<br> windward_isles | 1995-01-20 |  0.63 | pound sterling/kg<br> guatemala      | 1995-01-20 |  0.64 | pound sterling/kg<br> guadeloupe     | 1995-01-20 |  0.61 | pound sterling/kg<br> surinam        | 1995-01-20 |  0.63 | pound sterling/kg<br> windward_isles | 1995-01-27 |  0.71 | pound sterling/kg<br> guadeloupe     | 1995-01-27 |  0.61 | pound sterling/kg<br> surinam        | 1995-01-27 |   0.7 | pound sterling/kg<br> windward_isles | 1995-02-03 |  0.64 | pound sterling/kg<br> guadeloupe     | 1995-02-03 |  0.64 | pound sterling/kg<br> surinam        | 1995-02-10 |  0.82 | pound sterling/kg<br> windward_isles | 1995-02-10 |  0.81 | pound sterling/kg<br> guatemala      | 1995-02-10 |  0.73 | pound sterling/kg<br> surinam        | 1995-02-17 |  0.85 | pound sterling/kg<br> windward_isles | 1995-02-17 |   0.8 | pound sterling/kg<br> guadeloupe     | 1995-02-17 |  0.84 | pound sterling/kg<br> windward_isles | 1995-02-24 |   0.8 | pound sterling/kg<br> guadeloupe     | 1995-02-24 |  0.74 | pound sterling/kg<br>(20 rows)</pre><blockquote><strong>Note:- </strong>It is important to analyse the frequency of number of records inserted on daily basis, it will help you to decide how frequently you should run Automation to Archive the data and accordingly we need to prepare database query.</blockquote><ol><li>Create S3 URI</li></ol><p>The file path identifies where the export is stored in the Amazon S3 bucket. The file path consists of the following:</p><ul><li>An optional path prefix that identifies a virtual folder path.</li><li>A file prefix that identifies one or more files to be stored. Larger exports are stored in multiple files, each with a maximum size of approximately 6 GB. The additional file names have the same file prefix but with _part<em>XX</em> appended. The <em>XX</em> represents 2, then 3, and so on.</li><li>Syntax is as follows:</li></ul><pre>SELECT aws_commons.create_s3_uri(<br>   &#39;&lt;your_bucket_name&gt;&#39;,<br>   &#39;&lt;your_file_path&gt;&#39;,<br>   &#39;&lt;your_region&gt;&#39;<br>) AS s3_uri \gset</pre><ul><li>Run following query to create s3_uri (file path)</li></ul><pre>SELECT aws_commons.create_s3_uri(<br>   &#39;&lt;your_bucket_name&gt;&#39;,<br>   &#39;&lt;your_db_cluster_name&gt;/&lt;your_database_name&gt;/&lt;your_table_name&gt;/1995/01/year_1995_month_01.csv&#39;,<br>   &#39;&lt;your_region&gt;&#39;<br>) AS s3_uri \gset</pre><p><strong>Notes:-</strong></p><ul><li>2nd Argument in create_s3_uri completely depends on how would you like to store your data in S3 bucket. My recommended way is &lt;your_db_cluster_name&gt;/&lt;your_database_name&gt;/&lt;your_table_name&gt;/&lt;year&gt;/&lt;month&gt;/&lt;date&gt;.csv</li><li>I recommend to keep &lt;your_db_cluster_name&gt;/&lt;your_database_name&gt;/&lt;your_table_name&gt;/ as fixed prefix while storing data in S3 and remaining prefix depends on how frequently you are archiving data based on that you can add your prefix (Eg. daily / monthly / weekly / bi-weekly etc)</li><li>In my sample data the daily record is only 3–5 so I will Archive my data on monthly basis for my testing and hence I will store my data with prefix&lt;your_db_cluster_name&gt;/&lt;your_database_name&gt;/&lt;your_table_name&gt;/1995/01/ and file name will be year_1995_month_01.csv</li><li>Run following query to get all records of <strong>year 1995 and month 01</strong></li></ul><pre>select * from price where date::text like &#39;1995-01%&#39;;</pre><p>2. Export data from database to S3 Bucket in CSV format</p><ul><li>Run the following query to export the data</li></ul><pre>SELECT * from aws_s3.query_export_to_s3(&#39;select * from price where date::text like &#39;&#39;1995-01%&#39;&#39;&#39;, :&#39;s3_uri&#39;, options :=&#39;format csv&#39;);</pre><p><strong>Note:- </strong>The sting in condition should be surrounded with 2 single quotes Example: &#39;&#39;1995-01-01&#39;&#39; / &#39;&#39;my-test-string-condition&#39;&#39; / &#39;&#39;1995–01%&#39;&#39;</p><ul><li>Output will be as follows</li></ul><pre> rows_uploaded | files_uploaded | bytes_uploaded<br>---------------+----------------+----------------<br>            10 |              1 |            449<br>(1 row)</pre><p><strong>Congratulations!!! </strong>You have successfully Archived your data into S3 Bucket now you can safely delete data from database. You have backup of your data 😎</p><ul><li>Delete the data from database</li></ul><pre>delete from price where date::text like &#39;1995-01%&#39;;</pre><p><strong>Note:- </strong>I have used same “<strong>query condition”</strong> to delete the data from database which I wrote in aws_s3.query_export_to_s3 function, please make sure that you execute delete query carefully.</p><h3>Let’s Automate</h3><p>We will be writing lambda function to execute psql queries using psycopg2 python pacakge and that lambda will be automatically triggered by CloudWatch Events based on the frequency you want to archive the data</p><h4>Create VPC Endpoint for RDS</h4><p>Since our Postgres will be running in private subnet we need to create VPC Endpoint for RDS so that our Lambda function can connect to our DB Cluster / Instance</p><ol><li>Go to VPC console and click on Endpoints</li><li>Click on Create Endpoint</li><li>Give Name to Endpoint and Service category -&gt; AWS services</li><li>Search “RDS” and select “com.amazonaws.ap-south-1.rds”</li><li>Select your VPC</li><li>Select your subnets and Select Address Type</li><li>Select your Security Group (you can select default)</li><li>Click on Create Endpoint</li></ol><h4>Create Lambda Function</h4><ol><li>Go to AWS Lambda Console and Click on Function and Click on Create Function</li><li>Select Author from Scratch</li><li>Give name to your function (my-rds-function)</li><li>Select “<strong>python3.9” </strong>in runtime</li><li>Expand Avance Settings</li></ol><ul><li>Click on <strong>Enable VPC </strong>and select your VPC</li><li>Select Your Subnet</li><li>Select Your Security Group (you can select default)</li></ul><p>6. Click on Create Funtion</p><p><strong>Note:- </strong>If you have selected different Security group while creating V<strong>PC Endpoint for RDS</strong> then make sure you allow lambda function Security Group in VPC Endpoint Security Group</p><blockquote><strong>Important:- </strong>Update Your RDS Security Group and allow access for VPC Endpoint Security Group</blockquote><h4>Write Code and Deploy Lambda Function</h4><ul><li>Create directory</li></ul><pre>mkdir my-function<br>cd my-function</pre><ul><li>Create and activate virtual environment</li></ul><pre>pip3 install virtualenv<br><br>virtualenv -p python3.9 venv<br><br>source venv/bin/activate</pre><ul><li>Install required packages make sure your virtualenv is active</li></ul><pre>pip3 install boto3<br><br>pip3 install psycopg2-binary</pre><ul><li>Add Environment Variables in Lambda Function</li></ul><ol><li>Click on your Lambda Function and go to Configuration</li><li>Click on Environment Variables and click on Edit</li><li>Add following Environment Variables<br>bucket_name -&gt; your_bucket_name, <br>dbname -&gt; your_database_name,<br>dbpassword -&gt; your_database_password,<br>dbport -&gt; 5432,<br>dbuser -&gt; your_db_user,<br>rds_endpoint -&gt; your_rds_endpoint,<br>region -&gt; your_region</li><li>Click on Save</li></ol><ul><li>Open file lambda_function.py and add following content</li></ul><pre>import json<br>import boto3<br>import psycopg2<br>import psycopg2.extras<br>from datetime import date<br>from dateutil.relativedelta import relativedelta<br>import os<br>from botocore.exceptions import ClientError<br><br>dbuser = os.environ.get(&#39;dbuser&#39;)<br>port = os.environ.get(&#39;dbport&#39;)<br>dbpassword = os.environ.get(&#39;dbpassword&#39;)<br>dbname = os.environ.get(&#39;dbname&#39;)<br>rds_endpoint = os.environ.get(&#39;rds_endpoint&#39;)<br>bucket_name = os.environ.get(&#39;bucket_name&#39;)<br>region = os.environ.get(&#39;region&#39;)<br><br>db_client = boto3.client(&#39;rds&#39;, region_name=&quot;ap-south-1&quot;)<br><br>def make_connection(conn_str):<br>    conn = psycopg2.connect(conn_str)<br>    conn.autocommit = True<br>    return conn<br><br>def date_time_json_converter(o):<br>    if isinstance(o, datetime.datetime):<br>        return o.__str__()<br>    elif isinstance(o, datetime.timedelta):<br>        return o.__str__()<br><br>def get_query_result(cursor):<br>    all_result = cursor.fetchall()<br>    dict_result = []<br>    for row in all_result:<br>        dict_result.append(dict(row))<br>    # return json.dumps(dict_result, default = date_time_json_converter)<br>    print(dict_result)<br>    return dict_result<br><br><br>def lambda_handler(event, context):<br>    try:<br>        table_name = &quot;&lt;your_table_name&gt;&quot;<br>        conn_str = &quot;host={0} dbname={1} user={2} password={3} port={4}&quot;.format(<br>                    rds_endpoint, dbname, dbuser, dbpassword, port)<br>        cnx = make_connection(conn_str)<br>        cursor = cnx.cursor(cursor_factory=psycopg2.extras.DictCursor)<br><br>        # get date<br>        three_month=date.today()+relativedelta(months=-3)<br><br>        # create s3_prefix<br>        month = &quot;{0:0=2d}&quot;.format(three_month.month)<br>        s3_prefix = &quot;{0}/{1}/{2}/{3}/{4}/&quot;.format(&quot;&lt;your_db_cluster_name&gt;&quot;, dbname, table_name, three_month.year, month)<br>        print(s3_prefix)<br><br>        # Crate S3 URI<br>        day = &quot;{0:0=2d}&quot;.format(three_month.day)<br>        s3_uri = &quot;aws_commons.create_s3_uri(&#39;{0}&#39;, &#39;{1}{2}.csv&#39;, &#39;{3}&#39;)&quot;.format(bucket_name, s3_prefix, day, region)<br>        print(s3_uri)<br><br>        # Prepare DB Query<br>        query = &quot;&quot;&quot;<br>        SELECT * from aws_s3.query_export_to_s3(&#39;select * from price where date::text = &#39;&#39;{0}-{1}-{2}&#39;&#39;&#39;, {3}, options :=&#39;format csv&#39;);<br>        &quot;&quot;&quot;.format(three_month.year, month, day, s3_uri)<br>        print(query)<br><br>        print(&quot;Executing Query&quot;)<br>        cursor.execute(query)<br>        query_result_list = get_query_result(cursor)<br>        print(query_result_list)<br>        print(&quot;Query Completed&quot;)<br>        print(&quot;------------------------------------&quot;)<br>        cnx.close()<br>    except Exception as e:<br>        print(&quot;Error&quot;, e)</pre><p><strong>Notes:-</strong></p><ol><li>As example I have decided to keep <strong>last 3 months data</strong> and archive my data on daily basis so, In my lambda function I have written a query to archive data on daily basis</li><li>Since I have written query to Archive data daily I will trigger this lambda using CloudWatch events on daily basis</li><li>If you deploy your lambda function on let say 12-Dec-2022 then last 3 month date will be 12–09–2022 so, as per SQL query it will archive data of 12–09–2022.</li><li>If you have any <strong>historical data older then 3 months</strong> (12–09–2022) then it needs to archived manually or you can do some modifications in above code</li><li>I have not added a delete query to purge record from database, if required you can add delete query once CSV is uploaded to S3, It is always a good idea to perform few checks before running delete query</li></ol><ul><li>Create deployment package</li></ul><ol><li>Note the path of virtual environment</li></ol><pre>pip3 -V<br><br># Example output<br>/root/my-function/venv/lib/python3.9/site-packages/pip</pre><p>2. deactivate the virtual env</p><pre>deactivate</pre><p>3. Zip dependency from virtualenv</p><pre>cd venv/lib/python3.9/site-packages/<br><br>zip -r ../../../../my-deployment-package.zip .</pre><p>4. Add function code files to the root of your deployment package</p><pre>cd ../../../../<br><br>zip -g my-deployment-package.zip lambda_function.py</pre><p>5. Deploy your .zip file to the function</p><pre>aws lambda update-function-code --function-name &lt;your_lambda_function_name&gt; --zip-file fileb://my-deployment-package.zip --region &lt;your_region&gt;</pre><p>6. Test Lambda ( optional )</p><p>a. Click on your Lambda Function and click on “<strong>Test”</strong> tab</p><p>b. Give Event Name</p><p>c. Select Private in Event sharing settings</p><p>d. Click on Save</p><p>e. Click on Test</p><p>f. Go to Cloudwatch Logs and select log group created by lambda and You will see the output of lambda function</p><h4>Add Cloudwatch Event Trigger</h4><ol><li>Click on your Lambda Function</li><li>Click on Add Trigger</li><li>Select EventBridge (Cloudwatch Events) from dropdown</li><li>Click on Create new rule</li><li>Give Rule name and Description</li><li>Select Schedule Expression in Rule Type</li><li>Add cron(0 10 * * ? *) in Schedule expression (Run at 10:00 am (UTC) every day)</li><li>Click Add</li></ol><p><strong>Note:- </strong>I have added cron to trigger Lambda Function everyday because I have written SQL query to archive data daily, you need to add cron expression according to your query.</p><p><strong>Congratulations !!! </strong>You have archived your data into S3 Bucket 😊</p><h4>Thank You!</h4><h4>Reference:</h4><ol><li><a href="https://www.gov.uk/government/statistical-data-sets/banana-prices">https://www.gov.uk/government/statistical-data-sets/banana-prices</a></li><li><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postgresql-s3-export.html">https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postgresql-s3-export.html</a></li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/python-package.html">https://docs.aws.amazon.com/lambda/latest/dg/python-package.html</a></li><li><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html">https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html</a></li><li><a href="https://stackoverflow.com/questions/58616005/import-postgres-data-into-rds-using-s3-and-aws-s3">https://stackoverflow.com/questions/58616005/import-postgres-data-into-rds-using-s3-and-aws-s3</a></li></ol><h3>About the Author</h3><p>I’m <a href="https://topmate.io/thakharsh/"><strong>Harsh Thakkar</strong></a>, Results-driven Senior DevOps &amp; Site Reliability Engineer with 7+ years of experience in architecting, automating, and optimizing cloud-native infrastructure and CI/CD pipelines. Proven expertise in Kubernetes, AWS, and observability solutions, delivering cost savings of up to 20%, reducing deployment time by 40%, and ensuring high availability for mission-critical SaaS platforms. Skilled in designing scalable systems, implementing security best practices, and mentoring teams on modern DevOps methodologies. Passionate about driving innovation through automation, performance optimization, and cloud security.</p><p>If you’re looking for practical guidance, feel free to connect with me or book a session:</p><ul><li>Connect with me on <a href="http://www.linkedin.com/in/thakharsh">LinkedIn</a></li><li>Book <strong>1:1 mock interviews, career guidance or mentoring </strong>on <a href="https://topmate.io/thakharsh">Topmate</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4dd08c6fb6f" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/automate-data-archival-and-purging-postgresql-f4dd08c6fb6f">Automate Data Archival and Purging: PostgreSQL</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>